Page MenuHomec4science

No OneTemporary

File Metadata

Created
Mon, Jul 21, 07:55
This file is larger than 256 KB, so syntax highlighting was skipped.
diff --git a/.gitignore b/.gitignore
index 26bba45cc..20c9fa253 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,57 +1,59 @@
*.clisp.mem
*.cmucl.core
*.fas
*.fasl
+*.kdev4
*.kdevelop
*.kdevses
*.lib
*.pyc
*.sbcl.core
*.sse2f
*.x86f
*~
.coverage
+.kdev4/
.noseids
.project
.pydevproject
.settings
.version
Invenio.egg-info
Makefile
Makefile.in
TAGS
aclocal.m4
autom4te.cache
bower_components/
build
compile
compile
config.cache
config.guess
config.log
config.nice
config.status
config.status.lineno
config.sub
configure
configure.lineno
dist
docs/_build
docs/db
docs/static
dump.rdb
install-sh
instance/
invenio-autotools.conf
invenio/base/translations/*/LC_MESSAGES/messages.mo
missing
node_modules/
org.eclipse.core.resources.prefs
po/*.gmo
po/*.mo
po/*.sed
po/POTFILES
po/POTFILES-py
po/POTFILES-webdoc
po/stamp-po
tags
diff --git a/AUTHORS b/AUTHORS
index 7d35016ca..1c49b7715 100644
--- a/AUTHORS
+++ b/AUTHORS
@@ -1,387 +1,387 @@
Invenio AUTHORS
===============
Invenio is being co-developed by an international collaboration
comprising institutes such as CERN, CfA, DESY, EPFL, FNAL, SLAC.
The CERN development team currently consists of:
- Jean-Yves Le Meur <jean-yves.le.meur@cern.ch>
CERN Digital Library Services team leader.
- Tibor Simko <tibor.simko@cern.ch>
CERN Digital Library Technology team leader. Head Developer of
Invenio. General system architecture, release management.
WebSearch, BibIndex, BibSched, WebStat, WebStyle, WebSession,
WebHelp, and more.
- Jerome Caffaro <jerome.caffaro@cern.ch>
BibFormat, redesign and rewrite in Python. BibConvert
XML-oriented mode. OAI harvester improvements. Improvements to
BibEdit. WebDoc tool. WebJournal refactoring and rewrite.
WebComment rounds and threads. WebSubmit asynchronous upload
support. Improvements to numerous modules.
- Samuele Kaplun <samuele.kaplun@cern.ch>
Authentication and user preferences rewrite and improvements.
Firewall-like access control RBAC system. Fulltext file
management rewrite and upload feature. Intbitset Python C
extension for the indexer. Improvents to the task scheduler and
session hander. Improvements to numerous modules.
- Ludmila Marian <ludmila.marian@gmail.com>
Citerank family of ranking methods. Fixes to numerous modules.
- - Jaime Garcia Llopis <jaime.garcia.llopis@cern.ch>
- Improvements to the BibCirculation module.
-
- Flavio Costa <flavio.costa@cern.ch>
Contributions to the Italian translation.
- Jiri Kuncar <jiri.kuncar@cern.ch>
General system architecture of Invenio 2.0. Contributions to
the Czech translation.
- Esteban J. G. Gabancho <esteban.jose.garcia.gabancho@cern.ch>
Initial release of WebApiKey, enhancements for WebSubmit.
- Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
Initial release of Invenio Upgrader and jsonutils; patches for
- pluginutils,
+ pluginutils.
- Patrick Glauner <patrick.oliver.glauner@cern.ch>
Cleanup of SQL queries for several modules.
- Raquel Jimenez Encinar <raquel.jimenez.encinar@cern.ch>
Errorlib refactoring, improvements to search UI, discussion tab,
merged record redirect, adaptation to new web test framework.
- Grzegorz Szpura <grzegorz.szpura@cern.ch>
Better browsing of fuzzy indexes.
- Thorsten Schwander <thorsten.schwander@gmail.com>
Improvements to dbdump.
- Jan Aage Lavik <jan.age.lavik@cern.ch>
Improvements to BibMatch with remote matching capabilities,
improvements to plot extractor, improvements to harvesting and
other small fixes.
- - Piotr Praczyk <piotr.praczyk@gmail.com>
- OAI harvesting from arXiv. Test harvesting interface, for
- OAIHarvest. Record comparison library functions, for BibRecord.
- Numerous improvements to BibEdit, e.g. holding pen, copy/paste,
- undo/redo.
-
- Samuele Carli <samuele.carli@cern.ch>
Initial implementation of BibAuthorID module, with Henning
Weiler. Fixes for basket export facility and adding external
items to baskets.
- Alessio Deiana <alessio.deiana@cern.ch>
Fix for BibFormat element initialisation. Improvements to data
cacher and cite summary.
- Wojciech Ziolek <wojciech.ziolek@cern.ch>
Fixes for OAI holding pen facility, Strftime improvements for
dateutils.
- Sebastian Witowski <sebastian.witowski@cern.ch>
Improvements to multi-record editor.
- Laura Rueda <laura.rueda@cern.ch>
Mechanize compatibility for Invenio Connector.
- Annette Holtkamp <annette.holtkamp@cern.ch>
Updates to "Howto MARC" guide.
- Jocelyne Jerdelet <jocelyne.jerdelet@cern.ch>
Updates to "Howto MARC" guide.
The EPFL development team currently consists of:
- Gregory Favre <gregory.favre@cern.ch>
Rewrite of WebBasket. WebMessage. Improvements to WebComment.
Other contributions and improvements.
The SLAC development team currently consists of:
- Mike Sullivan <sul@slac.stanford.edu>
Improvements to author pages.
- Eduardo Benavidez <eduardo.benavidez@gmail.com>
Improvements to BibCatalog.
The Harvard-Smithsonian Center for Astrophysics development team
currently consists of:
- Alberto Accomazzi <aaccomazzi@cfa.harvard.edu>
Team leader.
- - Giovanni Di Milia <gdimilia@cfa.harvard.edu>
- Recognition of /record/sysno URLs, ADS formatting.
-
- Jay Luker <lbjay@reallywow.com>
Improvements to the emergency notification sending facility.
- Roman Chyla <roman.chyla@cern.ch>
WSGI handler accepts UTF-8 strings.
Many former team members (mostly CERN staff and fellows, technical
students, diploma students, summer students) contributed to the
project since 2002. In an approximately decreasing chronological
order:
+ - Jaime Garcia Llopis <jaime.garcia.llopis@cern.ch>
+ Improvements to the BibCirculation module.
+
+ - Piotr Praczyk <piotr.praczyk@gmail.com>
+ OAI harvesting from arXiv. Test harvesting interface, for
+ BibHarvest. Record comparison library functions, for BibRecord.
+ Numerous improvements to BibEdit, e.g. holding pen, copy/paste,
+ undo/redo.
+
+ - Giovanni Di Milia <gdimilia@cfa.harvard.edu>
+ Recognition of /record/sysno URLs, ADS formatting.
+
- Daniel Stanculescu <daniel.stanculescu@cern.ch>
Improvements to Unicode treatment for textutils.
- Vasanth Venkatraman <vasanth.venkatraman@cern.ch>
Improvements to BibUpload version treatment, monotask and
sequence tasks for BibSched.
- Peter Halliday <phalliday@cornell.edu>
Configurable /record URL name space, improvements to dbquery.
- Chris Montarbaud <christiane.montarbaud@cern.ch>
Multimedia and photo management.
- Joe Blaylock <jrbl@slac.stanford.edu>
Rewrite of SPIRES query syntax parser, support for nested
parenthesis for WebSearch, fuzzy author name tokenizer,
enrichment of author pages with h-index.
- Benoit Thiell <bthiell@cfa.harvard.edu>
Fixes for BibRecord library, detailed record links, improvements
to code kwalitee in numerous modules. Improvements to
BibClassify.
- Nikola Yolov <nikola.yolov@cern.ch>
Improvements and refactoring of BibAuthorID, fixes for
WebAuthorProfile.
- Lewis Barnes <lewis.barnes@cern.ch>
Amendments for INSPIRE linking style.
- Olivier Canévet <olivier.canevet@cern.ch>
Fixes for WebComment report abuse facility.
- Belinda Chan <belinda.chan@cern.ch>
User documentation for personal features like alerts and baskets.
- Carmen Alvarez Perez <carmen.alvarez.perez@cern.ch>
Improvements to WebStat.
- Henning Weiler <henning.weiler@cern.ch>
Initial implementation of BibAuthorID module, with Samuele Carli.
- Juan Francisco Pereira Corral <juan.francisco.pereira.corral@cern.ch>
Fix taxonomy regression test, for BibKnowledge.
- Stamen Todorov Peev <stamen.peev@cern.ch>
Enrichment of Dublin Core XSL stylesheet.
- Jan Iwaszkiewicz <jan.iwaszkiewicz@cern.ch>
Full-text snippet feature for full-text search.
- Björn Oltmanns <bjoern.oltmanns@gmail.com>
Initial release of BibEncode, multi-node support for BibSched,
style refactoring for WebComment.
- Christopher Dickinson <christopher.dickinson@cern.ch>
Patch for auto-suggest facility.
- Christopher Hayward <christopher.james.hayward@cern.ch>
Improvements to the reference extraction tool.
- Travis Brooks <travis@slac.stanford.edu>
Support for SPIRES search syntax and other improvements.
- Juliusz Sompolski <julsomp@gmail.com>
Reimplementation of pdf2hocr2pdf.
- Jurga Girdzijauskaite <jurga.gird@gmail.com>
Contributions to the Lithuanian translation.
- Tony Ohls <tony.ohls@cern.ch>
Fixes for regexp treatment in BibConvert.
- Marko Niinimaki <manzikki@gmail.com>
Contributions to the BibRank citation module and WebSearch
summary output formats. Initial implementation of BibCatalog and
BibKnowledge.
- Mathieu Barras <mbarras@gmail.com>
Initial implementation of SWORD client application.
- Fabio Souto <fsoutomoure@gmail.com>
Initial implementation of the invenio config dumper/loader.
- Pablo Vázquez Caderno <pcaderno@cern.ch>
Prevent loop in collection trees, for WebSearch.
- Victor Engmark <victor.engmark@cern.ch>
Bash completion for inveniocfg, patche for dist-packages.
- Javier Martin <javier.martin.montull@cern.ch>
Moderation tools for WebComment, improvements to BibEdit, initial
implementation of the batch uploader.
- Nikolaos Kasioumis <nikolaos.kasioumis@cern.ch>
Hosted collections for WebSearch, rewrite of WebBasket UI,
improvements to WebAlert.
- Valkyrie Savage <vasavage@gmail.com>
Initial implementation of the plot extractor library.
- Miguel Martinez Pedreira <miguel.martinez.pedreira@cern.ch>
Tool for manipulated embedded metadata in full-text files.
- Jorge Aranda Sumarroca <jorge.aranda.sumarroca@cern.ch>
Support for FCKeditor-uploaded files for WebSubmit.
- Glenn Gard <glenng4@aol.com>
Implemented many unit, regression and web tests for WebAlert,
WebJournal, WebSubmit, WebComment, WebMessage, WebSession
modules.
- Christopher Parker <chris.parker.za@gmail.com>
Improvements to the submission approval workflow.
- Martin Vesely <martin.vesely@cern.ch>
OAIHarvest, OAIRepository, OAI daemon and admin
interface. BibConvert text-oriented mode. BibMatch.
- Tony Osborne <tony.osborne@cern.ch>
Improvements to the reference extractor.
- Radoslav Ivanov <radoslav.ivanov@cern.ch>
Contributions to the WebBasket module test suite. Support for
parentheses and SPIRES search syntax in WebSearch. Initial
implementation of the multi-record editor. Initial
implementation of BibExport.
- Joaquim Rodrigues Silvestre <joaquim.rodrigues.silvestre@cern.ch>
Initial implementation of the BibCirculation module to handle
physical item copies.
- Kyriakos Liakopoulos <kyriakos.liakopoulos@cern.ch>
Initial implementation of BibMerge. Improvements to BibEdit.
- Lars Christian Raae <lars.christian.raae@cern.ch>
Record locking, per-collection curating authentication, reverting
older record versions, for the BibEdit. Rewrite of BibEdit in
Ajax.
- Ruben Pollan <ruben.pollan@cern.ch>
Contributions to the WebStat module.
- Nicholas Robinson <nicholas.robinson@cern.ch>
WebSubmit. Reference extraction for the BibEdit module.
- Gabriel Hase <gabriel.hase@konnektiv.ch>
WebJournal module.
- Diane Berkovits <diane.berkovits@cern.ch>
Ranking by downloads, for the BibRank and WebSubmit
modules. Group management for WebSession.
- Joël Vogt <joel.vogt@unifr.ch>
Contributions to the BibClassify module.
- Marcus Johansson <marcus.johansson@cern.ch>
Contributions to the WebStat module.
- Jan Brice Krause <jan.brice.krause@cern.ch>
Original implementation of the fulltext file transfer mode for
BibUpload.
- Axel Voitier <axel.voitier@gmail.com>
Complex approval and refereeing subsystem, for WebSubmit.
- Alberto Pepe <alberto.pepe@cern.ch>
BibClassify, OAIHarvest Admin.
- Øyvind Østlund <oyvind.ostlund@cern.ch>
Sample BibTeX to MARCXML conversion, for BibConvert.
- Nikolay Dyankov <ndyankov@gmail.com>
XML-based BFX formatting engine, for BibFormat.
- Olivier Serres <olivier.serres@gmail.com>
External collections searching, for WebSearch.
- Eric Stahl <eric.stahl@utbm.fr>
Rewrite of BibUpload in Python.
- Frederic Gobry <frederic.gobry@gmail.com>
Contributions to the templating system, the URL handler, the
gettext infrastructure, the regression test suite infrastructure,
numerous patches for many modules.
- Krzysztof Jedrzejek <krzysztof.jedrzejek@gmail.com>
Improvements to ElmSubmit.
- Yohann Paris <paris.yohann@gmail.com>
BibEdit Admin.
- Paulo Cabral <cabralp@gmail.com>
WebComment, error library, design of collaborative features.
- Thomas Baron <thomas.baron@cern.ch>
WebSubmit and BibUpload. Improvements to BibSched.
- Maja Gracco <maja.gracco@cern.ch>
System librarian, MARC21 expertise.
- Tiberiu Dondera <tiberiu.dondera@pronet-consulting.com>
Patches for the WebSubmit engine and the admin interface.
Templatizing codebase.
- Anna Afshar <anna.afshar@novell.com>
Ranking by citations, for the BibRank module.
- Trond Aksel Myklebust <tamyk@online.no>
Ranking engine, the BibRank module. Stemming and stopwords for
the BibIndex module. Site access policies and external
authentication methods, for the WebAccess module and its clients.
Administration interfaces to WebSearch, BibIndex, BibRank, and
additions to WebAccess.
- Hector Sanchez <hector.sanchez.sanmartin@gmail.com>
Metadata output formatter, the BibFormat module. Session
management, for the WebSession module.
- Richard Owen <ro@tes.la>
Electronic mail submission system, the ElmSubmit module.
- Alexandra Silva <xana@correio.ci.uminho.pt>
Rewriting and enhancing BibRecord XML MARC and record handling
library, for the BibEdit module.
- Arturo Montejo Raez <amontejo@ujaen.es>
Automatic text classification and keyword indexing. (upcoming)
- Mikael Vik <mikael.vik@cern.ch>
Role-based access control engine and its admin interface,
the WebAccess module. Guest user sessions garbage collector,
for the WebSession module.
- Erik Simon <erik.simon@unine.ch>, Eric Simon <eric.simon@unine.ch>
Alert engine, for the WebAlert module.
- Roberta Faggian <roberta.faggian@cern.ch>
Rewrite of the alert and basket user interfaces, for the WebAlert
and the WebBasket modules.
- Julio Pernia Aznar <jpernia@altransdb.com>
Parts of user and session management, for the WebSession module.
- Franck Grenier <franckgrenier@wanadoo.fr>
Parts of web design and graphics, for the WebStyle module.
- Eduardo Margallo
Enhancements to the indexing engine, for the BibWords module.
Initial implementation of the task scheduler, for the BibSched
module.
- end of file -
diff --git a/Makefile.am b/Makefile.am
index bdf1d7785..3c8591742 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -1,689 +1,692 @@
## This file is part of Invenio.
## Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
confignicedir = $(sysconfdir)/build
confignice_SCRIPTS=config.nice
SUBDIRS = config
EXTRA_DIST = UNINSTALL THANKS RELEASE-NOTES configure-tests.py config.nice.in \
config.rpath
# current MathJax version and packages
# See also modules/miscutil/lib/htmlutils.py (get_mathjax_header)
MJV = 2.3
MATHJAX = http://invenio-software.org/download/mathjax/MathJax-v$(MJV).zip
# current CKeditor version
CKV = 3.6.6
CKEDITOR = ckeditor_$(CKV).zip
# current MediaElement.js version
MEV = master
MEDIAELEMENT = http://github.com/johndyer/mediaelement/zipball/$(MEV)
#for solrutils
INVENIO_JAVA_PATH = org/invenio_software/solr
solrdirname = apache-solr-3.1.0
solrdir = $(prefix)/lib/$(solrdirname)
solrutils_dir=$(CURDIR)/modules/miscutil/lib/solrutils
# for recline.js
RECLINEVER=master
CLASSPATH=.:${solrdir}/dist/solrj-lib/commons-io-1.4.jar:${solrdir}/dist/apache-solr-core-*jar:${solrdir}/contrib/jzlib-1.0.7.jar:${solrdir}/dist/apache-solr-solrj-3.1.0.jar:${solrdir}/dist/solrj-lib/slf4j-api-1.5.5.jar:${solrdir}/dist/*:${solrdir}/contrib/basic-lucene-libs/*:${solrdir}/contrib/analysis-extras/lucene-libs/*:${solrdir}/dist/solrj-lib/*
# git-version-get stuff:
BUILT_SOURCES = $(top_srcdir)/.version
$(top_srcdir)/.version:
echo $(VERSION) > $@-t && mv $@-t $@
dist-hook:
echo $(VERSION) > $(distdir)/.tarball-version
# Bootstrap version
BOOTSTRAPV = 3.0.2
# Hogan.js version
HOGANVER = 2.0.0
check-custom-templates:
$(PYTHON) $(top_srcdir)/modules/webstyle/lib/template.py --check-custom-templates $(top_srcdir)
kwalitee-check:
@$(PYTHON) $(top_srcdir)/scripts/kwalitee.py --stats $(top_srcdir)
kwalitee-check-errors-only:
@$(PYTHON) $(top_srcdir)/scripts/kwalitee.py --check-errors $(top_srcdir)
kwalitee-check-variables:
@$(PYTHON) $(top_srcdir)/scripts/kwalitee.py --check-variables $(top_srcdir)
kwalitee-check-indentation:
@$(PYTHON) $(top_srcdir)/scripts/kwalitee.py --check-indentation $(top_srcdir)
kwalitee-check-sql-queries:
@$(PYTHON) $(top_srcdir)/scripts/kwalitee.py --check-sql $(top_srcdir)
etags:
\rm -f $(top_srcdir)/TAGS
(cd $(top_srcdir) && find $(top_srcdir) -name "*.py" -print | xargs etags)
install-data-local:
for d in / /cache /cache/RTdata /log /tmp /tmp-shared /data /run /tmp-shared/bibencode/jobs/done /tmp-shared/bibedit-cache; do \
mkdir -p $(localstatedir)$$d ; \
done
@echo "************************************************************"
@echo "** Invenio software has been successfully installed! **"
@echo "** **"
@echo "** You may proceed to customizing your installation now. **"
@echo "************************************************************"
install-mathjax-plugin:
@echo "***********************************************************"
@echo "** Installing MathJax plugin, please wait... **"
@echo "***********************************************************"
rm -rf /tmp/invenio-mathjax-plugin
mkdir /tmp/invenio-mathjax-plugin
rm -fr ${prefix}/var/www/MathJax
mkdir -p ${prefix}/var/www/MathJax
(cd /tmp/invenio-mathjax-plugin && \
wget '$(MATHJAX)' -O mathjax.zip && \
unzip -q mathjax.zip && cd mathjax-MathJax-* && cp -r * \
${prefix}/var/www/MathJax)
rm -fr /tmp/invenio-mathjax-plugin
@echo "************************************************************"
@echo "** The MathJax plugin was successfully installed. **"
@echo "** Please do not forget to properly set the option **"
@echo "** CFG_WEBSEARCH_USE_MATHJAX_FOR_FORMATS and **"
@echo "** CFG_WEBSUBMIT_USE_MATHJAX in invenio.conf. **"
@echo "************************************************************"
uninstall-mathjax-plugin:
@rm -rvf ${prefix}/var/www/MathJax
@echo "***********************************************************"
@echo "** The MathJax plugin was successfully uninstalled. **"
@echo "***********************************************************"
install-jscalendar-plugin:
@echo "***********************************************************"
@echo "** Installing jsCalendar plugin, please wait... **"
@echo "***********************************************************"
rm -rf /tmp/invenio-jscalendar-plugin
mkdir /tmp/invenio-jscalendar-plugin
(cd /tmp/invenio-jscalendar-plugin && \
wget 'http://www.dynarch.com/static/jscalendar-1.0.zip' && \
unzip -u jscalendar-1.0.zip && \
mkdir -p ${prefix}/var/www/jsCalendar && \
cp jscalendar-1.0/img.gif ${prefix}/var/www/jsCalendar/jsCalendar.gif && \
cp jscalendar-1.0/calendar.js ${prefix}/var/www/jsCalendar/ && \
cp jscalendar-1.0/calendar-setup.js ${prefix}/var/www/jsCalendar/ && \
cp jscalendar-1.0/lang/calendar-en.js ${prefix}/var/www/jsCalendar/ && \
cp jscalendar-1.0/calendar-blue.css ${prefix}/var/www/jsCalendar/)
rm -fr /tmp/invenio-jscalendar-plugin
@echo "***********************************************************"
@echo "** The jsCalendar plugin was successfully installed. **"
@echo "***********************************************************"
uninstall-jscalendar-plugin:
@rm -rvf ${prefix}/var/www/jsCalendar
@echo "***********************************************************"
@echo "** The jsCalendar plugin was successfully uninstalled. **"
@echo "***********************************************************"
install-js-test-driver:
@echo "*******************************************************"
@echo "** Installing js-test-driver, please wait... **"
@echo "*******************************************************"
mkdir -p $(prefix)/lib/java/js-test-driver && \
cd $(prefix)/lib/java/js-test-driver && \
wget http://invenio-software.org/download/js-test-driver/JsTestDriver-1.3.5.jar -O JsTestDriver.jar
uninstall-js-test-driver:
@rm -rvf ${prefix}/lib/java/js-test-driver
@echo "*********************************************************"
@echo "** The js-test-driver was successfully uninstalled. **"
@echo "*********************************************************"
install-jquery-plugins:
@echo "***********************************************************"
@echo "** Installing various jQuery plugins, please wait... **"
@echo "***********************************************************"
mkdir -p ${prefix}/var/www/js
mkdir -p $(prefix)/var/www/css
mkdir -p $(prefix)/var/www/img
(cd ${prefix}/var/www/js && \
wget http://invenio-software.org/download/jquery/jquery-1.7.1.min.js && \
mv jquery-1.7.1.min.js jquery.min.js && \
wget http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.17/jquery-ui.min.js && \
wget http://invenio-software.org/download/jquery/jquery.jeditable.custom.min.js -O jquery.jeditable.mini.js && \
wget https://raw.github.com/malsup/form/master/jquery.form.js --no-check-certificate && \
wget http://jquery-multifile-plugin.googlecode.com/svn/trunk/jquery.MultiFile.pack.js && \
wget -O jquery.tablesorter.zip http://invenio-software.org/download/jquery/jquery.tablesorter.20111208.zip && \
wget http://invenio-software.org/download/jquery/uploadify-v2.1.4.zip -O uploadify.zip && \
wget http://www.datatables.net/download/build/jquery.dataTables.min.js && \
rm -rf /tmp/invenio-dt-bootstrap && \
mkdir /tmp/invenio-dt-bootstrap && \
(cd /tmp/invenio-dt-bootstrap && \
git clone 'https://github.com/DataTables/Plugins.git' && \
cp Plugins/integration/bootstrap/2/dataTables.bootstrap.css ${prefix}/var/www/css/dataTables.bootstrap.css && \
cp Plugins/integration/bootstrap/2/dataTables.bootstrap.js ${prefix}/var/www/js/dataTables.bootstrap.js && \
cp Plugins/integration/bootstrap/images/*.png ${prefix}/var/www/img/ && \
ln -s ${prefix}/var/www/img/ ${prefix}/var/www/images) && \
rm -rf /tmp/invenio-dt-bootstrap/plugins && \
mkdir /tmp/invenio-dt-bootstrap/plugins && \
(cd /tmp/invenio-dt-bootstrap/plugins && \
git clone 'https://github.com/DataTables/ColVis' && \
cp ColVis/media/css/ColVis.css ${prefix}/var/www/css/ColVis.css && \
cp ColVis/media/js/ColVis.js ${prefix}/var/www/js/ColVis.js && \
cp ColVis/media/images/*.png ${prefix}/var/www/img/) && \
cd ${prefix}/var/www/js && \
rm -rf /tmp/invenio-dt-bootstrap/plugins && \
mkdir /tmp/invenio-dt-bootstrap/plugins && \
(cd /tmp/invenio-dt-bootstrap/plugins && \
git clone 'https://github.com/LeaVerou/prism' && \
cp prism/themes/prism.css ${prefix}/var/www/css/prism.css && \
cp prism/prism.js ${prefix}/var/www/js/prism.js) && \
rm -rf /tmp/invenio-dt-bootstrap/plugins && \
wget http://invenio-software.org/download/jquery/jquery.bookmark.package-1.4.0.zip && \
unzip jquery.tablesorter.zip -d tablesorter && \
wget http://invenio-software.org/download/jquery/jquery.fileTree-1.01.zip && \
unzip jquery.fileTree-1.01.zip && \
rm jquery.fileTree-1.01.zip && \
wget http://invenio-software.org/download/jquery/ColVis.min.js && \
mv ColVis.min.js jquery.dataTables.ColVis.min.js && \
rm jquery.tablesorter.zip && \
rm -rf uploadify && \
unzip -u uploadify.zip -d uploadify && \
wget http://invenio-software.org/download/jquery/flot-0.6.zip && \
wget http://www.csspace.net/tmp/jquery-lightbox-0.5.zip && \
rm -rf jquery-lightbox && \
unzip -u jquery-lightbox-0.5.zip -d jquery-lightbox && \
sed -i 's/images\//\/js\/jquery-lightbox\/images\//g' jquery-lightbox/js/jquery.lightbox-0.5.js && \
rm -rf jquery-lightbox-0.5.zip && \
wget -O jquery-ui-timepicker-addon.js http://invenio-software.org/download/jquery/jquery-ui-timepicker-addon-1.0.3.js && \
unzip -u flot-0.6.zip && \
mv flot/jquery.flot.selection.min.js flot/jquery.flot.min.js flot/excanvas.min.js ./ && \
rm flot-0.6.zip && rm -r flot && \
mv uploadify/swfobject.js ./ && \
mv uploadify/cancel.png uploadify/uploadify.css uploadify/uploadify.allglyphs.swf uploadify/uploadify.fla uploadify/uploadify.swf ../img/ && \
mv uploadify/jquery.uploadify.v2.1.4.min.js ./jquery.uploadify.min.js && \
rm uploadify.zip && rm -r uploadify && \
wget --no-check-certificate https://github.com/douglascrockford/JSON-js/raw/master/json2.js && \
wget http://invenio-software.org/download/jquery/jquery.hotkeys-0.8.js -O jquery.hotkeys.js && \
- wget http://jquery.bassistance.de/treeview/jquery.treeview.zip && \
+ wget http://invenio-software.org/download/jquery/jquery.treeview.zip && \
unzip jquery.treeview.zip -d jquery-treeview && \
rm jquery.treeview.zip && \
wget http://invenio-software.org/download/jquery/v1.5/js/jquery.ajaxPager.js && \
unzip jquery.bookmark.package-1.4.0.zip && \
rm -f jquery.bookmark.ext.* bookmarks-big.png bookmarkBasic.html jquery.bookmark.js jquery.bookmark.pack.js && \
mv bookmarks.png ../img/ && \
mv jquery.bookmark.css ../css/ && \
- wget --no-check-certificate https://raw.github.com/0x000000/OmniWindow/master/jquery.omniwindow.js && \
- wget --no-check-certificate http://malsup.github.com/jquery.blockUI.js && \
- wget --no-check-certificate https://github.com/Darsain/sly/raw/master/dist/sly.min.js &&\
- wget --no-check-certificate https://raw.github.com/guillaumepotier/Parsley.js/master/parsley.js &&\
+ wget --no-check-certificate http://invenio-software.org/download/jquery/jquery.omniwindow.js && \
+ wget --no-check-certificate http://invenio-software.org/download/jquery/jquery.blockUI.js && \
+ wget --no-check-certificate http://invenio-software.org/download/jquery/sly.min.js &&\
+ wget --no-check-certificate http://invenio-software.org/download/jquery/parsley.js &&\
wget --no-check-certificate http://invenio-software.org/download/jquery/spin.min.js &&\
rm -f jquery.bookmark.package-1.4.0.zip && \
mkdir -p ${prefix}/var/www/img && \
cd ${prefix}/var/www/img && \
wget -r -np -nH --cut-dirs=4 -A "png,css" -P jquery-ui/themes http://jquery-ui.googlecode.com/svn/tags/1.8.17/themes/base/ && \
wget -r -np -nH --cut-dirs=4 -A "png,css" -P jquery-ui/themes http://jquery-ui.googlecode.com/svn/tags/1.8.17/themes/smoothness/ && \
wget -r -np -nH --cut-dirs=4 -A "png,css" -P jquery-ui/themes http://jquery-ui.googlecode.com/svn/tags/1.8.17/themes/redmond/ && \
wget --no-check-certificate -O datatables_jquery-ui.css https://github.com/DataTables/DataTables/raw/master/media/css/demo_table_jui.css && \
wget http://jquery-ui.googlecode.com/svn/tags/1.8.17/themes/redmond/jquery-ui.css && \
wget http://jquery-ui.googlecode.com/svn/tags/1.8.17/demos/images/calendar.gif && \
wget -r -np -nH --cut-dirs=5 -A "png" http://jquery-ui.googlecode.com/svn/tags/1.8.17/themes/redmond/images/)
@echo "***********************************************************"
@echo "** The jQuery plugins were successfully installed. **"
@echo "***********************************************************"
uninstall-jquery-plugins:
(cd ${prefix}/var/www/js && \
rm -f jquery.min.js && \
rm -f jquery.MultiFile.pack.js && \
rm -f jquery.jeditable.mini.js && \
rm -f jquery.flot.selection.min.js && \
rm -f jquery.flot.min.js && \
rm -f excanvas.min.js && \
rm -f jquery-ui-timepicker-addon.min.js && \
rm -f json2.js && \
rm -f jquery.uploadify.min.js && \
rm -rf tablesorter && \
rm -rf jquery-treeview && \
rm -f jquery.ajaxPager.js && \
rm -f jquery.form.js && \
rm -f jquery.dataTables.min.js && \
rm -f ui.core.js && \
rm -f jquery.bookmark.min.js && \
rm -f jquery.dataTables.ColVis.min.js && \
rm -f jquery.hotkeys.js && \
rm -f jquery.tablesorter.min.js && \
rm -f jquery-ui-1.7.3.custom.min.js && \
rm -f jquery.metadata.js && \
rm -f jquery-latest.js && \
rm -f jquery-ui.min.js)
(cd ${prefix}/var/www/img && \
rm -f cancel.png uploadify.css uploadify.swf uploadify.allglyphs.swf uploadify.fla && \
rm -f datatables_jquery-ui.css \
rm -f bookmarks.png) && \
(cd ${prefix}/var/www/css && \
rm -f jquery.bookmark.css)
@echo "***********************************************************"
@echo "** The jquery plugins were successfully uninstalled. **"
@echo "***********************************************************"
install-ckeditor-plugin:
@echo "***********************************************************"
@echo "** Installing CKeditor plugin, please wait... **"
@echo "***********************************************************"
rm -rf ${prefix}/lib/python/invenio/ckeditor/
rm -rf /tmp/invenio-ckeditor-plugin
mkdir /tmp/invenio-ckeditor-plugin
(cd /tmp/invenio-ckeditor-plugin && \
wget 'http://invenio-software.org/download/ckeditor/$(CKEDITOR)' && \
unzip -u -d ${prefix}/var/www $(CKEDITOR)) && \
find ${prefix}/var/www/ckeditor/ -depth -name '_*' -exec rm -rf {} \; && \
find ${prefix}/var/www/ckeditor/ckeditor* -maxdepth 0 ! -name "ckeditor.js" -exec rm -r {} \; && \
touch ${prefix}/var/www/ckeditor/invenio-ckeditor-config.js && \
rm -fr /tmp/invenio-ckeditor-plugin
@echo "* Installing Invenio-specific CKeditor config..."
@echo "Ignored: (cd $(top_srcdir)/modules/webstyle/etc && make install)"
@echo "***********************************************************"
@echo "** The CKeditor plugin was successfully installed. **"
@echo "** Please do not forget to properly set the option **"
@echo "** CFG_WEBCOMMENT_USE_RICH_TEXT_EDITOR in invenio.conf. **"
@echo "***********************************************************"
uninstall-ckeditor-plugin:
@rm -rvf ${prefix}/var/www/ckeditor
@rm -rvf ${prefix}/lib/python/invenio/ckeditor
@echo "***********************************************************"
@echo "** The CKeditor plugin was successfully uninstalled. **"
@echo "***********************************************************"
install-pdfa-helper-files:
@echo "***********************************************************"
@echo "** Installing PDF/A helper files, please wait... **"
@echo "***********************************************************"
wget 'http://invenio-software.org/download/invenio-demo-site-files/ISOCoatedsb.icc' -O ${prefix}/etc/websubmit/file_converter_templates/ISOCoatedsb.icc
@echo "***********************************************************"
@echo "** The PDF/A helper files were successfully installed. **"
@echo "***********************************************************"
install-mediaelement:
@echo "***********************************************************"
@echo "** MediaElement.js, please wait... **"
@echo "***********************************************************"
rm -rf /tmp/mediaelement
mkdir /tmp/mediaelement
wget 'http://github.com/johndyer/mediaelement/zipball/master' -O '/tmp/mediaelement/mediaelement.zip' --no-check-certificate
unzip -u -d '/tmp/mediaelement' '/tmp/mediaelement/mediaelement.zip'
rm -rf ${prefix}/var/www/mediaelement
mkdir ${prefix}/var/www/mediaelement
mv /tmp/mediaelement/johndyer-mediaelement-*/build/* ${prefix}/var/www/mediaelement
rm -rf /tmp/mediaelement
@echo "***********************************************************"
@echo "** MediaElement.js was successfully installed. **"
@echo "***********************************************************"
install-bootstrap:
@echo "***********************************************************"
@echo "** Installing Twitter Bootstrap, please wait... **"
@echo "***********************************************************"
rm -rf /tmp/invenio-bootstrap
mkdir /tmp/invenio-bootstrap
(cd /tmp/invenio-bootstrap && \
wget -O bootstrap.zip 'https://github.com/twbs/bootstrap/releases/download/v${BOOTSTRAPV}/bootstrap-${BOOTSTRAPV}-dist.zip' && \
unzip -u bootstrap.zip && \
cp dist/css/bootstrap.css ${prefix}/var/www/css/bootstrap.css && \
cp dist/css/bootstrap.min.css ${prefix}/var/www/css/bootstrap.min.css && \
cp dist/css/bootstrap-theme.css ${prefix}/var/www/css/bootstrap-theme.css && \
cp dist/css/bootstrap-theme.min.css ${prefix}/var/www/css/bootstrap-theme.min.css && \
mkdir -p ${prefix}/var/www/fonts && \
cp dist/fonts/glyphicons-halflings-regular.eot ${prefix}/var/www/fonts/glyphicons-halflings-regular.eot && \
cp dist/fonts/glyphicons-halflings-regular.svg ${prefix}/var/www/fonts/glyphicons-halflings-regular.svg && \
cp dist/fonts/glyphicons-halflings-regular.ttf ${prefix}/var/www/fonts/glyphicons-halflings-regular.ttf && \
cp dist/fonts/glyphicons-halflings-regular.woff ${prefix}/var/www/fonts/glyphicons-halflings-regular.woff && \
cp dist/js/bootstrap.js ${prefix}/var/www/js/bootstrap.js && \
cp dist/js/bootstrap.min.js ${prefix}/var/www/js/bootstrap.min.js && \
wget -O bootstrap-typeahead.zip 'http://twitter.github.com/typeahead.js/releases/latest/typeahead.js.zip' && \
unzip -u bootstrap-typeahead.zip && \
cp typeahead.js/typeahead.js ${prefix}/var/www/js/typeahead.js && \
cp typeahead.js/typeahead.min.js ${prefix}/var/www/js/typeahead.min.js && \
wget -O typeahead.js-bootstrap.css 'https://raw.github.com/jharding/typeahead.js-bootstrap.css/master/typeahead.js-bootstrap.css' && \
mv typeahead.js-bootstrap.css ${prefix}/var/www/css/typeahead.js-bootstrap.css && \
rm -fr /tmp/invenio-bootstrap )
@echo "***********************************************************"
@echo "** The Twitter Bootstrap was successfully installed. **"
@echo "***********************************************************"
uninstall-bootstrap:
rm ${prefix}/var/www/css/bootstrap.css && \
rm ${prefix}/var/www/css/bootstrap.min.css && \
rm ${prefix}/var/www/css/bootstrap-theme.css && \
rm ${prefix}/var/www/css/bootstrap-theme.min.css && \
rm ${prefix}/var/www/css/typeahead.js-bootstrap.css && \
rm ${prefix}/var/www/fonts/glyphicons-halflings-regular.eot && \
rm ${prefix}/var/www/fonts/glyphicons-halflings-regular.svg && \
rm ${prefix}/var/www/fonts/glyphicons-halflings-regular.ttf && \
rm ${prefix}/var/www/fonts/glyphicons-halflings-regular.woff && \
rm ${prefix}/var/www/js/bootstrap.js && \
rm ${prefix}/var/www/js/bootstrap.min.js && \
rm ${prefix}/var/www/js/typeahead.js && \
rm ${prefix}/var/www/js/typeahead.min.js
@echo "***********************************************************"
@echo "** The Twitter Bootstrap was successfully uninstalled. **"
@echo "***********************************************************"
install-hogan-plugin:
@echo "***********************************************************"
@echo "** Installing Hogan.js, please wait... **"
@echo "***********************************************************"
rm -rf /tmp/hogan
mkdir /tmp/hogan
(cd /tmp/hogan && \
wget -O hogan-${HOGANVER}.js 'http://twitter.github.com/hogan.js/builds/${HOGANVER}/hogan-${HOGANVER}.js' && \
cp hogan-${HOGANVER}.js ${prefix}/var/www/js/hogan.js && \
rm -fr /tmp/hogan )
@echo "***********************************************************"
@echo "** Hogan.js was successfully installed. **"
@echo "***********************************************************"
uninstall-hogan-plugin:
rm ${prefix}/var/www/js/hogan.js
@echo "***********************************************************"
@echo "** Hogan.js was successfully uninstalled. **"
@echo "***********************************************************"
install-typeahead-plugin:
@echo "***********************************************************"
@echo "** Installing typeahead.js, please wait... **"
@echo "***********************************************************"
rm -rf /tmp/typeahead
mkdir /tmp/typeahead
(cd /tmp/typeahead && \
wget -O typeahead.min.js 'http://twitter.github.com/typeahead.js/releases/latest/typeahead.min.js' && \
cp typeahead.min.js ${prefix}/var/www/js/typeahead.min.js && \
wget -O typeahead.js-bootstrap.css 'https://raw.github.com/jharding/typeahead.js-bootstrap.css/master/typeahead.js-bootstrap.css' && \
cp typeahead.js-bootstrap.css ${prefix}/var/www/css/typeahead.js-bootstrap.css && \
rm -fr /tmp/typeahead )
@echo "***********************************************************"
@echo "** typeahead.js was successfully installed. **"
@echo "***********************************************************"
uninstall-typeahead-plugin:
rm ${prefix}/var/www/js/typeahead.min.js
@echo "***********************************************************"
@echo "** typeahead.js was successfully uninstalled. **"
install-recline:
@echo "***********************************************************"
@echo "** Installing Recline JS, please wait... **"
@echo "***********************************************************"
rm -rf /tmp/invenio-recline
mkdir /tmp/invenio-recline
(cd /tmp/invenio-recline && \
wget -O recline.zip 'https://github.com/okfn/recline/archive/${RECLINEVER}.zip' && \
unzip -u recline.zip && \
rm recline.zip && \
mv *recline* recline && \
mkdir -p ${prefix}/var/www/js/recline/css && \
mkdir -p ${prefix}/var/www/js/recline/dist && \
mkdir -p ${prefix}/var/www/js/recline/src && \
mkdir -p ${prefix}/var/www/js/recline/vendor && \
cp -R recline/css ${prefix}/var/www/js/recline/ && \
cp -R recline/src ${prefix}/var/www/js/recline/ && \
cp -R recline/dist ${prefix}/var/www/js/recline/ && \
cp -R recline/vendor ${prefix}/var/www/js/recline/ && \
rm -rf /tmp/invenio-recline )
@echo "***********************************************************"
@echo "** The Recline JS was successfully installed. **"
@echo "***********************************************************"
uninstall-recline:
rm -Rf ${prefix}/var/www/js/recline
@echo "***********************************************************"
@echo "** The Recline JS was successfully uninstalled. **"
@echo "***********************************************************"
install-jquery-tokeninput:
@echo "***********************************************************"
@echo "** Installing JQuery Tokeninput, please wait... **"
@echo "***********************************************************"
rm -rf /tmp/jquery-tokeninput
mkdir /tmp/jquery-tokeninput
(cd /tmp/jquery-tokeninput && \
wget -O jquery-tokeninput-master.zip 'https://github.com/loopj/jquery-tokeninput/archive/master.zip' --no-check-certificate && \
unzip -u jquery-tokeninput-master.zip && \
cp jquery-tokeninput-master/styles/token-input-facebook.css ${prefix}/var/www/css/token-input-facebook.css && \
cp jquery-tokeninput-master/styles/token-input-mac.css ${prefix}/var/www/css/token-input-mac.css && \
cp jquery-tokeninput-master/styles/token-input.css ${prefix}/var/www/css/token-input.css && \
cp jquery-tokeninput-master/src/jquery.tokeninput.js ${prefix}/var/www/js/jquery.tokeninput.js && \
rm -fr /tmp/jquery-tokeninput )
@echo "***********************************************************"
@echo "** The JQuery Tokeninput was successfully installed. **"
@echo "***********************************************************"
uninstall-jquery-tokeninput:
rm ${prefix}/var/www/css/token-input-facebook.css && \
rm ${prefix}/var/www/css/token-input-mac.css && \
rm ${prefix}/var/www/css/token-input.css && \
rm ${prefix}/var/www/js/jquery.tokeninput.js
@echo "***********************************************************"
@echo "** The JQuery Tokeninput was successfully uninstalled. **"
@echo "***********************************************************"
install-plupload-plugin:
@echo "***********************************************************"
@echo "** Installing Plupload plugin, please wait... **"
@echo "***********************************************************"
rm -rf /tmp/plupload-plugin
mkdir /tmp/plupload-plugin
(cd /tmp/plupload-plugin && \
wget -O plupload-plugin.zip 'http://invenio-software.org/download/jquery/plupload-1.5.5.zip' && \
unzip -u plupload-plugin.zip && \
mkdir -p ${prefix}/var/www/js/plupload/i18n/ && \
cp -R plupload/js/jquery.plupload.queue ${prefix}/var/www/js/plupload/ && \
cp -R plupload/js/jquery.ui.plupload ${prefix}/var/www/js/plupload/ && \
cp plupload/js/plupload.browserplus.js ${prefix}/var/www/js/plupload/plupload.browserplus.js && \
cp plupload/js/plupload.flash.js ${prefix}/var/www/js/plupload/plupload.flash.js && \
cp plupload/js/plupload.flash.swf ${prefix}/var/www/js/plupload/plupload.flash.swf && \
cp plupload/js/plupload.full.js ${prefix}/var/www/js/plupload/plupload.full.js && \
cp plupload/js/plupload.gears.js ${prefix}/var/www/js/plupload/plupload.gears.js && \
cp plupload/js/plupload.html4.js ${prefix}/var/www/js/plupload/plupload.html4.js && \
cp plupload/js/plupload.html5.js ${prefix}/var/www/js/plupload/plupload.html5.js && \
cp plupload/js/plupload.js ${prefix}/var/www/js/plupload/plupload.js && \
cp plupload/js/plupload.silverlight.js ${prefix}/var/www/js/plupload/plupload.silverlight.js && \
cp plupload/js/plupload.silverlight.xap ${prefix}/var/www/js/plupload/plupload.silverlight.xap && \
cp plupload/js/i18n/*.js ${prefix}/var/www/js/plupload/i18n/ && \
rm -fr /tmp/plupload-plugin )
@echo "***********************************************************"
@echo "** The Plupload plugin was successfully installed. **"
@echo "***********************************************************"
uninstall-plupload-plugin:
rm -rf ${prefix}/var/www/js/plupload
@echo "***********************************************************"
@echo "** The Plupload was successfully uninstalled. **"
@echo "***********************************************************"
uninstall-pdfa-helper-files:
rm -f ${prefix}/etc/websubmit/file_converter_templates/ISOCoatedsb.icc
@echo "***********************************************************"
@echo "** The PDF/A helper files were successfully uninstalled. **"
@echo "***********************************************************"
#Solrutils allows automatic installation, running and searching of an external Solr index.
install-solrutils:
@echo "***********************************************************"
@echo "** Installing Solrutils and solr, please wait... **"
@echo "***********************************************************"
cd $(prefix)/lib && \
if test -d apache-solr*; then echo A solr directory already exists in `pwd` . \
Please remove it manually, if you are sure it is not needed; exit 2; fi ; \
if test -f apache-solr*; then echo solr tarball already exists in `pwd` . \
Please remove it manually.; exit 2; fi ; \
wget http://archive.apache.org/dist/lucene/solr/3.1.0/apache-solr-3.1.0.tgz && \
tar -xzf apache-solr-3.1.0.tgz && \
rm apache-solr-3.1.0.tgz
cd $(solrdir)/contrib/ ;\
wget http://mirrors.ibiblio.org/pub/mirrors/maven2/com/jcraft/jzlib/1.0.7/jzlib-1.0.7.jar && \
cd $(solrdir)/contrib/ ;\
jar -xf ../example/webapps/solr.war WEB-INF/lib/lucene-core-3.1.0.jar ; \
if test -d basic-lucene-libs; then rm -rf basic-lucene-libs; fi ; \
mv WEB-INF/lib/ basic-lucene-libs ; \
cp $(solrutils_dir)/schema.xml $(solrdir)/example/solr/conf/
cp $(solrutils_dir)/solrconfig.xml $(solrdir)/example/solr/conf/
cd $(solrutils_dir) && \
javac -classpath $(CLASSPATH) -d $(solrdir)/contrib @$(solrutils_dir)/java_sources.txt && \
cd $(solrdir)/contrib/ && \
jar -cf invenio-solr.jar org/invenio_software/solr/*class
update-v0.99.0-tables:
cat $(top_srcdir)/invenio/legacy/miscutil/sql/tabcreate.sql | grep -v 'INSERT INTO upgrade' | ${prefix}/bin/dbexec
echo "DROP TABLE IF EXISTS oaiREPOSITORY;" | ${prefix}/bin/dbexec
echo "ALTER TABLE bibdoc ADD COLUMN more_info mediumblob NULL default NULL;" | ${prefix}/bin/dbexec
echo "ALTER TABLE schTASK ADD COLUMN priority tinyint(4) NOT NULL default 0;" | ${prefix}/bin/dbexec
echo "ALTER TABLE schTASK ADD KEY priority (priority);" | ${prefix}/bin/dbexec
echo "ALTER TABLE rnkCITATIONDATA DROP PRIMARY KEY;" | ${prefix}/bin/dbexec
echo "ALTER TABLE rnkCITATIONDATA ADD PRIMARY KEY (id);" | ${prefix}/bin/dbexec
echo "ALTER TABLE rnkCITATIONDATA CHANGE id id mediumint(8) unsigned NOT NULL auto_increment;" | ${prefix}/bin/dbexec
echo "ALTER TABLE rnkCITATIONDATA ADD UNIQUE KEY object_name (object_name);" | ${prefix}/bin/dbexec
echo "ALTER TABLE sbmPARAMETERS CHANGE value value text NOT NULL default '';" | ${prefix}/bin/dbexec
echo "ALTER TABLE sbmAPPROVAL ADD note text NOT NULL default '';" | ${prefix}/bin/dbexec
echo "ALTER TABLE hstDOCUMENT CHANGE docsize docsize bigint(15) unsigned NOT NULL;" | ${prefix}/bin/dbexec
echo "ALTER TABLE cmtACTIONHISTORY CHANGE client_host client_host int(10) unsigned default NULL;" | ${prefix}/bin/dbexec
update-v0.99.1-tables:
@echo "Nothing to do; table structure did not change between v0.99.1 and v0.99.2."
update-v0.99.2-tables:
@echo "Nothing to do; table structure did not change between v0.99.2 and v0.99.3."
update-v0.99.3-tables:
@echo "Nothing to do; table structure did not change between v0.99.3 and v0.99.4."
update-v0.99.4-tables:
@echo "Nothing to do; table structure did not change between v0.99.4 and v0.99.5."
update-v0.99.5-tables:
@echo "Nothing to do; table structure did not change between v0.99.5 and v0.99.6."
update-v0.99.6-tables:
@echo "Nothing to do; table structure did not change between v0.99.6 and v0.99.7."
update-v0.99.7-tables:
@echo "Nothing to do; table structure did not change between v0.99.7 and v0.99.8."
-update-v0.99.8-tables: # from v0.99.8 to v1.0.0-rc0
+update-v0.99.8-tables:
+ @echo "Nothing to do; table structure did not change between v0.99.8 and v0.99.9."
+
+update-v0.99.9-tables: # from v0.99.9 to v1.0.0-rc0
echo "RENAME TABLE oaiARCHIVE TO oaiREPOSITORY;" | ${prefix}/bin/dbexec
cat $(top_srcdir)/invenio/legacy/miscutil/sql/tabcreate.sql | grep -v 'INSERT INTO upgrade' | ${prefix}/bin/dbexec
echo "INSERT INTO knwKB (id,name,description,kbtype) SELECT id,name,description,'' FROM fmtKNOWLEDGEBASES;" | ${prefix}/bin/dbexec
echo "INSERT INTO knwKBRVAL (id,m_key,m_value,id_knwKB) SELECT id,m_key,m_value,id_fmtKNOWLEDGEBASES FROM fmtKNOWLEDGEBASEMAPPINGS;" | ${prefix}/bin/dbexec
echo "ALTER TABLE sbmPARAMETERS CHANGE name name varchar(40) NOT NULL default '';" | ${prefix}/bin/dbexec
echo "ALTER TABLE bibdoc CHANGE docname docname varchar(250) COLLATE utf8_bin NOT NULL default 'file';" | ${prefix}/bin/dbexec
echo "ALTER TABLE bibdoc CHANGE status status text NOT NULL default '';" | ${prefix}/bin/dbexec
echo "ALTER TABLE bibdoc ADD COLUMN text_extraction_date datetime NOT NULL default '0000-00-00';" | ${prefix}/bin/dbexec
echo "ALTER TABLE collection DROP COLUMN restricted;" | ${prefix}/bin/dbexec
echo "ALTER TABLE schTASK CHANGE host host varchar(255) NOT NULL default '';" | ${prefix}/bin/dbexec
echo "ALTER TABLE hstTASK CHANGE host host varchar(255) NOT NULL default '';" | ${prefix}/bin/dbexec
echo "ALTER TABLE bib85x DROP INDEX kv, ADD INDEX kv (value(100));" | ${prefix}/bin/dbexec
echo "UPDATE clsMETHOD SET location='http://invenio-software.org/download/invenio-demo-site-files/HEP.rdf' WHERE name='HEP' AND location='';" | ${prefix}/bin/dbexec
echo "UPDATE clsMETHOD SET location='http://invenio-software.org/download/invenio-demo-site-files/NASA-subjects.rdf' WHERE name='NASA-subjects' AND location='';" | ${prefix}/bin/dbexec
echo "UPDATE accACTION SET name='runoairepository', description='run oairepositoryupdater task' WHERE name='runoaiarchive';" | ${prefix}/bin/dbexec
echo "UPDATE accACTION SET name='cfgoaiharvest', description='configure OAI Harvest' WHERE name='cfgbibharvest';" | ${prefix}/bin/dbexec
echo "ALTER TABLE accARGUMENT CHANGE value value varchar(255);" | ${prefix}/bin/dbexec
echo "UPDATE accACTION SET allowedkeywords='doctype,act,categ' WHERE name='submit';" | ${prefix}/bin/dbexec
echo "INSERT INTO accARGUMENT(keyword,value) VALUES ('categ','*');" | ${prefix}/bin/dbexec
echo "INSERT INTO accROLE_accACTION_accARGUMENT(id_accROLE,id_accACTION,id_accARGUMENT,argumentlistid) SELECT DISTINCT raa.id_accROLE,raa.id_accACTION,accARGUMENT.id,raa.argumentlistid FROM accROLE_accACTION_accARGUMENT as raa JOIN accACTION on id_accACTION=accACTION.id,accARGUMENT WHERE accACTION.name='submit' and accARGUMENT.keyword='categ' and accARGUMENT.value='*';" | ${prefix}/bin/dbexec
echo "UPDATE accACTION SET allowedkeywords='name,with_editor_rights' WHERE name='cfgwebjournal';" | ${prefix}/bin/dbexec
echo "INSERT INTO accARGUMENT(keyword,value) VALUES ('with_editor_rights','yes');" | ${prefix}/bin/dbexec
echo "INSERT INTO accROLE_accACTION_accARGUMENT(id_accROLE,id_accACTION,id_accARGUMENT,argumentlistid) SELECT DISTINCT raa.id_accROLE,raa.id_accACTION,accARGUMENT.id,raa.argumentlistid FROM accROLE_accACTION_accARGUMENT as raa JOIN accACTION on id_accACTION=accACTION.id,accARGUMENT WHERE accACTION.name='cfgwebjournal' and accARGUMENT.keyword='with_editor_rights' and accARGUMENT.value='yes';" | ${prefix}/bin/dbexec
echo "ALTER TABLE bskEXTREC CHANGE id id int(15) unsigned NOT NULL auto_increment;" | ${prefix}/bin/dbexec
echo "ALTER TABLE bskEXTREC ADD external_id int(15) NOT NULL default '0';" | ${prefix}/bin/dbexec
echo "ALTER TABLE bskEXTREC ADD collection_id int(15) unsigned NOT NULL default '0';" | ${prefix}/bin/dbexec
echo "ALTER TABLE bskEXTREC ADD original_url text;" | ${prefix}/bin/dbexec
echo "ALTER TABLE cmtRECORDCOMMENT ADD status char(2) NOT NULL default 'ok';" | ${prefix}/bin/dbexec
echo "ALTER TABLE cmtRECORDCOMMENT ADD KEY status (status);" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmALLFUNCDESCR VALUES ('Move_Photos_to_Storage','Attach/edit the pictures uploaded with the \"create_photos_manager_interface()\" function');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFIELDDESC VALUES ('Upload_Photos',NULL,'','R',NULL,NULL,NULL,NULL,NULL,'\"\"\"\r\nThis is an example of element that creates a photos upload interface.\r\nClone it, customize it and integrate it into your submission. Then add function \r\n\'Move_Photos_to_Storage\' to your submission functions list, in order for files \r\nuploaded with this interface to be attached to the record. More information in \r\nthe WebSubmit admin guide.\r\n\"\"\"\r\n\r\nfrom invenio.websubmit_functions.ParamFile import ParamFromFile\r\nfrom invenio.websubmit_functions.Move_Photos_to_Storage import read_param_file, create_photos_manager_interface, get_session_id\r\n\r\n# Retrieve session id\r\ntry:\r\n # User info is defined only in MBI/MPI actions...\r\n session_id = get_session_id(None, uid, user_info) \r\nexcept:\r\n session_id = get_session_id(req, uid, {})\r\n\r\n# Retrieve context\r\nindir = curdir.split(\'/\')[-3]\r\ndoctype = curdir.split(\'/\')[-2]\r\naccess = curdir.split(\'/\')[-1]\r\n\r\n# Get the record ID, if any\r\nsysno = ParamFromFile(\"%s/%s\" % (curdir,\'SN\')).strip()\r\n\r\n\"\"\"\r\nModify below the configuration of the photos manager interface.\r\nNote: \'can_reorder_photos\' parameter is not yet fully taken into consideration\r\n\r\nDocumentation of the function is available by running:\r\necho -e \'from invenio.websubmit_functions.Move_Photos_to_Storage import create_photos_manager_interface as f\\nprint f.__doc__\' | python\r\n\"\"\"\r\ntext += create_photos_manager_interface(sysno, session_id, uid,\r\n doctype, indir, curdir, access,\r\n can_delete_photos=True,\r\n can_reorder_photos=True,\r\n can_upload_photos=True,\r\n editor_width=700,\r\n editor_height=400,\r\n initial_slider_value=100,\r\n max_slider_value=200,\r\n min_slider_value=80)','0000-00-00','0000-00-00',NULL,NULL,0);" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_Photos_to_Storage','iconsize');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFIELDDESC VALUES ('Upload_Files',NULL,'','R',NULL,NULL,NULL,NULL,NULL,'\"\"\"\r\nThis is an example of element that creates a file upload interface.\r\nClone it, customize it and integrate it into your submission. Then add function \r\n\'Move_Uploaded_Files_to_Storage\' to your submission functions list, in order for files \r\nuploaded with this interface to be attached to the record. More information in \r\nthe WebSubmit admin guide.\r\n\"\"\"\r\nfrom invenio.websubmit_managedocfiles import create_file_upload_interface\r\nfrom invenio.websubmit_functions.Shared_Functions import ParamFromFile\r\n\r\nindir = ParamFromFile(os.path.join(curdir, \'indir\'))\r\ndoctype = ParamFromFile(os.path.join(curdir, \'doctype\'))\r\naccess = ParamFromFile(os.path.join(curdir, \'access\'))\r\ntry:\r\n sysno = int(ParamFromFile(os.path.join(curdir, \'SN\')).strip())\r\nexcept:\r\n sysno = -1\r\nln = ParamFromFile(os.path.join(curdir, \'ln\'))\r\n\r\n\"\"\"\r\nRun the following to get the list of parameters of function \'create_file_upload_interface\':\r\necho -e \'from invenio.websubmit_managedocfiles import create_file_upload_interface as f\\nprint f.__doc__\' | python\r\n\"\"\"\r\ntext = create_file_upload_interface(recid=sysno,\r\n print_outside_form_tag=False,\r\n include_headers=True,\r\n ln=ln,\r\n doctypes_and_desc=[(\'main\',\'Main document\'),\r\n (\'additional\',\'Figure, schema, etc.\')],\r\n can_revise_doctypes=[\'*\'],\r\n can_describe_doctypes=[\'main\'],\r\n can_delete_doctypes=[\'additional\'],\r\n can_rename_doctypes=[\'main\'],\r\n sbm_indir=indir, sbm_doctype=doctype, sbm_access=access)[1]\r\n','0000-00-00','0000-00-00',NULL,NULL,0);" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_Uploaded_Files_to_Storage','forceFileRevision');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmALLFUNCDESCR VALUES ('Create_Upload_Files_Interface','Display generic interface to add/revise/delete files. To be used before function \"Move_Uploaded_Files_to_Storage\"');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmALLFUNCDESCR VALUES ('Move_Uploaded_Files_to_Storage','Attach files uploaded with \"Create_Upload_Files_Interface\"')" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_Revised_Files_to_Storage','elementNameToDoctype');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_Revised_Files_to_Storage','createIconDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_Revised_Files_to_Storage','createRelatedFormats');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_Revised_Files_to_Storage','iconsize');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_Revised_Files_to_Storage','keepPreviousVersionDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmALLFUNCDESCR VALUES ('Move_Revised_Files_to_Storage','Revise files initially uploaded with \"Move_Files_to_Storage\"')" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','maxsize');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','minsize');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','doctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','restrictions');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canDeleteDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canReviseDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canDescribeDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canCommentDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canKeepDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canAddFormatDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canRestrictDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canRenameDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canNameNewFiles');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','createRelatedFormats');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','keepDefault');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','showLinks');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','fileLabel');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','filenameLabel');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','descriptionLabel');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','commentLabel');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','restrictionLabel');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','startDoc');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','endDoc');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','defaultFilenameDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','maxFilesDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_Uploaded_Files_to_Storage','iconsize');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_Uploaded_Files_to_Storage','createIconDoctypes');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Report_Number_Generation','nblength');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Second_Report_Number_Generation','2nd_nb_length');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Get_Recid','record_search_pattern');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmALLFUNCDESCR VALUES ('Move_FCKeditor_Files_to_Storage','Transfer files attached to the record with the FCKeditor');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_FCKeditor_Files_to_Storage','input_fields');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Stamp_Uploaded_Files','layer');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Stamp_Replace_Single_File_Approval','layer');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Stamp_Replace_Single_File_Approval','switch_file');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Stamp_Uploaded_Files','switch_file');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_Files_to_Storage','paths_and_restrictions');" | ${prefix}/bin/dbexec
echo "INSERT INTO sbmFUNDESC VALUES ('Move_Files_to_Storage','paths_and_doctypes');" | ${prefix}/bin/dbexec
echo "ALTER TABLE cmtRECORDCOMMENT ADD round_name varchar(255) NOT NULL default ''" | ${prefix}/bin/dbexec
echo "ALTER TABLE cmtRECORDCOMMENT ADD restriction varchar(50) NOT NULL default ''" | ${prefix}/bin/dbexec
echo "ALTER TABLE cmtRECORDCOMMENT ADD in_reply_to_id_cmtRECORDCOMMENT int(15) unsigned NOT NULL default '0'" | ${prefix}/bin/dbexec
echo "ALTER TABLE cmtRECORDCOMMENT ADD KEY in_reply_to_id_cmtRECORDCOMMENT (in_reply_to_id_cmtRECORDCOMMENT);" | ${prefix}/bin/dbexec
echo "ALTER TABLE bskRECORDCOMMENT ADD in_reply_to_id_bskRECORDCOMMENT int(15) unsigned NOT NULL default '0'" | ${prefix}/bin/dbexec
echo "ALTER TABLE bskRECORDCOMMENT ADD KEY in_reply_to_id_bskRECORDCOMMENT (in_reply_to_id_bskRECORDCOMMENT);" | ${prefix}/bin/dbexec
echo "ALTER TABLE cmtRECORDCOMMENT ADD reply_order_cached_data blob NULL default NULL;" | ${prefix}/bin/dbexec
echo "ALTER TABLE bskRECORDCOMMENT ADD reply_order_cached_data blob NULL default NULL;" | ${prefix}/bin/dbexec
echo "ALTER TABLE cmtRECORDCOMMENT ADD INDEX (reply_order_cached_data(40));" | ${prefix}/bin/dbexec
echo "ALTER TABLE bskRECORDCOMMENT ADD INDEX (reply_order_cached_data(40));" | ${prefix}/bin/dbexec
echo -e 'from invenio.legacy.webcomment.adminlib import migrate_comments_populate_threads_index;\
migrate_comments_populate_threads_index()' | $(PYTHON)
echo -e 'from invenio.modules.access.firerole import repair_role_definitions;\
repair_role_definitions()' | $(PYTHON)
CLEANFILES = *~ *.pyc *.tmp
diff --git a/NEWS b/NEWS
index a5e695169..b282ba22f 100644
--- a/NEWS
+++ b/NEWS
@@ -1,1868 +1,1958 @@
Invenio NEWS
============
Here is a short summary of the most notable changes in Invenio
releases. For more information about the current release, please
consult RELEASE-NOTES. For more information about changes, please
consult ChangeLog.
+Invenio v1.1.3 -- released 2014-02-25
+-------------------------------------
+
+ *) BatchUploader: rights to ::1 for robot upload; avoid
+ tempfile.tempdir redefinition (#1594)
+
+ *) BibCatalog: no newlines in subject for RT plugin
+
+ *) BibDocFile: RHEL6 magic bindings support (#1466)
+
+ *) BibFormat: fix for BibTeX regression tests; better BibTeX title
+ and collaboration
+
+ *) BibRank: temporary file storage in CFG_TMPDIR (#1594)
+
+ *) BibSword: author MARC tag definition fix
+
+ *) BibUpload: FFT replace warning in guide
+
+ *) I18N: PO file update for the release of v1.1.3; PO file update for
+ the release of v1.0.6; PO file update for the release of v0.99.9;
+ collection demo names for new translations
+
+ *) OAIHarvest: for for bad exception handling
+
+ *) OAIRepository: optional support for --notimechange
+
+ *) Travis CI: initial release of configuration
+
+ *) WebSearch: nonexisting record API test case fix (#1692); correct
+ record sums from hosted colls (#1651); space between records in
+ MARC HTML; fix for BibTeX regression tests; field-filtered MARCXML
+ API output (#1591); more complete API regression test suite;
+ get_fieldvalues_alephseq_like() utils; asciification of `oe`
+ grapheme (#1582); bug fix for SPIRES date math search
+
+ *) WebSession: fix mail cookie expiration (#1596)
+
+ *) WebSubmit: fix for typo in Shared_Functions; optional pdftk
+ regression tests
+
+ *) dbquery: closes redundant connection
+
+ *) git: addition of compile to gitignore; new entry in gitignore
+
+ *) global: language value always in link URLs
+
+ *) installation: pip requirement version updates; pip requirements;
+ no user prompt for warnings; empty Travis configuration; location
+ of jquery-1.7.1.min.js; location of flot; information about
+ unidecode; fix autotools rsync instructions
+
+ *) intbitset: no crash when intbitset is on rhs (#1287)
+
+ *) inveniocfg: fix for mod_headers
+
+ *) kwalitee: list comprehensions instead of lambdas; compatibility
+ with pylint 1.0.0
+
+Invenio v1.0.6 -- released 2014-01-31
+-------------------------------------
+
+ *) BatchUploader: avoid tempfile.tempdir redefinition (#1594)
+
+ *) BibRank: temporary file storage in CFG_TMPDIR (#1594)
+
+ *) BibUpload: FFT replace warning in guide
+
+ *) dbquery: closes redundant connection
+
+ *) global: language value always in link URLs
+
+ *) installation: fix autotools rsync instructions; pip requirements;
+ pip requirement version updates
+
+ *) intbitset: no crash when intbitset is on rhs (#1287)
+
+ *) WebSearch: asciification of `oe` grapheme (#1582); correct record
+ sums from hosted colls (#1651); nonexisting record API test case
+ fix (#1692); space between records in MARC HTML
+
+ *) WebSession: fix mail cookie expiration (#1596)
+
+ *) WebSubmit: fix for typo in Shared_Functions
+
+CDS Invenio v0.99.9 -- released 2014-01-31
+------------------------------------------
+
+ *) temporary file storage in CFG_TMPDIR (BibRank)
+
Invenio v1.1.2 -- released 2013-08-19
-------------------------------------
*) BibAuthorID: fix in name comparisons (#1313 #1314); improvements
and fixes; improvements, fixes and optimizations; UI and backend
improvements
*) BibCatalog: removal of print statement (#1337)
*) BibClassify: escape keywords in tag cloud and MARCXML
*) BibDocFile: better JS washing in web UI; display file upload
progress (#1020 #1021); display "Restricted" label correctly
(#1299); fix check-md5 with bibdocfsinfo cache (#1249); fix
check-md5 with bibdocfsinfo cache (#1249); fix error in calling
register_download (#1311); handling of exceptions in Md5Folder
(#1060); revert md5 property patch (#1249); support new magic
library (#1207)
*) BibEncode: minor fix in process_batch_job()
*) BibFormat: additional fulltext file display in HB (#1219); checks
for bibformat bin; fix CLI call to old PHP-based formatter; fixes
unit tests (#1320); fix for fulltext file format; fix snippets for
phrase queries (#1201); format_element initialisation fix; passing
of user_info for Excel format; replacement of CDS Invenio by
Invenio; setUp/tearDown in unit tests (#1319); skip hidden icons
in OpenGraph image tag
*) BibIndex: better wording for stemming in admin UI; replacement of
CDS Invenio by Invenio; synonym indexing speed up (#1484); use
human friendly index name (#1329)
*) BibKnowledge: /kb/export 500 error fix; optional memoisation of
KBR lookups (#1484)
*) BibMerge: delete cache file on submit
*) BibSched: bibupload max_priority check; bugfix for high-priority
monotasks; increases size of monitor columns;
parse_runtime_limit() fix (#1432); parse_runtime_limit() tests fix
(#1432)
*) BibUpload: FMT regression test case fix (#1152); indicators in
strong tags (#939)
*) CKEditor: updated to version 3.6.6
*) dateutils: strftime improvement (#1065); strptime for Python-2.4
compatibility
*) errorlib: hiding bibcatalog info in exception body
*) global: test suite nosification
*) htmlutils: fix single quote escaping; improve js string escaping;
MathJax 2.1 (#1050)
*) I18N: updates to Catalan and Spanish translations
*) installation: fix collectiondetailedrecordpagetabs (#1496); fix
for jQuery hotkeys add-on URL (#1507); fix for MathJax OS X
install issue (#1455); support for Apache-2.4 (#1552)
*) inveniocfg: tests runner file closure fix (#1327)
*) InvenioConnector: fix for CDS authentication; mechanize dependency
*) inveniogc: consider journal cache subdirs
*) memoiseutils: initial release
*) OAIHarvest: fix path for temporary authorlists; holding-pen UI
bugfixes (#1401)
*) OAIRepository: CFG_OAI_REPOSITORY_MARCXML_SIZE; no bibupload -n
*) RefExtract: replacement of CDS Invenio by Invenio
*) WebAccess: fix variable parsing in robot auth (#1456); IP-based
rules and offline user fix (#1233); replacement of CDS Invenio by
InveniO
*) WebApiKey: renames unit tests to regression tests (#1324)
*) WebAuthorProfile: fix XSS vulnerability
*) WebComment: escape review "title"; escape review "title"
*) WebSearch: 410 HTTP code for deleted records; advanced search
notification if no hits; better cleaning of word patterns; fix
infinite synonym lookup cases (#804); handles "find feb 12"
(#948); nicer browsing of fuzzy indexes (#1348); respect default
`rg` in Advanced Search; SPIRES date math search fixes (#431
#948); SPIRES invalid date search fix (#1467); tweaks SPIRES
two-digit search; unit test disabling for CFG_CERN_SITE; unit test
update (#1326)
*) WebSession: fix for list of admin activities (#1444); login_method
changes; unit vs regression test suite cleanup
*) WebStat: use CFG_JOURNAL_TAG instead of 773/909C4 (#546)
*) WebSubmit: new websubmitadmin CLI (#1334); replacement of CDS
Invenio v1.0.5 -- released 2013-08-19
-------------------------------------
*) BibClassify: escape keywords in tag cloud and MARCXML
*) BibDocFile: support new magic library
*) BibFormat: additional fulltext file display in HB; fix CLI call to
old PHP-based formatter; format_element initialisation fix
*) BibIndex: better wording for stemming in admin UI
*) BibKnowledge: /kb/export 500 error fix
*) BibUpload: FMT regression test case fix; indicators in strong tags
*) errorlib: hiding bibcatalog info in exception body
*) global: test suite nosification
*) installation: fix collectiondetailedrecordpagetabs; support for
Apache-2.4
*) WebAccess: IP-based rules and offline user fix; replacement of CDS
Invenio by InveniO
*) WebComment: escape review "title"
*) WebSearch: respect default `rg` in Advanced Search
*) WebSession: fix for list of admin activities; login_method changes
*) WebSubmit: new websubmitadmin CLI
CDS Invenio v0.99.8 -- released 2013-08-19
------------------------------------------
*) escape keywords in tag cloud and MARCXML (BibClassify)
*) fix CLI call to old PHP-based formatter; fix format_element
initialisation (BibFormat)
*) better wording for stemming in admin UI (BibIndex)
*) IP-based rules and offline user fix (WebAccess)
*) escape review "title" (WebComment)
*) fix collectiondetailedrecordpagetabs (installation)
Invenio v1.1.1 -- released 2012-12-21
-------------------------------------
*) BatchUploader: error reporting improvements
*) BibAuthorID: arXiv login upgrade; fix for small bug in claim
interface
*) BibConvert: fix bug with SPLITW function; target/source CLI flag
description fix
*) BibDocFile: better error report for unknown format; explicit
redirection to secure URL; fix for file upload in submissions
*) BibEdit: 'bibedit' CSS class addition to page body
*) BibFormat: clean Default_HTML_meta template; fix for js_quicktags
location; ISBN tag update for meta format; "ln" parameter in
bfe_record_url output; meta header output fix; relator code filter
in bfe_authors; fix for reformatting by record IDs
*) errorlib: register_exception improvements
*) global: login link using absolute URL redirection
*) installation: aidUSERINPUTLOG consistency upgrade; bigger
hstRECORD.marcxml size; fix for wrong name in tabcreate; inclusion
of JS quicktags in tarball; mark upgrade recipes as applied;
rephrase 1.1 upgrade recipe warning; safer upgrader bibsched
status parse; strip spaces in CFG list values
*) jQuery: tablesorter location standardisation
*) mailutils: authentication and TLS support
*) OAIRepository: Edit OAI Set page bug fix; fix for OAI set editing;
print_record() fixes
*) plotextractor: washing of captions and context
*) pluginutils: fix for failing bibformat test case
*) solrutils: addition of files into release tarball
*) WebAccess: admin interface usability improvement; guest unit tests
for firerole
*) WebAlert: new regression tests for alerts
*) WebComment: cleaner handling of non-reply comments
*) WebJournal: better language handling in widgets; CERN-specific
translation; explicit RSS icon dimensions; fix for
CFG_TMPSHAREDDIR; fix for retrieval of deleted articles; search
select form by name
*) WebSearch: fix for webcoll grid layout markup;
get_all_field_values() typo; next-hit/previous-hit numbering fix;
respect output format content-type; washing of 'as' argument
*) WebSession: fix for login-with-referer issue; fix for
merge_usera_into_userb()
*) WebStyle: dumb page loading fix Google Analytics documentation
update; memory leak fix in session handling; new /ping handler;
removal of excess language box call; req.is_https() fix;
*) WebSubmit: display login link on /submit page; fix for
Send_APP_Mail function; fix the approval URL for publiline
*) WebUser: fix for referer URL protocol
Invenio v1.0.4 -- released 2012-12-21
-------------------------------------
*) installation: inclusion of JS quicktags in tarball
*) bibdocfile: better error report for unknown format
*) WebAccess: admin interface usability improvement
Invenio v1.0.3 -- released 2012-12-19
-------------------------------------
*) BatchUploader: error reporting improvements
*) BibConvert: fix bug with SPLITW function; target/source CLI flag
description fix
*) BibEdit: 'bibedit' CSS class addition to page body
*) BibFormat: fix for js_quicktags location
*) jQuery: tablesorter location standardisation
*) WebComment: cleaner handling of non-reply comments
*) WebJournal: explicit RSS icon dimensions; fix for
CFG_TMPSHAREDDIR; fix for retrieval of deleted articles
*) WebSearch: external search pattern_list escape fix; respect output
format content-type; washing of 'as' argument
*) WebStyle: dumb page loading fix; Google Analytics documentation
update; memory leak fix in session handling; new /ping handler;
removal of excess language box call; req.is_https() fix
*) WebSubmit: fix for Send_APP_Mail function
*) WebUser: fix for referer URL protocol
CDS Invenio v0.99.7 -- released 2012-12-18
------------------------------------------
*) Google Analytics documentation update (WebStyle)
*) target/source CLI flag description fix (BibConvert)
Invenio v1.1.0 -- released 2012-10-21
-------------------------------------
*) BatchUploader: RESTful interface, runtime checks, TextMARC input,
job priority selection
*) BibAuthorID: new automatic author disambiguation and paper
claiming facility
*) BibCatalog: storage of ticket requestor, default RT user
*) BibCirculation: security fixes
*) BibClassify: UI improvements and refactoring
*) BibConvert: new BibTeX-to-MARCXML conversion, new oaidmf2marcxml
conversion, fixes for WORDS
*) BibDocFile: new filesystem cache for faster statistics, caseless
authorisation, disable HTTP range requests, improve file format
policies, and more
*) BibEdit: new options related to preview and printing, reference
curation, autocompletion, record and field template manager,
editing fields and subfields, per-collection authorisations, use
of knowledge bases, and more
*) BibEditMulti: new actions with conditions on fields, partial
matching for subfields, faster preview generation, and more
*) BibEncode: new audio and video media file processing tool, new
Video demo collection
*) BibFormat: new full-text snippet display facility, new
configuration for I18N caching, updates to EndNote, Excel, Dublin
Core and other formats, updates to formatting elements such as
DOI, author, updates to podcast output, updates to XSLT
processing, and more
*) OAIHarvest: new configurable workflow with reference extraction,
new author list extraction post process, upload priority, OpenAIRE
compliance, better handling of timeouts, and more
*) BibIndex: new full-text indexing via Solr, new support for author
ID indexing, better author tokeniser
*) BibKnowledge: dynamic knowledge bases for record editor, support
for JSON format
*) BibMatch: new matching of restricted collections
*) BibMerge: subfield order in slave record, confirmation pop up,
record selection bug fix
*) BibRank: new index term count ranking method, new support for flot
graphs, updates to citation graphs
*) BibRecord: new possibility to use lxml parser, sanity checks
*) BibSched: new motd-like facility for queue monitor, new
continuable error status for tasks, new tasklet framework, new
multi-node support, new monotask support, new support for task
sequences, improvements to scheduling algorithm
*) BibSort: new in-memory fast sorting tool using configurable
buckets
*) BibUpload: new automatic generation of MARC tag 005, new
`--callback-url' CLI parameter, fixes for appending existing
files, fixes for multiple 001 tags, and more
*) WebAccess: new external person ID support, performance
improvements, robot manager UI improvements, fixes for firerole
handling,
*) WebAlert: new alert description facility, fixes for restricted
collections
*) WebApiKey: new user-signed Web API key facility
*) WebAuthorProfile: new author pages with dynamic box layout
*) WebBasket: add to basket interface improvements, better XML
export, fixes for external records and other improvements
*) WebComment: new collapsible comment support, new permalink to
comments, loss prevention of unsubmitted comments, tidying up HTML
markup of comments, and more
*) WebJournal: new Open Graph markup, more customisable newsletter,
redirect to latest release of specific category, refresh chosen
collections on release, remove unnecessary encoding/decoding,
update weather widget for new APIs, and more
*) WebSearch: new index-time and search-time synonym support, new
Open Graph markup, new Google Scholar friendly metadata in page
header, new limit option for wildcard queries, new support for
access to merged records, new next/previous/back link support, new
`authorcount' indexing and searching, new relative date search
facility, clean OpenSearch support, improved speed, improvements
to SPIRES query syntax support, improvements to self-cite math,
primary collection guessing, other numerous fixes
*) WebSession: new useful guest sessions, reintroduces configurable
IP checking, enforcement of nickname refresh, several other fixes
*) WebStat: new login statistics, new custom query summary, error
analyser, custom event improvements
*) WebStyle: new display restriction flag for restricted records, new
initial right-to-left language support, authenticated user and
HTTPS support, IP check for proxy configurations, layout updates
and fixes for MSIE, and more
*) WebSubmit: new initial support for converting to PDF/X, new
embargo support, better LibreOffice compatibility, better async
file upload, enhancements for Link_Records, support for hiding
HIDDEN files in document manager, configurable initial value for
counter, make use of BibSched task sequences, and more
*) installation: updates to jQuery, CKEditor, unoconv, and other
prerequisites
*) dbdump: new compression support, reworked error handling
*) dbquery: new possibility to query DB slave nodes, new dict-like
output, fix for MySQL 5.5.3 and higher versions
*) errorlib: stack analysis improvements, outline style improvements
for invenio.err
*) htmlutils: improvements to HTML markup removal, HTML tidying
*) I18N: new Arabic and Lithuanian translations, updates to Catalan,
Czech, French, German, Greek, Italian, Russian, Slovak, Spanish
translations
*) intbitset: new performance improvements, new get item support, new
pickle support, several memory leak fixes
*) inveniocfg: new automated Invenio Upgrader tool
*) InvenioConnector: new search with retries, improved search
parameters, improved local site check, use of Invenio user agent
*) jsonutils: new JSON utility library
*) mailutils: possibility to specify Reply-To header, fixes to
multipart
*) plotextractor: better TeX detection, better PDF harvesting from
arXiv, configurable sleep timer
*) pluginutils: new create_enhanced_plugin_builder API, external
plugin loading
*) RefExtract: new daemon operation mode, new DOI recognition, better
author recognition, new author knowledge base
*) remote debugger: new remote debuggng support
*) sequtils: new sequence generator tool
*) solrutils: new support for full-text query dispatching to Solr
*) testutils: new Selenium web test framework
*) textutils: updates to string-to-ascii functions, LaTeX symbols to
Unicode
*) urlutils: fix for redirect_to_url
*) xmlmarclint: fix for error report formatting
*) ... and other numerous smaller fixes and improvements
Invenio v1.0.2 -- released 2012-10-19
-------------------------------------
*) BibConvert: fix for static files in admin guide
*) BibEdit: regression test case fix
*) BibFormat: fix call to bfe_primary_report_number; revert fix for
format validation report
*) BibHarvest: OAI harvesting via HTTP proxy
*) BibRank: begin_date initialisation in del_recids(); INSERT DELAYED
INTO rnkPAGEVIEWS; user-friendlier message for similar docs
*) BibUpload: clarify correct/replace mode help
*) WebJournal: catch ValueError when reading cache; use
CFG_TMPSHAREDDIR in admin UI
*) WebSearch: allow webcoll to query hidden tags; external collection
search fix; external search XSS vulnerability fix; fix for
parentheses inside quotes; get_collection_reclist() fix; more uses
of `rg` configurable default; 'verbose' mode available to admins
only; XSS and verbose improvements
*) WebSession: fix possibly undefined variables; prevent nickname
modification
*) WebStyle: workaround IE bug with cache and HTTPS
*) WebSubmit: configurable Document File Manager; fix JS check for
mandatory fields; unoconv calling fix
*) bibdocfile: guess_format_from_url() improvement;
guess_format_from_url() improvements; INSERT DELAYED INTO
rnkDOWNLOADS
*) global: removal of psyco
*) I18N: Spanish and Catalan updates to Search Tips; updates to
German translation
*) installation: fix for jQuery UI custom; fix md5sum example
arguments; new index on session.session_expiry
*) intbitset: fix memory leak
*) inveniogc: tmp directory removal improvements
*) urlutils: MS Office redirection workaround
CDS Invenio v0.99.6 -- released 2012-10-18
------------------------------------------
*) improved XSS safety in external collection searching (WebSearch)
*) verbose level in the search results pages is now available only to
admins, preventing potential restricted record ID disclosure even
though record content would remain restricted (WebSearch)
Invenio v1.0.1 -- released 2012-06-28
-------------------------------------
*) BibFormat: fix format validation report; fix opensearch prefix
exclusion in RSS; fix retrieval of collection identifier
*) BibIndex: new unit tests for the Greek stemmer
*) BibSched: improve low level submission arg parsing; set ERROR
status when wrong params; task can stop immediately when sleeping
*) BibSword: remove dangling documentation
*) BibUpload: fix setting restriction in -a/-ir modes
*) WebAlert: simplify HTML markup
*) WebComment: only logged users to use report abuse
*) WebJournal: hide deleted records
*) WebSearch: adapt test cases for citation summary; fix collection
order on the search page; look at access control when webcolling;
sorting in citesummary breakdown links
*) WebSession: simplify HTML markup
*) WebSubmit: capitalise doctypes in Doc File Manager; check
authorizations in endaction; check for problems when archiving;
ensure unique tmp file name for upload; fix email formatting; fix
Move_to_Done function; remove 8564_ field from demo templates;
skip file upload if necessary; update CERN-specific config
*) bibdocfile: BibRecDocs recID argument type check
*) data cacher: deletes cache before refilling it
*) dbquery: fix dbexec CLI WRT max allowed packet
*) I18N: updates to Greek translation
*) installation: fix circular install-jquery-plugins; fix demo user
initialisation; fix jQuery tablesorter download URL; fix jQuery
uploadify download URL; more info about max_allowed_packet; remove
unneeded rxp binary package
Invenio v1.0.0 -- released 2012-02-29
-------------------------------------
*) BatchUploader: fix retrieval of recs from extoaiid
*) BibCirculation: fix regexp for dictionary checking; security check
before eval
*) BibConvert: fix UP and DOWN for UTF-8 strings
*) bibdocfile: add missing normalize_format() calls;
check_bibdoc_authorization caseless; fix append WRT
description/restriction; fix cli_set_batch function; fix
documentation WRT --with-version; fix handling of embargo firerole
rule; fix parsing of complex subformats
*) BibEdit: fix crash in Ajax request; fix undefined dictionary key
*) BibFormat: better escape BFE in admin test UI; do not exit if no
XSLT processor found; fix regression test; fix URL to ejournal
resolver; fix XSLT formatting of MARCXML snippets; removes 'No
fulltext' message; special handling of INSPIRE-PUBLIC type; use
default namespace in XSL
*) BibHarvest: check for empty resumptionToken; fix MARCXML creation
in OAI updater; optional JSON dependency
*) BibIndex: fix author:Campbell-Wilson word query; fix
double-stemming upon indexing; fix Porter stemmer in multithread;
Greek stemmer improvements
*) BibKnowledge: make XML/XSLT libs optional
*) BibRank: CERN hack to inactivate similarity lists; fix citation
indexer time stamp updating; fix citation indexing of deleted
records; fix citedby/refersto for infinite sets; fix empty
citation data cacher; fix incremental citation indexer leaks; make
numpy optional; minimum x-axis in citation history graphs; run
citation indexer after word indexer
*) BibRecord: fix for record_get_field_instances()
*) BibSched: fix guess_apache_process_user_from_ps; use larger
timouts for launching tasks
*) BibUpload: FFT regression tests not to use CDS
*) htmlutils: fix FCKeditor upload URLs
*) installation: add note about optional hashlib; change table TYPE
to ENGINE in SQL; fix 'install-mathjax-plugin'; fix issue with
FCKeditor; fix 'make install-jquery-plugins'; fix output message
cosmetics; new 'make install-ckeditor-plugin'; re-enable WSGI
pre-loading
*) intbitset: fix never ending loop in __repr__; fix several memory
leaks
*) inveniocfg: fix resetting ranking method names
*) inveniogc: new CLI options check/optimise tables
*) kwalitee: grep-like output and exit status changes; use
`--check-some` as default CLI option
*) mailutils: remove unnecessary 'multipart/related'
*) plotextractor: fix INSPIRE unit test
*) textmarc2xmlmarc: fix handling of BOM
*) urlutils: new Indico request generator helper
*) WebAccess: fix Access policy page; fix FireRole handling integer
uid; fix retrieving emails from firerole
*) WebAlert: fix the display of records in alerts
*) WebBasket: fix missing return statement; fix number of items in
public baskets
*) WebComment: CERN-specific hack for ATLAS comments; fix discussion
display in bfe_comments; fix washing of email to admin; improve
sanity checks
*) WebHelp: HOWTO MARC document update
*) WebJournal: fix seminar widget encoding issue; fix seminar widget
for new Indico APIs; update weather widget for new APIs
*) WebSearch: add refersto:/a b c/ example to guide; CERN-specific
hack for journal sorting; CERN-specific hack for latest additions;
fix case-insensitive collection search; fix CDSIndico external
search; fix collection translation in admin UI; fix
get_fieldvalues() when recid is str; fix
get_index_id_from_field(); fix structured regexp query parsing;
fix symbol name typo in loop checking; parenthesised collection
definitions; remove accent-search warning in guide; remove Report
for INSPIRE author pages; replace CDS Indico by Indico; updates
some output phrases
*) WebSession: fix crash when no admin user exists
*) WebStyle: better service failure message; fix implementation of
req.get_hostname; fluid width of the menu; pre-load citation
dictionaries for web
*) WebSubmit: avoid printing empty doctype section;
check_user_can_view_record in publiline; fix filename bug in
document manager; fix handling of uploaded files; fix
record_search_pattern in DEMOJRN
*) xmlmarclint: 'no valid record detected' error
*) I18N: updates to Catalan, Czech, French, German, Greek, Italian,
Slovak, and Spanish translations
*) Note: for a complete list of new features in Invenio v1.0 release
series over Invenio v0.99 release series, please see:
<http://invenio-software.org/blog/invenio-1.0.0-rc0>
CDS Invenio v0.99.5 -- released 2012-02-21
------------------------------------------
*) improved sanity checks when reporting, voting, or replying to a
comment, or when accessing comment attachments, preventing URL
mangling attempts (WebComment)
CDS Invenio v0.99.4 -- released 2011-12-19
------------------------------------------
*) fixed double stemming during indexing (BibIndex)
*) fixed collection translation in admin UI (WebSearch)
*) fixed UP and DOWN functions for UTF-8 strings (BibConvert)
Invenio v1.0.0-rc0 -- released 2010-12-21
-----------------------------------------
*) CDS Invenio becomes Invenio as of this release
*) new facility of hosted collections; support for external records
in search collections, user alerts and baskets (WebSearch,
WebAlert, WebBasket)
*) support for nested parentheses in search query syntax (WebSearch)
*) new refersto/citedby search operators for second-order searches in
citation map (BibRank, WebSearch)
*) numerous improvements to SPIRES query syntax parser (WebSearch)
*) enhancement to search results summaries, e.g. co-author lists on
author pages, e.g. h-index (WebSearch)
*) new support for unAPI, Zotero, OpenSearch, AWS (WebSearch)
*) new phrase and word-pair indexes (BibIndex)
*) new fuzzy author name matching mode (BibIndex)
*) new time-dependent citation ranking family of methods (BibRank)
*) full-text search now shows context snippets (BibFormat)
*) improvements to the basket UI, basket export facility (WebBasket)
*) new support for FCKeditor in submissions and user comments,
possibility to attach files (WebComment, WebSubmit)
*) commenting facility enhanced with rounds and threads (WebComment)
*) new facility to moderate user comments (WebComment)
*) enhanced CLI tool for document file management bringing new
options such as hidden file flag (WebSubmit)
*) numerous improvements to the submission system, e.g. asynchronous
JavaScript upload support, derived document formats, icon
creation, support for automatic conversion of OpenOffice
documents, PDF/A, OCR (WebSubmit)
*) new full-text file metadata reader/writer tool (WebSubmit)
*) new experimental SWORD protocol client application (BibSword)
*) complete rewrite of the record editor using Ajax technology for
faster user operation, with new features such as field templates,
cloning, copy/paste, undo/redo, auto-completion, etc (BibEdit)
*) new multi-record editor to alter many records in one go (BibEdit)
*) new Ajax-based record differ and merger (BibMerge)
*) new fuzzy record matching mode, with possibility to match records
against remote Invenio installations (BibMatch)
*) new circulation and holdings module (BibCirculation)
*) new facility for matching provenance information when uploading
records (BibUpload)
*) new possibility of uploading incoming changes into holding pen
(BibUpload)
*) new batch uploader facility to support uploading of metadata files
and of full-text files either in CLI or over web (BibUpload)
*) new record exporting module supporting e.g. Sitemap and Google
Scholar export methods (BibExport)
*) improvements to the keyword classifier, e.g. author and core
keywords (BibClassify)
*) new facility for external robot-like login method (WebAccess)
*) numerous improvements to the journal creation facility, new
journal `Atlantis Times' demo journal (WebJournal)
*) refactored and improved OAI exporter and harvester (BibHarvest)
*) new taxonomy-based and dynamic-query knowledge base types
(BibKnowledge)
*) possibility to switch on/off user features such as alerts and
baskets based on RBAC rules (WebAccess and other modules)
*) various improvements to task scheduler, for example better
communication with tasks, possibility to run certain bibsched
tasks within given time limit, etc (BibSched)
*) new database dumper for backup purposes (MiscUtil)
*) new plotextractor library for extracting plots from compuscripts,
new figure caption index and the Plots tab (MiscUtil, BibIndex,
Webearch)
*) enhanced reference extrator, e.g. support for DOI, for author name
recognition (MiscUtil)
*) new register emergency feature e.g. to alert admins by SMS in case
the task queue stops (MiscUtil)
*) infrastructure move from mod_python to mod_wsgi, support for
mod_xsendfile (WebStyle and many modules)
*) infrastructure move from jsMath to MathJax (MiscUtil)
*) some notable backward-incompatible changes: removed authentication
methods related to Apache user and group files, changed BibFormat
element's API (BibFormat, many modules)
*) new translations (Afrikaans, Galician, Georgian, Romanian,
Kinyarwanda) plus many translation updates
*) other numerous improvements and bug fixes done in about 1600
commits over Invenio v0.99 series
CDS Invenio v0.99.3 -- released 2010-12-13
------------------------------------------
*) fixed issues in the harvesting daemon when harvesting from more
than one OAI repository (BibHarvest)
*) fixed failure in formatting engine when dealing with
not-yet-existing records (BibFormat)
*) fixed traversal of final URL parts in the URL dispatcher
(WebStyle)
*) improved bibdocfile URL recognition upon upload of MARC files
(BibUpload)
*) fixed bug in admin interface for adding authorizations (WebAccess)
*) keyword extractor is now compatible with rdflib releases older
than 2.3.2 (BibClassify)
*) output of `bibsched status' now shows the queue mode status as
AUTOMATIC or MANUAL to help queue monitoring (BibSched)
CDS Invenio v0.99.2 -- released 2010-10-20
------------------------------------------
*) stricter checking of access to restricted records: in order to
view a restricted record, users are now required to have
authorizations to access all restricted collections the given
record may belong to (WebSearch)
*) strict checking of user query history when setting up email
notification alert, preventing URL mangling attempts (WebAlert)
*) fixed possible Unix signal conflicts for tasks performing I/O
operations or running external processes, relevant notably to
full-text indexing of remote files (BibSched)
*) fixed full-text indexing and improved handling of files of
`unexpected' extensions (BibIndex, WebSubmit)
*) streaming of files of `unknown' MIME type now defaults to
application/octet-stream (WebSubmit)
*) fixed addition of new MARC fields in the record editor (BibEdit)
*) fixed issues in full-text file attachment via MARC (BibUpload)
*) fixed authaction CLI client (WebAccess)
*) ... plus other minor fixes and improvements
CDS Invenio v0.99.1 -- released 2008-07-10
------------------------------------------
*) search engine syntax now supports parentheses (WebSearch)
*) search engine syntax now supports SPIRES query language
(WebSearch)
*) strict respect for per-collection sort options on the search
results pages (WebSearch)
*) improved parsing of search query with respect to non-existing
field terms (WebSearch)
*) fixed "any collection" switch on the search results page
(WebSearch)
*) added possibility for progressive display of detailed record page
tabs (WebSearch)
*) added support for multi-page RSS output (WebSearch)
*) new search engine summarizer module with the cite summary output
format (WebSearch, BibRank)
*) "cited by" links are now generated only when needed (WebSearch)
*) new experimental comprehensive author page (WebSearch)
*) stemming for many indexes is now enabled by default (BibIndex)
*) new intelligent journal index (BibIndex)
*) new logging of missing citations (BibRank)
*) citation indexer and searcher improvements and caching (BibRank)
*) new low-level task submission facility (BibSched)
*) new options in bibsched task monitor: view task options, log and
error files; prune task to a history table; extended status
reporting; failed tasks now need acknowledgement in order to
restart the queue (BibSched)
*) safer handling of task sleeping and waking up (BibSched)
*) new experimental support for task priorities and concurrent task
execution (BibSched)
*) improved user-configured browser language matching (MiscUtil)
*) new default behaviour not differentiating between guest users;
this removes a need to keep sessions/uids for guests and robots
(WebSession)
*) optimized sessions and collecting external user information (WebSession)
*) improved logging conflicts for external vs internal users
(WebAccess)
*) improved Single Sign-On session preservation (WebAccess)
*) new 'become user' debugging facility for admins (WebAccess)
*) new bibdocfile CLI tool to manipulate full-text files archive
(WebSubmit)
*) optimized redirection of old URLs (WebSubmit)
*) new icon creation tool in the submission input chain (WebSubmit)
*) improved full-text file migration tool (WebSubmit)
*) improved stamping of full-text files (WebSubmit)
*) new approval-related end-submission functions (WebSubmit)
*) comments and descriptions of full-text files are now kept also in
bibdoc tables, not only in MARC; they are synchronized during
bibupload (WebSubmit, BibUpload)
*) fixed navigation in public baskets (WebBasket)
*) added detailed record page link to basket records (WebBasket)
*) new removal of HTML markup in alert notification emails (WebAlert)
*) improved OAI harvester logging and handling (BibHarvest)
*) improved error checking (BibConvert)
*) improvements to the record editing tool: subfield order change,
repetitive subfields; improved record locking features;
configurable per-collection curators (BibEdit)
*) fully refactored WebJournal module (WebJournal)
*) new RefWorks output format, thanks to Theodoros Theodoropoulos
(BibFormat)
*) fixed keyword detection tool's output; deactivated taxonomy
compilation (BibClassify)
*) new /stats URL for administrators (WebStat)
*) better filtering of unused translations (WebStyle)
*) updated French, Italian, Norwegian and Swedish translations;
updated Japanese translation (thanks to Makiko Matsumoto and Takao
Ishigaki); updated Greek translation (thanks to Theodoros
Theodoropoulos); new Hungarian translation (thanks to Eva Papp)
*) ... plus many other minor bug fixes and improvements
CDS Invenio v0.99.0 -- released 2008-03-27
------------------------------------------
*) new Invenio configuration language, new inveniocfg configuration
tool permitting more runtime changes and enabling separate local
customizations (MiscUtil)
*) phased out WML dependency everywhere (all modules)
*) new common RSS cache implementation (WebSearch)
*) improved access control to the detailed record pages (WebSearch)
*) when searching non-existing collections, do not revert to
searching in public Home anymore (WebSearch)
*) strict calculation of number of hits per multiple collections
(WebSearch)
*) propagate properly language environment in browse pages, thanks to
Ferran Jorba (WebSearch)
*) search results sorting made accentless, thanks to Ferran Jorba
(WebSearch)
*) new OpenURL interface (WebSearch)
*) added new search engine API argument to limit searches to record
creation/modification dates and times instead of hitherto creation
dates only (WebSearch)
*) do not allow HTTP POST method for searches to prevent hidden
mining (WebSearch)
*) added alert and RSS teaser for search engine queries (WebSearch)
*) new optimized index structure for fast integer bit vector
operations, leading to significant indexing time improvements
(MiscUtil, BibIndex, WebSearch)
*) new tab-based organisation of detailed record pages, with new URL
schema (/record/1/usage) and related CSS changes (BibFormat,
MiscUtil, WebComment, WebSearch, WebStyle, WebSubmit)
*) phased out old PHP based code; migration to Python-based output
formats recommended (BibFormat, WebSubmit)
*) new configurability to show/hide specific output formats for
specific collections (BibFormat, WebSearch)
*) new configurability to have specific stemming settings for
specific indexes (BibIndex, WebSearch)
*) optional removal of LaTeX markup for indexer (BibIndex, WebSearch)
*) performance optimization for webcoll and optional arguments to
refresh only parts of collection cache (WebSearch)
*) optional verbosity argument propagation to the output formatter
(BibFormat, WebSearch)
*) new convenient reindex option to the indexer (BibIndex)
*) fixed problem with indexing of some lengthy UTF-8 accented names,
thanks to Theodoros Theodoropoulos for reporting the problem
(BibIndex)
*) fixed full-text indexing of HTML pages (BibIndex)
*) new Stemmer module dependency, fixes issues on 64-bit systems
(BibIndex)
*) fixed download history graph display (BibRank)
*) improved citation ranking and history graphs, introduced
self-citation distinction, added new demo records (BibRank)
*) fixed range redefinition and output message printing problems in
the ranking indexer, thanks to Mike Marino (BibRank)
*) new XSLT output formatter support; phased out old BFX formats
(BibFormat)
*) I18N output messages are now translated in the output formatter
templates (BibFormat)
*) formats fixed to allow multiple author affiliations (BibFormat)
*) improved speed of the record output reformatter in case of large
sets (BibFormat)
*) support for displaying LaTeX formulas via JavaScript (BibFormat)
*) new and improved output formatter elements (BibFormat)
*) new escaping modes for format elements (BibFormat)
*) output format template editor cache and element dependency
checker improvements (BibFormat)
*) output formatter speed improvements in PHP-compatible mode
(BibFormat)
*) new demo submission configuration and approval workflow examples
(WebSubmit)
*) new submission full-text file stamper utility (WebSubmit)
*) new submission icon-creation utility (WebSubmit)
*) separated submission engine and database layer (WebSubmit)
*) submission functions can now access user information (WebSubmit)
*) implemented support for restricted icons (WebSubmit, WebAccess)
*) new full-text file URL and cleaner storage facility; requires file
names to be unique within a given record (WebSearch, WebSubmit)
*) experimental release of the complex approval and refereeing
workflow (WebSubmit)
*) new end-submission functions to move files to storage space
(WebSubmit)
*) added support for MD5 checking of full-text files (WebSubmit)
*) improved behaviour of the submission system with respect to the
browser "back" button (WebSubmit)
*) removed support for submission "cookies" (WebSubmit)
*) flexible report number generation during submission (WebSubmit)
*) added support for optional filtering step in the OAI harvesting
chain (BibHarvest)
*) new text-oriented converter functions IFDEFP, JOINMULTILINES
(BibConvert)
*) selective harvesting improvements, sets, non-standard responses,
safer resumption token handling (BibHarvest)
*) OAI archive configuration improvements: collections retrieval,
multiple set definitions, new clean mode, timezones, and more
(BibHarvest)
*) OAI gateway improvements: XSLT used to produce configurable output
(BibHarvest)
*) added support for "strong tags" that can resist metadata replace
mode (BibUpload)
*) added external OAI ID tag support to the uploader (BibUpload)
*) added support for full-text file transfer during uploading
(BibUpload)
*) preserving full history of all MARCXML versions of a record
(BibEdit, BibUpload)
*) XMLMARC to TextMarc improvements: empty indicators and more
(BibEdit)
*) numerous reference extraction tool improvements: year handling,
LaTeX handling, URLs, journal titles, output methods, and more
(BibEdit)
*) new classification daemon (BibClassify)
*) classification taxonomy caching resulting in speed optimization
(BibClassify)
*) new possibility to define more than one keyword taxonomy per
collection (BibClassify)
*) fixed non-standalone keyword detection, thanks to Annette Holtkamp
(BibClassify)
*) new embedded page generation profiler (WebStyle)
*) new /help pages layout and webdoc formatting tool (WebStyle)
*) new custom style template verification tool (WebStyle)
*) added support for the XML page() output format, suitable for AJAX
interfaces (WebStyle)
*) introduction of navigation menus (WebStyle)
*) general move from HTML to XHTML markup (all modules)
*) fixed alert deletion tool vulnerability (WebAlert)
*) do not advertise baskets/alerts much for guest users; show only
the login link (WebSession)
*) password reset interface improvements (WebSession)
*) new permanent "remember login" mechanism (WebSession, WebAccess)
*) local user passwords are now encrypted (WebSession, WebAccess)
*) new LDAP external authentication plugin (WebAccess)
*) new password reset mechanism using new secure mail cookies and
temporary role membership facilities (WebAccess, WebSession)
*) added support for Single Sign-On Shibboleth based authentication
method (WebAccess)
*) new firewall-like based role definition language, new demo
examples (WebAccess)
*) external authentication and groups improvements: nicknames,
account switching, and more (WebSession, WebAccess)
*) task log viewer integrated in the task monitor (BibSched)
*) new journal creation module (WebJournal)
*) new generic statistic gathering and display facility (WebStat)
*) deployed new common email sending facility (MiscUtil, WebAlert,
WebComment, WebSession, WebSubmit)
*) dropped support for MySQL-4.0, permitting to use clean and strict
UTF-8 storage methods; upgrade of MySQLdb to at least 1.2.1_p2
required (MiscUtil)
*) uncatched exceptions are now being sent by email to the
administrator (MiscUtil, WebStyle)
*) new general garbage collector with a possibility to run via the
task scheduler and a possibility to clean unreferenced
bibliographic values (MiscUtil)
*) new generic SQL and data cacher (MiscUtil)
*) new HTML page validator plugin (MiscUtil)
*) new web test suite running in a real browser (MiscUtil)
*) improved code kwalitee checker (MiscUtil)
*) translation updates: Spanish and Catalan (thanks to Ferran Jorba),
Japanese (Toru Tsuboyama), German (Benedikt Koeppel), Polish
(Zbigniew Szklarz and Zbigniew Leonowicz), Greek (Theodoros
Theodoropoulos), Russian (Yana Osborne), Swedish, Italian, French
*) new translations: Chinese traditional and Chinese simplified
(thanks to Kam-ming Ku)
*) ... plus many other minor bug fixes and improvements
CDS Invenio v0.92.1 -- released 2007-02-20
------------------------------------------
*) new support for external authentication systems (WebSession,
WebAccess)
*) new support for external user groups (WebSession)
*) new experimental version of the reference extraction program
(BibEdit)
*) new optional Greek stopwords list, thanks to Theodoropoulos
Theodoros (BibIndex)
*) new Get_Recid submission function (WebSubmit)
*) new config variable governing the display of the download history
graph (BibRank)
*) started deployment of user preferences (WebSession, WebSearch)
*) split presentation style for "Narrow search", "Focus on" and
"Search also" search interface boxes (WebSearch, WebStyle)
*) updated CERN Indico and KEK external collection searching facility
(WebSearch)
*) fixed search interface portalbox and collection definition
escaping behaviour (WebSearch Admin)
*) fixed problems with external system number and OAI ID matching
(BibUpload)
*) fixed problem with case matching behaviour (BibUpload)
*) fixed problems with basket record display and basket topic change
(WebBasket)
*) fixed output format template attribution behaviour (BibFormat)
*) improved language context propagation in output formats
(BibFormat)
*) improved output format treatment of HTML-aware fields (BibFormat)
*) improved BibFormat migration kit (BibFormat)
*) improved speed and eliminated set duplication of the OAI
repository gateway (BibHarvest)
*) fixed resumption token handling (BibHarvest)
*) improved record editing interface (BibEdit)
*) fixed problem with empty fields treatment (BibConvert)
*) updated Report_Number_Generation submission function to be able to
easily generate report numbers from any submission information
(WebSubmit)
*) fixed problem with submission field value escaping (WebSubmit)
*) fixed problem with submission collection ordering (WebSubmit)
*) fixed BibSched task signal handling inconsistency (BibSched)
*) fixed TEXT versus BLOB database problems for some tables/columns
*) minor updates to the HOWTO Migrate guide and several admin guides
(WebHelp, BibIndex, BibFormat)
*) minor bugfixes to several modules; see ChangeLog for details and
credits
CDS Invenio v0.92.0 -- released 2006-12-22
------------------------------------------
*) previously experimental output formatter in Python improved and
made default (BibFormat)
*) previously experimental new submission admin interface in Python
improved and made default (WebSubmit)
*) new XML-oriented output formatting mode (BibFormat)
*) new export-oriented output formats: EndNote, NLM (BibFormat)
*) RSS 2.0 latest additions feed service (WebSearch, BibFormat)
*) new XML-oriented metadata converter mode (BibConvert)
*) new metadata uploader in Python (BibUpload)
*) new integrated parallel external collection searching (WebSearch)
*) improved document classifier: composite keywords, wildcards, cloud
output (BibClassify)
*) improved UTF-8 fulltext indexing (BibIndex)
*) improved external login authentication subsystem (WebAccess)
*) added possibility to order submission categories (WebSubmit)
*) improved handling of cached search interface page formats,
preferential sort pattern functionality, international collection
names (WebSearch)
*) improved behaviour of OAI harvester: sets, deleted records,
harvested metadata transformation (BibHarvest)
*) improved MARCXML schema compatibility concerning indicators;
updates to the HTML MARC output format (BibEdit, BibUpload,
BibFormat, and other modules)
*) multiple minor bugs fixed thanks to the wider deployment of the
regression test suite (all modules)
*) new translation (Croatian) and several translation updates
(Catalan, Bulgarian, French, Greek, Spanish); thanks to Ferran
Jorba, Beatriu Piera, Alen Vodopijevec, Jasna Marković, Theodoros
Theodoropoulos, and Nikolay Dyankov (see also THANKS file)
*) removed dependency on PHP; not needed anymore
*) full compatibility with MySQL 4.1 and 5.0; upgrade from MySQL 4.0
now recommended
*) full compatibility with FreeBSD and Mac OS X
CDS Invenio v0.90.1 -- released 2006-07-23
------------------------------------------
*) output messages improved and enhanced to become more easily
translatable in various languages (all modules)
*) new translation (Bulgarian) and several updated translations
(Greek, French, Russian, Slovak)
*) respect langugage choice in various web application links
(WebAlert, WebBasket, WebComment, WebSession, WebSubmit)
*) fixed problem with commenting rights in a group-shared basket that
is also a public basket with lesser rights (WebBasket)
*) guest users are now forbidden to share baskets (WebBasket)
*) fixed guest user garbage collection, adapted to the new baskets
schema (WebSession)
*) added possibility to reject group membership requests; sending
informational messages when users are approved/refused by group
administrators (WebSession)
*) experimental release of the new BibFormat in Python (BibFormat)
*) started massive deployment of the regression test suite, checking
availability of all web interface pages (BibEdit, BibFormat,
BibHarvest, BibIndex, BibRank, MiscUtil, WebAccess, WebBasket,
WebComment, WebMessage, WebSearch, WebSession, WebSubmit)
*) updated developer documentation (I18N output messages policy, test
suite policy, coding style)
CDS Invenio v0.90.0 -- released 2006-06-30
------------------------------------------
*) formerly known as CDSware; the application name change clarifies
the relationship with respect to the CDS Sofware Consortium
producing two flagship applications (CDS Indico and Invenio)
*) version number increased to v0.90 in the anticipation of the
forthcoming v1.0 release after all the major codebase changes are
now over
*) new possibility to define user groups (WebGroup)
*) new personal basket organization in topics (WebBasket)
*) new basket sharing among user groups (WebBasket)
*) new open peer reviewing and commenting on documents (WebComment)
*) new user and group web messaging system (WebMessage)
*) new ontology-based document classification system (BibClassify)
*) new WebSubmit Admin (WebSubmit)
*) new record editing web interface (BibEdit)
*) new record matching tool (BibMatch)
*) new OAI repository administration tool (BibHarvest)
*) new OAI periodical harvesting tool (BibHarvest)
*) new web layout templating system (WebStyle)
*) new clean URL schema (e.g. /collection/Theses, /record/1234)
(WebStyle)
*) new BibTeX output format support (BibFormat)
*) new possibility of secure HTTPS authentication while keeping the
rest of the site non-HTTPS (WebSession)
*) new centralized error library (MiscUtil)
*) new gettext-based international translations, with two new beta
translations (Japanese, Polish)
*) new regression testing suite framework (MiscUtil)
*) new all prerequisites are now apt-gettable for Debian "Sarge"
GNU/Linux
*) new full support for Mac OS X
*) ... plus many fixes and changes worth one year of development
CDSware v0.7.1 -- released 2005-05-04
-------------------------------------
*) important bugfix for bibconvert's ``source data in a directory''
mode, as invoked by the web submission system (BibConvert)
*) minor bugfix in the search engine, thanks to Frederic Gobry
(WebSearch)
*) minor bugfix in the WebSearch Admin interface (WebSearch)
*) automatic linking to Google Print in the ``Haven't found what you
were looking for...'' page box (WebSearch)
*) BibFormat Admin Guide cleaned, thanks to Ferran Jorba
*) new Catalan translation, thanks to Ferran Jorba
*) updated Greek and Portuguese translations, thanks to Theodoros
Theodoropoulos and Flávio C. Coelho
*) updated Spanish translation
CDSware v0.7.0 -- released 2005-04-06
-------------------------------------
*) experimental release of the refextract program for automatic
reference extraction from PDF fulltext files (BibEdit)
*) experimental release of the citation and download ranking tools
(BibRank)
*) new module for gathering usage statistics out of Apache log files
(WebStat)
*) new similar-records-navigation tool exploring end-user viewing
habits: "people who viewed this page also viewed" (WebSearch,
BibRank)
*) OAI gateway validated against OAI Repository Explorer (BibHarvest)
*) fixed "records modified since" option for the indexer (BibIndex)
*) collection cache update is done only when the cache is not up to
date (WebSearch) [closing #WebSearch-016]
*) cleanup of user login mechanism (WebSession, WebAccess)
*) fixed uploading of already-existing records in the insertion mode
(BibUpload)
*) fixed submission in UTF-8 languages (WebSubmit)
*) updated HOWTO Run Your Existing CDSware Installation (WebHelp)
*) test suite improvements (WebSearch, BibHarvest, BibRank,
BibConvert)
*) German translation updated and new German stopwords list added,
thanks to Guido Pelzer
*) new Greek and Ukrainian translations, thanks to Theodoros
Theodoropoulos and Vasyl Ostrovskyi
*) all language codes now comply to RFC 1766 and ISO 639
*) numerous other small fixes and improvements, with many
contributions by the EPFL team headed by Frederic Gobry
(BibConvert, BibUpload, WebSearch, WebSubmit, WebSession)
CDSware v0.5.0 -- released 2004-12-17
-------------------------------------
*) new rank engine, featuring word similarity rank method and the
journal impact factor rank demo (BibRank)
*) search engine includes ranking option (WebSearch)
*) record similarity search based on word frequency (WebSearch,
BibRank)
*) stopwords possibility when ranking and indexing (BibRank, BibIndex)
*) stemming possibility when ranking and indexing (BibRank, BibIndex)
*) search engine boolean query processing stages improved (WebSearch)
*) search engine accent matching in phrase searches (WebSearch)
*) regular expression searching mode introduced into the Simple
Search interface too (WebSearch)
*) Search Tips split into a brief Search Tips page and detailed
Search Guide page (WebSearch)
*) improvements to the ``Try your search on'' hints (WebSearch)
*) author search hints introduced (WebSearch)
*) search interface respects title prologue/epilogue portalboxes
(WebSearch)
*) improvements to admin interfaces (WebSearch, BibIndex, BibRank,
WebAccess)
*) basket item ordering problem fixed (WebBasket)
*) access error messages introduced (WebAccess and its clients)
*) new account management to enable/disable guest users and
automatic vs to-be-approved account registration (WebAccess)
*) possibility for temporary read-only access to, and closure of, the
site; useful for backups (WebAccess and its clients)
*) possibility for external authentication login methods (WebAccess)
*) new XML MARC handling library (BibEdit)
*) when uploading, bad XML records are marked as errors (BibUpload)
*) improvements to the submission engine and its admin interface,
thanks to Tiberiu Dondera (WebSubmit)
*) preparations for electronic mail submission feature, not yet
functional (ElmSubmit)
*) added example on MARC usage at CERN (WebHelp)
*) legacy compatibility with MySQL 3.23.x assured (BibUpload)
*) legacy compatibility with Python 2.2 assured (WebSubmit)
*) test suite additions and corrections (BibRank, BibIndex,
WebSearch, BibEdit)
*) French translation fixes, thanks to Eric Grand
*) minor Czech and Slovak translation cleanup
CDSware v0.3.3 (DEVELOPMENT) -- released 2004-07-16
---------------------------------------------------
*) new international phrases, collection and field names; thanks to
Guido, Flavio, Tullio
*) collection international names are now respected by the search
engine and interfaces (WebSearch)
*) field international names are now respected by the search
engine and interfaces (WebSearch)
*) when no hits found in a given collection, do not display all
public hits straight away but only link to them (WebSearch)
*) records marked as DELETED aren't shown anymore in XML MARC and
other formats (WebSearch)
*) detailed record page now features record creation and modification
times (WebSearch)
*) improved XML MARC parsing and cumulative record count in case of
uploading of several files in one go (BibUpload)
*) personal `your admin activities' page introduced (WebSession)
*) added option to fulltext-index local files only (BibIndex)
*) initial release of the BibIndex Admin interface (BibIndex)
*) checking of mandatory selection box definitions (WebSubmit)
*) WebSearch Admin interface cleanup (WebSearch)
*) introducing common test suite infrastructure (WebSearch, BibIndex,
MiscUtil, WebHelp)
*) fixed accent and link problems for photo demo records (MiscUtil)
*) conference title exported via OAI XML DC (BibHarvest)
*) enabled building out of source directory; thanks to Frederic
CDSware v0.3.2 (DEVELOPMENT) -- released 2004-05-12
---------------------------------------------------
*) admin area improved: all the modules have now Admin Guides; some
guides were updated, some are still to be updated (WebHelp,
BibConvert, BibFormat, BibIndex, BibSched, WebAlert, WebSession,
WebSubmit, BibEdit, BibHarvest, BibRank, BibUpload, WebAccess,
WebBasket, WebSearch, WebStyle)
*) initial release of the WebSearch Admin interface (WebSearch)
*) initial release of the BibRank Admin interface (BibRank)
*) search cache expiry after insertion of new records (WebSearch)
*) search engine now does on-the-fly formatting via BibFormat CLI
call to handle restricted site situations (WebSearch)
*) webcoll default verbosity decreased for efficiency (WebSearch)
*) added BibConvert configuration example for converting XML Dublin
Core to XML MARC (BibConvert)
*) BibConvert knowledge base mode extended by various case-sensitive
matching possibilities (BibConvert)
*) fixed various problems with fulltext file names and the submission
from MS Windows platform (WebSubmit)
*) fixed problem with bibupload append mode not updating XML MARC
properly (BibUpload)
*) fixed small problems with the submission interface such as
multiple fields selection (WebSubmit)
*) session revoking and session expiry strengthened (WebSession)
*) page design and style sheet updated to better fit large variety of
browsers (WebStyle)
*) added output format argument for basket display (WebBasket)
*) new Swedish translation and updated German, Russian, and Spanish
translations; thanks to Urban, Guido, Lyuba, and Magaly
*) faster creation of I18N static HTML and PHP files during make
CDSware v0.3.1 (DEVELOPMENT) -- released 2004-03-12
---------------------------------------------------
*) security fix preventing exposure of local configuration variables
by malicious URL crafting (WebSearch, WebSubmit, WebAlert,
WebBasket, WebSession, BibHarvest, MiscUtil)
*) initial release of the ranking engine (BibRank)
*) new guide on HOWTO Run Your CDSware Installation (WebHelp)
*) fixed submit configurations with respect to fulltext links and
metadata tags (WebSubmit, MiscUtil)
*) Your Account personal corner now shows the list and the status
of submissions and approvals (WebSession)
*) uniform help and version number option for CLI executables
(WebSearch, BibSched, BibIndex, BibRank, BibHarvest, BibConvert,
WebAccess, BibFormat, WebSession, WebAlert)
*) uniform technique for on-the-fly formatting of search results via
`hb_' and `hd_' output format parameters (WebSearch)
*) check for presence of pcntl and mysql PHP libraries (BibUpload)
CDSware v0.3.0 (DEVELOPMENT) -- released 2004-03-05
---------------------------------------------------
*) new development branch release (important SQL table changes)
*) introducing a new submission engine and the end-user web
interface (WebSubmit)
*) bibupload is now a BibSched task with new options (BibUpload)
*) BibWords renamed into BibIndex in the view of future phrase
indexing changes (BibIndex)
*) more secure DB server connectivity (BibSched)
*) record matching functionality (BibConvert)
*) character encoding conversion tables (BibConvert)
*) Qualified Dublin Core conversion example (BibConvert)
*) OAI deleted records policy can now be specified (BibHarvest)
*) multi-language collection portalboxes (WebSearch)
*) HTML pages now respect language selections (WebSearch, WebHelp)
*) minor layout changes (WebStyle)
*) updated Russian and other translations
*) ChangeLog is now generated from CVS log messages
*) plus the usual set of bugfixes (see ChangeLog)
CDSware v0.1.2 (DEVELOPMENT) -- released 2003-12-21
---------------------------------------------------
*) development branch release
*) fix BibReformat task launching problem (BibFormat)
*) fix BibTeX -> XML MARC conversion example (BibConvert)
*) updated Spanish translation
CDSware v0.1.1 (DEVELOPMENT) -- released 2003-12-19
---------------------------------------------------
*) development branch release
*) access control engine now used by BibWords, BibFormat (admin and
bibreformat), WebSearch (webcoll), and BibTaskEx
*) access control engine admin guide started (WebAccess)
*) search engine support for sorting by more than one field (WebSearch)
*) more internationalization of the search engine messages (WebSearch)
*) new language: Norwegian (bokmål)
*) simple example for converting BibTeX into XML MARC (BibConvert)
*) new optional --with-python configuration option
*) Python module detection during configure
*) bugfixes: os.tempnam() warning, login page referer, and others
CDSware v0.1.0 (DEVELOPMENT) -- released 2003-12-04
---------------------------------------------------
*) development branch release
*) search engine redesign to yield five times more search performance
for larger sites (WebSearch, BibWords)
*) fulltext indexation of PDF, PostScript, MS Word, MS PowerPoint and
MS Excel files (WebSearch)
*) integrated combined metadata/fulltext/citation search (WebSearch)
*) multi-stage search guidance in cases of no exact match (WebSearch)
*) OAI-PMH harvestor (BibHarvest)
*) bibliographic task scheduler (BibSched)
*) automatic daemon mode of the indexer, the formatter and the
collection cache generator (BibWords, BibFormat, WebSearch)
*) user management and session handling rewrite (WebSession)
*) user personalization, document baskets and notification alert
system (WebBasket, WebAlert)
*) role-based access control engine (WebAccess)
*) internationalization of the interface started (currently with
Czech, German, English, Spanish, French, Italian, Portuguese,
Russian, and Slovak support)
*) web page design update (WebStyle)
*) introduction of programmer-oriented technical documentation corner
(WebHelp)
*) source tree reorganization, mod_python technology adopted for most
of the modules
CDSware v0.0.9 (STABLE) -- released 2002-08-01
----------------------------------------------
*) first "public" alpha release of CDSware
*) recently standardized Library of Congress' MARC XML schema adopted
in all CDSware modules as the new default internal XML file format
(BibConvert, BibFormat, BibUpload, WebSubmit, WebSearch)
*) support for OAI-PMH v2.0 in addition to OAI-PMH v1.1 (WebSearch)
*) search interface now honors multiple output formats per collection
(BibFormat, WebSearch)
*) search interface now honors search fields, search options, and
sort options from the database config tables (WebSearch,
WebSearch Admin)
*) search interface now honors words indexes from the database config
tables (BibWords, WebSearch)
*) easy reformatting of already uploaded bibliographic records via
web admin. tool (BibFormat Admin/Reformat Records)
*) new submission form field type ("response") allowing
greater flexibility (WebSubmit) [thanks to Frank Sudholt]
*) demo site "Atlantis Institute of Science" updated to demonstrate:
Pictures collection of photographs; specific per-collection
formats; references inside Articles and Preprints; "cited by"
search link; published version linking; subject category
searching; search within, search options, sort options in the web
collection pages.
- end of file -
diff --git a/README b/README
index 8ef556514..97e17b137 100644
--- a/README
+++ b/README
@@ -1,68 +1,72 @@
Invenio README
==============
-Invenio enables you to run your own electronic preprint server,
-your own online library catalogue or a digital document system on the
-web. It complies with the Open Archive Initiative metadata harvesting
-protocol and uses MARC21 as its underlying bibliographic standard.
-
-Invenio is developed at, and maintained by, the CERN Document
-Server Software Consortium. At CERN, Invenio installation manages
-over 600 collections of data, consisting of over 800,000 bibliographic
-records, including 400,000 fulltext documents, covering preprints,
-articles, books, journals, photographs, videos, and more. Please
-consult <http://cds.cern.ch/> for details.
+Invenio is a free software suite enabling you to run your own digital
+library or document repository on the web. The technology offered by
+the software covers all aspects of digital library management, from
+document ingestion through classification, indexing, and curation up
+to document dissemination. Invenio complies with standards such as
+the Open Archives Initiative and uses MARC 21 as its underlying
+bibliographic format. The flexibility and performance of Invenio make
+it a comprehensive solution for management of document repositories of
+moderate to large sizes.
+
+Invenio has been originally developed at CERN to run the CERN document
+server, managing over 1,000,000 bibliographic records in high-energy
+physics since 2002, covering articles, books, journals, photos,
+videos, and more. Invenio is nowadays co-developed by an
+international collaboration comprising institutes such as CERN, DESY,
+EPFL, FNAL, SLAC and is being used by many more scientific
+institutions worldwide.
We aim at user friendliness and speed. Among the features are:
- Navigable collection tree
. Documents organised in collections
. Regular and virtual collection trees
- . Customizable portalboxes for each collection
- . At CERN, over 800,000 documents in 500 collections
+ . Customisable portal pages for each collection
+ . At CERN, over 1,000,000 documents in 700 collections
- Powerful search engine
- . Specially designed indexes to provide Google-like search speeds
- for repositories of up to 1,500,000 records
- . Customizable simple and advanced search interfaces
+ . Specially designed indexes to provide fast search speed
+ for repositories of up to 3,000,000 records
+ . Customisable simple and advanced search interfaces
. Combined metadata, fulltext and citation search in one go
. Results clustering by collection
- Flexible metadata
. Standard metadata format (MARC)
. Handling articles, books, theses, photos, videos, museum objects
and more
- . Customizable batch imports and web submission lines
- . Customizable output display and linking rules
+ . Customisable batch import and web submission workflows
+ . Customisable output format display and linking rules
- - Collaborative features and personalization
+ - Collaborative features and personalisation
. User-defined document baskets
- . User-defined automated email notification alerts
- . Basket-sharing within user groups
- . Messaging between users
- . Open peer commentin and reviewing of repository documents and in
- group-shared baskets
-
-The Invenio is free software licenced under terms of GNU General
-Public Licence (GPL). It is provided on an "as is" basis. However,
-please note that a support contract can be arranged with the Invenio
-Development Team in which we, the developers, can help you to install,
-configure, and maintain the system (featuring email and telnet support
-throughout the year), including development of new local features
-should such a need arise. If you are interested in concluding a
-support contract with us, please contact the Invenio Development Team
-at <info@invenio-software.org>.
-
-Invenio runs on Unix-like systems and requires MySQL database
-server and Apache/Python web application server. Please consult the
-INSTALL file for more information.
-
-Good luck, and thanks for choosing Invenio.
+ . User-defined email notification alerts
+ . Personalised RSS queries
+ . Sharing documents of interest in user groups
+ . Peer reviewing and group commenting on documents
+
+Invenio is free software licenced under the terms of the GNU General
+Public Licence (GPL). It is provided on an "as is" basis, in the hope
+that it will be useful, but without any warranty. There is a
+possibility to get commercial support in case of interest.
+
+Invenio runs on Unix-like systems and requires MySQL database server
+and Apache/Python web application server. Please consult the INSTALL
+file for more information.
+
+Good luck and thanks for choosing Invenio.
- Invenio Development Team
- <info@invenio-software.org>
- <http://invenio-software.org/>
+
+ Email: info@invenio-software.org
+ IRC: #invenio on irc.freenode.net
+ Twitter: http://twitter.com/inveniosoftware
+ Github: http://github.com/inveniosoftware
+ URL: http://invenio-software.org
diff --git a/README.rst b/README.rst
new file mode 100644
index 000000000..b0a7401a1
--- /dev/null
+++ b/README.rst
@@ -0,0 +1,75 @@
+================
+ Invenio README
+================
+
+.. image:: https://travis-ci.org/inveniosoftware/invenio.png?branch=master
+ :target: https://travis-ci.org/inveniosoftware/invenio
+
+Invenio is a free software suite enabling you to run your own digital
+library or document repository on the web. The technology offered by
+the software covers all aspects of digital library management, from
+document ingestion through classification, indexing, and curation up
+to document dissemination. Invenio complies with standards such as
+the Open Archives Initiative and uses MARC 21 as its underlying
+bibliographic format. The flexibility and performance of Invenio make
+it a comprehensive solution for management of document repositories of
+moderate to large sizes.
+
+Invenio has been originally developed at CERN to run the CERN document
+server, managing over 1,000,000 bibliographic records in high-energy
+physics since 2002, covering articles, books, journals, photos,
+videos, and more. Invenio is nowadays co-developed by an
+international collaboration comprising institutes such as CERN, DESY,
+EPFL, FNAL, SLAC and is being used by many more scientific
+institutions worldwide.
+
+We aim at user friendliness and speed. Among the features are:
+
+- Navigable collection tree
+
+ - Documents organised in collections
+ - Regular and virtual collection trees
+ - Customisable portal pages for each collection
+ - At CERN, over 1,000,000 documents in 700 collections
+
+- Powerful search engine
+
+ - Specially designed indexes to provide fast search speed
+ for repositories of up to 3,000,000 records
+ - Customisable simple and advanced search interfaces
+ - Combined metadata, fulltext and citation search in one go
+ - Results clustering by collection
+
+- Flexible metadata
+
+ - Standard metadata format (MARC)
+ - Handling articles, books, theses, photos, videos, museum objects
+ and more
+ - Customisable batch import and web submission workflows
+ - Customisable output format display and linking rules
+
+- Collaborative features and personalisation
+
+ - User-defined document baskets
+ - User-defined email notification alerts
+ - Personalised RSS queries
+ - Sharing documents of interest in user groups
+ - Peer reviewing and group commenting on documents
+
+Invenio is free software licenced under the terms of the GNU General
+Public Licence (GPL). It is provided on an "as is" basis, in the hope
+that it will be useful, but without any warranty. There is a
+possibility to get commercial support in case of interest.
+
+Invenio runs on Unix-like systems and requires MySQL database server
+and Apache/Python web application server. Please consult the INSTALL
+file for more information.
+
+Good luck and thanks for choosing Invenio.
+
+| Invenio Development Team
+| Email: info@invenio-software.org
+| IRC: #invenio on irc.freenode.net
+| Twitter: http://twitter.com/inveniosoftware
+| Github: http://github.com/inveniosoftware
+| URL: http://invenio-software.org
diff --git a/RELEASE-NOTES b/RELEASE-NOTES
index 5f7ad94df..aa942efeb 100644
--- a/RELEASE-NOTES
+++ b/RELEASE-NOTES
@@ -1,172 +1,117 @@
--------------------------------------------------------------------
-Invenio v1.1.2 is released
-August 19, 2013
+Invenio v1.1.3 is released
+February 25, 2014
http://invenio-software.org/
--------------------------------------------------------------------
-Invenio v1.1.2 was released on August 19, 2013.
+Invenio v1.1.3 was released on February 25, 2014.
This stable release contains a number of minor fixes and improvements.
-It is recommended to all Invenio sites using v1.1.1 or previous stable
+It is recommended to all Invenio sites using v1.1.2 or previous stable
release series (v0.99, v1.0).
What's new:
-----------
- *) BibAuthorID: fix in name comparisons (#1313 #1314); improvements
- and fixes; improvements, fixes and optimizations; UI and backend
- improvements
+ *) BatchUploader: rights to ::1 for robot upload; avoid
+ tempfile.tempdir redefinition (#1594)
- *) BibCatalog: removal of print statement (#1337)
+ *) BibCatalog: no newlines in subject for RT plugin
- *) BibClassify: escape keywords in tag cloud and MARCXML
+ *) BibDocFile: RHEL6 magic bindings support (#1466)
- *) BibDocFile: better JS washing in web UI; display file upload
- progress (#1020 #1021); display "Restricted" label correctly
- (#1299); fix check-md5 with bibdocfsinfo cache (#1249); fix
- check-md5 with bibdocfsinfo cache (#1249); fix error in calling
- register_download (#1311); handling of exceptions in Md5Folder
- (#1060); revert md5 property patch (#1249); support new magic
- library (#1207)
+ *) BibFormat: fix for BibTeX regression tests; better BibTeX title
+ and collaboration
- *) BibEncode: minor fix in process_batch_job()
+ *) BibRank: temporary file storage in CFG_TMPDIR (#1594)
- *) BibFormat: additional fulltext file display in HB (#1219); checks
- for bibformat bin; fix CLI call to old PHP-based formatter; fixes
- unit tests (#1320); fix for fulltext file format; fix snippets for
- phrase queries (#1201); format_element initialisation fix; passing
- of user_info for Excel format; replacement of CDS Invenio by
- Invenio; setUp/tearDown in unit tests (#1319); skip hidden icons
- in OpenGraph image tag
+ *) BibSword: author MARC tag definition fix
- *) BibIndex: better wording for stemming in admin UI; replacement of
- CDS Invenio by Invenio; synonym indexing speed up (#1484); use
- human friendly index name (#1329)
+ *) BibUpload: FFT replace warning in guide
- *) BibKnowledge: /kb/export 500 error fix; optional memoisation of
- KBR lookups (#1484)
+ *) I18N: PO file update for the release of v1.1.3; PO file update for
+ the release of v1.0.6; PO file update for the release of v0.99.9;
+ collection demo names for new translations
- *) BibMerge: delete cache file on submit
+ *) OAIHarvest: for for bad exception handling
- *) BibSched: bibupload max_priority check; bugfix for high-priority
- monotasks; increases size of monitor columns;
- parse_runtime_limit() fix (#1432); parse_runtime_limit() tests fix
- (#1432)
+ *) OAIRepository: optional support for --notimechange
- *) BibUpload: FMT regression test case fix (#1152); indicators in
- strong tags (#939)
+ *) Travis CI: initial release of configuration
- *) CKEditor: updated to version 3.6.6
+ *) WebSearch: nonexisting record API test case fix (#1692); correct
+ record sums from hosted colls (#1651); space between records in
+ MARC HTML; fix for BibTeX regression tests; field-filtered MARCXML
+ API output (#1591); more complete API regression test suite;
+ get_fieldvalues_alephseq_like() utils; asciification of `oe`
+ grapheme (#1582); bug fix for SPIRES date math search
- *) dateutils: strftime improvement (#1065); strptime for Python-2.4
- compatibility
+ *) WebSession: fix mail cookie expiration (#1596)
- *) errorlib: hiding bibcatalog info in exception body
+ *) WebSubmit: fix for typo in Shared_Functions; optional pdftk
+ regression tests
- *) global: test suite nosification
+ *) dbquery: closes redundant connection
- *) htmlutils: fix single quote escaping; improve js string escaping;
- MathJax 2.1 (#1050)
+ *) git: addition of compile to gitignore; new entry in gitignore
- *) I18N: updates to Catalan and Spanish translations
+ *) global: language value always in link URLs
- *) installation: fix collectiondetailedrecordpagetabs (#1496); fix
- for jQuery hotkeys add-on URL (#1507); fix for MathJax OS X
- install issue (#1455); support for Apache-2.4 (#1552)
+ *) installation: pip requirement version updates; pip requirements;
+ no user prompt for warnings; empty Travis configuration; location
+ of jquery-1.7.1.min.js; location of flot; information about
+ unidecode; fix autotools rsync instructions
- *) inveniocfg: tests runner file closure fix (#1327)
+ *) intbitset: no crash when intbitset is on rhs (#1287)
- *) InvenioConnector: fix for CDS authentication; mechanize dependency
+ *) inveniocfg: fix for mod_headers
- *) inveniogc: consider journal cache subdirs
-
- *) memoiseutils: initial release
-
- *) OAIHarvest: fix path for temporary authorlists; holding-pen UI
- bugfixes (#1401)
-
- *) OAIRepository: CFG_OAI_REPOSITORY_MARCXML_SIZE; no bibupload -n
-
- *) RefExtract: replacement of CDS Invenio by Invenio
-
- *) WebAccess: fix variable parsing in robot auth (#1456); IP-based
- rules and offline user fix (#1233); replacement of CDS Invenio by
- InveniO
-
- *) WebApiKey: renames unit tests to regression tests (#1324)
-
- *) WebAuthorProfile: fix XSS vulnerability
-
- *) WebComment: escape review "title"; escape review "title"
-
- *) WebSearch: 410 HTTP code for deleted records; advanced search
- notification if no hits; better cleaning of word patterns; fix
- infinite synonym lookup cases (#804); handles "find feb 12"
- (#948); nicer browsing of fuzzy indexes (#1348); respect default
- `rg` in Advanced Search; SPIRES date math search fixes (#431
- #948); SPIRES invalid date search fix (#1467); tweaks SPIRES
- two-digit search; unit test disabling for CFG_CERN_SITE; unit test
- update (#1326)
-
- *) WebSession: fix for list of admin activities (#1444); login_method
- changes; unit vs regression test suite cleanup
-
- *) WebStat: use CFG_JOURNAL_TAG instead of 773/909C4 (#546)
-
- *) WebSubmit: new websubmitadmin CLI (#1334); replacement of CDS
- Invenio by Invenio; use PyPDF2 if available
+ *) kwalitee: list comprehensions instead of lambdas; compatibility
+ with pylint 1.0.0
Download:
---------
- <http://invenio-software.org/download/invenio-1.1.2.tar.gz>
- <http://invenio-software.org/download/invenio-1.1.2.tar.gz.md5>
- <http://invenio-software.org/download/invenio-1.1.2.tar.gz.sig>
+ <http://invenio-software.org/download/invenio-1.1.3.tar.gz>
+ <http://invenio-software.org/download/invenio-1.1.3.tar.gz.md5>
+ <http://invenio-software.org/download/invenio-1.1.3.tar.gz.sig>
Installation notes:
-------------------
Please follow the INSTALL file bundled in the distribution tarball.
Upgrade notes:
--------------
Please proceed as follows:
a) Stop your bibsched queue and your Apache server.
b) Install the update:
- $ tar xvfz invenio-1.1.2.tar.gz
- $ cd invenio-1.1.2
+ $ tar xvfz invenio-1.1.3.tar.gz
+ $ cd invenio-1.1.3
$ sudo rsync -a /opt/invenio/etc/ /opt/invenio/etc.OLD/
$ sh /opt/invenio/etc/build/config.nice
$ make
#FIXME: create new python virtual environment, install invenio and check upgrades
$ sudo -u www-data make install
$ sudo rsync -a /opt/invenio/etc.OLD/ \
--exclude invenio-autotools.conf \
- --exclude bibformat/format_templates/Default_HTML_brief.bft \
/opt/invenio/etc/
# Note: if you are upgrading from previous stable release
# series (v0.99 or v1.0), please don't rsync but diff, in order
# to inspect changes and adapt your old configuration to the
# new v1.1 release series.
$ sudo -u www-data /opt/invenio/bin/inveniocfg --update-all
$ sudo -u www-data /opt/invenio/bin/inveniocfg --upgrade
- c) Update your MathJax and CKeditor plugins:
-
- $ sudo -u www-data make install-mathjax-plugin
- $ sudo -u www-data make install-ckeditor-plugin
-
- d) Restart your Apache server and your bibsched queue.
-
-*** NOTE The items below will go into v1.0.6 release announcement ***
+ c) Restart your Apache server and your bibsched queue.
- x) If your records may contain `oe` UTF-8 graphemes, please schedule
+ d) If your records may contain `oe` UTF-8 graphemes, please schedule
reindexing of concerned indexes at a convenient time, for example:
$ sudo -u www-data /opt/invenio/bin/bibindex -w title -f100000 -R
- end of file -
diff --git a/UNINSTALL b/UNINSTALL
index 712f3a6d1..00746648b 100644
--- a/UNINSTALL
+++ b/UNINSTALL
@@ -1,68 +1,72 @@
Invenio UNINSTALL
=================
About
=====
This document specifies how to uninstall the Invenio. Please see
INSTALL and RELEASE-NOTES files if you are looking for information on
how to install or upgrade the system.
Contents
========
1. Quick instructions for the impatient Invenio admin
2. Detailed instructions for the patient Invenio admin
1. Quick instructions for the impatient Invenio admin
==================================================
$ cd /usr/local/src/invenio-0.90
$ make uninstall ## NOTE: This has not been tested yet!!
$ cd ..
$ rm -rf invenio-0.90.tar.gz invenio-0.90
$ mysql -h sqlserver.domain.com -u root -p mysql
mysql> DROP DATABASE invenio;
mysql> REVOKE ALL PRIVILEGES ON invenio.* FROM invenio@webserver.domain.com;
2. Detailed instructions for the patient Invenio admin
===================================================
$ cd /usr/local/src/invenio-0.90
Change to the directory where you have configured and built
the Invenio.
$ make uninstall ## NOTE: This has not been tested yet!!
This will uninstall all the installed web pages, binary
scripts and database utilities from their respective
directories.
Note that it may happen that you will need to clean manually
some directories and files that may have been added under
@prefix@.
$ cd ..
Go to the parent directory.
$ rm -rf invenio-0.90.tar.gz invenio-0.90
Wipe out the downloaded tarball and all the Invenio sources.
$ mysql -h sqlserver.domain.com -u root -p mysql
mysql> DROP DATABASE invenio;
mysql> REVOKE ALL PRIVILEGES ON invenio.* FROM invenio@webserver.domain.com;
Ask your MySQL administrator to execute the above commands
to drop Invenio databases and to revoke access rights
from the Invenio user.
Thanks for testing Invenio. We would be glad to hear from you
about your Invenio experience. Please tell us what you think to
help us to improve the system. Thanks!
- - Invenio Development Group
- <info@invenio-software.org>
- <http://invenio-software.org/>
+ - Invenio Development Team
+
+ Email: info@invenio-software.org
+ IRC: #invenio on irc.freenode.net
+ Twitter: http://twitter.com/inveniosoftware
+ Github: http://github.com/inveniosoftware
+ URL: http://invenio-software.org
diff --git a/config/invenio.conf b/config/invenio.conf
index 90e2ba939..da3e4e420 100644
--- a/config/invenio.conf
+++ b/config/invenio.conf
@@ -1,2572 +1,2613 @@
## This file is part of Invenio.
## Copyright (C) 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
###################################################
## About 'invenio.conf' and 'invenio-local.conf' ##
###################################################
## The 'invenio.conf' file contains the vanilla default configuration
## parameters of a Invenio installation, as coming out of the
## distribution. The file should be self-explanatory. Once installed
## in its usual location (usually /opt/invenio/etc), you could in
## principle go ahead and change the values according to your local
## needs, but this is not advised.
##
## If you would like to customize some of these parameters, you should
## rather create a file named 'invenio-local.conf' in the same
## directory where 'invenio.conf' lives and you should write there
## only the customizations that you want to be different from the
## vanilla defaults.
##
## Here is a realistic, minimalist, yet production-ready example of
## what you would typically put there:
##
## $ cat /opt/invenio/etc/invenio-local.conf
## [Invenio]
## CFG_SITE_NAME = John Doe's Document Server
## CFG_SITE_NAME_INTL_fr = Serveur des Documents de John Doe
## CFG_SITE_URL = http://your.site.com
## CFG_SITE_SECURE_URL = https://your.site.com
## CFG_SITE_ADMIN_EMAIL = john.doe@your.site.com
## CFG_SITE_SUPPORT_EMAIL = john.doe@your.site.com
## CFG_WEBALERT_ALERT_ENGINE_EMAIL = john.doe@your.site.com
## CFG_WEBCOMMENT_ALERT_ENGINE_EMAIL = john.doe@your.site.com
## CFG_WEBCOMMENT_DEFAULT_MODERATOR = john.doe@your.site.com
## CFG_BIBAUTHORID_AUTHOR_TICKET_ADMIN_EMAIL = john.doe@your.site.com
## CFG_BIBCATALOG_SYSTEM_EMAIL_ADDRESS = john.doe@your.site.com
## CFG_DATABASE_HOST = localhost
## CFG_DATABASE_NAME = invenio
## CFG_DATABASE_USER = invenio
## CFG_DATABASE_PASS = my123p$ss
##
## You should override at least the parameters mentioned above and the
## parameters mentioned in the `Part 1: Essential parameters' below in
## order to define some very essential runtime parameters such as the
## name of your document server (CFG_SITE_NAME and
## CFG_SITE_NAME_INTL_*), the visible URL of your document server
## (CFG_SITE_URL and CFG_SITE_SECURE_URL), the email address of the
## local Invenio administrator, comment moderator, and alert engine
## (CFG_SITE_SUPPORT_EMAIL, CFG_SITE_ADMIN_EMAIL, etc), and last but
## not least your database credentials (CFG_DATABASE_*).
##
## The Invenio system will then read both the default invenio.conf
## file and your customized invenio-local.conf file and it will
## override any default options with the ones you have specified in
## your local file. This cascading of configuration parameters will
## ease your future upgrades.
[Invenio]
###################################
## Part 1: Essential parameters ##
###################################
## This part defines essential Invenio internal parameters that
## everybody should override, like the name of the server or the email
## address of the local Invenio administrator.
## CFG_DATABASE_* - specify which MySQL server to use, the name of the
## database to use, and the database access credentials.
CFG_DATABASE_TYPE = mysql
CFG_DATABASE_HOST = localhost
CFG_DATABASE_PORT = 3306
CFG_DATABASE_NAME = invenio
CFG_DATABASE_USER = invenio
CFG_DATABASE_PASS = my123p$ss
## CFG_DATABASE_SLAVE - if you use DB replication, then specify the DB
## slave address credentials. (Assuming the same access rights to the
## DB slave as to the DB master.) If you don't use DB replication,
## then leave this option blank.
CFG_DATABASE_SLAVE =
## CFG_DATABASE_SLAVE_SU_USER - is a special user that is authorized to
## detach/attach the slave to the master, thus allowing consistent dbdump
## on the slave database.
CFG_DATABASE_SLAVE_SU_USER = slaveadmin
CFG_DATABASE_SLAVE_SU_PASS =
## CFG_DATABASE_PASSWORD_FILE - if you want to store your password
## in a separate special file that is only readable by the dbquery user
## specify here the path. The file should contain one user per row,
## with the format:
## username // password
## i.e. with the username, the 4 characters " // ", and the password.
CFG_DATABASE_PASSWORD_FILE =
## CFG_SITE_URL - specify URL under which your installation will be
## visible. For example, use "http://your.site.com". Do not leave
## trailing slash.
CFG_SITE_URL = http://localhost
## CFG_SITE_SECURE_URL - specify secure URL under which your
## installation secure pages such as login or registration will be
## visible. For example, use "https://your.site.com". Do not leave
## trailing slash. If you don't plan on using HTTPS, then you may
## leave this empty.
CFG_SITE_SECURE_URL = https://localhost
## CFG_SITE_NAME -- the visible name of your Invenio installation.
CFG_SITE_NAME = Atlantis Institute of Fictive Science
## CFG_SITE_NAME_INTL -- the international versions of CFG_SITE_NAME
## in various languages. (See also CFG_SITE_LANGS below.)
CFG_SITE_NAME_INTL_en = Atlantis Institute of Fictive Science
CFG_SITE_NAME_INTL_fr = Atlantis Institut des Sciences Fictives
CFG_SITE_NAME_INTL_de = Atlantis Institut der fiktiven Wissenschaft
CFG_SITE_NAME_INTL_es = Instituto de Ciencia Ficticia Atlantis
CFG_SITE_NAME_INTL_ca = Institut Atlantis de Ciència Fictícia
CFG_SITE_NAME_INTL_pt = Instituto Atlantis de Ciência Fictícia
CFG_SITE_NAME_INTL_it = Atlantis Istituto di Scienza Fittizia
CFG_SITE_NAME_INTL_ru = Институт Фиктивных Наук Атлантиды
CFG_SITE_NAME_INTL_sk = Atlantis Inštitút Fiktívnych Vied
CFG_SITE_NAME_INTL_cs = Atlantis Institut Fiktivních Věd
CFG_SITE_NAME_INTL_no = Atlantis Institutt for Fiktiv Vitenskap
CFG_SITE_NAME_INTL_sv = Atlantis Institut för Fiktiv Vetenskap
CFG_SITE_NAME_INTL_el = Ινστιτούτο Φανταστικών Επιστημών Ατλαντίδος
CFG_SITE_NAME_INTL_uk = Інститут вигаданих наук в Атлантісі
CFG_SITE_NAME_INTL_ja = Fictive 科学のAtlantis の協会
CFG_SITE_NAME_INTL_pl = Instytut Fikcyjnej Nauki Atlantis
CFG_SITE_NAME_INTL_bg = Институт за фиктивни науки Атлантис
CFG_SITE_NAME_INTL_hr = Institut Fiktivnih Znanosti Atlantis
CFG_SITE_NAME_INTL_zh_CN = 阿特兰提斯虚拟科学学院
CFG_SITE_NAME_INTL_zh_TW = 阿特蘭提斯虛擬科學學院
CFG_SITE_NAME_INTL_hu = Kitalált Tudományok Atlantiszi Intézete
CFG_SITE_NAME_INTL_af = Atlantis Instituut van Fiktiewe Wetenskap
CFG_SITE_NAME_INTL_gl = Instituto Atlantis de Ciencia Fictive
CFG_SITE_NAME_INTL_ro = Institutul Atlantis al Ştiinţelor Fictive
CFG_SITE_NAME_INTL_rw = Atlantis Ishuri Rikuru Ry'ubuhanga
CFG_SITE_NAME_INTL_ka = ატლანტიდის ფიქტიური მეცნიერების ინსტიტუტი
CFG_SITE_NAME_INTL_lt = Fiktyvių Mokslų Institutas Atlantis
CFG_SITE_NAME_INTL_ar = معهد أطلنطيس للعلوم الافتراضية
CFG_SITE_NAME_INTL_fa = موسسه علوم تخیلی آتلانتیس
## CFG_SITE_LANG -- the default language of the interface: '
CFG_SITE_LANG = en
## CFG_SITE_LANGS -- list of all languages the user interface should
## be available in, separated by commas. The order specified below
## will be respected on the interface pages. A good default would be
## to use the alphabetical order. Currently supported languages
## include Afrikaans, Arabic, Bulgarian, Catalan, Czech, German,
## Georgian, Greek, English, Spanish, Persian (Farsi), French,
## Croatian, Hungarian, Galician, Italian, Japanese, Kinyarwanda,
## Lithuanian, Norwegian, Polish, Portuguese, Romanian, Russian,
## Slovak, Swedish, Ukrainian, Chinese (China), Chinese (Taiwan), so
## that the eventual maximum you can currently select is
## "af,ar,bg,ca,cs,de,el,en,es,fa,fr,hr,gl,ka,it,rw,lt,hu,ja,no,pl,pt,ro,ru,sk,sv,uk,zh_CN,zh_TW".
CFG_SITE_LANGS = af,ar,bg,ca,cs,de,el,en,es,fa,fr,hr,gl,ka,it,rw,lt,hu,ja,no,pl,pt,ro,ru,sk,sv,uk,zh_CN,zh_TW
## CFG_EMAIL_BACKEND -- the backend to use for sending emails. Defaults to
## 'flask.ext.email.backends.smtp.Mail' if CFG_MISCUTIL_SMTP_HOST and
## CFG_MISCUTIL_SMTP_PORT are set. Possible values are:
## - flask.ext.email.backends.console.Mail
## - flask.ext.email.backends.dummy.Mail
## - flask.ext.email.backends.filebased.Mail
## - flask.ext.email.backends.locmem.Mail
## - flask.ext.email.backends.smtp.Mail
## - invenio.ext.email.backends.console_adminonly.Mail
## - invenio.ext.email.backends.smtp_adminonly.Mail
## * sends email only to the CFG_SITE_ADMIN_EMAIL address using SMTP
CFG_EMAIL_BACKEND = flask.ext.email.backends.smtp.Mail
## CFG_SITE_SUPPORT_EMAIL -- the email address of the support team for
## this installation:
CFG_SITE_SUPPORT_EMAIL = info@invenio-software.org
## CFG_SITE_ADMIN_EMAIL -- the email address of the 'superuser' for
## this installation. Enter your email address below and login with
## this address when using Invenio inistration modules. You
## will then be automatically recognized as superuser of the system.
CFG_SITE_ADMIN_EMAIL = info@invenio-software.org
## CFG_SITE_EMERGENCY_EMAIL_ADDRESSES -- list of email addresses to
## which an email should be sent in case of emergency (e.g. bibsched
## queue has been stopped because of an error). Configuration
## dictionary allows for different recipients based on weekday and
## time-of-day. Example:
##
## CFG_SITE_EMERGENCY_EMAIL_ADDRESSES = {
## 'Sunday 22:00-06:00': '0041761111111@email2sms.foo.com',
## '06:00-18:00': 'team-in-europe@foo.com,0041762222222@email2sms.foo.com',
## '18:00-06:00': 'team-in-usa@foo.com',
## '*': 'john.doe.phone@foo.com'}
##
## If you want the emergency email notifications to always go to the
## same address, just use the wildcard line in the above example.
CFG_SITE_EMERGENCY_EMAIL_ADDRESSES = {}
## CFG_SITE_ADMIN_EMAIL_EXCEPTIONS -- set this to 0 if you do not want
## to receive any captured exception via email to CFG_SITE_ADMIN_EMAIL
## address. Captured exceptions will still be available in
## var/log/invenio.err file. Set this to 1 if you want to receive
## some of the captured exceptions (this depends on the actual place
## where the exception is captured). Set this to 2 if you want to
## receive all captured exceptions.
CFG_SITE_ADMIN_EMAIL_EXCEPTIONS = 1
## CFG_SITE_RECORD -- what is the URI part representing detailed
## record pages? We recommend to leave the default value `record'
## unchanged.
CFG_SITE_RECORD = record
## CFG_SITE_SECRET_KEY --- which secret key should we use? This should be set
## to a random value per Invenio installation and must be kept secret, as it is
## used to protect against e.g. cross-site request forgery and other is the
## basis of other security measures in Invenio. A random value can be generated
## using the following command:
## python -c "import os;import re;print re.escape(os.urandom(24).__repr__()[1:-1])"
CFG_SITE_SECRET_KEY =
## CFG_ERRORLIB_RESET_EXCEPTION_NOTIFICATION_COUNTER_AFTER -- set this to
## the number of seconds after which to reset the exception notification
## counter. A given repetitive exception is notified via email with a
## logarithmic strategy: the first time it is seen it is sent via email,
## then the second time, then the fourth, then the eighth and so forth.
## If the number of seconds elapsed since the last time it was notified
## is greater than CFG_ERRORLIB_RESET_EXCEPTION_NOTIFICATION_COUNTER_AFTER
## then the internal counter is reset in order not to have exception
## notification become more and more rare.
CFG_ERRORLIB_RESET_EXCEPTION_NOTIFICATION_COUNTER_AFTER = 14400
+## CFG_ERRORLIB_SENTRY_URI -- optionally, if you use Sentry for logging.
+## Sentry notifies you when your users experience errors.
+## Enables logging of exceptions in invenio to given Sentry server.
+CFG_ERRORLIB_SENTRY_URI =
+
## CFG_CERN_SITE -- do we want to enable CERN-specific code?
## Put "1" for "yes" and "0" for "no".
CFG_CERN_SITE = 0
## CFG_INSPIRE_SITE -- do we want to enable INSPIRE-specific code?
## Put "1" for "yes" and "0" for "no".
CFG_INSPIRE_SITE = 0
## CFG_ADS_SITE -- do we want to enable ADS-specific code?
## Put "1" for "yes" and "0" for "no".
CFG_ADS_SITE = 0
## CFG_FLASK_CACHE_TYPE -- do we want to enable any cache engine?
## 'null', 'redis' or your own e.g. 'invenio.cache.my_cache_engine'
## NOTE: If you disable cache engine it WILL affect some
## functionality such as 'search facets'.
CFG_FLASK_CACHE_TYPE = null
## CFG_FLASK_DISABLED_BLUEPRINTS -- do we want to prevent certain blueprints from
## being loaded?
CFG_FLASK_DISABLED_BLUEPRINTS =
## CFG_FLASK_SERVE_STATIC_FILES -- do we want Flask to serve static files?
## Normally Apache serves static files, but during development and if you are
## using the Werkzeug standalone development server, you can set this flag to
## "1", to enable static file serving.
CFG_FLASK_SERVE_STATIC_FILES = 0
+## CFG_SCOAP3_SITE -- do we want to enable SCOAP3-specific code?
+## Put "1" for "yes" and "0" for "no".
+CFG_SCOAP3_SITE = 0
+
## Now you can tune whether to integrate with external authentication providers
## through the OpenID and OAuth protocols.
## The following variables let you fine-tune which authentication
## providers you want to authorize. You can override here most of
## the variables in lib/invenio/access_control_config.py.
## In particular you can put in these variable the consumer_key and
## consumer_secret of the desired services.
## Note: some providers don't supply an mail address.
## If you choose them, the users will be registered with a temporary email address.
## CFG_OPENID_PROVIDERS -- Comma-separated list of providers you want to enable
## through the OpenID protocol.
## E.g.: CFG_OPENID_PROVIDERS = google,yahoo,aol,wordpress,myvidoop,openid,verisign,myopenid,myspace,livejournal,blogger
CFG_OPENID_PROVIDERS =
## CFG_OAUTH1_PROVIDERS -- Comma-separated list of providers you want to enable
## through the OAuth1 protocol.
## Note: OAuth1 is in general deprecated in favour of OAuth2.
## E.g.: CFG_OAUTH1_PROVIDERS = twitter,linkedin,flickr,
CFG_OAUTH1_PROVIDERS =
## CFG_OAUTH2_PROVIDERS -- Comma-separated list of providers you want to enable
## through the OAuth2 protocol.
## Note: if you enable the "orcid" provider the full profile of the user
## in Orcid will be imported.
## E.g.: CFG_OAUTH2_PROVIDERS = facebook,yammer,foursquare,googleoauth2,instagram,orcid
-CFG_OAUTH2_PROVIDERS = orcid
+CFG_OAUTH2_PROVIDERS =
## CFG_OPENID_CONFIGURATIONS -- Mapping of special parameter to configure the
## desired OpenID providers. Use this variable to override out-of-the-box
## parameters already set in lib/python/invenio/access_control_config.py.
## E.g.: CFG_OPENID_CONFIGURATIONS = {'google': {
## 'identifier': 'https://www.google.com/accounts/o8/id',
## 'trust_email': True}}
CFG_OPENID_CONFIGURATIONS = {}
## CFG_OAUTH1_CONFIGURATIONS -- Mapping of special parameter to configure the
## desired OAuth1 providers. Use this variable to override out-of-the-box
## parameters already set in lib/python/invenio/access_control_config.py.
## E.g.: CFG_OAUTH1_CONFIGURATIONS = {'linkedin': {
## 'consumer_key' : 'MY_LINKEDIN_CONSUMER_KEY',
## 'consumer_secret' : 'MY_LINKEDIN_CONSUMER_SECRET'}}
CFG_OAUTH1_CONFIGURATIONS = {}
## CFG_OAUTH2_CONFIGURATIONS -- Mapping of special parameter to configure the
## desired OAuth2 providers. Use this variable to override out-of-the-box
## parameters already set in lib/python/invenio/access_control_config.py.
## E.g.: CFG_OAUTH2_CONFIGURATIONS = {'orcid': {
## 'consumer_key' : 'MY_ORCID_CONSUMER_KEY',
## 'consumer_secret' : 'MY_ORCID_CONSUMER_SECRET'}}
CFG_OAUTH2_CONFIGURATIONS = {}
## CFG_PROPAGATE_EXCEPTIONS -- Tells register_exception to propagate the
## exception instead of catching it and sending an email. Mostly for
## development, we have custom exceptions handlers that are handier than
## emails.
CFG_PROPAGATE_EXCEPTIONS = 0
################################
## Part 2: Web page style ##
################################
## The variables affecting the page style. The most important one is
## the 'template skin' you would like to use and the obfuscation mode
## for your email addresses. Please refer to the WebStyle Admin Guide
## for more explanation. The other variables are listed here mostly
## for backwards compatibility purposes only.
## CFG_WEBSTYLE_TEMPLATE_SKIN -- what template skin do you want to
## use?
CFG_WEBSTYLE_TEMPLATE_SKIN = default
## CFG_WEBSTYLE_EMAIL_ADDRESSES_OBFUSCATION_MODE. How do we "protect"
## email addresses from undesired automated email harvesters? This
## setting will not affect 'support' and 'admin' emails.
## NOTE: there is no ultimate solution to protect against email
## harvesting. All have drawbacks and can more or less be
## circumvented. Choose you preferred mode ([t] means "transparent"
## for the user):
## -1: hide all emails.
## [t] 0 : no protection, email returned as is.
## foo@example.com => foo@example.com
## 1 : basic email munging: replaces @ by [at] and . by [dot]
## foo@example.com => foo [at] example [dot] com
## [t] 2 : transparent name mangling: characters are replaced by
## equivalent HTML entities.
## foo@example.com => &#102;&#111;&#111;&#64;&#101;&#120;&#97;&#109;&#112;&#108;&#101;&#46;&#99;&#111;&#109;
## [t] 3 : javascript insertion. Requires Javascript enabled on client
## side.
## 4 : replaces @ and . characters by gif equivalents.
## foo@example.com => foo<img src="at.gif" alt=" [at] ">example<img src="dot.gif" alt=" [dot] ">com
CFG_WEBSTYLE_EMAIL_ADDRESSES_OBFUSCATION_MODE = 2
## (deprecated) CFG_WEBSTYLE_CDSPAGEBOXLEFTTOP -- eventual global HTML
## left top box:
CFG_WEBSTYLE_CDSPAGEBOXLEFTTOP =
## (deprecated) CFG_WEBSTYLE_CDSPAGEBOXLEFTBOTTOM -- eventual global
## HTML left bottom box:
CFG_WEBSTYLE_CDSPAGEBOXLEFTBOTTOM =
## (deprecated) CFG_WEBSTYLE_CDSPAGEBOXRIGHTTOP -- eventual global
## HTML right top box:
CFG_WEBSTYLE_CDSPAGEBOXRIGHTTOP =
## (deprecated) CFG_WEBSTYLE_CDSPAGEBOXRIGHTBOTTOM -- eventual global
## HTML right bottom box:
CFG_WEBSTYLE_CDSPAGEBOXRIGHTBOTTOM =
## CFG_WEBSTYLE_HTTP_STATUS_ALERT_LIST -- when certain HTTP status
## codes are raised to the WSGI handler, the corresponding exceptions
## and error messages can be sent to the system administrator for
## inspecting. This is useful to detect and correct errors. The
## variable represents a comma-separated list of HTTP statuses that
## should alert admin. Wildcards are possible. If the status is
## followed by an "r", it means that a referer is required to exist
## (useful to distinguish broken known links from URL typos when 404
## errors are raised).
CFG_WEBSTYLE_HTTP_STATUS_ALERT_LIST = 404r,400,5*,41*
## CFG_WEBSTYLE_HTTP_USE_COMPRESSION -- whether to enable deflate
## compression of your HTTP/HTTPS connections. This will affect the Apache
## configuration snippets created by inveniocfg --create-apache-conf and
## the OAI-PMH Identify response.
CFG_WEBSTYLE_HTTP_USE_COMPRESSION = 0
## CFG_WEBSTYLE_REVERSE_PROXY_IPS -- if you are setting a multinode
## environment where an HTTP proxy such as mod_proxy is sitting in
## front of the Invenio web application and is forwarding requests to
## worker nodes, set here the the list of IP addresses of the allowed
## HTTP proxies. This is needed in order to avoid IP address spoofing
## when worker nodes are also available on the public Internet and
## might receive forged HTTP requests. Only HTTP requests coming from
## the specified IP addresses will be considered as forwarded from a
## reverse proxy. E.g. set this to '123.123.123.123'.
CFG_WEBSTYLE_REVERSE_PROXY_IPS =
##################################
## Part 3: WebSearch parameters ##
##################################
## This section contains some configuration parameters for WebSearch
## module. Please note that WebSearch is mostly configured on
## run-time via its WebSearch Admin web interface. The parameters
## below are the ones that you do not probably want to modify very
## often during the runtime. (Note that you may modify them
## afterwards too, though.)
## CFG_WEBSEARCH_SEARCH_CACHE_SIZE -- do you want to enable search
## caching in global search cache engine (e.g. Redis)? This cache is
## used mainly for "next/previous page" functionality, but it caches
## "popular" user queries too if more than one user happen to search
## for the same thing. Note that if you disable the search caching
## features like "facets" will not work. We recommend a value to be
## kept at CFG_WEBSEARCH_SEARCH_CACHE_SIZE = 1.
CFG_WEBSEARCH_SEARCH_CACHE_SIZE = 1
## CFG_WEBSEARCH_SEARCH_CACHE_TIMEOUT -- how long should we keep a
## search result in the cache. The value should be more than 0 and
## the unit is second. [600 s = 10 minutes]
CFG_WEBSEARCH_SEARCH_CACHE_TIMEOUT = 600
## CFG_WEBSEARCH_FIELDS_CONVERT -- if you migrate from an older
## system, you may want to map field codes of your old system (such as
## 'ti') to Invenio/MySQL ("title"). Use Python dictionary syntax
## for the translation table, e.g. {'wau':'author', 'wti':'title'}.
## Usually you don't want to do that, and you would use empty dict {}.
CFG_WEBSEARCH_FIELDS_CONVERT = {}
## CFG_WEBSEARCH_LIGHTSEARCH_PATTERN_BOX_WIDTH -- width of the
## search pattern window in the light search interface, in
## characters. CFG_WEBSEARCH_LIGHTSEARCH_PATTERN_BOX_WIDTH = 60
CFG_WEBSEARCH_LIGHTSEARCH_PATTERN_BOX_WIDTH = 60
## CFG_WEBSEARCH_SIMPLESEARCH_PATTERN_BOX_WIDTH -- width of the search
## pattern window in the simple search interface, in characters.
CFG_WEBSEARCH_SIMPLESEARCH_PATTERN_BOX_WIDTH = 40
## CFG_WEBSEARCH_ADVANCEDSEARCH_PATTERN_BOX_WIDTH -- width of the
## search pattern window in the advanced search interface, in
## characters.
CFG_WEBSEARCH_ADVANCEDSEARCH_PATTERN_BOX_WIDTH = 30
## CFG_WEBSEARCH_NB_RECORDS_TO_SORT -- how many records do we still
## want to sort? For higher numbers we print only a warning and won't
## perform any sorting other than default 'latest records first', as
## sorting would be very time consuming then. We recommend a value of
## not more than a couple of thousands.
CFG_WEBSEARCH_NB_RECORDS_TO_SORT = 1000
## CFG_WEBSEARCH_CALL_BIBFORMAT -- if a record is being displayed but
## it was not preformatted in the "HTML brief" format, do we want to
## call BibFormatting on the fly? Put "1" for "yes" and "0" for "no".
## Note that "1" will display the record exactly as if it were fully
## preformatted, but it may be slow due to on-the-fly processing; "0"
## will display a default format very fast, but it may not have all
## the fields as in the fully preformatted HTML brief format. Note
## also that this option is active only for old (PHP) formats; the new
## (Python) formats are called on the fly by default anyway, since
## they are much faster. When usure, please set "0" here.
CFG_WEBSEARCH_CALL_BIBFORMAT = 0
## CFG_WEBSEARCH_USE_ALEPH_SYSNOS -- do we want to make old SYSNOs
## visible rather than MySQL's record IDs? You may use this if you
## migrate from a different e-doc system, and you store your old
## system numbers into 970__a. Put "1" for "yes" and "0" for
## "no". Usually you don't want to do that, though.
CFG_WEBSEARCH_USE_ALEPH_SYSNOS = 0
## CFG_WEBSEARCH_I18N_LATEST_ADDITIONS -- Put "1" if you want the
## "Latest Additions" in the web collection pages to show
## internationalized records. Useful only if your brief BibFormat
## templates contains internationalized strings. Otherwise put "0" in
## order not to slow down the creation of latest additions by WebColl.
CFG_WEBSEARCH_I18N_LATEST_ADDITIONS = 0
## CFG_WEBSEARCH_INSTANT_BROWSE -- the number of records to display
## under 'Latest Additions' in the web collection pages.
CFG_WEBSEARCH_INSTANT_BROWSE = 10
## CFG_WEBSEARCH_INSTANT_BROWSE_RSS -- the number of records to
## display in the RSS feed.
CFG_WEBSEARCH_INSTANT_BROWSE_RSS = 25
## CFG_WEBSEARCH_RSS_I18N_COLLECTIONS -- comma-separated list of
## collections that feature an internationalized RSS feed on their
## main seach interface page created by webcoll. Other collections
## will have RSS feed using CFG_SITE_LANG.
CFG_WEBSEARCH_RSS_I18N_COLLECTIONS =
## CFG_WEBSEARCH_RSS_TTL -- number of minutes that indicates how long
## a feed cache is valid.
CFG_WEBSEARCH_RSS_TTL = 360
## CFG_WEBSEARCH_RSS_MAX_CACHED_REQUESTS -- maximum number of request kept
## in cache. If the cache is filled, following request are not cached.
CFG_WEBSEARCH_RSS_MAX_CACHED_REQUESTS = 1000
## CFG_WEBSEARCH_AUTHOR_ET_AL_THRESHOLD -- up to how many author names
## to print explicitely; for more print "et al". Note that this is
## used in default formatting that is seldomly used, as usually
## BibFormat defines all the format. The value below is only used
## when BibFormat fails, for example.
CFG_WEBSEARCH_AUTHOR_ET_AL_THRESHOLD = 3
## CFG_WEBSEARCH_NARROW_SEARCH_SHOW_GRANDSONS -- whether to show or
## not collection grandsons in Narrow Search boxes (sons are shown by
## default, grandsons are configurable here). Use 0 for no and 1 for
## yes.
CFG_WEBSEARCH_NARROW_SEARCH_SHOW_GRANDSONS = 1
## CFG_WEBSEARCH_CREATE_SIMILARLY_NAMED_AUTHORS_LINK_BOX -- shall we
## create help links for Ellis, Nick or Ellis, Nicholas and friends
## when Ellis, N was searched for? Useful if you have one author
## stored in the database under several name formats, namely surname
## comma firstname and surname comma initial cataloging policy. Use 0
## for no and 1 for yes.
CFG_WEBSEARCH_CREATE_SIMILARLY_NAMED_AUTHORS_LINK_BOX = 1
## CFG_WEBSEARCH_USE_MATHJAX_FOR_FORMATS -- MathJax is a JavaScript
## library that renders (La)TeX mathematical formulas in the client
## browser. This parameter must contain a comma-separated list of
## output formats for which to apply the MathJax rendering, for example
## "hb,hd". If the list is empty, MathJax is disabled.
CFG_WEBSEARCH_USE_MATHJAX_FOR_FORMATS =
## CFG_WEBSEARCH_EXTERNAL_COLLECTION_SEARCH_TIMEOUT -- when searching
## external collections (e.g. SPIRES, CiteSeer, etc), how many seconds
## do we wait for reply before abandonning?
CFG_WEBSEARCH_EXTERNAL_COLLECTION_SEARCH_TIMEOUT = 5
## CFG_WEBSEARCH_EXTERNAL_COLLECTION_SEARCH_MAXRESULTS -- how many
## results do we fetch?
CFG_WEBSEARCH_EXTERNAL_COLLECTION_SEARCH_MAXRESULTS = 10
## CFG_WEBSEARCH_SPLIT_BY_COLLECTION -- do we want to split the search
## results by collection or not? Use 0 for not, 1 for yes.
CFG_WEBSEARCH_SPLIT_BY_COLLECTION = 1
## CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS -- the default number of
## records to display per page in the search results pages.
CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS = 10
## CFG_WEBSEARCH_MAX_RECORDS_IN_GROUPS -- in order to limit denial of
## service attacks the total number of records per group displayed as a
## result of a search query will be limited to this number. Only the superuser
## queries will not be affected by this limit.
CFG_WEBSEARCH_MAX_RECORDS_IN_GROUPS = 200
## CFG_WEBSEARCH_SHOW_COMMENT_COUNT -- do we want to show the 'N comments'
## links on the search engine pages? (useful only when you have allowed
## commenting)
CFG_WEBSEARCH_SHOW_COMMENT_COUNT = 1
## CFG_WEBSEARCH_SHOW_REVIEW_COUNT -- do we want to show the 'N reviews'
## links on the search engine pages? (useful only when you have allowed
## reviewing)
CFG_WEBSEARCH_SHOW_REVIEW_COUNT = 1
## CFG_WEBSEARCH_FULLTEXT_SNIPPETS_GENERATOR -- how do we want to generate
## fulltext? Can be generated by 'native' Invenio or 'SOLR'
CFG_WEBSEARCH_FULLTEXT_SNIPPETS_GENERATOR = native
## CFG_WEBSEARCH_FULLTEXT_SNIPPETS -- how many full-text snippets do
## we want to display for full-text searches? If you want to specify
## different values for different document status types, please add
## more items into this dictionary. (Unless specified, the empty
## value will be used as default.) This is useful if you have
## restricted files of different types with various restrictions on
## what we can show.
CFG_WEBSEARCH_FULLTEXT_SNIPPETS = {
'': 4,
}
## CFG_WEBSEARCH_FULLTEXT_SNIPPETS_CHARS -- what is the maximum size
## of a snippet to display around the pattern found in the full-text?
## If you want to specify different values for different document
## status types, please add more items into this dictionary. (Unless
## specified, the empty value will be used as default.) This is
## useful if you have restricted files of different types with various
## restrictions on what we can show.
CFG_WEBSEARCH_FULLTEXT_SNIPPETS_CHARS = {
'': 100,
}
## CFG_WEBSEARCH_WILDCARD_LIMIT -- some of the queries, wildcard
## queries in particular (ex: cern*, a*), but also regular expressions
## (ex: [a-z]+), may take a long time to respond due to the high
## number of hits. You can limit the number of terms matched by a
## wildcard by setting this variable. A negative value or zero means
## that none of the queries will be limited (which may be wanted by
## also prone to denial-of-service kind of attacks).
CFG_WEBSEARCH_WILDCARD_LIMIT = 50000
## CFG_WEBSEARCH_SYNONYM_KBRS -- defines which knowledge bases are to
## be used for which index in order to provide runtime synonym lookup
## of user-supplied terms, and what massaging function should be used
## upon search pattern before performing the KB lookup. (Can be one
## of `exact', 'leading_to_comma', `leading_to_number'.)
CFG_WEBSEARCH_SYNONYM_KBRS = {
'journal': ['SEARCH-SYNONYM-JOURNAL', 'leading_to_number'],
}
## CFG_SOLR_URL -- optionally, you may use Solr to serve full-text
## queries and ranking. If so, please specify the URL of your Solr instance.
## Example: http://localhost:8983/solr (default solr port)
CFG_SOLR_URL =
## CFG_XAPIAN_ENABLED -- optionally, you may use Xapian to serve full-text
## queries and ranking. If so, please enable it: 1 = enabled
CFG_XAPIAN_ENABLED =
## CFG_WEBSEARCH_PREV_NEXT_HIT_LIMIT -- specify the limit when
## the previous/next/back hit links are to be displayed on detailed record pages.
## In order to speeding up list manipulations, if a search returns lots of hits,
## more than this limit, then do not loose time calculating next/previous/back
## hits at all, but display page directly without these.
## Note also that Invenio installations that do not like
## to have the next/previous hit link functionality would be able to set this
## variable to zero and not see anything.
CFG_WEBSEARCH_PREV_NEXT_HIT_LIMIT = 1000
## CFG_WEBSEARCH_PREV_NEXT_HIT_FOR_GUESTS -- Set this to 0 if you want
## to disable the previous/next/back hit link functionality for guests
## users.
## Since the previous/next/back hit link functionality is causing the allocation
## of user session in the database even for guests users, it might be useful to
## be able to disable it e.g. when your site is bombarded by web request
## (a.k.a. Slashdot effect).
CFG_WEBSEARCH_PREV_NEXT_HIT_FOR_GUESTS = 1
## CFG_WEBSEARCH_VIEWRESTRCOLL_POLICY -- when a record belongs to more than one
## restricted collection, if the viewrestcoll policy is set to "ALL" (default)
## then the user must be authorized to all the restricted collections, in
## order to be granted access to the specific record. If the policy is set to
## "ANY", then the user need to be authorized to only one of the collections
## in order to be granted access to the specific record.
CFG_WEBSEARCH_VIEWRESTRCOLL_POLICY = ANY
## CFG_WEBSEARCH_SPIRES_SYNTAX -- variable to configure the use of the
## SPIRES query syntax in searches. Values: 0 = SPIRES syntax is
## switched off; 1 = leading 'find' is required; 9 = leading 'find' is
## not required (leading SPIRES operator, space-operator-space, etc
## are also accepted).
CFG_WEBSEARCH_SPIRES_SYNTAX = 1
## CFG_WEBSEARCH_DISPLAY_NEAREST_TERMS -- when user search does not
## return any direct result, what do we want to display? Set to 0 in
## order to display a generic message about search returning no hits.
## Set to 1 in order to display list of nearest terms from the indexes
## that may match user query. Note: this functionality may be slow,
## so you may want to disable it on bigger sites.
CFG_WEBSEARCH_DISPLAY_NEAREST_TERMS = 1
## CFG_WEBSEARCH_DETAILED_META_FORMAT -- the output format to use for
## detailed meta tags containing metadata as configured in the tag
## table. Default output format should be 'hdm', included. This
## format will be included in the header of /record/ pages. For
## efficiency this format should be pre-cached with BibReformat. See
## also CFG_WEBSEARCH_ENABLE_GOOGLESCHOLAR and
## CFG_WEBSEARCH_ENABLE_GOOGLESCHOLAR.
CFG_WEBSEARCH_DETAILED_META_FORMAT = hdm
## CFG_WEBSEARCH_ENABLE_GOOGLESCHOLAR -- decides if meta tags for
## Google Scholar shall be included in the detailed record page
## header, when using the standard formatting templates/elements. See
## also CFG_WEBSEARCH_DETAILED_META_FORMAT and
## CFG_WEBSEARCH_ENABLE_OPENGRAPH. When this variable is changed and
## output format defined in CFG_WEBSEARCH_DETAILED_META_FORMAT is
## cached, a bibreformat must be run for the cached records.
CFG_WEBSEARCH_ENABLE_GOOGLESCHOLAR = True
## CFG_WEBSEARCH_ENABLE_OPENGRAPH -- decides if meta tags for the Open
## Graph protocol shall be included in the detailed record page
## header, when using the standard formatting templates/elements. See
## also CFG_WEBSEARCH_DETAILED_META_FORMAT and
## CFG_WEBSEARCH_ENABLE_GOOGLESCHOLAR. When this variable is changed
## and output format defined in CFG_WEBSEARCH_DETAILED_META_FORMAT is
## cached, a bibreformat must be run for the cached records. Note that
## enabling Open Graph produces invalid XHTML/HTML5 markup.
CFG_WEBSEARCH_ENABLE_OPENGRAPH = False
## CFG_WEBSEARCH_CITESUMMARY_SCAN_THRESHOLD -- switches to filter a presorted
## self-citations list when # of recids is bigger than this
CFG_WEBSEARCH_CITESUMMARY_SCAN_THRESHOLD = 20000
## CFG_WEBSEARCH_COLLECTION_NAMES_SEARCH -- decides whether search for
## collection name is enabled (1), disabled (-1) or enabled only for
## the home collection (0), enabled for all collections including
## those not attached to the collection tree (2). This requires the
## CollectionNameSearchService search services to be enabled.
CFG_WEBSEARCH_COLLECTION_NAMES_SEARCH = 0
#######################################
## Part 4: BibHarvest OAI parameters ##
#######################################
## This part defines parameters for the Invenio OAI gateway.
## Useful if you are running Invenio as OAI data provider.
## CFG_OAI_ID_FIELD -- OAI identifier MARC field:
CFG_OAI_ID_FIELD = 909COo
## CFG_OAI_SET_FIELD -- OAI set MARC field:
CFG_OAI_SET_FIELD = 909COp
## CFG_OAI_SET_FIELD -- previous OAI set MARC field:
CFG_OAI_PREVIOUS_SET_FIELD = 909COq
## CFG_OAI_DELETED_POLICY -- OAI deletedrecordspolicy
## (no/transient/persistent):
CFG_OAI_DELETED_POLICY = persistent
## CFG_OAI_ID_PREFIX -- OAI identifier prefix:
CFG_OAI_ID_PREFIX = atlantis.cern.ch
## CFG_OAI_SAMPLE_IDENTIFIER -- OAI sample identifier:
CFG_OAI_SAMPLE_IDENTIFIER = oai:atlantis.cern.ch:123
## CFG_OAI_IDENTIFY_DESCRIPTION -- description for the OAI Identify verb:
CFG_OAI_IDENTIFY_DESCRIPTION = <description>
<eprints xmlns="http://www.openarchives.org/OAI/1.1/eprints"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.openarchives.org/OAI/1.1/eprints
http://www.openarchives.org/OAI/1.1/eprints.xsd">
<content>
<URL>%(CFG_SITE_URL)s</URL>
</content>
<metadataPolicy>
<text>Free and unlimited use by anybody with obligation to refer to original record</text>
</metadataPolicy>
<dataPolicy>
<text>Full content, i.e. preprints may not be harvested by robots</text>
</dataPolicy>
<submissionPolicy>
<text>Submission restricted. Submitted documents are subject of approval by OAI repository admins.</text>
</submissionPolicy>
</eprints>
</description>
## CFG_OAI_LOAD -- OAI number of records in a response:
CFG_OAI_LOAD = 500
## CFG_OAI_EXPIRE -- OAI resumptionToken expiration time:
CFG_OAI_EXPIRE = 90000
## CFG_OAI_SLEEP -- service unavailable between two consecutive
## requests for CFG_OAI_SLEEP seconds:
CFG_OAI_SLEEP = 2
## CFG_OAI_METADATA_FORMATS -- mapping between accepted metadataPrefixes and
## the corresponding output format to use, its schema and its metadataNamespace.
CFG_OAI_METADATA_FORMATS = {
'marcxml': ('XOAIMARC', 'http://www.openarchives.org/OAI/1.1/dc.xsd', 'http://purl.org/dc/elements/1.1/'),
'oai_dc': ('XOAIDC', 'http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd', 'http://www.loc.gov/MARC21/slim'),
}
## CFG_OAI_FRIENDS -- list of OAI baseURL of friend repositories. See:
## <http://www.openarchives.org/OAI/2.0/guidelines-friends.htm>
CFG_OAI_FRIENDS = http://cds.cern.ch/oai2d,http://openaire.cern.ch/oai2d,http://export.arxiv.org/oai2
## The following subfields are a completition to
## CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG. If CFG_OAI_PROVENANCE_BASEURL_SUBFIELD is
## set for a record, then the corresponding field is considered has being
## harvested via OAI-PMH
## CFG_OAI_PROVENANCE_BASEURL_SUBFIELD -- baseURL of the originDescription or a
## record
CFG_OAI_PROVENANCE_BASEURL_SUBFIELD = u
## CFG_OAI_PROVENANCE_DATESTAMP_SUBFIELD -- datestamp of the originDescription
## or a record
CFG_OAI_PROVENANCE_DATESTAMP_SUBFIELD = d
## CFG_OAI_PROVENANCE_METADATANAMESPACE_SUBFIELD -- metadataNamespace of the
## originDescription or a record
CFG_OAI_PROVENANCE_METADATANAMESPACE_SUBFIELD = m
## CFG_OAI_PROVENANCE_ORIGINDESCRIPTION_SUBFIELD -- originDescription of the
## originDescription or a record
CFG_OAI_PROVENANCE_ORIGINDESCRIPTION_SUBFIELD = d
## CFG_OAI_PROVENANCE_HARVESTDATE_SUBFIELD -- harvestDate of the
## originDescription or a record
CFG_OAI_PROVENANCE_HARVESTDATE_SUBFIELD = h
## CFG_OAI_PROVENANCE_ALTERED_SUBFIELD -- altered flag of the
## originDescription or a record
CFG_OAI_PROVENANCE_ALTERED_SUBFIELD = t
## CFG_OAI_FAILED_HARVESTING_STOP_QUEUE -- when harvesting OAI sources
## fails, shall we report an error with the task and stop BibSched
## queue, or simply wait for the next run of the task? A value of 0
## will stop the task upon errors, 1 will let the queue run if the
## next run of the oaiharvest task can safely recover the failure
## (this means that the queue will stop if the task is not set to run
## periodically)
CFG_OAI_FAILED_HARVESTING_STOP_QUEUE = 1
## CFG_OAI_FAILED_HARVESTING_EMAILS_ADMIN -- when
## CFG_OAI_FAILED_HARVESTING_STOP_QUEUE is set to leave the queue
## running upon errors, shall we send an email to admin to notify
## about the failure?
CFG_OAI_FAILED_HARVESTING_EMAILS_ADMIN = True
## NOTE: the following parameters are experimental
## -----------------------------------------------------------------------------
## CFG_OAI_RIGHTS_FIELD -- MARC field dedicated to storing Copyright information
CFG_OAI_RIGHTS_FIELD = 542__
## CFG_OAI_RIGHTS_HOLDER_SUBFIELD -- MARC subfield dedicated to storing the
## Copyright holder information
CFG_OAI_RIGHTS_HOLDER_SUBFIELD = d
## CFG_OAI_RIGHTS_DATE_SUBFIELD -- MARC subfield dedicated to storing the
## Copyright date information
CFG_OAI_RIGHTS_DATE_SUBFIELD = g
## CFG_OAI_RIGHTS_URI_SUBFIELD -- MARC subfield dedicated to storing the URI
## (URL or URN, more detailed statement about copyright status) information
CFG_OAI_RIGHTS_URI_SUBFIELD = u
## CFG_OAI_RIGHTS_CONTACT_SUBFIELD -- MARC subfield dedicated to storing the
## Copyright holder contact information
CFG_OAI_RIGHTS_CONTACT_SUBFIELD = e
## CFG_OAI_RIGHTS_STATEMENT_SUBFIELD -- MARC subfield dedicated to storing the
## Copyright statement as presented on the resource
CFG_OAI_RIGHTS_STATEMENT_SUBFIELD = f
## CFG_OAI_LICENSE_FIELD -- MARC field dedicated to storing terms governing
## use and reproduction (license)
CFG_OAI_LICENSE_FIELD = 540__
## CFG_OAI_LICENSE_TERMS_SUBFIELD -- MARC subfield dedicated to storing the
## Terms governing use and reproduction, e.g. CC License
CFG_OAI_LICENSE_TERMS_SUBFIELD = a
## CFG_OAI_LICENSE_PUBLISHER_SUBFIELD -- MARC subfield dedicated to storing the
## person or institute imposing the license (author, publisher)
CFG_OAI_LICENSE_PUBLISHER_SUBFIELD = b
## CFG_OAI_LICENSE_URI_SUBFIELD -- MARC subfield dedicated to storing the URI
## URI
CFG_OAI_LICENSE_URI_SUBFIELD = u
##------------------------------------------------------------------------------
###################################
## Part 5: BibDocFile parameters ##
###################################
## This section contains some configuration parameters for BibDocFile
## module.
## CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_DOCTYPES -- this is the list of
## doctypes (like 'Main' or 'Additional') and their description that admins
## can choose from when adding new files via the Document File Manager
## admin interface.
## - When no value is provided, admins cannot add new
## file (they can only revise/delete/add format)
## - When a single value is given, it is used as
## default doctype for all new documents
##
## Order is relevant
## Eg:
## [('main', 'Main document'), ('additional', 'Figure, schema. etc')]
CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_DOCTYPES = [
('Main', 'Main document'),
('LaTeX', 'LaTeX'),
('Source', 'Source'),
('Additional', 'Additional File'),
('Audio', 'Audio file'),
('Video', 'Video file'),
('Script', 'Script'),
('Data', 'Data'),
('Figure', 'Figure'),
('Schema', 'Schema'),
('Graph', 'Graph'),
('Image', 'Image'),
('Drawing', 'Drawing'),
('Slides', 'Slides')]
## CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_RESTRICTIONS -- this is the
## list of restrictions (like 'Restricted' or 'No Restriction') and their
## description that admins can choose from when adding or revising files.
## Restrictions can then be configured at the level of WebAccess.
## - When no value is provided, no restriction is
## applied
## - When a single value is given, it is used as
## default resctriction for all documents.
## - The first value of the list is used as default
## restriction if the user if not given the
## choice of the restriction. Order is relevant
##
## Eg:
## [('', 'No restriction'), ('restr', 'Restricted')]
CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_RESTRICTIONS = [
('', 'Public'),
('restricted', 'Restricted')]
## CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_MISC -- set here the other
## default flags and attributes to tune the Document File Manager admin
## interface.
## See the docstring of bibdocfile_managedocfiles.create_file_upload_interface
## to have a description of the available parameters and their syntax.
## In general you will rarely need to change this variable.
CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_MISC = {
'can_revise_doctypes': ['*'],
'can_comment_doctypes': ['*'],
'can_describe_doctypes': ['*'],
'can_delete_doctypes': ['*'],
'can_keep_doctypes': ['*'],
'can_rename_doctypes': ['*'],
'can_add_format_to_doctypes': ['*'],
'can_restrict_doctypes': ['*'],
}
## CFG_BIBDOCFILE_FILESYSTEM_BIBDOC_GROUP_LIMIT -- the fulltext
## documents are stored under "/opt/invenio/var/data/files/gX/Y"
## directories where X is 0,1,... and Y stands for bibdoc ID. Thusly
## documents Y are grouped into directories X and this variable
## indicates the maximum number of documents Y stored in each
## directory X. This limit is imposed solely for filesystem
## performance reasons in order not to have too many subdirectories in
## a given directory.
CFG_BIBDOCFILE_FILESYSTEM_BIBDOC_GROUP_LIMIT = 5000
## CFG_BIBDOCFILE_ADDITIONAL_KNOWN_FILE_EXTENSIONS -- a comma-separated
## list of document extensions not listed in Python standard mimetype
## library that should be recognized by Invenio.
CFG_BIBDOCFILE_ADDITIONAL_KNOWN_FILE_EXTENSIONS = hpg,link,lis,llb,mat,mpp,msg,docx,docm,xlsx,xlsm,xlsb,pptx,pptm,ppsx,ppsm
## CFG_BIBDOCFILE_ADDITIONAL_KNOWN_MIMETYPES -- a mapping of additional
## mimetypes that could be served or have to be recognized by this instance
## of Invenio (this is useful in order to patch old versions of the
## mimetypes Python module).
CFG_BIBDOCFILE_ADDITIONAL_KNOWN_MIMETYPES = {
"application/xml-dtd": ".dtd",
}
## CFG_BIBDOCFILE_PREFERRED_MIMETYPES_MAPPING -- the mimetype Python
## provides a mapping between mimetypes and file extensions.
## Unfortunately this is a one to many mapping and the system
## have to pick one extension for a given mimetype.
## This parameter let you specify the actual desired
## mapping that is going to override the mimetype one.
CFG_BIBDOCFILE_PREFERRED_MIMETYPES_MAPPING = {
'application/msword': '.doc',
'application/octet-stream': '.bin',
'application/postscript': '.ps',
'application/vnd.ms-excel': '.xls',
'application/vnd.ms-powerpoint': '.ppt',
'application/x-gtar-compressed': '.tgz',
'application/xhtml+xml': '.xhtml',
'application/xml': '.xml',
'audio/mpeg': '.mp3',
'audio/ogg': '.ogg',
'image/jpeg': '.jpeg',
'image/svg+xml': '.svg',
'image/tiff': '.tiff',
'message/rfc822': '.eml',
'text/calendar': '.ics',
'text/plain': '.txt',
'video/mpeg': '.mpeg',
}
## CFG_BIBDOCFILE_DESIRED_CONVERSIONS -- a dictionary having as keys
## a format and as values the corresponding list of desired converted
## formats.
CFG_BIBDOCFILE_DESIRED_CONVERSIONS = {
'pdf' : ('pdf;pdfa', ),
'ps.gz' : ('pdf;pdfa', ),
'djvu' : ('pdf', ),
'sxw': ('doc', 'odt', 'pdf;pdfa', ),
'docx' : ('doc', 'odt', 'pdf;pdfa', ),
'doc' : ('odt', 'pdf;pdfa', 'docx'),
'rtf' : ('pdf;pdfa', 'odt', ),
'odt' : ('pdf;pdfa', 'doc', ),
'pptx' : ('ppt', 'odp', 'pdf;pdfa', ),
'ppt' : ('odp', 'pdf;pdfa', 'pptx'),
'sxi': ('odp', 'pdf;pdfa', ),
'odp' : ('pdf;pdfa', 'ppt', ),
'xlsx' : ('xls', 'ods', 'csv'),
'xls' : ('ods', 'csv'),
'ods' : ('xls', 'xlsx', 'csv'),
'sxc': ('xls', 'xlsx', 'csv'),
'tiff' : ('pdf;pdfa', ),
'tif' : ('pdf;pdfa', ),}
## CFG_BIBDOCFILE_USE_XSENDFILE -- if your web server supports
## XSendfile header, you may want to enable this feature in order for
## to Invenio tell the web server to stream files for download (after
## proper authorization checks) by web server's means. This helps to
## liberate Invenio worker processes from being busy with sending big
## files to clients. The web server will take care of that. Note:
## this feature is still somewhat experimental. Note: when enabled
## (set to 1), then you have to also regenerate Apache vhost conf
## snippets (inveniocfg --update-config-py --create-apache-conf).
CFG_BIBDOCFILE_USE_XSENDFILE = 0
## CFG_BIBDOCFILE_MD5_CHECK_PROBABILITY -- a number between 0 and
## 1 that indicates probability with which MD5 checksum will be
## verified when streaming bibdocfile-managed files. (0.1 will cause
## the check to be performed once for every 10 downloads)
CFG_BIBDOCFILE_MD5_CHECK_PROBABILITY = 0.1
## CFG_BIBDOCFILE_BEST_FORMATS_TO_EXTRACT_TEXT_FROM -- a comma-separated
## list of document extensions in decrescent order of preference
## to suggest what is considered the best format to extract text from.
CFG_BIBDOCFILE_BEST_FORMATS_TO_EXTRACT_TEXT_FROM = ('txt', 'html', 'xml', 'odt', 'doc', 'docx', 'djvu', 'pdf', 'ps', 'ps.gz')
## CFG_BIBDOCFILE_ENABLE_BIBDOCFSINFO_CACHE -- whether to use the
## database table bibdocfsinfo as reference for filesystem
## information. The default is 0. Switch this to 1
## after you have run bibdocfile --fix-bibdocfsinfo-cache
## or on an empty system.
CFG_BIBDOCFILE_ENABLE_BIBDOCFSINFO_CACHE = 0
## CFG_BIBDOCFILE_AFS_VOLUME_PATTERN -- If documents are going to be stored
## on the AFS filesystem (e.g. in the case of the CDS and Inspire projects),
## this is the pattern to be used to create a volumes to be mounted when
## allocating new gXX directories. (e.g. p.cds.%s or p.inspire.%s).
## Note: this parameter is considered only if either CFG_CERN_SITE or
## CFG_INSPIRE_SITE is set to 1.
## See: CERN specific afs_admin man page for more information.
CFG_BIBDOCFILE_AFS_VOLUME_PATTERN = p.invenio.%s
## CFG_BIBDOCFILE_AFS_VOLUME_QUOTA -- If documents are going to be stored on
## the AFS filesystem (see CFG_BIBDOCFILE_AFS_VOLUME_PATTERN), this is the
## initial quota automatically allocated when creating new volumes for gXX
## directories.
## Note: this parameter is considered only if either CFG_CERN_SITE or
## CFG_INSPIRE_SITE is set to 1.
## See: CERN specific afs_admin man page for more information.
CFG_BIBDOCFILE_AFS_VOLUME_QUOTA = 10000000
## CFG_OPENOFFICE_SERVER_HOST -- the host where an OpenOffice Server is
## listening to. If localhost an OpenOffice server will be started
## automatically if it is not already running.
## Note: if you set this to an empty value this will disable the usage of
## OpenOffice for converting documents.
## If you set this to something different than localhost you'll have to take
## care to have an OpenOffice server running on the corresponding host and
## to install the same OpenOffice release both on the client and on the server
## side.
## In order to launch an OpenOffice server on a remote machine, just start
## the usual 'soffice' executable in this way:
## $> soffice -headless -nologo -nodefault -norestore -nofirststartwizard \
## .. -accept=socket,host=HOST,port=PORT;urp;StarOffice.ComponentContext
CFG_OPENOFFICE_SERVER_HOST = localhost
## CFG_OPENOFFICE_SERVER_PORT -- the port where an OpenOffice Server is
## listening to.
CFG_OPENOFFICE_SERVER_PORT = 2002
## CFG_OPENOFFICE_USER -- the user that will be used to launch the OpenOffice
## client. It is recommended to set this to a user who don't own files, like
## e.g. 'nobody'. You should also authorize your Apache server user to be
## able to become this user, e.g. by adding to your /etc/sudoers the following
## line:
## "apache ALL=(nobody) NOPASSWD: ALL"
## provided that apache is the username corresponding to the Apache user.
## On some machine this might be apache2 or www-data.
CFG_OPENOFFICE_USER = nobody
+## CFG_ICON_CREATION_FORMAT_MAPPINGS -- when using BibTasklet
+## 'bst_create_icons' to prepare icons, which format should be used to
+## create the icons for each input format/extension? You can specify
+## for each file type (eg, tiff, psd, etc.) in which formats its icons
+## should be created. Use keyword "*" to map any source extension not
+## covered by other rules. For eg:
+## {'eps': ['png', 'jpg'],
+## 'pdf': ['jpg'],
+## '*': ['png']}
+## would create png and jpg icons for any eps input file, jpg-only
+## icons for all PDFs, and png icons for any other supported file. if
+## "*" is dropped, the icons would be created in the same format as
+## the main file (if possible).
+CFG_ICON_CREATION_FORMAT_MAPPINGS = {'*': ['jpg']}
+
#################################
## Part 6: BibIndex parameters ##
#################################
## This section contains some configuration parameters for BibIndex
## module. Please note that BibIndex is mostly configured on run-time
## via its BibIndex Admin web interface. The parameters below are the
## ones that you do not probably want to modify very often during the
## runtime.
## CFG_BIBINDEX_FULLTEXT_INDEX_LOCAL_FILES_ONLY -- when fulltext indexing, do
## you want to index locally stored files only, or also external URLs?
## Use "0" to say "no" and "1" to say "yes".
CFG_BIBINDEX_FULLTEXT_INDEX_LOCAL_FILES_ONLY = 1
## (deprecated) CFG_BIBINDEX_REMOVE_STOPWORDS -- configuration moved to
## DB, variable kept here just for backwards compatibility purposes.
CFG_BIBINDEX_REMOVE_STOPWORDS =
## CFG_BIBINDEX_CHARS_ALPHANUMERIC_SEPARATORS -- characters considered as
## alphanumeric separators of word-blocks inside words. You probably
## don't want to change this.
CFG_BIBINDEX_CHARS_ALPHANUMERIC_SEPARATORS = \!\"\#\$\%\&\'\(\)\*\+\,\-\.\/\:\;\<\=\>\?\@\[\\\]\^\_\`\{\|\}\~
## CFG_BIBINDEX_CHARS_PUNCTUATION -- characters considered as punctuation
## between word-blocks inside words. You probably don't want to
## change this.
CFG_BIBINDEX_CHARS_PUNCTUATION = \.\,\:\;\?\!\"
## (deprecated) CFG_BIBINDEX_REMOVE_HTML_MARKUP -- now in database
CFG_BIBINDEX_REMOVE_HTML_MARKUP = 0
## (deprecated) CFG_BIBINDEX_REMOVE_LATEX_MARKUP -- now in database
CFG_BIBINDEX_REMOVE_LATEX_MARKUP = 0
## CFG_BIBINDEX_MIN_WORD_LENGTH -- minimum word length allowed to be added to
## index. The terms smaller then this amount will be discarded.
## Useful to keep the database clean, however you can safely leave
## this value on 0 for up to 1,000,000 documents.
CFG_BIBINDEX_MIN_WORD_LENGTH = 0
## CFG_BIBINDEX_URLOPENER_USERNAME and CFG_BIBINDEX_URLOPENER_PASSWORD --
## access credentials to access restricted URLs, interesting only if
## you are fulltext-indexing files located on a remote server that is
## only available via username/password. But it's probably better to
## handle this case via IP or some convention; the current scheme is
## mostly there for demo only.
CFG_BIBINDEX_URLOPENER_USERNAME = mysuperuser
CFG_BIBINDEX_URLOPENER_PASSWORD = mysuperpass
## CFG_INTBITSET_ENABLE_SANITY_CHECKS --
## Enable sanity checks for integers passed to the intbitset data
## structures. It is good to enable this during debugging
## and to disable this value for speed improvements.
CFG_INTBITSET_ENABLE_SANITY_CHECKS = False
## CFG_BIBINDEX_PERFORM_OCR_ON_DOCNAMES -- regular expression that matches
## docnames for which OCR is desired (set this to .* in order to enable
## OCR in general, set this to empty in order to disable it.)
CFG_BIBINDEX_PERFORM_OCR_ON_DOCNAMES = scan-.*
## CFG_BIBINDEX_SPLASH_PAGES -- key-value mapping where the key corresponds
## to a regular expression that matches the URLs of the splash pages of
## a given service and the value is a regular expression of the set of URLs
## referenced via <a> tags in the HTML content of the splash pages that are
## referring to documents that need to be indexed.
## NOTE: for backward compatibility reasons you can set this to a simple
## regular expression that will directly be used as the unique key of the
## map, with corresponding value set to ".*" (in order to match any URL)
CFG_BIBINDEX_SPLASH_PAGES = {
"http://documents\.cern\.ch/setlink\?.*": ".*",
"http://ilcagenda\.linearcollider\.org/subContributionDisplay\.py\?.*|http://ilcagenda\.linearcollider\.org/contributionDisplay\.py\?.*": "http://ilcagenda\.linearcollider\.org/getFile\.py/access\?.*|http://ilcagenda\.linearcollider\.org/materialDisplay\.py\?.*",
}
## CFG_BIBINDEX_AUTHOR_WORD_INDEX_EXCLUDE_FIRST_NAMES -- do we want
## the author word index to exclude first names to keep only last
## names? If set to True, then for the author `Bernard, Denis', only
## `Bernard' will be indexed in the word index, not `Denis'. Note
## that if you change this variable, you have to re-index the author
## index via `bibindex -w author -R'.
CFG_BIBINDEX_AUTHOR_WORD_INDEX_EXCLUDE_FIRST_NAMES = False
## (deprecated) CFG_BIBINDEX_SYNONYM_KBRS -- configuration moved to
## DB, variable kept here just for backwards compatibility purposes.
CFG_BIBINDEX_SYNONYM_KBRS = {}
#######################################
## Part 7: Access control parameters ##
#######################################
## This section contains some configuration parameters for the access
## control system. Please note that WebAccess is mostly configured on
## run-time via its WebAccess Admin web interface. The parameters
## below are the ones that you do not probably want to modify very
## often during the runtime. (If you do want to modify them during
## runtime, for example te deny access temporarily because of backups,
## you can edit access_control_config.py directly, no need to get back
## here and no need to redo the make process.)
## CFG_ACCESS_CONTROL_LEVEL_SITE -- defines how open this site is.
## Use 0 for normal operation of the site, 1 for read-only site (all
## write operations temporarily closed), 2 for site fully closed,
## 3 for also disabling any database connection.
## Useful for site maintenance.
CFG_ACCESS_CONTROL_LEVEL_SITE = 0
## CFG_ACCESS_CONTROL_LEVEL_GUESTS -- guest users access policy. Use
## 0 to allow guest users, 1 not to allow them (all users must login).
CFG_ACCESS_CONTROL_LEVEL_GUESTS = 0
## CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS -- account registration and
## activation policy. When 0, users can register and accounts are
## automatically activated. When 1, users can register but admin must
## activate the accounts. When 2, users cannot register nor update
## their email address, only admin can register accounts. When 3,
## users cannot register nor update email address nor password, only
## admin can register accounts. When 4, the same as 3 applies, nor
## user cannot change his login method. When 5, then the same as 4
## applies, plus info about how to get an account is hidden from the
## login page.
CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS = 0
## CFG_ACCESS_CONTROL_LIMIT_REGISTRATION_TO_DOMAIN -- limit account
## registration to certain email addresses? If wanted, give domain
## name below, e.g. "cern.ch". If not wanted, leave it empty.
CFG_ACCESS_CONTROL_LIMIT_REGISTRATION_TO_DOMAIN =
## CFG_ACCESS_CONTROL_NOTIFY_ADMIN_ABOUT_NEW_ACCOUNTS -- send a
## notification email to the administrator when a new account is
## created? Use 0 for no, 1 for yes.
CFG_ACCESS_CONTROL_NOTIFY_ADMIN_ABOUT_NEW_ACCOUNTS = 0
## CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_NEW_ACCOUNT -- send a
## notification email to the user when a new account is created in order to
## to verify the validity of the provided email address? Use
## 0 for no, 1 for yes.
CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_NEW_ACCOUNT = 1
## CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_ACTIVATION -- send a
## notification email to the user when a new account is activated?
## Use 0 for no, 1 for yes.
CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_ACTIVATION = 0
## CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_DELETION -- send a
## notification email to the user when a new account is deleted or
## account demand rejected? Use 0 for no, 1 for yes.
CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_DELETION = 0
## CFG_APACHE_PASSWORD_FILE -- the file where Apache user credentials
## are stored. Must be an absolute pathname. If the value does not
## start by a slash, it is considered to be the filename of a file
## located under prefix/var/tmp directory. This is useful for the
## demo site testing purposes. For the production site, if you plan
## to restrict access to some collections based on the Apache user
## authentication mechanism, you should put here an absolute path to
## your Apache password file.
CFG_APACHE_PASSWORD_FILE = demo-site-apache-user-passwords
## CFG_APACHE_GROUP_FILE -- the file where Apache user groups are
## defined. See the documentation of the preceding config variable.
CFG_APACHE_GROUP_FILE = demo-site-apache-user-groups
###################################
## Part 8: WebSession parameters ##
###################################
## This section contains some configuration parameters for tweaking
## session handling.
## CFG_WEBSESSION_EXPIRY_LIMIT_DEFAULT -- number of days after which a session
## and the corresponding cookie is considered expired.
CFG_WEBSESSION_EXPIRY_LIMIT_DEFAULT = 2
## CFG_WEBSESSION_EXPIRY_LIMIT_REMEMBER -- number of days after which a session
## and the corresponding cookie is considered expired, when the user has
## requested to permanently stay logged in.
CFG_WEBSESSION_EXPIRY_LIMIT_REMEMBER = 365
## CFG_WEBSESSION_RESET_PASSWORD_EXPIRE_IN_DAYS -- when user requested
## a password reset, for how many days is the URL valid?
CFG_WEBSESSION_RESET_PASSWORD_EXPIRE_IN_DAYS = 3
## CFG_WEBSESSION_ADDRESS_ACTIVATION_EXPIRE_IN_DAYS -- when an account
## activation email was sent, for how many days is the URL valid?
CFG_WEBSESSION_ADDRESS_ACTIVATION_EXPIRE_IN_DAYS = 3
## CFG_WEBSESSION_NOT_CONFIRMED_EMAIL_ADDRESS_EXPIRE_IN_DAYS -- when
## user won't confirm his email address and not complete
## registeration, after how many days will it expire?
CFG_WEBSESSION_NOT_CONFIRMED_EMAIL_ADDRESS_EXPIRE_IN_DAYS = 10
## CFG_WEBSESSION_DIFFERENTIATE_BETWEEN_GUESTS -- when set to 1, the session
## system allocates the same uid=0 to all guests users regardless of where they
## come from. 0 allocate a unique uid to each guest.
CFG_WEBSESSION_DIFFERENTIATE_BETWEEN_GUESTS = 0
## CFG_WEBSESSION_IPADDR_CHECK_SKIP_BITS -- to prevent session cookie
## stealing, Invenio checks that the IP address of a connection is the
## same as that of the connection which created the initial session.
## This variable let you decide how many bits should be skipped during
## this check. Set this to 0 in order to enable full IP address
## checking. Set this to 32 in order to disable IP address checking.
## Intermediate values (say 8) let you have some degree of security so
## that you can trust your local network only while helping to solve
## issues related to outside clients that configured their browser to
## use a web proxy for HTTP connection but not for HTTPS, thus
## potentially having two different IP addresses. In general, if use
## HTTPS in order to serve authenticated content, you can safely set
## CFG_WEBSESSION_IPADDR_CHECK_SKIP_BITS to 32.
CFG_WEBSESSION_IPADDR_CHECK_SKIP_BITS = 0
+## CFG_WEBSESSION_STORAGE -- where to store the sessions
+## possible choices are 'mysql' or 'redis'
+CFG_WEBSESSION_STORAGE = redis
+
+
################################
## Part 9: BibRank parameters ##
################################
## This section contains some configuration parameters for the ranking
## system.
## CFG_BIBRANK_SHOW_READING_STATS -- do we want to show reading
## similarity stats? ('People who viewed this page also viewed')
CFG_BIBRANK_SHOW_READING_STATS = 1
## CFG_BIBRANK_SHOW_DOWNLOAD_STATS -- do we want to show the download
## similarity stats? ('People who downloaded this document also
## downloaded')
CFG_BIBRANK_SHOW_DOWNLOAD_STATS = 1
## CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS -- do we want to show download
## history graph? (0=no | 1=classic/gnuplot | 2=flot)
CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS = 1
## CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS_CLIENT_IP_DISTRIBUTION -- do we
## want to show a graph representing the distribution of client IPs
## downloading given document? (0=no | 1=classic/gnuplot | 2=flot)
CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS_CLIENT_IP_DISTRIBUTION = 0
## CFG_BIBRANK_SHOW_CITATION_LINKS -- do we want to show the 'Cited
## by' links? (useful only when you have citations in the metadata)
CFG_BIBRANK_SHOW_CITATION_LINKS = 1
## CFG_BIBRANK_SHOW_CITATION_STATS -- de we want to show citation
## stats? ('Cited by M recors', 'Co-cited with N records')
CFG_BIBRANK_SHOW_CITATION_STATS = 1
## CFG_BIBRANK_SHOW_CITATION_GRAPHS -- do we want to show citation
## history graph? (0=no | 1=classic/gnuplot | 2=flot)
CFG_BIBRANK_SHOW_CITATION_GRAPHS = 1
## CFG_BIBRANK_SELFCITES_USE_BIBAUTHORID -- use authorids for computing
## self-citations
## falls back to hashing the author string
CFG_BIBRANK_SELFCITES_USE_BIBAUTHORID = 0
## CFG_BIBRANK_SELFCITES_PRECOMPUTE -- use precomputed self-citations
## when displaying itesummary. Precomputing citations allows use to
## speed up things
CFG_BIBRANK_SELFCITES_PRECOMPUTE = 0
####################################
## Part 10: WebComment parameters ##
####################################
## This section contains some configuration parameters for the
## commenting and reviewing facilities.
## CFG_WEBCOMMENT_ALLOW_COMMENTS -- do we want to allow users write
## public comments on records?
CFG_WEBCOMMENT_ALLOW_COMMENTS = 1
## CFG_WEBCOMMENT_ALLOW_REVIEWS -- do we want to allow users write
## public reviews of records?
CFG_WEBCOMMENT_ALLOW_REVIEWS = 1
## CFG_WEBCOMMENT_ALLOW_SHORT_REVIEWS -- do we want to allow short
## reviews, that is just the attribution of stars without submitting
## detailed review text?
CFG_WEBCOMMENT_ALLOW_SHORT_REVIEWS = 0
## CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN -- if users
## report a comment to be abusive, how many they have to be before the
## site admin is alerted?
CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN = 5
## CFG_WEBCOMMENT_NB_COMMENTS_IN_DETAILED_VIEW -- how many comments do
## we display in the detailed record page upon welcome?
CFG_WEBCOMMENT_NB_COMMENTS_IN_DETAILED_VIEW = 1
## CFG_WEBCOMMENT_NB_REVIEWS_IN_DETAILED_VIEW -- how many reviews do
## we display in the detailed record page upon welcome?
CFG_WEBCOMMENT_NB_REVIEWS_IN_DETAILED_VIEW = 1
## CFG_WEBCOMMENT_ADMIN_NOTIFICATION_LEVEL -- do we notify the site
## admin after every comment?
CFG_WEBCOMMENT_ADMIN_NOTIFICATION_LEVEL = 1
## CFG_WEBCOMMENT_TIMELIMIT_PROCESSING_COMMENTS_IN_SECONDS -- how many
## elapsed seconds do we consider enough when checking for possible
## multiple comment submissions by a user?
CFG_WEBCOMMENT_TIMELIMIT_PROCESSING_COMMENTS_IN_SECONDS = 20
## CFG_WEBCOMMENT_TIMELIMIT_PROCESSING_REVIEWS_IN_SECONDS -- how many
## elapsed seconds do we consider enough when checking for possible
## multiple review submissions by a user?
CFG_WEBCOMMENT_TIMELIMIT_PROCESSING_REVIEWS_IN_SECONDS = 20
## CFG_WEBCOMMENT_USE_RICH_EDITOR -- enable the WYSIWYG
## Javascript-based editor when user edits comments?
CFG_WEBCOMMENT_USE_RICH_TEXT_EDITOR = False
## CFG_WEBCOMMENT_ALERT_ENGINE_EMAIL -- the email address from which the
## alert emails will appear to be sent:
CFG_WEBCOMMENT_ALERT_ENGINE_EMAIL = info@invenio-software.org
## CFG_WEBCOMMENT_DEFAULT_MODERATOR -- if no rules are
## specified to indicate who is the comment moderator of
## a collection, this person will be used as default
CFG_WEBCOMMENT_DEFAULT_MODERATOR = info@invenio-software.org
## CFG_WEBCOMMENT_USE_MATHJAX_IN_COMMENTS -- do we want to allow the use
## of MathJax plugin to render latex input in comments?
CFG_WEBCOMMENT_USE_MATHJAX_IN_COMMENTS = 1
## CFG_WEBCOMMENT_AUTHOR_DELETE_COMMENT_OPTION -- allow comment author to
## delete its own comment?
CFG_WEBCOMMENT_AUTHOR_DELETE_COMMENT_OPTION = 1
# CFG_WEBCOMMENT_EMAIL_REPLIES_TO -- which field of the record define
# email addresses that should be notified of newly submitted comments,
# and for which collection. Use collection names as keys, and list of
# tags as values
CFG_WEBCOMMENT_EMAIL_REPLIES_TO = {
'Articles': ['506__d', '506__m'],
}
# CFG_WEBCOMMENT_RESTRICTION_DATAFIELD -- which field of the record
# define the restriction (must be linked to WebAccess
# 'viewrestrcomment') to apply to newly submitted comments, and for
# which collection. Use collection names as keys, and one tag as value
CFG_WEBCOMMENT_RESTRICTION_DATAFIELD = {
'Articles': '5061_a',
'Pictures': '5061_a',
'Theses': '5061_a',
}
# CFG_WEBCOMMENT_ROUND_DATAFIELD -- which field of the record define
# the current round of comment for which collection. Use collection
# name as key, and one tag as value
CFG_WEBCOMMENT_ROUND_DATAFIELD = {
'Articles': '562__c',
'Pictures': '562__c',
}
# CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE -- max file size per attached
# file, in bytes. Choose 0 if you don't want to limit the size
CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE = 5242880
# CFG_WEBCOMMENT_MAX_ATTACHED_FILES -- maxium number of files that can
# be attached per comment. Choose 0 if you don't want to limit the
# number of files. File uploads can be restricted with action
# "attachcommentfile".
CFG_WEBCOMMENT_MAX_ATTACHED_FILES = 5
# CFG_WEBCOMMENT_MAX_COMMENT_THREAD_DEPTH -- how many levels of
# indentation discussions can be. This can be used to ensure that
# discussions will not go into deep levels of nesting if users don't
# understand the difference between "reply to comment" and "add
# comment". When the depth is reached, any "reply to comment" is
# conceptually converted to a "reply to thread" (i.e. reply to this
# parent's comment). Use -1 for no limit, 0 for unthreaded (flat)
# discussions.
CFG_WEBCOMMENT_MAX_COMMENT_THREAD_DEPTH = 1
##################################
## Part 11: BibSched parameters ##
##################################
## This section contains some configuration parameters for the
## bibliographic task scheduler.
## CFG_BIBSCHED_REFRESHTIME -- how often do we want to refresh
## bibsched monitor? (in seconds)
CFG_BIBSCHED_REFRESHTIME = 5
## CFG_BIBSCHED_LOG_PAGER -- what pager to use to view bibsched task
## logs?
CFG_BIBSCHED_LOG_PAGER = /usr/bin/less
## CFG_BIBSCHED_EDITOR -- what editor to use to edit the marcxml
## code of the locked records
CFG_BIBSCHED_EDITOR = /usr/bin/vim
## CFG_BIBSCHED_GC_TASKS_OLDER_THAN -- after how many days to perform the
## gargbage collector of BibSched queue (i.e. removing/moving task to archive).
CFG_BIBSCHED_GC_TASKS_OLDER_THAN = 30
## CFG_BIBSCHED_GC_TASKS_TO_REMOVE -- list of BibTask that can be safely
## removed from the BibSched queue once they are DONE.
CFG_BIBSCHED_GC_TASKS_TO_REMOVE = bibindex,bibreformat,webcoll,bibrank,inveniogc
## CFG_BIBSCHED_GC_TASKS_TO_ARCHIVE -- list of BibTasks that should be safely
## archived out of the BibSched queue once they are DONE.
CFG_BIBSCHED_GC_TASKS_TO_ARCHIVE = bibupload,oairepositoryupdater
## CFG_BIBSCHED_MAX_NUMBER_CONCURRENT_TASKS -- maximum number of BibTasks
## that can run concurrently.
## NOTE: concurrent tasks are still considered as an experimental
## feature. Please keep this value set to 1 on production environments.
CFG_BIBSCHED_MAX_NUMBER_CONCURRENT_TASKS = 1
## CFG_BIBSCHED_PROCESS_USER -- bibsched and bibtask processes must
## usually run under the same identity as the Apache web server
## process in order to share proper file read/write privileges. If
## you want to force some other bibsched/bibtask user, e.g. because
## you are using a local `invenio' user that belongs to your
## `www-data' Apache user group and so shares writing rights with your
## Apache web server process in this way, then please set its username
## identity here. Otherwise we shall check whether your
## bibsched/bibtask processes are run under the same identity as your
## Apache web server process (in which case you can leave the default
## empty value here).
CFG_BIBSCHED_PROCESS_USER =
## CFG_BIBSCHED_NODE_TASKS -- specific nodes may be configured to
## run only specific tasks; if you want this, then this variable is a
## dictionary of the form {'hostname1': ['task1', 'task2']}. The
## default is that any node can run any task.
CFG_BIBSCHED_NODE_TASKS = {}
## CFG_BIBSCHED_MAX_ARCHIVED_ROWS_DISPLAY -- number of tasks displayed
## in the archive view (by pressing '1')
CFG_BIBSCHED_MAX_ARCHIVED_ROWS_DISPLAY = 500
## CFG_BIBSCHED_NON_CONCURRENT_TASKS -- tasks that should not be running at
## the same time (but one of them can be sleeping)
CFG_BIBSCHED_NON_CONCURRENT_TASKS = (
('bibupload', ),)
## CFG_BIBSCHED_INCOMPATIBLE_TASKS -- tasks that should not be running at
## the same time (even if the other one is sleeping)
CFG_BIBSCHED_INCOMPATIBLE_TASKS = ()
## CFG_BIBSCHED_LOGDIR -- location of bibtask logs such as
## bibsched_task_{ID}.(log|err). Add a trailing slash to support symlinks.
## If path is relative, CFG_LOGDIR will be joined as a prefix.
CFG_BIBSCHED_LOGDIR = bibsched
## CFG_BIBSCHED_FLUSH_LOGS -- flush logs after writing each message.
## This can be useful when using a filesystem that buffers yours writes
## like AFS and want to check the logs from a different server than the one
## the task is running on.
CFG_BIBSCHED_FLUSH_LOGS = 0
## CFG_BIBSCHED_NEVER_STOPS -- continue queue execution when a task fails
CFG_BIBSCHED_NEVER_STOPS = 0
###################################
## Part 12: WebBasket parameters ##
###################################
## CFG_WEBBASKET_MAX_NUMBER_OF_DISPLAYED_BASKETS -- a safety limit for
## a maximum number of displayed baskets
CFG_WEBBASKET_MAX_NUMBER_OF_DISPLAYED_BASKETS = 20
## CFG_WEBBASKET_USE_RICH_TEXT_EDITOR -- enable the WYSIWYG
## Javascript-based editor when user edits comments in WebBasket?
CFG_WEBBASKET_USE_RICH_TEXT_EDITOR = False
##################################
## Part 13: WebAlert parameters ##
##################################
## This section contains some configuration parameters for the
## automatic email notification alert system.
## CFG_WEBALERT_ALERT_ENGINE_EMAIL -- the email address from which the
## alert emails will appear to be sent:
CFG_WEBALERT_ALERT_ENGINE_EMAIL = info@invenio-software.org
## CFG_WEBALERT_MAX_NUM_OF_RECORDS_IN_ALERT_EMAIL -- how many records
## at most do we send in an outgoing alert email?
CFG_WEBALERT_MAX_NUM_OF_RECORDS_IN_ALERT_EMAIL = 20
## CFG_WEBALERT_MAX_NUM_OF_CHARS_PER_LINE_IN_ALERT_EMAIL -- number of
## chars per line in an outgoing alert email?
CFG_WEBALERT_MAX_NUM_OF_CHARS_PER_LINE_IN_ALERT_EMAIL = 72
## CFG_WEBALERT_SEND_EMAIL_NUMBER_OF_TRIES -- when sending alert
## emails fails, how many times we retry?
CFG_WEBALERT_SEND_EMAIL_NUMBER_OF_TRIES = 3
## CFG_WEBALERT_SEND_EMAIL_SLEEPTIME_BETWEEN_TRIES -- when sending
## alert emails fails, what is the sleeptime between tries? (in
## seconds)
CFG_WEBALERT_SEND_EMAIL_SLEEPTIME_BETWEEN_TRIES = 300
####################################
## Part 14: WebMessage parameters ##
####################################
## CFG_WEBMESSAGE_MAX_SIZE_OF_MESSAGE -- how large web messages do we
## allow?
CFG_WEBMESSAGE_MAX_SIZE_OF_MESSAGE = 20000
## CFG_WEBMESSAGE_MAX_NB_OF_MESSAGES -- how many messages for a
## regular user do we allow in its inbox?
CFG_WEBMESSAGE_MAX_NB_OF_MESSAGES = 30
## CFG_WEBMESSAGE_DAYS_BEFORE_DELETE_ORPHANS -- how many days before
## we delete orphaned messages?
CFG_WEBMESSAGE_DAYS_BEFORE_DELETE_ORPHANS = 60
##################################
## Part 15: MiscUtil parameters ##
##################################
## CFG_MISCUTIL_SQL_USE_SQLALCHEMY -- whether to use SQLAlchemy.pool
## in the DB engine of Invenio. It is okay to enable this flag
## even if you have not installed SQLAlchemy. Note that Invenio will
## loose some perfomance if this option is enabled.
CFG_MISCUTIL_SQL_USE_SQLALCHEMY = False
## CFG_MISCUTIL_SQL_RUN_SQL_MANY_LIMIT -- how many queries can we run
## inside run_sql_many() in one SQL statement? The limit value
## depends on MySQL's max_allowed_packet configuration.
CFG_MISCUTIL_SQL_RUN_SQL_MANY_LIMIT = 10000
## CFG_MISCUTIL_SMTP_HOST -- which server to use as outgoing mail server to
## send outgoing emails generated by the system, for example concerning
## submissions or email notification alerts.
CFG_MISCUTIL_SMTP_HOST = localhost
## CFG_MISCUTIL_SMTP_PORT -- which port to use on the outgoing mail server
## defined in the previous step.
CFG_MISCUTIL_SMTP_PORT = 25
## CFG_MISCUTIL_SMTP_USER -- which username to use on the outgoing mail server
## defined in CFG_MISCUTIL_SMTP_HOST. If either CFG_MISCUTIL_SMTP_USER or
## CFG_MISCUTIL_SMTP_PASS are empty Invenio won't attempt authentication.
CFG_MISCUTIL_SMTP_USER =
## CFG_MISCUTIL_SMTP_PASS -- which password to use on the outgoing mail
## server defined in CFG_MISCUTIL_SMTP_HOST. If either CFG_MISCUTIL_SMTP_USER
## or CFG_MISCUTIL_SMTP_PASS are empty Invenio won't attempt authentication.
CFG_MISCUTIL_SMTP_PASS =
## CFG_MISCUTIL_SMTP_TLS -- whether to use a TLS (secure) connection when
## talking to the SMTP server defined in CFG_MISCUTIL_SMTP_HOST.
CFG_MISCUTIL_SMTP_TLS = False
## CFG_MISCUTILS_DEFAULT_PROCESS_TIMEOUT -- the default number of seconds after
## which a process launched trough shellutils.run_process_with_timeout will
## be killed. This is useful to catch runaway processes.
CFG_MISCUTIL_DEFAULT_PROCESS_TIMEOUT = 300
## CFG_MATHJAX_HOSTING -- if you plan to use MathJax to display TeX
## formulas on HTML web pages, you can specify whether you wish to use
## 'local' hosting or 'cdn' hosting of MathJax libraries. (If set to
## 'local', you have to run 'make install-mathjax-plugin' as described
## in the INSTALL guide.) If set to 'local', users will use your site
## to download MathJax sources. If set to 'cdn', users will use
## centralized MathJax CDN servers instead. Please note that using
## CDN is suitable only for small institutes or for MathJax
## sponsors; see the MathJax website for more details. (Also, please
## note that if you plan to use MathJax on your site, you have to
## adapt CFG_WEBSEARCH_USE_MATHJAX_FOR_FORMATS and
## CFG_WEBCOMMENT_USE_MATHJAX_IN_COMMENTS configuration variables
## elsewhere in this file.)
CFG_MATHJAX_HOSTING = local
+## CFG_MATHJAX_RENDERS_MATHML -- whether MathJax is used to render MathML.
+CFG_MATHJAX_RENDERS_MATHML = 0
+
#################################
## Part 16: BibEdit parameters ##
#################################
## CFG_BIBEDIT_TIMEOUT -- when a user edits a record, this record is
## locked to prevent other users to edit it at the same time.
## How many seconds of inactivity before the locked record again will be free
## for other people to edit?
CFG_BIBEDIT_TIMEOUT = 3600
## CFG_BIBEDIT_LOCKLEVEL -- when a user tries to edit a record which there
## is a pending bibupload task for in the queue, this shouldn't be permitted.
## The lock level determines how thouroughly the queue should be investigated
## to determine if this is the case.
## Level 0 - always permits editing, doesn't look at the queue
## (unsafe, use only if you know what you are doing)
## Level 1 - permits editing if there are no queued bibedit tasks for this record
## (safe with respect to bibedit, but not for other bibupload maintenance jobs)
## Level 2 - permits editing if there are no queued bibupload tasks of any sort
## (safe, but may lock more than necessary if many cataloguers around)
## Level 3 - permits editing if no queued bibupload task concerns given record
## (safe, most precise locking, but slow,
## checks for 001/EXTERNAL_SYSNO_TAG/EXTERNAL_OAIID_TAG)
## The recommended level is 3 (default) or 2 (if you use maintenance jobs often).
CFG_BIBEDIT_LOCKLEVEL = 3
## CFG_BIBEDIT_PROTECTED_FIELDS -- a comma-separated list of fields that BibEdit
## will not allow to be added, edited or deleted. Wildcards are not supported,
## but conceptually a wildcard is added at the end of every field specification.
## Examples:
## 500A - protect all MARC fields with tag 500 and first indicator A
## 5 - protect all MARC fields in the 500-series.
## 909C_a - protect subfield a in tag 909 with first indicator C and empty
## second indicator
## Note that 001 is protected by default, but if protection of other
## identifiers or automated fields is a requirement, they should be added to
## this list.
CFG_BIBEDIT_PROTECTED_FIELDS =
## CFG_BIBEDIT_QUEUE_CHECK_METHOD -- how do we want to check for
## possible queue locking situations to prevent cataloguers from
## editing a record that may be waiting in the queue? Use 'bibrecord'
## for exact checking (always works, but may be slow), use 'regexp'
## for regular expression based checking (very fast, but may be
## inaccurate). When unsure, use 'bibrecord'.
CFG_BIBEDIT_QUEUE_CHECK_METHOD = bibrecord
## CFG_BIBEDIT_EXTEND_RECORD_WITH_COLLECTION_TEMPLATE -- a list
## containing which collections will be extended with a given template
## while being displayed in BibEdit UI. The collection corresponds with
## the value written in field 980. The order is not arbitrary, first match of
## a collection in CFG_BIBEDIT_EXTEND_RECORD_WITH_COLLECTION_TEMPLATE with a
## 980 field will be used.
CFG_BIBEDIT_EXTEND_RECORD_WITH_COLLECTION_TEMPLATE = [('POETRY', 'record_poem')]
## CFG_BIBEDIT_KB_SUBJECTS - Name of the KB used in the field 65017a
## to automatically convert codes into extended version. e.g
## a - Astrophysics
CFG_BIBEDIT_KB_SUBJECTS = Subjects
## CFG_BIBEDIT_KB_INSTITUTIONS - Name of the KB used for institution
## autocomplete. To be applied in fields defined in
## CFG_BIBEDIT_AUTOCOMPLETE_INSTITUTIONS_FIELDS
CFG_BIBEDIT_KB_INSTITUTIONS = InstitutionsCollection
## CFG_BIBEDIT_AUTOCOMPLETE_INSTITUTIONS_FIELDS - list of fields to
## be autocompleted with the KB CFG_BIBEDIT_KB_INSTITUTIONS
CFG_BIBEDIT_AUTOCOMPLETE_INSTITUTIONS_FIELDS = 100__u,700__u,701__u,502__c
## CFG_BIBEDIT_SHOW_HOLDING_PEN_REMOVED_FIELDS -- whether to show or not
## the fields and subfields that are removed in holding pen records.
## 0 for false, 1 for true.
CFG_BIBEDIT_SHOW_HOLDING_PEN_REMOVED_FIELDS = 0
## CFG_BIBEDITMULTI_LIMIT_INSTANT_PROCESSING -- maximum number of records
## that can be modified instantly using the multi-record editor. Above
## this limit, modifications will only be executed in limited hours.
CFG_BIBEDITMULTI_LIMIT_INSTANT_PROCESSING = 2000
## CFG_BIBEDITMULTI_LIMIT_DELAYED_PROCESSING -- maximum number of records
## that can be send for modification without having a superadmin role.
## If the number of records is between CFG_BIBEDITMULTI_LIMIT_INSTANT_PROCESSING
## and this number, the modifications will take place only in limited hours.
CFG_BIBEDITMULTI_LIMIT_DELAYED_PROCESSING = 20000
## CFG_BIBEDITMULTI_LIMIT_DELAYED_PROCESSING_TIME -- Allowed time to
## execute modifications on records, when the number exceeds
## CFG_BIBEDITMULTI_LIMIT_INSTANT_PROCESSING.
CFG_BIBEDITMULTI_LIMIT_DELAYED_PROCESSING_TIME = 22:00-05:00
## CFG_BIBEDIT_ADD_TICKET_RT_QUEUES -- list of queues that are
## are being displayed in create new ticket select box, so
## that user can create new ticket in one of these queues.
CFG_BIBEDIT_ADD_TICKET_RT_QUEUES=Authors,Conf_add+cor,Exp,Feedback,HEP_cor,HEP_ref,INST_add+cor,AUTHORS_long_list
###################################
## Part 17: BibUpload parameters ##
###################################
## CFG_BIBUPLOAD_REFERENCE_TAG -- where do we store references?
CFG_BIBUPLOAD_REFERENCE_TAG = 999
## CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG -- where do we store external
## system numbers? Useful for matching when our records come from an
## external digital library system.
CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG = 970__a
## CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG -- where do we store OAI ID tags
## of harvested records? Useful for matching when we harvest stuff
## via OAI that we do not want to reexport via Invenio OAI; so records
## may have only the source OAI ID stored in this tag (kind of like
## external system number too).
CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG = 035__a
## CFG_BIBUPLOAD_EXTERNAL_OAIID_PROVENANCE_TAG -- where do we store OAI SRC
## tags of harvested records? Useful for matching when we harvest stuff
## via OAI that we do not want to reexport via Invenio OAI; so records
## may have only the source OAI SRC stored in this tag (kind of like
## external system number too). Note that the field should be the same of
## CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG.
CFG_BIBUPLOAD_EXTERNAL_OAIID_PROVENANCE_TAG = 035__9
## CFG_BIBUPLOAD_STRONG_TAGS -- a comma-separated list of tags that
## are strong enough to resist the replace mode. Useful for tags that
## might be created from an external non-metadata-like source,
## e.g. the information about the number of copies left.
CFG_BIBUPLOAD_STRONG_TAGS = 964
## CFG_BIBUPLOAD_CONTROLLED_PROVENANCE_TAGS -- a comma-separated list
## of tags that contain provenance information that should be checked
## in the bibupload correct mode via matching provenance codes. (Only
## field instances of the same provenance information would be acted
## upon.) Please specify the whole tag info up to subfield codes.
CFG_BIBUPLOAD_CONTROLLED_PROVENANCE_TAGS = 6531_9
## CFG_BIBUPLOAD_FFT_ALLOWED_LOCAL_PATHS -- a comma-separated list of system
## paths from which it is allowed to take fulltextes that will be uploaded via
## FFT (CFG_TMPDIR is included by default).
CFG_BIBUPLOAD_FFT_ALLOWED_LOCAL_PATHS = /tmp,/home
## CFG_BIBUPLOAD_FFT_ALLOWED_EXTERNAL_URLS -- a dictionary containing
## external URLs that can be accessed by Invenio and specific HTTP
## headers that will be used for each URL. The keys of the dictionary
## are regular expressions matching a set of URLs, the values are
## dictionaries of headers as consumed by urllib2.Request. If a
## regular expression matching all URLs is created at the end of the
## list, it means that Invenio will download all URLs. Otherwise
## Invenio will just download authorized URLs. Note: by default, a
## User-Agent built after the current Invenio version, site name, and
## site URL will be used. The values of the header dictionary can
## also contain a call to a python function, in the form of a
## disctionary with two entries: the name of the function to be called
## as a value for the 'fnc' key, and the arguments to this function,
## as a value for the 'args' key (in the form of a dictionary).
## CFG_BIBUPLOAD_FFT_ALLOWED_EXTERNAL_URLS = [
## ('http://myurl.com/.*', {'User-Agent': 'Me'}),
## ('http://yoururl.com/.*', {'User-Agent': 'You', 'Accept': 'text/plain'}),
## ('http://thisurl.com/.*', {'Cookie': {'fnc':'read_cookie', 'args':{'cookiefile':'/tmp/cookies.txt'}}})
## ('http://.*', {'User-Agent': 'Invenio'}),
## ]
CFG_BIBUPLOAD_FFT_ALLOWED_EXTERNAL_URLS = [
('http(s)?://.*', {}),
]
## CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE -- do we want to serialize
## internal representation of records (Pythonic record structure) into
## the database? This can improve internal processing speed of some
## operations at the price of somewhat bigger disk space usage.
## If you change this value after some records have already been added
## to your installation, you may want to run:
## $ /opt/invenio/bin/inveniocfg --reset-recstruct-cache
## in order to either erase the cache thus freeing database space,
## or to fill the cache for all records that have not been cached yet.
CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE = 1
## CFG_BIBUPLOAD_DELETE_FORMATS -- which formats do we want bibupload
## to delete when a record is ingested? Enter comma-separated list of
## formats. For example, 'hb,hd' will delete pre-formatted HTML brief
## and defailed formats from cache, so that search engine will
## generate them on-the-fly. Useful to always present latest data of
## records upon record display, until the periodical bibreformat job
## runs next and updates the cache.
CFG_BIBUPLOAD_DELETE_FORMATS = hb
## CFG_BIBUPLOAD_DISABLE_RECORD_REVISIONS -- set to 1 if keeping
## history of record revisions is not necessary (e.g. because records
## and corresponding modifications are coming always from the same
## external system which already keeps revision history).
CFG_BIBUPLOAD_DISABLE_RECORD_REVISIONS = 0
## CFG_BIBUPLOAD_CONFLICTING_REVISION_TICKET_QUEUE -- Set the name of
## the BibCatalog ticket queue to be used when BibUpload can't
## automatically resolve a revision conflict and has therefore to put
## requested modifications in the holding pen.
CFG_BIBUPLOAD_CONFLICTING_REVISION_TICKET_QUEUE =
## CFG_BIBUPLOAD_MATCH_DELETED_RECORDS -- Whether matching of
## records (e.g. via DOI and other identifiers) should consider
## also deleted records.
CFG_BIBUPLOAD_MATCH_DELETED_RECORDS = 1
## CFG_BATCHUPLOADER_FILENAME_MATCHING_POLICY -- a comma-separated list
## indicating which fields match the file names of the documents to be
## uploaded.
## The matching will be done in the same order as the list provided.
CFG_BATCHUPLOADER_FILENAME_MATCHING_POLICY = reportnumber,recid
## CFG_BATCHUPLOADER_DAEMON_DIR -- Directory where the batchuploader daemon
## will look for the subfolders metadata and document by default.
## If path is relative, CFG_PREFIX will be joined as a prefix
CFG_BATCHUPLOADER_DAEMON_DIR = var/batchupload
## CFG_BATCHUPLOADER_WEB_ROBOT_AGENTS -- Regular expression to specify the
## agents permitted when calling batch uploader web interface
## cds.cern.ch/batchuploader/robotupload
## if using a curl, eg: curl xxx -A invenio
CFG_BATCHUPLOADER_WEB_ROBOT_AGENTS = invenio_webupload|Invenio-.*
## CFG_BATCHUPLOADER_WEB_ROBOT_RIGHTS -- Access list specifying for each
## IP address (or range), which collections are allowed using batch uploader robot
## interface.
## Note: you can also specify network ranges such as "127.0.0.0/8".
CFG_BATCHUPLOADER_WEB_ROBOT_RIGHTS = {
'::1': ['*'], # useful for testing
'127.0.0.1': ['*'], # useful for testing
'127.0.1.1': ['*'], # useful for testing
'10.0.0.1': ['BOOK', 'REPORT'], # Example 1
'10.0.0.2': ['POETRY', 'PREPRINT'], # Example 2
}
####################################
## Part 18: BibCatalog parameters ##
####################################
## CFG_BIBCATALOG_SYSTEM -- set desired catalog system. (RT or EMAIL)
CFG_BIBCATALOG_SYSTEM = EMAIL
## Email backend configuration:
CFG_BIBCATALOG_SYSTEM_EMAIL_ADDRESS = info@invenio-software.org
## RT backend configuration:
## CFG_BIBCATALOG_SYSTEM_RT_CLI -- path to the RT CLI client
CFG_BIBCATALOG_SYSTEM_RT_CLI = /usr/bin/rt
## CFG_BIBCATALOG_SYSTEM_RT_URL -- Base URL of the remote RT system
CFG_BIBCATALOG_SYSTEM_RT_URL = http://localhost/rt3
## CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_USER -- Set the username for a default RT account
## on remote system, with limited privileges, in order to only create and modify own tickets.
CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_USER =
## CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_PWD -- Set the password for the default RT account
## on remote system.
CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_PWD =
####################################
## Part 19: BibFormat parameters ##
####################################
## CFG_BIBFORMAT_HIDDEN_TAGS -- comma-separated list of MARC tags that
## are not shown to users not having cataloging authorizations.
CFG_BIBFORMAT_HIDDEN_TAGS = 595
## CFG_BIBFORMAT_HIDDEN_FILE_FORMATS -- comma-separated list of file formats
## that are not shown explicitly to user not having cataloging authorizations.
## e.g. pdf;pdfa,xml
CFG_BIBFORMAT_HIDDEN_FILE_FORMATS =
## CFG_BIBFORMAT_ADDTHIS_ID -- if you want to use the AddThis service from
## <http://www.addthis.com/>, set this value to the pubid parameter as
## provided by the service (e.g. ra-4ff80aae118f4dad), and add a call to
## <BFE_ADDTHIS /> formatting element in your formats, for example
## Default_HTML_detailed.bft.
CFG_BIBFORMAT_ADDTHIS_ID =
## CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS -- For each output
## format BibReformat currently creates a cache for only one language
## (CFG_SITE_LANG) per record. This means that visitors having set a
## different language than CFG_SITE_LANG will be served an on-the-fly
## output using the language of their choice. You can disable this
## behaviour by specifying below for which output format you would
## like to force the cache to be used whatever language is
## requested. If your format templates do not provide
## internationalization, you can optimize your site by setting for
## eg. hb,hd to always serve the precached output (if it exists) in
## the CFG_SITE_LANG
CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS =
## CFG_BIBFORMAT_CACHED_FORMATS -- Specify a list of cached formats
## We need to know which ones are cached because bibformat will save the
## of these in a db table
CFG_BIBFORMAT_CACHED_FORMATS =
####################################
## Part 20: BibMatch parameters ##
####################################
## CFG_BIBMATCH_LOCAL_SLEEPTIME -- Determines the amount of seconds to sleep
## between search queries on LOCAL system.
CFG_BIBMATCH_LOCAL_SLEEPTIME = 0.0
## CFG_BIBMATCH_REMOTE_SLEEPTIME -- Determines the amount of seconds to sleep
## between search queries on REMOTE systems.
CFG_BIBMATCH_REMOTE_SLEEPTIME = 2.0
## CFG_BIBMATCH_FUZZY_WORDLIMITS -- Determines the amount of words to extract
## from a certain fields value during fuzzy matching mode. Add/change field
## and appropriate number to the dictionary to configure this.
CFG_BIBMATCH_FUZZY_WORDLIMITS = {
'100__a': 2,
'245__a': 4
}
## CFG_BIBMATCH_FUZZY_EMPTY_RESULT_LIMIT -- Determines the amount of empty results
## to accept during fuzzy matching mode.
CFG_BIBMATCH_FUZZY_EMPTY_RESULT_LIMIT = 1
## CFG_BIBMATCH_QUERY_TEMPLATES -- Here you can set the various predefined querystrings
## used to standardize common matching queries. By default the following templates
## are given:
## title - standard title search. Taken from 245__a (default)
## title-author - title and author search (i.e. this is a title AND author a)
## Taken from 245__a and 100__a
## reportnumber - reportnumber search (i.e. reportnumber:REP-NO-123).
CFG_BIBMATCH_QUERY_TEMPLATES = {
'title' : '[title]',
'title-author' : '[title] [author]',
'reportnumber' : 'reportnumber:[reportnumber]'
}
## CFG_BIBMATCH_MATCH_VALIDATION_RULESETS -- Here you can define the various rulesets for
## validating search results done by BibMatch. Each ruleset contains a certain pattern mapped
## to a tuple defining a "matching-strategy".
##
## The rule-definitions must come in two parts:
##
## * The first part is a string containing a regular expression
## that is matched against the textmarc representation of each record.
## If a match is found, the final rule-set is updated with
## the given "sub rule-set", where identical tag rules are replaced.
##
## * The second item is a list of key->value mappings (dict) that indicates specific
## strategy parameters with corresponding validation rules.
##
## This strategy consists of five items:
##
## * MARC TAGS:
## These MARC tags represents the fields taken from original record and any records from search
## results. When several MARC tags are specified with a given match-strategy, all the fields
## associated with these tags are matched together (i.e. with key "100__a,700__a", all 100__a
## and 700__a fields are matched together. Which is useful when first-author can vary for
## certain records on different systems).
##
## * COMPARISON THRESHOLD:
## a value between 0.0 and 1.0 specifying the threshold for string matches
## to determine if it is a match or not (using normalized string-distance).
## Normally 0.8 (80% match) is considered to be a close match.
##
## * COMPARISON MODE:
## the parse mode decides how the record datafields are compared:
## - 'strict' : all (sub-)fields are compared, and all must match. Order is significant.
## - 'normal' : all (sub-)fields are compared, and all must match. Order is ignored.
## - 'lazy' : all (sub-)fields are compared with each other and at least one must match
## - 'ignored': the tag is ignored in the match. Used to disable previously defined rules.
##
## * MATCHING MODE:
## the comparison mode decides how the fieldvalues are matched:
## - 'title' : uses a method specialized for comparing titles, e.g. looking for subtitles
## - 'author' : uses a special authorname comparison. Will take initials into account.
## - 'identifier' : special matching for identifiers, stripping away punctuation
## - 'date': matches dates by extracting and comparing the year
## - 'normal': normal string comparison.
## Note: Fields are considered matching when all its subfields or values match.
##
## * RESULT MODE:
## the result mode decides how the results from the comparisons are handled further:
## - 'normal' : a failed match will cause the validation to immediately exit as a failure.
## a successful match will cause the validation to continue on other rules (if any)
## - 'final' : a failed match will cause the validation to immediately exit as a failure.
## a successful match will cause validation to immediately exit as a success.
## - 'joker' : a failed match will cause the validation to continue on other rules (if any).
## a successful match will cause validation to immediately exit as a success.
##
## You can add your own rulesets in the dictionary below. The 'default' ruleset is always applied,
## and should therefore NOT be removed, but can be changed. The tag-rules can also be overwritten
## by other rulesets.
##
## WARNING: Beware that the validation quality is only as good as given rules, so matching results
## are never guaranteed to be accurate, as it is very content-specific.
CFG_BIBMATCH_MATCH_VALIDATION_RULESETS = [('default', [{ 'tags' : '245__%,242__%',
'threshold' : 0.8,
'compare_mode' : 'lazy',
'match_mode' : 'title',
'result_mode' : 'normal' },
{ 'tags' : '037__a,088__a',
'threshold' : 1.0,
'compare_mode' : 'lazy',
'match_mode' : 'identifier',
'result_mode' : 'final' },
{ 'tags' : '100__a,700__a',
'threshold' : 0.8,
'compare_mode' : 'normal',
'match_mode' : 'author',
'result_mode' : 'normal' },
{ 'tags' : '773__a',
'threshold' : 1.0,
'compare_mode' : 'lazy',
'match_mode' : 'title',
'result_mode' : 'normal' }]),
('980__ \$\$a(THESIS|Thesis)', [{ 'tags' : '100__a',
'threshold' : 0.8,
'compare_mode' : 'strict',
'match_mode' : 'author',
'result_mode' : 'normal' },
{ 'tags' : '700__a,701__a',
'threshold' : 1.0,
'compare_mode' : 'lazy',
'match_mode' : 'author',
'result_mode' : 'normal' },
{ 'tags' : '100__a,700__a',
'threshold' : 0.8,
'compare_mode' : 'ignored',
'match_mode' : 'author',
'result_mode' : 'normal' }]),
('260__', [{ 'tags' : '260__c',
'threshold' : 0.8,
'compare_mode' : 'lazy',
'match_mode' : 'date',
'result_mode' : 'normal' }]),
('0247_', [{ 'tags' : '0247_a',
'threshold' : 1.0,
'compare_mode' : 'lazy',
'match_mode' : 'identifier',
'result_mode' : 'final' }]),
('020__', [{ 'tags' : '020__a',
'threshold' : 1.0,
'compare_mode' : 'lazy',
'match_mode' : 'identifier',
'result_mode' : 'final' }])
]
## CFG_BIBMATCH_FUZZY_MATCH_VALIDATION_LIMIT -- Determines the minimum percentage of the
## amount of rules to be positively matched when comparing two records. Should the number
## of matches be lower than required matches but equal to or above this limit,
## the match will be considered fuzzy.
CFG_BIBMATCH_FUZZY_MATCH_VALIDATION_LIMIT = 0.65
## CFG_BIBMATCH_SEARCH_RESULT_MATCH_LIMIT -- Determines the maximum amount of search results
## a single search can return before acting as a non-match.
CFG_BIBMATCH_SEARCH_RESULT_MATCH_LIMIT = 15
## CFG_BIBMATCH_MIN_VALIDATION_COMPARISONS -- Determines the minimum amount of comparisons
## required for a validation process to be fully "valid". Meaning, for the default value, that
## if for example at least 2 comparisons (rules) are made, regardless of success/failure, the validation
## may or may not return as a success. Should fewer comparisons have taken place, the validation
## returns as a failure.
CFG_BIBMATCH_MIN_VALIDATION_COMPARISONS = 2
######################################
## Part 21: BibAuthorID parameters ##
######################################
# CFG_BIBAUTHORID_MAX_PROCESSES is the max number of processes
# that may be spawned by the disambiguation algorithm
CFG_BIBAUTHORID_MAX_PROCESSES = 12
# CFG_BIBAUTHORID_PERSONID_SQL_MAX_THREADS is the max number of threads
# to parallelize sql queries during personID tables updates
CFG_BIBAUTHORID_PERSONID_SQL_MAX_THREADS = 12
# CFG_BIBAUTHORID_EXTERNAL_CLAIMED_RECORDS_KEY defines the user info
# keys for externally claimed records in an remote-login scenario--e.g. from arXiv.org
# e.g. "external_arxivids" for arXiv SSO
CFG_BIBAUTHORID_EXTERNAL_CLAIMED_RECORDS_KEY =
# CFG_BIBAUTHORID_AID_ENABLED
# Globally enable AuthorID Interfaces.
# If False: No guest, user or operator will have access to the system.
CFG_BIBAUTHORID_ENABLED = True
# CFG_BIBAUTHORID_AID_ON_AUTHORPAGES
# Enable AuthorID information on the author pages.
CFG_BIBAUTHORID_ON_AUTHORPAGES = True
# CFG_BIBAUTHORID_AUTHOR_TICKET_ADMIN_EMAIL defines the eMail address
# all ticket requests concerning authors will be sent to.
CFG_BIBAUTHORID_AUTHOR_TICKET_ADMIN_EMAIL = info@invenio-software.org
# CFG_BIBAUTHORID_UI_SKIP_ARXIV_STUB_PAGE defines if the optional arXive stub page is skipped
CFG_BIBAUTHORID_UI_SKIP_ARXIV_STUB_PAGE = False
# CFG_BIBAUTHORID_DB_MAX_DATACHUNK_PER_INSERT_QUERY defines the maximum datachunk size to
# be inserted to the bibauthorid search engine tables per single insert query
# (used when the search engine indexer is updating and populating the tables).
# Bigger size means better performance. Nevertheless take into account
# that if it is too big there is the risk that the mysql timeout will run out
# and connection with the db will be lost.
CFG_BIBAUTHORID_SEARCH_ENGINE_MAX_DATACHUNK_PER_INSERT_DB_QUERY = 10000000
# CFG_BIBAUTHORID_ENABLED_REMOTE_LOGIN_SYSTEMS defines the available remote login external
# systems that a user can log in through them on Inspire
CFG_BIBAUTHORID_ENABLED_REMOTE_LOGIN_SYSTEMS = arXiv
#########################################
## Part 22: BibCirculation parameters ##
#########################################
## CFG_BIBCIRCULATION_ITEM_STATUS_OPTIONAL -- comma-separated list of statuses
# Example: missing, order delayed, not published
# You can allways add a new status here, but you may want to run some script
# to update the database if you remove some statuses.
CFG_BIBCIRCULATION_ITEM_STATUS_OPTIONAL =
## Here you can edit the text of the statuses that have specific roles.
# You should run a script to update the database if you change them after having
# used the module for some time.
## Item statuses
# The book is on loan
CFG_BIBCIRCULATION_ITEM_STATUS_ON_LOAN = on loan
# Available for loan
CFG_BIBCIRCULATION_ITEM_STATUS_ON_SHELF = on shelf
# The book is being processed by the library (cataloguing, etc.)
CFG_BIBCIRCULATION_ITEM_STATUS_IN_PROCESS = in process
# The book has been ordered (bought)
CFG_BIBCIRCULATION_ITEM_STATUS_ON_ORDER = on order
# The order of the book has been cancelled
CFG_BIBCIRCULATION_ITEM_STATUS_CANCELLED = cancelled
# The order of the book has not arrived yet
CFG_BIBCIRCULATION_ITEM_STATUS_NOT_ARRIVED = not arrived
# The order of the book has not arrived yet and has been claimed
CFG_BIBCIRCULATION_ITEM_STATUS_CLAIMED = claimed
# The book has been proposed for acquisition and is under review.
CFG_BIBCIRCULATION_ITEM_STATUS_UNDER_REVIEW = under review
## Loan statuses
# This status should not be confussed with CFG_BIBCIRCULATION_ITEM_STATUS_ON_LOAN.
# If the item status is CFG_BIBCIRCULATION_ITEM_STATUS_ON_LOAN, then there is
# a loan with status CFG_BIBCIRCULATION_LOAN_STATUS_ON_LOAN or
# CFG_BIBCIRCULATION_LOAN_STATUS_EXPIRED.
# For each copy, there can only be one active loan ('on loan' or 'expired') at
# the time, since can be many 'returned' loans for the same copy.
CFG_BIBCIRCULATION_LOAN_STATUS_ON_LOAN = on loan
# The due date has come and the item has not been returned
CFG_BIBCIRCULATION_LOAN_STATUS_EXPIRED = expired
# The item has been returned.
CFG_BIBCIRCULATION_LOAN_STATUS_RETURNED = returned
## Request statuses
# There is at least one copy available, and this is the oldest request.
CFG_BIBCIRCULATION_REQUEST_STATUS_PENDING = pending
# There are no copies available, or there is another request with more priority.
CFG_BIBCIRCULATION_REQUEST_STATUS_WAITING = waiting
# The request has become a loan
CFG_BIBCIRCULATION_REQUEST_STATUS_DONE = done
# The request has been cancelled
CFG_BIBCIRCULATION_REQUEST_STATUS_CANCELLED = cancelled
# The request has been generated for a proposed book
CFG_BIBCIRCULATION_REQUEST_STATUS_PROPOSED = proposed
# ILL request statuses
CFG_BIBCIRCULATION_ILL_STATUS_NEW = new
CFG_BIBCIRCULATION_ILL_STATUS_REQUESTED = requested
CFG_BIBCIRCULATION_ILL_STATUS_ON_LOAN = on loan
CFG_BIBCIRCULATION_ILL_STATUS_RETURNED = returned
CFG_BIBCIRCULATION_ILL_STATUS_CANCELLED = cancelled
CFG_BIBCIRCULATION_ILL_STATUS_RECEIVED = received
#Book proposal statuses
CFG_BIBCIRCULATION_PROPOSAL_STATUS_NEW = proposal-new
CFG_BIBCIRCULATION_PROPOSAL_STATUS_ON_ORDER = proposal-on order
CFG_BIBCIRCULATION_PROPOSAL_STATUS_PUT_ASIDE = proposal-put aside
CFG_BIBCIRCULATION_PROPOSAL_STATUS_RECEIVED = proposal-received
# Purchase statuses
CFG_BIBCIRCULATION_ACQ_STATUS_NEW = new
CFG_BIBCIRCULATION_ACQ_STATUS_ON_ORDER = on order
CFG_BIBCIRCULATION_ACQ_STATUS_PARTIAL_RECEIPT = partial receipt
CFG_BIBCIRCULATION_ACQ_STATUS_RECEIVED = received
CFG_BIBCIRCULATION_ACQ_STATUS_CANCELLED = cancelled
## Library types
# Normal library where you have your books. I can also be a depot.
CFG_BIBCIRCULATION_LIBRARY_TYPE_INTERNAL = internal
# external libraries for ILL.
CFG_BIBCIRCULATION_LIBRARY_TYPE_EXTERNAL = external
# The main library is also an internal library.
# Since you may have several depots or small sites you can tag one of them as
# the main site.
CFG_BIBCIRCULATION_LIBRARY_TYPE_MAIN = main
# It is also an internal library. The copies in this type of library will NOT
# be displayed to borrowers. Use this for depots.
CFG_BIBCIRCULATION_LIBRARY_TYPE_HIDDEN = hidden
## Amazon access key. You will need your own key.
# Example: 1T6P5M3ZDMW9AWJ212R2
CFG_BIBCIRCULATION_AMAZON_ACCESS_KEY =
######################################
-## Part 22: BibClassify parameters ##
+## Part 23: BibClassify parameters ##
######################################
# CFG_BIBCLASSIFY_WEB_MAXKW -- maximum number of keywords to display
# in the Keywords tab web page.
CFG_BIBCLASSIFY_WEB_MAXKW = 100
########################################
-## Part 23: Plotextractor parameters ##
+## Part 24: Plotextractor parameters ##
########################################
## CFG_PLOTEXTRACTOR_SOURCE_BASE_URL -- for acquiring source tarballs for plot
## extraction, where should we look? If nothing is set, we'll just go
## to arXiv, but this can be a filesystem location, too
CFG_PLOTEXTRACTOR_SOURCE_BASE_URL = http://arxiv.org/
## CFG_PLOTEXTRACTOR_SOURCE_TARBALL_FOLDER -- for acquiring source tarballs for plot
## extraction, subfolder where the tarballs sit
CFG_PLOTEXTRACTOR_SOURCE_TARBALL_FOLDER = e-print/
## CFG_PLOTEXTRACTOR_SOURCE_PDF_FOLDER -- for acquiring source tarballs for plot
## extraction, subfolder where the pdf sit
CFG_PLOTEXTRACTOR_SOURCE_PDF_FOLDER = pdf/
## CFG_PLOTEXTRACTOR_DOWNLOAD_TIMEOUT -- a float representing the number of seconds
## to wait between each download of pdf and/or tarball from source URL.
CFG_PLOTEXTRACTOR_DOWNLOAD_TIMEOUT = 2.0
## CFG_PLOTEXTRACTOR_CONTEXT_LIMIT -- when extracting context of plots from
## TeX sources, this is the limitation of characters in each direction to extract
## context from. Default 750.
CFG_PLOTEXTRACTOR_CONTEXT_EXTRACT_LIMIT = 750
## CFG_PLOTEXTRACTOR_DISALLOWED_TEX -- when extracting context of plots from TeX
## sources, this is the list of TeX tags that will trigger 'end of context'.
CFG_PLOTEXTRACTOR_DISALLOWED_TEX = begin,end,section,includegraphics,caption,acknowledgements
## CFG_PLOTEXTRACTOR_CONTEXT_WORD_LIMIT -- when extracting context of plots from
## TeX sources, this is the limitation of words in each direction. Default 75.
CFG_PLOTEXTRACTOR_CONTEXT_WORD_LIMIT = 75
## CFG_PLOTEXTRACTOR_CONTEXT_SENTENCE_LIMIT -- when extracting context of plots from
## TeX sources, this is the limitation of sentences in each direction. Default 2.
CFG_PLOTEXTRACTOR_CONTEXT_SENTENCE_LIMIT = 2
######################################
-## Part 24: WebStat parameters ##
+## Part 25: WebStat parameters ##
######################################
# CFG_WEBSTAT_BIBCIRCULATION_START_YEAR defines the start date of the BibCirculation
# statistics. Value should have the format 'yyyy'. If empty, take all existing data.
CFG_WEBSTAT_BIBCIRCULATION_START_YEAR =
######################################
-## Part 25: Web API Key parameters ##
+## Part 26: Web API Key parameters ##
######################################
# CFG_WEB_API_KEY_ALLOWED_URL defines the web apps that are going to use the web
# API key. It has three values, the name of the web app, the time of life for the
# secure url and if a time stamp is needed.
#CFG_WEB_API_KEY_ALLOWED_URL = [('search/\?', 3600, True),
# ('rss', 0, False)]
CFG_WEB_API_KEY_ALLOWED_URL = []
##########################################
-## Part 26: WebAuthorProfile parameters ##
+## Part 27: WebAuthorProfile parameters ##
##########################################
# CFG_WEBAUTHORPROFILE_CACHE_EXPIRED_DELAY_LIVE consider a cached element expired after days
# when loading an authorpage, thus recomputing the content live
CFG_WEBAUTHORPROFILE_CACHE_EXPIRED_DELAY_LIVE = 7
# CFG_WEBAUTHORPROFILE_CACHE_EXPIRED_DELAY_BIBSCHED consider a cache element expired after days,
# thus recompute it, bibsched daemon
CFG_WEBAUTHORPROFILE_CACHE_EXPIRED_DELAY_BIBSCHED = 5
# CFG_WEBAUTHORPROFILE_MAX_COLLAB_LIST: limit collaboration list.
# Set to 0 to disable limit.
CFG_WEBAUTHORPROFILE_MAX_COLLAB_LIST = 100
# CFG_WEBAUTHORPROFILE_MAX_KEYWORD_LIST: limit keywords list
# Set to 0 to disable limit.
CFG_WEBAUTHORPROFILE_MAX_KEYWORD_LIST = 100
# CFG_WEBAUTHORPROFILE_MAX_FIELDCODE_LIST: limit frequent keyword list
# Set to 0 to disable limit.
CFG_WEBAUTHORPROFILE_MAX_FIELDCODE_LIST = 100
# CFG_WEBAUTHORPROFILE_MAX_AFF_LIST: limit affiliations list
# Set to 0 to disable limit.
CFG_WEBAUTHORPROFILE_MAX_AFF_LIST = 100
# CFG_WEBAUTHORPROFILE_MAX_COAUTHOR_LIST: limit coauthors list
# Set to 0 to disable limit.
CFG_WEBAUTHORPROFILE_MAX_COAUTHOR_LIST = 100
# CFG_WEBAUTHORPROFILE_MAX_HEP_CHOICES: limit HepRecords choices
# Set to 0 to disable limit.
CFG_WEBAUTHORPROFILE_MAX_HEP_CHOICES = 10
# CFG_WEBAUTHORPROFILE_USE_BIBAUTHORID: use bibauthorid or exactauthor
CFG_WEBAUTHORPROFILE_USE_BIBAUTHORID = False
# CFG_WEBAUTHORPROFILE_USE_ALLOWED_FIELDCODES: use allowed fieldcodes
CFG_WEBAUTHORPROFILE_USE_ALLOWED_FIELDCODES = True
# CFG_WEBAUTHORPROFILE_ALLOWED_FIELDCODES: allowed fieldcodes
CFG_WEBAUTHORPROFILE_ALLOWED_FIELDCODES = ['Astrophysics', 'Accelerators', 'Computing', \
'Experiment-HEP', 'Gravitation and Cosmology', 'Instrumentation', 'Lattice', \
'Math and Math Physics', 'Theory-Nucl', 'Other', 'Phenomenology-HEP', 'General Physics', \
'Theory-HEP', 'Experiment-Nucl']
CFG_WEBAUTHORPROFILE_CFG_HEPNAMES_EMAIL = authors@inspirehep.net
#CFG_WEBAUTHORPROFILE_SERIALIZER = [msgpack|pickle]
#Select which serializer is used to store caches in the database
CFG_WEBAUTHORPROFILE_SERIALIZER = msgpack
# CFG_WEBAUTHORPROFILE_ORCID_ENDPOINT_*
# Configure the endpoint server used to connect to ORCID
CFG_WEBAUTHORPROFILE_ORCID_ENDPOINT_PUBLIC = http://pub.orcid.org/
# http://sandbox-1.orcid.org/
CFG_WEBAUTHORPROFILE_ORCID_ENDPOINT_MEMBER = http://api.orcid.org/
# http://api.sandbox-1.orcid.org/
####################################
####################################
-## Part 27: BibSort parameters ##
+## Part 28: BibSort parameters ##
####################################
## CFG_BIBSORT_ENABLED -- whetever to enable or disable record sorting
## by bibsort.
CFG_BIBSORT_ENABLED = 1
## CFG_BIBSORT_BUCKETS -- the number of buckets bibsort should use.
## If 0, then no buckets will be used (bibsort will be inactive).
## If different from 0, bibsort will be used for sorting the records.
## The number of buckets should be set with regards to the size
## of the repository; having a larger number of buckets will increase
## the sorting performance for the top results but will decrease
## the performance for sorting the middle results.
## We recommend to to use 1 in case you have less than about
## 1,000,000 records.
## When modifying this variable, re-run rebalancing for all the bibsort
## methods, for having the database in synch.
CFG_BIBSORT_BUCKETS = 1
####################################
## Part 26: Developer options ##
####################################
## CFG_DEVEL_SITE -- is this a development site? If it is, you might
## prefer that it does not do certain things. For example, you might
## not want WebSubmit to send certain emails or trigger certain
## processes on a development site. Put "0" for "no" (meaning we are
## on production site), put "1" for "yes" (meaning we are on
## development site), or put "9" for "maximum debugging info" (which
## will be displayed to *all* users using Flask DebugToolbar, so
## please beware).
## If you do *NOT* want to send emails to their original recipients
## set up corresponding value to CFG_EMAIL_BACKEND (e.g. dummy, locmem).
CFG_DEVEL_SITE = 0
## CFG_DEVEL_TEST_DATABASE_ENGINES -- do we want to enable different testing
## database engines for testing Flask and SQLAlchemy? This setting
## will allow `*_flask_tests.py` to run on databases defined bellow.
## It uses `CFG_DATABASE_*` config variables as defaults for every
## specified engine. Put following keys to the testing database
## configuration dictionary in order to overwrite default values:
## * `engine`: SQLAlchemy engine + driver
## * `username`: The user name.
## * `password`: The database password.
## * `host`: The name of the host.
## * `port`: The port number.
## * `database`: The database name.
## EXAMPLE:
## CFG_DEVEL_TEST_DATABASE_ENGINES = {
## 'PostgreSQL': {'engine': 'postgresql'},
## 'SQLite': {'engine': 'sqlite+pysqlite', 'username': None,
## 'password': None, 'host': None, 'database': None}
## }
## }
CFG_DEVEL_TEST_DATABASE_ENGINES = {}
## CFG_DEVEL_TOOLS -- list of development tools to enable or disable.
## Currently supported tools are:
## * debug-toolbar: Flask Debug Toolbar
## * werkzeug-debugger: Werkzeug Debugger (for Apache)
## * sql-logger: Logging of run_sql SQL queries
## * inspect-templates: Template inspection (formerly CFG_WEBSTYLE_INSPECT_TEMPLATES)
## * no-https-redirect: Do not redirect HTTP to HTTPS
## * assets-debug: Jinja2 assets debugging (i.e. do not merge JavaScript files)
## * intercept-redirects: Intercept redirects (requires debug-toolbar enabled).
## * winpdb-local: Embedded WinPDB Debugger (default password is Change1Me)
## * winpdb-remote: Remote WinPDB Debugger (default password is Change1Me)
## * pydev: PyDev Remote Debugger
##
## IMPORTANT: For werkzeug-debugger, winpdb and pydev to work with Apache you
## must set WSGIDaemonProcess processes=1 threads=1 in invenio-apache-vhost.conf.
CFG_DEVEL_TOOLS =
########################################
-## Part 28: JsTestDriver parameters ##
+## Part 29: JsTestDriver parameters ##
########################################
## CFG_JSTESTDRIVER_PORT -- server port where JS tests will be run.
CFG_JSTESTDRIVER_PORT = 9876
############################
-## Part 29: RefExtract ##
+## Part 30: RefExtract ##
############################
## Refextract can automatically submit tickets (after extracting refereces)
## to CFG_REFEXTRACT_TICKET_QUEUE if it is set
CFG_REFEXTRACT_TICKET_QUEUE = None
## Override refextract kbs locations
CFG_REFEXTRACT_KBS_OVERRIDE = {}
##################################
-## Part 30: CrossRef parameters ##
+## Part 31: CrossRef parameters ##
##################################
## CFG_CROSSREF_USERNAME -- the username used when sending request
## to the Crossref site.
CFG_CROSSREF_USERNAME =
## CFG_CROSSREF_PASSWORD -- the password used when sending request
## to the Crossref site.
CFG_CROSSREF_PASSWORD =
## CFG_CROSSREF_EMAIL -- crossref query services email
CFG_CROSSREF_EMAIL =
#####################################
-## Part 31: WebLinkback parameters ##
+## Part 32: WebLinkback parameters ##
#####################################
## CFG_WEBLINKBACK_TRACKBACK_ENABLED -- whether to enable trackback support
## 1 to enable, 0 to disable it
CFG_WEBLINKBACK_TRACKBACK_ENABLED = 0
####################################
-## Part 32: WebSubmit parameters ##
+## Part 33: WebSubmit parameters ##
####################################
## CFG_WEBSUBMIT_USE_MATHJAX -- whether to use MathJax and math
## preview panel within submissions (1) or not (0). Customize your
## websubmit_template.tmpl_mathpreview_header() to enable for specific
## fields.
## See also CFG_WEBSEARCH_USE_MATHJAX_FOR_FORMATS
CFG_WEBSUBMIT_USE_MATHJAX = 0
############################
## Part 34: BibWorkflow ##
############################
## Setting worker that will be used to execut workflows.
## Allowed options: Celery
CFG_BIBWORKFLOW_WORKER = worker_celery
## Messages broker for worker
## RabbitMQ - amqp://guest@localhost//
## Redis - redis://localhost:6379/0
CFG_BROKER_URL = amqp://guest@localhost:5672//
## Broker backend
## RabbitMQ - amqp
## Redis - redis://localhost:6379/0
CFG_CELERY_RESULT_BACKEND = amqp
####################################
## Part 35: BibField parameters ##
####################################
## CFG_BIBFIELD_MASTER_FORMATS -- the name of all the allowed master formats
## that BibField will work with.
CFG_BIBFIELD_MASTER_FORMATS = marc
####################################
## Part 36: HEPData parameters ##
####################################
## CFG_HEPDATA_URL -- defines the URL of the server hosting HEPData
CFG_HEPDATA_URL = http://hepdata.cedar.ac.uk
## CFG_HEPDATA_PLOTSIZE -- defines the width and height of plots that are
## displayed in the hepdata tab on record pages
CFG_HEPDATA_PLOTSIZE = 200
## CFG_HEPDATA_THREADS_NUM defines the default number of concurrent
## harvesting threads used when retrieving HEPData records.
CFG_HEPDATA_THREADS_NUM = 1
## CFG_HEPDATA_INDEX -- describes a name of the index used to link to
## datasets to their parent records
CFG_HEPDATA_INDEX = hepdataparent
## CFG_HEPDATA_FIELD -- The name of the field which should be used for search
CFG_HEPDATA_FIELD = hepdataparent
##########################
## Part 37: PdfChecker ##
##########################
## CFG_ARXIV_URL_PATTERN -- the url to use for fetching arxiv pdfs
CFG_ARXIV_URL_PATTERN = http://export.arxiv.org/pdf/%sv%s.pdf
+#########################
+## Redis Configuration ##
+#########################
+
+## CFG_REDIS_HOSTS -- the list of redis hosts to connect too
+## This is a list of dictionaries with connection information.
+## e.g. CFG_REDIS_HOSTS = [{'db': 0, 'host': '127.0.0.1', 'port': 7001}]
+CFG_REDIS_HOSTS = {'default': [{'db': 0, 'host': '127.0.0.1', 'port': 6379}]}
+
##########################
## THAT's ALL, FOLKS! ##
##########################
diff --git a/invenio/base/static/css/invenio.css b/invenio/base/static/css/invenio.css
index dc9ace26b..ef8bd814e 100644
--- a/invenio/base/static/css/invenio.css
+++ b/invenio/base/static/css/invenio.css
@@ -1,4055 +1,4068 @@
/*
* -*- mode: text; coding: utf-8; -*-
This file is part of Invenio.
Copyright (C) 2009, 2010, 2011, 2012, 2013 CERN.
Invenio is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of the
License, or (at your option) any later version.
Invenio is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public License
along with Invenio; if not, write to the Free Software Foundation, Inc.,
59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
*/
body {
color: #000;
background: #fff;
margin: 0px;
padding: 0px;
font-family: sans-serif;
}
h1 {
font-size: 173%;
font-weight: 700;
margin-top: 5px;
margin-left: 0px;
color: #36c;
background: transparent;
}
.h1 {
font-size: 173%;
font-weight: 700;
margin-left: 0px;
color: #36c;
background: transparent;
}
h2 {
font-size: 144%;
font-weight: 700;
margin-left: 0px;
color: #36c;
background: transparent;
}
h3 {
font-size: 120%;
font-weight: 600;
font-variant: small-caps;
margin-top: 40px;
margin-left: 0px;
margin-bottom: 10px;
color: #36c;
background: transparent;
border-bottom: dotted 2px #36c;
width: 50%;
}
h4 {
font-size: 110%;
font-weight: 600;
font-style: italic;
color: #36c;
margin-left: 0px;
background: transparent;
}
h5 {
font-size: 110%;
font-weight: 400;
color: #36c;
margin-left: 0px;
background: transparent;
}
h6 {
font-size: 110%;
font-weight: 200;
font-style: italic;
color: #36c;
margin-left: 0px;
background: transparent;
}
a:link {
color: #00c;
background: transparent;
}
a:visited {
color: #006;
background: transparent;
}
a:active {
color: #00c;
background: transparent;
}
a:hover {
color: #00c;
background: transparent;
}
a.moreinfo:link {
color: #060;
background: transparent;
}
a.moreinfo:visited {
color: #060;
background: transparent;
}
a.moreinfo:active {
color: #060;
background: transparent;
}
a.moreinfo:hover {
color: #060;
background: transparent;
}
a.examplequery:link {
color: #00c;
background: transparent;
}
a.examplequery:visited {
color: #006;
background: transparent;
}
a.examplequery:active {
color: #00c;
background: transparent;
}
a.examplequery:hover {
color: #00c;
background: transparent;
}
a.img:hover {
color: #00c;
background: transparent;
}
a.img:active {
color: #00c;
background: transparent;
font-weight: normal;
}
a.note:link {
color: #666;
background: transparent;
}
a.note:visited {
color: #666;
background: transparent;
}
a.note:active {
color: #666;
background: transparent;
}
a.note:hover {
color: #666;
background: transparent;
}
a.nodecoration:link {
color: #000;
text-decoration: none;
}
a.nodecoration:visited {
color: #000;
text-decoration: none;
}
a.nodecoration:active {
color: #000;
text-decoration: none;
}
a.nodecoration:hover {
color: #000;
text-decoration: underline;
}
th.searchboxheader a:link {
color: #000;
text-decoration: none;
}
th.searchboxheader a:visited {
color: #000;
text-decoration: none;
}
th.searchboxheader a:active {
color: #000;
text-decoration: none;
}
th.searchboxheader a:hover {
color: #000;
}
a.searchoptionlink:link {
padding:3px;
border: 1px solid transparent;
color: #6E5907;
text-decoration:none;
}
a.searchoptionlink:visited {
padding:3px;
border: 1px solid transparent;
color: #6E5907;
text-decoration:none;
}
a.searchoptionlink:hover {
padding:3px;
border: 1px dotted #CBA50E;
color: #6E5907;
text-decoration:underline;
}
a.searchoptionlink:active, a.searchoptionlink:visited:active {
padding:3px;
border: 1px solid #CBA50E;
background-color: #D4D4A0;
color: #6E5907;
text-decoration:none;
}
a.searchoptionlinkselected:link {
border: 1px solid #CBA50E;
padding:3px;
background-color: #D4D4A0;
text-decoration:none;
color: #6E5907;
}
a.searchoptionlinkselected:visited {
border: 1px solid #CBA50E;
padding:3px;
background-color: #D4D4A0;
text-decoration:none;
color: #6E5907;
}
a.searchoptionlinkselected:hover {
border: 1px solid #CBA50E;
padding:3px;
background-color: #D4D4A0;
text-decoration:none;
color: #6E5907;
}
a.searchoptionlinkselected:active, a.searchoptionlinkselected:visited:active{
border: 1px dotted #CBA50E;
padding:3px;
background-color: transparent;
text-decoration:none;
color: #6E5907;
}
.list-group-item.active a {
color: #ffffff;
}
.headerbox {
color: #000;
/*background: transparent;*/
width: 100%;
padding: 0px;
margin-top: 0px;
margin-bottom: 0px;
/*border-collapse: collapse;*/
background-color: #E6E6FA;
-webkit-background-clip:padding;
/* background-color: #f00;*/
}
.headerlogo {
/*background-color: #79d;*/
/*background-color: #36c;*/
/*background-color: #369;*/
/*background-color: #E6E6FA;*/
/*background-image: url(header_background.gif);*/
}
.headerboxbody {
color: #000;
padding: 5px 0px 0px 0px;
margin: 0px;
}
.headerboxbodylogo a, .headerboxbodylogo a:visited{
padding-left: 40px;
background: transparent url(site_logo_small.gif) no-repeat top left;
color: #36c;
font-size: 20px;
font-weight: bold;
font-variant: small-caps;
letter-spacing: 3px;
padding-bottom: 6px;
float: left;
clear: none;
margin-top: 5px;
margin-bottom: 5px;
text-decoration: none;
}
.home .headerboxbodylogo a, .home .headerboxbodylogo a:visited{
margin-left: 5px;
padding-left: 90px;
background: transparent url(site_logo.gif) no-repeat top left;
height: 75px;
line-height: 3em;
font-size: 26px;
letter-spacing: 2px;
text-decoration: none;
}
.headermodulebox {
color: #fff;
background: transparent;
border-spacing: 0px;
margin: 0px;
padding: 0px;
}
.headermoduleboxbody {
color: #000;
background: #f6f6fa;
font-size: x-small;
font-weight: bold;
text-align: center;
border-left: 2px solid #79d;
border-right: 2px solid #79d;
border-top: 2px solid #79d;
margin: 0px;
padding: 2px 10px;
width: 75px;
border-bottom: 2px solid #36c;
}
.headermoduleboxbody:hover {
background: #fff;
}
.headermoduleboxbodyblank {
width: 12px;
padding: 2px 5px;
margin: 0px;
border-bottom: 2px solid #36c;
}
.headermoduleboxbodyblanklast {
padding: 0px;
margin: 0px;
width:100%;
border-bottom: 2px solid #36c;
}
.headermoduleboxbodyselected {
color: #36c;
background: #fff;
font-size: x-small;
font-weight: bold;
text-align: center;
/*border-bottom: 2px solid #fff;*/
border-left: 2px solid #36c;
border-right: 2px solid #36c;
border-top: 2px solid #36c;
margin: 5px;
padding: 2px 10px;
width: 75px;
}
a.header:link {
color: #68d;
text-decoration: none;
white-space: nowrap;
}
a.header:visited {
color: #68d;
text-decoration: none;
white-space: nowrap;
}
a.header:active {
color: #68d;
text-decoration: none;
white-space: nowrap;
}
a.header:hover {
color: #36c;
text-decoration: underline;
white-space: nowrap;
}
a.headerselected:link {
color: #36c;
text-decoration: none;
font-weight: bold;
white-space: nowrap;
}
a.headerselected:visited {
color: #36c;
text-decoration: none;
font-weight: bold;
white-space: nowrap;
}
a.headerselected:active {
color: #36c;
text-decoration: none;
font-weight: bold;
white-space: nowrap;
}
a.headerselected:hover {
color: #36c;
text-decoration: none;
font-weight: bold;
white-space: nowrap;
}
.navtrailbox {
color: #36c;
background: #fff;
padding: 0px;
margin-top: 7px;
border-spacing: 0px;
border-collapse: collapse;
font-size: x-small;
}
.navtrailboxbody {
color: #36c;
padding: 0px 0px 0px 10px;
border-spacing: 0px;
background: #fff;
font-size: x-small;
}
a.navtrail:link {
color: #36c;
background: transparent;
}
a.navtrail:visited {
color: #36c;
background: transparent;
}
a.navtrail:active {
color: #36c;
background: transparent;
}
a.navtrail:hover {
color: #36c;
background: transparent;
}
.info {
color: #060;
background: transparent;
}
.snapshot {
color: #000;
background: transparent;
border: 2px solid #999;
margin: 10px 10px 0px 40px;
}
.pageheader {
color: #999;
font-size: x-small;
background: transparent;
padding: 0px;
margin: 0px;
width: 100%;
}
.pagebody {
color: #000;
background: transparent;
margin:0px;
padding: 20px;
}
.pagebodystripeleft {
color: #000;
background: #fff;
font-size: x-small;
width: 120px;
margin: 0px;
padding-left: 10px;
float: left;
}
.pagebodystripemiddle {
color: #000;
background: #fff;
padding: 0px;
margin: 0px;
}
.pagebodystriperight {
color: #000;
background: #fff;
font-size: x-small;
width: 120px;
float: right;
}
.pageboxlefttop {
color: #000;
background: transparent;
font-size: x-small;
}
.pageboxlefttopadd {
color: #000;
background: transparent;
font-size: x-small;
}
.pageboxleftbottom {
color: #000;
background: transparent;
font-size: x-small;
}
.pageboxleftbottomadd {
color: #000;
background: transparent;
font-size: x-small;
}
.pageboxrighttop {
color: #000;
background: transparent;
font-size: x-small;
}
.pageboxrighttopadd {
color: #000;
background: transparent;
font-size: x-small;
}
.pageboxrightbottom {
color: #000;
background: transparent;
font-size: x-small;
}
.pageboxrightbottomadd {
color: #000;
background: transparent;
font-size: x-small;
}
.pagefooter {
color: #666;
background: #fff;
font-size: x-small;
margin-top: 50px;
padding: 0px;
border-top: 1px solid #666;
width: 100%;
clear: both;
}
.pagefooterstripeleft {
color: #666;
background: #fff;
font-size: x-small;
margin-left: 5px;
width: 60%;
float: left;
text-align: left;
}
.pagefooterstriperight {
color: #666;
background: #fff;
margin-right: 5px;
font-size: x-small;
text-align: right;
}
a.footer:link {
color: #666;
background: transparent;
}
a.footer:visited {
color: #666;
background: transparent;
}
a.footer:active {
color: #666;
background: transparent;
}
a.footer:hover {
color: #666;
background: transparent;
}
.errorbox {
color: #000;
background: #ffcccc;
padding: 1px;
margin: 5px 30px 5px 30px;
border-collapse: collapse;
border: 2px solid #900;
}
.errorboxheader {
color: #000;
background: #ffcccc;
padding: 3px;
border-spacing: 0px;
font-weight: bold;
text-align: left;
}
.errorboxbody {
color: #000;
background: #ffcccc;
padding: 3px;
}
.searchbox {
color: #000;
background: #fff;
padding: 1px;
margin: 5px 0px 5px 0px;
border-collapse: collapse;
/*border-top: 1px solid #36c;*/
}
.home .searchbox {
margin-top: 20px;
margin-bottom: 10px;
}
.lightsearch input.formbutton{
font-size: medium;
}
.searchboxheader {
color: #000;
/*background: #f1f1f1;
padding: 3px;*/
border-spacing: 0px;
font-size: small;
text-align: left;
}
.searchboxbody {
color: #000;
background: #fff;
padding: 3px;
}
.narrowsearchbox {
color: #000;
background: #fff;
padding: 1px;
margin: 20px 20px 5px 0px;
border-collapse: collapse;
/*border-top: 1px solid #36c;*/
}
.narrowsearchboxheader {
color: #000;
/*background: #f1f1f1;*/
padding: 3px;
border-spacing: 0px;
font-size: small;
text-align: left;
}
.narrowsearchboxbody {
color: #000;
/*background: #fff;*/
padding: 3px;
}
.focusonsearchbox {
color: #000;
/*background: #f0f8ff;*/
padding: 1px;
margin: 20px 20px 5px 0px;
border-collapse: collapse;
/*border-top: 1px solid #36c;*/
}
.focusonsearchboxheader {
color: #000;
/*background: #e6e6fa;*/
padding: 3px;
border-spacing: 0px;
font-size: small;
text-align: left;
}
.focusonsearchboxbody {
color: #000;
/*background: #f0f8ff;*/
padding: 3px;
}
.searchalsosearchbox {
color: #000;
/*background: #fffbf5;*/
padding: 1px;
margin: 20px 20px 5px 0px;
border-collapse: collapse;
/*border-top: 1px solid #36c;*/
}
+span.collection-father-has-grandchildren a{
+ font-weight:800;
+}
+ul.collection-second-level{
+ padding-left: 0;
+ margin: 0;
+}
+ul.collection-second-level li{
+ font-size:small;
+ list-style: none;
+ display: inline;
+ white-space: nowrap;
+}
.searchalsosearchboxheader {
color: #000;
/*background: #ffeacc;*/
padding: 3px;
border-spacing: 0px;
font-size: small;
text-align: left;
}
.searchalsosearchboxbody {
color: #444;
/*background: #fffbf5;*/
padding: 3px;
}
.latestadditionsbox {
color: #000;
background: #fff;
padding: 5px;
margin: 5px 20px 5px 0px;
border-spacing: 5px;
}
.latestadditionsboxtimebody {
color: #000;
background: #fff;
padding: 3px;
white-space: nowrap;
text-align: right;
vertical-align: top;
font-size: xx-small;
}
.latestadditionsboxrecordbody {
color: #000;
background: #fff;
padding: 3px;
text-align: left;
vertical-align: top;
font-size: small;
}
.portalbox {
color: #000;
background: #fff;
margin: 0px 0px 15px 0px;
border-collapse: collapse;
border-top: 1px solid #abb;
font-size: small;
width: 100%;
}
.portalboxheader {
color: #000;
background: #f1f1f1;
padding: 2px;
border-spacing: 0px;
border-bottom: 1px solid #999;
text-align: left;
font-size: small;
}
.portalboxbody {
color: #000;
background: #fff;
padding: 2px;
font-size: small;
}
.admin_wvar, .admin_w200, .admin_wvar_nomargin {
color: #000;
background: white;
padding: 1px;
margin: 0px 0px 5px 20px;
border-spacing: 0px;
border-top: 1px solid #36c;
}
.admin_w200 {
width: 250px;
}
.admin_wvar_nomargin {
margin: 0px;
}
tr.admin_row_highlight:hover{
background-color:#eee;
}
tr.admin_row_highlight:hover td.extracolumn{
background-color:#fff;
}
.admin_row_color{
background-color:#EBF7FF;
}
.adminlabel {
width: 100px;
font-size: small;
background: #f1f1f1;
vertical-align: top;
}
.adminheader, .adminheaderleft, .adminheadercenter, .adminheaderright {
color: #000;
background: #f1f1f1;
border-spacing: 0px;
font-size: small;
padding: 3px 5px;
text-align: center;
}
.adminheaderleft {
text-align: left;
}
.adminheaderright {
text-align: right;
}
.adminbutton {
color: #fff;
background: #36c;
font-weight: bold;
margin: 5px 10px 5px 10px;
border-collapse: collapse;
border-top: 1px solid #36c;
}
.admintd, .admintdleft, .admintdright {
font-size: small;
padding: 0px 10px;
text-align: center;
vertical-align: top;
}
.admintdleft {
text-align: left;
}
.admintdright {
text-align: right;
}
a.google:link {
color: #333;
background: transparent;
}
a.google:visited {
color: #333;
background: transparent;
}
a.google:active {
color: #333;
background: transparent;
}
a.google:hover {
color: #333;
background: transparent;
}
.googlebox {
color: #333;
background: #fff;
text-align: left;
margin-left: auto;
margin-right: auto;
margin-top: 50px;
padding: 10px;
font-size: small;
border-collapse: collapse;
border-top: 1px solid #fc0;
}
.googleboxheader {
color: #333;
background: #ffc;
font-weight: normal;
font-size: small;
vertical-align: top;
}
.googleboxbody {
color: #333;
background: #fff;
padding: 0px 5px 0px 5px;
font-size: small;
text-align: left;
vertical-align: top;
}
.youraccountbox {
color: #000;
background: #fff;
padding: 1px;
margin: 5px 0px 5px 0px;
border-collapse: collapse;
border-top: 1px solid #fc0;
}
.youraccountheader {
color: #333;
background: #ffc;
font-weight: normal;
font-size: small;
vertical-align: top;
text-align: left;
}
.youraccountbody {
color: #333;
background: #fff;
padding: 0px 5px 0px 5px;
margin-bottom:5px;
font-size: small;
text-align: left;
vertical-align: top;
}
th.youraccountheader a:link, th.youraccountheader a:visited {
color:#000000;
text-decoration:none;
}
th.youraccountheader a:hover {
text-decoration:underline;
}
.adminbox {
color: #000;
background: #f1f1f1;
margin: 0px;
padding: 0px;
width: 120px;
}
.adminboxheader {
color: #000;
background: #f1f1f1;
font-size: x-small;
text-align: left;
}
.adminboxbody {
color: #000;
background: #f1f1f1;
font-size: x-small;
}
.formbutton {
color: #fff;
background: #36c;
font-weight: bold;
}
.headline {
color: #36c;
background: transparent;
}
.quicknote {
color: #603;
background: transparent;
}
.important {
color: #f00;
background: transparent;
}
.popupselected {
color: #fff;
background: #006;
}
.searchresultsbox {
color: #000;
background: #ffe;
padding: 0px;
margin-top: 15px;
border-collapse: collapse;
border-top: 1px solid #fc0;
width: 100%;
}
.searchresultsboxheader {
color: #000;
background: #ffc;
padding: 2px;
border-spacing: 0px;
text-align: left;
font-weight: normal;
}
.searchresultsboxbody {
color: #000;
background: #ffe;
border-top: 1px dotted #fc0;
border-bottom: 1px dotted #fc0;
padding: 2px;
}
.searchresultsboxrecords {
color: #000;
background: transparent;
margin-left: 0px;
margin-right: 20px;
}
.nearesttermsbox {
color: #603;
background: #ffe;
padding: 0px;
border-collapse: collapse;
}
.nearesttermsboxheader {
color: #603;
background: #ffc;
padding: 0px;
border-spacing: 0px;
text-align: left;
font-weight: normal;
}
.nearesttermsboxbody {
color: #603;
background: #fff;
padding: 0px;
}
.searchservicebox {
margin-top: 15px;
margin-bottom: 4px;
font-size: small;
}
.searchservicebox a.searchserviceitem {
color: #333;
}
.searchservicebox a.searchserviceitem:hover {
color: #36c;
}
.searchservicebox a.lessserviceitemslink, .searchservicebox a.moreserviceitemslink {
font-size: smaller;
color: #060;
}
a.nearestterms:link {
color: #603;
background: transparent;
}
a.nearestterms:visited {
color: #603;
background: transparent;
}
a.nearestterms:active {
color: #603;
background: transparent;
}
a.nearestterms:hover {
color: #603;
background: transparent;
}
.nearesttermsboxbodyselected {
color: #999;
background: #fff;
padding: 0px;
}
a.nearesttermsselected:link {
color: #999;
background: transparent;
}
a.nearesttermsselected:visited {
color: #999;
background: transparent;
}
a.nearesttermsselected:active {
color: #999;
background: transparent;
}
a.nearesttermsselected:hover {
color: #999;
background: transparent;
}
.moreinfo {
color: #060;
font-size: small;
background: transparent;
}
.examplequery {
color: #060;
font-size: x-small;
background: transparent;
}
.rankscoreinfo {
color: #666;
font-size: x-small;
background: transparent;
}
.userinfobox {
color: #039;
font-size: x-small;
width: 150px;
margin-bottom: 15px;
}
.userinfoboxheader {
color: #039;
font-size: x-small;
font-weight: bold;
border-top: 1px solid #060;
border-bottom: 1px solid #060;
}
.userinfoboxbody {
color: #039;
padding: 5px 5px 2px 0px;
font-size: 11px;
font-weight: normal;
float: right;
clear: none;
}
a.userinfo:link {
color: #039;
background: transparent;
}
a.userinfo:visited {
color: #039;
background: transparent;
}
a.userinfo:active {
color: #039;
background: transparent;
}
a.userinfo:hover {
color: #039;
background: transparent;
}
a.langinfo:link {
color: #666;
background: transparent;
}
a.langinfo:visited {
color: #666;
background: transparent;
}
a.langinfo:active {
color: #666;
background: transparent;
}
a.langinfo:hover {
color: #666;
background: transparent;
}
.faq {
margin-left: 12%;
margin-right: 3%;
}
.faqq {
margin-left: 18%;
margin-right: 3%;
}
.exampleleader {
color: #060;
background: transparent;
}
.example {
color: #039;
background: transparent;
}
.blocknote {
color: #000;
background: #ccc;
}
.blocknotebis {
color: #000;
background: #999;
}
.devel {
color: #600;
background: #fff;
border-color: #600;
border-left-width: medium;
border-left-style: solid;
font-size: 90%;
}
.normal {
color: #000;
background: #fff;
}
.address {
font-style: normal;
font-size: x-small;
}
.note {
color: #666;
background: transparent;
}
.warning {
color: #603;
background: transparent;
}
.light {
color: #ccc;
background: transparent;
}
.nbdoccoll {
color: #666;
background: transparent;
}
hr {
width: 100%;
height: 1px;
color: #999;
background-color: #999;
border-width: 0;
}
form input[type="text"], form input[type="password"], form select {
color: #000;
background: #fff;
border: 1px solid #555;
}
form input[disabled], form input[disabled="true"]{
/* <http://reference.sitepoint.com/css/attributeselector#compatibilitysection> */
color: #666;
}
.wsactionbutton {
width: 150px;
height: 25px;
color: #039;
margin: 0px;
background-color: #fff;
border: 2px solid #039;
vertical-align: middle;
font-size: small;
padding: 5px 5px 0px 5px;
}
.wsactionbuttonh {
width: 150px;
height: 25px;
color: #039;
margin: 0px;
background-color: #9cf;
border: 2px solid #039;
vertical-align: middle;
font-size: small;
padding: 5px 5px 0px 5px;
}
.textbutton {
color: #039;
font-weight: bold;
text-decoration: none;
}
.submitBody {
color: #000;
background: #9cf;
}
.submitHeader {
color: #fff;
background: #006;
}
.submitCurrentPage {
color: #000;
background: #9cf;
border-top: 1px solid #039;
border-left: 1px solid #039;
border-right: 1px solid #039;
}
.submitEmptyPage {
color: #fff;
background: #fff;
border-bottom: 1px solid #039;
}
.submitPage {
color: #000;
background: #fff;
border-top: 1px solid #039;
border-left: 1px solid #039;
border-right: 1px solid #039;
}
.mycdscell {
border-right: 1px solid #fff;
}
.guideimg {
border: 2px dotted #777;
padding: 5px;
margin: 5px;
}
.guideheader {
font-size: 120%;
font-weight: 600;
font-variant: small-caps;
color: #36c;
margin-left: 10px;
background: transparent;
}
.recordlastmodifiedbox {
text-align: left;
font-size: small;
color: #603;
background: #fff;
}
.commentbox {
/*color: #000;*/
width: 100%;
padding: 0px 10px 10px 10px;
border-left: 2px solid #36c;
margin-left: 10px;
}
/* Write comment */
#comment-write {
border:1px solid #ccc;
margin-top:30px;
margin-right:50px;
padding:25px 20px 5px ; /* 25 20 20 */
position:relative;
clear:right;
}
#comment-write h2 {
background:#fff;
font-size:1.4em;
font-weight:400;
left:40px;
padding:0 5px;
position:absolute;
top:-32px;
}
#comment-write .submit-area {
margin:20px 0 0 -5px;
}
/* Subscribe to comment*/
.comment-subscribe {
/*color:#f00;*/
font-size:small;
overflow:hidden;
padding:7px 0 5px 5px;
position:relative;
width:95%;
background:#dfe6f2;
/*border:1px solid #ccc;*/
}
.warninggreen {
color: #060;
background: transparent;
}
.warningred {
color: #f00;
background: transparent;
}
.reportabuse {
color: #000;
background: #fff;
font-size: small;
vertical-align: bottom;
}
/* WebMessage module */
.mailbox{
border-collapse: collapse;
color: #000;
margin-top: 15px;
padding: 0px;
width: auto;
}
.mailboxheader tr{
background: #ffc;
}
.inboxheader {
text-align:center;
padding: 5px 30px 5px 30px;
border-top: 1px solid #fc0;
border-bottom: 1px solid #fc0;
}
.messageheader{
width: 100%;
padding: 0px;
border: 0px;
}
.mailboxinput{
width: 100%;
}
.mailboxlabel{
white-space: nowrap;
padding-right: 15px;
}
.mailboxbody{
background: #ffe;
}
.mailboxrecord{
/* each record */
}
.mailboxrecord td{
/* each cell of a record */
padding: 4px 30px 4px 30px;
border-top: 1px dashed #fff;
}
.mailboxfooter{
background-color: #fff;
}
.mailboxfooter td{
padding: 10px 0px 0px 0px;
border-top: 1px solid #fc0;
border-bottom: none;
border-left: none;
border-right: none;
}
.mailboxsearch td {
padding-top: 10px;
padding-bottom: 10px;
}
.mailboxresults td {
padding-bottom: 5px;
border-bottom: 1px solid #fc0;
}
.nonsubmitbutton {
color: #000;
background: #fc0;
font-weight: bold;
}
.confirmoperation{
margin: auto;
width: 400px;
height: 100px;
background-color: #ddf;
}
.confirmmessage{
font-weight: bold;
text-align: center;
}
.infobox{
background-color: #ffc;
padding: 7px;
border-collapse: collapse;
border: 1px solid #fc0;
}
.warningbox{
background-color: #cff;
padding: 7px;
border-collapse: collapse;
border: 1px solid #ccff00;
}
.quotabox{
background-color: #ffc;
width: 200px;
height: 15px;
border: 1px solid #fc0;
margin: 3px 0px 3px 0px;
}
.quotabar{
background-color: #fc0;
border: 0px none black;
height: 15px;
}
/* WebBasket module */
#bskcontainer{
/* float: left;*/
background: transparent;
width: 100%;
}
ul.bsk_export_as_list{
list-style-image: url(arrow_link-icon-15x11-right.gif) ;
padding-left: 1em;
margin:0.05em;
}
.bsk_export_as_list, .bsk_export_as_list a{
text-decoration:none;
color:#555;
font-weight:700;
font-size:small;
}
/*START The search box container*/
#webbasket_container_search_box{
width: 100%;
}
/*END The search box container*/
/*START The search results container*/
#webbasket_container_search_results{
width: 100%;
}
.webbasket_search_results_results_overview_cell{
border-top: 1px #fc0 solid;
border-bottom: 1px #fc0 dotted;
background-color: #ffc;
}
.webbasket_search_results_number_of_items{
color: gray;
}
.webbasket_search_results_basket{
font-size: 80%;
padding: 0;
}
.webbasket_search_results_basket ol{
padding-top: 0;
margin-top: 0;
}
/*END The search box container*/
/*START The directory box container*/
#bskcontainerdirectory{
width: 100%;
}
.bsk_directory_box{
width: 100%;
}
.bsk_directory_box_tabs{
width: 100%;
}
.bsk_directory_box_tab_separator{
width: 1px;
border-bottom: 1px solid #fc0;
}
.bsk_directory_box_tab_separator_end{
width: 100%;
border-bottom: 1px solid #fc0;
}
.bsk_directory_box_tab_content{
border: 1px solid #fc0;
background: #fda;
padding: 5px;
white-space: nowrap;
}
.bsk_directory_box_tab_content a, .bsk_directory_box_tab_content a:link, .bsk_directory_box_tab_content a:visited, .bsk_directory_box_tab_content a:active{
text-decoration: none;
font-weight: bold;
color: #444;
}
.bsk_directory_box_tab_content a:hover{
text-decoration: underline;
font-weight: bold;
color: #000;
}
.bsk_directory_box_tab_content_selected{
border-top: 1px solid #fc0;
border-right: 1px solid #fc0;
border-left: 1px solid #fc0;
background: #ffc;
padding: 5px;
white-space: nowrap;
}
.bsk_directory_box_tab_content_selected a, .bsk_directory_box_tab_content_selected a:link, .bsk_directory_box_tab_content_selected a:visited, .bsk_directory_box_tab_content_selected a:active{
text-decoration: none;
font-weight: bold;
color: #000;
}
.bsk_directory_box_tab_content_selected a:hover{
text-decoration: underline;
font-weight: bold;
color: #000;
}
.bsk_directory_box_tab_content_inactive{
border: 1px solid #fc0;
background: #ffd;
padding: 5px;
white-space: nowrap;
font-weight: bold;
color: #d3d3d3;
}
.bsk_directory_box_nav_tab_content{
width: 100%;
border-bottom: 1px solid #fc0;
border-top: 1px solid #fc0;
border-left: 1px solid #fc0;
background: #ffc;
padding: 5px;
white-space: nowrap;
}
.bsk_directory_box_nav_tab_content a, .bsk_directory_box_nav_tab_content a:link, .bsk_directory_box_nav_tab_content a:visited, .bsk_directory_box_nav_tab_content a:active{
text-decoration: none;
font-weight: bold;
color: #000;
}
.bsk_directory_box_nav_tab_content a:hover{
text-decoration: underline;
font-weight: bold;
color: #000;
}
.bsk_directory_box_nav_tab_options{
border-bottom: 1px solid #fc0;
border-top: 1px solid #fc0;
border-right: 1px solid #fc0;
background: #ffc;
padding: 5px;
white-space: nowrap;
}
.bsk_directory_box_nav_tab_options img{
vertical-align: middle;
margin-right: 3px;
border: none;
}
.bsk_directory_box_nav_tab_options a, .bsk_directory_box_nav_tab_options a:link, .bsk_directory_box_nav_tab_options a:visited, .bsk_directory_box_nav_tab_options a:active{
text-decoration: none;
color: #000;
font-size: small;
}
.bsk_directory_box_nav_tab_options a:hover{
text-decoration: underline;
color: #000;
font-size: small;
}
.bsk_directory_box_nav_extra_options{
padding: 5px;
white-space: nowrap;
text-align: right;
}
.bsk_directory_box_nav_extra_options img{
vertical-align: middle;
margin-right: 3px;
border: none;
}
.bsk_directory_box_nav_extra_options a, .bsk_directory_box_nav_extra_options a:link, .bsk_directory_box_nav_extra_options a:visited, .bsk_directory_box_nav_extra_options a:active{
text-decoration: none;
color: #000;
font-size: small;
}
.bsk_directory_box_nav_extra_options a:hover{
text-decoration: underline;
color: #000;
font-size: small;
}
.bsk_directory_box_content{
width: 100%;
}
.bsk_directory_box_content_list_topics_groups{
width: 100%;
text-align: center;
border-left: 1px solid #fc0;
border-right: 1px solid #fc0;
border-bottom: 1px solid #fc0;
background: #ffc;
padding: 5px;
}
.bsk_directory_box_content_list_baskets{
width: 100%;
text-align: center;
border-left: 1px solid #fc0;
border-right: 1px solid #fc0;
border-bottom: 1px solid #fc0;
background: #fff;
padding: 5px;
}
.bsk_directory_box_content_list_cell{
text-align: left;
white-space: nowrap;
padding: 5px;
vertical-align: top;
}
.bsk_directory_box_content_list_cell a, .bsk_directory_box_content_list_cell a:link, .bsk_directory_box_content_list_cell a:visited, .bsk_directory_box_content_list_cell a:active{
text-decoration: none;
color: #444;
}
.bsk_directory_box_content_list_cell a:hover{
text-decoration: underline;
color: #000;
}
.bsk_directory_box_content_list_number_of{
color: #808080;
}
/*END The directory box container*/
/*START List public baskets*/
.bsk_list_public_baskets{
margin-bottom: 20px;
border-collapse: collapse;
border: 1px solid #fc0;
background-color:white;
}
.bsk_list_public_baskets_header{
background-color: #ffc;
padding: 10px;
border-bottom: 1px solid #fc0;
border-collapse: collapse;
vertical-align: top;
white-space: nowrap;
font-weight: bold;
}
.bsk_list_public_baskets_header a, .bsk_list_public_baskets_header a:link, .bsk_list_public_baskets_header a:visited, .bsk_list_public_baskets_header a:active{
text-decoration: none;
color: #000;
}
.bsk_list_public_baskets_header a:hover{
text-decoration: underline;
color: #000;
}
.bsk_list_public_baskets_header img{
border: none;
vertical-align: bottom;
}
.bsk_list_public_baskets_footer{
background-color: #ffc;
padding: 10px;
border-top: 1px solid #fc0;
border-collapse: collapse;
vertical-align: bottom;
font-size: small;
}
.bsk_list_public_baskets_footer a, .bsk_list_public_baskets_footer a:link, .bsk_list_public_baskets_footer a:visited, .bsk_list_public_baskets_footer a:active{
text-decoration: none;
color: #000;
}
.bsk_list_public_baskets_footer a:hover{
text-decoration: underline;
color: #000;
}
.bsk_list_public_baskets_footer img{
border: none;
vertical-align: bottom;
}
.bsk_list_public_baskets_basket_left{
padding-top: 5px;
padding-bottom: 5px;
padding-right: 15px;
padding-left: 10px;
text-align: left;
border-right: 1px solid #fc0;
}
.bsk_list_public_baskets_basket_right{
padding-top: 5px;
padding-bottom: 5px;
padding-right: 10px;
padding-left: 15px;
text-align: right;
border-right: 1px solid #fc0;
}
/*END List public baskets*/
/*START The content container*/
#bskcontainercontent{
width: 100%;
}
#bskcontent{
float: left;
width: 100%;
border-collapse: collapse;
background: #ffe;
border: none;
}
.bsktopic{
white-space: nowrap;
font-weight: bold;
}
.bsktopic a{
font-weight: normal;
text-decoration: none;
color: #000;
}
.bsktopic a:hover{
text-decoration: underline;
}
.bsktopic_{
white-space: nowrap;
font-weight: bold;
}
.bsktopic_ a{
font-weight: normal;
text-decoration: none;
color: #000;
}
.bsktopic_ a:hover{
text-decoration: underline;
}
#bskbaskets{
padding: 10px;
}
#bskinfos{
background-color: transparent;
}
.bskbasket{
margin-bottom: 20px;
border-collapse: collapse;
border: 1px solid #fc0;
background-color:white;
}
.bskbasketheader{
background-color: #ffc;
padding: 5px;
border: 1px solid #fc0;
border-collapse: collapse;
vertical-align: top;
}
.bskbasketheadertitle{
width: 100%;
border: none;
}
.bskbasketheaderoptions{
text-align: right;
vertical-align: top;
white-space: nowrap;
border: none;
font-size: small;
}
.bskbasketheaderoptions a, .bskbasketheaderoptions a:link, .bskbasketheaderoptions a:visited, .bskbasketheaderoptions a:active{
text-decoration: none;
color: black;
}
.bskbasketheaderoptions a:hover{
text-decoration: underline;
color: black;
}
.bskbasketheaderoptions img{
vertical-align: middle;
margin-right: 3px;
border: none;
}
.bskbasketfooter{
background-color: #ffc;
padding: 5px;
border: 1px solid #fc0;
border-collapse: collapse;
vertical-align: bottom;
}
.bskbasketfootertitle{
width: 100%;
border: none;
vertical-align: bottom;
text-align: left;
}
.bskbasketfooteroptions{
text-align: right;
vertical-align: bottom;
white-space: nowrap;
border: none;
font-size: small;
}
.bskbasketfooteroptions a, .bskbasketfooteroptionsoptions a:link, .bskbasketfooteroptionsoptions a:visited, .bskbasketfooteroptionsoptions a:active{
text-decoration: none;
color: black;
}
.bskbasketfooteroptions a:hover{
text-decoration: underline;
color: black;
}
.bskbasketfootertitle a, .bskbasketfootertitleoptions a:link, .bskbasketfootertitleoptions a:visited, .bskbasketfootertitleoptions a:active{
text-decoration: underline;
color: black;
}
.bskbasketfootertitle a:hover{
text-decoration: underline;
color: black;
}
.bskbasketfooter img{
vertical-align: middle;
margin-right: 3px;
border: none;
}
/*START Notes header, content and footer*/
.bsknotesheadertitle{
width: 100%;
border: none;
}
.bsknotesheaderoptions{
text-align: right;
vertical-align: bottom;
white-space: nowrap;
border: none;
font-size: small;
}
.bsknotesheaderoptions a, .bsknotesheaderoptions a:link, .bsknotesheaderoptions a:visited, .bsknotesheaderoptions a:active{
text-decoration: none;
color: black;
}
.bsknotesheaderoptions a:hover{
text-decoration: underline;
color: black;
}
.bsknotesheaderoptions img{
vertical-align: middle;
margin-right: 3px;
border: none;
}
.bsknotesfootertitle{
width: 100%;
border: none;
vertical-align: top;
text-align: left;
}
.bsknotesfooteroptions{
text-align: right;
vertical-align: top;
white-space: nowrap;
border: none;
font-size: small;
}
.bsknotesfooteroptions a, .bsknotesfooteroptions a:link, .bsknotesfooteroptions a:visited, .bsknotesfooteroptions a:active{
text-decoration: none;
color: black;
}
.bsknotesfooteroptions a:hover{
text-decoration: underline;
color: black;
}
.bsknotesfooteroptions img{
vertical-align: middle;
margin-right: 3px;
border: none;
}
.bsknotescontent {
width: 100%;
padding-left: 20px;
padding-top: 20px;
padding-right: 20px;
}
.bsknotescontent a, .bsknotescontent a:link, .bsknotescontent a:visited, .bsknotescontent a:active{
text-decoration: none;
color: black;
}
.bsknotescontent a:hover{
text-decoration: underline;
color: black;
}
.bsknotescontent img{
margin-right: 3px;
border: none;
}
.bsknotescontentnote {
width: 100%;
}
.bsknotescontentaddnote {
width: 100%;
}
.bsknotescontentaddform {
width: 100%;
}
.bsknotescontenttitle {
width: 100%;
border-top: 1px solid #ddd;
border-left: 1px solid #ddd;
border-right: 1px solid #ddd;
font-size: small;
}
.bsknotescontentbody {
width: 100%;
border-bottom: 1px solid #ddd;
border-left: 1px solid #ddd;
border-right: 1px solid #ddd;
font-size: small;
}
.bsknotescontentoptions {
width: 100%;
font-size: small;
text-align: right;
padding: 3px;
}
.bskcomment {
margin-bottom:20px;
margin-left:30px;
background:#F9F9F9;
border:1px solid #DDD;
font-size:small;
width: 90%;
}
.bskcommentheader {
background-color:#EEE;
padding:2px;
}
/*END Notes header, content and footer*/
.bskbasketheaderactions{
text-align: center;
white-space: nowrap;
}
.bskbasketheaderactions td{
border: none;
}
.bskbasketheaderactions img{
border: 0px;
margin: 2px;
}
.bskbasketheaderactions a{
font-size: small;
color: #000;
}
.bskcontentcount{
text-align: right;
white-space: nowrap;
vertical-align: top;
padding-top: 5px;
padding-left: 5px;
padding-bottom: 5px;
}
.bskcontentoptions{
text-align: right;
white-space: nowrap;
vertical-align: top;
padding-left: 5px;
}
.bskcontentoptions img{
border: none;
padding-right: 5px;
}
.webbasket_basket_content_item_cell {
border: 1px solid #fc0;
}
.bskactions{
text-align: center;
white-space: nowrap;
}
.bskactions td{
border: none;
}
.bskactions img{
border: 0px;
margin: 5px;
}
.bskactions a{
font-size: x-small;
}
.bsktitle {
width: 100%;
}
.bskcmtcol{
white-space: nowrap;
text-align: right;
}
.bskcontentcol{
padding-top: 5px;
padding-right: 5px;
width: 100%;
}
.bskcontentcol a{
font-size: small;
}
.bsk_create_link{
padding-top: 5px;
padding-bottom: 10px;
background-color: transparent;
}
.bsk_create_link a{
color: black;
}
.bsk_create_link img{
border: none;
}
dd{
margin-bottom: 10px;
}
.cmtsubround {
margin: 5px 15px 5px;
border-bottom: 1px dashed #bbb;
}
.cmtfilesblock{
background-color:#f5f5f5;
border-top:1px solid #eee;
padding:2px;
}
.cmtfilesblock a{
background:url(file-icon-text-12x16.gif) no-repeat;
padding-left: 14px;
margin-left: 2px;
}
/* end of WebBasket module */
/* WebSubmit module */
form.hyperlinkform {
/* used in form tag for a form that should be hidden, but contains a button styled like a hyperlink */
display: inline;
padding: 0;
margin: 0;
height: 0;
width: 0;
}
form input.hyperlinkformHiddenInput {
/* used in a hidden input tag for a form that should be hidden, but contains a button styled like a hyperlink */
display: inline;
padding: 0;
margin: 0;
height: 0;
width: 0;
}
form input.hyperlinkformSubmitButton {
/* used in a submit input tag for a form that should be hidden, but contains a button styled like a hyperlink */
display: inline;
padding: 0;
margin: 0;
border: 0;
background-color: transparent;
font-size: 1em;
line-height: 1em;
text-decoration: underline;
cursor: pointer;
color: blue;
}
/* end of WebSubmit module */
/* BibEdit module */
/* BibEdit - old interface */
.bibEditTable{
background: #ececec;
border: 0px;
padding: 0px;
border-collapse: collapse;
}
.bibEditTable th{
background: #ccc;
text-align: left;
padding: 5px;
}
.bibEditCellRight{
font-size: small;
text-align: right;
padding: 0px;
}
.bibEditCellTag{
font-size: small;
text-align: right;
vertical-align: top;
padding: 2px 5px 2px 5px;
font-weight: bold;
}
.bibEditHistView{
float: left;
margin: 0px 10px 5px 0px;
width: 70%;
}
.bibEditHistCompare{
float: left;
margin: 0px 10px 5px 0px;
font-size: small;
width: 70%;
}
.bibEditHistForms{
margin: 5px 0px 5px 10px;
}
/* BibEdit - new interface */
.bibEditBtnBold {
font-weight: bold
}
.bibEditImgCtrlEnabled {
cursor: pointer;
opacity: 1.0;
vertical-align: bottom;
}
.bibEditImgCtrlDisabled {
cursor: default;
opacity: 0.4;
vertical-align: bottom;
}
#bibEditTable {
background-color: rgb(255, 255, 255);
border: 1px solid #A1A1A1;
border-collapse: collapse;
width: 100%;
}
.bibEditColFieldBox {
min-width: 14px;
max-width: 14px;
width: 14px;
}
#bibEditColFieldTag {
min-width: 48px;
max-width: 100px;
width: 48px;
border-right: 1px solid #A1A1A1;
}
#bibEditColSubfieldTag {
width: 28px;
min-width: 28px;
max-width: 80px;
border-right: 1px solid #A1A1A1;
}
#bibEditColSubfieldAdd {
width: 16px;
min-width: 16px;
max-width: 16px;
}
#bibEditTable .bibEditCellContent:hover,
#bibEditTable .bibEditCellContentProtected:hover{
background: lightyellow;
}
#bibEditTable .bibEditFieldColored {
background-color: rgb(235, 235, 235);
border: 1px solid #A1A1A1;
}
#bibEditTable .bibEditSelected {
background: lightblue;
}
#bibEditTable .bibEditVolatileSubfield {
color: grey;
}
#bibEditTable td {
padding: 1px 1px;
font-size: 0.8em;
}
#bibEditTable .bibEditCellField,
#bibEditTable .bibEditCellFieldTag,
#bibEditTable .bibEditCellSubfield,
#bibEditTable .bibEditCellSubfieldTag {
vertical-align: top;
}
#bibEditTable .bibEditCellFieldTag {
font-weight: bold;
}
#bibEditTable .bibEditCellContentProtected {
font-weight: bold;
}
#bibEditTable .bibEditCellAddSubfields {
vertical-align: bottom;
}
#bibEditTable img {
cursor: pointer;
}
#bibEditTable .bibEditBtnMoveSubfieldUp {
float: left;
}
#bibEditTable .bibEditBtnMoveSubfieldDown {
float: right;
}
#bibEditTable input[type="text"]:not(.bibEditInputError){
border: 2px inset grey;
}
#bibEditTable input[type="checkbox"] {
margin: 1px 0px 0px;
padding: 0px;
}
#bibEditTable .bibEditBtnClear {
margin-left: 20px;
}
#bibEditTable .bibEditTxtTag {
width: 34px;
}
#bibEditTable .bibEditTxtInd {
width: 14px;
}
#bibEditTable .bibEditCellAddSubfieldCode {
text-align: right;
}
#bibEditTable .bibEditTxtSubfieldCode {
width: 14px;
}
#bibEditTable .bibEditTxtValue {
width: 100%;
}
#bibEditTable .bibEditInputError {
border: 2px solid red;
}
#bibEditMessage {
background: rgb(250, 209, 99);
}
.bibEditAddFieldManipulationsBar {
display: table;
}
.bibEditAddFieldFormSelectTemplate {
display: table-cell;
width: 400px;
}
.bibEditAddFieldFormCreateSimilar {
display: table-cell;
}
.bibeditscrollArea {
width: 200px;
height: 75px;
padding-left: 5px;
padding-right: 5px;
border-color: #6699CC;
border-width: 1px;
border-style: solid;
float: left;
overflow: auto;
}
.bibeditscrollArea ul {
margin-top: 0px;
margin-bottom: 0px;
list-style: none;
margin-left: 0;
padding-left: 0;
}
.bibeditHPCorrection{
border-width: 1px;
background-color: #FAEBD7;
font-size: smaller;
border-style: groove;
display: inline-block;
border-color: RosyBrown;
margin:5px 3px 3px 5px;
padding: 2px;
}
#bibeditHPChanges{
overflow-y: auto;
overflow-x: hidden;
max-height: 150px;
}
.bibeditHPHiddenElement{
visibility: hidden;
display: none;
}
.bibeditHPPanelEntry{
border-width: 1px;
border-style: dotted;
display: table;
background-color: #feffff;
border-color: #cbcccc;
margin-bottom: 5px;
}
.bibeditHPContentPreviewBox {
position: absolute;
z-index:0;
border-style: dotted;
border-width: 1px;
background-color: #fefeff;
top: 0px;
height: 500px;
overflow: auto;
width: 500px;
}
.bibeditHPControl{
display:table-cell;
width:20%;
margin:0px;
}
.bibeditHPEntryNumber{
display: table-cell;
width: 80%;
}
.bibeditHPEntryDateSection{
font-size: smaller;
width: 60%;
margin: 0px;
display: table-cell;
}
.bibeditHPDetailsOpener{
display: table-cell;
width: 80%;
cursor: pointer;
}
.bibeditHPInformationsSection{
}
.bibeditHPDetailsOpener{
display: table-cell;
}
.bibeditHPEntryControlsPanel{
display: table-row;
}
.bibeditHPEntryRow1{
display: table-row;
}
.bibeditHPEntryRow2{
display: table-row;
}
.bibeditHPEntryCol1{
display: table-cell;
vertical-align: middle;
}
.bibeditHPEntryCol2{
display: table-cell;
max-width: 28px;
min-width: 28px;
margin: 0px;
}
.bibeditHPPanelEntryDisabled {
background-color: #eeeeee;
color: #BBBBBB;
}
.bibEditRevHistorySelectedEntry{
background-color: #cccccc;
display: table-row;
}
.bibEditRevHistoryEntry{
background-color: #efefef;
display: table-row;
}
.bibEditRevHistoryEntryContent{
display: table-cell;
font-size: 80%;
cursor: pointer;
}
.bibEditRevHistoryEntryControls{
display: table-cell;
}
.bibEditRevHistoryLink{
display: table-cell;
border-style: none;
}
.bibEditRevHistoryLinkImg{
border-width: 0px;
border-style: none;
}
.bibEditRevHistoryMenuSection{
max-height: 200px;
overflow-y: auto;
overflow-x: hidden;
}
.bibEditHiddenElement {
display: none;
}
.bibEditURPreviewBox {
border-color: #dddddd;
border-style: dotted;
}
.bibEditURPairRow {
background-color: white;
}
.bibEditUROddRow {
background-color: #eeeeee;
}
.bibEditURPreviewHeader{
font-weight: bolder;
}
.bibEditURDescHeader{
font-weight: bolder;
}
.bibEditURDescEmptyCell{
width:20px;
}
.bibEditURDescChar{
font-style: italic;
}
.bibEditURDescVal{
}
.bibEditURMenuSection{
display: table-row;
width: 138px;
max-width: 138px;
overflow: visible;
}
.bibEditURDetailsSection{
display: table-cell;
max-width: 64px;
width: 64px;
}
.bibEditURPreviewBox{
max-width: 138px;
width: 138px;
}
.bibEditURDescEntry {
max-width: 138px;
width: 138px;
max-height: 20px;
overflow: visible;
}
.bibEditURDescEntryDetails {
position: relative;
left: 138px;
top: -20px;
overflow: visible;
background-color: white;
border-style: dotted;
border-width: 1px;
width: 300px;
}
.bibEditURDescEntrySelected{
background-color: #aaaaaa;
}
.bibEditBibCircPanel{
font-weight: normal;
}
/* end of BibEdit module */
/* BibMerge module */
#bibMergePanel {
background: white;
width: 173px;
position: fixed;
top: 220px;
left: 5px;
margin-bottom: 20px;
font-size: 0.7em;
}
.bibMergeMenuSectionHeader {
font-weight: bold;
font-size: 1.2em !important;
}
.bibMergeImgCompressMenuSection {
margin: 0px;
text-align: left;
vertical-align: bottom;
}
#bibMergePanel button {
font-size: 0.8em;
font-weight: bold;
}
.bibMergeRecNumInput {
width: 75px;
}
a.bibMergeImgClickable img {
border: 0px;
}
a:hover.bibMergeImgClickable img {
background: orange;
}
#bibMergeSearchInput {
width: 130px;
}
#bibMergeSelectList option{
border-bottom: 1px solid grey;
color: blue;
text-decoration: underline;
cursor:pointer;
width: 130px;
}
.bibMergeSelectListSelected {
background: blue !important;
color: white !important;
}
#bibMergeMessage {
border-style: solid;
border-width: 4px 1px 1px 1px;
padding: 1px;
}
#bibMergeContent {
min-height: 300px;
font-size:0.8em;
margin-left:160px;
}
#bibMergeContent a:hover img {
background: orange;
}
#bibMergeContent img {
border: 0px;
}
.bibMergeFieldGroupDiv {
width: 100%;
margin-bottom: 4px;
}
.bibMergeFieldGroupHeaderDiv {
width: 150px;
font-size:1.2em;
background-color: #6699CC;
border-bottom: 3px solid #FFCC99;
color: #FFFFFF;
padding: 1px 1px;
}
.bibMergeFieldGroupHeaderDiv > * {
color: inherit !important;
}
.bibMergeFieldTable {
border: 0.5px solid #FFCC99;
border-top: 2.5px solid #6699CC;
border-left: 2.5px solid #6699CC;
border-right: 2.5px solid #6699CC;
border-bottom: 2.5px solid #6699CC;
border-collapse:collapse;
width:100%;
table-layout: fixed;
}
.bibMergeColHeaderLeft span, .bibMergeColHeaderRight span {
font-weight: bold;
}
.bibMergeFieldTable td {
border: 0.5px solid #FFCC99;
word-wrap:break-word;
}
.bibMergeFieldTable td div {
overflow:auto;
}
.bibMergeColSubfieldTag {
border-left: 1px solid #FFCC99;
min-width: 30px;
max-width: 30px;
width: 25px;
}
.bibMergeColContent {
border-left: 1px solid #FFCC99;
width: 50%;
}
.bibMergeColDiff {
border-left: 1px solid #FFCC99;
min-width: 30px;
max-width: 30px;
width: 20px;
}
.bibMergeColContentLeft { background:#FFFCF9;}
.bibMergeColContentRight { background:#F1F1F1;}
.bibMergeColHeaderLeft { background:#FFCC99;}
.bibMergeColHeaderRight { background:#BDD5DD;}
.bibMergeColHeaderLeft a { float:right; }
.bibMergeColHeaderRight a { float:left; }
.bibMergeColActions {
border-left: 1px solid #FFCC99;
min-width: 60px;
max-width: 60px;
width: 60px;
}
.bibMergeFieldTable td {
vertical-align:top;
}
.bibMergeCellSimilarityRed {
color: red;
border-right: 5px solid red;
}
.bibMergeCellSimilarityGreen {
color: green;
border-right: 5px solid green;
}
.bibMergeDiffSpanSame {
}
.bibMergeDiffSpanIns {
background:#BDD5DD;
}
.bibMergeDiffSpanDel {
background:#FFCC99;
}
.bibMergeDiffSpanSub {
background:Pink;
}
.bibMergeDiffHighlight {
background:Yellow;
}
/* end of BibMerge module */
/* WebAlert module */
.alrtTable{
border: 1px solid black;
width: 100%;
padding: 0px;
border-collapse: collapse;
}
.alrtTable td{
border: 1px solid black;
padding: 3px;
}
/* end of WebAlert module */
/* BibClassify module */
.bibclassify {
text-align: center;
}
.bibclassify-kwlist {
width: 60%;
float: top;
margin-left: 20%;
margin-top: 20px;
margin-bottom: 20px;
font-family: Arial, Helvetica, sans-serif;
}
.bibclassify .tagcloud {
width: 60%;
float: top;
margin-left: 20%;
margin-top: 20px;
margin-bottom: 20px;
}
.bibclassify .cloud {
text-align: center;
color: red;
font-size: 80%;
margin-top: 15px;
}
.bibclassify-kwlist hr {
display: block;
clear: left;
margin: -0.66em 0;
visibility: hidden;
}
.kw-list {
float: left;
width: 47%;
overflow: auto;
height:100%;
padding:1%;
text-align: left;
}
.bibclassify .acronym {
display: none;
}
.bibclassify .author-kw {
color: #993300;
}
.bibclassify .other-kw {
color: green;
}
.bibclassify .keyword.composite.f695 {
color: green;
}
.bibclassify .keyword {
white-space:nowrap;
text-decoration: none;
}
.bibclassify .keyword.composite{
color: #3366CC;
}
.bibclassify .keyword.single {
color: #666666;
}
.bibclassify a.keyword:hover {
text-decoration: underline;
}
.bibclassify .nav-links {
text-align: left;
margin-left: 20%;
font-size: small;
padding-top:10px;
padding-bottom:10px;
}
.bibclassify-top {
width: 60%;
float: top;
margin-left: 20%;
padding: 10px;
border:2px solid #CC6633;
}
.bibclassify-nav {
background-color:#d3d3d3;
text-align: center;
}
.bibclassify-bottom {
background-color:#d3d3d3;
font-size:smaller;
text-align:center;
}
.bibclassify-marcxml {
margin-left: 10%;
text-align: left;
}
/* end of BibClassify module */
/* externalcollections */
.externalcollectionsbox {
color: #333;
background: #fffbf5;
text-align: left;
margin-left: auto;
margin-right: auto;
margin-top: 50px;
padding: 10px;
font-size: small;
border-collapse: collapse;
border-top: 1px solid #fc0;
}
.externalcollectionsboxheader {
color: #333;
background: #ffeacc;
font-weight: normal;
font-size: small;
vertical-align: top;
}
.externalcollectionsboxbody {
color: #333;
background: #fffbf5;
padding: 0px 5px 0px 5px;
font-size: small;
text-align: left;
vertical-align: top;
}
.externalcollectionsresultsbox {
color: #000;
background: #fffbf5;
padding: 0px;
margin-top: 15px;
border-collapse: collapse;
border-top: 1px solid #fc0;
width: 100%;
}
.externalcollectionsresultsboxheader {
color: #000;
background: #ffeacc;
padding: 2px;
border-spacing: 0px;
text-align: left;
font-weight: normal;
}
.externalcollectionsresultsboxbody {
color: #000;
background: #fffbf5;
border-top: 1px dotted #fc0;
border-bottom: 1px dotted #fc0;
padding: 2px;
}
.externalcollectionsresultsboxrecords {
color: #000;
background: transparent;
margin-left: 0px;
margin-right: 20px;
}
/* Start 'detailed record' boxes*/
div.detailedrecordbox, div.detailedrecordminipanel {
width:90%;
margin:auto;
position:relative;
max-width: 1280px;
display: table;
}
div.detailedrecordbox img, div.detailedrecordminipanel img{
border:none;
}
div.detailedrecordtabs{
border-bottom: 2px solid #36c;
position:relative;
}
div.detailedrecordtabs div {
width:100%;
margin:0 auto;
position:relative;
z-index:1;
width:100%;
padding:0;
bottom:-2px;
}
div.detailedrecordbox div.detailedrecordboxcontent{
border-bottom: 2px solid #36c;
border-right: 2px solid #36c;
border-left: 2px solid #36c;
padding-bottom:3px;
padding-top:1px; /* Needed, otherwise tabs with paperclip get
messed up, but created a small gap at top
of box. Override with !important if needed*/
}
div.detailedrecordminipanel{
border: 1px solid #ddd;
padding-top:3px;
padding-bottom:3px;
}
div.detailedrecordminipanel{
background-color:#f7f7f7;
}
.notopgap { margin: 0; height:0;}
.nobottomgap { margin-bottom: 0; height:0;}
.top-left-folded {
height: 10px;
}
* html .top-left-folded, * html .top-right-folded{
/*IE6 hack*/
display:none
}
/* tabs as used in detailed record pages */
ul.detailedrecordtabs{
margin-top:0px;
margin-bottom:0px;
list-style-type:none;
margin-left:0;
padding-left:0;
width:100%;
text-align:center;
}
.detailedrecordtabs li{
display:block;
margin:0;
float: left;
}
* html .detailedrecordtabs li{
/*IE 6 hack*/
width:1%;
white-space: nowrap;
}
*html #tabsSpacer {
/*IE 6 hack*/
display: none;
}
.detailedrecordtabs li a{
border-top: 2px solid #79D;
border-right: 2px solid #79D;
border-bottom: 2px solid #36c;
border-left: 2px solid #79D;
margin-right:10px;
background-color: #FFF;
padding:4px;
color: #79d;
font-size:x-small;
font-weight:bold;
text-decoration:none;
position:relative;
display:block;
height:100%;
zoom:100%; /*We loose CSS validity here, but necessary because of ie7*/
}
.detailedrecordtabs li a:hover{
text-decoration:underline;
}
.detailedrecordtabs li.on a{
color:#36c;
border-bottom: 2px solid #fff;
border-top: 2px solid #36c;
border-right: 2px solid #36c;
border-left: 2px solid #36c;
}
.detailedrecordtabs li.on a:hover, .detailedrecordtabs li.disabled a:hover{
text-decoration:none;
}
.detailedrecordtabs li.disabled a{
color:#ccf;
cursor:default;
border-top: 2px solid #ccf;
border-right: 2px solid #ccf;
border-left: 2px solid #ccf;
}
.detailedrecordtabs li.first a{
border-left: 2px solid #79d;
}
.detailedrecordtabs li.first.on a{
border-left: 2px solid #36c;
}
div.restrictedflag {
background-color: #f11;
color: #fff;
font-weight: bold;
text-transform: uppercase;
padding: 4px;
cursor: default;
font-size: small;
clear: none;
letter-spacing: 1px;
padding: 4px;
/* Define transformation origin*/
transform-origin: 0 0 ;
-moz-transform-origin: 0 0 ;
-webkit-transform-origin: 0 0 ;
-ms-transform-origin: 0 0 ;
-o-transform-origin: 0 0 ;
/* Rotate and translate */
-moz-transform: rotate(-90deg) translate(-100%, 0); /* FF3.5+ */
-o-transform: rotate(-90deg) translate(-100%, 0); /* Opera 10.5 */
-webkit-transform: rotate(-90deg) translate(-100%, 0); /* Saf3.1+, Chrome */
transform: rotate(-90deg) translate(-100%, 0); /* CSS3 (for when it gets supported) */
ms-filter: "progid:DXImageTransform.Microsoft.Matrix(M11=3.061515884555943e-16, M12=1, M21=-1, M22=3.061515884555943e-16, SizingMethod='auto expand')"; /* IE 8+ */
filter: progid:DXImageTransform.Microsoft.Matrix(M11=3.061515884555943e-16, M12=1, M21=-1, M22=3.061515884555943e-16, SizingMethod='auto expand'); /* IE 6/7 */
/* See CSS conditional statement in webstyle_templates.page_header for compatibility with ie9 */
margin-top: 50px;
-ms-transform: rotate(-90deg) translate(-150%, 0); /* IE 9+*/
position: fixed;
left: 0;
}
* html div.restrictedflag {
/*IE6 hack*/
width: 100px;
float:left;
text-align:center;
}
div.restrictedflag-pending {
background-color: #FF7011;
}
.restrictedfilerowheader{
background-color: #f11;
color: #fff;
font-weight: bold;
text-transform: uppercase;
text-align:center;
cursor: default;
font-size: small;
letter-spacing: 1px;
}
/* Actions in mini-panel of detailed record view*/
ul.detailedrecordactions{
list-style-image: url(arrow_link-icon-15x11-right.gif) ;
padding-left: 1em;
text-align:left;
margin:0.05em;
}
.detailedrecordactions, .detailedrecordactions a{
text-decoration:none;
color:#555;
font-weight:700;
font-size:small;
}
/* Grading stars for mini-panel in detailed record view*/
#detailedrecordminipanelreview div.star a{
text-indent:-9000px; /*Hide text*/
height:30px;
width:30px;
float:left;
}
#detailedrecordminipanelreview div.star{
height:30px !important;
float:left;
clear:none;
width:auto;
text-align:left;
background:url(star_empty-icon-30x30.gif) no-repeat;
}
#detailedrecordminipanelreview div.star:hover{
background-image:url(star-icon-30x30.gif) !important;
}
#detailedrecordminipanelreview:hover div.full, #detailedrecordminipanelreview:hover div.half{
background-image:url(star_empty-icon-30x30.gif) ;
}
#detailedrecordminipanelreview div.full{
background-image:url(star-icon-30x30.gif) ;
}
#detailedrecordminipanelreview div.half{
background-image:url(star_half-icon-30x30.gif) ;
}
/* Short reminder displayed as sheet of paper in detailed record pages */
#detailedrecordshortreminder {
background: #fff url(paper-texture-128x128.gif) repeat top left;
margin: 10px 10px 0px 10px;
padding:4px;
border-bottom: 1px solid #CCC;
border-right: 1px solid #CCC;
border-top: 1px solid #DDD;
border-left: 1px solid #DDD;
}
#detailedrecordshortreminder #clip {
position:relative;
clear:none;
float:left;
left: -45px;
top:-15px;
height:72px;
width:72px;
background: transparent url(paper_clip-72x72.gif) no-repeat top left;
}
#detailedrecordshortreminder #HB{
position:relative;
left: -50px;
}
/* end detailed record page*/
/* definitions for webjournal module */
.webjournalBoxList{
/*list-style-image:url("star_dot-icon-30x30.gif");*/
}
.webjournalBoxList a:visited, .webjournalBoxList a:link, .webjournalBoxList a:active{
text-decoration: none;
}
.webjournalBoxList a:hover{
text-decoration:underline;
}
/* cite summary output */
#citesummary td {
padding-left: 30px;
}
/*Table columns*/
.oddtablecolumn
{
background-color: #f0f0ff;
}
.pairtablecolumn
{
background-color: #fafaff;
}
.scrollableframe
{
overflow-y: auto;
min-height: 10px;
max-height: 300px;
height: auto;
width: 900px;
}
.normalframe
{
width: 600px;
}
.brtable
{
width: 100%;
}
/* BibCirculation CSS */
#bibcircmenu h2 {
display:none
}
#bibcircmenu ul {
list-style:none;
margin:0;
padding:0 0 0 4px;
}
#bibcircmenu {
clear:both;
width:100%;
border-bottom:2px solid #36c;
border-top:2px solid #36c;
height:auto;
padding:0;
position:relative;
padding-bottom:9px;
z-index:5;
}
#bibcircmenu ul {
padding-top:9px;
position:relative;
}
* html #bibcircmenu ul {
/* Needed for ie < 7 ONLY
That's why the star hack is used
*/
width:99%;
}
#bibcircmenu li.on a, #bibcircmenu li.on a:hover, #bibcircmenu li.on a:visited{
text-decoration:none;
color:#555;
}
#bibcircmenu li {
display:block;
float:left;
text-decoration:none;
padding: 5px 5px 5px 7px;
height:100%;
padding-right:5px;
margin-right:10px;
background: transparent url(list_sep_blue.gif) no-repeat left center;
}
#bibcircmenu li.hassubmenu{
position:relative;
}
#bibcircmenu li.hassubmenu a{
padding-right:12px;
background: transparent url(drop_down_menu_arrow_down.gif) no-repeat right center;
}
#bibcircmenu li.on.hassubmenu a{
background: transparent url(drop_down_menu_arrow_down_w.gif) no-repeat right center;
}
#bibcircmenu li.on {
background: #eee url(list_sep_grey.gif) no-repeat left center;
height:100%;
}
#bibcircmenu li a img {
vertical-align:middle;
}
#bibcircmenu li img{
border:none;
}
#bibcircmenu li.right {
float:right;
margin-right:2px;
padding-right:0;
padding-left:0;
margin-left:0;
background-image:none;
}
#bibcircmenu li a{
white-space:nowrap;
text-decoration:none;
margin:0;
padding:3px 6px;
}
#bibcircmenu li ul.submenu {
display:none;
position: absolute;
background-image:none;
top: 1em;
left: 0px;
z-index:10;
}
#bibcircmenu li ul.submenu li{
background:#FFFFFF none repeat scroll 0 0;
border-bottom:1px solid #3366CC;
border-left:1px solid #3366CC;
border-right:1px solid #3366CC;
display:block !important;
float:none !important;
position:relative;
}
#bibcircmenu li ul.subsubmenu{
left: 7em;
top: -5px;
background:#FFFFFF none repeat scroll 0 0;
border-bottom:none;
border-left:none;
border-right:none;
display:none !important;
float:left !important;
position:absolute;
z-index:20;
}
#bibcircmenu li ul.subsubmenu li{
background:#FFFFFF none repeat scroll 0 0;
border-bottom:1px solid #3366CC;
border-left:1px solid #3366CC;
border-right:1px solid #3366CC;
border-top:1px solid #3366CC;
display:block !important;
float:none !important;
position:relative;
z-index:20;
}
#bibcircmenu li ul.submenu li:hover ul.subsubmenu{
display:block !important;
}
#bibcircmenu li ul.submenu li ul.subsubmenu:hover{
display:block !important;
}
#bibcircmenu li ul.submenu li a, #bibcircmenu li ul.submenu li a:hover, #bibcircmenu li ul.submenu li a:visited{
display:block !important;
color:#36c;
}
#bibcircmenu li ul.submenu li a:hover{
text-decoration:underline
}
#bibcircmenu li:hover ul {
display:block;
}
div.bibcircbottom {
clear:both;
width:100%;
border-bottom:2px solid #36c;
height:auto;
padding:0;
position:relative;
padding-bottom:9px;
z-index:0;
}
.bibcirctable {
font-size: 82%;
width:100%;
margin:auto;
max-width: 1280px;
}
.bibcirctable_contents {
color: #79d;
font-size: small;
width:90%;
margin:auto;
max-width: 300px;
}
.bibcirctableheader {
color: #79d;
font-size: small;
font-weight: bold;
font-variant: small-caps;
}
.bibcircwarning {
color: #f11010;
}
.bibcircpending {
color: #f16f10;
}
.clear {
clear: both;
}
.bibcircok {
color: #060;
}
.bibcirccontent {
color: black;
}
.formbutton {
color: #fff;
background: #36c;
font-weight: bold;
}
.bibcircbutton {
background-color:#fff;
border:1px solid;
border-color:#cdcdcd;
}
.bibcircbuttonover {
background-color:#cadff5;
border:1px solid;
border-color:#cdcdcd;
}
.bibcircsubtitle {
width: 600px;
padding-left: 5px;
color: #36c;
font-size: 16px;
font-variant: small-caps;
padding-bottom:2px;
}
.bibcircnotes {
border-width: 0px;
border-spacing: 1px;
border-style: none;
border-color: #cfcfcf;
border-collapse: collapse;
background-color: white;
}
.bibcircnotes th {
border-width: 1px;
padding: 2px;
border-style: solid;
border-color: #cfcfcf;
background-color: white;
-moz-border-radius: 0px;
}
.bibcircnotes td {
border-width: 1px;
padding: 2px;
border-style: solid;
border-color: #cfcfcf;
background-color: white;
-moz-border-radius: 0px;
}
.bibcircinfoboxsuccess{
background-color: #dcfdc1;
padding: 20px;
color: #3e5e25;
border-collapse: collapse;
border: 1px solid #88d04d;
text-align: left;
width: 50%;
}
.bibcircinfoboxmsg{
background-color: #cadff5;
padding: 20px;
color: #336598;
border-collapse: collapse;
border: 1px solid #336598;
text-align: left;
width: 50%;
}
/* Rollover menus */
.menu ul {
list-style:none;
margin:0;
padding:0 0 0 0px;
}
.menu li {
display:block;
float:left;
text-decoration:none;
padding: 7px;
height:100%;
margin-right:10px;
/*/*//*/
display : none;
/* */
}
.menu .hassubmenu {
position:relative;
}
.menu .hassubmenu a {
/*display: inline-block; /* Necessary on IE */
padding-right:12px;
background: transparent url(drop_down_menu_arrow_down_b.gif) no-repeat right center;
}
.menu .on.hassubmenu a {
background: transparent url(drop_down_menu_arrow_down_b.gif) no-repeat right center;
}
.menu .on {
height:100%;
}
.menu a img {
vertical-align:middle;
}
.menu img {
border:none;
}
.menu .right {
float:right;
margin-right:2px;
padding-right:0;
padding-left:0;
margin-left:0;
background-image:none;
}
.menu a {
white-space:nowrap;
text-decoration:none;
margin:0;
padding:0px 6px;
}
.menu ul.subsubmenu {
display:none;
position: absolute;
background-image:none;
top: 1em;
left: 0;
z-index: 99;
text-align: left;
}
.menu ul.subsubmenu li {
display:list-item !important;
float: none !important;
position: relative;
border-bottom: 1px solid #36c;
border-left: 1px solid #36c;
border-right: 1px solid #36c;
background-color:#fff;
background-image:none !important;
}
.menu ul.subsubmenu li a, .menu div ul.subsubmenu li a:hover, .menu div ul.subsubmenu li a:visited {
background-color:transparent;
background-image:none !important;
display:block;
text-transform:capitalize;
color: #36c;
}
.menu ul.subsubmenu li a:hover {
text-decoration:underline
}
.menu :hover ul {
display:block;
position: absolute;
}
/* Override/Customize rollover menus in some cases*/
.menu .headermoduleboxbody .hassubmenu a {
/*background: transparent url(drop_down_menu_arrow_down_lb.gif) no-repeat right center;*/
background-image:url(drop_down_menu_arrow_down_lb.gif)
}
.menu .headermoduleboxbody .hassubmenu a.header:hover {
/*background: transparent url(drop_down_menu_arrow_down_b.gif) no-repeat right center;*/
background-image:url(drop_down_menu_arrow_down_b.gif)
}
.snippetbox {
margin: 5px 2px 2px 10px;
padding: 3px;
color: #222;
background: #f5f9ff;
border: solid 1px #ddd;
font-size: small;
}
.oaiHarvestPagerNavigationBar{
display: table-row;
}
.oaiHarvestPagerNavigationLink{
display: table-cell;
padding: 5px;
cursor: default;
}
.oaiHarvestPagerHiddenNavigationLink{
display: none;
}
.oaiHarvestPagerHiddenPage{
display: none;
}
.oaiHarvestPagerCurrentPageLink{
background-color: #a0a0ff;
}
.oaiHarvestPagerPage{
}
/* WebComment Refactor */
.webcomment_collapse_ctr_down {
outline: 0;
width:16px;
height:16px;
margin:10px 5px;
background: transparent url(webbasket_down.png) no-repeat center center !important;
display:block;
}
.webcomment_collapse_ctr_right {
outline: 0;
width:16px;
height:16px;
margin:10px 5px;
background: transparent url(webbasket_right.png) no-repeat center center !important;
display:block;
}
.webcomment_permalink, .webcomment_permalink:link,
.webcomment_permalink:visited{
visibility: hidden;
color: #C60F0F;
font-size: medium;
padding: 0 4px;
text-decoration: none;
}
.webcomment_permalink:hover {
color: white;
background-color: #C60F0F;
}
.webcomment_comment_title:hover .webcomment_permalink{
visibility: visible;
}
.webcomment_review_title_table {
border: 0px;
border-collapse: separate;
border-spacing: 5px;
padding: 5px;
width: 100%;
}
.webcomment_deleted_comment_message,
.webcomment_reported_comment_message,
.webcomment_comment_pending_approval_message,
.webcomment_review_pending_approval_message,
.webcomment_deleted_review_message {
color:#a3a3a3;font-style:italic;
}
.webcomment_deleted_comment_undelete,
.webcomment_reported_comment_unreport,
.webcomment_deleted_review_undelete,
.webcomment_reported_review_unreport,
.webcomment_comment_delete,
.webcomment_review_delete {
color:#8B0000;
}
.webcomment_comment_box {
margin-top:20px;
background:#F9F9F9;
border:1px solid #DDD;
}
.webcomment_comment_box blockquote {
margin: 10px;
}
.webcomment_comment_options {
margin: 0 0px 5px 0;
float:right;
}
.webcomment_comment_title {
background-color:#EEE;
padding:2px;
height: 26px;
line-height: 26px;
vertical-align: middle;
}
.webcomment_review_box {
background:#F9F9F9;
border:1px solid #DDD;
}
.webcomment_review_box_inner {
background-color:#EEE;
padding:2px;
}
.webcomment_review_box_inner > img {
margin-right:10px;
}
.webcomment_review_title {
font-weight: bold;
}
.webcomment_review_label_reviewed {
}
.webcomment_review_label_useful {
}
.webcomment_comment_table {
border: 0px solid black;
width: 95%;
margin:10px;
font-size:small;
}
.webcomment_container {
margin-left:10px;
margin-right:10px;
}
.clearer {
clear: both;
}
.webcomment_header_comments {
}
.webcomment_header_ratings {
}
.websomment_header_comments_label {
}
.webcomment_comment_report {
}
.webcomment_comment_reply {
}
.webcomment_comment_delete {
}
.webcomment_comment_date {
display: inline;
margin: 0 0 0 10px;
color: #666;
}
.webcomment_comment_author {
}
.webcomment_comment_avatar {
float: left;
margin: 2px 10px 2px 5px;
}
.webcomment_toggle_visibility {
float: left;
}
.webcomment_thread_block{
border-left: 1px dashed transparent;
padding-top:1px;
padding-left: 20px;
}
.webcomment_thread_block:hover{
border-left: 1px dashed #aaa;
background-color: #F9F9F9;
}
.webcomment_comment_content {
}
.webcomment_comment_options > a {
margin: 0 10px 0 0;
}
.commentbox {
width: auto;
}
.webcomment_comment_depth_1 {
margin-left: 20px;
}
.webcomment_comment_round_header {
}
.webcomment_view_all_comments {
}
.webcomment_review_first_introduction {
}
#yourcommentsmaincontent {
max-width: 1280px;
margin-left: auto;
margin-right: auto
}
.yourcommentsdisplayoptionsform {
text-align: center;
}
#yourcommentsdisplayoptions {
margin-bottom: 30px;
padding: 12px;
font-size: small;
display: inline-block;
border: 1px solid #bbb;
}
#yourcommentsdisplayoptions legend{
color: #555;
}
#yourcommentsdisplayoptions label{
font-weight: 700;
margin-left: 10px;
}
.yourcommentsrecordgroup {
margin-bottom: 20px;
}
#yourcommentsnavigationlinks {
margin-top: 20px;
margin-left: auto;
margin-right: auto;
text-align: center;
}
/* DEMOVID SUBMISSION */
.websubmit_demovid_form table{
background-color: #d6e5f4;
}
.websubmit_demovid_radio {
display: inline;
margin: 10px 15px 0 0;
}
#websubmit_demovid_samples_wrapper {
background-color: #FFF;
border: 1px solid #85b1de;
padding: 3px;
margin-bottom: 10px;
width: 533px;
height: 300px;
-webkit-border-radius: 3px;
-moz-border-radius: 3px;
border-radius: 3px;
}
#websubmit_demovid_samples {
position: relative;
width: 533px;
height: 300px;
background-color: #000;
-webkit-border-radius: 3px;
-moz-border-radius: 3px;
border-radius: 3px;
}
#websubmit_demovid_samples img {
position:absolute;
top:0;
left:0;
z-index:8;
}
#websubmit_demovid_samples img.active {
z-index:10;
}
#websubmit_demovid_samples img.last-active {
z-index:9;
}
.websubmit_demovid_form label {
display: block;
font-size: 12px;
font-weight: normal;
}
.websubmit_demovid_form > label.error {
color: #C00;
margin-bottom: 15px;
font-weight: bold;
}
.websubmit_demovid_form > div > label.error {
color: #C00;
position: relative;
top: 22px;
left: 130px;
font-weight: bold;
}
.websubmit_demovid_form > div > object {
}
.websubmit_demovid_form > input {
border: 1px solid #85b1de;
display: block;
margin-bottom: 15px;
padding: 3px;
font-size: 14px;
height: 22px;
-webkit-border-radius: 3px;
-moz-border-radius: 3px;
border-radius: 3px;
}
.websubmit_demovid_form > input:focus, .websubmit_demovid_form > textarea:focus {
border: solid 1px #33677F;
}
.websubmit_demovid_form > textarea {
border: 1px solid #85b1de;
display: block;
margin-bottom: 13px;
font-size: 14px;
padding: 3px;
line-height: 22px;
-webkit-border-radius: 3px;
-moz-border-radius: 3px;
border-radius: 3px;
}
.websubmit_demovid_form > input.error {
display: block;
margin-bottom: 3px;
border: 1px solid #C00;
}
.websubmit_demovid_form > textarea.error {
display: block;
margin-bottom: 3px;
border: 1px solid #C00;
}
.websubmit_demovid_form object {
display: block;
margin-bottom: 0px;
}
#websubmit_demovid_error {
border: 1px solid #C00;
background-color: #fff;
padding: 10px;
margin: 15px 0 15px 0;
width: 516px;
-webkit-border-radius: 3px;
-moz-border-radius: 3px;
border-radius: 3px;
}
#websubmit_demovid_preview {
margin: 0 0 10px 0;
font-size: 12px;
font-weight: normal;
-webkit-border-radius: 3px;
-moz-border-radius: 3px;
border-radius: 3px;
}
.websubmit_demovid_radio input {
border: 1px solid #85b1de;
text-align: center;
-webkit-border-radius: 3px;
-moz-border-radius: 3px;
border-radius: 3px;
}
.websubmit_demovid_submit {
position: relative;
font-weight: bold;
top: -30px;
right: 0px;
height: 29px;
padding: 0 15px 0 15px;
text-align: center;
border: none;
background-color: #444;
color: #fff;
font-size: 13px;
border-radius: 2px;
-moz-border-radius: 3px;
-webkit-border-radius: 3px;
background-image: -webkit-gradient(
linear,
left bottom,
left top,
color-stop(0, rgb(65,65,65)),
color-stop(0.7, rgb(95,95,95))
);
background-image: -moz-linear-gradient(
center bottom,
rgb(65,65,65) 0%,
rgb(95,95,95) 70%
);
}
.websubmit_demovid_form .uploadifyQueueItem {
border: 1px solid #666;
-webkit-border-radius: 3px;
-moz-border-radius: 3px;
border-radius: 3px;
}
#uploadify_loading {
display: none;
float: left;
background-color: #fff;
border: 1px solid #85B1DE;
height: 29px;
width: 29px;
border-radius: 2px;
-moz-border-radius: 3px;
-webkit-border-radius: 3px;
margin-right: 5px;
}
#uploadify_loading > img {
height: 29px;
width: 29px;
}
.websubmit_demovid_radio label {
display: inline;
}
/* VIDEO FORMAT ELEMENTS */
.video_brief_container {
}
.video_brief_thumb {
padding: 2px;
border: 1px solid #ccc;
float: left;
margin: 0 15px 0 0;
}
.video_brief_abstract {
}
.video_brief_info {
}
/* Math preview panel */
#mathpreviewarea {
display: none;
float:left;
padding:0;
border: 1px solid #eec520;
color: #000;
background-color:#ffffea;
-webkit-box-shadow: 1px 2px 5px 1px rgba(0,0,0,0.5);
-moz-box-shadow: 1px 2px 5px 1px rgba(0,0,0,0.5);
box-shadow: 1px 2px 5px 1px rgba(0,0,0,0.5);}
#mathpreviewareabar {
border-bottom:1px solid #eec520;
background-color:#eec970;
font-size:small;
color: #3B2324
}
#mathpreviewareacontent {
padding:5px;
}
#mathpreviewareaclose {
float: right;
clear: none;
margin-left: 10px;
}
#mathpreviewareabar span{float:left; clear:none}
/* Classes used for diff between two records */
.diff_field_deleted {
color: #ff0000;
}
.diff_field_added {
color: #00aa00;
}
/* end of Classes used for diff between two records */
/* BibHarvest Holding Pen style*/
#holdingpencontainer h1{
font-size:20px;
}
#holdingpencontainer h2{
font-size:16px;
}
#holdingpencontainer h3{
margin:0 0 10px 0;
border-bottom:none;
}
#holdingpencontainer > div > *{
margin-left:20px;
}
#holdingpencontainer > div > div > *{
margin-left:20px;
}
/* end od BibHarvest Holding Pen style*/
div.login_button {
margin: 5px;
border-radius: 5px;
margin-left: auto;
float: left;
}
div.login_button:hover {
box-shadow: -1px 1px #888;
}
div.login_button_big {
width: 48px;
height: 48px;
border-radius: 5px;
}
div.login_button_small {
width: 24px;
height: 24px;
border-radius: 3px;
}
div.provider_img {
float: left;
width: 32px;
cursor: pointer;
}
div.login_content {
float: left;
margin: 5px;
width: 208px;
}
a.openid_url {
text-decoration: none;
color: inherit;
}
input.openid_input {
width: 140px;
}
label.openid_label {
font-size: 10px;
}
div.with_label {
margin-top: -4px;
}
div#buttons {
float: right;
width: 50%;
}
div#big_buttons, div#small_buttons {
width: 80%;
float: left;
}
#form_field {
float: left;
}
img.login_button_big {
border-radius: 5px;
}
img.login_button_small {
border-radius: 2px;
}
.outof {
padding-left: 10px;
padding-right: 10px;
}
.linksbox {
text-align: right;
}
.empty-element {
display: none;
}
.iconlink {
text-decoration: none;
}
.iconlink:hover {
text-decoration: none;
cursor: pointer;
}
.sortlink:hover {
cursor: move;
}
.remove-element {
padding-left: 10px;
}
.add-element {
text-decoration: none;
}
.add-element:hover {
cursor: pointer;
text-decoration: none;
}
.error:input {
border-color: #b94a48;
color: #b94a48;
}
.list-group-item.active a {
color: #ffffff;
}
.panel-bot-margin {
margin-bottom: 10px;
}
/* end of invenio.css */
diff --git a/invenio/base/static/js/ckeditor/journal-editor-styles.js b/invenio/base/static/js/ckeditor/journal-editor-styles.js
index ea940314b..0beec1e12 100644
--- a/invenio/base/static/js/ckeditor/journal-editor-styles.js
+++ b/invenio/base/static/js/ckeditor/journal-editor-styles.js
@@ -1,31 +1,31 @@
/*
Defines the default styles available in the CKEditor used for the
submission of journal articles. These styles should be in sync with
the CSS styles and the HTML templates of the journal
See config.stylesSet in the configuration file.
*/
CKEDITOR.stylesSet.add( 'journal-editor-style',
[
// Inline styles
- { name : 'Article Header', element : 'span', attributes : { 'class' : 'articleHeader' } },
+ { name : 'Article Header', element : 'p', attributes : { 'class' : 'articleHeader' } },
{ name : 'Caption', element : 'span', styles : { 'class' : 'articleHeader' } },
{ name : 'Computer Code', element : 'code' },
{ name : 'Keyboard Phrase', element : 'kbd' },
{ name : 'Sample Text', element : 'samp' },
{ name : 'Variable', element : 'var' },
{ name : 'Deleted Text', element : 'del' },
{ name : 'Inserted Text', element : 'ins' },
{ name : 'Cited Work', element : 'cite' },
{ name : 'Inline Quotation', element : 'q' },
// Object styles
{ name : 'Image on Left', element : 'img', attributes : { 'class' : 'phl' } },
{ name : 'Image on Right', element : 'img', attributes : { 'class' : 'phr'} },
{ name : 'Centered Image', element : 'img', attributes : { 'class' : 'ph'} },
{ name : 'Image With Caption on Left', element : 'table', attributes : { 'class' : 'phl'} },
{ name : 'Image With Caption on Right', element : 'table', attributes : { 'class' : 'phr'} },
{ name : 'Centered Image With Caption', element : 'table', attributes : { 'class' : 'ph'} }
]);
diff --git a/invenio/ext/legacy/handler.py b/invenio/ext/legacy/handler.py
index 4787caffe..71c4e3b6d 100644
--- a/invenio/ext/legacy/handler.py
+++ b/invenio/ext/legacy/handler.py
@@ -1,450 +1,452 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013 CERN.
+## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
Apache request handler mechanism.
It gives the tools to map url to functions, handles the legacy url
scheme (/search.py queries), HTTP/HTTPS switching, language
specification,...
"""
__revision__ = "$Id$"
## Import the remote debugger as a first thing, if allowed
try:
import invenio.utils.remote_debugger as remote_debugger
except:
remote_debugger = None
import urlparse
import cgi
import sys
import re
import os
import gc
from flask import session
from invenio.utils import apache
from invenio.config import CFG_SITE_URL, CFG_SITE_SECURE_URL, \
CFG_SITE_RECORD, CFG_ACCESS_CONTROL_LEVEL_SITE
from invenio.base.i18n import wash_language
from invenio.utils.url import redirect_to_url
from invenio.ext.logging import register_exception
from invenio.legacy.webuser import get_preferred_user_language, isGuestUser, \
getUid, isUserSuperAdmin, collect_user_info, setUid
from invenio.legacy.wsgi.utils import StringField
from invenio.modules import apikeys as web_api_key
+from invenio.webinterface_handler_wsgi_utils import StringField
+from invenio.modules.access.engine import acc_authorize_action
## The following variable is True if the installation make any difference
## between HTTP Vs. HTTPS connections.
CFG_HAS_HTTPS_SUPPORT = CFG_SITE_SECURE_URL.startswith("https://")
## The following variable is True if HTTPS is used for *any* URL.
CFG_FULL_HTTPS = CFG_SITE_URL.lower().startswith("https://")
## Set this to True in order to log some more information.
DEBUG = False
# List of URIs for which the 'ln' argument must not be added
# automatically
CFG_NO_LANG_RECOGNITION_URIS = ['/rss',
'/oai2d',
'/journal']
RE_SLASHES = re.compile('/+')
RE_SPECIAL_URI = re.compile('^/%s/\d+|^/collection/.+' % CFG_SITE_RECORD)
_RE_BAD_MSIE = re.compile("MSIE\s+(\d+\.\d+)")
def _debug(req, msg):
"""
Log the message.
@param req: the request.
@param msg: the message.
@type msg: string
"""
if DEBUG:
req.log_error(msg)
def _check_result(req, result):
"""
Check that a page handler actually wrote something, and
properly finish the apache request.
@param req: the request.
@param result: the produced output.
@type result: string
@return: an apache error code
@rtype: int
@raise apache.SERVER_RETURN: in case of a HEAD request.
@note: that this function actually takes care of writing the result
to the client.
"""
if result or req.bytes_sent > 0:
if result is None:
result = ""
else:
result = str(result)
# unless content_type was manually set, we will attempt
# to guess it
if not req.content_type_set_p:
# make an attempt to guess content-type
if result[:100].strip()[:6].lower() == '<html>' \
or result.find('</') > 0:
req.content_type = 'text/html'
else:
req.content_type = 'text/plain'
if req.method == 'HEAD':
if req.status in (apache.HTTP_NOT_FOUND, ):
raise apache.SERVER_RETURN, req.status
else:
req.write(result)
return apache.OK
else:
req.log_error("publisher: %s returned nothing." % `object`)
return apache.HTTP_INTERNAL_SERVER_ERROR
class TraversalError(Exception):
"""
Exception raised in case of an error in parsing the URL of the request.
"""
pass
class WebInterfaceDirectory(object):
"""
A directory groups web pages, and can delegate dispatching of
requests to the actual handler. This has been heavily borrowed
from Quixote's dispatching mechanism, with specific adaptations.
"""
# Lists the valid URLs contained in this directory.
_exports = []
# Set this to True in order to redirect queries over HTTPS
_force_https = False
def _translate(self, component):
"""(component : string) -> string | None
Translate a path component into a Python identifier. Returning
None signifies that the component does not exist.
"""
if component in self._exports:
if component == '':
return 'index' # implicit mapping
else:
return component
else:
# check for an explicit external to internal mapping
for value in self._exports:
if isinstance(value, tuple):
if value[0] == component:
return value[1]
else:
return None
def _lookup(self, component, path):
""" Override this method if you need to map dynamic URLs.
It can eat up as much of the remaining path as needed, and
return the remaining parts, so that the traversal can
continue.
"""
return None, path
def _traverse(self, req, path, do_head=False, guest_p=True):
""" Locate the handler of an URI by traversing the elements of
the path."""
_debug(req, 'traversing %r' % path)
component, path = path[0], path[1:]
name = self._translate(component)
if name is None:
obj, path = self._lookup(component, path)
else:
obj = getattr(self, name)
if obj is None:
_debug(req, 'could not resolve %s' % repr((component, path)))
raise TraversalError()
# We have found the next segment. If we know that from this
# point our subpages are over HTTPS, do the switch.
if (CFG_FULL_HTTPS or CFG_HAS_HTTPS_SUPPORT and (self._force_https or session.need_https())) and not req.is_https():
# We need to isolate the part of the URI that is after
# CFG_SITE_URL, and append that to our CFG_SITE_SECURE_URL.
original_parts = urlparse.urlparse(req.unparsed_uri)
plain_prefix_parts = urlparse.urlparse(CFG_SITE_URL)
secure_prefix_parts = urlparse.urlparse(CFG_SITE_SECURE_URL)
# Compute the new path
plain_path = original_parts[2]
plain_path = secure_prefix_parts[2] + \
plain_path[len(plain_prefix_parts[2]):]
# ...and recompose the complete URL
final_parts = list(secure_prefix_parts)
final_parts[2] = plain_path
final_parts[-3:] = original_parts[-3:]
target = urlparse.urlunparse(final_parts)
## The following condition used to allow certain URLs to
## by-pass the forced SSL redirect. Since SSL certificates
## are deployed on INSPIRE, this is no longer needed
## Will be left here for reference.
#from invenio.config import CFG_INSPIRE_SITE
#if not CFG_INSPIRE_SITE or plain_path.startswith('/youraccount/login'):
redirect_to_url(req, target)
# Continue the traversal. If there is a path, continue
# resolving, otherwise call the method as it is our final
# renderer. We even pass it the parsed form arguments.
if path:
if hasattr(obj, '_traverse'):
return obj._traverse(req, path, do_head, guest_p)
else:
raise apache.SERVER_RETURN, apache.HTTP_NOT_FOUND
if do_head:
req.content_type = "text/html; charset=UTF-8"
raise apache.SERVER_RETURN, apache.DONE
form = req.form
#if 'ln' not in form and \
# req.uri not in CFG_NO_LANG_RECOGNITION_URIS:
# ln = get_preferred_user_language(req)
# form.add_field('ln', ln)
result = _check_result(req, obj(req, form))
return result
def __call__(self, req, form):
""" Maybe resolve the final / of a directory """
# When this method is called, we either are a directory which
# has an 'index' method, and we redirect to it, or we don't
# have such a method, in which case it is a traversal error.
if "" in self._exports:
if not form:
# Fix missing trailing slash as a convenience, unless
# we are processing a form (in which case it is better
# to fix the form posting).
redirect_to_url(req, req.uri + "/", apache.HTTP_MOVED_PERMANENTLY)
_debug(req, 'directory %r is not callable' % self)
raise TraversalError()
def create_handler(root):
""" Return a handler function that will dispatch apache requests
through the URL layout passed in parameter."""
def _handler(req):
""" This handler is invoked by mod_python with the apache request."""
allowed_methods = ("GET", "POST", "HEAD", "OPTIONS", "PUT")
#req.allow_methods(allowed_methods, 1)
#if req.method not in allowed_methods:
# raise apache.SERVER_RETURN, apache.HTTP_METHOD_NOT_ALLOWED
if req.method == 'OPTIONS':
## OPTIONS is used to now which method are allowed
req.headers_out['Allow'] = ', '.join(allowed_methods)
raise apache.SERVER_RETURN, apache.OK
# Set user agent for fckeditor.py, which needs it here
os.environ["HTTP_USER_AGENT"] = req.headers_in.get('User-Agent', '')
# Check if REST authentication can be performed
if req.args:
args = cgi.parse_qs(req.args)
if 'apikey' in args and req.is_https():
uid = web_api_key.acc_get_uid_from_request()
if uid < 0:
raise apache.SERVER_RETURN, apache.HTTP_UNAUTHORIZED
else:
setUid(req=req, uid=uid)
guest_p = isGuestUser(getUid(req), run_on_slave=False)
uri = req.uri
if uri == '/':
path = ['']
else:
## Let's collapse multiple slashes into a single /
uri = RE_SLASHES.sub('/', uri)
path = uri[1:].split('/')
if CFG_ACCESS_CONTROL_LEVEL_SITE > 1:
## If the site is under maintainance mode let's return
## 503 to casual crawler to avoid having the site being
## indexed
req.status = 503
g = _RE_BAD_MSIE.search(req.headers_in.get('User-Agent', "MSIE 6.0"))
bad_msie = g and float(g.group(1)) < 9.0
if uri.startswith('/yours') or not guest_p:
## Private/personalized request should not be cached
if bad_msie and req.is_https():
req.headers_out['Cache-Control'] = 'private, max-age=0, must-revalidate'
else:
req.headers_out['Cache-Control'] = 'private, no-cache, no-store, max-age=0, must-revalidate'
req.headers_out['Pragma'] = 'no-cache'
req.headers_out['Vary'] = '*'
elif not (bad_msie and req.is_https()):
req.headers_out['Cache-Control'] = 'public, max-age=3600'
req.headers_out['Vary'] = 'Cookie, ETag, Cache-Control'
try:
if req.header_only and not RE_SPECIAL_URI.match(req.uri):
return root._traverse(req, path, True, guest_p)
else:
## bibdocfile have a special treatment for HEAD
return root._traverse(req, path, False, guest_p)
except TraversalError:
raise apache.SERVER_RETURN, apache.HTTP_NOT_FOUND
except apache.SERVER_RETURN:
## This is one of mod_python way of communicating
raise
except IOError as exc:
if 'Write failed, client closed connection' not in "%s" % exc:
## Workaround for considering as false positive exceptions
## rised by mod_python when the user close the connection
## or in some other rare and not well identified cases.
register_exception(req=req, alert_admin=True)
raise
except Exception:
# send the error message, much more convenient than log hunting
if remote_debugger:
args = {}
if req.args:
args = cgi.parse_qs(req.args)
if 'debug' in args:
remote_debugger.error_msg(args['debug'])
register_exception(req=req, alert_admin=True)
raise
# Serve an error by default.
raise apache.SERVER_RETURN, apache.HTTP_NOT_FOUND
return _handler
def wash_urlargd(form, content):
"""
Wash the complete form based on the specification in
content. Content is a dictionary containing the field names as a
key, and a tuple (type, default) as value.
'type' can be list, str, invenio.legacy.wsgi.utils.StringField, int, tuple, or
invenio.legacy.wsgi.utils.Field (for
file uploads).
The specification automatically includes the 'ln' field, which is
common to all queries.
Arguments that are not defined in 'content' are discarded.
Note that in case {list,tuple} were asked for, we assume that
{list,tuple} of strings is to be returned. Therefore beware when
you want to use wash_urlargd() for multiple file upload forms.
@Return: argd dictionary that can be used for passing function
parameters by keywords.
"""
result = {}
content['ln'] = (str, '')
for k, (dst_type, default) in content.items():
try:
value = form[k]
except KeyError:
result[k] = default
continue
#FIXES problems with unicode arguments from Flask
if isinstance(value, unicode):
value = value.encode('utf-8')
src_type = type(value)
# First, handle the case where we want all the results. In
# this case, we need to ensure all the elements are strings,
# and not Field instances.
if src_type in (list, tuple):
if dst_type is list:
result[k] = [str(x) for x in value]
continue
if dst_type is tuple:
result[k] = tuple([str(x) for x in value])
continue
# in all the other cases, we are only interested in the
# first value.
value = value[0]
# Maybe we already have what is expected? Then don't change
# anything.
if isinstance(value, dst_type):
if isinstance(value, StringField):
result[k] = str(value)
else:
result[k] = value
continue
# Since we got here, 'value' is sure to be a single symbol,
# not a list kind of structure anymore.
if dst_type in (str, int):
try:
result[k] = dst_type(value)
except:
result[k] = default
elif dst_type is tuple:
result[k] = (str(value), )
elif dst_type is list:
result[k] = [str(value)]
else:
raise ValueError('cannot cast form value %s of type %r into type %r' % (value, src_type, dst_type))
result['ln'] = wash_language(result['ln'])
return result
diff --git a/invenio/legacy/bibauthorid/webinterface.py b/invenio/legacy/bibauthorid/webinterface.py
index b59866512..3a22cc961 100644
--- a/invenio/legacy/bibauthorid/webinterface.py
+++ b/invenio/legacy/bibauthorid/webinterface.py
@@ -1,3790 +1,3789 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2011, 2012, 2013 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
""" Bibauthorid Web Interface Logic and URL handler. """
# pylint: disable=W0105
# pylint: disable=C0301
# pylint: disable=W0613
from cgi import escape
from pprint import pformat
from operator import itemgetter
import re
try:
from invenio.utils.json import json, CFG_JSON_AVAILABLE
except:
CFG_JSON_AVAILABLE = False
json = None
from invenio.legacy.bibauthorid.webapi import add_cname_to_hepname_record
from invenio.config import CFG_SITE_URL, CFG_BASE_URL
from invenio.legacy.bibauthorid.config import AID_ENABLED, PERSON_SEARCH_RESULTS_SHOW_PAPERS_PERSON_LIMIT, \
BIBAUTHORID_UI_SKIP_ARXIV_STUB_PAGE, VALID_EXPORT_FILTERS, PERSONS_PER_PAGE, \
MAX_NUM_SHOW_PAPERS
from invenio.config import CFG_SITE_LANG, CFG_SITE_NAME, CFG_INSPIRE_SITE, CFG_SITE_SECURE_URL
from invenio.legacy.bibauthorid.name_utils import most_relevant_name
from invenio.legacy.webpage import page
from invenio.base.i18n import gettext_set_language #, wash_language
from invenio.legacy.template import load
from invenio.ext.legacy.handler import wash_urlargd, WebInterfaceDirectory
from invenio.utils.url import redirect_to_url
from invenio.legacy.webuser import (
getUid,
page_not_authorized,
collect_user_info,
set_user_preferences,
get_user_preferences,
email_valid_p,
emailUnique,
get_email_from_username,
get_uid_from_email,
isGuestUser)
from invenio.modules.access.control import acc_find_user_role_actions
from invenio.legacy.search_engine import perform_request_search
from invenio.legacy.search_engine.utils import get_fieldvalues
import invenio.legacy.bibauthorid.webapi as webapi
from flask import session
from invenio.legacy.bibauthorid.config import CREATE_NEW_PERSON
import invenio.utils.apache as apache
import invenio.legacy.webauthorprofile.interface as webauthorapi
from invenio.legacy.bibauthorid.general_utils import is_valid_orcid
from invenio.legacy.bibauthorid.backinterface import update_external_ids_of_authors, get_orcid_id_of_author, \
get_validated_request_tickets_for_author, get_title_of_paper, get_claimed_papers_of_author
from invenio.legacy.bibauthorid.dbinterface import defaultdict, remove_arxiv_papers_of_author
from invenio.webauthorprofile_orcidutils import get_dois_from_orcid
from invenio.legacy.bibauthorid.webauthorprofileinterface import is_valid_canonical_id, get_person_id_from_canonical_id, \
get_person_redirect_link, author_has_papers
from invenio.legacy.bibauthorid.templates import WebProfileMenu, WebProfilePage
# Imports related to hepnames update form
from invenio.bibedit_utils import get_bibrecord
from invenio.bibrecord import record_get_field_value, record_get_field_values, \
record_get_field_instances, field_get_subfield_values
TEMPLATE = load('bibauthorid')
class WebInterfaceBibAuthorIDClaimPages(WebInterfaceDirectory):
'''
Handles /author/claim pages and AJAX requests.
Supplies the methods:
/author/claim/<string>
/author/claim/action
/author/claim/claimstub
/author/claim/export
/author/claim/generate_autoclaim_data
/author/claim/merge_profiles_ajax
/author/claim/search_box_ajax
/author/claim/tickets_admin
/author/claim/search
'''
_exports = ['',
'action',
'claimstub',
'export',
'generate_autoclaim_data',
'merge_profiles_ajax',
'search_box_ajax',
'tickets_admin'
]
def _lookup(self, component, path):
'''
This handler parses dynamic URLs:
- /author/profile/1332 shows the page of author with id: 1332
- /author/profile/100:5522,1431 shows the page of the author
identified by the bibrefrec: '100:5522,1431'
'''
if not component in self._exports:
return WebInterfaceBibAuthorIDClaimPages(component), path
def _is_profile_owner(self, pid):
return self.person_id == int(pid)
def _is_admin(self, pinfo):
return pinfo['ulevel'] == 'admin'
def __init__(self, identifier=None):
'''
Constructor of the web interface.
@param identifier: identifier of an author. Can be one of:
- an author id: e.g. "14"
- a canonical id: e.g. "J.R.Ellis.1"
- a bibrefrec: e.g. "100:1442,155"
@type identifier: str
'''
self.person_id = -1 # -1 is a non valid author identifier
if identifier is None or not isinstance(identifier, str):
return
# check if it's a canonical id: e.g. "J.R.Ellis.1"
pid = int(webapi.get_person_id_from_canonical_id(identifier))
if pid >= 0:
self.person_id = pid
return
# check if it's an author id: e.g. "14"
try:
pid = int(identifier)
if webapi.author_has_papers(pid):
self.person_id = pid
return
except ValueError:
pass
# check if it's a bibrefrec: e.g. "100:1442,155"
if webapi.is_valid_bibref(identifier):
pid = int(webapi.get_person_id_from_paper(identifier))
if pid >= 0:
self.person_id = pid
return
def __call__(self, req, form):
'''
Serve the main person page.
Will use the object's person id to get a person's information.
@param req: apache request object
@type req: apache request object
@param form: POST/GET variables of the request
@type form: dict
@return: a full page formatted in HTML
@rtype: str
'''
webapi.session_bareinit(req)
pinfo = session['personinfo']
ulevel = pinfo['ulevel']
argd = wash_urlargd(form, {'ln': (str, CFG_SITE_LANG),
'open_claim': (str, None),
'ticketid': (int, -1),
'verbose': (int, 0)})
debug = "verbose" in argd and argd["verbose"] > 0
ln = argd['ln']
req.argd = argd # needed for perform_req_search
if self.person_id < 0:
return redirect_to_url(req, '%s/author/search' % (CFG_SITE_URL))
no_access = self._page_access_permission_wall(req, [self.person_id])
if no_access:
return no_access
pinfo['claim_in_process'] = True
- prefs = get_user_preferences(req)
- prefs['precached_viewclaimlink'] = pinfo['claim_in_process']
- set_user_preferences(pinfo['uid'], prefs)
+ user_info = collect_user_info(req)
+ user_info['precached_viewclaimlink'] = pinfo['claim_in_process']
+ session.dirty = True
if self.person_id != -1:
pinfo['claimpaper_admin_last_viewed_pid'] = self.person_id
rt_ticket_id = argd['ticketid']
if rt_ticket_id != -1:
pinfo["admin_requested_ticket_id"] = rt_ticket_id
session.dirty = True
## Create menu and page using templates
cname = webapi.get_canonical_id_from_person_id(self.person_id)
menu = WebProfileMenu(str(cname), "claim", ln, self._is_profile_owner(pinfo['pid']), self._is_admin(pinfo))
profile_page = WebProfilePage("claim", webapi.get_longest_name_from_pid(self.person_id))
profile_page.add_profile_menu(menu)
gboxstatus = self.person_id
gpid = self.person_id
gNumOfWorkers = 3 # to do: read it from conf file
gReqTimeout = 3000
gPageTimeout = 12000
profile_page.add_bootstrapped_data(json.dumps({
"other": "var gBOX_STATUS = '%s';var gPID = '%s'; var gNumOfWorkers= '%s'; var gReqTimeout= '%s'; var gPageTimeout= '%s';" % (gboxstatus, gpid, gNumOfWorkers, gReqTimeout, gPageTimeout),
"backbone": """
(function(ticketbox) {
var app = ticketbox.app;
app.userops.set(%s);
app.bodyModel.set({userLevel: "%s", guestPrompt: true});
})(ticketbox);""" % (WebInterfaceAuthorTicketHandling.bootstrap_status(pinfo, "user"), ulevel)
}))
if debug:
profile_page.add_debug_info(pinfo)
# content += self._generate_person_info_box(ulevel, ln) #### Name variants
# metaheaderadd = self._scripts() + '\n <meta name="robots" content="nofollow" />'
# body = self._generate_optional_menu(ulevel, req, form)
content = self._generate_tabs(ulevel, req)
content += self._generate_footer(ulevel)
content = content.decode('utf-8', 'strict')
webapi.history_log_visit(req, 'claim', pid=self.person_id)
return page(title=self._generate_title(ulevel),
metaheaderadd=profile_page.get_head().encode('utf-8'),
body=profile_page.get_wrapped_body(content).encode('utf-8'),
req=req,
language=ln,
show_title_p=False)
def _page_access_permission_wall(self, req, req_pid=None, req_level=None):
'''
Display an error page if user not authorized to use the interface.
@param req: Apache Request Object for session management
@type req: Apache Request Object
@param req_pid: Requested person id
@type req_pid: int
@param req_level: Request level required for the page
@type req_level: string
'''
uid = getUid(req)
pinfo = session["personinfo"]
uinfo = collect_user_info(req)
if 'ln' in pinfo:
ln = pinfo["ln"]
else:
ln = CFG_SITE_LANG
_ = gettext_set_language(ln)
is_authorized = True
pids_to_check = []
if not AID_ENABLED:
return page_not_authorized(req, text=_("Fatal: Author ID capabilities are disabled on this system."))
if req_level and 'ulevel' in pinfo and pinfo["ulevel"] != req_level:
return page_not_authorized(req, text=_("Fatal: You are not allowed to access this functionality."))
if req_pid and not isinstance(req_pid, list):
pids_to_check = [req_pid]
elif req_pid and isinstance(req_pid, list):
pids_to_check = req_pid
if (not (uinfo['precached_usepaperclaim']
or uinfo['precached_usepaperattribution'])
and 'ulevel' in pinfo
and not pinfo["ulevel"] == "admin"):
is_authorized = False
if is_authorized and not webapi.user_can_view_CMP(uid):
is_authorized = False
if is_authorized and 'ticket' in pinfo:
for tic in pinfo["ticket"]:
if 'pid' in tic:
pids_to_check.append(tic['pid'])
if pids_to_check and is_authorized:
user_pid = webapi.get_pid_from_uid(uid)
if not uinfo['precached_usepaperattribution']:
if (not user_pid in pids_to_check
and 'ulevel' in pinfo
and not pinfo["ulevel"] == "admin"):
is_authorized = False
elif (user_pid in pids_to_check
and 'ulevel' in pinfo
and not pinfo["ulevel"] == "admin"):
for tic in list(pinfo["ticket"]):
if not tic["pid"] == user_pid:
pinfo['ticket'].remove(tic)
if not is_authorized:
return page_not_authorized(req, text=_("Fatal: You are not allowed to access this functionality."))
else:
return ""
def _generate_title(self, ulevel):
'''
Generates the title for the specified user permission level.
@param ulevel: user permission level
@type ulevel: str
@return: title
@rtype: str
'''
def generate_title_guest():
title = 'Assign papers'
if self.person_id:
title = 'Assign papers for: ' + str(webapi.get_person_redirect_link(self.person_id))
return title
def generate_title_user():
title = 'Assign papers'
if self.person_id:
title = 'Assign papers (user interface) for: ' + str(webapi.get_person_redirect_link(self.person_id))
return title
def generate_title_admin():
title = 'Assign papers'
if self.person_id:
title = 'Assign papers (administrator interface) for: ' + str(webapi.get_person_redirect_link(self.person_id))
return title
generate_title = {'guest': generate_title_guest,
'user': generate_title_user,
'admin': generate_title_admin}
return generate_title[ulevel]()
def _generate_optional_menu(self, ulevel, req, form):
'''
Generates the menu for the specified user permission level.
@param ulevel: user permission level
@type ulevel: str
@param req: apache request object
@type req: apache request object
@param form: POST/GET variables of the request
@type form: dict
@return: menu
@rtype: str
'''
def generate_optional_menu_guest(req, form):
argd = wash_urlargd(form, {'ln': (str, CFG_SITE_LANG),
'verbose': (int, 0)})
menu = TEMPLATE.tmpl_person_menu(self.person_id, argd['ln'])
if "verbose" in argd and argd["verbose"] > 0:
pinfo = session['personinfo']
menu += "\n<pre>" + pformat(pinfo) + "</pre>\n"
return menu
def generate_optional_menu_user(req, form):
argd = wash_urlargd(form, {'ln': (str, CFG_SITE_LANG),
'verbose': (int, 0)})
menu = TEMPLATE.tmpl_person_menu(self.person_id, argd['ln'])
if "verbose" in argd and argd["verbose"] > 0:
pinfo = session['personinfo']
menu += "\n<pre>" + pformat(pinfo) + "</pre>\n"
return menu
def generate_optional_menu_admin(req, form):
argd = wash_urlargd(form, {'ln': (str, CFG_SITE_LANG),
'verbose': (int, 0)})
menu = TEMPLATE.tmpl_person_menu_admin(self.person_id, argd['ln'])
if "verbose" in argd and argd["verbose"] > 0:
pinfo = session['personinfo']
menu += "\n<pre>" + pformat(pinfo) + "</pre>\n"
return menu
generate_optional_menu = {'guest': generate_optional_menu_guest,
'user': generate_optional_menu_user,
'admin': generate_optional_menu_admin}
return "<div class=\"clearfix\">" + generate_optional_menu[ulevel](req, form) + "</div>"
def _generate_ticket_box(self, ulevel, req):
'''
Generates the semi-permanent info box for the specified user permission
level.
@param ulevel: user permission level
@type ulevel: str
@param req: apache request object
@type req: apache request object
@return: info box
@rtype: str
'''
def generate_ticket_box_guest(req):
pinfo = session['personinfo']
ticket = pinfo['ticket']
results = list()
pendingt = list()
for t in ticket:
if 'execution_result' in t:
for res in t['execution_result']:
results.append(res)
else:
pendingt.append(t)
box = ""
if pendingt:
box += TEMPLATE.tmpl_ticket_box('in_process', 'transaction', len(pendingt))
if results:
failed = [messages for status, messages in results if not status]
if failed:
box += TEMPLATE.tmpl_transaction_box('failure', failed)
successfull = [messages for status, messages in results if status]
if successfull:
box += TEMPLATE.tmpl_transaction_box('success', successfull)
return box
def generate_ticket_box_user(req):
return generate_ticket_box_guest(req)
def generate_ticket_box_admin(req):
return generate_ticket_box_guest(req)
generate_ticket_box = {'guest': generate_ticket_box_guest,
'user': generate_ticket_box_user,
'admin': generate_ticket_box_admin}
return generate_ticket_box[ulevel](req)
def _generate_person_info_box(self, ulevel, ln):
'''
Generates the name info box for the specified user permission level.
@param ulevel: user permission level
@type ulevel: str
@param ln: page display language
@type ln: str
@return: name info box
@rtype: str
'''
def generate_person_info_box_guest(ln):
names = webapi.get_person_names_from_id(self.person_id)
box = TEMPLATE.tmpl_admin_person_info_box(ln, person_id=self.person_id,
names=names)
return box
def generate_person_info_box_user(ln):
return generate_person_info_box_guest(ln)
def generate_person_info_box_admin(ln):
return generate_person_info_box_guest(ln)
generate_person_info_box = {'guest': generate_person_info_box_guest,
'user': generate_person_info_box_user,
'admin': generate_person_info_box_admin}
return generate_person_info_box[ulevel](ln)
def _generate_tabs(self, ulevel, req):
'''
Generates the tabs content for the specified user permission level.
@param ulevel: user permission level
@type ulevel: str
@param req: apache request object
@type req: apache request object
@return: tabs content
@rtype: str
'''
from invenio.legacy.bibauthorid.templates import verbiage_dict as tmpl_verbiage_dict
from invenio.legacy.bibauthorid.templates import buttons_verbiage_dict as tmpl_buttons_verbiage_dict
def generate_tabs_guest(req):
links = list() # ['delete', 'commit','del_entry','commit_entry']
tabs = ['records', 'repealed', 'review']
return generate_tabs_admin(req, show_tabs=tabs, ticket_links=links,
open_tickets=list(),
verbiage_dict=tmpl_verbiage_dict['guest'],
buttons_verbiage_dict=tmpl_buttons_verbiage_dict['guest'],
show_reset_button=False)
def generate_tabs_user(req):
links = ['delete', 'del_entry']
tabs = ['records', 'repealed', 'review', 'tickets']
pinfo = session['personinfo']
uid = getUid(req)
user_is_owner = 'not_owner'
if pinfo["claimpaper_admin_last_viewed_pid"] == webapi.get_pid_from_uid(uid):
user_is_owner = 'owner'
open_tickets = webapi.get_person_request_ticket(self.person_id)
tickets = list()
for t in open_tickets:
owns = False
for row in t[0]:
if row[0] == 'uid-ip' and row[1].split('||')[0] == str(uid):
owns = True
if owns:
tickets.append(t)
return generate_tabs_admin(req, show_tabs=tabs, ticket_links=links,
open_tickets=tickets,
verbiage_dict=tmpl_verbiage_dict['user'][user_is_owner],
buttons_verbiage_dict=tmpl_buttons_verbiage_dict['user'][user_is_owner])
def generate_tabs_admin(req, show_tabs=['records', 'repealed', 'review', 'comments', 'tickets', 'data'],
ticket_links=['delete', 'commit', 'del_entry', 'commit_entry'], open_tickets=None,
verbiage_dict=None, buttons_verbiage_dict=None, show_reset_button=True):
personinfo = dict()
try:
personinfo = session["personinfo"]
except KeyError:
return ""
if 'ln' in personinfo:
ln = personinfo["ln"]
else:
ln = CFG_SITE_LANG
all_papers = webapi.get_papers_by_person_id(self.person_id, ext_out=True)
records = [{'recid': paper[0],
'bibref': paper[1],
'flag': paper[2],
'authorname': paper[3],
'authoraffiliation': paper[4],
'paperdate': paper[5],
'rt_status': paper[6],
'paperexperiment': paper[7]} for paper in all_papers]
rejected_papers = [row for row in records if row['flag'] < -1]
rest_of_papers = [row for row in records if row['flag'] >= -1]
review_needed = webapi.get_review_needing_records(self.person_id)
if len(review_needed) < 1:
if 'review' in show_tabs:
show_tabs.remove('review')
if open_tickets == None:
open_tickets = webapi.get_person_request_ticket(self.person_id)
else:
if len(open_tickets) < 1 and 'tickets' in show_tabs:
show_tabs.remove('tickets')
rt_tickets = None
if "admin_requested_ticket_id" in personinfo:
rt_tickets = personinfo["admin_requested_ticket_id"]
if verbiage_dict is None:
verbiage_dict = translate_dict_values(tmpl_verbiage_dict['admin'], ln)
if buttons_verbiage_dict is None:
buttons_verbiage_dict = translate_dict_values(tmpl_buttons_verbiage_dict['admin'], ln)
# send data to the template function
tabs = TEMPLATE.tmpl_admin_tabs(ln, person_id=self.person_id,
rejected_papers=rejected_papers,
rest_of_papers=rest_of_papers,
review_needed=review_needed,
rt_tickets=rt_tickets,
open_rt_tickets=open_tickets,
show_tabs=show_tabs,
ticket_links=ticket_links,
verbiage_dict=verbiage_dict,
buttons_verbiage_dict=buttons_verbiage_dict,
show_reset_button=show_reset_button)
return tabs
def translate_dict_values(dictionary, ln):
def translate_str_values(dictionary, f=lambda x: x):
translated_dict = dict()
for key, value in dictionary.iteritems():
if isinstance(value, str):
translated_dict[key] = f(value)
elif isinstance(value, dict):
translated_dict[key] = translate_str_values(value, f)
else:
raise TypeError("Value should be either string or dictionary.")
return translated_dict
return translate_str_values(dictionary, f=gettext_set_language(ln))
generate_tabs = {'guest': generate_tabs_guest,
'user': generate_tabs_user,
'admin': generate_tabs_admin}
return generate_tabs[ulevel](req)
def _generate_footer(self, ulevel):
'''
Generates the footer for the specified user permission level.
@param ulevel: user permission level
@type ulevel: str
@return: footer
@rtype: str
'''
def generate_footer_guest():
return TEMPLATE.tmpl_invenio_search_box()
def generate_footer_user():
return generate_footer_guest()
def generate_footer_admin():
return generate_footer_guest()
generate_footer = {'guest': generate_footer_guest,
'user': generate_footer_user,
'admin': generate_footer_admin}
return generate_footer[ulevel]()
def _ticket_dispatch_end(self, req):
'''
The ticket dispatch is finished, redirect to the original page of
origin or to the last_viewed_pid or return to the papers autoassigned box to populate its data
'''
pinfo = session["personinfo"]
webapi.session_bareinit(req)
if 'claim_in_process' in pinfo:
pinfo['claim_in_process'] = False
if "merge_ticket" in pinfo and pinfo['merge_ticket']:
pinfo['merge_ticket'] = []
- prefs = get_user_preferences(req)
- prefs['precached_viewclaimlink'] = True
- uid = getUid(req)
- set_user_preferences(uid, prefs)
+ user_info = collect_user_info(req)
+ user_info['precached_viewclaimlink'] = True
+ session.dirty = True
if "referer" in pinfo and pinfo["referer"]:
referer = pinfo["referer"]
del(pinfo["referer"])
session.dirty = True
return redirect_to_url(req, referer)
# if we are coming fromt he autoclaim box we should not redirect and just return to the caller function
if 'autoclaim' in pinfo and pinfo['autoclaim']['review_failed'] == False and pinfo['autoclaim']['begin_autoclaim'] == True:
pinfo['autoclaim']['review_failed'] = False
pinfo['autoclaim']['begin_autoclaim'] = False
session.dirty = True
else:
redirect_page = webapi.history_get_last_visited_url(pinfo['visit_diary'], limit_to_page=['manage_profile', 'claim'])
if not redirect_page:
redirect_page = webapi.get_fallback_redirect_link(req)
if 'autoclaim' in pinfo and pinfo['autoclaim']['review_failed'] == True and pinfo['autoclaim']['checkout'] == True:
redirect_page = '%s/author/claim/action?checkout=True' % (CFG_SITE_URL,)
pinfo['autoclaim']['checkout'] = False
session.dirty = True
elif not 'manage_profile' in redirect_page:
pinfo['autoclaim']['review_failed'] = False
pinfo['autoclaim']['begin_autoclaim'] == False
pinfo['autoclaim']['checkout'] = True
session.dirty = True
redirect_page = '%s/author/claim/%s?open_claim=True' % (CFG_SITE_URL, webapi.get_person_redirect_link(pinfo["claimpaper_admin_last_viewed_pid"]))
else:
pinfo['autoclaim']['review_failed'] = False
pinfo['autoclaim']['begin_autoclaim'] == False
pinfo['autoclaim']['checkout'] = True
session.dirty = True
return redirect_to_url(req, redirect_page)
# redirect_link = diary('get_redirect_link', caller='_ticket_dispatch_end', parameters=[('open_claim','True')])
# return redirect_to_url(req, redirect_link)
# need review if should be deleted
def __user_is_authorized(self, req, action):
'''
Determines if a given user is authorized to perform a specified action
@param req: Apache Request Object
@type req: Apache Request Object
@param action: the action the user wants to perform
@type action: string
@return: True if user is allowed to perform the action, False if not
@rtype: boolean
'''
if not req:
return False
if not action:
return False
else:
action = escape(action)
uid = getUid(req)
if not isinstance(uid, int):
return False
if uid == 0:
return False
allowance = [i[1] for i in acc_find_user_role_actions({'uid': uid})
if i[1] == action]
if allowance:
return True
return False
@staticmethod
def _scripts(kill_browser_cache=False):
'''
Returns html code to be included in the meta header of the html page.
The actual code is stored in the template.
@return: html formatted Javascript and CSS inclusions for the <head>
@rtype: string
'''
return TEMPLATE.tmpl_meta_includes(kill_browser_cache)
def _check_user_fields(self, req, form):
argd = wash_urlargd(
form,
{'ln': (str, CFG_SITE_LANG),
'user_first_name': (str, None),
'user_last_name': (str, None),
'user_email': (str, None),
'user_comments': (str, None)})
pinfo = session["personinfo"]
ulevel = pinfo["ulevel"]
skip_checkout_faulty_fields = False
if ulevel in ['user', 'admin']:
skip_checkout_faulty_fields = True
if not ("user_first_name_sys" in pinfo and pinfo["user_first_name_sys"]):
if "user_first_name" in argd and argd['user_first_name']:
if not argd["user_first_name"] and not skip_checkout_faulty_fields:
pinfo["checkout_faulty_fields"].append("user_first_name")
else:
pinfo["user_first_name"] = escape(argd["user_first_name"])
if not ("user_last_name_sys" in pinfo and pinfo["user_last_name_sys"]):
if "user_last_name" in argd and argd['user_last_name']:
if not argd["user_last_name"] and not skip_checkout_faulty_fields:
pinfo["checkout_faulty_fields"].append("user_last_name")
else:
pinfo["user_last_name"] = escape(argd["user_last_name"])
if not ("user_email_sys" in pinfo and pinfo["user_email_sys"]):
if "user_email" in argd and argd['user_email']:
if not email_valid_p(argd["user_email"]):
pinfo["checkout_faulty_fields"].append("user_email")
else:
pinfo["user_email"] = escape(argd["user_email"])
if (ulevel == "guest"
and emailUnique(argd["user_email"]) > 0):
pinfo["checkout_faulty_fields"].append("user_email_taken")
else:
pinfo["checkout_faulty_fields"].append("user_email")
if "user_comments" in argd:
if argd["user_comments"]:
pinfo["user_ticket_comments"] = escape(argd["user_comments"])
else:
pinfo["user_ticket_comments"] = ""
session.dirty = True
def action(self, req, form):
'''
Initial step in processing of requests: ticket generation/update.
Also acts as action dispatcher for interface mass action requests.
Valid mass actions are:
- add_external_id: add an external identifier to an author
- add_missing_external_ids: add missing external identifiers of an author
- bibref_check_submit:
- cancel: clean the session (erase tickets and so on)
- cancel_rt_ticket:
- cancel_search_ticket:
- cancel_stage:
- checkout:
- checkout_continue_claiming:
- checkout_remove_transaction:
- checkout_submit:
- claim: claim papers for an author
- commit_rt_ticket:
- confirm: confirm assignments to an author
- delete_external_ids: delete external identifiers of an author
- repeal: repeal assignments from an author
- reset: reset assignments of an author
- set_canonical_name: set/swap the canonical name of an author
- to_other_person: assign a document from an author to another author
@param req: apache request object
@type req: apache request object
@param form: parameters sent via GET or POST request
@type form: dict
@return: a full page formatted in HTML
@return: str
'''
webapi.session_bareinit(req)
pinfo = session["personinfo"]
argd = wash_urlargd(form,
{'autoclaim_show_review':(str, None),
'canonical_name': (str, None),
'existing_ext_ids': (list, None),
'ext_id': (str, None),
'uid': (int, None),
'ext_system': (str, None),
'ln': (str, CFG_SITE_LANG),
'pid': (int, -1),
'primary_profile':(str, None),
'search_param': (str, None),
'rt_action': (str, None),
'rt_id': (int, None),
'selection': (list, None),
# permitted actions
'add_external_id': (str, None),
'set_uid': (str, None),
'add_missing_external_ids': (str, None),
'associate_profile': (str, None),
'bibref_check_submit': (str, None),
'cancel': (str, None),
'cancel_merging': (str, None),
'cancel_rt_ticket': (str, None),
'cancel_search_ticket': (str, None),
'cancel_stage': (str, None),
'checkout': (str, None),
'checkout_continue_claiming': (str, None),
'checkout_remove_transaction': (str, None),
'checkout_submit': (str, None),
'assign': (str, None),
'commit_rt_ticket': (str, None),
'confirm': (str, None),
'delete_external_ids': (str, None),
'merge': (str, None),
'reject': (str, None),
'repeal': (str, None),
'reset': (str, None),
'send_message': (str, None),
'set_canonical_name': (str, None),
'to_other_person': (str, None)})
ulevel = pinfo["ulevel"]
ticket = pinfo["ticket"]
uid = getUid(req)
ln = argd['ln']
action = None
permitted_actions = ['add_external_id',
'set_uid',
'add_missing_external_ids',
'associate_profile',
'bibref_check_submit',
'cancel',
'cancel_merging',
'cancel_rt_ticket',
'cancel_search_ticket',
'cancel_stage',
'checkout',
'checkout_continue_claiming',
'checkout_remove_transaction',
'checkout_submit',
'assign',
'commit_rt_ticket',
'confirm',
'delete_external_ids',
'merge',
'reject',
'repeal',
'reset',
'send_message',
'set_canonical_name',
'to_other_person']
for act in permitted_actions:
# one action (the most) is enabled in the form
if argd[act] is not None:
action = act
no_access = self._page_access_permission_wall(req, None)
if no_access and action not in ["assign"]:
return no_access
# incomplete papers (incomplete paper info or other problems) trigger action function without user's interference
# in order to fix those problems and claim papers or remove them from the ticket
if (action is None
and "bibref_check_required" in pinfo
and pinfo["bibref_check_required"]):
if "bibref_check_reviewed_bibrefs" in pinfo:
del(pinfo["bibref_check_reviewed_bibrefs"])
session.dirty = True
def add_external_id():
'''
associates the user with pid to the external id ext_id
'''
if argd['pid'] > -1:
pid = argd['pid']
else:
return self._error_page(req, ln,
"Fatal: cannot add external id to unknown person")
if argd['ext_system'] is not None:
ext_sys = argd['ext_system']
else:
return self._error_page(req, ln,
"Fatal: cannot add an external id without specifying the system")
if argd['ext_id'] is not None:
ext_id = argd['ext_id']
else:
return self._error_page(req, ln,
"Fatal: cannot add a custom external id without a suggestion")
userinfo = "%s||%s" % (uid, req.remote_ip)
webapi.add_person_external_id(pid, ext_sys, ext_id, userinfo)
return redirect_to_url(req, "%s/author/manage_profile/%s" % (CFG_SITE_URL, webapi.get_person_redirect_link(pid)))
def set_uid():
'''
associates the user with pid to the external id ext_id
'''
if argd['pid'] > -1:
pid = argd['pid']
else:
return self._error_page(req, ln,
"Fatal: current user is unknown")
if argd['uid'] is not None:
dest_uid = int(argd['uid'])
else:
return self._error_page(req, ln,
"Fatal: user id is not valid")
userinfo = "%s||%s" % (uid, req.remote_ip)
webapi.set_person_uid(pid, dest_uid, userinfo)
# remove arxiv pubs of current pid
remove_arxiv_papers_of_author(pid)
dest_uid_pid = webapi.get_pid_from_uid(dest_uid)
if dest_uid_pid > -1:
# move the arxiv pubs of the dest_uid to the current pid
dest_uid_arxiv_papers = webapi.get_arxiv_papers_of_author(dest_uid_pid)
webapi.add_arxiv_papers_to_author(dest_uid_arxiv_papers, pid)
return redirect_to_url(req, "%s/author/manage_profile/%s" % (CFG_SITE_URL, webapi.get_person_redirect_link(pid)))
def add_missing_external_ids():
if argd['pid'] > -1:
pid = argd['pid']
else:
return self._error_page(req, ln,
"Fatal: cannot recompute external ids for an unknown person")
update_external_ids_of_authors([pid], overwrite=False)
return redirect_to_url(req, "%s/author/manage_profile/%s" % (CFG_SITE_URL, webapi.get_person_redirect_link(pid)))
def associate_profile():
'''
associates the user with user id to the person profile with pid
'''
if argd['pid'] > -1:
pid = argd['pid']
else:
return self._error_page(req, ln,
"Fatal: cannot associate profile without a person id.")
uid = getUid(req)
pid, profile_claimed = webapi.claim_profile(uid, pid)
redirect_pid = pid
if profile_claimed:
pinfo['pid'] = pid
pinfo['should_check_to_autoclaim'] = True
pinfo["login_info_message"] = "confirm_success"
session.dirty = True
redirect_to_url(req, '%s/author/manage_profile/%s' % (CFG_SITE_URL, redirect_pid))
# if someone have already claimed this profile it redirects to choose_profile with an error message
else:
param=''
if 'search_param' in argd and argd['search_param']:
param = '&search_param=' + argd['search_param']
redirect_to_url(req, '%s/author/choose_profile?failed=%s%s' % (CFG_SITE_URL, True, param))
def bibref_check_submit():
pinfo["bibref_check_reviewed_bibrefs"] = list()
add_rev = pinfo["bibref_check_reviewed_bibrefs"].append
if ("bibrefs_auto_assigned" in pinfo
or "bibrefs_to_confirm" in pinfo):
person_reviews = list()
if ("bibrefs_auto_assigned" in pinfo
and pinfo["bibrefs_auto_assigned"]):
person_reviews.append(pinfo["bibrefs_auto_assigned"])
if ("bibrefs_to_confirm" in pinfo
and pinfo["bibrefs_to_confirm"]):
person_reviews.append(pinfo["bibrefs_to_confirm"])
for ref_review in person_reviews:
for person_id in ref_review:
for bibrec in ref_review[person_id]["bibrecs"]:
rec_grp = "bibrecgroup%s" % bibrec
elements = list()
if rec_grp in form:
if isinstance(form[rec_grp], str):
elements.append(form[rec_grp])
elif isinstance(form[rec_grp], list):
elements += form[rec_grp]
else:
continue
for element in elements:
test = element.split("||")
if test and len(test) > 1 and test[1]:
tref = test[1] + "," + str(bibrec)
tpid = webapi.wash_integer_id(test[0])
if (webapi.is_valid_bibref(tref)
and tpid > -1):
add_rev(element + "," + str(bibrec))
session.dirty = True
def cancel():
self.__session_cleanup(req)
return self._ticket_dispatch_end(req)
def cancel_merging():
'''
empties the session out of merge content and redirects to the manage profile page
that the user was viewing before the merge
'''
if argd['primary_profile']:
primary_cname = argd['primary_profile']
else:
return self._error_page(req, ln,
"Fatal: Couldn't redirect to the previous page")
webapi.session_bareinit(req)
pinfo = session['personinfo']
if pinfo['merge_profiles']:
pinfo['merge_profiles'] = list()
session.dirty = True
redirect_url = "%s/author/manage_profile/%s" % (CFG_SITE_URL, primary_cname)
return redirect_to_url(req, redirect_url)
def cancel_rt_ticket():
if argd['selection'] is not None:
bibrefrecs = argd['selection']
else:
return self._error_page(req, ln,
"Fatal: cannot cancel unknown ticket")
if argd['pid'] > -1:
pid = argd['pid']
else:
return self._error_page(req, ln, "Fatal: cannot cancel unknown ticket")
if argd['rt_id'] is not None and argd['rt_action'] is not None:
rt_id = int(argd['rt_id'])
rt_action = argd['rt_action']
for bibrefrec in bibrefrecs:
webapi.delete_transaction_from_request_ticket(pid, rt_id, rt_action, bibrefrec)
else:
rt_id = int(bibrefrecs[0])
webapi.delete_request_ticket(pid, rt_id)
return redirect_to_url(req, "%s/author/claim/%s" % (CFG_SITE_URL, pid))
def cancel_search_ticket(without_return=False):
if 'search_ticket' in pinfo:
del(pinfo['search_ticket'])
session.dirty = True
if "claimpaper_admin_last_viewed_pid" in pinfo:
pid = pinfo["claimpaper_admin_last_viewed_pid"]
if not without_return:
return redirect_to_url(req, "%s/author/claim/%s" % (CFG_SITE_URL, webapi.get_person_redirect_link(pid)))
if not without_return:
return self.search(req, form)
def cancel_stage():
if 'bibref_check_required' in pinfo:
del(pinfo['bibref_check_required'])
if 'bibrefs_auto_assigned' in pinfo:
del(pinfo['bibrefs_auto_assigned'])
if 'bibrefs_to_confirm' in pinfo:
del(pinfo['bibrefs_to_confirm'])
for tt in [row for row in ticket if 'incomplete' in row]:
ticket.remove(tt)
session.dirty = True
return self._ticket_dispatch_end(req)
def checkout():
pass
# return self._ticket_final_review(req)
def checkout_continue_claiming():
pinfo["checkout_faulty_fields"] = list()
self._check_user_fields(req, form)
return self._ticket_dispatch_end(req)
def checkout_remove_transaction():
bibref = argd['checkout_remove_transaction']
if webapi.is_valid_bibref(bibref):
for rmt in [row for row in ticket if row["bibref"] == bibref]:
ticket.remove(rmt)
pinfo["checkout_confirmed"] = False
session.dirty = True
# return self._ticket_final_review(req)
def checkout_submit():
pinfo["checkout_faulty_fields"] = list()
self._check_user_fields(req, form)
if not ticket:
pinfo["checkout_faulty_fields"].append("tickets")
pinfo["checkout_confirmed"] = True
if pinfo["checkout_faulty_fields"]:
pinfo["checkout_confirmed"] = False
session.dirty = True
# return self._ticket_final_review(req)
def claim():
if argd['selection'] is not None:
bibrefrecs = argd['selection']
else:
return self._error_page(req, ln,
"Fatal: cannot create ticket without any bibrefrec")
if argd['pid'] > -1:
pid = argd['pid']
else:
return self._error_page(req, ln,
"Fatal: cannot claim papers to an unknown person")
if action == 'assign':
claimed_recs = [paper[2] for paper in get_claimed_papers_of_author(pid)]
for bibrefrec in list(bibrefrecs):
_, rec = webapi.split_bibrefrec(bibrefrec)
if rec in claimed_recs:
bibrefrecs.remove(bibrefrec)
for bibrefrec in bibrefrecs:
operation_parts = {'pid': pid,
'action': action,
'bibrefrec': bibrefrec}
operation_to_be_added = webapi.construct_operation(operation_parts, pinfo, uid)
if operation_to_be_added is None:
continue
ticket = pinfo['ticket']
webapi.add_operation_to_ticket(operation_to_be_added, ticket)
session.dirty = True
return redirect_to_url(req, "%s/author/claim/%s" % (CFG_SITE_URL, webapi.get_person_redirect_link(pid)))
def claim_to_other_person():
if argd['selection'] is not None:
bibrefrecs = argd['selection']
else:
return self._error_page(req, ln,
"Fatal: cannot create ticket without any bibrefrec")
return self._ticket_open_assign_to_other_person(req, bibrefrecs, form)
def commit_rt_ticket():
if argd['selection'] is not None:
tid = argd['selection'][0]
else:
return self._error_page(req, ln,
"Fatal: cannot cancel unknown ticket")
if argd['pid'] > -1:
pid = argd['pid']
else:
return self._error_page(req, ln,
"Fatal: cannot cancel unknown ticket")
return self._commit_rt_ticket(req, tid, pid)
def confirm_repeal_reset():
if argd['pid'] > -1 or int(argd['pid']) == CREATE_NEW_PERSON:
pid = argd['pid']
cancel_search_ticket(without_return = True)
else:
return self._ticket_open_assign_to_other_person(req, argd['selection'], form)
#return self._error_page(req, ln, "Fatal: cannot create ticket without a person id! (crr %s)" %repr(argd))
bibrefrecs = argd['selection']
if argd['confirm']:
action = 'assign'
elif argd['repeal']:
action = 'reject'
elif argd['reset']:
action = 'reset'
else:
return self._error_page(req, ln, "Fatal: not existent action!")
for bibrefrec in bibrefrecs:
form['jsondata'] = json.dumps({'pid': str(pid),
'action': action,
'bibrefrec': bibrefrec,
'on': 'user'})
t = WebInterfaceAuthorTicketHandling()
t.add_operation(req, form)
return redirect_to_url(req, "%s/author/claim/%s" % (CFG_SITE_URL, webapi.get_person_redirect_link(pid)))
def delete_external_ids():
'''
deletes association between the user with pid and the external id ext_id
'''
if argd['pid'] > -1:
pid = argd['pid']
else:
return self._error_page(req, ln,
"Fatal: cannot delete external ids from an unknown person")
if argd['existing_ext_ids'] is not None:
existing_ext_ids = argd['existing_ext_ids']
else:
return self._error_page(req, ln,
"Fatal: you must select at least one external id in order to delete it")
userinfo = "%s||%s" % (uid, req.remote_ip)
webapi.delete_person_external_ids(pid, existing_ext_ids, userinfo)
return redirect_to_url(req, "%s/author/manage_profile/%s" % (CFG_SITE_URL, webapi.get_person_redirect_link(pid)))
def none_action():
return self._error_page(req, ln,
"Fatal: cannot create ticket if no action selected.")
def merge():
'''
performs a merge if allowed on the profiles that the user chose
'''
if argd['primary_profile']:
primary_cname = argd['primary_profile']
else:
return self._error_page(req, ln,
"Fatal: cannot perform a merge without a primary profile!")
if argd['selection']:
profiles_to_merge = argd['selection']
else:
return self._error_page(req, ln,
"Fatal: cannot perform a merge without any profiles selected!")
webapi.session_bareinit(req)
pinfo = session['personinfo']
uid = getUid(req)
primary_pid = webapi.get_person_id_from_canonical_id(primary_cname)
pids_to_merge = [webapi.get_person_id_from_canonical_id(cname) for cname in profiles_to_merge]
is_admin = False
if pinfo['ulevel'] == 'admin':
is_admin = True
# checking if there are restrictions regarding this merge
can_perform_merge, preventing_pid = webapi.merge_is_allowed(primary_pid, pids_to_merge, is_admin)
if not can_perform_merge:
# when redirected back to the merge profiles page display an error message about the currently attempted merge
pinfo['merge_info_message'] = ("failure", "confirm_failure")
session.dirty = True
redirect_url = "%s/author/merge_profiles?primary_profile=%s" % (CFG_SITE_URL, primary_cname)
return redirect_to_url(req, redirect_url)
if is_admin:
webapi.merge_profiles(primary_pid, pids_to_merge)
# when redirected back to the manage profile page display a message about the currently attempted merge
pinfo['merge_info_message'] = ("success", "confirm_success")
else:
name = ''
if 'user_last_name' in pinfo:
name = pinfo['user_last_name']
if 'user_first_name' in pinfo:
name += pinfo['user_first_name']
email = ''
if 'user_email' in pinfo:
email = pinfo['user_email']
selection_str = "&selection=".join(profiles_to_merge)
userinfo = {'uid-ip': "userid: %s (from %s)" % (uid, req.remote_ip),
'name': name,
'email': email,
'merge link': "%s/author/merge_profiles?primary_profile=%s&selection=%s" % (CFG_SITE_URL, primary_cname, selection_str)}
# a message is sent to the admin with info regarding the currently attempted merge
webapi.create_request_message(userinfo, subj='Merge profiles request')
# when redirected back to the manage profile page display a message about the merge
pinfo['merge_info_message'] = ("success", "confirm_operation")
pinfo['merge_profiles'] = list()
session.dirty = True
redirect_url = "%s/author/manage_profile/%s" % (CFG_SITE_URL, primary_cname)
return redirect_to_url(req, redirect_url)
def send_message():
'''
sends a message from the user to the admin
'''
webapi.session_bareinit(req)
pinfo = session['personinfo']
#pp = pprint.PrettyPrinter(indent=4)
#session_dump = pp.pprint(pinfo)
session_dump = str(pinfo)
name = ''
name_changed = False
name_given = ''
email = ''
email_changed = False
email_given = ''
comment = ''
last_page_visited = ''
if "user_last_name" in pinfo:
name = pinfo["user_last_name"]
if "user_first_name" in pinfo:
name += pinfo["user_first_name"]
name = name.rstrip()
if "user_email" in pinfo:
email = pinfo["user_email"]
email = email.rstrip()
if 'Name' in form:
if not name:
name = form['Name']
elif name != form['Name']:
name_given = form['Name']
name_changed = True
name = name.rstrip()
if 'E-mail'in form:
if not email:
email = form['E-mail']
elif name != form['E-mail']:
email_given = form['E-mail']
email_changed = True
email = email.rstrip()
if 'Comment' in form:
comment = form['Comment']
comment = comment.rstrip()
if not name or not comment or not email:
redirect_to_url(req, '%s/author/help?incomplete_params=%s' % (CFG_SITE_URL, True))
if 'last_page_visited' in form:
last_page_visited = form['last_page_visited']
uid = getUid(req)
userinfo = {'uid-ip': "userid: %s (from %s)" % (uid, req.remote_ip),
'name': name,
'email': email,
'comment': comment,
'last_page_visited': last_page_visited,
'session_dump': session_dump,
'name_given': name_given,
'email_given': email_given,
'name_changed': name_changed,
'email_changed': email_changed}
webapi.create_request_message(userinfo)
def set_canonical_name():
if argd['pid'] > -1:
pid = argd['pid']
else:
return self._error_page(req, ln,
"Fatal: cannot set canonical name to unknown person")
if argd['canonical_name'] is not None:
cname = argd['canonical_name']
else:
return self._error_page(req, ln,
"Fatal: cannot set a custom canonical name without a suggestion")
userinfo = "%s||%s" % (uid, req.remote_ip)
if webapi.is_valid_canonical_id(cname):
webapi.swap_person_canonical_name(pid, cname, userinfo)
else:
webapi.update_person_canonical_name(pid, cname, userinfo)
return redirect_to_url(req, "%s/author/claim/%s%s" % (CFG_SITE_URL, webapi.get_person_redirect_link(pid), '#tabData'))
action_functions = {'add_external_id': add_external_id,
'set_uid': set_uid,
'add_missing_external_ids': add_missing_external_ids,
'associate_profile': associate_profile,
'bibref_check_submit': bibref_check_submit,
'cancel': cancel,
'cancel_merging': cancel_merging,
'cancel_rt_ticket': cancel_rt_ticket,
'cancel_search_ticket': cancel_search_ticket,
'cancel_stage': cancel_stage,
'checkout': checkout,
'checkout_continue_claiming': checkout_continue_claiming,
'checkout_remove_transaction': checkout_remove_transaction,
'checkout_submit': checkout_submit,
'assign': claim,
'commit_rt_ticket': commit_rt_ticket,
'confirm': confirm_repeal_reset,
'delete_external_ids': delete_external_ids,
'merge': merge,
'reject': claim,
'repeal': confirm_repeal_reset,
'reset': confirm_repeal_reset,
'send_message': send_message,
'set_canonical_name': set_canonical_name,
'to_other_person': claim_to_other_person,
None: none_action}
return action_functions[action]()
def _ticket_open_claim(self, req, bibrefs, ln):
'''
Generate page to let user choose how to proceed
@param req: Apache Request Object
@type req: Apache Request Object
@param bibrefs: list of record IDs to perform an action on
@type bibrefs: list of int
@param ln: language to display the page in
@type ln: string
'''
uid = getUid(req)
uinfo = collect_user_info(req)
pinfo = session["personinfo"]
if 'ln' in pinfo:
ln = pinfo["ln"]
else:
ln = CFG_SITE_LANG
_ = gettext_set_language(ln)
no_access = self._page_access_permission_wall(req)
session.dirty = True
pid = -1
search_enabled = True
if not no_access and uinfo["precached_usepaperclaim"]:
tpid = webapi.get_pid_from_uid(uid)
if tpid > -1:
pid = tpid
last_viewed_pid = False
if (not no_access
and "claimpaper_admin_last_viewed_pid" in pinfo
and pinfo["claimpaper_admin_last_viewed_pid"]):
names = webapi.get_person_names_from_id(pinfo["claimpaper_admin_last_viewed_pid"])
names = sorted([i for i in names], key=lambda k: k[1], reverse=True)
if len(names) > 0:
if len(names[0]) > 0:
last_viewed_pid = [pinfo["claimpaper_admin_last_viewed_pid"], names[0][0]]
if no_access:
search_enabled = False
pinfo["referer"] = uinfo["referer"]
session.dirty = True
body = TEMPLATE.tmpl_open_claim(bibrefs, pid, last_viewed_pid,
search_enabled=search_enabled)
body = TEMPLATE.tmpl_person_detail_layout(body)
title = _('Claim this paper')
metaheaderadd = WebInterfaceBibAuthorIDClaimPages._scripts(kill_browser_cache=True)
return page(title=title,
metaheaderadd=metaheaderadd,
body=body,
req=req,
language=ln)
def _ticket_open_assign_to_other_person(self, req, bibrefs, form):
'''
Initializes search to find a person to attach the selected records to
@param req: Apache request object
@type req: Apache request object
@param bibrefs: list of record IDs to consider
@type bibrefs: list of int
@param form: GET/POST request parameters
@type form: dict
'''
pinfo = session["personinfo"]
pinfo["search_ticket"] = dict()
search_ticket = pinfo["search_ticket"]
search_ticket['action'] = 'assign'
search_ticket['bibrefs'] = bibrefs
session.dirty = True
return self.search(req, form)
def _cancel_rt_ticket(self, req, tid, pid):
'''
deletes an RT ticket
'''
webapi.delete_request_ticket(pid, tid)
return redirect_to_url(req, "%s/author/claim/%s" %
(CFG_SITE_URL, webapi.get_person_redirect_link(str(pid))))
def _cancel_transaction_from_rt_ticket(self, tid, pid, action, bibref):
'''
deletes a transaction from an rt ticket
'''
webapi.delete_transaction_from_request_ticket(pid, tid, action, bibref)
def _commit_rt_ticket(self, req, tid, pid):
'''
Commit of an rt ticket: creates a real ticket and commits.
'''
pinfo = session["personinfo"]
ticket = pinfo["ticket"]
uid = getUid(req)
tid = int(tid)
rt_ticket = get_validated_request_tickets_for_author(pid, tid)[0]
for action, bibrefrec in rt_ticket['operations']:
operation_parts = {'pid': pid,
'action': action,
'bibrefrec': bibrefrec}
operation_to_be_added = webapi.construct_operation(operation_parts, pinfo, uid)
webapi.add_operation_to_ticket(operation_to_be_added, ticket)
session.dirty = True
webapi.delete_request_ticket(pid, tid)
redirect_to_url(req, "%s/author/claim/%s" % (CFG_SITE_URL, pid))
def _error_page(self, req, ln=CFG_SITE_LANG, message=None, intro=True):
'''
Create a page that contains a message explaining the error.
@param req: Apache Request Object
@type req: Apache Request Object
@param ln: language
@type ln: string
@param message: message to be displayed
@type message: string
'''
body = []
_ = gettext_set_language(ln)
if not message:
message = "No further explanation available. Sorry."
if intro:
body.append(_("<p>We're sorry. An error occurred while "
"handling your request. Please find more information "
"below:</p>"))
body.append("<p><strong>%s</strong></p>" % message)
return page(title=_("Notice"),
body="\n".join(body),
description="%s - Internal Error" % CFG_SITE_NAME,
keywords="%s, Internal Error" % CFG_SITE_NAME,
language=ln,
req=req)
def __session_cleanup(self, req):
'''
Cleans the session from all bibauthorid specific settings and
with that cancels any transaction currently in progress.
@param req: Apache Request Object
@type req: Apache Request Object
'''
try:
pinfo = session["personinfo"]
except KeyError:
return
if "ticket" in pinfo:
pinfo['ticket'] = []
if "search_ticket" in pinfo:
pinfo['search_ticket'] = dict()
# clear up bibref checker if it's done.
if ("bibref_check_required" in pinfo
and not pinfo["bibref_check_required"]):
if 'bibrefs_to_confirm' in pinfo:
del(pinfo['bibrefs_to_confirm'])
if "bibrefs_auto_assigned" in pinfo:
del(pinfo["bibrefs_auto_assigned"])
del(pinfo["bibref_check_required"])
if "checkout_confirmed" in pinfo:
del(pinfo["checkout_confirmed"])
if "checkout_faulty_fields" in pinfo:
del(pinfo["checkout_faulty_fields"])
# pinfo['ulevel'] = ulevel
# pinfo["claimpaper_admin_last_viewed_pid"] = -1
pinfo["admin_requested_ticket_id"] = -1
session.dirty = True
def _generate_search_ticket_box(self, req):
'''
Generate the search ticket to remember a pending search for Person
entities in an attribution process
@param req: Apache request object
@type req: Apache request object
'''
pinfo = session["personinfo"]
search_ticket = None
if 'search_ticket' in pinfo:
search_ticket = pinfo['search_ticket']
if not search_ticket:
return ''
else:
return TEMPLATE.tmpl_search_ticket_box('person_search', 'assign_papers', search_ticket['bibrefs'])
def search_box(self, query, shown_element_functions):
'''
collecting the persons' data that the search function returned
@param req: Apache request object
@type req: Apache request object
@param query: the query string
@type query: string
@param shown_element_functions: contains the functions that will tell to the template which columns to show and what buttons to print
@type shown_element_functions: dict
@return: html body
@rtype: string
'''
pid_list = self._perform_search(query)
search_results = []
for pid in pid_list:
result = defaultdict(list)
result['pid'] = pid
result['canonical_id'] = webapi.get_canonical_id_from_person_id(pid)
result['name_variants'] = webapi.get_person_names_from_id(pid)
result['external_ids'] = webapi.get_external_ids_from_person_id(pid)
# this variable shows if we want to use the following data in the search template
if 'pass_status' in shown_element_functions and shown_element_functions['pass_status']:
result['status'] = webapi.is_profile_available(pid)
search_results.append(result)
body = TEMPLATE.tmpl_author_search(query, search_results, shown_element_functions)
body = TEMPLATE.tmpl_person_detail_layout(body)
return body
def search(self, req, form):
'''
Function used for searching a person based on a name with which the
function is queried.
@param req: Apache Request Object
@type form: dict
@return: a full page formatted in HTML
@rtype: string
'''
webapi.session_bareinit(req)
pinfo = session['personinfo']
ulevel = pinfo['ulevel']
person_id = self.person_id
uid = getUid(req)
argd = wash_urlargd(
form,
{'ln': (str, CFG_SITE_LANG),
'verbose': (int, 0),
'q': (str, None)})
debug = "verbose" in argd and argd["verbose"] > 0
ln = argd['ln']
cname = ''
is_owner = False
last_visited_pid = webapi.history_get_last_visited_pid(session['personinfo']['visit_diary'])
if last_visited_pid is not None:
cname = webapi.get_canonical_id_from_person_id(last_visited_pid)
try:
int(cname)
except ValueError:
is_owner = False
else:
is_owner = self._is_profile_owner(last_visited_pid)
menu = WebProfileMenu(str(cname), "search", ln, is_owner, self._is_admin(pinfo))
title = "Person search"
# Create Wrapper Page Markup
profile_page = WebProfilePage("search", title, no_cache=True)
profile_page.add_profile_menu(menu)
profile_page.add_bootstrapped_data(json.dumps({
"other": "var gBOX_STATUS = '10';var gPID = '10'; var gNumOfWorkers= '10'; var gReqTimeout= '10'; var gPageTimeout= '10';",
"backbone": """
(function(ticketbox) {
var app = ticketbox.app;
app.userops.set(%s);
app.bodyModel.set({userLevel: "%s"});
})(ticketbox);""" % (WebInterfaceAuthorTicketHandling.bootstrap_status(pinfo, "user"), ulevel)
}))
if debug:
profile_page.add_debug_info(pinfo)
no_access = self._page_access_permission_wall(req)
shown_element_functions = dict()
shown_element_functions['show_search_bar'] = TEMPLATE.tmpl_general_search_bar()
if no_access:
return no_access
search_ticket = None
bibrefs = []
if 'search_ticket' in pinfo:
search_ticket = pinfo['search_ticket']
for r in search_ticket['bibrefs']:
bibrefs.append(r)
if search_ticket and "ulevel" in pinfo:
if pinfo["ulevel"] == "admin":
shown_element_functions['new_person_gen'] = TEMPLATE.tmpl_assigning_search_new_person_generator(bibrefs)
content = ""
if search_ticket:
shown_element_functions['button_gen'] = TEMPLATE.tmpl_assigning_search_button_generator(bibrefs)
content = content + self._generate_search_ticket_box(req)
query = None
if 'q' in argd:
if argd['q']:
query = escape(argd['q'])
content += self.search_box(query, shown_element_functions)
body = profile_page.get_wrapped_body(content)
parameter = None
if query:
parameter = '?search_param=%s' + query
webapi.history_log_visit(req, 'search', params = parameter)
return page(title=title,
metaheaderadd=profile_page.get_head().encode('utf-8'),
body=body.encode('utf-8'),
req=req,
language=ln,
show_title_p=False)
def merge_profiles(self, req, form):
'''
begginig of the proccess that performs the merge over multipe person profiles
@param req: Apache Request Object
@type form: dict
@return: a full page formatted in HTML
@rtype: string
'''
argd = wash_urlargd(form, {'ln': (str, CFG_SITE_LANG),
'primary_profile': (str, None),
'search_param': (str, ''),
'selection': (list, None),
'verbose': (int, 0)})
ln = argd['ln']
primary_cname = argd['primary_profile']
search_param = argd['search_param']
selection = argd['selection']
debug = 'verbose' in argd and argd['verbose'] > 0
webapi.session_bareinit(req)
pinfo = session['personinfo']
profiles_to_merge = pinfo['merge_profiles']
_ = gettext_set_language(ln)
if not primary_cname:
return page_not_authorized(req, text=_('This page is not accessible directly.'))
no_access = self._page_access_permission_wall(req)
if no_access:
return no_access
if selection is not None:
profiles_to_merge_session = [cname for cname, is_available in profiles_to_merge]
for profile in selection:
if profile not in profiles_to_merge_session:
pid = webapi.get_person_id_from_canonical_id(profile)
is_available = webapi.is_profile_available(pid)
pinfo['merge_profiles'].append([profile, '1' if is_available else '0'])
session.dirty = True
primary_pid = webapi.get_person_id_from_canonical_id(primary_cname)
is_available = webapi.is_profile_available(primary_pid)
body = ''
cname = ''
is_owner = False
last_visited_pid = webapi.history_get_last_visited_pid(session['personinfo']['visit_diary'])
if last_visited_pid is not None:
cname = webapi.get_canonical_id_from_person_id(last_visited_pid)
is_owner = self._is_profile_owner(last_visited_pid)
title = 'Merge Profiles'
menu = WebProfileMenu(str(cname), "manage_profile", ln, is_owner, self._is_admin(pinfo))
merge_page = WebProfilePage("merge_profile", title, no_cache=True)
merge_page.add_profile_menu(menu)
if debug:
merge_page.add_debug_info(pinfo)
# display status for any previously attempted merge
if pinfo['merge_info_message']:
teaser_key, message = pinfo['merge_info_message']
body += TEMPLATE.tmpl_merge_transaction_box(teaser_key, [message])
pinfo['merge_info_message'] = None
session.dirty = True
body += TEMPLATE.tmpl_merge_ticket_box('person_search', 'merge_profiles', primary_cname)
shown_element_functions = dict()
shown_element_functions['show_search_bar'] = TEMPLATE.tmpl_merge_profiles_search_bar(primary_cname)
shown_element_functions['button_gen'] = TEMPLATE.merge_profiles_button_generator()
shown_element_functions['pass_status'] = 'True'
merge_page.add_bootstrapped_data(json.dumps({
"other": "var gMergeProfile = %s; var gMergeList = %s;" % ([primary_cname, '1' if is_available else '0'], profiles_to_merge)
}))
body += self.search_box(search_param, shown_element_functions)
body = merge_page.get_wrapped_body(body)
return page(title=title,
metaheaderadd=merge_page.get_head().encode('utf-8'),
body=body.encode('utf-8'),
req=req,
language=ln,
show_title_p=False)
def _perform_search(self, search_param):
'''
calls the search function on the search_param and returns the results
@param search_param: query string
@type search_param: String
@return: list of pids that the search found they match with the search query
@return: list
'''
pid_canditates_list = []
nquery = None
if search_param:
if search_param.count(":"):
try:
left, right = search_param.split(":")
try:
nsearch_param = str(right)
except (ValueError, TypeError):
try:
nsearch_param = str(left)
except (ValueError, TypeError):
nsearch_param = search_param
except ValueError:
nsearch_param = search_param
else:
nsearch_param = search_param
sorted_results = webapi.search_person_ids_by_name(nsearch_param)
for result in sorted_results:
pid_canditates_list.append(result[0])
return pid_canditates_list
def merge_profiles_ajax(self, req, form):
'''
Function used for handling Ajax requests used in order to add/remove profiles
in/from the merging profiles list, which is saved in the session.
@param req: Apache Request Object
@type req: Apache Request Object
@param form: Parameters sent via Ajax request
@type form: dict
@return: json data
'''
# Abort if the simplejson module isn't available
if not CFG_JSON_AVAILABLE:
print "Json not configurable"
# If it is an Ajax request, extract any JSON data.
ajax_request = False
# REcent papers request
if form.has_key('jsondata'):
json_data = json.loads(str(form['jsondata']))
# Deunicode all strings (Invenio doesn't have unicode
# support).
json_data = json_unicode_to_utf8(json_data)
ajax_request = True
json_response = {'resultCode': 0}
# Handle request.
if ajax_request:
req_type = json_data['requestType']
if req_type == 'addProfile':
if json_data.has_key('profile'):
profile = json_data['profile']
person_id = webapi.get_person_id_from_canonical_id(profile)
if person_id != -1:
webapi.session_bareinit(req)
profiles_to_merge = session["personinfo"]["merge_profiles"]
profile_availability = webapi.is_profile_available(person_id)
if profile_availability:
profile_availability = "1"
else:
profile_availability = "0"
if profile not in [el[0] for el in profiles_to_merge]:
profiles_to_merge.append([profile, profile_availability])
session.dirty = True
# TODO check access rights and get profile from db
json_response.update({'resultCode': 1})
json_response.update({'addedPofile': profile})
json_response.update({'addedPofileAvailability': profile_availability})
else:
json_response.update({'result': 'Error: Profile does not exist'})
else:
json_response.update({'result': 'Error: Profile was already in the list'})
else:
json_response.update({'result': 'Error: Missing profile'})
elif req_type == 'removeProfile':
if json_data.has_key('profile'):
profile = json_data['profile']
if webapi.get_person_id_from_canonical_id(profile) != -1:
webapi.session_bareinit(req)
profiles_to_merge = session["personinfo"]["merge_profiles"]
# print (str(profiles_to_merge))
if profile in [el[0] for el in profiles_to_merge]:
for prof in list(profiles_to_merge):
if prof[0] == profile:
profiles_to_merge.remove(prof)
session.dirty = True
# TODO check access rights and get profile from db
json_response.update({'resultCode': 1})
json_response.update({'removedProfile': profile})
else:
json_response.update({'result': 'Error: Profile was missing already from the list'})
else:
json_response.update({'result': 'Error: Profile does not exist'})
else:
json_response.update({'result': 'Error: Missing profile'})
elif req_type == 'setPrimaryProfile':
if json_data.has_key('profile'):
profile = json_data['profile']
profile_id = webapi.get_person_id_from_canonical_id(profile)
if profile_id != -1:
webapi.session_bareinit(req)
profile_availability = webapi.is_profile_available(profile_id)
if profile_availability:
profile_availability = "1"
else:
profile_availability = "0"
profiles_to_merge = session["personinfo"]["merge_profiles"]
if profile in [el[0] for el in profiles_to_merge if el and el[0]]:
for prof in list(profiles_to_merge):
if prof[0] == profile:
profiles_to_merge.remove(prof)
primary_profile = session["personinfo"]["merge_primary_profile"]
if primary_profile not in profiles_to_merge:
profiles_to_merge.append(primary_profile)
session["personinfo"]["merge_primary_profile"] = [profile, profile_availability]
session.dirty = True
json_response.update({'resultCode': 1})
json_response.update({'primaryProfile': profile})
json_response.update({'primaryPofileAvailability': profile_availability})
else:
json_response.update({'result': 'Error: Profile was already in the list'})
else:
json_response.update({'result': 'Error: Missing profile'})
else:
json_response.update({'result': 'Error: Wrong request type'})
return json.dumps(json_response)
def search_box_ajax(self, req, form):
'''
Function used for handling Ajax requests used in the search box.
@param req: Apache Request Object
@type req: Apache Request Object
@param form: Parameters sent via Ajax request
@type form: dict
@return: json data
'''
# Abort if the simplejson module isn't available
if not CFG_JSON_AVAILABLE:
print "Json not configurable"
# If it is an Ajax request, extract any JSON data.
ajax_request = False
# REcent papers request
if form.has_key('jsondata'):
json_data = json.loads(str(form['jsondata']))
# Deunicode all strings (Invenio doesn't have unicode
# support).
json_data = json_unicode_to_utf8(json_data)
ajax_request = True
json_response = {'resultCode': 0}
# Handle request.
if ajax_request:
req_type = json_data['requestType']
if req_type == 'getPapers':
if json_data.has_key('personId'):
pId = json_data['personId']
papers = sorted([[p[0]] for p in webapi.get_papers_by_person_id(int(pId), -1)],
key=itemgetter(0))
papers_html = TEMPLATE.tmpl_gen_papers(papers[0:MAX_NUM_SHOW_PAPERS])
json_response.update({'result': "\n".join(papers_html)})
json_response.update({'totalPapers': len(papers)})
json_response.update({'resultCode': 1})
json_response.update({'pid': str(pId)})
else:
json_response.update({'result': 'Error: Missing person id'})
elif req_type == 'getNames':
if json_data.has_key('personId'):
pId = json_data['personId']
names = webapi.get_person_names_from_id(int(pId))
names_html = TEMPLATE.tmpl_gen_names(names)
json_response.update({'result': "\n".join(names_html)})
json_response.update({'resultCode': 1})
json_response.update({'pid': str(pId)})
elif req_type == 'getIDs':
if json_data.has_key('personId'):
pId = json_data['personId']
ids = webapi.get_external_ids_from_person_id(int(pId))
ids_html = TEMPLATE.tmpl_gen_ext_ids(ids)
json_response.update({'result': "\n".join(ids_html)})
json_response.update({'resultCode': 1})
json_response.update({'pid': str(pId)})
elif req_type == 'isProfileClaimed':
if json_data.has_key('personId'):
pId = json_data['personId']
isClaimed = webapi.get_uid_from_personid(pId)
if isClaimed != -1:
json_response.update({'resultCode': 1})
json_response.update({'pid': str(pId)})
else:
json_response.update({'result': 'Error: Wrong request type'})
return json.dumps(json_response)
def choose_profile(self, req, form):
'''
Generate SSO landing/choose_profile page
@param req: Apache request object
@type req: Apache request object
@param form: GET/POST request params
@type form: dict
'''
argd = wash_urlargd(form, {'ln': (str, CFG_SITE_LANG),
'search_param': (str, None),
'failed': (str, None),
'verbose': (int, 0)})
ln = argd['ln']
debug = "verbose" in argd and argd["verbose"] > 0
req.argd = argd # needed for perform_req_search
search_param = argd['search_param']
webapi.session_bareinit(req)
uid = getUid(req)
pinfo = session['personinfo']
failed = True
if not argd['failed']:
failed = False
_ = gettext_set_language(ln)
if not CFG_INSPIRE_SITE:
return page_not_authorized(req, text=_("This page is not accessible directly."))
params = WebInterfaceBibAuthorIDClaimPages.get_params_to_check_login_info(session)
login_info = webapi.get_login_info(uid, params)
if 'arXiv' not in login_info['logged_in_to_remote_systems']:
return page_not_authorized(req, text=_("This page is not accessible directly."))
pid = webapi.get_user_pid(login_info['uid'])
# Create Wrapper Page Markup
is_owner = False
menu = WebProfileMenu('', "choose_profile", ln, is_owner, self._is_admin(pinfo))
choose_page = WebProfilePage("choose_profile", "Choose your profile", no_cache=True)
choose_page.add_profile_menu(menu)
if debug:
choose_page.add_debug_info(pinfo)
content = TEMPLATE.tmpl_choose_profile(failed)
body = choose_page.get_wrapped_body(content)
#In any case, when we step by here, an autoclaim should be performed right after!
pinfo = session["personinfo"]
pinfo['should_check_to_autoclaim'] = True
session.dirty = True
last_visited_pid = webapi.history_get_last_visited_pid(session['personinfo']['visit_diary'])
# if already logged in then redirect the user to the page he was viewing
if pid != -1:
redirect_pid = pid
if last_visited_pid:
redirect_pid = last_visited_pid
redirect_to_url(req, '%s/author/manage_profile/%s' % (CFG_SITE_URL, str(redirect_pid)))
else:
# get name strings and email addresses from SSO/Oauth logins: {'system':{'name':[variant1,...,variantn], 'email':'blabla@bla.bla', 'pants_size':20}}
remote_login_systems_info = webapi.get_remote_login_systems_info(req, login_info['logged_in_to_remote_systems'])
# get union of recids that are associated to the ids from all the external systems: set(inspire_recids_list)
recids = webapi.get_remote_login_systems_recids(req, login_info['logged_in_to_remote_systems'])
# this is the profile with the biggest intersection of papers so it's more probable that this is the profile the user seeks
probable_pid = webapi.match_profile(req, recids, remote_login_systems_info)
# if not search_param and probable_pid > -1 and probable_pid == last_visited_pid:
# # try to assign the user to the profile he chose. If for some reason the profile is not available we assign him to an empty profile
# redirect_pid, profile_claimed = webapi.claim_profile(login_info['uid'], probable_pid)
# if profile_claimed:
# redirect_to_url(req, '%s/author/claim/action?associate_profile=True&redirect_pid=%s' % (CFG_SITE_URL, str(redirect_pid)))
probable_profile_suggestion_info = None
last_viewed_profile_suggestion_info = None
if last_visited_pid > -1 and webapi.is_profile_available(last_visited_pid):
# get information about the most probable profile and show it to the user
last_viewed_profile_suggestion_info = webapi.get_profile_suggestion_info(req, last_visited_pid, recids)
if probable_pid > -1 and webapi.is_profile_available(probable_pid):
# get information about the most probable profile and show it to the user
probable_profile_suggestion_info = webapi.get_profile_suggestion_info(req, probable_pid, recids )
if not search_param:
# we prefil the search with most relevant among the names that we get from external systems
name_variants = webapi.get_name_variants_list_from_remote_systems_names(remote_login_systems_info)
search_param = most_relevant_name(name_variants)
body = body + TEMPLATE.tmpl_probable_profile_suggestion(probable_profile_suggestion_info, last_viewed_profile_suggestion_info, search_param)
shown_element_functions = dict()
shown_element_functions['button_gen'] = TEMPLATE.tmpl_choose_profile_search_button_generator()
shown_element_functions['new_person_gen'] = TEMPLATE.tmpl_choose_profile_search_new_person_generator()
shown_element_functions['show_search_bar'] = TEMPLATE.tmpl_choose_profile_search_bar()
# show in the templates the column status (if profile is bound to a user or not)
shown_element_functions['show_status'] = True
# pass in the templates the data of the column status (if profile is bound to a user or not)
# we might need the data without having to show them in the columne (fi merge_profiles
shown_element_functions['pass_status'] = True
# show search results to the user
body = body + self.search_box(search_param, shown_element_functions)
body = body + TEMPLATE.tmpl_choose_profile_footer()
title = _(' ')
return page(title=title,
metaheaderadd=choose_page.get_head().encode('utf-8'),
body=body,
req=req,
language=ln)
@staticmethod
def _arxiv_box(req, login_info, person_id, user_pid):
'''
Proccess and collect data for arXiv box
@param req: Apache request object
@type req: Apache request object
@param login_info: status of login in the following format: {'logged_in': True, 'uid': 2, 'logged_in_to_remote_systems':['Arxiv', ...]}
@type login_info: dict
@param login_info: person id of the current page's profile
@type login_info: int
@param login_info: person id of the user
@type login_info: int
@return: data required to built the arXiv box
@rtype: dict
'''
pinfo = session["personinfo"]
arxiv_data = dict()
arxiv_data['view_own_profile'] = person_id == user_pid
# if the user is not a guest and he is connected through arXiv
arxiv_data['login'] = login_info['logged_in']
arxiv_data['user_pid'] = user_pid
arxiv_data['user_has_pid'] = user_pid != -1
# if the profile the use is logged in is the same with the profile of the page that the user views
arxiv_data['view_own_profile'] = user_pid == person_id
return arxiv_data
@staticmethod
def _orcid_box(arxiv_logged_in, person_id, user_pid, ulevel):
'''
Proccess and collect data for orcid box
@param req: Apache request object
@type req: Apache request object
@param arxiv_logged_in: shows if the user is logged in through arXiv or not
@type arxiv_logged_in: boolean
@param person_id: person id of the current page's profile
@type person_id: int
@param user_pid: person id of the user
@type user_pid: int
@param ulevel: user's level
@type ulevel: string
@return: data required to built the orcid box
@rtype: dict
'''
orcid_data = dict()
orcid_data['arxiv_login'] = arxiv_logged_in
orcid_data['orcids'] = None
orcid_data['add_power'] = False
orcid_data['own_profile'] = False
orcid_data['pid'] = person_id
# if the profile the use is logged in is the same with the profile of the page that the user views
if person_id == user_pid:
orcid_data['own_profile'] = True
# if the user is an admin then he can add an existing orcid to the profile
if ulevel == "admin":
orcid_data['add_power'] = True
orcids = webapi.get_orcids_by_pid(person_id)
if orcids:
orcid_data['orcids'] = orcids
return orcid_data
@staticmethod
def _autoclaim_papers_box(req, person_id, user_pid, remote_logged_in_systems):
'''
Proccess and collect data for orcid box
@param req: Apache request object
@type req: Apache request object
@param person_id: person id of the current page's profile
@type person_id: int
@param user_pid: person id of the user
@type user_pid: int
@param remote_logged_in_systems: the remote logged in systems
@type remote_logged_in_systems: list
@return: data required to built the autoclaim box
@rtype: dict
'''
autoclaim_data = dict()
# if no autoclaim should occur or had occured and results should be shown then the box should remain hidden
autoclaim_data['hidden'] = True
autoclaim_data['person_id'] = person_id
# if the profile the use is logged in is the same with the profile of the page that the user views
if person_id == user_pid:
recids_to_autoclaim = webapi.get_remote_login_systems_recids(req, remote_logged_in_systems)
autoclaim_data['hidden'] = False
autoclaim_data['num_of_claims'] = len(recids_to_autoclaim)
return autoclaim_data
############################################
# New autoclaim functions #
############################################
def generate_autoclaim_data(self, req, form):
# Abort if the simplejson module isn't available
assert CFG_JSON_AVAILABLE, "Json not available"
# Fail if no json data exists in the Ajax request
if not form.has_key('jsondata'):
return self._fail(req, apache.HTTP_NOT_FOUND)
json_data = json.loads(str(form['jsondata']))
json_data = json_unicode_to_utf8(json_data)
try:
pid = int(json_data['personId'])
except:
raise NotImplementedError("Some error with the parameter from the Ajax request occured.")
webapi.session_bareinit(req)
pinfo = session['personinfo']
# If autoclaim was done already and no new remote systems exist
# in order to autoclaim new papers send the cached result
if not pinfo['orcid']['import_pubs'] and pinfo['autoclaim']['res'] is not None:
autoclaim_data = pinfo['autoclaim']['res']
json_response = {'resultCode': 1, 'result': TEMPLATE.tmpl_autoclaim_box(autoclaim_data, CFG_SITE_LANG, add_box=False, loading=False)}
return json.dumps(json_response)
external_pubs_association = pinfo['autoclaim']['external_pubs_association']
autoclaim_ticket = pinfo['autoclaim']['ticket']
ulevel = pinfo['ulevel']
uid = getUid(req)
params = WebInterfaceBibAuthorIDClaimPages.get_params_to_check_login_info(session)
login_status = webapi.get_login_info(uid, params)
remote_systems = login_status['logged_in_to_remote_systems']
papers_to_autoclaim = set(webapi.get_papers_from_remote_systems(remote_systems, params, external_pubs_association))
already_claimed_recids = set([rec for _, _, rec in get_claimed_papers_of_author(pid)]) & papers_to_autoclaim
papers_to_autoclaim = papers_to_autoclaim - set([rec for _, _, rec in get_claimed_papers_of_author(pid)])
for paper in papers_to_autoclaim:
operation_parts = {'pid': pid,
'action': 'assign',
'bibrefrec': str(paper)}
operation_to_be_added = webapi.construct_operation(operation_parts, pinfo, uid)
if operation_to_be_added is None:
# In case the operation could not be created (because of an
# erroneous bibrefrec) ignore it and continue with the rest
continue
webapi.add_operation_to_ticket(operation_to_be_added, autoclaim_ticket)
additional_info = {'first_name': '', 'last_name': '', 'email': '',
'comments': 'Assigned automatically when autoclaim was triggered.'}
userinfo = webapi.fill_out_userinfo(additional_info, uid, req.remote_ip, ulevel, strict_check=False)
webapi.commit_operations_from_ticket(autoclaim_ticket, userinfo, uid, ulevel)
autoclaim_data = dict()
autoclaim_data['hidden'] = False
autoclaim_data['person_id'] = pid
autoclaim_data['successfull_recids'] = set([op['rec'] for op in webapi.get_ticket_status(autoclaim_ticket) if 'execution_result' in op]) | already_claimed_recids
webapi.clean_ticket(autoclaim_ticket)
autoclaim_data['unsuccessfull_recids'] = [op['rec'] for op in webapi.get_ticket_status(autoclaim_ticket)]
autoclaim_data['num_of_unsuccessfull_recids'] = len(autoclaim_data['unsuccessfull_recids'])
autoclaim_data['recids_to_external_ids'] = dict()
for key, value in external_pubs_association.iteritems():
ext_system, ext_id = key
rec = value
title = get_title_of_paper(rec)
autoclaim_data['recids_to_external_ids'][rec] = title
# cache the result in the session
pinfo['autoclaim']['res'] = autoclaim_data
if pinfo['orcid']['import_pubs']:
pinfo['orcid']['import_pubs'] = False
session.dirty = True
json_response = {'resultCode': 1, 'result': TEMPLATE.tmpl_autoclaim_box(autoclaim_data, CFG_SITE_LANG, add_box=False, loading=False)}
req.write(json.dumps(json_response))
@staticmethod
def get_params_to_check_login_info(session):
def get_params_to_check_login_info_of_arxiv(session):
try:
return session['user_info']
except KeyError:
return None
def get_params_to_check_login_info_of_orcid(session):
pinfo = session['personinfo']
try:
pinfo['orcid']['has_orcid_id'] = bool(get_orcid_id_of_author(pinfo['pid'])[0][0] and pinfo['orcid']['import_pubs'])
except:
pinfo['orcid']['has_orcid_id'] = False
session.dirty = True
return pinfo['orcid']
get_params_for_remote_system = {'arXiv': get_params_to_check_login_info_of_arxiv,
'orcid': get_params_to_check_login_info_of_orcid}
params = dict()
for system, get_params in get_params_for_remote_system.iteritems():
params[system] = get_params(session)
return params
@staticmethod
def _claim_paper_box(person_id):
'''
Proccess and collect data for claim paper box
@param person_id: person id of the current page's profile
@type person_id: int
@return: data required to built the claim paper box
@rtype: dict
'''
claim_paper_data = dict()
claim_paper_data['canonical_id'] = str(webapi.get_canonical_id_from_person_id(person_id))
return claim_paper_data
@staticmethod
def _support_box():
'''
Proccess and collect data for support box
@return: data required to built the support box
@rtype: dict
'''
support_data = dict()
return support_data
@staticmethod
def _merge_box(person_id):
'''
Proccess and collect data for merge box
@param person_id: person id of the current page's profile
@type person_id: int
@return: data required to built the merge box
@rtype: dict
'''
merge_data = dict()
search_param = webapi.get_canonical_id_from_person_id(person_id)
name_variants = [element[0] for element in webapi.get_person_names_from_id(person_id)]
relevant_name = most_relevant_name(name_variants)
if relevant_name:
search_param = relevant_name.split(",")[0]
merge_data['search_param'] = search_param
merge_data['canonical_id'] = webapi.get_canonical_id_from_person_id(person_id)
return merge_data
@staticmethod
def _internal_ids_box(person_id, user_pid, ulevel):
'''
Proccess and collect data for external_ids box
@param person_id: person id of the current page's profile
@type person_id: int
@param user_pid: person id of the user
@type user_pid: int
@param remote_logged_in_systems: the remote logged in systems
@type remote_logged_in_systems: list
@return: data required to built the external_ids box
@rtype: dict
'''
external_ids_data = dict()
external_ids_data['uid'],external_ids_data['old_uids'] = webapi.get_internal_user_id_from_person_id(person_id)
external_ids_data['person_id'] = person_id
external_ids_data['user_pid'] = user_pid
external_ids_data['ulevel'] = ulevel
return external_ids_data
@staticmethod
def _external_ids_box(person_id, user_pid, ulevel):
'''
Proccess and collect data for external_ids box
@param person_id: person id of the current page's profile
@type person_id: int
@param user_pid: person id of the user
@type user_pid: int
@param remote_logged_in_systems: the remote logged in systems
@type remote_logged_in_systems: list
@return: data required to built the external_ids box
@rtype: dict
'''
internal_ids_data = dict()
internal_ids_data['ext_ids'] = webapi.get_external_ids_from_person_id(person_id)
internal_ids_data['person_id'] = person_id
internal_ids_data['user_pid'] = user_pid
internal_ids_data['ulevel'] = ulevel
return internal_ids_data
@staticmethod
def _hepnames_box(person_id):
return webapi.get_hepnames(person_id)
def tickets_admin(self, req, form):
'''
Generate SSO landing/welcome page
@param req: Apache request object
@type req: Apache request object
@param form: GET/POST request params
@type form: dict
'''
argd = wash_urlargd(form, {'ln': (str, CFG_SITE_LANG)})
ln = argd['ln']
webapi.session_bareinit(req)
no_access = self._page_access_permission_wall(req, req_level='admin')
if no_access:
return no_access
pinfo = session['personinfo']
cname = ''
is_owner = False
last_visited_pid = webapi.history_get_last_visited_pid(pinfo['visit_diary'])
if last_visited_pid is not None:
cname = webapi.get_canonical_id_from_person_id(last_visited_pid)
is_owner = self._is_profile_owner(last_visited_pid)
menu = WebProfileMenu(str(cname), "open_tickets", ln, is_owner, self._is_admin(pinfo))
title = "Open RT tickets"
profile_page = WebProfilePage("help", title, no_cache=True)
profile_page.add_profile_menu(menu)
tickets = webapi.get_persons_with_open_tickets_list()
tickets = list(tickets)
for t in list(tickets):
tickets.remove(t)
tickets.append([webapi.get_most_frequent_name_from_pid(int(t[0])),
webapi.get_person_redirect_link(t[0]), t[0], t[1]])
content = TEMPLATE.tmpl_tickets_admin(tickets)
content = TEMPLATE.tmpl_person_detail_layout(content)
body = profile_page.get_wrapped_body(content)
return page(title=title,
metaheaderadd=profile_page.get_head().encode('utf-8'),
body=body.encode('utf-8'),
req=req,
language=ln,
show_title_p=False)
def help(self, req, form):
argd = wash_urlargd(form, {'ln': (str, CFG_SITE_LANG)})
ln = argd['ln']
_ = gettext_set_language(ln)
if not CFG_INSPIRE_SITE:
return page_not_authorized(req, text=_("This page is not accessible directly."))
webapi.session_bareinit(req)
pinfo = session['personinfo']
cname = ''
is_owner = False
last_visited_pid = webapi.history_get_last_visited_pid(pinfo['visit_diary'])
if last_visited_pid is not None:
cname = webapi.get_canonical_id_from_person_id(last_visited_pid)
is_owner = self._is_profile_owner(last_visited_pid)
menu = WebProfileMenu(str(cname), "help", ln, is_owner, self._is_admin(pinfo))
title = "Help page"
profile_page = WebProfilePage("help", title, no_cache=True)
profile_page.add_profile_menu(menu)
content = TEMPLATE.tmpl_help_page()
body = profile_page.get_wrapped_body(content)
return page(title=title,
metaheaderadd=profile_page.get_head().encode('utf-8'),
body=body.encode('utf-8'),
req=req,
language=ln,
show_title_p=False)
def export(self, req, form):
'''
Generate JSONized export of Person data
@param req: Apache request object
@type req: Apache request object
@param form: GET/POST request params
@type form: dict
'''
argd = wash_urlargd(
form,
{'ln': (str, CFG_SITE_LANG),
'request': (str, None),
'userid': (str, None)})
if not CFG_JSON_AVAILABLE:
return "500_json_not_found__install_package"
request = None
userid = None
if "userid" in argd and argd['userid']:
userid = argd['userid']
else:
return "404_user_not_found"
if "request" in argd and argd['request']:
request = argd["request"]
# find user from ID
user_email = get_email_from_username(userid)
if user_email == userid:
return "404_user_not_found"
uid = get_uid_from_email(user_email)
uinfo = collect_user_info(uid)
# find person by uid
pid = webapi.get_pid_from_uid(uid)
# find papers py pid that are confirmed through a human.
papers = webapi.get_papers_by_person_id(pid, 2)
# filter by request param, e.g. arxiv
if not request:
return "404__no_filter_selected"
if not request in VALID_EXPORT_FILTERS:
return "500_filter_invalid"
if request == "arxiv":
query = "(recid:"
query += " OR recid:".join(papers)
query += ") AND 037:arxiv"
db_docs = perform_request_search(p=query, rg=0)
nickmail = ""
nickname = ""
db_arxiv_ids = []
try:
nickname = uinfo["nickname"]
except KeyError:
pass
if not nickname:
try:
nickmail = uinfo["email"]
except KeyError:
nickmail = user_email
nickname = nickmail
db_arxiv_ids = get_fieldvalues(db_docs, "037__a")
construct = {"nickname": nickname,
"claims": ";".join(db_arxiv_ids)}
jsondmp = json.dumps(construct)
signature = webapi.sign_assertion("arXiv", jsondmp)
construct["digest"] = signature
return json.dumps(construct)
index = __call__
class WebInterfaceBibAuthorIDManageProfilePages(WebInterfaceDirectory):
_exports = ['',
'import_orcid_pubs',
'connect_author_with_hepname',
'connect_author_with_hepname_ajax',
'suggest_orcid',
'suggest_orcid_ajax']
def _lookup(self, component, path):
'''
This handler parses dynamic URLs:
- /author/profile/1332 shows the page of author with id: 1332
- /author/profile/100:5522,1431 shows the page of the author
identified by the bibrefrec: '100:5522,1431'
'''
if not component in self._exports:
return WebInterfaceBibAuthorIDManageProfilePages(component), path
def _is_profile_owner(self, pid):
return self.person_id == int(pid)
def _is_admin(self, pinfo):
return pinfo['ulevel'] == 'admin'
def __init__(self, identifier=None):
'''
Constructor of the web interface.
@param identifier: identifier of an author. Can be one of:
- an author id: e.g. "14"
- a canonical id: e.g. "J.R.Ellis.1"
- a bibrefrec: e.g. "100:1442,155"
@type identifier: str
'''
self.person_id = -1 # -1 is a non valid author identifier
if identifier is None or not isinstance(identifier, str):
return
self.original_identifier = identifier
# check if it's a canonical id: e.g. "J.R.Ellis.1"
try:
pid = int(identifier)
except ValueError:
pid = int(webapi.get_person_id_from_canonical_id(identifier))
if pid >= 0:
self.person_id = pid
return
# check if it's an author id: e.g. "14"
try:
pid = int(identifier)
if webapi.author_has_papers(pid):
self.person_id = pid
return
except ValueError:
pass
# check if it's a bibrefrec: e.g. "100:1442,155"
if webapi.is_valid_bibref(identifier):
pid = int(webapi.get_person_id_from_paper(identifier))
if pid >= 0:
self.person_id = pid
return
def __call__(self, req, form):
'''
Generate SSO landing/author management page
@param req: Apache request object
@type req: Apache request object
@param form: GET/POST request params
@type form: dict
'''
webapi.session_bareinit(req)
pinfo = session['personinfo']
ulevel = pinfo['ulevel']
person_id = self.person_id
uid = getUid(req)
pinfo['claim_in_process'] = True
argd = wash_urlargd(form, {
'ln': (str, CFG_SITE_LANG),
'verbose': (int, 0)})
debug = "verbose" in argd and argd["verbose"] > 0
ln = argd['ln']
_ = gettext_set_language(ln)
if not CFG_INSPIRE_SITE or self.person_id is None:
return page_not_authorized(req, text=_("This page is not accessible directly."))
if person_id < 0:
return self._error_page(req, message=("Identifier %s is not a valid person identifier or does not exist anymore!" % self.original_identifier))
# log the visit
webapi.history_log_visit(req, 'manage_profile', pid=person_id)
# store the arxiv papers the user owns
if uid > 0 and not pinfo['arxiv_status']:
uinfo = collect_user_info(req)
arxiv_papers = list()
if 'external_arxivids' in uinfo and uinfo['external_arxivids']:
arxiv_papers = uinfo['external_arxivids'].split(';')
if arxiv_papers:
webapi.add_arxiv_papers_to_author(arxiv_papers, person_id)
pinfo['arxiv_status'] = True
params = WebInterfaceBibAuthorIDClaimPages.get_params_to_check_login_info(session)
login_info = webapi.get_login_info(uid, params)
title_message = _('Profile management')
ssl_param = 0
if req.is_https():
ssl_param = 1
# Create Wrapper Page Markup
cname = webapi.get_canonical_id_from_person_id(self.person_id)
if cname == self.person_id:
return page_not_authorized(req, text=_("This page is not accessible directly."))
menu = WebProfileMenu(cname, "manage_profile", ln, self._is_profile_owner(pinfo['pid']), self._is_admin(pinfo))
profile_page = WebProfilePage("manage_profile", webapi.get_longest_name_from_pid(self.person_id), no_cache=True)
profile_page.add_profile_menu(menu)
gboxstatus = self.person_id
gpid = self.person_id
gNumOfWorkers = 3 # to do: read it from conf file
gReqTimeout = 3000
gPageTimeout = 12000
profile_page.add_bootstrapped_data(json.dumps({
"other": "var gBOX_STATUS = '%s';var gPID = '%s'; var gNumOfWorkers= '%s'; var gReqTimeout= '%s'; var gPageTimeout= '%s';" % (gboxstatus, gpid, gNumOfWorkers, gReqTimeout, gPageTimeout),
"backbone": """
(function(ticketbox) {
var app = ticketbox.app;
app.userops.set(%s);
app.bodyModel.set({userLevel: "%s"});
})(ticketbox);""" % (WebInterfaceAuthorTicketHandling.bootstrap_status(pinfo, "user"), ulevel)
}))
if debug:
profile_page.add_debug_info(pinfo)
user_pid = webapi.get_user_pid(login_info['uid'])
person_data = webapi.get_person_info_by_pid(person_id)
# proccess and collect data for every box [LEGACY]
arxiv_data = WebInterfaceBibAuthorIDClaimPages._arxiv_box(req, login_info, person_id, user_pid)
orcid_data = WebInterfaceBibAuthorIDClaimPages._orcid_box(arxiv_data['login'], person_id, user_pid, ulevel)
claim_paper_data = WebInterfaceBibAuthorIDClaimPages._claim_paper_box(person_id)
support_data = WebInterfaceBibAuthorIDClaimPages._support_box()
ext_ids_data = None
int_ids_data = None
if ulevel == 'admin':
ext_ids_data = WebInterfaceBibAuthorIDClaimPages._external_ids_box(person_id, user_pid, ulevel)
int_ids_data = WebInterfaceBibAuthorIDClaimPages._internal_ids_box(person_id, user_pid, ulevel)
autoclaim_data = WebInterfaceBibAuthorIDClaimPages._autoclaim_papers_box(req, person_id, user_pid, login_info['logged_in_to_remote_systems'])
merge_data = WebInterfaceBibAuthorIDClaimPages._merge_box(person_id)
hepnames_data = WebInterfaceBibAuthorIDClaimPages._hepnames_box(person_id)
content = ''
# display status for any previously attempted merge
if pinfo['merge_info_message']:
teaser_key, message = pinfo['merge_info_message']
content += TEMPLATE.tmpl_merge_transaction_box(teaser_key, [message])
pinfo['merge_info_message'] = None
session.dirty = True
content += TEMPLATE.tmpl_profile_management(ln, person_data, arxiv_data,
orcid_data, claim_paper_data,
int_ids_data, ext_ids_data,
autoclaim_data, support_data,
merge_data, hepnames_data)
body = profile_page.get_wrapped_body(content)
return page(title=title_message,
metaheaderadd=profile_page.get_head().encode('utf-8'),
body=body.encode('utf-8'),
req=req,
language=ln,
show_title_p=False)
def import_orcid_pubs(self, req, form):
webapi.session_bareinit(req)
pinfo = session['personinfo']
orcid_info = pinfo['orcid']
# author should have already an orcid if this method was triggered
orcid_id = get_orcid_id_of_author(pinfo['pid'])[0][0]
orcid_dois = get_dois_from_orcid(orcid_id)
# TODO: what to do in case some ORCID server error occurs?
if orcid_dois is None:
redirect_to_url(req, "%s/author/manage_profile/%s" % (CFG_SITE_SECURE_URL, pinfo['pid']))
# TODO: it would be smarter if:
# 1. we save in the db the orcid_dois
# 2. to expire only the external pubs box in the profile page
webauthorapi.expire_all_cache_for_personid(pinfo['pid'])
orcid_info['imported_pubs'] = orcid_dois
orcid_info['import_pubs'] = True
session.dirty = True
redirect_to_url(req, "%s/author/manage_profile/%s" % (CFG_SITE_SECURE_URL, pinfo['pid']))
def connect_author_with_hepname(self, req, form):
argd = wash_urlargd(form, {'cname':(str, None),
'hepname': (str, None),
'ln': (str, CFG_SITE_LANG)})
ln = argd['ln']
if argd['cname'] is not None:
cname = argd['cname']
else:
return self._error_page(req, ln, "Fatal: cannot associate a hepname without a person id.")
if argd['hepname'] is not None:
hepname = argd['hepname']
else:
return self._error_page(req, ln, "Fatal: cannot associate an author with a non valid hepname.")
webapi.connect_author_with_hepname(cname, hepname)
webapi.session_bareinit(req)
pinfo = session['personinfo']
last_visited_page = webapi.history_get_last_visited_url(pinfo['visit_diary'], just_page=True)
redirect_to_url(req, "%s/author/%s/%s" % (CFG_SITE_URL, last_visited_page, cname))
def connect_author_with_hepname_ajax(self, req, form):
'''
Function used for handling Ajax requests.
@param req: apache request object
@type req: apache request object
@param form: parameters sent via Ajax request
@type form: dict
@return:
@rtype: json data
'''
# Abort if the simplejson module isn't available
assert CFG_JSON_AVAILABLE, "Json not available"
# Fail if no json data exists in the Ajax request
if not form.has_key('jsondata'):
return self._fail(req, apache.HTTP_NOT_FOUND)
json_data = json.loads(str(form['jsondata']))
json_data = json_unicode_to_utf8(json_data)
try:
cname = json_data['cname']
hepname = json_data['hepname']
except:
return self._fail(req, apache.HTTP_NOT_FOUND)
pinfo = session['personinfo']
if not self._is_admin(pinfo):
webapi.connect_author_with_hepname(cname, hepname)
else:
uid = getUid(req)
add_cname_to_hepname_record(cname, hepname, uid)
def suggest_orcid(self, req, form):
argd = wash_urlargd(form, {'orcid':(str, None),
'pid': (int, -1),
'ln': (str, CFG_SITE_LANG)})
ln = argd['ln']
if argd['pid'] > -1:
pid = argd['pid']
else:
return self._error_page(req, ln, "Fatal: cannot associate an orcid without a person id.")
if argd['orcid'] is not None and is_valid_orcid(argd['orcid']):
orcid = argd['orcid']
else:
return self._error_page(req, ln, "Fatal: cannot associate an author with a non valid ORCiD.")
webapi.connect_author_with_orcid(webapi.get_canonical_id_from_person_id(pid), orcid)
redirect_to_url(req, "%s/author/manage_profile/%s" % (CFG_SITE_URL, pid))
def suggest_orcid_ajax(self, req, form):
'''
Function used for handling Ajax requests.
@param req: apache request object
@type req: apache request object
@param form: parameters sent via Ajax request
@type form: dict
@return:
@rtype: json data
'''
# Abort if the simplejson module isn't available
assert CFG_JSON_AVAILABLE, "Json not available"
# Fail if no json data exists in the Ajax request
if not form.has_key('jsondata'):
return self._fail(req, apache.HTTP_NOT_FOUND)
json_data = json.loads(str(form['jsondata']))
json_data = json_unicode_to_utf8(json_data)
try:
orcid = json_data['orcid']
pid = json_data['pid']
except:
return self._fail(req, apache.HTTP_NOT_FOUND)
if not is_valid_orcid(orcid):
return self._fail(req, apache.HTTP_NOT_FOUND)
webapi.connect_author_with_orcid(webapi.get_canonical_id_from_person_id(pid), orcid)
def _fail(self, req, code):
req.status = code
return
def _error_page(self, req, ln=CFG_SITE_LANG, message=None, intro=True):
'''
Create a page that contains a message explaining the error.
@param req: Apache Request Object
@type req: Apache Request Object
@param ln: language
@type ln: string
@param message: message to be displayed
@type message: string
'''
body = []
_ = gettext_set_language(ln)
if not message:
message = "No further explanation available. Sorry."
if intro:
body.append(_("<p>We're sorry. An error occurred while "
"handling your request. Please find more information "
"below:</p>"))
body.append("<p><strong>%s</strong></p>" % message)
return page(title=_("Notice"),
body="\n".join(body),
description="%s - Internal Error" % CFG_SITE_NAME,
keywords="%s, Internal Error" % CFG_SITE_NAME,
language=ln,
req=req)
index = __call__
class WebInterfaceAuthorTicketHandling(WebInterfaceDirectory):
_exports = ['get_status',
'update_status',
'add_operation',
'modify_operation',
'remove_operation',
'commit',
'abort']
@staticmethod
def bootstrap_status(pinfo, on_ticket):
'''
Function used for generating get_status json bootstrapping.
@param pinfo: person_info
@type req: dict
@param on_ticket: ticket target
@type on_ticket: str
@return:
@rtype: json data
'''
# Abort if the simplejson module isn't available
assert CFG_JSON_AVAILABLE, "Json not available"
author_ticketing = WebInterfaceAuthorTicketHandling()
ticket = author_ticketing._get_according_ticket(on_ticket, pinfo)
if ticket is None:
return "{}"
ticket_status = webapi.get_ticket_status(ticket)
return json.dumps(ticket_status)
def get_status(self, req, form):
'''
Function used for handling Ajax requests.
@param req: apache request object
@type req: apache request object
@param form: parameters sent via Ajax request
@type form: dict
@return:
@rtype: json data
'''
# Abort if the simplejson module isn't available
assert CFG_JSON_AVAILABLE, "Json not available"
# Fail if no json data exists in the Ajax request
if not form.has_key('jsondata'):
return self._fail(req, apache.HTTP_NOT_FOUND)
json_data = json.loads(str(form['jsondata']))
json_data = json_unicode_to_utf8(json_data)
try:
on_ticket = json_data['on']
except:
return self._fail(req, apache.HTTP_NOT_FOUND)
webapi.session_bareinit(req)
pinfo = session['personinfo']
ticket = self._get_according_ticket(on_ticket, pinfo)
if ticket is None:
return self._fail(req, apache.HTTP_NOT_FOUND)
ticket_status = webapi.get_ticket_status(ticket)
session.dirty = True
req.content_type = 'application/json'
req.write(json.dumps(ticket_status))
def update_status(self, req, form):
'''
Function used for handling Ajax requests.
@param req: apache request object
@type req: apache request object
@param form: parameters sent via Ajax request
@type form: dict
@return:
@rtype: json data
'''
# Abort if the simplejson module isn't available
assert CFG_JSON_AVAILABLE, "Json not available"
# Fail if no json data exists in the Ajax request
if not form.has_key('jsondata'):
return self._fail(req, apache.HTTP_NOT_FOUND)
json_data = json.loads(str(form['jsondata']))
json_data = json_unicode_to_utf8(json_data)
try:
on_ticket = json_data['on']
except:
return self._fail(req, apache.HTTP_NOT_FOUND)
webapi.session_bareinit(req)
pinfo = session['personinfo']
ticket = self._get_according_ticket(on_ticket, pinfo)
if ticket is None:
return self._fail(req, apache.HTTP_NOT_FOUND)
webapi.update_ticket_status(ticket)
session.dirty = True
def add_operation(self, req, form):
'''
Function used for handling Ajax requests.
@param req: apache request object
@type req: apache request object
@param form: parameters sent via Ajax request
@type form: dict
@return:
@rtype: json data
'''
# Abort if the simplejson module isn't available
assert CFG_JSON_AVAILABLE, "Json not available"
# Fail if no json data exists in the Ajax request
if not form.has_key('jsondata'):
return self._fail(req, apache.HTTP_NOT_FOUND)
json_data = json.loads(str(form['jsondata']))
json_data = json_unicode_to_utf8(json_data)
try:
operation_parts = {'pid': int(json_data['pid']),
'action': json_data['action'],
'bibrefrec': json_data['bibrefrec']}
on_ticket = json_data['on']
except:
return self._fail(req, apache.HTTP_NOT_FOUND)
webapi.session_bareinit(req)
pinfo = session['personinfo']
uid = getUid(req)
operation_to_be_added = webapi.construct_operation(operation_parts, pinfo, uid)
if operation_to_be_added is None:
return self._fail(req, apache.HTTP_NOT_FOUND)
ticket = self._get_according_ticket(on_ticket, pinfo)
if ticket is None:
return self._fail(req, apache.HTTP_NOT_FOUND)
webapi.add_operation_to_ticket(operation_to_be_added, ticket)
session.dirty = True
def modify_operation(self, req, form):
'''
Function used for handling Ajax requests.
@param req: apache request object
@type req: apache request object
@param form: parameters sent via Ajax request
@type form: dict
@return:
@rtype: json data
'''
# Abort if the simplejson module isn't available
assert CFG_JSON_AVAILABLE, "Json not available"
# Fail if no json data exists in the Ajax request
if not form.has_key('jsondata'):
return self._fail(req, apache.HTTP_NOT_FOUND)
json_data = json.loads(str(form['jsondata']))
json_data = json_unicode_to_utf8(json_data)
try:
operation_parts = {'pid': int(json_data['pid']),
'action': json_data['action'],
'bibrefrec': json_data['bibrefrec']}
on_ticket = json_data['on']
except:
return self._fail(req, apache.HTTP_NOT_FOUND)
webapi.session_bareinit(req)
pinfo = session['personinfo']
uid = getUid(req)
operation_to_be_modified = webapi.construct_operation(operation_parts, pinfo, uid, should_have_bibref=False)
if operation_to_be_modified is None:
return self._fail(req, apache.HTTP_NOT_FOUND)
ticket = self._get_according_ticket(on_ticket, pinfo)
if ticket is None:
return self._fail(req, apache.HTTP_NOT_FOUND)
operation_is_modified = webapi.modify_operation_from_ticket(operation_to_be_modified, ticket)
if not operation_is_modified:
# Operation couldn't be modified because it doesn't exist in the
# ticket. Wrong parameters were given hence we should fail!
return self._fail(req, apache.HTTP_NOT_FOUND)
session.dirty = True
def remove_operation(self, req, form):
'''
Function used for handling Ajax requests.
@param req: apache request object
@type req: apache request object
@param form: parameters sent via Ajax request
@type form: dict
@return:
@rtype: json data
'''
# Abort if the simplejson module isn't available
assert CFG_JSON_AVAILABLE, "Json not available"
# Fail if no json data exists in the Ajax request
if not form.has_key('jsondata'):
return self._fail(req, apache.HTTP_NOT_FOUND)
json_data = json.loads(str(form['jsondata']))
json_data = json_unicode_to_utf8(json_data)
try:
operation_parts = {'pid': int(json_data['pid']),
'action': json_data['action'],
'bibrefrec': json_data['bibrefrec']}
on_ticket = json_data['on']
except:
return self._fail(req, apache.HTTP_NOT_FOUND)
webapi.session_bareinit(req)
pinfo = session['personinfo']
uid = getUid(req)
operation_to_be_removed = webapi.construct_operation(operation_parts, pinfo, uid)
if operation_to_be_removed is None:
return self._fail(req, apache.HTTP_NOT_FOUND)
ticket = self._get_according_ticket(on_ticket, pinfo)
if ticket is None:
return self._fail(req, apache.HTTP_NOT_FOUND)
operation_is_removed = webapi.remove_operation_from_ticket(operation_to_be_removed, ticket)
if not operation_is_removed:
# Operation couldn't be removed because it doesn't exist in the
# ticket. Wrong parameters were given hence we should fail!
return self._fail(req, apache.HTTP_NOT_FOUND)
session.dirty = True
def commit(self, req, form):
'''
Function used for handling Ajax requests.
@param req: apache request object
@type req: apache request object
@param form: parameters sent via Ajax request
@type form: dict
@return:
@rtype: json data
'''
# Abort if the simplejson module isn't available
assert CFG_JSON_AVAILABLE, "Json not available"
# Fail if no json data exists in the Ajax request
if not form.has_key('jsondata'):
return self._fail(req, apache.HTTP_NOT_FOUND)
json_data = json.loads(str(form['jsondata']))
json_data = json_unicode_to_utf8(json_data)
try:
additional_info = {'first_name': json_data.get('first_name',"Default"),
'last_name': json_data.get('last_name',"Default"),
'email': json_data.get('email',"Default"),
'comments': json_data['comments']}
on_ticket = json_data['on']
except:
return self._fail(req, apache.HTTP_NOT_FOUND)
webapi.session_bareinit(req)
pinfo = session['personinfo']
ulevel = pinfo['ulevel']
uid = getUid(req)
user_is_guest = isGuestUser(uid)
if not user_is_guest:
try:
additional_info['first_name'] = session['user_info']['external_firstname']
additional_info['last_name'] = session['user_info']['external_familyname']
additional_info['email'] = session['user_info']['email']
except KeyError:
additional_info['first_name'] = additional_info['last_name'] = additional_info['email'] = str(uid)
ticket = self._get_according_ticket(on_ticket, pinfo)
if ticket is None:
return self._fail(req, apache.HTTP_NOT_FOUND)
# When a guest is claiming we should not commit if he
# doesn't provide us his full personal information
strict_check = user_is_guest
userinfo = webapi.fill_out_userinfo(additional_info, uid, req.remote_ip, ulevel, strict_check=strict_check)
if userinfo is None:
return self._fail(req, apache.HTTP_NOT_FOUND)
webapi.commit_operations_from_ticket(ticket, userinfo, uid, ulevel)
session.dirty = True
def abort(self, req, form):
'''
Function used for handling Ajax requests.
@param req: apache request object
@type req: apache request object
@param form: parameters sent via Ajax request
@type form: dict
@return:
@rtype: json data
'''
# Abort if the simplejson module isn't available
assert CFG_JSON_AVAILABLE, "Json not available"
# Fail if no json data exists in the Ajax request
if not form.has_key('jsondata'):
return self._fail(req, apache.HTTP_NOT_FOUND)
json_data = json.loads(str(form['jsondata']))
json_data = json_unicode_to_utf8(json_data)
try:
on_ticket = json_data['on']
except:
return self._fail(req, apache.HTTP_NOT_FOUND)
webapi.session_bareinit(req)
pinfo = session['personinfo']
ticket = self._get_according_ticket(on_ticket, pinfo)
if ticket is None:
return self._fail(req, apache.HTTP_NOT_FOUND)
# When a user is claiming we should completely delete his ticket if he
# aborts the claiming procedure
delete_ticket = (on_ticket == 'user')
webapi.abort_ticket(ticket, delete_ticket=delete_ticket)
session.dirty = True
def _get_according_ticket(self, on_ticket, pinfo):
ticket = None
if on_ticket == 'user':
ticket = pinfo['ticket']
elif on_ticket == 'autoclaim':
ticket = pinfo['autoclaim']['ticket']
return ticket
def _fail(self, req, code):
req.status = code
return
class WebAuthorSearch(WebInterfaceDirectory):
"""
Provides an interface to profile search using AJAX queries.
"""
_exports = ['list',
'details']
# This class requires JSON libraries
assert CFG_JSON_AVAILABLE, "[WebAuthorSearch] JSON must be enabled."
class QueryPerson(WebInterfaceDirectory):
_exports = ['']
MIN_QUERY_LENGTH = 2
QUERY_REGEX = re.compile(r"[\w\s\.\-,@]+$", re.UNICODE)
def __init__(self, query=None):
self.query = query
def _lookup(self, component, path):
if component not in self._exports:
return WebAuthorSearch.QueryPerson(component), path
def __call__(self, req, form):
if self.query is None or len(self.query) < self.MIN_QUERY_LENGTH:
req.status = apache.HTTP_BAD_REQUEST
return "Query too short"
if not self.QUERY_REGEX.match(self.query):
req.status = apache.HTTP_BAD_REQUEST
return "Invalid query."
pid_results = [{"pid": pid[0]} for pid in webapi.search_person_ids_by_name(self.query)]
req.content_type = 'application/json'
return json.dumps(pid_results)
# Request for index handled by __call__
index = __call__
def _JSON_received(self, form):
try:
return "jsondata" in form
except TypeError:
return False
def _extract_JSON(self, form):
try:
json_data = json.loads(str(form['jsondata']))
json_data = json_unicode_to_utf8(json_data)
return json_data
except ValueError:
return None
def _get_pid_details(self, pid):
details = webapi.get_person_info_by_pid(pid)
details.update({
"names": [{"name": x, "paperCount": y} for x, y in webapi.get_person_names_from_id(pid)],
"externalIds": [{x: y} for x, y in webapi.get_external_ids_from_person_id(pid).items()]
})
details['cname'] = details.pop("canonical_name", None)
return details
def details(self, req, form):
if self._JSON_received(form):
try:
json_data = self._extract_JSON(form)
pids = json_data['pids']
req.content_type = 'application/json'
details = [self._get_pid_details(pid) for pid in pids]
return json.dumps(details)
except (TypeError, KeyError):
req.status = apache.HTTP_BAD_REQUEST
return "Invalid query."
else:
req.status = apache.HTTP_BAD_REQUEST
return "Incorrect query format."
list = QueryPerson()
class WebInterfaceAuthor(WebInterfaceDirectory):
'''
Handles /author/* pages.
Supplies the methods:
/author/choose_profile
/author/claim/
/author/help
/author/manage_profile
/author/merge_profiles
/author/profile/
/author/search
/author/ticket/
'''
_exports = ['',
'choose_profile',
'claim',
'help',
'manage_profile',
'merge_profiles',
'profile',
'search',
'search_ajax',
'ticket']
from invenio.webauthorprofile_webinterface import WebAuthorPages
claim = WebInterfaceBibAuthorIDClaimPages()
profile = WebAuthorPages()
choose_profile = claim.choose_profile
help = claim.help
manage_profile = WebInterfaceBibAuthorIDManageProfilePages()
merge_profiles = claim.merge_profiles
search = claim.search
search_ajax = WebAuthorSearch()
ticket = WebInterfaceAuthorTicketHandling()
def _lookup(self, component, path):
if component not in self._exports:
return WebInterfaceAuthor(component), path
def __init__(self, component=None):
self.path = component
def __call__(self, req, form):
if self.path is None or len(self.path) < 1:
redirect_to_url(req, "%s/author/search" % CFG_BASE_URL)
# Check if canonical id: e.g. "J.R.Ellis.1"
pid = get_person_id_from_canonical_id(self.path)
if pid >= 0:
url = "%s/author/profile/%s" % (CFG_BASE_URL, get_person_redirect_link(pid))
redirect_to_url(req, url, redirection_type=apache.HTTP_MOVED_PERMANENTLY)
return
else:
try:
pid = int(self.path)
except ValueError:
redirect_to_url(req, "%s/author/search?q=%s" % (CFG_BASE_URL, self.path))
return
else:
if author_has_papers(pid):
cid = get_person_redirect_link(pid)
if is_valid_canonical_id(cid):
redirect_id = cid
else:
redirect_id = pid
url = "%s/author/profile/%s" % (CFG_BASE_URL, redirect_id)
redirect_to_url(req, url, redirection_type=apache.HTTP_MOVED_PERMANENTLY)
return
redirect_to_url(req, "%s/author/search" % CFG_BASE_URL)
return
index = __call__
class WebInterfacePerson(WebInterfaceDirectory):
'''
Handles /person/* pages.
Supplies the methods:
/person/welcome
'''
_exports = ['welcome','update', 'you']
def welcome(self, req, form):
redirect_to_url(req, "%s/author/choose_profile" % CFG_SITE_SECURE_URL)
def you(self, req, form):
redirect_to_url(req, "%s/author/choose_profile" % CFG_SITE_SECURE_URL)
def update(self, req, form):
"""
Generate hepnames update form
"""
argd = wash_urlargd(form,
{'ln': (str, CFG_SITE_LANG),
'email': (str, ''),
'IRN': (str, ''),
})
# Retrieve info for HEP name based on email or IRN
recids = []
if argd['email']:
recids = perform_request_search(p="371__m:%s" % argd['email'], cc="HepNames")
elif argd['IRN']:
recids = perform_request_search(p="001:%s" % argd['IRN'], cc="HepNames")
else:
redirect_to_url(req, "%s/collection/HepNames" % (CFG_SITE_URL))
if not recids:
redirect_to_url(req, "%s/collection/HepNames" % (CFG_SITE_URL))
else:
hepname_bibrec = get_bibrecord(recids[0])
# Extract all info from recid that should be included in the form
full_name = record_get_field_value(hepname_bibrec, tag="100", ind1="", ind2="", code="a")
display_name = record_get_field_value(hepname_bibrec, tag="880", ind1="", ind2="", code="a")
email = record_get_field_value(hepname_bibrec, tag="371", ind1="", ind2="", code="m")
status = record_get_field_value(hepname_bibrec, tag="100", ind1="", ind2="", code="g")
keynumber = record_get_field_value(hepname_bibrec, tag="970", ind1="", ind2="", code="a")
try:
keynumber = keynumber.split('-')[1]
except IndexError:
pass
research_field_list = record_get_field_values(hepname_bibrec, tag="650", ind1="1", ind2="7", code="a")
institution_list = []
for instance in record_get_field_instances(hepname_bibrec, tag="371", ind1="", ind2=""):
if not instance or field_get_subfield_values(instance, "m"):
continue
institution_info = ["", "", "", "", ""]
if field_get_subfield_values(instance, "a"):
institution_info[0] = field_get_subfield_values(instance, "a")[0]
if field_get_subfield_values(instance, "r"):
institution_info[1] = field_get_subfield_values(instance, "r")[0]
if field_get_subfield_values(instance, "s"):
institution_info[2] = field_get_subfield_values(instance, "s")[0]
if field_get_subfield_values(instance, "t"):
institution_info[3] = field_get_subfield_values(instance, "t")[0]
if field_get_subfield_values(instance, "z"):
institution_info[4] = field_get_subfield_values(instance, "z")[0]
institution_list.append(institution_info)
phd_advisor_list = record_get_field_values(hepname_bibrec, tag="701", ind1="", ind2="", code="a")
experiment_list = record_get_field_values(hepname_bibrec, tag="693", ind1="", ind2="", code="e")
web_page = record_get_field_value(hepname_bibrec, tag="856", ind1="1", ind2="", code="u")
# Create form and pass as parameters all the content from the record
body = TEMPLATE.tmpl_update_hep_name(full_name, display_name, email,
status, research_field_list,
institution_list, phd_advisor_list,
experiment_list, web_page)
title = "HEPNames"
return page(title=title,
metaheaderadd = TEMPLATE.tmpl_update_hep_name_headers(),
body=body,
req=req,
)
# pylint: enable=C0301
# pylint: enable=W0613
diff --git a/invenio/legacy/bibcatalog/system_rt.py b/invenio/legacy/bibcatalog/system_rt.py
index 132539333..a6c7b9008 100644
--- a/invenio/legacy/bibcatalog/system_rt.py
+++ b/invenio/legacy/bibcatalog/system_rt.py
@@ -1,469 +1,470 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2009, 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
Provide a "ticket" interface with a request tracker.
This is a subclass of BibCatalogSystem
"""
import os
import re
import invenio.legacy.webuser
from invenio.utils.shell import run_shell_command, escape_shell_arg
from invenio.legacy.bibcatalog.system import BibCatalogSystem, get_bibcat_from_prefs
from invenio.config import CFG_BIBCATALOG_SYSTEM, \
CFG_BIBCATALOG_SYSTEM_RT_CLI, \
CFG_BIBCATALOG_SYSTEM_RT_URL, \
CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_USER, \
CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_PWD, \
CFG_BIBEDIT_ADD_TICKET_RT_QUEUES
class BibCatalogSystemRT(BibCatalogSystem):
BIBCATALOG_RT_SERVER = "" # construct this by http://user:password@RT_URL
def check_system(self, uid=None):
"""return an error string if there are problems"""
if uid:
rtuid, rtpw = get_bibcat_from_prefs(uid)
else:
# Assume default RT user
rtuid = CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_USER
rtpw = CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_PWD
if not rtuid and not rtpw:
return "No valid RT user login specified"
if not CFG_BIBCATALOG_SYSTEM == 'RT':
return "CFG_BIBCATALOG_SYSTEM is not RT though this is an RT module"
if not CFG_BIBCATALOG_SYSTEM_RT_CLI:
return "CFG_BIBCATALOG_SYSTEM_RT_CLI not defined or empty"
if not os.path.exists(CFG_BIBCATALOG_SYSTEM_RT_CLI):
return "CFG_BIBCATALOG_SYSTEM_RT_CLI " + CFG_BIBCATALOG_SYSTEM_RT_CLI + " file does not exists"
# Check that you can execute the binary.. this is a safe call unless someone can fake CFG_BIBCATALOG_SYSTEM_RT_CLI (unlikely)
dummy, myout, myerr = run_shell_command(CFG_BIBCATALOG_SYSTEM_RT_CLI + " help")
helpfound = False
if myerr.count("help") > 0:
helpfound = True
if not helpfound:
return "Execution of CFG_BIBCATALOG_SYSTEM_RT_CLI " + CFG_BIBCATALOG_SYSTEM_RT_CLI + " help did not produce output 'help'"
if not CFG_BIBCATALOG_SYSTEM_RT_URL:
return "CFG_BIBCATALOG_SYSTEM_RT_URL not defined or empty"
# Construct URL, split RT_URL at //
if not CFG_BIBCATALOG_SYSTEM_RT_URL.startswith('http://') and \
not CFG_BIBCATALOG_SYSTEM_RT_URL.startswith('https://'):
return "CFG_BIBCATALOG__SYSTEM_RT_URL does not start with 'http://' or 'https://'"
httppart, siteandpath = CFG_BIBCATALOG_SYSTEM_RT_URL.split("//")
# Assemble by http://user:password@RT_URL
bibcatalog_rt_server = httppart + "//" + rtuid + ":" + rtpw + "@" + siteandpath
#set as env var
os.environ["RTUSER"] = rtuid
os.environ["RTSERVER"] = bibcatalog_rt_server
#try to talk to RT server
#this is a safe call since rtpw is the only variable in it, and it is escaped
rtpw = escape_shell_arg(rtpw)
dummy, myout, myerr = run_shell_command("echo "+rtpw+" | " + CFG_BIBCATALOG_SYSTEM_RT_CLI + " ls \"Subject like 'F00'\"")
if len(myerr) > 0:
return "could not connect to " + bibcatalog_rt_server + " " + myerr
#finally, check that there is some sane output like tickets or 'No matching result'
saneoutput = (myout.count('matching') > 0) or (myout.count('1') > 0)
if not saneoutput:
return CFG_BIBCATALOG_SYSTEM_RT_CLI + " returned " + myout + " instead of 'matching' or '1'"
return ""
def ticket_search(self, uid, recordid=-1, subject="", text="", creator="",
owner="", date_from="", date_until="", status="",
priority="", queue=""):
"""returns a list of ticket ID's related to this record or by
matching the subject, creator or owner of the ticket."""
search_atoms = [] # the search expression will be made by and'ing these
if (recordid > -1):
#search by recid
search_atoms.append("CF.{RecordID} = " + escape_shell_arg(str(recordid)))
if subject:
#search by subject
search_atoms.append("Subject like " + escape_shell_arg(str(subject)))
if text:
search_atoms.append("Content like " + escape_shell_arg(str(text)))
if str(creator):
#search for this person's bibcatalog_username in preferences
creatorprefs = invenio.legacy.webuser.get_user_preferences(creator)
creator = "Nobody can Have This Kind of Name"
if "bibcatalog_username" in creatorprefs:
creator = creatorprefs["bibcatalog_username"]
search_atoms.append("Creator = " + escape_shell_arg(str(creator)))
if str(owner):
ownerprefs = invenio.legacy.webuser.get_user_preferences(owner)
owner = "Nobody can Have This Kind of Name"
if "bibcatalog_username" in ownerprefs:
owner = ownerprefs["bibcatalog_username"]
search_atoms.append("Owner = " + escape_shell_arg(str(owner)))
if date_from:
search_atoms.append("Created >= " + escape_shell_arg(str(date_from)))
if date_until:
search_atoms.append("Created <= " + escape_shell_arg(str(date_until)))
if str(status) and isinstance(status, type("this is a string")):
search_atoms.append("Status = " + escape_shell_arg(str(status)))
if str(priority):
# Try to convert to int
intpri = -1
try:
intpri = int(priority)
except ValueError:
pass
if intpri > -1:
search_atoms.append("Priority = " + str(intpri))
if queue:
search_atoms.append("Queue = " + escape_shell_arg(queue))
searchexp = " and ".join(search_atoms)
tickets = []
if len(searchexp) == 0:
return tickets
command = CFG_BIBCATALOG_SYSTEM_RT_CLI + " ls -l \"" + searchexp + "\""
command_out = self._run_rt_command(command, uid)
if command_out is None:
return tickets
statuses = []
for line in command_out.split("\n"):
#if there are matching lines they will look like NUM:subj.. so pick num
if line.count('id: ticket/') > 0:
dummy, tnum = line.split('/') # get the ticket id
try:
dummy = int(tnum)
except ValueError:
pass
else:
tickets.append(tnum)
if line.count('Status: ') > 0:
dummy, tstatus = line.split('Status: ')
statuses.append(tstatus)
if isinstance(status, list):
#take only those tickets whose status matches with one of the status list
alltickets = tickets
tickets = []
for i in range(len(alltickets)):
tstatus = statuses[i]
tnum = alltickets[i]
if status.count(tstatus) > 0: # match
tickets.append(tnum)
return tickets
def ticket_submit(self, uid=None, subject="", recordid=-1, text="",
queue="", priority="", owner="", requestor=""):
comment = False
if "\n" in text:
# The RT client does not support newlines in the initial body
# We need to add the ticket then add a comment.
comment = True
res = self._ticket_submit(uid=uid, subject=subject,
queue=queue,
recordid=recordid)
else:
res = self._ticket_submit(uid=uid, subject=subject,
queue=queue,
text=text,
recordid=recordid)
try:
# The BibCatalog API returns int if successful or
# a string explaining the error if unsuccessful.
ticketid = int(res)
except (ValueError, TypeError), e:
# Not a number. Must be an error string
raise Exception("%s\n%s" % (res, str(e)))
if comment:
self.ticket_comment(uid=uid,
ticketid=ticketid,
comment=text)
return ticketid
def _ticket_submit(self, uid=None, subject="", recordid=-1, text="",
queue="", priority="", owner="", requestor=""):
"""creates a ticket. return ticket num on success, otherwise None"""
queueset = ""
textset = ""
priorityset = ""
ownerset = ""
subjectset = ""
requestorset = ""
if subject:
- subjectset = " subject=" + escape_shell_arg(subject)
+ cleaned_subject = " ".join(str(subject).splitlines())
+ subjectset = " subject=" + escape_shell_arg(cleaned_subject)
recidset = " CF-RecordID=" + escape_shell_arg(str(recordid))
if priority:
priorityset = " priority=" + escape_shell_arg(str(priority))
if queue:
queueset = " queue=" + escape_shell_arg(queue)
if requestor:
requestorset = " requestor=" + escape_shell_arg(requestor)
if owner:
#get the owner name from prefs
ownerprefs = invenio.legacy.webuser.get_user_preferences(owner)
if "bibcatalog_username" in ownerprefs:
owner = ownerprefs["bibcatalog_username"]
ownerset = " owner=" + escape_shell_arg(owner)
if text:
if '\n' in text:
# contains newlines (\n) return with error
raise Exception("Newlines are not allowed in text parameter. Use ticket_comment() instead.")
else:
textset = " text=" + escape_shell_arg(text)
# make a command.. note that all set 'set' parts have been escaped
command = CFG_BIBCATALOG_SYSTEM_RT_CLI + " create -t ticket set " + subjectset + recidset + \
queueset + textset + priorityset + ownerset + requestorset
command_out = self._run_rt_command(command, uid)
if command_out is None:
return None
ticket_id = None
ticket_created_re = re.compile(r'Ticket (\d+) created')
for line in command_out.split("\n"):
matches = ticket_created_re.search(line)
if matches:
ticket_id = int(matches.groups()[0])
return ticket_id
def ticket_comment(self, uid, ticketid, comment):
"""comment on a given ticket. Returns 1 on success, 0 on failure"""
command = '%s comment -m %s %s' % (CFG_BIBCATALOG_SYSTEM_RT_CLI,
escape_shell_arg(comment),
str(ticketid))
command_out = self._run_rt_command(command, uid)
if command_out is None:
return None
return 1
def ticket_assign(self, uid, ticketid, to_user):
"""assign a ticket to an RT user. Returns 1 on success, 0 on failure"""
return self.ticket_set_attribute(uid, ticketid, 'owner', to_user)
def ticket_steal(self, uid, ticketid):
"""steal a ticket from an RT user. Returns 1 on success, 0 on failure"""
command = CFG_BIBCATALOG_SYSTEM_RT_CLI + " steal " + str(ticketid)
command_out = self._run_rt_command(command, uid)
if command_out is None:
return 0
return 1
def ticket_set_attribute(self, uid, ticketid, attribute, new_value):
"""change the ticket's attribute. Returns 1 on success, 0 on failure"""
#check that the attribute is accepted..
if attribute not in BibCatalogSystem.TICKET_ATTRIBUTES:
return 0
#we cannot change read-only values.. including text that is an attachment. pity
if attribute in ['creator', 'date', 'ticketid', 'url_close', 'url_display', 'recordid', 'text']:
return 0
#check attribute
setme = ""
if attribute == 'priority':
try:
dummy = int(new_value)
except ValueError:
return 0
setme = "set Priority=" + str(new_value)
if attribute == 'subject':
subject = escape_shell_arg(new_value)
setme = "set Subject='" + subject + "'"
if attribute == 'owner':
#convert from invenio to RT
ownerprefs = invenio.legacy.webuser.get_user_preferences(new_value)
if "bibcatalog_username" not in ownerprefs:
return 0
else:
owner = escape_shell_arg(ownerprefs["bibcatalog_username"])
setme = " set owner='" + owner + "'"
if attribute == 'status':
setme = " set status='" + escape_shell_arg(new_value) + "'"
if attribute == 'queue':
setme = " set queue='" + escape_shell_arg(new_value) + "'"
#make sure ticketid is numeric
try:
dummy = int(ticketid)
except ValueError:
return 0
command = CFG_BIBCATALOG_SYSTEM_RT_CLI + " edit ticket/" + str(ticketid) + setme
command_out = self._run_rt_command(command, uid)
if command_out is None:
return 0
mylines = command_out.split("\n")
for line in mylines:
if line.count('updated') > 0:
return 1
return 0
def ticket_get_attribute(self, uid, ticketid, attribute):
"""return an attribute of a ticket"""
ticinfo = self.ticket_get_info(uid, ticketid, [attribute])
if attribute in ticinfo:
return ticinfo[attribute]
return None
def ticket_get_info(self, uid, ticketid, attributes = None):
"""return ticket info as a dictionary of pre-defined attribute names.
Or just those listed in attrlist.
Returns None on failure"""
# Make sure ticketid is numeric
try:
dummy = int(ticketid)
except ValueError:
return 0
if attributes is None:
attributes = []
command = CFG_BIBCATALOG_SYSTEM_RT_CLI + " show ticket/" + str(ticketid)
command_out = self._run_rt_command(command, uid)
if command_out is None:
return 0
tdict = {}
for line in command_out.split("\n"):
if line.count(": ") > 0:
tattr, tvaluen = line.split(": ")[0], ": ".join(line.split(": ")[1:])
tvalue = tvaluen.rstrip()
tdict[tattr] = tvalue
# Query again to get attachments -> Contents
command = CFG_BIBCATALOG_SYSTEM_RT_CLI + " show ticket/" + str(ticketid) + "/attachments/"
command_out = self._run_rt_command(command, uid)
if command_out is None:
return 0
attachments = []
regex = re.compile(r"[0-9]*:\s{2}[(]") # attachment's format: 557408: (text/plain / 131b)
for line in command_out.split("\n"):
for match in regex.findall(line):
attachments.append(match.split(":")[0])
# Query again for each attachment
for att in attachments:
command = CFG_BIBCATALOG_SYSTEM_RT_CLI + " show ticket/" + str(ticketid) + "/attachments/" + att
command_out = self._run_rt_command(command, uid)
if command_out is None:
return 0
# Get the contents line
for line in command_out.split("\n"):
if line.count("Content: ") > 0:
cstuff = line.split("Content: ")
tdict['Text'] = cstuff[1].rstrip()
if (len(tdict) > 0):
# Iterate over TICKET_ATTRIBUTES to make a canonical ticket
candict = {}
for f in BibCatalogSystem.TICKET_ATTRIBUTES:
tcased = f.title()
if tcased in tdict:
candict[f] = tdict[tcased]
if 'CF.{RecordID}' in tdict:
candict['recordid'] = tdict['CF.{RecordID}']
if 'id' in tdict:
candict['ticketid'] = tdict['id']
# Make specific URL attributes:
url_display = CFG_BIBCATALOG_SYSTEM_RT_URL + "/Ticket/Display.html?id=" + str(ticketid)
candict['url_display'] = url_display
url_close = CFG_BIBCATALOG_SYSTEM_RT_URL + "/Ticket/Update.html?Action=Comment&DefaultStatus=resolved&id=" + str(ticketid)
candict['url_close'] = url_close
url_modify = CFG_BIBCATALOG_SYSTEM_RT_URL + "/Ticket/ModifyAll.html?id=" + str(ticketid)
candict['url_modify'] = url_modify
# Change the ticket owner into invenio UID
if 'owner' in tdict:
rt_owner = tdict["owner"]
uid = invenio.legacy.webuser.get_uid_based_on_pref("bibcatalog_username", rt_owner)
candict['owner'] = uid
if len(attributes) == 0: # return all fields
return candict
else: # return only the fields that were requested
tdict = {}
for myatt in attributes:
if myatt in candict:
tdict[myatt] = candict[myatt]
return tdict
else:
return None
def get_queues(self, uid):
"""get all the queues from RT. Returns a list of queues"""
# get all queues with id from 1-100 in order to get all the available queues.
# Then filters the queues keeping these selected in the configuration variable
command = CFG_BIBCATALOG_SYSTEM_RT_CLI + " show -t queue -f name 1-100"
command_out = self._run_rt_command(command, uid)
spl = command_out.split("\n")
queues = []
for i, line in enumerate(spl):
if 'id' in line:
_id = line.split("/")[1]
# name is on the next line after id
name = spl[i+1].split(": ")[1]
if name in CFG_BIBEDIT_ADD_TICKET_RT_QUEUES:
queue = {'id': _id, 'name': name}
queues.append(queue)
return queues
def _run_rt_command(self, command, uid=None):
"""
This function will run a RT CLI command as given user. If no user is specified
the default RT user will be used, if configured.
Should any of the configuration parameters be missing this function will return
None. Otherwise it will return the standard output from the CLI command.
@param command: RT CLI command to execute
@type command: string
@param uid: the Invenio user id to submit on behalf of. Optional.
@type uid: int
@return: standard output from the command given. None, if any errors.
@rtype: string
"""
if not CFG_BIBCATALOG_SYSTEM_RT_URL:
return None
if uid:
username, passwd = get_bibcat_from_prefs(uid)
else:
username = CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_USER
passwd = CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_PWD
httppart, siteandpath = CFG_BIBCATALOG_SYSTEM_RT_URL.split("//")
bibcatalog_rt_server = httppart + "//" + username + ":" + passwd + "@" + siteandpath
#set as env var
os.environ["RTUSER"] = username
os.environ["RTSERVER"] = bibcatalog_rt_server
passwd = escape_shell_arg(passwd)
error_code, myout, dummyerr = run_shell_command("echo " + passwd + " | " + command)
if error_code > 0:
raise ValueError('Problem running "%s": %d' % (command, error_code))
return myout
diff --git a/invenio/legacy/bibdocfile/api.py b/invenio/legacy/bibdocfile/api.py
index 65fa2d0cf..bd4b7450b 100644
--- a/invenio/legacy/bibdocfile/api.py
+++ b/invenio/legacy/bibdocfile/api.py
@@ -1,4886 +1,4886 @@
## This file is part of Invenio.
## Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
"""
This module implements the low-level API for dealing with fulltext files.
- All the files associated to a I{record} (identified by a I{recid}) can be
managed via an instance of the C{BibRecDocs} class.
- A C{BibRecDocs} is a wrapper of the list of I{documents} attached to the
record.
- Each document is represented by an instance of the C{BibDoc} class.
- A document is identified by a C{docid} and name (C{docname}). The docname
must be unique within the record. A document is the set of all the
formats and revisions of a piece of information.
- A document has a type called C{doctype} and can have a restriction.
- Each physical file, i.e. the concretization of a document into a
particular I{version} and I{format} is represented by an instance of the
C{BibDocFile} class.
- The format is infact the extension of the physical file.
- A comment and a description and other information can be associated to a
BibDocFile.
- A C{bibdoc} is a synonim for a document, while a C{bibdocfile} is a
synonim for a physical file.
@group Main classes: BibRecDocs,BibDoc,BibDocFile
@group Other classes: BibDocMoreInfo,Md5Folder,InvenioBibDocFileError
@group Main functions: decompose_file,stream_file,bibdocfile_*,download_url
@group Configuration Variables: CFG_*
"""
__revision__ = "$Id$"
import os
import re
import shutil
import filecmp
import time
import random
import socket
import urllib2
import urllib
import tempfile
from six.moves import cPickle
import base64
import binascii
import cgi
import sys
try:
import magic
if hasattr(magic, "open"):
CFG_HAS_MAGIC = 1
if not hasattr(magic, "MAGIC_MIME_TYPE"):
## Patching RHEL6/CentOS6 version
magic.MAGIC_MIME_TYPE = 16
elif hasattr(magic, "Magic"):
CFG_HAS_MAGIC = 2
except ImportError:
CFG_HAS_MAGIC = 0
from datetime import datetime
from mimetypes import MimeTypes
from thread import get_ident
from six import iteritems
from invenio.utils import apache
## Let's set a reasonable timeout for URL request (e.g. FFT)
socket.setdefaulttimeout(40)
if sys.hexversion < 0x2040000:
# pylint: disable=W0622
from sets import Set as set
# pylint: enable=W0622
from invenio.utils.shell import escape_shell_arg, run_shell_command
from invenio.legacy.dbquery import run_sql, DatabaseError
from invenio.ext.logging import register_exception
from invenio.legacy.bibrecord import record_get_field_instances, \
field_get_subfield_values, field_get_subfield_instances, \
encode_for_xml
from invenio.utils.url import create_url, make_user_agent_string
from invenio.utils.text import nice_size
from invenio.modules.access.engine import acc_authorize_action
from invenio.modules.access.control import acc_is_user_in_role, acc_get_role_id
from invenio.modules.access.firerole import compile_role_definition, acc_firerole_check_user
from invenio.modules.access.local_config import SUPERADMINROLE, CFG_WEBACCESS_WARNING_MSGS
from invenio.config import CFG_SITE_URL, \
CFG_WEBDIR, CFG_BIBDOCFILE_FILEDIR,\
CFG_BIBDOCFILE_ADDITIONAL_KNOWN_FILE_EXTENSIONS, \
CFG_BIBDOCFILE_FILESYSTEM_BIBDOC_GROUP_LIMIT, CFG_SITE_SECURE_URL, \
CFG_BIBUPLOAD_FFT_ALLOWED_LOCAL_PATHS, \
CFG_TMPDIR, CFG_TMPSHAREDDIR, CFG_PATH_MD5SUM, \
CFG_WEBSUBMIT_STORAGEDIR, \
CFG_BIBDOCFILE_USE_XSENDFILE, \
CFG_BIBDOCFILE_MD5_CHECK_PROBABILITY, \
CFG_SITE_RECORD, CFG_PYLIBDIR, \
CFG_BIBUPLOAD_FFT_ALLOWED_EXTERNAL_URLS, \
CFG_BIBDOCFILE_ENABLE_BIBDOCFSINFO_CACHE, \
- CFG_BIBINDEX_PERFORM_OCR_ON_DOCNAMES, \
CFG_BIBDOCFILE_ADDITIONAL_KNOWN_MIMETYPES, \
CFG_BIBDOCFILE_PREFERRED_MIMETYPES_MAPPING, \
CFG_BIBCATALOG_SYSTEM
from invenio.legacy.bibcatalog.api import BIBCATALOG_SYSTEM
from invenio.legacy.bibdocfile.config import CFG_BIBDOCFILE_ICON_SUBFORMAT_RE, \
CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT
from invenio.base.utils import import_submodules_from_packages
from invenio.utils.hash import md5
import invenio.legacy.template
def _plugin_bldr(plugin_code):
"""Preparing the plugin dictionary structure"""
if not plugin_code.__name__.split('.')[-1].startswith('bom_'):
return
ret = {}
ret['create_instance'] = getattr(plugin_code, "create_instance", None)
ret['supports'] = getattr(plugin_code, "supports", None)
return ret
_CFG_BIBDOC_PLUGINS = None
def get_plugins():
"""
Lazy loading of plugins
"""
global _CFG_BIBDOC_PLUGINS
if _CFG_BIBDOC_PLUGINS is None:
_CFG_BIBDOC_PLUGINS = filter(None, map(
_plugin_bldr, import_submodules_from_packages(
'plugins', packages=['invenio.legacy.bibdocfile'])))
return _CFG_BIBDOC_PLUGINS
bibdocfile_templates = invenio.legacy.template.load('bibdocfile')
## The above flag controls whether HTTP range requests are supported or not
## when serving static files via Python. This is disabled by default as
## it currently breaks support for opening PDF files on Windows platforms
## using Acrobat reader brower plugin.
CFG_ENABLE_HTTP_RANGE_REQUESTS = False
#: block size when performing I/O.
CFG_BIBDOCFILE_BLOCK_SIZE = 1024 * 8
#: threshold used do decide when to use Python MD5 of CLI MD5 algorithm.
CFG_BIBDOCFILE_MD5_THRESHOLD = 256 * 1024
#: chunks loaded by the Python MD5 algorithm.
CFG_BIBDOCFILE_MD5_BUFFER = 1024 * 1024
#: whether to normalize e.g. ".JPEG" and ".jpg" into .jpeg.
CFG_BIBDOCFILE_STRONG_FORMAT_NORMALIZATION = False
#: flags that can be associated to files.
CFG_BIBDOCFILE_AVAILABLE_FLAGS = (
'PDF/A',
'STAMPED',
'PDFOPT',
'HIDDEN',
'CONVERTED',
'PERFORM_HIDE_PREVIOUS',
'OCRED'
)
DBG_LOG_QUERIES = False
#: constant used if FFT correct with the obvious meaning.
KEEP_OLD_VALUE = 'KEEP-OLD-VALUE'
_CFG_BIBUPLOAD_FFT_ALLOWED_EXTERNAL_URLS = [(re.compile(_regex), _headers)
for _regex, _headers in CFG_BIBUPLOAD_FFT_ALLOWED_EXTERNAL_URLS]
_mimes = MimeTypes(strict=False)
_mimes.suffix_map.update({'.tbz2' : '.tar.bz2'})
_mimes.encodings_map.update({'.bz2' : 'bzip2'})
if CFG_BIBDOCFILE_ADDITIONAL_KNOWN_MIMETYPES:
for key, value in iteritems(CFG_BIBDOCFILE_ADDITIONAL_KNOWN_MIMETYPES):
_mimes.add_type(key, value)
del key, value
_magic_cookies = {}
if CFG_HAS_MAGIC == 1:
def _get_magic_cookies():
"""
@return: a tuple of magic object.
@rtype: (MAGIC_NONE, MAGIC_COMPRESS, MAGIC_MIME, MAGIC_COMPRESS + MAGIC_MIME)
@note: ... not real magic. Just see: man file(1)
"""
thread_id = get_ident()
if thread_id not in _magic_cookies:
_magic_cookies[thread_id] = {
magic.MAGIC_NONE: magic.open(magic.MAGIC_NONE),
magic.MAGIC_COMPRESS: magic.open(magic.MAGIC_COMPRESS),
magic.MAGIC_MIME: magic.open(magic.MAGIC_MIME),
magic.MAGIC_COMPRESS + magic.MAGIC_MIME: magic.open(magic.MAGIC_COMPRESS + magic.MAGIC_MIME),
magic.MAGIC_MIME_TYPE: magic.open(magic.MAGIC_MIME_TYPE),
}
for key in _magic_cookies[thread_id].keys():
_magic_cookies[thread_id][key].load()
return _magic_cookies[thread_id]
elif CFG_HAS_MAGIC == 2:
def _magic_wrapper(local_path, mime=True, mime_encoding=False):
thread_id = get_ident()
if (thread_id, mime, mime_encoding) not in _magic_cookies:
magic_object = _magic_cookies[thread_id, mime, mime_encoding] = magic.Magic(mime=mime, mime_encoding=mime_encoding)
else:
magic_object = _magic_cookies[thread_id, mime, mime_encoding]
return magic_object.from_file(local_path) # pylint: disable=E1103
def _generate_extensions():
"""
Generate the regular expression to match all the known extensions.
@return: the regular expression.
@rtype: regular expression object
"""
_tmp_extensions = _mimes.encodings_map.keys() + \
_mimes.suffix_map.keys() + \
_mimes.types_map[1].keys() + \
CFG_BIBDOCFILE_ADDITIONAL_KNOWN_FILE_EXTENSIONS
extensions = []
for ext in _tmp_extensions:
if ext.startswith('.'):
extensions.append(ext)
else:
extensions.append('.' + ext)
extensions.sort()
extensions.reverse()
extensions = set([ext.lower() for ext in extensions])
extensions = '\\' + '$|\\'.join(extensions) + '$'
extensions = extensions.replace('+', '\\+')
return re.compile(extensions, re.I)
#: Regular expression to recognized extensions.
_extensions = _generate_extensions()
class InvenioBibDocFileError(Exception):
"""
Exception raised in case of errors related to fulltext files.
"""
pass
class InvenioBibdocfileUnauthorizedURL(InvenioBibDocFileError):
"""
Exception raised in case of errors related to fulltext files.
"""
## NOTE: this is a legacy Exception
pass
def _val_or_null(val, eq_name = None, q_str = None, q_args = None):
"""
Auxiliary function helpful while building WHERE clauses of SQL queries
that should contain field=val or field is val
If optional parameters q_str and q_args are provided, lists are updated
if val == None, a statement of the form "eq_name is Null" is returned
otherwise, otherwise the function returns a parametrised comparison
"eq_name=%s" with val as an argument added to the query args list.
Using parametrised queries diminishes the likelihood of having
SQL injection.
@param val Value to compare with
@type val
@param eq_name The name of the database column
@type eq_name string
@param q_str Query string builder - list of clauses
that should be connected by AND operator
@type q_str list
@param q_args Query arguments list. This list will be applied as
a second argument of run_sql command
@type q_args list
@result string of a single part of WHERE clause
@rtype string
"""
res = ""
if eq_name != None:
res += eq_name
if val == None:
if eq_name != None:
res += " is "
res += "NULL"
if q_str != None:
q_str.append(res)
return res
else:
if eq_name != None:
res += "="
res += "%s"
if q_str != None:
q_str.append(res)
if q_args != None:
q_args.append(str(val))
return res
def _sql_generate_conjunctive_where(to_process):
"""Generating WHERE clause of a SQL statement, consisting of conjunction
of declared terms. Terms are defined by the to_process argument.
the method creates appropriate entries different in the case, value
should be NULL (None in the list) and in the case of not-none arguments.
In the second case, parametrised query is generated decreasing the
chance of an SQL-injection.
@param to_process List of tuples (value, database_column)
@type to_process list"""
q_str = []
q_args = []
for entry in to_process:
q_str.append(_val_or_null(entry[0], eq_name = entry[1], q_args = q_args))
return (" AND ".join(q_str), q_args)
def file_strip_ext(afile, skip_version=False, only_known_extensions=False, allow_subformat=True):
"""
Strip in the best way the extension from a filename.
>>> file_strip_ext("foo.tar.gz")
'foo'
>>> file_strip_ext("foo.buz.gz")
'foo.buz'
>>> file_strip_ext("foo.buz")
'foo'
>>> file_strip_ext("foo.buz", only_known_extensions=True)
'foo.buz'
>>> file_strip_ext("foo.buz;1", skip_version=False,
... only_known_extensions=True)
'foo.buz;1'
>>> file_strip_ext("foo.gif;icon")
'foo'
>>> file_strip_ext("foo.gif:icon", allow_subformat=False)
'foo.gif:icon'
@param afile: the path/name of a file.
@type afile: string
@param skip_version: whether to skip a trailing ";version".
@type skip_version: bool
@param only_known_extensions: whether to strip out only known extensions or
to consider as extension anything that follows a dot.
@type only_known_extensions: bool
@param allow_subformat: whether to consider also subformats as part of
the extension.
@type allow_subformat: bool
@return: the name/path without the extension (and version).
@rtype: string
"""
if skip_version or allow_subformat:
afile = afile.split(';')[0]
nextfile = _extensions.sub('', afile)
if nextfile == afile and not only_known_extensions:
nextfile = os.path.splitext(afile)[0]
while nextfile != afile:
afile = nextfile
nextfile = _extensions.sub('', afile)
return nextfile
def normalize_format(docformat, allow_subformat=True):
"""
Normalize the format, e.g. by adding a dot in front.
@param format: the format/extension to be normalized.
@type format: string
@param allow_subformat: whether to consider also subformats as part of
the extension.
@type allow_subformat: bool
@return: the normalized format.
@rtype; string
"""
if not docformat:
return ''
if allow_subformat:
subformat = docformat[docformat.rfind(';'):]
docformat = docformat[:docformat.rfind(';')]
else:
subformat = ''
if docformat and docformat[0] != '.':
docformat = '.' + docformat
if CFG_BIBDOCFILE_STRONG_FORMAT_NORMALIZATION:
if docformat not in ('.Z', '.H', '.C', '.CC'):
docformat = docformat.lower()
docformat = {
'.jpg' : '.jpeg',
'.htm' : '.html',
'.tif' : '.tiff'
}.get(docformat, docformat)
return docformat + subformat
def guess_format_from_url(url):
"""
Given a URL tries to guess it's extension.
Different method will be used, including HTTP HEAD query,
downloading the resource and using mime
@param url: the URL for which the extension should be guessed.
@type url: string
@return: the recognized extension or '.bin' if it's impossible to
recognize it.
@rtype: string
"""
def guess_via_magic(local_path):
try:
if CFG_HAS_MAGIC == 1:
magic_cookie = _get_magic_cookies()[magic.MAGIC_MIME_TYPE]
mimetype = magic_cookie.file(local_path)
elif CFG_HAS_MAGIC == 2:
mimetype = _magic_wrapper(local_path, mime=True, mime_encoding=False)
if CFG_HAS_MAGIC:
if mimetype in CFG_BIBDOCFILE_PREFERRED_MIMETYPES_MAPPING:
return normalize_format(CFG_BIBDOCFILE_PREFERRED_MIMETYPES_MAPPING[mimetype])
else:
return normalize_format(_mimes.guess_extension(mimetype))
except Exception:
pass
## Let's try to guess the extension by considering the URL as a filename
ext = decompose_file(url, skip_version=True, only_known_extensions=True)[2]
if ext.startswith('.'):
return ext
if is_url_a_local_file(url):
## The URL corresponds to a local file, so we can safely consider
## traditional extensions after the dot.
ext = decompose_file(url, skip_version=True, only_known_extensions=False)[2]
if ext.startswith('.'):
return ext
## No extensions? Let's use Magic.
ext = guess_via_magic(url)
if ext:
return ext
else:
## Since the URL is remote, let's try to perform a HEAD request
## and see the corresponding headers
try:
response = open_url(url, head_request=True)
except (InvenioBibdocfileUnauthorizedURL, urllib2.URLError):
return ".bin"
ext = get_format_from_http_response(response)
if ext:
return ext
if CFG_HAS_MAGIC:
## Last solution: let's download the remote resource
## and use the Python magic library to guess the extension
filename = ""
try:
try:
filename = download_url(url, docformat='')
ext = guess_via_magic(filename)
if ext:
return ext
except Exception:
pass
finally:
if os.path.exists(filename):
## Let's free space
os.remove(filename)
return ".bin"
_docname_re = re.compile(r'[^-\w.]*')
def normalize_docname(docname):
"""
Normalize the docname.
At the moment the normalization is just returning the same string.
@param docname: the docname to be normalized.
@type docname: string
@return: the normalized docname.
@rtype: string
"""
#return _docname_re.sub('', docname)
return docname
def normalize_version(version):
"""
Normalize the version.
The version can be either an integer or the keyword 'all'. Any other
value will be transformed into the empty string.
@param version: the version (either a number or 'all').
@type version: integer or string
@return: the normalized version.
@rtype: string
"""
try:
int(version)
except ValueError:
if version.lower().strip() == 'all':
return 'all'
else:
return ''
return str(version)
def compose_file(dirname, extension, subformat=None, version=None, storagename=None):
"""
Construct back a fullpath given the separate components.
@param
@param storagename Name under which the file should be stored in the filesystem
@type storagename string
@return a fullpath to the file
@rtype string
"""
if version:
version = ";%i" % int(version)
else:
version = ""
if subformat:
if not subformat.startswith(";"):
subformat = ";%s" % subformat
else:
subformat = ""
if extension and not extension.startswith("."):
extension = ".%s" % extension
if not storagename:
storagename = "content"
return os.path.join(dirname, storagename + extension + subformat + version)
def compose_format(extension, subformat=None):
"""
Construct the format string
"""
if not extension.startswith("."):
extension = ".%s" % extension
if subformat:
if not subformat.startswith(";"):
subformat = ";%s" % subformat
else:
subformat = ""
return extension + subformat
def decompose_file(afile, skip_version=False, only_known_extensions=False,
allow_subformat=True):
"""
Decompose a file/path into its components dirname, basename and extension.
>>> decompose_file('/tmp/foo.tar.gz')
('/tmp', 'foo', '.tar.gz')
>>> decompose_file('/tmp/foo.tar.gz;1', skip_version=True)
('/tmp', 'foo', '.tar.gz')
>>> decompose_file('http://www.google.com/index.html')
('http://www.google.com', 'index', '.html')
@param afile: the path/name of a file.
@type afile: string
@param skip_version: whether to skip a trailing ";version".
@type skip_version: bool
@param only_known_extensions: whether to strip out only known extensions or
to consider as extension anything that follows a dot.
@type only_known_extensions: bool
@param allow_subformat: whether to consider also subformats as part of
the extension.
@type allow_subformat: bool
@return: a tuple with the directory name, the basename and extension.
@rtype: (dirname, basename, extension)
@note: if a URL is provided, the scheme will be part of the dirname.
@see: L{file_strip_ext} for the algorithm used to retrieve the extension.
"""
if skip_version:
version = afile.split(';')[-1]
try:
int(version)
afile = afile[:-len(version)-1]
except ValueError:
pass
basename = os.path.basename(afile)
dirname = afile[:-len(basename)-1]
base = file_strip_ext(
basename,
only_known_extensions=only_known_extensions,
allow_subformat=allow_subformat)
extension = basename[len(base) + 1:]
if extension:
extension = '.' + extension
return (dirname, base, extension)
def decompose_file_with_version(afile):
"""
Decompose a file into dirname, basename, extension and version.
>>> decompose_file_with_version('/tmp/foo.tar.gz;1')
('/tmp', 'foo', '.tar.gz', 1)
@param afile: the path/name of a file.
@type afile: string
@return: a tuple with the directory name, the basename, extension and
version.
@rtype: (dirname, basename, extension, version)
@raise ValueError: in case version does not exist it will.
@note: if a URL is provided, the scheme will be part of the dirname.
"""
version_str = afile.split(';')[-1]
version = int(version_str)
afile = afile[:-len(version_str)-1]
basename = os.path.basename(afile)
dirname = afile[:-len(basename)-1]
base = file_strip_ext(basename)
extension = basename[len(base) + 1:]
if extension:
extension = '.' + extension
return (dirname, base, extension, version)
def get_subformat_from_format(docformat):
"""
@return the subformat if any.
@rtype: string
>>> get_subformat_from_format('foo;bar')
'bar'
>>> get_subformat_from_format('foo')
''
"""
try:
return docformat[docformat.rindex(';') + 1:]
except ValueError:
return ''
def get_superformat_from_format(docformat):
"""
@return the superformat if any.
@rtype: string
>>> get_superformat_from_format('foo;bar')
'foo'
>>> get_superformat_from_format('foo')
'foo'
"""
try:
return docformat[:docformat.rindex(';')]
except ValueError:
return docformat
def propose_next_docname(docname):
"""
Given a I{docname}, suggest a new I{docname} (useful when trying to generate
a unique I{docname}).
>>> propose_next_docname('foo')
'foo_1'
>>> propose_next_docname('foo_1')
'foo_2'
>>> propose_next_docname('foo_10')
'foo_11'
@param docname: the base docname.
@type docname: string
@return: the next possible docname based on the given one.
@rtype: string
"""
if '_' in docname:
split_docname = docname.split('_')
try:
split_docname[-1] = str(int(split_docname[-1]) + 1)
docname = '_'.join(split_docname)
except ValueError:
docname += '_1'
else:
docname += '_1'
return docname
class BibRecDocs(object):
"""
This class represents all the files attached to one record.
@param recid: the record identifier.
@type recid: integer
@param deleted_too: whether to consider deleted documents as normal
documents (useful when trying to recover deleted information).
@type deleted_too: bool
@param human_readable: whether numbers should be printed in human readable
format (e.g. 2048 bytes -> 2Kb)
@ivar id: the record identifier as passed to the constructor.
@type id: integer
@ivar human_readable: the human_readable flag as passed to the constructor.
@type human_readable: bool
@ivar deleted_too: the deleted_too flag as passed to the constructor.
@type deleted_too: bool
@ivar bibdocs: the list of documents attached to the record.
@type bibdocs: list of BibDoc
"""
def __init__(self, recid, deleted_too=False, human_readable=False):
try:
self.id = int(recid)
except ValueError:
raise ValueError("BibRecDocs: recid is %s but must be an integer." % repr(recid))
self.human_readable = human_readable
self.deleted_too = deleted_too
self.attachment_types = {} # dictionary docname->attachment type
self._bibdocs = []
self.dirty = True
@property
def bibdocs(self):
if self.dirty:
self.build_bibdoc_list()
return self._bibdocs
def __repr__(self):
"""
@return: the canonical string representation of the C{BibRecDocs}.
@rtype: string
"""
return 'BibRecDocs(%s%s%s)' % (self.id,
self.deleted_too and ', True' or '',
self.human_readable and ', True' or ''
)
def __str__(self):
"""
@return: an easy to be I{grepped} string representation of the
whole C{BibRecDocs} content.
@rtype: string
"""
out = '%i::::total bibdocs attached=%i\n' % (self.id, len(self.bibdocs))
out += '%i::::total size latest version=%s\n' % (self.id, nice_size(self.get_total_size_latest_version()))
out += '%i::::total size all files=%s\n' % (self.id, nice_size(self.get_total_size()))
for (docname, (bibdoc, dummy)) in self.bibdocs.items():
out += str(docname) + ":" + str(bibdoc)
return out
def empty_p(self):
"""
@return: True when the record has no attached documents.
@rtype: bool
"""
return len(self.bibdocs) == 0
def deleted_p(self):
"""
@return: True if the correxsponding record has been deleted.
@rtype: bool
"""
from invenio.legacy.search_engine import record_exists
return record_exists(self.id) == -1
def get_xml_8564(self):
"""
Return a snippet of I{MARCXML} representing the I{8564} fields
corresponding to the current state.
@return: the MARCXML representation.
@rtype: string
"""
from invenio.legacy.search_engine import get_record
out = ''
record = get_record(self.id)
fields = record_get_field_instances(record, '856', '4', ' ')
for field in fields:
urls = field_get_subfield_values(field, 'u')
if urls and not bibdocfile_url_p(urls[0]):
out += '\t<datafield tag="856" ind1="4" ind2=" ">\n'
for subfield, value in field_get_subfield_instances(field):
out += '\t\t<subfield code="%s">%s</subfield>\n' % (subfield, encode_for_xml(value))
out += '\t</datafield>\n'
for afile in self.list_latest_files(list_hidden=False):
out += '\t<datafield tag="856" ind1="4" ind2=" ">\n'
url = afile.get_url()
description = afile.get_description()
comment = afile.get_comment()
if url:
out += '\t\t<subfield code="u">%s</subfield>\n' % encode_for_xml(url)
if description:
out += '\t\t<subfield code="y">%s</subfield>\n' % encode_for_xml(description)
if comment:
out += '\t\t<subfield code="z">%s</subfield>\n' % encode_for_xml(comment)
out += '\t</datafield>\n'
return out
def get_total_size_latest_version(self):
"""
Returns the total size used on disk by all the files belonging
to this record and corresponding to the latest version.
@return: the total size.
@rtype: integer
"""
size = 0
for (bibdoc, _) in self.bibdocs.values():
size += bibdoc.get_total_size_latest_version()
return size
def get_total_size(self):
"""
Return the total size used on disk of all the files belonging
to this record of any version (not only the last as in
L{get_total_size_latest_version}).
@return: the total size.
@rtype: integer
"""
size = 0
for (bibdoc, _) in self.bibdocs.values():
size += bibdoc.get_total_size()
return size
def build_bibdoc_list(self):
"""
This method must be called everytime a I{bibdoc} is added, removed or
modified.
"""
self._bibdocs = {}
if self.deleted_too:
res = run_sql("""SELECT brbd.id_bibdoc, brbd.docname, brbd.type FROM bibrec_bibdoc as brbd JOIN
bibdoc as bd ON bd.id=brbd.id_bibdoc WHERE brbd.id_bibrec=%s
ORDER BY brbd.docname ASC""", (self.id,))
else:
res = run_sql("""SELECT brbd.id_bibdoc, brbd.docname, brbd.type FROM bibrec_bibdoc as brbd JOIN
bibdoc as bd ON bd.id=brbd.id_bibdoc WHERE brbd.id_bibrec=%s AND
bd.status<>'DELETED' ORDER BY brbd.docname ASC""", (self.id,))
for row in res:
cur_doc = BibDoc.create_instance(docid=row[0], recid=self.id,
human_readable=self.human_readable)
self._bibdocs[row[1]] = (cur_doc, row[2])
self.dirty = False
def list_bibdocs_by_names(self, doctype=None):
"""
Returns the dictionary of all bibdocs object belonging to a recid.
Keys in the dictionary are names of documetns and values are BibDoc objects.
If C{doctype} is set, it returns just the bibdocs of that doctype.
@param doctype: the optional doctype.
@type doctype: string
@return: the dictionary of bibdocs.
@rtype: dictionary of Dcname -> BibDoc
"""
if not doctype:
return dict((k, v) for (k, (v, _)) in iteritems(self.bibdocs))
res = {}
for (docname, (doc, attachmenttype)) in iteritems(self.bibdocs):
if attachmenttype == doctype:
res[docname] = doc
return res
def list_bibdocs(self, doctype=None, rel_type=None):
"""
Returns the list all bibdocs object belonging to a recid.
If C{doctype} is set, it returns just the bibdocs of that doctype.
@param doctype: the optional doctype.
@type doctype: string
@return: the list of bibdocs.
@rtype: list of BibDoc
"""
return [bibdoc for (bibdoc, rtype) in self.bibdocs.values()
if (not doctype or doctype == bibdoc.doctype) and
(rel_type is None or rel_type == rtype)]
def get_bibdoc_names(self, doctype=None):
"""
Returns all the names of the documents associated with the bibrec.
If C{doctype} is set, restrict the result to all the matching doctype.
@param doctype: the optional doctype.
@type doctype: string
@return: the list of document names.
@rtype: list of string
"""
return [docname for (docname, dummy) in self.list_bibdocs_by_names(doctype).items()]
def check_file_exists(self, path, f_format):
"""
Check if a file with the same content of the file pointed in C{path}
is already attached to this record.
@param path: the file to be checked against.
@type path: string
@return: True if a file with the requested content is already attached
to the record.
@rtype: bool
"""
size = os.path.getsize(path)
# Let's consider all the latest files
files = self.list_latest_files()
# Let's consider all the latest files with same size
potential = [afile for afile in files if afile.get_size() == size and afile.format == f_format]
if potential:
checksum = calculate_md5(path)
# Let's consider all the latest files with the same size and the
# same checksum
potential = [afile for afile in potential if afile.get_checksum() == checksum]
if potential:
potential = [afile for afile in potential if
filecmp.cmp(afile.get_full_path(), path)]
if potential:
return True
else:
# Gosh! How unlucky, same size, same checksum but not same
# content!
pass
return False
def propose_unique_docname(self, docname):
"""
Given C{docname}, return a new docname that is not already attached to
the record.
@param docname: the reference docname.
@type docname: string
@return: a docname not already attached.
@rtype: string
"""
docname = normalize_docname(docname)
goodname = docname
i = 1
while goodname in self.get_bibdoc_names():
i += 1
goodname = "%s_%s" % (docname, i)
return goodname
def merge_bibdocs(self, docname1, docname2):
"""
This method merge C{docname2} into C{docname1}.
1. Given all the formats of the latest version of the files
attached to C{docname2}, these files are added as new formats
into C{docname1}.
2. C{docname2} is marked as deleted.
@raise InvenioBibDocFileError: if at least one format in C{docname2}
already exists in C{docname1}. (In this case the two bibdocs are
preserved)
@note: comments and descriptions are also copied.
@note: if C{docname2} has a I{restriction}(i.e. if the I{status} is
set) and C{docname1} doesn't, the restriction is imported.
"""
bibdoc1 = self.get_bibdoc(docname1)
bibdoc2 = self.get_bibdoc(docname2)
## Check for possibility
for bibdocfile in bibdoc2.list_latest_files():
docformat = bibdocfile.get_format()
if bibdoc1.format_already_exists_p(docformat):
raise InvenioBibDocFileError('Format %s already exists in bibdoc %s of record %s. It\'s impossible to merge bibdoc %s into it.' % (docformat, docname1, self.id, docname2))
## Importing restriction if needed.
restriction1 = bibdoc1.get_status()
restriction2 = bibdoc2.get_status()
if restriction2 and not restriction1:
bibdoc1.set_status(restriction2)
## Importing formats
for bibdocfile in bibdoc2.list_latest_files():
docformat = bibdocfile.get_format()
comment = bibdocfile.get_comment()
description = bibdocfile.get_description()
bibdoc1.add_file_new_format(bibdocfile.get_full_path(),
description=description,
comment=comment, docformat=docformat)
## Finally deleting old bibdoc2
bibdoc2.delete()
self.dirty = True
def get_docid(self, docname):
"""
@param docname: the document name.
@type docname: string
@return: the identifier corresponding to the given C{docname}.
@rtype: integer
@raise InvenioBibDocFileError: if the C{docname} does not
corresponds to a document attached to this record.
"""
if docname in self.bibdocs:
return self.bibdocs[docname][0].id
raise InvenioBibDocFileError, "Recid '%s' is not connected with a " \
"docname '%s'" % (self.id, docname)
def get_docname(self, docid):
"""
@param docid: the document identifier.
@type docid: integer
@return: the name of the document corresponding to the given document
identifier.
@rtype: string
@raise InvenioBibDocFileError: if the C{docid} does not
corresponds to a document attached to this record.
"""
for (docname, (bibdoc, _)) in self.bibdocs.items():
if bibdoc.id == docid:
return docname
raise InvenioBibDocFileError, "Recid '%s' is not connected with a " \
"docid '%s'" % (self.id, docid)
def change_name(self, newname, oldname=None, docid=None):
"""
Renames document of a given name.
@param newname: the new name.
@type newname: string
@raise InvenioBibDocFileError: if the new name corresponds to
a document already attached to the record owning this document.
"""
if not oldname and not docid:
raise StandardError("Trying to rename unspecified document")
if not oldname:
oldname = self.get_docname(docid)
if not docid:
docid = self.get_docid(oldname)
doc, atttype = self.bibdocs[oldname]
newname = normalize_docname(newname)
res = run_sql("SELECT id_bibdoc FROM bibrec_bibdoc WHERE id_bibrec=%s AND docname=%s", (self.id, newname))
if res:
raise InvenioBibDocFileError, "A bibdoc called %s already exists for recid %s" % (newname, self.id)
doc.change_name(self.id, newname)
# updating the record structure
del self._bibdocs[oldname]
self._bibdocs[newname] = (doc, atttype)
def has_docname_p(self, docname):
"""
@param docname: the document name,
@type docname: string
@return: True if a document with the given name is attached to this
record.
@rtype: bool
"""
return docname in self.bibdocs.keys()
def get_bibdoc(self, docname):
"""
@return: the bibdoc with a particular docname associated with
this recid"""
if docname in self.bibdocs:
return self.bibdocs[docname][0]
raise InvenioBibDocFileError, "Recid '%s' is not connected with " \
" docname '%s'" % (self.id, docname)
def delete_bibdoc(self, docname):
"""
Deletes the document with the specified I{docname}.
@param docname: the document name.
@type docname: string
"""
if docname in self.bibdocs:
self.bibdocs[docname][0].delete()
self.dirty = True
def add_bibdoc(self, doctype="Main", docname='file', never_fail=False):
"""
Add a new empty document object (a I{bibdoc}) to the list of
documents of this record.
@param doctype: the document type.
@type doctype: string
@param docname: the document name.
@type docname: string
@param never_fail: if True, this procedure will not fail, even if
a document with the given name is already attached to this
record. In this case a new name will be generated (see
L{propose_unique_docname}).
@type never_fail: bool
@return: the newly created document object.
@rtype: BibDoc
@raise InvenioBibDocFileError: in case of any error.
"""
try:
docname = normalize_docname(docname)
if never_fail:
docname = self.propose_unique_docname(docname)
if docname in self.get_bibdoc_names():
raise InvenioBibDocFileError, \
"%s has already a bibdoc with docname %s" % (self.id, docname)
else:
bibdoc = BibDoc.create_instance(recid=self.id, doctype=doctype,
docname=docname,
human_readable=self.human_readable)
self.dirty = True
return bibdoc
except Exception as e:
register_exception()
raise InvenioBibDocFileError(str(e))
def add_new_file(self, fullpath, doctype="Main", docname=None,
never_fail=False, description=None, comment=None,
docformat=None, flags=None, modification_date=None):
"""
Directly add a new file to this record.
Adds a new file with the following policy:
- if the C{docname} is not set it is retrieved from the name of the
file.
- If a bibdoc with the given docname doesn't already exist, it is
created and the file is added to it.
- It it exist but it doesn't contain the format that is being
added, the new format is added.
- If the format already exists then if C{never_fail} is True a new
bibdoc is created with a similar name but with a progressive
number as a suffix and the file is added to it (see
L{propose_unique_docname}).
@param fullpath: the filesystme path of the document to be added.
@type fullpath: string
@param doctype: the type of the document.
@type doctype: string
@param docname: the document name.
@type docname: string
@param never_fail: if True, this procedure will not fail, even if
a document with the given name is already attached to this
record. In this case a new name will be generated (see
L{propose_unique_docname}).
@type never_fail: bool
@param description: an optional description of the file.
@type description: string
@param comment: an optional comment to the file.
@type comment: string
@param format: the extension of the file. If not specified it will
be guessed (see L{guess_format_from_url}).
@type format: string
@param flags: a set of flags to be associated with the file (see
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS})
@type flags: list of string
@return: the elaborated document object.
@rtype: BibDoc
@raise InvenioBibDocFileError: in case of error.
"""
if docname is None:
docname = decompose_file(fullpath)[1]
if docformat is None:
docformat = decompose_file(fullpath)[2]
docname = normalize_docname(docname)
try:
bibdoc = self.get_bibdoc(docname)
except InvenioBibDocFileError:
# bibdoc doesn't already exists!
bibdoc = self.add_bibdoc(doctype, docname, False)
bibdoc.add_file_new_version(fullpath, description=description, comment=comment, docformat=docformat, flags=flags, modification_date=modification_date)
else:
try:
bibdoc.add_file_new_format(fullpath, description=description, comment=comment, docformat=docformat, flags=flags, modification_date=modification_date)
except InvenioBibDocFileError as dummy:
# Format already exist!
if never_fail:
bibdoc = self.add_bibdoc(doctype, docname, True)
bibdoc.add_file_new_version(fullpath, description=description, comment=comment, docformat=docformat, flags=flags, modification_date=modification_date)
else:
raise
return bibdoc
def add_new_version(self, fullpath, docname=None, description=None, comment=None, docformat=None, flags=None):
"""
Adds a new file to an already existent document object as a new
version.
@param fullpath: the filesystem path of the file to be added.
@type fullpath: string
@param docname: the document name. If not specified it will be
extracted from C{fullpath} (see L{decompose_file}).
@type docname: string
@param description: an optional description for the file.
@type description: string
@param comment: an optional comment to the file.
@type comment: string
@param format: the extension of the file. If not specified it will
be guessed (see L{guess_format_from_url}).
@type format: string
@param flags: a set of flags to be associated with the file (see
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS})
@type flags: list of string
@return: the elaborated document object.
@rtype: BibDoc
@raise InvenioBibDocFileError: in case of error.
@note: previous files associated with the same document will be
considered obsolete.
"""
if docname is None:
docname = decompose_file(fullpath)[1]
if docformat is None:
docformat = decompose_file(fullpath)[2]
if flags is None:
flags = []
if 'pdfa' in get_subformat_from_format(docformat).split(';') and not 'PDF/A' in flags:
flags.append('PDF/A')
bibdoc = self.get_bibdoc(docname=docname)
bibdoc.add_file_new_version(fullpath, description=description, comment=comment, docformat=docformat, flags=flags)
return bibdoc
def add_new_format(self, fullpath, docname=None, description=None, comment=None, docformat=None, flags=None, modification_date=None):
"""
Adds a new file to an already existent document object as a new
format.
@param fullpath: the filesystem path of the file to be added.
@type fullpath: string
@param docname: the document name. If not specified it will be
extracted from C{fullpath} (see L{decompose_file}).
@type docname: string
@param description: an optional description for the file.
@type description: string
@param comment: an optional comment to the file.
@type comment: string
@param format: the extension of the file. If not specified it will
be guessed (see L{guess_format_from_url}).
@type format: string
@param flags: a set of flags to be associated with the file (see
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS})
@type flags: list of string
@return: the elaborated document object.
@rtype: BibDoc
@raise InvenioBibDocFileError: in case the same format already
exists.
"""
if docname is None:
docname = decompose_file(fullpath)[1]
if docformat is None:
docformat = decompose_file(fullpath)[2]
if flags is None:
flags = []
if 'pdfa' in get_subformat_from_format(docformat).split(';') and not 'PDF/A' in flags:
flags.append('PDF/A')
bibdoc = self.get_bibdoc(docname=docname)
bibdoc.add_file_new_format(fullpath, description=description, comment=comment, docformat=docformat, flags=flags, modification_date=modification_date)
return bibdoc
def list_latest_files(self, doctype=None, list_hidden=True):
"""
Returns a list of the latest files.
@param doctype: if set, only document of the given type will be listed.
@type doctype: string
@param list_hidden: if True, will list also files with the C{HIDDEN}
flag being set.
@type list_hidden: bool
@return: the list of latest files.
@rtype: list of BibDocFile
"""
docfiles = []
for bibdoc in self.list_bibdocs(doctype):
docfiles += bibdoc.list_latest_files(list_hidden=list_hidden)
return docfiles
def fix(self, docname):
"""
Algorithm that transform a broken/old bibdoc into a coherent one.
Think of it as being the fsck of BibDocs.
- All the files in the bibdoc directory will be renamed according
to the document name. Proper .recid, .type, .md5 files will be
created/updated.
- In case of more than one file with the same format version a new
bibdoc will be created in order to put does files.
@param docname: the document name that need to be fixed.
@type docname: string
@return: the list of newly created bibdocs if any.
@rtype: list of BibDoc
@raise InvenioBibDocFileError: in case of issues that can not be
fixed automatically.
"""
bibdoc = self.get_bibdoc(docname)
versions = {}
res = []
new_bibdocs = [] # List of files with the same version/format of
# existing file which need new bibdoc.
counter = 0
zero_version_bug = False
if os.path.exists(bibdoc.basedir):
from invenio.config import CFG_CERN_SITE, CFG_INSPIRE_SITE, CFG_BIBDOCFILE_AFS_VOLUME_PATTERN, CFG_BIBDOCFILE_AFS_VOLUME_QUOTA
if os.path.realpath(bibdoc.basedir).startswith('/afs') and (CFG_CERN_SITE or CFG_INSPIRE_SITE):
## We are on AFS at CERN! Let's allocate directories the CERN/AFS way. E.g.
## $ afs_admin create -q 1000000 /afs/cern.ch/project/cds/files/g40 p.cds.g40
## NOTE: This might be extended to use low-level OpenAFS CLI tools
## so that this technique could be extended to other AFS users outside CERN.
mount_point = os.path.dirname(os.path.realpath(bibdoc.basedir))
if not os.path.exists(mount_point):
volume = CFG_BIBDOCFILE_AFS_VOLUME_PATTERN % os.path.basename(mount_point)
quota = str(CFG_BIBDOCFILE_AFS_VOLUME_QUOTA)
exit_code, stdout, stderr = run_shell_command("afs_admin create -q %s %s %s", (quota, mount_point, volume))
if exit_code or stderr:
raise IOError("Error in creating AFS mount point %s with quota %s and volume %s: exit_code=%s. Captured stdout:\n: %s\nCaptured stderr:\n: %s" % (mount_point, quota, volume, exit_code, stdout, stderr))
for filename in os.listdir(bibdoc.basedir):
if filename[0] != '.' and ';' in filename:
- name, version = filename.split(';')
+ name, version = filename.rsplit(';', 1)
try:
version = int(version)
except ValueError:
# Strange name
register_exception()
raise InvenioBibDocFileError, "A file called %s exists under %s. This is not a valid name. After the ';' there must be an integer representing the file version. Please, manually fix this file either by renaming or by deleting it." % (filename, bibdoc.basedir)
if version == 0:
zero_version_bug = True
docformat = name[len(file_strip_ext(name)):]
docformat = normalize_format(docformat)
if version not in versions:
versions[version] = {}
new_name = 'FIXING-%s-%s' % (str(counter), name)
try:
shutil.move('%s/%s' % (bibdoc.basedir, filename), '%s/%s' % (bibdoc.basedir, new_name))
except Exception as e:
register_exception()
raise InvenioBibDocFileError, "Error in renaming '%s' to '%s': '%s'" % ('%s/%s' % (bibdoc.basedir, filename), '%s/%s' % (bibdoc.basedir, new_name), e)
if docformat in versions[version]:
new_bibdocs.append((new_name, version))
else:
versions[version][docformat] = new_name
counter += 1
elif filename[0] != '.':
# Strange name
register_exception()
raise InvenioBibDocFileError, "A file called %s exists under %s. This is not a valid name. There should be a ';' followed by an integer representing the file version. Please, manually fix this file either by renaming or by deleting it." % (filename, bibdoc.basedir)
else:
# we create the corresponding storage directory
old_umask = os.umask(0o022)
os.makedirs(bibdoc.basedir)
# and save the father record id if it exists
try:
if self.id != "":
recid_fd = open("%s/.recid" % bibdoc.basedir, "w")
recid_fd.write(str(self.id))
recid_fd.close()
if bibdoc.doctype != "":
type_fd = open("%s/.type" % bibdoc.basedir, "w")
type_fd.write(str(bibdoc.doctype))
type_fd.close()
except Exception as e:
register_exception()
raise InvenioBibDocFileError, e
os.umask(old_umask)
if not versions:
bibdoc.delete()
self.dirty = True
else:
for version, formats in iteritems(versions):
if zero_version_bug:
version += 1
for docformat, filename in iteritems(formats):
destination = '%s%s;%i' % (docname, docformat, version)
try:
shutil.move('%s/%s' % (bibdoc.basedir, filename), '%s/%s' % (bibdoc.basedir, destination))
except Exception as e:
register_exception()
raise InvenioBibDocFileError, "Error in renaming '%s' to '%s': '%s'" % ('%s/%s' % (bibdoc.basedir, filename), '%s/%s' % (bibdoc.basedir, destination), e)
try:
recid_fd = open("%s/.recid" % bibdoc.basedir, "w")
recid_fd.write(str(self.id))
recid_fd.close()
type_fd = open("%s/.type" % bibdoc.basedir, "w")
type_fd.write(str(bibdoc.doctype))
type_fd.close()
except Exception as e:
register_exception()
raise InvenioBibDocFileError, "Error in creating .recid and .type file for '%s' folder: '%s'" % (bibdoc.basedir, e)
res = []
for (filename, version) in new_bibdocs:
if zero_version_bug:
version += 1
new_bibdoc = self.add_bibdoc(doctype=bibdoc.doctype, docname=docname, never_fail=True)
new_bibdoc.add_file_new_format('%s/%s' % (bibdoc.basedir, filename), version)
res.append(new_bibdoc)
try:
os.remove('%s/%s' % (bibdoc.basedir, filename))
except Exception as e:
register_exception()
raise InvenioBibDocFileError, "Error in removing '%s': '%s'" % ('%s/%s' % (bibdoc.basedir, filename), e)
Md5Folder(bibdoc.basedir).update(only_new=False)
bibdoc._build_file_list()
for (bibdoc, dummyatttype) in self.bibdocs.values():
if not run_sql('SELECT data_value FROM bibdocmoreinfo WHERE id_bibdoc=%s', (bibdoc.id,)):
## Import from MARC only if the bibdoc has never had
## its more_info initialized.
try:
bibdoc.import_descriptions_and_comments_from_marc()
except Exception as e:
register_exception()
raise InvenioBibDocFileError, "Error in importing description and comment from %s for record %s: %s" % (repr(bibdoc), self.id, e)
return res
def check_format(self, docname):
"""
Check for any format related issue.
In case L{CFG_BIBDOCFILE_ADDITIONAL_KNOWN_FILE_EXTENSIONS} is
altered or Python version changes, it might happen that a docname
contains files which are no more docname + .format ; version, simply
because the .format is now recognized (and it was not before, so
it was contained into the docname).
This algorithm verify if it is necessary to fix (seel L{fix_format}).
@param docname: the document name whose formats should be verified.
@type docname: string
@return: True if format is correct. False if a fix is needed.
@rtype: bool
@raise InvenioBibDocFileError: in case of any error.
"""
bibdoc = self.get_bibdoc(docname)
correct_docname = decompose_file(docname + '.pdf')[1]
if docname != correct_docname:
return False
for filename in os.listdir(bibdoc.basedir):
if not filename.startswith('.'):
try:
dummy, dummy, docformat, version = decompose_file_with_version(filename)
except Exception:
raise InvenioBibDocFileError('Incorrect filename "%s" for docname %s for recid %i' % (filename, docname, self.id))
if '%s%s;%i' % (correct_docname, docformat, version) != filename:
return False
return True
def check_duplicate_docnames(self):
"""
Check wethever the record is connected with at least tho documents
with the same name.
@return: True if everything is fine.
@rtype: bool
"""
docnames = set()
for docname in self.get_bibdoc_names():
if docname in docnames:
return False
else:
docnames.add(docname)
return True
def uniformize_bibdoc(self, docname):
"""
This algorithm correct wrong file name belonging to a bibdoc.
@param docname: the document name whose formats should be verified.
@type docname: string
"""
bibdoc = self.get_bibdoc(docname)
for filename in os.listdir(bibdoc.basedir):
if not filename.startswith('.'):
try:
dummy, dummy, docformat, version = decompose_file_with_version(filename)
except ValueError:
register_exception(alert_admin=True, prefix= "Strange file '%s' is stored in %s" % (filename, bibdoc.basedir))
else:
os.rename(os.path.join(bibdoc.basedir, filename), os.path.join(bibdoc.basedir, '%s%s;%i' % (docname, docformat, version)))
Md5Folder(bibdoc.basedir).update()
bibdoc.touch('rename')
def fix_format(self, docname, skip_check=False):
"""
Fixes format related inconsistencies.
@param docname: the document name whose formats should be verified.
@type docname: string
@param skip_check: if True assume L{check_format} has already been
called and the need for fix has already been found.
If False, will implicitly call L{check_format} and skip fixing
if no error is found.
@type skip_check: bool
@return: in case merging two bibdocs is needed but it's not possible.
@rtype: bool
"""
if not skip_check:
if self.check_format(docname):
return True
bibdoc = self.get_bibdoc(docname)
correct_docname = decompose_file(docname + '.pdf')[1]
need_merge = False
if correct_docname != docname:
need_merge = self.has_docname_p(correct_docname)
if need_merge:
proposed_docname = self.propose_unique_docname(correct_docname)
run_sql('UPDATE bibdoc SET docname=%s WHERE id=%s', (proposed_docname, bibdoc.id))
self.dirty = True
self.uniformize_bibdoc(proposed_docname)
try:
self.merge_bibdocs(docname, proposed_docname)
except InvenioBibDocFileError:
return False
else:
run_sql('UPDATE bibdoc SET docname=%s WHERE id=%s', (correct_docname, bibdoc.id))
self.dirty = True
self.uniformize_bibdoc(correct_docname)
else:
self.uniformize_bibdoc(docname)
return True
def fix_duplicate_docnames(self, skip_check=False):
"""
Algotirthm to fix duplicate docnames.
If a record is connected with at least two bibdoc having the same
docname, the algorithm will try to merge them.
@param skip_check: if True assume L{check_duplicate_docnames} has
already been called and the need for fix has already been found.
If False, will implicitly call L{check_duplicate_docnames} and skip
fixing if no error is found.
@type skip_check: bool
"""
if not skip_check:
if self.check_duplicate_docnames():
return
docnames = set()
for bibdoc in self.list_bibdocs():
docname = self.get_docname(bibdoc.id)
if docname in docnames:
new_docname = self.propose_unique_docname(self.get_docname(bibdoc.id))
self.change_name(docid=bibdoc.id, newname=new_docname)
self.merge_bibdocs(docname, new_docname)
docnames.add(docname)
def get_text(self, extract_text_if_necessary=True):
"""
@return: concatenated texts of all bibdocs separated by " ": string
"""
texts = []
for bibdoc in self.list_bibdocs():
if hasattr(bibdoc, 'has_text'):
if extract_text_if_necessary and not bibdoc.has_text(require_up_to_date=True):
- re_perform_ocr = re.compile(CFG_BIBINDEX_PERFORM_OCR_ON_DOCNAMES)
- doc_name = bibdoc.get_docname() or ""
-
- perform_ocr = bool(re_perform_ocr.match(doc_name))
- from invenio.legacy.bibsched.bibtask import write_message
- write_message("... will extract words from %s (docid: %s) %s" % (doc_name, bibdoc.get_id(), perform_ocr and 'with OCR' or ''), verbose=2)
+ perform_ocr = hasattr(bibdoc, 'is_ocr_required') and bibdoc.is_ocr_required()
+ from invenio.bibtask import write_message
+ write_message("... will extract words from %s %s" % (bibdoc, perform_ocr and 'with OCR' or ''), verbose=2)
bibdoc.extract_text(perform_ocr=perform_ocr)
texts.append(bibdoc.get_text())
return " ".join(texts)
class BibDoc(object):
"""
This class represents one document (i.e. a set of files with different
formats and with versioning information that consitutes a piece of
information.
To instanciate a new document, the recid and the docname are mandatory.
To instanciate an already existing document, either the recid and docname
or the docid alone are sufficient to retrieve it.
@param docid: the document identifier.
@type docid: integer
@param recid: the record identifier of the record to which this document
belongs to. If the C{docid} is specified the C{recid} is automatically
retrieven from the database.
@type recid: integer
@param docname: the document name.
@type docname: string
@param doctype: the document type (used when instanciating a new document).
@type doctype: string
@param human_readable: whether sizes should be represented in a human
readable format.
@type human_readable: bool
@raise InvenioBibDocFileError: in case of error.
"""
@staticmethod
def create_new_document(doc_type="Main", rec_links=None):
if rec_links is None:
rec_links = []
status = ''
doc_id = run_sql("INSERT INTO bibdoc (status, creation_date, modification_date, doctype) "
"values(%s,NOW(),NOW(), %s)", (status, doc_type))
if not doc_id:
raise InvenioBibDocFileError, "New docid cannot be created"
# creating the representation on disk ... preparing the directory
try:
BibDoc.prepare_basedir(doc_id)
except Exception as e:
run_sql('DELETE FROM bibdoc WHERE id=%s', (doc_id, ))
register_exception(alert_admin=True)
raise InvenioBibDocFileError, e
# the object has been created: linking to bibliographical records
doc = BibDoc(doc_id)
for link in rec_links:
if "rec_id" in link and link["rec_id"]:
rec_id = link["rec_id"]
doc_name = normalize_docname(link["doc_name"])
a_type = link["a_type"]
doc.attach_to_record(rec_id, str(a_type), str(doc_name))
return doc_id
def __init__(self, docid, human_readable=False, initial_data=None):
"""Constructor of a bibdoc. At least the docid or the recid/docname
pair is needed.
specifying recid, docname and doctype without specifying docid results in
attaching newly created document to a record
"""
# docid is known, the document already exists
res2 = run_sql("SELECT id_bibrec, type, docname FROM bibrec_bibdoc WHERE id_bibdoc=%s", (docid,))
self.bibrec_types = [(r[0], r[1], r[2]) for r in res2 ] # just in case the result was behaving like tuples but was something else
if not res2:
# fake attachment
self.bibrec_types = [(0, None, "fake_name_for_unattached_document")]
if initial_data is None:
initial_data = BibDoc._retrieve_data(docid)
self._docfiles = []
self.__md5s = None
self._related_files = {}
self.human_readable = human_readable
self.cd = initial_data["cd"] # creation date
self.md = initial_data["md"] # modification date
self.td = initial_data["td"] # text extraction date # should be moved from here !!!!
self.bibrec_links = initial_data["bibrec_links"]
self.id = initial_data["id"]
self.status = initial_data["status"]
self.basedir = initial_data["basedir"]
self.doctype = initial_data["doctype"]
self.storagename = initial_data["storagename"] # the old docname -> now used as a storage name for old records
self.more_info = BibDocMoreInfo(self.id)
self.dirty = True
self.dirty_related_files = True
self.last_action = 'init'
def __del__(self):
if self.dirty and self.last_action != 'init':
## The object is dirty and we did something more than initializing it
self._build_file_list()
@property
def docfiles(self):
if self.dirty:
self._build_file_list(self.last_action)
self.dirty = False
return self._docfiles
@property
def related_files(self):
if self.dirty_related_files:
self._build_related_file_list()
self.dirty_related_files = False
return self._related_files
@staticmethod
def prepare_basedir(doc_id):
"""Prepares the directory serving as root of a BibDoc"""
basedir = _make_base_dir(doc_id)
# we create the corresponding storage directory
if not os.path.exists(basedir):
from invenio.config import CFG_CERN_SITE, CFG_INSPIRE_SITE, CFG_BIBDOCFILE_AFS_VOLUME_PATTERN, CFG_BIBDOCFILE_AFS_VOLUME_QUOTA
if os.path.realpath(basedir).startswith('/afs') and (CFG_CERN_SITE or CFG_INSPIRE_SITE):
## We are on AFS at CERN! Let's allocate directories the CERN/AFS way. E.g.
## $ afs_admin create -q 1000000 /afs/cern.ch/project/cds/files/g40 p.cds.g40
## NOTE: This might be extended to use low-level OpenAFS CLI tools
## so that this technique could be extended to other AFS users outside CERN.
mount_point = os.path.dirname(os.path.realpath(basedir))
if not os.path.exists(mount_point):
volume = CFG_BIBDOCFILE_AFS_VOLUME_PATTERN % os.path.basename(mount_point)
quota = str(CFG_BIBDOCFILE_AFS_VOLUME_QUOTA)
exit_code, stdout, stderr = run_shell_command("afs_admin create -q %s %s %s", (quota, mount_point, volume))
if exit_code or stderr:
raise IOError("Error in creating AFS mount point %s with quota %s and volume %s: exit_code=%s. Captured stdout:\n: %s\nCaptured stderr:\n: %s" % (mount_point, quota, volume, exit_code, stdout, stderr))
old_umask = os.umask(022)
os.makedirs(basedir)
os.umask(old_umask)
def _update_additional_info_files(self):
"""Update the hidden file in the document directory ... the file contains all links to records"""
try:
reclinks_fd = open("%s/.reclinks" % (self.basedir, ), "w")
reclinks_fd.write("RECID DOCNAME TYPE\n")
for link in self.bibrec_links:
reclinks_fd.write("%(recid)s %(docname)s %(doctype)s\n" % link)
reclinks_fd.close()
except Exception as e:
register_exception(alert_admin=True)
raise InvenioBibDocFileError, e
@staticmethod
def _retrieve_data(docid = None):
"""
Filling information about a document from the database entry
"""
container = {}
container["bibrec_links"] = []
container["id"] = docid
container["basedir"] = _make_base_dir(container["id"])
# retrieving links betwen records and documents
res = run_sql("SELECT id_bibrec, type, docname FROM bibrec_bibdoc WHERE id_bibdoc=%s", (str(docid),), 1)
if res:
for r in res:
container["bibrec_links"].append({"recid": r[0], "doctype": r[1], "docname": r[2]})
# gather the other information
res = run_sql("SELECT status, creation_date, modification_date, text_extraction_date, doctype, docname FROM bibdoc WHERE id=%s LIMIT 1", (docid,), 1)
if res:
container["status"] = res[0][0]
container["cd"] = res[0][1]
container["md"] = res[0][2]
container["td"] = res[0][3]
container["doctype"] = res[0][4]
container["storagename"] = res[0][5]
else:
# this bibdoc doesn't exist
raise InvenioBibDocFileError, "The docid %s does not exist." % docid
# retreiving all available formats
fprefix = container["storagename"] or "content"
if CFG_BIBDOCFILE_ENABLE_BIBDOCFSINFO_CACHE:
## We take all extensions from the existing formats in the DB.
container["extensions"] = set([ext[0] for ext in run_sql("SELECT format FROM bibdocfsinfo WHERE id_bibdoc=%s", (docid, ))])
else:
## We take all the extensions by listing the directory content, stripping name
## and version.
container["extensions"] = set([fname[len(fprefix):].rsplit(";", 1)[0] for fname in filter(lambda x: x.startswith(fprefix), os.listdir(container["basedir"]))])
return container
@staticmethod
def create_instance(docid=None, recid=None, docname=None,
doctype='Fulltext', a_type = 'Main', human_readable=False):
"""
Parameters of an attachement to the record:
a_type, recid, docname
@param a_type Type of the attachment to the record (by default Main)
@type a_type String
@param doctype Type of the document itself (by default Fulltext)
@type doctype String
"""
# first try to retrieve existing record based on obtained data
data = None
extensions = []
if docid is not None:
data = BibDoc._retrieve_data(docid)
doctype = data["doctype"]
extensions = data["extensions"]
# Loading an appropriate plugin (by default a generic BibDoc)
used_plugin = None
for plugin in get_plugins():
if plugin['supports'](doctype, extensions):
used_plugin = plugin
if not docid:
rec_links = []
if recid:
rec_links.append({"rec_id": recid, "doc_name" : docname, "a_type": a_type})
if used_plugin and 'create_new' in used_plugin:
docid = used_plugin['create_new'](doctype, rec_links)
else:
docid = BibDoc.create_new_document(doctype, rec_links)
if used_plugin:
return used_plugin['create_instance'](docid=docid,
human_readable=human_readable,
initial_data=data)
return BibDoc(docid=docid,
human_readable=human_readable,
initial_data=data)
def attach_to_record(self, recid, a_type, docname):
""" Attaches given document to a record given by its identifier.
@param recid The identifier of the record
@type recid Integer
@param a_type Function of a document in the record
@type a_type String
@param docname Name of a document inside of a record
@type docname String
"""
run_sql("INSERT INTO bibrec_bibdoc (id_bibrec, id_bibdoc, type, docname) VALUES (%s,%s,%s,%s)",
(str(recid), str(self.id), a_type, docname))
self._update_additional_info_files()
def __repr__(self):
"""
@return: the canonical string representation of the C{BibDoc}.
@rtype: string
"""
return 'BibDoc(%s, %s, %s)' % (repr(self.id), repr(self.doctype), repr(self.human_readable))
def format_recids(self):
"""Returns a string representation of related record ids"""
if len(self.bibrec_links) == 1:
return self.bibrec_links[0]["recid"]
return "[" + ",".join([str(el["recid"]) for el in self.bibrec_links]) + "]"
def __str__(self):
"""
@return: an easy to be I{grepped} string representation of the
whole C{BibDoc} content.
@rtype: string
"""
recids = self.format_recids()
out = '%s:%i:::doctype=%s\n' % (recids, self.id, self.doctype)
out += '%s:%i:::status=%s\n' % (recids, self.id, self.status)
out += '%s:%i:::basedir=%s\n' % (recids, self.id, self.basedir)
out += '%s:%i:::creation date=%s\n' % (recids, self.id, self.cd)
out += '%s:%i:::modification date=%s\n' % (recids, self.id, self.md)
out += '%s:%i:::text extraction date=%s\n' % (recids, self.id, self.td)
out += '%s:%i:::total file attached=%s\n' % (recids, self.id, len(self.docfiles))
if self.human_readable:
out += '%s:%i:::total size latest version=%s\n' % (recids, self.id, nice_size(self.get_total_size_latest_version()))
out += '%s:%i:::total size all files=%s\n' % (recids, self.id, nice_size(self.get_total_size()))
else:
out += '%s:%i:::total size latest version=%s\n' % (recids, self.id, self.get_total_size_latest_version())
out += '%s:%i:::total size all files=%s\n' % (recids, self.id, self.get_total_size())
for docfile in self.docfiles:
out += str(docfile)
return out
def get_md5s(self):
"""
@return: an instance of the Md5Folder class to access MD5 information
of the current BibDoc
@rtype: Md5Folder
"""
if self.__md5s is None:
self.__md5s = Md5Folder(self.basedir)
return self.__md5s
md5s = property(get_md5s)
def format_already_exists_p(self, docformat):
"""
@param format: a format to be checked.
@type format: string
@return: True if a file of the given format already exists among the
latest files.
@rtype: bool
"""
docformat = normalize_format(docformat)
for afile in self.list_latest_files():
if docformat == afile.get_format():
return True
return False
def get_status(self):
"""
@return: the status information.
@rtype: string
"""
return self.status
@staticmethod
def get_fileprefix(basedir, storagename=None):
fname = "%s" % (storagename or "content", )
return os.path.join(basedir, fname )
def get_filepath(self, docformat, version):
""" Generaters the path inside of the filesystem where the document should be stored.
@param format The format of the document
@type format string
@param version version to be stored in the file
@type version string
TODO: this should be completely replaced. File storage (and so, also path building)
should be abstracted from BibDoc and be using loadable extensions
@param format Format of the document to be stored
@type format string
@param version Version of the document to be stored
@type version String
@return Full path to the file encoding a particular version and format of the document
@trype string
"""
return "%s%s;%i" % (BibDoc.get_fileprefix(self.basedir, self.storagename), docformat, version)
def get_docname(self):
"""Obsolete !! (will return empty String for new format documents"""
return self.storagename
def get_doctype(self, recid):
"""Retrieves the type of this document in the scope of a given recid"""
link_types = [attachement["doctype"] for attachement in
self.bibrec_links
if str(attachement["recid"]) == str(recid)]
if link_types:
return link_types[0]
return ""
def touch(self, action=''):
"""
Update the modification time of the bibdoc (as in the UNIX command
C{touch}).
"""
run_sql('UPDATE bibdoc SET modification_date=NOW() WHERE id=%s', (self.id, ))
self.dirty = True
self.last_action = action
def change_doctype(self, new_doctype):
"""
Modify the doctype of a BibDoc
"""
run_sql('UPDATE bibdoc SET doctype=%s WHERE id=%s', (new_doctype, self.id))
run_sql('UPDATE bibrec_bibdoc SET type=%s WHERE id_bibdoc=%s', (new_doctype, self.id))
self.dirty = True
def set_status(self, new_status):
"""
Set a new status. A document with a status information is a restricted
document that can be accessed only to user which as an authorization
to the I{viewrestrdoc} WebAccess action with keyword status with value
C{new_status}.
@param new_status: the new status. If empty the document will be
unrestricted.
@type new_status: string
@raise InvenioBibDocFileError: in case the reserved word
'DELETED' is used.
"""
if new_status != KEEP_OLD_VALUE:
if new_status == 'DELETED':
raise InvenioBibDocFileError('DELETED is a reserved word and can not be used for setting the status')
run_sql('UPDATE bibdoc SET status=%s WHERE id=%s', (new_status, self.id))
self.status = new_status
self.touch('status')
def add_file_new_version(self, filename, description=None, comment=None, docformat=None, flags=None, modification_date=None):
"""
Add a new version of a file. If no physical file is already attached
to the document a the given file will have version 1. Otherwise the
new file will have the current version number plus one.
@param filename: the local path of the file.
@type filename: string
@param description: an optional description for the file.
@type description: string
@param comment: an optional comment to the file.
@type comment: string
@param format: the extension of the file. If not specified it will
be retrieved from the filename (see L{decompose_file}).
@type format: string
@param flags: a set of flags to be associated with the file (see
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS})
@type flags: list of string
@raise InvenioBibDocFileError: in case of error.
"""
latestVersion = self.get_latest_version()
if latestVersion == 0:
myversion = 1
else:
myversion = latestVersion + 1
if os.path.exists(filename):
if not os.path.getsize(filename) > 0:
raise InvenioBibDocFileError, "%s seems to be empty" % filename
if docformat is None:
docformat = decompose_file(filename)[2]
else:
docformat = normalize_format(docformat)
destination = self.get_filepath(docformat, myversion)
if run_sql("SELECT id_bibdoc FROM bibdocfsinfo WHERE id_bibdoc=%s AND version=%s AND format=%s", (self.id, myversion, docformat)):
raise InvenioBibDocFileError("According to the database a file of format %s is already attached to the docid %s" % (docformat, self.id))
try:
shutil.copyfile(filename, destination)
os.chmod(destination, 0644)
if modification_date: # if the modification time of the file needs to be changed
update_modification_date_of_file(destination, modification_date)
except Exception as e:
register_exception()
raise InvenioBibDocFileError("Encountered an exception while copying '%s' to '%s': '%s'" % (filename, destination, e))
self.more_info.set_description(description, docformat, myversion)
self.more_info.set_comment(comment, docformat, myversion)
if flags is None:
flags = []
if 'pdfa' in get_subformat_from_format(docformat).split(';') and not 'PDF/A' in flags:
flags.append('PDF/A')
for flag in flags:
if flag == 'PERFORM_HIDE_PREVIOUS':
for afile in self.list_all_files():
docformat = afile.get_format()
version = afile.get_version()
if version < myversion:
self.more_info.set_flag('HIDDEN', docformat, myversion)
else:
self.more_info.set_flag(flag, docformat, myversion)
else:
raise InvenioBibDocFileError("'%s' does not exists!" % filename)
self.touch('newversion')
Md5Folder(self.basedir).update()
just_added_file = self.get_file(docformat, myversion)
run_sql("INSERT INTO bibdocfsinfo(id_bibdoc, version, format, last_version, cd, md, checksum, filesize, mime) VALUES(%s, %s, %s, true, %s, %s, %s, %s, %s)", (self.id, myversion, docformat, just_added_file.cd, just_added_file.md, just_added_file.get_checksum(), just_added_file.get_size(), just_added_file.mime))
run_sql("UPDATE bibdocfsinfo SET last_version=false WHERE id_bibdoc=%s AND version<%s", (self.id, myversion))
def add_file_new_format(self, filename, version=None, description=None, comment=None, docformat=None, flags=None, modification_date=None):
"""
Add a file as a new format.
@param filename: the local path of the file.
@type filename: string
@param version: an optional specific version to which the new format
should be added. If None, the last version will be used.
@type version: integer
@param description: an optional description for the file.
@type description: string
@param comment: an optional comment to the file.
@type comment: string
@param format: the extension of the file. If not specified it will
be retrieved from the filename (see L{decompose_file}).
@type format: string
@param flags: a set of flags to be associated with the file (see
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS})
@type flags: list of string
@raise InvenioBibDocFileError: if the given format already exists.
"""
if version is None:
version = self.get_latest_version()
if version == 0:
version = 1
if os.path.exists(filename):
if not os.path.getsize(filename) > 0:
raise InvenioBibDocFileError, "%s seems to be empty" % filename
if docformat is None:
docformat = decompose_file(filename)[2]
else:
docformat = normalize_format(docformat)
if run_sql("SELECT id_bibdoc FROM bibdocfsinfo WHERE id_bibdoc=%s AND version=%s AND format=%s", (self.id, version, docformat)):
raise InvenioBibDocFileError("According to the database a file of format %s is already attached to the docid %s" % (docformat, self.id))
destination = self.get_filepath(docformat, version)
if os.path.exists(destination):
raise InvenioBibDocFileError, "A file for docid '%s' already exists for the format '%s'" % (str(self.id), docformat)
try:
shutil.copyfile(filename, destination)
os.chmod(destination, 0644)
if modification_date: # if the modification time of the file needs to be changed
update_modification_date_of_file(destination, modification_date)
except Exception, e:
register_exception()
raise InvenioBibDocFileError, "Encountered an exception while copying '%s' to '%s': '%s'" % (filename, destination, e)
self.more_info.set_comment(comment, docformat, version)
self.more_info.set_description(description, docformat, version)
if flags is None:
flags = []
if 'pdfa' in get_subformat_from_format(docformat).split(';') and not 'PDF/A' in flags:
flags.append('PDF/A')
for flag in flags:
if flag != 'PERFORM_HIDE_PREVIOUS':
self.more_info.set_flag(flag, docformat, version)
else:
raise InvenioBibDocFileError, "'%s' does not exists!" % filename
Md5Folder(self.basedir).update()
self.touch('newformat')
just_added_file = self.get_file(docformat, version)
run_sql("INSERT INTO bibdocfsinfo(id_bibdoc, version, format, last_version, cd, md, checksum, filesize, mime) VALUES(%s, %s, %s, true, %s, %s, %s, %s, %s)", (self.id, version, docformat, just_added_file.cd, just_added_file.md, just_added_file.get_checksum(), just_added_file.get_size(), just_added_file.mime))
def change_docformat(self, oldformat, newformat):
"""
Renames a format name on disk and in all BibDoc structures.
The change will touch only the last version files.
The change will take place only if the newformat doesn't already exist.
@param oldformat: the format that needs to be renamed
@type oldformat: string
@param newformat: the format new name
@type newformat: string
"""
oldformat = normalize_format(oldformat)
newformat = normalize_format(newformat)
if self.format_already_exists_p(newformat):
# same format already exists in the latest files, abort
return
for bibdocfile in self.list_latest_files():
if bibdocfile.get_format() == oldformat:
# change format -> rename x.oldformat -> x.newformat
dirname, base, docformat, version = decompose_file_with_version(bibdocfile.get_full_path())
os.rename(bibdocfile.get_full_path(), os.path.join(dirname, '%s%s;%i' %(base, newformat, version)))
Md5Folder(self.basedir).update()
self.touch('rename')
self._sync_to_db()
return
def purge(self):
"""
Physically removes all the previous version of the given bibdoc.
Everything but the last formats will be erased.
"""
version = self.get_latest_version()
if version > 1:
for afile in self.docfiles:
if afile.get_version() < version:
self.more_info.unset_comment(afile.get_format(), afile.get_version())
self.more_info.unset_description(afile.get_format(), afile.get_version())
for flag in CFG_BIBDOCFILE_AVAILABLE_FLAGS:
self.more_info.unset_flag(flag, afile.get_format(), afile.get_version())
try:
os.remove(afile.get_full_path())
except Exception as dummy:
register_exception()
Md5Folder(self.basedir).update()
self.touch('purge')
run_sql("DELETE FROM bibdocfsinfo WHERE id_bibdoc=%s AND version<%s", (self.id, version))
def expunge(self):
"""
Physically remove all the traces of a given document.
@note: an expunged BibDoc object shouldn't be used anymore or the
result might be unpredicted.
"""
self.more_info.delete()
del self.more_info
os.system('rm -rf %s' % escape_shell_arg(self.basedir))
run_sql('DELETE FROM bibrec_bibdoc WHERE id_bibdoc=%s', (self.id, ))
run_sql('DELETE FROM bibdoc_bibdoc WHERE id_bibdoc1=%s OR id_bibdoc2=%s', (self.id, self.id))
run_sql('DELETE FROM bibdoc WHERE id=%s', (self.id, ))
run_sql('INSERT INTO hstDOCUMENT(action, docname, docformat, docversion, docsize, docchecksum, id_bibdoc, doctimestamp) VALUES("EXPUNGE", %s, %s, %s, %s, %s, %s, NOW())',
('', self.doctype, self.get_latest_version(), self.get_total_size_latest_version(), '', self.id, ))
run_sql('DELETE FROM bibdocfsinfo WHERE id_bibdoc=%s', (self.id, ))
del self._docfiles
del self.id
del self.cd
del self.md
del self.td
del self.basedir
del self.doctype
del self.bibrec_links
def revert(self, version):
"""
Revert the document to a given version. All the formats corresponding
to that version are copied forward to a new version.
@param version: the version to revert to.
@type version: integer
@raise InvenioBibDocFileError: in case of errors
"""
version = int(version)
docfiles = self.list_version_files(version)
if docfiles:
self.add_file_new_version(docfiles[0].get_full_path(), description=docfiles[0].get_description(), comment=docfiles[0].get_comment(), docformat=docfiles[0].get_format(), flags=docfiles[0].flags)
for docfile in docfiles[1:]:
self.add_file_new_format(docfile.filename, description=docfile.get_description(), comment=docfile.get_comment(), docformat=docfile.get_format(), flags=docfile.flags)
def import_descriptions_and_comments_from_marc(self, record=None):
"""
Import descriptions and comments from the corresponding MARC metadata.
@param record: the record (if None it will be calculated).
@type record: bibrecord recstruct
@note: If record is passed it is directly used, otherwise it is retrieved
from the MARCXML stored in the database.
"""
## Let's get the record
from invenio.legacy.search_engine import get_record
if record is None:
record = get_record(self.id)
fields = record_get_field_instances(record, '856', '4', ' ')
global_comment = None
global_description = None
local_comment = {}
local_description = {}
for field in fields:
url = field_get_subfield_values(field, 'u')
if url:
## Given a url
url = url[0]
if re.match('%s/%s/[0-9]+/files/' % (CFG_SITE_URL, CFG_SITE_RECORD), url):
## If it is a traditional /CFG_SITE_RECORD/1/files/ one
## We have global description/comment for all the formats
description = field_get_subfield_values(field, 'y')
if description:
global_description = description[0]
comment = field_get_subfield_values(field, 'z')
if comment:
global_comment = comment[0]
elif bibdocfile_url_p(url):
## Otherwise we have description/comment per format
dummy, docname, docformat = decompose_bibdocfile_url(url)
brd = BibRecDocs(self.id)
if docname == brd.get_docname(self.id):
description = field_get_subfield_values(field, 'y')
if description:
local_description[docformat] = description[0]
comment = field_get_subfield_values(field, 'z')
if comment:
local_comment[docformat] = comment[0]
## Let's update the tables
version = self.get_latest_version()
for docfile in self.list_latest_files():
docformat = docfile.get_format()
if docformat in local_comment:
self.set_comment(local_comment[docformat], docformat, version)
else:
self.set_comment(global_comment, docformat, version)
if docformat in local_description:
self.set_description(local_description[docformat], docformat, version)
else:
self.set_description(global_description, docformat, version)
self.dirty = True
def get_icon(self, subformat_re=CFG_BIBDOCFILE_ICON_SUBFORMAT_RE, display_hidden=True):
"""
@param subformat_re: by default the convention is that
L{CFG_BIBDOCFILE_ICON_SUBFORMAT_RE} is used as a subformat indicator to
mean that a particular format is to be used as an icon.
Specifiy a different subformat if you need to use a different
convention.
@type subformat_re: compiled regular expression
@return: the bibdocfile corresponding to the icon of this document, or
None if any icon exists for this document.
@rtype: BibDocFile
@warning: before I{subformat} were introduced this method was
returning a BibDoc, while now is returning a BibDocFile. Check
if your client code is compatible with this.
"""
for docfile in self.list_latest_files(list_hidden=display_hidden):
if subformat_re.match(docfile.get_subformat()):
return docfile
return None
def add_icon(self, filename, docformat=None, subformat=CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT, modification_date=None):
"""
Attaches icon to this document.
@param filename: the local filesystem path to the icon.
@type filename: string
@param format: an optional format for the icon. If not specified it
will be calculated after the filesystem path.
@type format: string
@param subformat: by default the convention is that
CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT is used as a subformat indicator to
mean that a particular format is to be used as an icon.
Specifiy a different subformat if you need to use a different
convention.
@type subformat: string
@raise InvenioBibDocFileError: in case of errors.
"""
#first check if an icon already exists
if not docformat:
docformat = decompose_file(filename)[2]
if subformat:
docformat += ";%s" % subformat
self.add_file_new_format(filename, docformat=docformat, modification_date=modification_date)
def delete_icon(self, subformat_re=CFG_BIBDOCFILE_ICON_SUBFORMAT_RE):
"""
@param subformat_re: by default the convention is that
L{CFG_BIBDOCFILE_ICON_SUBFORMAT_RE} is used as a subformat indicator to
mean that a particular format is to be used as an icon.
Specifiy a different subformat if you need to use a different
convention.
@type subformat: compiled regular expression
Removes the icon attached to the document if it exists.
"""
for docfile in self.list_latest_files():
if subformat_re.match(docfile.get_subformat()):
self.delete_file(docfile.get_format(), docfile.get_version())
def change_name(self, recid, newname):
"""
Renames this document in connection with a given record.
@param newname: the new name.
@type newname: string
@raise InvenioBibDocFileError: if the new name corresponds to
a document already attached to the record owning this document.
"""
newname = normalize_docname(newname)
res = run_sql("SELECT id_bibdoc FROM bibrec_bibdoc WHERE id_bibrec=%s AND docname=%s", (recid, newname))
if res:
raise InvenioBibDocFileError, "A bibdoc called %s already exists for recid %s" % (newname, recid)
run_sql("update bibrec_bibdoc set docname=%s where id_bibdoc=%s and id_bibrec=%s", (newname, self.id, recid))
# docid is known, the document already exists
res2 = run_sql("SELECT id_bibrec, type, docname FROM bibrec_bibdoc WHERE id_bibdoc=%s", (self.id,))
## Refreshing names and types.
self.bibrec_types = [(r[0], r[1], r[2]) for r in res2 ] # just in case the result was behaving like tuples but was something else
if not res2:
# fake attachment
self.bibrec_types = [(0, None, "fake_name_for_unattached_document")]
self.touch('rename')
def set_comment(self, comment, docformat, version=None):
"""
Updates the comment of a specific format/version of the document.
@param comment: the new comment.
@type comment: string
@param format: the specific format for which the comment should be
updated.
@type format: string
@param version: the specific version for which the comment should be
updated. If not specified the last version will be used.
@type version: integer
"""
if version is None:
version = self.get_latest_version()
docformat = normalize_format(docformat)
self.more_info.set_comment(comment, docformat, version)
self.dirty = True
def set_description(self, description, docformat, version=None):
"""
Updates the description of a specific format/version of the document.
@param description: the new description.
@type description: string
@param format: the specific format for which the description should be
updated.
@type format: string
@param version: the specific version for which the description should be
updated. If not specified the last version will be used.
@type version: integer
"""
if version is None:
version = self.get_latest_version()
docformat = normalize_format(docformat)
self.more_info.set_description(description, docformat, version)
self.dirty = True
def set_flag(self, flagname, docformat, version=None):
"""
Sets a flag for a specific format/version of the document.
@param flagname: a flag from L{CFG_BIBDOCFILE_AVAILABLE_FLAGS}.
@type flagname: string
@param format: the specific format for which the flag should be
set.
@type format: string
@param version: the specific version for which the flag should be
set. If not specified the last version will be used.
@type version: integer
"""
if version is None:
version = self.get_latest_version()
docformat = normalize_format(docformat)
self.more_info.set_flag(flagname, docformat, version)
self.dirty = True
def has_flag(self, flagname, docformat, version=None):
"""
Checks if a particular flag for a format/version is set.
@param flagname: a flag from L{CFG_BIBDOCFILE_AVAILABLE_FLAGS}.
@type flagname: string
@param format: the specific format for which the flag should be
set.
@type format: string
@param version: the specific version for which the flag should be
set. If not specified the last version will be used.
@type version: integer
@return: True if the flag is set.
@rtype: bool
"""
if version is None:
version = self.get_latest_version()
docformat = normalize_format(docformat)
return self.more_info.has_flag(flagname, docformat, version)
def unset_flag(self, flagname, docformat, version=None):
"""
Unsets a flag for a specific format/version of the document.
@param flagname: a flag from L{CFG_BIBDOCFILE_AVAILABLE_FLAGS}.
@type flagname: string
@param format: the specific format for which the flag should be
unset.
@type format: string
@param version: the specific version for which the flag should be
unset. If not specified the last version will be used.
@type version: integer
"""
if version is None:
version = self.get_latest_version()
docformat = normalize_format(docformat)
self.more_info.unset_flag(flagname, docformat, version)
self.dirty = True
def get_comment(self, docformat, version=None):
"""
Retrieve the comment of a specific format/version of the document.
@param format: the specific format for which the comment should be
retrieved.
@type format: string
@param version: the specific version for which the comment should be
retrieved. If not specified the last version will be used.
@type version: integer
@return: the comment.
@rtype: string
"""
if version is None:
version = self.get_latest_version()
docformat = normalize_format(docformat)
return self.more_info.get_comment(docformat, version)
def get_description(self, docformat, version=None):
"""
Retrieve the description of a specific format/version of the document.
@param format: the specific format for which the description should be
retrieved.
@type format: string
@param version: the specific version for which the description should
be retrieved. If not specified the last version will be used.
@type version: integer
@return: the description.
@rtype: string
"""
if version is None:
version = self.get_latest_version()
docformat = normalize_format(docformat)
return self.more_info.get_description(docformat, version)
def hidden_p(self, docformat, version=None):
"""
Returns True if the file specified by the given format/version is
hidden.
@param format: the specific format for which the description should be
retrieved.
@type format: string
@param version: the specific version for which the description should
be retrieved. If not specified the last version will be used.
@type version: integer
@return: True if hidden.
@rtype: bool
"""
if version is None:
version = self.get_latest_version()
return self.more_info.has_flag('HIDDEN', docformat, version)
def get_base_dir(self):
"""
@return: the base directory on the local filesystem for this document
(e.g. C{/soft/cdsweb/var/data/files/g0/123})
@rtype: string
"""
return self.basedir
def get_type(self):
"""
@return: the type of this document.
@rtype: string"""
return self.doctype
def get_id(self):
"""
@return: the id of this document.
@rtype: integer
"""
return self.id
- def get_file(self, docformat, version=""):
+ def get_file(self, docformat, version="", exact_docformat=False):
"""
Returns a L{BibDocFile} instance of this document corresponding to the
specific format and version.
@param format: the specific format.
@type format: string
@param version: the specific version for which the description should
be retrieved. If not specified the last version will be used.
@type version: integer
+ @param exact_docformat: if True, consider always the
+ complete docformat (including subformat if any)
+ @type exact_docformat: bool
@return: the L{BibDocFile} instance.
@rtype: BibDocFile
"""
if version == "":
docfiles = self.list_latest_files()
else:
version = int(version)
docfiles = self.list_version_files(version)
docformat = normalize_format(docformat)
for docfile in docfiles:
if (docfile.get_format() == docformat or not docformat):
return docfile
## Let's skip the subformat specification and consider just the
## superformat
- superformat = get_superformat_from_format(docformat)
- for docfile in docfiles:
- if get_superformat_from_format(docfile.get_format()) == superformat:
- return docfile
+ if not exact_docformat:
+ superformat = get_superformat_from_format(docformat)
+ for docfile in docfiles:
+ if get_superformat_from_format(docfile.get_format()) == superformat:
+ return docfile
raise InvenioBibDocFileError("No file for doc %i of format '%s', version '%s'" % (self.id, docformat, version))
def list_versions(self):
"""
@return: the list of existing version numbers for this document.
@rtype: list of integer
"""
versions = []
for docfile in self.docfiles:
if not docfile.get_version() in versions:
versions.append(docfile.get_version())
versions.sort()
return versions
def delete(self, recid=None):
"""
Delete this document.
@see: L{undelete} for how to undelete the document.
@raise InvenioBibDocFileError: in case of errors.
"""
try:
today = datetime.today()
recids = []
if recid:
recids = [recid]
else:
recids = [link["recid"] for link in self.bibrec_links]
for rid in recids:
brd = BibRecDocs(rid)
docname = brd.get_docname(self.id)
# if the document is attached to some records
brd.change_name(docid=self.id, newname = 'DELETED-%s%s-%s' % (today.strftime('%Y%m%d%H%M%S'), today.microsecond, docname))
run_sql("UPDATE bibdoc SET status='DELETED' WHERE id=%s", (self.id,))
self.status = 'DELETED'
except Exception as e:
register_exception(alert_admin=True)
raise InvenioBibDocFileError, "It's impossible to delete bibdoc %s: %s" % (self.id, e)
def deleted_p(self):
"""
@return: True if this document has been deleted.
@rtype: bool
"""
return self.status == 'DELETED'
def empty_p(self):
"""
@return: True if this document is empty, i.e. it has no bibdocfile
connected.
@rtype: bool
"""
return len(self.docfiles) == 0
def undelete(self, previous_status='', recid=None):
"""
Undelete a deleted file (only if it was actually deleted via L{delete}).
The previous C{status}, i.e. the restriction key can be provided.
Otherwise the undeleted document will be public.
@param previous_status: the previous status the should be restored.
@type previous_status: string
@raise InvenioBibDocFileError: in case of any error.
"""
try:
run_sql("UPDATE bibdoc SET status=%s WHERE id=%s AND status='DELETED'", (previous_status, self.id))
except Exception as e:
raise InvenioBibDocFileError, "It's impossible to undelete bibdoc %s: %s" % (self.id, e)
if recid:
bibrecdocs = BibRecDocs(recid)
docname = bibrecdocs.get_docname(self.id)
if docname.startswith('DELETED-'):
try:
# Let's remove DELETED-20080214144322- in front of the docname
original_name = '-'.join(docname.split('-')[2:])
original_name = bibrecdocs.propose_unique_docname(original_name)
bibrecdocs.change_name(docid=self.id, newname=original_name)
except Exception as e:
raise InvenioBibDocFileError, "It's impossible to restore the previous docname %s. %s kept as docname because: %s" % (original_name, docname, e)
else:
raise InvenioBibDocFileError, "Strange just undeleted docname isn't called DELETED-somedate-docname but %s" % docname
def delete_file(self, docformat, version):
"""
Delete a specific format/version of this document on the filesystem.
@param format: the particular format to be deleted.
@type format: string
@param version: the particular version to be deleted.
@type version: integer
@note: this operation is not reversible!"""
try:
afile = self.get_file(docformat, version)
except InvenioBibDocFileError:
return
try:
os.remove(afile.get_full_path())
run_sql("DELETE FROM bibdocfsinfo WHERE id_bibdoc=%s AND version=%s AND format=%s", (self.id, afile.get_version(), afile.get_format()))
last_version = run_sql("SELECT max(version) FROM bibdocfsinfo WHERE id_bibdoc=%s", (self.id, ))[0][0]
if last_version:
## Updating information about last version
run_sql("UPDATE bibdocfsinfo SET last_version=true WHERE id_bibdoc=%s AND version=%s", (self.id, last_version))
run_sql("UPDATE bibdocfsinfo SET last_version=false WHERE id_bibdoc=%s AND version<>%s", (self.id, last_version))
except OSError:
pass
self.touch('delete')
def get_history(self):
"""
@return: a human readable and parsable string that represent the
history of this document.
@rtype: string
"""
ret = []
hst = run_sql("""SELECT action, docname, docformat, docversion,
docsize, docchecksum, doctimestamp
FROM hstDOCUMENT
WHERE id_bibdoc=%s ORDER BY doctimestamp ASC""", (self.id, ))
for row in hst:
ret.append("%s %s '%s', format: '%s', version: %i, size: %s, checksum: '%s'" % (row[6].strftime('%Y-%m-%d %H:%M:%S'), row[0], row[1], row[2], row[3], nice_size(row[4]), row[5]))
return ret
def _build_file_list(self, context=''):
"""
Lists all files attached to the bibdoc. This function should be
called everytime the bibdoc is modified.
As a side effect it log everything that has happened to the bibdocfiles
in the log facility, according to the context:
"init": means that the function has been called;
for the first time by a constructor, hence no logging is performed
"": by default means to log every deleted file as deleted and every
added file as added;
"rename": means that every appearently deleted file is logged as
renamef and every new file as renamet.
"""
def log_action(action, docid, docname, docformat, version, size, checksum, timestamp=''):
"""Log an action into the bibdoclog table."""
try:
if timestamp:
run_sql('INSERT INTO hstDOCUMENT(action, id_bibdoc, docname, docformat, docversion, docsize, docchecksum, doctimestamp) VALUES(%s, %s, %s, %s, %s, %s, %s, %s)', (action, docid, docname, docformat, version, size, checksum, timestamp))
else:
run_sql('INSERT INTO hstDOCUMENT(action, id_bibdoc, docname, docformat, docversion, docsize, docchecksum, doctimestamp) VALUES(%s, %s, %s, %s, %s, %s, %s, NOW())', (action, docid, docname, docformat, version, size, checksum))
except DatabaseError:
register_exception()
def make_removed_added_bibdocfiles(previous_file_list):
"""Internal function for build the log of changed files."""
# Let's rebuild the previous situation
old_files = {}
for bibdocfile in previous_file_list:
old_files[(bibdocfile.name, bibdocfile.format, bibdocfile.version)] = (bibdocfile.size, bibdocfile.checksum, bibdocfile.md)
# Let's rebuild the new situation
new_files = {}
for bibdocfile in self._docfiles:
new_files[(bibdocfile.name, bibdocfile.format, bibdocfile.version)] = (bibdocfile.size, bibdocfile.checksum, bibdocfile.md)
# Let's subtract from added file all the files that are present in
# the old list, and let's add to deleted files that are not present
# added file.
added_files = dict(new_files)
deleted_files = {}
for key, value in iteritems(old_files):
if key in added_files:
del added_files[key]
else:
deleted_files[key] = value
return (added_files, deleted_files)
if context != ('init', 'init_from_disk'):
previous_file_list = list(self._docfiles)
res = run_sql("SELECT status, creation_date,"
"modification_date FROM bibdoc WHERE id=%s", (self.id,))
self.cd = res[0][1]
self.md = res[0][2]
self.status = res[0][0]
self.more_info = BibDocMoreInfo(self.id)
self._docfiles = []
if CFG_BIBDOCFILE_ENABLE_BIBDOCFSINFO_CACHE and context == 'init':
## In normal init context we read from DB
res = run_sql("SELECT version, format, cd, md, checksum, filesize FROM bibdocfsinfo WHERE id_bibdoc=%s", (self.id, ))
for version, docformat, cd, md, checksum, size in res:
filepath = self.get_filepath(docformat, version)
self._docfiles.append(BibDocFile(
filepath, self.bibrec_types,
version, docformat, self.id, self.status, checksum,
self.more_info, human_readable=self.human_readable, cd=cd, md=md, size=size, bibdoc=self))
else:
if os.path.exists(self.basedir):
files = os.listdir(self.basedir)
files.sort()
for afile in files:
if not afile.startswith('.'):
try:
filepath = os.path.join(self.basedir, afile)
dummy, dummy, docformat, fileversion = decompose_file_with_version(filepath)
checksum = self.md5s.get_checksum(afile)
self._docfiles.append(BibDocFile(filepath, self.bibrec_types,
fileversion, docformat,
self.id, self.status, checksum,
self.more_info, human_readable=self.human_readable, bibdoc=self))
except Exception as e:
register_exception()
raise InvenioBibDocFileError, e
if context in ('init', 'init_from_disk'):
return
else:
added_files, deleted_files = make_removed_added_bibdocfiles(previous_file_list)
deletedstr = "DELETED"
addedstr = "ADDED"
if context == 'rename':
deletedstr = "RENAMEDFROM"
addedstr = "RENAMEDTO"
for (docname, docformat, version), (size, checksum, md) in iteritems(added_files):
if context == 'rename':
md = '' # No modification time
log_action(addedstr, self.id, docname, docformat, version, size, checksum, md)
for (docname, docformat, version), (size, checksum, md) in iteritems(deleted_files):
if context == 'rename':
md = '' # No modification time
log_action(deletedstr, self.id, docname, docformat, version, size, checksum, md)
def _sync_to_db(self):
"""
Update the content of the bibdocfile table by taking what is available on the filesystem.
"""
self._build_file_list('init_from_disk')
run_sql("DELETE FROM bibdocfsinfo WHERE id_bibdoc=%s", (self.id,))
for afile in self.docfiles:
run_sql("INSERT INTO bibdocfsinfo(id_bibdoc, version, format, last_version, cd, md, checksum, filesize, mime) VALUES(%s, %s, %s, false, %s, %s, %s, %s, %s)", (self.id, afile.get_version(), afile.get_format(), afile.cd, afile.md, afile.get_checksum(), afile.get_size(), afile.mime))
run_sql("UPDATE bibdocfsinfo SET last_version=true WHERE id_bibdoc=%s AND version=%s", (self.id, self.get_latest_version()))
def _build_related_file_list(self):
"""Lists all files attached to the bibdoc. This function should be
called everytime the bibdoc is modified within e.g. its icon.
@deprecated: use subformats instead.
"""
self.related_files = {}
res = run_sql("SELECT ln.id_bibdoc2,ln.rel_type,bibdoc.status FROM "
"bibdoc_bibdoc AS ln,bibdoc WHERE bibdoc.id=ln.id_bibdoc2 AND "
"ln.id_bibdoc1=%s", (str(self.id),))
for row in res:
docid = row[0]
doctype = row[1]
if row[2] != 'DELETED':
if doctype not in self.related_files:
self.related_files[doctype] = []
cur_doc = BibDoc.create_instance(docid=docid, human_readable=self.human_readable)
self.related_files[doctype].append(cur_doc)
def get_total_size_latest_version(self):
"""Return the total size used on disk of all the files belonging
to this bibdoc and corresponding to the latest version."""
ret = 0
for bibdocfile in self.list_latest_files():
ret += bibdocfile.get_size()
return ret
def get_total_size(self):
"""Return the total size used on disk of all the files belonging
to this bibdoc."""
ret = 0
for bibdocfile in self.list_all_files():
ret += bibdocfile.get_size()
return ret
def list_all_files(self, list_hidden=True):
"""Returns all the docfiles linked with the given bibdoc."""
if list_hidden:
return self.docfiles
else:
return [afile for afile in self.docfiles if not afile.hidden_p()]
def list_latest_files(self, list_hidden=True):
"""Returns all the docfiles within the last version."""
return self.list_version_files(self.get_latest_version(), list_hidden=list_hidden)
def list_version_files(self, version, list_hidden=True):
"""Return all the docfiles of a particular version."""
version = int(version)
return [docfile for docfile in self.docfiles if docfile.get_version() == version and (list_hidden or not docfile.hidden_p())]
def get_latest_version(self):
""" Returns the latest existing version number for the given bibdoc.
If no file is associated to this bibdoc, returns '0'.
"""
version = 0
for bibdocfile in self.docfiles:
if bibdocfile.get_version() > version:
version = bibdocfile.get_version()
return version
def get_file_number(self):
"""Return the total number of files."""
return len(self.docfiles)
def register_download(self, ip_address, version, docformat, userid=0, recid=0):
"""Register the information about a download of a particular file."""
docformat = normalize_format(docformat)
if docformat[:1] == '.':
docformat = docformat[1:]
docformat = docformat.upper()
if not version:
version = self.get_latest_version()
return run_sql("INSERT INTO rnkDOWNLOADS "
"(id_bibrec,id_bibdoc,file_version,file_format,"
"id_user,client_host,download_time) VALUES "
"(%s,%s,%s,%s,%s,INET_ATON(%s),NOW())",
(recid, self.id, version, docformat,
userid, ip_address,))
def get_incoming_relations(self, rel_type=None):
"""Return all relations in which this BibDoc appears on target position
@param rel_type: Type of the relation, to which we want to limit our search. None = any type
@type rel_type: string
@return: List of BibRelation instances
@rtype: list
"""
return BibRelation.get_relations(rel_type = rel_type,
bibdoc2_id = self.id)
def get_outgoing_relations(self, rel_type=None):
"""Return all relations in which this BibDoc appears on target position
@param rel_type: Type of the relation, to which we want to limit our search. None = any type
@type rel_type: string
@return: List of BibRelation instances
@rtype: list
"""
return BibRelation.get_relations(rel_type = rel_type,
bibdoc1_id = self.id)
def create_outgoing_relation(self, bibdoc2, rel_type):
"""
Create an outgoing relation between current BibDoc and a different one
"""
return BibRelation.create(bibdoc1_id = self.id, bibdoc2_id = bibdoc2.id, rel_type = rel_type)
def create_incoming_relation(self, bibdoc1, rel_type):
"""
Create an outgoing relation between a particular version of
current BibDoc and a particular version of a different BibDoc
"""
return BibRelation.create(bibdoc1_id = bibdoc1.id, bibdoc2_id = self.id, rel_type = rel_type)
def generic_path2bidocfile(fullpath):
"""
Returns a BibDocFile objects that wraps the given fullpath.
@note: the object will contain the minimum information that can be
guessed from the fullpath (e.g. docname, format, subformat, version,
md5, creation_date, modification_date). It won't contain for example
a comment, a description, a doctype, a restriction.
"""
fullpath = os.path.abspath(fullpath)
try:
path, name, docformat, version = decompose_file_with_version(fullpath)
except ValueError:
## There is no version
version = 0
path, name, docformat = decompose_file(fullpath)
md5folder = Md5Folder(path)
checksum = md5folder.get_checksum(os.path.basename(fullpath))
return BibDocFile(fullpath=fullpath,
recid_doctypes=[(0, None, name)],
version=version,
docformat=docformat,
docid=0,
status=None,
checksum=checksum,
more_info=None)
class BibDocFile(object):
"""This class represents a physical file in the Invenio filesystem.
It should never be instantiated directly"""
def __init__(self, fullpath, recid_doctypes, version, docformat, docid, status, checksum, more_info=None, human_readable=False, cd=None, md=None, size=None, bibdoc = None):
self.fullpath = os.path.abspath(fullpath)
self.docid = docid
self.recids_doctypes = recid_doctypes
self.version = version
self.status = status
self.checksum = checksum
self.human_readable = human_readable
self.name = recid_doctypes[0][2]
self.bibdoc = bibdoc
if more_info:
self.description = more_info.get_description(docformat, version)
self.comment = more_info.get_comment(docformat, version)
self.flags = more_info.get_flags(docformat, version)
else:
self.description = None
self.comment = None
self.flags = []
self.format = normalize_format(docformat)
self.superformat = get_superformat_from_format(self.format)
self.subformat = get_subformat_from_format(self.format)
if docformat:
self.recids_doctypes = [(a,b,c+self.superformat) for (a,b,c) in self.recids_doctypes]
self.mime, self.encoding = _mimes.guess_type(self.recids_doctypes[0][2])
if self.mime is None:
self.mime = "application/octet-stream"
self.more_info = more_info
self.hidden = 'HIDDEN' in self.flags
self.size = size or os.path.getsize(fullpath)
self.md = md or datetime.fromtimestamp(os.path.getmtime(fullpath))
try:
self.cd = cd or datetime.fromtimestamp(os.path.getctime(fullpath))
except OSError:
self.cd = self.md
self.dir = os.path.dirname(fullpath)
if self.subformat:
self.url = create_url('%s/%s/%s/files/%s%s' % (CFG_SITE_URL, CFG_SITE_RECORD, self.recids_doctypes[0][0], self.name, self.superformat), {'subformat' : self.subformat})
self.fullurl = create_url('%s/%s/%s/files/%s%s' % (CFG_SITE_URL, CFG_SITE_RECORD, self.recids_doctypes[0][0], self.name, self.superformat), {'subformat' : self.subformat, 'version' : self.version})
else:
self.url = create_url('%s/%s/%s/files/%s%s' % (CFG_SITE_URL, CFG_SITE_RECORD, self.recids_doctypes[0][0], self.name, self.superformat), {})
self.fullurl = create_url('%s/%s/%s/files/%s%s' % (CFG_SITE_URL, CFG_SITE_RECORD, self.recids_doctypes[0][0], self.name, self.superformat), {'version' : self.version})
self.etag = '"%i%s%i"' % (self.docid, self.format, self.version)
self.magic = None
def __repr__(self):
return ('BibDocFile(%s, %i, %s, %s, %i, %i, %s, %s, %s, %s)' % (repr(self.fullpath), self.version, repr(self.name), repr(self.format), self.recids_doctypes[0][0], self.docid, repr(self.status), repr(self.checksum), repr(self.more_info), repr(self.human_readable)))
def format_recids(self):
if self.bibdoc:
return self.bibdoc.format_recids()
return "0"
def __str__(self):
recids = self.format_recids()
out = '%s:%s:%s:%s:fullpath=%s\n' % (recids, self.docid, self.version, self.format, self.fullpath)
out += '%s:%s:%s:%s:name=%s\n' % (recids, self.docid, self.version, self.format, self.name)
out += '%s:%s:%s:%s:subformat=%s\n' % (recids, self.docid, self.version, self.format, get_subformat_from_format(self.format))
out += '%s:%s:%s:%s:status=%s\n' % (recids, self.docid, self.version, self.format, self.status)
out += '%s:%s:%s:%s:checksum=%s\n' % (recids, self.docid, self.version, self.format, self.checksum)
if self.human_readable:
out += '%s:%s:%s:%s:size=%s\n' % (recids, self.docid, self.version, self.format, nice_size(self.size))
else:
out += '%s:%s:%s:%s:size=%s\n' % (recids, self.docid, self.version, self.format, self.size)
out += '%s:%s:%s:%s:creation time=%s\n' % (recids, self.docid, self.version, self.format, self.cd)
out += '%s:%s:%s:%s:modification time=%s\n' % (recids, self.docid, self.version, self.format, self.md)
out += '%s:%s:%s:%s:magic=%s\n' % (recids, self.docid, self.version, self.format, self.get_magic())
out += '%s:%s:%s:%s:mime=%s\n' % (recids, self.docid, self.version, self.format, self.mime)
out += '%s:%s:%s:%s:encoding=%s\n' % (recids, self.docid, self.version, self.format, self.encoding)
out += '%s:%s:%s:%s:url=%s\n' % (recids, self.docid, self.version, self.format, self.url)
out += '%s:%s:%s:%s:fullurl=%s\n' % (recids, self.docid, self.version, self.format, self.fullurl)
out += '%s:%s:%s:%s:description=%s\n' % (recids, self.docid, self.version, self.format, self.description)
out += '%s:%s:%s:%s:comment=%s\n' % (recids, self.docid, self.version, self.format, self.comment)
out += '%s:%s:%s:%s:hidden=%s\n' % (recids, self.docid, self.version, self.format, self.hidden)
out += '%s:%s:%s:%s:flags=%s\n' % (recids, self.docid, self.version, self.format, self.flags)
out += '%s:%s:%s:%s:etag=%s\n' % (recids, self.docid, self.version, self.format, self.etag)
return out
def is_restricted(self, user_info):
"""Returns restriction state. (see acc_authorize_action return values)"""
if self.status not in ('', 'DELETED'):
return check_bibdoc_authorization(user_info, status=self.status)
elif self.status == 'DELETED':
return (1, 'File has ben deleted')
else:
return (0, '')
def is_icon(self, subformat_re=CFG_BIBDOCFILE_ICON_SUBFORMAT_RE):
"""
@param subformat_re: by default the convention is that
L{CFG_BIBDOCFILE_ICON_SUBFORMAT_RE} is used as a subformat indicator to
mean that a particular format is to be used as an icon.
Specifiy a different subformat if you need to use a different
convention.
@type subformat: compiled regular expression
@return: True if this file is an icon.
@rtype: bool
"""
return bool(subformat_re.match(self.subformat))
def hidden_p(self):
return self.hidden
def get_url(self):
return self.url
def get_type(self):
"""Returns the first type connected with the bibdoc of this file."""
return self.recids_doctypes[0][1]
def get_path(self):
return self.fullpath
def get_bibdocid(self):
return self.docid
def get_name(self):
return self.name
def get_full_name(self):
"""Returns the first name connected with the bibdoc of this file."""
return self.recids_doctypes[0][2]
def get_full_path(self):
return self.fullpath
def get_format(self):
return self.format
def get_subformat(self):
return self.subformat
def get_superformat(self):
return self.superformat
def get_size(self):
return self.size
def get_version(self):
return self.version
def get_checksum(self):
return self.checksum
def get_description(self):
return self.description
def get_comment(self):
return self.comment
def get_content(self):
"""Returns the binary content of the file."""
content_fd = open(self.fullpath, 'rb')
content = content_fd.read()
content_fd.close()
return content
def get_recid(self):
"""Returns the first recid connected with the bibdoc of this file."""
return self.recids_doctypes[0][0]
def get_status(self):
"""Returns the status of the file, i.e. either '', 'DELETED' or a
restriction keyword."""
return self.status
def get_magic(self):
"""Return all the possible guesses from the magic library about
the content of the file."""
if self.magic is None:
if CFG_HAS_MAGIC == 1:
magic_cookies = _get_magic_cookies()
magic_result = []
for key in magic_cookies.keys():
magic_result.append(magic_cookies[key].file(self.fullpath))
self.magic = tuple(magic_result)
elif CFG_HAS_MAGIC == 2:
magic_result = []
for key in ({'mime': False, 'mime_encoding': False},
{'mime': True, 'mime_encoding': False},
{'mime': False, 'mime_encoding': True}):
magic_result.append(_magic_wrapper(self.fullpath, **key))
self.magic = tuple(magic_result)
return self.magic
def check(self):
"""Return True if the checksum corresponds to the file."""
return calculate_md5(self.fullpath) == self.checksum
def stream(self, req, download=False):
"""Stream the file. Note that no restriction check is being
done here, since restrictions have been checked previously
inside websubmit_webinterface.py."""
if os.path.exists(self.fullpath):
if random.random() < CFG_BIBDOCFILE_MD5_CHECK_PROBABILITY and calculate_md5(self.fullpath) != self.checksum:
raise InvenioBibDocFileError, "File %s, version %i, is corrupted!" % (self.recids_doctypes[0][2], self.version)
stream_file(req, self.fullpath, "%s%s" % (self.name, self.superformat), self.mime, self.encoding, self.etag, self.checksum, self.fullurl, download=download)
raise apache.SERVER_RETURN, apache.DONE
else:
req.status = apache.HTTP_NOT_FOUND
raise InvenioBibDocFileError, "%s does not exists!" % self.fullpath
_RE_STATUS_PARSER = re.compile(r'^(?P<type>email|group|egroup|role|firerole|status):\s*(?P<value>.*)$', re.S + re.I)
def check_bibdoc_authorization(user_info, status):
"""
Check if the user is authorized to access a document protected with the given status.
L{status} is a string of the form::
auth_type: auth_value
where C{auth_type} can have values in::
email, group, role, firerole, status
and C{auth_value} has a value interpreted againsta C{auth_type}:
- C{email}: the user can access the document if his/her email matches C{auth_value}
- C{group}: the user can access the document if one of the groups (local or
external) of which he/she is member matches C{auth_value}
- C{role}: the user can access the document if he/she belongs to the WebAccess
role specified in C{auth_value}
- C{firerole}: the user can access the document if he/she is implicitly matched
by the role described by the firewall like role definition in C{auth_value}
- C{status}: the user can access the document if he/she is authorized to
for the action C{viewrestrdoc} with C{status} paramter having value
C{auth_value}
@note: If no C{auth_type} is specified or if C{auth_type} is not one of the
above, C{auth_value} will be set to the value contained in the
parameter C{status}, and C{auth_type} will be considered to be C{status}.
@param user_info: the user_info dictionary
@type: dict
@param status: the status of the document.
@type status: string
@return: a tuple, of the form C{(auth_code, auth_message)} where auth_code is 0
if the authorization is granted and greater than 0 otherwise.
@rtype: (int, string)
@raise ValueError: in case of unexpected parsing error.
"""
if not status:
return (0, CFG_WEBACCESS_WARNING_MSGS[0])
def parse_status(status):
g = _RE_STATUS_PARSER.match(status)
if g:
return (g.group('type').lower(), g.group('value'))
else:
return ('status', status)
if acc_is_user_in_role(user_info, acc_get_role_id(SUPERADMINROLE)):
return (0, CFG_WEBACCESS_WARNING_MSGS[0])
auth_type, auth_value = parse_status(status)
if auth_type == 'status':
return acc_authorize_action(user_info, 'viewrestrdoc', status=auth_value)
elif auth_type == 'email':
if not auth_value.lower().strip() == user_info['email'].lower().strip():
return (1, 'You must be member of the group %s in order to access this document' % repr(auth_value))
elif auth_type == 'group':
if not auth_value in user_info['group']:
return (1, 'You must be member of the group %s in order to access this document' % repr(auth_value))
elif auth_type == 'role':
if not acc_is_user_in_role(user_info, acc_get_role_id(auth_value)):
return (1, 'You must be member in the role %s in order to access this document' % repr(auth_value))
elif auth_type == 'firerole':
if not acc_firerole_check_user(user_info, compile_role_definition(auth_value)):
return (1, 'You must be authorized in order to access this document')
else:
raise ValueError, 'Unexpected authorization type %s for %s' % (repr(auth_type), repr(auth_value))
return (0, CFG_WEBACCESS_WARNING_MSGS[0])
## TODO for future reimplementation of stream_file
#class StreamFileException(Exception):
# def __init__(self, value):
# self.value = value
_RE_BAD_MSIE = re.compile("MSIE\s+(\d+\.\d+)")
def stream_file(req, fullpath, fullname=None, mime=None, encoding=None, etag=None, md5str=None, location=None, download=False):
"""This is a generic function to stream a file to the user.
If fullname, mime, encoding, and location are not provided they will be
guessed based on req and fullpath.
md5str should be passed as an hexadecimal string.
"""
## TODO for future reimplementation of stream_file
# from flask import send_file
# if fullname is None:
# fullname = fullpath.split('/')[-1]
# response = send_file(fullpath,
# attachment_filename=fullname.replace('"', '\\"'),
# as_attachment=False)
# if not download:
# response.headers['Content-Disposition'] = 'inline; filename="%s"' % fullname.replace('"', '\\"')
#
# raise StreamFileException(response)
def normal_streaming(size):
req.set_content_length(size)
req.send_http_header()
if req.method != 'HEAD':
req.sendfile(fullpath)
return ""
def single_range(size, the_range):
req.set_content_length(the_range[1])
req.headers_out['Content-Range'] = 'bytes %d-%d/%d' % (the_range[0], the_range[0] + the_range[1] - 1, size)
req.status = apache.HTTP_PARTIAL_CONTENT
req.send_http_header()
if req.method != 'HEAD':
req.sendfile(fullpath, the_range[0], the_range[1])
return ""
def multiple_ranges(size, ranges, mime):
req.status = apache.HTTP_PARTIAL_CONTENT
boundary = '%s%04d' % (time.strftime('THIS_STRING_SEPARATES_%Y%m%d%H%M%S'), random.randint(0, 9999))
req.content_type = 'multipart/byteranges; boundary=%s' % boundary
content_length = 0
for arange in ranges:
content_length += len('--%s\r\n' % boundary)
content_length += len('Content-Type: %s\r\n' % mime)
content_length += len('Content-Range: bytes %d-%d/%d\r\n' % (arange[0], arange[0] + arange[1] - 1, size))
content_length += len('\r\n')
content_length += arange[1]
content_length += len('\r\n')
content_length += len('--%s--\r\n' % boundary)
req.set_content_length(content_length)
req.send_http_header()
if req.method != 'HEAD':
for arange in ranges:
req.write('--%s\r\n' % boundary, 0)
req.write('Content-Type: %s\r\n' % mime, 0)
req.write('Content-Range: bytes %d-%d/%d\r\n' % (arange[0], arange[0] + arange[1] - 1, size), 0)
req.write('\r\n', 0)
req.sendfile(fullpath, arange[0], arange[1])
req.write('\r\n', 0)
req.write('--%s--\r\n' % boundary)
req.flush()
return ""
def parse_date(date):
"""According to <http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.3>
a date can come in three formats (in order of preference):
Sun, 06 Nov 1994 08:49:37 GMT ; RFC 822, updated by RFC 1123
Sunday, 06-Nov-94 08:49:37 GMT ; RFC 850, obsoleted by RFC 1036
Sun Nov 6 08:49:37 1994 ; ANSI C's asctime() format
Moreover IE is adding some trailing information after a ';'.
Wrong dates should be simpled ignored.
This function return the time in seconds since the epoch GMT or None
in case of errors."""
if not date:
return None
try:
date = date.split(';')[0].strip() # Because of IE
## Sun, 06 Nov 1994 08:49:37 GMT
return time.mktime(time.strptime(date, '%a, %d %b %Y %X %Z'))
except:
try:
## Sun, 06 Nov 1994 08:49:37 GMT
return time.mktime(time.strptime(date, '%A, %d-%b-%y %H:%M:%S %Z'))
except:
try:
## Sun, 06 Nov 1994 08:49:37 GMT
return time.mktime(date)
except:
return None
def parse_ranges(ranges):
"""According to <http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35>
a (multiple) range request comes in the form:
bytes=20-30,40-60,70-,-80
with the meaning:
from byte to 20 to 30 inclusive (11 bytes)
from byte to 40 to 60 inclusive (21 bytes)
from byte 70 to (size - 1) inclusive (size - 70 bytes)
from byte size - 80 to (size - 1) inclusive (80 bytes)
This function will return the list of ranges in the form:
[[first_byte, last_byte], ...]
If first_byte or last_byte aren't specified they'll be set to None
If the list is not well formatted it will return None
"""
try:
if ranges.startswith('bytes') and '=' in ranges:
ranges = ranges.split('=')[1].strip()
else:
return None
ret = []
for arange in ranges.split(','):
arange = arange.strip()
if arange.startswith('-'):
ret.append([None, int(arange[1:])])
elif arange.endswith('-'):
ret.append([int(arange[:-1]), None])
else:
ret.append(map(int, arange.split('-')))
return ret
except:
return None
def parse_tags(tags):
"""Return a list of tags starting from a comma separated list."""
return [tag.strip() for tag in tags.split(',')]
def fix_ranges(ranges, size):
"""Complementary to parse_ranges it will transform all the ranges
into (first_byte, length), adjusting all the value based on the
actual size provided.
"""
ret = []
for arange in ranges:
if (arange[0] is None and arange[1] > 0) or arange[0] < size:
if arange[0] is None:
arange[0] = size - arange[1]
elif arange[1] is None:
arange[1] = size - arange[0]
else:
arange[1] = arange[1] - arange[0] + 1
arange[0] = max(0, arange[0])
arange[1] = min(size - arange[0], arange[1])
if arange[1] > 0:
ret.append(arange)
return ret
def get_normalized_headers():
"""Strip and lowerize all the keys of the headers dictionary plus
strip, lowerize and transform known headers value into their value."""
ret = {
'if-match' : None,
'unless-modified-since' : None,
'if-modified-since' : None,
'range' : None,
'if-range' : None,
'if-none-match' : None,
}
for key, value in iteritems(req.headers_in):
key = key.strip().lower()
value = value.strip()
if key in ('unless-modified-since', 'if-modified-since'):
value = parse_date(value)
elif key == 'range':
value = parse_ranges(value)
elif key == 'if-range':
value = parse_date(value) or parse_tags(value)
elif key in ('if-match', 'if-none-match'):
value = parse_tags(value)
if value:
ret[key] = value
return ret
headers = get_normalized_headers()
g = _RE_BAD_MSIE.search(headers.get('user-agent', "MSIE 6.0"))
bad_msie = g and float(g.group(1)) < 9.0
if CFG_BIBDOCFILE_USE_XSENDFILE:
## If XSendFile is supported by the server, let's use it.
if os.path.exists(fullpath):
if fullname is None:
fullname = os.path.basename(fullpath)
if bad_msie:
## IE is confused by quotes
req.headers_out["Content-Disposition"] = 'attachment; filename=%s' % fullname.replace('"', '\\"')
elif download:
req.headers_out["Content-Disposition"] = 'attachment; filename="%s"' % fullname.replace('"', '\\"')
else:
## IE is confused by inline
req.headers_out["Content-Disposition"] = 'inline; filename="%s"' % fullname.replace('"', '\\"')
req.headers_out["X-Sendfile"] = fullpath
if mime is None:
(mime, encoding) = _mimes.guess_type(fullpath)
if mime is None:
mime = "application/octet-stream"
if not bad_msie:
## IE is confused by not supported mimetypes
req.content_type = mime
return ""
else:
raise apache.SERVER_RETURN, apache.HTTP_NOT_FOUND
if headers['if-match']:
if etag is not None and etag not in headers['if-match']:
raise apache.SERVER_RETURN, apache.HTTP_PRECONDITION_FAILED
if os.path.exists(fullpath):
mtime = os.path.getmtime(fullpath)
if fullname is None:
fullname = os.path.basename(fullpath)
if mime is None:
(mime, encoding) = _mimes.guess_type(fullpath)
if mime is None:
mime = "application/octet-stream"
if location is None:
location = req.uri
if not bad_msie:
## IE is confused by not supported mimetypes
req.content_type = mime
req.encoding = encoding
req.filename = fullname
req.headers_out["Last-Modified"] = time.strftime('%a, %d %b %Y %X GMT', time.gmtime(mtime))
if CFG_ENABLE_HTTP_RANGE_REQUESTS:
req.headers_out["Accept-Ranges"] = "bytes"
else:
req.headers_out["Accept-Ranges"] = "none"
req.headers_out["Content-Location"] = location
if etag is not None:
req.headers_out["ETag"] = etag
if md5str is not None:
req.headers_out["Content-MD5"] = base64.encodestring(binascii.unhexlify(md5str.upper()))[:-1]
if bad_msie:
## IE is confused by quotes
req.headers_out["Content-Disposition"] = 'attachment; filename=%s' % fullname.replace('"', '\\"')
elif download:
req.headers_out["Content-Disposition"] = 'attachment; filename="%s"' % fullname.replace('"', '\\"')
else:
## IE is confused by inline
req.headers_out["Content-Disposition"] = 'inline; filename="%s"' % fullname.replace('"', '\\"')
size = os.path.getsize(fullpath)
if not size:
try:
raise Exception, '%s exists but is empty' % fullpath
except Exception:
register_exception(req=req, alert_admin=True)
raise apache.SERVER_RETURN, apache.HTTP_NOT_FOUND
if headers['if-modified-since'] and headers['if-modified-since'] >= mtime:
raise apache.SERVER_RETURN, apache.HTTP_NOT_MODIFIED
if headers['if-none-match']:
if etag is not None and etag in headers['if-none-match']:
raise apache.SERVER_RETURN, apache.HTTP_NOT_MODIFIED
if headers['unless-modified-since'] and headers['unless-modified-since'] < mtime:
return normal_streaming(size)
if CFG_ENABLE_HTTP_RANGE_REQUESTS and headers['range']:
try:
if headers['if-range']:
if etag is None or etag not in headers['if-range']:
return normal_streaming(size)
ranges = fix_ranges(headers['range'], size)
except:
return normal_streaming(size)
if len(ranges) > 1:
return multiple_ranges(size, ranges, mime)
elif ranges:
return single_range(size, ranges[0])
else:
raise apache.SERVER_RETURN, apache.HTTP_RANGE_NOT_SATISFIABLE
else:
return normal_streaming(size)
else:
raise apache.SERVER_RETURN, apache.HTTP_NOT_FOUND
def stream_restricted_icon(req):
"""Return the content of the "Restricted Icon" file."""
stream_file(req, '%s/img/restricted.gif' % CFG_WEBDIR)
raise apache.SERVER_RETURN, apache.DONE
#def list_versions_from_array(docfiles):
# """Retrieve the list of existing versions from the given docfiles list."""
# versions = []
# for docfile in docfiles:
# if not docfile.get_version() in versions:
# versions.append(docfile.get_version())
# versions.sort()
# versions.reverse()
# return versions
def _make_base_dir(docid):
"""Given a docid it returns the complete path that should host its files."""
group = "g" + str(int(int(docid) / CFG_BIBDOCFILE_FILESYSTEM_BIBDOC_GROUP_LIMIT))
return os.path.join(CFG_BIBDOCFILE_FILEDIR, group, str(docid))
class Md5Folder(object):
"""Manage all the Md5 checksum about a folder"""
def __init__(self, folder):
"""Initialize the class from the md5 checksum of a given path"""
self.folder = folder
self.load()
def update(self, only_new=True):
"""Update the .md5 file with the current files. If only_new
is specified then only not already calculated file are calculated."""
if not only_new:
self.md5s = {}
if os.path.exists(self.folder):
for filename in os.listdir(self.folder):
if filename not in self.md5s and not filename.startswith('.'):
self.md5s[filename] = calculate_md5(os.path.join(self.folder, filename))
self.store()
def store(self):
"""Store the current md5 dictionary into .md5"""
try:
old_umask = os.umask(0o022)
md5file = open(os.path.join(self.folder, ".md5"), "w")
for key, value in self.md5s.items():
md5file.write('%s *%s\n' % (value, key))
md5file.close()
os.umask(old_umask)
except Exception as e:
register_exception(alert_admin=True)
raise InvenioBibDocFileError("Encountered an exception while storing .md5 for folder '%s': '%s'" % (self.folder, e))
def load(self):
"""Load .md5 into the md5 dictionary"""
self.md5s = {}
md5_path = os.path.join(self.folder, ".md5")
if os.path.exists(md5_path):
for row in open(md5_path, "r"):
md5hash = row[:32]
filename = row[34:].strip()
self.md5s[filename] = md5hash
else:
self.update()
def check(self, filename=''):
"""Check the specified file or all the files for which it exists a hash
for being coherent with the stored hash."""
if filename and filename in self.md5s.keys():
try:
return self.md5s[filename] == calculate_md5(os.path.join(self.folder, filename))
except Exception as e:
register_exception(alert_admin=True)
raise InvenioBibDocFileError("Encountered an exception while loading '%s': '%s'" % (os.path.join(self.folder, filename), e))
else:
for filename, md5hash in self.md5s.items():
try:
if calculate_md5(os.path.join(self.folder, filename)) != md5hash:
return False
except Exception as e:
register_exception(alert_admin=True)
raise InvenioBibDocFileError("Encountered an exception while loading '%s': '%s'" % (os.path.join(self.folder, filename), e))
return True
def get_checksum(self, filename):
"""Return the checksum of a physical file."""
md5hash = self.md5s.get(filename, None)
if md5hash is None:
self.update()
# Now it should not fail!
md5hash = self.md5s[filename]
return md5hash
def calculate_md5_external(filename):
"""Calculate the md5 of a physical file through md5sum Command Line Tool.
This is suitable for file larger than 256Kb."""
try:
md5_result = os.popen(CFG_PATH_MD5SUM + ' -b %s' % escape_shell_arg(filename))
ret = md5_result.read()[:32]
md5_result.close()
if len(ret) != 32:
# Error in running md5sum. Let's fallback to internal
# algorithm.
return calculate_md5(filename, force_internal=True)
else:
return ret
except Exception as e:
raise InvenioBibDocFileError("Encountered an exception while calculating md5 for file '%s': '%s'" % (filename, e))
def calculate_md5(filename, force_internal=False):
"""Calculate the md5 of a physical file. This is suitable for files smaller
than 256Kb."""
if not CFG_PATH_MD5SUM or force_internal or os.path.getsize(filename) < CFG_BIBDOCFILE_MD5_THRESHOLD:
try:
to_be_read = open(filename, "rb")
computed_md5 = md5()
while True:
buf = to_be_read.read(CFG_BIBDOCFILE_MD5_BUFFER)
if buf:
computed_md5.update(buf)
else:
break
to_be_read.close()
return computed_md5.hexdigest()
except Exception as e:
register_exception(alert_admin=True)
raise InvenioBibDocFileError("Encountered an exception while calculating md5 for file '%s': '%s'" % (filename, e))
else:
return calculate_md5_external(filename)
def bibdocfile_url_to_bibrecdocs(url):
"""Given an URL in the form CFG_SITE_[SECURE_]URL/CFG_SITE_RECORD/xxx/files/... it returns
a BibRecDocs object for the corresponding recid."""
recid = decompose_bibdocfile_url(url)[0]
return BibRecDocs(recid)
def bibdocfile_url_to_bibdoc(url):
"""Given an URL in the form CFG_SITE_[SECURE_]URL/CFG_SITE_RECORD/xxx/files/... it returns
a BibDoc object for the corresponding recid/docname."""
docname = decompose_bibdocfile_url(url)[1]
return bibdocfile_url_to_bibrecdocs(url).get_bibdoc(docname)
def bibdocfile_url_to_bibdocfile(url):
"""Given an URL in the form CFG_SITE_[SECURE_]URL/CFG_SITE_RECORD/xxx/files/... it returns
a BibDocFile object for the corresponding recid/docname/format."""
docformat = decompose_bibdocfile_url(url)[2]
return bibdocfile_url_to_bibdoc(url).get_file(docformat)
def bibdocfile_url_to_fullpath(url):
"""Given an URL in the form CFG_SITE_[SECURE_]URL/CFG_SITE_RECORD/xxx/files/... it returns
the fullpath for the corresponding recid/docname/format."""
return bibdocfile_url_to_bibdocfile(url).get_full_path()
def bibdocfile_url_p(url):
"""Return True when the url is a potential valid url pointing to a
fulltext owned by a system."""
if url.startswith('%s/getfile.py' % CFG_SITE_URL) or url.startswith('%s/getfile.py' % CFG_SITE_SECURE_URL):
return True
if not (url.startswith('%s/%s/' % (CFG_SITE_URL, CFG_SITE_RECORD)) or url.startswith('%s/%s/' % (CFG_SITE_SECURE_URL, CFG_SITE_RECORD))):
return False
splitted_url = url.split('/files/')
return len(splitted_url) == 2 and splitted_url[0] != '' and splitted_url[1] != ''
def get_docid_from_bibdocfile_fullpath(fullpath):
"""Given a bibdocfile fullpath (e.g. "CFG_BIBDOCFILE_FILEDIR/g0/123/bar.pdf;1")
returns the docid (e.g. 123)."""
if not fullpath.startswith(os.path.join(CFG_BIBDOCFILE_FILEDIR, 'g')):
raise InvenioBibDocFileError, "Fullpath %s doesn't correspond to a valid bibdocfile fullpath" % fullpath
dirname = decompose_file_with_version(fullpath)[0]
try:
return int(dirname.split('/')[-1])
except:
raise InvenioBibDocFileError, "Fullpath %s doesn't correspond to a valid bibdocfile fullpath" % fullpath
def decompose_bibdocfile_fullpath(fullpath):
"""Given a bibdocfile fullpath (e.g. "CFG_BIBDOCFILE_FILEDIR/g0/123/bar.pdf;1")
returns a quadruple (recid, docname, format, version)."""
if not fullpath.startswith(os.path.join(CFG_BIBDOCFILE_FILEDIR, 'g')):
raise InvenioBibDocFileError, "Fullpath %s doesn't correspond to a valid bibdocfile fullpath" % fullpath
dirname, dummy, extension, version = decompose_file_with_version(fullpath)
try:
docid = int(dirname.split('/')[-1])
return {"doc_id" : docid, "extension": extension, "version": version}
except:
raise InvenioBibDocFileError, "Fullpath %s doesn't correspond to a valid bibdocfile fullpath" % fullpath
def decompose_bibdocfile_url(url):
"""Given a bibdocfile_url return a triple (recid, docname, format)."""
if url.startswith('%s/getfile.py' % CFG_SITE_URL) or url.startswith('%s/getfile.py' % CFG_SITE_SECURE_URL):
return decompose_bibdocfile_very_old_url(url)
if url.startswith('%s/%s/' % (CFG_SITE_URL, CFG_SITE_RECORD)):
recid_file = url[len('%s/%s/' % (CFG_SITE_URL, CFG_SITE_RECORD)):]
elif url.startswith('%s/%s/' % (CFG_SITE_SECURE_URL, CFG_SITE_RECORD)):
recid_file = url[len('%s/%s/' % (CFG_SITE_SECURE_URL, CFG_SITE_RECORD)):]
else:
raise InvenioBibDocFileError, "Url %s doesn't correspond to a valid record inside the system." % url
recid_file = recid_file.replace('/files/', '/')
recid, docname, docformat = decompose_file(urllib.unquote(recid_file)) # this will work in the case of URL... not file !
if not recid and docname.isdigit():
## If the URL was something similar to CFG_SITE_URL/CFG_SITE_RECORD/123
return (int(docname), '', '')
return (int(recid), docname, docformat)
re_bibdocfile_old_url = re.compile(r'/%s/(\d*)/files/' % CFG_SITE_RECORD)
def decompose_bibdocfile_old_url(url):
"""Given a bibdocfile old url (e.g. CFG_SITE_URL/CFG_SITE_RECORD/123/files)
it returns the recid."""
g = re_bibdocfile_old_url.search(url)
if g:
return int(g.group(1))
raise InvenioBibDocFileError('%s is not a valid old bibdocfile url' % url)
def decompose_bibdocfile_very_old_url(url):
"""Decompose an old /getfile.py? URL"""
if url.startswith('%s/getfile.py' % CFG_SITE_URL) or url.startswith('%s/getfile.py' % CFG_SITE_SECURE_URL):
params = urllib.splitquery(url)[1]
if params:
try:
params = cgi.parse_qs(params)
if 'docid' in params:
docid = int(params['docid'][0])
bibdoc = BibDoc.create_instance(docid)
if bibdoc.bibrec_links:
recid = bibdoc.bibrec_links[0]["rec_id"]
docname = bibdoc.bibrec_links[0]["doc_name"]
else:
raise InvenioBibDocFileError("Old style URL pointing to an unattached document")
elif 'recid' in params:
recid = int(params['recid'][0])
if 'name' in params:
docname = params['name'][0]
else:
docname = ''
else:
raise InvenioBibDocFileError('%s has not enough params to correspond to a bibdocfile.' % url)
docformat = normalize_format(params.get('format', [''])[0])
return (recid, docname, docformat)
except Exception as e:
raise InvenioBibDocFileError('Problem with %s: %s' % (url, e))
else:
raise InvenioBibDocFileError('%s has no params to correspond to a bibdocfile.' % url)
else:
raise InvenioBibDocFileError('%s is not a valid very old bibdocfile url' % url)
def get_docname_from_url(url):
"""Return a potential docname given a url"""
path = urllib2.urlparse.urlsplit(urllib.unquote(url))[2]
filename = os.path.split(path)[-1]
return file_strip_ext(filename)
def get_format_from_url(url):
"""Return a potential format given a url"""
path = urllib2.urlparse.urlsplit(urllib.unquote(url))[2]
filename = os.path.split(path)[-1]
return filename[len(file_strip_ext(filename)):]
def clean_url(url):
"""Given a local url e.g. a local path it render it a realpath."""
if is_url_a_local_file(url):
path = urllib2.urlparse.urlsplit(urllib.unquote(url))[2]
return os.path.abspath(path)
else:
return url
def is_url_a_local_file(url):
"""Return True if the given URL is pointing to a local file."""
protocol = urllib2.urlparse.urlsplit(url)[0]
return protocol in ('', 'file')
def check_valid_url(url):
"""
Check for validity of a url or a file.
@param url: the URL to check
@type url: string
@raise StandardError: if the URL is not a valid URL.
"""
try:
if is_url_a_local_file(url):
path = urllib2.urlparse.urlsplit(urllib.unquote(url))[2]
if os.path.abspath(path) != path:
raise StandardError, "%s is not a normalized path (would be %s)." % (path, os.path.normpath(path))
for allowed_path in CFG_BIBUPLOAD_FFT_ALLOWED_LOCAL_PATHS + [CFG_TMPDIR, CFG_TMPSHAREDDIR, CFG_WEBSUBMIT_STORAGEDIR]:
if path.startswith(allowed_path):
dummy_fd = open(path)
dummy_fd.close()
return
raise StandardError, "%s is not in one of the allowed paths." % path
else:
try:
open_url(url)
except InvenioBibdocfileUnauthorizedURL as e:
raise StandardError, str(e)
except Exception as e:
raise StandardError, "%s is not a correct url: %s" % (url, e)
def safe_mkstemp(suffix, prefix='bibdocfile_'):
"""Create a temporary filename that don't have any '.' inside a part
from the suffix."""
tmpfd, tmppath = tempfile.mkstemp(suffix=suffix, prefix=prefix, dir=CFG_TMPDIR)
# Close the file and leave the responsability to the client code to
# correctly open/close it.
os.close(tmpfd)
if '.' not in suffix:
# Just in case format is empty
return tmppath
while '.' in os.path.basename(tmppath)[:-len(suffix)]:
os.remove(tmppath)
tmpfd, tmppath = tempfile.mkstemp(suffix=suffix, prefix=prefix, dir=CFG_TMPDIR)
os.close(tmpfd)
return tmppath
def download_local_file(filename, docformat=None):
"""
Copies a local file to Invenio's temporary directory.
@param filename: the name of the file to copy
@type filename: string
@param format: the format of the file to copy (will be found if not
specified)
@type format: string
@return: the path of the temporary file created
@rtype: string
@raise StandardError: if something went wrong
"""
# Make sure the format is OK.
if docformat is None:
docformat = guess_format_from_url(filename)
else:
docformat = normalize_format(docformat)
tmppath = ''
# Now try to copy.
try:
path = urllib2.urlparse.urlsplit(urllib.unquote(filename))[2]
if os.path.abspath(path) != path:
raise StandardError, "%s is not a normalized path (would be %s)." \
% (path, os.path.normpath(path))
for allowed_path in CFG_BIBUPLOAD_FFT_ALLOWED_LOCAL_PATHS + [CFG_TMPDIR,
CFG_WEBSUBMIT_STORAGEDIR]:
if path.startswith(allowed_path):
tmppath = safe_mkstemp(docformat)
shutil.copy(path, tmppath)
if os.path.getsize(tmppath) == 0:
os.remove(tmppath)
raise StandardError, "%s seems to be empty" % filename
break
else:
raise StandardError, "%s is not in one of the allowed paths." % path
except Exception as e:
raise StandardError, "Impossible to copy the local file '%s': %s" % \
(filename, str(e))
return tmppath
def download_external_url(url, docformat=None, progress_callback=None):
"""
Download a url (if it corresponds to a remote file) and return a
local url to it.
@param url: the URL to download
@type url: string
@param format: the format of the file (will be found if not specified)
@type format: string
@return: the path to the download local file
@rtype: string
@raise StandardError: if the download failed
"""
tmppath = None
# Make sure the format is OK.
if docformat is None:
# First try to find a known extension to the URL
docformat = decompose_file(url, skip_version=True,
only_known_extensions=True)[2]
if not docformat:
# No correct format could be found. Will try to get it from the
# HTTP message headers.
docformat = ''
else:
docformat = normalize_format(docformat)
from_file, to_file, tmppath = None, None, ''
try:
from_file = open_url(url)
except InvenioBibdocfileUnauthorizedURL as e:
raise StandardError, str(e)
except urllib2.URLError as e:
raise StandardError, 'URL could not be opened: %s' % str(e)
if not docformat:
# We could not determine the format from the URL, so let's try
# to read it from the HTTP headers.
docformat = get_format_from_http_response(from_file)
try:
tmppath = safe_mkstemp(docformat)
if progress_callback:
total_size = int(from_file.info().getheader('Content-Length').strip())
progress_size = 0
to_file = open(tmppath, 'w')
while True:
block = from_file.read(CFG_BIBDOCFILE_BLOCK_SIZE)
if not block:
break
to_file.write(block)
if progress_callback:
progress_size += CFG_BIBDOCFILE_BLOCK_SIZE
progress_callback(progress_size, CFG_BIBDOCFILE_BLOCK_SIZE,
total_size)
to_file.close()
from_file.close()
if os.path.getsize(tmppath) == 0:
raise StandardError, "%s seems to be empty" % url
except Exception as e:
# Try to close and remove the temporary file.
try:
to_file.close()
except Exception:
pass
try:
os.remove(tmppath)
except Exception:
pass
raise StandardError, "Error when downloading %s into %s: %s" % \
(url, tmppath, e)
return tmppath
def get_format_from_http_response(response):
"""
Tries to retrieve the format of the file from the message headers of the
HTTP response.
@param response: the HTTP response
@type response: file-like object (as returned by urllib.urlopen)
@return: the format of the remote resource
@rtype: string
"""
def parse_content_type(text):
return text.split(';')[0].strip()
def parse_content_disposition(text):
for item in text.split(';'):
item = item.strip()
if item.strip().startswith('filename='):
return item[len('filename="'):-len('"')]
info = response.info()
docformat = ''
content_disposition = info.getheader('Content-Disposition')
if content_disposition:
filename = parse_content_disposition(content_disposition)
if filename:
docformat = decompose_file(filename, only_known_extensions=False)[2]
if docformat:
return docformat
content_type = info.getheader('Content-Type')
if content_type:
content_type = parse_content_type(content_type)
if content_type not in ('text/plain', 'application/octet-stream'):
## We actually ignore these mimetypes since they are the
## defaults often returned by Apache in case the mimetype
## was not known
if content_type in CFG_BIBDOCFILE_PREFERRED_MIMETYPES_MAPPING:
docformat = normalize_format(CFG_BIBDOCFILE_PREFERRED_MIMETYPES_MAPPING[content_type])
else:
ext = _mimes.guess_extension(content_type)
if ext:
docformat = normalize_format(ext)
return docformat
def download_url(url, docformat=None):
"""
Download a url (if it corresponds to a remote file) and return a
local url to it.
"""
tmppath = None
try:
if is_url_a_local_file(url):
tmppath = download_local_file(url, docformat = docformat)
else:
tmppath = download_external_url(url, docformat = docformat)
except StandardError:
raise
return tmppath
class MoreInfo(object):
"""This class represents a genering MoreInfo dictionary.
MoreInfo object can be attached to bibdoc, bibversion, format or BibRelation.
The entity where a particular MoreInfo object is attached has to be specified using the
constructor parametes.
This class is a thin wrapper around the database table.
"""
def __init__(self, docid = None, version = None, docformat = None,
relation = None, cache_only = False, cache_reads = True, initial_data = None):
"""
@param cache_only Determines if MoreInfo object should be created in
memory only or reflected in the database
@type cache_only boolean
@param cache_reads Determines if reads should be executed on the
in-memory cache or should be redirected to the
database. If this is true, cache can be entirely
regenerated from the database only upon an explicit
request. If the value is not present in the cache,
the database is queried
@type cache_reads boolean
@param initial_data Allows to specify initial content of the cache.
This parameter is useful when we create an in-memory
instance from serialised value
@type initial_data string
"""
self.docid = docid
self.version = version
self.format = docformat
self.relation = relation
self.cache_only = cache_only
if initial_data != None:
self.cache = initial_data
self.dirty = initial_data
if not self.cache_only:
self._flush_cache() #inserts new entries
else:
self.cache = {}
self.dirty = {}
self.cache_reads = cache_reads
if not self.cache_only:
self.populate_from_database()
@staticmethod
def create_from_serialised(ser_str, docid = None, version = None, docformat = None,
relation = None, cache_only = False, cache_reads = True):
"""Creates an instance of MoreInfo
using serialised data as the cache content"""
data = cPickle.loads(base64.b64decode(ser_str))
return MoreInfo(docid = docid, version = version, docformat = docformat,
relation = relation, cache_only = cache_only,
cache_reads = cache_reads, initial_data = data);
def serialise_cache(self):
"""Returns a serialised representation of the cache"""
return base64.b64encode(cPickle.dumps(self.get_cache()))
def populate_from_database(self):
"""Retrieves all values of MoreInfo and places them in the cache"""
where_str, where_args = self._generate_where_query_args()
query_str = "SELECT namespace, data_key, data_value FROM bibdocmoreinfo WHERE %s" % (where_str, )
res = run_sql(query_str, where_args)
if res:
for row in res:
namespace, data_key, data_value_ser = row
data_value = cPickle.loads(data_value_ser)
if not namespace in self.cache:
self.cache[namespace] = {}
self.cache[namespace][data_key] = data_value
def _mark_dirty(self, namespace, data_key):
"""Marks a data key dirty - that should be saved into the database"""
if not namespace in self.dirty:
self.dirty[namespace] = {}
self.dirty[namespace][data_key] = True
def _database_get_distinct_string_list(self, column, namespace = None):
"""A private method reading an unique list of strings from the
moreinfo database table"""
where_str, where_args = self._generate_where_query_args(
namespace = namespace)
query_str = "SELECT DISTINCT %s FROM bibdocmoreinfo WHERE %s" % \
( column, where_str, )
if DBG_LOG_QUERIES:
from invenio.legacy.bibsched.bibtask import write_message
write_message("Executing query: " + query_str + " ARGS: " + repr(where_args))
print("Executing query: " + query_str + " ARGS: " + repr(where_args))
res = run_sql(query_str, where_args)
return (res and [x[0] for x in res]) or [] # after migrating to python 2.6, can be rewritten using x if y else z syntax: return [x[0] for x in res] if res else []
def _database_get_namespaces(self):
"""Read the database to discover namespaces declared in a given MoreInfo"""
return self._database_get_distinct_string_list("namespace")
def _database_get_keys(self, namespace):
"""Returns all keys assigned in a given namespace of a MoreInfo instance"""
return self._database_get_distinct_string_list("data_key", namespace=namespace)
def _database_contains_key(self, namespace, key):
return self._database_read_value(namespace, key) != None
def _database_save_value(self, namespace, key, value):
"""Write changes into the database"""
#TODO: this should happen within one transaction
serialised_val = cPickle.dumps(value)
# on duplicate key will not work here as miltiple null values are permitted by the index
if not self._database_contains_key(namespace, key):
#insert new value
query_parts = []
query_args = []
to_process = [(self.docid, "id_bibdoc"), (self.version, "version"),
(self.format, "format"), (self.relation, "id_rel"),
(str(namespace), "namespace"), (str(key), "data_key"),
(str(serialised_val), "data_value")]
for entry in to_process:
_val_or_null(entry[0], q_str = query_parts, q_args = query_args)
columns_str = ", ".join(map(lambda x: x[1], to_process))
values_str = ", ".join(query_parts)
query_str = "INSERT INTO bibdocmoreinfo (%s) VALUES(%s)" % \
(columns_str, values_str)
if DBG_LOG_QUERIES:
from invenio.legacy.bibsched.bibtask import write_message
write_message("Executing query: " + query_str + " ARGS: " + repr(query_args))
print("Executing query: " + query_str + " ARGS: " + repr(query_args))
run_sql(query_str, query_args)
else:
#Update existing value
where_str, where_args = self._generate_where_query_args(namespace, key)
query_str = "UPDATE bibdocmoreinfo SET data_value=%s WHERE " + where_str
query_args = [str(serialised_val)] + where_args
if DBG_LOG_QUERIES:
from invenio.legacy.bibsched.bibtask import write_message
write_message("Executing query: " + query_str + " ARGS: " + repr(query_args))
print("Executing query: " + query_str + " ARGS: " + repr(query_args))
run_sql(query_str, query_args )
def _database_read_value(self, namespace, key):
"""Reads a value directly from the database
@param namespace - namespace of the data to be read
@param key - key of the data to be read
"""
where_str, where_args = self._generate_where_query_args(namespace = namespace, data_key = key)
query_str = "SELECT data_value FROM bibdocmoreinfo WHERE " + where_str
res = run_sql(query_str, where_args)
if DBG_LOG_QUERIES:
from invenio.legacy.bibsched.bibtask import write_message
write_message("Executing query: " + query_str + " ARGS: " + repr(where_args) + "WITH THE RESULT: " + str(res))
s_ = ""
if res:
s_ = cPickle.loads(res[0][0])
print("Executing query: " + query_str + " ARGS: " + repr(where_args) + " WITH THE RESULT: " + str(s_))
if res and res[0][0]:
try:
return cPickle.loads(res[0][0])
except:
raise Exception("Error when deserialising value for %s key=%s retrieved value=%s" % (repr(self), str(key), str(res[0][0])))
return None
def _database_remove_value(self, namespace, key):
"""Removes an entry directly in the database"""
where_str, where_args = self._generate_where_query_args(namespace = namespace, data_key = key)
query_str = "DELETE FROM bibdocmoreinfo WHERE " + where_str
if DBG_LOG_QUERIES:
from invenio.legacy.bibsched.bibtask import write_message
write_message("Executing query: " + query_str + " ARGS: " + repr(where_args))
print("Executing query: " + query_str + " ARGS: " + repr(where_args))
run_sql(query_str, where_args)
return None
def _flush_cache(self):
"""Writes all the dirty cache entries into the database"""
for namespace in self.dirty:
for data_key in self.dirty[namespace]:
if namespace in self.cache and data_key in self.cache[namespace]\
and not self.cache[namespace][data_key] is None:
self._database_save_value(namespace, data_key, self.cache[namespace][data_key])
else:
# This might happen if a value has been removed from the cache
self._database_remove_value(namespace, data_key)
self.dirty = {}
def _generate_where_query_args(self, namespace = None, data_key = None):
"""Private method generating WHERE clause of SQL statements"""
ns = []
if namespace != None:
ns = [(namespace, "namespace")]
dk = []
if data_key != None:
dk = [(data_key, "data_key")]
to_process = [(self.docid, "id_bibdoc"), (self.version, "version"),
(self.format, "format"), (self.relation, "id_rel")] + \
ns + dk
return _sql_generate_conjunctive_where(to_process)
def set_data(self, namespace, key, value):
"""setting data directly in the database dictionary"""
if not namespace in self.cache:
self.cache[namespace] = {}
self.cache[namespace][key] = value
self._mark_dirty(namespace, key)
if not self.cache_only:
self._flush_cache()
def get_data(self, namespace, key):
"""retrieving data from the database"""
if self.cache_reads or self.cache_only:
if namespace in self.cache and key in self.cache[namespace]:
return self.cache[namespace][key]
if not self.cache_only:
# we have a permission to read from the database
value = self._database_read_value(namespace, key)
if value:
if not namespace in self.cache:
self.cache[namespace] = {}
self.cache[namespace][key] = value
return value
return None
def del_key(self, namespace, key):
"""retrieving data from the database"""
if not namespace in self.cache:
return None
del self.cache[namespace][key]
self._mark_dirty(namespace, key)
if not self.cache_only:
self._flush_cache()
def contains_key(self, namespace, key):
return self.get_data(namespace, key) != None
# the dictionary interface -> updating the default namespace
def __setitem__(self, key, value):
self.set_data("", key, value) #the default value
def __getitem__(self, key):
return self.get_data("", key)
def __delitem__(self, key):
self.del_key("", key)
def __contains__(self, key):
return self.contains_key("", key)
def __repr__(self):
return "MoreInfo(docid=%s, version=%s, docformat=%s, relation=%s)" % \
(self.docid, self.version, self.format, self.relation)
def delete(self):
"""Remove all entries associated with this MoreInfo"""
self.cache = {}
if not self.cache_only:
where_str, query_args = self._generate_where_query_args()
query_str = "DELETE FROM bibdocmoreinfo WHERE %s" % (where_str, )
if DBG_LOG_QUERIES:
from invenio.legacy.bibsched.bibtask import write_message
write_message("Executing query: " + query_str + " ARGS: " + repr(query_args))
print("Executing query: " + query_str + " ARGS: " + repr(query_args))
run_sql(query_str, query_args)
def get_cache(self):
"""Returns the content of the cache
@return The content of the MoreInfo cache
@rtype dictionary {namespace: {key1: value1, ... }, namespace2: {}}
"""
return self.cache
def get_namespaces(self):
"""Returns a list of namespaces present in the MoreInfo structure.
If the object is permitted access to the database, the data should
be always read from there. Unlike when reading a particular value,
we can not check if value is missing in the cache
"""
if self.cache_only and self.cache_reads:
return self.cache.keys()
return self._database_get_namespaces()
def get_keys(self, namespace):
"""Returns a list of keys present in a given namespace"""
if self.cache_only and self.cache_reads:
res = []
if namespace in self.cache:
res = self.cache[namespace].keys()
return res
else:
return self._database_get_keys(namespace)
def flush(self):
"""Flush the content into the database"""
self._flush_cache()
class BibDocMoreInfo(MoreInfo):
"""
This class wraps contextual information of the documents, such as the
- comments
- descriptions
- flags.
Such information is kept separately per every format/version instance of
the corresponding document and is searialized in the database, ready
to be retrieved (but not searched).
@param docid: the document identifier.
@type docid: integer
@param more_info: a serialized version of an already existing more_info
object. If not specified this information will be readed from the
database, and othewise an empty dictionary will be allocated.
@raise ValueError: if docid is not a positive integer.
@ivar docid: the document identifier as passed to the constructor.
@type docid: integer
@ivar more_info: the more_info dictionary that will hold all the
additional document information.
@type more_info: dict of dict of dict
@note: in general this class is never instanciated in client code and
never used outside bibdocfile module.
@note: this class will be extended in the future to hold all the new auxiliary
information about a document.
"""
def __init__(self, docid, cache_only = False, initial_data = None):
if not (type(docid) in (long, int) and docid > 0):
raise ValueError("docid is not a positive integer, but %s." % docid)
MoreInfo.__init__(self, docid, cache_only = cache_only, initial_data = initial_data)
if 'descriptions' not in self:
self['descriptions'] = {}
if 'comments' not in self:
self['comments'] = {}
if 'flags' not in self:
self['flags'] = {}
if DBG_LOG_QUERIES:
from invenio.legacy.bibsched.bibtask import write_message
write_message("Creating BibDocMoreInfo :" + repr(self["comments"]))
print("Creating BibdocMoreInfo :" + repr(self["comments"]))
def __repr__(self):
"""
@return: the canonical string representation of the C{BibDocMoreInfo}.
@rtype: string
"""
return 'BibDocMoreInfo(%i, %s)' % (self.docid, repr(cPickle.dumps(self)))
def set_flag(self, flagname, docformat, version):
"""
Sets a flag.
@param flagname: the flag to set (see
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS}).
@type flagname: string
@param format: the format for which the flag should set.
@type format: string
@param version: the version for which the flag should set:
@type version: integer
@raise ValueError: if the flag is not in
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS}
"""
if flagname in CFG_BIBDOCFILE_AVAILABLE_FLAGS:
flags = self['flags']
if not flagname in flags:
flags[flagname] = {}
if not version in flags[flagname]:
flags[flagname][version] = {}
if not docformat in flags[flagname][version]:
flags[flagname][version][docformat] = {}
flags[flagname][version][docformat] = True
self['flags'] = flags
else:
raise ValueError, "%s is not in %s" % \
(flagname, CFG_BIBDOCFILE_AVAILABLE_FLAGS)
def get_comment(self, docformat, version):
"""
Returns the specified comment.
@param format: the format for which the comment should be
retrieved.
@type format: string
@param version: the version for which the comment should be
retrieved.
@type version: integer
@return: the specified comment.
@rtype: string
"""
try:
assert(type(version) is int)
docformat = normalize_format(docformat)
return self['comments'].get(version, {}).get(docformat)
except:
register_exception()
raise
def get_description(self, docformat, version):
"""
Returns the specified description.
@param format: the format for which the description should be
retrieved.
@type format: string
@param version: the version for which the description should be
retrieved.
@type version: integer
@return: the specified description.
@rtype: string
"""
try:
assert(type(version) is int)
docformat = normalize_format(docformat)
return self['descriptions'].get(version, {}).get(docformat)
except:
register_exception()
raise
def has_flag(self, flagname, docformat, version):
"""
Return True if the corresponding has been set.
@param flagname: the name of the flag (see
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS}).
@type flagname: string
@param format: the format for which the flag should be checked.
@type format: string
@param version: the version for which the flag should be checked.
@type version: integer
@return: True if the flag is set for the given format/version.
@rtype: bool
@raise ValueError: if the flagname is not in
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS}
"""
if flagname in CFG_BIBDOCFILE_AVAILABLE_FLAGS:
return self['flags'].get(flagname, {}).get(version, {}).get(docformat, False)
else:
raise ValueError, "%s is not in %s" % (flagname, CFG_BIBDOCFILE_AVAILABLE_FLAGS)
def get_flags(self, docformat, version):
"""
Return the list of all the enabled flags.
@param format: the format for which the list should be returned.
@type format: string
@param version: the version for which the list should be returned.
@type version: integer
@return: the list of enabled flags (from
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS}).
@rtype: list of string
"""
return [flag for flag in self['flags'] if docformat in self['flags'][flag].get(version, {})]
def set_comment(self, comment, docformat, version):
"""
Set a comment.
@param comment: the comment to be set.
@type comment: string
@param format: the format for which the comment should be set.
@type format: string
@param version: the version for which the comment should be set:
@type version: integer
"""
try:
assert(type(version) is int and version > 0)
docformat = normalize_format(docformat)
if comment == KEEP_OLD_VALUE:
comment = self.get_comment(docformat, version) or self.get_comment(docformat, version - 1)
if not comment:
self.unset_comment(docformat, version)
return
if not version in self['comments']:
comments = self['comments']
comments[version] = {}
self['comments'] = comments
comments = self['comments']
comments[version][docformat] = comment
self['comments'] = comments
except:
register_exception()
raise
def set_description(self, description, docformat, version):
"""
Set a description.
@param description: the description to be set.
@type description: string
@param format: the format for which the description should be set.
@type format: string
@param version: the version for which the description should be set:
@type version: integer
"""
try:
assert(type(version) is int and version > 0)
docformat = normalize_format(docformat)
if description == KEEP_OLD_VALUE:
description = self.get_description(docformat, version) or self.get_description(docformat, version - 1)
if not description:
self.unset_description(docformat, version)
return
descriptions = self['descriptions']
if not version in descriptions:
descriptions[version] = {}
descriptions[version][docformat] = description
self.set_data("", 'descriptions', descriptions)
except:
register_exception()
raise
def unset_comment(self, docformat, version):
"""
Unset a comment.
@param format: the format for which the comment should be unset.
@type format: string
@param version: the version for which the comment should be unset:
@type version: integer
"""
try:
assert(type(version) is int and version > 0)
comments = self['comments']
del comments[version][docformat]
self['comments'] = comments
except KeyError:
pass
except:
register_exception()
raise
def unset_description(self, docformat, version):
"""
Unset a description.
@param format: the format for which the description should be unset.
@type format: string
@param version: the version for which the description should be unset:
@type version: integer
"""
try:
assert(type(version) is int and version > 0)
descriptions = self['descriptions']
del descriptions[version][docformat]
self['descriptions'] = descriptions
except KeyError:
pass
except:
register_exception()
raise
def unset_flag(self, flagname, docformat, version):
"""
Unset a flag.
@param flagname: the flag to be unset (see
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS}).
@type flagname: string
@param format: the format for which the flag should be unset.
@type format: string
@param version: the version for which the flag should be unset:
@type version: integer
@raise ValueError: if the flag is not in
L{CFG_BIBDOCFILE_AVAILABLE_FLAGS}
"""
if flagname in CFG_BIBDOCFILE_AVAILABLE_FLAGS:
try:
flags = self['flags']
del flags[flagname][version][docformat]
self['flags'] = flags
except KeyError:
pass
else:
raise ValueError, "%s is not in %s" % (flagname, CFG_BIBDOCFILE_AVAILABLE_FLAGS)
_bib_relation__any_value = -1
class BibRelation(object):
"""
A representation of a relation between documents or their particular versions
"""
def __init__(self, rel_type = None,
bibdoc1_id = None, bibdoc2_id = None,
bibdoc1_ver = None, bibdoc2_ver = None,
bibdoc1_fmt = None, bibdoc2_fmt = None,
rel_id = None):
"""
The constructor of the class representing a relation between two
documents.
If the more_info parameter is specified, no data is retrieved from
the database and the internal dictionary is initialised with
the passed value. If the more_info is not provided, the value is
read from the database. In the case of non-existing record, an
empty dictionary is assigned.
If a version of whichever record is not specified, the resulting
object desctibes a relation of all version of a given BibDoc.
@param bibdoc1
@type bibdoc1 BibDoc
@param bibdoc1_ver
@type version1_ver int
@param bibdoc2
@type bibdoc2 BibDco
@param bibdoc2_ver
@type bibdoc2_ver int
@param bibdoc1_fmt format of the first document
@type bibdoc1_fmt string
@param bibdoc2_fmt format of the second document
@type bibdoc2_fmt string
@param rel_type
@type rel_type string
@param more_info The serialised representation of the more_info
@type more_info string
@param rel_id allows to specify the identifier of the newly created relation
@type rel_ide unsigned int
"""
self.id = rel_id
self.bibdoc1_id = bibdoc1_id
self.bibdoc2_id = bibdoc2_id
self.bibdoc1_ver = bibdoc1_ver
self.bibdoc2_ver = bibdoc2_ver
self.bibdoc1_fmt = bibdoc1_fmt
self.bibdoc2_fmt = bibdoc2_fmt
self.rel_type = rel_type
if rel_id == None:
self._fill_id_from_data()
else:
self._fill_data_from_id()
self.more_info = MoreInfo(relation = self.id)
def _fill_data_from_id(self):
"""Fill all the relation data from the relation identifier
"""
query = "SELECT id_bibdoc1, version1, format1, id_bibdoc2, version2, format2, rel_type FROM bibdoc_bibdoc WHERE id=%s"
res = run_sql(query, (str(self.id), ))
if res != None and res[0] != None:
self.bibdoc1_id = res[0][0]
self.bibdoc1_ver = res[0][1]
self.bibdoc1_fmt = res[0][2]
self.bibdoc2_id = res[0][3]
self.bibdoc2_ver = res[0][4]
self.bibdoc2_fmt = res[0][5]
self.rel_type = res[0][6]
def _fill_id_from_data(self):
"""Fill the relation identifier based on the data provided"""
where_str, where_args = self._get_where_clauses()
query = "SELECT id FROM bibdoc_bibdoc WHERE %s" % (where_str, )
res = run_sql(query, where_args)
if res and res[0][0]:
self.id = int(res[0][0])
def _get_value_column_mapping(self):
"""
Returns a list of tuples each tuple consists of a value and a name
of a database column where this value should fit
"""
return [(self.rel_type, "rel_type"), (self.bibdoc1_id, "id_bibdoc1"),
(self.bibdoc1_ver, "version1"),
(self.bibdoc1_fmt, "format1"),
(self.bibdoc2_id, "id_bibdoc2"),
(self.bibdoc2_ver, "version2"),
(self.bibdoc2_fmt, "format2")]
def _get_where_clauses(self):
"""Private function returning part of the SQL statement identifying
current relation
@return
@rtype tuple
"""
return _sql_generate_conjunctive_where(self._get_value_column_mapping())
@staticmethod
def create(bibdoc1_id = None, bibdoc1_ver = None,
bibdoc1_fmt = None, bibdoc2_id = None,
bibdoc2_ver = None, bibdoc2_fmt = None,
rel_type = ""):
"""
Create a relation and return instance.
Ommiting an argument means that a particular relation concerns any value of the parameter
"""
# check if there is already entry corresponding to parameters
existing = BibRelation.get_relations(rel_type = rel_type,
bibdoc1_id = bibdoc1_id,
bibdoc2_id = bibdoc2_id,
bibdoc1_ver = bibdoc1_ver,
bibdoc2_ver = bibdoc2_ver,
bibdoc1_fmt = bibdoc1_fmt,
bibdoc2_fmt = bibdoc2_fmt)
if len(existing) > 0:
return existing[0]
# build the insert query and execute it
to_process = [(rel_type, "rel_type"), (bibdoc1_id, "id_bibdoc1"),
(bibdoc1_ver, "version1"), (bibdoc1_fmt, "format1"),
(bibdoc2_id, "id_bibdoc2"), (bibdoc2_ver, "version2"),
(bibdoc2_fmt, "format2")]
values_list = []
args_list = []
columns_list = []
for entry in to_process:
columns_list.append(entry[1])
if entry[0] == None:
values_list.append("NULL")
else:
values_list.append("%s")
args_list.append(entry[0])
query = "INSERT INTO bibdoc_bibdoc (%s) VALUES (%s)" % (", ".join(columns_list), ", ".join(values_list))
# print "Query: %s Args: %s" % (query, str(args_list))
rel_id = run_sql(query, args_list)
return BibRelation(rel_id = rel_id)
def delete(self):
""" Removes a relation between objects from the database.
executing the flush function on the same object will restore
the relation
"""
where_str, where_args = self._get_where_clauses()
run_sql("DELETE FROM bibdoc_bibdoc WHERE %s" % (where_str,), where_args) # kwalitee: disable=sql
# removing associated MoreInfo
self.more_info.delete()
def get_more_info(self):
return self.more_info
@staticmethod
def get_relations(rel_type = _bib_relation__any_value,
bibdoc1_id = _bib_relation__any_value,
bibdoc2_id = _bib_relation__any_value,
bibdoc1_ver = _bib_relation__any_value,
bibdoc2_ver = _bib_relation__any_value,
bibdoc1_fmt = _bib_relation__any_value,
bibdoc2_fmt = _bib_relation__any_value):
"""Retrieves list of relations satisfying condtions.
If a parameter is specified, its value has to match exactly.
If a parameter is ommited, any of its values will be accepted"""
to_process = [(rel_type, "rel_type"), (bibdoc1_id, "id_bibdoc1"),
(bibdoc1_ver, "version1"), (bibdoc1_fmt, "format1"),
(bibdoc2_id, "id_bibdoc2"), (bibdoc2_ver, "version2"),
(bibdoc2_fmt, "format2")]
where_str, where_args = _sql_generate_conjunctive_where(
filter(lambda x: x[0] != _bib_relation__any_value, to_process))
if where_str:
where_str = "WHERE " + where_str # in case of nonempty where, we need a where clause
query_str = "SELECT id FROM bibdoc_bibdoc %s" % (where_str, )
# print "running query : %s with arguments %s on the object %s" % (query_str, str(where_args), repr(self))
try:
res = run_sql(query_str, where_args)
except:
raise Exception(query_str + " " + str(where_args))
results = []
if res != None:
for res_row in res:
results.append(BibRelation(rel_id=res_row[0]))
return results
# Access to MoreInfo
def set_data(self, category, key, value):
"""assign additional information to this relation"""
self.more_info.set_data(category, key, value)
def get_data(self, category, key):
"""read additional information assigned to this relation"""
return self.more_info.get_data(category, key)
#the dictionary interface allowing to set data bypassing the namespaces
def __setitem__(self, key, value):
self.more_info[key] = value
def __getitem__(self, key):
return self.more_info[key]
def __contains__(self, key):
return self.more_info.__contains__(key)
def __repr__(self):
return "BibRelation(id_bibdoc1 = %s, version1 = %s, format1 = %s, id_bibdoc2 = %s, version2 = %s, format2 = %s, rel_type = %s)" % \
(self.bibdoc1_id, self.bibdoc1_ver, self.bibdoc1_fmt,
self.bibdoc2_id, self.bibdoc2_ver, self.bibdoc2_fmt,
self.rel_type)
def readfile(filename):
"""
Read a file.
@param filename: the name of the file to be read.
@type filename: string
@return: the text contained in the file.
@rtype: string
@note: Returns empty string in case of any error.
@note: this function is useful for quick implementation of websubmit
functions.
"""
try:
return open(filename).read()
except Exception:
return ''
class HeadRequest(urllib2.Request):
"""
A request object to perform a HEAD request.
"""
def get_method(self):
return 'HEAD'
def read_cookie(cookiefile):
"""
Parses a cookie file and returns a string as needed for the urllib2 headers
The file should respect the Netscape cookie specifications
"""
cookie_data = ''
cfile = open(cookiefile, 'r')
for line in cfile.readlines():
tokens = line.split('\t')
if len(tokens) == 7: # we are on a cookie line
cookie_data += '%s=%s; ' % (tokens[5], tokens[6].replace('\n', ''))
cfile.close()
return cookie_data
def open_url(url, headers=None, head_request=False):
"""
Opens a URL. If headers are passed as argument, no check is performed and
the URL will be opened. Otherwise checks if the URL is present in
CFG_BIBUPLOAD_FFT_ALLOWED_EXTERNAL_URLS and uses the headers specified in
the config variable.
@param url: the URL to open
@type url: string
@param headers: the headers to use
@type headers: dictionary
@param head_request: if True, perform a HEAD request, otherwise a POST
request
@type head_request: boolean
@return: a file-like object as returned by urllib2.urlopen.
"""
headers_to_use = None
if headers is None:
for regex, headers in _CFG_BIBUPLOAD_FFT_ALLOWED_EXTERNAL_URLS:
if regex.match(url) is not None:
headers_to_use = headers
break
if headers_to_use is None:
# URL is not allowed.
raise InvenioBibdocfileUnauthorizedURL, "%s is not an authorized " \
"external URL." % url
else:
headers_to_use = headers
request_obj = head_request and HeadRequest or urllib2.Request
request = request_obj(url)
request.add_header('User-Agent', make_user_agent_string('bibdocfile'))
for key, value in headers_to_use.items():
try:
value = globals()[value['fnc']](**value['args'])
except (KeyError, TypeError):
pass
request.add_header(key, value)
return urllib2.urlopen(request)
def update_modification_date_of_file(filepath, modification_date):
"""Update the modification time and date of the file with the modification_date
@param filepath: the full path of the file that needs to be updated
@type filepath: string
@param modification_date: the new modification date and time
@type modification_date: datetime.datetime object
"""
try:
modif_date_in_seconds = time.mktime(modification_date.timetuple()) # try to get the time in seconds
except (AttributeError, TypeError):
modif_date_in_seconds = 0
if modif_date_in_seconds:
statinfo = os.stat(filepath) # we need to keep the same access time
os.utime(filepath, (statinfo.st_atime, modif_date_in_seconds)) #update the modification time
diff --git a/invenio/legacy/bibdocfile/managedocfiles.py b/invenio/legacy/bibdocfile/managedocfiles.py
index 5eadb94bf..3bdd3c3ba 100644
--- a/invenio/legacy/bibdocfile/managedocfiles.py
+++ b/invenio/legacy/bibdocfile/managedocfiles.py
@@ -1,2938 +1,2970 @@
## $Id: Revise_Files.py,v 1.37 2009/03/26 15:11:05 jerome Exp $
## This file is part of Invenio.
-## Copyright (C) 2010, 2011, 2012, 2013 CERN.
+## Copyright (C) 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
BibDocFile Upload File Interface utils
=====================================
Tools to help with creation of file management interfaces.
Contains the two main functions `create_file_upload_interface' and
`move_uploaded_files_to_storage', which must be run one after the
other:
- create_file_upload_interface: Generates the HTML of an interface to
revise files of a given record. The actions on the files are
recorded in a working directory, but not applied to the record.
- move_uploaded_files_to_storage: Applies/executes the modifications
on files as recorded by the `create_file_upload_interface'
function.
Theses functions are a complex interplay of HTML, Javascript and HTTP
requests. They are not meant to be used in any type of scenario, but
require to be used in extremely specific contexts (Currently in
WebSubmit Response Elements, WebSubmit functions and the BibDocFile
File Management interface).
NOTES:
======
- Comments are not considered as a property of bibdocfiles, but
bibdocs: this conflicts with the APIs
FIXME:
======
- refactor into smaller components. Eg. form processing in
create_file_upload_interface could be move outside the function.
- better differentiate between revised file, and added format
(currently when adding a format, the whole bibdoc is marked as
updated, and all links are removed)
- After a file has been revised or added, add a 'check' icon
- One issue: if we allow deletion or renaming, we might lose track of
a bibdoc: someone adds X, renames X->Y, and adds again another file
with name X: when executing actions, we will add the second X, and
rename it to Y
-> need to go back in previous action when renaming... or check
that name has never been used..
DEPENDENCIES:
=============
- jQuery Form plugin U{http://jquery.malsup.com/form/}
"""
from six.moves import cPickle
import os
import time
import cgi
from urllib import urlencode
from invenio.config import \
CFG_SITE_LANG, \
CFG_SITE_URL, \
CFG_WEBSUBMIT_STORAGEDIR, \
CFG_TMPSHAREDDIR, \
CFG_SITE_SUPPORT_EMAIL, \
CFG_CERN_SITE, \
CFG_SITE_RECORD
from invenio.base.i18n import gettext_set_language
from invenio.legacy.bibdocfile.cli import cli_fix_marc
from invenio.legacy.bibdocfile.api import BibRecDocs, \
decompose_file, calculate_md5, BibDocFile, \
InvenioBibDocFileError, BibDocMoreInfo
from invenio.legacy.websubmit.functions.Shared_Functions import \
createRelatedFormats
-from invenio.ext.logging import register_exception
-from invenio.legacy.dbquery import run_sql
from invenio.legacy.websubmit.icon_creator import \
create_icon, InvenioWebSubmitIconCreatorError
+from invenio.legacy.websubmit.file_converter import get_missing_formats
+from invenio.ext.logging import register_exception
+from invenio.legacy.dbquery import run_sql
from invenio.utils.url import create_html_mailto
from invenio.utils.html import escape_javascript_string
-from invenio.legacy.bibdocfile.config import CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT
-
+from invenio.bibtask import task_low_level_submission, bibtask_allocate_sequenceid
CFG_ALLOWED_ACTIONS = ['revise', 'delete', 'add', 'addFormat']
params_id = 0
def create_file_upload_interface(recid,
form=None,
print_outside_form_tag=True,
print_envelope=True,
include_headers=False,
ln=CFG_SITE_LANG,
minsize='', maxsize='',
doctypes_and_desc=None,
can_delete_doctypes=None,
can_revise_doctypes=None,
can_describe_doctypes=None,
can_comment_doctypes=None,
can_keep_doctypes=None,
can_rename_doctypes=None,
can_add_format_to_doctypes=None,
create_related_formats=True,
can_name_new_files=True,
keep_default=True, show_links=True,
file_label=None, filename_label=None,
description_label=None, comment_label=None,
restrictions_and_desc=None,
can_restrict_doctypes=None,
restriction_label=None,
doctypes_to_default_filename=None,
max_files_for_doctype=None,
sbm_indir=None, sbm_doctype=None, sbm_access=None,
uid=None, sbm_curdir=None,
- display_hidden_files=False, protect_hidden_files=True):
+ display_hidden_files=False, protect_hidden_files=True,
+ defer_related_formats_creation=True):
"""
Returns the HTML for the file upload interface.
@param recid: the id of the record to edit files
@type recid: int or None
@param form: the form sent by the user's browser in response to a
user action. This is used to read and record user's
actions.
@param form: as returned by the interface handler.
@param print_outside_form_tag: display encapsulating <form> tag or
not
@type print_outside_form_tag: boolean
@param print_envelope: (internal parameter) if True, return the
encapsulating initial markup, otherwise
skip it.
@type print_envelope: boolean
@param include_headers: include javascript and css headers in the
body of the page. If you set this to
False, you must take care of including
these headers in your page header. Setting
this parameter to True is useful if you
cannot change the page header.
@type include_headers: boolean
@param ln: language
@type ln: string
@param minsize: the minimum size (in bytes) allowed for the
uploaded files. Files not big enough are
discarded.
@type minsize: int
@param maxsize: the maximum size (in bytes) allowed for the
uploaded files. Files too big are discarded.
@type maxsize: int
@param doctypes_and_desc: the list of doctypes (like 'Main' or
'Additional') and their description that users
can choose from when adding new files.
- When no value is provided, users cannot add new
file (they can only revise/delete/add format)
- When a single value is given, it is used as
default doctype for all new documents
Order is relevant
Eg:
[('main', 'Main document'), ('additional', 'Figure, schema. etc')]
@type doctypes_and_desc: list(tuple(string, string))
@param restrictions_and_desc: the list of restrictions (like 'Restricted' or
'No Restriction') and their description that
users can choose from when adding or revising
files. Restrictions can then be configured at
the level of WebAccess.
- When no value is provided, no restriction is
applied
- When a single value is given, it is used as
default resctriction for all documents.
- The first value of the list is used as default
restriction if the user if not given the
choice of the restriction. Order is relevant
Eg:
[('', 'No restriction'), ('restr', 'Restricted')]
@type restrictions_and_desc: list(tuple(string, string))
@param can_delete_doctypes: the list of doctypes that users are
allowed to delete.
Eg: ['main', 'additional']
Use ['*'] for "all doctypes"
@type can_delete_doctypes: list(string)
@param can_revise_doctypes: the list of doctypes that users are
allowed to revise
Eg: ['main', 'additional']
Use ['*'] for "all doctypes"
@type can_revise_doctypes: list(string)
@param can_describe_doctypes: the list of doctypes that users are
allowed to describe
Eg: ['main', 'additional']
Use ['*'] for "all doctypes"
@type can_describe_doctypes: list(string)
@param can_comment_doctypes: the list of doctypes that users are
allowed to comment
Eg: ['main', 'additional']
Use ['*'] for "all doctypes"
@type can_comment_doctypes: list(string)
@param can_keep_doctypes: the list of doctypes for which users can
choose to keep previous versions visible when
revising a file (i.e. 'Keep previous version'
checkbox). See also parameter 'keepDefault'.
Note that this parameter is ~ignored when
revising the attributes of a file (comment,
description) without uploading a new
file. See also parameter
Move_Uploaded_Files_to_Storage.force_file_revision
Eg: ['main', 'additional']
Use ['*'] for "all doctypes"
@type can_keep_doctypes: list(string)
@param can_add_format_to_doctypes: the list of doctypes for which users can
add new formats. If there is no value,
then no 'add format' link nor warning
about losing old formats are displayed.
Eg: ['main', 'additional']
Use ['*'] for "all doctypes"
@type can_add_format_to_doctypes: list(string)
@param can_restrict_doctypes: the list of doctypes for which users can
choose the access restrictions when adding or
revising a file. If no value is given:
- no restriction is applied if none is defined
in the 'restrictions' parameter.
- else the *first* value of the 'restrictions'
parameter is used as default restriction.
Eg: ['main', 'additional']
Use ['*'] for "all doctypes"
@type can_restrict_doctypes : list(string)
@param can_rename_doctypes: the list of doctypes that users are allowed
to rename (when revising)
Eg: ['main', 'additional']
Use ['*'] for "all doctypes"
@type can_rename_doctypes: list(string)
@param can_name_new_files: if user can choose the name of the files they
upload or not
@type can_name_new_files: boolean
@param doctypes_to_default_filename: Rename uploaded files to admin-chosen
values. To rename to a value found in a file in curdir,
use 'file:' prefix to specify the file to read from.
Eg:
{'main': 'file:RN', 'additional': 'foo'}
If the same doctype is submitted
several times, a"-%i" suffix is added
to the name defined in the file.
When using 'file:' prefix, the name
is only resolved at the end of the
submission, when attaching the file.
The default filenames are overriden
by user-chosen names if you allow
'can_name_new_files' or
'can_rename_doctypes', excepted if the
name is prefixed with 'file:'.
@type doctypes_to_default_filename: dict
@param max_files_for_doctype: the maximum number of files that users can
upload for each doctype.
Eg: {'main': 1, 'additional': 2}
Do not specify the doctype here to have an
unlimited number of files for a given
doctype.
@type max_files_for_doctype: dict
@param create_related_formats: if uploaded files get converted to
whatever format we can or not
@type create_related_formats: boolean
@param keep_default: the default behaviour for keeping or not previous
version of files when users cannot choose (no
value in can_keep_doctypes).
Note that this parameter is ignored when revising
the attributes of a file (comment, description)
without uploading a new file. See also parameter
Move_Uploaded_Files_to_Storage.force_file_revision
@type keep_default: boolean
@param show_links: if we display links to files when possible or
not
@type show_links: boolean
@param file_label: the label for the file field
@type file_label: string
@param filename_label: the label for the file name field
@type filename_label: string
@param description_label: the label for the description field
@type description_label: string
@param comment_label: the label for the comments field
@type comment_label: string
@param restriction_label: the label in front of the restrictions list
@type restriction_label: string
@param sbm_indir: the submission indir parameter, in case the
function is used in a WebSubmit submission
context.
This value will be used to retrieve where to
read the current state of the interface and
store uploaded files
@type sbm_indir : string
@param sbm_doctype: the submission doctype parameter, in case the
function is used in a WebSubmit submission
context.
This value will be used to retrieve where to
read the current state of the interface and
store uploaded files
@type sbm_doctype: string
@param sbm_access: the submission access parameter. Must be
specified in the context of WebSubmit
submission, as well when used in the
WebSubmit Admin file management interface.
This value will be used to retrieve where to
read the current state of the interface and
store uploaded files
@type sbm_access: string
@param sbm_curdir: the submission curdir parameter. Must be
specified in the context of WebSubmit
function Create_Upload_File_Interface.
This value will be used to retrieve where to
read the current state of the interface and
store uploaded files.
@type sbm_curdir: string
@param uid: the user id
@type uid: int
@param display_hidden_files: if bibdoc containing bibdocfiles
flagged as 'HIDDEN' should be
displayed or not.
@type display_hidden_files: boolean
@param protect_hidden_files: if bibdoc containing bibdocfiles
flagged as 'HIDDEN' can be edited
(revise, delete, add format) or not.
@type protect_hidden_files: boolean
+ @param defer_related_formats_creation: should the creation of
+ "related formats" (See
+ C{create_related_formats})
+ be created at offline at a
+ later stage (default) or
+ immediately after upload
+ (False)?
+ @type defer_related_formats_creation: boolean
+
@return Tuple (errorcode, html)
"""
# Clean and set up a few parameters
_ = gettext_set_language(ln)
body = ''
if not file_label:
file_label = _('Choose a file')
if not filename_label:
filename_label = _('Name')
if not description_label:
description_label = _('Description')
if not comment_label:
comment_label = _('Comment')
if not restriction_label:
restriction_label = _('Access')
if not doctypes_and_desc:
doctypes_and_desc = []
if not can_delete_doctypes:
can_delete_doctypes = []
if not can_revise_doctypes:
can_revise_doctypes = []
if not can_describe_doctypes:
can_describe_doctypes = []
if not can_comment_doctypes:
can_comment_doctypes = []
if not can_keep_doctypes:
can_keep_doctypes = []
if not can_rename_doctypes:
can_rename_doctypes = []
if not can_add_format_to_doctypes:
can_add_format_to_doctypes = []
if not restrictions_and_desc:
restrictions_and_desc = []
if not can_restrict_doctypes:
can_restrict_doctypes = []
if not doctypes_to_default_filename:
doctypes_to_default_filename = {}
if not max_files_for_doctype:
max_files_for_doctype = {}
doctypes = [doctype for (doctype, desc) in doctypes_and_desc]
# Retrieve/build a working directory to save uploaded files and
# states + configuration.
working_dir = None
if sbm_indir and sbm_doctype and sbm_access:
# Write/read configuration to/from working_dir (WebSubmit mode).
# Retrieve the interface configuration from the current
# submission directory.
working_dir = os.path.join(CFG_WEBSUBMIT_STORAGEDIR,
sbm_indir,
sbm_doctype,
sbm_access)
try:
assert(working_dir == os.path.abspath(working_dir))
except AssertionError:
register_exception(prefix='Cannot create file upload interface: ' + \
+ 'missing parameter ',
alert_admin=True)
return (1, "Unauthorized parameters")
form_url_params = "?" + urlencode({'access': sbm_access,
'indir': sbm_indir,
'doctype': sbm_doctype})
elif uid and sbm_access:
# WebSubmit File Management (admin) interface mode.
# Working directory is in CFG_TMPSHAREDDIR
working_dir = os.path.join(CFG_TMPSHAREDDIR,
'websubmit_upload_interface_config_' + str(uid),
sbm_access)
try:
assert(working_dir == os.path.abspath(working_dir))
except AssertionError:
register_exception(prefix='Some user tried to access ' \
+ working_dir + \
' which is different than ' + \
os.path.abspath(working_dir),
alert_admin=True)
return (1, "Unauthorized parameters")
if not os.path.exists(working_dir):
os.makedirs(working_dir)
form_url_params = "?" + urlencode({'access': sbm_access})
elif sbm_curdir:
# WebSubmit Create_Upload_File_Interface.py function
working_dir = sbm_curdir
form_url_params = None
else:
register_exception(prefix='Some user tried to access ' \
+ working_dir + \
' which is different than ' + \
os.path.abspath(working_dir),
alert_admin=True)
return (1, "Unauthorized parameters")
# Save interface configuration, if this is the first time we come
# here, or else load parameters
try:
parameters = _read_file_revision_interface_configuration_from_disk(working_dir)
(minsize, maxsize, doctypes_and_desc, doctypes,
can_delete_doctypes, can_revise_doctypes,
can_describe_doctypes,
can_comment_doctypes, can_keep_doctypes,
can_rename_doctypes,
can_add_format_to_doctypes, create_related_formats,
can_name_new_files, keep_default, show_links,
file_label, filename_label, description_label,
comment_label, restrictions_and_desc,
can_restrict_doctypes,
restriction_label, doctypes_to_default_filename,
max_files_for_doctype, print_outside_form_tag,
- display_hidden_files, protect_hidden_files) = parameters
+ display_hidden_files, protect_hidden_files,
+ defer_related_formats_creation) = parameters
except:
# Initial display of the interface: save configuration to
# disk for later reuse
parameters = (minsize, maxsize, doctypes_and_desc, doctypes,
can_delete_doctypes, can_revise_doctypes,
can_describe_doctypes,
can_comment_doctypes, can_keep_doctypes,
can_rename_doctypes,
can_add_format_to_doctypes, create_related_formats,
can_name_new_files, keep_default, show_links,
file_label, filename_label, description_label,
comment_label, restrictions_and_desc,
can_restrict_doctypes,
restriction_label, doctypes_to_default_filename,
max_files_for_doctype, print_outside_form_tag,
- display_hidden_files, protect_hidden_files)
+ display_hidden_files, protect_hidden_files,
+ defer_related_formats_creation)
_write_file_revision_interface_configuration_to_disk(working_dir, parameters)
# Get the existing bibdocs as well as the actions performed during
# the former revise sessions of the user, to build an updated list
# of documents. We will use it to check if last action performed
# by user is allowed.
performed_actions = read_actions_log(working_dir)
if recid:
bibrecdocs = BibRecDocs(recid)
# Create the list of files based on current files and performed
# actions
bibdocs = bibrecdocs.list_bibdocs()
else:
bibdocs = []
# "merge":
abstract_bibdocs = build_updated_files_list(bibdocs,
performed_actions,
recid or -1,
display_hidden_files)
# If any, process form submitted by user
if form:
## Get and clean parameters received from user
(file_action, file_target, file_target_doctype,
keep_previous_files, file_description, file_comment, file_rename,
file_doctype, file_restriction, uploaded_filename, uploaded_filepath) = \
wash_form_parameters(form, abstract_bibdocs, can_keep_doctypes,
keep_default, can_describe_doctypes, can_comment_doctypes,
can_rename_doctypes, can_name_new_files, can_restrict_doctypes,
doctypes_to_default_filename, working_dir)
if protect_hidden_files and \
(file_action in ['revise', 'addFormat', 'delete']) and \
is_hidden_for_docname(file_target, abstract_bibdocs):
# Sanity check. We should not let editing
file_action = ''
body += '<script>alert("%s");</script>' % \
_("The file you want to edit is protected against modifications. Your action has not been applied")
## Check the last action performed by user, and log it if
## everything is ok
if uploaded_filepath and \
((file_action == 'add' and (file_doctype in doctypes)) or \
(file_action == 'revise' and \
((file_target_doctype in can_revise_doctypes) or \
'*' in can_revise_doctypes)) or
(file_action == 'addFormat' and \
((file_target_doctype in can_add_format_to_doctypes) or \
'*' in can_add_format_to_doctypes))):
# A file has been uploaded (user has revised or added a file,
# or a format)
dirname, filename, extension = decompose_file(uploaded_filepath)
os.unlink("%s/myfile" % working_dir)
if minsize.isdigit() and os.path.getsize(uploaded_filepath) < int(minsize):
os.unlink(uploaded_filepath)
body += '<script>alert("%s");</script>' % \
(_("The uploaded file is too small (<%(x_size)i o) and has therefore not been considered",
x_size=int(minsize))).replace('"', '\\"')
elif maxsize.isdigit() and os.path.getsize(uploaded_filepath) > int(maxsize):
os.unlink(uploaded_filepath)
body += '<script>alert("%s");</script>' % \
(_("The uploaded file is too big (>%(x_size)i o) and has therefore not been considered",
x_size=int(maxsize))).replace('"', '\\"')
elif len(filename) + len(extension) + 4 > 255:
# Max filename = 256, including extension and version that
# will be appended later by BibDoc
os.unlink(uploaded_filepath)
body += '<script>alert("%s");</script>' % \
_("The uploaded file name is too long and has therefore not been considered").replace('"', '\\"')
elif file_action == 'add' and \
file_doctype in max_files_for_doctype and \
max_files_for_doctype[file_doctype] < \
(len([bibdoc for bibdoc in abstract_bibdocs \
if bibdoc['get_type'] == file_doctype]) + 1):
# User has tried to upload more than allowed for this
# doctype. Should never happen, unless the user did some
# nasty things
os.unlink(uploaded_filepath)
body += '<script>alert("%s");</script>' % \
_("You have already reached the maximum number of files for this type of document").replace('"', '\\"')
else:
# Prepare to move file to
# working_dir/files/updated/doctype/bibdocname/
folder_doctype = file_doctype or \
bibrecdocs.get_bibdoc(file_target).get_type()
folder_bibdocname = file_rename or file_target or filename
new_uploaded_filepath = os.path.join(working_dir, 'files', 'updated',
folder_doctype,
folder_bibdocname, uploaded_filename)
# First check that we do not conflict with an already
# existing bibdoc name
if file_action == "add" and \
((filename in [bibdoc['get_docname'] for bibdoc \
in abstract_bibdocs] and not file_rename) or \
file_rename in [bibdoc['get_docname'] for bibdoc \
in abstract_bibdocs]):
# A file with that name already exist. Cancel action
# and tell user.
os.unlink(uploaded_filepath)
body += '<script>alert("%s");</script>' % \
(_("A file named %(x_name)s already exists. Please choose another name.",
x_name=(file_rename or filename))).replace('"', '\\"')
elif file_action == "revise" and \
file_rename != file_target and \
file_rename in [bibdoc['get_docname'] for bibdoc \
in abstract_bibdocs]:
# A file different from the one to revise already has
# the same bibdocname
os.unlink(uploaded_filepath)
body += '<script>alert("%s");</script>' % \
(_("A file named %(x_name)s already exists. Please choose another name.",
x_name=file_rename)).replace('"', '\\"')
elif file_action == "addFormat" and \
(extension in \
get_extensions_for_docname(file_target,
abstract_bibdocs)):
# A file with that extension already exists. Cancel
# action and tell user.
os.unlink(uploaded_filepath)
body += '<script>alert("%s");</script>' % \
(_("A file with format '%(x_name)s' already exists. Please upload another format.",
x_name=extension)).replace('"', '\\"')
elif '.' in file_rename or '/' in file_rename or "\\" in file_rename or \
not os.path.abspath(new_uploaded_filepath).startswith(os.path.join(working_dir, 'files', 'updated')):
# We forbid usage of a few characters, for the good of
# everybody...
os.unlink(uploaded_filepath)
body += '<script>alert("%s");</script>' % \
_("You are not allowed to use dot '.', slash '/', or backslash '\\\\' in file names. Choose a different name and upload your file again. In particular, note that you should not include the extension in the renaming field.").replace('"', '\\"')
else:
# No conflict with file name
# When revising, delete previously uploaded files for
# this entry, so that we do not execute the
# corresponding action
if file_action == "revise":
for path_to_delete in \
get_uploaded_files_for_docname(working_dir, file_target):
delete_file(working_dir, path_to_delete)
# Move uploaded file to working_dir/files/updated/doctype/bibdocname/
os.renames(uploaded_filepath, new_uploaded_filepath)
if file_action == "add":
# no need to check bibrecdocs.check_file_exists(new_uploaded_filepath, new_uploaded_format): was done before
# Log
if file_rename != '':
# at this point, bibdocname is specified
# name, no need to 'rename'
filename = file_rename
log_action(working_dir, file_action, filename,
new_uploaded_filepath, file_rename,
file_description, file_comment,
file_doctype, keep_previous_files,
- file_restriction)
+ file_restriction,
+ create_related_formats and defer_related_formats_creation)
# Automatically create additional formats when
- # possible.
+ # possible AND wanted
additional_formats = []
- if create_related_formats:
+ if create_related_formats and not defer_related_formats_creation:
additional_formats = createRelatedFormats(new_uploaded_filepath,
overwrite=False)
- for additional_format in additional_formats:
- # Log
- log_action(working_dir, 'addFormat', filename,
- additional_format, file_rename,
- file_description, file_comment,
- file_doctype, True, file_restriction)
+ for additional_format in additional_formats:
+ # Log
+ log_action(working_dir, 'addFormat', filename,
+ additional_format, file_rename,
+ file_description, file_comment,
+ file_doctype, True, file_restriction)
if file_action == "revise" and file_target != "":
# Log
log_action(working_dir, file_action, file_target,
new_uploaded_filepath, file_rename,
file_description, file_comment,
file_target_doctype, keep_previous_files,
- file_restriction)
+ file_restriction, create_related_formats and defer_related_formats_creation)
+
# Automatically create additional formats when
- # possible.
+ # possible AND wanted
additional_formats = []
- if create_related_formats:
+ if create_related_formats and not defer_related_formats_creation:
additional_formats = createRelatedFormats(new_uploaded_filepath,
overwrite=False)
- for additional_format in additional_formats:
- # Log
- log_action(working_dir, 'addFormat',
- (file_rename or file_target),
- additional_format, file_rename,
- file_description, file_comment,
- file_target_doctype, True,
- file_restriction)
+ for additional_format in additional_formats:
+ # Log
+ log_action(working_dir, 'addFormat',
+ (file_rename or file_target),
+ additional_format, file_rename,
+ file_description, file_comment,
+ file_target_doctype, True,
+ file_restriction)
if file_action == "addFormat" and file_target != "":
# We have already checked above that this format does
# not already exist.
# Log
log_action(working_dir, file_action, file_target,
new_uploaded_filepath, file_rename,
file_description, file_comment,
file_target_doctype, keep_previous_files,
file_restriction)
elif file_action in ["add", "addFormat"]:
# No file found, but action involved adding file: ask user to
# select a file
body += """<script>
alert("You did not specify a file. Please choose one before uploading.");
</script>"""
elif file_action == "revise" and file_target != "":
# User has chosen to revise attributes of a file (comment,
# name, etc.) without revising the file itself.
if file_rename != file_target and \
file_rename in [bibdoc['get_docname'] for bibdoc \
in abstract_bibdocs]:
# A file different from the one to revise already has
# the same bibdocname
body += '<script>alert("%s");</script>' % \
(_("A file named %(x_name)s already exists. Please choose another name.",
x_name=file_rename)).replace('"', '\\"')
elif file_rename != file_target and \
('.' in file_rename or '/' in file_rename or "\\" in file_rename):
# We forbid usage of a few characters, for the good of
# everybody...
body += '<script>alert("%s");</script>' % \
_("You are not allowed to use dot '.', slash '/', or backslash '\\\\' in file names. Choose a different name and upload your file again. In particular, note that you should not include the extension in the renaming field.").replace('"', '\\"')
else:
# Log
log_action(working_dir, file_action, file_target,
"", file_rename,
file_description, file_comment,
file_target_doctype, keep_previous_files,
file_restriction)
elif file_action == "delete" and file_target != "" and \
((file_target_doctype in can_delete_doctypes) or \
'*' in can_delete_doctypes):
# Delete previously uploaded files for this entry
for path_to_delete in get_uploaded_files_for_docname(working_dir, file_target):
delete_file(working_dir, path_to_delete)
# Log
log_action(working_dir, file_action, file_target, "", file_rename,
file_description, file_comment, "",
keep_previous_files, file_restriction)
## Display
performed_actions = read_actions_log(working_dir)
#performed_actions = []
if recid:
bibrecdocs = BibRecDocs(recid)
# Create the list of files based on current files and performed
# actions
bibdocs = bibrecdocs.list_bibdocs()
else:
bibdocs = []
abstract_bibdocs = build_updated_files_list(bibdocs, performed_actions,
recid or -1, display_hidden_files)
abstract_bibdocs.sort(lambda x, y: x['order'] - y['order'])
# Display form and necessary CSS + Javscript
#body += '<div>'
#body += css
js_can_describe_doctypes = repr({}.fromkeys(can_describe_doctypes, ''))
js_can_comment_doctypes = repr({}.fromkeys(can_comment_doctypes, ''))
js_can_restrict_doctypes = repr({}.fromkeys(can_restrict_doctypes, ''))
# Prepare to display file revise panel "balloon". Check if we
# should display the list of doctypes or if it is not necessary (0
# or 1 doctype). Also make sure that we do not exceed the maximum
# number of files specified per doctype. The markup of the list of
# doctypes is prepared here, and will be passed as parameter to
# the display_revise_panel function
cleaned_doctypes = [doctype for doctype in doctypes if
doctype not in max_files_for_doctype or
(max_files_for_doctype[doctype] > \
len([bibdoc for bibdoc in abstract_bibdocs \
if bibdoc['get_type'] == doctype]))]
doctypes_list = ""
if len(cleaned_doctypes) > 1:
doctypes_list = '<select id="fileDoctype" name="fileDoctype" onchange="var idx=this.selectedIndex;var doctype=this.options[idx].value;updateForm(doctype,'+','.join([js_can_describe_doctypes, js_can_comment_doctypes, js_can_restrict_doctypes])+');">' + \
'\n'.join(['<option value="' + cgi.escape(doctype, True) + '">' + \
cgi.escape(description) + '</option>' \
for (doctype, description) \
in doctypes_and_desc if \
doctype in cleaned_doctypes]) + \
'</select>'
elif len(cleaned_doctypes) == 1:
doctypes_list = '<input id="fileDoctype" name="fileDoctype" type="hidden" value="%s" />' % cleaned_doctypes[0]
# Check if we should display the list of access restrictions or if
# it is not necessary
restrictions_list = ""
if len(restrictions_and_desc) > 1:
restrictions_list = '<select id="fileRestriction" name="fileRestriction">' + \
'\n'.join(['<option value="' + cgi.escape(restriction, True) + '">' + \
cgi.escape(description) + '</option>' \
for (restriction, description) \
in restrictions_and_desc]) + \
'</select>'
restrictions_list = '''<label for="restriction">%(restriction_label)s:</label>&nbsp;%(restrictions_list)s&nbsp;<small>[<a href="" onclick="alert('%(restriction_help)s');return false;">?</a>]</small>''' % \
{'restrictions_list': restrictions_list,
'restriction_label': restriction_label,
'restriction_help': _('Choose how you want to restrict access to this file.').replace("'", "\\'")}
elif len(restrictions_and_desc) == 1:
restrictions_list = '<select style="display:none" id="fileRestriction" name="fileRestriction"><option value="%(restriction_attr)s">%(restriction)s</option></select>' % {
'restriction': cgi.escape(restrictions_and_desc[0][0]),
'restriction_attr': cgi.escape(restrictions_and_desc[0][0], True)
}
else:
restrictions_list = '<select style="display:none" id="fileRestriction" name="fileRestriction"></select>'
# List the files
body += '''
<div id="reviseControl">
<table class="reviseControlBrowser">'''
i = 0
for bibdoc in abstract_bibdocs:
if bibdoc['list_latest_files']:
i += 1
body += create_file_row(bibdoc, can_delete_doctypes,
can_rename_doctypes,
can_revise_doctypes,
can_describe_doctypes,
can_comment_doctypes,
can_keep_doctypes,
can_add_format_to_doctypes,
doctypes_list,
show_links,
can_restrict_doctypes,
even=not (i % 2),
ln=ln,
form_url_params=form_url_params,
protect_hidden_files=protect_hidden_files)
body += '</table>'
if len(cleaned_doctypes) > 0:
(revise_panel, javascript_prefix) = javascript_display_revise_panel(action='add', target='', show_doctypes=True, show_keep_previous_versions=False, show_rename=can_name_new_files, show_description=True, show_comment=True, bibdocname='', description='', comment='', show_restrictions=True, restriction=len(restrictions_and_desc) > 0 and restrictions_and_desc[0][0] or '', doctypes=doctypes_list)
body += '''%(javascript_prefix)s<input type="button" onclick="%(display_revise_panel)s;updateForm('%(defaultSelectedDoctype)s', %(can_describe_doctypes)s, %(can_comment_doctypes)s, %(can_restrict_doctypes)s);return false;" value="%(add_new_file)s"/>''' % \
{'display_revise_panel': revise_panel,
'javascript_prefix': javascript_prefix,
'defaultSelectedDoctype': escape_javascript_string(cleaned_doctypes[0], escape_quote_for_html=True),
'add_new_file': _("Add new file"),
'can_describe_doctypes':js_can_describe_doctypes,
'can_comment_doctypes': repr({}.fromkeys(can_comment_doctypes, '')),
'can_restrict_doctypes': repr({}.fromkeys(can_restrict_doctypes, ''))}
body += '</div>'
if print_envelope:
# We should print this only if we display for the first time
body = '<div id="uploadFileInterface">' + body + '</div>'
if include_headers:
body = get_upload_file_interface_javascript(form_url_params) + \
get_upload_file_interface_css() + \
body
# Display markup of the revision panel. This one is also
# printed only at the beginning, so that it does not need to
# be returned with each response
body += revise_balloon % \
{'CFG_SITE_URL': CFG_SITE_URL,
'file_label': file_label,
'filename_label': filename_label,
'description_label': description_label,
'comment_label': comment_label,
'restrictions': restrictions_list,
'previous_versions_help': _('You can decide to hide or not previous version(s) of this file.').replace("'", "\\'"),
'revise_format_help': _('When you revise a file, the additional formats that you might have previously uploaded are removed, since they no longer up-to-date with the new file.').replace("'", "\\'"),
'revise_format_warning': _('Alternative formats uploaded for current version of this file will be removed'),
'previous_versions_label': _('Keep previous versions'),
'cancel': _('Cancel'),
'upload': _('Upload'),
'uploading_label': _('Uploading...'),
'postprocess_label': _('Please wait...'),
'submit_or_button': form_url_params and 'button' or 'submit'}
body += '''
<input type="hidden" name="recid" value="%(recid)i"/>
<input type="hidden" name="ln" value="%(ln)s"/>
''' % \
{'recid': recid or -1,
'ln': ln}
# End submission button
if sbm_curdir:
body += '''<br /><div style="font-size:small">
<input type="button" class="adminbutton" name="Submit" id="applyChanges" value="%(apply_changes)s" onClick="nextStep();"></div>''' % \
{'apply_changes': _("Apply changes")}
# Display a link to support email in case users have problem
# revising/adding files
mailto_link = create_html_mailto(email=CFG_SITE_SUPPORT_EMAIL,
subject=_("Need help revising or adding files to record %(recid)s") % \
{'recid': recid or ''},
body=_("""Dear Support,
I would need help to revise or add a file to record %(recid)s.
I have attached the new version to this email.
Best regards""") % {'recid': recid or ''})
problem_revising = _('Having a problem revising a file? Send the revised version to %(mailto_link)s.') % {'mailto_link': mailto_link}
if len(cleaned_doctypes) > 0:
# We can add files, so change note
problem_revising = _('Having a problem adding or revising a file? Send the new/revised version to %(mailto_link)s.') % {'mailto_link': mailto_link}
body += '<br />'
body += problem_revising
if print_envelope and print_outside_form_tag:
body = '<form method="post" action="/%s/managedocfilesasync" id="uploadFileForm">' % CFG_SITE_RECORD + body + '</form>'
return (0, body)
def create_file_row(abstract_bibdoc, can_delete_doctypes,
can_rename_doctypes, can_revise_doctypes,
can_describe_doctypes, can_comment_doctypes,
can_keep_doctypes, can_add_format_to_doctypes,
doctypes_list, show_links, can_restrict_doctypes,
even=False, ln=CFG_SITE_LANG, form_url_params='',
protect_hidden_files=True):
"""
Creates a row in the files list representing the given abstract_bibdoc
@param abstract_bibdoc: list of "fake" BibDocs: it is a list of dictionaries
with keys 'list_latest_files' and 'get_docname' with
values corresponding to what you would expect to receive
when calling their counterpart function on a real BibDoc
object.
@param can_delete_doctypes: list of doctypes for which we allow users to delete
documents
@param can_revise_doctypes: the list of doctypes that users are
allowed to revise.
@param can_describe_doctypes: the list of doctypes that users are
allowed to describe.
@param can_comment_doctypes: the list of doctypes that users are
allowed to comment.
@param can_keep_doctypes: the list of doctypes for which users can
choose to keep previous versions visible
when revising a file (i.e. 'Keep previous
version' checkbox).
@param can_rename_doctypes: the list of doctypes that users are
allowed to rename (when revising)
@param can_add_format_to_doctypes: the list of doctypes for which users can
add new formats
@param show_links: if we display links to files
@param even: if the row is even or odd on the list
@type even: boolean
@param ln: language
@type ln: string
@param form_url_params: the
@type form_url_params: string
@param protect_hidden_files: if bibdoc containing bibdocfiles
flagged as 'HIDDEN' can be edited
(revise, delete, add format) or not.
@type protect_hidden_files: boolean
@return: an HTML formatted "file" row
@rtype: string
"""
_ = gettext_set_language(ln)
# Try to retrieve "main format", to display as link for the
# file. There is no such concept in BibDoc, but let's just try to
# get the pdf file if it exists
main_bibdocfile = [bibdocfile for bibdocfile in abstract_bibdoc['list_latest_files'] \
if bibdocfile.get_format().strip('.').lower() == 'pdf']
if len(main_bibdocfile) > 0:
main_bibdocfile = main_bibdocfile[0]
else:
main_bibdocfile = abstract_bibdoc['list_latest_files'][0]
main_bibdocfile_description = main_bibdocfile.get_description()
if main_bibdocfile_description is None:
main_bibdocfile_description = ''
updated = abstract_bibdoc['updated'] # Has BibDoc been updated?
hidden_p = abstract_bibdoc['hidden_p']
# Main file row
out = '<tr%s>' % (even and ' class="even"' or '')
out += '<td class="reviseControlFileColumn"%s>' % (hidden_p and ' style="color:#99F"' or '')
if not updated and show_links and not hidden_p:
out += '<a target="_blank" href="' + main_bibdocfile.get_url() \
+ '">'
out += cgi.escape(abstract_bibdoc['get_docname'])
if hidden_p:
out += ' <span style="font-size:small;font-style:italic;color:#888">(hidden)</span>'
if not updated and show_links and not hidden_p:
out += '</a>'
if main_bibdocfile_description:
out += ' (<em>' + cgi.escape(main_bibdocfile_description) + '</em>)'
out += '</td>'
(description, comment) = get_description_and_comment(abstract_bibdoc['list_latest_files'])
restriction = abstract_bibdoc['get_status']
# Revise link
out += '<td class="reviseControlActionColumn">'
if main_bibdocfile.get_type() in can_revise_doctypes or \
'*' in can_revise_doctypes and not (hidden_p and protect_hidden_files):
(revise_panel, javascript_prefix) = javascript_display_revise_panel(
action='revise',
target=abstract_bibdoc['get_docname'],
show_doctypes=False,
show_keep_previous_versions=(main_bibdocfile.get_type() in can_keep_doctypes) or '*' in can_keep_doctypes,
show_rename=(main_bibdocfile.get_type() in can_rename_doctypes) or '*' in can_rename_doctypes,
show_description=(main_bibdocfile.get_type() in can_describe_doctypes) or '*' in can_describe_doctypes,
show_comment=(main_bibdocfile.get_type() in can_comment_doctypes) or '*' in can_comment_doctypes,
bibdocname=abstract_bibdoc['get_docname'],
description=description,
comment=comment,
show_restrictions=(main_bibdocfile.get_type() in can_restrict_doctypes) or '*' in can_restrict_doctypes,
restriction=restriction,
doctypes=doctypes_list)
out += '%(javascript_prefix)s[<a href="" onclick="%(display_revise_panel)s;return false;">%(revise)s</a>]' % \
{'display_revise_panel': revise_panel,
'javascript_prefix': javascript_prefix,
'revise': _("revise")
}
# Delete link
if main_bibdocfile.get_type() in can_delete_doctypes or \
'*' in can_delete_doctypes and not (hidden_p and protect_hidden_files):
global params_id
params_id += 1
out += '''
<script type="text/javascript">
/*<![CDATA[*/
var delete_panel_params_%(id)i = "%(bibdocname)s";
/*]]>*/
</script>
[<a href="" onclick="return askDelete(delete_panel_params_%(id)i, '%(form_url_params)s')">%(delete)s</a>]
''' % {'bibdocname': escape_javascript_string(abstract_bibdoc['get_docname'], escape_for_html=False),
'delete': _("delete"),
'form_url_params': form_url_params or '',
'id': params_id}
out += '''</td>'''
# Format row
out += '''<tr%s>
<td class="reviseControlFormatColumn"%s>
<img src="%s/img/tree_branch.gif" alt="">
''' % (even and ' class="even"' or '', hidden_p and ' style="color:#999"' or '', CFG_SITE_URL)
for bibdocfile in abstract_bibdoc['list_latest_files']:
if not updated and show_links and not hidden_p:
out += '<a target="_blank" href="' + bibdocfile.get_url() + '">'
out += bibdocfile.get_format().strip('.')
if not updated and show_links and not hidden_p:
out += '</a>'
out += ' '
+ for format_to_be_created in abstract_bibdoc['formats_to_be_created']:
+ if not hidden_p:
+ out += '<span class="reviseControlFormatToBeCreated">' + format_to_be_created.strip('.') + '</span> '
+
# Add format link
out += '<td class="reviseControlActionColumn">'
if main_bibdocfile.get_type() in can_add_format_to_doctypes or \
'*' in can_add_format_to_doctypes and not (hidden_p and protect_hidden_files):
(revise_panel, javascript_prefix) = javascript_display_revise_panel(
action='addFormat',
target=abstract_bibdoc['get_docname'],
show_doctypes=False,
show_keep_previous_versions=False,
show_rename=False,
show_description=False,
show_comment=False,
bibdocname='',
description='',
comment='',
show_restrictions=False,
restriction=restriction,
doctypes=doctypes_list)
out += '%(javascript_prefix)s[<a href="" onclick="%(display_revise_panel)s;return false;">%(add_format)s</a>]' % \
{'display_revise_panel': revise_panel,
'javascript_prefix': javascript_prefix,
'add_format':_("add format")}
out += '</td></tr>'
return out
def build_updated_files_list(bibdocs, actions, recid, display_hidden_files=False):
"""
Parses the list of BibDocs and builds an updated version to reflect
the changes performed by the user of the file
It is necessary to abstract the BibDocs since user wants to
perform action on the files that are committed only at the end of
the session.
@param bibdocs: the original list of bibdocs on which we want to
build a new updated list
@param actions: the list of actions performed by the user on the
files, and that we want to consider to build an
updated file list
@param recid: the record ID to which the files belong
@param display_hidden_files: if bibdoc containing bibdocfiles
flagged as 'HIDDEN' should be
displayed or not.
@type display_hidden_files: boolean
"""
abstract_bibdocs = {}
+ create_related_formats_for_bibdocs = {}
i = 0
for bibdoc in bibdocs:
hidden_p = True in [bibdocfile.hidden_p() for bibdocfile in bibdoc.list_latest_files()]
if CFG_CERN_SITE:
hidden_p = False # Temporary workaround. See Ticket #846
if not display_hidden_files and hidden_p:
# Do not consider hidden files
continue
i += 1
status = bibdoc.get_status()
if status == "DELETED":
status = ''
brd = BibRecDocs(recid)
abstract_bibdocs[brd.get_docname(bibdoc.id)] = \
{'list_latest_files': bibdoc.list_latest_files(),
'get_docname': brd.get_docname(bibdoc.id),
'updated': False,
'get_type': bibdoc.get_type(),
'get_status': status,
'order': i,
- 'hidden_p': hidden_p}
+ 'hidden_p': hidden_p,
+ 'formats_to_be_created': []}
for action, bibdoc_name, file_path, rename, description, \
comment, doctype, keep_previous_versions, \
- file_restriction in actions:
+ file_restriction, create_related_formats in actions:
dirname, filename, fileformat = decompose_file(file_path)
i += 1
if action in ["add", "revise"] and \
os.path.exists(file_path):
checksum = calculate_md5(file_path)
order = i
if action == "revise" and \
bibdoc_name in abstract_bibdocs:
# Keep previous values
order = abstract_bibdocs[bibdoc_name]['order']
doctype = abstract_bibdocs[bibdoc_name]['get_type']
if bibdoc_name.strip() == '' and rename.strip() == '':
bibdoc_name = os.path.extsep.join(filename.split(os.path.extsep)[:-1])
elif rename.strip() != '' and \
bibdoc_name in abstract_bibdocs:
# Keep previous position
del abstract_bibdocs[bibdoc_name]
# First instantiate a fake BibDocMoreInfo object, without any side effect
more_info = BibDocMoreInfo(1, cache_only = False, initial_data = {})
if description is not None:
more_info['descriptions'] = {1: {fileformat:description}}
if comment is not None:
more_info['comments'] = {1: {fileformat:comment}}
abstract_bibdocs[(rename or bibdoc_name)] = \
{'list_latest_files': [BibDocFile(file_path, [(int(recid), doctype,(rename or bibdoc_name))], version=1,
docformat=fileformat,
docid=-1,
status=file_restriction,
checksum=checksum,
more_info=more_info)],
'get_docname': rename or bibdoc_name,
'get_type': doctype,
'updated': True,
'get_status': file_restriction,
'order': order,
- 'hidden_p': False}
+ 'hidden_p': False,
+ 'formats_to_be_created': []}
abstract_bibdocs[(rename or bibdoc_name)]['updated'] = True
+ if create_related_formats:
+ create_related_formats_for_bibdocs[(rename or bibdoc_name)] = True
elif action == "revise" and not file_path:
# revision of attributes of a file (description, name,
# comment or restriction) but no new file.
abstract_bibdocs[bibdoc_name]['get_docname'] = rename or bibdoc_name
abstract_bibdocs[bibdoc_name]['get_status'] = file_restriction
set_description_and_comment(abstract_bibdocs[bibdoc_name]['list_latest_files'],
description, comment)
abstract_bibdocs[bibdoc_name]['updated'] = True
elif action == "delete":
if bibdoc_name in abstract_bibdocs:
del abstract_bibdocs[bibdoc_name]
elif action == "addFormat" and \
os.path.exists(file_path):
checksum = calculate_md5(file_path)
# Preserve type and status
doctype = abstract_bibdocs[bibdoc_name]['get_type']
file_restriction = abstract_bibdocs[bibdoc_name]['get_status']
# First instantiate a fake BibDocMoreInfo object, without any side effect
more_info = BibDocMoreInfo(1, cPickle.dumps({}))
if description is not None:
more_info['descriptions'] = {1: {fileformat:description}}
if comment is not None:
more_info['comments'] = {1: {fileformat:comment}}
abstract_bibdocs[bibdoc_name]['list_latest_files'].append(\
BibDocFile(file_path, [(int(recid), doctype, (rename or bibdoc_name))], version=1,
docformat=fileformat,
docid=-1, status='',
checksum=checksum, more_info=more_info))
abstract_bibdocs[bibdoc_name]['updated'] = True
+ # For each BibDoc for which we would like to create related
+ # formats, do build the list of formats that should be created
+ for docname in create_related_formats_for_bibdocs.keys():
+ current_files_for_bibdoc = [bibdocfile.get_path() for bibdocfile in abstract_bibdocs[docname]['list_latest_files']]
+ missing_formats = []
+ for missing_formats_group in get_missing_formats(current_files_for_bibdoc).values():
+ missing_formats.extend(missing_formats_group)
+ abstract_bibdocs[docname]['formats_to_be_created'] = missing_formats
+
return abstract_bibdocs.values()
def _read_file_revision_interface_configuration_from_disk(working_dir):
"""
Read the configuration of the file revision interface from disk
@param working_dir: the path to the working directory where we can find
the configuration file
"""
input_file = open(os.path.join(working_dir, 'upload_interface.config'), 'rb')
configuration = cPickle.load(input_file)
input_file.close()
return configuration
def _write_file_revision_interface_configuration_to_disk(working_dir, parameters):
"""
Write the configuration of the file revision interface to disk
@param working_dir: the path to the working directory where we should
write the configuration.
@param parameters: the parameters to write to disk
"""
output = open(os.path.join(working_dir, 'upload_interface.config'), 'wb')
cPickle.dump(parameters, output)
output.close()
def log_action(log_dir, action, bibdoc_name, file_path, rename,
description, comment, doctype, keep_previous_versions,
- file_restriction):
+ file_restriction, create_related_formats=False):
"""
Logs a new action performed by user on a BibDoc file.
The log file record one action per line, each column being split
by '<--->' ('---' is escaped from values 'rename', 'description',
'comment' and 'bibdoc_name'). The original request for this
format was motivated by the need to have it easily readable by
other scripts. Not sure it still makes sense nowadays...
Newlines are also reserved, and are escaped from the input values
(necessary for the 'comment' field, which is the only one allowing
newlines from the browser)
Each line starts with the time of the action in the following
format: '2008-06-20 08:02:04 --> '
@param log_dir: directory where to save the log (ie. working_dir)
@param action: the performed action (one of 'revise', 'delete',
'add', 'addFormat')
@param bibdoc_name: the name of the bibdoc on which the change is
applied
@param file_path: the path to the file that is going to be
integrated as bibdoc, if any (should be""
in case of action="delete", or action="revise"
when revising only attributes of a file)
@param rename: the name used to display the bibdoc, instead of the
filename (can be None for no renaming)
@param description: a description associated with the file
@param comment: a comment associated with the file
@param doctype: the category in which the file is going to be
integrated
@param keep_previous_versions: if the previous versions of this
file are to be hidden (0) or not (1)
@param file_restriction: the restriction applied to the
file. Empty string if no restriction
+
+ @param create_related_formats: shall we created related formats
+ for this action? Valid only if
+ C{action} is 'add' or 'revise'.
"""
log_file = os.path.join(log_dir, 'bibdocactions.log')
try:
file_desc = open(log_file, "a+")
# We must escape new lines from comments in some way:
comment = str(comment).replace('\\', '\\\\').replace('\r\n', '\\n\\r')
msg = action + '<--->' + \
bibdoc_name.replace('---', '___') + '<--->' + \
file_path + '<--->' + \
str(rename).replace('---', '___') + '<--->' + \
str(description).replace('---', '___') + '<--->' + \
comment.replace('---', '___') + '<--->' + \
doctype + '<--->' + \
str(int(keep_previous_versions)) + '<--->' + \
- file_restriction + '\n'
+ file_restriction + '<--->' + \
+ str(create_related_formats) + '\n'
file_desc.write("%s --> %s" %(time.strftime("%Y-%m-%d %H:%M:%S"), msg))
file_desc.close()
except Exception as e:
raise e
def read_actions_log(log_dir):
"""
Reads the logs of action to be performed on files
See log_action(..) for more information about the structure of the
log file.
@param log_dir: the path to the directory from which to read the
log file
@type log_dir: string
"""
actions = []
log_file = os.path.join(log_dir, 'bibdocactions.log')
try:
file_desc = open(log_file, "r")
for line in file_desc.readlines():
(timestamp, action) = line.split(' --> ', 1)
try:
(action, bibdoc_name, file_path, rename, description,
comment, doctype, keep_previous_versions,
- file_restriction) = action.rstrip('\n').split('<--->')
+ file_restriction, create_related_formats) = action.rstrip('\n').split('<--->')
except ValueError as e:
# Malformed action log
pass
# Clean newline-escaped comment:
comment = comment.replace('\\n\\r', '\r\n').replace('\\\\', '\\')
# Perform some checking
if action not in CFG_ALLOWED_ACTIONS:
# Malformed action log
pass
try:
keep_previous_versions = int(keep_previous_versions)
except:
# Malformed action log
keep_previous_versions = 1
pass
+ create_related_formats = create_related_formats == 'True' and True or False
+
actions.append((action, bibdoc_name, file_path, rename, \
description, comment, doctype,
- keep_previous_versions, file_restriction))
+ keep_previous_versions, file_restriction,
+ create_related_formats))
file_desc.close()
except:
pass
return actions
def javascript_display_revise_panel(action, target, show_doctypes, show_keep_previous_versions, show_rename, show_description, show_comment, bibdocname, description, comment, show_restrictions, restriction, doctypes):
"""
Returns a correctly encoded call to the javascript function to
display the revision panel.
"""
global params_id
params_id += 1
javascript_prefix = '''
<script type="text/javascript">
/*<![CDATA[*/
var revise_panel_params_%(id)i = {"action": "%(action)s",
"target": "%(target)s",
"showDoctypes": %(showDoctypes)s,
"showKeepPreviousVersions": %(showKeepPreviousVersions)s,
"showRename": %(showRename)s,
"showDescription": %(showDescription)s,
"showComment": %(showComment)s,
"bibdocname": "%(bibdocname)s",
"description": "%(description)s",
"comment": "%(comment)s",
"showRestrictions": %(showRestrictions)s,
"restriction": "%(restriction)s",
"doctypes": "%(doctypes)s"}
/*]]>*/
</script>''' % {'id': params_id,
'action': action,
'showDoctypes': show_doctypes and 'true' or 'false',
'target': escape_javascript_string(target, escape_for_html=False),
'bibdocname': escape_javascript_string(bibdocname, escape_for_html=False),
'showRename': show_rename and 'true' or 'false',
'showKeepPreviousVersions': show_keep_previous_versions and 'true' or 'false',
'showComment': show_comment and 'true' or 'false',
'showDescription': show_description and 'true' or 'false',
'description': description and escape_javascript_string(description, escape_for_html=False) or '',
'comment': comment and escape_javascript_string(comment, escape_for_html=False) or '',
'showRestrictions': show_restrictions and 'true' or 'false',
'restriction': escape_javascript_string(restriction, escape_for_html=False),
'doctypes': escape_javascript_string(doctypes, escape_for_html=False)}
return ('display_revise_panel(this, revise_panel_params_%(id)i)' % {'id': params_id},
javascript_prefix)
def get_uploaded_files_for_docname(log_dir, docname):
"""
Given a docname, returns the paths to the files uploaded for this
revision session.
@param log_dir: the path to the directory that should contain the
uploaded files.
@param docname: the name of the bibdoc for which we want to
retrieve files.
"""
return [file_path for action, bibdoc_name, file_path, rename, \
description, comment, doctype, keep_previous_versions , \
- file_restriction in read_actions_log(log_dir) \
+ file_restriction, create_related_formats in read_actions_log(log_dir) \
if bibdoc_name == docname and os.path.exists(file_path)]
def get_bibdoc_for_docname(docname, abstract_bibdocs):
"""
Given a docname, returns the corresponding bibdoc from the
'abstract' bibdocs.
Return None if not found
@param docname: the name of the bibdoc we want to retrieve
@param abstract_bibdocs: the list of bibdocs from which we want to
retrieve the bibdoc
"""
bibdocs = [bibdoc for bibdoc in abstract_bibdocs \
if bibdoc['get_docname'] == docname]
if len(bibdocs) > 0:
return bibdocs[0]
else:
return None
def get_extensions_for_docname(docname, abstract_bibdocs):
"""
Returns the list of extensions that exists for given bibdoc
name in the given 'abstract' bibdocs.
@param docname: the name of the bibdoc for wich we want to
retrieve the available extensions
@param abstract_bibdocs: the list of bibdocs from which we want to
retrieve the bibdoc extensions
"""
bibdocfiles = [bibdoc['list_latest_files'] for bibdoc \
in abstract_bibdocs \
if bibdoc['get_docname'] == docname]
if len(bibdocfiles) > 0:
# There should always be at most 1 matching docname, or 0 if
# it is a new file
return [bibdocfile.get_format() for bibdocfile \
in bibdocfiles[0]]
return []
def is_hidden_for_docname(docname, abstract_bibdocs):
"""
Returns True if the bibdoc with given docname in abstract_bibdocs
should be hidden. Also return True if docname cannot be found in
abstract_bibdocs.
@param docname: the name of the bibdoc for wich we want to
check if it is hidden or not
@param abstract_bibdocs: the list of bibdocs from which we want to
look for the given docname
"""
bibdocs = [bibdoc for bibdoc in abstract_bibdocs \
if bibdoc['get_docname'] == docname]
if len(bibdocs) > 0:
return bibdocs[0]['hidden_p']
return True
def get_description_and_comment(bibdocfiles):
"""
Returns the first description and comment as tuple (description,
comment) found in the given list of bibdocfile
description and/or comment can be None.
This function is needed since we do consider that there is one
comment/description per bibdoc, and not per bibdocfile as APIs
state.
@param bibdocfiles: the list of files of a given bibdoc for which
we want to extract the description and comment.
"""
description = None
comment = None
all_descriptions = [bibdocfile.get_description() for bibdocfile \
in bibdocfiles
if bibdocfile.get_description() not in ['', None]]
if len(all_descriptions) > 0:
description = all_descriptions[0]
all_comments = [bibdocfile.get_comment() for bibdocfile \
in bibdocfiles
if bibdocfile.get_comment() not in ['', None]]
if len(all_comments) > 0:
comment = all_comments[0]
return (description, comment)
def set_description_and_comment(abstract_bibdocfiles, description, comment):
"""
Set the description and comment to the given (abstract)
bibdocfiles.
description and/or comment can be None.
This function is needed since we do consider that there is one
comment/description per bibdoc, and not per bibdocfile as APIs
state.
@param abstract_bibdocfiles: the list of 'abstract' files of a
given bibdoc for which we want to set the
description and comment.
@param description: the new description
@param comment: the new comment
"""
for bibdocfile in abstract_bibdocfiles:
bibdocfile.description = description
bibdocfile.comment = comment
def delete_file(working_dir, file_path):
"""
Deletes a file at given path from the file.
In fact, we just move it to working_dir/files/trash
@param working_dir: the path to the working directory
@param file_path: the path to the file to delete
"""
if os.path.exists(file_path):
filename = os.path.split(file_path)[1]
move_to = os.path.join(working_dir, 'files', 'trash',
filename +'_' + str(time.time()))
os.renames(file_path, move_to)
def wash_form_parameters(form, abstract_bibdocs, can_keep_doctypes,
keep_default, can_describe_doctypes,
can_comment_doctypes, can_rename_doctypes,
can_name_new_files, can_restrict_doctypes,
doctypes_to_default_filename, working_dir):
"""
Washes the (user-defined) form parameters, taking into account the
current state of the files and the admin defaults.
@param form: the form of the function
@param abstract_bibdocs: a representation of the current state of
the files, as returned by
build_updated_file_list(..)
@param can_keep_doctypes: the list of doctypes for which we allow
users to choose to keep or not the
previous versions when revising.
@type can_keep_doctypes: list
@param keep_default: the admin-defined default for when users
cannot choose to keep or not previous version
of a revised file
@type keep_default: boolean
@param can_describe_doctypes: the list of doctypes for which we
let users define descriptions.
@type can_describe_doctypes: list
@param can_comment_doctypes: the list of doctypes for which we let
users define comments.
@type can_comment_doctypes: list
@param can_rename_doctypes: the list of doctypes for which we let
users rename bibdoc when revising.
@type can_rename_doctypes: list
@param can_name_new_files: if we let users choose a name when
adding new files.
@type can_name_new_files: boolean
@param can_restrict_doctypes: the list of doctypes for which we
let users define access
restrictions.
@type can_restrict_doctypes: list
@param doctypes_to_default_filename: mapping from doctype to
admin-chosen name for
uploaded file.
@type doctypes_to_default_filename: dict
@param working_dir: the path to the current working directory
@type working_dir: string
@return: tuple (file_action, file_target, file_target_doctype,
keep_previous_files, file_description, file_comment,
file_rename, file_doctype, file_restriction) where::
file_action: *str* the performed action ('add',
'revise','addFormat' or 'delete')
file_target: *str* the bibdocname of the file on which the
action is performed (empty string when
file_action=='add')
file_target_doctype: *str* the doctype of the file we will
work on. Eg: ('main',
'additional'). Empty string with
file_action=='add'.
keep_previous_files: *bool* if we keep the previous version of
the file or not. Only useful when
revising files.
file_description: *str* the user-defined description to apply
to the file. Empty string when no
description defined or when not applicable
file_comment: *str* the user-defined comment to apply to the
file. Empty string when no comment defined or
when not applicable
file_rename: *str* the new name chosen by user for the
bibdoc. Empty string when not defined or when not
applicable.
file_doctype: *str* the user-chosen doctype for the bibdoc
when file_action=='add', or the current doctype
of the file_target in other cases (doctype must
be preserved).
file_restriction: *str* the user-selected restriction for the
file. Emptry string if not defined or when
not applicable.
file_name: *str* the original name of the uploaded file. None
if no file uploaded
file_path: *str* the full path to the file
@rtype: tuple(string, string, string, boolean, string, string,
string, string, string, string, string)
"""
# Action performed ...
if "fileAction" in form and \
form['fileAction'] in CFG_ALLOWED_ACTIONS:
file_action = str(form['fileAction']) # "add", "revise",
# "addFormat" or "delete"
else:
file_action = ""
# ... on file ...
if "fileTarget" in form:
file_target = str(form['fileTarget']) # contains bibdocname
# Also remember its doctype to make sure we do valid actions
# on it
corresponding_bibdoc = get_bibdoc_for_docname(file_target,
abstract_bibdocs)
if corresponding_bibdoc is not None:
file_target_doctype = corresponding_bibdoc['get_type']
else:
file_target_doctype = ""
else:
file_target = ""
file_target_doctype = ""
# ... with doctype?
# Only useful when adding file: otherwise fileTarget doctype is
# preserved
file_doctype = file_target_doctype
if "fileDoctype" in form and \
file_action == 'add':
file_doctype = str(form['fileDoctype'])
# ... keeping previous version? ...
if file_target_doctype != '' and \
"keepPreviousFiles" not in form:
# no corresponding key. Two possibilities:
if file_target_doctype in can_keep_doctypes or \
'*' in can_keep_doctypes:
# User decided no to keep
keep_previous_files = 0
else:
# No choice for user. Use default admin has chosen
keep_previous_files = keep_default
else:
# Checkbox seems to be checked ...
if file_target_doctype in can_keep_doctypes or \
'*' in can_keep_doctypes:
# ...and this is allowed
keep_previous_files = 1
else:
# ...but this is not allowed
keep_previous_files = keep_default
# ... and decription? ...
if "description" in form and \
(((file_action == 'revise' and \
(file_target_doctype in can_describe_doctypes)) or \
(file_action == 'add' and \
(file_doctype in can_describe_doctypes))) \
or '*' in can_describe_doctypes):
file_description = str(form['description'])
else:
file_description = ''
# ... and comment? ...
if "comment" in form and \
(((file_action == 'revise' and \
(file_target_doctype in can_comment_doctypes)) or \
(file_action == 'add' and \
(file_doctype in can_comment_doctypes))) \
or '*' in can_comment_doctypes):
file_comment = str(form['comment'])
else:
file_comment = ''
# ... and rename to ? ...
if "rename" in form and \
((file_action == "revise" and \
((file_target_doctype in can_rename_doctypes) or \
'*' in can_rename_doctypes)) or \
(file_action == "add" and \
can_name_new_files)):
file_rename = str(form['rename']) # contains new bibdocname if applicable
elif file_action == "add" and \
file_doctype in doctypes_to_default_filename:
# Admin-chosen name.
file_rename = doctypes_to_default_filename[file_doctype]
if file_rename.lower().startswith('file:'):
# We will define name at a later stage, i.e. when
# submitting the file with bibdocfile. The name will be
# chosen by reading content of a file in curdir
file_rename = ''
else:
# Ensure name is unique, by appending a suffix
file_rename = doctypes_to_default_filename[file_doctype]
file_counter = 2
while get_bibdoc_for_docname(file_rename, abstract_bibdocs):
if file_counter == 2:
file_rename += '-2'
else:
file_rename = file_rename[:-len(str(file_counter))] + \
str(file_counter)
file_counter += 1
else:
file_rename = ''
# ... and file restriction ? ...
file_restriction = ''
if "fileRestriction" in form:
# We cannot clean that value as it could be a restriction
# declared in another submission. We keep this value.
file_restriction = str(form['fileRestriction'])
# ... and the file itself ? ...
if 'myfile' in form and \
hasattr(form['myfile'], "filename") and \
form['myfile'].filename:
dir_to_open = os.path.join(working_dir, 'files', 'myfile')
if not os.path.exists(dir_to_open):
try:
os.makedirs(dir_to_open)
except:
pass
# Shall we continue?
if os.path.exists(dir_to_open):
form_field = form['myfile']
file_name = form_field.filename
form_file = form_field.file
## Before saving the file to disk, wash the filename (in particular
## washing away UNIX and Windows (e.g. DFS) paths):
file_name = os.path.basename(file_name.split('\\')[-1])
file_name = file_name.strip()
if file_name != "":
# This may be dangerous if the file size is bigger than
# the available memory
file_path = os.path.join(dir_to_open, file_name)
if not os.path.exists(file_path):
# If file already exists, it means that it was
# handled by WebSubmit
fp = file(file_path, "wb")
chunk = form_file.read(10240)
while chunk:
fp.write(chunk)
chunk = form_file.read(10240)
fp.close()
fp = open(os.path.join(working_dir, "lastuploadedfile"), "w")
fp.write(file_name)
fp.close()
fp = open(os.path.join(working_dir, 'myfile'), "w")
fp.write(file_name)
fp.close()
else:
file_name = None
file_path = None
return (file_action, file_target, file_target_doctype,
keep_previous_files, file_description, file_comment,
file_rename, file_doctype, file_restriction, file_name,
file_path)
def move_uploaded_files_to_storage(working_dir, recid, icon_sizes,
create_icon_doctypes,
force_file_revision):
"""
Apply the modifications on files (add/remove/revise etc.) made by
users with one of the compatible interfaces (WebSubmit function
`Create_Upload_Files_Interface.py'; WebSubmit element or WebSubmit
File management interface using function
`create_file_upload_interface').
This function needs a "working directory" (working_dir) that contains a
bibdocactions.log file with the list of actions to perform.
@param working_dir: a path to the working directory containing actions to perform and files to attach
@type working_dir: string
@param recid: the recid to modify
@type recid: int
@param icon_sizes: the sizes of icons to create, as understood by
the websubmit icon creation tool
@type icon_sizes: list(string)
@param create_icon_doctypes: a list of doctype for which we want
to create icons
@type create_icon_doctypes: list(string)
@param force_file_revision: when revising attributes of a file
(comment, description) without
uploading a new file, force a revision
of the current version (so that old
comment, description, etc. is kept
or not)
@type force_file_revision: bool
"""
# We need to remember of some actions that cannot be performed,
# because files have been deleted or moved after a renaming.
# Those pending action must be applied when revising the bibdoc
# with a file that exists (that means that the bibdoc has not been
# deleted nor renamed by a later action)
pending_bibdocs = {}
newly_added_bibdocs = [] # Does not consider new formats/revisions
-
+ create_related_formats_for_bibdocs = {}
+ create_icons_for_bibdocs = {}
performed_actions = read_actions_log(working_dir)
+ sequence_id = bibtask_allocate_sequenceid(working_dir)
for action, bibdoc_name, file_path, rename, description, \
comment, doctype, keep_previous_versions, \
- file_restriction in performed_actions:
+ file_restriction, create_related_formats in performed_actions:
# FIXME: get this out of the loop once changes to bibrecdocs
# are immediately visible. For the moment, reload the
# structure from scratch at each step
bibrecdocs = BibRecDocs(recid)
if action == 'add':
new_bibdoc = \
add(file_path, bibdoc_name, rename, doctype, description,
comment, file_restriction, recid, working_dir, icon_sizes,
create_icon_doctypes, pending_bibdocs, bibrecdocs)
if new_bibdoc:
newly_added_bibdocs.append(new_bibdoc)
+ if create_related_formats:
+ # Schedule creation of related formats when possible.
+ create_related_formats_for_bibdocs[rename or bibdoc_name] = True
+
+ if doctype in create_icon_doctypes or '*' in create_icon_doctypes:
+ # Schedule creation of icons when possible.
+ create_icons_for_bibdocs[rename or bibdoc_name] = True
+
elif action == 'addFormat':
add_format(file_path, bibdoc_name, recid, doctype, working_dir,
icon_sizes, create_icon_doctypes,
pending_bibdocs, bibrecdocs)
+ if doctype in create_icon_doctypes or '*' in create_icon_doctypes:
+ # Schedule creation of icons when possible.
+ create_icons_for_bibdocs[rename or bibdoc_name] = True
+
elif action == 'revise':
new_bibdoc = \
revise(file_path, bibdoc_name, rename, doctype,
description, comment, file_restriction, icon_sizes,
create_icon_doctypes, keep_previous_versions,
recid, working_dir, pending_bibdocs,
bibrecdocs, force_file_revision)
if new_bibdoc:
newly_added_bibdocs.append(new_bibdoc)
+
+
+ if create_related_formats:
+ # Schedule creation of related formats
+ create_related_formats_for_bibdocs[rename or bibdoc_name] = True
+
+ if doctype in create_icon_doctypes or '*' in create_icon_doctypes:
+ # Schedule creation of icons when possible.
+ create_icons_for_bibdocs[rename or bibdoc_name] = True
+
elif action == 'delete':
delete(bibdoc_name, recid, working_dir, pending_bibdocs,
bibrecdocs)
# Finally rename bibdocs that should be named according to a file in
# curdir (eg. naming according to report number). Only consider
# file that have just been added.
parameters = _read_file_revision_interface_configuration_from_disk(working_dir)
new_names = []
doctypes_to_default_filename = parameters[22]
for bibdoc_to_rename in newly_added_bibdocs:
bibdoc_to_rename_doctype = bibdoc_to_rename.doctype
rename_to = doctypes_to_default_filename.get(bibdoc_to_rename_doctype, '')
if rename_to.startswith('file:'):
# This BibDoc must be renamed. Look for name in working dir
name_at_filepath = os.path.join(working_dir, rename_to[5:])
if os.path.exists(name_at_filepath) and \
os.path.abspath(name_at_filepath).startswith(working_dir):
try:
rename = file(name_at_filepath).read()
except:
register_exception(prefix='Move_Uploaded_Files_to_Storage ' \
'could not read file %s in curdir to rename bibdoc' % \
(name_at_filepath,),
alert_admin=True)
if rename:
file_counter = 2
new_filename = rename
while bibrecdocs.has_docname_p(new_filename) or (new_filename in new_names):
new_filename = rename + '_%i' % file_counter
file_counter += 1
+ if create_related_formats_for_bibdocs.has_key(bibdoc_to_rename.get_docname()):
+ create_related_formats_for_bibdocs[bibdoc_to_rename.get_docname()] = new_filename
+ if create_icons_for_bibdocs.has_key(bibdoc_to_rename.get_docname()):
+ create_icons_for_bibdocs[bibdoc_to_rename.get_docname()] = new_filename
bibdoc_to_rename.change_name(new_filename)
new_names.append(new_filename) # keep track of name, or we have to reload bibrecdoc...
_do_log(working_dir, 'Renamed ' + bibdoc_to_rename.get_docname())
# Delete the HB BibFormat cache in the DB, so that the fulltext
# links do not point to possible dead files
run_sql("DELETE LOW_PRIORITY from bibfmt WHERE format='HB' AND id_bibrec=%s", (recid,))
# Update the MARC
cli_fix_marc(None, [recid], interactive=False)
+ # Schedule related formats creation for selected BibDoc
+ if create_related_formats_for_bibdocs:
+ additional_params = []
+ # Add task sequence ID
+ additional_params.append('-I')
+ additional_params.append(str(sequence_id))
+ additional_params.append("-a")
+ additional_params.append("docnames=%s" % '/'.join(create_related_formats_for_bibdocs.keys()))
+ task_low_level_submission('bibtasklet', 'bibdocfile', '-N', 'createFormats', '-T', 'bst_create_related_formats', '-a', 'recid=%s' % recid, *additional_params)
+
+ # Schedule icons creation for selected BibDoc
+ additional_params = []
+ # Add task sequence ID
+ additional_params.append('-I')
+ additional_params.append(str(sequence_id))
+ additional_params.append("-a")
+ additional_params.append("docnames=%s" % '/'.join(create_icons_for_bibdocs.keys()))
+ additional_params.append("-a")
+ additional_params.append("icon_sizes=%s" % ','.join(icon_sizes))
+ additional_params.append('-a')
+ additional_params.append("add_default_icon=1")
+ additional_params.append('-a')
+ additional_params.append("inherit_moreinfo=1")
+ task_low_level_submission('bibtasklet', 'bibdocfile', '-N', 'createIcons', '-T', 'bst_create_icons', '-a', 'recid=%s' % recid, *additional_params)
def add(file_path, bibdoc_name, rename, doctype, description, comment,
file_restriction, recid, working_dir, icon_sizes, create_icon_doctypes,
pending_bibdocs, bibrecdocs):
"""
Adds the file using bibdocfile CLI
Return the bibdoc that has been newly added.
"""
try:
brd = BibRecDocs(recid)
if os.path.exists(file_path):
# Add file
bibdoc = bibrecdocs.add_new_file(file_path,
doctype,
rename or bibdoc_name,
never_fail=True)
_do_log(working_dir, 'Added ' + brd.get_docname(bibdoc.id) + ': ' + \
file_path)
- # Add icon
- iconpath = ''
- has_added_default_icon_subformat_p = False
- for icon_size in icon_sizes:
- if doctype in create_icon_doctypes or \
- '*' in create_icon_doctypes:
- iconpath = _create_icon(file_path, icon_size)
- if iconpath is not None:
- try:
- if not has_added_default_icon_subformat_p:
- bibdoc.add_icon(iconpath)
- has_added_default_icon_subformat_p = True
- else:
- icon_suffix = icon_size.replace('>', '').replace('<', '').replace('^', '').replace('!', '')
- bibdoc.add_icon(iconpath, subformat=CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT + "-" + icon_suffix)
- _do_log(working_dir, 'Added icon to ' + \
- brd.get_docname(bibdoc.id) + ': ' + iconpath)
- except InvenioBibDocFileError as e:
- # Most probably icon already existed.
- pass
-
# Add description
if description:
bibdocfiles = bibdoc.list_latest_files()
for bibdocfile in bibdocfiles:
bibdoc.set_description(description,
bibdocfile.get_format())
_do_log(working_dir, 'Described ' + \
brd.get_docname(bibdoc.id) + ': ' + description)
# Add comment
if comment:
bibdocfiles = bibdoc.list_latest_files()
for bibdocfile in bibdocfiles:
bibdoc.set_comment(comment,
bibdocfile.get_format())
_do_log(working_dir, 'Commented ' + \
brd.get_docname(bibdoc.id) + ': ' + comment)
# Set restriction
bibdoc.set_status(file_restriction)
_do_log(working_dir, 'Set restriction of ' + \
brd.get_docname(bibdoc.id) + ': ' + \
file_restriction or '(no restriction)')
return bibdoc
else:
# File has been later renamed or deleted.
# Remember to add it later if file is found (ie
# it was renamed)
pending_bibdocs[bibdoc_name] = (doctype, comment, description, [])
except InvenioBibDocFileError as e:
# Format already existed. How come? We should
# have checked this in Create_Upload_Files_Interface.py
register_exception(prefix='Move_Uploaded_Files_to_Storage ' \
'tried to add already existing file %s ' \
'with name %s to record %i.' % \
(file_path, bibdoc_name, recid),
alert_admin=True)
def add_format(file_path, bibdoc_name, recid, doctype, working_dir,
icon_sizes, create_icon_doctypes, pending_bibdocs,
bibrecdocs):
"""
Adds a new format to a bibdoc using bibdocfile CLI
"""
try:
brd = BibRecDocs(recid)
if os.path.exists(file_path):
# We must retrieve previous description and comment as
# adding a file using the APIs reset these values
prev_desc, prev_comment = None, None
if bibrecdocs.has_docname_p(bibdoc_name):
(prev_desc, prev_comment) = \
get_description_and_comment(bibrecdocs.get_bibdoc(bibdoc_name).list_latest_files())
# Add file
bibdoc = bibrecdocs.add_new_format(file_path,
bibdoc_name,
prev_desc,
prev_comment)
_do_log(working_dir, 'Added new format to ' + \
brd.get_docname(bibdoc.id) + ': ' + file_path)
- # Add icons
- has_added_default_icon_subformat_p = False
- for icon_size in icon_sizes:
- iconpath = ''
- if doctype in create_icon_doctypes or \
- '*' in create_icon_doctypes:
- iconpath = _create_icon(file_path, icon_size)
- if iconpath is not None:
- try:
- if not has_added_default_icon_subformat_p:
- bibdoc.add_icon(iconpath)
- has_added_default_icon_subformat_p = True
- else:
- # We have already added the "default" icon subformat
- icon_suffix = icon_size.replace('>', '').replace('<', '').replace('^', '').replace('!', '')
-
- bibdoc.add_icon(iconpath, subformat=CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT + "-" + icon_suffix)
- _do_log(working_dir, 'Added icon to ' + \
- brd.get_docname(bibdoc.id) + ': ' + iconpath)
- except InvenioBibDocFileError as e:
- # Most probably icon already existed.
- pass
else:
# File has been later renamed or deleted.
# Remember to add it later if file is found
if bibdoc_name in pending_bibdocs:
pending_bibdocs[bibdoc_name][3].append(file_path)
# else: we previously added a file by mistake. Do
# not care, it will be deleted
except InvenioBibDocFileError as e:
# Format already existed. How come? We should
# have checked this in Create_Upload_Files_Interface.py
register_exception(prefix='Move_Uploaded_Files_to_Storage ' \
'tried to add already existing format %s ' \
'named %s in record %i.' % \
(file_path, bibdoc_name, recid),
alert_admin=True)
def revise(file_path, bibdoc_name, rename, doctype, description,
comment, file_restriction, icon_sizes, create_icon_doctypes,
keep_previous_versions, recid, working_dir, pending_bibdocs,
bibrecdocs, force_file_revision):
"""
Revises the given bibdoc with a new file.
Return the bibdoc that has been newly added. (later: if needed,
return as tuple the bibdoc that has been revised, or deleted,
etc.)
"""
added_bibdoc = None
try:
if os.path.exists(file_path) or not file_path:
brd = BibRecDocs(recid)
# Perform pending actions
if bibdoc_name in pending_bibdocs:
# We have some pending actions to apply before
# going further.
if description == '':
# Last revision did not include a description.
# Use the one of the pending actions
description = pending_bibdocs[bibdoc_name][2]
if comment == '':
# Last revision did not include a comment.
# Use the one of the pending actions
comment = pending_bibdocs[bibdoc_name][1]
original_bibdoc_name = pending_bibdocs[bibdoc_name][0]
if not bibrecdocs.has_docname_p(original_bibdoc_name) and file_path:
# the bibdoc did not originaly exist, so it
# must be added first
bibdoc = bibrecdocs.add_new_file(file_path,
pending_bibdocs[bibdoc_name][0],
bibdoc_name,
never_fail=True)
_do_log(working_dir, 'Added ' + brd.get_docname(bibdoc.id) + ': ' + \
file_path)
added_bibdoc = bibdoc
# Set restriction
bibdoc.set_status(file_restriction)
_do_log(working_dir, 'Set restriction of ' + \
bibrecdocs.get_docname(bibdoc.id) + ': ' + \
file_restriction or '(no restriction)')
# We must retrieve previous description and comment as
# revising a file using the APIs reset these values
prev_desc, prev_comment = None, None
if bibrecdocs.has_docname_p(bibdoc_name):
(prev_desc, prev_comment) = \
get_description_and_comment(bibrecdocs.get_bibdoc(bibdoc_name).list_latest_files())
# Do we have additional formats?
for additional_format in pending_bibdocs[bibdoc_name][3]:
if os.path.exists(additional_format):
bibdoc.add_file_new_format(additional_format,
description=bibdoc.get_description(),
comment=bibdoc.get_comment())
_do_log(working_dir, 'Added new format to' + \
brd.get_docname(bibdoc.id) + ': ' + file_path)
# All pending modification have been applied,
# so delete
del pending_bibdocs[bibdoc_name]
# We must retrieve previous description and comment as
# revising a file using the APIs reset these values
prev_desc, prev_comment = None, None
if bibrecdocs.has_docname_p(bibdoc_name):
(prev_desc, prev_comment) = \
get_description_and_comment(bibrecdocs.get_bibdoc(bibdoc_name).list_latest_files())
if keep_previous_versions and file_path:
# Standard procedure, keep previous version
bibdoc = bibrecdocs.add_new_version(file_path,
bibdoc_name,
prev_desc,
prev_comment)
_do_log(working_dir, 'Revised ' + brd.get_docname(bibdoc.id) + \
' with : ' + file_path)
elif file_path:
# Soft-delete previous versions, and add new file
# (we need to get the doctype before deleting)
if bibrecdocs.has_docname_p(bibdoc_name):
# Delete only if bibdoc originally
# existed
bibrecdocs.delete_bibdoc(bibdoc_name)
_do_log(working_dir, 'Deleted ' + bibdoc_name)
try:
bibdoc = bibrecdocs.add_new_file(file_path,
doctype,
bibdoc_name,
never_fail=True,
description=prev_desc,
comment=prev_comment)
_do_log(working_dir, 'Added ' + brd.get_docname(bibdoc.id) + ': ' + \
file_path)
except InvenioBibDocFileError as e:
_do_log(working_dir, str(e))
register_exception(prefix='Move_Uploaded_Files_to_Storage ' \
'tried to revise a file %s ' \
'named %s in record %i.' % \
(file_path, bibdoc_name, recid),
alert_admin=True)
else:
# User just wanted to change attribute of the file,
# not the file itself
bibdoc = bibrecdocs.get_bibdoc(bibdoc_name)
(prev_desc, prev_comment) = \
get_description_and_comment(bibdoc.list_latest_files())
if prev_desc is None:
prev_desc = ""
if prev_comment is None:
prev_comment = ""
if force_file_revision and \
(description != prev_desc or comment != prev_comment):
# FIXME: If we are going to create a new version,
# then we should honour the keep_previous_versions
# parameter (soft-delete, then add bibdoc, etc)
# But it is a bit complex right now...
# Trick: we revert to current version, which
# creates a revision of the BibDoc
bibdoc.revert(bibdoc.get_latest_version())
bibdoc = bibrecdocs.get_bibdoc(bibdoc_name)
# Rename
if rename and rename != bibdoc_name:
bibrecdocs.change_name(newname=rename, docid=bibdoc.id)
_do_log(working_dir, 'renamed ' + bibdoc_name +' to '+ rename)
- # Add icons
- if file_path:
- has_added_default_icon_subformat_p = False
- for icon_size in icon_sizes:
- iconpath = ''
- if doctype in create_icon_doctypes or \
- '*' in create_icon_doctypes:
- iconpath = _create_icon(file_path, icon_size)
- if iconpath is not None:
- try:
- if not has_added_default_icon_subformat_p:
- bibdoc.add_icon(iconpath)
- has_added_default_icon_subformat_p = True
- else:
- # We have already added the "default" icon subformat
- icon_suffix = icon_size.replace('>', '').replace('<', '').replace('^', '').replace('!', '')
- bibdoc.add_icon(iconpath, subformat=CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT + "-" + icon_suffix)
- _do_log(working_dir, 'Added icon to ' + \
- brd.get_docname(bibdoc.id) + ': ' + iconpath)
- except InvenioBibDocFileError as e:
- # Most probably icon already existed.
- pass
-
# Description
if description:
bibdocfiles = bibdoc.list_latest_files()
for bibdocfile in bibdocfiles:
bibdoc.set_description(description,
bibdocfile.get_format())
_do_log(working_dir, 'Described ' + \
brd.get_docname(bibdoc.id) + ': ' + description)
# Comment
if comment:
bibdocfiles = bibdoc.list_latest_files()
for bibdocfile in bibdocfiles:
bibdoc.set_comment(comment,
bibdocfile.get_format())
_do_log(working_dir, 'Commented ' + \
brd.get_docname(bibdoc.id) + ': ' + comment)
# Set restriction
bibdoc.set_status(file_restriction)
_do_log(working_dir, 'Set restriction of ' + \
brd.get_docname(bibdoc.id) + ': ' + \
file_restriction or '(no restriction)')
else:
# File has been later renamed or deleted.
# Remember it
if rename and rename != bibdoc_name:
pending_bibdocs[rename] = pending_bibdocs[bibdoc_name]
except InvenioBibDocFileError as e:
# Format already existed. How come? We should
# have checked this in Create_Upload_Files_Interface.py
register_exception(prefix='Move_Uploaded_Files_to_Storage ' \
'tried to revise a file %s ' \
'named %s in record %i.' % \
(file_path, bibdoc_name, recid),
alert_admin=True)
return added_bibdoc
def delete(bibdoc_name, recid, working_dir, pending_bibdocs,
bibrecdocs):
"""
Deletes the given bibdoc
"""
try:
if bibrecdocs.has_docname_p(bibdoc_name):
bibrecdocs.delete_bibdoc(bibdoc_name)
_do_log(working_dir, 'Deleted ' + bibdoc_name)
if bibdoc_name in pending_bibdocs:
del pending_bibdocs[bibdoc_name]
except InvenioBibDocFileError as e:
# Mmh most probably we deleted two files at the same
# second. Sleep 1 second and retry... This might go
# away one bibdoc improves its way to delete files
try:
time.sleep(1)
bibrecdocs.delete_bibdoc(bibdoc_name)
_do_log(working_dir, 'Deleted ' + bibdoc_name)
if bibdoc_name in pending_bibdocs:
del pending_bibdocs[bibdoc_name]
except InvenioBibDocFileError as e:
_do_log(working_dir, str(e))
_do_log(working_dir, repr(bibrecdocs.list_bibdocs()))
register_exception(prefix='Move_Uploaded_Files_to_Storage ' \
'tried to delete a file' \
'named %s in record %i.' % \
(bibdoc_name, recid),
alert_admin=True)
def _do_log(log_dir, msg):
"""
Log what we have done, in case something went wrong.
Nice to compare with bibdocactions.log
Should be removed when the development is over.
@param log_dir: the path to the working directory
@type log_dir: string
@param msg: the message to log
@type msg: string
"""
log_file = os.path.join(log_dir, 'performed_actions.log')
file_desc = open(log_file, "a+")
file_desc.write("%s --> %s\n" %(time.strftime("%Y-%m-%d %H:%M:%S"), msg))
file_desc.close()
+
def _create_icon(file_path, icon_size, docformat='gif', verbosity=9):
"""
Creates icon of given file.
Returns path to the icon. If creation fails, return None, and
register exception (send email to admin).
@param file_path: full path to icon
@type file_path: string
@param icon_size: the scaling information to be used for the
creation of the new icon.
@type icon_size: int
@param verbosity: the verbosity level under which the program
is to run;
@type verbosity: int
"""
icon_path = None
try:
filename = os.path.splitext(os.path.basename(file_path))[0]
(icon_dir, icon_name) = create_icon(
{'input-file':file_path,
'icon-name': "icon-%s" % filename,
'multipage-icon': False,
'multipage-icon-delay': 0,
'icon-scale': icon_size,
'icon-file-format': docformat,
'verbosity': verbosity})
icon_path = icon_dir + os.sep + icon_name
except InvenioWebSubmitIconCreatorError as e:
register_exception(prefix='Icon for file %s could not be created: %s' % \
(file_path, str(e)),
alert_admin=False)
return icon_path
+
def get_upload_file_interface_javascript(form_url_params):
"""
Returns the Javascript code necessary to run the upload file
interface.
"""
javascript = '''
<script type="text/javascript" src="/js/jquery.form.js"></script>
<script type="text/javascript">
<!--
'''
if form_url_params:
javascript += '''
// prepare the form when the DOM is ready
$(document).ready(function() {
var progress = $('.progress');
var rotatingprogress = $('.rotatingprogress');
var bar = $('.bar');
var percent = $('.percent');
var options = {
target: '#uploadFileInterface', // target element(s) to be updated with server response
uploadProgress: function(event, position, total, percentComplete) {
update_progress(progress, bar, percent, percentComplete, rotatingprogress);},
beforeSubmit: function(arr, $form, options) {
show_upload_progress();
return true;},
success: showResponse, // post-submit callback
url: '/%(CFG_SITE_RECORD)s/managedocfilesasync%(form_url_params)s' // override for form's 'action' attribute
};
// bind form using 'ajaxForm'
var this_form = $('form:has(#balloonReviseFileInput)')
$('#bibdocfilemanagedocfileuploadbutton').click(function() {
this_form.bibdocfilemanagedocfileuploadbuttonpressed=true;
this_form.ajaxSubmit(options);
})
});
// post-submit callback
function showResponse(responseText, statusText) {
hide_upload_progress();
hide_revise_panel();
}
''' % {
'form_url_params': form_url_params,
'CFG_SITE_RECORD': CFG_SITE_RECORD}
javascript += '''
/* Record position of the last clicked link that triggered the display
* of the revise panel
*/
var last_clicked_link = null;
function display_revise_panel(link, params){
var action = params['action'];
var target = params['target'];
var showDoctypes = params['showDoctypes'];
var showKeepPreviousVersions = params['showKeepPreviousVersions'];
var showRename = params['showRename'];
var showDescription = params['showDescription'];
var showComment = params['showComment'];
var bibdocname = params['bibdocname'];
var description = params['description'];
var comment = params['comment'];
var showRestrictions = params['showRestrictions'];
var restriction = params['restriction'];
var doctypes = params['doctypes'];
var balloon = document.getElementById("balloon");
var file_input_block = document.getElementById("balloonReviseFileInputBlock");
var doctype = document.getElementById("fileDoctypesRow");
var warningFormats = document.getElementById("warningFormats");
var keepPreviousVersions = document.getElementById("keepPreviousVersions");
var renameBox = document.getElementById("renameBox");
var descriptionBox = document.getElementById("descriptionBox");
var commentBox = document.getElementById("commentBox");
var restrictionBox = document.getElementById("restrictionBox");
var apply_button = document.getElementById("applyChanges");
var mainForm = getMainForm();
last_clicked_link = link;
var pos;
/* Show/hide parts of the form */
if (showDoctypes) {
doctype.style.display = ''
} else {
doctype.style.display = 'none'
}
if (action == 'revise' && showKeepPreviousVersions == true){
warningFormats.style.display = ''
} else {
warningFormats.style.display = 'none'
}
if ((action == 'revise' || action == 'add') && showRename == true){
renameBox.style.display = ''
} else {
renameBox.style.display = 'none'
}
if ((action == 'revise' || action == 'add') && showDescription == true){
descriptionBox.style.display = ''
} else {
descriptionBox.style.display = 'none'
}
if ((action == 'revise' || action == 'add') && showComment == true){
commentBox.style.display = ''
} else {
commentBox.style.display = 'none'
}
if ((action == 'revise' || action == 'add') && showRestrictions == true){
restrictionBox.style.display = ''
} else {
restrictionBox.style.display = 'none'
}
if (action == 'revise' && showKeepPreviousVersions == true) {
keepPreviousVersions.style.display = ''
} else {
keepPreviousVersions.style.display = 'none'
}
if (action == 'add') {
updateForm();
}
/* Reset values */
file_input_block.innerHTML = file_input_block.innerHTML; // Trick to reset input field
doctype.innerHTML = doctypes;
mainForm.balloonReviseFileKeep.checked = true;
mainForm.rename.value = bibdocname;
mainForm.comment.value = comment;
mainForm.description.value = description;
var fileRestrictionFound = false;
for (var i=0; i < mainForm.fileRestriction.length; i++) {
if (mainForm.fileRestriction[i].value == restriction) {
mainForm.fileRestriction.selectedIndex = i;
fileRestrictionFound = true;
}
}
if (!fileRestrictionFound) {
var restrictionItem = new Option(restriction, restriction);
mainForm.fileRestriction.appendChild(restrictionItem);
var lastIndex = mainForm.fileRestriction.length - 1;
mainForm.fileRestriction.selectedIndex = lastIndex;
}
/* Display and move to correct position*/
pos = findPosition(link)
balloon.style.display = '';
balloon.style.position="absolute";
balloon.style.left = pos[0] + link.offsetWidth +"px";
balloon.style.top = pos[1] - Math.round(balloon.offsetHeight/2) + 5 + "px";
balloon.style.zIndex = 1001;
balloon.style.display = '';
/* Set the correct action and target file*/
mainForm.fileAction.value = action;
mainForm.fileTarget.value = target;
/* Disable other controls */
if (apply_button) {
apply_button.disabled = true;
}
/*gray_out(true);*/
}
function hide_revise_panel(){
var balloon = document.getElementById("balloon");
var apply_button = document.getElementById("applyChanges");
balloon.style.display = 'none';
if (apply_button) {
apply_button.disabled = false;
}
/*gray_out(false);*/
}
/* Intercept ESC key in order to close revise panel*/
document.onkeyup = keycheck;
function keycheck(e){
var KeyID = (window.event) ? event.keyCode : e.keyCode;
var upload_in_progress_p = $('.progress').is(":visible") || $('.rotatingprogress').is(":visible")
if(KeyID==27){
if (upload_in_progress_p) {
hide_upload_progress();
} else {
hide_revise_panel();
}
}
}
/* Update progress bar, show if necessary (and then hide rotating progress indicator) */
function update_progress(progress, bar, percent, percentComplete, rotatingprogress){
if (rotatingprogress.is(":visible")) {
$('.rotatingprogress').hide();
$('.progress').show();
}
var percentVal = percentComplete + '%%';
bar.width(percentVal)
percent.html(percentVal);
if (percentComplete == '100') {
// There might be some lengthy post-processing to do.
show_upload_progress(post_process_label=true);
}
}
/* Hide upload/cancel button, show rotating progress indicator */
function show_upload_progress(post_process_label_p) {
if (!post_process_label_p) { post_process_label_p = false;}
if (post_process_label_p) {
/* Show post-process label */
$('.progress').hide();
$('.rotatingprogress').hide();
$('.rotatingpostprocess').show();
} else {
/* Show uploading label */
$('#canceluploadbuttongroup').hide();
$('.rotatingprogress').show();
}
}
/* show upload/cancel button, hide any progress indicator */
function hide_upload_progress() {
$('.progress').hide();
$('.rotatingprogress').hide();
$('.rotatingpostprocess').hide();
$('#canceluploadbuttongroup').show();
$('.percent').html('0%%');
}
function findPosition( oElement ) {
/*Return the x,y position on page of the given object*/
if( typeof( oElement.offsetParent ) != 'undefined' ) {
for( var posX = 0, posY = 0; oElement; oElement = oElement.offsetParent ) {
posX += oElement.offsetLeft;
posY += oElement.offsetTop;
}
return [ posX, posY ];
} else {
return [ oElement.x, oElement.y ];
}
}
function getMainForm()
{
return $('form:has(#balloonReviseFileInput)')[0];
}
function nextStep()
{
if(confirm("You are about to submit the files and end the upload process."))
{
var mainForm = getMainForm();
mainForm.step.value = 2;
user_must_confirm_before_leaving_page = false;
mainForm.submit();
}
return true;
}
function updateForm(doctype, can_describe_doctypes, can_comment_doctypes, can_restrict_doctypes) {
/* Update the revision panel to hide or not part of the interface
* based on selected doctype
*
* Note: we use a small trick here to use the javascript 'in' operator, which
* does not work for arrays, but for object => we transform our arrays into
* objects literal
*/
/* Get the elements we are going to affect */
var renameBox = document.getElementById("renameBox");
var descriptionBox = document.getElementById("descriptionBox");
var commentBox = document.getElementById("commentBox");
var restrictionBox = document.getElementById("restrictionBox");
if (!can_describe_doctypes) {var can_describe_doctypes = [];}
if (!can_comment_doctypes) {var can_comment_doctypes = [];}
if (!can_restrict_doctypes) {var can_restrict_doctypes = [];}
if ((doctype in can_describe_doctypes) ||
('*' in can_describe_doctypes)){
descriptionBox.style.display = ''
} else {
descriptionBox.style.display = 'none'
}
if ((doctype in can_comment_doctypes) ||
('*' in can_comment_doctypes)){
commentBox.style.display = ''
} else {
commentBox.style.display = 'none'
}
if ((doctype in can_restrict_doctypes) ||
('*' in can_restrict_doctypes)){
restrictionBox.style.display = ''
} else {
restrictionBox.style.display = 'none'
}
/* Move the revise panel accordingly */
var balloon = document.getElementById("balloon");
pos = findPosition(last_clicked_link)
balloon.style.display = '';
balloon.style.position="absolute";
balloon.style.left = pos[0] + last_clicked_link.offsetWidth +"px";
balloon.style.top = pos[1] - Math.round(balloon.offsetHeight/2) + 5 + "px";
balloon.style.zIndex = 1001;
balloon.style.display = '';
}
function askDelete(bibdocname, form_url_params){
/*
Ask user if she wants to delete file
*/
if (confirm('Are you sure you want to delete '+bibdocname+'?'))
{
if (form_url_params) {
var mainForm = getMainForm();
mainForm.fileTarget.value = bibdocname;
mainForm.fileAction.value='delete';
user_must_confirm_before_leaving_page = false;
var options = {
target: '#uploadFileInterface',
success: showResponse,
url: '/%(CFG_SITE_RECORD)s/managedocfilesasync' + form_url_params
};
$(mainForm).ajaxSubmit(options);
} else {
/*WebSubmit function*/
document.forms[0].fileTarget.value = bibdocname;
document.forms[0].fileAction.value='delete';
user_must_confirm_before_leaving_page = false;
document.forms[0].submit();
}
}
return false;
}
function gray_out(visible) {
/* Gray out the screen so that user cannot click anywhere else.
Based on <http://www.hunlock.com/blogs/Snippets:_Howto_Grey-Out_The_Screen>
*/
var modalShield = document.getElementById('modalShield');
if (!modalShield) {
var tbody = document.getElementsByTagName("body")[0];
var tnode = document.createElement('div');
tnode.style.position = 'absolute';
tnode.style.top = '0px';
tnode.style.left = '0px';
tnode.style.overflow = 'hidden';
tnode.style.display = 'none';
tnode.id = 'modalShield';
tbody.appendChild(tnode);
modalShield = document.getElementById('modalShield');
}
if (visible){
// Calculate the page width and height
var pageWidth = '100%%';
var pageHeight = '100%%';
//set the shader to cover the entire page and make it visible.
modalShield.style.opacity = 0.7;
modalShield.style.MozOpacity = 0.7;
modalShield.style.filter = 'alpha(opacity=70)';
modalShield.style.zIndex = 1000;
modalShield.style.backgroundColor = '#000000';
modalShield.style.width = pageWidth;
modalShield.style.height = pageHeight;
modalShield.style.display = 'block';
} else {
modalShield.style.display = 'none';
}
}
-->
</script>
''' % {'CFG_SITE_RECORD': CFG_SITE_RECORD}
return javascript
def get_upload_file_interface_css():
"""
Returns the CSS to embed in the page for the upload file interface.
"""
# The CSS embedded in the page for the revise panel
css = '''
<style type="text/css">
<!--
#reviseControl{
overflow:auto;
width: 600px;
padding:1px;
}
.reviseControlBrowser{
padding:5px;
background-color:#fff;
border-collapse:collapse;
border-spacing: 0px;
border: 1px solid #999;
}
.reviseControlFileColumn {
padding-right:60px;
padding-left:5px;
text-align: left;
color:#00f;
}
.reviseControlActionColumn,
.reviseControlFormatColumn{
font-size:small;
}
.reviseControlActionColumn,
.reviseControlActionColumn a,
.reviseControlActionColumn a:link,
.reviseControlActionColumn a:hover
.reviseControlActionColumn a:visited{
font-size:small;
color: #060;
text-align:right;
}
.reviseControlFormatColumn,
.reviseControlFormatColumn a,
.reviseControlFormatColumn a:link,
.reviseControlFormatColumn a:hover
.reviseControlFormatColumn a:visited{
font-size:small;
color: #555;
text-align:left;
}
+.reviseControlFormatToBeCreated{
+font-style:italic;
+color: #aaa;
+}
.optional{
color: #555;
font-size:0.9em;
font-weight:normal
}
.even{
background-color:#ecf3fe;
}
/*
.buttonLikeLink, .buttonLikeLink:visited, .buttonLikeLink:hover{
background-color:#fff;
border:2px outset #555;
color:#000;
padding: 2px 5px;
display:inline-block;
margin:2px;
text-decoration:none;
font-size:small;
cursor: default
}
*/
#balloon table{
border-collapse:collapse;
border-spacing: 0px;
}
#balloon table td.topleft{
background: transparent url(%(CFG_SITE_URL)s/img/balloon_top_left_shadow.png) no-repeat bottom right;
}
#balloon table td.bottomleft{
background: transparent url(%(CFG_SITE_URL)s/img/balloon_bottom_left_shadow.png) no-repeat top right;
}
#balloon table td.topright{
background: transparent url(%(CFG_SITE_URL)s/img/balloon_top_right_shadow.png) no-repeat bottom left;
}
#balloon table td.bottomright{
background: transparent url(%(CFG_SITE_URL)s/img/balloon_bottom_right_shadow.png) no-repeat top left;
}
#balloon table td.top{
background: transparent url(%(CFG_SITE_URL)s/img/balloon_top_shadow.png) repeat-x bottom left;
}
#balloon table td.bottom{
background: transparent url(%(CFG_SITE_URL)s/img/balloon_bottom_shadow.png) repeat-x top left;
}
#balloon table td.left{
background: transparent url(%(CFG_SITE_URL)s/img/balloon_left_shadow.png) repeat-y top right;
text-align:right;
padding:0;
}
#balloon table td.right{
background: transparent url(%(CFG_SITE_URL)s/img/balloon_right_shadow.png) repeat-y top left;
}
#balloon table td.arrowleft{
background: transparent url(%(CFG_SITE_URL)s/img/balloon_arrow_left_shadow.png) no-repeat bottom right;
width:24px;
height:27px;
}
#balloon table td.center{
background-color:#ffffea;
}
#balloon label{
font-size:small;
}
#balloonReviseFile{
width:220px;
text-align:left;
}
#warningFormats{
color:#432e11;
font-size:x-small;
text-align:center;
margin: 4px auto 4px auto;
}
#fileDoctype {
margin-bottom:3px;
}
#renameBox, #descriptionBox, #commentBox, #keepPreviousVersions{
margin-top:6px;
}
#description, #comment, #rename {
width:90%%;
}
.rotatingprogress, .rotatingpostprocess {
position:relative;
float:right;
padding: 1px;
font-style:italic;
font-size:small;
margin-right: 5px;
display:none;
}
.progress {
position:relative;
width:100%%;
float:left;
border: 1px solid #ddd;
padding: 1px;
border-radius: 3px;
display:none;
}
.bar {
background-color: #dd9700;
width:0%%; height:20px;
border-radius: 3px; }
.percent {
position:absolute;
display:inline-block;
top:3px;
left:45%%;
font-size:small;
color: #514100;
}
-->
</style>
''' % {'CFG_SITE_URL': CFG_SITE_URL}
return css
# The HTML markup of the revise panel
revise_balloon = '''
<div id="balloon" style="display:none;">
<input type="hidden" name="fileAction" value="" />
<input type="hidden" name="fileTarget" value="" />
<table>
<tr>
<td class="topleft">&nbsp;</td>
<td class="top">&nbsp;</td>
<td class="topright">&nbsp;</td>
</tr>
<tr>
<td class="left" vertical-align="center" width="24"><img alt=" " src="../img/balloon_arrow_left_shadow.png" /></td>
<td class="center">
<table id="balloonReviseFile">
<tr>
<td><label for="balloonReviseFileInput">%(file_label)s:</label><br/>
<div style="display:none" id="fileDoctypesRow"></div>
<div id="balloonReviseFileInputBlock"><input type="file" name="myfile" id="balloonReviseFileInput" size="20" /></div>
<!-- <input type="file" name="myfile" id="balloonReviseFileInput" size="20" onchange="var name=getElementById('rename');var filename=this.value.split('/').pop().split('.')[0];name.value=filename;"/> -->
<div id="renameBox" style=""><label for="rename">%(filename_label)s:</label><br/><input type="text" name="rename" id="rename" size="20" autocomplete="off"/></div>
<div id="descriptionBox" style=""><label for="description">%(description_label)s:</label><br/><input type="text" name="description" id="description" size="20" autocomplete="off"/></div>
<div id="commentBox" style=""><label for="comment">%(comment_label)s:</label><br/><textarea name="comment" id="comment" rows="3"/></textarea></div>
<div id="restrictionBox" style="display:none;white-space:nowrap;">%(restrictions)s</div>
<div id="keepPreviousVersions" style="display:none"><input type="checkbox" id="balloonReviseFileKeep" name="keepPreviousFiles" checked="checked" /><label for="balloonReviseFileKeep">%(previous_versions_label)s</label>&nbsp;<small>[<a href="" onclick="alert('%(previous_versions_help)s');return false;">?</a>]</small></div>
<p id="warningFormats" style="display:none"><img src="%(CFG_SITE_URL)s/img/warning.png" alt="Warning"/> %(revise_format_warning)s&nbsp;[<a href="" onclick="alert('%(revise_format_help)s');return false;">?</a>]</p>
<div class="progress"><div class="bar"></div ><div class="percent">0%%</div ></div>
<div class="rotatingprogress"><img src="/img/ui-anim_basic_16x16.gif" /> %(uploading_label)s</div><div class="rotatingpostprocess"><img src="/img/ui-anim_basic_16x16.gif" /> %(postprocess_label)s</div><div id="canceluploadbuttongroup" style="text-align:right;margin-top:5px"><input type="button" value="%(cancel)s" onclick="javascript:hide_revise_panel();"/> <input type="%(submit_or_button)s" id="bibdocfilemanagedocfileuploadbutton" onclick="show_upload_progress()" value="%(upload)s"/></div>
</td>
</tr>
</table>
</td>
<td class="right">&nbsp;</td>
</tr>
<tr>
<td class="bottomleft">&nbsp;</td>
<td class="bottom">&nbsp;</td>
<td class="bottomright">&nbsp;</td>
</tr>
</table>
</div>
'''
diff --git a/invenio/legacy/bibdocfile/plugins/bom_textdoc.py b/invenio/legacy/bibdocfile/plugins/bom_textdoc.py
index a1e3d38f5..ece543202 100644
--- a/invenio/legacy/bibdocfile/plugins/bom_textdoc.py
+++ b/invenio/legacy/bibdocfile/plugins/bom_textdoc.py
@@ -1,142 +1,155 @@
## This file is part of Invenio.
-## Copyright (C) 2007, 2008, 2009, 2010, 2011 CERN.
+## Copyright (C) 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""BibObject Module providing BibObject prividing features for documents containing text (not necessarily as the main part of the content)"""
+import os
+import re
+from datetime import datetime
+from invenio.config import CFG_BIBINDEX_PERFORM_OCR_ON_DOCNAMES
from invenio.legacy.bibdocfile.api import BibDoc, InvenioBibDocFileError
from invenio.legacy.dbquery import run_sql
-from datetime import datetime
from invenio.ext.logging import register_exception
-import os
+
+_RE_PERFORM_OCR = re.compile(CFG_BIBINDEX_PERFORM_OCR_ON_DOCNAMES)
class BibTextDoc(BibDoc):
def get_text(self, version=None):
"""
@param version: the requested version. If not set, the latest version
will be used.
@type version: integer
@return: the textual content corresponding to the specified version
of the document.
@rtype: string
"""
if version is None:
version = self.get_latest_version()
if self.has_text(version):
return open(os.path.join(self.basedir, '.text;%i' % version)).read()
else:
return ""
+ def is_ocr_required(self):
+ """
+ Return True if this document require OCR in order to extract text from it.
+ """
+ for bibrec_link in self.bibrec_links:
+ if _RE_PERFORM_OCR.match(bibrec_link['docname']):
+ return True
+ return False
+
def get_text_path(self, version=None):
"""
@param version: the requested version. If not set, the latest version
will be used.
@type version: int
@return: the full path to the textual content corresponding to the specified version
of the document.
@rtype: string
"""
if version is None:
version = self.get_latest_version()
if self.has_text(version):
return os.path.join(self.basedir, '.text;%i' % version)
else:
return ""
def extract_text(self, version=None, perform_ocr=False, ln='en'):
"""
Try what is necessary to extract the textual information of a document.
@param version: the version of the document for which text is required.
If not specified the text will be retrieved from the last version.
@type version: integer
@param perform_ocr: whether to perform OCR.
@type perform_ocr: bool
@param ln: a two letter language code to give as a hint to the OCR
procedure.
@type ln: string
@raise InvenioBibDocFileError: in case of error.
@note: the text is extracted and cached for later use. Use L{get_text}
to retrieve it.
"""
from invenio.legacy.websubmit.file_converter import get_best_format_to_extract_text_from, convert_file, InvenioWebSubmitFileConverterError
if version is None:
version = self.get_latest_version()
docfiles = self.list_version_files(version)
## We try to extract text only from original or OCRed documents.
filenames = [docfile.get_full_path() for docfile in docfiles if 'CONVERTED' not in docfile.flags or 'OCRED' in docfile.flags]
try:
filename = get_best_format_to_extract_text_from(filenames)
except InvenioWebSubmitFileConverterError:
## We fall back on considering all the documents
filenames = [docfile.get_full_path() for docfile in docfiles]
try:
filename = get_best_format_to_extract_text_from(filenames)
except InvenioWebSubmitFileConverterError:
open(os.path.join(self.basedir, '.text;%i' % version), 'w').write('')
return
try:
convert_file(filename, os.path.join(self.basedir, '.text;%i' % version), '.txt', perform_ocr=perform_ocr, ln=ln)
if version == self.get_latest_version():
run_sql("UPDATE bibdoc SET text_extraction_date=NOW() WHERE id=%s", (self.id, ))
except InvenioWebSubmitFileConverterError as e:
register_exception(alert_admin=True, prefix="Error in extracting text from bibdoc %i, version %i" % (self.id, version))
raise InvenioBibDocFileError, str(e)
def pdf_a_p(self):
"""
@return: True if this document contains a PDF in PDF/A format.
@rtype: bool"""
return self.has_flag('PDF/A', 'pdf')
def has_text(self, require_up_to_date=False, version=None):
"""
Return True if the text of this document has already been extracted.
@param require_up_to_date: if True check the text was actually
extracted after the most recent format of the given version.
@type require_up_to_date: bool
@param version: a version for which the text should have been
extracted. If not specified the latest version is considered.
@type version: integer
@return: True if the text has already been extracted.
@rtype: bool
"""
if version is None:
version = self.get_latest_version()
if os.path.exists(os.path.join(self.basedir, '.text;%i' % version)):
if not require_up_to_date:
return True
else:
docfiles = self.list_version_files(version)
text_md = datetime.fromtimestamp(os.path.getmtime(os.path.join(self.basedir, '.text;%i' % version)))
for docfile in docfiles:
if text_md <= docfile.md:
return False
return True
return False
def __repr__(self):
return 'BibTextDoc(%s, %s, %s)' % (repr(self.id), repr(self.doctype), repr(self.human_readable))
def supports(doctype, extensions):
return doctype == "Fulltext" or reduce(lambda x, y: x or y.startswith(".pdf") or y.startswith(".ps") , extensions, False)
def create_instance(docid=None, doctype='Main', human_readable=False, # pylint: disable=W0613
initial_data = None):
return BibTextDoc(docid=docid, human_readable=human_readable,
initial_data = initial_data)
diff --git a/invenio/legacy/bibdocfile/webinterface.py b/invenio/legacy/bibdocfile/webinterface.py
index 434c35a2f..831e8e1b2 100644
--- a/invenio/legacy/bibdocfile/webinterface.py
+++ b/invenio/legacy/bibdocfile/webinterface.py
@@ -1,549 +1,554 @@
## This file is part of Invenio.
## Copyright (C) 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
import cgi
import os
import time
import shutil
from six import iteritems
from invenio.config import \
CFG_ACCESS_CONTROL_LEVEL_SITE, \
CFG_SITE_LANG, \
CFG_TMPSHAREDDIR, \
CFG_SITE_URL, \
CFG_SITE_SECURE_URL, \
CFG_WEBSUBMIT_STORAGEDIR, \
CFG_SITE_RECORD, \
CFG_INSPIRE_SITE
from invenio.legacy.bibdocfile.config import CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_DOCTYPES, \
CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_MISC, \
CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_RESTRICTIONS, \
CFG_BIBDOCFILE_ICON_SUBFORMAT_RE
from invenio.utils import apache
from invenio.modules.access.local_config import VIEWRESTRCOLL
from invenio.modules.access.mailcookie import mail_cookie_create_authorize_action
from invenio.modules.access.engine import acc_authorize_action
from invenio.modules.access.control import acc_is_role
from invenio.legacy.webpage import page, pageheaderonly, \
pagefooteronly, warning_page, write_warning
from invenio.legacy.webuser import getUid, page_not_authorized, collect_user_info, isUserSuperAdmin, \
isGuestUser
from invenio.legacy.webjournal import utils as webjournal_utils
from invenio.ext.legacy.handler import wash_urlargd, WebInterfaceDirectory
from invenio.utils.url import make_canonical_urlargd, redirect_to_url
from invenio.base.i18n import gettext_set_language
from invenio.legacy.search_engine import \
guess_primary_collection_of_a_record, get_colID, record_exists, \
create_navtrail_links, check_user_can_view_record, record_empty, \
is_user_owner_of_record
from invenio.legacy.bibdocfile.api import BibRecDocs, normalize_format, file_strip_ext, \
stream_restricted_icon, BibDoc, InvenioBibDocFileError, \
get_subformat_from_format
from invenio.ext.logging import register_exception
-from invenio.legacy.websearch.adminlib import get_detailed_page_tabs
+from invenio.legacy.websearch.adminlib import get_detailed_page_tabs, get_detailed_page_tabs_counts
import invenio.legacy.template
bibdocfile_templates = invenio.legacy.template.load('bibdocfile')
webstyle_templates = invenio.legacy.template.load('webstyle')
websubmit_templates = invenio.legacy.template.load('websubmit')
websearch_templates = invenio.legacy.template.load('websearch')
from invenio.legacy.bibdocfile.managedocfiles import \
create_file_upload_interface, \
get_upload_file_interface_javascript, \
get_upload_file_interface_css, \
move_uploaded_files_to_storage
class WebInterfaceFilesPages(WebInterfaceDirectory):
def __init__(self, recid):
self.recid = recid
def _lookup(self, component, path):
# after /<CFG_SITE_RECORD>/<recid>/files/ every part is used as the file
# name
filename = component
def getfile(req, form):
args = wash_urlargd(form, bibdocfile_templates.files_default_urlargd)
ln = args['ln']
_ = gettext_set_language(ln)
uid = getUid(req)
user_info = collect_user_info(req)
verbose = args['verbose']
if verbose >= 1 and not isUserSuperAdmin(user_info):
# Only SuperUser can see all the details!
verbose = 0
if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE > 1:
return page_not_authorized(req, "/%s/%s" % (CFG_SITE_RECORD, self.recid),
navmenuid='submit')
if record_exists(self.recid) < 1:
msg = "<p>%s</p>" % _("Requested record does not seem to exist.")
return warning_page(msg, req, ln)
if record_empty(self.recid):
msg = "<p>%s</p>" % _("Requested record does not seem to have been integrated.")
return warning_page(msg, req, ln)
(auth_code, auth_message) = check_user_can_view_record(user_info, self.recid)
if auth_code and user_info['email'] == 'guest':
if webjournal_utils.is_recid_in_released_issue(self.recid):
# We can serve the file
pass
else:
cookie = mail_cookie_create_authorize_action(VIEWRESTRCOLL, {'collection' : guess_primary_collection_of_a_record(self.recid)})
target = CFG_SITE_SECURE_URL + '/youraccount/login' + \
make_canonical_urlargd({'action': cookie, 'ln' : ln, 'referer' : \
CFG_SITE_SECURE_URL + user_info['uri']}, {})
return redirect_to_url(req, target, norobot=True)
elif auth_code:
if webjournal_utils.is_recid_in_released_issue(self.recid):
# We can serve the file
pass
else:
return page_not_authorized(req, "../", \
text = auth_message)
readonly = CFG_ACCESS_CONTROL_LEVEL_SITE == 1
# From now on: either the user provided a specific file
# name (and a possible version), or we return a list of
# all the available files. In no case are the docids
# visible.
try:
bibarchive = BibRecDocs(self.recid)
except InvenioBibDocFileError:
register_exception(req=req, alert_admin=True)
msg = "<p>%s</p><p>%s</p>" % (
_("The system has encountered an error in retrieving the list of files for this document."),
_("The error has been logged and will be taken in consideration as soon as possible."))
return warning_page(msg, req, ln)
if bibarchive.deleted_p():
req.status = apache.HTTP_GONE
return warning_page(_("Requested record does not seem to exist."), req, ln)
docname = ''
docformat = ''
version = ''
warn = ''
if filename:
# We know the complete file name, guess which docid it
# refers to
## TODO: Change the extension system according to ext.py from setlink
## and have a uniform extension mechanism...
docname = file_strip_ext(filename)
docformat = filename[len(docname):]
if docformat and docformat[0] != '.':
docformat = '.' + docformat
if args['subformat']:
docformat += ';%s' % args['subformat']
else:
docname = args['docname']
if not docformat:
docformat = args['format']
if args['subformat']:
docformat += ';%s' % args['subformat']
if not version:
version = args['version']
## Download as attachment
is_download = False
if args['download']:
is_download = True
# version could be either empty, or all or an integer
try:
int(version)
except ValueError:
if version != 'all':
version = ''
display_hidden = isUserSuperAdmin(user_info)
if version != 'all':
# search this filename in the complete list of files
for doc in bibarchive.list_bibdocs():
if docname == bibarchive.get_docname(doc.id):
try:
try:
docfile = doc.get_file(docformat, version)
except InvenioBibDocFileError as msg:
req.status = apache.HTTP_NOT_FOUND
if not CFG_INSPIRE_SITE and req.headers_in.get('referer'):
## There must be a broken link somewhere.
## Maybe it's good to alert the admin
register_exception(req=req, alert_admin=True)
warn += write_warning(_("The format %(x_form)s does not exist for the given version: %(x_vers)s",
x_form=cgi.escape(docformat), x_vers=cgi.escape(str(msg))))
break
(auth_code, auth_message) = docfile.is_restricted(user_info)
if auth_code != 0 and not is_user_owner_of_record(user_info, self.recid):
if CFG_BIBDOCFILE_ICON_SUBFORMAT_RE.match(get_subformat_from_format(docformat)):
return stream_restricted_icon(req)
if user_info['email'] == 'guest':
cookie = mail_cookie_create_authorize_action('viewrestrdoc', {'status' : docfile.get_status()})
target = CFG_SITE_SECURE_URL + '/youraccount/login' + \
make_canonical_urlargd({'action': cookie, 'ln' : ln, 'referer' : \
CFG_SITE_SECURE_URL + user_info['uri']}, {})
redirect_to_url(req, target)
else:
req.status = apache.HTTP_UNAUTHORIZED
warn += write_warning(_("This file is restricted: ") + str(auth_message))
break
if not docfile.hidden_p():
if not readonly:
ip = str(req.remote_ip)
doc.register_download(ip, docfile.get_version(), docformat, uid, self.recid)
try:
return docfile.stream(req, download=is_download)
except InvenioBibDocFileError as msg:
register_exception(req=req, alert_admin=True)
req.status = apache.HTTP_INTERNAL_SERVER_ERROR
warn += write_warning(_("An error has happened in trying to stream the request file."))
else:
req.status = apache.HTTP_UNAUTHORIZED
warn += write_warning(_("The requested file is hidden and can not be accessed."))
except InvenioBibDocFileError as msg:
register_exception(req=req, alert_admin=True)
if docname and docformat and not warn:
req.status = apache.HTTP_NOT_FOUND
warn += write_warning(_("Requested file does not seem to exist."))
# filelist = bibarchive.display("", version, ln=ln, verbose=verbose, display_hidden=display_hidden)
filelist = bibdocfile_templates.tmpl_display_bibrecdocs(bibarchive, "", version, ln=ln, verbose=verbose, display_hidden=display_hidden)
t = warn + bibdocfile_templates.tmpl_filelist(
ln=ln,
filelist=filelist)
cc = guess_primary_collection_of_a_record(self.recid)
unordered_tabs = get_detailed_page_tabs(get_colID(cc), self.recid, ln)
ordered_tabs_id = [(tab_id, values['order']) for (tab_id, values) in iteritems(unordered_tabs)]
ordered_tabs_id.sort(lambda x, y: cmp(x[1], y[1]))
link_ln = ''
if ln != CFG_SITE_LANG:
link_ln = '?ln=%s' % ln
- tabs = [(unordered_tabs[tab_id]['label'], \
- '%s/%s/%s/%s%s' % (CFG_SITE_URL, CFG_SITE_RECORD, self.recid, tab_id, link_ln), \
+ tabs = [(unordered_tabs[tab_id]['label'],
+ '%s/%s/%s/%s%s' % (CFG_SITE_URL, CFG_SITE_RECORD, self.recid, tab_id, link_ln),
tab_id == 'files',
- unordered_tabs[tab_id]['enabled']) \
+ unordered_tabs[tab_id]['enabled'])
for (tab_id, dummy_order) in ordered_tabs_id
- if unordered_tabs[tab_id]['visible'] == True]
+ if unordered_tabs[tab_id]['visible'] is True]
+
+ tabs_counts = get_detailed_page_tabs_counts(self.recid)
top = webstyle_templates.detailed_record_container_top(self.recid,
tabs,
- args['ln'])
+ args['ln'],
+ citationnum=tabs_counts['Citations'],
+ referencenum=tabs_counts['References'],
+ discussionnum=tabs_counts['Discussions'])
bottom = webstyle_templates.detailed_record_container_bottom(self.recid,
tabs,
args['ln'])
title, description, keywords = websearch_templates.tmpl_record_page_header_content(req, self.recid, args['ln'])
return pageheaderonly(title=title,
navtrail=create_navtrail_links(cc=cc, aas=0, ln=ln) + \
''' &gt; <a class="navtrail" href="%s/%s/%s">%s</a>
&gt; %s''' % \
(CFG_SITE_URL, CFG_SITE_RECORD, self.recid, title, _("Access to Fulltext")),
description=description,
keywords=keywords,
uid=uid,
language=ln,
req=req,
navmenuid='search',
navtrail_append_title_p=0) + \
websearch_templates.tmpl_search_pagestart(ln) + \
top + t + bottom + \
websearch_templates.tmpl_search_pageend(ln) + \
pagefooteronly(language=ln, req=req)
return getfile, []
def __call__(self, req, form):
"""Called in case of URLs like /CFG_SITE_RECORD/123/files without
trailing slash.
"""
args = wash_urlargd(form, bibdocfile_templates.files_default_urlargd)
ln = args['ln']
link_ln = ''
if ln != CFG_SITE_LANG:
link_ln = '?ln=%s' % ln
return redirect_to_url(req, '%s/%s/%s/files/%s' % (CFG_SITE_URL, CFG_SITE_RECORD, self.recid, link_ln))
def bibdocfile_legacy_getfile(req, form):
""" Handle legacy /getfile.py URLs """
args = wash_urlargd(form, {
'recid': (int, 0),
'docid': (int, 0),
'version': (str, ''),
'name': (str, ''),
'format': (str, ''),
'ln' : (str, CFG_SITE_LANG)
})
_ = gettext_set_language(args['ln'])
def _getfile_py(req, recid=0, docid=0, version="", name="", docformat="", ln=CFG_SITE_LANG):
if not recid:
## Let's obtain the recid from the docid
if docid:
try:
bibdoc = BibDoc(docid=docid)
recid = bibdoc.bibrec_links[0]["recid"]
except InvenioBibDocFileError:
return warning_page(_("An error has happened in trying to retrieve the requested file."), req, ln)
else:
return warning_page(_('Not enough information to retrieve the document'), req, ln)
else:
brd = BibRecDocs(recid)
if not name and docid:
## Let's obtain the name from the docid
try:
name = brd.get_docname(docid)
except InvenioBibDocFileError:
return warning_page(_("An error has happened in trying to retrieving the requested file."), req, ln)
docformat = normalize_format(docformat)
redirect_to_url(req, '%s/%s/%s/files/%s%s?ln=%s%s' % (CFG_SITE_URL, CFG_SITE_RECORD, recid, name, docformat, ln, version and 'version=%s' % version or ''), apache.HTTP_MOVED_PERMANENTLY)
return _getfile_py(req, **args)
# --------------------------------------------------
class WebInterfaceManageDocFilesPages(WebInterfaceDirectory):
_exports = ['', 'managedocfiles', 'managedocfilesasync']
def managedocfiles(self, req, form):
"""
Display admin interface to manage files of a record
"""
argd = wash_urlargd(form, {
'ln': (str, ''),
'access': (str, ''),
'recid': (int, None),
'do': (int, 0),
'cancel': (str, None),
})
_ = gettext_set_language(argd['ln'])
uid = getUid(req)
user_info = collect_user_info(req)
# Check authorization
(auth_code, auth_msg) = acc_authorize_action(req,
'runbibdocfile')
if auth_code and user_info['email'] == 'guest':
# Ask to login
target = CFG_SITE_SECURE_URL + '/youraccount/login' + \
make_canonical_urlargd({'ln' : argd['ln'],
'referer' : CFG_SITE_SECURE_URL + user_info['uri']}, {})
return redirect_to_url(req, target)
elif auth_code:
return page_not_authorized(req, referer="/%s/managedocfiles" % CFG_SITE_RECORD,
uid=uid, text=auth_msg,
ln=argd['ln'],
navmenuid="admin")
# Prepare navtrail
navtrail = '''<a class="navtrail" href="%(CFG_SITE_URL)s/help/admin">Admin Area</a> &gt; %(manage_files)s''' \
% {'CFG_SITE_URL': CFG_SITE_URL,
'manage_files': _("Manage Document Files")}
body = ''
if argd['do'] != 0 and not argd['cancel']:
# Apply modifications
working_dir = os.path.join(CFG_TMPSHAREDDIR,
'websubmit_upload_interface_config_' + str(uid),
argd['access'])
if not os.path.isdir(working_dir):
# We accessed the url without preliminary steps
# (we did not upload a file)
# Our working dir does not exist
# Display the file manager
argd['do'] = 0
else:
move_uploaded_files_to_storage(working_dir=working_dir,
recid=argd['recid'],
icon_sizes=['180>', '700>'],
create_icon_doctypes=['*'],
force_file_revision=False)
# Clean temporary directory
shutil.rmtree(working_dir)
# Confirm modifications
body += '<p style="color:#0f0">%s</p>' % \
(_('Your modifications to record #%i have been submitted') % argd['recid'])
elif argd['cancel']:
# Clean temporary directory
working_dir = os.path.join(CFG_TMPSHAREDDIR,
'websubmit_upload_interface_config_' + str(uid),
argd['access'])
shutil.rmtree(working_dir)
body += '<p style="color:#c00">%s</p>' % \
(_('Your modifications to record #%(x_num)i have been cancelled', x_num=argd['recid']))
if not argd['recid'] or argd['do'] != 0:
body += '''
<form method="post" action="%(CFG_SITE_URL)s/%(CFG_SITE_RECORD)s/managedocfiles">
<label for="recid">%(edit_record)s:</label>
<input type="text" name="recid" id="recid" />
<input type="submit" value="%(edit)s" class="adminbutton" />
</form>
''' % {'edit': _('Edit'),
'edit_record': _('Edit record'),
'CFG_SITE_URL': CFG_SITE_URL,
'CFG_SITE_RECORD': CFG_SITE_RECORD}
access = time.strftime('%Y%m%d_%H%M%S')
if argd['recid'] and argd['do'] == 0:
# Displaying interface to manage files
# Prepare navtrail
title, dummy_description, dummy_keywords = websearch_templates.tmpl_record_page_header_content(req, argd['recid'],
argd['ln'])
navtrail = '''<a class="navtrail" href="%(CFG_SITE_URL)s/help/admin">Admin Area</a> &gt;
<a class="navtrail" href="%(CFG_SITE_URL)s/%(CFG_SITE_RECORD)s/managedocfiles">%(manage_files)s</a> &gt;
%(record)s: %(title)s
''' \
% {'CFG_SITE_URL': CFG_SITE_URL,
'title': title,
'manage_files': _("Document File Manager"),
'record': _("Record #%(x_rec)i", x_rec=argd['recid']),
'CFG_SITE_RECORD': CFG_SITE_RECORD}
body += create_file_upload_interface(\
recid=argd['recid'],
ln=argd['ln'],
uid=uid,
sbm_access=access,
display_hidden_files=True,
restrictions_and_desc=CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_RESTRICTIONS,
doctypes_and_desc=CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_DOCTYPES,
**CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_MISC)[1]
body += '''<br />
<form method="post" action="%(CFG_SITE_URL)s/%(CFG_SITE_RECORD)s/managedocfiles">
<input type="hidden" name="recid" value="%(recid)s" />
<input type="hidden" name="do" value="1" />
<input type="hidden" name="access" value="%(access)s" />
<input type="hidden" name="ln" value="%(ln)s" />
<div style="font-size:small">
<input type="submit" name="cancel" value="%(cancel_changes)s" />
<input type="submit" onclick="user_must_confirm_before_leaving_page=false;return true;" class="adminbutton" name="submit" id="applyChanges" value="%(apply_changes)s" />
</div></form>''' % \
{'apply_changes': _("Apply changes"),
'cancel_changes': _("Cancel all changes"),
'recid': argd['recid'],
'access': access,
'ln': argd['ln'],
'CFG_SITE_URL': CFG_SITE_URL,
'CFG_SITE_RECORD': CFG_SITE_RECORD}
body += websubmit_templates.tmpl_page_do_not_leave_submission_js(argd['ln'], enabled=True)
return page(title = _("Document File Manager") + (argd['recid'] and (': ' + _("Record #%(x_rec)i", x_rec=argd['recid'])) or ''),
navtrail=navtrail,
navtrail_append_title_p=0,
metaheaderadd = get_upload_file_interface_javascript(form_url_params='?access='+access) + \
get_upload_file_interface_css(),
body = body,
uid = uid,
language=argd['ln'],
req=req,
navmenuid='admin')
def managedocfilesasync(self, req, form):
"Upload file and returns upload interface"
argd = wash_urlargd(form, {
'ln': (str, ''),
'recid': (int, 1),
'doctype': (str, ''),
'access': (str, ''),
'indir': (str, ''),
})
user_info = collect_user_info(req)
include_headers = False
# User submitted either through WebSubmit, or admin interface.
if 'doctype' in form and 'indir' in form \
and 'access' in form:
# Submitted through WebSubmit. Check rights
include_headers = True
working_dir = os.path.join(CFG_WEBSUBMIT_STORAGEDIR,
argd['indir'], argd['doctype'],
argd['access'])
try:
assert(working_dir == os.path.abspath(working_dir))
except AssertionError:
raise apache.SERVER_RETURN(apache.HTTP_UNAUTHORIZED)
try:
# Retrieve recid from working_dir, safer.
recid_fd = file(os.path.join(working_dir, 'SN'))
recid = int(recid_fd.read())
recid_fd.close()
except:
recid = ""
try:
act_fd = file(os.path.join(working_dir, 'act'))
action = act_fd.read()
act_fd.close()
except:
action = ""
# Is user authorized to perform this action?
auth_code = acc_authorize_action(user_info,
"submit",
authorized_if_no_roles=not isGuestUser(getUid(req)),
doctype=argd['doctype'],
act=action)[0]
if auth_code and not acc_is_role("submit", doctype=argd['doctype'], act=action):
# There is NO authorization plugged. User should have access
auth_code = 0
else:
# User must be allowed to attach files
auth_code = acc_authorize_action(user_info, 'runbibdocfile')[0]
recid = argd['recid']
if auth_code:
raise apache.SERVER_RETURN(apache.HTTP_UNAUTHORIZED)
return create_file_upload_interface(recid=recid,
ln=argd['ln'],
print_outside_form_tag=False,
print_envelope=False,
form=form,
include_headers=include_headers,
sbm_indir=argd['indir'],
sbm_access=argd['access'],
sbm_doctype=argd['doctype'],
uid=user_info['uid'])[1]
__call__ = managedocfiles
diff --git a/invenio/legacy/bibedit/engine.py b/invenio/legacy/bibedit/engine.py
index 4d8671fd7..c43f8e03c 100644
--- a/invenio/legacy/bibedit/engine.py
+++ b/invenio/legacy/bibedit/engine.py
@@ -1,1834 +1,1840 @@
## This file is part of Invenio.
## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
# pylint: disable=C0103
"""Invenio BibEdit Engine."""
__revision__ = "$Id"
from datetime import datetime
import re
import zlib
import copy
import urllib
import urllib2
import cookielib
import json
from flask import url_for
from invenio.modules import formatter as bibformat
from invenio.errorlib import register_exception
from invenio.utils.json import CFG_JSON_AVAILABLE
from invenio.utils.url import auto_version_url
from invenio.legacy.bibrecord.scripts.xmlmarc2textmarc import create_marc_record
from invenio.legacy.bibedit.config import CFG_BIBEDIT_AJAX_RESULT_CODES, \
CFG_BIBEDIT_JS_CHECK_SCROLL_INTERVAL, CFG_BIBEDIT_JS_HASH_CHECK_INTERVAL, \
CFG_BIBEDIT_JS_CLONED_RECORD_COLOR, \
CFG_BIBEDIT_JS_CLONED_RECORD_COLOR_FADE_DURATION, \
CFG_BIBEDIT_JS_NEW_ADD_FIELD_FORM_COLOR, \
CFG_BIBEDIT_JS_NEW_ADD_FIELD_FORM_COLOR_FADE_DURATION, \
CFG_BIBEDIT_JS_NEW_CONTENT_COLOR, \
CFG_BIBEDIT_JS_NEW_CONTENT_COLOR_FADE_DURATION, \
CFG_BIBEDIT_JS_NEW_CONTENT_HIGHLIGHT_DELAY, \
CFG_BIBEDIT_JS_STATUS_ERROR_TIME, CFG_BIBEDIT_JS_STATUS_INFO_TIME, \
CFG_BIBEDIT_JS_TICKET_REFRESH_DELAY, CFG_BIBEDIT_MAX_SEARCH_RESULTS, \
CFG_BIBEDIT_TAG_FORMAT, CFG_BIBEDIT_AJAX_RESULT_CODES_REV, \
CFG_BIBEDIT_AUTOSUGGEST_TAGS, CFG_BIBEDIT_AUTOCOMPLETE_TAGS_KBS,\
CFG_BIBEDIT_KEYWORD_TAXONOMY, CFG_BIBEDIT_KEYWORD_TAG, \
CFG_BIBEDIT_KEYWORD_RDFLABEL, CFG_BIBEDIT_REQUESTS_UNTIL_SAVE, \
CFG_BIBEDIT_DOI_LOOKUP_FIELD, CFG_DOI_USER_AGENT, \
CFG_BIBEDIT_DISPLAY_REFERENCE_TAGS, CFG_BIBEDIT_DISPLAY_AUTHOR_TAGS, \
CFG_BIBEDIT_EXCLUDE_CURATOR_TAGS, CFG_BIBEDIT_AUTHOR_DISPLAY_THRESHOLD
from invenio.config import (CFG_SITE_LANG, CFG_DEVEL_SITE,
CFG_BIBCATALOG_SYSTEM_RT_URL, CFG_BIBEDIT_SHOW_HOLDING_PEN_REMOVED_FIELDS,
CFG_BIBCATALOG_SYSTEM)
from invenio.legacy.bibedit.db_layer import get_name_tags_all, reserve_record_id, \
get_related_hp_changesets, get_hp_update_xml, delete_hp_change, \
get_record_last_modification_date, get_record_revision_author, \
get_marcxml_of_record_revision, delete_related_holdingpen_changes, \
get_record_revisions, get_info_of_record_revision, \
deactivate_cache
from invenio.legacy.bibedit.utils import cache_exists, cache_expired, \
create_cache, delete_cache, get_bibrecord, \
get_cache_contents, get_cache_mtime, get_record_templates, \
get_record_template, latest_record_revision, record_locked_by_other_user, \
record_locked_by_queue, save_xml_record, touch_cache, \
update_cache_contents, get_field_templates, get_marcxml_of_revision, \
revision_to_timestamp, timestamp_to_revision, \
get_record_revision_timestamps, record_revision_exists, \
can_record_have_physical_copies, extend_record_with_template, \
replace_references, merge_record_with_template, record_xml_output, \
record_is_conference, add_record_cnum, get_xml_from_textmarc, \
record_locked_by_user_details, crossref_process_template, \
modify_record_timestamp, get_affiliation_for_paper
from invenio.legacy.bibrecord import create_record, print_rec, record_add_field, \
record_add_subfield_into, record_delete_field, \
record_delete_subfield_from, \
record_modify_subfield, record_move_subfield, \
create_field, record_replace_field, record_move_fields, \
record_modify_controlfield, record_get_field_values, \
record_get_subfields, record_get_field_instances, record_add_fields, \
record_strip_empty_fields, record_strip_empty_volatile_subfields, \
record_strip_controlfields, record_order_subfields, \
field_add_subfield, field_get_subfield_values
from invenio.config import CFG_BIBEDIT_PROTECTED_FIELDS, CFG_CERN_SITE, \
CFG_SITE_URL, CFG_SITE_RECORD, CFG_BIBEDIT_KB_SUBJECTS, \
CFG_BIBEDIT_KB_INSTITUTIONS, CFG_BIBEDIT_AUTOCOMPLETE_INSTITUTIONS_FIELDS, \
CFG_INSPIRE_SITE
from invenio.legacy.search_engine import record_exists, perform_request_search
from invenio.legacy.webuser import session_param_get, session_param_set
from invenio.legacy.bibcatalog.api import BIBCATALOG_SYSTEM
from invenio.legacy.webpage import page
from invenio.utils.html import get_mathjax_header
from invenio.utils.text import wash_for_xml, show_diff
from invenio.modules.knowledge.api import get_kbd_values_for_bibedit, get_kbr_values, \
get_kbt_items_for_bibedit, kb_exists
from invenio.legacy.batchuploader.engine import perform_upload_check
from invenio.legacy.bibcirculation.db_layer import get_number_copies, has_copies
from invenio.legacy.bibcirculation.utils import create_item_details_url
from invenio.legacy.refextract.api import FullTextNotAvailable \
get_pdf_doc, \
record_has_fulltext
from invenio.legacy.bibrecord.scripts import xmlmarc2textmarc as xmlmarc2textmarc
from invenio.utils.crossref import get_marcxml_for_doi, CrossrefError
import invenio.legacy.template
bibedit_templates = invenio.legacy.template.load('bibedit')
try:
BIBCATALOG_SYSTEM.ticket_search(0)
CFG_CAN_SEARCH_FOR_TICKET = True
except NotImplementedError:
CFG_CAN_SEARCH_FOR_TICKET = False
re_revdate_split = re.compile(r'^(\d\d\d\d)(\d\d)(\d\d)(\d\d)(\d\d)(\d\d)')
def get_empty_fields_templates():
"""
Returning the templates of empty fields::
-an empty data field
-an empty control field
"""
return [{
"name": "Empty field",
"description": "The data field not containing any " +
"information filled in",
"tag" : "",
"ind1" : "",
"ind2" : "",
"subfields" : [("", "")],
"isControlfield" : False
}, {
"name" : "Empty control field",
"description" : "The controlfield not containing any " +
"data or tag description",
"isControlfield" : True,
"tag" : "",
"value" : ""
}]
def get_available_fields_templates():
"""
A method returning all the available field templates
Returns a list of descriptors. Each descriptor has
the same structure as a full field descriptor inside the
record
"""
templates = get_field_templates()
result = get_empty_fields_templates()
for template in templates:
tplTag = template[3].keys()[0]
field = template[3][tplTag][0]
if (field[0] == []):
# if the field is a controlField, add different structure
result.append({
"name" : template[1],
"description" : template[2],
"isControlfield" : True,
"tag" : tplTag,
"value" : field[3]
})
else:
result.append({
"name": template[1],
"description": template[2],
"tag" : tplTag,
"ind1" : field[1],
"ind2" : field[2],
"subfields" : field[0],
"isControlfield" : False
})
return result
def perform_request_init(uid, ln, req, lastupdated):
"""Handle the initial request by adding menu and JavaScript to the page."""
errors = []
warnings = []
body = ''
# Add script data.
record_templates = get_record_templates()
record_templates.sort()
tag_names = get_name_tags_all()
protected_fields = ['001']
protected_fields.extend(CFG_BIBEDIT_PROTECTED_FIELDS.split(','))
cern_site = 'false'
if not CFG_JSON_AVAILABLE:
title = 'Record Editor'
body = '''Sorry, the record editor cannot operate when the
`simplejson' module is not installed. Please see the INSTALL
file.'''
return page(title = title,
body = body,
errors = [],
warnings = [],
uid = uid,
language = ln,
navtrail = "",
lastupdated = lastupdated,
req = req)
body += '<link rel="stylesheet" type="text/css" href="/img/jquery-ui.css" />'
body += '<link rel="stylesheet" type="text/css" href="%s" />' % (
url_for('editor.static', filename='editor/base.css'), )
if CFG_CERN_SITE:
cern_site = 'true'
data = {'gRECORD_TEMPLATES': record_templates,
'gTAG_NAMES': tag_names,
'gPROTECTED_FIELDS': protected_fields,
'gSITE_URL': '"' + CFG_SITE_URL + '"',
'gSITE_RECORD': '"' + CFG_SITE_RECORD + '"',
'gCERN_SITE': cern_site,
'gINSPIRE_SITE': CFG_INSPIRE_SITE,
'gHASH_CHECK_INTERVAL': CFG_BIBEDIT_JS_HASH_CHECK_INTERVAL,
'gCHECK_SCROLL_INTERVAL': CFG_BIBEDIT_JS_CHECK_SCROLL_INTERVAL,
'gSTATUS_ERROR_TIME': CFG_BIBEDIT_JS_STATUS_ERROR_TIME,
'gSTATUS_INFO_TIME': CFG_BIBEDIT_JS_STATUS_INFO_TIME,
'gCLONED_RECORD_COLOR':
'"' + CFG_BIBEDIT_JS_CLONED_RECORD_COLOR + '"',
'gCLONED_RECORD_COLOR_FADE_DURATION':
CFG_BIBEDIT_JS_CLONED_RECORD_COLOR_FADE_DURATION,
'gNEW_ADD_FIELD_FORM_COLOR':
'"' + CFG_BIBEDIT_JS_NEW_ADD_FIELD_FORM_COLOR + '"',
'gNEW_ADD_FIELD_FORM_COLOR_FADE_DURATION':
CFG_BIBEDIT_JS_NEW_ADD_FIELD_FORM_COLOR_FADE_DURATION,
'gNEW_CONTENT_COLOR': '"' + CFG_BIBEDIT_JS_NEW_CONTENT_COLOR + '"',
'gNEW_CONTENT_COLOR_FADE_DURATION':
CFG_BIBEDIT_JS_NEW_CONTENT_COLOR_FADE_DURATION,
'gNEW_CONTENT_HIGHLIGHT_DELAY':
CFG_BIBEDIT_JS_NEW_CONTENT_HIGHLIGHT_DELAY,
'gTICKET_REFRESH_DELAY': CFG_BIBEDIT_JS_TICKET_REFRESH_DELAY,
'gRESULT_CODES': CFG_BIBEDIT_AJAX_RESULT_CODES,
'gAUTOSUGGEST_TAGS' : CFG_BIBEDIT_AUTOSUGGEST_TAGS,
'gAUTOCOMPLETE_TAGS' : CFG_BIBEDIT_AUTOCOMPLETE_TAGS_KBS.keys(),
'gKEYWORD_TAG' : '"' + CFG_BIBEDIT_KEYWORD_TAG + '"',
'gREQUESTS_UNTIL_SAVE' : CFG_BIBEDIT_REQUESTS_UNTIL_SAVE,
'gAVAILABLE_KBS': get_available_kbs(),
'gTagsToAutocomplete': CFG_BIBEDIT_AUTOCOMPLETE_INSTITUTIONS_FIELDS,
'gDOILookupField': '"' + CFG_BIBEDIT_DOI_LOOKUP_FIELD + '"',
'gDisplayReferenceTags': CFG_BIBEDIT_DISPLAY_REFERENCE_TAGS,
'gDisplayAuthorTags': CFG_BIBEDIT_DISPLAY_AUTHOR_TAGS,
'gExcludeCuratorTags': CFG_BIBEDIT_EXCLUDE_CURATOR_TAGS,
'gSHOW_HP_REMOVED_FIELDS': CFG_BIBEDIT_SHOW_HOLDING_PEN_REMOVED_FIELDS,
'gBIBCATALOG_SYSTEM_RT_URL': repr(CFG_BIBCATALOG_SYSTEM_RT_URL)
}
body += '<script type="text/javascript">\n'
for key in data:
body += ' var %s = %s;\n' % (key, data[key])
body += ' </script>\n'
# Adding the information about field templates
fieldTemplates = get_available_fields_templates()
body += "<script>\n" + \
" var fieldTemplates = %s\n" % (json.dumps(fieldTemplates), ) + \
"</script>\n"
# Add scripts (the ordering is NOT irrelevant).
scripts = ['jquery-ui.min.js', 'jquery.jeditable.mini.js',
'jquery.hotkeys.js', 'json2.js']
bibedit_scripts = ['refextract.js', 'display.js', 'engine.js', 'keys.js',
'menu.js', 'holdingpen.js', 'marcxml.js',
'clipboard.js']
for script in scripts:
body += ' <script type="text/javascript" src="%s">' \
'</script>\n' % (url_for('static', filename='js/' + script), )
for script in bibedit_scripts:
body += ' <script type="text/javascript" src="%s">' \
'</script>\n' % (url_for('editor.static',
filename='js/editor/' + script), )
# Init BibEdit
body += '<script>$(init_bibedit);</script>'
# Build page structure and menu.
# rec = create_record(format_record(235, "xm"))[0]
#oaiId = record_extract_oai_id(rec)
body += bibedit_templates.menu()
body += bibedit_templates.focuson()
body += """<div id="bibEditContent">
<div class="revisionLine"></div>
<div id="Toptoolbar"></div>
<div id="bibEditMessage"></div>
<div id="bibEditContentTable"></div>
</div>"""
return body, errors, warnings
def get_available_kbs():
"""
Return list of KBs that are available in the system to be used with
BibEdit
"""
kb_list = [CFG_BIBEDIT_KB_INSTITUTIONS, CFG_BIBEDIT_KB_SUBJECTS]
available_kbs = [kb for kb in kb_list if kb_exists(kb)]
return available_kbs
def get_marcxml_of_revision_id(recid, revid):
"""
Return MARCXML string with corresponding to revision REVID
(=RECID.REVDATE) of a record. Return empty string if revision
does not exist.
"""
job_date = "%s-%s-%s %s:%s:%s" % re_revdate_split.search(revid).groups()
tmp_res = get_marcxml_of_record_revision(recid, job_date)
if tmp_res:
for row in tmp_res:
xml = zlib.decompress(row[0]) + "\n"
# xml contains marcxml of record
# now we create a record object from this xml and sort fields and subfields
# and return marcxml
rec = create_record(xml)[0]
record_order_subfields(rec)
marcxml = record_xml_output(rec, order_fn="_order_by_tags")
return marcxml
def perform_request_compare(ln, recid, rev1, rev2):
"""Handle a request for comparing two records"""
body = ""
errors = []
warnings = []
person1 = ""
person2 = ""
if (not record_revision_exists(recid, rev1)) or \
(not record_revision_exists(recid, rev2)):
body = "The requested record revision does not exist !"
else:
xml1 = get_marcxml_of_revision_id(recid, rev1)
xml2 = get_marcxml_of_revision_id(recid, rev2)
# Create MARC representations of the records
marc1 = create_marc_record(create_record(xml1)[0], '', {"text-marc": 1, "aleph-marc": 0})
marc2 = create_marc_record(create_record(xml2)[0], '', {"text-marc": 1, "aleph-marc": 0})
- comparison = show_diff(marc1, marc2)
+ comparison = show_diff(marc1,
+ marc2,
+ prefix="<pre>", suffix="</pre>",
+ prefix_removed='<strong class="diff_field_deleted">',
+ suffix_removed='</strong>',
+ prefix_added='<strong class="diff_field_added">',
+ suffix_added='</strong>')
job_date1 = "%s-%s-%s %s:%s:%s" % re_revdate_split.search(rev1).groups()
job_date2 = "%s-%s-%s %s:%s:%s" % re_revdate_split.search(rev2).groups()
# Geting the author of each revision
info1 = get_info_of_record_revision(recid, job_date1)
info2 = get_info_of_record_revision(recid, job_date2)
if info1:
person1 = info1[0][1]
if info2:
person2 = info2[0][1]
body += bibedit_templates.history_comparebox(ln, job_date1, job_date2,
person1, person2, comparison)
return body, errors, warnings
def perform_request_newticket(recid, uid):
"""create a new ticket with this record's number
@param recid: record id
@param uid: user id
@return: (error_msg, url)
"""
t_url = ""
errmsg = ""
if CFG_BIBCATALOG_SYSTEM is not None:
t_id = BIBCATALOG_SYSTEM.ticket_submit(uid, "", recid, "")
if t_id:
#get the ticket's URL
t_url = BIBCATALOG_SYSTEM.ticket_get_attribute(uid, t_id, 'url_modify')
else:
errmsg = "ticket_submit failed"
else:
errmsg = "No ticket system configured"
return (errmsg, t_url)
def perform_request_ajax(req, recid, uid, data, isBulk = False):
"""Handle Ajax requests by redirecting to appropriate function."""
response = {}
request_type = data['requestType']
undo_redo = None
if "undoRedo" in data:
undo_redo = data["undoRedo"]
# Call function based on request type.
if request_type == 'searchForRecord':
# Search request.
response.update(perform_request_bibedit_search(data, req))
elif request_type in ['changeTagFormat']:
# User related requests.
response.update(perform_request_user(req, request_type, recid, data))
elif request_type in ('getRecord', 'submit', 'cancel', 'newRecord',
'deleteRecord', 'deleteRecordCache', 'prepareRecordMerge', 'revert',
'updateCacheRef', 'submittextmarc'):
# 'Major' record related requests.
response.update(perform_request_record(req, request_type, recid, uid,
data))
elif request_type in ('addField', 'addSubfields',
'addFieldsSubfieldsOnPositions', 'modifyContent',
'modifySubfieldTag', 'modifyFieldTag',
'moveSubfield', 'deleteFields', 'moveField',
'modifyField', 'otherUpdateRequest',
'disableHpChange', 'deactivateHoldingPenChangeset'):
# Record updates.
cacheMTime = data['cacheMTime']
if 'hpChanges' in data:
hpChanges = data['hpChanges']
else:
hpChanges = {}
response.update(perform_request_update_record(request_type, recid,
uid, cacheMTime, data,
hpChanges, undo_redo,
isBulk))
elif request_type in ('autosuggest', 'autocomplete', 'autokeyword'):
response.update(perform_request_autocomplete(request_type, recid, uid,
data))
elif request_type in ('getTickets', 'closeTicket', 'openTicket'):
# BibCatalog requests.
response.update(perform_request_bibcatalog(request_type, uid, data))
elif request_type in ('getHoldingPenUpdates', ):
response.update(perform_request_holdingpen(request_type, recid))
elif request_type in ('getHoldingPenUpdateDetails',
'deleteHoldingPenChangeset'):
updateId = data['changesetNumber']
response.update(perform_request_holdingpen(request_type, recid,
updateId))
elif request_type in ('applyBulkUpdates', ):
# a general version of a bulk request
changes = data['requestsData']
cacheMTime = data['cacheMTime']
response.update(perform_bulk_request_ajax(req, recid, uid, changes,
undo_redo, cacheMTime))
elif request_type in ('preview', ):
response.update(perform_request_preview_record(request_type, recid, uid, data))
elif request_type in ('get_pdf_url', ):
response.update(perform_request_get_pdf_url(recid))
elif request_type in ('refextract', ):
txt = None
if 'txt' in data:
txt = data["txt"]
response.update(perform_request_ref_extract(recid, uid, txt))
elif request_type in ('refextracturl', ):
response.update(perform_request_ref_extract_url(recid, uid, data['url']))
elif request_type == 'getTextMarc':
response.update(perform_request_get_textmarc(recid, uid))
elif request_type == "getTableView":
response.update(perform_request_get_tableview(recid, uid, data))
elif request_type == "DOISearch":
response.update(perform_doi_search(data['doi']))
elif request_type == "deactivateRecordCache":
deactivate_cache(recid, uid)
response.update({"cacheMTime": data['cacheMTime']})
elif request_type == "guessAffiliations":
response.update(perform_guess_affiliations(uid, data))
return response
def perform_bulk_request_ajax(req, recid, uid, reqsData, undoRedo, cacheMTime):
""" An AJAX handler used when treating bulk updates """
lastResult = {}
lastTime = cacheMTime
if get_cache_mtime(recid, uid) != cacheMTime:
return {"resultCode": 107}
isFirst = True
for data in reqsData:
assert data is not None
data['cacheMTime'] = lastTime
if isFirst and undoRedo is not None:
# we add the undo/redo handler to the first operation in order to
# save the handler on the server side !
data['undoRedo'] = undoRedo
isFirst = False
lastResult = perform_request_ajax(req, recid, uid, data, isBulk=True)
try:
lastTime = lastResult['cacheMTime']
except KeyError:
raise Exception(str(lastResult))
return lastResult
def perform_request_bibedit_search(data, req):
"""Handle search requests."""
response = {}
searchType = data['searchType']
if searchType is None:
searchType = "anywhere"
searchPattern = data['searchPattern']
if searchType == 'anywhere':
pattern = searchPattern
else:
pattern = searchType + ':' + searchPattern
pattern = urllib.unquote(pattern)
result_set = list(perform_request_search(req=req, p=pattern))
response['resultCode'] = 1
response['resultSet'] = result_set[0:CFG_BIBEDIT_MAX_SEARCH_RESULTS]
return response
def perform_request_user(req, request_type, recid, data):
"""Handle user related requests."""
response = {}
if request_type == 'changeTagFormat':
tagformat_settings = session_param_get(req, 'bibedit_tagformat', {})
tagformat_settings[recid] = data['tagFormat']
session_param_set(req, 'bibedit_tagformat', tagformat_settings)
response['resultCode'] = 2
return response
def perform_request_holdingpen(request_type, recId, changeId=None):
"""
A method performing the holdingPen ajax request. The following types of
requests can be made::
-getHoldingPenUpdates: retrieving the holding pen updates pending
for a given record
"""
response = {}
if request_type == 'getHoldingPenUpdates':
changeSet = get_related_hp_changesets(recId)
changes = []
for change in changeSet:
changes.append((str(change[0]), str(change[1])))
changes.reverse() # newest to older order
response["changes"] = changes
elif request_type == 'getHoldingPenUpdateDetails':
# returning the list of changes related to the holding pen update
# the format based on what the record difference xtool returns
assert(changeId is not None)
hpContent = get_hp_update_xml(changeId)
holdingPenRecord = create_record(hpContent[0], "xm")[0]
if not holdingPenRecord:
response['resultCode'] = 107
else:
template_to_merge = extend_record_with_template(recId)
if template_to_merge:
merged_record = merge_record_with_template(holdingPenRecord,
template_to_merge)
if merged_record:
holdingPenRecord = merged_record
# order subfields alphabetically
record_order_subfields(holdingPenRecord)
# databaseRecord = get_record(hpContent[1])
response['record'] = holdingPenRecord
response['changeset_number'] = changeId
elif request_type == 'deleteHoldingPenChangeset':
assert(changeId is not None)
delete_hp_change(changeId)
return response
def perform_request_record(req, request_type, recid, uid, data, ln=CFG_SITE_LANG):
"""Handle 'major' record related requests like fetching, submitting or
deleting a record, cancel editing or preparing a record for merging.
"""
response = {}
if request_type == 'newRecord':
# Create a new record.
new_recid = reserve_record_id()
new_type = data['newType']
if new_type == 'empty':
# Create a new empty record.
create_cache(recid, uid)
response['resultCode'], response['newRecID'] = 6, new_recid
elif new_type == 'template':
# Create a new record from XML record template.
template_filename = data['templateFilename']
template = get_record_template(template_filename)
if not template:
response['resultCode'] = 108
else:
record = create_record(template)[0]
if not record:
response['resultCode'] = 109
else:
record_add_field(record, '001',
controlfield_value=str(new_recid))
create_cache(new_recid, uid, record, True)
response['cacheMTime'] = get_cache_mtime(new_recid, uid)
response['resultCode'], response['newRecID'] = 7, new_recid
elif new_type == 'import':
# Import data from external source, using DOI
doi = data['doi']
if not doi:
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_no_doi_specified']
else:
try:
marcxml_template = get_marcxml_for_doi(doi)
except CrossrefError as inst:
response['resultCode'] = \
CFG_BIBEDIT_AJAX_RESULT_CODES_REV[inst.code]
except:
response['resultCode'] = 0
else:
record = crossref_process_template(marcxml_template, CFG_INSPIRE_SITE)
if not record:
response['resultCode'] = 109
else:
record_add_field(record, '001',
controlfield_value=str(new_recid))
template_to_merge = extend_record_with_template(recstruct=record)
if template_to_merge:
merged_record = merge_record_with_template(record, template_to_merge)
if merged_record:
record = merged_record
create_cache(new_recid, uid, record, True)
response['cacheMTime'] = get_cache_mtime(new_recid, uid)
response['resultCode'], response['newRecID'] = 7, new_recid
elif new_type == 'clone':
# Clone an existing record (from the users cache).
existing_cache = cache_exists(recid, uid)
if existing_cache:
cache = get_cache_contents(recid, uid)
if cache:
record = cache[2]
else:
# if, for example, the cache format was wrong (outdated)
record = get_bibrecord(recid)
else:
# Cache missing. Fall back to using original version.
record = get_bibrecord(recid)
record_delete_field(record, '001')
record_delete_field(record, '005')
record_add_field(record, '001', controlfield_value=str(new_recid))
create_cache(new_recid, uid, record, True)
response['resultCode'], response['newRecID'] = 8, new_recid
elif request_type == 'getRecord':
# Fetch the record. Possible error situations:
# - Non-existing record
# - Deleted record
# - Record locked by other user
# - Record locked by queue
# A cache file will be created if it does not exist.
# If the cache is outdated (i.e., not based on the latest DB revision),
# cacheOutdated will be set to True in the response.
record_status = record_exists(recid)
existing_cache = cache_exists(recid, uid)
read_only_mode = False
if "inReadOnlyMode" in data:
read_only_mode = data['inReadOnlyMode']
if data.get('deleteRecordCache'):
delete_cache(recid, uid)
existing_cache = False
pending_changes = []
disabled_hp_changes = {}
if record_status == 0:
response['resultCode'] = 102
elif not read_only_mode and not existing_cache and \
record_locked_by_other_user(recid, uid):
name, email, locked_since = record_locked_by_user_details(recid, uid)
response['locked_details'] = {'name': name,
'email': email,
'locked_since': locked_since}
response['resultCode'] = 104
elif not read_only_mode and existing_cache and \
cache_expired(recid, uid) and \
record_locked_by_other_user(recid, uid):
response['resultCode'] = 104
elif not read_only_mode and record_locked_by_queue(recid):
response['resultCode'] = 105
else:
if read_only_mode:
if 'recordRevision' in data and data['recordRevision'] != 'sampleValue':
record_revision_ts = data['recordRevision']
record_xml = get_marcxml_of_revision(recid,
record_revision_ts)
record = create_record(record_xml)[0]
record_revision = timestamp_to_revision(record_revision_ts)
pending_changes = []
disabled_hp_changes = {}
else:
# a normal cacheless retrieval of a record
record = get_bibrecord(recid)
record_revision = get_record_last_modification_date(recid)
if record_revision is None:
record_revision = datetime.now().timetuple()
pending_changes = []
disabled_hp_changes = {}
cache_dirty = False
mtime = 0
undo_list = []
redo_list = []
else:
try:
cache_dirty, record_revision, record, pending_changes, \
disabled_hp_changes, undo_list, redo_list = \
get_cache_contents(recid, uid)
except TypeError:
# No cache found in the DB
record_revision, record = create_cache(recid, uid)
if not record:
response['resultCode'] = 103
return response
pending_changes = []
disabled_hp_changes = {}
cache_dirty = False
undo_list = []
redo_list = []
else:
touch_cache(recid, uid)
if not latest_record_revision(recid, record_revision) and \
get_record_revisions(recid) != ():
# This sould prevent from using old cache in case of
# viewing old version. If there are no revisions,
# it means we should skip this step because this
# is a new record
response['cacheOutdated'] = True
mtime = get_cache_mtime(recid, uid)
if data.get('clonedRecord', ''):
response['resultCode'] = 9
else:
response['resultCode'] = 3
revision_author = get_record_revision_author(recid, record_revision)
latest_revision = get_record_last_modification_date(recid)
if latest_revision is None:
latest_revision = datetime.now().timetuple()
last_revision_ts = revision_to_timestamp(latest_revision)
revisions_history = get_record_revision_timestamps(recid)
number_of_physical_copies = get_number_copies(recid)
bibcirc_details_URL = create_item_details_url(recid, ln)
can_have_copies = can_record_have_physical_copies(recid)
# For some collections, merge template with record
template_to_merge = extend_record_with_template(recid)
if template_to_merge and not read_only_mode:
merged_record = merge_record_with_template(record, template_to_merge)
if merged_record:
record = merged_record
mtime = update_cache_contents(recid, uid, record_revision,
record, pending_changes,
disabled_hp_changes,
undo_list, redo_list)
if record_status == -1:
# The record was deleted
response['resultCode'] = 103
response['record_has_pdf'] = record_has_fulltext(recid)
response['record_hide_authors'] = check_hide_authors(record)
response['cacheDirty'], response['record'], \
response['cacheMTime'], response['recordRevision'], \
response['revisionAuthor'], response['lastRevision'], \
response['revisionsHistory'], response['inReadOnlyMode'], \
response['pendingHpChanges'], response['disabledHpChanges'], \
response['undoList'], response['redoList'] = cache_dirty, \
record, mtime, revision_to_timestamp(record_revision), \
revision_author, last_revision_ts, revisions_history, \
read_only_mode, pending_changes, disabled_hp_changes, \
undo_list, redo_list
response['numberOfCopies'] = number_of_physical_copies
response['bibCirculationUrl'] = bibcirc_details_URL
response['canRecordHavePhysicalCopies'] = can_have_copies
# Set tag format from user's session settings.
tagformat_settings = session_param_get(req, 'bibedit_tagformat')
tagformat = (tagformat_settings is not None) and tagformat_settings.get(recid, CFG_BIBEDIT_TAG_FORMAT) or CFG_BIBEDIT_TAG_FORMAT
response['tagFormat'] = tagformat
# KB information
response['KBSubject'] = CFG_BIBEDIT_KB_SUBJECTS
response['KBInstitution'] = CFG_BIBEDIT_KB_INSTITUTIONS
elif request_type == 'submit':
# Submit the record. Possible error situations:
# - Missing cache file
# - Cache file modified in other editor
# - Record locked by other user
# - Record locked by queue
# If the cache is outdated cacheOutdated will be set to True in the
# response.
if not cache_exists(recid, uid):
response['resultCode'] = 106
elif not get_cache_mtime(recid, uid) == data['cacheMTime']:
response['resultCode'] = 107
elif cache_expired(recid, uid) and \
record_locked_by_other_user(recid, uid):
response['resultCode'] = 104
elif record_locked_by_queue(recid):
response['resultCode'] = 105
else:
try:
tmp_result = get_cache_contents(recid, uid)
record_revision = tmp_result[1]
record = tmp_result[2]
pending_changes = tmp_result[3]
# disabled_changes = tmp_result[4]
xml_record = wash_for_xml(print_rec(record))
record, status_code, list_of_errors = create_record(xml_record)
# Simulate upload to catch errors
errors_upload = perform_upload_check(xml_record, '--replace')
if errors_upload:
response['resultCode'], response['errors'] = 113, \
errors_upload
return response
elif status_code == 0:
response['resultCode'], response['errors'] = 110, \
list_of_errors
if not data['force'] and not latest_record_revision(recid, record_revision):
response['cacheOutdated'] = True
else:
if record_is_conference(record):
new_cnum = add_record_cnum(recid, uid)
if new_cnum:
response["new_cnum"] = new_cnum
save_xml_record(recid, uid)
response['resultCode'] = 4
except Exception as e:
register_exception()
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV[
'error_wrong_cache_file_format']
if CFG_DEVEL_SITE: # return debug information in the request
response['exception_message'] = e.__str__()
elif request_type == 'revert':
revId = data['revId']
job_date = "%s-%s-%s %s:%s:%s" % re_revdate_split.search(revId).groups()
revision_xml = get_marcxml_of_revision(recid, job_date)
# Modify the 005 tag in order to merge with the latest version of record
last_revision_ts = data['lastRevId'] + ".0"
revision_xml = modify_record_timestamp(revision_xml, last_revision_ts)
save_xml_record(recid, uid, revision_xml)
if (cache_exists(recid, uid)):
delete_cache(recid, uid)
response['resultCode'] = 4
elif request_type == 'cancel':
# Cancel editing by deleting the cache file. Possible error situations:
# - Cache file modified in other editor
if cache_exists(recid, uid):
if get_cache_mtime(recid, uid) == data['cacheMTime']:
delete_cache(recid, uid)
response['resultCode'] = 5
else:
response['resultCode'] = 107
else:
response['resultCode'] = 5
elif request_type == 'deleteRecord':
# Submit the record. Possible error situations:
# - Record locked by other user
# - Record locked by queue
# As the user is requesting deletion we proceed even if the cache file
# is missing and we don't check if the cache is outdated or has
# been modified in another editor.
existing_cache = cache_exists(recid, uid)
pending_changes = []
if has_copies(recid):
response['resultCode'] = \
CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_physical_copies_exist']
elif existing_cache and cache_expired(recid, uid) and \
record_locked_by_other_user(recid, uid):
response['resultCode'] = \
CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_rec_locked_by_user']
elif record_locked_by_queue(recid):
response['resultCode'] = \
CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_rec_locked_by_queue']
else:
if not existing_cache:
create_cache(recid, uid)
record_revision, record, pending_changes, \
deactivated_hp_changes, undo_list, redo_list = \
get_cache_contents(recid, uid)[1:]
else:
try:
record_revision, record, pending_changes, \
deactivated_hp_changes, undo_list, redo_list = \
get_cache_contents(recid, uid)[1:]
except:
record_revision, record, pending_changes, \
deactivated_hp_changes = create_cache(recid, uid)
record_add_field(record, '980', ' ', ' ', '', [('c', 'DELETED')])
undo_list = []
redo_list = []
update_cache_contents(recid, uid, record_revision, record,
pending_changes, deactivated_hp_changes,
undo_list, redo_list)
save_xml_record(recid, uid)
delete_related_holdingpen_changes(recid) # we don't need any changes
# related to a deleted record
response['resultCode'] = 10
elif request_type == 'deleteRecordCache':
# Delete the cache file. Ignore the request if the cache has been
# modified in another editor.
if 'cacheMTime' in data:
if cache_exists(recid, uid) and get_cache_mtime(recid, uid) == \
data['cacheMTime']:
delete_cache(recid, uid)
response['resultCode'] = 11
elif request_type == 'updateCacheRef':
# Update cache with the contents coming from BibEdit JS interface
# Used when updating references using ref extractor
record_revision, record, pending_changes, \
deactivated_hp_changes, undo_list, redo_list = \
get_cache_contents(recid, uid)[1:]
record = create_record(data['recXML'])[0]
response['cacheMTime'] = update_cache_contents(recid,
uid,
record_revision,
record,
pending_changes,
deactivated_hp_changes,
undo_list,
redo_list)
response['cacheDirty'] = True
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['cache_updated_with_references']
elif request_type == 'prepareRecordMerge':
# We want to merge the cache with the current DB version of the record,
# so prepare an XML file from the file cache, to be used by BibMerge.
# Possible error situations:
# - Missing cache file
# - Record locked by other user
# - Record locked by queue
# We don't check if cache is outdated (a likely scenario for this
# request) or if it has been modified in another editor.
if not cache_exists(recid, uid):
response['resultCode'] = 106
elif cache_expired(recid, uid) and \
record_locked_by_other_user(recid, uid):
response['resultCode'] = 104
elif record_locked_by_queue(recid):
response['resultCode'] = 105
else:
save_xml_record(recid, uid, to_upload=False, to_merge=True)
response['resultCode'] = 12
elif request_type == 'submittextmarc':
# Textmarc content coming from the user
textmarc_record = data['textmarc']
xml_conversion_status = get_xml_from_textmarc(recid, textmarc_record, uid)
if xml_conversion_status['resultMsg'] == "textmarc_parsing_error":
response.update(xml_conversion_status)
return response
# Simulate upload to catch errors
errors_upload = perform_upload_check(xml_conversion_status['resultXML'], '--replace')
if errors_upload:
response['resultCode'], response['errors'] = 113, \
errors_upload
return response
response.update(xml_conversion_status)
if xml_conversion_status['resultMsg'] == 'textmarc_parsing_success':
create_cache(recid, uid,
create_record(response['resultXML'])[0])
save_xml_record(recid, uid)
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV["record_submitted"]
return response
def perform_request_update_record(request_type, recid, uid, cacheMTime, data,
hpChanges, undoRedoOp, isBulk=False):
"""
Handle record update requests like adding, modifying, moving or deleting
of fields or subfields. Possible common error situations::
- Missing cache file
- Cache file modified in other editor
@param undoRedoOp: Indicates in "undo"/"redo"/undo_descriptor operation is
performed by a current request.
"""
response = {}
if not cache_exists(recid, uid):
response['resultCode'] = 106
elif get_cache_mtime(recid, uid) != cacheMTime and isBulk is False:
# In case of a bulk request, the changes are deliberately performed
# immediately one after another
response['resultCode'] = 107
else:
try:
record_revision, record, pending_changes, deactivated_hp_changes, \
undo_list, redo_list = get_cache_contents(recid, uid)[1:]
except:
register_exception()
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV[
'error_wrong_cache_file_format']
return response
# process all the Holding Pen changes operations ... regardles the
# request type
if "toDisable" in hpChanges:
for changeId in hpChanges["toDisable"]:
pending_changes[changeId]["applied_change"] = True
if "toEnable" in hpChanges:
for changeId in hpChanges["toEnable"]:
pending_changes[changeId]["applied_change"] = False
if "toOverride" in hpChanges:
pending_changes = hpChanges["toOverride"]
if "changesetsToDeactivate" in hpChanges:
for changesetId in hpChanges["changesetsToDeactivate"]:
deactivated_hp_changes[changesetId] = True
if "changesetsToActivate" in hpChanges:
for changesetId in hpChanges["changesetsToActivate"]:
deactivated_hp_changes[changesetId] = False
# processing the undo/redo entries
if undoRedoOp == "undo":
try:
redo_list = [undo_list[-1]] + redo_list
undo_list = undo_list[:-1]
except:
raise Exception("An exception occured when undoing previous" +
" operation. Undo list: " + str(undo_list) +
" Redo list " + str(redo_list))
elif undoRedoOp == "redo":
try:
undo_list = undo_list + [redo_list[0]]
redo_list = redo_list[1:]
except:
raise Exception("An exception occured when redoing previous" +
" operation. Undo list: " + str(undo_list) +
" Redo list " + str(redo_list))
else:
# This is a genuine operation - we have to add a new descriptor
# to the undo list and cancel the redo unless the operation is
# a bulk operation
if undoRedoOp is not None:
undo_list = undo_list + [undoRedoOp]
redo_list = []
else:
assert isBulk is True
field_position_local = data.get('fieldPosition')
if field_position_local is not None:
field_position_local = int(field_position_local)
if request_type == 'otherUpdateRequest':
# An empty request. Might be useful if we want to perform
# operations that require only the actions performed globally,
# like modifying the holdingPen changes list
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV[
'editor_modifications_changed']
elif request_type == 'deactivateHoldingPenChangeset':
# the changeset has been marked as processed ( user applied it in
# the editor). Marking as used in the cache file.
# CAUTION: This function has been implemented here because logically
# it fits with the modifications made to the cache file.
# No changes are made to the Holding Pen physically. The
# changesets are related to the cache because we want to
# cancel the removal every time the cache disappears for
# any reason
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV[
'disabled_hp_changeset']
elif request_type == 'addField':
if data['controlfield']:
record_add_field(record, data['tag'],
controlfield_value=data['value'])
response['resultCode'] = 20
else:
record_add_field(record, data['tag'], data['ind1'],
data['ind2'], subfields=data['subfields'],
field_position_local=field_position_local)
response['resultCode'] = 21
elif request_type == 'addSubfields':
subfields = data['subfields']
for subfield in subfields:
record_add_subfield_into(record, data['tag'], subfield[0],
subfield[1], subfield_position=None,
field_position_local=field_position_local)
if len(subfields) == 1:
response['resultCode'] = 22
else:
response['resultCode'] = 23
elif request_type == 'addFieldsSubfieldsOnPositions':
#1) Sorting the fields by their identifiers
fieldsToAdd = data['fieldsToAdd']
subfieldsToAdd = data['subfieldsToAdd']
for tag in fieldsToAdd.keys():
positions = fieldsToAdd[tag].keys()
positions.sort()
for position in positions:
# now adding fields at a position
isControlfield = (len(fieldsToAdd[tag][position][0]) == 0)
# if there are n subfields, this is a control field
if isControlfield:
controlfieldValue = fieldsToAdd[tag][position][3]
record_add_field(record, tag,
field_position_local=int(position),
controlfield_value=controlfieldValue)
else:
subfields = fieldsToAdd[tag][position][0]
ind1 = fieldsToAdd[tag][position][1]
ind2 = fieldsToAdd[tag][position][2]
record_add_field(record, tag, ind1, ind2,
subfields=subfields,
field_position_local=int(position))
# now adding the subfields
for tag in subfieldsToAdd.keys():
for fieldPosition in subfieldsToAdd[tag].keys(): # now the fields
# order not important !
subfieldsPositions = subfieldsToAdd[tag][fieldPosition]. \
keys()
subfieldsPositions.sort()
for subfieldPosition in subfieldsPositions:
subfield = subfieldsToAdd[tag][fieldPosition][subfieldPosition]
record_add_subfield_into(record, tag, subfield[0], subfield[1],
subfield_position=int(subfieldPosition),
field_position_local=int(fieldPosition))
response['resultCode'] = \
CFG_BIBEDIT_AJAX_RESULT_CODES_REV['added_positioned_subfields']
elif request_type == 'modifyField': # changing the field structure
# first remove subfields and then add new... change the indices
subfields = data['subFields'] # parse the JSON representation of
# the subfields here
new_field = create_field(subfields, data['ind1'], data['ind2'])
record_replace_field(record, data['tag'], new_field,
field_position_local=data['fieldPosition'])
response['resultCode'] = 26
elif request_type == 'modifyContent':
if data['subfieldIndex'] is not None:
record_modify_subfield(record, data['tag'],
data['subfieldCode'], data['value'],
int(data['subfieldIndex']),
field_position_local=field_position_local)
else:
record_modify_controlfield(record, data['tag'], data["value"],
field_position_local=field_position_local)
response['resultCode'] = 24
elif request_type == 'modifySubfieldTag':
record_add_subfield_into(record, data['tag'], data['subfieldCode'],
data["value"], subfield_position= int(data['subfieldIndex']),
field_position_local=field_position_local)
record_delete_subfield_from(record, data['tag'], int(data['subfieldIndex']) + 1,
field_position_local=field_position_local)
response['resultCode'] = 24
elif request_type == 'modifyFieldTag':
subfields = record_get_subfields(record, data['oldTag'],
field_position_local=field_position_local)
record_add_field(record, data['newTag'], data['ind1'],
data['ind2'] , subfields=subfields)
record_delete_field(record, data['oldTag'], ind1=data['oldInd1'],
ind2=data['oldInd2'], field_position_local=field_position_local)
response['resultCode'] = 32
elif request_type == 'moveSubfield':
record_move_subfield(record, data['tag'],
int(data['subfieldIndex']), int(data['newSubfieldIndex']),
field_position_local=field_position_local)
response['resultCode'] = 25
elif request_type == 'moveField':
if data['direction'] == 'up':
final_position_local = field_position_local-1
else: # direction is 'down'
final_position_local = field_position_local+1
record_move_fields(record, data['tag'], [field_position_local],
final_position_local)
response['resultCode'] = 32
elif request_type == 'deleteFields':
to_delete = data['toDelete']
deleted_fields = 0
deleted_subfields = 0
for tag in to_delete:
#Sorting the fields in a edcreasing order by the local position!
fieldsOrder = to_delete[tag].keys()
fieldsOrder.sort(lambda a, b: int(b) - int(a))
for field_position_local in fieldsOrder:
if not to_delete[tag][field_position_local]:
# No subfields specified - delete entire field.
record_delete_field(record, tag,
field_position_local=int(field_position_local))
deleted_fields += 1
else:
for subfield_position in \
to_delete[tag][field_position_local][::-1]:
# Delete subfields in reverse order (to keep the
# indexing correct).
record_delete_subfield_from(record, tag,
int(subfield_position),
field_position_local=int(field_position_local))
deleted_subfields += 1
if deleted_fields == 1 and deleted_subfields == 0:
response['resultCode'] = 26
elif deleted_fields and deleted_subfields == 0:
response['resultCode'] = 27
elif deleted_subfields == 1 and deleted_fields == 0:
response['resultCode'] = 28
elif deleted_subfields and deleted_fields == 0:
response['resultCode'] = 29
else:
response['resultCode'] = 30
response['cacheMTime'] = update_cache_contents(recid,
uid,
record_revision,
record,
pending_changes,
deactivated_hp_changes,
undo_list, redo_list)
response['cacheDirty'] = True
return response
def perform_request_autocomplete(request_type, recid, uid, data):
"""
Perfrom an AJAX request associated with the retrieval of autocomplete
data.
@param request_type: Type of the currently served request
@param recid: the identifer of the record
@param uid: The identifier of the user being currently logged in
@param data: The request data containing possibly important additional
arguments
"""
response = {}
# get the values based on which one needs to search
searchby = data['value']
# we check if the data is properly defined
fulltag = ''
if 'maintag' in data and 'subtag1' in data and \
'subtag2' in data and 'subfieldcode' in data:
maintag = data['maintag']
subtag1 = data['subtag1']
subtag2 = data['subtag2']
u_subtag1 = subtag1
u_subtag2 = subtag2
if (not subtag1) or (subtag1 == ' '):
u_subtag1 = '_'
if (not subtag2) or (subtag2 == ' '):
u_subtag2 = '_'
subfieldcode = data['subfieldcode']
fulltag = maintag+u_subtag1+u_subtag2+subfieldcode
if (request_type == 'autokeyword'):
# call the keyword-form-ontology function
if fulltag and searchby:
items = get_kbt_items_for_bibedit(CFG_BIBEDIT_KEYWORD_TAXONOMY,
CFG_BIBEDIT_KEYWORD_RDFLABEL,
searchby)
response['autokeyword'] = items
if (request_type == 'autosuggest'):
# call knowledge base function to put the suggestions in an array..
if fulltag and searchby and len(searchby) > 3:
# add trailing '*' wildcard for 'search_unit_in_bibxxx()' if not already present
suggest_values = get_kbd_values_for_bibedit(fulltag, "", searchby+"*")
# remove ..
new_suggest_vals = []
for sugg in suggest_values:
if sugg.startswith(searchby):
new_suggest_vals.append(sugg)
response['autosuggest'] = new_suggest_vals
if (request_type == 'autocomplete'):
# call the values function with the correct kb_name
if fulltag in CFG_BIBEDIT_AUTOCOMPLETE_TAGS_KBS:
kbname = CFG_BIBEDIT_AUTOCOMPLETE_TAGS_KBS[fulltag]
# check if the seachby field has semicolons. Take all
# the semicolon-separated items..
items = []
vals = []
if searchby:
if searchby.rfind(';'):
items = searchby.split(';')
else:
items = [searchby.strip()]
for item in items:
item = item.strip()
kbrvals = get_kbr_values(kbname, item, '', 'e') # we want an exact match
if kbrvals and kbrvals[0]: # add the found val into vals
vals.append(kbrvals[0])
#check that the values are not already contained in other
#instances of this field
record = get_cache_contents(recid, uid)[2]
xml_rec = wash_for_xml(print_rec(record))
record, status_code, dummy_errors = create_record(xml_rec)
existing_values = []
if (status_code != 0):
existing_values = record_get_field_values(record,
maintag,
subtag1,
subtag2,
subfieldcode)
#get the new values.. i.e. vals not in existing
new_vals = vals
for val in new_vals:
if val in existing_values:
new_vals.remove(val)
response['autocomplete'] = new_vals
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['autosuggestion_scanned']
return response
def perform_request_bibcatalog(request_type, uid, data):
"""Handle request to BibCatalog (RT).
"""
response = {}
if request_type == 'getTickets':
# Insert the tickets data in the response, if possible
if not CFG_BIBCATALOG_SYSTEM or not CFG_CAN_SEARCH_FOR_TICKET:
response['tickets'] = "<!--No ticket system configured-->"
elif uid:
bibcat_resp = BIBCATALOG_SYSTEM.check_system(uid)
if bibcat_resp == "":
tickets_found = BIBCATALOG_SYSTEM.ticket_search(uid,
status=['new', 'open'], recordid=data['recID'])
tickets = []
for t_id in tickets_found:
ticket_info = BIBCATALOG_SYSTEM.ticket_get_info(
uid, t_id, ['url_display', 'url_close', 'subject', 'text', 'queue', 'created'])
t_url = ticket_info['url_display']
t_close_url = ticket_info['url_close']
t_subject = ticket_info['subject']
t_text = ticket_info['text']
t_queue = ticket_info['queue']
date_string = ticket_info['created']
date_splitted = date_string.split(" ")
# convert date to readable format
try:
t_date = date_splitted[2] + ' ' + date_splitted[1] + " " + date_splitted[4] +\
" " + date_splitted[3].split(":")[0] + ":" + date_splitted[3].split(":")[1]
except IndexError:
t_date = date_string
ticket = {"id": str(t_id), "queue": t_queue, "date": t_date, "url": t_url,
"close_url": t_close_url, "subject": t_subject, "text": t_text}
tickets.append(ticket)
response['tickets'] = tickets
# add available queues
response['queues'] = BIBCATALOG_SYSTEM.get_queues(uid)
response['resultCode'] = 31
else:
# put something in the tickets container, for debug
response['tickets'] = "Error connecting to RT<!--" + bibcat_resp + "-->"
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_rt_connection']
# closeTicket usecase
elif request_type == 'closeTicket':
if not CFG_BIBCATALOG_SYSTEM or not CFG_CAN_SEARCH_FOR_TICKET:
response['ticket_closed_description'] = "<!--No ticket system configured-->"
response['ticket_closed_code'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_ticket_closed']
elif uid:
bibcat_resp = BIBCATALOG_SYSTEM.check_system(uid)
if bibcat_resp == "":
un, pw = get_bibcat_from_prefs(uid)
if un and pw:
ticket_assigned = BIBCATALOG_SYSTEM.ticket_steal(uid, data['ticketid'])
ticket_closed = BIBCATALOG_SYSTEM.ticket_set_attribute(uid, data['ticketid'], 'status', 'resolved')
if ticket_closed == 1:
response['ticket_closed_description'] = 'Ticket resolved'
response['ticket_closed_code'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['ticket_closed']
else:
response['ticket_closed_description'] = 'Ticket could not be resolved.Try again'
response['ticket_closed_code'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_ticket_closed']
else:
response['ticket_closed_description'] = 'RT user does not exist'
response['ticket_closed_code'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_ticket_closed']
else:
#put something in the tickets container, for debug
response['ticket_closed_description'] = "Error connecting to RT<!--" + bibcat_resp + "-->"
response['ticket_closed_code'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_rt_connection']
response['ticketid'] = data['ticketid']
elif request_type == 'openTicket':
if not CFG_BIBCATALOG_SYSTEM or not CFG_CAN_SEARCH_FOR_TICKET:
response['ticket_opened_description'] = "<!--No ticket system configured-->"
response['ticket_opened_code'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_ticket_opened']
elif uid:
bibcat_resp = BIBCATALOG_SYSTEM.check_system(uid)
if bibcat_resp == "":
un, pw = get_bibcat_from_prefs(uid)
if un and pw:
ticket_opened = BIBCATALOG_SYSTEM.ticket_set_attribute(uid, data['ticketid'], 'status', 'open')
if ticket_opened == 1:
response['ticket_opened_description'] = 'Ticket opened'
response['ticket_opened_code'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['ticket_opened']
else:
response['ticket_opened_description'] = 'Ticket could not be opened.Try again'
response['ticket_opened_code'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_ticket_opened']
else:
response['ticket_opened_description'] = 'RT user does not exist'
response['ticket_opened_code'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_ticket_opened']
else:
#put something in the tickets container, for debug
response['ticket_opened_description'] = "Error connecting to RT<!--" + bibcat_resp + "-->"
response['ticket_opened_code'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['error_rt_connection']
response['ticketid'] = data['ticketid']
return response
def _add_curated_references_to_record(recid, uid, bibrec):
"""
Adds references from the cache that have been curated (contain $$9CURATOR)
to the bibrecord object
@param recid: record id, used to retrieve cache
@param uid: id of the current user, used to retrieve cache
@param bibrec: bibrecord object to add references to
"""
dummy1, dummy2, record, dummy3, dummy4, dummy5, dummy6 = get_cache_contents(recid, uid)
for field_instance in record_get_field_instances(record, "999", "C", "5"):
for subfield_instance in field_instance[0]:
if subfield_instance[0] == '9' and subfield_instance[1] == 'CURATOR':
# Add reference field on top of references, removing first $$o
field_instance = ([subfield for subfield in field_instance[0]
if subfield[0] != 'o'], field_instance[1],
field_instance[2], field_instance[3],
field_instance[4])
record_add_fields(bibrec, '999', [field_instance],
field_position_local=0)
def _xml_to_textmarc_references(bibrec):
"""
Convert XML record to textmarc and return the lines related to references
@param bibrec: bibrecord object to be converted
@return: textmarc lines with references
@rtype: string
"""
sysno = ""
options = {"aleph-marc": 0, "correct-mode": 1, "append-mode": 0,
"delete-mode": 0, "insert-mode": 0, "replace-mode": 0,
"text-marc": 1}
# Using deepcopy as function create_marc_record() modifies the record passed
textmarc_references = [line.strip() for line
in xmlmarc2textmarc.create_marc_record(copy.deepcopy(bibrec),
sysno, options).split('\n')
if '999C5' in line]
return textmarc_references
def perform_request_ref_extract_url(recid, uid, url):
"""
Making use of the refextractor API, extract references from the url
received from the client
@param recid: opened record id
@param uid: active user id
@param url: URL to extract references from
@return response to be returned to the client code
"""
response = {}
try:
recordExtended = replace_references(recid, uid, url=url)
except FullTextNotAvailable:
response['ref_xmlrecord'] = False
response['ref_msg'] = "File not found. Server returned code 404"
return response
except:
response['ref_xmlrecord'] = False
response['ref_msg'] = """Error while fetching PDF. Bad URL or file could
not be retrieved """
return response
if not recordExtended:
response['ref_msg'] = """No references were found in the given PDF """
return response
ref_bibrecord = create_record(recordExtended)[0]
_add_curated_references_to_record(recid, uid, ref_bibrecord)
response['ref_bibrecord'] = ref_bibrecord
response['ref_xmlrecord'] = record_xml_output(ref_bibrecord)
textmarc_references = _xml_to_textmarc_references(ref_bibrecord)
response['ref_textmarc'] = '<div class="refextracted">' + '<br />'.join(textmarc_references) + "</div>"
return response
def perform_request_ref_extract(recid, uid, txt=None):
""" Handle request to extract references in the given record
@param recid: record id from which the references should be extracted
@type recid: str
@param txt: string containing references
@type txt: str
@param uid: user id
@type uid: int
@return: xml record with references extracted
@rtype: dictionary
"""
text_no_references_found_msg = """ No references extracted. The automatic
extraction did not recognize any reference in the
pasted text.<br /><br />If you want to add the references
manually, an easily recognizable format is:<br/><br/>
&nbsp;&nbsp;&nbsp;&nbsp;[1] Phys. Rev A71 (2005) 42<br />
&nbsp;&nbsp;&nbsp;&nbsp;[2] ATLAS-CMS-2007-333
"""
pdf_no_references_found_msg = """ No references were found in the attached
PDF.
"""
response = {}
response['ref_xmlrecord'] = False
recordExtended = None
try:
if txt:
recordExtended = replace_references(recid, uid,
txt=txt.decode('utf-8'))
if not recordExtended:
response['ref_msg'] = text_no_references_found_msg
else:
recordExtended = replace_references(recid, uid)
if not recordExtended:
response['ref_msg'] = pdf_no_references_found_msg
except FullTextNotAvailable:
response['ref_msg'] = """ The fulltext is not available.
"""
except:
response['ref_msg'] = """ An error ocurred while extracting references.
"""
if not recordExtended:
return response
ref_bibrecord = create_record(recordExtended)[0]
_add_curated_references_to_record(recid, uid, ref_bibrecord)
response['ref_bibrecord'] = ref_bibrecord
response['ref_xmlrecord'] = record_xml_output(ref_bibrecord)
textmarc_references = _xml_to_textmarc_references(ref_bibrecord)
response['ref_textmarc'] = '<div class="refextracted">' + '<br />'.join(textmarc_references) + "</div>"
return response
def perform_request_preview_record(request_type, recid, uid, data):
""" Handle request to preview record with formatting
"""
response = {}
if request_type == "preview":
if data["submitMode"] == "textmarc":
textmarc_record = data['textmarc']
xml_conversion_status = get_xml_from_textmarc(recid, textmarc_record, uid)
if xml_conversion_status['resultMsg'] == 'textmarc_parsing_error':
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['textmarc_parsing_error']
response.update(xml_conversion_status)
return response
record = create_record(xml_conversion_status["resultXML"])[0]
elif cache_exists(recid, uid):
dummy1, dummy2, record, dummy3, dummy4, dummy5, dummy6 = get_cache_contents(recid, uid)
else:
record = get_bibrecord(recid)
# clean the record from unfilled volatile fields
record_strip_empty_volatile_subfields(record)
record_strip_empty_fields(record)
response['html_preview'] = _get_formated_record(record, data['new_window'])
# clean the record from unfilled volatile fields
record_strip_empty_volatile_subfields(record)
record_strip_empty_fields(record)
response['html_preview'] = _get_formated_record(record, data['new_window'])
return response
def perform_request_get_pdf_url(recid):
""" Handle request to get the URL of the attached PDF
"""
response = {}
doc = get_pdf_doc(recid)
if doc:
response['pdf_url'] = doc.get_url()
else:
response['pdf_url'] = ""
return response
def perform_request_get_textmarc(recid, uid):
""" Get record content from cache, convert it to textmarc and return it
"""
textmarc_options = {"aleph-marc": 0, "correct-mode": 1, "append-mode": 0,
"delete-mode": 0, "insert-mode": 0, "replace-mode": 0,
"text-marc": 1}
bibrecord = get_cache_contents(recid, uid)[2]
record_strip_empty_fields(bibrecord)
record_strip_controlfields(bibrecord)
textmarc = xmlmarc2textmarc.create_marc_record(
copy.deepcopy(bibrecord), sysno="", options=textmarc_options)
return {'textmarc': textmarc}
def perform_request_get_tableview(recid, uid, data):
""" Convert textmarc inputed by user to marcxml and if there are no
parsing errors, create cache file
"""
response = {}
textmarc_record = data['textmarc']
if not textmarc_record:
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['tableview_change_success']
xml_conversion_status = get_xml_from_textmarc(recid, textmarc_record, uid)
response.update(xml_conversion_status)
if xml_conversion_status['resultMsg'] == 'textmarc_parsing_error':
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['textmarc_parsing_error']
else:
create_cache(recid, uid,
create_record(xml_conversion_status['resultXML'])[0], data['recordDirty'])
response['resultCode'] = CFG_BIBEDIT_AJAX_RESULT_CODES_REV['tableview_change_success']
response['cacheMTime'] = get_cache_mtime(recid, uid)
return response
def _get_formated_record(record, new_window):
"""Returns a record in a given format
@param record: BibRecord object
@param new_window: Boolean, indicates if it is needed to add all the headers
to the page (used when clicking Preview button)
"""
from invenio.config import CFG_WEBSTYLE_TEMPLATE_SKIN
xml_record = wash_for_xml(record_xml_output(record))
result = ''
if new_window:
result = """ <html><head><title>Record preview</title>
<script type="text/javascript" src="%(site_url)s/js/jquery.min.js"></script>
<link rel="stylesheet" href="%(site_url)s/img/invenio%(cssskin)s.css" type="text/css"></head>
""" % {'site_url': CFG_SITE_URL,
'cssskin': CFG_WEBSTYLE_TEMPLATE_SKIN != 'default' and '_' + CFG_WEBSTYLE_TEMPLATE_SKIN or ''
}
result += get_mathjax_header(True) + '<body>'
result += "<h2> Brief format preview </h2><br />"
result += bibformat.format_record(0,
of="hb",
xml_record=xml_record) + "<br />"
result += "<br /><h2> Detailed format preview </h2><br />"
result += bibformat.format_record(0,
of="hd",
xml_record=xml_record)
#Preview references
result += "<br /><h2> References </h2><br />"
result += bibformat.format_record(0,
'hdref',
xml_record=xml_record)
result += """<script>
$('#referenceinp_link').hide();
$('#referenceinp_link_span').hide();
</script>
"""
if new_window:
result += "</body></html>"
return result
########### Functions related to templates web interface #############
def perform_request_init_template_interface():
"""Handle a request to manage templates"""
errors = []
warnings = []
body = ''
# Add script data.
record_templates = get_record_templates()
record_templates.sort()
data = {'gRECORD_TEMPLATES': record_templates,
'gSITE_RECORD': '"' + CFG_SITE_RECORD + '"',
'gSITE_URL': '"' + CFG_SITE_URL + '"'}
body += '<script type="text/javascript">\n'
for key in data:
body += ' var %s = %s;\n' % (key, data[key])
body += ' </script>\n'
# Add scripts (the ordering is NOT irrelevant).
scripts = ['jquery-ui.min.js', 'json2.js']
bibedit_scripts = ['display.js', 'template_interface.js']
for script in scripts:
body += ' <script type="text/javascript" src="%s">' \
'</script>\n' % (url_for('static', filename='js/' + script), )
for script in bibedit_scripts:
body += ' <script type="text/javascript" src="%s">' \
'</script>\n' % (url_for('editor.static',
filename='js/editor/' + script), )
body += ' <div id="bibEditTemplateList"></div>\n'
body += ' <div id="bibEditTemplateEdit"></div>\n'
return body, errors, warnings
def perform_request_ajax_template_interface(data):
"""Handle Ajax requests by redirecting to appropriate function."""
response = {}
request_type = data['requestType']
if request_type == 'editTemplate':
# Edit a template request.
response.update(perform_request_edit_template(data))
return response
def perform_request_edit_template(data):
""" Handle request to edit a template """
response = {}
template_filename = data['templateFilename']
template = get_record_template(template_filename)
if not template:
response['resultCode'] = 1
else:
response['templateMARCXML'] = template
return response
def perform_doi_search(doi):
"""Search for DOI on the dx.doi.org page
@return: the url returned by this page"""
response = {}
url = "http://dx.doi.org/"
val = {'hdl': doi}
url_data = urllib.urlencode(val)
cj = cookielib.CookieJar()
header = [('User-Agent', CFG_DOI_USER_AGENT)]
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
opener.addheaders = header
try:
resp = opener.open(url, url_data)
except:
return response
else:
response['doi_url'] = resp.geturl()
return response
def check_hide_authors(record):
""" Check if authors should be hidden by default in the user interface """
return sum([len(record.get(tag, [])) for tag in CFG_BIBEDIT_DISPLAY_AUTHOR_TAGS]) > CFG_BIBEDIT_AUTHOR_DISPLAY_THRESHOLD
def perform_guess_affiliations(uid, data):
response = {}
recid = data["recID"]
record_revision, record, pending_changes, deactivated_hp_changes, \
undo_list, redo_list = get_cache_contents(recid, uid)[1:]
# Let's guess affiliations
result = {}
for tag in CFG_BIBEDIT_DISPLAY_AUTHOR_TAGS:
result[tag] = {}
author_field_instances = record_get_field_instances(record, tag)
for field_pos, instance in enumerate(author_field_instances):
subfields_to_add = []
current_affilations = field_get_subfield_values(instance, code="u")
if not current_affilations or current_affilations[0].startswith("VOLATILE:"):
# This author does not have affiliation
try:
author_name = field_get_subfield_values(instance, code="a")[0]
except IndexError:
author_name = author_name[0]
aff_guess = get_affiliation_for_paper(recid, author_name)
if aff_guess:
for aff in aff_guess:
field_add_subfield(instance, code="u", value=aff)
subfields_to_add.append(["u", aff])
if subfields_to_add:
result[tag][field_pos] = subfields_to_add
response['cacheMTime'] = update_cache_contents(recid, uid, record_revision,
record, pending_changes,
deactivated_hp_changes,
undo_list, redo_list)
response['subfieldsToAdd'] = result
return response
diff --git a/invenio/legacy/bibexport/googlescholar.py b/invenio/legacy/bibexport/googlescholar.py
index 606e44ce5..70ebbe920 100644
--- a/invenio/legacy/bibexport/googlescholar.py
+++ b/invenio/legacy/bibexport/googlescholar.py
@@ -1,291 +1,208 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2008, 2010, 2011 CERN.
+## Copyright (C) 2008, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
BibExport plugin implementing 'googlescholar' exporting method.
The main function is run_export_method(jobname) defined at the end.
This is what BibExport daemon calls for all the export jobs that use
this exporting method.
-The Google Scholar exporting method answers this use case: every first
-of the month, please export all records modified during the last month
-and matching these search criteria in an NLM format in such a way that
-the output is split into files containing not more than 1000 records
-and compressed via gzip and placed in this place from where Google
-Scholar would fetch them. The output files would be organized like
-this:
+The Google Scholar exporting method produces a sitemap of all the
+records matching the collections defined in "googlescholar.cfg", as
+well as a second sitemap that list all records modified in the last
+months in these collections. The produced files contain at most
+MAX_RECORDS records and weight at most MAX_SIZE bytes. The output
+files would be organized like this:
* all exportable records:
- /export/googlescholar/all-index.html - links to parts below
+ /export/googlescholar/all-index.xml.gz - links to parts below
/export/googlescholar/all-part1.xml.gz - first batch of 1000 records
/export/googlescholar/all-part2.xml.gz - second batch of 1000 records
...
/export/googlescholar/all-partM.xml.gz - last batch of 1000 records
* records modified in the last month:
- /export/googlescholar/lastmonth-index.html - links to parts below
+ /export/googlescholar/lastmonth-index.xml.gz - links to parts below
/export/googlescholar/lastmonth-part1.xml.gz - first batch of 1000 records
/export/googlescholar/lastmonth-part2.xml.gz - second batch of 1000 records
...
/export/googlescholar/lastmonth-partN.xml.gz - last batch of 1000 records
"""
-
-from invenio.config import CFG_WEBDIR, CFG_CERN_SITE
-from invenio.legacy.bibsched.bibtask import write_message
-from invenio.legacy.search_engine import perform_request_search, print_record
import os
-import gzip
import datetime
-def run_export_method(jobname):
- """Main function, reading params and running the task."""
- # FIXME: read jobname's cfg file to detect collection and fulltext status arguments
- write_message("bibexport_sitemap: job %s started." % jobname)
-
- try:
- output_directory = CFG_WEBDIR + os.sep + "export" + os.sep + "googlescholar"
- exporter = GoogleScholarExporter(output_directory)
- exporter.export()
- except GoogleScholarExportException as ex:
- write_message("%s Exception: %s" %(ex.get_error_message(), ex.get_inner_exception()))
-
- write_message("bibexport_sitemap: job %s finished." % jobname)
-
-class GoogleScholarExporter:
- """Export data for google scholar"""
-
- _output_directory = ""
- _records_with_fulltext_only = True
- #FIXME: Read collections from configuration file
- _collections = ["Theses"]
- if CFG_CERN_SITE:
- _collections = ["CERN Theses"]
-
- def __init__(self, output_directory):
- """Constructor of GoogleScholarExporter
-
- output_directory - directory where files will be placed
- """
-
- self.set_output_directory(output_directory)
-
- def export(self):
- """Export all records and records modified last month"""
- LAST_MONTH_FILE_NAME_PATTERN = "lastmonth"
- ALL_MONTH_FILE_NAME_PATTERN = "all"
- SPLIT_BY_RECORDS = 1000
-
- # Export records modified last month
- records = self._get_records_modified_last_month()
- self._delete_files(self._output_directory, LAST_MONTH_FILE_NAME_PATTERN)
- self._split_records_into_files(records, SPLIT_BY_RECORDS, LAST_MONTH_FILE_NAME_PATTERN, self._output_directory)
-
- # Export all records
- all_records = self._get_all_records()
- self._delete_files(self._output_directory, ALL_MONTH_FILE_NAME_PATTERN)
- self._split_records_into_files(all_records, SPLIT_BY_RECORDS, ALL_MONTH_FILE_NAME_PATTERN, self._output_directory)
-
- def set_output_directory(self, path_to_directory):
- """Check if directory exists. If it does not exists it creates it."""
-
- directory = path_to_directory
- # remove the slash from the end of the path if exists
- if directory[-1] == os.sep:
- directory = directory[:-1]
-
- # if directory does not exists then create it
- if not os.path.exists(directory):
- try:
- os.makedirs(directory)
- except(IOError, OSError), exception:
- self._report_error("Directory %s does not exist and cannot be ctreated." % (directory, ), exception)
-
- # if it is not path to a directory report an error
- if not os.path.isdir(directory):
- self._report_error("%s is not a directory." % (directory, ))
- return
-
- self._output_directory = directory
-
- def _get_records_modified_last_month(self):
- """Returns all records modified last month and matching the criteria."""
- current_date = datetime.date.today()
- one_month_ago = current_date - datetime.timedelta(days = 31)
-
- #FIXME: Return only records with full texts available for Google Scholar
- #FIXME: There is a problem with searching in modification date. It searches only in creation date
- return perform_request_search(dt="m", c = self._collections, d1y = one_month_ago.year, d1m = one_month_ago.month, d1d = one_month_ago.day)
-
- def _get_all_records(self):
- """Return all records matching the criteria no matter of their modification date."""
- #FIXME: Return only records with full texts available for Google Scholar
- return perform_request_search(c = self._collections)
-
- def _split_records_into_files(self, records, max_records_per_file, file_name_pattern, output_directory):
- """Split and save records into files containing not more than max_records_per_file records.
-
- records - list of record numbers
-
- max_records_per_file - the maximum number of records per file
+from invenio.config import \
+ CFG_WEBDIR, \
+ CFG_SITE_URL, \
+ CFG_SITE_RECORD
+from invenio.legacy.bibsched.bibtask import write_message, task_update_progress, task_sleep_now_if_required
+from invenio.legacy.search_engine import get_collection_reclist, get_all_restricted_recids
+from intbitset import intbitset
+from invenio.legacy.dbquery import run_sql
+from invenio.legacy.bibexport.sitemap import \
+ get_config_parameter, \
+ SitemapWriter, \
+ SitemapIndexWriter, \
+ get_all_public_records
- file_name_pattern - the pattern used to name the files. Filenames will start with this
- pattern.
- output_directory - directory where all the files will be placed
- """
- file_number = 1
- file_name = self._get_part_file_name(file_name_pattern, file_number)
- begin = 0
- number_of_records = len(records)
+DEFAULT_PRIORITY_RECORDS = 0.8
+DEFAULT_CHANGEFREQ_RECORDS = 'weekly'
- if 0 == number_of_records:
- return
+LAST_MONTH_FILE_NAME_PATTERN = "lastmonth"
+ALL_MONTH_FILE_NAME_PATTERN = "all"
- for end in xrange(max_records_per_file, number_of_records, max_records_per_file):
- self._save_records_into_file(records[begin:end], file_name, output_directory)
- begin = end
- file_number = file_number + 1
- file_name = self._get_part_file_name(file_name_pattern, file_number)
+MAX_RECORDS = 50000
+MAX_SIZE = 10000000
- if(begin != number_of_records):
- self._save_records_into_file(records[begin:number_of_records], file_name, output_directory)
-
- self._create_index_file(file_number, file_name_pattern, output_directory)
-
- def _get_part_file_name(self, file_name_pattern, file_number):
- """Returns name of the file containing part of the records
-
- file_name_pattern - the pattetn used to create the filename
-
- file_number - the number of the file in the sequence of files
-
- The result is filename like lastmonth-part2.xml.gz
- where lastmonth is the file_name_pattern and 2 is the file_number
- """
- file_name = "%s-part%d.xml.gz" % (file_name_pattern, file_number)
-
- return file_name
-
- def _create_index_file(self, number_of_files, file_name_pattern, output_directory):
- """Creates HTML file containing links to all files containing records"""
-
- try:
- index_file = open(output_directory + os.sep +file_name_pattern+"-index.html", "w")
- index_file.write("<html><body>\n")
-
- for file_number in xrange(1, number_of_files + 1):
- file_name = self._get_part_file_name(file_name_pattern, file_number)
- index_file.write('<a href="%s">%s</a><br>\n' % (file_name, file_name))
-
- index_file.write("</body></html>\n")
- except (IOError, OSError) as exception:
- self._report_error("Failed to create index file.", exception)
-
- if index_file is not None:
- index_file.close()
-
- def _save_records_into_file(self, records, file_name, output_directory):
- """Save all the records into file in proper format (currently
- National Library of Medicine XML).
-
- file_name - the name of the file where records will be saved
-
- output_directory - directory where the file will be placed"""
-
- output_file = self._open_output_file(file_name, output_directory)
- self._write_to_output_file(output_file, "<articles>\n")
-
- for record in records:
- nlm_xml = self._get_record_NLM_XML(record)
- output_file.write(nlm_xml)
+def run_export_method(jobname):
+ """Main function, reading params and running the task."""
+ write_message("bibexport_sitemap: job %s started." % jobname)
- self._write_to_output_file(output_file, "\n</articles>")
- self._close_output_file(output_file)
+ collections = get_config_parameter(jobname=jobname, parameter_name="collection", is_parameter_collection = True)
+ output_directory = CFG_WEBDIR + os.sep + "export" + os.sep + "googlescholar"
- def _open_output_file(self, file_name, output_directory):
- """Opens new file for writing.
+ try:
+ init_output_directory(output_directory)
+ except GoogleScholarExportException, ex:
+ write_message("%s Exception: %s" %(ex.get_error_message(), ex.get_inner_exception()))
+ return
- file_name - the name of the file without the extention.
+ # Export records modified last month
+ records = get_all_public_records_modified_last_month(collections)
+ _delete_files(output_directory, LAST_MONTH_FILE_NAME_PATTERN)
+ generate_sitemaps_index(records, output_directory, LAST_MONTH_FILE_NAME_PATTERN)
- output_directory - the directory where file will be created"""
+ # Export all records
+ all_records = get_all_public_records(collections)
+ _delete_files(output_directory, ALL_MONTH_FILE_NAME_PATTERN)
+ generate_sitemaps_index(all_records, output_directory, ALL_MONTH_FILE_NAME_PATTERN)
- path = output_directory + os.sep + file_name
+ write_message("bibexport_sitemap: job %s finished." % jobname)
+def generate_sitemaps_index(records, output_directory, sitemap_name):
+ """main function. Generates the sitemap index and the sitemaps
+
+ @param records: the list of (recid, modification_date) tuples to process
+ @param output_directory: directory where to store the sitemaps
+ @param sitemap_name: the name (prefix) of the sitemap files(s)
+ """
+ write_message("Generating all sitemaps...")
+ sitemap_index_writer = SitemapIndexWriter(os.path.join(output_directory, sitemap_name + '-index.xml.gz'))
+ generate_sitemaps(sitemap_index_writer, records, output_directory, sitemap_name)
+
+def generate_sitemaps(sitemap_index_writer, records, output_directory, sitemap_name):
+ """
+ Generate sitemaps themselves.
+
+ @param sitemap_index_writer: the instance of SitemapIndexWriter that will refer to these sitemaps
+ @param records: the list of (recid, modification_date) tuples to process
+ @param output_directory: directory where to store the sitemaps
+ @param sitemap_name: the name (prefix) of the sitemap files(s)
+ """
+ sitemap_id = 1
+ writer = SitemapWriter(sitemap_id, output_directory, sitemap_name)
+ sitemap_index_writer.add_url(writer.get_sitemap_url())
+ nb_urls = 0
+ write_message("... Getting sitemap '%s'..." % sitemap_name)
+ write_message("... Generating urls for %s records..." % len(records))
+ task_sleep_now_if_required(can_stop_too=True)
+ for i, (recid, lastmod) in enumerate(records):
+ if nb_urls % 100 == 0 and (writer.get_size() >= MAX_SIZE or nb_urls >= MAX_RECORDS):
+ sitemap_id += 1
+ writer = SitemapWriter(sitemap_id, output_directory, sitemap_name)
+ sitemap_index_writer.add_url(writer.get_sitemap_url())
+ nb_urls = writer.add_url(CFG_SITE_URL + '/%s/%s' % (CFG_SITE_RECORD, recid),
+ lastmod = lastmod,
+ changefreq = DEFAULT_CHANGEFREQ_RECORDS,
+ priority = DEFAULT_PRIORITY_RECORDS)
+ if i % 100 == 0:
+ task_update_progress("Google Scholar sitemap '%s' for recid %s/%s" % (sitemap_name, i + 1, len(records)))
+ task_sleep_now_if_required(can_stop_too=True)
+
+def init_output_directory(path_to_directory):
+ """Check if directory exists. If it does not exists it creates it."""
+ directory = path_to_directory
+ # remove the slash from the end of the path if exists
+ if directory[-1] == os.sep:
+ directory = directory[:-1]
+
+ # if directory does not exists then create it
+ if not os.path.exists(directory):
try:
- output_file = gzip.GzipFile(filename = path, mode = "w")
- return output_file
- except (IOError, OSError) as exception:
- self._report_error("Failed to open file file %s." % (path, ), exception)
- return None
-
- def _close_output_file(self, output_file):
- """Closes the file"""
- if output_file is None:
- return
- output_file.close()
-
- def _write_to_output_file(self, output_file, text_to_write):
- """"Wirtes a the text passed as a parameter to file"""
- try:
- output_file.write(text_to_write)
- except (IOError, OSError) as exception:
- self._report_error("Failed to write to file " + output_file.name, exception)
-
- def _get_record_NLM_XML(self, record):
- """Returns the record in National Library of Medicine XML format."""
- return print_record(record, format='xn')
+ os.makedirs(directory)
+ except(IOError, OSError) as exception:
+ raise GoogleScholarExportException("Directory %s does not exist and cannot be ctreated." % (directory, ), exception)
- def _delete_files(self, path_to_directory, name_pattern):
- """Deletes files with file name starting with name_pattern
- from directory specified by path_to_directory"""
+ # if it is not path to a directory report an error
+ if not os.path.isdir(directory):
+ raise GoogleScholarExportException("%s is not a directory." % (directory, ))
+def _delete_files(path_to_directory, name_pattern):
+ """Deletes files with file name starting with name_pattern
+ from directory specified by path_to_directory"""
+ try:
files = os.listdir(path_to_directory)
-
- for current_file in files:
- if current_file.startswith(name_pattern):
- path_to_file = path_to_directory + os.sep + current_file
- os.remove(path_to_file)
-
- def _report_error(self, error_message, exception = None):
- """Reprts an error during exprotring"""
- raise GoogleScholarExportException(error_message, exception)
+ except OSError:
+ return
+
+ for current_file in files:
+ if current_file.startswith(name_pattern):
+ path_to_file = path_to_directory + os.sep + current_file
+ os.remove(path_to_file)
+
+def get_all_public_records_modified_last_month(collections):
+ """ Get all records which exist (i.e. not suppressed ones) and are in
+ accessible collection.
+ returns list of (recid, last_modification) tuples
+ """
+ all_restricted_recids = get_all_restricted_recids()
+ current_date = datetime.date.today()
+ one_month_ago = current_date - datetime.timedelta(days = 31)
+ recids = intbitset()
+ for collection in collections:
+ recids += get_collection_reclist(collection)
+ recids = recids.difference(all_restricted_recids)
+ query = 'SELECT id, modification_date FROM bibrec WHERE modification_date > %s'
+ res = run_sql(query, (one_month_ago,))
+ return [(recid, lastmod) for (recid, lastmod) in res if recid in recids]
class GoogleScholarExportException(Exception):
"""Exception indicating an error during exportting for Google scholar."""
_error_message = ""
_inner_exception = None
def __init__(self, error_message, inner_exception = None):
"""Constructor of the exception"""
Exception.__init__(self, error_message, inner_exception)
self._error_message = error_message
self._inner_exception = inner_exception
def get_error_message(self):
"""Returns the error message that explains the reason for the exception"""
return self._error_message
def get_inner_exception(self):
"""Returns the inner exception that is the cause for the current exception"""
return self._inner_exception
diff --git a/invenio/legacy/bibexport/sitemap.py b/invenio/legacy/bibexport/sitemap.py
index 366c8fcde..013053047 100644
--- a/invenio/legacy/bibexport/sitemap.py
+++ b/invenio/legacy/bibexport/sitemap.py
@@ -1,439 +1,449 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2008, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
BibExport plugin implementing 'sitemap' exporting method.
The main function is run_export_method(jobname) defined at the end.
This is what BibExport daemon calls for all the export jobs that use
this exporting method.
"""
from datetime import datetime
from urllib import quote
from ConfigParser import ConfigParser
from six import iteritems
import os
import gzip
import time
from invenio.legacy.bibdocfile.api import BibRecDocs
-from invenio.legacy.search_engine import get_collection_reclist
+from invenio.legacy.search_engine import get_collection_reclist, get_all_restricted_recids
from invenio.legacy.dbquery import run_sql
from invenio.config import CFG_SITE_URL, CFG_WEBDIR, CFG_ETCDIR, \
CFG_SITE_RECORD, CFG_SITE_LANGS, CFG_TMPSHAREDDIR
from intbitset import intbitset
from invenio.legacy.websearch.webcoll import Collection
from invenio.legacy.bibsched.bibtask import write_message, task_update_progress, task_sleep_now_if_required
from invenio.utils.text import encode_for_xml
from invenio.utils.url import get_canonical_and_alternates_urls
DEFAULT_TIMEZONE = '+01:00'
DEFAULT_PRIORITY_HOME = 1
DEFAULT_CHANGEFREQ_HOME = 'hourly'
DEFAULT_PRIORITY_RECORDS = 0.8
DEFAULT_CHANGEFREQ_RECORDS = 'weekly'
DEFAULT_PRIORITY_COMMENTS = 0.4
DEFAULT_CHANGEFREQ_COMMENTS = 'weekly'
DEFAULT_PRIORITY_REVIEWS = 0.6
DEFAULT_CHANGEFREQ_REVIEWS = 'weekly'
DEFAULT_PRIORITY_FULLTEXTS = 0.9
DEFAULT_CHANGEFREQ_FULLTEXTS = 'weekly'
DEFAULT_PRIORITY_COLLECTIONS = 0.3
DEFAULT_CHANGEFREQ_COLLECTIONS = 'hourly'
MAX_RECORDS = 50000
MAX_SIZE = 10000000
_CFG_FORCE_RECRAWLING_TIMESTAMP_PATH = os.path.join(CFG_TMPSHAREDDIR, "bibexport_sitemap_force_recrawling_timestamp.txt")
def get_minimum_timestamp():
"""
Return the minimum timestamp to be used when exporting.
"""
if os.path.exists(_CFG_FORCE_RECRAWLING_TIMESTAMP_PATH):
return datetime.fromtimestamp(os.path.getmtime(_CFG_FORCE_RECRAWLING_TIMESTAMP_PATH))
else:
return datetime(1970, 1, 1)
def get_all_public_records(collections):
""" Get all records which exist (i.e. not suppressed ones) and are in
accessible collection.
returns list of (recid, last_modification) tuples
"""
+ all_restricted_recids = get_all_restricted_recids()
recids = intbitset()
minimum_timestamp = get_minimum_timestamp()
for collection in collections:
recids += get_collection_reclist(collection)
+ recids = recids.difference(all_restricted_recids)
query = 'SELECT id, modification_date FROM bibrec'
res = run_sql(query)
return [(recid, max(lastmod, minimum_timestamp)) for (recid, lastmod) in res if recid in recids]
def get_all_public_collections(base_collections):
""" Return a list of (collection.name, last_modification) tuples for all
collections and subcollections of base_collections
"""
minimum_timestamp = get_minimum_timestamp()
def get_collection_last_modification(collection):
""" last modification = modification date fo latest added record """
last_mod = None
query_last_mod = "SELECT modification_date FROM bibrec WHERE id=%s"
try:
latest_recid = collection.reclist.tolist()[-1]
except IndexError:
# this collection is empty
return last_mod
res = run_sql(query_last_mod, (latest_recid,))
if res and res[0][0]:
last_mod = res[0][0]
return max(minimum_timestamp, last_mod)
output = []
for coll_name in base_collections:
mother_collection = Collection(coll_name)
if not mother_collection.restricted_p():
last_mod = get_collection_last_modification(mother_collection)
output.append((coll_name, last_mod))
for descendant in mother_collection.get_descendants(type='r'):
if not descendant.restricted_p():
last_mod = get_collection_last_modification(descendant)
output.append((descendant.name, last_mod))
for descendant in mother_collection.get_descendants(type='v'):
if not descendant.restricted_p():
last_mod = get_collection_last_modification(descendant)
output.append((descendant.name, last_mod))
return output
def filter_fulltexts(recids, fulltext_type=None):
""" returns list of records having a fulltext of type fulltext_type.
If fulltext_type is empty, return all records having a fulltext"""
recids = dict(recids)
minimum_timestamp = get_minimum_timestamp()
if fulltext_type:
query = """SELECT id_bibrec, max(modification_date)
FROM bibrec_bibdoc
LEFT JOIN bibdoc ON bibrec_bibdoc.id_bibdoc=bibdoc.id
WHERE type=%s
GROUP BY id_bibrec"""
res = run_sql(query, (fulltext_type,))
else:
query = """SELECT id_bibrec, max(modification_date)
FROM bibrec_bibdoc
LEFT JOIN bibdoc ON bibrec_bibdoc.id_bibdoc=bibdoc.id
GROUP BY id_bibrec"""
res = run_sql(query)
return [(recid, max(lastmod, minimum_timestamp)) for (recid, lastmod) in res if recid in recids and BibRecDocs(recid).list_latest_files(list_hidden=False)]
def filter_comments(recids):
""" Retrieve recids having a comment. return (recid, last_review_date)"""
minimum_timestamp = get_minimum_timestamp()
recids = dict(recids)
query = """SELECT id_bibrec, max(date_creation)
FROM cmtRECORDCOMMENT
WHERE star_score=0
GROUP BY id_bibrec"""
res = run_sql(query)
return [(recid, max(minimum_timestamp, lastmod)) for (recid, lastmod) in res if recid in recids]
def filter_reviews(recids):
""" Retrieve recids having a review. return (recid, last_review_date)"""
minimum_timestamp = get_minimum_timestamp()
recids = dict(recids)
query = """SELECT id_bibrec, max(date_creation)
FROM cmtRECORDCOMMENT
WHERE star_score>0
GROUP BY id_bibrec"""
res = run_sql(query)
return [(recid, max(lastmod, minimum_timestamp)) for (recid, lastmod) in res if recid in recids]
SITEMAP_HEADER = """\
<?xml version="1.0" encoding="UTF-8"?>
<urlset
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xhtml="http://www.w3.org/1999/xhtml"
xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">"""
SITEMAP_FOOTER = '\n</urlset>\n'
class SitemapWriter(object):
""" Writer for sitemaps"""
- def __init__(self, sitemap_id):
+ def __init__(self, sitemap_id, base_dir=None, name=None):
""" Constructor.
name: path to the sitemap file to be created
"""
self.header = SITEMAP_HEADER
self.footer = SITEMAP_FOOTER
self.sitemap_id = sitemap_id
- self.name = os.path.join(CFG_WEBDIR, 'sitemap-%02d.xml.gz' % sitemap_id)
+ if name:
+ sitemap_name = name
+ else:
+ sitemap_name = 'sitemap'
+ if base_dir:
+ self.name = os.path.join(base_dir, sitemap_name + '-%02d.xml.gz' % sitemap_id)
+ else:
+ self.name = os.path.join(CFG_WEBDIR, sitemap_name + '-%02d.xml.gz' % sitemap_id)
self.filedescriptor = gzip.open(self.name + '.part', 'w')
self.num_urls = 0
self.file_size = 0
self.filedescriptor.write(self.header)
self.file_size += len(self.footer)
def add_url(self, url, lastmod=datetime(1900, 1, 1), changefreq="", priority="", alternate=False):
""" create a new url node. Returns the number of url nodes in sitemap"""
self.num_urls += 1
canonical_url, alternate_urls = get_canonical_and_alternates_urls(url, drop_ln=not alternate)
url_node = u"""
<url>
<loc>%s</loc>%s
</url>"""
optional = ''
if lastmod:
optional += u"""
<lastmod>%s</lastmod>""" % lastmod.strftime('%Y-%m-%dT%H:%M:%S' + \
DEFAULT_TIMEZONE)
if changefreq:
optional += u"""
<changefreq>%s</changefreq>""" % changefreq
if priority:
optional += u"""
<priority>%s</priority>""" % priority
if alternate:
for ln, alternate_url in iteritems(alternate_urls):
ln = ln.replace('_', '-') ## zh_CN -> zh-CN
optional += u"""
<xhtml:link rel="alternate" hreflang="%s" href="%s" />""" % (ln, encode_for_xml(alternate_url, quote=True))
url_node %= (encode_for_xml(canonical_url), optional)
self.file_size += len(url_node)
self.filedescriptor.write(url_node)
return self.num_urls
def get_size(self):
""" File size. Should not be > 10MB """
return self.file_size + len(self.footer)
def get_number_of_urls(self):
""" Number of urls in the sitemap. Should not be > 50'000"""
return self.num_urls
def get_name(self):
""" Returns the filename """
return self.name
def get_sitemap_url(self):
""" Returns the sitemap URL"""
- return CFG_SITE_URL + '/' + os.path.basename(self.name)
+ return self.name.replace(CFG_WEBDIR, CFG_SITE_URL, 1)
def __del__(self):
""" Writes the whole sitemap """
self.filedescriptor.write(self.footer)
self.filedescriptor.close()
os.rename(self.name + '.part', self.name)
SITEMAP_INDEX_HEADER = \
'<?xml version="1.0" encoding="UTF-8"?>\n' \
'<sitemapindex\n' \
' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"\n' \
' xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9\n' \
' http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd"\n' \
' xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
SITEMAP_INDEX_FOOTER = '\n</sitemapindex>\n'
class SitemapIndexWriter(object):
"""class for writing Sitemap Index files."""
def __init__(self, name):
""" Constructor.
name: path to the sitemap index file to be created
"""
self.header = SITEMAP_INDEX_HEADER
self.footer = SITEMAP_INDEX_FOOTER
self.name = name
self.filedescriptor = gzip.open(self.name + '.part', 'w')
self.num_urls = 0
self.file_size = 0
self.filedescriptor.write(self.header)
self.file_size += len(self.footer)
def add_url(self, url):
""" create a new url node. Returns the number of url nodes in sitemap"""
self.num_urls += 1
url_node = u"""
<sitemap>
<loc>%s</loc>%s
</sitemap>"""
optional = u"""
<lastmod>%s</lastmod>""" % time.strftime('%Y-%m-%dT%H:%M:%S' +\
DEFAULT_TIMEZONE)
url_node %= (url, optional)
self.file_size += len(url_node)
self.filedescriptor.write(url_node)
return self.num_urls
def __del__(self):
""" Writes the whole sitemap """
self.filedescriptor.write(self.footer)
self.filedescriptor.close()
os.rename(self.name + '.part', self.name)
def generate_sitemaps(sitemap_index_writer, collection_names, fulltext_filter=''):
"""
Generate sitemaps themselves. Return list of generated sitemaps files
"""
sitemap_id = 1
writer = SitemapWriter(sitemap_id)
sitemap_index_writer.add_url(writer.get_sitemap_url())
nb_urls = 0
for lang in CFG_SITE_LANGS:
writer.add_url(CFG_SITE_URL + '/?ln=%s' % lang,
lastmod=datetime.today(),
changefreq=DEFAULT_CHANGEFREQ_HOME,
- priority=DEFAULT_PRIORITY_HOME)
+ priority=DEFAULT_PRIORITY_HOME,
+ alternate=True)
nb_urls += 1
write_message("... Getting all public records...")
recids = get_all_public_records(collection_names)
write_message("... Generating urls for %s records..." % len(recids))
task_sleep_now_if_required(can_stop_too=True)
for i, (recid, lastmod) in enumerate(recids):
if nb_urls % 100 == 0 and (writer.get_size() >= MAX_SIZE or nb_urls >= MAX_RECORDS):
sitemap_id += 1
writer = SitemapWriter(sitemap_id)
sitemap_index_writer.add_url(writer.get_sitemap_url())
nb_urls = writer.add_url(CFG_SITE_URL + '/%s/%s' % (CFG_SITE_RECORD, recid),
lastmod = lastmod,
changefreq = DEFAULT_CHANGEFREQ_RECORDS,
priority = DEFAULT_PRIORITY_RECORDS)
if i % 100 == 0:
task_update_progress("Sitemap for recid %s/%s" % (i + 1, len(recids)))
task_sleep_now_if_required(can_stop_too=True)
write_message("... Generating urls for collections...")
collections = get_all_public_collections(collection_names)
for i, (collection, lastmod) in enumerate(collections):
for lang in CFG_SITE_LANGS:
if nb_urls % 100 == 0 and (writer.get_size() >= MAX_SIZE or nb_urls >= MAX_RECORDS):
sitemap_id += 1
writer = SitemapWriter(sitemap_id)
sitemap_index_writer.add_url(writer.get_sitemap_url())
nb_urls = writer.add_url('%s/collection/%s?ln=%s' % (CFG_SITE_URL, quote(collection), lang),
lastmod = lastmod,
changefreq = DEFAULT_CHANGEFREQ_COLLECTIONS,
priority = DEFAULT_PRIORITY_COLLECTIONS,
alternate=True)
if i % 100 == 0:
task_update_progress("Sitemap for collection %s/%s" % (i + 1, len(collections)))
task_sleep_now_if_required(can_stop_too=True)
write_message("... Generating urls for fulltexts...")
recids = filter_fulltexts(recids, fulltext_filter)
for i, (recid, lastmod) in enumerate(recids):
if nb_urls % 100 == 0 and (writer.get_size() >= MAX_SIZE or nb_urls >= MAX_RECORDS):
sitemap_id += 1
writer = SitemapWriter(sitemap_id)
sitemap_index_writer.add_url(writer.get_sitemap_url())
nb_urls = writer.add_url(CFG_SITE_URL + '/%s/%s/files' % (CFG_SITE_RECORD, recid),
lastmod = lastmod,
changefreq = DEFAULT_CHANGEFREQ_FULLTEXTS,
priority = DEFAULT_PRIORITY_FULLTEXTS)
if i % 100 == 0:
task_update_progress("Sitemap for files page %s/%s" % (i, len(recids)))
task_sleep_now_if_required(can_stop_too=True)
write_message("... Generating urls for comments...")
recids = filter_comments(recids)
for i, (recid, lastmod) in enumerate(recids):
if nb_urls % 100 == 0 and (writer.get_size() >= MAX_SIZE or nb_urls >= MAX_RECORDS):
sitemap_id += 1
writer = SitemapWriter(sitemap_id)
sitemap_index_writer.add_url(writer.get_sitemap_url())
nb_urls = writer.add_url(CFG_SITE_URL + '/%s/%s/comments' % (CFG_SITE_RECORD, recid),
lastmod = lastmod,
changefreq = DEFAULT_CHANGEFREQ_COMMENTS,
priority = DEFAULT_PRIORITY_COMMENTS)
if i % 100 == 0:
task_update_progress("Sitemap for comments page %s/%s" % (i, len(recids)))
task_sleep_now_if_required(can_stop_too=True)
write_message("... Generating urls for reviews")
recids = filter_reviews(recids)
for i, (recid, lastmod) in enumerate(recids):
if nb_urls % 100 == 0 and (writer.get_size() >= MAX_SIZE or nb_urls >= MAX_RECORDS):
sitemap_id += 1
write_message("")
writer = SitemapWriter(sitemap_id)
sitemap_index_writer.add_url(writer.get_sitemap_url())
nb_urls = writer.add_url(CFG_SITE_URL + '/%s/%s/reviews' % (CFG_SITE_RECORD, recid),
lastmod = lastmod,
changefreq = DEFAULT_CHANGEFREQ_REVIEWS,
priority = DEFAULT_PRIORITY_REVIEWS)
if i % 100 == 0:
task_update_progress("Sitemap for reviews page %s/%s" % (i, len(recids)))
task_sleep_now_if_required(can_stop_too=True)
def generate_sitemaps_index(collection_list, fulltext_filter=None):
"""main function. Generates the sitemap index and the sitemaps
collection_list: list of collection names to add in sitemap
fulltext_filter: if provided the parser will intergrate only give fulltext
types
"""
write_message("Generating all sitemaps...")
sitemap_index_writer = SitemapIndexWriter(CFG_WEBDIR + '/sitemap-index.xml.gz')
generate_sitemaps(sitemap_index_writer, collection_list, fulltext_filter)
def run_export_method(jobname):
"""Main function, reading params and running the task."""
write_message("bibexport_sitemap: job %s started." % jobname)
collections = get_config_parameter(jobname=jobname, parameter_name="collection", is_parameter_collection=True)
fulltext_type = get_config_parameter(jobname=jobname, parameter_name="fulltext_status")
generate_sitemaps_index(collections, fulltext_type)
write_message("bibexport_sitemap: job %s finished." % jobname)
def get_config_parameter(jobname, parameter_name, is_parameter_collection = False):
"""Detect export method of JOBNAME. Basically, parse JOBNAME.cfg
and return export_method. Return None if problem found."""
jobconfig = ConfigParser()
jobconffile = CFG_ETCDIR + os.sep + 'bibexport' + os.sep + jobname + '.cfg'
if not os.path.exists(jobconffile):
write_message("ERROR: cannot find config file %s." % jobconffile)
return None
jobconfig.read(jobconffile)
if is_parameter_collection:
all_items = jobconfig.items(section='export_job')
parameters = []
for item_name, item_value in all_items:
if item_name.startswith(parameter_name):
parameters.append(item_value)
return parameters
else:
parameter = jobconfig.get('export_job', parameter_name)
return parameter
diff --git a/invenio/legacy/bibformat/adminlib.py b/invenio/legacy/bibformat/adminlib.py
index be682b4a7..e686adced 100644
--- a/invenio/legacy/bibformat/adminlib.py
+++ b/invenio/legacy/bibformat/adminlib.py
@@ -1,1615 +1,1615 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN.
+## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
Handle requests from the web interface to configure BibFormat.
"""
__revision__ = "$Id$"
import os
import re
import stat
import time
import cgi
from invenio.config import CFG_SITE_LANG, CFG_SITE_URL, CFG_ETCDIR
from invenio.modules.formatter.config import \
CFG_BIBFORMAT_TEMPLATES_PATH, \
CFG_BIBFORMAT_OUTPUTS_PATH, \
CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION, \
InvenioBibFormatError
from invenio.utils.url import wash_url_argument
from invenio.ext.logging import register_exception
from invenio.base.i18n import gettext_set_language, wash_language, language_list_long
from invenio.legacy.search_engine import perform_request_search
import invenio.modules.formatter.api as bibformat_dblayer
from invenio.modules.formatter import engine as bibformat_engine
from invenio.modules.formatter import registry
from invenio.utils.text import encode_for_xml
import invenio.legacy.template
bibformat_templates = invenio.legacy.template.load('bibformat')
def getnavtrail(previous = '', ln=CFG_SITE_LANG):
"""
Get the navtrail
@param previous: suffix of the navtrail
@param ln: language
@return: HTML markup of the navigation trail
"""
previous = wash_url_argument(previous, 'str')
ln = wash_language(ln)
_ = gettext_set_language(ln)
navtrail = '''<a class="navtrail" href="%s/help/admin">%s</a> &gt; <a class="navtrail" href="%s/admin/bibformat/bibformatadmin.py?ln=%s">%s</a> ''' % \
(CFG_SITE_URL, _("Admin Area"), CFG_SITE_URL, ln, _("BibFormat Admin"))
navtrail = navtrail + previous
return navtrail
def perform_request_index(ln=CFG_SITE_LANG, warnings=None, is_admin=False):
"""
Returns the main BibFormat admin page.
@param ln: language
@param is_admin: indicate if user is authorized to use BibFormat
@return: the main admin page
"""
return bibformat_templates.tmpl_admin_index(ln, warnings, is_admin)
def perform_request_format_templates_management(ln=CFG_SITE_LANG, checking=0):
"""
Returns the main management console for format templates
@param ln: language
@param checking: the level of checking (0: basic, 1:extensive (time consuming) )
@return: the main page for format templates management
"""
# Reload in case a format was changed
bibformat_engine.clear_caches()
# Get formats lists of attributes
formats = bibformat_engine.get_format_templates(with_attributes=True)
formats_attrs = []
for filename in formats:
attrs = formats[filename]['attrs']
attrs['filename'] = filename
if filename.endswith('.xsl'):
attrs['name'] += ' (XSL)'
attrs['editable'] = can_write_format_template(filename)
path = CFG_BIBFORMAT_TEMPLATES_PATH + os.sep + filename
try:
attrs['last_mod_date'] = time.ctime(os.stat(path)[stat.ST_MTIME])
except OSError:
# File does not exist. Happens with temporary files
# created by editors.
continue
status = check_format_template(filename, checking)
import string
if len(status) > 1 or (len(status)==1 and status[0].find('Could not read format template named') == -1):
status = '''
<a style="color: rgb(255, 0, 0);"
href="%(siteurl)s/admin/bibformat/bibformatadmin.py/validate_format?ln=%(ln)s&amp;bft=%(bft)s">Not OK</a>
''' % {'siteurl':CFG_SITE_URL,
'ln':ln,
'bft':filename}
else:
status = '<span style="color: rgb(0, 255, 0);">OK</span>'
attrs['status'] = status
formats_attrs.append(attrs)
def sort_by_attr(seq):
"""
Sort 'seq' by attribute name.
@param seq: a list of dictionaries, containing each one key named 'name'
"""
intermed = [ (x['name'].lower(), i, x) for i, x in enumerate(seq)]
intermed.sort()
return [x[-1] for x in intermed]
sorted_format_templates = sort_by_attr(formats_attrs)
return bibformat_templates.tmpl_admin_format_templates_management(ln, sorted_format_templates)
def perform_request_format_template_show(bft, ln=CFG_SITE_LANG, code=None,
ln_for_preview=CFG_SITE_LANG, pattern_for_preview="",
content_type_for_preview="text/html"):
"""
Returns the editor for format templates.
@param ln: language
@param bft: the template to edit
@param code: the code being edited
@param ln_for_preview: the language for the preview (for bfo)
@param pattern_for_preview: the search pattern to be used for the preview (for bfo)
@param content_type_for_preview: content-type to use to serve preview
@return: the main page for formats management
"""
format_template = bibformat_engine.get_format_template(filename=bft, with_attributes=True)
# Either use code being edited, or the original code inside template
if code is None:
code = cgi.escape(format_template['code'])
# Build a default pattern if it is empty
if pattern_for_preview == "":
recIDs = perform_request_search()
if len(recIDs) > 0:
recID = recIDs[0]
pattern_for_preview = "recid:%s" % recID
editable = can_write_format_template(bft)
# Look for all existing content_types
content_types = bibformat_dblayer.get_existing_content_types()
# Add some standard content types if not already there
standard_content_types = ['text/xml', 'application/rss+xml', 'text/plain', 'text/html']
content_types.extend([content_type for content_type in standard_content_types
if content_type not in content_types])
return bibformat_templates.tmpl_admin_format_template_show(ln,
code,
bft,
ln_for_preview=ln_for_preview,
pattern_for_preview=pattern_for_preview,
editable=editable,
content_type_for_preview=content_type_for_preview,
content_types=content_types)
def perform_request_format_template_show_dependencies(bft, ln=CFG_SITE_LANG):
"""
Show the dependencies (on elements) of the given format.
@param ln: language
@param bft: the filename of the template to show
@return: HTML markup
"""
format_template = bibformat_engine.get_format_template(filename=bft, with_attributes=True)
name = format_template['attrs']['name']
output_formats = get_outputs_that_use_template(bft)
format_elements = get_elements_used_by_template(bft)
tags = []
for output_format in output_formats:
for tag in output_format['tags']:
tags.append(tag)
for format_element in format_elements:
for tag in format_element['tags']:
tags.append(tag)
tags.sort()
return bibformat_templates.tmpl_admin_format_template_show_dependencies(ln,
name,
bft,
output_formats,
format_elements,
tags)
def perform_request_format_template_show_attributes(bft, ln=CFG_SITE_LANG, new=False):
"""
Page for template name and descrition attributes edition.
If format template is new, offer the possibility to
make a duplicate of an existing format template.
@param ln: language
@param bft: the template to edit
@param new: if True, the template has just been added (is new)
@return: the main page for format templates attributes edition
"""
all_templates = []
if new:
all_templates_attrs = bibformat_engine.get_format_templates(with_attributes=True)
if bft in all_templates_attrs: # Sanity check. Should always be true at this stage
del all_templates_attrs[bft] # Remove in order not to make a duplicate of self..
# Sort according to name, inspired from Python Cookbook
def sort_by_name(seq, keys):
"""
Sort the sequence 'seq' by 'keys'
"""
intermed = [(x['attrs']['name'], keys[i], i, x) for i, x in enumerate(seq)]
intermed.sort()
return [(x[1], x[0]) for x in intermed]
all_templates = sort_by_name(all_templates_attrs.values(), all_templates_attrs.keys())
#keys = all_templates_attrs.keys()
#keys.sort()
#all_templates = map(lambda x: (x, all_templates_attrs.get(x)['attrs']['name']), keys)
format_template = bibformat_engine.get_format_template(filename=bft, with_attributes=True)
name = format_template['attrs']['name']
description = format_template['attrs']['description']
editable = can_write_format_template(bft)
return bibformat_templates.tmpl_admin_format_template_show_attributes(ln,
name,
description,
bft,
editable,
all_templates,
new)
def perform_request_format_template_show_short_doc(ln=CFG_SITE_LANG, search_doc_pattern=""):
"""
Returns the format elements documentation to be included inside format templated editor.
Keep only elements that have 'search_doc_pattern' text inside description,
if pattern not empty
@param ln: language
@param search_doc_pattern: a search pattern that specified which elements to display
@return: a brief version of the format element documentation
"""
# Get format elements lists of attributes
elements = bibformat_engine.get_format_elements(with_built_in_params=True)
keys = elements.keys()
keys.sort()
elements = map(elements.get, keys)
def filter_elem(element):
"""Keep element if is string representation contains all keywords of search_doc_pattern,
and if its name does not start with a number (to remove 'garbage' from elements in tags table)"""
if element['type'] != 'python' and \
element['attrs']['name'][0] in ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']:
return False
text = str(element).upper() # Basic text representation
if search_doc_pattern != "":
for word in search_doc_pattern.split():
if word.upper() != "AND" and text.find(word.upper()) == -1:
return False
return True
elements = filter(filter_elem, elements)
return bibformat_templates.tmpl_admin_format_template_show_short_doc(elements)
def perform_request_format_elements_documentation(ln=CFG_SITE_LANG):
"""
Returns the main management console for format elements.
Includes list of format elements and associated administration tools.
@param ln: language
@return: the main page for format elements management
"""
# Get format elements lists of attributes
elements = bibformat_engine.get_format_elements(with_built_in_params=True)
keys = elements.keys()
keys.sort()
elements = map(elements.get, keys)
# Remove all elements found in table and that begin with a number (to remove 'garbage')
filtered_elements = [element for element in elements \
if element is not None and \
element['type'] == 'python' and \
element['attrs']['name'][0] not in \
['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']]
return bibformat_templates.tmpl_admin_format_elements_documentation(ln, filtered_elements)
def perform_request_format_element_show_dependencies(bfe, ln=CFG_SITE_LANG):
"""
Show the dependencies of the given format.
@param ln: language
@param bfe: the filename of the format element to show
@return: HTML markup of elements dependencies page
"""
format_templates = get_templates_that_use_element(bfe)
tags = get_tags_used_by_element(bfe)
return bibformat_templates.tmpl_admin_format_element_show_dependencies(ln,
bfe,
format_templates,
tags)
def perform_request_format_element_test(bfe, ln=CFG_SITE_LANG, param_values=None, user_info=None):
"""
Show the dependencies of the given format.
'param_values' is the list of values to pass to 'format'
function of the element as parameters, in the order ...
If params is None, this means that they have not be defined by user yet.
@param ln: language
@param bfe: the name of the format element to show
@param param_values: the list of parameters to pass to element format function
@param user_info: the user_info of this request
@return: HTML markup of elements test page
"""
_ = gettext_set_language(ln)
format_element = bibformat_engine.get_format_element(bfe, with_built_in_params=True)
# Load parameter names and description
##
param_names = []
param_descriptions = []
# First value is a search pattern to choose the record
param_names.append(_("Test with record:")) # Caution: keep in sync with same text below
param_descriptions.append(_("Enter a search query here."))
# Parameters defined in this element
for param in format_element['attrs']['params']:
param_names.append(param['name'])
param_descriptions.append(param['description'])
# Parameters common to all elements of a kind
for param in format_element['attrs']['builtin_params']:
param_names.append(param['name'])
param_descriptions.append(param['description'])
# Load parameters values
##
if param_values is None: #First time the page is loaded
param_values = []
# Propose an existing record id by default
recIDs = perform_request_search()
if len(recIDs) > 0:
recID = recIDs[0]
param_values.append("recid:%s" % recID)
# Default values defined in this element
for param in format_element['attrs']['params']:
param_values.append(param['default'])
#Parameters common to all elements of a kind
for param in format_element['attrs']['builtin_params']:
param_values.append(param['default'])
# Execute element with parameters
##
params = dict(zip(param_names, param_values))
# Find a record corresponding to search pattern
search_pattern = params[_("Test with record:")] # Caution keep in sync with same text above and below
recIDs = perform_request_search(p=search_pattern)
del params[_("Test with record:")] # Caution keep in sync with same text above
if len(recIDs) > 0:
bfo = bibformat_engine.BibFormatObject(recID = recIDs[0],
ln = ln,
search_pattern = search_pattern.split(' '),
xml_record = None,
user_info = user_info)
- (result, dummy, dummy) = bibformat_engine.eval_format_element(format_element, bfo, params)
+ (result, dummy) = bibformat_engine.eval_format_element(format_element, bfo, params)
else:
try:
raise InvenioBibFormatError(_('No Record Found for %(x_rec)s.', x_rec=search_pattern))
except InvenioBibFormatError as exc:
register_exception()
result = exc.message
return bibformat_templates.tmpl_admin_format_element_test(ln,
bfe,
format_element['attrs']['description'],
param_names,
param_values,
param_descriptions,
result)
def perform_request_output_formats_management(ln=CFG_SITE_LANG, sortby="code"):
"""
Returns the main management console for output formats.
Includes list of output formats and associated administration tools.
@param ln: language
@param sortby: the sorting crieteria (can be 'code' or 'name')
@return: the main page for output formats management
"""
# Reload in case a format was changed
bibformat_engine.clear_caches()
# Get output formats lists of attributes
output_formats_list = bibformat_engine.get_output_formats(with_attributes=True)
output_formats = {}
for filename in output_formats_list:
output_format = output_formats_list[filename]
code = output_format['attrs']['code']
path = CFG_BIBFORMAT_OUTPUTS_PATH + os.sep + filename
output_format['editable'] = can_write_output_format(code)
try:
output_format['last_mod_date'] = time.ctime(os.stat(path)[stat.ST_MTIME])
except OSError:
# File does not exist. Happens with temporary files
# created by editors.
continue
# Validate the output format
status = check_output_format(code)
# If there is an error but the error is just 'format is not writable', do not display as error
if len(status) > 1 or (len(status)==1 and status[0].find('BibFormat could not write to output format') == -1):
status = '''
<a style="color: rgb(255, 0, 0);"
href="%(siteurl)s/admin/bibformat/bibformatadmin.py/validate_format?ln=%(ln)s&bfo=%(bfo)s">Not OK</a>
''' % {'siteurl':CFG_SITE_URL,
'ln':ln,
'bfo':code}
else:
status = '<span style="color: rgb(0, 255, 0);">OK</span>'
output_format['status'] = status
output_formats[filename] = output_format
# Sort according to code or name, inspired from Python Cookbook
def get_attr(dic, attr):
"""
Returns the value given by 'attr' in the dictionary 'dic', representing
an output format attributes.
If attr is equal to 'code', returns the code attribute of the dictionary.
Else returns the generic name
@param dic: a dictionary of the attribute of an output format, as returned by bibformat_engine.get_output_format
@param the: attribute we want to fetch. Either 'code' or any other string
"""
if attr == "code":
return dic['attrs']['code']
else:
return dic['attrs']['names']['generic']
def sort_by_attr(seq, attr):
"""
Sort dictionaries given in 'seq' according to parameter 'attr'
"""
intermed = [ (get_attr(x, attr), i, x) for i, x in enumerate(seq)]
intermed.sort()
return [x[-1] for x in intermed]
if sortby != "code" and sortby != "name":
sortby = "code"
sorted_output_formats = sort_by_attr(output_formats.values(), sortby)
return bibformat_templates.tmpl_admin_output_formats_management(ln, sorted_output_formats)
def perform_request_output_format_show(bfo, ln=CFG_SITE_LANG, r_fld=[], r_val=[], r_tpl=[], default="", r_upd="", args={}):
"""
Returns the editing tools for a given output format.
The page either shows the output format from file, or from user's
POST session, as we want to let him edit the rules without
saving. Policy is: r_fld, r_val, rules_tpl are list of attributes
of the rules. If they are empty, load from file. Else use
POST. The i th value of each list is one of the attributes of rule
i. Rule i is the i th rule in order of evaluation. All list have
the same number of item.
r_upd contains an action that has to be performed on rules. It
can composed of a number (i, the rule we want to modify) and an
operator : "save" to save the rules, "add" or "del".
syntax: operator [number]
For eg: r_upd = _("Save Changes") saves all rules (no int should be specified).
For eg: r_upd = _("Add New Rule") adds a rule (no int should be specified).
For eg: r_upd = _("Remove Rule") + " 5" deletes rule at position 5.
The number is used only for operation delete.
An action can also be in **args. We must look there for string starting
with '(+|-) [number]' to increase (+) or decrease (-) a rule given by its
index (number).
For example "+ 5" increase priority of rule 5 (put it at fourth position).
The string in **args can be followed by some garbage that looks like .x
or .y, as this is returned as the coordinate of the click on the
<input type="image">. We HAVE to use args and reason on its keys, because for <input> of
type image, iexplorer does not return the value of the tag, but only the name.
Action is executed only if we are working from user's POST session
(means we must have loaded the output format first, which is
totally normal and expected behaviour)
IMPORTANT: we display rules evaluation index starting at 1 in
interface, but we start internally at 0
@param ln: language
@param bfo: the filename of the output format to show
@param r_fld: the list of 'field' attribute for each rule
@param r_val: the list of 'value' attribute for each rule
@param r_tpl: the list of 'template' attribute for each rule
@param default: the default format template used by this output format
@param r_upd: the rule that we want to increase/decrease in order of evaluation
@param args: additional parameters to move rules. See above
@return: HTML markuo for editing tools of a given output format.
"""
output_format = bibformat_engine.get_output_format(bfo, with_attributes=True)
format_templates = bibformat_engine.get_format_templates(with_attributes=True)
name = output_format['attrs']['names']['generic']
rules = []
debug = ""
if len(r_fld) == 0 and r_upd=="":
# Retrieve rules from file
rules = output_format['rules']
default = output_format['default']
else:
# Retrieve rules from given lists
# Transform a single rule (not considered as a list with length
# 1 by the templating system) into a list
if not isinstance(r_fld, list):
r_fld = [r_fld]
r_val = [r_val]
r_tpl = [r_tpl]
for i in range(len(r_fld)):
rule = {'field': r_fld[i],
'value': r_val[i],
'template': r_tpl[i]}
rules.append(rule)
# Execute action
_ = gettext_set_language(ln)
if r_upd.startswith(_("Remove Rule")):
# Remove rule
index = int(r_upd.split(" ")[-1]) -1
del rules[index]
elif r_upd.startswith(_("Save Changes")):
# Save
update_output_format_rules(bfo, rules, default)
elif r_upd.startswith(_("Add New Rule")):
# Add new rule
rule = {'field': "",
'value': "",
'template': ""}
rules.append(rule)
else:
# Get the action in 'args'
# The action must be constructed from string of the kind:
# + 5 or - 4 or + 5.x or -4.y
for button_val in args.keys():#for all elements of form not handled yet
action = button_val.split(" ")
if action[0] == '-' or action[0] == '+':
index = int(action[1].split(".")[0]) -1
if action[0] == '-':
# Decrease priority
rule = rules[index]
del rules[index]
rules.insert(index + 1, rule)
# debug = 'Decrease rule '+ str(index)
break
elif action[0] == '+':
# Increase priority
rule = rules[index]
del rules[index]
rules.insert(index - 1, rule)
# debug = 'Increase rule ' + str(index)
break
editable = can_write_output_format(bfo)
return bibformat_templates.tmpl_admin_output_format_show(ln,
bfo,
rules,
default,
format_templates,
editable)
def perform_request_output_format_show_dependencies(bfo, ln=CFG_SITE_LANG):
"""
Show the dependencies of the given format.
@param ln: language
@param bfo: the filename of the output format to show
@return: HTML markup of the output format dependencies pages
"""
output_format = bibformat_engine.get_output_format(code=bfo, with_attributes=True)
name = output_format['attrs']['names']['generic']
format_templates = get_templates_used_by_output(bfo)
return bibformat_templates.tmpl_admin_output_format_show_dependencies(ln,
name,
bfo,
format_templates)
def perform_request_output_format_show_attributes(bfo, ln=CFG_SITE_LANG):
"""
Page for output format names and description attributes edition.
@param ln: language
@param bfo: filename of output format to edit
@return: the main page for output format attributes edition
"""
output_format = bibformat_engine.get_output_format(code=bfo, with_attributes=True)
name = output_format['attrs']['names']['generic']
description = output_format['attrs']['description']
content_type = output_format['attrs']['content_type']
visible = output_format['attrs']['visibility']
# Get translated names. Limit to long names now.
# Translation are given in order of languages in language_list_long()
names_trans = []
for lang in language_list_long():
name_trans = output_format['attrs']['names']['ln'].get(lang[0], "")
names_trans.append({'lang':lang[1], 'trans':name_trans})
editable = can_write_output_format(bfo)
return bibformat_templates.tmpl_admin_output_format_show_attributes(ln,
name,
description,
content_type,
bfo,
names_trans,
editable,
visible)
def add_format_template():
"""
Adds a new format template (mainly create file with unique name)
@return: the filename of the created format
"""
(filename, name) = bibformat_engine.get_fresh_format_template_filename("Untitled")
out = ""
if not filename.endswith(".xsl"):
out = '<name>%(name)s</name><description></description>' % {'name': name}
path = CFG_BIBFORMAT_TEMPLATES_PATH + os.sep + filename
format = open(path, 'w')
format.write(out)
format.close
return filename
def delete_format_template(filename):
"""
Delete a format template given by its filename
If format template is not writable, do not remove
@param filename: the format template filename
@return: None
"""
if not can_write_format_template(filename):
return
path = CFG_BIBFORMAT_TEMPLATES_PATH + os.sep + filename
os.remove(path)
bibformat_engine.clear_caches()
def update_format_template_code(filename, code=""):
"""
Saves code inside template given by filename
@param filename: filename of the template to edit
@param code: content of the template
@return: None
"""
format_template = bibformat_engine.get_format_template_attrs(filename)
name = format_template['name']
description = format_template['description']
code = re.sub("\r\n", "\n", code)
out = ""
if not filename.endswith(".xsl"):
out = """<name>%(name)s</name>
<description>%(description)s</description>
""" % {'name': name, 'description': description}
out += code
path = CFG_BIBFORMAT_TEMPLATES_PATH + os.sep + filename
format = open(path, 'w')
format.write(out)
format.close
bibformat_engine.clear_caches()
def update_format_template_attributes(filename, name="", description="", duplicate=None):
"""
Saves name and description inside template given by filename.
the filename must change according to name, and every output format
having reference to filename must be updated.
If name already exist, use fresh filename (we never overwrite other templates) amd
remove old one.
if duplicate is different from None and is not empty string, then it means that we must copy
the code of the template whoose filename is given in 'duplicate' for the code
of our template.
@param filename: filename of the template to update
@param name: name to use for the template
@param description: description to use for the template
@param duplicate: the filename of a template that we want to copy
@return: the filename of the modified format
"""
if filename.endswith('.'+CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION):
format_template = bibformat_engine.get_format_template(filename, with_attributes=True)
if duplicate is not None and duplicate != "":
format_template_to_copy = bibformat_engine.get_format_template(duplicate)
code = format_template_to_copy['code']
else:
code = format_template['code']
if format_template['attrs']['name'] != name:
# Name has changed, so update filename
old_filename = filename
old_path = CFG_BIBFORMAT_TEMPLATES_PATH + os.sep + old_filename
# Remove old one
os.remove(old_path)
(filename, name) = bibformat_engine.get_fresh_format_template_filename(name)
# Change output formats that calls this template
output_formats = bibformat_engine.get_output_formats()
for output_format_filename in output_formats:
if can_read_output_format(output_format_filename) and can_write_output_format(output_format_filename):
output_path = CFG_BIBFORMAT_OUTPUTS_PATH + os.sep + output_format_filename
format = open(output_path, 'r')
output_text = format.read()
format.close
output_pattern = re.compile("---(\s)*" + old_filename, re.IGNORECASE)
mod_output_text = output_pattern.sub("--- " + filename, output_text)
if output_text != mod_output_text:
format = open(output_path, 'w')
format.write(mod_output_text)
format.close
description = cgi.escape(description)
name = cgi.escape(name)
# Write updated format template
out = ""
if not filename.endswith(".xsl"):
out = """<name>%(name)s</name><description>%(description)s</description>""" % {'name': name,
'description': description,}
out += code
path = CFG_BIBFORMAT_TEMPLATES_PATH + os.sep + filename
format = open(path, 'w')
format.write(out)
format.close
bibformat_engine.clear_caches()
return filename
def add_output_format():
"""
Adds a new output format (mainly create file with unique name)
@return: the code of the created format, or None if it could not be created
"""
if not os.access(CFG_BIBFORMAT_OUTPUTS_PATH, os.W_OK):
return None
(filename, code) = bibformat_engine.get_fresh_output_format_filename("UNTLD")
# Add entry in database
bibformat_dblayer.add_output_format(code)
bibformat_dblayer.set_output_format_name(code, "Untitled", lang="generic")
bibformat_dblayer.set_output_format_content_type(code, "text/html")
# Add file
out = ""
path = CFG_BIBFORMAT_OUTPUTS_PATH + os.sep + filename
format = open(path, 'w')
format.write(out)
format.close
return code
def delete_output_format(code):
"""
Delete a format template given by its code
if file is not writable, don't remove
@param code: the 6 letters code of the output format to remove
@return: None
"""
if not can_write_output_format(code):
return
# Remove entry from database
bibformat_dblayer.remove_output_format(code)
# Remove file
filename = bibformat_engine.resolve_output_format_filename(code)
path = CFG_BIBFORMAT_OUTPUTS_PATH + os.sep + filename
os.remove(path)
bibformat_engine.clear_caches()
def update_output_format_rules(code, rules=[], default=""):
"""
Saves rules inside output format given by code
@param code: the code of the output format to update
@param rules: the rules to apply for the output format
@param default: the default template when no rule match
@return: None
"""
# Generate output format syntax
# Try to group rules by field
previous_field = ""
out = ""
for rule in rules:
field = rule["field"]
value = rule["value"]
template = rule["template"]
if previous_field != field:
out += "tag %s:\n" % field
out +="%(value)s --- %(template)s\n" % {'value':value, 'template':template}
previous_field = field
out += "default: %s" % default
filename = bibformat_engine.resolve_output_format_filename(code)
path = CFG_BIBFORMAT_OUTPUTS_PATH + os.sep + filename
format = open(path, 'w')
format.write(out)
format.close
bibformat_engine.clear_caches()
def update_output_format_attributes(code, name="", description="", new_code="",
content_type="", names_trans=[], visibility=1):
"""
Saves name and description inside output format given by filename.
If new_code already exist, use fresh code (we never overwrite other output).
@param description: the new description
@param name: the new name
@param new_code: the new short code (== new bfo) of the output format
@param code: the code of the output format to update
@param names_trans: the translations in the same order as the languages from get_languages()
@param content_type: the new content_type of the output format
@param visibility: the visibility of the output format in the output formats list (public pages)
@return: the filename of the modified format
"""
bibformat_dblayer.set_output_format_description(code, description)
bibformat_dblayer.set_output_format_content_type(code, content_type)
bibformat_dblayer.set_output_format_visibility(code, visibility)
bibformat_dblayer.set_output_format_name(code, name, lang="generic")
i = 0
for lang in language_list_long():
if names_trans[i] != "":
bibformat_dblayer.set_output_format_name(code, names_trans[i], lang[0])
i += 1
new_code = new_code.upper()
if code != new_code:
# If code has changed, we must update filename with a new unique code
old_filename = bibformat_engine.resolve_output_format_filename(code)
old_path = CFG_BIBFORMAT_OUTPUTS_PATH + os.sep + old_filename
(new_filename, new_code) = bibformat_engine.get_fresh_output_format_filename(new_code)
new_path = CFG_BIBFORMAT_OUTPUTS_PATH + os.sep + new_filename
os.rename(old_path, new_path)
bibformat_dblayer.change_output_format_code(code, new_code)
bibformat_engine.clear_caches()
return new_code
def can_read_format_template(filename):
"""
Returns 0 if we have read permission on given format template, else
returns other integer
@param filename: name of a format template
@return: True if template X{bft} can be read or not
"""
path = "%s%s%s" % (CFG_BIBFORMAT_TEMPLATES_PATH, os.sep, filename)
return os.access(path, os.R_OK)
def can_read_output_format(bfo):
"""
Returns 0 if we have read permission on given output format, else
returns other integer
@param bfo: name of an output format
@return: True if output format X{bfo} can be read or not
"""
filename = bibformat_engine.resolve_output_format_filename(bfo)
path = "%s%s%s" % (CFG_BIBFORMAT_OUTPUTS_PATH, os.sep, filename)
return os.access(path, os.R_OK)
def can_read_format_element(name):
"""
Returns 0 if we have read permission on given format element, else
returns other integer
@param name: name of a format element
@return: True if element X{name} can be read or not
"""
filename = bibformat_engine.resolve_format_element_filename(name)
path = bibformat_engine.get_format_element_path(filename)
return os.access(path, os.R_OK)
def can_write_format_template(bft):
"""
Returns 0 if we have write permission on given format template, else
returns other integer
@param bft: name of a format template
@return: True if template X{bft} can be edited or not
"""
if not can_read_format_template(bft):
return False
path = "%s%s%s" % (CFG_BIBFORMAT_TEMPLATES_PATH, os.sep, bft)
return os.access(path, os.W_OK)
def can_write_output_format(bfo):
"""
Returns 0 if we have write permission on given output format, else
returns other integer
@param bfo: name of an output format
@return: True if output format X{bfo} can be edited or not
"""
if not can_read_output_format(bfo):
return False
filename = bibformat_engine.resolve_output_format_filename(bfo)
path = "%s%s%s" % (CFG_BIBFORMAT_OUTPUTS_PATH, os.sep, filename)
return os.access(path, os.W_OK)
def can_write_etc_bibformat_dir():
"""
Returns true if we can write in etc/bibformat dir.
@return: True if can write, or False
"""
path = "%s%sbibformat" % (CFG_ETCDIR, os.sep)
return os.access(path, os.W_OK)
def get_outputs_that_use_template(filename):
"""
Returns a list of output formats that call the given format template.
The returned output formats also give their dependencies on tags.
We don't return the complete output formats but some reference to
them (filename + names)::
[ {'filename':"filename_1.bfo"
'names': {'en':"a name", 'fr': "un nom", 'generic':"a name"}
'tags': ['710__a', '920__']
},
...
]
@param filename: a format template filename
@return: output formats references sorted by (generic) name
"""
output_formats_list = {}
tags = []
output_formats = bibformat_engine.get_output_formats(with_attributes=True)
for output_format in output_formats:
name = output_formats[output_format]['attrs']['names']['generic']
# First look at default template, and add it if necessary
if output_formats[output_format]['default'] == filename:
output_formats_list[name] = {'filename':output_format,
'names':output_formats[output_format]['attrs']['names'],
'tags':[]}
# Second look at each rule
found = False
for rule in output_formats[output_format]['rules']:
if rule['template'] == filename:
found = True
tags.append(rule['field']) #Also build dependencies on tags
# Finally add dependency on template from rule (overwrite default dependency,
# which is weaker in term of tag)
if found:
output_formats_list[name] = {'filename':output_format,
'names':output_formats[output_format]['attrs']['names'],
'tags':tags}
keys = output_formats_list.keys()
keys.sort()
return map(output_formats_list.get, keys)
def get_elements_used_by_template(filename):
"""
Returns a list of format elements that are called by the given format template.
The returned elements also give their dependencies on tags.
Dependencies on tag might be approximative. See get_tags_used_by_element()
doc string.
We must handle usage of bfe_field in a special way if we want to retrieve
used tag: used tag is given in "tag" parameter, not inside element code.
The list is returned sorted by name::
[ {'filename':"filename_1.py"
'name':"filename_1"
'tags': ['710__a', '920__']
},
...
]
@param filename: a format template filename
@return: elements sorted by name
"""
format_elements = {}
format_template = bibformat_engine.get_format_template(filename=filename, with_attributes=True)
code = format_template['code']
format_elements_iter = bibformat_engine.pattern_tag.finditer(code)
for result in format_elements_iter:
function_name = result.group("function_name").lower()
if function_name is not None and function_name not in format_elements \
and not function_name == "field":
filename = bibformat_engine.resolve_format_element_filename("BFE_"+function_name)
if filename is not None:
tags = get_tags_used_by_element(filename)
format_elements[function_name] = {'name':function_name.lower(),
'filename':filename,
'tags':tags}
elif function_name == "field":
# Handle bfe_field element in a special way
if function_name not in format_elements:
#Indicate usage of bfe_field if not already done
filename = bibformat_engine.resolve_format_element_filename("BFE_"+function_name)
format_elements[function_name] = {'name':function_name.lower(),
'filename':filename,
'tags':[]}
# Retrieve value of parameter "tag"
all_params = result.group('params')
function_params_iterator = bibformat_engine.pattern_function_params.finditer(all_params)
for param_match in function_params_iterator:
name = param_match.group('param')
if name == "tag":
value = param_match.group('value')
if not value in format_elements[function_name]['tags']:
format_elements[function_name]['tags'].append(value)
break
keys = format_elements.keys()
keys.sort()
return map(format_elements.get, keys)
# Format Elements Dependencies
##
def get_tags_used_by_element(filename):
"""
Returns a list of tags used by given format element
APPROXIMATIVE RESULTS: the tag are retrieved in field(), fields()
and control_field() function. If they are used computed, or saved
in a variable somewhere else, they are not retrieved
@TODO: There is room for improvements. For example catch
call to BibRecord functions.
@param filename: a format element filename
@return: tags sorted by value
"""
tags = {}
format_element = bibformat_engine.get_format_element(filename)
if format_element is None:
return []
elif format_element['type']=="field":
tags = format_element['attrs']['tags']
return tags
filename = bibformat_engine.resolve_format_element_filename(filename)
path = bibformat_engine.get_format_element_path(filename)
format = open(path, 'r')
code = format.read()
format.close
tags_pattern = re.compile('''
(field|fields|control_field)\s* #Function call
\(\s* #Opening parenthesis
[\'"]+ #Single or double quote
(?P<tag>.+?) #Tag
[\'"]+\s* #Single or double quote
(,[^\)]+)* #Additional function param
\) #Closing parenthesis
''', re.VERBOSE | re.MULTILINE)
tags_iter = tags_pattern.finditer(code)
for result in tags_iter:
tags[result.group("tag")] = result.group("tag")
return tags.values()
def get_templates_that_use_element(name):
"""
Returns a list of format templates that call the given format element.
The returned format templates also give their dependencies on tags::
[ {'filename':"filename_1.bft"
'name': "a name"
'tags': ['710__a', '920__']
},
...
]
@param name: a format element name
@return: templates sorted by name
"""
format_templates = {}
tags = []
files = os.listdir(CFG_BIBFORMAT_TEMPLATES_PATH) #Retrieve all templates
for possible_template in files:
if possible_template.endswith(CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION):
format_elements = get_elements_used_by_template(possible_template) #Look for elements used in template
format_elements = map(lambda x: x['name'].lower(), format_elements)
try: #Look for element
format_elements.index(name.lower()) #If not found, get out of "try" statement
format_template = bibformat_engine.get_format_template(filename=possible_template, with_attributes=True)
template_name = format_template['attrs']['name']
format_templates[template_name] = {'name':template_name,
'filename':possible_template}
except:
pass
keys = format_templates.keys()
keys.sort()
return map(format_templates.get, keys)
# Output Formats Dependencies
##
def get_templates_used_by_output(code):
"""
Returns a list of templates used inside an output format give by its code
The returned format templates also give their dependencies on elements and tags::
[ {'filename':"filename_1.bft"
'name': "a name"
'elements': [{'filename':"filename_1.py", 'name':"filename_1", 'tags': ['710__a', '920__']
}, ...]
},
...
]
@param code: outpout format code
@return: templates sorted by name
"""
format_templates = {}
output_format = bibformat_engine.get_output_format(code, with_attributes=True)
filenames = map(lambda x: x['template'], output_format['rules'])
if output_format['default'] != "":
filenames.append(output_format['default'])
for filename in filenames:
template = bibformat_engine.get_format_template(filename, with_attributes=True)
name = template['attrs']['name']
elements = get_elements_used_by_template(filename)
format_templates[name] = {'name':name,
'filename':filename,
'elements':elements}
keys = format_templates.keys()
keys.sort()
return map(format_templates.get, keys)
# Validation tools
##
def perform_request_format_validate(ln=CFG_SITE_LANG, bfo=None, bft=None, bfe=None):
"""
Returns a page showing the status of an output format or format
template or format element. This page is called from output
formats management page or format template management page or
format elements documentation.
The page only shows the status of one of the format, depending on
the specified one. If multiple are specified, shows the first one.
@param ln: language
@param bfo: an output format 6 chars code
@param bft: a format element filename
@param bfe: a format element name
@return: HTML markup
"""
if bfo is not None:
messages = check_output_format(bfo)
elif bft is not None:
messages = check_format_template(bft, checking=1)
elif bfe is not None:
messages = check_format_element(bfe)
if messages is None:
messages = []
messages = [encode_for_xml(message) for message in messages]
return bibformat_templates.tmpl_admin_validate_format(ln, messages)
def check_output_format(code):
"""
Returns the list of errors in the output format given by code
The errors are the formatted errors defined in bibformat_config.py file.
@param code: the 6 chars code of the output format to check
@return: a list of errors
"""
_ = gettext_set_language(CFG_SITE_LANG)
errors = []
filename = bibformat_engine.resolve_output_format_filename(code)
if can_read_output_format(code):
path = CFG_BIBFORMAT_OUTPUTS_PATH + os.sep + filename
format = open(path)
current_tag = ''
i = 0
for line in format:
i += 1
if line.strip() == "":
# Ignore blank lines
continue
clean_line = line.rstrip("\n\r ") #remove spaces and eol
if line.strip().endswith(":") or (line.strip().lower().startswith("tag") and line.find('---') == -1):
# Check tag
if not clean_line.endswith(":"):
# Column misses at the end of line
try:
raise InvenioBibFormatError(_('Tag specification "%(x_line)s" must end with column ":" at line %(x_col)s.',
x_line=line, x_col=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
if not clean_line.lower().startswith("tag"):
# Tag keyword is missing
try:
raise InvenioBibFormatError(_('Tag specification "%(x_line)s" must start with "tag" at line %(x_col)s.',
x_line=line, x_col=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
elif not clean_line.startswith("tag"):
# Tag was not lower case
try:
raise InvenioBibFormatError(_('"tag" must be lowercase in "%(x_line)s" at line %(x_col)s.',
x_line=line, x_col=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
clean_line = clean_line.rstrip(": ") #remove : and spaces at the end of line
current_tag = "".join(clean_line.split()[1:]).strip() #the tag starts at second position
if len(clean_line.split()) > 2: #We should only have 'tag' keyword and tag
try:
raise InvenioBibFormatError(_('Should be "tag field_number:" at line %(x_line)s.', x_line=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
else:
if len(check_tag(current_tag)) > 0:
# Invalid tag
try:
raise InvenioBibFormatError(_('Invalid tag "%(x_line)s" at line %(x_col)s.', x_line=current_tag, x_col=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
if not clean_line.startswith("tag"):
try:
raise InvenioBibFormatError(_('Should be "tag field_number:" at line %(x_line)s.', x_line=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
elif line.find('---') != -1:
# Check condition
if current_tag == "":
try:
raise InvenioBibFormatError(_('Condition "%(x_line)s" is outside a tag specification at line %(x_col)s.',
x_line=line, x_col=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
words = line.split('---')
if len(words) != 2:
try:
raise InvenioBibFormatError(_('Condition "%(x_line)s" can only have a single separator --- at line %(x_col)s.',
x_line=line, x_col=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
template = words[-1].strip()
path = CFG_BIBFORMAT_TEMPLATES_PATH + os.sep + template
if not os.path.exists(path):
try:
raise InvenioBibFormatError(_('Template "%(x_tem)s" does not exist at line %(x_line)s.',
x_tem=template, x_line=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
elif line.find(':') != -1 or (line.strip().lower().startswith("default") and line.find('---') == -1):
# Check default template
clean_line = line.strip()
if line.find(':') == -1:
# Column misses after default
try:
raise InvenioBibFormatError(_('Missing column ":" after "default" in "%(x_line)s" at line %(x_col)s.',
x_line=line, x_col=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
if not clean_line.startswith("default"):
# Default keyword is missing
try:
raise InvenioBibFormatError(_('Default template specification "%(x_line)s" must start with "default :" at line %(x_col)s.',
x_line=line, x_col=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
if not clean_line.startswith("default"):
# Default was not lower case
try:
raise InvenioBibFormatError(_('"default" keyword must be lowercase in "%(x_line)s" at line %(x_col)s.',
x_line=line, x_col=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
default = "".join(line.split(':')[1]).strip()
path = CFG_BIBFORMAT_TEMPLATES_PATH + os.sep + default
if not os.path.exists(path):
try:
raise InvenioBibFormatError(_('Template "%(x_tem)s" does not exist at line %(x_col)s.',
x_tem=default, x_col=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
else:
# Check others
try:
raise InvenioBibFormatError(_('Line %(x_line)s could not be understood at line %(x_col)s.',
x_line=line, x_col=i))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
else:
try:
raise InvenioBibFormatError(_('Output format %(x_file)s cannot not be read. %(x_text)s',
x_file=filename, x_text=""))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
return errors
def check_format_template(filename, checking=0):
"""
Returns the list of errors in the format template given by its filename
The errors are the formatted errors defined in bibformat_config.py file.
@param filename: the filename of the format template to check
@param checking: the level of checking (0:basic, >=1 extensive (time-consuming))
@return: a list of errors
"""
errors = []
_ = gettext_set_language(CFG_SITE_LANG)
if can_read_format_template(filename):#Can template be read?
if filename.endswith('.'+CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION):
#format_template = bibformat_engine.get_format_template(filename, with_attributes=True)
format = open("%s%s%s" % (CFG_BIBFORMAT_TEMPLATES_PATH, os.sep, filename))
code = format.read()
format.close()
# Look for name
match = bibformat_engine.pattern_format_template_name.search(code)
if match is None:#Is tag <name> defined in template?
try:
raise InvenioBibFormatError(_('Could not find a name specified in tag "<name>" inside format template %(x_file)s.',
x_file=filename))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
# Look for description
match = bibformat_engine.pattern_format_template_desc.search(code)
if match is None:#Is tag <description> defined in template?
try:
raise InvenioBibFormatError(_('Could not find a description specified in tag "<description>" inside format template %(x_file)s.',
x_file=filename))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
format_template = bibformat_engine.get_format_template(filename, with_attributes=False)
code = format_template['code']
# Look for calls to format elements
# Check existence of elements and attributes used in call
elements_call = bibformat_engine.pattern_tag.finditer(code)
for element_match in elements_call:
element_name = element_match.group("function_name")
filename = bibformat_engine.resolve_format_element_filename(element_name)
if filename is None and not bibformat_dblayer.tag_exists_for_name(element_name): #Is element defined?
try:
raise InvenioBibFormatError(_('Format template %(x_file)s calls undefined element "%(x_elem)s".',
x_file=filename, x_elem=element_name))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
else:
format_element = bibformat_engine.get_format_element(element_name, with_built_in_params=True)
if format_element is None:#Can element be loaded?
if not can_read_format_element(element_name):
try:
raise InvenioBibFormatError(_('Format template %(x_file)s calls unreadable element "%(x_elem)s". Check element file permissions.',
x_file=filename, x_elem=element_name))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
else:
try:
raise InvenioBibFormatError(_('Cannot load element "%(x_file)s" in template %(x_elem)s. Check element code.',
x_file=filename, x_elem=element_name))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
else:
# Are the parameters used defined in element?
params_call = bibformat_engine.pattern_function_params.finditer(element_match.group())
all_params = {}
for param_match in params_call:
param = param_match.group("param")
value = param_match.group("value")
all_params[param] = value
allowed_params = []
# Built-in params
for allowed_param in format_element['attrs']['builtin_params']:
allowed_params.append(allowed_param['name'])
# Params defined in element
for allowed_param in format_element['attrs']['params']:
allowed_params.append(allowed_param['name'])
if not param in allowed_params:
try:
raise InvenioBibFormatError(_('Format element %(x_elem)s uses unknown parameter "%(x_param)s" in format template %(x_file)s.',
x_elem=element_name, x_param=param, x_file=filename))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
# The following code is too much time consuming. Only do where really requested
if checking > 0:
# Try to evaluate, with any object and pattern
recIDs = perform_request_search()
if len(recIDs) > 0:
recID = recIDs[0]
bfo = bibformat_engine.BibFormatObject(recID, search_pattern="Test")
- (result, errors_, dummy) = bibformat_engine.eval_format_element(format_element, bfo, all_params, verbose=7)
+ result, errors_ = bibformat_engine.eval_format_element(format_element, bfo, all_params, verbose=7)
errors.extend(errors_)
else:# Template cannot be read
try:
raise InvenioBibFormatError(_('Could not read format template named %(x_file)s. %(x_text)s',
x_file=filename, x_text=""))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
return errors
def check_format_element(name):
"""
Returns the list of errors in the format element given by its name
The errors are the formatted errors defined in bibformat_config.py file.
@param name: the name of the format element to check
@return: a list of errors
"""
errors = []
_ = gettext_set_language(CFG_SITE_LANG)
filename = bibformat_engine.resolve_format_element_filename(name)
if filename is not None:#Can element be found in files?
if can_read_format_element(name):#Can element be read?
# Try to load
try:
module_name = filename
if module_name.endswith(".py"):
module_name = module_name[:-3]
module = __import__("invenio.modules.formatter.format_elements."+module_name)
try:
function_format = module.bibformat_elements.__dict__[module_name].format_element
except AttributeError as e:
function_format = module.bibformat_elements.__dict__[module_name].format
# Try to evaluate, with any object and pattern
recIDs = perform_request_search()
if len(recIDs) > 0:
recID = recIDs[0]
bfo = bibformat_engine.BibFormatObject(recID, search_pattern="Test")
element = bibformat_engine.get_format_element(name)
- (result, errors_, dummy) = bibformat_engine.eval_format_element(element, bfo, verbose=7)
+ result, errors_ = bibformat_engine.eval_format_element(element, bfo, verbose=7)
errors.extend(errors_)
except Exception as e:
try:
raise InvenioBibFormatError(_('Error in format element %(x_elem)s. %(x_e)s.', x_elem=name, x_e=e))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
else:
try:
raise InvenioBibFormatError(_('Format element %(x_file)s cannot not be read. %(x_text)s',
x_file=filename, x_text=""))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
elif bibformat_dblayer.tag_exists_for_name(name):#Can element be found in database?
pass
else:
try:
raise InvenioBibFormatError(_('Could not find format element named %(x_name)s.', x_name=name))
except InvenioBibFormatError as exc:
register_exception()
errors.append(exc.message)
return errors
def check_tag(tag):
"""
Checks the validity of a tag
@param tag: tag to check
@return: list of errors for the tag
"""
errors = []
return errors
def perform_request_dreamweaver_floater():
"""
Returns a floater for Dreamweaver with all Format Elements available.
@return: HTML markup (according to Dreamweaver specs)
"""
# Get format elements lists of attributes
elements = bibformat_engine.get_format_elements(with_built_in_params=True)
keys = elements.keys()
keys.sort()
elements = map(elements.get, keys)
def filter_elem(element):
"""Keep element if is string representation contains all keywords of search_doc_pattern,
and if its name does not start with a number (to remove 'garbage' from elements in tags table)"""
if element['type'] != 'python' and \
element['attrs']['name'][0] in ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']:
return False
else:
return True
elements = filter(filter_elem, elements)
return bibformat_templates.tmpl_dreamweaver_floater(CFG_SITE_LANG, elements)
diff --git a/invenio/legacy/bibindex/adminlib.py b/invenio/legacy/bibindex/adminlib.py
index 8442ea65d..8d1794c1d 100644
--- a/invenio/legacy/bibindex/adminlib.py
+++ b/invenio/legacy/bibindex/adminlib.py
@@ -1,2773 +1,2828 @@
## This file is part of Invenio.
## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""Invenio BibIndex Administrator Interface."""
__revision__ = "$Id$"
import random
from six import iteritems
from invenio.config import \
CFG_SITE_LANG, \
CFG_SITE_URL, \
CFG_BINDIR
from invenio.legacy.bibrank.adminlib import write_outcome, modify_translations, \
get_def_name, get_name, get_languages, addadminbox, tupletotable, \
createhiddenform
from invenio.modules.access.engine import acc_authorize_action
from invenio.legacy.dbquery import run_sql, get_table_status_info, wash_table_column_name
from invenio.legacy.bibindex.engine_stemmer import get_stemming_language_map
import invenio.legacy.template
from invenio.legacy.bibindex.engine_config import CFG_BIBINDEX_SYNONYM_MATCH_TYPE, \
CFG_BIBINDEX_COLUMN_VALUE_SEPARATOR
from invenio.modules.knowledge.dblayer import get_all_kb_names
from invenio.legacy.bibindex.engine_utils import load_tokenizers, \
get_idx_indexer, \
get_all_indexes, \
get_all_virtual_indexes, \
get_virtual_index_building_blocks, \
get_index_name_from_index_id, \
get_all_index_names_and_column_values, \
- is_index_virtual
+ is_index_virtual, \
+ get_index_virtual_indexes
from invenio.utils.datastructures import LazyDict
_TOKENIZERS = LazyDict(load_tokenizers)
websearch_templates = invenio.legacy.template.load('websearch')
def getnavtrail(previous = ''):
"""Get the navtrail"""
navtrail = """<a class="navtrail" href="%s/help/admin">Admin Area</a> """ % (CFG_SITE_URL,)
navtrail = navtrail + previous
return navtrail
def perform_index(ln=CFG_SITE_LANG, mtype='', content='', **params):
"""start area for modifying indexes
mtype - the method that called this method.
content - the output from that method."""
fin_output = """
<table>
<tr>
<td>0.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/index?ln=%s">Show all</a></small></td>
<td>1.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/index?ln=%s&amp;mtype=perform_showindexoverview#1">Overview of indexes</a></small></td>
<td>2.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/index?ln=%s&amp;mtype=perform_showvirtualindexoverview#2">Overview of virtual indexes</a></small></td>
<td>3.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/index?ln=%s&amp;mtype=perform_editindexes#2">Edit index</a></small></td>
<td>4.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/index?ln=%s&amp;mtype=perform_addindex#3">Add new index</a></small></td>
<td>5.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/field?ln=%s">Manage logical fields</a></small></td>
<td>6.&nbsp;<small><a href="%s/help/admin/bibindex-admin-guide">Guide</a></small></td>
</tr>
</table>
""" % (CFG_SITE_URL, ln, CFG_SITE_URL, ln, CFG_SITE_URL, ln, CFG_SITE_URL, ln, CFG_SITE_URL, ln, CFG_SITE_URL, ln, CFG_SITE_URL)
if mtype == "perform_showindexoverview" and content:
fin_output += content
elif mtype == "perform_showindexoverview" or not mtype:
fin_output += perform_showindexoverview(ln, callback='', **params)
if mtype == "perform_showvirtualindexoverview" and content:
fin_output += content
elif mtype == "perform_showvirtualindexoverview" or not mtype:
fin_output += perform_showvirtualindexoverview(ln, callback='', **params)
if mtype == "perform_editindexes" and content:
fin_output += content
elif mtype == "perform_editindexes" or not mtype:
fin_output += perform_editindexes(ln, callback='', **params)
if mtype == "perform_addindex" and content:
fin_output += content
elif mtype == "perform_addindex" or not mtype:
fin_output += perform_addindex(ln, callback='', **params)
if mtype == "perform_editvirtualindexes" and content:
fin_output += content
elif mtype == "perform_editvirtualindexes":
#not visible in 'show all' view of 'Manage Indexes'
fin_output += perform_editvirtualindexes(ln, callback='', **params)
if mtype == "perform_addvirtualindex" and content:
fin_output += content
elif mtype == "perform_addvirtualindex":
#not visible in 'show all' view of 'Manage Indexes'
fin_output += perform_addvirtualindex(ln, callback='', **params)
if mtype == "perform_deletevirtualindex" and content:
fin_output += content
elif mtype == "perform_deletevirtualindex":
#not visible in 'show all' view of 'Manage Indexes'
fin_output += perform_deletevirtualindex(ln, callback='', **params)
return addadminbox("<b>Menu</b>", [fin_output])
def perform_field(ln=CFG_SITE_LANG, mtype='', content=''):
"""Start area for modifying fields
mtype - the method that called this method.
content - the output from that method."""
fin_output = """
<table>
<tr>
<td>0.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/field?ln=%s">Show all</a></small></td>
<td>1.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/field?ln=%s&amp;mtype=perform_showfieldoverview#1">Overview of logical fields</a></small></td>
<td>2.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/field?ln=%s&amp;mtype=perform_editfields#2">Edit logical field</a></small></td>
<td>3.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/field?ln=%s&amp;mtype=perform_addfield#3">Add new logical field</a></small></td>
<td>4.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/index?ln=%s">Manage Indexes</a></small></td>
<td>5.&nbsp;<small><a href="%s/help/admin/bibindex-admin-guide">Guide</a></small></td>
</tr>
</table>
""" % (CFG_SITE_URL, ln, CFG_SITE_URL, ln, CFG_SITE_URL, ln, CFG_SITE_URL, ln, CFG_SITE_URL, ln, CFG_SITE_URL)
if mtype == "perform_showfieldoverview" and content:
fin_output += content
elif mtype == "perform_showfieldoverview" or not mtype:
fin_output += perform_showfieldoverview(ln, callback='')
if mtype == "perform_editfields" and content:
fin_output += content
elif mtype == "perform_editfields" or not mtype:
fin_output += perform_editfields(ln, callback='')
if mtype == "perform_addfield" and content:
fin_output += content
elif mtype == "perform_addfield" or not mtype:
fin_output += perform_addfield(ln, callback='')
return addadminbox("<b>Menu</b>", [fin_output])
def perform_editfield(fldID, ln=CFG_SITE_LANG, mtype='', content='', callback='yes', confirm=-1):
"""form to modify a field. this method is calling other methods which again is calling this and sending back the output of the method.
if callback, the method will call perform_editcollection, if not, it will just return its output.
fldID - id of the field
mtype - the method that called this method.
content - the output from that method."""
fld_dict = dict(get_def_name('', "field"))
if fldID in [-1, "-1"]:
return addadminbox("Edit logical field", ["""<b><span class="info">Please go back and select a logical field</span></b>"""])
fin_output = """
<table>
<tr>
<td><b>Menu</b></td>
</tr>
<tr>
<td>0.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editfield?fldID=%s&amp;ln=%s">Show all</a></small></td>
<td>1.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editfield?fldID=%s&amp;ln=%s&amp;mtype=perform_modifyfield">Modify field code</a></small></td>
<td>2.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editfield?fldID=%s&amp;ln=%s&amp;mtype=perform_modifyfieldtranslations">Modify translations</a></small></td>
<td>3.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editfield?fldID=%s&amp;ln=%s&amp;mtype=perform_modifyfieldtags">Modify MARC tags</a></small></td>
<td>4.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editfield?fldID=%s&amp;ln=%s&amp;mtype=perform_deletefield">Delete field</a></small></td>
</tr><tr>
<td>5.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editfield?fldID=%s&amp;ln=%s&amp;mtype=perform_showdetailsfield">Show field usage</a></small></td>
</tr>
</table>
""" % (CFG_SITE_URL, fldID, ln, CFG_SITE_URL, fldID, ln, CFG_SITE_URL, fldID, ln, CFG_SITE_URL, fldID, ln, CFG_SITE_URL, fldID, ln, CFG_SITE_URL, fldID, ln)
if mtype == "perform_modifyfield" and content:
fin_output += content
elif mtype == "perform_modifyfield" or not mtype:
fin_output += perform_modifyfield(fldID, ln, callback='')
if mtype == "perform_modifyfieldtranslations" and content:
fin_output += content
elif mtype == "perform_modifyfieldtranslations" or not mtype:
fin_output += perform_modifyfieldtranslations(fldID, ln, callback='')
if mtype == "perform_modifyfieldtags" and content:
fin_output += content
elif mtype == "perform_modifyfieldtags" or not mtype:
fin_output += perform_modifyfieldtags(fldID, ln, callback='')
if mtype == "perform_deletefield" and content:
fin_output += content
elif mtype == "perform_deletefield" or not mtype:
fin_output += perform_deletefield(fldID, ln, callback='')
return addadminbox("Edit logical field '%s'" % fld_dict[int(fldID)], [fin_output])
def perform_editindex(idxID, ln=CFG_SITE_LANG, mtype='', content='', callback='yes', confirm=-1):
"""form to modify a index. this method is calling other methods which again is calling this and sending back the output of the method.
idxID - id of the index
mtype - the method that called this method.
content - the output from that method."""
if idxID in [-1, "-1"]:
return addadminbox("Edit index", ["""<b><span class="info">Please go back and select a index</span></b>"""])
fin_output = """
<table>
<tr>
<td><b>Menu</b></td>
</tr>
<tr>
<td>0.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s">Show all</a></small></td>
<td>1.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s&amp;mtype=perform_modifyindex">Modify index name / descriptor</a></small></td>
<td>2.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s&amp;mtype=perform_modifyindextranslations">Modify translations</a></small></td>
<td>3.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s&amp;mtype=perform_modifyindexfields">Modify index fields</a></small></td>
<td>4.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s&amp;mtype=perform_modifyindexstemming">Modify index stemming language</a></small></td>
<td>5.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s&amp;mtype=perform_modifysynonymkb">Modify synonym knowledge base</a></small></td>
<td>6.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s&amp;mtype=perform_modifystopwords">Modify remove stopwords</a></small></td>
<td>7.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s&amp;mtype=perform_modifyremovehtml">Modify remove HTML markup</a></small></td>
<td>8.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s&amp;mtype=perform_modifyremovelatex">Modify remove latex markup</a></small></td>
<td>9.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s&amp;mtype=perform_modifytokenizer">Modify tokenizer</a></small></td>
<td>10.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s&amp;mtype=perform_modifyindexer">Modify indexer</a></small></td>
<td>11.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s&amp;mtype=perform_deleteindex">Delete index</a></small></td>
</tr>
</table>
""" % (CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln)
if mtype == "perform_modifyindex" and content:
fin_output += content
elif mtype == "perform_modifyindex" or not mtype:
fin_output += perform_modifyindex(idxID, ln, callback='')
if mtype == "perform_modifyindextranslations" and content:
fin_output += content
elif mtype == "perform_modifyindextranslations" or not mtype:
fin_output += perform_modifyindextranslations(idxID, ln, callback='')
if mtype == "perform_modifyindexfields" and content:
fin_output += content
elif mtype == "perform_modifyindexfields" or not mtype:
fin_output += perform_modifyindexfields(idxID, ln, callback='')
if mtype == "perform_modifyindexstemming" and content:
fin_output += content
elif mtype == "perform_modifyindexstemming" or not mtype:
fin_output += perform_modifyindexstemming(idxID, ln, callback='')
if mtype == "perform_modifysynonymkb" and content:
fin_output += content
elif mtype == "perform_modifysynonymkb" or not mtype:
fin_output += perform_modifysynonymkb(idxID, ln, callback='')
if mtype == "perform_modifystopwords" and content:
fin_output += content
elif mtype == "perform_modifystopwords" or not mtype:
fin_output += perform_modifystopwords(idxID, ln, callback='')
if mtype == "perform_modifyremovehtml" and content:
fin_output += content
elif mtype == "perform_modifyremovehtml" or not mtype:
fin_output += perform_modifyremovehtml(idxID, ln, callback='')
if mtype == "perform_modifyremovelatex" and content:
fin_output += content
elif mtype == "perform_modifyremovelatex" or not mtype:
fin_output += perform_modifyremovelatex(idxID, ln, callback='')
if mtype == "perform_modifytokenizer" and content:
fin_output += content
elif mtype == "perform_modifytokenizer" or not mtype:
fin_output += perform_modifytokenizer(idxID, ln, callback='')
if mtype == "perform_modifyindexer" and content:
fin_output += content
elif mtype == "perform_modifyindexer" or not mtype:
fin_output += perform_modifyindexer(idxID, ln, callback='')
if mtype == "perform_deleteindex" and content:
fin_output += content
elif mtype == "perform_deleteindex" or not mtype:
fin_output += perform_deleteindex(idxID, ln, callback='')
return addadminbox("Edit index", [fin_output])
def perform_editvirtualindex(idxID, ln=CFG_SITE_LANG, mtype='', content='', callback='yes', confirm=-1):
if idxID in [-1, "-1"]:
return addadminbox("Edit virtual index", ["""<b><span class="info">Please go back and select an index</span></b>"""])
fin_output = """
<table>
<tr>
<td><b>Menu</b></td>
</tr>
<tr>
<td>0.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editvirtualindex?idxID=%s&amp;ln=%s">Show all</a></small></td>
<td>1.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/editvirtualindex?idxID=%s&amp;ln=%s&amp;mtype=perform_modifydependentindexes">Modify depedent indexes</a></small></td>
<td>2.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/index?ln=%s&amp;mtype=perform_showvirtualindexoverview#2">Overview of virtual indexes</a></small></td>
</tr>
</table>
""" % (CFG_SITE_URL, idxID, ln, CFG_SITE_URL, idxID, ln, CFG_SITE_URL, ln)
if mtype == "perform_modifydependentindexes" and content:
fin_output += content
elif mtype == "perform_modifydependentindexes" or not mtype:
fin_output += perform_modifydependentindexes(idxID, ln, callback='')
index_name = "( %s )" % get_index_name_from_index_id(idxID)
return addadminbox("Edit virtual index %s" % index_name, [fin_output])
def perform_showindexoverview(ln=CFG_SITE_LANG, callback='', confirm=0):
subtitle = """<a name="1"></a>1. Overview of all indexes"""
output = """<table cellpadding="3" border="1">"""
output += """<tr><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></tr>""" % ("ID", "Name", "Fwd.Idx Size", "Rev.Idx Size", "Fwd.Idx Words", "Rev.Idx Records", "Last updated", "Fields", "Translations", "Stemming Language", "Synonym knowledge base", "Remove stopwords", "Remove HTML markup", "Remove Latex markup", "Tokenizer", "Indexer type")
idx = get_idx()
idx_dict = dict(get_def_name('', "idxINDEX"))
stemming_language_map = get_stemming_language_map()
stemming_language_map_reversed = dict([(elem[1], elem[0]) for elem in iteritems(stemming_language_map)])
virtual_indexes = dict(get_all_virtual_indexes())
for idxID, idxNAME, idxDESC, idxUPD, idxSTEM, idxSYNKB, idxSTOPWORDS, idxHTML, idxLATEX, idxTOK in idx:
forward_table_status_info = get_table_status_info('idxWORD%sF' % (idxID < 10 and '0%s' % idxID or idxID))
reverse_table_status_info = get_table_status_info('idxWORD%sR' % (idxID < 10 and '0%s' % idxID or idxID))
if str(idxUPD)[-3:] == ".00":
idxUPD = str(idxUPD)[0:-3]
lang = get_lang_list("idxINDEXNAME", "id_idxINDEX", idxID)
idx_fld = get_idx_fld(idxID)
fld = ""
for row in idx_fld:
fld += row[3] + ", "
if fld.endswith(", "):
fld = fld[:-2]
if len(fld) == 0:
fld = """<strong><span class="info">None</span></strong>"""
date = (idxUPD and idxUPD or """<strong><span class="info">Not updated</span></strong>""")
stemming_lang = stemming_language_map_reversed.get(idxSTEM, None)
if not stemming_lang:
stemming_lang = """<strong><span class="info">None</span></strong>"""
synonym_kb = get_idx_synonym_kb(idxID)
if not synonym_kb:
synonym_kb = """<strong><span class="info">None</span></strong>"""
remove_stopwords = get_idx_remove_stopwords(idxID)
if not remove_stopwords:
remove_stopwords = """<strong><span class="info">None</span></strong>"""
remove_html_markup = get_idx_remove_html_markup(idxID)
if not remove_html_markup:
remove_html_markup = """<strong><span class="info">None</span></strong>"""
remove_latex_markup = get_idx_remove_latex_markup(idxID)
if not remove_latex_markup:
remove_latex_markup = """<strong><span class="info">None</span></strong>"""
tokenizer = get_idx_tokenizer(idxID)
if not remove_latex_markup:
tokenizer = """<strong><span class="info">None</span></strong>"""
type_of_indexer = virtual_indexes.get(idxID) and "virtual" or get_idx_indexer(idxNAME)
if forward_table_status_info and reverse_table_status_info:
output += """<tr><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td></tr>""" % \
(idxID,
"""<a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s" title="%s">%s</a>""" % (CFG_SITE_URL, idxID, ln, idxDESC, idx_dict.get(idxID, idxNAME)),
"%s MB" % websearch_templates.tmpl_nice_number(forward_table_status_info['Data_length'] / 1048576.0, max_ndigits_after_dot=3),
"%s MB" % websearch_templates.tmpl_nice_number(reverse_table_status_info['Data_length'] / 1048576.0, max_ndigits_after_dot=3),
websearch_templates.tmpl_nice_number(forward_table_status_info['Rows']),
websearch_templates.tmpl_nice_number(reverse_table_status_info['Rows'], max_ndigits_after_dot=3),
date,
fld,
lang,
stemming_lang,
synonym_kb,
remove_stopwords,
remove_html_markup,
remove_latex_markup,
tokenizer,
type_of_indexer)
elif not forward_table_status_info:
output += """<tr><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td></tr>""" % \
(idxID,
"""<a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s">%s</a>""" % (CFG_SITE_URL, idxID, ln, idx_dict.get(idxID, idxNAME)),
"Error", "%s MB" % websearch_templates.tmpl_nice_number(reverse_table_status_info['Data_length'] / 1048576.0, max_ndigits_after_dot=3),
"Error",
websearch_templates.tmpl_nice_number(reverse_table_status_info['Rows'], max_ndigits_after_dot=3),
date,
"",
lang,
synonym_kb,
remove_stopwords,
remove_html_markup,
remove_latex_markup,
tokenizer,
type_of_indexer)
elif not reverse_table_status_info:
output += """<tr><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td></tr>""" % \
(idxID,
"""<a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&amp;ln=%s">%s</a>""" % (CFG_SITE_URL, idxID, ln, idx_dict.get(idxID, idxNAME)),
"%s MB" % websearch_templates.tmpl_nice_number(forward_table_status_info['Data_length'] / 1048576.0, max_ndigits_after_dot=3),
"Error", websearch_templates.tmpl_nice_number(forward_table_status_info['Rows'], max_ndigits_after_dot=3),
"Error",
date,
"",
lang,
synonym_kb,
remove_stopwords,
remove_html_markup,
remove_latex_markup,
tokenizer,
type_of_indexer)
output += "</table>"
body = [output]
if callback:
return perform_index(ln, "perform_showindexoverview", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_showvirtualindexoverview(ln=CFG_SITE_LANG, callback='', confirm=0):
subtitle = """<a name="1"></a>2. Overview of virtual indexes"""
output = """
<table>
<tr>
<td>1.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/index?ln=%s&amp;mtype=perform_editvirtualindexes#1">Edit virtual index</a></small></td>
<td>2.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/index?ln=%s&amp;mtype=perform_addvirtualindex#2">Add new virtual index</a></small></td>
<td>3.&nbsp;<small><a href="%s/admin/bibindex/bibindexadmin.py/index?ln=%s&amp;mtype=perform_deletevirtualindex#3">Delete virtual index</a></small></td>
</tr>
</table>
""" % (CFG_SITE_URL, ln, CFG_SITE_URL, ln, CFG_SITE_URL, ln)
output += """<table cellpadding="3" border="1">"""
output += """<tr><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td></tr>""" % ("ID", "Virtual index", "Dependent indexes")
idx = get_all_virtual_indexes()
for idxID, idxNAME in idx:
normal_indexes = zip(*get_virtual_index_building_blocks(idxID))[1]
output += """<tr><td>%s</td><td>%s</td><td>%s</td></tr>""" % \
(idxID,
"""<a href="%s/admin/bibindex/bibindexadmin.py/editvirtualindex?idxID=%s&amp;ln=%s">%s</a>""" % (CFG_SITE_URL, idxID, ln, idxNAME),
", ".join(normal_indexes))
output += "</table>"
body = [output]
if callback:
return perform_index(ln, "perform_showvirtualindexoverview", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_editindexes(ln=CFG_SITE_LANG, callback='yes', content='', confirm=-1):
"""show a list of indexes that can be edited."""
subtitle = """<a name="3"></a>3. Edit index&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % (CFG_SITE_URL)
fin_output = ''
idx = get_idx()
output = ""
if len(idx) > 0:
text = """
<span class="adminlabel">Index name</span>
<select name="idxID" class="admin_w200">
<option value="-1">- Select a index -</option>
"""
for (idxID, idxNAME, idxDESC, idxUPD, idxSTEM, idxSYNKB, idxSTOPWORDS, idxHTML, idxLATEX, idxTOK) in idx:
text += """<option value="%s">%s</option>""" % (idxID, idxNAME)
text += """</select>"""
output += createhiddenform(action="%s/admin/bibindex/bibindexadmin.py/editindex" % CFG_SITE_URL,
text=text,
button="Edit",
ln=ln,
confirm=1)
else:
output += """No indexes exists"""
body = [output]
if callback:
return perform_index(ln, "perform_editindexes", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_editvirtualindexes(ln=CFG_SITE_LANG, callback='yes', content='', confirm=-1):
"""show a list of indexes that can be edited."""
subtitle = """<a name="2"></a>1. Edit virtual index&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % (CFG_SITE_URL)
idx = get_all_virtual_indexes()
output = ""
if len(idx) > 0:
text = """
<span class="adminlabel">Virtual index name</span>
<select name="idxID" class="admin_w200">
<option value="-1">- Select a index -</option>
"""
for (idxID, idxNAME) in idx:
text += """<option value="%s">%s</option>""" % (idxID, idxNAME)
text += """</select>"""
output += createhiddenform(action="%s/admin/bibindex/bibindexadmin.py/editvirtualindex" % CFG_SITE_URL,
text=text,
button="Edit",
ln=ln,
confirm=1)
else:
output += """No indexes exist"""
body = [output]
if callback:
return perform_index(ln, "perform_editvirtualindexes", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_editfields(ln=CFG_SITE_LANG, callback='yes', content='', confirm=-1):
"""show a list of all logical fields that can be edited."""
subtitle = """<a name="4"></a>4. Edit logical field&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % (CFG_SITE_URL)
fin_output = ''
res = get_fld()
output = ""
if len(res) > 0:
text = """
<span class="adminlabel">Field name</span>
<select name="fldID" class="admin_w200">
<option value="-1">- Select a field -</option>
"""
for (fldID, name, code) in res:
text += """<option value="%s">%s</option>""" % (fldID, name)
text += """</select>"""
output += createhiddenform(action="%s/admin/bibindex/bibindexadmin.py/editfield" % CFG_SITE_URL,
text=text,
button="Edit",
ln=ln,
confirm=1)
else:
output += """No logical fields exists"""
body = [output]
if callback:
return perform_field(ln, "perform_editfields", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_addindex(ln=CFG_SITE_LANG, idxNAME='', callback="yes", confirm=-1):
"""form to add a new index.
idxNAME - the name of the new index"""
output = ""
subtitle = """<a name="3"></a>3. Add new index"""
text = """
<span class="adminlabel">Index name</span>
<input class="admin_w200" type="text" name="idxNAME" value="%s" /><br />
""" % idxNAME
output = createhiddenform(action="%s/admin/bibindex/bibindexadmin.py/addindex" % CFG_SITE_URL,
text=text,
ln=ln,
button="Add index",
confirm=1)
if idxNAME and confirm in ["1", 1]:
res = add_idx(idxNAME)
output += write_outcome(res) + """<br /><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&ln=%s">Configure this index</a>.""" % (CFG_SITE_URL, res[1], ln)
elif confirm not in ["-1", -1]:
output += """<b><span class="info">Please give the index a name.</span></b>
"""
body = [output]
if callback:
return perform_index(ln, "perform_addindex", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_addvirtualindex(ln=CFG_SITE_LANG, idxNEWVID='', idxNEWPID='', callback="yes", confirm=-1):
"""form to add a new virtual index from the set of physical indexes.
idxID - the name of the new virtual index"""
idx = get_all_indexes(virtual=False, with_ids=True)
output = ""
subtitle = """<a name="3"></a>2. Add new virtual index"""
if len(idx) > 0:
text = """
<span class="adminlabel">Choose new virtual index</span>
<select name="idxNEWVID" class="admin_w200">
<option value="-1">- Select an index -</option>
"""
for (idxID, idxNAME) in idx:
checked = str(idxNEWVID) == str(idxID) and 'selected="selected"' or ''
text += """<option value="%s" %s>%s</option>
""" % (idxID, checked, idxNAME)
text += """</select>"""
text += """&nbsp;&nbsp;
<span class="adminlabel">Add physical index</span>
<select name="idxNEWPID" class="admin_w200">
<option value="-1">- Select an index -</option>
"""
for (idxID, idxNAME) in idx:
text += """<option value="%s">%s</option>""" % (idxID, idxNAME)
text += """</select>"""
output += createhiddenform(action="%s/admin/bibindex/bibindexadmin.py/addvirtualindex" % CFG_SITE_URL,
text=text,
button="Add index",
ln=ln,
confirm=1)
else:
output += """No index exists"""
if idxNEWVID not in ['', "-1", -1] and idxNEWPID not in ['', "-1", -1] and confirm in ["1", 1]:
res = add_virtual_idx(idxNEWVID, idxNEWPID)
output += write_outcome(res)
output += """<br /><span class="info">Please note you must run as soon as possible:
<pre>$> %s/bibindex --reindex -w %s</pre></span>""" % (CFG_BINDIR, dict(idx)[int(idxNEWPID)])
elif confirm not in ["-1", -1] or idxNEWVID in ["-1", -1] or idxNEWPID in ["-1", -1]:
output += """<b><span class="info">Please specify the index.</span></b>"""
body = [output]
if callback:
return perform_index(ln, "perform_addvirtualindex", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifyindextranslations(idxID, ln=CFG_SITE_LANG, sel_type='', trans=[], confirm=-1, callback='yes'):
"""Modify the translations of a index
sel_type - the nametype to modify
trans - the translations in the same order as the languages from get_languages()"""
output = ''
subtitle = ''
langs = get_languages()
if confirm in ["2", 2] and idxID:
finresult = modify_translations(idxID, langs, sel_type, trans, "idxINDEX")
idx_dict = dict(get_def_name('', "idxINDEX"))
if idxID and int(idxID) in idx_dict:
idxID = int(idxID)
subtitle = """<a name="2"></a>2. Modify translations for index.&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % CFG_SITE_URL
if type(trans) is str:
trans = [trans]
if sel_type == '':
sel_type = get_idx_nametypes()[0][0]
header = ['Language', 'Translation']
actions = []
types = get_idx_nametypes()
if len(types) > 1:
text = """
<span class="adminlabel">Name type</span>
<select name="sel_type" class="admin_w200">
"""
for (key, value) in types:
text += """<option value="%s" %s>%s""" % (key, key == sel_type and 'selected="selected"' or '', value)
trans_names = get_name(idxID, ln, key, "field")
if trans_names and trans_names[0][0]:
text += ": %s" % trans_names[0][0]
text += "</option>"
text += """</select>"""
output += createhiddenform(action="modifyindextranslations#2",
text=text,
button="Select",
idxID=idxID,
ln=ln,
confirm=0)
if confirm in [-1, "-1", 0, "0"]:
trans = []
for (key, value) in langs:
try:
trans_names = get_name(idxID, key, sel_type, "idxINDEX")
trans.append(trans_names[0][0])
except StandardError as e:
trans.append('')
for nr in range(0,len(langs)):
actions.append(["%s" % (langs[nr][1],)])
actions[-1].append('<input type="text" name="trans" size="30" value="%s"/>' % trans[nr])
text = tupletotable(header=header, tuple=actions)
output += createhiddenform(action="modifyindextranslations#2",
text=text,
button="Modify",
idxID=idxID,
sel_type=sel_type,
ln=ln,
confirm=2)
if sel_type and len(trans):
if confirm in ["2", 2]:
output += write_outcome(finresult)
body = [output]
if callback:
return perform_editindex(idxID, ln, "perform_modifyindextranslations", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifyfieldtranslations(fldID, ln=CFG_SITE_LANG, sel_type='', trans=[], confirm=-1, callback='yes'):
"""Modify the translations of a field
sel_type - the nametype to modify
trans - the translations in the same order as the languages from get_languages()"""
output = ''
subtitle = ''
langs = get_languages()
if confirm in ["2", 2] and fldID:
finresult = modify_translations(fldID, langs, sel_type, trans, "field")
fld_dict = dict(get_def_name('', "field"))
if fldID and int(fldID) in fld_dict:
fldID = int(fldID)
subtitle = """<a name="3"></a>3. Modify translations for logical field '%s'&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % (fld_dict[fldID], CFG_SITE_URL)
if type(trans) is str:
trans = [trans]
if sel_type == '':
sel_type = get_fld_nametypes()[0][0]
header = ['Language', 'Translation']
actions = []
types = get_fld_nametypes()
if len(types) > 1:
text = """
<span class="adminlabel">Name type</span>
<select name="sel_type" class="admin_w200">
"""
for (key, value) in types:
text += """<option value="%s" %s>%s""" % (key, key == sel_type and 'selected="selected"' or '', value)
trans_names = get_name(fldID, ln, key, "field")
if trans_names and trans_names[0][0]:
text += ": %s" % trans_names[0][0]
text += "</option>"
text += """</select>"""
output += createhiddenform(action="modifyfieldtranslations#3",
text=text,
button="Select",
fldID=fldID,
ln=ln,
confirm=0)
if confirm in [-1, "-1", 0, "0"]:
trans = []
for (key, value) in langs:
try:
trans_names = get_name(fldID, key, sel_type, "field")
trans.append(trans_names[0][0])
except StandardError as e:
trans.append('')
for nr in range(0,len(langs)):
actions.append(["%s" % (langs[nr][1],)])
actions[-1].append('<input type="text" name="trans" size="30" value="%s"/>' % trans[nr])
text = tupletotable(header=header, tuple=actions)
output += createhiddenform(action="modifyfieldtranslations#3",
text=text,
button="Modify",
fldID=fldID,
sel_type=sel_type,
ln=ln,
confirm=2)
if sel_type and len(trans):
if confirm in ["2", 2]:
output += write_outcome(finresult)
body = [output]
if callback:
return perform_editfield(fldID, ln, "perform_modifytranslations", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_showdetailsfieldtag(fldID, tagID, ln=CFG_SITE_LANG, callback="yes", confirm=-1):
"""form to add a new field.
fldNAME - the name of the new field
code - the field code"""
fld_dict = dict(get_def_name('', "field"))
fldID = int(fldID)
tagname = run_sql("SELECT name from tag where id=%s", (tagID, ))[0][0]
output = ""
subtitle = """<a name="4.1"></a>Showing details for MARC tag '%s'""" % tagname
output += "<br /><b>This MARC tag is used directly in these logical fields:</b>&nbsp;"
fld_tag = get_fld_tags('', tagID)
exist = {}
for (id_field,id_tag, tname, tvalue, score) in fld_tag:
output += "%s, " % fld_dict[int(id_field)]
exist[id_field] = 1
output += "<br /><b>This MARC tag is used indirectly in these logical fields:</b>&nbsp;"
tag = run_sql("SELECT value from tag where id=%s", (id_tag, ))
tag = tag[0][0]
for i in range(0, len(tag) - 1):
res = run_sql("SELECT id_field,id_tag FROM field_tag,tag WHERE tag.id=field_tag.id_tag AND tag.value=%s", ('%' + tag[0:i] + '%',))
for (id_field, id_tag) in res:
output += "%s, " % fld_dict[int(id_field)]
exist[id_field] = 1
res = run_sql("SELECT id_field,id_tag FROM field_tag,tag WHERE tag.id=field_tag.id_tag AND tag.value like %s", (tag, ))
for (id_field, id_tag) in res:
if id_field not in exist:
output += "%s, " % fld_dict[int(id_field)]
body = [output]
if callback:
return perform_modifyfieldtags(fldID, ln, "perform_showdetailsfieldtag", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_showdetailsfield(fldID, ln=CFG_SITE_LANG, callback="yes", confirm=-1):
"""form to add a new field.
fldNAME - the name of the new field
code - the field code"""
fld_dict = dict(get_def_name('', "field"))
col_dict = dict(get_def_name('', "collection"))
fldID = int(fldID)
col_fld = get_col_fld('', '', fldID)
sort_types = dict(get_sort_nametypes())
fin_output = ""
subtitle = """<a name="1"></a>5. Show usage for logical field '%s'""" % fld_dict[fldID]
output = "This logical field is used in these collections:<br />"
ltype = ''
exist = {}
for (id_collection, id_field, id_fieldvalue, ftype, score, score_fieldvalue) in col_fld:
if ltype != ftype:
output += "<br /><b>%s:&nbsp;</b>" % sort_types[ftype]
ltype = ftype
exist = {}
if id_collection not in exist:
output += "%s, " % col_dict[int(id_collection)]
exist[id_collection] = 1
if not col_fld:
output = "This field is not used by any collections."
fin_output = addadminbox('Collections', [output])
body = [fin_output]
if callback:
return perform_editfield(ln, "perform_showdetailsfield", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_addfield(ln=CFG_SITE_LANG, fldNAME='', code='', callback="yes", confirm=-1):
"""form to add a new field.
fldNAME - the name of the new field
code - the field code"""
output = ""
subtitle = """<a name="3"></a>3. Add new logical field"""
code = str.replace(code,' ', '')
text = """
<span class="adminlabel">Field name</span>
<input class="admin_w200" type="text" name="fldNAME" value="%s" /><br />
<span class="adminlabel">Field code</span>
<input class="admin_w200" type="text" name="code" value="%s" /><br />
""" % (fldNAME, code)
output = createhiddenform(action="%s/admin/bibindex/bibindexadmin.py/addfield" % CFG_SITE_URL,
text=text,
ln=ln,
button="Add field",
confirm=1)
if fldNAME and code and confirm in ["1", 1]:
res = add_fld(fldNAME, code)
output += write_outcome(res)
elif confirm not in ["-1", -1]:
output += """<b><span class="info">Please give the logical field a name and code.</span></b>
"""
body = [output]
if callback:
return perform_field(ln, "perform_addfield", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_deletefield(fldID, ln=CFG_SITE_LANG, callback='yes', confirm=0):
"""form to remove a field.
fldID - the field id from table field.
"""
fld_dict = dict(get_def_name('', "field"))
if int(fldID) not in fld_dict:
return """<b><span class="info">Field does not exist</span></b>"""
subtitle = """<a name="4"></a>4. Delete the logical field '%s'&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % (fld_dict[int(fldID)], CFG_SITE_URL)
output = ""
if fldID:
fldID = int(fldID)
if confirm in ["0", 0]:
check = run_sql("SELECT id_field from idxINDEX_field where id_field=%s", (fldID, ))
text = ""
if check:
text += """<b><span class="info">This field is used in an index, deletion may cause problems.</span></b><br />"""
text += """Do you want to delete the logical field '%s' and all its relations and definitions.""" % (fld_dict[fldID])
output += createhiddenform(action="deletefield#4",
text=text,
button="Confirm",
fldID=fldID,
confirm=1)
elif confirm in ["1", 1]:
res = delete_fld(fldID)
if res[0] == 1:
return """<br /><b><span class="info">Field deleted.</span></b>""" + write_outcome(res)
else:
output += write_outcome(res)
body = [output]
if callback:
return perform_editfield(fldID, ln, "perform_deletefield", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_deleteindex(idxID, ln=CFG_SITE_LANG, callback='yes', confirm=0):
"""form to delete an index.
idxID - the index id from table idxINDEX.
"""
if idxID:
subtitle = """<a name="5"></a>11. Delete the index.&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % CFG_SITE_URL
output = ""
if confirm in ["0", 0]:
idx = get_idx(idxID)
if idx:
text = ""
text += """<b><span class="info">By deleting an index, you may also loose any indexed data in the forward and reverse table for this index.</span></b><br />"""
text += """Do you want to delete the index '%s' and all its relations and definitions.""" % (idx[0][1])
output += createhiddenform(action="deleteindex#5",
text=text,
button="Confirm",
idxID=idxID,
confirm=1)
else:
return """<br /><b><span class="info">Index specified does not exist.</span></b>"""
elif confirm in ["1", 1]:
res = delete_idx(idxID)
if res[0] == 1:
return """<br /><b><span class="info">Index deleted.</span></b>""" + write_outcome(res)
else:
output += write_outcome(res)
body = [output]
if callback:
return perform_editindex(idxID, ln, "perform_deleteindex", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_deletevirtualindex(ln=CFG_SITE_LANG, idxID='', callback='yes', confirm=-1):
"""form to delete a virtual index.
idxID - the index id from table idxINDEX.
"""
output = ""
subtitle = """<a name="3"></a>3. Delete virtual index"""
idx = get_all_virtual_indexes()
if len(idx) > 0:
text = """<span class="adminlabel">Choose a virtual index</span>
<select name="idxID" class="admin_w200">
<option value="-1">- Select an index -</option>
"""
for idx_id, idx_name in idx:
selected = str(idxID) == str(idx_id) and 'selected="selected"' or ''
text += """<option value="%s" %s>%s</option>""" % (idx_id, selected, idx_name)
text += """</select>"""
output += createhiddenform(action="deletevirtualindex#3",
text=text,
button="Confirm",
confirm=1)
else:
output = "No index specified"
if confirm in ["1", 1] and idxID not in ['', "-1", -1]:
res = delete_virtual_idx(int(idxID))
if res[0] == 1:
output += """<br /><b><span class="info">Virtual index deleted.</span></b><br />"""
output += write_outcome(res)
else:
output += write_outcome(res)
elif idxID in ["-1", -1]:
output += """<b><span class="info">Please specify the index.</span></b>"""
body = [output]
if callback:
return perform_index(ln, "perform_deletevirtualindex", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifydependentindexes(idxID, ln=CFG_SITE_LANG, newIDs=[], callback='yes', confirm=-1):
"""page on which dependent indexes for specific virtual index
can be chosen"""
subtitle = ""
output = ""
non_virtual_indexes = dict(get_all_indexes(virtual=False, with_ids=True)) #[(id1, name1), (id2, name2)..]
already_dependent = dict(get_virtual_index_building_blocks(idxID))
if not already_dependent:
idxID = -1
if idxID not in [-1, "-1"]:
subtitle = """<a name="1"></a>1. Modify dependent indexes.&nbsp;&nbsp;&nbsp;
<small>
[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]
</small>""" % CFG_SITE_URL
if confirm in [-1, "-1"]:
newIDs = []
if not newIDs:
newIDs = []
tick_list = ""
checked_values = already_dependent.values()
if confirm > -1:
checked_values = newIDs
for index_name in non_virtual_indexes.values():
checked = index_name in checked_values and 'checked="checked"' or ''
tick_list += """<input type="checkbox" name='newIDs' value="%s" %s >%s </br>""" % \
(index_name, checked, index_name)
output += createhiddenform(action="modifydependentindexes#1",
text=tick_list,
button="Modify",
idxID=idxID,
ln=ln,
confirm=0)
if confirm in [0, "0"] and newIDs == []:
output += "</br>"
text = """
<span class="important">Removing all dependent indexes
means removing virtual index.</span>
<br /> <strong>Are you sure you want to do this?</strong>"""
output += createhiddenform(action="modifydependentindexes#1",
text=text,
button="Confirm",
idxID=idxID,
newIDs=newIDs,
ln=ln,
confirm=1)
elif confirm in [0, "0"]:
output += "</br>"
text = """
<span class="important">You are about to change dependent indexes</span>.<br /> <strong>Are you sure you want to do this?</strong>"""
output += createhiddenform(action="modifydependentindexes#1",
text=text,
button="Confirm",
idxID=idxID,
newIDs=newIDs,
ln=ln,
confirm=1)
elif idxID > -1 and confirm in [1, "1"]:
output += "</br>"
to_add, to_remove = find_dependent_indexes_to_change(idxID, newIDs)
- res = modify_dependent_indexes(idxID, to_add, to_remove)
+ # NOTE: we don't need to take care of indexes to remove, because
+ # -w <<virutal_index>> --remove-dependent-index will take care of everything
+ # so it's enough to just post a message
+ res = modify_dependent_indexes(idxID, to_add)
output += write_outcome(res)
if len(to_remove) + len(to_add) > 0:
output += """<br /><span class="info">Please note you should run as soon as possible:"""
- for index in to_add:
+ if len(to_add) > 0:
output += """<pre>$> %s/bibindex --reindex -w %s</pre>
- """ % (CFG_BINDIR, index)
+ """ % (CFG_BINDIR, get_index_name_from_index_id(idxID))
for index in to_remove:
output += """<pre>$> %s/bibindex -w %s --remove-dependent-index %s</pre>
""" % (CFG_BINDIR, get_index_name_from_index_id(idxID), index)
if len(to_remove) + len(to_add) > 0:
output += "</span>"
elif confirm in [1, "1"]:
output += """<br /><b><span class="info">Please give a name for the index.</span></b>"""
else:
output = """It seems that this index is not virtual."""
body = [output]
if callback:
return perform_editvirtualindex(idxID, ln, "perform_modifydependentindexes", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def find_dependent_indexes_to_change(idxID, new_indexes):
"""From new set of dependent indexes finds out
which indexes should be added and which should be removed
from database (idxINDEX_idxINDEX table)
@param idxID: id of the virtual index
@param new_indexes: future set of dependent indexes
"""
if not type(new_indexes) is list:
new_indexes = [new_indexes]
dependent_indexes = dict(get_virtual_index_building_blocks(idxID)).values()
to_add = set(new_indexes) - set(dependent_indexes)
to_remove = set(dependent_indexes) - set(new_indexes)
return list(to_add), list(to_remove)
def perform_showfieldoverview(ln=CFG_SITE_LANG, callback='', confirm=0):
subtitle = """<a name="1"></a>1. Logical fields overview"""
output = """<table cellpadding="3" border="1">"""
output += """<tr><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td></tr>""" % ("Field", "MARC Tags", "Translations")
query = "SELECT id,name FROM field"
res = run_sql(query)
col_dict = dict(get_def_name('', "collection"))
fld_dict = dict(get_def_name('', "field"))
for field_id,field_name in res:
query = "SELECT tag.value FROM tag, field_tag WHERE tag.id=field_tag.id_tag AND field_tag.id_field=%s ORDER BY field_tag.score DESC,tag.value ASC"
res = run_sql(query, (field_id, ))
field_tags = ""
for row in res:
field_tags = field_tags + row[0] + ", "
if field_tags.endswith(", "):
field_tags = field_tags[:-2]
if not field_tags:
field_tags = """<b><span class="info">None</span></b>"""
lang = get_lang_list("fieldname", "id_field", field_id)
output += """<tr><td>%s</td><td>%s</td><td>%s</td></tr>""" % ("""<a href="%s/admin/bibindex/bibindexadmin.py/editfield?fldID=%s&ln=%s">%s</a>""" % (CFG_SITE_URL, field_id, ln, fld_dict[field_id]), field_tags, lang)
output += "</table>"
body = [output]
if callback:
return perform_field(ln, "perform_showfieldoverview", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifyindex(idxID, ln=CFG_SITE_LANG, idxNAME='', idxDESC='', callback='yes', confirm=-1):
"""form to modify an index name.
idxID - the index name to change.
idxNAME - new name of index
idxDESC - description of index content"""
subtitle = ""
output = ""
idx = get_idx(idxID)
if not idx:
idxID = -1
if idxID not in [-1, "-1"]:
subtitle = """<a name="2"></a>1. Modify index name.&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % CFG_SITE_URL
if confirm in [-1, "-1"]:
idxNAME = idx[0][1]
idxDESC = idx[0][2]
text = """
<span class="adminlabel">Index name</span>
<input class="admin_w200" type="text" name="idxNAME" value="%s" /><br />
<span class="adminlabel">Index description</span>
<textarea class="admin_w200" name="idxDESC">%s</textarea><br />
""" % (idxNAME, idxDESC)
output += createhiddenform(action="modifyindex#1",
text=text,
button="Modify",
idxID=idxID,
ln=ln,
confirm=1)
if idxID > -1 and idxNAME and confirm in [1, "1"]:
res = modify_idx(idxID, idxNAME, idxDESC)
output += write_outcome(res)
elif confirm in [1, "1"]:
output += """<br /><b><span class="info">Please give a name for the index.</span></b>"""
else:
output = """No index to modify."""
body = [output]
if callback:
return perform_editindex(idxID, ln, "perform_modifyindex", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifyindexstemming(idxID, ln=CFG_SITE_LANG, idxSTEM='', callback='yes', confirm=-1):
"""form to modify an index name.
idxID - the index name to change.
idxSTEM - new stemming language code"""
subtitle = ""
output = ""
stemming_language_map = get_stemming_language_map()
stemming_language_map['None'] = ''
idx = get_idx(idxID)
if not idx:
idxID = -1
if idxID not in [-1, "-1"]:
subtitle = """<a name="4"></a>4. Modify index stemming language.&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % CFG_SITE_URL
if confirm in [-1, "-1"]:
idxSTEM = idx[0][4]
if not idxSTEM:
idxSTEM = ''
language_html_element = """<select name="idxSTEM" class="admin_w200">"""
languages = stemming_language_map.keys()
languages.sort()
for language in languages:
if stemming_language_map[language] == idxSTEM:
selected = 'selected="selected"'
else:
selected = ""
language_html_element += """<option value="%s" %s>%s</option>""" % (stemming_language_map[language], selected, language)
language_html_element += """</select>"""
text = """
<span class="adminlabel">Index stemming language</span>
""" + language_html_element
output += createhiddenform(action="modifyindexstemming#4",
text=text,
button="Modify",
idxID=idxID,
ln=ln,
confirm=0)
if confirm in [0, "0"] and get_idx(idxID)[0][4] == idxSTEM:
output += """<span class="info">Stemming language has not been changed</span>"""
elif confirm in [0, "0"]:
text = """
<span class="important">You are about to either disable or change the stemming language setting for this index. Please note that it is not recommended to enable stemming for structured-data indexes like "report number", "year", "author" or "collection". On the contrary, it is advisable to enable stemming for indexes like "fulltext", "abstract", "title", etc. since this would overall improve the retrieval quality. <br /> Beware, however, that after disabling or changing the stemming language setting of an index you will have to reindex it. It is a good idea to change the stemming language and to reindex during low usage hours of your service, since searching results will be potentially affected by the discrepancy between search terms now being (not) stemmed and indexes still using the previous settings until the reindexing is completed</span>.<br /> <strong>Are you sure you want to disable/change the stemming language setting of this index?</strong>"""
output += createhiddenform(action="modifyindexstemming#4",
text=text,
button="Modify",
idxID=idxID,
idxSTEM=idxSTEM,
ln=ln,
confirm=1)
elif idxID > -1 and confirm in [1, "1"]:
res = modify_idx_stemming(idxID, idxSTEM)
output += write_outcome(res)
output += """<br /><span class="info">Please note you must run as soon as possible:
<pre>$> %s/bibindex --reindex -w %s</pre></span>
""" % (CFG_BINDIR, get_idx(idxID)[0][1])
elif confirm in [1, "1"]:
output += """<br /><b><span class="info">Please give a name for the index.</span></b>"""
else:
output = """No index to modify."""
body = [output]
if callback:
return perform_editindex(idxID, ln, "perform_modifyindexstemming", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifyindexer(idxID, ln=CFG_SITE_LANG, indexer='', callback='yes', confirm=-1):
"""form to modify an indexer.
idxID - the index name to change.
idexer - indexer type: native/SOLR/XAPIAN/virtual"""
subtitle = ""
output = ""
idx = get_idx(idxID)
if idx:
current_indexer = is_index_virtual(idx[0][0]) and "virtual" or get_idx_indexer(idx[0][1])
subtitle = """<a name="4"></a>5. Modify indexer.&nbsp;&nbsp;&nbsp;
<small>
[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]
</small>""" % CFG_SITE_URL
if confirm in [-1, "-1"]:
indexer = current_indexer or ''
items = ["native"]
if idx[0][1] == "fulltext":
items.extend(["SOLR", "XAPIAN"])
else:
items.extend(["virtual"])
html_element = """<select name="indexer" class="admin_w200">"""
for item in items:
selected = indexer==item and 'selected="selected"' or ''
html_element += """<option value="%s" %s>%s</option>""" % (item, selected, item)
html_element += """</select>"""
text = """<span class="adminlabel">Indexer type</span>""" + html_element
output += createhiddenform(action="modifyindexer#5",
text=text,
button="Modify",
idxID=idxID,
ln=ln,
confirm=1)
if confirm in [1, "1"] and idx[0][1]=="fulltext":
res = modify_idx_indexer(idxID, indexer)
output += write_outcome(res)
output += """<br /><span class="info">Please note you should run:
<pre>$> %s/bibindex --reindex -w fulltext</pre></span>""" % CFG_BINDIR
elif confirm in [1, "1"]:
if indexer=="virtual" and current_indexer == "native":
params = {'idxNEWVID': idxID}
return perform_index(ln, "perform_addvirtualindex", "", **params)
elif indexer=="native" and current_indexer == "virtual":
params = {'idxID':idxID}
return perform_index(ln, "perform_deletevirtualindex", "", **params)
else:
output = """No index to modify."""
body = [output]
if callback:
return perform_editindex(idxID, ln, "perform_modifyindexer", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifysynonymkb(idxID, ln=CFG_SITE_LANG, idxKB='', idxMATCH='', callback='yes', confirm=-1):
"""form to modify the knowledge base for the synonym lookup.
idxID - the index name to change.
idxKB - new knowledge base name
idxMATCH - new match type
"""
subtitle = ""
output = ""
idx = get_idx(idxID)
if not idx:
idxID = -1
if idxID not in [-1, "-1"]:
subtitle = """<a name="4"></a>5. Modify knowledge base for synonym lookup.&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % CFG_SITE_URL
if confirm in [-1, "-1"]:
field_value = get_idx_synonym_kb(idxID)
if CFG_BIBINDEX_COLUMN_VALUE_SEPARATOR in field_value:
idxKB, idxMATCH = field_value.split(CFG_BIBINDEX_COLUMN_VALUE_SEPARATOR)
if not idxKB:
idxKB = ''
idxMATCH = ''
kb_html_element = """<select name="idxKB" class="admin_w200">"""
knowledge_base_names = get_all_kb_names()
knowledge_base_names.append(CFG_BIBINDEX_SYNONYM_MATCH_TYPE["None"])
knowledge_base_names.sort()
for knowledge_base_name in knowledge_base_names:
if knowledge_base_name == idxKB:
selected = 'selected="selected"'
else:
selected = ""
kb_html_element += """<option value="%s" %s>%s</option>""" % (knowledge_base_name, selected, knowledge_base_name)
kb_html_element += """</select>"""
match_html_element = """<select name="idxMATCH" class="admin_w200">"""
match_names = CFG_BIBINDEX_SYNONYM_MATCH_TYPE.values()
match_names.sort()
for match_name in match_names:
if match_name == idxMATCH:
selected = 'selected="selected"'
else:
selected = ""
match_html_element += """<option value="%s" %s>%s</option>""" % (match_name, selected, match_name)
match_html_element += """</select>"""
text = """<span class="adminlabel">Knowledge base name and match type</span>""" + kb_html_element + match_html_element
output += createhiddenform(action="modifysynonymkb#4",
text=text,
button="Modify",
idxID=idxID,
ln=ln,
confirm=0)
if confirm in [0, "0"] and get_idx(idxID)[0][5] == idxKB + CFG_BIBINDEX_COLUMN_VALUE_SEPARATOR + idxMATCH:
output += """<span class="info">Knowledge base has not been changed</span>"""
elif confirm in [0, "0"]:
text = """
<span class="important">You are going to change the knowledge base for this index.<br /> <strong>Are you sure you want
to change the knowledge base of this index?</strong>"""
output += createhiddenform(action="modifysynonymkb#4",
text=text,
button="Modify",
idxID=idxID,
idxKB=idxKB,
idxMATCH=idxMATCH,
ln=ln,
confirm=1)
elif idxID > -1 and confirm in [1, "1"]:
res = modify_idx_synonym_kb(idxID, idxKB, idxMATCH)
output += write_outcome(res)
output += """<br /><span class="info">Please note that you must run as soon as possible:
<pre>$> %s/bibindex --reindex -w %s</pre></span>""" % (CFG_BINDIR, get_idx(idxID)[0][1])
elif confirm in [1, "1"]:
output += """<br /><b><span class="info">Please give a name for the index.</span></b>"""
else:
output = """No index to modify."""
body = [output]
if callback:
return perform_editindex(idxID, ln, "perform_modifysynonymkb", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifystopwords(idxID, ln=CFG_SITE_LANG, idxSTOPWORDS='', callback='yes', confirm=-1):
"""Form to modify the stopwords configuration
@param idxID: id of the index on which modification will be performed.
@param idxSTOPWORDS: remove stopwords or not ('Yes' or 'No')
"""
subtitle = ""
output = ""
idx = get_idx(idxID)
if not idx:
idxID = -1
if idxID not in [-1, "-1"]:
subtitle = """<a name="4"></a>6. Modify remove stopwords.&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % CFG_SITE_URL
if confirm in [-1, "-1"]:
idxSTOPWORDS = get_idx_remove_stopwords(idxID)
if not idxSTOPWORDS:
idxSTOPWORDS = ''
if isinstance(idxSTOPWORDS, tuple):
idxSTOPWORDS = ''
stopwords_html_element = """<input class="admin_w200" type="text" name="idxSTOPWORDS" value="%s" /><br />""" % idxSTOPWORDS
text = """<span class="adminlabel">Remove stopwords</span><br />""" + stopwords_html_element
output += createhiddenform(action="modifystopwords#4",
text=text,
button="Modify",
idxID=idxID,
ln=ln,
confirm=0)
if confirm in [0, "0"] and get_idx(idxID)[0][6] == idxSTOPWORDS:
output += """<span class="info">Stopwords have not been changed</span>"""
elif confirm in [0, "0"] and idxSTOPWORDS == '':
output += """<span class="info">You need to provide a name of the file with stopwords</span>"""
elif confirm in [0, "0"]:
text = """<span class="important">You are going to change the stopwords configuration for this index.<br />
<strong>Are you sure you want to do this?</strong>"""
output += createhiddenform(action="modifystopwords#4",
text=text,
button="Modify",
idxID=idxID,
idxSTOPWORDS=idxSTOPWORDS,
ln=ln,
confirm=1)
elif idxID > -1 and confirm in [1, "1"]:
res = modify_idx_stopwords(idxID, idxSTOPWORDS)
output += write_outcome(res)
output += """<br /><span class="info">Please note you must run as soon as possible:
<pre>$> %s/bibindex --reindex -w %s</pre></span>""" % (CFG_BINDIR, get_idx(idxID)[0][1])
elif confirm in [1, "1"]:
output += """<br /><b><span class="info">Please give a name for the index.</span></b>"""
else:
output = """No index to modify."""
body = [output]
if callback:
return perform_editindex(idxID, ln, "perform_modifystopwords", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifyremovehtml(idxID, ln=CFG_SITE_LANG, idxHTML='', callback='yes', confirm=-1):
"""Form to modify the 'remove html' configuration.
@param idxID: id of the index on which modification will be performed.
@param idxHTML: remove html markup or not ('Yes' or 'No')"""
subtitle = ""
output = ""
idx = get_idx(idxID)
if not idx:
idxID = -1
if idxID not in [-1, "-1"]:
subtitle = """<a name="4"></a>7. Modify remove HTML markup.&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % CFG_SITE_URL
if confirm in [-1, "-1"]:
idxHTML = get_idx_remove_html_markup(idxID)
if not idxHTML:
idxHTML = ''
remove_html_element = """<select name="idxHTML" class="admin_w200">"""
if idxHTML == 'Yes':
remove_html_element += """<option value="Yes" selected ="selected">Yes</option>"""
remove_html_element += """<option value="No">No</option>"""
elif idxHTML == 'No':
remove_html_element += """<option value="Yes">Yes</option>"""
remove_html_element += """<option value="No" selected ="selected">No</option>"""
else:
remove_html_element += """<option value="Yes">Yes</option>"""
remove_html_element += """<option value="No">No</option>"""
remove_html_element += """</select>"""
text = """<span class="adminlabel">Remove HTML markup</span>""" + remove_html_element
output += createhiddenform(action="modifyremovehtml#4",
text=text,
button="Modify",
idxID=idxID,
ln=ln,
confirm=0)
if confirm in [0, "0"] and get_idx_remove_html_markup(idxID) == idxHTML:
output += """<span class="info">Remove HTML markup parameter has not been changed</span>"""
elif confirm in [0, "0"]:
text = """<span class="important">You are going to change the remove HTML markup for this index.<br />
<strong>Are you sure you want to change the remove HTML markup of this index?</strong>"""
output += createhiddenform(action="modifyremovehtml#4",
text=text,
button="Modify",
idxID=idxID,
idxHTML=idxHTML,
ln=ln,
confirm=1)
elif idxID > -1 and confirm in [1, "1"]:
res = modify_idx_html_markup(idxID, idxHTML)
output += write_outcome(res)
output += """<br /><span class="info">Please note you must run as soon as possible:
<pre>$> %s/bibindex --reindex -w %s</pre></span>""" % (CFG_BINDIR, get_idx(idxID)[0][1])
elif confirm in [1, "1"]:
output += """<br /><b><span class="info">Please give a name for the index.</span></b>"""
else:
output = """No index to modify."""
body = [output]
if callback:
return perform_editindex(idxID, ln, "perform_modifyremovehtml", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifyremovelatex(idxID, ln=CFG_SITE_LANG, idxLATEX='', callback='yes', confirm=-1):
"""Form to modify the 'remove latex' configuration.
@param idxID: id of the index on which modification will be performed.
@param idxLATEX: remove latex markup or not ('Yes' or 'No')"""
subtitle = ""
output = ""
idx = get_idx(idxID)
if not idx:
idxID = -1
if idxID not in [-1, "-1"]:
subtitle = """<a name="4"></a>8. Modify remove latex markup.&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % CFG_SITE_URL
if confirm in [-1, "-1"]:
idxLATEX = get_idx_remove_latex_markup(idxID)
if not idxLATEX:
idxLATEX = ''
remove_latex_element = """<select name="idxLATEX" class="admin_w200">"""
if idxLATEX == 'Yes':
remove_latex_element += """<option value="Yes" selected ="selected">Yes</option>"""
remove_latex_element += """<option value="No">No</option>"""
elif idxLATEX == 'No':
remove_latex_element += """<option value="Yes">Yes</option>"""
remove_latex_element += """<option value="No" selected ="selected">No</option>"""
else:
remove_latex_element += """<option value="Yes">Yes</option>"""
remove_latex_element += """<option value="No">No</option>"""
remove_latex_element += """</select>"""
text = """<span class="adminlabel">Remove latex markup</span>""" + remove_latex_element
output += createhiddenform(action="modifyremovelatex#4",
text=text,
button="Modify",
idxID=idxID,
ln=ln,
confirm=0)
if confirm in [0, "0"] and get_idx_remove_latex_markup(idxID) == idxLATEX:
output += """<span class="info">Remove latex markup parameter has not been changed</span>"""
elif confirm in [0, "0"]:
text = """<span class="important">You are going to change the remove latex markup for this index.<br />
<strong>Are you sure you want to change the remove latex markup of this index?</strong>"""
output += createhiddenform(action="modifyremovelatex#4",
text=text,
button="Modify",
idxID=idxID,
idxLATEX=idxLATEX,
ln=ln,
confirm=1)
elif idxID > -1 and confirm in [1, "1"]:
res = modify_idx_latex_markup(idxID, idxLATEX)
output += write_outcome(res)
output += """<br /><span class="info">Please note you must run as soon as possible:
<pre>$> %s/bibindex --reindex -w %s</pre></span>""" % (CFG_BINDIR, get_idx(idxID)[0][1])
elif confirm in [1, "1"]:
output += """<br /><b><span class="info">Please give a name for the index.</span></b>"""
else:
output = """No index to modify."""
body = [output]
if callback:
return perform_editindex(idxID, ln, "perform_modifyremovelatex", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifytokenizer(idxID, ln=CFG_SITE_LANG, idxTOK='', callback='yes', confirm=-1):
"""Form to modify the 'tokenizer' configuration.
@param idxID: id of the index on which modification will be performed.
@param idxTOK: tokenizer name"""
subtitle = ""
output = ""
idx = get_idx(idxID)
if not idx:
idxID = -1
if idxID not in [-1, "-1"]:
subtitle = """<a name="4"></a>9. Modify tokenizer.&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % CFG_SITE_URL
if confirm in [-1, "-1"]:
idxTOK = get_idx_tokenizer(idxID)
if not idxTOK:
idxTOK = ''
tokenizer_element = """<select name="idxTOK" class="admin_w200">"""
- for key in _TOKENIZERS:
+ tokenizers = [tokenizer for tokenizer in _TOKENIZERS if _TOKENIZERS[tokenizer]().implemented]
+ for key in tokenizers:
if key == idxTOK:
tokenizer_element += """<option value="%s" selected ="selected">%s</option>""" % (key, key)
else:
tokenizer_element += """<option value="%s">%s</option>""" % (key, key)
tokenizer_element += """</select>"""
text = """<span class="adminlabel">Tokenizer</span>""" + tokenizer_element
output += createhiddenform(action="modifytokenizer#4",
text=text,
button="Modify",
idxID=idxID,
ln=ln,
confirm=0)
if confirm in [0, "0"] and get_idx_tokenizer(idxID) == idxTOK:
output += """<span class="info">Tokenizer has not been changed</span>"""
elif confirm in [0, "0"]:
text = """<span class="important">You are going to change a tokenizer for this index.<br />
<strong>Are you sure you want to do this?</strong>"""
output += createhiddenform(action="modifytokenizer#4",
text=text,
button="Modify",
idxID=idxID,
idxTOK=idxTOK,
ln=ln,
confirm=1)
elif idxID > -1 and confirm in [1, "1"]:
res = modify_idx_tokenizer(idxID, idxTOK)
output += write_outcome(res)
output += """<br /><span class="info">Please note you must run as soon as possible:
<pre>$> %s/bibindex --reindex -w %s</pre></span>""" % (CFG_BINDIR, get_idx(idxID)[0][1])
elif confirm in [1, "1"]:
output += """<br /><b><span class="info">Please give a name for the index.</span></b>"""
else:
output = """No index to modify."""
body = [output]
if callback:
return perform_editindex(idxID, ln, "perform_modifytokenizer", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifyfield(fldID, ln=CFG_SITE_LANG, code='', callback='yes', confirm=-1):
"""form to modify a field.
fldID - the field to change."""
subtitle = ""
output = ""
fld_dict = dict(get_def_name('', "field"))
if fldID not in [-1, "-1"]:
if confirm in [-1, "-1"]:
res = get_fld(fldID)
code = res[0][2]
else:
code = str.replace("%s" % code, " ", "")
fldID = int(fldID)
subtitle = """<a name="2"></a>1. Modify field code for logical field '%s'&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % (fld_dict[int(fldID)], CFG_SITE_URL)
text = """
<span class="adminlabel">Field code</span>
<input class="admin_w200" type="text" name="code" value="%s" /><br />
""" % code
output += createhiddenform(action="modifyfield#2",
text=text,
button="Modify",
fldID=fldID,
ln=ln,
confirm=1)
if fldID > -1 and confirm in [1, "1"]:
fldID = int(fldID)
res = modify_fld(fldID, code)
output += write_outcome(res)
else:
output = """No field to modify.
"""
body = [output]
if callback:
return perform_editfield(fldID, ln, "perform_modifyfield", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifyindexfields(idxID, ln=CFG_SITE_LANG, callback='yes', content='', confirm=-1):
"""Modify which logical fields to use in this index.."""
output = ''
subtitle = """<a name="3"></a>3. Modify index fields.&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % CFG_SITE_URL
output = """<dl>
<dt>Menu</dt>
<dd><a href="%s/admin/bibindex/bibindexadmin.py/addindexfield?idxID=%s&amp;ln=%s#3.1">Add field to index</a></dd>
<dd><a href="%s/admin/bibindex/bibindexadmin.py/field?ln=%s">Manage fields</a></dd>
</dl>
""" % (CFG_SITE_URL, idxID, ln, CFG_SITE_URL, ln)
header = ['Field', '']
actions = []
idx_fld = get_idx_fld(idxID)
if len(idx_fld) > 0:
for (idxID, idxNAME,fldID, fldNAME, regexp_punct, regexp_alpha_sep) in idx_fld:
actions.append([fldNAME])
for col in [(('Remove','removeindexfield'),)]:
actions[-1].append('<a href="%s/admin/bibindex/bibindexadmin.py/%s?idxID=%s&amp;fldID=%s&amp;ln=%s#3.1">%s</a>' % (CFG_SITE_URL, col[0][1], idxID, fldID, ln, col[0][0]))
for (str, function) in col[1:]:
actions[-1][-1] += ' / <a href="%s/admin/bibindex/bibindexadmin.py/%s?fldID=%s&amp;flID=%s&amp;ln=%s#4.1">%s</a>' % (CFG_SITE_URL, function, idxID, fldID, ln, str)
output += tupletotable(header=header, tuple=actions)
else:
output += """No index fields exists"""
output += content
body = [output]
if callback:
return perform_editindex(idxID, ln, "perform_modifyindexfields", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifyfieldtags(fldID, ln=CFG_SITE_LANG, callback='yes', content='', confirm=-1):
"""show the sort fields of this collection.."""
output = ''
fld_dict = dict(get_def_name('', "field"))
fld_type = get_fld_nametypes()
fldID = int(fldID)
subtitle = """<a name="4"></a>3. Modify MARC tags for the logical field '%s'&nbsp;&nbsp;&nbsp;<small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % (fld_dict[int(fldID)], CFG_SITE_URL)
output = """<dl>
<dt>Menu</dt>
<dd><a href="%s/admin/bibindex/bibindexadmin.py/addtag?fldID=%s&amp;ln=%s#4.1">Add MARC tag</a></dd>
<dd><a href="%s/admin/bibindex/bibindexadmin.py/deletetag?fldID=%s&amp;ln=%s#4.1">Delete unused MARC tags</a></dd>
</dl>
""" % (CFG_SITE_URL, fldID, ln, CFG_SITE_URL, fldID, ln)
header = ['', 'Value', 'Comment', 'Actions']
actions = []
res = get_fld_tags(fldID)
if len(res) > 0:
i = 0
for (fldID, tagID, tname, tvalue, score) in res:
move = ""
if i != 0:
move += """<a href="%s/admin/bibindex/bibindexadmin.py/switchtagscore?fldID=%s&amp;id_1=%s&amp;id_2=%s&amp;ln=%s&amp=rand=%s#4"><img border="0" src="%s/img/smallup.gif" title="Move tag up"></a>""" % (CFG_SITE_URL, fldID, tagID, res[i - 1][1], ln, random.randint(0, 1000), CFG_SITE_URL)
else:
move += "&nbsp;&nbsp;&nbsp;"
i += 1
if i != len(res):
move += '<a href="%s/admin/bibindex/bibindexadmin.py/switchtagscore?fldID=%s&amp;id_1=%s&amp;id_2=%s&amp;ln=%s&amp;rand=%s#4"><img border="0" src="%s/img/smalldown.gif" title="Move tag down"></a>' % (CFG_SITE_URL, fldID, tagID, res[i][1], ln, random.randint(0, 1000), CFG_SITE_URL)
actions.append([move, tvalue, tname])
for col in [(('Details','showdetailsfieldtag'), ('Modify','modifytag'),('Remove','removefieldtag'),)]:
actions[-1].append('<a href="%s/admin/bibindex/bibindexadmin.py/%s?fldID=%s&amp;tagID=%s&amp;ln=%s#4.1">%s</a>' % (CFG_SITE_URL, col[0][1], fldID, tagID, ln, col[0][0]))
for (str, function) in col[1:]:
actions[-1][-1] += ' / <a href="%s/admin/bibindex/bibindexadmin.py/%s?fldID=%s&amp;tagID=%s&amp;ln=%s#4.1">%s</a>' % (CFG_SITE_URL, function, fldID, tagID, ln, str)
output += tupletotable(header=header, tuple=actions)
else:
output += """No fields exists"""
output += content
body = [output]
if callback:
return perform_editfield(fldID, ln, "perform_modifyfieldtags", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_addtag(fldID, ln=CFG_SITE_LANG, value=['',-1], name='', callback="yes", confirm=-1):
"""form to add a new field.
fldNAME - the name of the new field
code - the field code"""
output = ""
subtitle = """<a name="4.1"></a>Add MARC tag to logical field"""
text = """
Add new tag:<br />
<span class="adminlabel">Tag value</span>
<input class="admin_w200" maxlength="6" type="text" name="value" value="%s" /><br />
<span class="adminlabel">Tag comment</span>
<input class="admin_w200" type="text" name="name" value="%s" /><br />
""" % ((name=='' and value[0] or name), value[0])
text += """Or existing tag:<br />
<span class="adminlabel">Tag</span>
<select name="value" class="admin_w200">
<option value="-1">- Select a tag -</option>
"""
fld_tags = get_fld_tags(fldID)
tags = get_tags()
fld_tags = dict(map(lambda x: (x[1], x[0]), fld_tags))
for (id_tag, tname, tvalue) in tags:
if id_tag not in fld_tags:
text += """<option value="%s" %s>%s</option>""" % (tvalue, (tvalue==value[1] and 'selected="selected"' or ''), "%s - %s" % (tvalue, tname))
text += """</select>"""
output = createhiddenform(action="%s/admin/bibindex/bibindexadmin.py/addtag" % CFG_SITE_URL,
text=text,
fldID=fldID,
ln=ln,
button="Add tag",
confirm=1)
if (value[0] and value[1] in [-1, "-1"]) or (not value[0] and value[1] not in [-1, "-1"]):
if confirm in ["1", 1]:
res = add_fld_tag(fldID, name, (value[0] !='' and value[0] or value[1]))
output += write_outcome(res)
elif confirm not in ["-1", -1]:
output += """<b><span class="info">Please choose to add either a new or an existing MARC tag, but not both.</span></b>
"""
body = [output]
if callback:
return perform_modifyfieldtags(fldID, ln, "perform_addtag", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_modifytag(fldID, tagID, ln=CFG_SITE_LANG, name='', value='', callback='yes', confirm=-1):
"""form to modify a field.
fldID - the field to change."""
subtitle = ""
output = ""
fld_dict = dict(get_def_name('', "field"))
fldID = int(fldID)
tagID = int(tagID)
tag = get_tags(tagID)
if confirm in [-1, "-1"] and not value and not name:
name = tag[0][1]
value = tag[0][2]
subtitle = """<a name="3.1"></a>Modify MARC tag"""
text = """
Any modifications will apply to all logical fields using this tag.<br />
<span class="adminlabel">Tag value</span>
<input class="admin_w200" type="text" name="value" value="%s" /><br />
<span class="adminlabel">Comment</span>
<input class="admin_w200" type="text" name="name" value="%s" /><br />
""" % (value, name)
output += createhiddenform(action="modifytag#4.1",
text=text,
button="Modify",
fldID=fldID,
tagID=tagID,
ln=ln,
confirm=1)
if name and value and confirm in [1, "1"]:
res = modify_tag(tagID, name, value)
output += write_outcome(res)
body = [output]
if callback:
return perform_modifyfieldtags(fldID, ln, "perform_modifytag", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_removefieldtag(fldID, tagID, ln=CFG_SITE_LANG, callback='yes', confirm=0):
"""form to remove a tag from a field.
fldID - the current field, remove the tag from this field.
tagID - remove the tag with this id"""
subtitle = """<a name="4.1"></a>Remove MARC tag from logical field"""
output = ""
fld_dict = dict(get_def_name('', "field"))
if fldID and tagID:
fldID = int(fldID)
tagID = int(tagID)
tag = get_fld_tags(fldID, tagID)
if confirm not in ["1", 1]:
text = """Do you want to remove the tag '%s - %s ' from the field '%s'.""" % (tag[0][3], tag[0][2], fld_dict[fldID])
output += createhiddenform(action="removefieldtag#4.1",
text=text,
button="Confirm",
fldID=fldID,
tagID=tagID,
confirm=1)
elif confirm in ["1", 1]:
res = remove_fldtag(fldID, tagID)
output += write_outcome(res)
body = [output]
if callback:
return perform_modifyfieldtags(fldID, ln, "perform_removefieldtag", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_addindexfield(idxID, ln=CFG_SITE_LANG, fldID='', callback="yes", confirm=-1):
"""form to add a new field.
fldNAME - the name of the new field
code - the field code"""
output = ""
subtitle = """<a name="4.1"></a>Add logical field to index"""
text = """
<span class="adminlabel">Field name</span>
<select name="fldID" class="admin_w200">
<option value="-1">- Select a field -</option>
"""
fld = get_fld()
for (fldID2, fldNAME, fldCODE) in fld:
text += """<option value="%s" %s>%s</option>""" % (fldID2, (fldID==fldID2 and 'selected="selected"' or ''), fldNAME)
text += """</select>"""
output = createhiddenform(action="%s/admin/bibindex/bibindexadmin.py/addindexfield" % CFG_SITE_URL,
text=text,
idxID=idxID,
ln=ln,
button="Add field",
confirm=1)
if fldID and not fldID in [-1, "-1"] and confirm in ["1", 1]:
res = add_idx_fld(idxID, fldID)
output += write_outcome(res)
elif confirm in ["1", 1]:
output += """<b><span class="info">Please select a field to add.</span></b>"""
body = [output]
if callback:
return perform_modifyindexfields(idxID, ln, "perform_addindexfield", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_removeindexfield(idxID, fldID, ln=CFG_SITE_LANG, callback='yes', confirm=0):
"""form to remove a field from an index.
idxID - the current index, remove the field from this index.
fldID - remove the field with this id"""
subtitle = """<a name="3.1"></a>Remove field from index"""
output = ""
if fldID and idxID:
fldID = int(fldID)
idxID = int(idxID)
fld = get_fld(fldID)
idx = get_idx(idxID)
if fld and idx and confirm not in ["1", 1]:
text = """Do you want to remove the field '%s' from the index '%s'.""" % (fld[0][1], idx[0][1])
output += createhiddenform(action="removeindexfield#3.1",
text=text,
button="Confirm",
idxID=idxID,
fldID=fldID,
confirm=1)
elif confirm in ["1", 1]:
res = remove_idxfld(idxID, fldID)
output += write_outcome(res)
body = [output]
if callback:
return perform_modifyindexfields(idxID, ln, "perform_removeindexfield", addadminbox(subtitle, body))
else:
return addadminbox(subtitle, body)
def perform_switchtagscore(fldID, id_1, id_2, ln=CFG_SITE_LANG):
"""Switch the score of id_1 and id_2 in the table type.
colID - the current collection
id_1/id_2 - the id's to change the score for.
type - like "format" """
output = ""
name_1 = run_sql("select name from tag where id=%s", (id_1, ))[0][0]
name_2 = run_sql("select name from tag where id=%s", (id_2, ))[0][0]
res = switch_score(fldID, id_1, id_2)
output += write_outcome(res)
return perform_modifyfieldtags(fldID, ln, content=output)
def perform_deletetag(fldID, ln=CFG_SITE_LANG, tagID=-1, callback='yes', confirm=-1):
"""form to delete an MARC tag not in use.
fldID - the collection id of the current collection.
fmtID - the format id to delete."""
subtitle = """<a name="10.3"></a>Delete an unused MARC tag"""
output = """
<dl>
<dd>Deleting an MARC tag will also delete the translations associated.</dd>
</dl>
"""
fldID = int(fldID)
if tagID not in [-1," -1"] and confirm in [1, "1"]:
ares = delete_tag(tagID)
fld_tag = get_fld_tags()
fld_tag = dict(map(lambda x: (x[1], x[0]), fld_tag))
tags = get_tags()
text = """
<span class="adminlabel">MARC tag</span>
<select name="tagID" class="admin_w200">
"""
text += """<option value="-1">- Select MARC tag -"""
i = 0
for (id, name, value) in tags:
if id not in fld_tag:
text += """<option value="%s" %s>%s</option>""" % (id, id == int(tagID) and 'selected="selected"' or '', "%s - %s" % (value, name))
i += 1
text += """</select><br />"""
if i == 0:
output += """<b><span class="info">No unused MARC tags</span></b><br />"""
else:
output += createhiddenform(action="deletetag#4.1",
text=text,
button="Delete",
fldID=fldID,
ln=ln,
confirm=0)
if tagID not in [-1,"-1"]:
tagID = int(tagID)
tags = get_tags(tagID)
if confirm in [0, "0"]:
text = """<b>Do you want to delete the MARC tag '%s'.</b>""" % tags[0][2]
output += createhiddenform(action="deletetag#4.1",
text=text,
button="Confirm",
fldID=fldID,
tagID=tagID,
ln=ln,
confirm=1)
elif confirm in [1, "1"]:
output += write_outcome(ares)
elif confirm not in [-1, "-1"]:
output += """<b><span class="info">Choose a MARC tag to delete.</span></b>"""
body = [output]
output = "<br />" + addadminbox(subtitle, body)
return perform_modifyfieldtags(fldID, ln, content=output)
def compare_on_val(first, second):
"""Compare the two values"""
return cmp(first[1], second[1])
def get_col_fld(colID=-1, type = '', id_field=''):
"""Returns either all portalboxes associated with a collection, or based on either colID or language or both.
colID - collection id
ln - language id"""
sql = "SELECT id_collection,id_field,id_fieldvalue,type,score,score_fieldvalue FROM collection_field_fieldvalue, field WHERE id_field=field.id"
params = []
try:
if id_field:
sql += " AND id_field=%s"
params.append(id_field)
sql += " ORDER BY type, score desc, score_fieldvalue desc"
res = run_sql(sql, tuple(params))
return res
except StandardError as e:
return ""
def get_idx(idxID=''):
sql = "SELECT id,name,description,last_updated,stemming_language, synonym_kbrs,remove_stopwords,remove_html_markup,remove_latex_markup,tokenizer FROM idxINDEX"
params = []
try:
if idxID:
sql += " WHERE id=%s"
params.append(idxID)
sql += " ORDER BY id asc"
res = run_sql(sql, tuple(params))
return res
except StandardError as e:
return ""
def get_idx_synonym_kb(idxID):
"""Returns a synonym knowledge base field value"""
try:
return run_sql("SELECT synonym_kbrs FROM idxINDEX WHERE ID=%s", (idxID, ))[0][0]
except StandardError as e:
return e.__str__()
def get_idx_remove_stopwords(idxID):
"""Returns a stopwords field value"""
try:
return run_sql("SELECT remove_stopwords FROM idxINDEX WHERE ID=%s", (idxID, ))[0][0]
except StandardError as e:
return (0, e)
def get_idx_remove_html_markup(idxID):
"""Returns a remove html field value"""
try:
return run_sql("SELECT remove_html_markup FROM idxINDEX WHERE ID=%s", (idxID, ))[0][0]
except StandardError as e:
return (0, e)
def get_idx_remove_latex_markup(idxID):
"""Returns a remove latex field value"""
try:
return run_sql("SELECT remove_latex_markup FROM idxINDEX WHERE ID=%s", (idxID, ))[0][0]
except StandardError as e:
return (0, e)
def get_idx_tokenizer(idxID):
"""Returns a tokenizer field value"""
try:
return run_sql("SELECT tokenizer FROM idxINDEX WHERE ID=%s", (idxID, ))[0][0]
except StandardError as e:
return (0, e)
def get_fld_tags(fldID='', tagID=''):
"""Returns tags associated with a field.
fldID - field id
tagID - tag id"""
sql = "SELECT id_field,id_tag, tag.name, tag.value, score FROM field_tag,tag WHERE tag.id=field_tag.id_tag"
params = []
try:
if fldID:
sql += " AND id_field=%s"
params.append(fldID)
if tagID:
sql += " AND id_tag=%s"
params.append(tagID)
sql += " ORDER BY score desc, tag.value, tag.name"
res = run_sql(sql, tuple(params))
return res
except StandardError as e:
return ""
def get_tags(tagID=''):
"""Returns all or a given tag.
tagID - tag id
ln - language id"""
sql = "SELECT id, name, value FROM tag"
params = []
try:
if tagID:
sql += " WHERE id=%s"
params.append(tagID)
sql += " ORDER BY name, value"
res = run_sql(sql, tuple(params))
return res
except StandardError as e:
return ""
def get_fld(fldID=''):
"""Returns all fields or only the given field"""
try:
if not fldID:
res = run_sql("SELECT id, name, code FROM field ORDER by name, code")
else:
res = run_sql("SELECT id, name, code FROM field WHERE id=%s ORDER by name, code", (fldID, ))
return res
except StandardError as e:
return ""
def get_fld_id(fld_name=''):
"""Returns field id for a field name"""
try:
res = run_sql('SELECT id FROM field WHERE name=%s', (fld_name,))
return res[0][0]
except StandardError as e:
return ''
def get_fld_value(fldvID = ''):
"""Returns fieldvalue"""
try:
sql = "SELECT id, name, value FROM fieldvalue"
params = []
if fldvID:
sql += " WHERE id=%s"
params.append(fldvID)
res = run_sql(sql, tuple(params))
return res
except StandardError as e:
return ""
def get_idx_fld(idxID=''):
"""Return a list of fields associated with one or all indexes"""
try:
sql = "SELECT id_idxINDEX, idxINDEX.name, id_field, field.name, regexp_punctuation, regexp_alphanumeric_separators FROM idxINDEX, field, idxINDEX_field WHERE idxINDEX.id = idxINDEX_field.id_idxINDEX AND field.id = idxINDEX_field.id_field"
params = []
if idxID:
sql += " AND id_idxINDEX=%s"
params.append(idxID)
sql += " ORDER BY id_idxINDEX asc"
res = run_sql(sql, tuple(params))
return res
except StandardError as e:
return ""
def get_col_nametypes():
"""Return a list of the various translationnames for the fields"""
type = []
type.append(('ln', 'Long name'))
return type
def get_fld_nametypes():
"""Return a list of the various translationnames for the fields"""
type = []
type.append(('ln', 'Long name'))
return type
def get_idx_nametypes():
"""Return a list of the various translationnames for the index"""
type = []
type.append(('ln', 'Long name'))
return type
def get_sort_nametypes():
"""Return a list of the various translationnames for the fields"""
type = {}
type['soo'] = 'Sort options'
type['seo'] = 'Search options'
type['sew'] = 'Search within'
return type
def remove_fld(colID,fldID, fldvID=''):
"""Removes a field from the collection given.
colID - the collection the format is connected to
fldID - the field which should be removed from the collection."""
try:
sql = "DELETE FROM collection_field_fieldvalue WHERE id_collection=%s AND id_field=%s"
params = [colID, fldID]
if fldvID:
sql += " AND id_fieldvalue=%s"
params.append(fldvID)
res = run_sql(sql, tuple(params))
return (1, "")
except StandardError as e:
return (0, e)
def remove_idxfld(idxID, fldID):
"""Remove a field from a index in table idxINDEX_field
idxID - index id from idxINDEX
fldID - field id from field table"""
try:
sql = "DELETE FROM idxINDEX_field WHERE id_field=%s and id_idxINDEX=%s"
res = run_sql(sql, (fldID, idxID))
return (1, "")
except StandardError as e:
return (0, e)
def remove_fldtag(fldID,tagID):
"""Removes a tag from the field given.
fldID - the field the tag is connected to
tagID - the tag which should be removed from the field."""
try:
sql = "DELETE FROM field_tag WHERE id_field=%s AND id_tag=%s"
res = run_sql(sql, (fldID, tagID))
return (1, "")
except StandardError as e:
return (0, e)
def delete_tag(tagID):
"""Deletes all data for the given field
fldID - delete all data in the tables associated with field and this id """
try:
res = run_sql("DELETE FROM tag where id=%s", (tagID, ))
return (1, "")
except StandardError as e:
return (0, e)
def delete_idx(idxID):
"""Deletes all data for the given index together with the idxWORDXXR and idxWORDXXF tables"""
try:
idxID = int(idxID)
res = run_sql("DELETE FROM idxINDEX WHERE id=%s", (idxID, ))
res = run_sql("DELETE FROM idxINDEXNAME WHERE id_idxINDEX=%s", (idxID, ))
res = run_sql("DELETE FROM idxINDEX_field WHERE id_idxINDEX=%s", (idxID, ))
res = run_sql("DROP TABLE idxWORD%02dF" % idxID) # kwalitee: disable=sql
res = run_sql("DROP TABLE idxWORD%02dR" % idxID) # kwalitee: disable=sql
res = run_sql("DROP TABLE idxPAIR%02dF" % idxID) # kwalitee: disable=sql
res = run_sql("DROP TABLE idxPAIR%02dR" % idxID) # kwalitee: disable=sql
res = run_sql("DROP TABLE idxPHRASE%02dF" % idxID) # kwalitee: disable=sql
res = run_sql("DROP TABLE idxPHRASE%02dR" % idxID) # kwalitee: disable=sql
return (1, "")
except StandardError as e:
return (0, e)
def delete_virtual_idx(idxID):
"""Deletes this virtual index - it means that function
changes type of the index from 'virtual' to 'normal'
@param idxID -id of the virtual index to delete/change into normal idx
"""
try:
run_sql("""UPDATE idxINDEX SET indexer='native'
WHERE id=%s""", (idxID, ))
run_sql("""DELETE FROM idxINDEX_idxINDEX
WHERE id_virtual=%s""", (idxID, ))
+ drop_queue_tables(idxID)
return (1, "")
except StandardError as e:
return (0, e)
def delete_fld(fldID):
"""Deletes all data for the given field
fldID - delete all data in the tables associated with field and this id """
try:
res = run_sql("DELETE FROM collection_field_fieldvalue WHERE id_field=%s", (fldID, ))
res = run_sql("DELETE FROM field_tag WHERE id_field=%s", (fldID, ))
res = run_sql("DELETE FROM idxINDEX_field WHERE id_field=%s", (fldID, ))
res = run_sql("DELETE FROM field WHERE id=%s", (fldID, ))
return (1, "")
except StandardError as e:
return (0, e)
def add_idx(idxNAME):
"""Add a new index. returns the id of the new index.
idxID - the id for the index, number
idxNAME - the default name for the default language of the format."""
try:
idxID = 0
res = run_sql("SELECT id from idxINDEX WHERE name=%s", (idxNAME,))
if res:
return (0, (0, "A index with the given name already exists."))
for i in xrange(1, 100):
res = run_sql("SELECT id from idxINDEX WHERE id=%s", (i, ))
res2 = get_table_status_info("idxWORD%02d%%" % i)
if not res and not res2:
idxID = i
break
if idxID == 0:
return (0, (0, "Not possible to create new indexes, delete an index and try again."))
res = run_sql("INSERT INTO idxINDEX (id, name) VALUES (%s,%s)", (idxID, idxNAME))
type = get_idx_nametypes()[0][0]
res = run_sql("INSERT INTO idxINDEXNAME (id_idxINDEX, ln, type, value) VALUES (%s,%s,%s,%s)",
(idxID, CFG_SITE_LANG, type, idxNAME))
res = run_sql("""CREATE TABLE IF NOT EXISTS idxWORD%02dF (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM""" % idxID)
res = run_sql("""CREATE TABLE IF NOT EXISTS idxWORD%02dR (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type),
KEY type (type)
) ENGINE=MyISAM""" % idxID)
res = run_sql("""CREATE TABLE IF NOT EXISTS idxPAIR%02dF (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM""" % idxID)
res = run_sql("""CREATE TABLE IF NOT EXISTS idxPAIR%02dR (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type),
KEY type (type)
) ENGINE=MyISAM""" % idxID)
res = run_sql("""CREATE TABLE IF NOT EXISTS idxPHRASE%02dF (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM""" % idxID)
res = run_sql("""CREATE TABLE IF NOT EXISTS idxPHRASE%02dR (
id_bibrec mediumint(9) unsigned NOT NULL default '0',
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type),
KEY type (type)
) ENGINE=MyISAM""" % idxID)
res = run_sql("SELECT id from idxINDEX WHERE id=%s", (idxID, ))
res2 = get_table_status_info("idxWORD%02dF" % idxID)
res3 = get_table_status_info("idxWORD%02dR" % idxID)
if res and res2 and res3:
return (1, res[0][0])
elif not res:
return (0, (0, "Could not add the new index to idxINDEX"))
elif not res2:
return (0, (0, "Forward table not created for unknown reason."))
elif not res3:
return (0, (0, "Reverse table not created for unknown reason."))
except StandardError as e:
return (0, e)
+def create_queue_tables(index_id):
+ """Creates queue tables for virtual index.
+ Queue tables store orders for virtual index
+ from its dependent indexes.
+ @param index_id: id of the index we want to create queue tables for
+ """
+ query = """
+ CREATE TABLE IF NOT EXISTS idx%s%02dQ (
+ id mediumint(10) unsigned NOT NULL auto_increment,
+ runtime datetime NOT NULL default '0000-00-00 00:00:00',
+ id_bibrec_low mediumint(9) unsigned NOT NULL,
+ id_bibrec_high mediumint(9) unsigned NOT NULL,
+ index_name varchar(50) NOT NULL default '',
+ mode varchar(50) NOT NULL default 'update',
+ PRIMARY KEY (id),
+ INDEX (index_name),
+ INDEX (runtime)
+ ) ENGINE=MyISAM;"""
+ run_sql(query % ("WORD", int(index_id)))
+ run_sql(query % ("PAIR", int(index_id)))
+ run_sql(query % ("PHRASE", int(index_id)))
+
+
+def drop_queue_tables(index_id):
+ """
+ Drops queue tables.
+ @param index_id: id of the index we want to drop tables for
+ """
+ query = """DROP TABLE IF EXISTS idx%s%02dQ"""
+ run_sql(query % ("WORD", int(index_id)))
+ run_sql(query % ("PAIR", int(index_id)))
+ run_sql(query % ("PHRASE", int(index_id)))
+
+
def add_virtual_idx(id_virtual, id_normal):
"""Adds new virtual index and its first dependent index.
Doesn't change index's settings, but they're not
used anymore.
Uses function add_dependent_index, because
query in both cases is the same.
"""
try:
run_sql("""UPDATE idxINDEX SET indexer='virtual'
WHERE id=%s""", (id_virtual, ))
+ create_queue_tables(id_virtual)
return add_dependent_index(id_virtual, id_normal)
except StandardError as e:
return (0, e)
-def modify_dependent_indexes(idxID, indexes_to_add, indexes_to_remove):
- """Adds and removes dependent indexes"""
+def modify_dependent_indexes(idxID, indexes_to_add=[]):
+ """
+ Adds indexes to a list of dependent indexes of
+ a specific virtual index.
+ @param idxID: id of the virtual index
+ @param indexes_to_add: list of names of indexes which
+ should be added as new dependent
+ indexes for a virtual index
+ """
all_indexes = dict(get_all_index_names_and_column_values("id"))
for index_name in indexes_to_add:
res = add_dependent_index(idxID, all_indexes[index_name])
if res[0] == 0:
return res
- for index_name in indexes_to_remove:
- res = remove_dependent_index(idxID, all_indexes[index_name])
- if res[0] == 0:
- return res
return (1, "")
def add_dependent_index(id_virtual, id_normal):
"""Adds dependent index to specific virtual index"""
try:
query = """INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal)
VALUES (%s, %s)""" % (id_virtual, id_normal)
res = run_sql(query)
return (1, "")
except StandardError as e:
return (0, e)
-def remove_dependent_index(id_virtual, id_normal):
- """Remove dependent index to specific virtual index"""
- try:
- query = """DELETE FROM idxINDEX_idxINDEX
- WHERE id_virtual=%s AND
- id_normal=%s
- """ % (id_virtual, id_normal)
- res = run_sql(query)
- return (1, "")
- except StandardError as e:
- return (0, e)
-
-
-
-
def add_fld(name, code):
"""Add a new logical field. Returns the id of the field.
code - the code for the field,
name - the default name for the default language of the field."""
try:
type = get_fld_nametypes()[0][0]
res = run_sql("INSERT INTO field (name, code) VALUES (%s,%s)", (name, code))
fldID = run_sql("SELECT id FROM field WHERE code=%s", (code,))
res = run_sql("INSERT INTO fieldname (id_field, type, ln, value) VALUES (%s,%s,%s,%s)", (fldID[0][0], type, CFG_SITE_LANG, name))
if fldID:
return (1, fldID[0][0])
else:
raise StandardError
except StandardError as e:
return (0, e)
def add_fld_tag(fldID, name, value):
"""Add a sort/search/field to the collection.
colID - the id of the collection involved
fmtID - the id of the format.
score - the score of the format, decides sorting, if not given, place the format on top"""
try:
res = run_sql("SELECT score FROM field_tag WHERE id_field=%s ORDER BY score desc", (fldID, ))
if res:
score = int(res[0][0]) + 1
else:
score = 0
res = run_sql("SELECT id FROM tag WHERE value=%s", (value,))
if not res:
if name == '':
name = value
res = run_sql("INSERT INTO tag (name, value) VALUES (%s,%s)", (name, value))
res = run_sql("SELECT id FROM tag WHERE value=%s", (value,))
res = run_sql("INSERT INTO field_tag(id_field, id_tag, score) values(%s, %s, %s)", (fldID, res[0][0], score))
return (1, "")
except StandardError as e:
return (0, e)
def add_idx_fld(idxID, fldID):
"""Add a field to an index"""
try:
sql = "SELECT id_idxINDEX FROM idxINDEX_field WHERE id_idxINDEX=%s and id_field=%s"
res = run_sql(sql, (idxID, fldID))
if res:
return (0, (0, "The field selected already exists for this index"))
sql = "INSERT INTO idxINDEX_field(id_idxINDEX, id_field) values (%s, %s)"
res = run_sql(sql, (idxID, fldID))
return (1, "")
except StandardError as e:
return (0, e)
+
+def update_all_queue_tables_with_new_name(idxID, idxNAME_new, idxNAME_old):
+ """
+ Updates queue tables for all virtual indexes connected to this index
+ with new name of this index.
+ @param idxID: id of the index
+ @param idxNAME_new: new name for specified index
+ @param idxNAME_old: old name of specified index
+ """
+ virtual_indexes = get_index_virtual_indexes(idxID)
+ for index in virtual_indexes:
+ id_virtual, name = index
+ query = """UPDATE idxWORD%02dQ SET index_name=%%s WHERE index_name=%%s""" % id_virtual
+ run_sql(query, (idxNAME_new, idxNAME_old))
+ query = """UPDATE idxPAIR%02dQ SET index_name=%%s WHERE index_name=%%s""" % id_virtual
+ run_sql(query, (idxNAME_new, idxNAME_old))
+ query = """UPDATE idxPHRASE%02dQ SET index_name=%%s WHERE index_name=%%s""" % id_virtual
+ run_sql(query, (idxNAME_new, idxNAME_old))
+
+
def modify_idx(idxID, idxNAME, idxDESC):
"""Modify index name or index description in idxINDEX table"""
-
- try:
- res = run_sql("UPDATE idxINDEX SET name=%s WHERE id=%s", (idxNAME, idxID))
- res = run_sql("UPDATE idxINDEX SET description=%s WHERE ID=%s", (idxDESC, idxID))
- return (1, "")
- except StandardError as e:
- return (0, e)
+ query = """SELECT proc,status FROM schTASK WHERE proc='bibindex' AND status='RUNNING'"""
+ res = run_sql(query)
+ if len(res) == 0:
+ idxNAME_old = get_index_name_from_index_id(idxID)
+ try:
+ update_all_queue_tables_with_new_name(idxID, idxNAME, idxNAME_old)
+ res = run_sql("UPDATE idxINDEX SET name=%s WHERE id=%s", (idxNAME, idxID))
+ res = run_sql("UPDATE idxINDEX SET description=%s WHERE ID=%s", (idxDESC, idxID))
+ return (1, "")
+ except StandardError as e:
+ return (0, e)
+ else:
+ return (0, "Try again later. Cannot change details of an index when bibindex is running.")
def modify_idx_stemming(idxID, idxSTEM):
"""Modify the index stemming language in idxINDEX table"""
try:
run_sql("UPDATE idxINDEX SET stemming_language=%s WHERE ID=%s", (idxSTEM, idxID))
return (1, "")
except StandardError as e:
return (0, e)
def modify_idx_indexer(idxID, indexer):
"""Modify an indexer type in idxINDEX table"""
try:
res = run_sql("UPDATE idxINDEX SET indexer=%s WHERE ID=%s", (indexer, idxID))
return (1, "")
except StandardError as e:
return (0, e)
def modify_idx_synonym_kb(idxID, idxKB, idxMATCH):
"""Modify the knowledge base for the synonym lookup in idxINDEX table
@param idxID: id of the index in idxINDEX table
@param idxKB: name of the knowledge base (for example: INDEX-SYNONYM-TITLE)
@param idxMATCH: type of match in the knowledge base: exact, leading-to-coma, leading-to-number
"""
try:
field_value = ""
if idxKB != CFG_BIBINDEX_SYNONYM_MATCH_TYPE["None"] and idxMATCH != CFG_BIBINDEX_SYNONYM_MATCH_TYPE["None"]:
field_value = idxKB + CFG_BIBINDEX_COLUMN_VALUE_SEPARATOR + idxMATCH
run_sql("UPDATE idxINDEX SET synonym_kbrs=%s WHERE ID=%s", (field_value, idxID))
return (1, "")
except StandardError as e:
return (0, e)
def modify_idx_stopwords(idxID, idxSTOPWORDS):
"""Modify the stopwords in idxINDEX table
@param idxID: id of the index which we modify
@param idxSTOPWORDS: tells if stopwords should be removed ('Yes' or 'No')
"""
try:
run_sql("UPDATE idxINDEX SET remove_stopwords=%s WHERE ID=%s", (idxSTOPWORDS, idxID))
return (1, "")
except StandardError as e:
return (0, e)
def modify_idx_html_markup(idxID, idxHTML):
"""Modify the index remove html markup in idxINDEX table"""
try:
run_sql("UPDATE idxINDEX SET remove_html_markup=%s WHERE ID=%s", (idxHTML, idxID))
return (1, "")
except StandardError as e:
return (0, e)
def modify_idx_latex_markup(idxID, idxLATEX):
"""Modify the index remove latex markup in idxINDEX table"""
try:
run_sql("UPDATE idxINDEX SET remove_latex_markup=%s WHERE ID=%s", (idxLATEX, idxID))
return (1, "")
except StandardError as e:
return (0, e)
def modify_idx_tokenizer(idxID, idxTOK):
"""Modify a tokenizer in idxINDEX table for given index"""
try:
run_sql("UPDATE idxINDEX SET tokenizer=%s WHERE ID=%s", (idxTOK, idxID))
return (1, "")
except StandardError as e:
return (0, e)
def modify_fld(fldID, code):
"""Modify the code of field
fldID - the id of the field to modify
code - the new code"""
try:
sql = "UPDATE field SET code=%s"
sql += " WHERE id=%s"
res = run_sql(sql, (code, fldID))
return (1, "")
except StandardError as e:
return (0, e)
def modify_tag(tagID, name, value):
"""Modify the name and value of a tag.
tagID - the id of the tag to modify
name - the new name of the tag
value - the new value of the tag"""
try:
sql = "UPDATE tag SET name=%s WHERE id=%s"
res = run_sql(sql, (name, tagID))
sql = "UPDATE tag SET value=%s WHERE id=%s"
res = run_sql(sql, (value, tagID))
return (1, "")
except StandardError as e:
return (0, e)
def switch_score(fldID, id_1, id_2):
"""Switch the scores of id_1 and id_2 in the table given by the argument.
colID - collection the id_1 or id_2 is connected to
id_1/id_2 - id field from tables like format..portalbox...
table - name of the table"""
try:
res1 = run_sql("SELECT score FROM field_tag WHERE id_field=%s and id_tag=%s", (fldID, id_1))
res2 = run_sql("SELECT score FROM field_tag WHERE id_field=%s and id_tag=%s", (fldID, id_2))
res = run_sql("UPDATE field_tag SET score=%s WHERE id_field=%s and id_tag=%s", (res2[0][0], fldID, id_1))
res = run_sql("UPDATE field_tag SET score=%s WHERE id_field=%s and id_tag=%s", (res1[0][0], fldID, id_2))
return (1, "")
except StandardError as e:
return (0, e)
def get_lang_list(table, field, id):
langs = run_sql("SELECT ln FROM %s WHERE %s=%%s" % (wash_table_column_name(table), wash_table_column_name(field)), (id, )) # kwalitee: disable=sql
exists = {}
lang = ''
for lng in langs:
if lng[0] not in exists:
lang += lng[0] + ", "
exists[lng[0]] = 1
if lang.endswith(", "):
lang = lang [:-2]
if len(exists) == 0:
lang = """<b><span class="info">None</span></b>"""
return lang
def check_user(req, role, adminarea=2, authorized=0):
# FIXME: Add doctype.
# This function is similar to the one found in
# oairepository/lib/oai_repository_admin.py, bibrank/lib/bibrankadminlib.py and
# websubmit/lib/websubmitadmin_engine.py.
auth_code, auth_message = acc_authorize_action(req, role)
if not authorized and auth_code != 0:
return ("false", auth_message)
return ("", auth_message)
diff --git a/invenio/legacy/bibindex/doc/admin/bibindex-admin-guide.webdoc b/invenio/legacy/bibindex/doc/admin/bibindex-admin-guide.webdoc
index 91e4d3515..edfd02f1a 100644
--- a/invenio/legacy/bibindex/doc/admin/bibindex-admin-guide.webdoc
+++ b/invenio/legacy/bibindex/doc/admin/bibindex-admin-guide.webdoc
@@ -1,332 +1,359 @@
## -*- mode: html; coding: utf-8; -*-
## This file is part of Invenio.
-## Copyright (C) 2007, 2008, 2009, 2010, 2011 CERN.
+## Copyright (C) 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
<!-- WebDoc-Page-Title: BibIndex Admin Guide -->
<!-- WebDoc-Page-Navtrail: <a class="navtrail" href="<CFG_SITE_URL>/help/admin<lang:link/>">_(Admin Area)_</a> -->
<!-- WebDoc-Page-Revision: $Id$ -->
<p><table class="errorbox">
<thead>
<tr>
<th class="errorboxheader">
WARNING: BIBINDEX ADMIN GUIDE IS UNDER DEVELOPMENT
</th>
</tr>
</thead>
<tbody>
<tr>
<td class="errorboxbody">
BibIndex Admin Guide is not yet completed. Most of admin-level
functionality for BibIndex exists only in commandline mode. We
are in the process of developing both the guide as well as the
web admin interface. If you are interested in seeing some
specific things implemented with high priority, please contact us
at <CFG_SITE_SUPPORT_EMAIL>. Thanks for your interest!
</td>
</tr>
</tbody>
</table></p>
<h2>Contents</h2>
<strong>1.<a href="#1">Overview</a></strong><br/>
<strong>2. <a href="#2">Configure Metadata Tags and Fields</a></strong><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2.1 <a href="#2.1">Configure Physical MARC Tags</a><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2.2 <a href="#2.2">Configure Logical Fields</a><br/>
-<strong>3. <a href="#3">Configure Word/Phrase Indexes</a></strong><br/>
+<strong>3. <a href="#3">Configure Normal Word/Phrase Indexes</a></strong><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.1 <a href="#3.1">Define New Index</a><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.2 <a href="#3.2">Configure Word-Breaking Procedure</a><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.3 <a href="#3.3">Configure Stopwords List</a><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.4 <a href="#3.4">Configure Stemming</a><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.5 <a href="#3.5">Configure Word Length</a><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.6 <a href="#3.6">Configure Removal of HTML Code</a><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.7 <a href="#3.7">Configure Accent Stripping</a><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.8 <a href="#3.8">Configure Fulltext Indexing</a><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.8.1 <a href="#3.8.1">Configure Solr Fulltext Indexing</a><br/>
-<strong>4. <a href="#4">Run BibIndex Daemon</a></strong><br/>
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4.1 <a href="#4.1">Run BibIndex daemon</a><br/>
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4.2 <a href="#4.2">Checking and repairing indexes</a><br/>
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4.3 <a href="#4.3">Reindexing</a><br/>
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4.4 <a href="#4.4">Solr fulltext indexing</a><br/>
+<strong>4. <a href="#4">Configure Virtual Word/Phrase Indexes</a></strong><br/>
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4.1 <a href="#4.1">Define New Virtual Index</a><br/>
+<strong>5. <a href="#5">Run BibIndex Daemon</a></strong><br/>
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5.1 <a href="#5.1">Run BibIndex daemon</a><br/>
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5.2 <a href="#5.2">Checking and repairing indexes</a><br/>
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5.3 <a href="#5.3">Reindexing</a><br/>
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5.4 <a href="#5.4">Solr fulltext indexing</a><br/>
<a name="1"></a><h2>1. Overview</h2>
<a name="2"></a><h2>2. Configure Metadata Tags and Fields</h2>
<a name="2.1"></a><h3>2.1 Configure Physical MARC Tags</h3>
<a name="2.2"></a><h3>2.2 Configure Logical Fields</h3>
<a name="3"></a><h2>3. Configure Word/Phrase Indexes</h2>
<a name="3.1"></a><h3>3.1 Define New Index</h3>
<p>To define a new index you must first give the index a internal
name. An empty index is then created by preparing the database tables.</p>
<p>Before the index can be used for searching, the fields that should be
included in the index must be selected.</p>
<p>When desired to fill the index based on the fields selected, you can
schedule the update by running <b>bibindex -w indexname</b> together
with other desired parameters.</p>
<a name="3.2"></a><h3>3.2 Configure Word-Breaking Procedure</h3>
<p>Can be configured by changing
<b>CFG_BIBINDEX_CHARS_ALPHANUMERIC_SEPARATORS</b> and
<b>CFG_BIBINDEX_CHARS_PUNCTUATION</b> in the general config file.</p>
<p>How the words are broken up defines what is added to the index. Should
only "director-general" be added, or should "director", "general" and
"director-general" be added? The index can vary between 300 000 and 3
000 000 terms based the policy for breaking words.</p>
<a name="3.3"></a><h3>3.3 Configure Stopwords List</h3>
<p>BibIndex supports stopword removal by not adding words which exists in
a given stopword list to the index. Stopword removal makes the index
smaller by removing much used words.</p>
<p>Which stopword list that should be used can be configured in the
general config file file by changing the value of the variable
CFG_BIBINDEX_PATH_TO_STOPWORDS_FILE. If no stopword list should be
used, the value should be 0.</p>
<a name="3.4"></a><h3>3.4 Configure stemming</h3>
<p>The BibIndex indexer supports stemming, removing the ending of
words thus creating a smaller indexer. For example, using English, the
word "information" will be stemmed to "inform"; or "looking", "looks",
and "looked" will be all stemmed to "look", thus giving more hits to each
word.</p>
<p>Currently you can configure the stemming language on a per-index basis. All
searches referring a stemmed index will also be stemmed based on the same
language.</p>
<a name="3.5"></a><h3>3.5 Configure Word Length</h3>
<p>By setting the value of <b>CFG_BIBINDEX_MIN_WORD_LENGTH</b> in the
general config file higher than 0, only words with the number of
characters higher than this will be added to the index.</p>
<a name="3.6"></a><h3>3.6 Configure Removal of HTML and LaTeX Code</h3>
<p>If you set the <b>Remove HTML Markup</b> parameter in the admin interface
to 'Yes' the indexer will try to remove all HTML code
from documents before indexing, and index only the text left. (HTML
code is defined as everything between '&lt;' and '>' in a text.)</p>
<p>If you set the <b>Remove LATEX Markup</b> parameter in the admin interface
to 'Yes', the indexer will try to remove all LaTeX code
from documents before indexing, and index only the text left. (LaTeX
code is defined as everything between '\command{' and '}' in a text, or
'{\command ' and '}').</p>
<a name="3.7"></a><h3>3.7 Configure Accent Stripping</h3>
<a name="3.8"></a><h3>3.8 Configure Fulltext Indexing</h3>
<p>The metadata tags are usually indexed by its content. There are
special cases however, such as the fulltext indexing. In this case
the tag contains an URL to the fulltext material and we would like to
fetch this material and index words found in this material rather than
in the metadata itself. This is possible via special tag assignement
via <code>tagToWordsFunctions</code> variable.</p>
<p>The default setup is configured in the way that if the indexer sees
that it has to index tag <code>8564_u</code>, it switches into the
fulltext indexing mode described above. It can index locally stored
files or even fetch them from external URLs, depending on the value of
the <b>CFG_BIBINDEX_FULLTEXT_INDEX_LOCAL_FILES_ONLY</b> configuration
variable. When fetching files from remote URLs, when it ends on a
splash page (an intermediate page before getting to fulltext file
itself), it can find and follow any further links to fulltext files.</p>
<p>The default setup also differentiate between metadata and fulltext
indexing, so that <code>any field</code> index does process only
metadata, not fulltext. If you want to have the fulltext indexed
together with the metadata, so that both are searched by default, you
can go to BibIndex Admin interface and in the Manage Logical Fields
explicitly add the tag <code>8564_u</code> under <code>any
field</code> field.</p>
<a name="3.8.1"></a><h3>3.8.1 Configure Solr Fulltext Indexing</h3>
<p>Solr can be used to index fulltext and to serve fulltext queries. To use it, the following steps are necessary:</p>
<p>First, Solr is installed:</p>
<blockquote>
<pre>
$ cd &lt;invenio source tree&gt;
$ sudo make install-solrutils
</pre>
</blockquote>
<p>Second, <code>invenio-local.conf</code> is amended:</p>
<blockquote>
<pre>
CFG_SOLR_URL = http://localhost:8983/solr
</pre>
</blockquote>
<p>Third, Solr is set to index fulltext:</p>
<blockquote>
<pre>
UPDATE idxINDEX SET indexer='SOLR' WHERE name='fulltext'
</pre>
</blockquote>
<p>Fourth, Solr is started:</p>
<blockquote>
<pre>
&lt;invenio installation&gt;/lib/apache-solr-3.1.0/example$ sudo -u www-data java -jar start.jar
</pre>
</blockquote>
-<a name="4"></a><h2>4. Run BibIndex Daemon</h2>
+<a name="4"></a><h2>4. Configure Virtual Word/Phrase Indexes</h2>
-<a name="4.1"></a><h3>4.1 Run BibIndex daemon</h3>
+<a name="4.1"></a><h3>4.1 Define New Virtual Index</h3>
+
+<p>An index can be <em>virtual</em> in which case it is composed of
+other <em>normal</em> indexes that were described in the preceding
+section. For example, the <em>global</em> index is by default virtual
+and it is composed of several normal indexes such
+as <em>title</em>, <em>author</em>, <em>abstract</em>, etc.</p>
+
+<p>When indexing a virtual index, the indexer does not tokenises terms
+anew as is the case for normal index. Rather, the indexer simply
+collects terms that were generated previously when dependent normal
+indexes were running. When a normal index run, say title, it created
+list of terms from title fields and it updated not only its own title
+index, but it also submitted its terms to the overall global index's
+processing queue. When the global index process runs, it simply
+processes its incoming queue and updates its term list, without any
+record lookups.</p>
+
+<p>In this manner, a site can run several parallel indexing processes
+for independent normal indexes, and several parallel indexing
+processes for various virtual indexes, taking full advantage of
+multi-core multi-node architecture.</p>
+
+<a name="5"></a><h2>5. Run BibIndex Daemon</h2>
+
+<a name="5.1"></a><h3>5.1 Run BibIndex daemon</h3>
<p>To index your newly created or modified documents, bibindex must be
run periodically via bibsched. This is achieved by the sleep option
(-s) to bibindex. For more information please see <a
href="howto-run">HOWTO Run</a> admin guide.</p>
-<a name="4.2"></a><h3>4.2 Checking and repairing indexes</h3>
+<a name="5.2"></a><h3>5.2 Checking and repairing indexes</h3>
<p>Upon each indexing run, bibindex checks and reports any
inconsistencies in the indexes. You can also manually check for the
index corruption yourself by using the check (-k) option to bibindex.</p>
<p>If a problem is found during the check, bibindex hints you to run
repairing (-r). If you run it, then during repair bibindex tries to
correct problems automatically by its own means. Usually it succeeds.</p>
<p>When the automatic repairing does not succeed though, then manual
intervention is required. The easiest thing to get the indexes back
to shape are commands like: (assuming the problem is with the index ID
1):
<blockquote>
<pre>
$ echo "DELETE FROM idxWORD01R WHERE type='TEMPORARY' or type='FUTURE';" | \
/opt/invenio/bin/dbexec
</pre>
</blockquote>
to leave only the 'CURRENT' reverse index. After that you can rerun
the index checking procedure (-k) and, if successful, continue with
the normal web site operation. However, a full reindexing should be
scheduled for the forthcoming night or weekend.</p>
-<a name="4.3"></a><h3>4.3 Reindexing</h3>
+<a name="5.3"></a><h3>5.3 Reindexing</h3>
<p>The procedure of reindexing is taking place into the real indexes that
are also used for searching. Therefore the end users will feel
immediately any change in the indexes. If you need to reindex your
records from scratch, then the best procedure is the following: reindex
the collection index only (fast operation), recreate collection cache,
and only after that reindex all the other indexes (slow operation).
This will ensure that the records in your system will be at least
browsable while the indexes are being rebuilt. The steps to perform are:</p>
<p>First we reindex the collection index:
<blockquote>
<pre>
$ bibindex --reindex -f50000 -wcollection # reindex the collection index (fast)
$ echo "UPDATE collection SET reclist=NULL;" | \
/opt/invenio/bin/dbexec # clean collection cache
$ webcoll -f # recreate the collection cache
$ bibsched # run the two above-submitted tasks
$ sudo apachectl restart
</pre>
</blockquote></p>
<p>Then we launch (slower) reindexing of the remaining indexes:
<blockquote>
<pre>
$ bibindex --reindex -f50000 # reindex other indexes (slow)
$ webcoll -f
$ bibsched # run the two above-submitted tasks, and put the queue back in auto mode
$ sudo apachectl restart
</pre>
</blockquote></p>
<p>You may optionally want to reindex the word ranking tables:
<blockquote>
<pre>
$ bibsched # wait for all active tasks to finish, and put the queue into manual mode
$ cd invenio-0.92.1 # source dir
$ grep rnkWORD ./modules/miscutil/sql/tabbibclean.sql | \
/opt/invenio/bin/dbexec # truncate rank indexes
$ echo "UPDATE rnkMETHOD SET last_updated='0000-00-00 00:00:00';" | \
/opt/invenio/bin/dbexec # rewind the last ranking time
</pre>
</blockquote></p>
<p>Secondly, if you have been using custom ranking methods using new
rnkWORD* tables (most probably you have not), you would have to
truncate them too:
<blockquote>
<pre>
&nbsp; # find out which custom ranking indexes were added:
&nbsp; $ echo "SELECT id FROM rnkMETHOD" | /opt/invenio/bin/dbexec
&nbsp; id
&nbsp; 66
&nbsp; 67
&nbsp; [...]
&nbsp;
&nbsp; # for every ranking index id, truncate corresponding ranking tables:
&nbsp; $ echo "TRUNCATE rnkWORD66F" | /opt/invenio/bin/dbexec
&nbsp; $ echo "TRUNCATE rnkWORD66R" | /opt/invenio/bin/dbexec
&nbsp; $ echo "TRUNCATE rnkWORD67F" | /opt/invenio/bin/dbexec
&nbsp; $ echo "TRUNCATE rnkWORD67R" | /opt/invenio/bin/dbexec
</pre>
</blockquote></p>
<p>At last, we launch reindexing of the ranking indexes:
<blockquote>
<pre>
$ bibrank -f50000
$ bibsched # run the three above-submitted tasks, and put the queue back in auto mode
$ sudo apachectl restart
</pre>
</blockquote>
and we are done.</p>
<p>In the future Invenio should ideally run indexing into
invisible tables that would be switched against the production ones
once the indexing process is successfully over. For the time being,
if reindexing takes several hours in your installation (e.g. if you
have 1,000,000 records), you may want to mysqlhotcopy your tables and
run reindexing on those copies yourself.</p>
-<a name="4.4"></a><h3>4.4 Solr fulltext indexing</h3>
+<a name="5.4"></a><h3>5.4 Solr fulltext indexing</h3>
<p>If Solr is used for both fulltext and ranking, only the <code>BibRank</code>
daemon shall run. Since Solr documents can only be overriden and not updated, the
<code>BibRank</code> daemon also indexes fulltext.</p>
diff --git a/invenio/legacy/bibindex/engine.py b/invenio/legacy/bibindex/engine.py
index 50ce851fc..1a31186ab 100644
--- a/invenio/legacy/bibindex/engine.py
+++ b/invenio/legacy/bibindex/engine.py
@@ -1,1964 +1,2400 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
+## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009,
+## 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
"""
-BibIndex indexing engine implementation. See bibindex executable for entry point.
+BibIndex indexing engine implementation.
+See bibindex executable for entry point.
"""
__revision__ = "$Id$"
import re
import sys
import time
import fnmatch
+import inspect
from datetime import datetime
from six import iteritems
-from time import strptime
from invenio.config import CFG_SOLR_URL
from invenio.legacy.bibindex.engine_config import CFG_MAX_MYSQL_THREADS, \
CFG_MYSQL_THREAD_TIMEOUT, \
CFG_CHECK_MYSQL_THREADS, \
CFG_BIBINDEX_COLUMN_VALUE_SEPARATOR, \
CFG_BIBINDEX_INDEX_TABLE_TYPE, \
CFG_BIBINDEX_ADDING_RECORDS_STARTED_STR, \
- CFG_BIBINDEX_UPDATE_MESSAGE
+ CFG_BIBINDEX_UPDATE_MESSAGE, \
+ CFG_BIBINDEX_UPDATE_MODE, \
+ CFG_BIBINDEX_TOKENIZER_TYPE, \
+ CFG_BIBINDEX_WASH_INDEX_TERMS, \
+ CFG_BIBINDEX_SPECIAL_TAGS
from invenio.legacy.bibauthority.config import \
- CFG_BIBAUTHORITY_CONTROLLED_FIELDS_BIBLIOGRAPHIC, \
- CFG_BIBAUTHORITY_RECORD_CONTROL_NUMBER_FIELD
+ CFG_BIBAUTHORITY_CONTROLLED_FIELDS_BIBLIOGRAPHIC
from invenio.legacy.bibauthority.engine import get_index_strings_by_control_no,\
get_control_nos_from_recID
from invenio.legacy.bibindex.adminlib import get_idx_remove_html_markup, \
get_idx_remove_latex_markup, \
get_idx_remove_stopwords
from invenio.legacy.bibdocfile.api import BibRecDocs
from invenio.legacy.search_engine import perform_request_search, \
get_index_stemming_language, \
get_synonym_terms, \
search_pattern, \
search_unit_in_bibrec
from invenio.legacy.dbquery import run_sql, DatabaseError, serialize_via_marshal, \
deserialize_via_marshal, wash_table_column_name
from invenio.legacy.bibindex.engine_washer import wash_index_term
from invenio.legacy.bibsched.bibtask import task_init, write_message, get_datetime, \
task_set_option, task_get_option, task_get_task_param, \
task_update_progress, task_sleep_now_if_required
from intbitset import intbitset
from invenio.ext.logging import register_exception
from invenio.legacy.bibrank.adminlib import get_def_name
from invenio.legacy.miscutil.solrutils_bibindex_indexer import solr_commit
from invenio.modules.indexer.tokenizers.BibIndexJournalTokenizer import \
CFG_JOURNAL_TAG, \
CFG_JOURNAL_PUBINFO_STANDARD_FORM, \
CFG_JOURNAL_PUBINFO_STANDARD_FORM_REGEXP_CHECK
from invenio.legacy.bibindex.engine_utils import load_tokenizers, \
get_all_index_names_and_column_values, \
- get_idx_indexer, \
get_index_tags, \
- get_field_tags, \
get_tag_indexes, \
get_all_indexes, \
- get_all_virtual_indexes, \
get_index_virtual_indexes, \
- is_index_virtual, \
get_virtual_index_building_blocks, \
get_index_id_from_index_name, \
- get_index_name_from_index_id, \
run_sql_drop_silently, \
get_min_last_updated, \
- remove_inexistent_indexes
+ remove_inexistent_indexes, \
+ filter_for_virtual_indexes, \
+ get_records_range_for_index, \
+ make_prefix, \
+ UnknownTokenizer
from invenio.legacy.bibrecord import get_fieldvalues
from invenio.modules.records.api import get_record
from invenio.utils.memoise import Memoise
## precompile some often-used regexp for speed reasons:
re_subfields = re.compile('\$\$\w')
re_datetime_shift = re.compile("([-\+]{0,1})([\d]+)([dhms])")
-
+re_prefix = re.compile('__[a-zA-Z1-9]*__')
nb_char_in_line = 50 # for verbose pretty printing
chunksize = 1000 # default size of chunks that the records will be treated by
base_process_size = 4500 # process base size
_last_word_table = None
_TOKENIZERS = load_tokenizers()
def list_union(list1, list2):
"Returns union of the two lists."
union_dict = {}
for e in list1:
union_dict[e] = 1
for e in list2:
union_dict[e] = 1
return union_dict.keys()
+
def list_unique(_list):
"""Returns a _list with duplicates removed."""
_dict = {}
for e in _list:
_dict[e] = 1
return _dict.keys()
## safety function for killing slow DB threads:
-def kill_sleepy_mysql_threads(max_threads=CFG_MAX_MYSQL_THREADS, thread_timeout=CFG_MYSQL_THREAD_TIMEOUT):
+def kill_sleepy_mysql_threads(max_threads=CFG_MAX_MYSQL_THREADS,
+ thread_timeout=CFG_MYSQL_THREAD_TIMEOUT):
"""Check the number of DB threads and if there are more than
MAX_THREADS of them, lill all threads that are in a sleeping
state for more than THREAD_TIMEOUT seconds. (This is useful
for working around the the max_connection problem that appears
during indexation in some not-yet-understood cases.) If some
threads are to be killed, write info into the log file.
"""
res = run_sql("SHOW FULL PROCESSLIST")
if len(res) > max_threads:
for row in res:
r_id, dummy, dummy, dummy, r_command, r_time, dummy, dummy = row
if r_command == "Sleep" and int(r_time) > thread_timeout:
- run_sql("KILL %s", (r_id,))
- write_message("WARNING: too many DB threads, killing thread %s" % r_id, verbose=1)
+ run_sql("KILL %s", (r_id, ))
+ write_message("WARNING: too many DB threads, " + \
+ "killing thread %s" % r_id, verbose=1)
return
+
def get_associated_subfield_value(recID, tag, value, associated_subfield_code):
"""Return list of ASSOCIATED_SUBFIELD_CODE, if exists, for record
RECID and TAG of value VALUE. Used by fulltext indexer only.
Note: TAG must be 6 characters long (tag+ind1+ind2+sfcode),
otherwise en empty string is returned.
FIXME: what if many tag values have the same value but different
associated_subfield_code? Better use bibrecord library for this.
"""
out = ""
if len(tag) != 6:
return out
bibXXx = "bib" + tag[0] + tag[1] + "x"
bibrec_bibXXx = "bibrec_" + bibXXx
query = """SELECT bb.field_number, b.tag, b.value FROM %s AS b, %s AS bb
WHERE bb.id_bibrec=%%s AND bb.id_bibxxx=b.id AND tag LIKE
%%s%%""" % (bibXXx, bibrec_bibXXx)
res = run_sql(query, (recID, tag[:-1]))
field_number = -1
for row in res:
if row[1] == tag and row[2] == value:
field_number = row[0]
if field_number > 0:
for row in res:
if row[0] == field_number and row[1] == tag[:-1] + associated_subfield_code:
out = row[2]
break
return out
def get_author_canonical_ids_for_recid(recID):
"""
Return list of author canonical IDs (e.g. `J.Ellis.1') for the
given record. Done by consulting BibAuthorID module.
"""
from invenio.legacy.bibauthorid.dbinterface import get_data_of_papers
lwords = []
res = get_data_of_papers([recID])
if res is None:
## BibAuthorID is not enabled
return lwords
else:
dpersons, dpersoninfos = res
for aid in dpersoninfos.keys():
author_canonical_id = dpersoninfos[aid].get('canonical_id', '')
if author_canonical_id:
lwords.append(author_canonical_id)
return lwords
+
def swap_temporary_reindex_tables(index_id, reindex_prefix="tmp_"):
"""Atomically swap reindexed temporary table with the original one.
Delete the now-old one."""
- is_virtual = is_index_virtual(index_id)
- if is_virtual:
- write_message("Removing %s index tables for id %s" % (reindex_prefix, index_id))
- query = """DROP TABLE IF EXISTS %%sidxWORD%02dR, %%sidxWORD%02dF,
- %%sidxPAIR%02dR, %%sidxPAIR%02dF,
- %%sidxPHRASE%02dR, %%sidxPHRASE%02dF
- """ % ((index_id,)*6)
- query = query % ((reindex_prefix,)*6)
- run_sql(query)
- else:
- write_message("Putting new tmp index tables for id %s into production" % index_id)
- run_sql(
- "RENAME TABLE " +
- "idxWORD%02dR TO old_idxWORD%02dR," % (index_id, index_id) +
- "%sidxWORD%02dR TO idxWORD%02dR," % (reindex_prefix, index_id, index_id) +
- "idxWORD%02dF TO old_idxWORD%02dF," % (index_id, index_id) +
- "%sidxWORD%02dF TO idxWORD%02dF," % (reindex_prefix, index_id, index_id) +
- "idxPAIR%02dR TO old_idxPAIR%02dR," % (index_id, index_id) +
- "%sidxPAIR%02dR TO idxPAIR%02dR," % (reindex_prefix, index_id, index_id) +
- "idxPAIR%02dF TO old_idxPAIR%02dF," % (index_id, index_id) +
- "%sidxPAIR%02dF TO idxPAIR%02dF," % (reindex_prefix, index_id, index_id) +
- "idxPHRASE%02dR TO old_idxPHRASE%02dR," % (index_id, index_id) +
- "%sidxPHRASE%02dR TO idxPHRASE%02dR," % (reindex_prefix, index_id, index_id) +
- "idxPHRASE%02dF TO old_idxPHRASE%02dF," % (index_id, index_id) +
- "%sidxPHRASE%02dF TO idxPHRASE%02dF;" % (reindex_prefix, index_id, index_id)
- )
- write_message("Dropping old index tables for id %s" % index_id)
- run_sql_drop_silently("DROP TABLE old_idxWORD%02dR, old_idxWORD%02dF, old_idxPAIR%02dR, old_idxPAIR%02dF, old_idxPHRASE%02dR, old_idxPHRASE%02dF" % (index_id, index_id, index_id, index_id, index_id, index_id)) # kwalitee: disable=sql
+ write_message("Putting new tmp index tables " + \
+ "for id %s into production" % index_id)
+ run_sql(
+ "RENAME TABLE " +
+ "idxWORD%02dR TO old_idxWORD%02dR," % (index_id, index_id) +
+ "%sidxWORD%02dR TO idxWORD%02dR," % (reindex_prefix, index_id, index_id) +
+ "idxWORD%02dF TO old_idxWORD%02dF," % (index_id, index_id) +
+ "%sidxWORD%02dF TO idxWORD%02dF," % (reindex_prefix, index_id, index_id) +
+ "idxPAIR%02dR TO old_idxPAIR%02dR," % (index_id, index_id) +
+ "%sidxPAIR%02dR TO idxPAIR%02dR," % (reindex_prefix, index_id, index_id) +
+ "idxPAIR%02dF TO old_idxPAIR%02dF," % (index_id, index_id) +
+ "%sidxPAIR%02dF TO idxPAIR%02dF," % (reindex_prefix, index_id, index_id) +
+ "idxPHRASE%02dR TO old_idxPHRASE%02dR," % (index_id, index_id) +
+ "%sidxPHRASE%02dR TO idxPHRASE%02dR," % (reindex_prefix, index_id, index_id) +
+ "idxPHRASE%02dF TO old_idxPHRASE%02dF," % (index_id, index_id) +
+ "%sidxPHRASE%02dF TO idxPHRASE%02dF;" % (reindex_prefix, index_id, index_id)
+ )
+ write_message("Dropping old index tables for id %s" % index_id)
+ run_sql_drop_silently("""DROP TABLE old_idxWORD%02dR,
+ old_idxWORD%02dF,
+ old_idxPAIR%02dR,
+ old_idxPAIR%02dF,
+ old_idxPHRASE%02dR,
+ old_idxPHRASE%02dF""" % ((index_id, )* 6)
+ ) # kwalitee: disable=sql
def init_temporary_reindex_tables(index_id, reindex_prefix="tmp_"):
"""Create reindexing temporary tables."""
write_message("Creating new tmp index tables for id %s" % index_id)
- run_sql_drop_silently("""DROP TABLE IF EXISTS %sidxWORD%02dF""" % (wash_table_column_name(reindex_prefix), index_id)) # kwalitee: disable=sql
+
+ query = """DROP TABLE IF EXISTS %sidxWORD%02dF""" % \
+ (wash_table_column_name(reindex_prefix), index_id)
+ run_sql_drop_silently(query) # kwalitee: disable=sql
+
run_sql("""CREATE TABLE %sidxWORD%02dF (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM""" % (reindex_prefix, index_id))
- run_sql_drop_silently("""DROP TABLE IF EXISTS %sidxWORD%02dR""" % (wash_table_column_name(reindex_prefix), index_id)) # kwalitee: disable=sql
+ query = """DROP TABLE IF EXISTS %sidxWORD%02dR""" % \
+ (wash_table_column_name(reindex_prefix), index_id)
+ run_sql_drop_silently(query) # kwalitee: disable=sql
+
run_sql("""CREATE TABLE %sidxWORD%02dR (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM""" % (reindex_prefix, index_id))
- run_sql_drop_silently("""DROP TABLE IF EXISTS %sidxPAIR%02dF""" % (wash_table_column_name(reindex_prefix), index_id)) # kwalitee: disable=sql
+ query = """DROP TABLE IF EXISTS %sidxPAIR%02dF""" % \
+ (wash_table_column_name(reindex_prefix), index_id)
+ run_sql_drop_silently(query) # kwalitee: disable=sql
+
run_sql("""CREATE TABLE %sidxPAIR%02dF (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM""" % (reindex_prefix, index_id))
- run_sql_drop_silently("""DROP TABLE IF EXISTS %sidxPAIR%02dR""" % (wash_table_column_name(reindex_prefix), index_id)) # kwalitee: disable=sql
+ query = """DROP TABLE IF EXISTS %sidxPAIR%02dR""" % \
+ (wash_table_column_name(reindex_prefix), index_id)
+ run_sql_drop_silently(query) # kwalitee: disable=sql
+
run_sql("""CREATE TABLE %sidxPAIR%02dR (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM""" % (reindex_prefix, index_id))
- run_sql_drop_silently("""DROP TABLE IF EXISTS %sidxPHRASE%02dF""" % (wash_table_column_name(reindex_prefix), index_id)) # kwalitee: disable=sql
+ query = """DROP TABLE IF EXISTS %sidxPHRASE%02dF""" % \
+ (wash_table_column_name(reindex_prefix), index_id)
+ run_sql_drop_silently(query) # kwalitee: disable=sql
+
run_sql("""CREATE TABLE %sidxPHRASE%02dF (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM""" % (reindex_prefix, index_id))
- run_sql_drop_silently("""DROP TABLE IF EXISTS %sidxPHRASE%02dR""" % (wash_table_column_name(reindex_prefix), index_id)) # kwalitee: disable=sql
+ query = """DROP TABLE IF EXISTS %sidxPHRASE%02dR""" % \
+ (wash_table_column_name(reindex_prefix), index_id)
+ run_sql_drop_silently(query) # kwalitee: disable=sql
+
run_sql("""CREATE TABLE %sidxPHRASE%02dR (
id_bibrec mediumint(9) unsigned NOT NULL default '0',
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM""" % (reindex_prefix, index_id))
def remove_subfields(s):
"Removes subfields from string, e.g. 'foo $$c bar' becomes 'foo bar'."
return re_subfields.sub(' ', s)
def get_field_indexes(field):
"""Returns indexes names and ids corresponding to the given field"""
if field[0:3].isdigit():
#field is actually a tag
return get_tag_indexes(field, virtual=False)
else:
#future implemeptation for fields
return []
get_field_indexes_memoised = Memoise(get_field_indexes)
def get_all_synonym_knowledge_bases():
- """Returns a dictionary of name key and knowledge base name and match type tuple value
- information of all defined words indexes that have knowledge base information.
+ """
+ Returns a dictionary of name key and knowledge base name
+ and match type tuple value information of all defined words indexes
+ that have knowledge base information.
Returns empty dictionary in case there are no tags indexed.
- Example: output['global'] = ('INDEX-SYNONYM-TITLE', 'exact'), output['title'] = ('INDEX-SYNONYM-TITLE', 'exact')."""
+ Example: output['global'] = ('INDEX-SYNONYM-TITLE', 'exact'),
+ output['title'] = ('INDEX-SYNONYM-TITLE', 'exact').
+ """
res = get_all_index_names_and_column_values("synonym_kbrs")
out = {}
for row in res:
kb_data = row[1]
# ignore empty strings
if len(kb_data):
out[row[0]] = tuple(kb_data.split(CFG_BIBINDEX_COLUMN_VALUE_SEPARATOR))
return out
def get_index_remove_stopwords(index_id):
"""Returns value of a remove_stopword field from idxINDEX database table
if it's not 'No'. If it's 'No' returns False.
Just for consistency with WordTable.
@param index_id: id of the index
"""
result = get_idx_remove_stopwords(index_id)
if isinstance(result, tuple):
return False
if result == 'No' or result == '':
return False
return result
def get_index_remove_html_markup(index_id):
""" Gets remove_html_markup parameter from database ('Yes' or 'No') and
changes it to True, False.
Just for consistency with WordTable."""
result = get_idx_remove_html_markup(index_id)
if result == 'Yes':
return True
return False
def get_index_remove_latex_markup(index_id):
""" Gets remove_latex_markup parameter from database ('Yes' or 'No') and
changes it to True, False.
Just for consistency with WordTable."""
result = get_idx_remove_latex_markup(index_id)
if result == 'Yes':
return True
return False
def get_index_tokenizer(index_id):
"""Returns value of a tokenizer field from idxINDEX database table
@param index_id: id of the index
"""
query = "SELECT tokenizer FROM idxINDEX WHERE id=%s" % index_id
out = None
try:
res = run_sql(query)
if res:
out = _TOKENIZERS[res[0][0]]
except DatabaseError:
- write_message("Exception caught for SQL statement: %s; column tokenizer might not exist" % query, sys.stderr)
+ write_message("Exception caught for SQL statement: %s; " + \
+ "column tokenizer might not exist" % query, sys.stderr)
except KeyError:
write_message("Exception caught: there is no such tokenizer")
out = None
return out
+def detect_tokenizer_type(tokenizer):
+ """
+ Checks what is the main type of the tokenizer.
+ For more information on tokenizer types take
+ a look at BibIndexTokenizer class.
+ @param tokenizer: instance of a tokenizer
+ """
+ from invenio.bibindex_tokenizers.BibIndexStringTokenizer import BibIndexStringTokenizer
+ from invenio.bibindex_tokenizers.BibIndexRecJsonTokenizer import BibIndexRecJsonTokenizer
+ from invenio.bibindex_tokenizers.BibIndexMultiFieldTokenizer import BibIndexMultiFieldTokenizer
+
+ tokenizer_inheritance_tree = inspect.getmro(tokenizer.__class__)
+ if BibIndexStringTokenizer in tokenizer_inheritance_tree:
+ return CFG_BIBINDEX_TOKENIZER_TYPE['string']
+ if BibIndexMultiFieldTokenizer in tokenizer_inheritance_tree:
+ return CFG_BIBINDEX_TOKENIZER_TYPE['multifield']
+ if BibIndexRecJsonTokenizer in tokenizer_inheritance_tree:
+ return CFG_BIBINDEX_TOKENIZER_TYPE['recjson']
+ return CFG_BIBINDEX_TOKENIZER_TYPE['unknown']
+
+
def get_last_updated_all_indexes():
"""Returns last modification date for all defined indexes"""
query= """SELECT name, last_updated FROM idxINDEX"""
res = run_sql(query)
return res
def split_ranges(parse_string):
"""Parse a string a return the list or ranges."""
recIDs = []
ranges = parse_string.split(",")
for arange in ranges:
tmp_recIDs = arange.split("-")
if len(tmp_recIDs) == 1:
recIDs.append([int(tmp_recIDs[0]), int(tmp_recIDs[0])])
else:
if int(tmp_recIDs[0]) > int(tmp_recIDs[1]): # sanity check
tmp = tmp_recIDs[0]
tmp_recIDs[0] = tmp_recIDs[1]
tmp_recIDs[1] = tmp
recIDs.append([int(tmp_recIDs[0]), int(tmp_recIDs[1])])
return recIDs
+
def get_word_tables(tables):
""" Given a list of table names it return a list of tuples
(index_id, index_name, index_tags).
"""
wordTables = []
if tables:
for index in tables:
index_id = get_index_id_from_index_name(index)
if index_id:
wordTables.append((index_id, index, get_index_tags(index)))
else:
- write_message("Error: There is no %s words table." % index, sys.stderr)
+ write_message("Error: There is no %s words table." % \
+ index, sys.stderr)
return wordTables
+
def get_date_range(var):
"Returns the two dates contained as a low,high tuple"
limits = var.split(",")
if len(limits) == 1:
low = get_datetime(limits[0])
return low, None
if len(limits) == 2:
low = get_datetime(limits[0])
high = get_datetime(limits[1])
return low, high
return None, None
+
def create_range_list(res):
"""Creates a range list from a recID select query result contained
in res. The result is expected to have ascending numerical order."""
if not res:
return []
row = res[0]
if not row:
return []
else:
range_list = [[row, row]]
for row in res[1:]:
row_id = row
if row_id == range_list[-1][1] + 1:
range_list[-1][1] = row_id
else:
range_list.append([row_id, row_id])
return range_list
+
def beautify_range_list(range_list):
"""Returns a non overlapping, maximal range list"""
ret_list = []
for new in range_list:
found = 0
for old in ret_list:
if new[0] <= old[0] <= new[1] + 1 or new[0] - 1 <= old[1] <= new[1]:
old[0] = min(old[0], new[0])
old[1] = max(old[1], new[1])
found = 1
break
if not found:
ret_list.append(new)
return ret_list
def truncate_index_table(index_name):
"""Properly truncate the given index."""
index_id = get_index_id_from_index_name(index_name)
if index_id:
- write_message('Truncating %s index table in order to reindex.' % index_name, verbose=2)
- run_sql("UPDATE idxINDEX SET last_updated='0000-00-00 00:00:00' WHERE id=%s", (index_id,))
+ write_message('Truncating %s index table in order to reindex.' % \
+ index_name, verbose=2)
+ run_sql("""UPDATE idxINDEX SET last_updated='0000-00-00 00:00:00'
+ WHERE id=%s""", (index_id, ))
run_sql("TRUNCATE idxWORD%02dF" % index_id) # kwalitee: disable=sql
run_sql("TRUNCATE idxWORD%02dR" % index_id) # kwalitee: disable=sql
run_sql("TRUNCATE idxPHRASE%02dF" % index_id) # kwalitee: disable=sql
run_sql("TRUNCATE idxPHRASE%02dR" % index_id) # kwalitee: disable=sql
def update_index_last_updated(indexes, starting_time=None):
"""Update last_updated column of the index table in the database.
- Puts starting time there so that if the task was interrupted for record download,
+ Puts starting time there so that if the task
+ was interrupted for record download,
the records will be reindexed next time.
@param indexes: list of indexes names
"""
if starting_time is None:
return None
for index_name in indexes:
- write_message("updating last_updated to %s...for %s index" % (starting_time, index_name), verbose=9)
- run_sql("UPDATE idxINDEX SET last_updated=%s WHERE name=%s", (starting_time, index_name,))
+ write_message("updating last_updated to %s...for %s index" % \
+ (starting_time, index_name), verbose=9)
+ run_sql("UPDATE idxINDEX SET last_updated=%s WHERE name=%s",
+ (starting_time, index_name))
def get_percentage_completed(num_done, num_total):
""" Return a string containing the approx. percentage completed """
percentage_remaining = 100.0 * float(num_done) / float(num_total)
if percentage_remaining:
- percentage_display = "(%.1f%%)" % (percentage_remaining,)
+ percentage_display = "(%.1f%%)" % (percentage_remaining, )
else:
percentage_display = ""
return percentage_display
+
def _fill_dict_of_indexes_with_empty_sets():
"""find_affected_records internal function.
Creates dict: {'index_name1':set([]), ...}
"""
index_dict = {}
tmp_all_indexes = get_all_indexes(virtual=False)
for index in tmp_all_indexes:
index_dict[index] = set([])
return index_dict
+
def find_affected_records_for_index(indexes=[], recIDs=[], force_all_indexes=False):
"""
Function checks which records need to be changed/reindexed
for given index/indexes.
- Makes use of hstRECORD table where different revisions of record
- are kept.
- If parameter force_all_indexes is set function will assign all recIDs to all indexes.
+ Makes use of hstRECORD table where
+ different revisions of record are kept.
+ If parameter force_all_indexes is set
+ function will assign all recIDs to all indexes.
@param indexes: names of indexes for reindexation separated by coma
- @param recIDs: recIDs for reindexation in form: [[range1_down, range1_up],[range2_down, range2_up]..]
+ @param recIDs: recIDs for reindexation in form:
+ [[range1_down, range1_up],[range2_down, range2_up]..]
@param force_all_indexes: should we index all indexes?
"""
tmp_dates = dict(get_last_updated_all_indexes())
- modification_dates = dict([(date, tmp_dates[date] or datetime(1000,1,1,1,1,1)) for date in tmp_dates])
+ modification_dates = dict([(date, tmp_dates[date] or datetime(1000, 1, 1, 1, 1, 1))
+ for date in tmp_dates])
tmp_all_indexes = get_all_indexes(virtual=False)
indexes = remove_inexistent_indexes(indexes, leave_virtual=False)
if not indexes:
return {}
def _should_reindex_for_revision(index_name, revision_date):
try:
- if modification_dates[index_name] < revision_date and index_name in indexes:
+ if modification_dates[index_name] < revision_date and \
+ index_name in indexes:
return True
return False
except KeyError:
return False
if force_all_indexes:
records_for_indexes = {}
all_recIDs = []
for recIDs_range in recIDs:
all_recIDs.extend(range(recIDs_range[0], recIDs_range[1]+1))
for index in indexes:
records_for_indexes[index] = all_recIDs
return records_for_indexes
- min_last_updated = get_min_last_updated(indexes)[0][0] or datetime(1000,1,1,1,1,1)
+ min_last_updated = get_min_last_updated(indexes)[0][0] or \
+ datetime(1000, 1, 1, 1, 1, 1)
indexes_to_change = _fill_dict_of_indexes_with_empty_sets()
recIDs_info = []
for recIDs_range in recIDs:
- query = """SELECT id_bibrec,job_date,affected_fields FROM hstRECORD WHERE
- id_bibrec BETWEEN %s AND %s AND job_date > '%s'""" % (recIDs_range[0], recIDs_range[1], min_last_updated)
+ query = """SELECT id_bibrec,job_date,affected_fields FROM hstRECORD
+ WHERE id_bibrec BETWEEN %s AND %s AND
+ job_date > '%s'""" % \
+ (recIDs_range[0], recIDs_range[1], min_last_updated)
res = run_sql(query)
if res:
recIDs_info.extend(res)
for recID_info in recIDs_info:
- recID, revision, affected_fields = recID_info
+ recID, revision, affected_fields = recID_info
affected_fields = affected_fields.split(",")
indexes_for_recID = set()
for field in affected_fields:
if field:
field_indexes = get_field_indexes_memoised(field) or []
indexes_names = set([idx[1] for idx in field_indexes])
indexes_for_recID |= indexes_names
else:
- #record was inserted, all fields were changed, no specific affected fields
+ # record was inserted, all fields were changed,
+ # no specific affected fields
indexes_for_recID |= set(tmp_all_indexes)
indexes_for_recID_filtered = [ind for ind in indexes_for_recID if _should_reindex_for_revision(ind, revision)]
for index in indexes_for_recID_filtered:
indexes_to_change[index].add(recID)
indexes_to_change = dict((k, list(sorted(v))) for k, v in iteritems(indexes_to_change) if v)
-
return indexes_to_change
-#def update_text_extraction_date(first_recid, last_recid):
- #"""for all the bibdoc connected to the specified recid, set
- #the text_extraction_date to the task_starting_time."""
- #run_sql("UPDATE bibdoc JOIN bibrec_bibdoc ON id=id_bibdoc SET text_extraction_date=%s WHERE id_bibrec BETWEEN %s AND %s", (task_get_task_param('task_starting_time'), first_recid, last_recid))
+def chunk_generator(rng):
+ """
+ Splits one range into several smaller ones
+ with respect to global chunksize variable.
+ @param rng: range of records
+ @type rng: list in the form: [1, 2000]
+ """
+ global chunksize
+ current_low = rng[0]
+ current_high = rng[0]
+ if rng[0] == None or rng[1] == None:
+ raise StopIteration
+ if rng[1] - rng[0] + 1 <= chunksize:
+ yield rng
+ else:
+ while current_high - 1 < rng[1]:
+ current_high += chunksize
+ yield current_low, min(current_high - 1, rng[1])
+ current_low += chunksize
-class WordTable:
- "A class to hold the words table."
- def __init__(self, index_name, index_id, fields_to_index, table_name_pattern, wordtable_type, tag_to_tokenizer_map, wash_index_terms=50):
- """Creates words table instance.
- @param index_name: the index name
- @param index_id: the index integer identificator
- @param fields_to_index: a list of fields to index
- @param table_name_pattern: i.e. idxWORD%02dF or idxPHRASE%02dF
- @parm wordtable_type: type of the wordtable: Words, Pairs, Phrases
- @param tag_to_tokenizer_map: a mapping to specify particular tokenizer to
- extract words from particular metdata (such as 8564_u)
- @param wash_index_terms: do we wash index terms, and if yes (when >0),
- how many characters do we keep in the index terms; see
- max_char_length parameter of wash_index_term()
- """
+class AbstractIndexTable(object):
+ """
+ This class represents an index table in database.
+ An index consists of three different kinds of tables:
+ table which stores only words in db,
+ table which stores pairs of words and
+ table which stores whole phrases.
+ The class represents only one table. Another instance of
+ the class must be created in order to store different
+ type of terms.
+
+ This class is an abstract class. It contains methods
+ to connect to db and methods which facilitate
+ inserting/modifing/removing terms from it. The class
+ also contains methods which help managing the memory.
+ All specific methods for indexing can be found in corresponding
+ classes for virtual and regular indexes.
+ """
+
+ def __init__(self, index_name, table_type, table_prefix="", wash_index_terms=50):
self.index_name = index_name
- self.index_id = index_id
- self.tablename = table_name_pattern % index_id
- self.virtual_tablename_pattern = table_name_pattern[table_name_pattern.find('idx'):-1]
- self.humanname = get_def_name('%s' % (str(index_id),), "idxINDEX")[0][1]
- self.recIDs_in_mem = []
- self.fields_to_index = fields_to_index
- self.value = {}
- try:
- self.stemming_language = get_index_stemming_language(index_id)
- except KeyError:
- self.stemming_language = ''
- self.remove_stopwords = get_index_remove_stopwords(index_id)
- self.remove_html_markup = get_index_remove_html_markup(index_id)
- self.remove_latex_markup = get_index_remove_latex_markup(index_id)
- self.tokenizer = get_index_tokenizer(index_id)(self.stemming_language,
- self.remove_stopwords,
- self.remove_html_markup,
- self.remove_latex_markup)
- self.default_tokenizer_function = self.tokenizer.get_tokenizing_function(wordtable_type)
+ self.index_id = get_index_id_from_index_name(index_name)
+ self.table_type = table_type
self.wash_index_terms = wash_index_terms
- self.is_virtual = is_index_virtual(self.index_id)
- self.virtual_indexes = get_index_virtual_indexes(self.index_id)
-
- # tagToTokenizer mapping. It offers an indirection level necessary for
- # indexing fulltext.
- self.tag_to_words_fnc_map = {}
- for k in tag_to_tokenizer_map.keys():
- special_tokenizer_for_tag = _TOKENIZERS[tag_to_tokenizer_map[k]](self.stemming_language,
- self.remove_stopwords,
- self.remove_html_markup,
- self.remove_latex_markup)
- special_tokenizer_function = special_tokenizer_for_tag.get_tokenizing_function(wordtable_type)
- self.tag_to_words_fnc_map[k] = special_tokenizer_function
-
- if self.stemming_language and self.tablename.startswith('idxWORD'):
- write_message('%s has stemming enabled, language %s' % (self.tablename, self.stemming_language))
-
-
- def turn_off_virtual_indexes(self):
- self.virtual_indexes = []
-
- def turn_on_virtual_indexes(self):
- self.virtual_indexes = get_index_virtual_indexes(self.index_id)
+ self.table_name = wash_table_column_name(table_prefix + \
+ "idx" + \
+ table_type + \
+ ("%02d" % self.index_id) + "F")
+ self.table_prefix = table_prefix
- def get_field(self, recID, tag):
- """Returns list of values of the MARC-21 'tag' fields for the
- record 'recID'."""
-
- out = []
- bibXXx = "bib" + tag[0] + tag[1] + "x"
- bibrec_bibXXx = "bibrec_" + bibXXx
- query = """SELECT value FROM %s AS b, %s AS bb
- WHERE bb.id_bibrec=%%s AND bb.id_bibxxx=b.id
- AND tag LIKE %%s""" % (bibXXx, bibrec_bibXXx)
- res = run_sql(query, (recID, tag))
- for row in res:
- out.append(row[0])
- return out
-
- def clean(self):
- "Cleans the words table."
- self.value = {}
+ self.value = {} # cache
+ self.recIDs_in_mem = []
def put_into_db(self, mode="normal"):
"""Updates the current words table in the corresponding DB
idxFOO table. Mode 'normal' means normal execution,
mode 'emergency' means words index reverting to old state.
- """
- write_message("%s %s wordtable flush started" % (self.tablename, mode))
+ """
+ write_message("%s %s wordtable flush started" % \
+ (self.table_name, mode))
write_message('...updating %d words into %s started' % \
- (len(self.value), self.tablename))
- task_update_progress("(%s:%s) flushed %d/%d words" % (self.tablename, self.humanname, 0, len(self.value)))
+ (len(self.value), self.table_name))
+ task_update_progress("(%s:%s) flushed %d/%d words" % \
+ (self.table_name, self.index_name, 0, len(self.value)))
self.recIDs_in_mem = beautify_range_list(self.recIDs_in_mem)
- all_indexes = [(self.index_id, self.humanname)]
- if self.virtual_indexes:
- all_indexes.extend(self.virtual_indexes)
- for ind_id, ind_name in all_indexes:
- tab_name = self.tablename[:-1] + "R"
- if ind_id != self.index_id:
- tab_name = self.virtual_tablename_pattern % ind_id + "R"
- if mode == "normal":
- for group in self.recIDs_in_mem:
- query = """UPDATE %s SET type='TEMPORARY' WHERE id_bibrec
- BETWEEN %%s AND %%s AND type='CURRENT'""" % tab_name
- write_message(query % (group[0], group[1]), verbose=9)
- run_sql(query, (group[0], group[1]))
-
- nb_words_total = len(self.value)
- nb_words_report = int(nb_words_total / 10.0)
- nb_words_done = 0
- for word in self.value.keys():
- self.put_word_into_db(word, ind_id)
- nb_words_done += 1
- if nb_words_report != 0 and ((nb_words_done % nb_words_report) == 0):
- write_message('......processed %d/%d words' % (nb_words_done, nb_words_total))
- percentage_display = get_percentage_completed(nb_words_done, nb_words_total)
- task_update_progress("(%s:%s) flushed %d/%d words %s" % (tab_name, ind_name, nb_words_done, nb_words_total, percentage_display))
- write_message('...updating %d words into %s ended' % \
- (nb_words_total, tab_name))
-
- write_message('...updating reverse table %s started' % tab_name)
- if mode == "normal":
- for group in self.recIDs_in_mem:
- query = """UPDATE %s SET type='CURRENT' WHERE id_bibrec
- BETWEEN %%s AND %%s AND type='FUTURE'""" % tab_name
- write_message(query % (group[0], group[1]), verbose=9)
- run_sql(query, (group[0], group[1]))
- query = """DELETE FROM %s WHERE id_bibrec
- BETWEEN %%s AND %%s AND type='TEMPORARY'""" % tab_name
- write_message(query % (group[0], group[1]), verbose=9)
- run_sql(query, (group[0], group[1]))
- #if self.is_fulltext_index:
- #update_text_extraction_date(group[0], group[1])
- write_message('End of updating wordTable into %s' % tab_name, verbose=9)
- elif mode == "emergency":
- for group in self.recIDs_in_mem:
- query = """UPDATE %s SET type='CURRENT' WHERE id_bibrec
- BETWEEN %%s AND %%s AND type='TEMPORARY'""" % tab_name
- write_message(query % (group[0], group[1]), verbose=9)
- run_sql(query, (group[0], group[1]))
- query = """DELETE FROM %s WHERE id_bibrec
- BETWEEN %%s AND %%s AND type='FUTURE'""" % tab_name
- write_message(query % (group[0], group[1]), verbose=9)
- run_sql(query, (group[0], group[1]))
- write_message('End of emergency flushing wordTable into %s' % tab_name, verbose=9)
- write_message('...updating reverse table %s ended' % tab_name)
+ tab_name = self.table_name[:-1] + "R"
+ if mode == "normal":
+ for group in self.recIDs_in_mem:
+ query = """UPDATE %s SET type='TEMPORARY' WHERE id_bibrec
+ BETWEEN %%s AND %%s AND type='CURRENT'""" % tab_name
+ write_message(query % (group[0], group[1]), verbose=9)
+ run_sql(query, (group[0], group[1]))
+
+ nb_words_total = len(self.value)
+ nb_words_report = int(nb_words_total / 10.0)
+ nb_words_done = 0
+ for word in self.value.keys():
+ self.put_word_into_db(word)
+ nb_words_done += 1
+ if nb_words_report != 0 and ((nb_words_done % nb_words_report) == 0):
+ write_message('......processed %d/%d words' % \
+ (nb_words_done, nb_words_total))
+ percentage_display = get_percentage_completed(nb_words_done, nb_words_total)
+ task_update_progress("(%s:%s) flushed %d/%d words %s" % \
+ (tab_name, self.index_name,
+ nb_words_done, nb_words_total,
+ percentage_display))
+
+ write_message('...updating %d words into %s ended' % \
+ (nb_words_total, tab_name))
+
+ write_message('...updating reverse table %s started' % tab_name)
+ if mode == "normal":
+ for group in self.recIDs_in_mem:
+ query = """UPDATE %s SET type='CURRENT' WHERE id_bibrec
+ BETWEEN %%s AND %%s AND type='FUTURE'""" % tab_name
+ write_message(query % (group[0], group[1]), verbose=9)
+ run_sql(query, (group[0], group[1]))
+ query = """DELETE FROM %s WHERE id_bibrec
+ BETWEEN %%s AND %%s AND type='TEMPORARY'""" % tab_name
+ write_message(query % (group[0], group[1]), verbose=9)
+ run_sql(query, (group[0], group[1]))
+ write_message('End of updating wordTable into %s' % \
+ tab_name, verbose=9)
+ elif mode == "emergency":
+ for group in self.recIDs_in_mem:
+ query = """UPDATE %s SET type='CURRENT' WHERE id_bibrec
+ BETWEEN %%s AND %%s AND type='TEMPORARY'""" % tab_name
+ write_message(query % (group[0], group[1]), verbose=9)
+ run_sql(query, (group[0], group[1]))
+ query = """DELETE FROM %s WHERE id_bibrec
+ BETWEEN %%s AND %%s AND type='FUTURE'""" % tab_name
+ write_message(query % (group[0], group[1]), verbose=9)
+ run_sql(query, (group[0], group[1]))
+ write_message('End of emergency flushing wordTable into %s' % \
+ tab_name, verbose=9)
+ write_message('...updating reverse table %s ended' % tab_name)
self.clean()
self.recIDs_in_mem = []
- write_message("%s %s wordtable flush ended" % (self.tablename, mode))
- task_update_progress("(%s:%s) flush ended" % (self.tablename, self.humanname))
+ write_message("%s %s wordtable flush ended" % \
+ (self.table_name, mode))
+ task_update_progress("(%s:%s) flush ended" % \
+ (self.table_name, self.index_name))
- def load_old_recIDs(self, word, index_id=None):
+ def put_word_into_db(self, word):
+ """Flush a single word to the database and delete it from memory"""
+ set = self.load_old_recIDs(word)
+ if set is not None: # merge the word recIDs found in memory:
+ hitlist_was_changed = self.merge_with_old_recIDs(word, set)
+ if not hitlist_was_changed:
+ # nothing to update:
+ write_message("......... unchanged hitlist for ``%s''" % \
+ word, verbose=9)
+ else:
+ # yes there were some new words:
+ write_message("......... updating hitlist for ``%s''" % \
+ word, verbose=9)
+ run_sql("UPDATE %s SET hitlist=%%s WHERE term=%%s" % wash_table_column_name(self.table_name), (set.fastdump(), word)) # kwalitee: disable=sql
+
+ else: # the word is new, will create new set:
+ write_message("......... inserting hitlist for ``%s''" % \
+ word, verbose=9)
+ set = intbitset(self.value[word].keys())
+ try:
+ run_sql("INSERT INTO %s (term, hitlist) VALUES (%%s, %%s)" % wash_table_column_name(self.table_name), (word, set.fastdump())) # kwalitee: disable=sql
+ except Exception, e:
+ ## We send this exception to the admin only when is not
+ ## already reparing the problem.
+ register_exception(prefix="Error when putting the term '%s' into db (hitlist=%s): %s\n" % (repr(word), set, e), alert_admin=(task_get_option('cmd') != 'repair'))
+
+ if not set: # never store empty words
+ run_sql("DELETE FROM %s WHERE term=%%s" % wash_table_column_name(self.table_name), (word,)) # kwalitee: disable=sql
+
+ def put(self, recID, word, sign):
+ """Keeps track of changes done during indexing
+ and stores these changes in memory for further use.
+ Indexing process needs this information later while
+ filling in the database.
+
+ @param recID: recID of the record we want to update in memory
+ @param word: word we want to update
+ @param sing: sign of the word, 1 means keep this word in database,
+ -1 remove word from database
+ """
+ value = self.value
+ try:
+ if self.wash_index_terms:
+ word = wash_index_term(word, self.wash_index_terms)
+ if word in value:
+ # the word 'word' exist already: update sign
+ value[word][recID] = sign
+ else:
+ value[word] = {recID: sign}
+ except Exception as e:
+ write_message("Error: Cannot put word %s with sign %d for recID %s." % \
+ (word, sign, recID))
+
+ def load_old_recIDs(self, word):
"""Load existing hitlist for the word from the database index files."""
- tab_name = self.tablename
- if index_id != self.index_id:
- tab_name = self.virtual_tablename_pattern % index_id + "F"
- query = "SELECT hitlist FROM %s WHERE term=%%s" % tab_name
- res = run_sql(query, (word,))
+ query = "SELECT hitlist FROM %s WHERE term=%%s" % self.table_name
+ res = run_sql(query, (word, ))
if res:
return intbitset(res[0][0])
else:
return None
def merge_with_old_recIDs(self, word, set):
- """Merge the system numbers stored in memory (hash of recIDs with value +1 or -1
- according to whether to add/delete them) with those stored in the database index
- and received in set universe of recIDs for the given word.
+ """Merge the system numbers stored in memory
+ (hash of recIDs with value +1 or -1 according
+ to whether to add/delete them) with those stored
+ in the database index and received in set universe
+ of recIDs for the given word.
Return False in case no change was done to SET, return True in case SET
was changed.
"""
oldset = intbitset(set)
set.update_with_signs(self.value[word])
return set != oldset
- def put_word_into_db(self, word, index_id):
- """Flush a single word to the database and delete it from memory"""
- tab_name = self.tablename
- if index_id != self.index_id:
- tab_name = self.virtual_tablename_pattern % index_id + "F"
- set = self.load_old_recIDs(word, index_id)
- if set is not None: # merge the word recIDs found in memory:
- if not self.merge_with_old_recIDs(word, set):
- # nothing to update:
- write_message("......... unchanged hitlist for ``%s''" % word, verbose=9)
+ def clean(self):
+ "Cleans the cache."
+ self.value = {}
+
+
+class VirtualIndexTable(AbstractIndexTable):
+ """
+ There are two types of indexes: virtual and regular/normal.
+ Check WordTable class for more on normal indexes.
+
+ This class represents a single index table for virtual index
+ (see also: AbstractIndexTable).
+ Virtual index doesn't store its own terms,
+ it accumulates terms from other indexes.
+ Good example of virtual index is the global index which stores
+ terms from title, abstract, keyword, author and so on.
+
+ This class contains methods for indexing virtual indexes.
+ See also: run_update()
+ """
+
+ def __init__(self, index_name, table_type, table_prefix="", wash_index_terms=50):
+ """
+ Creates VirtualIndexTable instance.
+ @param index_name: name of the index we want to reindex
+ @param table_type: words, pairs or phrases
+ @param table_prefix: add "tmp_" if you want to
+ reindex to temporary table
+ """
+ AbstractIndexTable.__init__(self, index_name,
+ table_type,
+ table_prefix,
+ wash_index_terms)
+ self.mode = "normal"
+ self.dependent_indexes = dict(get_virtual_index_building_blocks(self.index_id))
+
+ def set_reindex_mode(self):
+ """
+ Sets reindex mode. VirtualIndexTable will
+ remove all its content from database and
+ use insert_index function to repopulate it.
+ """
+ self.mode = "reindex"
+
+ def run_update(self, flush=10000):
+ """
+ Function starts all updating processes for virtual index.
+ It will take all information about pending changes from database
+ from queue tables (idxWORD/PAIR/PHRASExxQ), process them
+ and trigger appropriate indexing functions.
+ @param flush: how many records we will put in one go
+ into database (at most);
+ see also: opt_flush in WordTable class
+ """
+ global chunksize
+
+ if self.mode == "reindex":
+ self.clean_database()
+ for index_id, index_name in self.dependent_indexes.iteritems():
+ rng = get_records_range_for_index(index_id)
+ flush_count = 0
+ if not rng:
+ continue
+ write_message('Virtual index: %s is being reindexed for %s index' % \
+ (self.index_name, index_name))
+ chunks = chunk_generator(rng)
+ try:
+ while True:
+ task_sleep_now_if_required()
+ chunk = chunks.next()
+ self.insert_index(index_id, chunk[0], chunk[1])
+ flush_count = flush_count + chunk[1] - chunk[0] + 1
+ self.recIDs_in_mem.append(list(chunk))
+ if flush_count >= flush:
+ flush_count = 0
+ self.put_into_db()
+ except StopIteration:
+ if flush_count > 0:
+ self.put_into_db()
+ self.clean_queue_table(index_name)
+ else:
+ for index_id, index_name in self.dependent_indexes.iteritems():
+ query = """SELECT id_bibrec_low, id_bibrec_high, mode FROM %s
+ WHERE index_name=%%s
+ ORDER BY runtime ASC""" % \
+ (self.table_name[:-1] + "Q")
+ entries = self.remove_duplicates(run_sql(query, (index_name, )))
+ if entries:
+ write_message('Virtual index: %s is being updated for %s index' % \
+ (self.index_name, index_name))
+ for entry in entries:
+ operation = None
+ recID_low, recID_high, mode = entry
+
+ if mode == CFG_BIBINDEX_UPDATE_MODE["Update"]:
+ operation = self.update_index
+ elif mode == CFG_BIBINDEX_UPDATE_MODE["Remove"]:
+ operation = self.remove_index
+ elif mode == CFG_BIBINDEX_UPDATE_MODE["Insert"]:
+ operation = self.insert_index
+
+ flush_count = 0
+ chunks = chunk_generator([recID_low, recID_high])
+ try:
+ while True:
+ task_sleep_now_if_required()
+ chunk = chunks.next()
+ operation(index_id, chunk[0], chunk[1])
+ flush_count = flush_count + chunk[1] - chunk[0] + 1
+ self.recIDs_in_mem.append(list(chunk))
+ if flush_count >= flush:
+ flush_count = 0
+ self.put_into_db()
+ except StopIteration:
+ if flush_count > 0:
+ self.put_into_db()
+ self.clean_queue_table(index_name)
+
+ def retrieve_new_values_from_index(self, index_id, records_range):
+ """
+ Retrieves new values from dependent index
+ for specific range of records.
+ @param index_id: id of the dependent index
+ @param records_range: the smallest and the biggest id
+ in the range: [id_low, id_high]
+ """
+
+ tab_name = "idx" + self.table_type + ("%02d" % index_id) + "R"
+ query = """SELECT id_bibrec, termlist FROM %s WHERE id_bibrec
+ BETWEEN %%s AND %%s""" % tab_name
+ new_regular_values = run_sql(query, (records_range[0], records_range[1]))
+ if new_regular_values:
+ zipped = zip(*new_regular_values)
+ new_regular_values = dict(zip(zipped[0], map(deserialize_via_marshal, zipped[1])))
+ else:
+ new_regular_values = dict()
+ return new_regular_values
+
+ def retrieve_old_values(self, records_range):
+ """
+ Retrieves old values from database for this virtual index
+ for specific records range.
+ @param records_range: the smallest and the biggest id
+ in the range: [id_low, id_high]
+ """
+
+ virtual_tab_name = self.table_name[:-1] + "R"
+ query = """SELECT id_bibrec, termlist FROM %s
+ WHERE type='CURRENT' AND
+ id_bibrec BETWEEN %%s AND %%s""" % virtual_tab_name
+ old_virtual_values = run_sql(query, (records_range[0], records_range[1]))
+ if old_virtual_values:
+ zipped = zip(*old_virtual_values)
+ old_virtual_values = dict(zip(zipped[0], map(deserialize_via_marshal, zipped[1])))
+ else:
+ old_virtual_values = dict()
+ return old_virtual_values
+
+ def update_index(self, index_id, recID_low, recID_high):
+ """
+ Updates the state of virtual index for records in range:
+ recID_low, recID_high for index specified by index_id.
+ Function stores terms in idxWORD/PAIR/PHRASExxR tables with
+ prefixes for specific index, for example term 'ellis'
+ from author index will be stored in reversed table as:
+ '__author__ellis'. It allows fast operations on only part of terms
+ @param index_id: id of the dependent index we want to remove
+ @param recID_low: first recID from the range of considered recIDs
+ @param recID_high: last recID from the range of considered recIDs
+ """
+ index_name = self.dependent_indexes[index_id]
+ update_cache_for_record = self.update_cache_for_record
+ virtual_tab_name = self.table_name[:-1] + "R"
+
+ # take new values
+ new_regular_values = self.retrieve_new_values_from_index(index_id, [recID_low, recID_high])
+
+ # take old values
+ old_virtual_values = self.retrieve_old_values([recID_low, recID_high])
+
+ # update reversed table
+ for recID in xrange(recID_low, recID_high + 1):
+ new_values = new_regular_values.get(recID) or []
+ old_values = old_virtual_values.get(recID) or []
+ to_serialize = update_cache_for_record(index_name, recID, old_values, new_values)
+ if len(to_serialize) == 0:
+ continue
+ run_sql("""INSERT INTO %s (id_bibrec,termlist,type)
+ VALUES (%%s,%%s,'FUTURE')""" % \
+ wash_table_column_name(virtual_tab_name),
+ (recID, serialize_via_marshal(to_serialize))) # kwalitee: disable=sql
+ try:
+ run_sql("INSERT INTO %s (id_bibrec,termlist,type) VALUES (%%s,%%s,'CURRENT')" % wash_table_column_name(virtual_tab_name), (recID, serialize_via_marshal([]))) # kwalitee: disable=sql
+ except DatabaseError:
+ pass
+
+ def insert_index(self, index_id, recID_low, recID_high):
+ """
+ Inserts terms from dependent index to virtual table
+ without looking what's inside the virtual table and
+ what terms are being added. It's faster than 'updating',
+ but it can only be used when virtual table is free of
+ terms from this dependent index.
+ @param index_id: id of the dependent index we want to remove
+ @param recID_low: first recID from the range of considered recIDs
+ @param recID_high: last recID from the range of considered recIDs
+ """
+ index_name = self.dependent_indexes[index_id]
+ insert_to_cache_for_record = self.insert_to_cache_for_record
+ virtual_tab_name = self.table_name[:-1] + "R"
+
+ # take new values
+ new_regular_values = self.retrieve_new_values_from_index(index_id, [recID_low, recID_high])
+
+ # take old values
+ old_virtual_values = self.retrieve_old_values([recID_low, recID_high])
+
+ # update reversed table
+ for recID in xrange(recID_low, recID_high + 1):
+ new_values = new_regular_values.get(recID) or []
+ old_values = old_virtual_values.get(recID) or []
+ to_serialize = insert_to_cache_for_record(index_name, recID, old_values, new_values)
+ if len(to_serialize) == 0:
+ continue
+ run_sql("INSERT INTO %s (id_bibrec,termlist,type) VALUES (%%s,%%s,'FUTURE')" % wash_table_column_name(virtual_tab_name), (recID, serialize_via_marshal(to_serialize))) # kwalitee: disable=sql
+ try:
+ run_sql("INSERT INTO %s (id_bibrec,termlist,type) VALUES (%%s,%%s,'CURRENT')" % wash_table_column_name(virtual_tab_name), (recID, serialize_via_marshal([]))) # kwalitee: disable=sql
+ except DatabaseError:
pass
+
+ def remove_index(self, index_id, recID_low, recID_high):
+ """
+ Removes words found in dependent index from reversed
+ table of virtual index. Updates the state of the memory
+ (for future removal from forward table).
+ Takes into account that given words can be found in more
+ that one dependent index and it won't mark these words
+ for the removal process.
+ @param index_id: id of the dependent index we want to remove
+ @param recID_low: first recID from the range of considered recIDs
+ @param recID_high: last recID from the range of considered recIDs
+ """
+ index_name = self.dependent_indexes[index_id]
+ remove_from_cache_for_record = self.remove_from_cache_for_record
+ virtual_tab_name = self.table_name[:-1] + "R"
+
+ # take old values
+ old_virtual_values = self.retrieve_old_values([recID_low, recID_high])
+
+ # update reversed table
+ for recID in xrange(recID_low, recID_high + 1):
+ old_values = old_virtual_values.get(recID) or []
+ to_serialize = remove_from_cache_for_record(index_name, recID, old_values)
+ if len(to_serialize) == 0:
+ continue
+ run_sql("INSERT INTO %s (id_bibrec,termlist,type) VALUES (%%s,%%s,'FUTURE')" % wash_table_column_name(virtual_tab_name), (recID, serialize_via_marshal(to_serialize))) # kwalitee: disable=sql
+ try:
+ run_sql("INSERT INTO %s (id_bibrec,termlist,type) VALUES (%%s,%%s,'CURRENT')" % wash_table_column_name(virtual_tab_name), (recID, serialize_via_marshal([]))) # kwalitee: disable=sql
+ except DatabaseError:
+ pass
+
+ def update_cache_for_record(self, index_name, recID, old_values, new_values):
+ """
+ Updates memory (cache) with information on what to
+ remove/add/modify in forward table for specified record.
+ It also returns new terms which should be indexed for given record.
+ @param index_name: index name of dependent index
+ @param recID: considered record
+ @param old_values: all old values from all dependent indexes
+ for this virtual index for recID
+ @param new_values: new values from some dependent index
+ which should be added
+ """
+ prefix = make_prefix(index_name)
+ put = self.put
+ new_values_prefix = [prefix + term for term in new_values]
+ part_values = []
+ tmp_old_values_prefix = []
+ # split old values from v.index into those with 'prefix' and those without
+ for term in old_values:
+ if term.startswith(prefix):
+ term_without_prefix = re.sub(re_prefix, '', term)
+ part_values.append(term_without_prefix)
+ put(recID, term_without_prefix, -1)
else:
- # yes there were some new words:
- write_message("......... updating hitlist for ``%s''" % word, verbose=9)
- run_sql("UPDATE %s SET hitlist=%%s WHERE term=%%s" % wash_table_column_name(tab_name), (set.fastdump(), word)) # kwalitee: disable=sql
+ tmp_old_values_prefix.append(term)
+
+ # remember not to remove words that occur more than once
+ part_values = set(part_values)
+ for value in tmp_old_values_prefix:
+ term_without_prefix = re.sub(re_prefix, '', value)
+ if term_without_prefix in part_values:
+ put(recID, term_without_prefix, 1)
+ for term_without_prefix in new_values:
+ put(recID, term_without_prefix, 1)
+
+ tmp_new_values_prefix = list(tmp_old_values_prefix)
+ tmp_new_values_prefix.extend(new_values_prefix)
+ return tmp_new_values_prefix
+
+ def insert_to_cache_for_record(self, index_name, recID, old_values, new_values):
+ """
+ Updates cache with terms which should be inserted to database.
+ Used in insert_index function. See also: update_cache_for_record
+ which is analogous for update_index function.
+ """
+ prefix = make_prefix(index_name)
- else: # the word is new, will create new set:
- write_message("......... inserting hitlist for ``%s''" % word, verbose=9)
- set = intbitset(self.value[word].keys())
+ append = old_values.append
+ put = self.put
+ for term in new_values:
+ append(prefix + term)
+ put(recID, term, 1)
+ return old_values
+
+ def remove_from_cache_for_record(self, index_name, recID, old_values):
+ """
+ Updates information in cache with terms which should be removed
+ from virtual table. Used in remove_index function.
+ """
+ prefix = make_prefix(index_name)
+ tmp_rest = []
+ tmp_removed = []
+ tmp_new_values = []
+
+ append_to_new = tmp_new_values.append
+ append_to_rest = tmp_rest.append
+ append_to_removed = tmp_removed.append
+ put = self.put
+ for term in old_values:
+ if term.startswith(prefix):
+ term_without_prefix = re.sub(re_prefix, '', term)
+ append_to_removed(term_without_prefix)
+ put(recID, term_without_prefix, -1)
+ else:
+ append_to_rest(re.sub(re_prefix, '', term))
+ append_to_new(term)
+
+ to_remember = set(tmp_rest) & set(tmp_removed)
+ for term_without_prefix in to_remember:
+ put(recID, term_without_prefix, 1)
+ return tmp_new_values
+
+ def clean_database(self):
+ """Removes all entries from corresponding tables in database"""
+ query = """DELETE FROM %s""" % self.table_name
+ run_sql(query)
+ query = """DELETE FROM %s""" % self.table_name[:-1] + "R"
+ run_sql(query)
+
+ def clean_queue_table(self, index_name):
+ """
+ Cleans queue table (i.e. idxWORD/PAIR/PHRASExxQ)
+ for specific index. It means that function will remove
+ all entries from db from queue table for this index.
+ """
+ query = "DELETE FROM %s WHERE index_name='%s'" % \
+ (self.table_name[:-1].lstrip(self.table_prefix) + "Q",
+ index_name)
+ run_sql(query)
+
+ def remove_duplicates(self, entries):
+ """
+ Removes duplicates from a list of entries (taken from Queue table)
+ in order to process a single command only once.
+ Queue table may look like this:
+ id (..) id_bibrec_low id_bibrec_high index_name mode
+ ...
+ 12 1 100 title update
+ 13 1 100 title update
+ We don't want to perform the same operation twice. First we want to
+ squash the same commands into one.
+ @param entries: list of entries taken from the database
+ """
+ unique = set()
+ return [entry for entry in entries if entry not in unique and not unique.add(entry)]
+
+
+ def remove_dependent_index(self, index_name):
+ """
+ Removes dependent index from this virtual index.
+ It means removing all words from all records with prefix:
+ __index_name__ from reversed table, and removing some of
+ them from forward table if they don't appear in another
+ dependent index.
+ @param index_name: name of the dependent index to remove
+ """
+ flush = 10000
+ dependent = self.dependent_indexes.values()
+ if len(dependent) == 0:
+ write_message("Specified index is not virtual...")
+ return
+ if index_name not in dependent:
+ write_message("Dependent index already removed...")
+ return
+ index_id = get_index_id_from_index_name(index_name)
+ records_range = get_records_range_for_index(index_id)
+ write_message("Removing an index: %s" % index_name)
+ if records_range:
+ flush_count = 0
+ chunks = chunk_generator([records_range[0], records_range[1]])
try:
- run_sql("INSERT INTO %s (term, hitlist) VALUES (%%s, %%s)" % wash_table_column_name(tab_name), (word, set.fastdump())) # kwalitee: disable=sql
- except Exception as e:
- ## We send this exception to the admin only when is not
- ## already reparing the problem.
- register_exception(prefix="Error when putting the term '%s' into db (hitlist=%s): %s\n" % (repr(word), set, e), alert_admin=(task_get_option('cmd') != 'repair'))
+ while True:
+ task_sleep_now_if_required()
+ chunk = chunks.next()
+ self.remove_index(index_id, chunk[0], chunk[1])
+ flush_count = flush_count + chunk[1] - chunk[0] + 1
+ self.recIDs_in_mem.append(chunk)
+ if flush_count >= flush:
+ flush_count = 0
+ self.put_into_db()
+ except StopIteration:
+ if flush_count > 0:
+ self.put_into_db()
- if not set: # never store empty words
- run_sql("DELETE FROM %s WHERE term=%%s" % wash_table_column_name(tab_name), (word,)) # kwalitee: disable=sql
+class WordTable(AbstractIndexTable):
+ """
+ This class represents a single index table of regular index
+ (regular means it doesn't accumulates data from other indexes,
+ but it takes data directly from metadata of records which
+ are being indexed; for other type of index check: VirtualIndexTable).
+
+ To start indexing process one need to invoke add_recIDs() method.
+ For furher reading see description of this method.
+ """
+
+ def __init__(self, index_name, fields_to_index, table_type, table_prefix="", wash_index_terms=50):
+ """Creates words table instance.
+ @param index_name: the index name
+ @param index_id: the index integer identificator
+ @param fields_to_index: a list of fields to index
+ @param table_name_pattern: i.e. idxWORD%02dF or idxPHRASE%02dF
+ @param table_type: type of the wordtable: Words, Pairs, Phrases
+ @param wash_index_terms: do we wash index terms, and if yes (when >0),
+ how many characters do we keep in the index terms; see
+ max_char_length parameter of wash_index_term()
+ """
+ AbstractIndexTable.__init__(self, index_name, table_type, table_prefix, wash_index_terms)
+ self.fields_to_index = fields_to_index
+ self.timestamp = datetime.now()
+
+ self.virtual_indexes = get_index_virtual_indexes(self.index_id)
+ self.virtual_index_update_mode = CFG_BIBINDEX_UPDATE_MODE["Update"]
+
+ try:
+ self.stemming_language = get_index_stemming_language(self.index_id)
+ except KeyError:
+ self.stemming_language = ''
+ self.remove_stopwords = get_index_remove_stopwords(self.index_id)
+ self.remove_html_markup = get_index_remove_html_markup(self.index_id)
+ self.remove_latex_markup = get_index_remove_latex_markup(self.index_id)
+ self.tokenizer = get_index_tokenizer(self.index_id)(self.stemming_language,
+ self.remove_stopwords,
+ self.remove_html_markup,
+ self.remove_latex_markup)
+ self.tokenizer_type = detect_tokenizer_type(self.tokenizer)
+ self.default_tokenizer_function = self.tokenizer.get_tokenizing_function(table_type)
+
+ self.special_tags = self._handle_special_tags()
+
+ if self.stemming_language and self.table_name.startswith('idxWORD'):
+ write_message('%s has stemming enabled, language %s' % (self.table_name, self.stemming_language))
+
+ def _handle_special_tags(self):
+ """
+ Fills in a dict with special tags which
+ always use the same tokenizer and this
+ tokenizer is independent of index.
+ """
+ special_tags = {}
+ for tag in self.fields_to_index:
+ if tag in CFG_BIBINDEX_SPECIAL_TAGS:
+
+ for t in CFG_BIBINDEX_INDEX_TABLE_TYPE:
+ if self.table_type == CFG_BIBINDEX_INDEX_TABLE_TYPE[t]:
+ tokenizer_name = CFG_BIBINDEX_SPECIAL_TAGS[tag][t]
+ tokenizer = _TOKENIZERS[tokenizer_name]
+ instance = tokenizer(self.stemming_language,
+ self.remove_stopwords,
+ self.remove_html_markup,
+ self.remove_latex_markup)
+ special_tags[tag] = instance.get_tokenizing_function(self.table_type)
+ break
+ return special_tags
+
+ def turn_off_virtual_indexes(self):
+ """
+ Prevents from reindexing related virtual indexes.
+ """
+ self.virtual_indexes = []
+
+ def turn_on_virtual_indexes(self):
+ """
+ Turns on indexing related virtual indexes.
+ """
+ self.virtual_indexes = get_index_virtual_indexes(self.index_id)
+
+ def get_field(self, recID, tag):
+ """Returns list of values of the MARC-21 'tag' fields for the
+ record 'recID'."""
+
+ out = []
+ bibXXx = "bib" + tag[0] + tag[1] + "x"
+ bibrec_bibXXx = "bibrec_" + bibXXx
+ query = """SELECT value FROM %s AS b, %s AS bb
+ WHERE bb.id_bibrec=%%s AND bb.id_bibxxx=b.id
+ AND tag LIKE %%s""" % (bibXXx, bibrec_bibXXx)
+ res = run_sql(query, (recID, tag))
+ for row in res:
+ out.append(row[0])
+ return out
+
+ def notify_virtual_indexes(self, recID_ranges):
+ """
+ Informs all related virtual indexes about index change.
+ Function leaves information about the change for each index
+ in proper table in database (idxSOMETHINGxxQ).
+ @param recID_ranges: low and high recIDs of ranges
+ @type recID_ranges: list [[low_id1, high_id1], [low_id2, high_id2]...]
+ """
+ query = """INSERT INTO %s (runtime, id_bibrec_low, id_bibrec_high, index_name, mode)
+ VALUES (%%s, %%s, %%s, %%s, %%s)"""
+ for index_id, index_name in self.virtual_indexes:
+ tab_name = "idx%s%02dQ" % (self.table_type, index_id)
+ full_query = query % tab_name
+ for recID_range in recID_ranges:
+ run_sql(full_query, (self.timestamp,
+ recID_range[0],
+ recID_range[1],
+ self.index_name,
+ self.virtual_index_update_mode))
def display(self):
"Displays the word table."
keys = self.value.keys()
keys.sort()
for k in keys:
write_message("%s: %s" % (k, self.value[k]))
def count(self):
"Returns the number of words in the table."
return len(self.value)
def info(self):
"Prints some information on the words table."
write_message("The words table contains %d words." % self.count())
def lookup_words(self, word=""):
"Lookup word from the words table."
if not word:
done = 0
while not done:
try:
word = raw_input("Enter word: ")
done = 1
except (EOFError, KeyboardInterrupt):
return
if word in self.value:
write_message("The word '%s' is found %d times." \
% (word, len(self.value[word])))
else:
write_message("The word '%s' does not exist in the word file."\
% word)
def add_recIDs(self, recIDs, opt_flush):
"""Fetches records which id in the recIDs range list and adds
them to the wordTable. The recIDs range list is of the form:
[[i1_low,i1_high],[i2_low,i2_high], ..., [iN_low,iN_high]].
"""
- if self.is_virtual:
- return
global chunksize, _last_word_table
flush_count = 0
records_done = 0
records_to_go = 0
for arange in recIDs:
records_to_go = records_to_go + arange[1] - arange[0] + 1
time_started = time.time() # will measure profile time
for arange in recIDs:
i_low = arange[0]
chunksize_count = 0
while i_low <= arange[1]:
task_sleep_now_if_required()
# calculate chunk group of recIDs and treat it:
i_high = min(i_low + opt_flush - flush_count - 1, arange[1])
i_high = min(i_low + chunksize - chunksize_count - 1, i_high)
try:
self.chk_recID_range(i_low, i_high)
except StandardError:
if self.index_name == 'fulltext' and CFG_SOLR_URL:
solr_commit()
raise
write_message(CFG_BIBINDEX_ADDING_RECORDS_STARTED_STR % \
- (self.tablename, i_low, i_high))
+ (self.table_name, i_low, i_high))
if CFG_CHECK_MYSQL_THREADS:
kill_sleepy_mysql_threads()
percentage_display = get_percentage_completed(records_done, records_to_go)
- task_update_progress("(%s:%s) adding recs %d-%d %s" % (self.tablename, self.humanname, i_low, i_high, percentage_display))
+ task_update_progress("(%s:%s) adding recs %d-%d %s" % (self.table_name, self.index_name, i_low, i_high, percentage_display))
self.del_recID_range(i_low, i_high)
just_processed = self.add_recID_range(i_low, i_high)
flush_count = flush_count + i_high - i_low + 1
chunksize_count = chunksize_count + i_high - i_low + 1
records_done = records_done + just_processed
write_message(CFG_BIBINDEX_ADDING_RECORDS_STARTED_STR % \
- (self.tablename, i_low, i_high))
+ (self.table_name, i_low, i_high))
if chunksize_count >= chunksize:
chunksize_count = 0
# flush if necessary:
if flush_count >= opt_flush:
self.put_into_db()
self.clean()
if self.index_name == 'fulltext' and CFG_SOLR_URL:
solr_commit()
- write_message("%s backing up" % (self.tablename))
+ write_message("%s backing up" % (self.table_name))
flush_count = 0
self.log_progress(time_started, records_done, records_to_go)
# iterate:
i_low = i_high + 1
if flush_count > 0:
self.put_into_db()
if self.index_name == 'fulltext' and CFG_SOLR_URL:
solr_commit()
self.log_progress(time_started, records_done, records_to_go)
+ self.notify_virtual_indexes(recIDs)
def add_recID_range(self, recID1, recID2):
"""Add records from RECID1 to RECID2."""
wlist = {}
self.recIDs_in_mem.append([recID1, recID2])
# special case of author indexes where we also add author
# canonical IDs:
if self.index_name in ('author', 'firstauthor', 'exactauthor', 'exactfirstauthor'):
for recID in range(recID1, recID2 + 1):
if recID not in wlist:
wlist[recID] = []
wlist[recID] = list_union(get_author_canonical_ids_for_recid(recID),
wlist[recID])
- if len(self.fields_to_index) == 0:
- #'no tag' style of indexing - use bibfield instead of directly consulting bibrec
- tokenizing_function = self.default_tokenizer_function
+ tokenizing_function = self.default_tokenizer_function
+ if self.tokenizer_type == CFG_BIBINDEX_TOKENIZER_TYPE["recjson"]:
+ # use bibfield instead of directly consulting bibrec
for recID in range(recID1, recID2 + 1):
record = get_record(recID)
if record:
new_words = tokenizing_function(record)
if recID not in wlist:
wlist[recID] = []
wlist[recID] = list_union(new_words, wlist[recID])
- # case of special indexes:
- elif self.index_name in ('authorcount', 'journal'):
- for tag in self.fields_to_index:
- tokenizing_function = self.tag_to_words_fnc_map.get(tag, self.default_tokenizer_function)
- for recID in range(recID1, recID2 + 1):
- new_words = tokenizing_function(recID)
- if recID not in wlist:
- wlist[recID] = []
- wlist[recID] = list_union(new_words, wlist[recID])
- # usual tag-by-tag indexing for the rest:
- else:
+ elif self.tokenizer_type == CFG_BIBINDEX_TOKENIZER_TYPE["multifield"]:
+ # tokenizers operating on multiple fields/tags at a time
+ for recID in range(recID1, recID2 + 1):
+ new_words = tokenizing_function(recID)
+ if recID not in wlist:
+ wlist[recID] = []
+ wlist[recID] = list_union(new_words, wlist[recID])
+ elif self.tokenizer_type == CFG_BIBINDEX_TOKENIZER_TYPE["string"]:
+ # tokenizers operating on one field/tag and phrase at a time
for tag in self.fields_to_index:
- tokenizing_function = self.tag_to_words_fnc_map.get(tag, self.default_tokenizer_function)
+ tokenizing_function = self.special_tags.get(tag, self.default_tokenizer_function)
phrases = self.get_phrases_for_tokenizing(tag, recID1, recID2)
for row in sorted(phrases):
recID, phrase = row
if recID not in wlist:
wlist[recID] = []
new_words = tokenizing_function(phrase)
wlist[recID] = list_union(new_words, wlist[recID])
+ else:
+ raise UnknownTokenizer("Tokenizer has not been recognized: %s" \
+ % self.tokenizer.__class__.__name__)
# lookup index-time synonyms:
synonym_kbrs = get_all_synonym_knowledge_bases()
if self.index_name in synonym_kbrs:
if len(wlist) == 0: return 0
recIDs = wlist.keys()
for recID in recIDs:
for word in wlist[recID]:
word_synonyms = get_synonym_terms(word,
synonym_kbrs[self.index_name][0],
synonym_kbrs[self.index_name][1],
use_memoise=True)
if word_synonyms:
wlist[recID] = list_union(word_synonyms, wlist[recID])
# were there some words for these recIDs found?
recIDs = wlist.keys()
for recID in recIDs:
# was this record marked as deleted?
if "DELETED" in self.get_field(recID, "980__c"):
wlist[recID] = []
write_message("... record %d was declared deleted, removing its word list" % recID, verbose=9)
write_message("... record %d, termlist: %s" % (recID, wlist[recID]), verbose=9)
- self.index_virtual_indexes_reversed(wlist, recID1, recID2)
-
if len(wlist) == 0: return 0
# put words into reverse index table with FUTURE status:
for recID in recIDs:
- run_sql("INSERT INTO %sR (id_bibrec,termlist,type) VALUES (%%s,%%s,'FUTURE')" % wash_table_column_name(self.tablename[:-1]), (recID, serialize_via_marshal(wlist[recID]))) # kwalitee: disable=sql
+ run_sql("INSERT INTO %sR (id_bibrec,termlist,type) VALUES (%%s,%%s,'FUTURE')" % wash_table_column_name(self.table_name[:-1]), (recID, serialize_via_marshal(wlist[recID]))) # kwalitee: disable=sql
# ... and, for new records, enter the CURRENT status as empty:
try:
- run_sql("INSERT INTO %sR (id_bibrec,termlist,type) VALUES (%%s,%%s,'CURRENT')" % wash_table_column_name(self.tablename[:-1]), (recID, serialize_via_marshal([]))) # kwalitee: disable=sql
+ run_sql("INSERT INTO %sR (id_bibrec,termlist,type) VALUES (%%s,%%s,'CURRENT')" % wash_table_column_name(self.table_name[:-1]), (recID, serialize_via_marshal([]))) # kwalitee: disable=sql
except DatabaseError:
# okay, it's an already existing record, no problem
pass
# put words into memory word list:
put = self.put
for recID in recIDs:
for w in wlist[recID]:
put(recID, w, 1)
return len(recIDs)
def get_phrases_for_tokenizing(self, tag, first_recID, last_recID):
"""Gets phrases for later tokenization for a range of records and
specific tag.
@param tag: MARC tag
@param first_recID: first recID from the range of recIDs to index
@param last_recID: last recID from the range of recIDs to index
"""
bibXXx = "bib" + tag[0] + tag[1] + "x"
bibrec_bibXXx = "bibrec_" + bibXXx
query = """SELECT bb.id_bibrec,b.value FROM %s AS b, %s AS bb
WHERE bb.id_bibrec BETWEEN %%s AND %%s
AND bb.id_bibxxx=b.id AND tag LIKE %%s""" % (bibXXx, bibrec_bibXXx)
phrases = run_sql(query, (first_recID, last_recID, tag))
if tag == '8564_u':
## FIXME: Quick hack to be sure that hidden files are
## actually indexed.
phrases = set(phrases)
for recid in xrange(int(first_recID), int(last_recID) + 1):
for bibdocfile in BibRecDocs(recid).list_latest_files():
phrases.add((recid, bibdocfile.get_url()))
#authority records
pattern = tag.replace('%', '*')
matches = fnmatch.filter(CFG_BIBAUTHORITY_CONTROLLED_FIELDS_BIBLIOGRAPHIC.keys(), pattern)
if not len(matches):
return phrases
phrases = set(phrases)
for tag_match in matches:
authority_tag = tag_match[0:3] + "__0"
for recID in xrange(int(first_recID), int(last_recID) + 1):
control_nos = get_fieldvalues(recID, authority_tag)
for control_no in control_nos:
new_strings = get_index_strings_by_control_no(control_no)
for string_value in new_strings:
phrases.add((recID, string_value))
return phrases
-
- def index_virtual_indexes_reversed(self, wlist, recID1, recID2):
- """Inserts indexed words into all virtual indexes connected to
- this index"""
- #first: need to take old values from given index to remove
- #them from virtual indexes
- query = """SELECT id_bibrec, termlist FROM %sR WHERE id_bibrec
- BETWEEN %%s AND %%s""" % wash_table_column_name(self.tablename[:-1])
- old_index_values = run_sql(query, (recID1, recID2))
- if old_index_values:
- zipped = zip(*old_index_values)
- old_index_values = dict(zip(zipped[0], map(deserialize_via_marshal, zipped[1])))
- else:
- old_index_values = dict()
- recIDs = wlist.keys()
-
- for vindex_id, vindex_name in self.virtual_indexes:
- #second: need to take old values from virtual index
- #to have a list of words from which we can remove old values from given index
- tab_name = self.virtual_tablename_pattern % vindex_id + "R"
- query = """SELECT id_bibrec, termlist FROM %s WHERE type='CURRENT' AND id_bibrec
- BETWEEN %%s AND %%s""" % tab_name
- old_virtual_index_values = run_sql(query, (recID1, recID2))
- if old_virtual_index_values:
- zipped = zip(*old_virtual_index_values)
- old_virtual_index_values = dict(zip(zipped[0], map(deserialize_via_marshal, zipped[1])))
- else:
- old_virtual_index_values = dict()
- for recID in recIDs:
- to_serialize = list((set(old_virtual_index_values.get(recID) or []) - set(old_index_values.get(recID) or [])) | set(wlist[recID]))
- run_sql("INSERT INTO %s (id_bibrec,termlist,type) VALUES (%%s,%%s,'FUTURE')" % wash_table_column_name(tab_name), (recID, serialize_via_marshal(to_serialize))) # kwalitee: disable=sql
- try:
- run_sql("INSERT INTO %s (id_bibrec,termlist,type) VALUES (%%s,%%s,'CURRENT')" % wash_table_column_name(tab_name), (recID, serialize_via_marshal([]))) # kwalitee: disable=sql
- except DatabaseError:
- pass
- if len(recIDs) != (recID2 - recID1 + 1):
- #for records in range(recID1, recID2) which weren't updated:
- #need to prevent them from being deleted by function: 'put_into_db'
- #which deletes all records with 'CURRENT' status
- query = """INSERT INTO %s (id_bibrec, termlist, type)
- SELECT id_bibrec, termlist, 'FUTURE' FROM %s
- WHERE id_bibrec BETWEEN %%s AND %%s
- AND type='CURRENT'
- AND id_bibrec IN (
- SELECT id_bibrec FROM %s
- WHERE id_bibrec BETWEEN %%s AND %%s
- GROUP BY id_bibrec HAVING COUNT(id_bibrec) = 1
- )
- """ % ((wash_table_column_name(tab_name),)*3)
- run_sql(query, (recID1, recID2, recID1, recID2))
-
-
def log_progress(self, start, done, todo):
"""Calculate progress and store it.
start: start time,
done: records processed,
todo: total number of records"""
time_elapsed = time.time() - start
# consistency check
if time_elapsed == 0 or done > todo:
return
time_recs_per_min = done / (time_elapsed / 60.0)
write_message("%d records took %.1f seconds to complete.(%1.f recs/min)"\
% (done, time_elapsed, time_recs_per_min))
if time_recs_per_min:
write_message("Estimated runtime: %.1f minutes" % \
((todo - done) / time_recs_per_min))
def put(self, recID, word, sign):
- """Adds/deletes a word to the word list."""
+ """Keeps track of changes done during indexing
+ and stores these changes in memory for further use.
+ Indexing process needs this information later while
+ filling in the database.
+
+ @param recID: recID of the record we want to update in memory
+ @param word: word we want to update
+ @param sing: sign of the word, 1 means keep this word in database,
+ -1 remove word from database
+ """
+ value = self.value
try:
if self.wash_index_terms:
word = wash_index_term(word, self.wash_index_terms)
if word in self.value:
# the word 'word' exist already: update sign
- self.value[word][recID] = sign
+ value[word][recID] = sign
else:
- self.value[word] = {recID: sign}
+ value[word] = {recID: sign}
except:
write_message("Error: Cannot put word %s with sign %d for recID %s." % (word, sign, recID))
+
def del_recIDs(self, recIDs):
"""Fetches records which id in the recIDs range list and adds
them to the wordTable. The recIDs range list is of the form:
[[i1_low,i1_high],[i2_low,i2_high], ..., [iN_low,iN_high]].
"""
count = 0
for arange in recIDs:
task_sleep_now_if_required()
self.del_recID_range(arange[0], arange[1])
count = count + arange[1] - arange[0]
+ self.virtual_index_update_mode = CFG_BIBINDEX_UPDATE_MODE["Remove"]
self.put_into_db()
+ self.notify_virtual_indexes(recIDs)
if self.index_name == 'fulltext' and CFG_SOLR_URL:
solr_commit()
def del_recID_range(self, low, high):
"""Deletes records with 'recID' system number between low
and high from memory words index table."""
write_message("%s fetching existing words for records #%d-#%d started" % \
- (self.tablename, low, high), verbose=3)
+ (self.table_name, low, high), verbose=3)
self.recIDs_in_mem.append([low, high])
query = """SELECT id_bibrec,termlist FROM %sR as bb WHERE bb.id_bibrec
- BETWEEN %%s AND %%s""" % (self.tablename[:-1])
+ BETWEEN %%s AND %%s""" % (self.table_name[:-1])
recID_rows = run_sql(query, (low, high))
for recID_row in recID_rows:
recID = recID_row[0]
wlist = deserialize_via_marshal(recID_row[1])
for word in wlist:
self.put(recID, word, -1)
write_message("%s fetching existing words for records #%d-#%d ended" % \
- (self.tablename, low, high), verbose=3)
+ (self.table_name, low, high), verbose=3)
def check_bad_words(self):
"""
Finds bad words in reverse tables. Returns True in case of bad words.
"""
query = """SELECT 1 FROM %sR WHERE type IN ('TEMPORARY','FUTURE') LIMIT 1""" \
- % (self.tablename[:-1],)
+ % (self.table_name[:-1],)
res = run_sql(query)
return bool(res)
def report_on_table_consistency(self):
"""Check reverse words index tables (e.g. idxWORD01R) for
interesting states such as 'TEMPORARY' state.
Prints small report (no of words, no of bad words).
"""
# find number of words:
- query = """SELECT COUNT(1) FROM %s""" % (self.tablename)
+ query = """SELECT COUNT(1) FROM %s""" % (self.table_name)
+
res = run_sql(query, None, 1)
if res:
nb_words = res[0][0]
else:
nb_words = 0
# report stats:
- write_message("%s contains %d words" % (self.tablename, nb_words))
+ write_message("%s contains %d words" % (self.table_name, nb_words))
# find possible bad states in reverse tables:
if self.check_bad_words():
write_message("EMERGENCY: %s needs to be repaired" %
- (self.tablename, ))
+ (self.table_name, ))
else:
- write_message("%s is in consistent state" % (self.tablename))
+ write_message("%s is in consistent state" % (self.table_name))
def repair(self, opt_flush):
"""Repair the whole table"""
# find possible bad states in reverse tables:
if not self.check_bad_words():
return
query = """SELECT id_bibrec FROM %sR WHERE type IN ('TEMPORARY','FUTURE')""" \
- % (self.tablename[:-1])
+ % (self.table_name[:-1])
res = intbitset(run_sql(query))
recIDs = create_range_list(list(res))
flush_count = 0
records_done = 0
records_to_go = 0
for arange in recIDs:
records_to_go = records_to_go + arange[1] - arange[0] + 1
time_started = time.time() # will measure profile time
for arange in recIDs:
i_low = arange[0]
chunksize_count = 0
while i_low <= arange[1]:
task_sleep_now_if_required()
# calculate chunk group of recIDs and treat it:
i_high = min(i_low + opt_flush - flush_count - 1, arange[1])
i_high = min(i_low + chunksize - chunksize_count - 1, i_high)
self.fix_recID_range(i_low, i_high)
flush_count = flush_count + i_high - i_low + 1
chunksize_count = chunksize_count + i_high - i_low + 1
records_done = records_done + i_high - i_low + 1
if chunksize_count >= chunksize:
chunksize_count = 0
# flush if necessary:
if flush_count >= opt_flush:
self.put_into_db("emergency")
self.clean()
flush_count = 0
self.log_progress(time_started, records_done, records_to_go)
# iterate:
i_low = i_high + 1
if flush_count > 0:
self.put_into_db("emergency")
self.log_progress(time_started, records_done, records_to_go)
- write_message("%s inconsistencies repaired." % self.tablename)
+ write_message("%s inconsistencies repaired." % self.table_name)
def chk_recID_range(self, low, high):
"""Check if the reverse index table is in proper state"""
## check db
query = """SELECT 1 FROM %sR WHERE type IN ('TEMPORARY','FUTURE')
- AND id_bibrec BETWEEN %%s AND %%s LIMIT 1""" % self.tablename[:-1]
+ AND id_bibrec BETWEEN %%s AND %%s LIMIT 1""" % self.table_name[:-1]
res = run_sql(query, (low, high), 1)
if not res:
- write_message("%s for %d-%d is in consistent state" % (self.tablename, low, high))
+ write_message("%s for %d-%d is in consistent state" % (self.table_name, low, high))
return # okay, words table is consistent
## inconsistency detected!
- write_message("EMERGENCY: %s inconsistencies detected..." % self.tablename)
+ write_message("EMERGENCY: %s inconsistencies detected..." % self.table_name)
error_message = "Errors found. You should check consistency of the " \
"%s - %sR tables.\nRunning 'bibindex --repair' is " \
- "recommended." % (self.tablename, self.tablename[:-1])
+ "recommended." % (self.table_name, self.table_name[:-1])
write_message("EMERGENCY: " + error_message, stream=sys.stderr)
raise StandardError(error_message)
def fix_recID_range(self, low, high):
- """Try to fix reverse index database consistency (e.g. table idxWORD01R) in the low,high doc-id range.
+ """Try to fix reverse index database consistency
+ (e.g. table idxWORD01R) in the low,high doc-id range.
Possible states for a recID follow:
CUR TMP FUT: very bad things have happened: warn!
CUR TMP : very bad things have happened: warn!
CUR FUT: delete FUT (crash before flushing)
CUR : database is ok
TMP FUT: add TMP to memory and del FUT from memory
flush (revert to old state)
TMP : very bad things have happened: warn!
FUT: very bad things have happended: warn!
"""
state = {}
query = "SELECT id_bibrec,type FROM %sR WHERE id_bibrec BETWEEN %%s AND %%s"\
- % self.tablename[:-1]
+ % self.table_name[:-1]
res = run_sql(query, (low, high))
for row in res:
if row[0] not in state:
state[row[0]] = []
state[row[0]].append(row[1])
ok = 1 # will hold info on whether we will be able to repair
for recID in state.keys():
if not 'TEMPORARY' in state[recID]:
if 'FUTURE' in state[recID]:
if 'CURRENT' not in state[recID]:
write_message("EMERGENCY: Index record %d is in inconsistent state. Can't repair it." % recID)
ok = 0
else:
write_message("EMERGENCY: Inconsistency in index record %d detected" % recID)
query = """DELETE FROM %sR
- WHERE id_bibrec=%%s""" % self.tablename[:-1]
+ WHERE id_bibrec=%%s""" % self.table_name[:-1]
run_sql(query, (recID,))
write_message("EMERGENCY: Inconsistency in record %d repaired." % recID)
else:
if 'FUTURE' in state[recID] and not 'CURRENT' in state[recID]:
self.recIDs_in_mem.append([recID, recID])
# Get the words file
query = """SELECT type,termlist FROM %sR
- WHERE id_bibrec=%%s""" % self.tablename[:-1]
+ WHERE id_bibrec=%%s""" % self.table_name[:-1]
write_message(query, verbose=9)
res = run_sql(query, (recID,))
for row in res:
wlist = deserialize_via_marshal(row[1])
write_message("Words are %s " % wlist, verbose=9)
if row[0] == 'TEMPORARY':
sign = 1
else:
sign = -1
for word in wlist:
self.put(recID, word, sign)
else:
write_message("EMERGENCY: %s for %d is in inconsistent "
- "state. Couldn't repair it." % (self.tablename,
+ "state. Couldn't repair it." % (self.table_name,
recID), stream=sys.stderr)
ok = 0
if not ok:
error_message = "Unrepairable errors found. You should check " \
"consistency of the %s - %sR tables. Deleting affected " \
"TEMPORARY and FUTURE entries from these tables is " \
"recommended; see the BibIndex Admin Guide." % \
- (self.tablename, self.tablename[:-1])
+ (self.table_name, self.table_name[:-1])
write_message("EMERGENCY: " + error_message, stream=sys.stderr)
raise StandardError(error_message)
- def remove_dependent_index(self, id_dependent):
- """Removes terms found in dependent index from virtual index.
- Function finds words for removal and then removes them from
- forward and reversed tables term by term.
- @param id_dependent: id of an index which we want to remove from this
- virtual index
- """
- if not self.is_virtual:
- write_message("Index is not virtual...")
- return
-
- global chunksize
- terms_current_counter = 0
- terms_done = 0
- terms_to_go = 0
-
- for_full_removal, for_partial_removal = self.get_words_to_remove(id_dependent, misc_lookup=False)
- query = """SELECT t.term, m.hitlist FROM %s%02dF as t INNER JOIN %s%02dF as m
- ON t.term=m.term""" % (self.tablename[:-3], self.index_id, self.tablename[:-3], id_dependent)
- terms_and_hitlists = dict(run_sql(query))
- terms_to_go = len(for_full_removal) + len(for_partial_removal)
- task_sleep_now_if_required()
- #full removal
- for term in for_full_removal:
- terms_current_counter += 1
- hitlist = intbitset(terms_and_hitlists[term])
- for recID in hitlist:
- self.remove_single_word_reversed_table(term, recID)
- self.remove_single_word_forward_table(term)
- if terms_current_counter % chunksize == 0:
- terms_done += terms_current_counter
- terms_current_counter = 0
- write_message("removed %s/%s terms..." % (terms_done, terms_to_go))
- task_sleep_now_if_required()
- terms_done += terms_current_counter
- terms_current_counter = 0
- #partial removal
- for term, indexes in iteritems(for_partial_removal):
- self.value = {}
- terms_current_counter += 1
- hitlist = intbitset(terms_and_hitlists[term])
- if len(indexes) > 0:
- hitlist -= self._find_common_hitlist(term, id_dependent, indexes)
- for recID in hitlist:
- self.remove_single_word_reversed_table(term, recID)
- if term in self.value:
- self.value[term][recID] = -1
- else:
- self.value[term] = {recID: -1}
- if self.value:
- self.put_word_into_db(term, self.index_id)
- if terms_current_counter % chunksize == 0:
- terms_done += terms_current_counter
- terms_current_counter = 0
- write_message("removed %s/%s terms..." % (terms_done, terms_to_go))
- task_sleep_now_if_required()
-
-
- def remove_single_word_forward_table(self, word):
- """Immediately and irreversibly removes a word from forward table"""
- run_sql("""DELETE FROM %s WHERE term=%%s""" % self.tablename, (word, )) # kwalitee: disable=sql
-
- def remove_single_word_reversed_table(self, word, recID):
- """Removes single word from temlist for given recID"""
- old_set = run_sql("""SELECT termlist FROM %sR WHERE id_bibrec=%%s""" % \
- wash_table_column_name(self.tablename[:-1]), (recID, ))
- new_set = []
- if old_set:
- new_set = deserialize_via_marshal(old_set[0][0])
- if word in new_set:
- new_set.remove(word)
- if new_set:
- run_sql("""UPDATE %sR SET termlist=%%s
- WHERE id_bibrec=%%s AND
- type='CURRENT'""" % \
- wash_table_column_name(self.tablename[:-1]), (serialize_via_marshal(new_set), recID))
-
- def _find_common_hitlist(self, term, id_dependent, indexes):
- """Checks 'indexes' for records that have 'term' indexed
- and returns intersection between found records
- and records that have a 'term' inside index
- defined by id_dependent parameter"""
- query = """SELECT m.hitlist FROM idxWORD%02dF as t INNER JOIN idxWORD%02dF as m
- ON t.term=m.term WHERE t.term='%s'"""
- common_hitlist = intbitset([])
- for _id in indexes:
- res = run_sql(query % (id_dependent, _id, term))
- if res:
- common_hitlist |= intbitset(res[0][0])
- return common_hitlist
-
- def get_words_to_remove(self, id_dependent, misc_lookup=False):
- """Finds words in dependent index which should be removed from virtual index.
- Example:
- Virtual index 'A' consists of 'B' and 'C' dependent indexes and we want to
- remove 'B' from virtual index 'A'.
- First we need to check if 'B' and 'C' have common words. If they have
- we need to be careful not to remove common words from 'A', because we want
- to remove only words from 'B'.
- Then we need to check common words for 'A' and 'B'. These are potential words
- for removal. We need to substract common words for 'B' and 'C' from common words
- for 'A' and 'B' to be sure that correct words are removed.
- @return: (list, dict), list contains terms/words for full removal, dict
- contains words for partial removal together with ids of indexes in which
- given term/word also exists
- """
-
- query = """SELECT t.term FROM %s%02dF as t INNER JOIN %s%02dF as m
- ON t.term=m.term"""
- dependent_indexes = get_virtual_index_building_blocks(self.index_id)
- other_ids = list(dependent_indexes and zip(*dependent_indexes)[0] or [])
- if id_dependent in other_ids:
- other_ids.remove(id_dependent)
- if not misc_lookup:
- misc_id = get_index_id_from_index_name('miscellaneous')
- if misc_id in other_ids:
- other_ids.remove(misc_id)
-
- #intersections between dependent indexes
- left_in_other_indexes = {}
- for _id in other_ids:
- intersection = zip(*run_sql(query % (self.tablename[:-3], id_dependent, self.tablename[:-3], _id))) # kwalitee: disable=sql
- terms = bool(intersection) and intersection[0] or []
- for term in terms:
- if term in left_in_other_indexes:
- left_in_other_indexes[term].append(_id)
- else:
- left_in_other_indexes[term] = [_id]
-
- #intersection between virtual index and index we want to remove
- main_intersection = zip(*run_sql(query % (self.tablename[:-3], self.index_id, self.tablename[:-3], id_dependent))) # kwalitee: disable=sql
- terms_main = set(bool(main_intersection) and main_intersection[0] or [])
- return list(terms_main - set(left_in_other_indexes.keys())), left_in_other_indexes
-
-
def main():
"""Main that construct all the bibtask."""
task_init(authorization_action='runbibindex',
authorization_msg="BibIndex Task Submission",
description="""Examples:
\t%s -a -i 234-250,293,300-500 -u admin@localhost
\t%s -a -w author,fulltext -M 8192 -v3
\t%s -d -m +4d -A on --flush=10000\n""" % ((sys.argv[0],) * 3), help_specific_usage=""" Indexing options:
-a, --add\t\tadd or update words for selected records
-d, --del\t\tdelete words for selected records
-i, --id=low[-high]\t\tselect according to doc recID
-m, --modified=from[,to]\tselect according to modification date
-c, --collection=c1[,c2]\tselect according to collection
-R, --reindex\treindex the selected indexes from scratch
Repairing options:
-k, --check\t\tcheck consistency for all records in the table(s)
-r, --repair\t\ttry to repair all records in the table(s)
Specific options:
-w, --windex=w1[,w2]\tword/phrase indexes to consider (all)
-M, --maxmem=XXX\tmaximum memory usage in kB (no limit)
-f, --flush=NNN\t\tfull consistent table flush after NNN records (10000)
- --force\tforce indexing of all records for provided indexes
- -Z, --remove-dependent-index=w\tname of an index for removing from virtual index
+ --force\t\tforce indexing of all records for provided indexes
+ -Z, --remove-dependent-index=w name of an index for removing from virtual index
+ -l --all-virtual\t\t set of all virtual indexes; the same as: -w virtual_ind1, virtual_ind2, ...
""",
version=__revision__,
- specific_params=("adi:m:c:w:krRM:f:oZ:", [
+ specific_params=("adi:m:c:w:krRM:f:oZ:l", [
"add",
"del",
"id=",
"modified=",
"collection=",
"windex=",
"check",
"repair",
"reindex",
"maxmem=",
"flush=",
"force",
- "remove-dependent-index="
+ "remove-dependent-index=",
+ "all-virtual"
]),
task_stop_helper_fnc=task_stop_table_close_fnc,
task_submit_elaborate_specific_parameter_fnc=task_submit_elaborate_specific_parameter,
task_run_fnc=task_run_core,
task_submit_check_options_fnc=task_submit_check_options)
def task_submit_check_options():
"""Check for options compatibility."""
if task_get_option("reindex"):
if task_get_option("cmd") != "add" or task_get_option('id') or task_get_option('collection'):
print("ERROR: You can use --reindex only when adding modified record.", file=sys.stderr)
return False
return True
def task_submit_elaborate_specific_parameter(key, value, opts, args):
""" Given the string key it checks it's meaning, eventually using the
value. Usually it fills some key in the options dict.
It must return True if it has elaborated the key, False, if it doesn't
know that key.
eg:
if key in ['-n', '--number']:
self.options['number'] = value
return True
return False
"""
if key in ("-a", "--add"):
task_set_option("cmd", "add")
if ("-x", "") in opts or ("--del", "") in opts:
raise StandardError("Can not have --add and --del at the same time!")
elif key in ("-k", "--check"):
task_set_option("cmd", "check")
elif key in ("-r", "--repair"):
task_set_option("cmd", "repair")
elif key in ("-d", "--del"):
task_set_option("cmd", "del")
elif key in ("-i", "--id"):
task_set_option('id', task_get_option('id') + split_ranges(value))
elif key in ("-m", "--modified"):
task_set_option("modified", get_date_range(value))
elif key in ("-c", "--collection"):
task_set_option("collection", value)
elif key in ("-R", "--reindex"):
task_set_option("reindex", True)
elif key in ("-w", "--windex"):
task_set_option("windex", value)
elif key in ("-M", "--maxmem"):
task_set_option("maxmem", int(value))
if task_get_option("maxmem") < base_process_size + 1000:
raise StandardError("Memory usage should be higher than %d kB" % \
(base_process_size + 1000))
elif key in ("-f", "--flush"):
task_set_option("flush", int(value))
elif key in ("-o", "--force"):
task_set_option("force", True)
elif key in ("-Z", "--remove-dependent-index",):
task_set_option("remove-dependent-index", value)
+ elif key in ("-l", "--all-virtual",):
+ task_set_option("all-virtual", True)
else:
return False
return True
def task_stop_table_close_fnc():
""" Close tables to STOP. """
global _last_word_table
if _last_word_table:
_last_word_table.put_into_db()
def get_recIDs_by_date_bibliographic(dates, index_name, force_all=False):
""" Finds records that were modified between DATES[0] and DATES[1]
for given index.
If DATES is not set, then finds records that were modified since
the last update of the index.
@param wordtable_type: can be 'Words', 'Pairs' or 'Phrases'
"""
index_id = get_index_id_from_index_name(index_name)
if not dates:
query = """SELECT last_updated FROM idxINDEX WHERE id=%s"""
res = run_sql(query, (index_id,))
if not res:
return set([])
if not res[0][0] or force_all:
dates = ("0000-00-00", None)
else:
dates = (res[0][0], None)
if dates[1] is None:
res = intbitset(run_sql("""SELECT b.id FROM bibrec AS b WHERE b.modification_date >= %s""",
(dates[0],)))
if index_name == 'fulltext':
res |= intbitset(run_sql("""SELECT id_bibrec FROM bibrec_bibdoc JOIN bibdoc ON id_bibdoc=id
WHERE text_extraction_date <= modification_date AND
modification_date >= %s
AND status<>'DELETED'""",
(dates[0],)))
elif dates[0] is None:
res = intbitset(run_sql("""SELECT b.id FROM bibrec AS b WHERE b.modification_date <= %s""",
(dates[1],)))
if index_name == 'fulltext':
res |= intbitset(run_sql("""SELECT id_bibrec FROM bibrec_bibdoc JOIN bibdoc ON id_bibdoc=id
WHERE text_extraction_date <= modification_date
AND modification_date <= %s
AND status<>'DELETED'""",
(dates[1],)))
else:
res = intbitset(run_sql("""SELECT b.id FROM bibrec AS b
WHERE b.modification_date >= %s AND
b.modification_date <= %s""",
(dates[0], dates[1])))
if index_name == 'fulltext':
res |= intbitset(run_sql("""SELECT id_bibrec FROM bibrec_bibdoc JOIN bibdoc ON id_bibdoc=id
WHERE text_extraction_date <= modification_date AND
modification_date >= %s AND
modification_date <= %s AND
status<>'DELETED'""",
(dates[0], dates[1],)))
# special case of author indexes where we need to re-index
# those records that were affected by changed BibAuthorID attributions:
if index_name in ('author', 'firstauthor', 'exactauthor', 'exactfirstauthor'):
from invenio.legacy.bibauthorid.personid_maintenance import get_recids_affected_since
# dates[1] is ignored, since BibAuthorID API does not offer upper limit search
rec_list_author = intbitset(get_recids_affected_since(dates[0]))
res = res | rec_list_author
return set(res)
def get_recIDs_by_date_authority(dates, index_name, force_all=False):
""" Finds records that were modified between DATES[0] and DATES[1]
for given index.
If DATES is not set, then finds records that were modified since
the last update of the index.
Searches for bibliographic records connected to authority records
that have been changed.
"""
index_id = get_index_id_from_index_name(index_name)
index_tags = get_index_tags(index_name)
if not dates:
query = """SELECT last_updated FROM idxINDEX WHERE id=%s"""
res = run_sql(query, (index_id,))
if not res:
return set([])
if not res[0][0] or force_all:
dates = ("0000-00-00", None)
else:
dates = (res[0][0], None)
res = intbitset()
for tag in index_tags:
pattern = tag.replace('%', '*')
matches = fnmatch.filter(CFG_BIBAUTHORITY_CONTROLLED_FIELDS_BIBLIOGRAPHIC.keys(), pattern)
if not len(matches):
continue
for tag_match in matches:
# get the type of authority record associated with this field
auth_type = CFG_BIBAUTHORITY_CONTROLLED_FIELDS_BIBLIOGRAPHIC.get(tag_match)
# find updated authority records of this type
# dates[1] is ignored, needs dates[0] to find res
now = datetime.now()
auth_recIDs = search_pattern(p='980__a:' + auth_type) \
& search_unit_in_bibrec(str(dates[0]), str(now), search_type='m')
# now find dependent bibliographic records
for auth_recID in auth_recIDs:
# get the fix authority identifier of this authority record
control_nos = get_control_nos_from_recID(auth_recID)
# there may be multiple control number entries! (the '035' field is repeatable!)
for control_no in control_nos:
# get the bibrec IDs that refer to AUTHORITY_ID in TAG
tag_0 = tag_match[:5] + '0' # possibly do the same for '4' subfields ?
fieldvalue = '"' + control_no + '"'
res |= search_pattern(p=tag_0 + ':' + fieldvalue)
return set(res)
def get_not_updated_recIDs(modified_dates, indexes, force_all=False):
"""Finds not updated recIDs in database for indexes.
@param modified_dates: between this dates we should look for modified records
@type modified_dates: [date_old, date_new]
@param indexes: list of indexes
@type indexes: string separated by coma
@param force_all: if True all records will be taken
"""
found_recIDs = set()
write_message(CFG_BIBINDEX_UPDATE_MESSAGE)
for index in indexes:
found_recIDs |= get_recIDs_by_date_bibliographic(modified_dates, index, force_all)
found_recIDs |= get_recIDs_by_date_authority(modified_dates, index, force_all)
return list(sorted(found_recIDs))
def get_recIDs_from_cli(indexes=[]):
"""
Gets recIDs ranges from CLI for indexing when
user specified 'id' or 'collection' option or
search for modified recIDs for provided indexes
when recIDs are not specified.
@param indexes: it's a list of specified indexes, which
can be obtained from CLI with use of:
get_indexes_from_cli() function.
@type indexes: list of strings
"""
# need to first update idxINDEX table to find proper recIDs for reindexing
if task_get_option("reindex"):
for index_name in indexes:
run_sql("""UPDATE idxINDEX SET last_updated='0000-00-00 00:00:00'
WHERE name=%s""", (index_name,))
if task_get_option("id"):
return task_get_option("id")
elif task_get_option("collection"):
l_of_colls = task_get_option("collection").split(",")
recIDs = perform_request_search(c=l_of_colls)
recIDs_range = []
for recID in recIDs:
recIDs_range.append([recID, recID])
return recIDs_range
elif task_get_option("cmd") == "add":
recs = get_not_updated_recIDs(task_get_option("modified"),
indexes,
task_get_option("force"))
recIDs_range = beautify_range_list(create_range_list(recs))
return recIDs_range
return []
def get_indexes_from_cli():
"""
Gets indexes from CLI and checks if they are
valid. If indexes weren't specified function
will return all known indexes.
"""
indexes = task_get_option("windex")
- if not indexes:
+ all_virtual = task_get_option("all-virtual")
+ if all_virtual:
+ indexes = filter_for_virtual_indexes(get_all_indexes())
+ elif not indexes:
indexes = get_all_indexes()
else:
indexes = indexes.split(",")
indexes = remove_inexistent_indexes(indexes, leave_virtual=True)
return indexes
def remove_dependent_index(virtual_indexes, dependent_index):
"""
Removes dependent index from virtual indexes.
@param virtual_indexes: names of virtual_indexes
@type virtual_indexes: list of strings
@param dependent_index: name of dependent index
@type dependent_index: string
"""
if not virtual_indexes:
write_message("You should specify a name of a virtual index...")
+ return
id_dependent = get_index_id_from_index_name(dependent_index)
wordTables = get_word_tables(virtual_indexes)
for index_id, index_name, index_tags in wordTables:
- wordTable = WordTable(index_name=index_name,
- index_id=index_id,
- fields_to_index=index_tags,
- table_name_pattern='idxWORD%02dF',
- wordtable_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Words"],
- tag_to_tokenizer_map={'8564_u': "BibIndexEmptyTokenizer"},
- wash_index_terms=50)
- wordTable.remove_dependent_index(id_dependent)
-
- wordTable.report_on_table_consistency()
- task_sleep_now_if_required()
-
- wordTable = WordTable(index_name=index_name,
- index_id=index_id,
- fields_to_index=index_tags,
- table_name_pattern='idxPAIR%02dF',
- wordtable_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Pairs"],
- tag_to_tokenizer_map={'8564_u': "BibIndexEmptyTokenizer"},
- wash_index_terms=50)
- wordTable.remove_dependent_index(id_dependent)
+ for type_ in CFG_BIBINDEX_INDEX_TABLE_TYPE.itervalues():
+ vit = VirtualIndexTable(index_name, type_)
+ vit.remove_dependent_index(dependent_index)
+ task_sleep_now_if_required()
- wordTable.report_on_table_consistency()
- task_sleep_now_if_required()
+ query = """DELETE FROM idxINDEX_idxINDEX WHERE id_virtual=%s AND id_normal=%s"""
+ run_sql(query, (index_id, id_dependent))
- wordTable = WordTable(index_name=index_name,
- index_id=index_id,
- fields_to_index=index_tags,
- table_name_pattern='idxPHRASE%02dF',
- wordtable_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Words"],
- tag_to_tokenizer_map={'8564_u': "BibIndexEmptyTokenizer"},
- wash_index_terms=50)
- wordTable.remove_dependent_index(id_dependent)
+def should_update_virtual_indexes():
+ """
+ Decides if any virtual indexes should be updated.
+ Decision is made based on arguments obtained
+ from CLI.
+ """
+ return task_get_option("all-virtual") or task_get_option("windex")
- wordTable.report_on_table_consistency()
+def update_virtual_indexes(virtual_indexes, reindex=False):
+ """
+ Function will update all specified virtual_indexes.
+ @param virtual_indexes: list of index names
+ @param reindex: shall we reindex given v.indexes from scratch?
+ """
+ kwargs = {}
+ if reindex:
+ kwargs.update({'table_prefix': 'tmp_'})
+
+ for index_name in virtual_indexes:
+ if reindex:
+ index_id = get_index_id_from_index_name(index_name)
+ init_temporary_reindex_tables(index_id)
+
+ for key, type_ in CFG_BIBINDEX_INDEX_TABLE_TYPE.iteritems():
+ kwargs.update({'wash_index_terms': CFG_BIBINDEX_WASH_INDEX_TERMS[key]})
+ vit = VirtualIndexTable(index_name, type_, **kwargs)
+ vit.set_reindex_mode()
+ vit.run_update()
+
+ swap_temporary_reindex_tables(index_id)
+ update_index_last_updated([index_name], task_get_task_param('task_starting_time'))
+ task_sleep_now_if_required(can_stop_too=True)
+ else:
+ for key, type_ in CFG_BIBINDEX_INDEX_TABLE_TYPE.iteritems():
+ kwargs.update({'wash_index_terms': CFG_BIBINDEX_WASH_INDEX_TERMS[key]})
+ vit = VirtualIndexTable(index_name, type_, **kwargs)
+ vit.run_update()
- query = """DELETE FROM idxINDEX_idxINDEX WHERE id_virtual=%s AND id_normal=%s"""
- run_sql(query, (index_id, id_dependent))
+ task_sleep_now_if_required(can_stop_too=True)
def task_run_core():
"""Runs the task by fetching arguments from the BibSched task queue.
This is what BibSched will be invoking via daemon call.
"""
global _last_word_table
indexes = get_indexes_from_cli()
if len(indexes) == 0:
write_message("Specified indexes can't be found.")
return True
+ virtual_indexes = filter_for_virtual_indexes(indexes)
+ regular_indexes = list(set(indexes) - set(virtual_indexes))
+
# check tables consistency
if task_get_option("cmd") == "check":
wordTables = get_word_tables(indexes)
for index_id, index_name, index_tags in wordTables:
wordTable = WordTable(index_name=index_name,
- index_id=index_id,
fields_to_index=index_tags,
- table_name_pattern='idxWORD%02dF',
- wordtable_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Words"],
- tag_to_tokenizer_map={'8564_u': "BibIndexFulltextTokenizer"},
+ table_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Words"],
wash_index_terms=50)
_last_word_table = wordTable
wordTable.report_on_table_consistency()
task_sleep_now_if_required(can_stop_too=True)
wordTable = WordTable(index_name=index_name,
- index_id=index_id,
fields_to_index=index_tags,
- table_name_pattern='idxPAIR%02dF',
- wordtable_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Pairs"],
- tag_to_tokenizer_map={'8564_u': "BibIndexEmptyTokenizer"},
+ table_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Pairs"],
wash_index_terms=100)
_last_word_table = wordTable
wordTable.report_on_table_consistency()
task_sleep_now_if_required(can_stop_too=True)
wordTable = WordTable(index_name=index_name,
- index_id=index_id,
fields_to_index=index_tags,
- table_name_pattern='idxPHRASE%02dF',
- wordtable_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Phrases"],
- tag_to_tokenizer_map={'8564_u': "BibIndexEmptyTokenizer"},
+ table_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Phrases"],
wash_index_terms=0)
_last_word_table = wordTable
wordTable.report_on_table_consistency()
task_sleep_now_if_required(can_stop_too=True)
_last_word_table = None
return True
- #virtual index: remove dependent index
+ # virtual index: remove dependent index
if task_get_option("remove-dependent-index"):
remove_dependent_index(indexes,
task_get_option("remove-dependent-index"))
return True
- #initialization for Words,Pairs,Phrases
- recIDs_range = get_recIDs_from_cli(indexes)
- recIDs_for_index = find_affected_records_for_index(indexes,
+ # virtual index: update
+ if should_update_virtual_indexes():
+ update_virtual_indexes(virtual_indexes, task_get_option("reindex"))
+
+ if len(regular_indexes) == 0:
+ return True
+
+ # regular index: initialization for Words,Pairs,Phrases
+ recIDs_range = get_recIDs_from_cli(regular_indexes)
+ recIDs_for_index = find_affected_records_for_index(regular_indexes,
recIDs_range,
(task_get_option("force") or \
task_get_option("reindex") or \
task_get_option("cmd") == "del"))
wordTables = get_word_tables(recIDs_for_index.keys())
if not wordTables:
write_message("Selected indexes/recIDs are up to date.")
# Let's work on single words!
for index_id, index_name, index_tags in wordTables:
reindex_prefix = ""
if task_get_option("reindex"):
reindex_prefix = "tmp_"
init_temporary_reindex_tables(index_id, reindex_prefix)
wordTable = WordTable(index_name=index_name,
- index_id=index_id,
fields_to_index=index_tags,
- table_name_pattern=reindex_prefix + 'idxWORD%02dF',
- wordtable_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Words"],
- tag_to_tokenizer_map={'8564_u': "BibIndexFulltextTokenizer"},
+ table_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Words"],
+ table_prefix=reindex_prefix,
wash_index_terms=50)
_last_word_table = wordTable
wordTable.report_on_table_consistency()
try:
if task_get_option("cmd") == "del":
if task_get_option("id") or task_get_option("collection"):
wordTable.del_recIDs(recIDs_range)
task_sleep_now_if_required(can_stop_too=True)
else:
error_message = "Missing IDs of records to delete from " \
- "index %s." % wordTable.tablename
+ "index %s." % wordTable.table_name
write_message(error_message, stream=sys.stderr)
raise StandardError(error_message)
elif task_get_option("cmd") == "add":
final_recIDs = beautify_range_list(create_range_list(recIDs_for_index[index_name]))
wordTable.add_recIDs(final_recIDs, task_get_option("flush"))
task_sleep_now_if_required(can_stop_too=True)
elif task_get_option("cmd") == "repair":
wordTable.repair(task_get_option("flush"))
task_sleep_now_if_required(can_stop_too=True)
else:
error_message = "Invalid command found processing %s" % \
- wordTable.tablename
+ wordTable.table_name
write_message(error_message, stream=sys.stderr)
raise StandardError(error_message)
except StandardError as e:
write_message("Exception caught: %s" % e, sys.stderr)
register_exception(alert_admin=True)
if _last_word_table:
_last_word_table.put_into_db()
raise
wordTable.report_on_table_consistency()
task_sleep_now_if_required(can_stop_too=True)
# Let's work on pairs now
wordTable = WordTable(index_name=index_name,
- index_id=index_id,
fields_to_index=index_tags,
- table_name_pattern=reindex_prefix + 'idxPAIR%02dF',
- wordtable_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Pairs"],
- tag_to_tokenizer_map={'8564_u': "BibIndexEmptyTokenizer"},
+ table_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Pairs"],
+ table_prefix=reindex_prefix,
wash_index_terms=100)
_last_word_table = wordTable
wordTable.report_on_table_consistency()
try:
if task_get_option("cmd") == "del":
if task_get_option("id") or task_get_option("collection"):
wordTable.del_recIDs(recIDs_range)
task_sleep_now_if_required(can_stop_too=True)
else:
error_message = "Missing IDs of records to delete from " \
- "index %s." % wordTable.tablename
+ "index %s." % wordTable.table_name
write_message(error_message, stream=sys.stderr)
raise StandardError(error_message)
elif task_get_option("cmd") == "add":
final_recIDs = beautify_range_list(create_range_list(recIDs_for_index[index_name]))
wordTable.add_recIDs(final_recIDs, task_get_option("flush"))
task_sleep_now_if_required(can_stop_too=True)
elif task_get_option("cmd") == "repair":
wordTable.repair(task_get_option("flush"))
task_sleep_now_if_required(can_stop_too=True)
else:
error_message = "Invalid command found processing %s" % \
- wordTable.tablename
+ wordTable.table_name
write_message(error_message, stream=sys.stderr)
raise StandardError(error_message)
except StandardError as e:
write_message("Exception caught: %s" % e, sys.stderr)
register_exception()
if _last_word_table:
_last_word_table.put_into_db()
raise
wordTable.report_on_table_consistency()
task_sleep_now_if_required(can_stop_too=True)
# Let's work on phrases now
wordTable = WordTable(index_name=index_name,
- index_id=index_id,
fields_to_index=index_tags,
- table_name_pattern=reindex_prefix + 'idxPHRASE%02dF',
- wordtable_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Phrases"],
- tag_to_tokenizer_map={'8564_u': "BibIndexEmptyTokenizer"},
+ table_type=CFG_BIBINDEX_INDEX_TABLE_TYPE["Phrases"],
+ table_prefix=reindex_prefix,
wash_index_terms=0)
_last_word_table = wordTable
wordTable.report_on_table_consistency()
try:
if task_get_option("cmd") == "del":
if task_get_option("id") or task_get_option("collection"):
wordTable.del_recIDs(recIDs_range)
task_sleep_now_if_required(can_stop_too=True)
else:
error_message = "Missing IDs of records to delete from " \
- "index %s." % wordTable.tablename
+ "index %s." % wordTable.table_name
write_message(error_message, stream=sys.stderr)
raise StandardError(error_message)
elif task_get_option("cmd") == "add":
final_recIDs = beautify_range_list(create_range_list(recIDs_for_index[index_name]))
wordTable.add_recIDs(final_recIDs, task_get_option("flush"))
if not task_get_option("id") and not task_get_option("collection"):
update_index_last_updated([index_name], task_get_task_param('task_starting_time'))
task_sleep_now_if_required(can_stop_too=True)
elif task_get_option("cmd") == "repair":
wordTable.repair(task_get_option("flush"))
task_sleep_now_if_required(can_stop_too=True)
else:
error_message = "Invalid command found processing %s" % \
- wordTable.tablename
+ wordTable.table_name
write_message(error_message, stream=sys.stderr)
raise StandardError(error_message)
except StandardError as e:
write_message("Exception caught: %s" % e, sys.stderr)
register_exception()
if _last_word_table:
_last_word_table.put_into_db()
raise
wordTable.report_on_table_consistency()
task_sleep_now_if_required(can_stop_too=True)
if task_get_option("reindex"):
swap_temporary_reindex_tables(index_id, reindex_prefix)
update_index_last_updated([index_name], task_get_task_param('task_starting_time'))
task_sleep_now_if_required(can_stop_too=True)
# update modification date also for indexes that were up to date
if not task_get_option("id") and not task_get_option("collection") and \
task_get_option("cmd") == "add":
up_to_date = set(indexes) - set(recIDs_for_index.keys())
update_index_last_updated(list(up_to_date), task_get_task_param('task_starting_time'))
_last_word_table = None
return True
### okay, here we go:
if __name__ == '__main__':
main()
diff --git a/invenio/legacy/bibindex/engine_config.py b/invenio/legacy/bibindex/engine_config.py
index ac88ba0f2..c47bc3446 100644
--- a/invenio/legacy/bibindex/engine_config.py
+++ b/invenio/legacy/bibindex/engine_config.py
@@ -1,54 +1,71 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2010, 2011 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
BibIndex indexing engine configuration parameters.
"""
__revision__ = \
"$Id$"
import os
## configuration parameters read from the general config file:
from invenio.config import CFG_VERSION, CFG_PYLIBDIR
## version number:
BIBINDEX_ENGINE_VERSION = "Invenio/%s bibindex/%s" % (CFG_VERSION, CFG_VERSION)
## safety parameters concerning DB thread-multiplication problem:
CFG_CHECK_MYSQL_THREADS = 0 # to check or not to check the problem?
CFG_MAX_MYSQL_THREADS = 50 # how many threads (connections) we
# consider as still safe
CFG_MYSQL_THREAD_TIMEOUT = 20 # we'll kill threads that were sleeping
# for more than X seconds
-
-
CFG_BIBINDEX_SYNONYM_MATCH_TYPE = { 'None': '-None-',
'exact': 'exact',
'leading_to_comma': 'leading_to_comma',
'leading_to_number': 'leading_to_number'}
+
CFG_BIBINDEX_COLUMN_VALUE_SEPARATOR = ","
-CFG_BIBINDEX_INDEX_TABLE_TYPE = { 'Words': 'Words',
- 'Pairs': 'Pairs',
- 'Phrases': 'Phrases' }
+
+CFG_BIBINDEX_INDEX_TABLE_TYPE = { 'Words': 'WORD',
+ 'Pairs': 'PAIR',
+ 'Phrases': 'PHRASE' }
+
+CFG_BIBINDEX_WASH_INDEX_TERMS = { 'Words': 50,
+ 'Pairs': 100,
+ 'Phrases': 0}
CFG_BIBINDEX_ADDING_RECORDS_STARTED_STR = "%s adding records #%d-#%d started"
CFG_BIBINDEX_UPDATE_MESSAGE = "Searching for records which should be reindexed..."
+CFG_BIBINDEX_UPDATE_MODE = { 'Update': 'update',
+ 'Insert': 'insert',
+ 'Remove': 'remove' }
+
+CFG_BIBINDEX_TOKENIZER_TYPE = {"string": "string",
+ "multifield": "multifield",
+ "recjson": "recjson",
+ "unknown": "unknown"}
+
+CFG_BIBINDEX_SPECIAL_TAGS = {'8564_u': {'Words': 'BibIndexFulltextTokenizer',
+ 'Pairs': 'BibIndexEmptyTokenizer',
+ 'Phrases': 'BibIndexEmptyTokenizer'}
+ }
diff --git a/invenio/legacy/bibindex/engine_utils.py b/invenio/legacy/bibindex/engine_utils.py
index 9db80d103..59d5bbd68 100644
--- a/invenio/legacy/bibindex/engine_utils.py
+++ b/invenio/legacy/bibindex/engine_utils.py
@@ -1,329 +1,370 @@
# -*- coding:utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2010, 2011, 2012, 2013 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""bibindex.engine_utils: here are some useful regular experssions for tokenizers
and several helper functions.
"""
import re
import sys
from invenio.legacy.dbquery import run_sql, \
DatabaseError
from invenio.legacy.bibsched.bibtask import write_message
from invenio.legacy.bibrecord import get_fieldvalues
from invenio.config import \
CFG_BIBINDEX_CHARS_PUNCTUATION, \
CFG_BIBINDEX_CHARS_ALPHANUMERIC_SEPARATORS
latex_formula_re = re.compile(r'\$.*?\$|\\\[.*?\\\]')
phrase_delimiter_re = re.compile(r'[\.:;\?\!]')
space_cleaner_re = re.compile(r'\s+')
re_block_punctuation_begin = re.compile(r"^" + CFG_BIBINDEX_CHARS_PUNCTUATION + "+")
re_block_punctuation_end = re.compile(CFG_BIBINDEX_CHARS_PUNCTUATION + "+$")
re_punctuation = re.compile(CFG_BIBINDEX_CHARS_PUNCTUATION)
re_separators = re.compile(CFG_BIBINDEX_CHARS_ALPHANUMERIC_SEPARATORS)
re_arxiv = re.compile(r'^arxiv:\d\d\d\d\.\d\d\d\d')
re_pattern_fuzzy_author_trigger = re.compile(r'[\s\,\.]')
# FIXME: re_pattern_fuzzy_author_trigger could be removed and an
# BibAuthorID API function could be called instead after we
# double-check that there are no circular imports.
def load_tokenizers():
"""
Load all the bibindex tokenizers and returns it.
"""
from invenio.modules.indexer.registry import tokenizers
return dict((module.__name__.split('.')[-1],
getattr(module, module.__name__.split('.')[-1], ''))
for module in tokenizers)
def get_all_index_names_and_column_values(column_name):
"""Returns a list of tuples of name and another column of all defined words indexes.
Returns empty list in case there are no tags indexed in this index or in case
the column name does not exist.
Example: output=[('global', something), ('title', something)]."""
out = []
query = """SELECT name, %s FROM idxINDEX""" % column_name
try:
res = run_sql(query)
for row in res:
out.append((row[0], row[1]))
except DatabaseError:
write_message("Exception caught for SQL statement: %s; column %s might not exist" % (query, column_name), sys.stderr)
return out
def author_name_requires_phrase_search(p):
"""
Detect whether author query pattern p requires phrase search.
Notably, look for presence of spaces and commas.
"""
if re_pattern_fuzzy_author_trigger.search(p):
return True
return False
def get_field_count(recID, tags):
"""
Return number of field instances having TAGS in record RECID.
@param recID: record ID
@type recID: int
@param tags: list of tags to count, e.g. ['100__a', '700__a']
@type tags: list
@return: number of tags present in record
@rtype: int
@note: Works internally via getting field values, which may not be
very efficient. Could use counts only, or else retrieve stored
recstruct format of the record and walk through it.
"""
out = 0
for tag in tags:
out += len(get_fieldvalues(recID, tag))
return out
def run_sql_drop_silently(query):
"""
SQL DROP statement with IF EXISTS part generates
warning if table does not exist. To mute the warning
we can remove IF EXISTS and catch SQL exception telling
us that table does not exist.
"""
try:
query = query.replace(" IF EXISTS", "")
run_sql(query)
except Exception as e:
if str(e).find("Unknown table") > -1:
pass
else:
raise e
def get_idx_indexer(name):
"""Returns the indexer field value"""
try:
return run_sql("SELECT indexer FROM idxINDEX WHERE NAME=%s", (name, ))[0][0]
except StandardError as e:
return (0, e)
def get_all_indexes(virtual=True, with_ids=False):
"""Returns the list of the names of all defined words indexes.
Returns empty list in case there are no tags indexed in this index.
@param virtual: if True function will return also virtual indexes
@param with_ids: if True function will return also IDs of found indexes
Example: output=['global', 'author']."""
out = []
if virtual:
query = """SELECT %s name FROM idxINDEX"""
query = query % (with_ids and "id," or "")
else:
query = """SELECT %s w.name FROM idxINDEX AS w
WHERE w.id NOT IN (SELECT DISTINCT id_virtual FROM idxINDEX_idxINDEX)"""
query = query % (with_ids and "w.id," or "")
res = run_sql(query)
if with_ids:
out = [row for row in res]
else:
out = [row[0] for row in res]
return out
def get_all_virtual_indexes():
""" Returns all defined 'virtual' indexes. """
query = """SELECT DISTINCT v.id_virtual, w.name FROM idxINDEX_idxINDEX AS v,
idxINDEX AS w
WHERE v.id_virtual=w.id"""
res = run_sql(query)
return res
def get_index_virtual_indexes(index_id):
"""Returns 'virtual' indexes that should be indexed together with
given index."""
query = """SELECT v.id_virtual, w.name FROM idxINDEX_idxINDEX AS v,
idxINDEX AS w
WHERE v.id_virtual=w.id AND
v.id_normal=%s"""
res = run_sql(query, (index_id,))
return res
def is_index_virtual(index_id):
"""Checks if index is virtual"""
query = """SELECT id_virtual FROM idxINDEX_idxINDEX
WHERE id_virtual=%s"""
res = run_sql(query, (index_id,))
if res:
return True
return False
+def filter_for_virtual_indexes(index_list):
+ """
+ Function removes all non-virtual indexes
+ from given list of indexes.
+ @param index_list: list of index names
+ """
+ try:
+ virtual = zip(*get_all_virtual_indexes())[1]
+ selected = set(virtual) & set(index_list)
+ return list(selected)
+ except IndexError:
+ return []
+ return []
def get_virtual_index_building_blocks(index_id):
"""Returns indexes that made up virtual index of given index_id.
If index_id is an id of normal index (not virtual) returns
empty tuple.
"""
query = """SELECT v.id_normal, w.name FROM idxINDEX_idxINDEX AS v,
idxINDEX AS w
WHERE v.id_normal=w.id AND
v.id_virtual=%s"""
res = run_sql(query, (index_id,))
return res
def get_index_id_from_index_name(index_name):
"""Returns the words/phrase index id for INDEXNAME.
Returns empty string in case there is no words table for this index.
Example: field='author', output=4."""
out = 0
query = """SELECT w.id FROM idxINDEX AS w
WHERE w.name=%s LIMIT 1"""
res = run_sql(query, (index_name,), 1)
if res:
out = res[0][0]
return out
def get_index_name_from_index_id(index_id):
"""Returns the words/phrase index name for INDEXID.
Returns '' in case there is no words table for this indexid.
Example: field=9, output='fulltext'."""
res = run_sql("SELECT name FROM idxINDEX WHERE id=%s", (index_id,))
if res:
return res[0][0]
return ''
def get_field_tags(field):
"""Returns a list of MARC tags for the field code 'field'.
Returns empty list in case of error.
Example: field='author', output=['100__%','700__%']."""
out = []
query = """SELECT t.value FROM tag AS t, field_tag AS ft, field AS f
WHERE f.code=%s AND ft.id_field=f.id AND t.id=ft.id_tag
ORDER BY ft.score DESC"""
res = run_sql(query, (field,))
return [row[0] for row in res]
def get_tag_indexes(tag, virtual=True):
"""Returns indexes names and ids corresponding to the given tag
@param tag: MARC tag in one of the forms:
'xx%', 'xxx', 'xxx__a', 'xxx__%'
@param virtual: if True function will also return virtual indexes"""
tag2 = tag[0:2] + "%" #for tags in the form: 10%
tag3 = tag[:-1] + "%" #for tags in the form: 100__%
query = """SELECT DISTINCT w.id,w.name FROM idxINDEX AS w,
idxINDEX_field AS wf,
field_tag AS ft,
tag as t
WHERE (t.value=%%s OR
t.value=%%s OR
%s) AND
t.id=ft.id_tag AND
ft.id_field=wf.id_field AND
wf.id_idxINDEX=w.id"""
if tag[-1] == "%":
missing_piece = "t.value LIKE %s"
elif tag[-1] != "%" and len(tag) == 3:
missing_piece = "t.value LIKE %s"
tag3 = tag + "%" #for all tags which start from 'tag'
else:
missing_piece = "t.value=%s"
query = query % missing_piece
res = run_sql(query, (tag, tag2, tag3))
if res:
if virtual:
response = list(res)
index_ids = map(str, zip(*res)[0])
query = """SELECT DISTINCT v.id_virtual,w.name FROM idxINDEX_idxINDEX AS v,
idxINDEX as w
WHERE v.id_virtual=w.id AND
v.id_normal IN ("""
query = query + ", ".join(index_ids) + ")"
response.extend(run_sql(query))
return tuple(response)
return res
return None
def get_index_tags(indexname, virtual=True):
"""Returns the list of tags that are indexed inside INDEXNAME.
Returns empty list in case there are no tags indexed in this index.
Note: uses get_field_tags() defined before.
Example: field='author', output=['100__%', '700__%']."""
out = []
query = """SELECT f.code FROM idxINDEX AS w, idxINDEX_field AS wf,
field AS f WHERE w.name=%s AND w.id=wf.id_idxINDEX
AND f.id=wf.id_field"""
res = run_sql(query, (indexname,))
for row in res:
out.extend(get_field_tags(row[0]))
if not out and virtual:
index_id = get_index_id_from_index_name(indexname)
try:
dependent_indexes = map(str, zip(*get_virtual_index_building_blocks(index_id))[0])
except IndexError:
return out
tags = set()
query = """SELECT DISTINCT f.code FROM idxINDEX AS w, idxINDEX_field AS wf, field AS f
WHERE w.id=wf.id_idxINDEX AND
f.id=wf.id_field AND
w.id IN ("""
query = query + ", ".join(dependent_indexes) + ")"
res = run_sql(query)
for row in res:
tags |= set(get_field_tags(row[0]))
return list(tags)
return out
def get_min_last_updated(indexes):
"""Returns min modification date for 'indexes':
min(last_updated)
@param indexes: list of indexes
"""
query= """SELECT min(last_updated) FROM idxINDEX WHERE name IN ("""
for index in indexes:
query += "%s,"
query = query[:-1] + ")"
res = run_sql(query, tuple(indexes))
return res
def remove_inexistent_indexes(indexes, leave_virtual=False):
"""Removes indexes that don't exist from the given list of indexes.
@param indexes: list of indexes
@param leave_virtual: should we leave virtual indexes in the list?
"""
correct_indexes = get_all_indexes(leave_virtual)
cleaned = []
for index in indexes:
if index in correct_indexes:
cleaned.append(index)
return cleaned
+
+
+def get_records_range_for_index(index_id):
+ """
+ Get records range for given index.
+ """
+ try:
+ query = """SELECT min(id_bibrec), max(id_bibrec) FROM idxWORD%02dR""" % index_id
+ resp = run_sql(query)
+ if resp:
+ return resp[0]
+ return None
+ except Exception:
+ return None
+
+
+def make_prefix(index_name):
+ """
+ Creates a prefix for specific index which is added
+ to every word from this index stored in reversed table
+ of corresponding virtual index.
+ @param index_name: name of the dependent index we want to create prefix for
+ """
+ return "__" + index_name + "__"
+
+
+class UnknownTokenizer(Exception):
+ pass
diff --git a/invenio/legacy/bibrank/citation_indexer.py b/invenio/legacy/bibrank/citation_indexer.py
index ff8f3743c..54330a714 100644
--- a/invenio/legacy/bibrank/citation_indexer.py
+++ b/invenio/legacy/bibrank/citation_indexer.py
@@ -1,1258 +1,1292 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
__revision__ = "$Id$"
import re
import time
import os
import sys
import ConfigParser
from datetime import datetime
from itertools import islice
from intbitset import intbitset
from six import iteritems
-from invenio.legacy.dbquery import run_sql, serialize_via_marshal, \
- deserialize_via_marshal
+from intbitset import intbitset
+from invenio.legacy.dbquery import run_sql
from invenio.modules.indexer.tokenizers.BibIndexJournalTokenizer import \
CFG_JOURNAL_PUBINFO_STANDARD_FORM, \
CFG_JOURNAL_PUBINFO_STANDARD_FORM_REGEXP_CHECK
+from invenio.utils.redis import get_redis
from invenio.legacy.search_engine import search_pattern, \
search_unit, \
get_collection_reclist
from invenio.modules.formatter.utils import parse_tag
from invenio.modules.knowledge.api import get_kb_mappings
from invenio.legacy.bibsched.bibtask import write_message, task_get_option, \
task_update_progress, task_sleep_now_if_required, \
task_get_task_param
from invenio.legacy.bibindex.engine import get_field_tags
from invenio.legacy.docextract.record import get_record
+from invenio.legacy.dbquery import serialize_via_marshal
re_CFG_JOURNAL_PUBINFO_STANDARD_FORM_REGEXP_CHECK \
= re.compile(CFG_JOURNAL_PUBINFO_STANDARD_FORM_REGEXP_CHECK)
def compute_weights():
sql = "SELECT citee, COUNT(citer) FROM rnkCITATIONDICT GROUP BY citee"
weights = {}
for citee, c in run_sql(sql):
weights[citee] = c
return weights
def recids_cache(collections, cache={}):
if 'valid_recids' not in cache:
cache['valid_recids'] = intbitset()
for coll in collections.split(','):
cache['valid_recids'] += get_collection_reclist(coll)
return cache['valid_recids']
def deleted_recids_cache(cache={}):
if 'deleted_records' not in cache:
cache['deleted_records'] = search_unit(p='DELETED', f='980', m='a')
return cache['deleted_records']
def get_recids_matching_query(p, f, config, m='e'):
"""Return set of recIDs matching query for pattern p in field f.
@param p: pattern to search for
@type recID: unicode string
@param f: field to search in
@type recID: unicode string
@param config: bibrank configuration
@type recID: dict
@param m: type of matching (usually 'e' for exact or 'r' for regexp)
@type recID: string
"""
p = p.encode('utf-8')
f = f.encode('utf-8')
function = config.get("rank_method", "function")
collections = config.get(function, 'collections')
if collections:
ret = search_pattern(p=p, f=f, m=m) & recids_cache(collections)
else:
ret = search_pattern(p=p, f=f, m=m) - deleted_recids_cache()
return ret
def get_citation_weight(rank_method_code, config, chunk_size=25000):
"""return a dictionary which is used by bibrank daemon for generating
the index of sorted research results by citation information
"""
quick = task_get_option("quick") != "no"
# id option forces re-indexing a certain range
# even if there are no new recs
if task_get_option("id"):
# construct a range of records to index
updated_recids = []
for first, last in task_get_option("id"):
updated_recids += range(first, last+1)
if len(updated_recids) > 10000:
str_updated_recids = str(updated_recids[:10]) + ' ... ' + str(updated_recids[-10:])
else:
str_updated_recids = str(updated_recids)
write_message('Records to process: %s' % str_updated_recids)
index_update_time = None
else:
bibrank_update_time = get_bibrankmethod_lastupdate(rank_method_code)
if not quick:
bibrank_update_time = "0000-00-00 00:00:00"
write_message("bibrank: %s" % bibrank_update_time)
index_update_time = get_bibindex_update_time()
write_message("bibindex: %s" % index_update_time)
if index_update_time > datetime.now().strftime("%Y-%m-%d %H:%M:%S"):
index_update_time = "0000-00-00 00:00:00"
updated_recids = get_modified_recs(bibrank_update_time,
index_update_time)
if len(updated_recids) > 10000:
str_updated_recids = str(updated_recids[:10]) + ' ... ' + str(updated_recids[-10:])
else:
str_updated_recids = str(updated_recids)
write_message("%s records to update" % str_updated_recids)
if updated_recids:
begin_time = time.time()
try:
function = config.get("rank_method", "function")
config.get(function, 'collections')
except ConfigParser.NoOptionError:
config.set(function, 'collections', None)
# Process fully the updated records
weights = process_and_store(updated_recids, config, chunk_size)
end_time = time.time()
write_message("Total time of get_citation_weight(): %.2f sec" %
(end_time - begin_time))
task_update_progress("citation analysis done")
else:
weights = None
write_message("No new records added since last time this "
"rank method was executed")
return weights, index_update_time
def process_and_store(recids, config, chunk_size):
# Limit of # of citation we can loose in one chunk
function = config.get("rank_method", "function")
citation_loss_limit = int(config.get(function, "citation_loss_limit"))
# If we have nothing to process
# Do not update the weights dictionary
modified = False
# Process recent records first
# The older records were most likely added by the above steps
# to be reprocessed so they only have minor changes
recids_iter = iter(sorted(recids, reverse=True))
# Split records to process into chunks so that we do not
# fill up too much memory
while True:
task_sleep_now_if_required()
chunk = list(islice(recids_iter, chunk_size))
if not chunk:
break
write_message("Processing chunk #%s to #%s" % (chunk[0], chunk[-1]))
# The core work
cites, refs = process_chunk(chunk, config)
# Check that we haven't lost too many citations
cites_diff = compute_dicts_diff(chunk, refs, cites)
write_message("Citations balance %s" % cites_diff)
if citation_loss_limit and cites_diff <= -citation_loss_limit:
raise Exception('Lost too many references, aborting')
# Store processed citations/references
store_dicts(chunk, refs, cites)
modified = True
# Compute new weights dictionary
if modified:
weights = compute_weights()
else:
weights = None
+ store_weights_cache(weights)
+
return weights
+
+def store_weights_cache(weights):
+ """Store into key/value store"""
+ redis = get_redis()
+ redis.set('citations_weights', serialize_via_marshal(weights))
+
+
def process_chunk(recids, config):
tags = get_tags_config(config)
# call the procedure that does the hard work by reading fields of
# citations and references in the updated_recid's (but nothing else)!
write_message("Entering get_citation_informations", verbose=9)
citation_informations = get_citation_informations(recids, tags, config)
write_message("Entering ref_analyzer", verbose=9)
# call the analyser that uses the citation_informations to really
# search x-cites-y in the coll..
return ref_analyzer(citation_informations,
recids,
tags,
config)
def get_bibrankmethod_lastupdate(rank_method_code):
"""Return the last excution date of bibrank method
"""
query = """SELECT DATE_FORMAT(last_updated, '%%Y-%%m-%%d %%H:%%i:%%s')
FROM rnkMETHOD WHERE name =%s"""
last_update_time = run_sql(query, [rank_method_code])
try:
r = last_update_time[0][0]
except IndexError:
r = "0000-00-00 00:00:00"
return r
def get_bibindex_update_time():
"""Return the last indexing date of the journals and report number indexes
"""
try:
# check indexing times of `journal' and `reportnumber`
# indexes, and only fetch records which have been indexed
sql = "SELECT DATE_FORMAT(MIN(last_updated), " \
"'%%Y-%%m-%%d %%H:%%i:%%s') FROM idxINDEX WHERE name IN (%s,%s)"
index_update_time = run_sql(sql, ('journal', 'reportnumber'), 1)[0][0]
except IndexError:
write_message("Not running citation indexer since journal/reportnumber"
" indexes are not created yet.")
index_update_time = "0000-00-00 00:00:00"
return index_update_time
def get_modified_recs(bibrank_method_lastupdate, indexes_lastupdate):
"""Get records to be updated by bibrank indexing
Return the list of records which have been modified between the last
execution of bibrank method and the latest journal/report index updates.
The result is expected to have ascending id order.
"""
query = """SELECT id FROM bibrec
WHERE modification_date >= %s
AND modification_date < %s
ORDER BY id ASC"""
records = run_sql(query, (bibrank_method_lastupdate, indexes_lastupdate))
return [r[0] for r in records]
def format_journal(format_string, mappings):
"""format the publ infostring according to the format"""
def replace(char, data):
return data.get(char, char)
return ''.join(replace(c, mappings) for c in format_string)
def get_tags_config(config):
"""Fetch needs config from our config file"""
# Probably "citation" unless this file gets renamed
function = config.get("rank_method", "function")
write_message("config function %s" % function, verbose=9)
tags = {}
# 037a: contains (often) the "hep-ph/0501084" tag of THIS record
try:
tag = config.get(function, "primary_report_number")
except ConfigParser.NoOptionError:
tags['record_pri_number'] = None
else:
tags['record_pri_number'] = tagify(parse_tag(tag))
# 088a: additional short identifier for the record
try:
tag = config.get(function, "additional_report_number")
except ConfigParser.NoOptionError:
tags['record_add_number'] = None
else:
tags['record_add_number'] = tagify(parse_tag(tag))
# 999C5r. this is in the reference list, refers to other records.
# Looks like: hep-ph/0408002
try:
tag = config.get(function, "reference_via_report_number")
except ConfigParser.NoOptionError:
tags['refs_report_number'] = None
else:
tags['refs_report_number'] = tagify(parse_tag(tag))
# 999C5s. this is in the reference list, refers to other records.
# Looks like: Phys.Rev.,A21,78
try:
tag = config.get(function, "reference_via_pubinfo")
except ConfigParser.NoOptionError:
tags['refs_journal'] = None
else:
tags['refs_journal'] = tagify(parse_tag(tag))
# 999C5a. this is in the reference list, refers to other records.
# Looks like: 10.1007/BF03170733
try:
tag = config.get(function, "reference_via_doi")
except ConfigParser.NoOptionError:
tags['refs_doi'] = None
else:
tags['refs_doi'] = tagify(parse_tag(tag))
# 999C50. this is in the reference list, refers to other records.
# Looks like: 1205
try:
tag = config.get(function, "reference_via_record_id")
except ConfigParser.NoOptionError:
tags['refs_record_id'] = None
else:
tags['refs_record_id'] = tagify(parse_tag(tag))
# 999C5i. this is in the reference list, refers to other records.
# Looks like: 9781439520031
try:
tag = config.get(function, "reference_via_isbn")
except ConfigParser.NoOptionError:
tags['refs_isbn'] = None
else:
tags['refs_isbn'] = tagify(parse_tag(tag))
# Fields needed to construct the journals for this record
try:
tag = {
'pages': config.get(function, "pubinfo_journal_page"),
'year': config.get(function, "pubinfo_journal_year"),
'journal': config.get(function, "pubinfo_journal_title"),
'volume': config.get(function, "pubinfo_journal_volume"),
}
except ConfigParser.NoOptionError:
tags['publication'] = None
else:
tags['publication'] = {
'pages': tagify(parse_tag(tag['pages'])),
'year': tagify(parse_tag(tag['year'])),
'journal': tagify(parse_tag(tag['journal'])),
'volume': tagify(parse_tag(tag['volume'])),
}
# Fields needed to lookup the DOIs
tags['doi'] = get_field_tags('doi')
# Fields needed to lookup the ISBN
tags['isbn'] = get_field_tags('isbn')
# 999C5s. A standardized way of writing a reference in the reference list.
# Like: Nucl. Phys. B 710 (2000) 371
try:
tags['publication_format'] = config.get(function,
"pubinfo_journal_format")
except ConfigParser.NoOptionError:
tags['publication_format'] = CFG_JOURNAL_PUBINFO_STANDARD_FORM
# Print values of tags for debugging
write_message("tag values: %r" % [tags], verbose=9)
return tags
def get_journal_info(record, tags):
"""Fetch journal info from given record"""
record_info = []
journals_fields = record.find_fields(tags['publication']['journal'][:5])
for field in journals_fields:
# we store the tags and their values here
# like c->444 y->1999 p->"journal of foo",
# v->20
tagsvalues = {}
try:
tmp = field.get_subfield_values(tags['publication']['journal'][5])[0]
except IndexError:
pass
else:
tagsvalues["p"] = tmp
try:
tmp = field.get_subfield_values(tags['publication']['volume'][5])[0]
except IndexError:
pass
else:
tagsvalues["v"] = tmp
try:
tmp = field.get_subfield_values(tags['publication']['year'][5])[0]
except IndexError:
pass
else:
tagsvalues["y"] = tmp
try:
tmp = field.get_subfield_values(tags['publication']['pages'][5])[0]
except IndexError:
pass
else:
# if the page numbers have "x-y" take just x
tagsvalues["c"] = tmp.split('-', 1)[0]
# check if we have the required data
ok = True
for c in tags['publication_format']:
if c in ('p', 'v', 'y', 'c'):
if c not in tagsvalues:
ok = False
if ok:
publ = format_journal(tags['publication_format'], tagsvalues)
record_info += [publ]
alt_volume = get_alt_volume(tagsvalues['v'])
if alt_volume:
tagsvalues2 = tagsvalues.copy()
tagsvalues2['v'] = alt_volume
publ = format_journal(tags['publication_format'], tagsvalues2)
record_info += [publ]
# Add codens
for coden in get_kb_mappings('CODENS',
value=tagsvalues['p']):
tagsvalues2 = tagsvalues.copy()
tagsvalues2['p'] = coden['key']
publ = format_journal(tags['publication_format'], tagsvalues2)
record_info += [publ]
return record_info
def get_alt_volume(volume):
"""Get alternate volume form
We handle the inversed volume letter bug
Some metadata is wrong which leads to journals with the volume letter
at the end.
e.g. Phys.Rev.,51B,1 instead of Phys.Rev.,B51,1
"""
alt_volume = None
if re.match(ur'[a-zA-Z]\d+', volume, re.U|re.I):
alt_volume = volume[1:] + volume[0]
elif re.match(ur'\d+[a-zA-Z]', volume, re.U|re.I):
alt_volume = volume[-1] + volume[:-1]
return alt_volume
def get_citation_informations(recid_list, tags, config,
fetch_catchup_info=True):
"""Scans the collections searching references (999C5x -fields) and
citations for items in the recid_list
returns a 4 list of dictionaries that contains the citation information
of cds records
examples: [ {} {} {} {} ]
[ {5: 'SUT-DP-92-70-5'},
{ 93: ['astro-ph/9812088']},
{ 93: ['Phys. Rev. Lett. 96 (2006) 081301'] }, {} ]
NB: stuff here is for analysing new or changed records.
see "ref_analyzer" for more.
"""
begin_time = os.times()[4]
records_info = {
'report-numbers': {},
'journals': {},
'doi': {},
'hdl': {},
'isbn': {},
+ 'record_id': {},
}
references_info = {
'report-numbers': {},
'journals': {},
'doi': {},
'record_id': {},
'isbn': {},
'hdl': {},
}
# perform quick check to see if there are some records with
# reference tags, because otherwise get.cit.inf would be slow even
# if there is nothing to index:
for done, recid in enumerate(recid_list):
if done % 10 == 0:
task_sleep_now_if_required()
if done % 50 == 0:
mesg = "get cit.inf done %s of %s" % (done, len(recid_list))
write_message(mesg)
task_update_progress(mesg)
record = get_record(recid)
+ records_info['record_id'][recid] = [unicode(recid)]
function = config.get("rank_method", "function")
if config.get(function, 'collections'):
if recid not in recids_cache(config.get(function, 'collections')):
# do not treat this record since it is not in the collections
# we want to process
continue
elif recid in deleted_recids_cache():
# do not treat this record since it was deleted; we
# skip it like this in case it was only soft-deleted
# e.g. via bibedit (i.e. when collection tag 980 is
# DELETED but other tags like report number or journal
# publication info remained the same, so the calls to
# get_fieldvalues() below would return old values)
continue
if tags['refs_report_number']:
references_info['report-numbers'][recid] = [t.value for t in
record.find_subfields(tags['refs_report_number'])]
msg = "references_info['report-numbers'][%s] = %r" \
% (recid, references_info['report-numbers'][recid])
write_message(msg, verbose=9)
if tags['refs_journal']:
references_info['journals'][recid] = []
for ref in record.find_subfields(tags['refs_journal']):
try:
# Inspire specific parsing
journal, volume, page = ref.value.split(',')
except ValueError:
pass
else:
alt_volume = get_alt_volume(volume)
if alt_volume:
alt_ref = ','.join([journal, alt_volume, page])
references_info['journals'][recid] += [alt_ref]
references_info['journals'][recid] += [ref.value]
msg = "references_info['journals'][%s] = %r" \
% (recid, references_info['journals'][recid])
write_message(msg, verbose=9)
if tags['refs_doi']:
references = [t.value for t in
record.find_subfields(tags['refs_doi'])]
dois = []
hdls = []
for ref in references:
if ref.startswith("hdl:"):
hdls.append(ref[4:])
elif ref.startswith("doi:"):
dois.append(ref[4:])
else:
dois.append(ref)
references_info['doi'][recid] = dois
references_info['hdl'][recid] = hdls
msg = "references_info['doi'][%s] = %r" % (recid, dois)
write_message(msg, verbose=9)
msg = "references_info['hdl'][%s] = %r" % (recid, hdls)
write_message(msg, verbose=9)
if tags['refs_record_id']:
references_info['record_id'][recid] = [t.value for t in
record.find_subfields(tags['refs_record_id'])]
msg = "references_info['record_id'][%s] = %r" \
% (recid, references_info['record_id'][recid])
write_message(msg, verbose=9)
if tags['refs_isbn']:
references_info['isbn'][recid] = [t.value for t in
record.find_subfields(tags['refs_isbn'])]
msg = "references_info['isbn'][%s] = %r" \
% (recid, references_info['isbn'][recid])
write_message(msg, verbose=9)
if not fetch_catchup_info:
# We do not need the extra info
continue
if tags['record_pri_number'] or tags['record_add_number']:
records_info['report-numbers'][recid] = []
if tags['record_pri_number']:
records_info['report-numbers'][recid] += [t.value for t in
record.find_subfields(tags['record_pri_number'])]
if tags['record_add_number']:
records_info['report-numbers'][recid] += [t.value for t in
record.find_subfields(tags['record_add_number'])]
msg = "records_info[%s]['report-numbers'] = %r" \
% (recid, records_info['report-numbers'][recid])
write_message(msg, verbose=9)
if tags['doi']:
records_info['doi'][recid] = []
records_info['hdl'][recid] = []
for tag in tags['doi']:
for field in record.find_fields(tag[:5]):
if 'DOI' in field.get_subfield_values('2'):
dois = field.get_subfield_values('a')
records_info['doi'][recid].extend(dois)
elif 'HDL' in field.get_subfield_values('2'):
hdls = field.get_subfield_values('a')
records_info['hdl'][recid].extend(hdls)
msg = "records_info[%s]['doi'] = %r" \
% (recid, records_info['doi'][recid])
write_message(msg, verbose=9)
msg = "records_info[%s]['hdl'] = %r" \
% (recid, records_info['hdl'][recid])
write_message(msg, verbose=9)
if tags['isbn']:
records_info['isbn'][recid] = []
for tag in tags['isbn']:
values = [t.value for t in record.find_subfields(tag)]
records_info['isbn'][recid] += values
msg = "records_info[%s]['isbn'] = %r" \
% (recid, records_info['isbn'][recid])
write_message(msg, verbose=9)
# get a combination of
# journal vol (year) pages
if tags['publication']:
records_info['journals'][recid] = get_journal_info(record, tags)
msg = "records_info[%s]['journals'] = %r" \
% (recid, records_info['journals'][recid])
write_message(msg, verbose=9)
mesg = "get cit.inf done fully"
write_message(mesg)
task_update_progress(mesg)
end_time = os.times()[4]
write_message("Execution time for generating citation info "
"from record: %.2f sec" % (end_time - begin_time))
return records_info, references_info
def standardize_report_number(report_number):
"""Format the report number to a standard form.
Currently we:
* remove category for arxiv papers
"""
report_number = re.sub(ur'(?:arXiv:)?(\d{4}\.\d{4}) \[[a-zA-Z\.-]+\]',
ur'arXiv:\g<1>',
report_number,
re.I | re.U)
return report_number
def ref_analyzer(citation_informations, updated_recids, tags, config):
"""Analyze the citation informations and calculate the citation weight
and cited by list dictionary.
"""
citations = {}
for recid in updated_recids:
citations[recid] = set()
references = {}
for recid in updated_recids:
references[recid] = set()
def step(msg_prefix, recid, done, total):
if done % 30 == 0:
task_sleep_now_if_required()
if done % 1000 == 0:
mesg = "%s done %s of %s" % (msg_prefix, done, total)
write_message(mesg)
task_update_progress(mesg)
write_message("Processing: %s" % recid, verbose=9)
def add_to_cites(citer, citee):
# Make sure we don't add ourselves
# Workaround till we know why we are adding ourselves.
if citer == citee:
return
citations[citee].add(citer)
if citer in updated_recids:
references[citer].add(citee)
def add_to_refs(citer, citee):
# Make sure we don't add ourselves
# Workaround till we know why we are adding ourselves.
if citer == citee:
return
if citee in updated_recids:
citations[citee].add(citer)
references[citer].add(citee)
# dict of recid -> institute_give_publ_id
records_info, references_info = citation_informations
t1 = os.times()[4]
# Try to find references based on 999C5r
# e.g 8 -> ([astro-ph/9889],[hep-ph/768])
# meaning: rec 8 contains these in bibliography
write_message("Phase 1: Report numbers references")
done = 0
for thisrecid, refnumbers in iteritems(references_info['report-numbers']):
step("Report numbers references", thisrecid, done,
len(references_info['report-numbers']))
done += 1
for refnumber in (r for r in refnumbers if r):
field = 'reportnumber'
refnumber = standardize_report_number(refnumber)
# Search for "hep-th/5644654 or such" in existing records
recids = get_recids_matching_query(p=refnumber,
f=field,
config=config)
write_message("These match searching %s in %s: %s" %
(refnumber, field, list(recids)), verbose=9)
if not recids:
insert_into_missing(thisrecid, refnumber)
else:
remove_from_missing(refnumber)
if len(recids) > 1:
store_citation_warning('multiple-matches', refnumber)
msg = "Whoops: record '%d' report number value '%s' " \
"matches many records; taking only the first one. %s" % \
(thisrecid, refnumber, repr(recids))
write_message(msg, stream=sys.stderr)
for recid in list(recids)[:1]: # take only the first one
add_to_refs(thisrecid, recid)
mesg = "done fully"
write_message(mesg)
task_update_progress(mesg)
t2 = os.times()[4]
# Try to find references based on 999C5s
# e.g. Phys.Rev.Lett. 53 (1986) 2285
write_message("Phase 2: Journal references")
done = 0
for thisrecid, refs in iteritems(references_info['journals']):
step("Journal references", thisrecid, done,
len(references_info['journals']))
done += 1
for reference in (r for r in refs if r):
p = reference
field = 'journal'
# check reference value to see whether it is well formed:
if not re_CFG_JOURNAL_PUBINFO_STANDARD_FORM_REGEXP_CHECK.match(p):
store_citation_warning('not-well-formed', p)
msg = "Whoops, record '%d' reference value '%s' " \
"is not well formed; skipping it." % (thisrecid, p)
write_message(msg, stream=sys.stderr)
continue # skip this ill-formed value
recids = get_recids_matching_query(p=p,
f=field,
config=config)
write_message("These match searching %s in %s: %s"
% (reference, field, list(recids)), verbose=9)
if not recids:
insert_into_missing(thisrecid, p)
else:
remove_from_missing(p)
if len(recids) > 1:
store_citation_warning('multiple-matches', p)
msg = "Whoops: record '%d' reference value '%s' " \
"matches many records; taking only the first one. %s" % \
(thisrecid, p, repr(recids))
write_message(msg, stream=sys.stderr)
for recid in list(recids)[:1]: # take only the first one
add_to_refs(thisrecid, recid)
mesg = "done fully"
write_message(mesg)
task_update_progress(mesg)
t3 = os.times()[4]
# Try to find references based on 999C5a
# e.g. 10.1007/BF03170733
write_message("Phase 3: DOI references")
done = 0
for thisrecid, refs in iteritems(references_info['doi']):
step("DOI references", thisrecid, done, len(references_info['doi']))
done += 1
for reference in (r for r in refs if r):
p = reference
field = 'doi'
recids = get_recids_matching_query(p=p,
f=field,
config=config)
write_message("These match searching %s in %s: %s"
% (reference, field, list(recids)), verbose=9)
if not recids:
insert_into_missing(thisrecid, p)
else:
remove_from_missing(p)
if len(recids) > 1:
store_citation_warning('multiple-matches', p)
msg = "Whoops: record '%d' DOI value '%s' " \
"matches many records; taking only the first one. %s" % \
(thisrecid, p, repr(recids))
write_message(msg, stream=sys.stderr)
for recid in list(recids)[:1]: # take only the first one
add_to_refs(thisrecid, recid)
mesg = "done fully"
write_message(mesg)
task_update_progress(mesg)
t4 = os.times()[4]
# Try to find references based on 999C5a (hdl references)
# e.g. 4263537/4000
write_message("Phase 4: HDL references")
done = 0
for thisrecid, refs in references_info['hdl'].iteritems():
step("HDL references", thisrecid, done, len(references_info['hdl']))
done += 1
for reference in (r for r in refs if r):
p = reference
field = 'hdl'
recids = get_recids_matching_query(p=p,
f=field,
config=config)
write_message("These match searching %s in %s: %s"
% (reference, field, list(recids)), verbose=9)
if not recids:
insert_into_missing(thisrecid, p)
else:
remove_from_missing(p)
if len(recids) > 1:
store_citation_warning('multiple-matches', p)
msg = "Whoops: record '%d' HDL value '%s' " \
"matches many records; taking only the first one. %s" % \
(thisrecid, p, repr(recids))
write_message(msg, stream=sys.stderr)
for recid in list(recids)[:1]: # take only the first one
add_to_refs(thisrecid, recid)
mesg = "done fully"
write_message(mesg)
task_update_progress(mesg)
t5 = os.times()[4]
# Try to find references based on 999C50
# e.g. 1244
write_message("Phase 5: Record ID references")
done = 0
for thisrecid, refs in references_info['record_id'].iteritems():
step("Record ID references", thisrecid, done, len(references_info['record_id']))
done += 1
+ field = "001"
for recid in (r for r in refs if r):
- valid = get_recids_matching_query(p=recid, f="001", config=config)
+ valid = get_recids_matching_query(p=recid, f=field, config=config)
+ write_message("These match searching %s in %s: %s"
+ % (recid, field, list(valid)), verbose=9)
if valid:
add_to_refs(thisrecid, valid[0])
mesg = "done fully"
write_message(mesg)
task_update_progress(mesg)
t6 = os.times()[4]
# Try to find references based on 999C5i
# e.g. 978-3-942171-73-1
write_message("Phase 6: ISBN references")
done = 0
for thisrecid, refs in references_info['isbn'].iteritems():
step("ISBN references", thisrecid, done, len(references_info['isbn']))
done += 1
for reference in (r for r in refs if r):
p = reference
field = 'isbn'
recids = get_recids_matching_query(p=p,
f=field,
config=config)
write_message("These match searching %s in %s: %s"
% (reference, field, list(recids)), verbose=9)
if not recids:
insert_into_missing(thisrecid, p)
else:
remove_from_missing(p)
if len(recids) > 1:
store_citation_warning('multiple-matches', p)
msg = "Whoops: record '%d' ISBN value '%s' " \
"matches many records; taking only the first one. %s" % \
(thisrecid, p, repr(recids))
write_message(msg, stream=sys.stderr)
for recid in list(recids)[:1]: # take only the first one
add_to_refs(thisrecid, recid)
mesg = "done fully"
write_message(mesg)
task_update_progress(mesg)
t7 = os.times()[4]
# Search for stuff like CERN-TH-4859/87 in list of refs
write_message("Phase 7: report numbers catchup")
done = 0
for thisrecid, reportcodes in iteritems(records_info['report-numbers']):
step("Report numbers catchup", thisrecid, done,
len(records_info['report-numbers']))
done += 1
for reportcode in (r for r in reportcodes if r):
if reportcode.startswith('arXiv'):
std_reportcode = standardize_report_number(reportcode)
report_pattern = r'^%s( *\[[a-zA-Z.-]*\])?' % \
re.escape(std_reportcode)
recids = get_recids_matching_query(p=report_pattern,
f=tags['refs_report_number'],
m='r',
config=config)
else:
recids = get_recids_matching_query(p=reportcode,
f=tags['refs_report_number'],
config=config)
for recid in recids:
add_to_cites(recid, thisrecid)
mesg = "done fully"
write_message(mesg)
task_update_progress(mesg)
# Find this record's pubinfo in other records' bibliography
write_message("Phase 8: journals catchup")
done = 0
t8 = os.times()[4]
for thisrecid, rec_journals in iteritems(records_info['journals']):
step("Journals catchup", thisrecid, done,
len(records_info['journals']))
done += 1
for journal in rec_journals:
journal = journal.replace("\"", "")
# Search the publication string like
# Phys. Lett., B 482 (2000) 417 in 999C5s
recids = get_recids_matching_query(p=journal,
f=tags['refs_journal'],
config=config)
write_message("These records match %s in %s: %s"
% (journal, tags['refs_journal'], list(recids)), verbose=9)
for recid in recids:
add_to_cites(recid, thisrecid)
mesg = "done fully"
write_message(mesg)
task_update_progress(mesg)
write_message("Phase 9: DOI catchup")
done = 0
t9 = os.times()[4]
for thisrecid, dois in iteritems(records_info['doi']):
step("DOI catchup", thisrecid, done, len(records_info['doi']))
done += 1
for doi in dois:
recids = get_recids_matching_query(p=doi,
f=tags['refs_doi'],
config=config)
write_message("These records match %s in %s: %s"
% (doi, tags['refs_doi'], list(recids)), verbose=9)
for recid in recids:
add_to_cites(recid, thisrecid)
mesg = "done fully"
write_message(mesg)
task_update_progress(mesg)
write_message("Phase 10: HDL catchup")
done = 0
t10 = os.times()[4]
for thisrecid, hdls in records_info['hdl'].iteritems():
step("HDL catchup", thisrecid, done, len(records_info['hdl']))
done += 1
for hdl in hdls:
recids = get_recids_matching_query(p=hdl,
f=tags['refs_doi'],
config=config)
write_message("These records match %s in %s: %s"
% (hdl, tags['refs_doi'], list(recids)), verbose=9)
for recid in recids:
add_to_cites(recid, thisrecid)
mesg = "done fully"
write_message(mesg)
task_update_progress(mesg)
write_message("Phase 11: ISBN catchup")
done = 0
t11 = os.times()[4]
for thisrecid, isbns in records_info['isbn'].iteritems():
step("ISBN catchup", thisrecid, done, len(records_info['isbn']))
done += 1
for isbn in isbns:
recids = get_recids_matching_query(p=isbn,
f=tags['refs_isbn'],
config=config)
write_message("These records match %s in %s: %s"
% (isbn, tags['refs_isbn'], list(recids)), verbose=9)
for recid in recids:
add_to_cites(recid, thisrecid)
+ write_message("Phase 12: Record ID catchup")
+ done = 0
+ t12 = os.times()[4]
+ for thisrecid, record_ids in records_info['record_id'].iteritems():
+ step("Record ID catchup", thisrecid, done, len(records_info['record_id']))
+ done += 1
+
+ for record_id in record_ids:
+ recids = get_recids_matching_query(p=record_id,
+ f=tags['refs_record_id'],
+ config=config)
+ write_message("These records match %s in %s: %s"
+ % (record_id, tags['refs_record_id'], list(recids)), verbose=9)
+
+ for recid in recids:
+ add_to_cites(recid, thisrecid)
+
mesg = "done fully"
write_message(mesg)
task_update_progress(mesg)
if task_get_task_param('verbose') >= 3:
# Print only X first to prevent flood
write_message("citation_list (x is cited by y):")
write_message(dict(islice(iteritems(citations), 10)))
write_message("size: %s" % len(citations))
write_message("reference_list (x cites y):")
write_message(dict(islice(iteritems(references), 10)))
write_message("size: %s" % len(references))
- t12 = os.times()[4]
+ t13 = os.times()[4]
write_message("Execution time for analyzing the citation information "
"generating the dictionary:")
write_message("... checking ref report numbers: %.2f sec" % (t2-t1))
write_message("... checking ref journals: %.2f sec" % (t3-t2))
write_message("... checking ref DOI: %.2f sec" % (t4-t3))
write_message("... checking ref HDL: %.2f sec" % (t5-t4))
write_message("... checking ref Record ID: %.2f sec" % (t6-t5))
write_message("... checking ref ISBN: %.2f sec" % (t7-t6))
write_message("... checking rec report numbers: %.2f sec" % (t8-t7))
write_message("... checking rec journals: %.2f sec" % (t9-t8))
write_message("... checking rec DOI: %.2f sec" % (t10-t9))
write_message("... checking rec HDL: %.2f sec" % (t11-t10))
write_message("... checking rec ISBN: %.2f sec" % (t12-t11))
- write_message("... total time of ref_analyze: %.2f sec" % (t12-t1))
+ write_message("... checking rec Record ID: %.2f sec" % (t13-t12))
+ write_message("... total time of ref_analyze: %.2f sec" % (t13-t1))
return citations, references
def compute_refs_diff(recid, new_refs):
"""
Given a set of references for a record, returns how many references were
added to it. The value can be negative which means the record lost
citations.
"""
old_refs = set(row[0] for row in run_sql("""SELECT citee
FROM rnkCITATIONDICT
WHERE citer = %s""", [recid]))
refs_to_add = new_refs - old_refs
refs_to_delete = old_refs - new_refs
return len(refs_to_add) - len(refs_to_delete)
def compute_cites_diff(recid, new_cites):
"""
This function does the same thing as compute_refs_diff but with citations.
"""
old_cites = set(row[0] for row in run_sql("""SELECT citer
FROM rnkCITATIONDICT
WHERE citee = %s""", [recid]))
cites_to_add = new_cites - old_cites
cites_to_delete = old_cites - new_cites
return len(cites_to_add) - len(cites_to_delete)
def compute_dicts_diff(recids, refs, cites):
"""
Given the new dictionaries for references and citations, computes how
many references were added or removed by comparing them to the current
stored in the database.
"""
cites_diff = 0
for recid in recids:
cites_diff += compute_refs_diff(recid, refs[recid])
cites_diff += compute_cites_diff(recid, cites[recid])
return cites_diff
def store_dicts(recids, refs, cites):
"""Insert the reference and citation list into the database"""
for recid in recids:
replace_refs(recid, refs[recid])
replace_cites(recid, cites[recid])
def replace_refs(recid, new_refs):
"""
Given a set of references, replaces the references of given recid
in the database.
The changes are logged into rnkCITATIONLOG.
"""
old_refs = set(row[0] for row in run_sql("""SELECT citee
FROM rnkCITATIONDICT
WHERE citer = %s""", [recid]))
refs_to_add = new_refs - old_refs
refs_to_delete = old_refs - new_refs
for ref in refs_to_add:
write_message('adding ref %s %s' % (recid, ref), verbose=1)
now = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
run_sql("""INSERT INTO rnkCITATIONDICT (citer, citee, last_updated)
VALUES (%s, %s, %s)""", (recid, ref, now))
run_sql("""INSERT INTO rnkCITATIONLOG (citer, citee, type, action_date)
VALUES (%s, %s, %s, %s)""", (recid, ref, 'added', now))
for ref in refs_to_delete:
write_message('deleting ref %s %s' % (recid, ref), verbose=1)
now = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
run_sql("""DELETE FROM rnkCITATIONDICT
WHERE citer = %s and citee = %s""", (recid, ref))
run_sql("""INSERT INTO rnkCITATIONLOG (citer, citee, type, action_date)
VALUES (%s, %s, %s, %s)""", (recid, ref, 'removed', now))
def replace_cites(recid, new_cites):
"""
Given a set of citations, replaces the citations of given recid
in the database.
The changes are logged into rnkCITATIONLOG.
See @replace_refs
"""
old_cites = set(row[0] for row in run_sql("""SELECT citer
FROM rnkCITATIONDICT
WHERE citee = %s""", [recid]))
cites_to_add = new_cites - old_cites
cites_to_delete = old_cites - new_cites
for cite in cites_to_add:
write_message('adding cite %s %s' % (recid, cite), verbose=1)
now = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
run_sql("""INSERT INTO rnkCITATIONDICT (citee, citer, last_updated)
VALUES (%s, %s, %s)""", (recid, cite, now))
run_sql("""INSERT INTO rnkCITATIONLOG (citee, citer, type, action_date)
VALUES (%s, %s, %s, %s)""", (recid, cite, 'added', now))
for cite in cites_to_delete:
write_message('deleting cite %s %s' % (recid, cite), verbose=1)
now = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
run_sql("""DELETE FROM rnkCITATIONDICT
WHERE citee = %s and citer = %s""", (recid, cite))
run_sql("""INSERT INTO rnkCITATIONLOG (citee, citer, type, action_date)
VALUES (%s, %s, %s, %s)""", (recid, cite, 'removed', now))
def insert_into_missing(recid, report):
"""Mark reference string as missing.
If a reference is a report number / journal / DOI but we do not have
the corresping record in the database, we mark that particualar
reference string as missing, by adding a row in rnkCITATIONDATAEXT.
The recid represents the record containing the reference string.
"""
if len(report) >= 255:
# Invalid report, it is too long
# and does not fit in the database column
# (currently varchar 255)
return
wasalready = run_sql("""SELECT id_bibrec
FROM rnkCITATIONDATAEXT
WHERE id_bibrec = %s
AND extcitepubinfo = %s""",
(recid, report))
if not wasalready:
run_sql("""INSERT INTO rnkCITATIONDATAEXT(id_bibrec, extcitepubinfo)
VALUES (%s,%s)""", (recid, report))
def remove_from_missing(report):
"""Remove the reference string from the missing table
See @insert_into_missing"""
run_sql("""DELETE FROM rnkCITATIONDATAEXT
WHERE extcitepubinfo = %s""", (report,))
def print_missing(num):
"""
Print the contents of rnkCITATIONDATAEXT table containing external
records that were cited by NUM or more internal records.
NUM is by default taken from the -E command line option.
"""
if not num:
num = task_get_option("print-extcites")
write_message("Listing external papers cited by %i or more \
internal records:" % num)
res = run_sql("""SELECT COUNT(id_bibrec), extcitepubinfo
FROM rnkCITATIONDATAEXT
GROUP BY extcitepubinfo HAVING COUNT(id_bibrec) >= %s
ORDER BY COUNT(id_bibrec) DESC""", (num,))
for cnt, brec in res:
print(str(cnt), "\t", brec)
write_message("Listing done.")
def tagify(parsedtag):
"""aux auf to make '100__a' out of ['100','','','a']"""
tag = ""
for t in parsedtag:
if t == '':
t = '_'
tag += t
return tag
def store_citation_warning(warning_type, cit_info):
"""Store citation indexing warnings in the database
If we encounter a problem during the citation indexing, such as multiple
results for a report number, we store a warning in rnkCITATIONDATAERR
"""
r = run_sql("""SELECT 1 FROM rnkCITATIONDATAERR
WHERE type = %s
AND citinfo = %s""", (warning_type, cit_info))
if not r:
run_sql("""INSERT INTO rnkCITATIONDATAERR (type, citinfo)
VALUES (%s, %s)""", (warning_type, cit_info))
diff --git a/invenio/legacy/bibrank/citation_searcher.py b/invenio/legacy/bibrank/citation_searcher.py
index 673506448..b355256ac 100644
--- a/invenio/legacy/bibrank/citation_searcher.py
+++ b/invenio/legacy/bibrank/citation_searcher.py
@@ -1,331 +1,336 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
__revision__ = "$Id$"
import re
-from operator import itemgetter
-from six import iteritems
-
from invenio.legacy.dbquery import run_sql
from intbitset import intbitset
from invenio.legacy.miscutil.data_cacher import DataCacher
+from invenio.utils.redis import get_redis
+from invenio.legacy.dbquery import deserialize_via_marshal
+from operator import itemgetter
+from six import iteritems
class CitationDictsDataCacher(DataCacher):
"""
Cache holding all citation dictionaries (citationdict,
reversedict, selfcitdict, selfcitedbydict).
"""
def __init__(self):
- def initial_fill():
+
+ def fill():
alldicts = {}
from invenio.legacy.bibrank.tag_based_indexer import fromDB
- weights = fromDB('citation')
+ redis = get_redis()
+ serialized_weights = redis.get('citations_weights')
+ if serialized_weights:
+ weights = deserialize_via_marshal(serialized_weights)
+ else:
+ weights = fromDB('citation')
alldicts['citations_weights'] = weights
# for cited:M->N queries, it is interesting to cache also
# some preprocessed citationdict:
alldicts['citations_keys'] = intbitset(weights.keys())
# Citation counts
alldicts['citations_counts'] = [t for t in iteritems(weights)]
alldicts['citations_counts'].sort(key=itemgetter(1), reverse=True)
# Self-cites
- selfcites = fromDB('selfcites')
+ serialized_weights = redis.get('selfcites_weights')
+ if serialized_weights:
+ selfcites = deserialize_via_marshal(serialized_weights)
+ else:
+ selfcites = fromDB('selfcites')
selfcites_weights = {}
for recid, counts in alldicts['citations_counts']:
selfcites_weights[recid] = counts - selfcites.get(recid, 0)
alldicts['selfcites_weights'] = selfcites_weights
alldicts['selfcites_counts'] = [(recid, selfcites_weights.get(recid, cites)) for recid, cites in alldicts['citations_counts']]
alldicts['selfcites_counts'].sort(key=itemgetter(1), reverse=True)
return alldicts
- def incremental_fill():
- self.cache = None
- return initial_fill()
-
def cache_filler():
- if self.cache:
- cache = incremental_fill()
- else:
- cache = initial_fill()
- return cache
+ self.cache = None # misfire from pylint: disable=W0201
+ # this is really defined in DataCacher
+ return fill()
from invenio.bibrank_tag_based_indexer import get_lastupdated
def timestamp_verifier():
citation_lastupdate = get_lastupdated('citation')
if citation_lastupdate:
return citation_lastupdate.strftime("%Y-%m-%d %H:%M:%S")
else:
return "0000-00-00 00:00:00"
DataCacher.__init__(self, cache_filler, timestamp_verifier)
CACHE_CITATION_DICTS = None
def get_citation_dict(dictname):
"""
Returns a cached value of a citation dictionary. Performs lazy
loading, i.e. loads the dictionary the first time it is actually
used.
@param dictname: the name of the citation dictionary to return. Can
be citationdict, reversedict, selfcitdict, selfcitedbydict.
@type dictname: string
@return: a citation dictionary. The structure of the dictionary is
{ recid -> [list of recids] }.
@rtype: dictionary
"""
global CACHE_CITATION_DICTS
if CACHE_CITATION_DICTS is None:
CACHE_CITATION_DICTS = CitationDictsDataCacher()
else:
CACHE_CITATION_DICTS.recreate_cache_if_needed()
return CACHE_CITATION_DICTS.cache[dictname]
def get_refers_to(recordid):
"""Return a list of records referenced by this record"""
rows = run_sql("SELECT citee FROM rnkCITATIONDICT WHERE citer = %s",
[recordid])
return set(r[0] for r in rows)
def get_cited_by(recordid):
"""Return a list of records that cite recordid"""
rows = run_sql("SELECT citer FROM rnkCITATIONDICT WHERE citee = %s",
[recordid])
return set(r[0] for r in rows)
def get_cited_by_count(recordid):
"""Return how many records cite given RECORDID."""
rows = run_sql("SELECT 1 FROM rnkCITATIONDICT WHERE citee = %s",
[recordid])
return len(rows)
def get_records_with_num_cites(numstr, allrecs=intbitset([]),
exclude_selfcites=False):
"""Return an intbitset of record IDs that are cited X times,
X defined in numstr.
Warning: numstr is string and may not be numeric! It can
be 10,0->100 etc
"""
if exclude_selfcites:
cache_cited_by_dictionary_counts = get_citation_dict("selfcites_counts")
citations_keys = intbitset(get_citation_dict("selfcites_weights").keys())
else:
cache_cited_by_dictionary_counts = get_citation_dict("citations_counts")
citations_keys = get_citation_dict("citations_keys")
matches = intbitset()
#once again, check that the parameter is a string
if type(numstr) != type("thisisastring"):
return matches
numstr = numstr.replace(" ", '')
numstr = numstr.replace('"', '')
num = 0
#first, check if numstr is just a number
singlenum = re.findall("^\d+$", numstr)
if singlenum:
num = int(singlenum[0])
if num == 0:
#we return recids that are not in keys
return allrecs - citations_keys
else:
return intbitset([recid for recid, cit_count
in cache_cited_by_dictionary_counts
if cit_count == num])
# Try to get 1->10 or such
firstsec = re.findall("(\d+)->(\d+)", numstr)
if firstsec:
first = int(firstsec[0][0])
sec = int(firstsec[0][1])
if first == 0:
# Start with those that have no cites..
matches = allrecs - citations_keys
if first <= sec:
matches += intbitset([recid for recid, cit_count
in cache_cited_by_dictionary_counts
if first <= cit_count <= sec])
return matches
# Try to get 10+
firstsec = re.findall("(\d+)\+", numstr)
if firstsec:
first = int(firstsec[0])
matches = intbitset([recid for recid, cit_count
in cache_cited_by_dictionary_counts \
if cit_count > first])
return matches
def get_cited_by_list(recids):
"""Return a tuple of ([recid,list_of_citing_records],...) for all the
records in recordlist.
"""
if not recids:
return []
in_sql = ','.join('%s' for dummy in recids)
rows = run_sql("""SELECT citer, citee FROM rnkCITATIONDICT
WHERE citee IN (%s)""" % in_sql, recids)
cites = {}
for citer, citee in rows:
cites.setdefault(citee, set()).add(citer)
return [(recid, cites.get(recid, set())) for recid in recids]
def get_refers_to_list(recids):
"""Return a tuple of ([recid,list_of_citing_records],...) for all the
records in recordlist.
"""
if not recids:
return []
in_sql = ','.join('%s' for dummy in recids)
rows = run_sql("""SELECT citee, citer FROM rnkCITATIONDICT
WHERE citer IN (%s)""" % in_sql, recids)
refs = {}
for citee, citer in rows:
refs.setdefault(citer, set()).add(citee)
return [(recid, refs.get(recid, set())) for recid in recids]
def get_refersto_hitset(ahitset):
"""
Return a hitset of records that refers to (cite) some records from
the given ahitset. Useful for search engine's
refersto:author:ellis feature.
"""
out = intbitset()
if ahitset:
try:
iter(ahitset)
except OverflowError:
# ignore attempt to iterate over infinite ahitset
pass
else:
in_sql = ','.join('%s' for dummy in ahitset)
rows = run_sql("""SELECT citer FROM rnkCITATIONDICT
WHERE citee IN (%s)""" % in_sql, ahitset)
out = intbitset(rows)
return out
def get_one_cited_by_weight(recID):
"""Returns a number_of_citing_records for one record
"""
weight = get_citation_dict("citations_weights")
return weight.get(recID, 0)
def get_cited_by_weight(recordlist):
"""Return a tuple of ([recid,number_of_citing_records],...) for all the
records in recordlist.
"""
weights = get_citation_dict("citations_weights")
result = []
for recid in recordlist:
result.append([recid, weights.get(recid, 0)])
return result
def get_citedby_hitset(ahitset):
"""
Return a hitset of records that are cited by records in the given
ahitset. Useful for search engine's citedby:author:ellis feature.
"""
out = intbitset()
if ahitset:
try:
iter(ahitset)
except OverflowError:
# ignore attempt to iterate over infinite ahitset
pass
else:
in_sql = ','.join('%s' for dummy in ahitset)
rows = run_sql("""SELECT citee FROM rnkCITATIONDICT
WHERE citer IN (%s)""" % in_sql, ahitset)
out = intbitset(rows)
return out
def calculate_cited_by_list(record_id, sort_order="d"):
"""Return a tuple of ([recid,citation_weight],...) for all the
record citing RECORD_ID. The resulting recids is sorted by
ascending/descending citation weights depending or SORT_ORDER.
"""
result = []
citation_list = get_cited_by(record_id)
# Add weights i.e. records that cite each of the entries in citation_list
weights = get_citation_dict("citations_weights")
for c in citation_list:
result.append([c, weights.get(c, 0)])
# sort them
reverse = sort_order == "d"
result.sort(key=itemgetter(1), reverse=reverse)
return result
def calculate_co_cited_with_list(record_id, sort_order="d"):
"""Return a tuple of ([recid,co-cited weight],...) for records
that are co-cited with RECORD_ID. The resulting recids is sorted by
ascending/descending citation weights depending or SORT_ORDER.
"""
result = []
result_intermediate = {}
for cit_id in get_cited_by(record_id):
for ref_id in get_refers_to(cit_id):
if ref_id not in result_intermediate:
result_intermediate[ref_id] = 1
else:
result_intermediate[ref_id] += 1
for key, value in iteritems(result_intermediate):
if key != record_id:
result.append([key, value])
reverse = sort_order == "d"
result.sort(key=itemgetter(1), reverse=reverse)
return result
def get_citers_log(recid):
return run_sql("""SELECT citer, type, action_date
FROM rnkCITATIONLOG
WHERE citee = %s
ORDER BY action_date DESC""", [recid])
diff --git a/invenio/legacy/bibrank/record_sorter.py b/invenio/legacy/bibrank/record_sorter.py
index 8cbbd16b8..a727ee122 100644
--- a/invenio/legacy/bibrank/record_sorter.py
+++ b/invenio/legacy/bibrank/record_sorter.py
@@ -1,462 +1,462 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""Ranking of records using different parameters and methods on the fly."""
import time
import re
import ConfigParser
from operator import itemgetter
from six import iteritems
from invenio.config import \
CFG_SITE_LANG, \
CFG_ETCDIR, \
CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS, \
CFG_WEBSEARCH_CITESUMMARY_SCAN_THRESHOLD, \
CFG_ETCDIR, \
CFG_WEBSEARCH_CITESUMMARY_SCAN_THRESHOLD
from invenio.legacy.dbquery import run_sql, \
deserialize_via_marshal, \
wash_table_column_name
from invenio.ext.logging import register_exception
from invenio.legacy.webpage import adderrorbox
from invenio.legacy.bibindex.engine_stopwords import is_stopword
from invenio.legacy.bibrank.citation_searcher import get_cited_by, \
get_cited_by_weight
from intbitset import intbitset
from invenio.legacy.bibrank.word_searcher import find_similar
# Do not remove these lines
# it is necessary for func_object = globals().get(function)
from invenio.legacy.bibrank.word_searcher import word_similarity
from invenio.legacy.miscutil.solrutils_bibrank_searcher import word_similarity_solr
from invenio.legacy.miscutil.xapianutils_bibrank_searcher import word_similarity_xapian
from invenio.modules.ranker.registry import configuration
METHODS = {}
def compare_on_val(first, second):
return cmp(second[1], first[1])
def check_term(term, col_size, term_rec, max_occ, min_occ, termlength):
"""Check if the tem is valid for use
term - the term to check
col_size - the number of records in database
term_rec - the number of records which contains this term
max_occ - max frequency of the term allowed
min_occ - min frequence of the term allowed
termlength - the minimum length of the terms allowed"""
try:
if is_stopword(term) or (len(term) <= termlength) or ((float(term_rec) / float(col_size)) >= max_occ) or ((float(term_rec) / float(col_size)) <= min_occ):
return ""
if int(term):
return ""
except StandardError:
pass
return "true"
def create_external_ranking_settings(rank_method_code, config):
METHODS[rank_method_code]['fields'] = dict()
sections = config.sections()
field_pattern = re.compile('field[0-9]+')
for section in sections:
if field_pattern.search(section):
field_name = config.get(section, 'name')
METHODS[rank_method_code]['fields'][field_name] = dict()
for option in config.options(section):
if option != 'name':
create_external_ranking_option(section, option, METHODS[rank_method_code]['fields'][field_name], config)
elif section == 'find_similar_to_recid':
METHODS[rank_method_code][section] = dict()
for option in config.options(section):
create_external_ranking_option(section, option, METHODS[rank_method_code][section], config)
elif section == 'field_settings':
for option in config.options(section):
create_external_ranking_option(section, option, METHODS[rank_method_code], config)
def create_external_ranking_option(section, option, dictionary, config):
value = config.get(section, option)
if value.isdigit():
value = int(value)
dictionary[option] = value
def create_rnkmethod_cache():
"""Create cache with vital information for each rank method."""
bibrank_meths = run_sql("SELECT name from rnkMETHOD")
for (rank_method_code,) in bibrank_meths:
filepath = configuration.get(rank_method_code + '.cfg', '')
config = ConfigParser.ConfigParser()
try:
config.readfp(open(filepath))
except IOError:
pass
cfg_function = config.get("rank_method", "function")
if config.has_section(cfg_function):
METHODS[rank_method_code] = {}
METHODS[rank_method_code]["function"] = cfg_function
METHODS[rank_method_code]["prefix"] = config.get(cfg_function, "relevance_number_output_prologue")
METHODS[rank_method_code]["postfix"] = config.get(cfg_function, "relevance_number_output_epilogue")
METHODS[rank_method_code]["chars_alphanumericseparators"] = r"[1234567890\!\"\#\$\%\&\'\(\)\*\+\,\-\.\/\:\;\<\=\>\?\@\[\\\]\^\_\`\{\|\}\~]"
else:
raise Exception("Error in configuration config_file: %s" % (config_file + ".cfg", ))
i8n_names = run_sql("""SELECT ln,value from rnkMETHODNAME,rnkMETHOD where id_rnkMETHOD=rnkMETHOD.id and rnkMETHOD.name=%s""", (rank_method_code,))
for (ln, value) in i8n_names:
METHODS[rank_method_code][ln] = value
if config.has_option(cfg_function, "table"):
METHODS[rank_method_code]["rnkWORD_table"] = config.get(cfg_function, "table")
query = "SELECT count(*) FROM %sR" % wash_table_column_name(METHODS[rank_method_code]["rnkWORD_table"][:-1])
METHODS[rank_method_code]["col_size"] = run_sql(query)[0][0]
if config.has_option(cfg_function, "stemming") and config.get(cfg_function, "stemming"):
try:
METHODS[rank_method_code]["stemmer"] = config.get(cfg_function, "stemming")
except KeyError:
pass
if config.has_option(cfg_function, "stopword"):
METHODS[rank_method_code]["stopwords"] = config.get(cfg_function, "stopword")
if config.has_section("find_similar"):
METHODS[rank_method_code]["max_word_occurence"] = float(config.get("find_similar", "max_word_occurence"))
METHODS[rank_method_code]["min_word_occurence"] = float(config.get("find_similar", "min_word_occurence"))
METHODS[rank_method_code]["min_word_length"] = int(config.get("find_similar", "min_word_length"))
METHODS[rank_method_code]["min_nr_words_docs"] = int(config.get("find_similar", "min_nr_words_docs"))
METHODS[rank_method_code]["max_nr_words_upper"] = int(config.get("find_similar", "max_nr_words_upper"))
METHODS[rank_method_code]["max_nr_words_lower"] = int(config.get("find_similar", "max_nr_words_lower"))
METHODS[rank_method_code]["default_min_relevance"] = int(config.get("find_similar", "default_min_relevance"))
if cfg_function in ('word_similarity_solr', 'word_similarity_xapian'):
create_external_ranking_settings(rank_method_code, config)
if config.has_section("combine_method"):
i = 1
METHODS[rank_method_code]["combine_method"] = []
while config.has_option("combine_method", "method%s" % i):
METHODS[rank_method_code]["combine_method"].append(config.get("combine_method", "method%s" % i).split(","))
i += 1
def is_method_valid(colID, rank_method_code):
"""
Check if RANK_METHOD_CODE method is valid for the collection given.
If colID is None, then check for existence regardless of collection.
"""
if colID is None:
return run_sql("SELECT COUNT(*) FROM rnkMETHOD WHERE name=%s", (rank_method_code,))[0][0]
enabled_colls = dict(run_sql("SELECT id_collection, score from collection_rnkMETHOD,rnkMETHOD WHERE id_rnkMETHOD=rnkMETHOD.id AND name=%s", (rank_method_code,)))
try:
colID = int(colID)
except TypeError:
return 0
if colID in enabled_colls:
return 1
else:
while colID:
colID = run_sql("SELECT id_dad FROM collection_collection WHERE id_son=%s", (colID,))
if colID and colID[0][0] in enabled_colls:
return 1
elif colID:
colID = colID[0][0]
return 0
def get_bibrank_methods(colID, ln=CFG_SITE_LANG):
"""
Return a list of rank methods enabled for collection colID and the
name of them in the language defined by the ln parameter.
"""
if 'methods' not in globals():
create_rnkmethod_cache()
avail_methods = []
for rank_method_code, options in iteritems(METHODS):
if "function" in options and is_method_valid(colID, rank_method_code):
if ln in options:
avail_methods.append((rank_method_code, options[ln]))
elif CFG_SITE_LANG in options:
avail_methods.append((rank_method_code, options[CFG_SITE_LANG]))
else:
avail_methods.append((rank_method_code, rank_method_code))
return avail_methods
def citation(rank_method_code, related_to, hitset, rank_limit_relevance, verbose):
"""Sort records by number of citations"""
if related_to:
from invenio.search_engine import search_pattern
hits = intbitset()
for pattern in related_to:
hits |= hitset & intbitset(search_pattern(p='refersto:%s' % pattern))
else:
hits = hitset
return rank_by_citations(hits, verbose)
def rank_records(rank_method_code, rank_limit_relevance, hitset, related_to=[], verbose=0, field='', rg=None, jrec=None):
"""Sorts given records or related records according to given method
Parameters:
- rank_method_code: Sort records using this method
e.g. `jif' or `sbr' (word frequency vector model)
- rank_limit_relevance: A parameter given to the sorting method
e.g. `23' for `nbc' (number of citations)
or `0.10' for `vec'
This is ignored when sorting by
citations. But I don't know what it means.
- hitset: records to sort
- related_to: if specified, instead of sorting given records,
we first fetch the related records ("related" being
defined by the method), then we sort these related
records
- verbose, verbose level
- field: stuff
- rg: more stuff
- jrec: even more stuff
Output:
- list of records
- list of rank values
- prefix, useless it is always '('
- postfix, useless it is always ')'
- verbose_output
"""
voutput = ""
configcreated = ""
starttime = time.time()
afterfind = starttime - time.time()
aftermap = starttime - time.time()
try:
# We are receiving a global hitset
hitset_global = hitset
hitset = intbitset(hitset_global)
if 'methods' not in globals():
create_rnkmethod_cache()
function = METHODS[rank_method_code]["function"]
# Check if we have specific function for sorting by this method
func_object = globals().get(function)
if verbose > 0:
voutput += "function: %s <br/> " % function
voutput += "related_to: %s <br/>" % str(related_to)
if func_object and related_to and related_to[0][0:6] == "recid:" and function == "word_similarity":
result = find_similar(rank_method_code, related_to[0][6:], hitset, rank_limit_relevance, verbose, METHODS)
elif func_object:
if function == "word_similarity":
result = func_object(rank_method_code, related_to, hitset, rank_limit_relevance, verbose, METHODS)
elif function in ("word_similarity_solr", "word_similarity_xapian"):
if not rg:
rg = CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS
if not jrec:
jrec = 0
ranked_result_amount = rg + jrec
if verbose > 0:
voutput += "Ranked result amount: %s<br/><br/>" % ranked_result_amount
if verbose > 0:
voutput += "field: %s<br/>" % field
if function == "word_similarity_solr":
if verbose > 0:
voutput += "In Solr part:<br/>"
result = word_similarity_solr(related_to, hitset, METHODS[rank_method_code], verbose, field, ranked_result_amount)
if function == "word_similarity_xapian":
if verbose > 0:
voutput += "In Xapian part:<br/>"
result = word_similarity_xapian(related_to, hitset, METHODS[rank_method_code], verbose, field, ranked_result_amount)
else:
result = func_object(rank_method_code, related_to, hitset, rank_limit_relevance, verbose)
else:
result = rank_by_method(rank_method_code, related_to, hitset, rank_limit_relevance, verbose)
except Exception as e:
register_exception()
result = (None, "", adderrorbox("An error occured when trying to rank the search result "+rank_method_code, ["Unexpected error: %s<br />" % (e,)]), voutput)
afterfind = time.time() - starttime
if result[0] and result[1]: #split into two lists for search_engine
results_similar_recIDs = [x[0] for x in result[0]]
results_similar_relevances = [x[1] for x in result[0]]
result = (results_similar_recIDs, results_similar_relevances, result[1], result[2], "%s%s" % (configcreated, result[3]))
aftermap = time.time() - starttime
else:
result = (None, None, result[1], result[2], result[3])
#add stuff from here into voutput from result
tmp = voutput+result[4]
if verbose > 0:
tmp += "<br/>Elapsed time after finding: %s\nElapsed after mapping: %s" % (afterfind, aftermap)
result = (result[0], result[1], result[2], result[3], tmp)
#dbg = string.join(map(str,methods[rank_method_code].items()))
#result = (None, "", adderrorbox("Debug ",rank_method_code+" "+dbg),"",voutput)
return result
def combine_method(rank_method_code, pattern, hitset, rank_limit_relevance, verbose):
"""combining several methods into one based on methods/percentage in config file"""
voutput = ""
result = {}
try:
for (method, percent) in METHODS[rank_method_code]["combine_method"]:
function = METHODS[method]["function"]
func_object = globals().get(function)
percent = int(percent)
if func_object:
this_result = func_object(method, pattern, hitset, rank_limit_relevance, verbose)[0]
else:
this_result = rank_by_method(method, pattern, hitset, rank_limit_relevance, verbose)[0]
for i in range(0, len(this_result)):
(recID, value) = this_result[i]
if value > 0:
result[recID] = result.get(recID, 0) + int((float(i) / len(this_result)) * float(percent))
result = result.items()
result.sort(lambda x, y: cmp(x[1], y[1]))
return (result, "(", ")", voutput)
except Exception:
return (None, "Warning: %s method cannot be used for ranking your query." % rank_method_code, "", voutput)
def rank_by_method(rank_method_code, lwords, hitset, rank_limit_relevance, verbose):
"""Ranking of records based on predetermined values.
input:
rank_method_code - the code of the method, from the name field in rnkMETHOD, used to get predetermined values from
rnkMETHODDATA
lwords - a list of words from the query
hitset - a list of hits for the query found by search_engine
rank_limit_relevance - show only records with a rank value above this
verbose - verbose value
output:
reclist - a list of sorted records, with unsorted added to the end: [[23,34], [344,24], [1,01]]
prefix - what to show before the rank value
postfix - what to show after the rank value
voutput - contains extra information, content dependent on verbose value"""
voutput = ""
rnkdict = run_sql("SELECT relevance_data FROM rnkMETHODDATA,rnkMETHOD where rnkMETHOD.id=id_rnkMETHOD and rnkMETHOD.name=%s", (rank_method_code,))
if not rnkdict:
return (None, "Warning: Could not load ranking data for method %s." % rank_method_code, "", voutput)
max_recid = 0
res = run_sql("SELECT max(id) FROM bibrec")
if res and res[0][0]:
max_recid = int(res[0][0])
lwords_hitset = None
for j in range(0, len(lwords)): #find which docs to search based on ranges..should be done in search_engine...
if lwords[j] and lwords[j][:6] == "recid:":
if not lwords_hitset:
lwords_hitset = intbitset()
lword = lwords[j][6:]
if lword.find("->") > -1:
lword = lword.split("->")
if int(lword[0]) >= max_recid or int(lword[1]) >= max_recid + 1:
return (None, "Warning: Given record IDs are out of range.", "", voutput)
for i in range(int(lword[0]), int(lword[1])):
lwords_hitset.add(int(i))
elif lword < max_recid + 1:
lwords_hitset.add(int(lword))
else:
return (None, "Warning: Given record IDs are out of range.", "", voutput)
rnkdict = deserialize_via_marshal(rnkdict[0][0])
if verbose > 0:
voutput += "<br />Running rank method: %s, using rank_by_method function in bibrank_record_sorter<br />" % rank_method_code
voutput += "Ranking data loaded, size of structure: %s<br />" % len(rnkdict)
lrecIDs = list(hitset)
if verbose > 0:
voutput += "Number of records to rank: %s<br />" % len(lrecIDs)
reclist = []
reclist_addend = []
if not lwords_hitset: #rank all docs, can this be speed up using something else than for loop?
for recID in lrecIDs:
if recID in rnkdict:
reclist.append((recID, rnkdict[recID]))
del rnkdict[recID]
else:
reclist_addend.append((recID, 0))
else: #rank docs in hitset, can this be speed up using something else than for loop?
for recID in lwords_hitset:
if recID in rnkdict and recID in hitset:
reclist.append((recID, rnkdict[recID]))
del rnkdict[recID]
elif recID in hitset:
reclist_addend.append((recID, 0))
if verbose > 0:
voutput += "Number of records ranked: %s<br />" % len(reclist)
voutput += "Number of records not ranked: %s<br />" % len(reclist_addend)
reclist.sort(lambda x, y: cmp(x[1], y[1]))
return (reclist_addend + reclist, METHODS[rank_method_code]["prefix"], METHODS[rank_method_code]["postfix"], voutput)
def rank_by_citations(hitset, verbose):
"""Rank by the amount of citations.
Calculate the cited-by values for all the members of the hitset
Rreturns: ((recordid,weight),prefix,postfix,message)
"""
voutput = ""
if len(hitset) > CFG_WEBSEARCH_CITESUMMARY_SCAN_THRESHOLD:
cites_counts = get_citation_dict('citations_counts')
ret = [(recid, weight) for recid, weight in cites_counts
if recid in hitset]
recids_without_cites = hitset - get_citation_dict('citations_keys')
ret.extend([(recid, 0) for recid in recids_without_cites])
ret = list(reversed(ret))
else:
ret = get_cited_by_weight(hitset)
ret.sort(key=itemgetter(1))
if verbose > 0:
- voutput += "\nhitset %s\nfind_citations ret %s" % (hitset, ret)
+ voutput += "\nhitset %s\nrank_by_citations ret %s" % (hitset, ret)
if ret:
return ret, "(", ")", voutput
else:
return [], "", "", voutput
diff --git a/invenio/legacy/bibrank/selfcites_indexer.py b/invenio/legacy/bibrank/selfcites_indexer.py
index da38ebaa4..81d671e0f 100644
--- a/invenio/legacy/bibrank/selfcites_indexer.py
+++ b/invenio/legacy/bibrank/selfcites_indexer.py
@@ -1,346 +1,346 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""Self-citations indexer
We store the records and authors in a faster to access way than directly
accessing the bibrecs tables.
We have 3 tables:
1. rnkAUTHORS to associate records to authors in a speedy way
2. rnkEXTENDEDAUTHORS to associate co-authors with bibrecs
for a given bibrec, it provides a fast way to access all the authors of
the bibrec but also the people they have written papers with
3. rnkSELFCITES used by search_engine_summarizer for displaying the self-
citations count.
"""
from itertools import chain
import ConfigParser
from invenio.modules.formatter.utils import parse_tag
from invenio.legacy.bibrecord import get_fieldvalues
from invenio.legacy.bibrank.citation_indexer import tagify
from invenio.config import CFG_BIBRANK_SELFCITES_USE_BIBAUTHORID, \
CFG_BIBRANK_SELFCITES_PRECOMPUTE
from invenio.legacy.dbquery import run_sql
from invenio.legacy.bibrank.citation_searcher import get_cited_by
from invenio.modules.ranker.registry import configuration
def load_config_file(key):
"""Load config file containing the authors, co-authors tags #"""
filename = configuration.get(key + '.cfg', '')
config = ConfigParser.ConfigParser()
try:
config.readfp(open(filename))
except StandardError:
raise Exception('Unable to load config file %s' % filename)
return config
def get_personids_from_record(record):
"""Returns all the personids associated to a record.
We limit the result length to 20 authors, after which it returns an
empty set for performance reasons
"""
ids = get_authors_of_claimed_paper(record)
if 0 < len(ids) <= 20:
person_ids = set(ids)
else:
person_ids = set()
return person_ids
def get_authors_tags():
"""
Get the tags for main author, coauthors, alternative authors from config
"""
config = load_config_file('citation')
function = config.get("rank_method", "function")
tags_names = [
'first_author',
'additional_author',
'alternative_author_name',
'collaboration_name',
]
tags = {}
for t in tags_names:
r_tag = config.get(function, t)
tags[t] = tagify(parse_tag(r_tag))
return tags
def get_authors_from_record(recID, tags,
use_bibauthorid=CFG_BIBRANK_SELFCITES_USE_BIBAUTHORID):
"""Get all authors for a record
We need this function because there's 3 different types of authors
and to fetch each one of them we need look through MARC tags
"""
if use_bibauthorid:
authors = get_personids_from_record(recID)
else:
authors_list = chain(
get_fieldvalues(recID, tags['first_author']),
get_fieldvalues(recID, tags['additional_author']),
get_fieldvalues(recID, tags['alternative_author_name']))
- authors = set(hash(author) for author in list(authors_list)[:20])
+ authors = set(hash(author) for author in list(authors_list)[:21])
return authors
def get_collaborations_from_record(recID, tags):
"""Get all collaborations for a record"""
return get_fieldvalues(recID, tags['collaboration_name'])
def compute_self_citations(recid, tags, authors_fun):
"""Compute the self-citations
We return the total numbers of citations minus the number of self-citations
Args:
- recid: record id
- lciters: list of record ids citing this record
- authors_cache: the authors cache which will be used to store an author
friends (to not compute friends twice)
- tags: the tag number for author, coauthors, collaborations,
required since it depends on how the marc was defined
"""
citers = get_cited_by(recid)
if not citers:
return set()
self_citations = set()
authors = frozenset(get_authors_from_record(recid, tags))
collaborations = None
if not authors or len(authors) > 20:
collaborations = frozenset(
get_collaborations_from_record(recid, tags))
if collaborations:
# Use collaborations names
for cit in citers:
cit_collaborations = frozenset(
get_collaborations_from_record(cit, tags))
if collaborations.intersection(cit_collaborations):
self_citations.add(cit)
else:
# Use authors names
for cit in citers:
cit_authors = get_authors_from_record(cit, tags)
if (not authors or len(cit_authors) > 20) and \
get_collaborations_from_record(cit, tags):
# Record from a collaboration that cites
# a record from an author, it's fine
pass
else:
cit_coauthors = frozenset(authors_fun(cit, tags))
if authors.intersection(cit_coauthors):
self_citations.add(cit)
return self_citations
def fetch_references(recid):
"""Fetch the references stored in the self-citations table for given record
We need to store the references to make sure that when we do incremental
updates of the table, we update all the related records properly
"""
sql = "SELECT `references` FROM rnkSELFCITES WHERE id_bibrec = %s"
try:
references = run_sql(sql, (recid, ))[0][0]
except IndexError:
references = ''
if references:
ids = set(int(ref) for ref in references.split(','))
else:
ids = set()
return ids
def get_precomputed_self_cites_list(recids):
"""Fetch pre-computed self-cites data for given records"""
in_sql = ','.join('%s' for dummy in recids)
sql = """SELECT id_bibrec, count
FROM rnkSELFCITES
WHERE id_bibrec IN (%s)""" % in_sql
return run_sql(sql, recids)
def get_precomputed_self_cites(recid):
"""Fetch pre-computed self-cites data for given record"""
sql = "SELECT count FROM rnkSELFCITES WHERE id_bibrec = %s"
try:
r = run_sql(sql, (recid, ))[0][0]
except IndexError:
r = None
return r
def compute_friends_self_citations(recid, tags):
def coauthors(recid, tags):
return set(get_record_coauthors(recid)) \
| set(get_authors_from_record(recid, tags))
return compute_self_citations(recid, tags, coauthors)
def compute_simple_self_citations(recid, tags):
"""Simple compute self-citations
The purpose of this algorithm is to provide an alternate way to compute
self-citations that we can use at runtime.
Here, we only check for authors citing themselves.
"""
return compute_self_citations(recid, tags, get_authors_from_record)
def get_self_citations_count(recids, algorithm='simple',
precompute=CFG_BIBRANK_SELFCITES_PRECOMPUTE):
"""Depending on our site we config, we either:
* compute self-citations (using a simple algorithm)
* or fetch self-citations from pre-computed table"""
total_cites = 0
if not precompute:
tags = get_authors_tags()
selfcites_fun = ALL_ALGORITHMS[algorithm]
for recid in recids:
citers = get_cited_by(recid)
self_cites = selfcites_fun(recid, tags)
total_cites += len(citers) - len(self_cites)
else:
results = get_precomputed_self_cites_list(recids)
results_dict = {}
for r in results:
results_dict[r[0]] = r[1]
for r in recids:
citers = get_cited_by(r)
self_cites = results_dict.get(r, 0)
total_cites += len(citers) - self_cites
return total_cites
def update_self_cites_tables(recid, config, tags):
"""For a given record update all self-cites table if needed"""
authors = get_authors_from_record(recid, tags)
if 0 < len(authors) <= 20:
# Updated reords cache table
deleted_authors, added_authors = store_record(recid, authors)
if deleted_authors or added_authors:
# Update extended authors table
store_record_coauthors(recid,
authors,
deleted_authors,
added_authors,
config)
def store_record(recid, authors):
"""
For a given record, updates if needed the db table (rnkRECORDSCACHE)
storing the association of recids and authorids
Returns true if the database has been modified
"""
sql = 'SELECT authorid FROM rnkRECORDSCACHE WHERE id_bibrec = %s'
rows = run_sql(sql, (recid, ))
old_authors = set(r[0] for r in rows)
if authors != old_authors:
deleted_authors = old_authors.difference(authors)
added_authors = authors.difference(old_authors)
for authorid in deleted_authors:
run_sql("""DELETE FROM rnkRECORDSCACHE
WHERE id_bibrec = %s""", (recid, ))
for authorid in added_authors:
run_sql("""INSERT IGNORE INTO rnkRECORDSCACHE (id_bibrec, authorid)
VALUES (%s,%s)""", (recid, authorid))
return deleted_authors, added_authors
return set(), set()
def get_author_coauthors_list(personids, config):
"""
Get all the authors that have written a paper with any of the given authors
"""
personids = list(personids)
if not personids:
return ()
cluster_threshold = config['friends_threshold']
in_sql = ','.join('%s' for r in personids)
coauthors = (r[0] for r in run_sql("""
SELECT a.authorid FROM rnkRECORDSCACHE as a
JOIN rnkRECORDSCACHE as b ON a.id_bibrec = b.id_bibrec
WHERE b.authorid IN (%s)
GROUP BY a.authorid
HAVING count(a.authorid) >= %s""" % (in_sql, cluster_threshold),
personids))
return chain(personids, coauthors)
def store_record_coauthors(recid, authors, deleted_authors,
added_authors, config):
"""Fill table used by get_record_coauthors()"""
if deleted_authors:
to_process = authors
else:
to_process = added_authors
for personid in get_author_coauthors_list(deleted_authors, config):
run_sql('DELETE FROM rnkEXTENDEDAUTHORS WHERE'
' id = %s AND authorid = %s', (recid, personid))
for personid in get_author_coauthors_list(to_process, config):
run_sql('INSERT IGNORE INTO rnkEXTENDEDAUTHORS (id, authorid) '
'VALUES (%s,%s)', (recid, personid))
def get_record_coauthors(recid):
"""
Get all the authors that have written a paper with any of the authors of
given bibrec
"""
sql = 'SELECT authorid FROM rnkEXTENDEDAUTHORS WHERE id = %s'
return (r[0] for r in run_sql(sql, (recid, )))
SELFCITES_CONFIG = load_config_file('selfcites')
ALL_ALGORITHMS = {
'friends': compute_friends_self_citations,
'simple': compute_simple_self_citations,
}
diff --git a/invenio/legacy/bibrank/selfcites_task.py b/invenio/legacy/bibrank/selfcites_task.py
index 349fe0b83..2ca76ff55 100644
--- a/invenio/legacy/bibrank/selfcites_task.py
+++ b/invenio/legacy/bibrank/selfcites_task.py
@@ -1,324 +1,334 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
"""
Self citations task
Stores self-citations in a table for quick access
Examples:
(run a daemon job)
bibrank -w selfcites
(run on a set of records)
selfcites -i 1-20
(run on a collection)
selfcites -c "Reports"
This task handles the self-citations computation
It is run on modified records so that it can update the tables used for
displaying info in the citesummary format
"""
import sys
import ConfigParser
import time
from datetime import datetime
+from invenio.utils.redis import get_redis
+from invenio.legacy.dbquery import serialize_via_marshal
from intbitset import intbitset
from invenio.config import CFG_BIBRANK_SELFCITES_USE_BIBAUTHORID, \
CFG_ETCDIR
from invenio.legacy.bibsched.bibtask import \
task_get_option, write_message, \
task_sleep_now_if_required, \
task_update_progress
from invenio.legacy.dbquery import run_sql
from invenio.legacy.bibrank.selfcites_indexer import update_self_cites_tables, \
compute_friends_self_citations, \
compute_simple_self_citations, \
get_authors_tags
from invenio.legacy.bibrank.citation_searcher import get_refers_to
from invenio.legacy.bibauthorid.daemon import get_user_logs as bibauthorid_user_log
from invenio.legacy.bibrank.citation_indexer import get_bibrankmethod_lastupdate
from invenio.legacy.bibrank.tag_based_indexer import intoDB, fromDB
from invenio.modules.ranker.registry import configuration
def compute_and_store_self_citations(recid, tags, citations_fun, selfcites_dic,
verbose=False):
"""Compute and store self-cites in a table
Args:
- recid
- tags: used when bibauthorid is desactivated see get_author_tags()
in bibrank_selfcites_indexer
"""
assert recid
if verbose:
write_message("* processing %s" % recid)
references = get_refers_to(recid)
recids_to_check = set([recid]) | set(references)
placeholders = ','.join('%s' for r in recids_to_check)
rec_row = run_sql("SELECT MAX(`modification_date`) FROM `bibrec`"
" WHERE `id` IN (%s)" % placeholders, recids_to_check)
try:
rec_timestamp = rec_row[0]
except IndexError:
write_message("record not found")
return
cached_citations_row = run_sql("SELECT `count` FROM `rnkSELFCITES`"
" WHERE `last_updated` >= %s"
" AND `id_bibrec` = %s", (rec_timestamp[0], recid))
if cached_citations_row and cached_citations_row[0][0]:
if verbose:
write_message("%s found (cached)" % cached_citations_row[0])
else:
cites = citations_fun(recid, tags)
selfcites_dic[recid] = len(cites)
replace_cites(recid, cites)
sql = """REPLACE INTO rnkSELFCITES (`id_bibrec`, `count`, `references`,
`last_updated`) VALUES (%s, %s, %s, NOW())"""
references_string = ','.join(str(r) for r in references)
run_sql(sql, (recid, len(cites), references_string))
if verbose:
write_message("%s found" % len(cites))
def replace_cites(recid, new_cites):
"""Update database with new citations set
Given a set of self citations:
* stores the new ones in the database
* removes the old ones from the database
"""
old_cites = set(row[0] for row in run_sql("""SELECT citer
FROM rnkSELFCITEDICT
WHERE citee = %s""", [recid]))
cites_to_add = new_cites - old_cites
cites_to_delete = old_cites - new_cites
for cit in cites_to_add:
write_message('adding cite %s %s' % (recid, cit), verbose=1)
now = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
run_sql("""INSERT INTO rnkSELFCITEDICT (citee, citer, last_updated)
VALUES (%s, %s, %s)""", (recid, cit, now))
for cit in cites_to_delete:
write_message('deleting cite %s %s' % (recid, cit), verbose=1)
run_sql("""DELETE FROM rnkSELFCITEDICT
WHERE citee = %s and citer = %s""", (recid, cit))
def rebuild_tables(rank_method_code, config):
"""Rebuild the tables from scratch
Called by bibrank -w selfcites -R
"""
task_update_progress('emptying tables')
empty_self_cites_tables()
task_update_progress('filling tables')
fill_self_cites_tables(rank_method_code, config)
return True
def fetch_bibauthorid_last_update():
"""Fetch last runtime of bibauthorid"""
bibauthorid_log = bibauthorid_user_log(userinfo='daemon',
action='PID_UPDATE',
only_most_recent=True)
try:
bibauthorid_end_date = bibauthorid_log[0][2]
except IndexError:
bibauthorid_end_date = datetime(year=1900, month=1, day=1)
return bibauthorid_end_date.strftime("%Y-%m-%d %H:%M:%S")
def fetch_index_update():
"""Fetch last runtime of given task"""
end_date = get_bibrankmethod_lastupdate('citation')
if CFG_BIBRANK_SELFCITES_USE_BIBAUTHORID:
bibauthorid_end_date = fetch_bibauthorid_last_update()
end_date = min(end_date, bibauthorid_end_date)
return end_date
def fetch_records(start_date, end_date):
"""Filter records not indexed out of recids
We need to run after bibauthorid // bibrank citation indexer
"""
sql = """SELECT `id` FROM `bibrec`
WHERE `modification_date` <= %s
AND `modification_date` > %s"""
records = run_sql(sql, (end_date, start_date))
return intbitset(records)
def fetch_concerned_records(name, ids_param):
"""Fetch records that have been updated since the last run of the daemon"""
if ids_param:
recids = intbitset()
for first, last in ids_param:
recids += range(first, last+1)
end_date = None
else:
start_date = get_bibrankmethod_lastupdate(name)
end_date = fetch_index_update()
recids = fetch_records(start_date, end_date)
return recids, end_date
def store_last_updated(name, date):
"""Updates method last run date"""
run_sql("UPDATE rnkMETHOD SET last_updated=%s WHERE name=%s", (date, name))
def read_configuration(rank_method_code):
"""Load the config file from disk and parse it."""
filename = configuration.get(rank_method_code + '.cfg', '')
config = ConfigParser.ConfigParser()
try:
config.readfp(open(filename))
except StandardError:
write_message("Cannot find configuration file: %s" % filename, sys.stderr)
raise
return config
def process_updates(rank_method_code):
"""
This is what gets executed first when the task is started.
It handles the --rebuild option. If that option is not specified
we fall back to the process_one()
"""
write_message("Running rank method: %s" % rank_method_code, verbose=0)
selfcites_config = read_configuration(rank_method_code)
config = {
'algorithm': selfcites_config.get(rank_method_code, "algorithm"),
'friends_threshold': selfcites_config.get(rank_method_code, "friends_threshold")
}
quick = task_get_option("quick") != "no"
if not quick:
return rebuild_tables(rank_method_code, config)
tags = get_authors_tags()
recids, end_date = fetch_concerned_records(rank_method_code,
task_get_option("id"))
citations_fun = get_citations_fun(config['algorithm'])
- selfcites_dic = fromDB(rank_method_code)
+ weights = fromDB(rank_method_code)
write_message("recids %s" % str(recids))
total = len(recids)
for count, recid in enumerate(recids):
task_sleep_now_if_required(can_stop_too=True)
msg = "Extracting for %s (%d/%d)" % (recid, count + 1, total)
task_update_progress(msg)
write_message(msg)
- process_one(recid, tags, citations_fun, selfcites_dic)
+ process_one(recid, tags, citations_fun, weights)
- intoDB(selfcites_dic, end_date, rank_method_code)
+ intoDB(weights, end_date, rank_method_code)
+ store_weights_cache(weights)
write_message("Complete")
return True
def get_citations_fun(algorithm):
"""Returns the computation function given the algorithm name"""
if algorithm == 'friends':
citations_fun = compute_friends_self_citations
else:
citations_fun = compute_simple_self_citations
return citations_fun
def process_one(recid, tags, citations_fun, selfcites_dic):
"""Self-cites core func, executed on each recid"""
# First update this record then all its references
compute_and_store_self_citations(recid, tags, citations_fun, selfcites_dic)
references = get_refers_to(recid)
for recordid in references:
compute_and_store_self_citations(recordid,
tags,
citations_fun,
selfcites_dic)
def empty_self_cites_tables():
"""
This will empty all the self-cites tables
The purpose is to rebuild the tables from scratch in case there is problem
with them: inconsitencies, corruption,...
"""
run_sql('TRUNCATE rnkSELFCITES')
run_sql('TRUNCATE rnkEXTENDEDAUTHORS')
run_sql('TRUNCATE rnkRECORDSCACHE')
def fill_self_cites_tables(rank_method_code, config):
"""
This will fill the self-cites tables with data
The purpose of this function is to fill these tables on a website that
never ran the self-cites daemon
This is an optimization when running on empty tables, and we hope the
result is the same as the compute_and_store_self_citations.
"""
begin_date = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
algorithm = config['algorithm']
tags = get_authors_tags()
selfcites_dic = {}
all_ids = intbitset(run_sql('SELECT id FROM bibrec ORDER BY id'))
citations_fun = get_citations_fun(algorithm)
write_message('using %s' % citations_fun.__name__)
if algorithm == 'friends':
# We only needs this table for the friends algorithm or assimilated
# Fill intermediary tables
for index, recid in enumerate(all_ids):
if index % 1000 == 0:
msg = 'intermediate %d/%d' % (index, len(all_ids))
task_update_progress(msg)
write_message(msg)
task_sleep_now_if_required()
update_self_cites_tables(recid, config, tags)
# Fill self-cites table
for index, recid in enumerate(all_ids):
if index % 1000 == 0:
msg = 'final %d/%d' % (index, len(all_ids))
task_update_progress(msg)
write_message(msg)
task_sleep_now_if_required()
compute_and_store_self_citations(recid,
tags,
citations_fun,
selfcites_dic)
intoDB(selfcites_dic, begin_date, rank_method_code)
+ store_weights_cache(selfcites_dic)
+
+
+def store_weights_cache(weights):
+ """Store into key/value store"""
+ redis = get_redis()
+ redis.set('selfcites_weights', serialize_via_marshal(weights))
diff --git a/invenio/legacy/bibrecord/__init__.py b/invenio/legacy/bibrecord/__init__.py
index 8c4f97d76..fd8edc16c 100644
--- a/invenio/legacy/bibrecord/__init__.py
+++ b/invenio/legacy/bibrecord/__init__.py
@@ -1,2266 +1,2338 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013,
## 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
BibRecord - XML MARC processing library for Invenio.
BibRecord library offer a whole set of API function to handle record metadata.
Managing metadata with BibRecord
In order to work with bibrecord library you first need to have available a
record representation.
If you have a MARCXML representation of the record to be handled, you can use
the create_record function to obtain a bibrecord internal representation::
from invenio.bibrecord import create_record
record = create_record(marcxml)[0]
If you want to handle a record stored in the system and you know the record ID,
then you can easily exploit Invenio search_engine API to obtain the
corresponding marcxml::
from invenio.bibrecord import create_record
from invenio.legacy.search_engine import print_record
marcxml = print_record(rec_id, 'xm')
record = create_record(marcxml)[0]
Having an internal representation of a record you can manipulate it by means of
bibrecord functions like
:func:`~invenio.legacy.bibrecord.record_get_field_instances`,
:func:`~invenio.legacy.bibrecord.record_has_field`,
:func:`~invenio.legacy.bibrecord.record_add_field`,
:func:`~invenio.legacy.bibrecord.record_delete_field`,
:func:`~invenio.legacy.bibrecord.record_delete_subfield`,
:func:`~invenio.legacy.bibrecord.record_add_or_modify_subfield`,
:func:`~invenio.legacy.bibrecord.record_add_subfield`,
:func:`~invenio.legacy.bibrecord.record_does_field_exist`,
:func:`~invenio.legacy.bibrecord.record_filter_fields`,
:func:`~invenio.legacy.bibrecord.record_replace_in_subfields`,
:func:`~invenio.legacy.bibrecord.record_get_field_value`,
:func:`~invenio.legacy.bibrecord.record_get_field_values`...
At the end, if you want the MARCXML representation of the record you can use
record_xml_output::
from invenio.bibrecord import create_record
from invenio.legacy.search_engine import print_record
marcxml = print_record(rec_id, 'xm')
record = create_record(marcxml)[0]
# ... manipulation ...
new_marcxml = record_xml_output(record)
In order to write back such a record into the system you should use the
BibUpload utility.
Please referer to bibrecord.py for a complete and up-to-date description of the
API, see :func:`~invenio.legacy.bibrecordcreate_record`,
:func:`~invenio.legacy.bibrecordrecord_get_field_instances` and friends in the
source code of this file in the section entitled INTERFACE.
As always, a good entry point to the bibrecord library and its record structure
manipulating functions is to read the unit test cases that are located in
bibrecord_tests.py and bibupload_regression_tests.py.
"""
### IMPORT INTERESTING MODULES AND XML PARSERS
import re
import string
import sys
from six import StringIO
if sys.hexversion < 0x2040000:
# pylint: disable=W0622
from sets import Set as set
# pylint: enable=W0622
from invenio.base.globals import cfg
from invenio.legacy.bibrecord.bibrecord_config import CFG_MARC21_DTD, \
CFG_BIBRECORD_WARNING_MSGS, CFG_BIBRECORD_DEFAULT_VERBOSE_LEVEL, \
CFG_BIBRECORD_DEFAULT_CORRECT, CFG_BIBRECORD_PARSERS_AVAILABLE, \
InvenioBibRecordParserError, InvenioBibRecordFieldError
from invenio.utils.text import encode_for_xml
from invenio.legacy.dbquery import run_sql
from intbitset import intbitset
# Some values used for the RXP parsing.
TAG, ATTRS, CHILDREN = 0, 1, 2
# Find out about the best usable parser:
AVAILABLE_PARSERS = []
# Do we remove singletons (empty tags)?
# NOTE: this is currently set to True as there are some external workflow
# exploiting singletons, e.g. bibupload -c used to delete fields, and
# bibdocfile --fix-marc called on a record where the latest document
# has been deleted.
CFG_BIBRECORD_KEEP_SINGLETONS = True
try:
import pyRXP
if 'pyrxp' in CFG_BIBRECORD_PARSERS_AVAILABLE:
AVAILABLE_PARSERS.append('pyrxp')
except ImportError:
pass
try:
from lxml import etree
if 'lxml' in CFG_BIBRECORD_PARSERS_AVAILABLE:
AVAILABLE_PARSERS.append('lxml')
except ImportError:
pass
try:
import Ft.Xml.Domlette
if '4suite' in CFG_BIBRECORD_PARSERS_AVAILABLE:
AVAILABLE_PARSERS.append('4suite')
except ImportError:
pass
except Exception as err:
from warnings import warn
warn("Error when importing 4suite: %s" % err)
pass
try:
import xml.dom.minidom
import xml.parsers.expat
if 'minidom' in CFG_BIBRECORD_PARSERS_AVAILABLE:
AVAILABLE_PARSERS.append('minidom')
except ImportError:
pass
### INTERFACE / VISIBLE FUNCTIONS
def create_field(subfields=None, ind1=' ', ind2=' ', controlfield_value='',
global_position=-1):
"""
Return a field created with the provided elements.
Global position is set arbitrary to -1.
"""
if subfields is None:
subfields = []
ind1, ind2 = _wash_indicators(ind1, ind2)
field = (subfields, ind1, ind2, controlfield_value, global_position)
_check_field_validity(field)
return field
def create_records(marcxml, verbose=CFG_BIBRECORD_DEFAULT_VERBOSE_LEVEL,
correct=CFG_BIBRECORD_DEFAULT_CORRECT, parser='',
keep_singletons=CFG_BIBRECORD_KEEP_SINGLETONS):
"""
Create a list of records from the marcxml description.
:returns: a list of objects initiated by the function create_record().
Please see that function's docstring.
"""
# Use the DOTALL flag to include newlines.
regex = re.compile('<record.*?>.*?</record>', re.DOTALL)
record_xmls = regex.findall(marcxml)
return [create_record(record_xml, verbose=verbose, correct=correct,
parser=parser, keep_singletons=keep_singletons)
for record_xml in record_xmls]
def create_record(marcxml, verbose=CFG_BIBRECORD_DEFAULT_VERBOSE_LEVEL,
correct=CFG_BIBRECORD_DEFAULT_CORRECT, parser='',
sort_fields_by_indicators=False,
keep_singletons=CFG_BIBRECORD_KEEP_SINGLETONS):
"""Create a record object from the marcxml description.
Uses the best parser available in CFG_BIBRECORD_PARSERS_AVAILABLE or
the parser specified.
The returned object is a tuple (record, status_code, list_of_errors),
where status_code is 0 when there are errors, 1 when no errors.
The return record structure is as follows::
Record := {tag : [Field]}
Field := (Subfields, ind1, ind2, value)
Subfields := [(code, value)]
.. code-block:: none
.--------.
| record |
'---+----'
|
.------------------------+------------------------------------.
|record['001'] |record['909'] |record['520'] |
| | | |
[list of fields] [list of fields] [list of fields] ...
| | |
| .--------+--+-----------. |
| | | | |
|[0] |[0] |[1] ... |[0]
.----+------. .-----+-----. .--+--------. .---+-------.
| Field 001 | | Field 909 | | Field 909 | | Field 520 |
'-----------' '-----+-----' '--+--------' '---+-------'
| | | |
... | ... ...
|
.----------+-+--------+------------.
| | | |
|[0] |[1] |[2] |
[list of subfields] 'C' '4' ...
|
.----+---------------+------------------------+
| | |
('a', 'value') | ('a', 'value for another a')
('b', 'value for subfield b')
:param marcxml: an XML string representation of the record to create
:param verbose: the level of verbosity: 0 (silent), 1-2 (warnings),
3(strict:stop when errors)
:param correct: 1 to enable correction of marcxml syntax. Else 0.
:return: a tuple (record, status_code, list_of_errors), where status
code is 0 where there are errors, 1 when no errors
"""
# Select the appropriate parser.
parser = _select_parser(parser)
try:
if parser == 'pyrxp':
rec = _create_record_rxp(marcxml, verbose, correct,
keep_singletons=keep_singletons)
elif parser == 'lxml':
rec = _create_record_lxml(marcxml, verbose, correct,
keep_singletons=keep_singletons)
elif parser == '4suite':
rec = _create_record_4suite(marcxml,
keep_singletons=keep_singletons)
elif parser == 'minidom':
rec = _create_record_minidom(marcxml,
keep_singletons=keep_singletons)
except InvenioBibRecordParserError as ex1:
return (None, 0, str(ex1))
# _create_record = {
# 'pyrxp': _create_record_rxp,
# 'lxml': _create_record_lxml,
# '4suite': _create_record_4suite,
# 'minidom': _create_record_minidom,
# }
# try:
# rec = _create_record[parser](marcxml, verbose)
# except InvenioBibRecordParserError as ex1:
# return (None, 0, str(ex1))
if sort_fields_by_indicators:
_record_sort_by_indicators(rec)
errs = []
if correct:
# Correct the structure of the record.
errs = _correct_record(rec)
return (rec, int(not errs), errs)
def filter_field_instances(field_instances, filter_subcode, filter_value,
filter_mode='e'):
"""Filter the given field.
Filters given field and returns only that field instances that contain
filter_subcode with given filter_value. As an input for search function
accepts output from record_get_field_instances function. Function can be
run in three modes:
- 'e' - looking for exact match in subfield value
- 's' - looking for substring in subfield value
- 'r' - looking for regular expression in subfield value
Example:
record_filter_field(record_get_field_instances(rec, '999', '%', '%'),
'y', '2001')
In this case filter_subcode is 'y' and filter_value is '2001'.
:param field_instances: output from record_get_field_instances
:param filter_subcode: name of the subfield
:type filter_subcode: string
:param filter_value: value of the subfield
:type filter_value: string
:param filter_mode: 'e','s' or 'r'
"""
matched = []
if filter_mode == 'e':
to_match = (filter_subcode, filter_value)
for instance in field_instances:
if to_match in instance[0]:
matched.append(instance)
elif filter_mode == 's':
for instance in field_instances:
for subfield in instance[0]:
if subfield[0] == filter_subcode and \
subfield[1].find(filter_value) > -1:
matched.append(instance)
break
elif filter_mode == 'r':
reg_exp = re.compile(filter_value)
for instance in field_instances:
for subfield in instance[0]:
if subfield[0] == filter_subcode and \
reg_exp.match(subfield[1]) is not None:
matched.append(instance)
break
return matched
+def record_drop_duplicate_fields(record):
+ """
+ Return a record where all the duplicate fields have been removed.
+ Fields are considered identical considering also the order of their
+ subfields.
+ """
+ out = {}
+ position = 0
+ tags = sorted(record.keys())
+ for tag in tags:
+ fields = record[tag]
+ out[tag] = []
+ current_fields = set()
+ for full_field in fields:
+ field = (tuple(full_field[0]),) + full_field[1:4]
+ if field not in current_fields:
+ current_fields.add(field)
+ position += 1
+ out[tag].append(full_field[:4] + (position,))
+ return out
+
+
def records_identical(rec1, rec2, skip_005=True, ignore_field_order=False,
ignore_subfield_order=False,
ignore_duplicate_subfields=False,
ignore_duplicate_controlfields=False):
"""
Return True if rec1 is identical to rec2.
It does so regardless of a difference in the 005 tag (i.e. the timestamp).
"""
rec1_keys = set(rec1.keys())
rec2_keys = set(rec2.keys())
if skip_005:
rec1_keys.discard("005")
rec2_keys.discard("005")
if rec1_keys != rec2_keys:
return False
for key in rec1_keys:
if ignore_duplicate_controlfields and key.startswith('00'):
if set(field[3] for field in rec1[key]) != \
set(field[3] for field in rec2[key]):
return False
continue
rec1_fields = rec1[key]
rec2_fields = rec2[key]
if len(rec1_fields) != len(rec2_fields):
# They already differs in length...
return False
if ignore_field_order:
## We sort the fields, first by indicators and then by anything else
rec1_fields = sorted(rec1_fields, key=lambda elem: (elem[1], elem[2], elem[3], elem[0]))
rec2_fields = sorted(rec2_fields, key=lambda elem: (elem[1], elem[2], elem[3], elem[0]))
else:
## We sort the fields, first by indicators, then by global position and then by anything else
rec1_fields = sorted(rec1_fields, key=lambda elem: (elem[1], elem[2], elem[4], elem[3], elem[0]))
rec2_fields = sorted(rec2_fields, key=lambda elem: (elem[1], elem[2], elem[4], elem[3], elem[0]))
for field1, field2 in zip(rec1_fields, rec2_fields):
if ignore_duplicate_subfields:
if field1[1:4] != field2[1:4] or \
set(field1[0]) != set(field2[0]):
return False
elif ignore_subfield_order:
if field1[1:4] != field2[1:4] or \
sorted(field1[0]) != sorted(field2[0]):
return False
elif field1[:4] != field2[:4]:
return False
return True
def record_get_field_instances(rec, tag="", ind1=" ", ind2=" "):
"""
Return the list of field instances for the specified tag and indications.
Return empty list if not found.
If tag is empty string, returns all fields
Parameters (tag, ind1, ind2) can contain wildcard %.
:param rec: a record structure as returned by create_record()
:param tag: a 3 characters long string
:param ind1: a 1 character long string
:param ind2: a 1 character long string
:param code: a 1 character long string
:return: a list of field tuples (Subfields, ind1, ind2, value,
field_position_global) where subfields is list of (code, value)
"""
if not rec:
return []
if not tag:
return rec.items()
else:
out = []
ind1, ind2 = _wash_indicators(ind1, ind2)
if '%' in tag:
# Wildcard in tag. Check all possible
for field_tag in rec:
if _tag_matches_pattern(field_tag, tag):
for possible_field_instance in rec[field_tag]:
if (ind1 in ('%', possible_field_instance[1]) and
ind2 in ('%', possible_field_instance[2])):
out.append(possible_field_instance)
else:
# Completely defined tag. Use dict
for possible_field_instance in rec.get(tag, []):
if (ind1 in ('%', possible_field_instance[1]) and
ind2 in ('%', possible_field_instance[2])):
out.append(possible_field_instance)
return out
def record_add_field(rec, tag, ind1=' ', ind2=' ', controlfield_value='',
subfields=None, field_position_global=None,
field_position_local=None):
"""
Add a new field into the record.
If field_position_global or field_position_local is specified then
this method will insert the new field at the desired position.
Otherwise a global field position will be computed in order to
insert the field at the best position (first we try to keep the
order of the tags and then we insert the field at the end of the
fields with the same tag).
If both field_position_global and field_position_local are present,
then field_position_local takes precedence.
:param rec: the record data structure
:param tag: the tag of the field to be added
:param ind1: the first indicator
:param ind2: the second indicator
:param controlfield_value: the value of the controlfield
:param subfields: the subfields (a list of tuples (code, value))
:param field_position_global: the global field position (record wise)
:param field_position_local: the local field position (tag wise)
:return: the global field position of the newly inserted field or -1 if the
operation failed
"""
- error = validate_record_field_positions_global(rec)
+ error = _validate_record_field_positions_global(rec)
if error:
# FIXME one should write a message here
pass
# Clean the parameters.
if subfields is None:
subfields = []
ind1, ind2 = _wash_indicators(ind1, ind2)
if controlfield_value and (ind1 != ' ' or ind2 != ' ' or subfields):
return -1
# Detect field number to be used for insertion:
# Dictionaries for uniqueness.
tag_field_positions_global = {}.fromkeys([field[4]
for field in rec.get(tag, [])])
all_field_positions_global = {}.fromkeys([field[4]
for fields in rec.values()
for field in fields])
if field_position_global is None and field_position_local is None:
# Let's determine the global field position of the new field.
if tag in rec:
try:
field_position_global = max([field[4] for field in rec[tag]]) \
+ 1
except IndexError:
if tag_field_positions_global:
field_position_global = max(tag_field_positions_global) + 1
elif all_field_positions_global:
field_position_global = max(all_field_positions_global) + 1
else:
field_position_global = 1
else:
if tag in ('FMT', 'FFT', 'BDR', 'BDM'):
# Add the new tag to the end of the record.
if tag_field_positions_global:
field_position_global = max(tag_field_positions_global) + 1
elif all_field_positions_global:
field_position_global = max(all_field_positions_global) + 1
else:
field_position_global = 1
else:
# Insert the tag in an ordered way by selecting the
# right global field position.
immediate_lower_tag = '000'
for rec_tag in rec:
if (tag not in ('FMT', 'FFT', 'BDR', 'BDM') and
immediate_lower_tag < rec_tag < tag):
immediate_lower_tag = rec_tag
if immediate_lower_tag == '000':
field_position_global = 1
else:
field_position_global = rec[immediate_lower_tag][-1][4] + 1
field_position_local = len(rec.get(tag, []))
_shift_field_positions_global(rec, field_position_global, 1)
elif field_position_local is not None:
if tag in rec:
if field_position_local >= len(rec[tag]):
field_position_global = rec[tag][-1][4] + 1
else:
field_position_global = rec[tag][field_position_local][4]
_shift_field_positions_global(rec, field_position_global, 1)
else:
if all_field_positions_global:
field_position_global = max(all_field_positions_global) + 1
else:
# Empty record.
field_position_global = 1
elif field_position_global is not None:
# If the user chose an existing global field position, shift all the
# global field positions greater than the input global field position.
if tag not in rec:
if all_field_positions_global:
field_position_global = max(all_field_positions_global) + 1
else:
field_position_global = 1
field_position_local = 0
elif field_position_global < min(tag_field_positions_global):
field_position_global = min(tag_field_positions_global)
_shift_field_positions_global(rec, min(tag_field_positions_global),
1)
field_position_local = 0
elif field_position_global > max(tag_field_positions_global):
field_position_global = max(tag_field_positions_global) + 1
_shift_field_positions_global(rec,
max(tag_field_positions_global) + 1,
1)
field_position_local = len(rec.get(tag, []))
else:
if field_position_global in tag_field_positions_global:
_shift_field_positions_global(rec, field_position_global, 1)
field_position_local = 0
for position, field in enumerate(rec[tag]):
if field[4] == field_position_global + 1:
field_position_local = position
# Create the new field.
newfield = (subfields, ind1, ind2, str(controlfield_value),
field_position_global)
rec.setdefault(tag, []).insert(field_position_local, newfield)
# Return new field number:
return field_position_global
def record_has_field(rec, tag):
"""
Check if the tag exists in the record.
:param rec: the record data structure
:param the: field
:return: a boolean
"""
return tag in rec
def record_delete_field(rec, tag, ind1=' ', ind2=' ',
field_position_global=None, field_position_local=None):
"""
Delete the field with the given position.
If global field position is specified, deletes the field with the
corresponding global field position.
If field_position_local is specified, deletes the field with the
corresponding local field position and tag.
Else deletes all the fields matching tag and optionally ind1 and
ind2.
If both field_position_global and field_position_local are present,
then field_position_local takes precedence.
:param rec: the record data structure
:param tag: the tag of the field to be deleted
:param ind1: the first indicator of the field to be deleted
:param ind2: the second indicator of the field to be deleted
:param field_position_global: the global field position (record wise)
:param field_position_local: the local field position (tag wise)
:return: the list of deleted fields
"""
- error = validate_record_field_positions_global(rec)
+ error = _validate_record_field_positions_global(rec)
if error:
# FIXME one should write a message here.
pass
if tag not in rec:
return False
ind1, ind2 = _wash_indicators(ind1, ind2)
deleted = []
newfields = []
if field_position_global is None and field_position_local is None:
# Remove all fields with tag 'tag'.
for field in rec[tag]:
if field[1] != ind1 or field[2] != ind2:
newfields.append(field)
else:
deleted.append(field)
rec[tag] = newfields
elif field_position_global is not None:
# Remove the field with 'field_position_global'.
for field in rec[tag]:
if (field[1] != ind1 and field[2] != ind2 or
field[4] != field_position_global):
newfields.append(field)
else:
deleted.append(field)
rec[tag] = newfields
elif field_position_local is not None:
# Remove the field with 'field_position_local'.
try:
del rec[tag][field_position_local]
except IndexError:
return []
if not rec[tag]:
# Tag is now empty, remove it.
del rec[tag]
return deleted
def record_delete_fields(rec, tag, field_positions_local=None):
"""
Delete all/some fields defined with MARC tag 'tag' from record 'rec'.
:param rec: a record structure.
:type rec: tuple
:param tag: three letter field.
:type tag: string
:param field_position_local: if set, it is the list of local positions
within all the fields with the specified tag, that should be deleted.
If not set all the fields with the specified tag will be deleted.
:type field_position_local: sequence
:return: the list of deleted fields.
:rtype: list
:note: the record is modified in place.
"""
if tag not in rec:
return []
new_fields, deleted_fields = [], []
for position, field in enumerate(rec.get(tag, [])):
if field_positions_local is None or position in field_positions_local:
deleted_fields.append(field)
else:
new_fields.append(field)
if new_fields:
rec[tag] = new_fields
else:
del rec[tag]
return deleted_fields
def record_add_fields(rec, tag, fields, field_position_local=None,
field_position_global=None):
"""
Add the fields into the record at the required position.
The position is specified by the tag and the field_position_local in the
list of fields.
:param rec: a record structure
:param tag: the tag of the fields to be moved
:param field_position_local: the field_position_local to which the field
will be inserted. If not specified, appends
the fields to the tag.
:param a: list of fields to be added
:return: -1 if the operation failed, or the field_position_local if it was
successful
"""
if field_position_local is None and field_position_global is None:
for field in fields:
record_add_field(
rec, tag, ind1=field[1],
ind2=field[2], subfields=field[0],
controlfield_value=field[3])
else:
fields.reverse()
for field in fields:
record_add_field(
rec, tag, ind1=field[1], ind2=field[2],
subfields=field[0], controlfield_value=field[3],
field_position_local=field_position_local,
field_position_global=field_position_global)
return field_position_local
def record_move_fields(rec, tag, field_positions_local,
field_position_local=None):
"""
Move some fields to the position specified by 'field_position_local'.
:param rec: a record structure as returned by create_record()
:param tag: the tag of the fields to be moved
:param field_positions_local: the positions of the fields to move
:param field_position_local: insert the field before that
field_position_local. If unspecified, appends
the fields :return: the field_position_local
is the operation was successful
"""
fields = record_delete_fields(
rec, tag,
field_positions_local=field_positions_local)
return record_add_fields(
rec, tag, fields,
field_position_local=field_position_local)
def record_delete_subfield(rec, tag, subfield_code, ind1=' ', ind2=' '):
"""Delete all subfields with subfield_code in the record."""
ind1, ind2 = _wash_indicators(ind1, ind2)
for field in rec.get(tag, []):
if field[1] == ind1 and field[2] == ind2:
field[0][:] = [subfield for subfield in field[0]
if subfield_code != subfield[0]]
def record_get_field(rec, tag, field_position_global=None,
field_position_local=None):
"""
Return the the matching field.
One has to enter either a global field position or a local field position.
:return: a list of subfield tuples (subfield code, value).
:rtype: list
"""
if field_position_global is None and field_position_local is None:
raise InvenioBibRecordFieldError(
"A field position is required to "
"complete this operation.")
elif field_position_global is not None and \
field_position_local is not None:
raise InvenioBibRecordFieldError(
"Only one field position is required "
"to complete this operation.")
elif field_position_global:
if tag not in rec:
raise InvenioBibRecordFieldError("No tag '%s' in record." % tag)
for field in rec[tag]:
if field[4] == field_position_global:
return field
raise InvenioBibRecordFieldError(
"No field has the tag '%s' and the "
"global field position '%d'." % (tag, field_position_global))
else:
try:
return rec[tag][field_position_local]
except KeyError:
raise InvenioBibRecordFieldError("No tag '%s' in record." % tag)
except IndexError:
raise InvenioBibRecordFieldError(
"No field has the tag '%s' and "
"the local field position '%d'." % (tag, field_position_local))
def record_replace_field(rec, tag, new_field, field_position_global=None,
field_position_local=None):
"""Replace a field with a new field."""
if field_position_global is None and field_position_local is None:
raise InvenioBibRecordFieldError(
"A field position is required to "
"complete this operation.")
elif field_position_global is not None and \
field_position_local is not None:
raise InvenioBibRecordFieldError(
"Only one field position is required "
"to complete this operation.")
elif field_position_global:
if tag not in rec:
raise InvenioBibRecordFieldError("No tag '%s' in record." % tag)
replaced = False
for position, field in enumerate(rec[tag]):
if field[4] == field_position_global:
rec[tag][position] = new_field
replaced = True
if not replaced:
raise InvenioBibRecordFieldError(
"No field has the tag '%s' and "
"the global field position '%d'." %
(tag, field_position_global))
else:
try:
rec[tag][field_position_local] = new_field
except KeyError:
raise InvenioBibRecordFieldError("No tag '%s' in record." % tag)
except IndexError:
raise InvenioBibRecordFieldError(
"No field has the tag '%s' and "
"the local field position '%d'." % (tag, field_position_local))
def record_get_subfields(rec, tag, field_position_global=None,
field_position_local=None):
"""
Return the subfield of the matching field.
One has to enter either a global field position or a local field position.
:return: a list of subfield tuples (subfield code, value).
:rtype: list
"""
field = record_get_field(
rec, tag,
field_position_global=field_position_global,
field_position_local=field_position_local)
return field[0]
def record_delete_subfield_from(rec, tag, subfield_position,
field_position_global=None,
field_position_local=None):
"""
Delete subfield from position specified.
Specify the subfield by tag, field number and subfield position.
"""
subfields = record_get_subfields(
rec, tag,
field_position_global=field_position_global,
field_position_local=field_position_local)
try:
del subfields[subfield_position]
except IndexError:
from .scripts.xmlmarc2textmarc import create_marc_record
recordMarc = create_marc_record(rec, 0,
{"text-marc": 1, "aleph-marc": 0})
raise InvenioBibRecordFieldError(
"The record : %(recordCode)s does not contain the subfield "
"'%(subfieldIndex)s' inside the field (local: "
"'%(fieldIndexLocal)s, global: '%(fieldIndexGlobal)s' ) of tag "
"'%(tag)s'." %
{"subfieldIndex": subfield_position,
"fieldIndexLocal": str(field_position_local),
"fieldIndexGlobal": str(field_position_global),
"tag": tag,
"recordCode": recordMarc})
if not subfields:
if field_position_global is not None:
for position, field in enumerate(rec[tag]):
if field[4] == field_position_global:
del rec[tag][position]
else:
del rec[tag][field_position_local]
if not rec[tag]:
del rec[tag]
def record_add_subfield_into(rec, tag, subfield_code, value,
subfield_position=None,
field_position_global=None,
field_position_local=None):
"""Add subfield into specified position.
Specify the subfield by tag, field number and optionally by subfield
position.
"""
subfields = record_get_subfields(
rec, tag,
field_position_global=field_position_global,
field_position_local=field_position_local)
if subfield_position is None:
subfields.append((subfield_code, value))
else:
subfields.insert(subfield_position, (subfield_code, value))
def record_modify_controlfield(rec, tag, controlfield_value,
field_position_global=None,
field_position_local=None):
"""Modify controlfield at position specified by tag and field number."""
field = record_get_field(
rec, tag,
field_position_global=field_position_global,
field_position_local=field_position_local)
new_field = (field[0], field[1], field[2], controlfield_value, field[4])
record_replace_field(
rec, tag, new_field,
field_position_global=field_position_global,
field_position_local=field_position_local)
def record_modify_subfield(rec, tag, subfield_code, value, subfield_position,
field_position_global=None,
field_position_local=None):
"""Modify subfield at specified position.
Specify the subfield by tag, field number and subfield position.
"""
subfields = record_get_subfields(
rec, tag,
field_position_global=field_position_global,
field_position_local=field_position_local)
try:
subfields[subfield_position] = (subfield_code, value)
except IndexError:
raise InvenioBibRecordFieldError(
"There is no subfield with position '%d'." % subfield_position)
def record_move_subfield(rec, tag, subfield_position, new_subfield_position,
field_position_global=None,
field_position_local=None):
"""Move subfield at specified position.
Sspecify the subfield by tag, field number and subfield position to new
subfield position.
"""
subfields = record_get_subfields(
rec,
tag,
field_position_global=field_position_global,
field_position_local=field_position_local)
try:
subfield = subfields.pop(subfield_position)
subfields.insert(new_subfield_position, subfield)
except IndexError:
raise InvenioBibRecordFieldError(
"There is no subfield with position '%d'." % subfield_position)
def record_get_field_value(rec, tag, ind1=" ", ind2=" ", code=""):
"""Return first (string) value that matches specified field of the record.
Returns empty string if not found.
Parameters (tag, ind1, ind2, code) can contain wildcard %.
Difference between wildcard % and empty '':
- Empty char specifies that we are not interested in a field which
has one of the indicator(s)/subfield specified.
- Wildcard specifies that we are interested in getting the value
of the field whatever the indicator(s)/subfield is.
For e.g. consider the following record in MARC::
100C5 $$a val1
555AB $$a val2
555AB val3
555 $$a val4
555A val5
.. doctest::
>>> record_get_field_value(record, '555', 'A', '', '')
"val5"
>>> record_get_field_value(record, '555', 'A', '%', '')
"val3"
>>> record_get_field_value(record, '555', 'A', '%', '%')
"val2"
>>> record_get_field_value(record, '555', 'A', 'B', '')
"val3"
>>> record_get_field_value(record, '555', '', 'B', 'a')
""
>>> record_get_field_value(record, '555', '', '', 'a')
"val4"
>>> record_get_field_value(record, '555', '', '', '')
""
>>> record_get_field_value(record, '%%%', '%', '%', '%')
"val1"
:param rec: a record structure as returned by create_record()
:param tag: a 3 characters long string
:param ind1: a 1 character long string
:param ind2: a 1 character long string
:param code: a 1 character long string
:return: string value (empty if nothing found)
"""
# Note: the code is quite redundant for speed reasons (avoid calling
# functions or doing tests inside loops)
ind1, ind2 = _wash_indicators(ind1, ind2)
if '%' in tag:
# Wild card in tag. Must find all corresponding fields
if code == '':
# Code not specified.
for field_tag, fields in rec.items():
if _tag_matches_pattern(field_tag, tag):
for field in fields:
if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):
# Return matching field value if not empty
if field[3]:
return field[3]
elif code == '%':
# Code is wildcard. Take first subfield of first matching field
for field_tag, fields in rec.items():
if _tag_matches_pattern(field_tag, tag):
for field in fields:
if (ind1 in ('%', field[1]) and
ind2 in ('%', field[2]) and field[0]):
return field[0][0][1]
else:
# Code is specified. Take corresponding one
for field_tag, fields in rec.items():
if _tag_matches_pattern(field_tag, tag):
for field in fields:
if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):
for subfield in field[0]:
if subfield[0] == code:
return subfield[1]
else:
# Tag is completely specified. Use tag as dict key
if tag in rec:
if code == '':
# Code not specified.
for field in rec[tag]:
if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):
# Return matching field value if not empty
# or return "" empty if not exist.
if field[3]:
return field[3]
elif code == '%':
# Code is wildcard. Take first subfield of first matching field
for field in rec[tag]:
if ind1 in ('%', field[1]) and ind2 in ('%', field[2]) and\
field[0]:
return field[0][0][1]
else:
# Code is specified. Take corresponding one
for field in rec[tag]:
if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):
for subfield in field[0]:
if subfield[0] == code:
return subfield[1]
# Nothing was found
return ""
def record_get_field_values(rec, tag, ind1=" ", ind2=" ", code="",
filter_subfield_code="",
filter_subfield_value="",
filter_subfield_mode="e"):
"""Return the list of values for the specified field of the record.
List can be filtered. Use filter_subfield_code
and filter_subfield_value to search
only in fields that have these values inside them as a subfield.
filter_subfield_mode can have 3 different values:
'e' for exact search
's' for substring search
'r' for regexp search
Returns empty list if nothing was found.
Parameters (tag, ind1, ind2, code) can contain wildcard %.
:param rec: a record structure as returned by create_record()
:param tag: a 3 characters long string
:param ind1: a 1 character long string
:param ind2: a 1 character long string
:param code: a 1 character long string
:return: a list of strings
"""
tmp = []
ind1, ind2 = _wash_indicators(ind1, ind2)
if filter_subfield_code and filter_subfield_mode == "r":
reg_exp = re.compile(filter_subfield_value)
tags = []
if '%' in tag:
# Wild card in tag. Must find all corresponding tags and fields
tags = [k for k in rec if _tag_matches_pattern(k, tag)]
elif rec and tag in rec:
tags = [tag]
if code == '':
# Code not specified. Consider field value (without subfields)
for tag in tags:
for field in rec[tag]:
if (ind1 in ('%', field[1]) and ind2 in ('%', field[2]) and
field[3]):
tmp.append(field[3])
elif code == '%':
# Code is wildcard. Consider all subfields
for tag in tags:
for field in rec[tag]:
if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):
if filter_subfield_code:
if filter_subfield_mode == "e":
subfield_to_match = (filter_subfield_code,
filter_subfield_value)
if subfield_to_match in field[0]:
for subfield in field[0]:
tmp.append(subfield[1])
elif filter_subfield_mode == "s":
if (dict(field[0]).get(filter_subfield_code, '')) \
.find(filter_subfield_value) > -1:
for subfield in field[0]:
tmp.append(subfield[1])
elif filter_subfield_mode == "r":
if reg_exp.match(dict(field[0])
.get(filter_subfield_code, '')):
for subfield in field[0]:
tmp.append(subfield[1])
else:
for subfield in field[0]:
tmp.append(subfield[1])
else:
# Code is specified. Consider all corresponding subfields
for tag in tags:
for field in rec[tag]:
if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):
if filter_subfield_code:
if filter_subfield_mode == "e":
subfield_to_match = (filter_subfield_code,
filter_subfield_value)
if subfield_to_match in field[0]:
for subfield in field[0]:
if subfield[0] == code:
tmp.append(subfield[1])
elif filter_subfield_mode == "s":
if (dict(field[0]).get(filter_subfield_code, '')) \
.find(filter_subfield_value) > -1:
for subfield in field[0]:
if subfield[0] == code:
tmp.append(subfield[1])
elif filter_subfield_mode == "r":
if reg_exp.match(dict(field[0])
.get(filter_subfield_code, '')):
for subfield in field[0]:
if subfield[0] == code:
tmp.append(subfield[1])
else:
for subfield in field[0]:
if subfield[0] == code:
tmp.append(subfield[1])
# If tmp was not set, nothing was found
return tmp
def record_xml_output(rec, tags=None, order_fn=None):
"""Generate the XML for record 'rec'.
:param rec: record
:param tags: list of tags to be printed
:return: string
"""
if tags is None:
tags = []
if isinstance(tags, str):
tags = [tags]
if tags and '001' not in tags:
# Add the missing controlfield.
tags.append('001')
marcxml = ['<record>']
# Add the tag 'tag' to each field in rec[tag]
fields = []
if rec is not None:
for tag in rec:
if not tags or tag in tags:
for field in rec[tag]:
fields.append((tag, field))
if order_fn is None:
record_order_fields(fields)
else:
record_order_fields(fields, order_fn)
for field in fields:
marcxml.append(field_xml_output(field[1], field[0]))
marcxml.append('</record>')
return '\n'.join(marcxml)
def field_get_subfield_instances(field):
"""Return the list of subfields associated with field 'field'."""
return field[0]
def field_get_subfield_values(field_instance, code):
"""Return subfield CODE values of the field instance FIELD."""
return [subfield_value
for subfield_code, subfield_value in field_instance[0]
if subfield_code == code]
def field_get_subfield_codes(field_instance):
"""Return subfield codes of the field instance FIELD."""
return [subfield_code
for subfield_code, subfield_value in field_instance[0]]
def field_add_subfield(field, code, value):
"""Add a subfield to field 'field'."""
field[0].append((code, value))
def record_order_fields(rec, fun="_order_by_ord"):
"""Order field inside record 'rec' according to a function."""
rec.sort(eval(fun))
def field_xml_output(field, tag):
"""Generate the XML for field 'field' and returns it as a string."""
marcxml = []
if field[3]:
marcxml.append(' <controlfield tag="%s">%s</controlfield>' %
(tag, encode_for_xml(field[3])))
else:
marcxml.append(' <datafield tag="%s" ind1="%s" ind2="%s">' %
(tag, field[1], field[2]))
marcxml += [_subfield_xml_output(subfield) for subfield in field[0]]
marcxml.append(' </datafield>')
return '\n'.join(map(str, marcxml))
def record_extract_oai_id(record):
"""Return the OAI ID of the record."""
tag = cfg['CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG'][0:3]
ind1 = cfg['CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG'][3]
ind2 = cfg['CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG'][4]
subfield = cfg['CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG'][5]
values = record_get_field_values(record, tag, ind1, ind2, subfield)
oai_id_regex = re.compile("oai[a-zA-Z0-9/.:]+")
for value in [value.strip() for value in values]:
if oai_id_regex.match(value):
return value
return ""
def record_extract_dois(record):
"""Return the DOI(s) of the record."""
record_dois = []
tag = "024"
ind1 = "7"
ind2 = "_"
subfield_source_code = "2"
subfield_value_code = "a"
identifiers_fields = record_get_field_instances(record, tag, ind1, ind2)
for identifer_field in identifiers_fields:
if 'doi' in [val.lower() for val in
field_get_subfield_values(identifer_field,
subfield_source_code)]:
record_dois.extend(
field_get_subfield_values(
identifer_field,
subfield_value_code))
return record_dois
def print_rec(rec, format=1, tags=None):
"""
Print a record.
:param format: 1 XML, 2 HTML (not implemented)
:param tags: list of tags to be printed
"""
if tags is None:
tags = []
if format == 1:
text = record_xml_output(rec, tags)
else:
return ''
return text
def print_recs(listofrec, format=1, tags=None):
"""
Print a list of records.
:param format: 1 XML, 2 HTML (not implemented)
:param tags: list of tags to be printed
if 'listofrec' is not a list it returns empty string
"""
if tags is None:
tags = []
text = ""
if type(listofrec).__name__ != 'list':
return ""
else:
for rec in listofrec:
text = "%s\n%s" % (text, print_rec(rec, format, tags))
return text
def concat(alist):
"""Concatenate a list of lists."""
newl = []
for l in alist:
newl.extend(l)
return newl
def record_find_field(rec, tag, field, strict=False):
"""
Return the global and local positions of the first occurrence of the field.
:param rec: A record dictionary structure
:type rec: dictionary
:param tag: The tag of the field to search for
:type tag: string
:param field: A field tuple as returned by create_field()
:type field: tuple
:param strict: A boolean describing the search method. If strict
is False, then the order of the subfields doesn't
matter. Default search method is strict.
:type strict: boolean
:return: A tuple of (global_position, local_position) or a
tuple (None, None) if the field is not present.
:rtype: tuple
:raise InvenioBibRecordFieldError: If the provided field is invalid.
"""
try:
_check_field_validity(field)
except InvenioBibRecordFieldError:
raise
for local_position, field1 in enumerate(rec.get(tag, [])):
if _compare_fields(field, field1, strict):
return (field1[4], local_position)
return (None, None)
+def record_match_subfields(rec, tag, ind1=" ", ind2=" ", sub_key=None,
+ sub_value='', sub_key2=None, sub_value2='',
+ case_sensitive=True):
+ """ Finds subfield instances in a particular field and tests
+ values in 1 of 3 possible ways:
+ - Does a subfield code exist? (ie does 773__a exist?)
+ - Does a subfield have a particular value? (ie 773__a == 'PhysX')
+ - Do a pair of subfields have particular values?
+ (ie 035__2 == 'CDS' and 035__a == '123456')
+
+ Parameters:
+ * rec - dictionary: a bibrecord structure
+ * tag - string: the tag of the field (ie '773')
+ * ind1, ind2 - char: a single characters for the MARC indicators
+ * sub_key - char: subfield key to find
+ * sub_value - string: subfield value of that key
+ * sub_key2 - char: key of subfield to compare against
+ * sub_value2 - string: expected value of second subfield
+ * case_sensitive - bool: be case sensitive when matching values
+
+ Returns: false if no match found, else provides the field position (int) """
+ if sub_key is None:
+ raise TypeError("None object passed for parameter sub_key.")
+
+ if sub_key2 is not None and sub_value2 is '':
+ raise TypeError("Parameter sub_key2 defined but sub_value2 is None, "
+ + "function requires a value for comparrison.")
+ ind1, ind2 = _wash_indicators(ind1, ind2)
+
+ if not case_sensitive:
+ sub_value = sub_value.lower()
+ sub_value2 = sub_value2.lower()
+
+ for field in record_get_field_instances(rec, tag, ind1, ind2):
+ subfields = dict(field_get_subfield_instances(field))
+ if not case_sensitive:
+ for k, v in subfields.iteritems():
+ subfields[k] = v.lower()
+
+ if sub_key in subfields:
+ if sub_value is '':
+ return field[4]
+ else:
+ if sub_value == subfields[sub_key]:
+ if sub_key2 is None:
+ return field[4]
+ else:
+ if sub_key2 in subfields:
+ if sub_value2 == subfields[sub_key2]:
+ return field[4]
+ return False
def record_strip_empty_volatile_subfields(rec):
"""Remove unchanged volatile subfields from the record."""
for tag in rec.keys():
for field in rec[tag]:
field[0][:] = [subfield for subfield in field[0]
if subfield[1][:9] != "VOLATILE:"]
def record_strip_empty_fields(rec, tag=None):
"""
Remove empty subfields and fields from the record.
If 'tag' is not None, only a specific tag of the record will be stripped,
otherwise the whole record.
:param rec: A record dictionary structure
:type rec: dictionary
:param tag: The tag of the field to strip empty fields from
:type tag: string
"""
# Check whole record
if tag is None:
tags = rec.keys()
for tag in tags:
record_strip_empty_fields(rec, tag)
# Check specific tag of the record
elif tag in rec:
# in case of a controlfield
if tag[:2] == '00':
if len(rec[tag]) == 0 or not rec[tag][0][3]:
del rec[tag]
#in case of a normal field
else:
fields = []
for field in rec[tag]:
subfields = []
for subfield in field[0]:
# check if the subfield has been given a value
if subfield[1]:
# Always strip values
subfield = (subfield[0], subfield[1].strip())
subfields.append(subfield)
if len(subfields) > 0:
new_field = create_field(subfields, field[1], field[2],
field[3])
fields.append(new_field)
if len(fields) > 0:
rec[tag] = fields
else:
del rec[tag]
def record_strip_controlfields(rec):
"""
Remove all non-empty controlfields from the record.
:param rec: A record dictionary structure
:type rec: dictionary
"""
for tag in rec.keys():
if tag[:2] == '00' and rec[tag][0][3]:
del rec[tag]
def record_order_subfields(rec, tag=None):
"""
Order subfields from a record alphabetically based on subfield code.
If 'tag' is not None, only a specific tag of the record will be reordered,
otherwise the whole record.
:param rec: bibrecord
:type rec: bibrec
:param tag: tag where the subfields will be ordered
:type tag: string
"""
if rec is None:
return rec
if tag is None:
tags = rec.keys()
for tag in tags:
record_order_subfields(rec, tag)
elif tag in rec:
for i in xrange(len(rec[tag])):
field = rec[tag][i]
# Order subfields alphabetically by subfield code
ordered_subfields = sorted(field[0],
key=lambda subfield: subfield[0])
rec[tag][i] = (ordered_subfields, field[1], field[2], field[3],
field[4])
def record_empty(rec):
for key in rec.iterkeys():
if key not in ('001', '005'):
return False
return True
-
### IMPLEMENTATION / INVISIBLE FUNCTIONS
def _compare_fields(field1, field2, strict=True):
"""
Compare 2 fields.
If strict is True, then the order of the subfield will be taken care of, if
not then the order of the subfields doesn't matter.
:return: True if the field are equivalent, False otherwise.
"""
if strict:
# Return a simple equal test on the field minus the position.
return field1[:4] == field2[:4]
else:
if field1[1:4] != field2[1:4]:
# Different indicators or controlfield value.
return False
else:
# Compare subfields in a loose way.
return set(field1[0]) == set(field2[0])
def _check_field_validity(field):
"""
Check if a field is well-formed.
:param field: A field tuple as returned by create_field()
:type field: tuple
:raise InvenioBibRecordFieldError: If the field is invalid.
"""
if type(field) not in (list, tuple):
raise InvenioBibRecordFieldError(
"Field of type '%s' should be either "
"a list or a tuple." % type(field))
if len(field) != 5:
raise InvenioBibRecordFieldError(
"Field of length '%d' should have 5 "
"elements." % len(field))
if type(field[0]) not in (list, tuple):
raise InvenioBibRecordFieldError(
"Subfields of type '%s' should be "
"either a list or a tuple." % type(field[0]))
if type(field[1]) is not str:
raise InvenioBibRecordFieldError(
"Indicator 1 of type '%s' should be "
"a string." % type(field[1]))
if type(field[2]) is not str:
raise InvenioBibRecordFieldError(
"Indicator 2 of type '%s' should be "
"a string." % type(field[2]))
if type(field[3]) is not str:
raise InvenioBibRecordFieldError(
"Controlfield value of type '%s' "
"should be a string." % type(field[3]))
if type(field[4]) is not int:
raise InvenioBibRecordFieldError(
"Global position of type '%s' should "
"be an int." % type(field[4]))
for subfield in field[0]:
if (type(subfield) not in (list, tuple) or
len(subfield) != 2 or type(subfield[0]) is not str or
type(subfield[1]) is not str):
raise InvenioBibRecordFieldError(
"Subfields are malformed. "
"Should a list of tuples of 2 strings.")
def _shift_field_positions_global(record, start, delta=1):
"""
Shift all global field positions.
Shift all global field positions with global field positions
higher or equal to 'start' from the value 'delta'.
"""
if not delta:
return
for tag, fields in record.items():
newfields = []
for field in fields:
if field[4] < start:
newfields.append(field)
else:
# Increment the global field position by delta.
newfields.append(tuple(list(field[:4]) + [field[4] + delta]))
record[tag] = newfields
def _tag_matches_pattern(tag, pattern):
"""Return true if MARC 'tag' matches a 'pattern'.
'pattern' is plain text, with % as wildcard
Both parameters must be 3 characters long strings.
.. doctest::
>>> _tag_matches_pattern("909", "909")
True
>>> _tag_matches_pattern("909", "9%9")
True
>>> _tag_matches_pattern("909", "9%8")
False
:param tag: a 3 characters long string
:param pattern: a 3 characters long string
:return: False or True
"""
for char1, char2 in zip(tag, pattern):
if char2 not in ('%', char1):
return False
return True
-def validate_record_field_positions_global(record):
+def _validate_record_field_positions_global(record):
"""
Check if the global field positions in the record are valid.
I.e., no duplicate global field positions and local field positions in the
list of fields are ascending.
:param record: the record data structure
:return: the first error found as a string or None if no error was found
"""
all_fields = []
for tag, fields in record.items():
previous_field_position_global = -1
for field in fields:
if field[4] < previous_field_position_global:
return ("Non ascending global field positions in tag '%s'." %
tag)
previous_field_position_global = field[4]
if field[4] in all_fields:
return ("Duplicate global field position '%d' in tag '%s'" %
(field[4], tag))
def get_fieldvalues(recIDs, tag, repetitive_values=True, sort=True,
split_by=0):
"""
Return list of field values for field TAG for the given record.
Record can be ID or list of record IDs. (RECIDS can be both an integer or
a list of integers.)
If REPETITIVE_VALUES is set to True, then return all values even
if they are doubled. If set to False, then return unique values
only.
"""
out = []
try:
recIDs = int(recIDs)
except:
pass
if isinstance(recIDs, (int, long)):
recIDs = [recIDs]
if not isinstance(recIDs, (list, tuple, intbitset)):
return []
if len(recIDs) == 0:
return []
if tag == "001___":
# We have asked for tag 001 (=recID) that is not stored in bibXXx
# tables.
out = [str(recID) for recID in recIDs]
else:
# we are going to look inside bibXXx tables
digits = tag[0:2]
try:
intdigits = int(digits)
if intdigits < 0 or intdigits > 99:
raise ValueError
except ValueError:
# invalid tag value asked for
return []
bx = "bib%sx" % digits
bibx = "bibrec_bib%sx" % digits
if not repetitive_values:
queryselect = "DISTINCT(bx.value)"
else:
queryselect = "bx.value"
if sort:
sort_sql = "ORDER BY bibx.field_number, bx.tag ASC"
else:
sort_sql = ""
def get_res(recIDs):
query = "SELECT %s FROM %s AS bx, %s AS bibx " \
"WHERE bibx.id_bibrec IN (%s) AND bx.id=bibx.id_bibxxx " \
"AND bx.tag LIKE %%s %s" % \
(queryselect, bx, bibx, ("%s," * len(recIDs))[:-1],
sort_sql)
return [i[0] for i in run_sql(query, tuple(recIDs) + (tag,))]
if sort or split_by <= 0 or len(recIDs) <= split_by:
return get_res(recIDs)
else:
return [i for res in map(get_res, zip(*[iter(recIDs)] * split_by))
for i in res]
return out
def get_fieldvalues_alephseq_like(recID, tags_in, can_see_hidden=False):
"""
Return buffer of ALEPH sequential-like textual format.
Return buffer of ALEPH sequential-like textual format with fields found in
the list TAGS_IN for record RECID.
If can_see_hidden is True, just print everything. Otherwise hide fields
from CFG_BIBFORMAT_HIDDEN_TAGS.
"""
out = ""
if type(tags_in) is not list:
tags_in = [tags_in]
if len(tags_in) == 1 and len(tags_in[0]) == 6:
## case A: one concrete subfield asked, so print its value if found
## (use with care: can mislead if field has multiple
## occurrences)
out += string.join(get_fieldvalues(recID, tags_in[0]), "\n")
else:
## case B: print our "text MARC" format; works safely all the time
# find out which tags to output:
dict_of_tags_out = {}
if not tags_in:
for i in range(0, 10):
for j in range(0, 10):
dict_of_tags_out["%d%d%%" % (i, j)] = 1
else:
for tag in tags_in:
if len(tag) == 0:
for i in range(0, 10):
for j in range(0, 10):
dict_of_tags_out["%d%d%%" % (i, j)] = 1
elif len(tag) == 1:
for j in range(0, 10):
dict_of_tags_out["%s%d%%" % (tag, j)] = 1
elif len(tag) < 5:
dict_of_tags_out["%s%%" % tag] = 1
elif tag >= 6:
dict_of_tags_out[tag[0:5]] = 1
tags_out = dict_of_tags_out.keys()
tags_out.sort()
# search all bibXXx tables as needed:
for tag in tags_out:
digits = tag[0:2]
try:
intdigits = int(digits)
if intdigits < 0 or intdigits > 99:
raise ValueError
except ValueError:
# invalid tag value asked for
continue
if tag.startswith("001") or tag.startswith("00%"):
if out:
out += "\n"
out += "%09d %s %d" % (recID, "001__", recID)
bx = "bib%sx" % digits
bibx = "bibrec_bib%sx" % digits
query = "SELECT b.tag,b.value,bb.field_number FROM %s AS b, %s " \
"AS bb WHERE bb.id_bibrec=%%s AND b.id=bb.id_bibxxx AND" \
" b.tag LIKE %%s ORDER BY bb.field_number, b.tag ASC" % \
(bx, bibx)
res = run_sql(query, (recID, str(tag) + '%'))
# go through fields:
field_number_old = -999
field_old = ""
for row in res:
field, value, field_number = row[0], row[1], row[2]
ind1, ind2 = field[3], field[4]
printme = True
#check the stuff in hiddenfields
if not can_see_hidden:
for htag in cfg['CFG_BIBFORMAT_HIDDEN_TAGS']:
ltag = len(htag)
samelenfield = field[0:ltag]
if samelenfield == htag:
printme = False
if ind1 == "_":
ind1 = ""
if ind2 == "_":
ind2 = ""
# print field tag
if printme:
if field_number != field_number_old or \
field[:-1] != field_old[:-1]:
if out:
out += "\n"
out += "%09d %s " % (recID, field[:5])
field_number_old = field_number
field_old = field
# print subfield value
if field[0:2] == "00" and field[-1:] == "_":
out += value
else:
out += "$$%s%s" % (field[-1:], value)
return out
def _record_sort_by_indicators(record):
"""Sort the fields inside the record by indicators."""
for tag, fields in record.items():
record[tag] = _fields_sort_by_indicators(fields)
def _fields_sort_by_indicators(fields):
"""Sort a set of fields by their indicators.
Return a sorted list with correct global field positions.
"""
field_dict = {}
field_positions_global = []
for field in fields:
field_dict.setdefault(field[1:3], []).append(field)
field_positions_global.append(field[4])
indicators = field_dict.keys()
indicators.sort()
field_list = []
for indicator in indicators:
for field in field_dict[indicator]:
field_list.append(field[:4] + (field_positions_global.pop(0),))
return field_list
def _select_parser(parser=None):
"""
Select the more relevant parser.
Selection is based on the parsers available and on the parser desired by
the user.
"""
if not AVAILABLE_PARSERS:
# No parser is available. This is bad.
return None
if parser is None or parser not in AVAILABLE_PARSERS:
# Return the best available parser.
return AVAILABLE_PARSERS[0]
else:
return parser
def _create_record_lxml(marcxml,
verbose=CFG_BIBRECORD_DEFAULT_VERBOSE_LEVEL,
correct=CFG_BIBRECORD_DEFAULT_CORRECT,
keep_singletons=CFG_BIBRECORD_KEEP_SINGLETONS):
"""
Create a record object using the LXML parser.
If correct == 1, then perform DTD validation
If correct == 0, then do not perform DTD validation
If verbose == 0, the parser will not give warnings.
If 1 <= verbose <= 3, the parser will not give errors, but will warn
the user about possible mistakes (implement me!)
If verbose > 3 then the parser will be strict and will stop in case of
well-formedness errors or DTD errors.
"""
parser = etree.XMLParser(dtd_validation=correct,
recover=(verbose <= 3))
if correct:
marcxml = '<?xml version="1.0" encoding="UTF-8"?>\n' \
'<!DOCTYPE collection SYSTEM "file://%s">\n' \
'<collection>\n%s\n</collection>' % (CFG_MARC21_DTD, marcxml)
try:
tree = etree.parse(StringIO(marcxml), parser)
# parser errors are located in parser.error_log
# if 1 <= verbose <=3 then show them to the user?
# if verbose == 0 then continue
# if verbose >3 then an exception will be thrown
except Exception as e:
raise InvenioBibRecordParserError(str(e))
record = {}
field_position_global = 0
controlfield_iterator = tree.iter(tag='controlfield')
for controlfield in controlfield_iterator:
tag = controlfield.attrib.get('tag', '!').encode("UTF-8")
ind1 = ' '
ind2 = ' '
text = controlfield.text
if text is None:
text = ''
else:
text = text.encode("UTF-8")
subfields = []
if text or keep_singletons:
field_position_global += 1
record.setdefault(tag, []).append((subfields, ind1, ind2, text,
field_position_global))
datafield_iterator = tree.iter(tag='datafield')
for datafield in datafield_iterator:
tag = datafield.attrib.get('tag', '!').encode("UTF-8")
ind1 = datafield.attrib.get('ind1', '!').encode("UTF-8")
ind2 = datafield.attrib.get('ind2', '!').encode("UTF-8")
#ind1, ind2 = _wash_indicators(ind1, ind2)
if ind1 in ('', '_'):
ind1 = ' '
if ind2 in ('', '_'):
ind2 = ' '
subfields = []
subfield_iterator = datafield.iter(tag='subfield')
for subfield in subfield_iterator:
code = subfield.attrib.get('code', '!').encode("UTF-8")
text = subfield.text
if text is None:
text = ''
else:
text = text.encode("UTF-8")
if text or keep_singletons:
subfields.append((code, text))
if subfields or keep_singletons:
text = ''
field_position_global += 1
record.setdefault(tag, []).append((subfields, ind1, ind2, text,
field_position_global))
return record
def _create_record_rxp(marcxml,
verbose=CFG_BIBRECORD_DEFAULT_VERBOSE_LEVEL,
correct=CFG_BIBRECORD_DEFAULT_CORRECT,
keep_singletons=CFG_BIBRECORD_KEEP_SINGLETONS):
"""Create a record object using the RXP parser.
If verbose>3 then the parser will be strict and will stop in case of
well-formedness errors or DTD errors.
If verbose=0, the parser will not give warnings.
If 0 < verbose <= 3, the parser will not give errors, but will warn
the user about possible mistakes
correct != 0 -> We will try to correct errors such as missing
attributes
correct = 0 -> there will not be any attempt to correct errors
"""
if correct:
# Note that with pyRXP < 1.13 a memory leak has been found
# involving DTD parsing. So enable correction only if you have
# pyRXP 1.13 or greater.
marcxml = ('<?xml version="1.0" encoding="UTF-8"?>\n'
'<!DOCTYPE collection SYSTEM "file://%s">\n'
'<collection>\n%s\n</collection>' %
(CFG_MARC21_DTD, marcxml))
# Create the pyRXP parser.
pyrxp_parser = pyRXP.Parser(ErrorOnValidityErrors=0, ProcessDTD=1,
ErrorOnUnquotedAttributeValues=0,
srcName='string input')
if verbose > 3:
pyrxp_parser.ErrorOnValidityErrors = 1
pyrxp_parser.ErrorOnUnquotedAttributeValues = 1
try:
root = pyrxp_parser.parse(marcxml)
except pyRXP.error as ex1:
raise InvenioBibRecordParserError(str(ex1))
# If record is enclosed in a collection tag, extract it.
if root[TAG] == 'collection':
children = _get_children_by_tag_name_rxp(root, 'record')
if not children:
return {}
root = children[0]
record = {}
# This is needed because of the record_xml_output function, where we
# need to know the order of the fields.
field_position_global = 1
# Consider the control fields.
for controlfield in _get_children_by_tag_name_rxp(root, 'controlfield'):
if controlfield[CHILDREN]:
value = ''.join([n for n in controlfield[CHILDREN]])
# Construct the field tuple.
field = ([], ' ', ' ', value, field_position_global)
record.setdefault(controlfield[ATTRS]['tag'], []).append(field)
field_position_global += 1
elif keep_singletons:
field = ([], ' ', ' ', '', field_position_global)
record.setdefault(controlfield[ATTRS]['tag'], []).append(field)
field_position_global += 1
# Consider the data fields.
for datafield in _get_children_by_tag_name_rxp(root, 'datafield'):
subfields = []
for subfield in _get_children_by_tag_name_rxp(datafield, 'subfield'):
if subfield[CHILDREN]:
value = _get_children_as_string_rxp(subfield[CHILDREN])
subfields.append((subfield[ATTRS].get('code', '!'), value))
elif keep_singletons:
subfields.append((subfield[ATTRS].get('code', '!'), ''))
if subfields or keep_singletons:
# Create the field.
tag = datafield[ATTRS].get('tag', '!')
ind1 = datafield[ATTRS].get('ind1', '!')
ind2 = datafield[ATTRS].get('ind2', '!')
ind1, ind2 = _wash_indicators(ind1, ind2)
# Construct the field tuple.
field = (subfields, ind1, ind2, '', field_position_global)
record.setdefault(tag, []).append(field)
field_position_global += 1
return record
def _create_record_from_document(
document,
keep_singletons=CFG_BIBRECORD_KEEP_SINGLETONS):
"""
Create a record from the document.
Of type xml.dom.minidom.Document or Ft.Xml.Domlette.Document).
"""
root = None
for node in document.childNodes:
if node.nodeType == node.ELEMENT_NODE:
root = node
break
if root is None:
return {}
if root.tagName == 'collection':
children = _get_children_by_tag_name(root, 'record')
if not children:
return {}
root = children[0]
field_position_global = 1
record = {}
for controlfield in _get_children_by_tag_name(root, "controlfield"):
tag = controlfield.getAttributeNS(None, "tag").encode('utf-8')
text_nodes = controlfield.childNodes
value = ''.join([n.data for n in text_nodes]).encode("utf-8")
if value or keep_singletons:
field = ([], " ", " ", value, field_position_global)
record.setdefault(tag, []).append(field)
field_position_global += 1
for datafield in _get_children_by_tag_name(root, "datafield"):
subfields = []
for subfield in _get_children_by_tag_name(datafield, "subfield"):
value = _get_children_as_string(subfield.childNodes) \
.encode("utf-8")
if value or keep_singletons:
code = subfield.getAttributeNS(None, 'code').encode("utf-8")
subfields.append((code or '!', value))
if subfields or keep_singletons:
tag = datafield.getAttributeNS(None, "tag").encode("utf-8") or '!'
ind1 = datafield.getAttributeNS(None, "ind1").encode("utf-8")
ind2 = datafield.getAttributeNS(None, "ind2").encode("utf-8")
ind1, ind2 = _wash_indicators(ind1, ind2)
field = (subfields, ind1, ind2, "", field_position_global)
record.setdefault(tag, []).append(field)
field_position_global += 1
return record
def _create_record_minidom(marcxml,
keep_singletons=CFG_BIBRECORD_KEEP_SINGLETONS):
"""Create a record using minidom."""
try:
dom = xml.dom.minidom.parseString(marcxml)
except xml.parsers.expat.ExpatError as ex1:
raise InvenioBibRecordParserError(str(ex1))
return _create_record_from_document(dom, keep_singletons=keep_singletons)
def _create_record_4suite(marcxml,
keep_singletons=CFG_BIBRECORD_KEEP_SINGLETONS):
"""Create a record using the 4suite parser."""
try:
dom = Ft.Xml.Domlette.NonvalidatingReader.parseString(marcxml,
"urn:dummy")
except Ft.Xml.ReaderException as ex1:
raise InvenioBibRecordParserError(ex1.message)
return _create_record_from_document(dom, keep_singletons=keep_singletons)
def _concat(alist):
"""Concatenate a list of lists."""
return [element for single_list in alist for element in single_list]
def _subfield_xml_output(subfield):
"""Generate the XML for a subfield object and return it as a string."""
return ' <subfield code="%s">%s</subfield>' % \
(subfield[0], encode_for_xml(subfield[1]))
def _order_by_ord(field1, field2):
"""Function used to order the fields according to their ord value."""
return cmp(field1[1][4], field2[1][4])
def _order_by_tags(field1, field2):
"""Function used to order the fields according to the tags."""
return cmp(field1[0], field2[0])
def _get_children_by_tag_name(node, name):
"""Retrieve all children from node 'node' with name 'name'."""
try:
return [child for child in node.childNodes if child.nodeName == name]
except TypeError:
return []
def _get_children_by_tag_name_rxp(node, name):
"""Retrieve all children from 'children' with tag name 'tag'.
children is a list returned by the RXP parser
"""
try:
return [child for child in node[CHILDREN] if child[TAG] == name]
except TypeError:
return []
def _get_children_as_string(node):
"""Iterate through all the children of a node.
Returns one string containing the values from all the text-nodes
recursively.
"""
out = []
if node:
for child in node:
if child.nodeType == child.TEXT_NODE:
out.append(child.data)
else:
out.append(_get_children_as_string(child.childNodes))
return ''.join(out)
def _get_children_as_string_rxp(node):
"""
RXP version of _get_children_as_string().
Iterate through all the children of a node and returns one string
containing the values from all the text-nodes recursively.
"""
out = []
if node:
for child in node:
if type(child) is str:
out.append(child)
else:
out.append(_get_children_as_string_rxp(child[CHILDREN]))
return ''.join(out)
def _wash_indicators(*indicators):
"""
Wash the values of the indicators.
An empty string or an underscore is replaced by a blank space.
:param indicators: a series of indicators to be washed
:return: a list of washed indicators
"""
return [indicator in ('', '_') and ' ' or indicator
for indicator in indicators]
def _correct_record(record):
"""
Check and correct the structure of the record.
:param record: the record data structure
:return: a list of errors found
"""
errors = []
for tag in record.keys():
upper_bound = '999'
n = len(tag)
if n > 3:
i = n - 3
while i > 0:
upper_bound = '%s%s' % ('0', upper_bound)
i -= 1
# Missing tag. Replace it with dummy tag '000'.
if tag == '!':
errors.append((1, '(field number(s): ' +
str([f[4] for f in record[tag]]) + ')'))
record['000'] = record.pop(tag)
tag = '000'
elif not ('001' <= tag <= upper_bound or
tag in ('FMT', 'FFT', 'BDR', 'BDM')):
errors.append(2)
record['000'] = record.pop(tag)
tag = '000'
fields = []
for field in record[tag]:
# Datafield without any subfield.
if field[0] == [] and field[3] == '':
errors.append((8, '(field number: ' + str(field[4]) + ')'))
subfields = []
for subfield in field[0]:
if subfield[0] == '!':
errors.append((3, '(field number: ' + str(field[4]) + ')'))
newsub = ('', subfield[1])
else:
newsub = subfield
subfields.append(newsub)
if field[1] == '!':
errors.append((4, '(field number: ' + str(field[4]) + ')'))
ind1 = " "
else:
ind1 = field[1]
if field[2] == '!':
errors.append((5, '(field number: ' + str(field[4]) + ')'))
ind2 = " "
else:
ind2 = field[2]
fields.append((subfields, ind1, ind2, field[3], field[4]))
record[tag] = fields
return errors
def _warning(code):
"""
Return a warning message of code 'code'.
If code = (cd, str) it returns the warning message of code 'cd' and appends
str at the end
"""
if isinstance(code, str):
return code
message = ''
if isinstance(code, tuple):
if isinstance(code[0], str):
message = code[1]
code = code[0]
return CFG_BIBRECORD_WARNING_MSGS.get(code, '') + message
def _warnings(alist):
"""Apply the function _warning() to every element in alist."""
return [_warning(element) for element in alist]
def _compare_lists(list1, list2, custom_cmp):
"""Compare twolists using given comparing function.
:param list1: first list to compare
:param list2: second list to compare
:param custom_cmp: a function taking two arguments (element of
list 1, element of list 2) and
:return: True or False depending if the values are the same
"""
if len(list1) != len(list2):
return False
for element1, element2 in zip(list1, list2):
if not custom_cmp(element1, element2):
return False
return True
diff --git a/invenio/legacy/bibsched/bibtask.py b/invenio/legacy/bibsched/bibtask.py
index fc5387b6d..314047a9e 100644
--- a/invenio/legacy/bibsched/bibtask.py
+++ b/invenio/legacy/bibsched/bibtask.py
@@ -1,1360 +1,1359 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
"""Invenio Bibliographic Task Class.
BibTask class.
A BibTask is an executable under CFG_BINDIR, whose name is stored in
bibtask_config.CFG_BIBTASK_VALID_TASKS.
A valid task must call the task_init function with the proper parameters.
Generic task related parameters (user, sleeptime, runtime, task_id, task_name
verbose)
go to _TASK_PARAMS global dictionary accessible through task_get_task_param.
Option specific to the particular BibTask go to _OPTIONS global dictionary
and are accessible via task_get_option/task_set_option.
In order to log something properly, just use write_message(s) with the desired
verbose level.
task_update_status and task_update_progress can be used to update the status
of the task (DONE, FAILED, DONE WITH ERRORS...) and it's progress
(1 out 100..) within the bibsched monitor.
It is possible to enqueue a BibTask via API call by means of
task_low_level_submission.
"""
import getopt
import getpass
import marshal
import os
import pwd
import re
import signal
import sys
import time
import datetime
import traceback
import logging
import logging.handlers
import random
from flask import current_app
from intbitset import intbitset
from socket import gethostname
from invenio.legacy.dbquery import run_sql, _db_login
from invenio.modules.access.engine import acc_authorize_action
from invenio.config import CFG_PREFIX, \
CFG_BINDIR, \
CFG_BIBSCHED_PROCESS_USER, \
CFG_TMPDIR, \
CFG_SITE_SUPPORT_EMAIL, \
CFG_VERSION, \
CFG_BIBSCHED_FLUSH_LOGS
from invenio.ext.logging import register_exception
from invenio.modules.access.local_config import CFG_EXTERNAL_AUTH_USING_SSO, \
CFG_EXTERNAL_AUTHENTICATION
from invenio.legacy.webuser import get_user_preferences, get_email
from invenio.legacy.bibsched.bibtask_config import (
CFG_BIBTASK_VALID_TASKS,
CFG_BIBTASK_DEFAULT_TASK_SETTINGS,
CFG_BIBTASK_FIXEDTIMETASKS,
CFG_BIBTASK_DEFAULT_GLOBAL_TASK_SETTINGS,
CFG_BIBSCHED_LOGDIR,
CFG_BIBTASK_LOG_FORMAT)
from invenio.utils.date import parse_runtime_limit
from invenio.utils.shell import escape_shell_arg
from invenio.ext.email import send_email
from invenio.legacy.bibsched.cli import bibsched_set_host, \
bibsched_get_host
# Global _TASK_PARAMS dictionary.
_TASK_PARAMS = dict(CFG_BIBTASK_DEFAULT_GLOBAL_TASK_SETTINGS)
# Global _OPTIONS dictionary.
_OPTIONS = {}
# Which tasks don't need to ask the user for authorization?
CFG_VALID_PROCESSES_NO_AUTH_NEEDED = ("bibupload", )
CFG_TASK_IS_NOT_A_DEAMON = ("bibupload", )
class RecoverableError(StandardError):
pass
class InvalidParams(StandardError):
def __init__(self, err=None):
self.err = err
StandardError.__init__(self)
def fix_argv_paths(paths, argv=None):
"""Given the argv vector of cli parameters, and a list of path that
can be relative and may have been specified within argv,
it substitute all the occurencies of these paths in argv.
argv is changed in place and returned.
"""
if argv is None:
argv = sys.argv
for path in paths:
for count in xrange(len(argv)):
if path == argv[count]:
argv[count] = os.path.abspath(path)
return argv
def get_sleeptime(argv):
"""Try to get the runtime by analysing the arguments."""
sleeptime = ""
argv = list(argv)
while True:
try:
opts, dummy_args = getopt.gnu_getopt(argv, 's:', ['sleeptime='])
except getopt.GetoptError, err:
## We remove one by one all the non recognized parameters
if len(err.opt) > 1:
argv = [arg for arg in argv if arg != '--%s' % err.opt and not arg.startswith('--%s=' % err.opt)]
else:
argv = [arg for arg in argv if not arg.startswith('-%s' % err.opt)]
else:
break
for opt in opts:
if opt[0] in ('-s', '--sleeptime'):
try:
sleeptime = opt[1]
except ValueError:
pass
return sleeptime
def task_low_level_submission(name, user, *argv):
"""Let special lowlevel enqueuing of a task on the bibsche queue.
@param name: is the name of the bibtask. It must be a valid executable under
C{CFG_BINDIR}.
@type name: string
@param user: is a string that will appear as the "user" submitting the task.
Since task are submitted via API it make sense to set the
user to the name of the module/function that called
task_low_level_submission.
@type user: string
@param argv: are all the additional CLI parameters that would have been
passed on the CLI (one parameter per variable).
e.g.:
>>> task_low_level_submission('bibupload', 'admin', '-a', '/tmp/z.xml')
@type: strings
@return: the task identifier when the task is correctly enqueued.
@rtype: int
@note: use absolute paths in argv
"""
def get_priority(argv):
"""Try to get the priority by analysing the arguments."""
priority = 0
argv = list(argv)
while True:
try:
opts, dummy_args = getopt.gnu_getopt(argv, 'P:', ['priority='])
except getopt.GetoptError as err:
## We remove one by one all the non recognized parameters
if len(err.opt) > 1:
argv = [arg for arg in argv if arg != '--%s' % err.opt and not arg.startswith('--%s=' % err.opt)]
else:
argv = [arg for arg in argv if not arg.startswith('-%s' % err.opt)]
else:
break
for opt in opts:
if opt[0] in ('-P', '--priority'):
try:
priority = int(opt[1])
except ValueError:
pass
return priority
def get_special_name(argv):
"""Try to get the special name by analysing the arguments."""
special_name = ''
argv = list(argv)
while True:
try:
opts, dummy_args = getopt.gnu_getopt(argv, 'N:', ['name='])
except getopt.GetoptError as err:
## We remove one by one all the non recognized parameters
if len(err.opt) > 1:
argv = [arg for arg in argv if arg != '--%s' % err.opt and not arg.startswith('--%s=' % err.opt)]
else:
argv = [arg for arg in argv if not arg.startswith('-%s' % err.opt)]
else:
break
for opt in opts:
if opt[0] in ('-N', '--name'):
special_name = opt[1]
return special_name
def get_runtime(argv):
"""Try to get the runtime by analysing the arguments."""
runtime = time.strftime("%Y-%m-%d %H:%M:%S")
argv = list(argv)
while True:
try:
opts, dummy_args = getopt.gnu_getopt(argv, 't:', ['runtime='])
except getopt.GetoptError as err:
## We remove one by one all the non recognized parameters
if len(err.opt) > 1:
argv = [arg for arg in argv if arg != '--%s' % err.opt and not arg.startswith('--%s=' % err.opt)]
else:
argv = [arg for arg in argv if not arg.startswith('-%s' % err.opt)]
else:
break
for opt in opts:
if opt[0] in ('-t', '--runtime'):
try:
runtime = get_datetime(opt[1])
except ValueError:
pass
return runtime
def get_sequenceid(argv):
"""Try to get the sequenceid by analysing the arguments."""
sequenceid = None
argv = list(argv)
while True:
try:
opts, dummy_args = getopt.gnu_getopt(argv, 'I:', ['sequence-id='])
except getopt.GetoptError as err:
## We remove one by one all the non recognized parameters
if len(err.opt) > 1:
argv = [arg for arg in argv if arg != '--%s' % err.opt and not arg.startswith('--%s=' % err.opt)]
else:
argv = [arg for arg in argv if not arg.startswith('-%s' % err.opt)]
else:
break
for opt in opts:
if opt[0] in ('-I', '--sequence-id'):
try:
sequenceid = opt[1]
except ValueError:
pass
return sequenceid
def get_host(argv):
"""Try to get the sequenceid by analysing the arguments."""
host = ''
argv = list(argv)
while True:
try:
opts, dummy_args = getopt.gnu_getopt(argv, '', ['host='])
- except getopt.GetoptError, err:
except getopt.GetoptError as err:
## We remove one by one all the non recognized parameters
if len(err.opt) > 1:
argv = [arg for arg in argv if arg != '--%s' % err.opt and not arg.startswith('--%s=' % err.opt)]
else:
argv = [arg for arg in argv if not arg.startswith('-%s' % err.opt)]
else:
break
for opt in opts:
if opt[0] in ('--host',):
try:
host = opt[1]
except ValueError:
pass
return host
task_id = None
try:
if not name in CFG_BIBTASK_VALID_TASKS:
raise StandardError('%s is not a valid task name' % name)
new_argv = []
for arg in argv:
if isinstance(arg, unicode):
arg = arg.encode('utf8')
new_argv.append(arg)
argv = new_argv
priority = get_priority(argv)
special_name = get_special_name(argv)
runtime = get_runtime(argv)
sleeptime = get_sleeptime(argv)
sequenceid = get_sequenceid(argv)
host = get_host(argv)
argv = tuple([os.path.join(CFG_BINDIR, name)] + list(argv))
if special_name:
name = '%s:%s' % (name, special_name)
verbose_argv = 'Will execute: %s' % ' '.join([escape_shell_arg(str(arg)) for arg in argv])
## submit task:
task_id = run_sql("""INSERT INTO schTASK (proc,host,user,
runtime,sleeptime,status,progress,arguments,priority,sequenceid)
VALUES (%s,%s,%s,%s,%s,'WAITING',%s,%s,%s,%s)""",
(name, host, user, runtime, sleeptime, verbose_argv,
marshal.dumps(argv), priority, sequenceid))
except Exception:
register_exception(alert_admin=True)
if task_id:
run_sql("""DELETE FROM schTASK WHERE id=%s""", (task_id, ))
raise
return task_id
def bibtask_allocate_sequenceid(curdir=None):
"""
Returns an almost unique number to be used a task sequence ID.
In WebSubmit functions, set C{curdir} to the curdir (!) to read
the shared sequence ID for all functions of this submission (reading
"access number").
@param curdir: in WebSubmit functions (ONLY) the value retrieved
from the curdir parameter of the function
@return: an integer for the sequence ID. 0 is returned if the
sequence ID could not be allocated
@rtype: int
"""
if curdir:
try:
fd = file(os.path.join(curdir, 'access'), "r")
access = fd.readline().strip()
fd.close()
return access.replace("_", "")[-9:]
except (IOError, OSError):
return 0
else:
return random.randrange(1, 4294967296)
def task_log_path(task_id, log_type):
"""Returns the path to the log files of given task
Args:
- task_id
- log_type: either 'log' or 'err' to indiciate which type of log we want
"""
sub_dir = str(task_id / 10000)
dest_dir = os.path.join(CFG_BIBSCHED_LOGDIR, sub_dir)
return os.path.join(dest_dir, 'bibsched_task_%d.%s' % (task_id, log_type))
def get_and_create_task_log_path(task_id, log_type):
"""Returns and creates the path to the log files of given task
@see task_log_path
"""
log_dest = task_log_path(task_id, log_type)
# log_dest and err_dest are in the same folder
dest_dir = os.path.dirname(log_dest)
try:
os.makedirs(dest_dir)
except OSError, e:
# If directory already exists, ignore error
if e.errno != 17:
raise
return log_dest
def create_logfiles_handlers(task_id, formatter):
"""Create log handlers to write into tasks log files
Args:
- task_id
- Formatter is an instance of logging.Formatter
Returns:
- log handler for standard log
- log handler for error log
"""
std_dest = get_and_create_task_log_path(task_id, 'log')
err_dest = get_and_create_task_log_path(task_id, 'err')
std_logger = logging.handlers.RotatingFileHandler(std_dest, 'a', 5*1024*1024, 10)
err_logger = logging.handlers.RotatingFileHandler(err_dest, 'a', 5*1024*1024, 10)
std_logger.setFormatter(formatter)
err_logger.setFormatter(formatter)
std_logger.setLevel(logging.DEBUG)
err_logger.setLevel(logging.WARNING)
return std_logger, err_logger
def create_streams_handlers(formatter):
"""Create log handlers to print to stdout/stderr
Args:
- Formatter is an instance of logging.Formatter
Returns:
- log handler for standard log
- log handler for error log
"""
stdout_logger = logging.StreamHandler(sys.stdout)
stdout_logger.setFormatter(formatter)
stdout_logger.setLevel(logging.DEBUG)
stderr_logger = logging.StreamHandler(sys.stderr)
stderr_logger.setFormatter(formatter)
stderr_logger.setLevel(logging.WARNING)
return stdout_logger, stderr_logger
def setup_loggers(task_id=None):
"""Sets up the logging system."""
logger = logging.getLogger()
# Let's clean the handlers in case some piece of code has already
# fired any write_message, i.e. any call to debug, info, etc.
# which triggered a call to logging.basicConfig()
for handler in logger.handlers:
logger.removeHandler(handler)
formatter = logging.Formatter(*CFG_BIBTASK_LOG_FORMAT)
# Log files
if task_id is not None:
log_logger, err_logger = create_logfiles_handlers(task_id, formatter)
logger.addHandler(err_logger)
logger.addHandler(log_logger)
# Stream handlers
stdout_logger, stderr_logger = create_streams_handlers(formatter)
logger.addHandler(stdout_logger)
logger.addHandler(stderr_logger)
# Default log level
logger.setLevel(logging.INFO)
return logger
def task_init(authorization_action="",
authorization_msg="",
description="",
help_specific_usage="",
version=CFG_VERSION,
specific_params=("", []),
task_stop_helper_fnc=None,
task_submit_elaborate_specific_parameter_fnc=None,
task_submit_check_options_fnc=None,
task_run_fnc=None):
""" Initialize a BibTask.
@param authorization_action: is the name of the authorization action
connected with this task;
@param authorization_msg: is the header printed when asking for an
authorization password;
@param description: is the generic description printed in the usage page;
@param help_specific_usage: is the specific parameter help
@param task_stop_fnc: is a function that will be called
whenever the task is stopped
@param task_submit_elaborate_specific_parameter_fnc: will be called passing
a key and a value, for parsing specific cli parameters. Must return True if
it has recognized the parameter. Must eventually update the options with
bibtask_set_option;
@param task_submit_check_options: must check the validity of options (via
bibtask_get_option) once all the options where parsed;
@param task_run_fnc: will be called as the main core function. Must return
False in case of errors.
"""
global _TASK_PARAMS, _OPTIONS
_TASK_PARAMS = {
"version" : version,
"task_stop_helper_fnc" : task_stop_helper_fnc,
"task_name" : os.path.basename(sys.argv[0]),
"task_specific_name" : '',
"user" : '',
"verbose" : 1,
"sleeptime" : '',
"runtime" : time.strftime("%Y-%m-%d %H:%M:%S"),
"priority" : 0,
"runtime_limit" : None,
"profile" : [],
"post-process": [],
"sequence-id": None,
"stop_queue_on_error": False,
"fixed_time": False,
"host": '',
}
to_be_submitted = True
if len(sys.argv) == 2 and sys.argv[1].isdigit():
_TASK_PARAMS['task_id'] = int(sys.argv[1])
argv = task_get_options(_TASK_PARAMS['task_id'], _TASK_PARAMS['task_name'])
to_be_submitted = False
else:
argv = sys.argv
setup_loggers(_TASK_PARAMS.get('task_id'))
task_name = os.path.basename(sys.argv[0])
if task_name not in CFG_BIBTASK_VALID_TASKS or os.path.realpath(os.path.join(CFG_BINDIR, task_name)) != os.path.realpath(sys.argv[0]):
raise OSError("%s is not in the allowed modules" % sys.argv[0])
from invenio.base.factory import configure_warnings
configure_warnings()
if type(argv) is dict:
# FIXME: REMOVE AFTER MAJOR RELEASE 1.0
# This is needed for old task submitted before CLI parameters
# where stored in DB and _OPTIONS dictionary was stored instead.
_OPTIONS = argv
else:
_OPTIONS = {}
if task_name in CFG_BIBTASK_DEFAULT_TASK_SETTINGS:
_OPTIONS.update(CFG_BIBTASK_DEFAULT_TASK_SETTINGS[task_name])
try:
params = _task_build_params(_TASK_PARAMS['task_name'],
argv, specific_params,
task_submit_elaborate_specific_parameter_fnc,
task_submit_check_options_fnc)
except InvalidParams, e:
if e.err:
err_msg = str(e.err)
else:
err_msg = ""
_usage(1, err_msg,
help_specific_usage=help_specific_usage,
description=description)
except (SystemExit, Exception) as err:
if not to_be_submitted:
register_exception(alert_admin=True)
write_message("Error in parsing the parameters: %s." % err, sys.stderr)
write_message("Exiting.", sys.stderr)
task_update_status("ERROR")
raise
_TASK_PARAMS.update(params)
write_message('argv=%s' % (argv, ), verbose=9)
write_message('_OPTIONS=%s' % (_OPTIONS, ), verbose=9)
write_message('_TASK_PARAMS=%s' % (_TASK_PARAMS, ), verbose=9)
if params.get('display_help'):
_usage(0, help_specific_usage=help_specific_usage, description=description)
if params.get('display_version'):
print(_TASK_PARAMS["version"])
sys.exit(0)
if to_be_submitted:
_task_submit(argv, authorization_action, authorization_msg)
else:
try:
try:
if task_get_task_param('profile'):
try:
from six import StringIO
import pstats
filename = os.path.join(CFG_TMPDIR, 'bibsched_task_%s.pyprof' % _TASK_PARAMS['task_id'])
existing_sorts = pstats.Stats.sort_arg_dict_default.keys()
required_sorts = []
profile_dump = []
for sort in task_get_task_param('profile'):
if sort not in existing_sorts:
sort = 'cumulative'
if sort not in required_sorts:
required_sorts.append(sort)
if sys.hexversion < 0x02050000:
import hotshot
import hotshot.stats
pr = hotshot.Profile(filename)
ret = pr.runcall(_task_run, task_run_fnc)
for sort_type in required_sorts:
tmp_out = sys.stdout
sys.stdout = StringIO()
hotshot.stats.load(filename).strip_dirs().sort_stats(sort_type).print_stats()
# pylint: disable=E1103
# This is a hack. sys.stdout is a StringIO in this case.
profile_dump.append(sys.stdout.getvalue())
# pylint: enable=E1103
sys.stdout = tmp_out
else:
import cProfile
pr = cProfile.Profile()
ret = pr.runcall(_task_run, task_run_fnc)
pr.dump_stats(filename)
for sort_type in required_sorts:
strstream = StringIO()
pstats.Stats(filename, stream=strstream).strip_dirs().sort_stats(sort_type).print_stats()
profile_dump.append(strstream.getvalue())
profile_dump = '\n'.join(profile_dump)
profile_dump += '\nYou can use profile=%s' % existing_sorts
open(os.path.join(CFG_BIBSCHED_LOGDIR, 'bibsched_task_%d.log' % _TASK_PARAMS['task_id']), 'a').write("%s" % profile_dump)
os.remove(filename)
except ImportError:
ret = _task_run(task_run_fnc)
write_message("ERROR: The Python Profiler is not installed!", stream=sys.stderr)
else:
ret = _task_run(task_run_fnc)
if not ret:
write_message("Error occurred. Exiting.", sys.stderr)
except Exception as e:
# We want to catch all exceptions here because:
# We set the task status to error and we want to display
# an error traceback
if isinstance(e, SystemExit) and e.code == 0:
raise
register_exception(alert_admin=True)
write_message("Unexpected error occurred: %s." % e, sys.stderr)
write_message("Traceback is:", sys.stderr)
write_messages(''.join(traceback.format_tb(sys.exc_info()[2])), sys.stderr)
write_message("Exiting.", sys.stderr)
if task_get_task_param('stop_queue_on_error'):
task_update_status("ERROR")
elif isinstance(e, RecoverableError) and task_get_task_param('email_logs_to'):
task_update_status("ERRORS REPORTED")
else:
task_update_status("CERROR")
finally:
_task_email_logs()
logging.shutdown()
def _task_build_params(task_name,
argv,
specific_params=("", []),
task_submit_elaborate_specific_parameter_fnc=None,
task_submit_check_options_fnc=None):
""" Build the BibTask params.
@param argv: a list of string as in sys.argv
@param description: is the generic description printed in the usage page;
@param help_specific_usage: is the specific parameter help
@param task_submit_elaborate_specific_parameter_fnc: will be called passing
a key and a value, for parsing specific cli parameters. Must return True if
it has recognized the parameter. Must eventually update the options with
bibtask_set_option;
@param task_submit_check_options: must check the validity of options (via
bibtask_get_option) once all the options where parsed;
"""
params = {}
# set user-defined options:
try:
(short_params, long_params) = specific_params
opts, args = getopt.gnu_getopt(argv[1:], "hVv:u:s:t:P:N:L:I:" +
short_params, [
"help",
"version",
"verbose=",
"user=",
"sleep=",
"runtime=",
"priority=",
"name=",
"limit=",
"profile=",
"post-process=",
"sequence-id=",
"stop-on-error",
"continue-on-error",
"fixed-time",
"email-logs-to=",
"host=",
] + long_params)
except getopt.GetoptError as err:
raise InvalidParams(err)
try:
for opt in opts:
if opt[0] in ("-h", "--help"):
params["display_help"] = True
break
elif opt[0] in ("-V", "--version"):
params["display_version"] = True
break
elif opt[0] in ("-u", "--user"):
params["user"] = opt[1]
elif opt[0] in ("-v", "--verbose"):
params["verbose"] = int(opt[1])
elif opt[0] in ("-s", "--sleeptime"):
if task_name not in CFG_TASK_IS_NOT_A_DEAMON:
get_datetime(opt[1]) # see if it is a valid shift
params["sleeptime"] = opt[1]
elif opt[0] in ("-t", "--runtime"):
params["runtime"] = get_datetime(opt[1])
elif opt[0] in ("-P", "--priority"):
params["priority"] = int(opt[1])
elif opt[0] in ("-N", "--name"):
params["task_specific_name"] = opt[1]
elif opt[0] in ("-L", "--limit"):
params["runtime_limit"] = parse_runtime_limit(opt[1])
elif opt[0] in ("--profile", ):
params.setdefault("profile", []).extend(opt[1].split(','))
elif opt[0] in ("--post-process", ):
params.setdefault("post-process", []).append(opt[1])
elif opt[0] in ("-I", "--sequence-id"):
params["sequence-id"] = opt[1]
elif opt[0] in ("--stop-on-error", ):
params["stop_queue_on_error"] = True
elif opt[0] in ("--continue-on-error", ):
params["stop_queue_on_error"] = False
elif opt[0] in ("--fixed-time", ):
params["fixed_time"] = True
elif opt[0] in ("--email-logs-to",):
params["email_logs_to"] = opt[1].split(',')
elif opt[0] in ("--host",):
params["host"] = opt[1]
elif not callable(task_submit_elaborate_specific_parameter_fnc) or \
not task_submit_elaborate_specific_parameter_fnc(opt[0],
opt[1],
opts,
args):
raise InvalidParams()
except StandardError as e:
raise InvalidParams(e)
if callable(task_submit_check_options_fnc):
if not task_submit_check_options_fnc():
raise InvalidParams()
return params
def task_set_option(key, value):
"""Set an value to key in the option dictionary of the task"""
global _OPTIONS
try:
_OPTIONS[key] = value
except NameError:
_OPTIONS = {key : value}
def task_get_option(key, default=None):
"""Returns the value corresponding to key in the option dictionary of the task"""
try:
if key is None:
return _OPTIONS
else:
return _OPTIONS.get(key, default)
except NameError:
return default
def task_has_option(key):
"""Map the has_key query to _OPTIONS"""
try:
return key in _OPTIONS
except NameError:
return False
def task_get_task_param(key, default=None):
"""Returns the value corresponding to the particular task param"""
try:
if key is None:
return _TASK_PARAMS
else:
return _TASK_PARAMS.get(key, default)
except NameError:
return default
def task_set_task_param(key, value):
"""Set the value corresponding to the particular task param"""
global _TASK_PARAMS
try:
_TASK_PARAMS[key] = value
except NameError:
_TASK_PARAMS = {key : value}
def task_update_progress(msg):
"""Updates progress information in the BibSched task table."""
write_message("Updating task progress to %s." % msg, verbose=9)
if "task_id" in _TASK_PARAMS:
return run_sql("UPDATE schTASK SET progress=%s where id=%s",
(msg[:255], _TASK_PARAMS["task_id"]))
def task_update_status(val):
"""Updates status information in the BibSched task table."""
write_message("Updating task status to %s." % val, verbose=9)
if "task_id" in _TASK_PARAMS:
return run_sql("UPDATE schTASK SET status=%s where id=%s",
(val, _TASK_PARAMS["task_id"]))
def task_read_status():
"""Read status information in the BibSched task table."""
res = run_sql("SELECT status FROM schTASK where id=%s",
(_TASK_PARAMS['task_id'],), 1)
try:
out = res[0][0]
except IndexError:
out = 'UNKNOWN'
return out
def write_messages(msgs, stream=None, verbose=1):
"""Write many messages through write_message"""
if stream is None:
stream = sys.stdout
for msg in msgs.split('\n'):
write_message(msg, stream, verbose)
def write_message(msg, stream=None, verbose=1):
"""Write message and flush output stream (may be sys.stdout or sys.stderr).
Useful for debugging stuff.
@note: msg could be a callable with no parameters. In this case it is
been called in order to obtain the string to be printed.
"""
if stream is None:
stream = sys.stdout
if msg and _TASK_PARAMS['verbose'] >= verbose:
if callable(msg):
msg = msg()
if stream == sys.stdout:
logging.info(msg)
elif stream == sys.stderr:
logging.error(msg)
else:
sys.stderr.write("Unknown stream %s. [must be sys.stdout or sys.stderr]\n" % stream)
else:
logging.debug(msg)
if CFG_BIBSCHED_FLUSH_LOGS:
for handler in logging.root.handlers:
handler.flush()
_RE_SHIFT = re.compile(r"([-\+]{0,1})([\d]+)([dhms])")
def get_datetime(var, format_string="%Y-%m-%d %H:%M:%S", now=None):
"""Returns a date string according to the format string.
It can handle normal date strings and shifts with respect
to now."""
date = now or datetime.datetime.now()
factors = {"d": 24 * 3600, "h": 3600, "m": 60, "s": 1}
m = _RE_SHIFT.match(var)
if m:
sign = m.groups()[0] == "-" and -1 or 1
factor = factors[m.groups()[2]]
value = float(m.groups()[1])
delta = sign * factor * value
while delta > 0 and date < datetime.datetime.now():
date = date + datetime.timedelta(seconds=delta)
date = date.strftime(format_string)
else:
date = time.strptime(var, format_string)
date = time.strftime(format_string, date)
return date
def task_sleep_now_if_required(can_stop_too=False):
"""This function should be called during safe state of BibTask,
e.g. after flushing caches or outside of run_sql calls.
"""
status = task_read_status()
write_message('Entering task_sleep_now_if_required with status=%s' % status, verbose=9)
if status == 'ABOUT TO SLEEP':
write_message("sleeping...")
task_update_status("SLEEPING")
signal.signal(signal.SIGTSTP, cb_task_sig_dumb)
os.kill(os.getpid(), signal.SIGSTOP)
time.sleep(1)
if task_read_status() == 'NOW STOP':
if can_stop_too:
write_message("stopped")
task_update_status("STOPPED")
sys.exit(0)
else:
write_message("stopping as soon as possible...")
task_update_status('ABOUT TO STOP')
else:
write_message("... continuing...")
task_update_status("CONTINUING")
signal.signal(signal.SIGTSTP, cb_task_sig_sleep)
elif status == 'ABOUT TO STOP':
if can_stop_too:
write_message("stopped")
task_update_status("STOPPED")
sys.exit(0)
if can_stop_too:
runtime_limit = task_get_option("limit")
if runtime_limit is not None:
if not (runtime_limit[0] <= datetime.datetime.now() <= runtime_limit[1]):
write_message("stopped (outside runtime limit)")
task_update_status("STOPPED")
sys.exit(0)
def get_modified_records_since(modification_date):
"""
Return the set of modified record since the given
modification_date.
@param modification_date: Return records modified after this date
@type modification_date datetime
"""
results = run_sql("SELECT id FROM bibrec WHERE modification_date >= %s",
(modification_date,))
return intbitset(results)
def authenticate(user, authorization_action, authorization_msg=""):
"""Authenticate the user against the user database.
Check for its password, if it exists.
Check for authorization_action access rights.
Return user name upon authorization success,
do system exit upon authorization failure.
"""
#FIXME
return user
# With SSO it's impossible to check for pwd
if CFG_EXTERNAL_AUTH_USING_SSO or os.path.basename(sys.argv[0]) in CFG_VALID_PROCESSES_NO_AUTH_NEEDED:
return user
if authorization_msg:
print(authorization_msg)
print("=" * len(authorization_msg))
if user == "":
print("\rUsername: ", end=' ', file=sys.stdout)
try:
user = sys.stdin.readline().lower().strip()
except EOFError:
sys.stderr.write("\n")
sys.exit(1)
except KeyboardInterrupt:
sys.stderr.write("\n")
sys.exit(1)
else:
print("\rUsername:", user, file=sys.stdout)
## first check user:
# p_un passed may be an email or a nickname:
res = run_sql("select id from user where email=%s", (user,), 1) + \
run_sql("select id from user where nickname=%s", (user,), 1)
if not res:
print("Sorry, %s does not exist." % user)
sys.exit(1)
else:
uid = res[0][0]
ok = False
login_method = get_user_preferences(uid)['login_method']
if not CFG_EXTERNAL_AUTHENTICATION[login_method]:
#Local authentication, let's see if we want passwords.
res = run_sql("select id from user where id=%s "
"and password=AES_ENCRYPT(email,'')",
(uid,), 1)
if res:
ok = True
if not ok:
try:
password_entered = getpass.getpass()
except EOFError:
sys.stderr.write("\n")
sys.exit(1)
except KeyboardInterrupt:
sys.stderr.write("\n")
sys.exit(1)
if not CFG_EXTERNAL_AUTHENTICATION[login_method]:
res = run_sql("select id from user where id=%s "
"and password=AES_ENCRYPT(email, %s)",
(uid, password_entered), 1)
if res:
ok = True
else:
if CFG_EXTERNAL_AUTHENTICATION[login_method].auth_user(get_email(uid), password_entered):
ok = True
if not ok:
print("Sorry, wrong credentials for %s." % user)
sys.exit(1)
else:
## secondly check authorization for the authorization_action:
(auth_code, auth_message) = acc_authorize_action(uid, authorization_action)
if auth_code != 0:
print(auth_message)
sys.exit(1)
return user
def _task_submit(argv, authorization_action, authorization_msg):
"""Submits task to the BibSched task queue. This is what people will
be invoking via command line."""
## check as whom we want to submit?
check_running_process_user()
## sanity check: remove eventual "task" option:
## authenticate user:
_TASK_PARAMS['user'] = authenticate(_TASK_PARAMS["user"], authorization_action, authorization_msg)
## submit task:
if _TASK_PARAMS['task_specific_name']:
task_name = '%s:%s' % (_TASK_PARAMS['task_name'], _TASK_PARAMS['task_specific_name'])
else:
task_name = _TASK_PARAMS['task_name']
write_message("storing task options %s\n" % argv, verbose=9)
verbose_argv = 'Will execute: %s' % ' '.join([escape_shell_arg(str(arg)) for arg in argv])
_TASK_PARAMS['task_id'] = run_sql("""INSERT INTO schTASK (proc,user,
runtime,sleeptime,status,progress,arguments,priority,sequenceid,host)
VALUES (%s,%s,%s,%s,'WAITING',%s,%s,%s,%s,%s)""",
(task_name, _TASK_PARAMS['user'], _TASK_PARAMS["runtime"],
_TASK_PARAMS["sleeptime"], verbose_argv[:255], marshal.dumps(argv),
_TASK_PARAMS['priority'], _TASK_PARAMS['sequence-id'],
_TASK_PARAMS['host']))
## update task number:
write_message("Task #%d submitted." % _TASK_PARAMS['task_id'])
return _TASK_PARAMS['task_id']
def task_get_options(task_id, task_name):
"""Returns options for the task 'id' read from the BibSched task
queue table."""
out = {}
res = run_sql("SELECT arguments FROM schTASK WHERE id=%s AND proc LIKE %s",
(task_id, task_name+'%'))
try:
out = marshal.loads(res[0][0])
except ValueError:
write_message("Error: %s task %d does not seem to exist."
% (task_name, task_id), sys.stderr)
task_update_status('ERROR')
sys.exit(1)
write_message('Options retrieved: %s' % (out, ), verbose=9)
return out
def _task_email_logs():
"""
In case this was requested, emails the logs.
"""
email_logs_to = task_get_task_param('email_logs_to')
if not email_logs_to:
return
status = task_read_status()
task_name = task_get_task_param('task_name')
task_specific_name = task_get_task_param('task_specific_name')
if task_specific_name:
task_name += '%s:%s' % (task_name, task_specific_name)
runtime = task_get_task_param('runtime')
title = "Execution of %s: %s" % (task_name, status)
body = """
Attached you can find the stdout and stderr logs of the execution of
name: %s
id: %s
runtime: %s
options: %s
status: %s
""" % (task_name, _TASK_PARAMS['task_id'], runtime, _OPTIONS, status)
log_file = task_log_path(_TASK_PARAMS['task_id'], 'log')
err_file = task_log_path(_TASK_PARAMS['task_id'], 'err')
return send_email(CFG_SITE_SUPPORT_EMAIL, email_logs_to, title, body,
attachments=[(log_file, 'text/plain'), (err_file, 'text/plain')])
def get_task_old_runtime(task_params):
"""Fetch from the database the last time this task ran
Here we check if the task can shift away or has to be run at a fixed time.
"""
if task_params['fixed_time'] or \
task_params['task_name'] in CFG_BIBTASK_FIXEDTIMETASKS:
sql = "SELECT runtime FROM schTASK WHERE id=%s"
old_runtime = run_sql(sql, (task_params['task_id'], ))[0][0]
else:
old_runtime = None
return old_runtime
def get_task_new_runtime(task_params):
"""Compute the next time this task should run"""
return get_datetime(task_params['sleeptime'],
now=get_task_old_runtime(task_params))
def _task_run(task_run_fnc):
"""Runs the task by fetching arguments from the BibSched task queue.
This is what BibSched will be invoking via daemon call.
The task prints Fibonacci numbers for up to NUM on the stdout, and some
messages on stderr.
@param task_run_fnc: will be called as the main core function. Must return
False in case of errors.
Return True in case of success and False in case of failure."""
from invenio.legacy.bibsched.bibtasklet import _TASKLETS
## We prepare the pid file inside /prefix/var/run/taskname_id.pid
check_running_process_user()
try:
CFG_BIBTASK_RUN_DIR = os.path.join(current_app.instance_path, 'run')
if not os.path.exists(CFG_BIBTASK_RUN_DIR):
os.mkdir(CFG_BIBTASK_RUN_DIR)
pidfile_name = os.path.join(CFG_BIBTASK_RUN_DIR,
'bibsched_task_%d.pid' % _TASK_PARAMS['task_id'])
pidfile = open(pidfile_name, 'w')
pidfile.write(str(os.getpid()))
pidfile.close()
except OSError:
register_exception(alert_admin=True)
task_update_status("ERROR")
return False
## check task status:
task_status = task_read_status()
if task_status not in ("WAITING", "SCHEDULED"):
write_message("Error: The task #%d is %s. I expected WAITING or SCHEDULED." %
(_TASK_PARAMS['task_id'], task_status), sys.stderr)
return False
time_now = datetime.datetime.now()
if _TASK_PARAMS['runtime_limit'] is not None:
if not _TASK_PARAMS['runtime_limit'][0][0] <= time_now <= _TASK_PARAMS['runtime_limit'][0][1]:
if task_get_option('fixed_time'):
new_runtime = get_task_new_runtime(_TASK_PARAMS)
elif time_now <= _TASK_PARAMS['runtime_limit'][0][0]:
new_runtime = _TASK_PARAMS['runtime_limit'][0][0].strftime("%Y-%m-%d %H:%M:%S")
else:
new_runtime = _TASK_PARAMS['runtime_limit'][1][0].strftime("%Y-%m-%d %H:%M:%S")
progress = run_sql("SELECT progress FROM schTASK WHERE id=%s", (_TASK_PARAMS['task_id'], ))
if progress:
progress = progress[0][0]
else:
progress = ''
g = re.match(r'Postponed (\d+) time\(s\)', progress)
if g:
postponed_times = int(g.group(1))
else:
postponed_times = 0
if _TASK_PARAMS['sequence-id']:
## Also postponing other dependent tasks.
run_sql("UPDATE schTASK SET runtime=%s, progress=%s WHERE sequenceid=%s AND status='WAITING'", (new_runtime, 'Postponed as task %s' % _TASK_PARAMS['task_id'], _TASK_PARAMS['sequence-id'])) # kwalitee: disable=sql
run_sql("UPDATE schTASK SET runtime=%s, status='WAITING', progress=%s, host='' WHERE id=%s", (new_runtime, 'Postponed %d time(s)' % (postponed_times + 1), _TASK_PARAMS['task_id'])) # kwalitee: disable=sql
write_message("Task #%d postponed because outside of runtime limit" % _TASK_PARAMS['task_id'])
return True
# Make sure the host field is updated
# It will not be updated properly when we run
# a task from the cli (without using the bibsched monitor)
host = bibsched_get_host(_TASK_PARAMS['task_id'])
if host and host != gethostname():
write_message("Error: The task #%d is bound to %s." %
(_TASK_PARAMS['task_id'], host), sys.stderr)
return False
else:
bibsched_set_host(_TASK_PARAMS['task_id'], gethostname())
## initialize signal handler:
signal.signal(signal.SIGUSR2, cb_task_sig_debug)
signal.signal(signal.SIGTSTP, cb_task_sig_sleep)
signal.signal(signal.SIGTERM, cb_task_sig_stop)
signal.signal(signal.SIGQUIT, cb_task_sig_stop)
signal.signal(signal.SIGABRT, cb_task_sig_suicide)
signal.signal(signal.SIGINT, cb_task_sig_stop)
## we can run the task now:
write_message("Task #%d started." % _TASK_PARAMS['task_id'])
task_update_status("RUNNING")
## run the task:
_TASK_PARAMS['task_starting_time'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
sleeptime = _TASK_PARAMS['sleeptime']
try:
if callable(task_run_fnc) and task_run_fnc():
task_update_status("DONE")
elif task_get_task_param('email_logs_to'):
task_update_status("ERRORS REPORTED")
else:
task_update_status("DONE WITH ERRORS")
finally:
task_status = task_read_status()
if sleeptime:
argv = task_get_options(_TASK_PARAMS['task_id'], _TASK_PARAMS['task_name'])
verbose_argv = 'Will execute: %s' % ' '.join([escape_shell_arg(str(arg)) for arg in argv])
new_runtime = get_task_new_runtime(_TASK_PARAMS)
## The task is a daemon. We resubmit it
if task_status == 'DONE':
## It has finished in a good way. We recycle the database row
run_sql("UPDATE schTASK SET runtime=%s, status='WAITING', progress=%s, host=%s WHERE id=%s", (new_runtime, verbose_argv, _TASK_PARAMS['host'], _TASK_PARAMS['task_id']))
write_message("Task #%d finished and resubmitted." % _TASK_PARAMS['task_id'])
elif task_status == 'STOPPED':
run_sql("UPDATE schTASK SET status='WAITING', progress=%s, host='' WHERE id=%s", (verbose_argv, _TASK_PARAMS['task_id'], ))
write_message("Task #%d stopped and resubmitted." % _TASK_PARAMS['task_id'])
else:
## We keep the bad result and we resubmit with another id.
#res = run_sql('SELECT proc,user,sleeptime,arguments,priority FROM schTASK WHERE id=%s', (_TASK_PARAMS['task_id'], ))
#proc, user, sleeptime, arguments, priority = res[0]
#run_sql("""INSERT INTO schTASK (proc,user,
#runtime,sleeptime,status,arguments,priority)
#VALUES (%s,%s,%s,%s,'WAITING',%s, %s)""",
#(proc, user, new_runtime, sleeptime, arguments, priority))
write_message("Task #%d finished but not resubmitted. [%s]" % (_TASK_PARAMS['task_id'], task_status))
else:
## we are done:
write_message("Task #%d finished. [%s]" % (_TASK_PARAMS['task_id'], task_status))
## Removing the pid
os.remove(pidfile_name)
#Lets call the post-process tasklets
if task_get_task_param("post-process"):
split = re.compile(r"(bst_.*)\[(.*)\]")
for tasklet in task_get_task_param("post-process"):
if not split.match(tasklet): # wrong syntax
_usage(1, "There is an error in the post processing option "
"for this task.")
aux_tasklet = split.match(tasklet)
_TASKLETS[aux_tasklet.group(1)](**eval("dict(%s)" % (aux_tasklet.group(2))))
return True
def _usage(exitcode=1, msg="", help_specific_usage="", description=""):
"""Prints usage info."""
if msg:
sys.stderr.write("Error: %s.\n" % msg)
sys.stderr.write("Usage: %s [options]\n" % sys.argv[0])
if help_specific_usage:
sys.stderr.write("Command options:\n")
sys.stderr.write(help_specific_usage)
sys.stderr.write(" Scheduling options:\n")
sys.stderr.write(" -u, --user=USER\tUser name under which to submit this"
" task.\n")
sys.stderr.write(" -t, --runtime=TIME\tTime to execute the task. [default=now]\n"
"\t\t\tExamples: +15s, 5m, 3h, 2002-10-27 13:57:26.\n")
sys.stderr.write(" -s, --sleeptime=SLEEP\tSleeping frequency after"
" which to repeat the task.\n"
"\t\t\tExamples: 30m, 2h, 1d. [default=no]\n")
sys.stderr.write(" --fixed-time\t\tAvoid drifting of execution time when using --sleeptime\n")
sys.stderr.write(" -I, --sequence-id=SEQUENCE-ID\tSequence Id of the current process\n")
sys.stderr.write(" -L --limit=LIMIT\tTime limit when it is"
" allowed to execute the task.\n"
"\t\t\tExamples: 22:00-03:00, Sunday 01:00-05:00.\n"
"\t\t\tSyntax: [Wee[kday]] [hh[:mm][-hh[:mm]]].\n")
sys.stderr.write(" -P, --priority=PRI\tTask priority (0=default, 1=higher, etc).\n")
sys.stderr.write(" -N, --name=NAME\tTask specific name (advanced option).\n\n")
sys.stderr.write(" General options:\n")
sys.stderr.write(" -h, --help\t\tPrint this help.\n")
sys.stderr.write(" -V, --version\t\tPrint version information.\n")
sys.stderr.write(" -v, --verbose=LEVEL\tVerbose level (0=min,"
" 1=default, 9=max).\n")
sys.stderr.write(" --profile=STATS\tPrint profile information. STATS is a comma-separated\n\t\t\tlist of desired output stats (calls, cumulative,\n\t\t\tfile, line, module, name, nfl, pcalls, stdname, time).\n")
sys.stderr.write(" --stop-on-error\tIn case of unrecoverable error stop the bibsched queue.\n")
sys.stderr.write(" --continue-on-error\tIn case of unrecoverable error don't stop the bibsched queue.\n")
sys.stderr.write(" --post-process=BIB_TASKLET_NAME[parameters]\tPostprocesses the specified\n\t\t\tbibtasklet with the given parameters between square\n\t\t\tbrackets.\n")
sys.stderr.write("\t\t\tExample:--post-process \"bst_send_email[fromaddr=\n\t\t\t'foo@xxx.com', toaddr='bar@xxx.com', subject='hello',\n\t\t\tcontent='help']\"\n")
sys.stderr.write(" --email-logs-to=EMAILS Sends an email with the results of the execution\n\t\t\tof the task, and attached the logs (EMAILS could be a comma-\n\t\t\tseparated lists of email addresses)\n")
sys.stderr.write(" --host=HOSTNAME Bind the task to the specified host, it will only ever run on that host.\n")
if description:
sys.stderr.write(description)
sys.exit(exitcode)
def cb_task_sig_sleep(sig, frame):
"""Signal handler for the 'sleep' signal sent by BibSched."""
signal.signal(signal.SIGTSTP, signal.SIG_IGN)
write_message("task_sig_sleep(), got signal %s frame %s"
% (sig, frame), verbose=9)
write_message("sleeping as soon as possible...")
_db_login(relogin=1)
task_update_status("ABOUT TO SLEEP")
def cb_task_sig_stop(sig, frame):
"""Signal handler for the 'stop' signal sent by BibSched."""
write_message("task_sig_stop(), got signal %s frame %s"
% (sig, frame), verbose=9)
write_message("stopping as soon as possible...")
_db_login(relogin=1) # To avoid concurrency with an interrupted run_sql call
task_update_status("ABOUT TO STOP")
def cb_task_sig_suicide(sig, frame):
"""Signal handler for the 'suicide' signal sent by BibSched."""
write_message("task_sig_suicide(), got signal %s frame %s"
% (sig, frame), verbose=9)
write_message("suiciding myself now...")
task_update_status("SUICIDING")
write_message("suicided")
_db_login(relogin=1)
task_update_status("SUICIDED")
sys.exit(1)
def cb_task_sig_debug(sig, frame):
"""Signal handler for the 'debug' signal sent by BibSched.
This spawn a remote console server we can connect to to check
the task behavior at runtime."""
write_message("task_sig_debug(), got signal %s frame %s"
% (sig, frame), verbose=9)
from rfoo.utils import rconsole
rconsole.spawn_server()
def cb_task_sig_dumb(sig, frame):
"""Dumb signal handler."""
pass
_RE_PSLINE = re.compile(r'^\s*(\w+)\s+(\w+)')
def guess_apache_process_user_from_ps():
"""Guess Apache process user by parsing the list of running processes."""
apache_users = []
try:
# Tested on Linux, Sun and MacOS X
for line in os.popen('ps -A -o user,comm').readlines():
g = _RE_PSLINE.match(line)
if g:
username = g.group(1)
process = os.path.basename(g.group(2))
if process in ('apache', 'apache2', 'httpd'):
if username not in apache_users and username != 'root':
apache_users.append(username)
except Exception as e:
print("WARNING: %s" % e, file=sys.stderr)
return tuple(apache_users)
def guess_apache_process_user():
"""
Return the possible name of the user running the Apache server process.
(Look at running OS processes or look at OS users defined in /etc/passwd.)
"""
apache_users = guess_apache_process_user_from_ps() + ('apache2', 'apache', 'www-data')
for username in apache_users:
try:
userline = pwd.getpwnam(username)
return userline[0]
except KeyError:
pass
print("ERROR: Cannot detect Apache server process user. Please set the correct value in CFG_BIBSCHED_PROCESS_USER.", file=sys.stderr)
sys.exit(1)
def check_running_process_user():
"""
Check that the user running this program is the same as the user
configured in CFG_BIBSCHED_PROCESS_USER or as the user running the
Apache webserver process.
"""
running_as_user = pwd.getpwuid(os.getuid())[0]
if CFG_BIBSCHED_PROCESS_USER:
# We have the expected bibsched process user defined in config,
# so check against her, not against Apache.
if running_as_user != CFG_BIBSCHED_PROCESS_USER:
print("""ERROR: You must run "%(x_proc)s" as the user set up in your
CFG_BIBSCHED_PROCESS_USER (seems to be "%(x_user)s").
You may want to do "sudo -u %(x_user)s %(x_proc)s ..." to do so.
If you think this is not right, please set CFG_BIBSCHED_PROCESS_USER
appropriately and rerun "inveniocfg --update-config-py".""" % \
{'x_proc': os.path.basename(sys.argv[0]), 'x_user': CFG_BIBSCHED_PROCESS_USER}, file=sys.stderr)
sys.exit(1)
elif running_as_user != guess_apache_process_user(): # not defined in config, check against Apache
print("""ERROR: You must run "%(x_proc)s" as the same user that runs your Apache server
process (seems to be "%(x_user)s").
You may want to do "sudo -u %(x_user)s %(x_proc)s ..." to do so.
If you think this is not right, please set CFG_BIBSCHED_PROCESS_USER
appropriately and rerun "inveniocfg --update-config-py".""" % \
{'x_proc': os.path.basename(sys.argv[0]), 'x_user': guess_apache_process_user()}, file=sys.stderr)
sys.exit(1)
return
diff --git a/invenio/legacy/bibsched/cli.py b/invenio/legacy/bibsched/cli.py
index ec211afc6..8f1829993 100644
--- a/invenio/legacy/bibsched/cli.py
+++ b/invenio/legacy/bibsched/cli.py
@@ -1,1328 +1,1331 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
"""BibSched - task management, scheduling and executing system for Invenio
"""
import os
import sys
import time
import re
import datetime
import marshal
import getopt
from itertools import chain
from socket import gethostname
from subprocess import Popen
import signal
from invenio.legacy.bibsched.bibtask_config import \
CFG_BIBTASK_VALID_TASKS, \
CFG_BIBTASK_MONOTASKS, \
CFG_BIBTASK_FIXEDTIMETASKS
from invenio.config import \
CFG_PREFIX, \
CFG_TMPSHAREDDIR, \
CFG_BIBSCHED_REFRESHTIME, \
CFG_BINDIR, \
CFG_LOGDIR, \
CFG_BIBSCHED_GC_TASKS_OLDER_THAN, \
CFG_BIBSCHED_GC_TASKS_TO_REMOVE, \
CFG_BIBSCHED_GC_TASKS_TO_ARCHIVE, \
CFG_BIBSCHED_MAX_NUMBER_CONCURRENT_TASKS, \
CFG_SITE_URL, \
CFG_BIBSCHED_NODE_TASKS, \
CFG_INSPIRE_SITE, \
CFG_BIBSCHED_INCOMPATIBLE_TASKS, \
CFG_BIBSCHED_NON_CONCURRENT_TASKS, \
CFG_VERSION, \
CFG_BIBSCHED_NEVER_STOPS
from invenio.legacy.dbquery import run_sql, real_escape_string
from invenio.ext.logging import register_exception
from invenio.utils.shell import run_shell_command
CFG_VALID_STATUS = ('WAITING', 'SCHEDULED', 'RUNNING', 'CONTINUING',
'% DELETED', 'ABOUT TO STOP', 'ABOUT TO SLEEP', 'STOPPED',
'SLEEPING', 'KILLED', 'NOW STOP', 'ERRORS REPORTED')
CFG_MOTD_PATH = os.path.join(CFG_TMPSHAREDDIR, "bibsched.motd")
ACTIVE_STATUS = ('SCHEDULED', 'ABOUT TO SLEEP', 'ABOUT TO STOP',
'CONTINUING', 'RUNNING')
SHIFT_RE = re.compile(r"([-\+]{0,1})([\d]+)([dhms])")
def register_emergency(msg, recipients=None):
"""Launch an emergency. This means to send email messages to each
address in 'recipients'. By default recipients will be obtained via
get_emergency_recipients() which loads settings from
CFG_SITE_EMERGENCY_EMAIL_ADDRESSES
"""
from invenio.base.globals import cfg
from invenio.ext.email import send_email
if not recipients:
recipients = get_emergency_recipients()
recipients = set(recipients)
recipients.add(cfg['CFG_SITE_ADMIN_EMAIL'])
for address_str in recipients:
send_email(
cfg['CFG_SITE_SUPPORT_EMAIL'],
address_str,
"Emergency notification",
msg
)
def get_emergency_recipients(recipient_cfg=None):
"""
Parse a list of appropriate emergency email recipients from
CFG_SITE_EMERGENCY_EMAIL_ADDRESSES, or from a provided dictionary
comprised of 'time constraint' => 'comma separated list of addresses'
CFG_SITE_EMERGENCY_EMAIL_ADDRESSES format example:
CFG_SITE_EMERGENCY_EMAIL_ADDRESSES = {
'Sunday 22:00-06:00': '0041761111111@email2sms.foo.com',
'06:00-18:00': 'team-in-europe@foo.com,0041762222222@email2sms.foo.com',
'18:00-06:00': 'team-in-usa@foo.com',
'*': 'john.doe.phone@foo.com'}
"""
from invenio.base.globals import cfg
from invenio.utils.date import parse_runtime_limit
if recipient_cfg is None:
recipient_cfg = cfg['CFG_SITE_EMERGENCY_EMAIL_ADDRESSES']
recipients = set()
for time_condition, address_str in recipient_cfg.items():
if time_condition and time_condition is not '*':
(current_range, future_range) = parse_runtime_limit(time_condition)
if not current_range[0] \
<= datetime.datetime.now() <= current_range[1]:
continue
recipients.update([address_str])
return list(recipients)
def get_pager():
"""
Return the first available pager.
"""
paths = (
os.environ.get('PAGER', ''),
CFG_BIBSCHED_LOG_PAGER,
'/usr/bin/less',
'/bin/more'
)
for pager in paths:
if os.path.exists(pager):
return pager
def get_editor():
"""
Return the first available editor.
"""
paths = (
os.environ.get('EDITOR', ''),
CFG_BIBSCHED_EDITOR,
'/usr/bin/vim',
'/usr/bin/emacs',
'/usr/bin/vi',
'/usr/bin/nano',
)
for editor in paths:
if os.path.exists(editor):
return editor
class RecoverableError(StandardError):
pass
def get_datetime(var, format_string="%Y-%m-%d %H:%M:%S"):
"""Returns a date string according to the format string.
It can handle normal date strings and shifts with respect
to now."""
try:
date = time.time()
factors = {"d": 24*3600, "h": 3600, "m": 60, "s": 1}
m = SHIFT_RE.match(var)
if m:
sign = m.groups()[0] == "-" and -1 or 1
factor = factors[m.groups()[2]]
value = float(m.groups()[1])
date = time.localtime(date + sign * factor * value)
date = time.strftime(format_string, date)
else:
date = time.strptime(var, format_string)
date = time.strftime(format_string, date)
return date
except ValueError:
return None
def get_my_pid(process, args=''):
if sys.platform.startswith('freebsd'):
command = "ps -o pid,args | grep '%s %s' | grep -v 'grep' | sed -n 1p" % (process, args)
else:
command = "ps -C %s o '%%p%%a' | grep '%s %s' | grep -v 'grep' | sed -n 1p" % (process, process, args)
answer = run_shell_command(command)[1].strip()
if answer == '':
answer = 0
else:
answer = answer[:answer.find(' ')]
return int(answer)
def get_task_pid(task_id):
"""Return the pid of task_name/task_id"""
try:
path = os.path.join(CFG_PREFIX, 'var', 'run', 'bibsched_task_%d.pid' % task_id)
pid = int(open(path).read())
os.kill(pid, 0)
return pid
except (OSError, IOError):
return None
def get_last_taskid():
"""Return the last taskid used."""
return run_sql("SELECT MAX(id) FROM schTASK")[0][0]
def delete_task(task_id):
"""Delete the corresponding task."""
run_sql("DELETE FROM schTASK WHERE id=%s", (task_id, ))
def is_task_scheduled(task_name):
"""Check if a certain task_name is due for execution (WAITING or RUNNING)"""
sql = """SELECT COUNT(proc) FROM schTASK
WHERE proc = %s AND (status='WAITING' OR status='RUNNING')"""
return run_sql(sql, (task_name,))[0][0] > 0
def get_task_ids_by_descending_date(task_name, statuses=['SCHEDULED']):
"""Returns list of task ids, ordered by descending runtime."""
sql = """SELECT id FROM schTASK
WHERE proc=%s AND (%s)
ORDER BY runtime DESC""" \
% " OR ".join(["status = '%s'" % x for x in statuses])
return [x[0] for x in run_sql(sql, (task_name,))]
def get_task_options(task_id):
"""Returns options for task_id read from the BibSched task queue table."""
res = run_sql("SELECT arguments FROM schTASK WHERE id=%s", (task_id,))
try:
return marshal.loads(res[0][0])
except IndexError:
return list()
def gc_tasks(verbose=False, statuses=None, since=None, tasks=None): # pylint: disable=W0613
"""Garbage collect the task queue."""
if tasks is None:
tasks = CFG_BIBSCHED_GC_TASKS_TO_REMOVE + CFG_BIBSCHED_GC_TASKS_TO_ARCHIVE
if since is None:
since = '-%id' % CFG_BIBSCHED_GC_TASKS_OLDER_THAN
if statuses is None:
statuses = ['DONE']
statuses = [status.upper() for status in statuses if status.upper() != 'RUNNING']
date = get_datetime(since)
status_query = 'status in (%s)' % ','.join([repr(real_escape_string(status)) for status in statuses])
for task in tasks:
if task in CFG_BIBSCHED_GC_TASKS_TO_REMOVE:
res = run_sql("""DELETE FROM schTASK WHERE proc=%%s AND %s AND
runtime<%%s""" % status_query, (task, date))
write_message('Deleted %s %s tasks (created before %s) with %s'
% (res, task, date, status_query))
elif task in CFG_BIBSCHED_GC_TASKS_TO_ARCHIVE:
run_sql("""INSERT INTO hstTASK(id,proc,host,user,
runtime,sleeptime,arguments,status,progress)
SELECT id,proc,host,user,
runtime,sleeptime,arguments,status,progress
FROM schTASK WHERE proc=%%s AND %s AND
runtime<%%s""" % status_query, (task, date))
res = run_sql("""DELETE FROM schTASK WHERE proc=%%s AND %s AND
runtime<%%s""" % status_query, (task, date))
write_message('Archived %s %s tasks (created before %s) with %s'
% (res, task, date, status_query))
def spawn_task(command, wait=False):
"""
Spawn the provided command in a way that is detached from the current
group. In this way a signal received by bibsched is not going to be
automatically propagated to the spawned process.
"""
def preexec(): # Don't forward signals.
os.setsid()
devnull = open(os.devnull, "w")
process = Popen(command, preexec_fn=preexec, shell=True,
stderr=devnull, stdout=devnull)
if wait:
process.wait()
def bibsched_get_host(task_id):
"""Retrieve the hostname of the task"""
res = run_sql("SELECT host FROM schTASK WHERE id=%s LIMIT 1", (task_id, ), 1)
if res:
return res[0][0]
def bibsched_set_host(task_id, host=""):
"""Update the progress of task_id."""
return run_sql("UPDATE schTASK SET host=%s WHERE id=%s", (host, task_id))
def bibsched_get_status(task_id):
"""Retrieve the task status."""
res = run_sql("SELECT status FROM schTASK WHERE id=%s LIMIT 1", (task_id, ), 1)
if res:
return res[0][0]
def bibsched_set_status(task_id, status, when_status_is=None):
"""Update the status of task_id."""
if when_status_is is None:
return run_sql("UPDATE schTASK SET status=%s WHERE id=%s",
(status, task_id))
else:
return run_sql("UPDATE schTASK SET status=%s WHERE id=%s AND status=%s",
(status, task_id, when_status_is))
def bibsched_set_progress(task_id, progress):
"""Update the progress of task_id."""
return run_sql("UPDATE schTASK SET progress=%s WHERE id=%s", (progress, task_id))
def bibsched_set_priority(task_id, priority):
"""Update the priority of task_id."""
return run_sql("UPDATE schTASK SET priority=%s WHERE id=%s", (priority, task_id))
def bibsched_set_name(task_id, name):
"""Update the name of task_id."""
return run_sql("UPDATE schTASK SET proc=%s WHERE id=%s", (name, task_id))
def bibsched_set_sleeptime(task_id, sleeptime):
"""Update the sleeptime of task_id."""
return run_sql("UPDATE schTASK SET sleeptime=%s WHERE id=%s", (sleeptime, task_id))
def bibsched_set_runtime(task_id, runtime):
"""Update the sleeptime of task_id."""
return run_sql("UPDATE schTASK SET runtime=%s WHERE id=%s", (runtime, task_id))
def bibsched_send_signal(task_id, sig):
"""Send a signal to a given task."""
if bibsched_get_host(task_id) != gethostname():
return False
pid = get_task_pid(task_id)
if pid:
try:
os.kill(pid, sig)
return True
except OSError:
return False
return False
def is_monotask(proc):
#procname = proc.split(':')[0]
return proc in CFG_BIBTASK_MONOTASKS
def stop_task(task):
Log("Sending STOP signal to #%d (%s) which was in status %s" % (task.id, task.proc, task.status))
bibsched_set_status(task.id, 'ABOUT TO STOP', task.status)
def sleep_task(task):
Log("Sending SLEEP signal to #%d (%s) which was in status %s" % (task.id, task.proc, task.status))
bibsched_set_status(task.id, 'ABOUT TO SLEEP', task.status)
def fetch_debug_mode():
r = run_sql('SELECT value FROM schSTATUS WHERE name = "debug_mode"')
try:
debug_mode = bool(int(r[0][0]))
except (ValueError, IndexError):
# We insert the missing configuration variable in the DB
run_sql('INSERT INTO schSTATUS (name, value) VALUES ("debug_mode", "0")')
debug_mode = False
return debug_mode
class Task(object):
def __init__(self, task_id, proc, runtime, status, priority, host, sequenceid):
self.id = task_id
self.proc = proc
self.runtime = runtime
self.status = status
self.priority = priority
self.host = host
self.sequenceid = sequenceid
@staticmethod
def from_resultset(resultset):
return [Task(*row) for row in resultset]
def __eq__(self, other):
return self.id == other.id
def __lt__(self, other):
return self.id < other.id
def __unicode__(self):
msg = u"Task(id=%r, proc=%r, runtime=%r, status=%r, " \
u"priority=%r, host=%r, sequenceid=%r"
return msg % (self.id, self.proc, self.runtime, self.status,
self.priority, self.host, self.sequenceid)
def __str__(self):
return unicode(self).encode('utf-8')
def __repr__(self):
return unicode(self)
class BibSched(object):
def __init__(self, debug=False):
self.cycles_count = 0
self.debug = debug
self.hostname = gethostname()
self.helper_modules = CFG_BIBTASK_VALID_TASKS
## All the tasks in the queue that the node is allowed to manipulate
self.node_relevant_bibupload_tasks = ()
self.node_relevant_waiting_tasks = ()
self.node_sleeping_tasks = ()
self.node_active_tasks = ()
## All tasks of all nodes
self.sleeping_tasks_all_nodes = ()
self.waiting_tasks_all_nodes = ()
self.active_tasks_all_nodes = ()
self.mono_tasks_all_nodes = ()
self.allowed_task_types = CFG_BIBSCHED_NODE_TASKS.get(self.hostname, CFG_BIBTASK_VALID_TASKS)
def tie_task_to_host(self, task_id):
"""Sets the hostname of a task to the machine executing this script
@return: True if the scheduling was successful, False otherwise,
e.g. if the task was scheduled concurrently on a different host.
"""
r = run_sql("""UPDATE schTASK SET host=%s, status='SCHEDULED'
WHERE id=%s AND status='WAITING'""",
(self.hostname, task_id))
return bool(r)
def filter_for_allowed_tasks(self):
""" Removes all tasks that are not allowed in this Invenio instance
"""
def relevant_task(task):
procname = task.proc.split(':')[0]
if procname not in self.allowed_task_types:
return False
return True
def filter_tasks(tasks):
return tuple(t for t in tasks if relevant_task(t))
self.node_relevant_bibupload_tasks = filter_tasks(self.node_relevant_bibupload_tasks)
self.node_relevant_waiting_tasks = filter_tasks(self.waiting_tasks_all_nodes)
def is_task_compatible(self, task1, task2):
"""Return True when the two tasks can run concurrently or can run when
the other task is sleeping"""
procname1 = task1.proc.split(':')[0]
procname2 = task2.proc.split(':')[0]
for non_compatible_tasks in CFG_BIBSCHED_INCOMPATIBLE_TASKS:
if (task1.proc in non_compatible_tasks or procname1 in non_compatible_tasks) \
and (task2.proc in non_compatible_tasks or procname2 in non_compatible_tasks):
return False
if task1.proc == task2.proc == 'bibupload':
return True
return task1.proc != task2.proc
def is_task_non_concurrent(self, task1, task2):
for non_concurrent_tasks in CFG_BIBSCHED_NON_CONCURRENT_TASKS:
if (task1.proc.split(':')[0] in non_concurrent_tasks
or task1.proc in non_concurrent_tasks):
if (task2.proc.split(':')[0] in non_concurrent_tasks
or task2.proc in non_concurrent_tasks):
return True
return False
def get_tasks_to_sleep_and_stop(self, task, task_set):
"""Among the task_set, return the list of tasks to stop and the list
of tasks to sleep.
"""
def minimum_priority_task(task_set):
min_task = None
## For all the lower priority tasks...
for t in task_set:
if (min_task is None or t.priority < min_task.priority) \
and t.status != 'SLEEPING' and t.priority < task.priority \
and task.host == t.host:
# We don't put to sleep already sleeping task :-)
# And it only makes sense to free a spot on the local node
min_task = t
return min_task
to_stop = []
to_sleep = []
for t in task_set:
if not self.is_task_compatible(task, t):
to_stop.append(t)
if is_monotask(task.proc):
to_sleep = [t for t in task_set if t.status != 'SLEEPING']
else:
for t in task_set:
if t.status != 'SLEEPING' and self.is_task_non_concurrent(task, t):
to_sleep.append(t)
# Only needed if we are not freeing a spot already
# So to_stop and to_sleep should be empty
if not to_stop and not to_sleep and \
len(self.node_active_tasks) >= CFG_BIBSCHED_MAX_NUMBER_CONCURRENT_TASKS:
min_task = minimum_priority_task(task_set)
if min_task:
to_sleep = [min_task]
return to_stop, to_sleep
def split_active_tasks_by_priority(self, task):
"""Return two lists: the list of task_ids with lower priority and
those with higher or equal priority."""
higher = []
lower = []
for t in self.node_active_tasks:
if task.id == t.id:
continue
if t.priority < task.priority:
lower.append(t)
else:
higher.append(t)
return lower, higher
def handle_task(self, task):
"""Perform needed action of the row representing a task.
Return True when task_status need to be refreshed"""
debug = self.debug
Log(str(task), debug)
# If the task is active, we do scheduling stuff here
if task in self.node_active_tasks:
# For multi-node
# check if we need to sleep ourselves for monotasks
# to be able to run
for t in self.mono_tasks_all_nodes:
# 2 cases here
# If a monotask is running, we want to sleep
# If a monotask is waiting, we want to sleep if our priority
# is inferior
if task.priority < t.priority or t.status in ACTIVE_STATUS:
# Sleep ourselves
if task.status not in ('ABOUT TO STOP', 'ABOUT TO SLEEP', 'SCHEDULED'):
Log("Sleeping ourselves because of a monotask: %s" % t, debug)
sleep_task(task)
return True
else:
Log("A monotask is running but already going to sleep/stop", debug)
return False
# We try to run a task in waiting status here
elif task in self.node_relevant_waiting_tasks:
Log("Trying to run %r" % task, debug)
if task.priority < -10:
Log("Cannot run because priority < -10", debug)
return False
if task.host and task.host != self.hostname:
Log("Cannot run because this task is bound to a different machine", debug)
return False
lower, higher = self.split_active_tasks_by_priority(task)
Log('lower: %r' % lower, debug)
Log('higher: %r' % higher, debug)
for t in self.active_tasks_all_nodes:
if task.id != t.id and not self.is_task_compatible(task, t):
### !!! WE NEED TO CHECK FOR TASKS THAT CAN ONLY BE EXECUTED ON ONE MACHINE AT ONE TIME
### !!! FOR EXAMPLE BIBUPLOADS WHICH NEED TO BE EXECUTED SEQUENTIALLY AND NEVER CONCURRENTLY
## There's at least a higher priority task running that
## cannot run at the same time of the given task.
## We give up
Log("Cannot run because task_id: %s, proc: %s is in the queue and incompatible" % (t.id, t.proc), debug)
return False
if task.sequenceid:
## Let's normalize the prority of all tasks in a sequenceid to the
## max priority of the group
max_priority = run_sql("""SELECT MAX(priority) FROM schTASK
WHERE status IN ('WAITING', 'RUNNING',
'SLEEPING', 'ABOUT TO STOP',
'ABOUT TO SLEEP',
'SCHEDULED', 'CONTINUING')
AND sequenceid=%s""",
(task.sequenceid, ))[0][0]
if run_sql("""UPDATE schTASK SET priority=%s
WHERE status IN ('WAITING', 'RUNNING',
'SLEEPING', 'ABOUT TO STOP', 'ABOUT TO SLEEP',
'SCHEDULED', 'CONTINUING') AND sequenceid=%s""",
(max_priority, task.sequenceid)):
Log("Raised all waiting tasks with sequenceid "
"%s to the max priority %s" % (task.sequenceid, max_priority))
## Some priorities where raised
return True
## Let's normalize the runtime of all tasks in a sequenceid to
## the compatible runtime.
current_runtimes = run_sql("""SELECT id, runtime FROM schTASK WHERE sequenceid=%s AND status='WAITING' ORDER by id""", (task.sequenceid, ))
runtimes_adjusted = False
if current_runtimes:
last_runtime = current_runtimes[0][1]
for the_task_id, runtime in current_runtimes:
if runtime < last_runtime:
run_sql("""UPDATE schTASK SET runtime=%s WHERE id=%s""", (last_runtime, the_task_id))
Log("Adjusted runtime of task_id %s to %s in order to be executed in the correct sequenceid order" % (the_task_id, last_runtime), debug)
runtimes_adjusted = True
runtime = last_runtime
last_runtime = runtime
if runtimes_adjusted:
## Some runtime have been adjusted
return True
if task.sequenceid is not None:
for t in chain(self.active_tasks_all_nodes,
self.waiting_tasks_all_nodes):
if task.sequenceid == t.sequenceid and task.id > t.id:
Log('Task %s need to run after task %s since they have the same sequence id: %s' % (task.id, t.id, task.sequenceid), debug)
## If there is a task with same sequence number then do not run the current task
return False
if is_monotask(task.proc) and higher:
## This is a monotask
Log("Cannot run because this is a monotask and there are higher priority tasks: %s" % (higher, ), debug)
return False
## Check for monotasks wanting to run
for t in self.mono_tasks_all_nodes:
if task.priority < t.priority:
Log("Cannot run because there is a monotask with higher priority: %s %s" % (t.id, t.proc), debug)
return False
## We check if it is necessary to stop/put to sleep some lower priority
## task.
tasks_to_stop, tasks_to_sleep = self.get_tasks_to_sleep_and_stop(task, self.active_tasks_all_nodes)
Log('tasks_to_stop: %s' % tasks_to_stop, debug)
Log('tasks_to_sleep: %s' % tasks_to_sleep, debug)
if tasks_to_stop and task.priority < 100:
## Only tasks with priority higher than 100 have the power
## to put task to stop.
Log("Cannot run because there are task to stop: %s and priority < 100" % tasks_to_stop, debug)
return False
for t in tasks_to_sleep:
if not t.priority < task.priority:
Log("Cannot run because #%s with priority %s cannot be slept by this task" % (t.id, t.priority), debug)
return False
procname = task.proc.split(':')[0]
if not tasks_to_stop and not tasks_to_sleep:
if is_monotask(task.proc) and self.active_tasks_all_nodes:
Log("Cannot run because this is a monotask and there are other tasks running: %s" % (self.active_tasks_all_nodes, ), debug)
return False
if task.proc not in CFG_BIBTASK_FIXEDTIMETASKS and len(self.node_active_tasks) >= CFG_BIBSCHED_MAX_NUMBER_CONCURRENT_TASKS:
Log("Cannot run because all resources (%s) are used (%s), active: %s" % (CFG_BIBSCHED_MAX_NUMBER_CONCURRENT_TASKS, len(self.node_active_tasks), self.node_active_tasks), debug)
return False
for t in self.waiting_tasks_all_nodes:
if self.is_task_non_concurrent(task, t) and task.priority < t.priority:
Log("Cannot run because %s is non-concurrent and has higher priority" % t, debug)
return False
if task.status in ("SCHEDULED",):
Log("Task is already scheduled", debug)
return False
elif task.status in ("SLEEPING", "ABOUT TO SLEEP"):
if task.host != self.hostname:
Log("We can't wake up tasks that are not in the same node", debug)
return False
## We can only wake up tasks that are running on our own host
for t in self.node_active_tasks:
## But only if there are not other tasks still going to sleep, otherwise
## we might end up stealing the slot for an higher priority task.
if t.id != task.id and t.status in ('ABOUT TO SLEEP', 'ABOUT TO STOP'):
Log("Not yet waking up task #%d since there are other tasks (%s #%d) going to sleep (higher priority task incoming?)" % (task.id, t.proc, t.id), debug)
return False
bibsched_set_status(task.id, "CONTINUING", task.status)
if not bibsched_send_signal(task.id, signal.SIGCONT):
bibsched_set_status(task.id, "ERROR", "CONTINUING")
Log("Task #%d (%s) woken up but didn't existed anymore" % (task.id, task.proc))
return True
Log("Task #%d (%s) woken up" % (task.id, task.proc))
return True
elif procname in self.helper_modules:
program = os.path.join(CFG_BINDIR, procname)
## Trick to log in bibsched.log the task exiting
exit_str = '&& echo "`date "+%%Y-%%m-%%d %%H:%%M:%%S"` --> Task #%d (%s) exited" >> %s' % (task.id, task.proc, os.path.join(CFG_LOGDIR, 'bibsched.log'))
command = "%s %s %s" % (program, str(task.id), exit_str)
### Set the task to scheduled and tie it to this host
if self.tie_task_to_host(task.id):
Log("Task #%d (%s) started" % (task.id, task.proc))
### Relief the lock for the BibTask, it is safe now to do so
spawn_task(command, wait=is_monotask(task.proc))
count = 10
while run_sql("""SELECT status FROM schTASK
WHERE id=%s AND status='SCHEDULED'""",
(task.id, )):
## Polling to wait for the task to really start,
## in order to avoid race conditions.
if count <= 0:
Log("Process %s (task_id: %s) was launched but seems not to be able to reach RUNNING status." % (task.proc, task.id))
bibsched_set_status(task.id, "ERROR", "SCHEDULED")
return True
time.sleep(CFG_BIBSCHED_REFRESHTIME)
count -= 1
return True
else:
raise StandardError("%s is not in the allowed modules" % procname)
else:
## It's not still safe to run the task.
## We first need to stop task that should be stopped
## and to put to sleep task that should be put to sleep
changes = False
for t in tasks_to_stop:
if t.status not in ('ABOUT TO STOP', 'SCHEDULED'):
changes = True
stop_task(t)
else:
Log("Cannot run because we are waiting for #%s to stop" % t.id, debug)
for t in tasks_to_sleep:
if t.status not in ('ABOUT TO SLEEP', 'SCHEDULED', 'ABOUT TO STOP'):
changes = True
sleep_task(t)
else:
Log("Cannot run because we are waiting for #%s to sleep" % t.id, debug)
if changes:
time.sleep(CFG_BIBSCHED_REFRESHTIME)
return changes
def check_errors(self):
errors = run_sql("""SELECT id,proc,status FROM schTASK
WHERE status = 'ERROR'
OR status = 'DONE WITH ERRORS'
OR status = 'CERROR'""")
if errors:
error_msgs = []
error_recoverable = True
for e_id, e_proc, e_status in errors:
if run_sql("""UPDATE schTASK
SET status='ERRORS REPORTED'
WHERE id = %s AND (status='CERROR'
OR status='ERROR'
OR status='DONE WITH ERRORS')""", [e_id]):
msg = " #%s %s -> %s" % (e_id, e_proc, e_status)
error_msgs.append(msg)
if e_status in ('ERROR', 'DONE WITH ERRORS'):
error_recoverable = False
if error_msgs:
msg = "BibTask with ERRORS:\n%s" % '\n'.join(error_msgs)
if error_recoverable or CFG_BIBSCHED_NEVER_STOPS:
raise RecoverableError(msg)
else:
raise StandardError(msg)
def calculate_rows(self):
"""Return all the node_relevant_active_tasks to work on."""
try:
self.check_errors()
except RecoverableError as msg:
register_emergency('Light emergency from %s: BibTask failed: %s' % (CFG_SITE_URL, msg))
max_bibupload_priority, min_bibupload_priority = run_sql(
"""SELECT MAX(priority), MIN(priority)
FROM schTASK
WHERE status IN ('WAITING', 'RUNNING', 'SLEEPING',
'ABOUT TO STOP', 'ABOUT TO SLEEP',
'SCHEDULED', 'CONTINUING')
AND proc = 'bibupload'
AND runtime <= NOW()""")[0]
if max_bibupload_priority > min_bibupload_priority:
run_sql(
"""UPDATE schTASK SET priority = %s
WHERE status IN ('WAITING', 'RUNNING', 'SLEEPING',
'ABOUT TO STOP', 'ABOUT TO SLEEP',
'SCHEDULED', 'CONTINUING')
AND proc = 'bibupload'
AND runtime <= NOW()
AND priority < %s""", (max_bibupload_priority,
max_bibupload_priority))
# The bibupload tasks are sorted by id,
# which means by the order they were scheduled
self.node_relevant_bibupload_tasks = Task.from_resultset(run_sql(
"""SELECT id, proc, runtime, status, priority, host, sequenceid
FROM schTASK WHERE status IN ('WAITING', 'SLEEPING')
AND proc = 'bibupload'
AND runtime <= NOW()
ORDER BY FIELD(status, 'SLEEPING', 'WAITING'),
id ASC LIMIT 1""", n=1))
## The other tasks are sorted by priority
self.waiting_tasks_all_nodes = Task.from_resultset(run_sql(
"""SELECT id, proc, runtime, status, priority, host, sequenceid
FROM schTASK WHERE (status = 'WAITING' AND runtime <= NOW())
OR status = 'SLEEPING'
ORDER BY priority DESC, runtime ASC, id ASC"""))
self.sleeping_tasks_all_nodes = Task.from_resultset(run_sql(
"""SELECT id, proc, runtime, status, priority, host, sequenceid
FROM schTASK WHERE status = 'SLEEPING'
ORDER BY priority DESC, runtime ASC, id ASC"""))
self.active_tasks_all_nodes = Task.from_resultset(run_sql(
"""SELECT id, proc, runtime, status, priority, host, sequenceid
FROM schTASK WHERE status IN ('RUNNING', 'CONTINUING',
'SCHEDULED', 'ABOUT TO STOP',
'ABOUT TO SLEEP')"""))
self.mono_tasks_all_nodes = tuple(t for t in
chain(self.waiting_tasks_all_nodes, self.active_tasks_all_nodes)
if is_monotask(t.proc))
## Remove tasks that can not be executed on this host
def filter_by_host(tasks):
return tuple(t for t in tasks if t.host == self.hostname or not t.host)
self.node_active_tasks = filter_by_host(self.active_tasks_all_nodes)
self.node_sleeping_tasks = filter_by_host(self.sleeping_tasks_all_nodes)
self.filter_for_allowed_tasks()
def check_auto_mode(self):
"""Check if the queue is in automatic or manual mode"""
r = run_sql('SELECT value FROM schSTATUS WHERE name = "auto_mode"')
try:
status = int(r[0][0])
except (ValueError, IndexError):
# We insert the missing configuration variable in the DB
run_sql('INSERT INTO schSTATUS (name, value) VALUES ("auto_mode", "1")')
status = 1
if not status:
r = run_sql('SELECT value FROM schSTATUS WHERE name = "resume_after"')
try:
date_string = r[0][0]
except IndexError:
pass
else:
if date_string:
resume_after = datetime(*(time.strptime(date_string, "%Y-%m-%d %H:%M:%S")[0:6]))
if datetime.now() > resume_after:
run_sql('UPDATE schSTATUS SET value = "" WHERE name = "resume_after"')
run_sql('UPDATE schSTATUS SET value = "1" WHERE name = "auto_mode"')
status = 1
return status
def check_for_crashed_tasks(self):
for task in self.node_active_tasks:
Log('Checking %s' % task.id)
pid = get_task_pid(task.id)
if not pid:
Log('Task crashed %s' % task.id)
run_sql("""UPDATE schTASK SET status = 'CERROR'
WHERE id = %%s AND status IN (%s)"""
% ','.join("'%s'" % s for s in ACTIVE_STATUS),
[task.id])
def check_debug_mode(self):
debug_mode = fetch_debug_mode()
if debug_mode and not self.debug:
Log('Switching to debug mode')
elif self.debug and not debug_mode:
Log('Switching out of debug mode')
self.debug = debug_mode
def tick(self):
Log("New bibsched cycle", self.debug)
self.cycles_count += 1
self.check_debug_mode()
if self.cycles_count % 50 == 0:
self.check_for_crashed_tasks()
try:
self.check_errors()
except RecoverableError, msg:
register_emergency('Light emergency from %s: BibTask failed: %s'
% (CFG_SITE_URL, msg))
# Update our tasks list (to know who is running, sleeping, etc.)
self.calculate_rows()
# Let's first handle running tasks running on this node.
for task in self.node_active_tasks:
if self.handle_task(task):
break
else:
# If nothing has changed we can go on to run tasks.
for task in self.node_relevant_waiting_tasks:
if task.proc == 'bibupload' \
and self.node_relevant_bibupload_tasks:
## We switch in bibupload serial mode!
## which means we execute the first next bibupload.
if self.handle_task(self.node_relevant_bibupload_tasks[0]):
## Something has changed
break
elif self.handle_task(task):
## Something has changed
break
else:
time.sleep(CFG_BIBSCHED_REFRESHTIME)
def watch_loop(self):
## Cleaning up scheduled task not run because of bibsched being
## interrupted in the middle.
run_sql("""UPDATE schTASK
SET status = 'WAITING'
WHERE status = 'SCHEDULED'
AND host = %s""", (self.hostname, ))
try:
while True:
auto_mode = self.check_auto_mode()
if auto_mode:
self.tick()
else:
# If nothing has changed we can go on to run tasks.
for task in self.node_relevant_waiting_tasks:
if task[1] == 'bibupload' and self.node_relevant_bibupload_tasks:
## We switch in bibupload serial mode!
## which means we execute the first next bibupload.
if self.handle_task(*self.node_relevant_bibupload_tasks[0]):
## Something has changed
break
elif self.handle_task(*task):
## Something has changed
break
else:
time.sleep(CFG_BIBSCHED_REFRESHTIME)
except Exception as err:
register_exception(alert_admin=True)
try:
register_emergency('Emergency from %s: BibSched halted: %s'
% (CFG_SITE_URL, err))
except NotImplementedError:
pass
raise
def Log(message, debug=None):
if debug is False:
return
log = open(CFG_LOGDIR + "/bibsched.log", "a")
log.write(time.strftime("%Y-%m-%d %H:%M:%S --> ", time.localtime()))
log.write(message)
log.write("\n")
log.close()
def redirect_stdout_and_stderr():
"This function redirects stdout and stderr to bibsched.log and bibsched.err file."
old_stdout = sys.stdout
old_stderr = sys.stderr
sys.stdout = open(CFG_LOGDIR + "/bibsched.log", "a")
sys.stderr = open(CFG_LOGDIR + "/bibsched.err", "a")
return old_stdout, old_stderr
def restore_stdout_and_stderr(stdout, stderr):
sys.stdout = stdout
sys.stderr = stderr
def usage(exitcode=1, msg=""):
"""Prints usage info."""
if msg:
sys.stderr.write("Error: %s.\n" % msg)
sys.stderr.write("""\
Usage: %s [options] [start|stop|restart|monitor|status]
The following commands are available for bibsched:
start start bibsched in background
stop stop running bibtasks and the bibsched daemon safely
halt halt running bibsched while keeping bibtasks running
restart restart running bibsched
monitor enter the interactive monitor
status get report about current status of the queue
purge purge the scheduler queue from old tasks
General options:
-h, --help \t Print this help.
-V, --version \t Print version information.
-q, --quiet \t Quiet mode
-d, --debug \t Write debugging information in bibsched.log
Status options:
-s, --status=LIST\t Which BibTask status should be considered (default is Running,waiting)
-S, --since=TIME\t Since how long time to consider tasks e.g.: 30m, 2h, 1d (default
is all)
-t, --tasks=LIST\t Comma separated list of BibTask to consider (default
\t is all)
Purge options:
-s, --status=LIST\t Which BibTask status should be considered (default is DONE)
-S, --since=TIME\t Since how long time to consider tasks e.g.: 30m, 2h, 1d (default
is %s days)
-t, --tasks=LIST\t Comma separated list of BibTask to consider (default
\t is %s)
""" % (sys.argv[0], CFG_BIBSCHED_GC_TASKS_OLDER_THAN, ','.join(CFG_BIBSCHED_GC_TASKS_TO_REMOVE + CFG_BIBSCHED_GC_TASKS_TO_ARCHIVE)))
sys.exit(exitcode)
pidfile = os.path.join(CFG_PREFIX, 'var', 'run', 'bibsched.pid')
def error(msg):
print("error: %s" % msg, file=sys.stderr)
sys.exit(1)
def warning(msg):
print("warning: %s" % msg, file=sys.stderr)
def server_pid(ping_the_process=True, check_is_really_bibsched=True):
# The pid must be stored on the filesystem
try:
pid = int(open(pidfile).read())
except IOError:
return None
if ping_the_process:
# Even if the pid is available, we check if it corresponds to an
# actual process, as it might have been killed externally
try:
os.kill(pid, signal.SIGCONT)
except OSError:
warning("pidfile %s found referring to pid %s which is not running" % (pidfile, pid))
return None
if check_is_really_bibsched:
output = run_shell_command("ps p %s -o args=", (str(pid), ))[1]
if not 'bibsched' in output:
warning("pidfile %s found referring to pid %s which does not correspond to bibsched: cmdline is %s" % (pidfile, pid, output))
return None
return pid
def write_server_pid(pid):
open(pidfile, 'w').write('%d' % pid)
def start(verbose=True, debug=False):
""" Fork this process in the background and start processing
requests. The process PID is stored in a pid file, so that it can
be stopped later on."""
if verbose:
sys.stdout.write("starting bibsched: ")
sys.stdout.flush()
pid = server_pid(ping_the_process=False)
if pid:
pid2 = server_pid()
if pid2:
error("another instance of bibsched (pid %d) is running" % pid2)
else:
warning("%s exist but the corresponding bibsched (pid %s) seems not be running" % (pidfile, pid))
warning("erasing %s and continuing..." % (pidfile, ))
os.remove(pidfile)
if debug:
pid = os.getpid()
write_server_pid(pid)
else:
# start the child process using the "double fork" technique
pid = os.fork()
if pid > 0:
sys.exit(0)
os.setsid()
os.chdir('/')
pid = os.fork()
if pid > 0:
if verbose:
sys.stdout.write('pid %d\n' % pid)
Log("daemon started (pid %d)" % pid)
write_server_pid(pid)
return
sys.stdin.close()
redirect_stdout_and_stderr()
sched = BibSched(debug=debug)
try:
sched.watch_loop()
finally:
try:
os.remove(pidfile)
except OSError:
pass
def halt(verbose=True, soft=False, debug=False): # pylint: disable=W0613
pid = server_pid()
if not pid:
if soft:
print('bibsched seems not to be running.', file=sys.stderr)
return
else:
error('bibsched seems not to be running.')
try:
os.kill(pid, signal.SIGKILL)
except OSError:
print('no bibsched process found', file=sys.stderr)
Log("daemon stopped (pid %d)" % pid)
if verbose:
print("stopping bibsched: pid %d" % pid)
os.unlink(pidfile)
def write_message(msg, stream=None, verbose=1): # pylint: disable=W0613
"""Write message and flush output stream (may be sys.stdout or sys.stderr).
Useful for debugging stuff."""
if stream is None:
stream = sys.stdout
if msg:
if stream == sys.stdout or stream == sys.stderr:
stream.write(time.strftime("%Y-%m-%d %H:%M:%S --> ",
time.localtime()))
try:
stream.write("%s\n" % msg)
except UnicodeEncodeError:
stream.write("%s\n" % msg.encode('ascii', 'backslashreplace'))
stream.flush()
else:
sys.stderr.write("Unknown stream %s. [must be sys.stdout or sys.stderr]\n" % stream)
def report_queue_status(verbose=True, status=None, since=None, tasks=None): # pylint: disable=W0613
"""
Report about the current status of BibSched queue on standard output.
"""
def report_about_processes(status='RUNNING', since=None, tasks=None):
"""
Helper function to report about processes with the given status.
"""
if tasks is None:
task_query = ''
else:
task_query = 'AND proc IN (%s)' % (
','.join([repr(real_escape_string(task)) for task in tasks]))
if since is None:
since_query = ''
else:
# We're not interested in future task
if since.startswith('+') or since.startswith('-'):
since = since[1:]
since = '-' + since
since_query = "AND runtime >= '%s'" % get_datetime(since)
res = run_sql("""SELECT id, proc, runtime, status, priority, host,
sequenceid
FROM schTASK WHERE status=%%s %(task_query)s
%(since_query)s ORDER BY id ASC""" % {
'task_query': task_query,
'since_query' : since_query},
(status,))
write_message("%s processes: %d" % (status, len(res)))
for t in Task.from_resultset(res):
write_message(' * %s' % t)
return
write_message("BibSched queue status report for %s:" % gethostname())
daemon_status = server_pid() and "UP" or "DOWN"
write_message("BibSched daemon status: %s" % daemon_status)
- r = run_sql('SELECT value FROM schSTATUS WHERE name = "auto_mode"')
- try:
- mode = bool(int(r[0][0]))
- except (ValueError, IndexError):
- mode = True
+ if run_sql("show tables like 'schSTATUS'"):
+ r = run_sql('SELECT value FROM schSTATUS WHERE name = "auto_mode"')
+ try:
+ mode = bool(int(r[0][0]))
+ except (ValueError, IndexError):
+ mode = True
+ else:
+ mode = False
mode_str = mode and 'AUTOMATIC' or 'MANUAL'
write_message("BibSched queue running mode: %s" % mode_str)
if status is None:
report_about_processes('Running', since, tasks)
report_about_processes('Waiting', since, tasks)
else:
for state in status:
report_about_processes(state, since, tasks)
write_message("Done.")
def restart(verbose=True, debug=False):
halt(verbose, soft=True, debug=debug)
start(verbose, debug=debug)
def stop(verbose=True, debug=False):
"""
* Stop bibsched
* Send stop signal to all the running tasks
* wait for all the tasks to stop
* return
"""
if verbose:
print("Stopping BibSched if running")
halt(verbose, soft=True, debug=debug)
run_sql("UPDATE schTASK SET status='WAITING' WHERE status='SCHEDULED'")
res = run_sql("""SELECT id, status FROM schTASK
WHERE status NOT LIKE 'DONE'
AND status NOT LIKE '%_DELETED'
AND (status='RUNNING'
OR status='ABOUT TO STOP'
OR status='ABOUT TO SLEEP'
OR status='SLEEPING'
OR status='CONTINUING')""")
if verbose:
print("Stopping all running BibTasks")
for task_id, status in res:
if status == 'SLEEPING':
bibsched_send_signal(task_id, signal.SIGCONT)
time.sleep(CFG_BIBSCHED_REFRESHTIME)
bibsched_set_status(task_id, 'ABOUT TO STOP')
while run_sql("""SELECT id FROM schTASK
WHERE status NOT LIKE 'DONE'
AND status NOT LIKE '%_DELETED'
AND (status='RUNNING'
OR status='ABOUT TO STOP'
OR status='ABOUT TO SLEEP'
OR status='SLEEPING'
OR status='CONTINUING')"""):
if verbose:
sys.stdout.write('.')
sys.stdout.flush()
time.sleep(CFG_BIBSCHED_REFRESHTIME)
if verbose:
print("\nStopped")
Log("BibSched and all BibTasks stopped")
def main():
from invenio.legacy.bibsched.monitor import monitor
from invenio.legacy.bibsched.bibtask import check_running_process_user
check_running_process_user()
verbose = True
status = None
since = None
tasks = None
debug = False
try:
opts, args = getopt.gnu_getopt(sys.argv[1:], "hVdqS:s:t:", [
"help", "version", "debug", "quiet", "since=", "status=", "task="])
except getopt.GetoptError as err:
Log("Error: %s" % err)
usage(1, err)
for opt, arg in opts:
if opt in ["-h", "--help"]:
usage(0)
elif opt in ["-V", "--version"]:
print(CFG_VERSION)
sys.exit(0)
elif opt in ['-q', '--quiet']:
verbose = False
elif opt in ['-s', '--status']:
status = arg.split(',')
elif opt in ['-S', '--since']:
since = arg
elif opt in ['-t', '--task']:
tasks = arg.split(',')
elif opt in ['-d', '--debug']:
debug = True
else:
usage(1)
try:
cmd = args[0]
except IndexError:
cmd = 'monitor'
try:
if cmd in ('status', 'purge'):
{'status' : report_queue_status,
'purge' : gc_tasks}[cmd](verbose, status, since, tasks)
else:
{'start': start,
'halt': halt,
'stop': stop,
'restart': restart,
'monitor': monitor}[cmd](verbose=verbose, debug=debug)
except KeyError:
usage(1, 'unkown command: %s' % cmd)
diff --git a/invenio/legacy/bibupload/engine.py b/invenio/legacy/bibupload/engine.py
index 7afe26dc7..8700ba117 100644
--- a/invenio/legacy/bibupload/engine.py
+++ b/invenio/legacy/bibupload/engine.py
@@ -1,3005 +1,3011 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
"""
BibUpload: Receive MARC XML file and update the appropriate database
tables according to options.
"""
__revision__ = "$Id$"
import os
import re
import sys
import time
from datetime import datetime
from six import iteritems
from zlib import compress
import socket
import marshal
import copy
import tempfile
import urlparse
import urllib2
import urllib
from invenio.config import CFG_OAI_ID_FIELD, \
CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG, \
CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG, \
CFG_BIBUPLOAD_EXTERNAL_OAIID_PROVENANCE_TAG, \
CFG_BIBUPLOAD_STRONG_TAGS, \
CFG_BIBUPLOAD_CONTROLLED_PROVENANCE_TAGS, \
CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE, \
CFG_BIBUPLOAD_DELETE_FORMATS, \
CFG_SITE_URL, \
CFG_SITE_SECURE_URL, \
CFG_SITE_RECORD, \
CFG_OAI_PROVENANCE_ALTERED_SUBFIELD, \
CFG_BIBUPLOAD_DISABLE_RECORD_REVISIONS, \
CFG_BIBUPLOAD_CONFLICTING_REVISION_TICKET_QUEUE, \
CFG_CERN_SITE, \
CFG_BIBUPLOAD_MATCH_DELETED_RECORDS
from invenio.utils.json import json, CFG_JSON_AVAILABLE
from invenio.legacy.bibupload.config import CFG_BIBUPLOAD_CONTROLFIELD_TAGS, \
CFG_BIBUPLOAD_SPECIAL_TAGS, \
CFG_BIBUPLOAD_DELETE_CODE, \
CFG_BIBUPLOAD_DELETE_VALUE, \
CFG_BIBUPLOAD_OPT_MODES
from invenio.legacy.dbquery import run_sql
from invenio.legacy.bibrecord import create_records, \
record_add_field, \
record_delete_field, \
record_xml_output, \
record_get_field_instances, \
record_get_field_value, \
record_get_field_values, \
field_get_subfield_values, \
field_get_subfield_instances, \
record_modify_subfield, \
record_delete_subfield_from, \
record_delete_fields, \
record_add_subfield_into, \
record_find_field, \
record_extract_oai_id, \
record_extract_dois, \
record_has_field, \
- records_identical
+ records_identical, \
+ record_drop_duplicate_fields
from invenio.legacy.search_engine import get_record, record_exists, search_pattern
from invenio.utils.date import convert_datestruct_to_datetext
from invenio.ext.logging import register_exception
from invenio.legacy.bibcatalog.api import BIBCATALOG_SYSTEM
from intbitset import intbitset
from invenio.utils.url import make_user_agent_string
from invenio.config import CFG_BIBDOCFILE_FILEDIR
from invenio.legacy.bibsched.bibtask import task_init, write_message, \
task_set_option, task_get_option, task_get_task_param, \
task_update_progress, task_sleep_now_if_required, fix_argv_paths, \
RecoverableError
from invenio.legacy.bibdocfile.api import BibRecDocs, file_strip_ext, normalize_format, \
get_docname_from_url, check_valid_url, download_url, \
KEEP_OLD_VALUE, decompose_bibdocfile_url, InvenioBibDocFileError, \
bibdocfile_url_p, CFG_BIBDOCFILE_AVAILABLE_FLAGS, guess_format_from_url, \
BibRelation, MoreInfo
from invenio.legacy.search_engine import search_pattern
from invenio.legacy.bibupload.revisionverifier import RevisionVerifier, \
InvenioBibUploadConflictingRevisionsError, \
InvenioBibUploadInvalidRevisionError, \
InvenioBibUploadMissing005Error, \
InvenioBibUploadUnchangedRecordError
#Statistic variables
stat = {}
stat['nb_records_to_upload'] = 0
stat['nb_records_updated'] = 0
stat['nb_records_inserted'] = 0
stat['nb_errors'] = 0
stat['nb_holdingpen'] = 0
stat['exectime'] = time.localtime()
_WRITING_RIGHTS = None
CFG_BIBUPLOAD_ALLOWED_SPECIAL_TREATMENTS = ('oracle', )
CFG_HAS_BIBCATALOG = "UNKNOWN"
def check_bibcatalog():
"""
Return True if bibcatalog is available.
"""
global CFG_HAS_BIBCATALOG # pylint: disable=W0603
if CFG_HAS_BIBCATALOG != "UNKNOWN":
return CFG_HAS_BIBCATALOG
CFG_HAS_BIBCATALOG = True
if BIBCATALOG_SYSTEM is not None:
bibcatalog_response = BIBCATALOG_SYSTEM.check_system()
else:
bibcatalog_response = "No ticket system configured"
if bibcatalog_response != "":
write_message("BibCatalog error: %s\n" % (bibcatalog_response,))
CFG_HAS_BIBCATALOG = False
return CFG_HAS_BIBCATALOG
## Let's set a reasonable timeout for URL request (e.g. FFT)
socket.setdefaulttimeout(40)
def parse_identifier(identifier):
"""Parse the identifier and determine if it is temporary or fixed"""
id_str = str(identifier)
if not id_str.startswith("TMP:"):
return (False, identifier)
else:
return (True, id_str[4:])
def resolve_identifier(tmps, identifier):
"""Resolves an identifier. If the identifier is not temporary, this
function is an identity on the second argument. Otherwise, a resolved
value is returned or an exception raised"""
is_tmp, tmp_id = parse_identifier(identifier)
if is_tmp:
if not tmp_id in tmps:
raise StandardError("Temporary identifier %s not present in the dictionary" % (tmp_id, ))
if tmps[tmp_id] == -1:
# the identifier has been signalised but never assigned a value - probably error during processing
raise StandardError("Temporary identifier %s has been declared, but never assigned a value. Probably an error during processign of an appropriate FFT has happened. Please see the log" % (tmp_id, ))
return int(tmps[tmp_id])
else:
return int(identifier)
_re_find_001 = re.compile('<controlfield\\s+tag=("001"|\'001\')\\s*>\\s*(\\d*)\\s*</controlfield>', re.S)
def bibupload_pending_recids():
"""This function embed a bit of A.I. and is more a hack than an elegant
algorithm. It should be updated in case bibupload/bibsched are modified
in incompatible ways.
This function return the intbitset of all the records that are being
(or are scheduled to be) touched by other bibuploads.
"""
options = run_sql("""SELECT arguments FROM schTASK WHERE status<>'DONE' AND
proc='bibupload' AND (status='RUNNING' OR status='CONTINUING' OR
status='WAITING' OR status='SCHEDULED' OR status='ABOUT TO STOP' OR
status='ABOUT TO SLEEP')""")
ret = intbitset()
xmls = []
if options:
for arguments in options:
arguments = marshal.loads(arguments[0])
for argument in arguments[1:]:
if argument.startswith('/'):
# XMLs files are recognizable because they're absolute
# files...
xmls.append(argument)
for xmlfile in xmls:
# Let's grep for the 001
try:
xml = open(xmlfile).read()
ret += [int(group[1]) for group in _re_find_001.findall(xml)]
except:
continue
return ret
### bibupload engine functions:
def bibupload(record, opt_mode=None, opt_notimechange=0, oai_rec_id="", pretend=False,
tmp_ids=None, tmp_vers=None):
"""Main function: process a record and fit it in the tables
bibfmt, bibrec, bibrec_bibxxx, bibxxx with proper record
metadata.
Return (error_code, recID) of the processed record.
"""
if tmp_ids is None:
tmp_ids = {}
if tmp_vers is None:
tmp_vers = {}
if opt_mode == 'reference':
## NOTE: reference mode has been deprecated in favour of 'correct'
opt_mode = 'correct'
assert(opt_mode in CFG_BIBUPLOAD_OPT_MODES)
try:
record_xml_output(record).decode('utf-8')
except UnicodeDecodeError:
msg = " Failed: Invalid utf-8 characters."
write_message(msg, verbose=1, stream=sys.stderr)
return (1, -1, msg)
error = None
affected_tags = {}
original_record = {}
rec_old = {}
now = datetime.now() # will hold record creation/modification date
record_had_altered_bit = False
is_opt_mode_delete = False
# Extraction of the Record Id from 001, SYSNO or OAIID or DOI tags:
rec_id = retrieve_rec_id(record, opt_mode, pretend=pretend)
if rec_id == -1:
msg = " Failed: either the record already exists and insert was " \
"requested or the record does not exists and " \
"replace/correct/append has been used"
write_message(msg, verbose=1, stream=sys.stderr)
return (1, -1, msg)
elif rec_id > 0:
write_message(" -Retrieve record ID (found %s): DONE." % rec_id, verbose=2)
(unique_p, msg) = check_record_doi_is_unique(rec_id, record)
if not unique_p:
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
if '001' not in record:
# Found record ID by means of SYSNO or OAIID or DOI, and the
# input MARCXML buffer does not have this 001 tag, so we
# should add it now:
error = record_add_field(record, '001', controlfield_value=rec_id)
if error is None:
msg = " Failed: Error during adding the 001 controlfield " \
"to the record"
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
else:
error = None
write_message(" -Added tag 001: DONE.", verbose=2)
write_message(" -Check if the xml marc file is already in the database: DONE" , verbose=2)
record_deleted_p = False
if opt_mode == 'insert' or \
(opt_mode == 'replace_or_insert') and rec_id is None:
insert_mode_p = True
# Insert the record into the bibrec databases to have a recordId
rec_id = create_new_record(pretend=pretend)
write_message(" -Creation of a new record id (%d): DONE" % rec_id, verbose=2)
# we add the record Id control field to the record
error = record_add_field(record, '001', controlfield_value=rec_id)
if error is None:
msg = " Failed: Error during adding the 001 controlfield " \
"to the record"
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
else:
error = None
if '005' not in record:
error = record_add_field(record, '005', controlfield_value=now.strftime("%Y%m%d%H%M%S.0"))
if error is None:
msg = " ERROR: during adding to 005 controlfield to record"
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
else:
error = None
else:
write_message(" Note: 005 already existing upon inserting of new record. Keeping it.", verbose=2)
elif opt_mode != 'insert':
insert_mode_p = False
# Update Mode
# Retrieve the old record to update
rec_old = get_record(rec_id)
record_had_altered_bit = record_get_field_values(rec_old, CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[:3], CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[3], CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[4], CFG_OAI_PROVENANCE_ALTERED_SUBFIELD)
# Also save a copy to restore previous situation in case of errors
original_record = get_record(rec_id)
if rec_old is None:
msg = " Failed during the creation of the old record!"
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
else:
write_message(" -Retrieve the old record to update: DONE", verbose=2)
# flag to check whether the revisions have been verified and patch generated.
# If revision verification failed, then we need to manually identify the affected tags
# and process them
revision_verified = False
rev_verifier = RevisionVerifier()
#check for revision conflicts before updating record
if record_has_field(record, '005') and not CFG_BIBUPLOAD_DISABLE_RECORD_REVISIONS:
write_message(" -Upload Record has 005. Verifying Revision", verbose=2)
try:
rev_res = rev_verifier.verify_revision(record, original_record, opt_mode)
if rev_res:
opt_mode = rev_res[0]
record = rev_res[1]
affected_tags = rev_res[2]
revision_verified = True
write_message(lambda: " -Patch record generated. Changing opt_mode to correct.\nPatch:\n%s " % record_xml_output(record), verbose=2)
else:
write_message(" -No Patch Record.", verbose=2)
except InvenioBibUploadUnchangedRecordError as err:
msg = " -ISSUE: %s" % err
write_message(msg, verbose=1, stream=sys.stderr)
write_message(msg, " Continuing anyway in case there are FFT or other tags")
except InvenioBibUploadConflictingRevisionsError as err:
msg = " -ERROR: Conflicting Revisions - %s" % err
write_message(msg, verbose=1, stream=sys.stderr)
submit_ticket_for_holding_pen(rec_id, err, "Conflicting Revisions. Inserting record into holding pen.", pretend=pretend)
insert_record_into_holding_pen(record, str(rec_id), pretend=pretend)
return (2, int(rec_id), msg)
except InvenioBibUploadInvalidRevisionError as err:
msg = " -ERROR: Invalid Revision - %s" % err
write_message(msg)
submit_ticket_for_holding_pen(rec_id, err, "Invalid Revisions. Inserting record into holding pen.", pretend=pretend)
insert_record_into_holding_pen(record, str(rec_id), pretend=pretend)
return (2, int(rec_id), msg)
except InvenioBibUploadMissing005Error as err:
msg = " -ERROR: Missing 005 - %s" % err
write_message(msg)
submit_ticket_for_holding_pen(rec_id, err, "Missing 005. Inserting record into holding pen.", pretend=pretend)
insert_record_into_holding_pen(record, str(rec_id), pretend=pretend)
return (2, int(rec_id), msg)
else:
write_message(" - No 005 Tag Present. Resuming normal flow.", verbose=2)
# dictionaries to temporarily hold original recs tag-fields
existing_tags = {}
retained_tags = {}
# in case of delete operation affected tags should be deleted in delete_bibrec_bibxxx
# but should not be updated again in STAGE 4
# utilising the below flag
is_opt_mode_delete = False
if not revision_verified:
# either 005 was not present or opt_mode was not correct/replace
# in this case we still need to find out affected tags to process
write_message(" - Missing 005 or opt_mode!=Replace/Correct.Revision Verifier not called.", verbose=2)
# Identify affected tags
if opt_mode == 'correct' or opt_mode == 'replace' or opt_mode == 'replace_or_insert':
rec_diff = rev_verifier.compare_records(record, original_record, opt_mode)
affected_tags = rev_verifier.retrieve_affected_tags_with_ind(rec_diff)
elif opt_mode == 'delete':
# populate an intermediate dictionary
# used in upcoming step related to 'delete' mode
is_opt_mode_delete = True
for tag, fields in iteritems(original_record):
existing_tags[tag] = [tag + (field[1] != ' ' and field[1] or '_') + (field[2] != ' ' and field[2] or '_') for field in fields]
elif opt_mode == 'append':
for tag, fields in iteritems(record):
if tag not in CFG_BIBUPLOAD_CONTROLFIELD_TAGS:
affected_tags[tag] = [(field[1], field[2]) for field in fields]
# In Replace mode, take over old strong tags if applicable:
if opt_mode == 'replace' or \
opt_mode == 'replace_or_insert':
copy_strong_tags_from_old_record(record, rec_old)
# Delete tags to correct in the record
if opt_mode == 'correct':
delete_tags_to_correct(record, rec_old)
write_message(" -Delete the old tags to correct in the old record: DONE",
verbose=2)
# Delete tags specified if in delete mode
if opt_mode == 'delete':
record = delete_tags(record, rec_old)
for tag, fields in iteritems(record):
retained_tags[tag] = [tag + (field[1] != ' ' and field[1] or '_') + (field[2] != ' ' and field[2] or '_') for field in fields]
#identify the tags that have been deleted
for tag in existing_tags.keys():
if tag not in retained_tags:
for item in existing_tags[tag]:
tag_to_add = item[0:3]
ind1, ind2 = item[3], item[4]
if tag_to_add in affected_tags and (ind1, ind2) not in affected_tags[tag_to_add]:
affected_tags[tag_to_add].append((ind1, ind2))
else:
affected_tags[tag_to_add] = [(ind1, ind2)]
else:
deleted = list(set(existing_tags[tag]) - set(retained_tags[tag]))
for item in deleted:
tag_to_add = item[0:3]
ind1, ind2 = item[3], item[4]
if tag_to_add in affected_tags and (ind1, ind2) not in affected_tags[tag_to_add]:
affected_tags[tag_to_add].append((ind1, ind2))
else:
affected_tags[tag_to_add] = [(ind1, ind2)]
write_message(" -Delete specified tags in the old record: DONE", verbose=2)
# Append new tag to the old record and update the new record with the old_record modified
if opt_mode == 'append' or opt_mode == 'correct':
record = append_new_tag_to_old_record(record, rec_old)
write_message(" -Append new tags to the old record: DONE", verbose=2)
write_message(" -Affected Tags found after comparing upload and original records: %s"%(str(affected_tags)), verbose=2)
# 005 tag should be added everytime the record is modified
# If an exiting record is modified, its 005 tag should be overwritten with a new revision value
if '005' in record:
record_delete_field(record, '005')
write_message(" Deleted the existing 005 tag.", verbose=2)
last_revision = run_sql("SELECT MAX(job_date) FROM hstRECORD WHERE id_bibrec=%s", (rec_id, ))[0][0]
if last_revision and last_revision.strftime("%Y%m%d%H%M%S.0") == now.strftime("%Y%m%d%H%M%S.0"):
## We are updating the same record within the same seconds! It's less than
## the minimal granularity. Let's pause for 1 more second to take a breath :-)
time.sleep(1)
now = datetime.now()
error = record_add_field(record, '005', controlfield_value=now.strftime("%Y%m%d%H%M%S.0"))
if error is None:
write_message(" Failed: Error during adding to 005 controlfield to record", verbose=1, stream=sys.stderr)
return (1, int(rec_id))
else:
error=None
write_message(lambda: " -Added tag 005: DONE. " + str(record_get_field_value(record, '005', '', '')), verbose=2)
# adding 005 to affected tags will delete the existing 005 entry
# and update with the latest timestamp.
if '005' not in affected_tags:
affected_tags['005'] = [(' ', ' ')]
write_message(" -Stage COMPLETED", verbose=2)
record_deleted_p = False
try:
if not record_is_valid(record):
msg = "ERROR: record is not valid"
write_message(msg, verbose=1, stream=sys.stderr)
return (1, -1, msg)
# Have a look if we have FFT tags
write_message("Stage 2: Start (Process FFT tags if exist).", verbose=2)
record_had_FFT = False
bibrecdocs = None
if extract_tag_from_record(record, 'FFT') is not None:
record_had_FFT = True
if not writing_rights_p():
msg = "ERROR: no rights to write fulltext files"
write_message(" Stage 2 failed: %s" % msg,
verbose=1, stream=sys.stderr)
raise StandardError(msg)
try:
bibrecdocs = BibRecDocs(rec_id)
record = elaborate_fft_tags(record, rec_id, opt_mode,
pretend=pretend, tmp_ids=tmp_ids,
tmp_vers=tmp_vers, bibrecdocs=bibrecdocs)
except Exception as e:
register_exception()
msg = " Stage 2 failed: ERROR: while elaborating FFT tags: %s" % e
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
if record is None:
msg = " Stage 2 failed: ERROR: while elaborating FFT tags"
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
write_message(" -Stage COMPLETED", verbose=2)
else:
write_message(" -Stage NOT NEEDED", verbose=2)
# Have a look if we have FFT tags
write_message("Stage 2B: Start (Synchronize 8564 tags).", verbose=2)
if record_had_FFT or extract_tag_from_record(record, '856') is not None:
try:
if bibrecdocs is None:
bibrecdocs = BibRecDocs(rec_id)
record = synchronize_8564(rec_id, record, record_had_FFT, bibrecdocs, pretend=pretend)
# in case if FFT is in affected list make appropriate changes
if not insert_mode_p: # because for insert, all tags are affected
if ('4', ' ') not in affected_tags.get('856', []):
if '856' not in affected_tags:
affected_tags['856'] = [('4', ' ')]
elif ('4', ' ') not in affected_tags['856']:
affected_tags['856'].append(('4', ' '))
write_message(" -Modified field list updated with FFT details: %s" % str(affected_tags), verbose=2)
except Exception as e:
register_exception(alert_admin=True)
msg = " Stage 2B failed: ERROR: while synchronizing 8564 tags: %s" % e
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
if record is None:
msg = " Stage 2B failed: ERROR: while synchronizing 8564 tags"
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
write_message(" -Stage COMPLETED", verbose=2)
else:
write_message(" -Stage NOT NEEDED", verbose=2)
write_message("Stage 3: Start (Apply fields deletion requests).", verbose=2)
write_message(lambda: " Record before deletion:\n%s" % record_xml_output(record), verbose=9)
# remove fields with __DELETE_FIELDS__
# NOTE:creating a temporary deep copy of record for iteration to avoid RunTimeError
# RuntimeError due to change in dictionary size during iteration
tmp_rec = copy.deepcopy(record)
for tag in tmp_rec:
for data_tuple in record[tag]:
if (CFG_BIBUPLOAD_DELETE_CODE, CFG_BIBUPLOAD_DELETE_VALUE) in data_tuple[0]:
# delete the tag with particular indicator pairs from original record
record_delete_field(record, tag, data_tuple[1], data_tuple[2])
write_message(lambda: " Record after cleaning up fields to be deleted:\n%s" % record_xml_output(record), verbose=9)
+ if opt_mode == 'append':
+ write_message("Stage 3b: Drop duplicate fields in append mode.", verbose=2)
+ record = record_drop_duplicate_fields(record)
+ write_message(lambda: " Record after dropping duplicate fields:\n%s" % record_xml_output(record), verbose=9)
+
# Update of the BibFmt
write_message("Stage 4: Start (Update bibfmt).", verbose=2)
updates_exist = not records_identical(record, original_record)
if updates_exist:
# if record_had_altered_bit, this must be set to true, since the
# record has been altered.
if record_had_altered_bit:
oai_provenance_fields = record_get_field_instances(record, CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[:3], CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[3], CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[4])
for oai_provenance_field in oai_provenance_fields:
for i, (code, dummy_value) in enumerate(oai_provenance_field[0]):
if code == CFG_OAI_PROVENANCE_ALTERED_SUBFIELD:
oai_provenance_field[0][i] = (code, 'true')
tmp_indicators = (CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[3] == '_' and ' ' or CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[3], CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[4] == '_' and ' ' or CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[4])
if tmp_indicators not in affected_tags.get(CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[:3], []):
if CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[:3] not in affected_tags:
affected_tags[CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[:3]] = [tmp_indicators]
else:
affected_tags[CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[:3]].append(tmp_indicators)
write_message(lambda: " Updates exists:\n%s\n!=\n%s" % (record, original_record), verbose=9)
# format the single record as xml
rec_xml_new = record_xml_output(record)
# Update bibfmt with the format xm of this record
modification_date = time.strftime('%Y-%m-%d %H:%M:%S', time.strptime(record_get_field_value(record, '005'), '%Y%m%d%H%M%S.0'))
error = update_bibfmt_format(rec_id, rec_xml_new, 'xm', modification_date, pretend=pretend)
# Fire record signals.
from invenio.base import signals
if record_had_altered_bit:
signals.record_after_update.send(
'bibupload', recid=rec_id)
else:
signals.record_after_create.send(
'bibupload', recid=rec_id)
if error == 1:
msg = " Failed: ERROR: during update_bibfmt_format 'xm'"
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
if CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE:
error = update_bibfmt_format(rec_id, marshal.dumps(record), 'recstruct', modification_date, pretend=pretend)
if error == 1:
msg = " Failed: ERROR: during update_bibfmt_format 'recstruct'"
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
if not CFG_BIBUPLOAD_DISABLE_RECORD_REVISIONS:
# archive MARCXML format of this record for version history purposes:
if insert_mode_p:
error = archive_marcxml_for_history(rec_id, affected_fields={}, pretend=pretend)
else:
error = archive_marcxml_for_history(rec_id, affected_fields=affected_tags, pretend=pretend)
if error == 1:
msg = " ERROR: Failed to archive MARCXML for history"
write_message(msg, verbose=1, stream=sys.stderr)
return (1, int(rec_id), msg)
else:
write_message(" -Archived MARCXML for history: DONE", verbose=2)
# delete some formats like HB upon record change:
if updates_exist or record_had_FFT:
for format_to_delete in CFG_BIBUPLOAD_DELETE_FORMATS:
try:
delete_bibfmt_format(rec_id, format_to_delete, pretend=pretend)
except:
# OK, some formats like HB could not have been deleted, no big deal
pass
write_message(" -Stage COMPLETED", verbose=2)
## Let's assert that one and only one 005 tag is existing at this stage.
assert len(record['005']) == 1
# Update the database MetaData
write_message("Stage 5: Start (Update the database with the metadata).",
verbose=2)
if insert_mode_p:
update_database_with_metadata(record, rec_id, oai_rec_id, pretend=pretend)
write_message(" -Stage COMPLETED", verbose=2)
elif opt_mode in ('replace', 'replace_or_insert',
'append', 'correct', 'delete') and updates_exist:
# now we clear all the rows from bibrec_bibxxx from the old
record_deleted_p = True
delete_bibrec_bibxxx(rec_old, rec_id, affected_tags, pretend=pretend)
# metadata update will insert tags that are available in affected_tags.
# but for delete, once the tags have been deleted from bibrec_bibxxx, they dont have to be inserted
# except for 005.
if is_opt_mode_delete:
tmp_affected_tags = copy.deepcopy(affected_tags)
for tag in tmp_affected_tags:
if tag != '005':
affected_tags.pop(tag)
write_message(" -Clean bibrec_bibxxx: DONE", verbose=2)
update_database_with_metadata(record, rec_id, oai_rec_id, affected_tags, pretend=pretend)
write_message(" -Stage COMPLETED", verbose=2)
else:
write_message(" -Stage NOT NEEDED in mode %s" % opt_mode,
verbose=2)
record_deleted_p = False
# Finally we update the bibrec table with the current date
write_message("Stage 6: Start (Update bibrec table with current date).",
verbose=2)
if opt_notimechange == 0 and (updates_exist or record_had_FFT):
bibrec_now = convert_datestruct_to_datetext(time.localtime())
write_message(" -Retrieved current localtime: DONE", verbose=2)
update_bibrec_date(bibrec_now, rec_id, insert_mode_p, pretend=pretend)
write_message(" -Stage COMPLETED", verbose=2)
else:
write_message(" -Stage NOT NEEDED", verbose=2)
# Increase statistics
if insert_mode_p:
stat['nb_records_inserted'] += 1
else:
stat['nb_records_updated'] += 1
# Upload of this record finish
write_message("Record "+str(rec_id)+" DONE", verbose=1)
return (0, int(rec_id), "")
finally:
if record_deleted_p:
## BibUpload has failed living the record deleted. We should
## back the original record then.
update_database_with_metadata(original_record, rec_id, oai_rec_id, pretend=pretend)
write_message(" Restored original record", verbose=1, stream=sys.stderr)
def record_is_valid(record):
"""
Check if the record is valid. Currently this simply checks if the record
has exactly one rec_id.
@param record: the record
@type record: recstruct
@return: True if the record is valid
@rtype: bool
"""
rec_ids = record_get_field_values(record, tag="001")
if len(rec_ids) != 1:
write_message(" The record is not valid: it has not a single rec_id: %s" % (rec_ids), stream=sys.stderr)
return False
return True
def find_record_ids_by_oai_id(oaiId):
"""
A method finding the records identifier provided the oai identifier
returns a list of identifiers matching a given oai identifier
"""
# Is this record already in invenio (matching by oaiid)
if oaiId:
recids = search_pattern(p=oaiId, f=CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG, m='e')
# Is this record already in invenio (matching by reportnumber i.e.
# particularly 037. Idea: to avoid double insertions)
repnumber = oaiId.split(":")[-1]
if repnumber:
recids |= search_pattern(p = repnumber,
f = "reportnumber",
m = 'e' )
# Is this record already in invenio (matching by reportnumber i.e.
# particularly 037. Idea: to avoid double insertions)
repnumber = "arXiv:" + oaiId.split(":")[-1]
recids |= search_pattern(p = repnumber,
f = "reportnumber",
m = 'e' )
if CFG_BIBUPLOAD_MATCH_DELETED_RECORDS:
return recids
else:
if CFG_CERN_SITE:
return recids - (search_pattern(p='DELETED', f='980__%', m='e') | search_pattern(p='DUMMY', f='980__%', m='e'))
else:
return recids - search_pattern(p='DELETED', f='980__%', m='e')
else:
return intbitset()
def bibupload_post_phase(record, mode=None, rec_id="", pretend=False,
tmp_ids=None, tmp_vers=None):
def _elaborate_tag(record, tag, fun):
if extract_tag_from_record(record, tag) is not None:
try:
record = fun()
except Exception as e:
register_exception()
write_message(" Stage failed: ERROR: while elaborating %s tags: %s" % (tag, e),
verbose=1, stream=sys.stderr)
return (1, int(rec_id)) # TODO: ?
if record is None:
write_message(" Stage failed: ERROR: while elaborating %s tags" % (tag, ),
verbose=1, stream=sys.stderr)
return (1, int(rec_id))
write_message(" -Stage COMPLETED", verbose=2)
else:
write_message(" -Stage NOT NEEDED", verbose=2)
if tmp_ids is None:
tmp_ids = {}
if tmp_vers is None:
tmp_vers = {}
_elaborate_tag(record, "BDR", lambda: elaborate_brt_tags(record, rec_id = rec_id,
mode = mode,
pretend = pretend,
tmp_ids = tmp_ids,
tmp_vers = tmp_vers))
_elaborate_tag(record, "BDM", lambda: elaborate_mit_tags(record, rec_id = rec_id,
mode = mode,
pretend = pretend,
tmp_ids = tmp_ids,
tmp_vers = tmp_vers))
def submit_ticket_for_holding_pen(rec_id, err, msg, pretend=False):
"""
Submit a ticket via BibCatalog to report about a record that has been put
into the Holding Pen.
@rec_id: the affected record
@err: the corresponding Exception
msg: verbose message
"""
from invenio.legacy.bibsched import bibtask
from invenio.legacy.webuser import get_email_from_username, get_uid_from_email
user = task_get_task_param("user")
uid = None
if user:
try:
uid = get_uid_from_email(get_email_from_username(user))
except Exception as err:
write_message("WARNING: can't reliably retrieve uid for user %s: %s" % (user, err), stream=sys.stderr)
if check_bibcatalog():
text = """
%(msg)s found for record %(rec_id)s: %(err)s
See: <%(siteurl)s/record/edit/#state=edit&recid=%(rec_id)s>
BibUpload task information:
task_id: %(task_id)s
task_specific_name: %(task_specific_name)s
user: %(user)s
task_params: %(task_params)s
task_options: %(task_options)s""" % {
"msg": msg,
"rec_id": rec_id,
"err": err,
"siteurl": CFG_SITE_SECURE_URL,
"task_id": task_get_task_param("task_id"),
"task_specific_name": task_get_task_param("task_specific_name"),
"user": user,
"task_params": bibtask._TASK_PARAMS,
"task_options": bibtask._OPTIONS}
if not pretend:
BIBCATALOG_SYSTEM.ticket_submit(subject="%s: %s by %s" % (msg, rec_id, user), recordid=rec_id, text=text, queue=CFG_BIBUPLOAD_CONFLICTING_REVISION_TICKET_QUEUE, owner=uid)
def insert_record_into_holding_pen(record, oai_id, pretend=False):
query = "INSERT INTO bibHOLDINGPEN (oai_id, changeset_date, changeset_xml, id_bibrec) VALUES (%s, NOW(), %s, %s)"
xml_record = record_xml_output(record)
bibrec_ids = find_record_ids_by_oai_id(oai_id) # here determining the identifier of the record
if len(bibrec_ids) > 0:
bibrec_id = bibrec_ids.pop()
else:
# id not found by using the oai_id, let's use a wider search based
# on any information we might have.
bibrec_id = retrieve_rec_id(record, 'holdingpen', pretend=pretend)
if bibrec_id is None:
bibrec_id = 0
if not pretend:
run_sql(query, (oai_id, compress(xml_record), bibrec_id))
# record_id is logged as 0! ( We are not inserting into the main database)
log_record_uploading(oai_id, task_get_task_param('task_id', 0), 0, 'H', pretend=pretend)
stat['nb_holdingpen'] += 1
def print_out_bibupload_statistics():
"""Print the statistics of the process"""
out = "Task stats: %(nb_input)d input records, %(nb_updated)d updated, " \
"%(nb_inserted)d inserted, %(nb_errors)d errors, %(nb_holdingpen)d inserted to holding pen. " \
"Time %(nb_sec).2f sec." % { \
'nb_input': stat['nb_records_to_upload'],
'nb_updated': stat['nb_records_updated'],
'nb_inserted': stat['nb_records_inserted'],
'nb_errors': stat['nb_errors'],
'nb_holdingpen': stat['nb_holdingpen'],
'nb_sec': time.time() - time.mktime(stat['exectime']) }
write_message(out)
def open_marc_file(path):
"""Open a file and return the data"""
try:
# open the file containing the marc document
marc_file = open(path, 'r')
marc = marc_file.read()
marc_file.close()
except IOError as erro:
write_message("ERROR: %s" % erro, verbose=1, stream=sys.stderr)
if erro.errno == 2:
# No such file or directory
# Not scary
e = RecoverableError('File does not exist: %s' % path)
else:
e = StandardError('File not accessible: %s' % path)
raise e
return marc
def xml_marc_to_records(xml_marc):
"""create the records"""
# Creation of the records from the xml Marc in argument
recs = create_records(xml_marc, 1, 1)
if recs == []:
msg = "ERROR: Cannot parse MARCXML file."
write_message(msg, verbose=1, stream=sys.stderr)
raise StandardError(msg)
elif recs[0][0] is None:
msg = "ERROR: MARCXML file has wrong format: %s" % recs
write_message(msg, verbose=1, stream=sys.stderr)
raise RecoverableError(msg)
else:
recs = map((lambda x:x[0]), recs)
return recs
def find_record_format(rec_id, bibformat):
"""Look whether record REC_ID is formatted in FORMAT,
i.e. whether FORMAT exists in the bibfmt table for this record.
Return the number of times it is formatted: 0 if not, 1 if yes,
2 if found more than once (should never occur).
"""
out = 0
query = """SELECT COUNT(*) FROM bibfmt WHERE id_bibrec=%s AND format=%s"""
params = (rec_id, bibformat)
res = []
res = run_sql(query, params)
out = res[0][0]
return out
def find_record_from_recid(rec_id):
"""
Try to find record in the database from the REC_ID number.
Return record ID if found, None otherwise.
"""
res = run_sql("SELECT id FROM bibrec WHERE id=%s",
(rec_id,))
if res:
return res[0][0]
else:
return None
def find_record_from_sysno(sysno):
"""
Try to find record in the database from the external SYSNO number.
Return record ID if found, None otherwise.
"""
bibxxx = 'bib'+CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG[0:2]+'x'
bibrec_bibxxx = 'bibrec_' + bibxxx
res = run_sql("""SELECT bb.id_bibrec FROM %(bibrec_bibxxx)s AS bb,
%(bibxxx)s AS b WHERE b.tag=%%s AND b.value=%%s
AND bb.id_bibxxx=b.id""" %
{'bibxxx': bibxxx,
'bibrec_bibxxx': bibrec_bibxxx},
(CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG, sysno,))
for recid in res:
if CFG_BIBUPLOAD_MATCH_DELETED_RECORDS:
return recid[0]
else:
if record_exists(recid[0]) > 0: ## Only non deleted records
return recid[0]
return None
def find_records_from_extoaiid(extoaiid, extoaisrc=None):
"""
Try to find records in the database from the external EXTOAIID number.
Return list of record ID if found, None otherwise.
"""
assert(CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[:5] == CFG_BIBUPLOAD_EXTERNAL_OAIID_PROVENANCE_TAG[:5])
bibxxx = 'bib'+CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[0:2]+'x'
bibrec_bibxxx = 'bibrec_' + bibxxx
write_message(' Looking for extoaiid="%s" with extoaisrc="%s"' % (extoaiid, extoaisrc), verbose=9)
id_bibrecs = intbitset(run_sql("""SELECT bb.id_bibrec FROM %(bibrec_bibxxx)s AS bb,
%(bibxxx)s AS b WHERE b.tag=%%s AND b.value=%%s
AND bb.id_bibxxx=b.id""" %
{'bibxxx': bibxxx,
'bibrec_bibxxx': bibrec_bibxxx},
(CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG, extoaiid,)))
write_message(' Partially found %s for extoaiid="%s"' % (id_bibrecs, extoaiid), verbose=9)
ret = intbitset()
for id_bibrec in id_bibrecs:
if not CFG_BIBUPLOAD_MATCH_DELETED_RECORDS:
if record_exists(id_bibrec) < 1:
## We don't match not existing records
continue
record = get_record(id_bibrec)
instances = record_get_field_instances(record, CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[0:3], CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[3], CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[4])
write_message(' recid %s -> instances "%s"' % (id_bibrec, instances), verbose=9)
for instance in instances:
this_extoaisrc = field_get_subfield_values(instance, CFG_BIBUPLOAD_EXTERNAL_OAIID_PROVENANCE_TAG[5])
this_extoaisrc = this_extoaisrc and this_extoaisrc[0] or None
this_extoaiid = field_get_subfield_values(instance, CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[5])
this_extoaiid = this_extoaiid and this_extoaiid[0] or None
write_message(" this_extoaisrc -> %s, this_extoaiid -> %s" % (this_extoaisrc, this_extoaiid), verbose=9)
if this_extoaiid == extoaiid:
write_message(' recid %s -> provenance "%s"' % (id_bibrec, this_extoaisrc), verbose=9)
if this_extoaisrc == extoaisrc:
write_message('Found recid %s for extoaiid="%s" with provenance="%s"' % (id_bibrec, extoaiid, extoaisrc), verbose=9)
ret.add(id_bibrec)
break
if this_extoaisrc is None:
write_message('WARNING: Found recid %s for extoaiid="%s" that doesn\'t specify any provenance, while input record does.' % (id_bibrec, extoaiid), stream=sys.stderr)
if extoaisrc is None:
write_message('WARNING: Found recid %s for extoaiid="%s" that specify a provenance (%s), while input record does not have a provenance.' % (id_bibrec, extoaiid, this_extoaisrc), stream=sys.stderr)
return ret
def find_record_from_oaiid(oaiid):
"""
Try to find record in the database from the OAI ID number and OAI SRC.
Return record ID if found, None otherwise.
"""
bibxxx = 'bib'+CFG_OAI_ID_FIELD[0:2]+'x'
bibrec_bibxxx = 'bibrec_' + bibxxx
res = run_sql("""SELECT bb.id_bibrec FROM %(bibrec_bibxxx)s AS bb,
%(bibxxx)s AS b WHERE b.tag=%%s AND b.value=%%s
AND bb.id_bibxxx=b.id""" %
{'bibxxx': bibxxx,
'bibrec_bibxxx': bibrec_bibxxx},
(CFG_OAI_ID_FIELD, oaiid,))
for recid in res:
if CFG_BIBUPLOAD_MATCH_DELETED_RECORDS:
return recid[0]
else:
if record_exists(recid[0]) > 0: ## Only non deleted records
return recid[0]
return None
def find_record_from_doi(doi):
"""
Try to find record in the database from the given DOI.
Return record ID if found, None otherwise.
"""
bibxxx = 'bib02x'
bibrec_bibxxx = 'bibrec_' + bibxxx
res = run_sql("""SELECT bb.id_bibrec, bb.field_number
FROM %(bibrec_bibxxx)s AS bb, %(bibxxx)s AS b
WHERE b.tag=%%s AND b.value=%%s
AND bb.id_bibxxx=b.id""" %
{'bibxxx': bibxxx,
'bibrec_bibxxx': bibrec_bibxxx},
('0247_a', doi,))
# For each of the result, make sure that it is really tagged as doi
for (id_bibrec, field_number) in res:
if not CFG_BIBUPLOAD_MATCH_DELETED_RECORDS:
if record_exists(id_bibrec) < 1:
## We don't match not existing records
continue
res = run_sql("""SELECT bb.id_bibrec
FROM %(bibrec_bibxxx)s AS bb, %(bibxxx)s AS b
WHERE b.tag=%%s AND b.value=%%s
AND bb.id_bibxxx=b.id and bb.field_number=%%s and bb.id_bibrec=%%s""" %
{'bibxxx': bibxxx,
'bibrec_bibxxx': bibrec_bibxxx},
('0247_2', "doi", field_number, id_bibrec))
if res and res[0][0] == id_bibrec:
return res[0][0]
return None
def extract_tag_from_record(record, tag_number):
""" Extract the tag_number for record."""
# first step verify if the record is not already in the database
if record:
return record.get(tag_number, None)
return None
def retrieve_rec_id(record, opt_mode, pretend=False, post_phase = False):
"""Retrieve the record Id from a record by using tag 001 or SYSNO or OAI ID or DOI
tag. opt_mod is the desired mode.
@param post_phase Tells if we are calling this method in the postprocessing phase. If true, we accept presence of
001 fields even in the insert mode
@type post_phase boolean
"""
rec_id = None
# 1st step: we look for the tag 001
tag_001 = extract_tag_from_record(record, '001')
if tag_001 is not None:
# We extract the record ID from the tag
rec_id = tag_001[0][3]
# if we are in insert mode => error
if opt_mode == 'insert' and not post_phase:
write_message(" Failed: tag 001 found in the xml"
" submitted, you should use the option replace,"
" correct or append to replace an existing"
" record. (-h for help)",
verbose=1, stream=sys.stderr)
return -1
else:
# we found the rec id and we are not in insert mode => continue
# we try to match rec_id against the database:
if find_record_from_recid(rec_id) is not None:
# okay, 001 corresponds to some known record
return int(rec_id)
elif opt_mode in ('replace', 'replace_or_insert'):
if task_get_option('force'):
# we found the rec_id but it's not in the system and we are
# requested to replace records. Therefore we create on the fly
# a empty record allocating the recid.
write_message(" Warning: tag 001 found in the xml with"
" value %(rec_id)s, but rec_id %(rec_id)s does"
" not exist. Since the mode replace was"
" requested the rec_id %(rec_id)s is allocated"
" on-the-fly." % {"rec_id": rec_id},
stream=sys.stderr)
return create_new_record(rec_id=rec_id, pretend=pretend)
else:
# Since --force was not used we are going to raise an error
write_message(" Failed: tag 001 found in the xml"
" submitted with value %(rec_id)s. The"
" corresponding record however does not"
" exists. If you want to really create"
" such record, please use the --force"
" parameter when calling bibupload." % {
"rec_id": rec_id}, stream=sys.stderr)
return -1
else:
# The record doesn't exist yet. We shall have try to check
# the SYSNO or OAI or DOI id later.
write_message(" -Tag 001 value not found in database.",
verbose=9)
rec_id = None
else:
write_message(" -Tag 001 not found in the xml marc file.", verbose=9)
if rec_id is None:
# 2nd step we look for the SYSNO
sysnos = record_get_field_values(record,
CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG[0:3],
CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG[3:4] != "_" and \
CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG[3:4] or "",
CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG[4:5] != "_" and \
CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG[4:5] or "",
CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG[5:6])
if sysnos:
sysno = sysnos[0] # there should be only one external SYSNO
write_message(" -Checking if SYSNO " + sysno + \
" exists in the database", verbose=9)
# try to find the corresponding rec id from the database
rec_id = find_record_from_sysno(sysno)
if rec_id is not None:
# rec_id found
pass
else:
# The record doesn't exist yet. We will try to check
# external and internal OAI ids later.
write_message(" -Tag SYSNO value not found in database.",
verbose=9)
rec_id = None
else:
write_message(" -Tag SYSNO not found in the xml marc file.",
verbose=9)
if rec_id is None:
# 2nd step we look for the external OAIID
extoai_fields = record_get_field_instances(record,
CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[0:3],
CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[3:4] != "_" and \
CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[3:4] or "",
CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[4:5] != "_" and \
CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[4:5] or "")
if extoai_fields:
for field in extoai_fields:
extoaiid = field_get_subfield_values(field, CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[5:6])
extoaisrc = field_get_subfield_values(field, CFG_BIBUPLOAD_EXTERNAL_OAIID_PROVENANCE_TAG[5:6])
if extoaiid:
extoaiid = extoaiid[0]
if extoaisrc:
extoaisrc = extoaisrc[0]
else:
extoaisrc = None
write_message(" -Checking if EXTOAIID %s (%s) exists in the database" % (extoaiid, extoaisrc), verbose=9)
# try to find the corresponding rec id from the database
rec_ids = find_records_from_extoaiid(extoaiid, extoaisrc)
if rec_ids:
# rec_id found
rec_id = rec_ids.pop()
break
else:
# The record doesn't exist yet. We will try to check
# OAI id later.
write_message(" -Tag EXTOAIID value not found in database.",
verbose=9)
rec_id = None
else:
write_message(" -Tag EXTOAIID not found in the xml marc file.", verbose=9)
if rec_id is None:
# 4th step we look for the OAI ID
oaiidvalues = record_get_field_values(record,
CFG_OAI_ID_FIELD[0:3],
CFG_OAI_ID_FIELD[3:4] != "_" and \
CFG_OAI_ID_FIELD[3:4] or "",
CFG_OAI_ID_FIELD[4:5] != "_" and \
CFG_OAI_ID_FIELD[4:5] or "",
CFG_OAI_ID_FIELD[5:6])
if oaiidvalues:
oaiid = oaiidvalues[0] # there should be only one OAI ID
write_message(" -Check if local OAI ID " + oaiid + \
" exist in the database", verbose=9)
# try to find the corresponding rec id from the database
rec_id = find_record_from_oaiid(oaiid)
if rec_id is not None:
# rec_id found
pass
else:
write_message(" -Tag OAI ID value not found in database.",
verbose=9)
rec_id = None
else:
write_message(" -Tag SYSNO not found in the xml marc file.",
verbose=9)
if rec_id is None:
# 5th step we look for the DOI.
record_dois = record_extract_dois(record)
matching_recids = set()
if record_dois:
# try to find the corresponding rec id from the database
for record_doi in record_dois:
possible_recid = find_record_from_doi(record_doi)
if possible_recid:
matching_recids.add(possible_recid)
if len(matching_recids) > 1:
# Oops, this record refers to DOI existing in multiple records.
# Dunno which one to choose.
write_message(" Failed: Multiple records found in the" \
" database %s that match the DOI(s) in the input" \
" MARCXML %s" % (repr(matching_recids), repr(record_dois)),
verbose=1, stream=sys.stderr)
return -1
elif len(matching_recids) == 1:
rec_id = matching_recids.pop()
if opt_mode == 'insert':
write_message(" Failed: DOI tag matching record #%s found in the xml" \
" submitted, you should use the option replace," \
" correct or append to replace an existing" \
" record. (-h for help)" % rec_id,
verbose=1, stream=sys.stderr)
return -1
else:
write_message(" - Tag DOI value not found in database.",
verbose=9)
rec_id = None
else:
write_message(" -Tag DOI not found in the xml marc file.",
verbose=9)
# Now we should have detected rec_id from SYSNO or OAIID
# tags. (None otherwise.)
if rec_id:
if opt_mode == 'insert':
write_message(" Failed: Record found in the database," \
" you should use the option replace," \
" correct or append to replace an existing" \
" record. (-h for help)",
verbose=1, stream=sys.stderr)
return -1
else:
if opt_mode != 'insert' and \
opt_mode != 'replace_or_insert':
write_message(" Failed: Record not found in the database."\
" Please insert the file before updating it."\
" (-h for help)", verbose=1, stream=sys.stderr)
return -1
return rec_id and int(rec_id) or None
def check_record_doi_is_unique(rec_id, record):
"""
Check that DOI found in 'record' does not exist in any other
record than 'recid'.
Return (boolean, msg) where 'boolean' would be True if the DOI is
unique.
"""
record_dois = record_extract_dois(record)
if record_dois:
matching_recids = set()
for record_doi in record_dois:
possible_recid = find_record_from_doi(record_doi)
if possible_recid:
matching_recids.add(possible_recid)
if len(matching_recids) > 1:
# Oops, this record refers to DOI existing in multiple records.
msg = " Failed: Multiple records found in the" \
" database %s that match the DOI(s) in the input" \
" MARCXML %s" % (repr(matching_recids), repr(record_dois))
return (False, msg)
elif len(matching_recids) == 1:
matching_recid = matching_recids.pop()
if str(matching_recid) != str(rec_id):
# Oops, this record refers to DOI existing in a different record.
msg = " Failed: DOI(s) %s found in this record (#%s)" \
" already exist(s) in another other record (#%s)" % \
(repr(record_dois), rec_id, matching_recid)
return (False, msg)
return (True, "")
### Insert functions
def create_new_record(rec_id=None, pretend=False):
"""
Create new record in the database
@param rec_id: if specified the new record will have this rec_id.
@type rec_id: int
@return: the allocated rec_id
@rtype: int
@note: in case of errors will be returned None
"""
if rec_id is not None:
try:
rec_id = int(rec_id)
except (ValueError, TypeError) as error:
write_message(" ERROR: during the creation_new_record function: %s "
% error, verbose=1, stream=sys.stderr)
return None
if run_sql("SELECT id FROM bibrec WHERE id=%s", (rec_id, )):
write_message(" ERROR: during the creation_new_record function: the requested rec_id %s already exists." % rec_id)
return None
if pretend:
if rec_id:
return rec_id
else:
return run_sql("SELECT max(id)+1 FROM bibrec")[0][0]
if rec_id is not None:
return run_sql("INSERT INTO bibrec (id, creation_date, modification_date) VALUES (%s, NOW(), NOW())", (rec_id, ))
else:
return run_sql("INSERT INTO bibrec (creation_date, modification_date) VALUES (NOW(), NOW())")
def insert_bibfmt(id_bibrec, marc, bibformat, modification_date='1970-01-01 00:00:00', pretend=False):
"""Insert the format in the table bibfmt"""
# compress the marc value
pickled_marc = compress(marc)
try:
time.strptime(modification_date, "%Y-%m-%d %H:%M:%S")
except ValueError:
modification_date = '1970-01-01 00:00:00'
query = """INSERT LOW_PRIORITY INTO bibfmt (id_bibrec, format, last_updated, value)
VALUES (%s, %s, %s, %s)"""
if not pretend:
row_id = run_sql(query, (id_bibrec, bibformat, modification_date, pickled_marc))
return row_id
else:
return 1
def insert_record_bibxxx(tag, value, pretend=False):
"""Insert the record into bibxxx"""
# determine into which table one should insert the record
table_name = 'bib'+tag[0:2]+'x'
# check if the tag, value combination exists in the table
query = """SELECT id,value FROM %s """ % table_name
query += """ WHERE tag=%s AND value=%s"""
params = (tag, value)
res = None
res = run_sql(query, params)
# Note: compare now the found values one by one and look for
# string binary equality (e.g. to respect lowercase/uppercase
# match), regardless of the charset etc settings. Ideally we
# could use a BINARY operator in the above SELECT statement, but
# we would have to check compatibility on various MySQLdb versions
# etc; this approach checks all matched values in Python, not in
# MySQL, which is less cool, but more conservative, so it should
# work better on most setups.
if res:
for row in res:
row_id = row[0]
row_value = row[1]
if row_value == value:
return (table_name, row_id)
# We got here only when the tag, value combination was not found,
# so it is now necessary to insert the tag, value combination into
# bibxxx table as new.
query = """INSERT INTO %s """ % table_name
query += """ (tag, value) values (%s , %s)"""
params = (tag, value)
if not pretend:
row_id = run_sql(query, params)
else:
return (table_name, 1)
return (table_name, row_id)
def insert_record_bibrec_bibxxx(table_name, id_bibxxx,
field_number, id_bibrec, pretend=False):
"""Insert the record into bibrec_bibxxx"""
# determine into which table one should insert the record
full_table_name = 'bibrec_'+ table_name
# insert the proper row into the table
query = """INSERT INTO %s """ % full_table_name
query += """(id_bibrec,id_bibxxx, field_number) values (%s , %s, %s)"""
params = (id_bibrec, id_bibxxx, field_number)
if not pretend:
res = run_sql(query, params)
else:
return 1
return res
def synchronize_8564(rec_id, record, record_had_FFT, bibrecdocs, pretend=False):
"""
Synchronize 8564_ tags and BibDocFile tables.
This function directly manipulate the record parameter.
@type rec_id: positive integer
@param rec_id: the record identifier.
@param record: the record structure as created by bibrecord.create_record
@type record_had_FFT: boolean
@param record_had_FFT: True if the incoming bibuploaded-record used FFT
@return: the manipulated record (which is also modified as a side effect)
"""
def merge_marc_into_bibdocfile(field, pretend=False):
"""
Internal function that reads a single field and stores its content
in BibDocFile tables.
@param field: the 8564_ field containing a BibDocFile URL.
"""
write_message('Merging field: %s' % (field, ), verbose=9)
url = field_get_subfield_values(field, 'u')[:1] or field_get_subfield_values(field, 'q')[:1]
description = field_get_subfield_values(field, 'y')[:1]
comment = field_get_subfield_values(field, 'z')[:1]
if url:
recid, docname, docformat = decompose_bibdocfile_url(url[0])
if recid != rec_id:
write_message("INFO: URL %s is not pointing to a fulltext owned by this record (%s)" % (url, recid), stream=sys.stderr)
else:
try:
bibdoc = bibrecdocs.get_bibdoc(docname)
if description and not pretend:
bibdoc.set_description(description[0], docformat)
if comment and not pretend:
bibdoc.set_comment(comment[0], docformat)
except InvenioBibDocFileError:
## Apparently the referenced docname doesn't exist anymore.
## Too bad. Let's skip it.
write_message("WARNING: docname %s does not seem to exist for record %s. Has it been renamed outside FFT?" % (docname, recid), stream=sys.stderr)
def merge_bibdocfile_into_marc(field, subfields):
"""
Internal function that reads BibDocFile table entries referenced by
the URL in the given 8564_ field and integrate the given information
directly with the provided subfields.
@param field: the 8564_ field containing a BibDocFile URL.
@param subfields: the subfields corresponding to the BibDocFile URL
generated after BibDocFile tables.
"""
write_message('Merging subfields %s into field %s' % (subfields, field), verbose=9)
subfields = dict(subfields) ## We make a copy not to have side-effects
subfield_to_delete = []
for subfield_position, (code, value) in enumerate(field_get_subfield_instances(field)):
## For each subfield instance already existing...
if code in subfields:
## ...We substitute it with what is in BibDocFile tables
record_modify_subfield(record, '856', code, subfields[code],
subfield_position, field_position_global=field[4])
del subfields[code]
else:
## ...We delete it otherwise
subfield_to_delete.append(subfield_position)
subfield_to_delete.sort()
for counter, position in enumerate(subfield_to_delete):
## FIXME: Very hackish algorithm. Since deleting a subfield
## will alterate the position of following subfields, we
## are taking note of this and adjusting further position
## by using a counter.
record_delete_subfield_from(record, '856', position - counter,
field_position_global=field[4])
subfields = subfields.items()
subfields.sort()
for code, value in subfields:
## Let's add non-previously existing subfields
record_add_subfield_into(record, '856', code, value,
field_position_global=field[4])
def get_bibdocfile_managed_info():
"""
Internal function, returns a dictionary of
BibDocFile URL -> wanna-be subfields.
This information is retrieved from internal BibDoc
structures rather than from input MARC XML files
@rtype: mapping
@return: BibDocFile URL -> wanna-be subfields dictionary
"""
ret = {}
latest_files = bibrecdocs.list_latest_files(list_hidden=False)
for afile in latest_files:
url = afile.get_url()
ret[url] = {'u': url}
description = afile.get_description()
comment = afile.get_comment()
subformat = afile.get_subformat()
size = afile.get_size()
if description:
ret[url]['y'] = description
if comment:
ret[url]['z'] = comment
if subformat:
ret[url]['x'] = subformat
ret[url]['s'] = str(size)
return ret
write_message("Synchronizing MARC of recid '%s' with:\n%s" % (rec_id, record), verbose=9)
tags856s = record_get_field_instances(record, '856', '%', '%')
write_message("Original 856%% instances: %s" % tags856s, verbose=9)
tags8564s_to_add = get_bibdocfile_managed_info()
write_message("BibDocFile instances: %s" % tags8564s_to_add, verbose=9)
positions_tags8564s_to_remove = []
for local_position, field in enumerate(tags856s):
if field[1] == '4' and field[2] == ' ':
write_message('Analysing %s' % (field, ), verbose=9)
for url in field_get_subfield_values(field, 'u') + field_get_subfield_values(field, 'q'):
if url in tags8564s_to_add:
# there exists a link in the MARC of the record and the connection exists in BibDoc tables
if record_had_FFT:
merge_bibdocfile_into_marc(field, tags8564s_to_add[url])
else:
merge_marc_into_bibdocfile(field, pretend=pretend)
del tags8564s_to_add[url]
break
elif bibdocfile_url_p(url) and decompose_bibdocfile_url(url)[0] == rec_id:
# The link exists and is potentially correct-looking link to a document
# moreover, it refers to current record id ... but it does not exist in
# internal BibDoc structures. This could have happen in the case of renaming a document
# or its removal. In both cases we have to remove link... a new one will be created
positions_tags8564s_to_remove.append(local_position)
write_message("%s to be deleted and re-synchronized" % (field, ), verbose=9)
break
record_delete_fields(record, '856', positions_tags8564s_to_remove)
tags8564s_to_add = tags8564s_to_add.values()
tags8564s_to_add.sort()
## FIXME: we are not yet able to preserve the sorting
## of 8564 tags WRT FFT in BibUpload.
## See ticket #1606.
for subfields in tags8564s_to_add:
subfields = subfields.items()
subfields.sort()
record_add_field(record, '856', '4', ' ', subfields=subfields)
write_message('Final record: %s' % record, verbose=9)
return record
def _get_subfield_value(field, subfield_code, default=None):
res = field_get_subfield_values(field, subfield_code)
if res != [] and res != None:
return res[0]
else:
return default
def elaborate_mit_tags(record, rec_id, mode, pretend = False, tmp_ids = {},
tmp_vers = {}):
"""
Uploading MoreInfo -> BDM tags
"""
tuple_list = extract_tag_from_record(record, 'BDM')
# Now gathering information from BDR tags - to be processed later
write_message("Processing BDM entries of the record ")
recordDocs = BibRecDocs(rec_id)
if tuple_list:
for mit in record_get_field_instances(record, 'BDM', ' ', ' '):
relation_id = _get_subfield_value(mit, "r")
bibdoc_id = _get_subfield_value(mit, "i")
# checking for a possibly temporary ID
if not (bibdoc_id is None):
bibdoc_id = resolve_identifier(tmp_ids, bibdoc_id)
bibdoc_ver = _get_subfield_value(mit, "v")
if not (bibdoc_ver is None):
bibdoc_ver = resolve_identifier(tmp_vers, bibdoc_ver)
bibdoc_name = _get_subfield_value(mit, "n")
bibdoc_fmt = _get_subfield_value(mit, "f")
moreinfo_str = _get_subfield_value(mit, "m")
if bibdoc_id == None:
if bibdoc_name == None:
raise StandardError("Incorrect relation. Neither name nor identifier of the first obejct has been specified")
else:
# retrieving the ID based on the document name (inside current record)
# The document is attached to current record.
try:
bibdoc_id = recordDocs.get_docid(bibdoc_name)
except:
raise StandardError("BibDoc of a name %s does not exist within a record" % (bibdoc_name, ))
else:
if bibdoc_name != None:
write_message("WARNING: both name and id of the first document of a relation have been specified. Ignoring the name", stream=sys.stderr)
if (moreinfo_str is None or mode in ("replace", "correct")) and (not pretend):
MoreInfo(docid=bibdoc_id , version = bibdoc_ver,
docformat = bibdoc_fmt, relation = relation_id).delete()
if (not moreinfo_str is None) and (not pretend):
MoreInfo.create_from_serialised(moreinfo_str,
docid=bibdoc_id,
version = bibdoc_ver,
docformat = bibdoc_fmt,
relation = relation_id)
return record
def elaborate_brt_tags(record, rec_id, mode, pretend=False, tmp_ids = {}, tmp_vers = {}):
"""
Process BDR tags describing relations between existing objects
"""
tuple_list = extract_tag_from_record(record, 'BDR')
# Now gathering information from BDR tags - to be processed later
relations_to_create = []
write_message("Processing BDR entries of the record ")
recordDocs = BibRecDocs(rec_id) #TODO: check what happens if there is no record yet ! Will the class represent an empty set?
if tuple_list:
for brt in record_get_field_instances(record, 'BDR', ' ', ' '):
relation_id = _get_subfield_value(brt, "r")
bibdoc1_id = None
bibdoc1_name = None
bibdoc1_ver = None
bibdoc1_fmt = None
bibdoc2_id = None
bibdoc2_name = None
bibdoc2_ver = None
bibdoc2_fmt = None
if not relation_id:
bibdoc1_id = _get_subfield_value(brt, "i")
bibdoc1_name = _get_subfield_value(brt, "n")
if bibdoc1_id == None:
if bibdoc1_name == None:
raise StandardError("Incorrect relation. Neither name nor identifier of the first obejct has been specified")
else:
# retrieving the ID based on the document name (inside current record)
# The document is attached to current record.
try:
bibdoc1_id = recordDocs.get_docid(bibdoc1_name)
except:
raise StandardError("BibDoc of a name %s does not exist within a record" % \
(bibdoc1_name, ))
else:
# resolving temporary identifier
bibdoc1_id = resolve_identifier(tmp_ids, bibdoc1_id)
if bibdoc1_name != None:
write_message("WARNING: both name and id of the first document of a relation have been specified. Ignoring the name", stream=sys.stderr)
bibdoc1_ver = _get_subfield_value(brt, "v")
if not (bibdoc1_ver is None):
bibdoc1_ver = resolve_identifier(tmp_vers, bibdoc1_ver)
bibdoc1_fmt = _get_subfield_value(brt, "f")
bibdoc2_id = _get_subfield_value(brt, "j")
bibdoc2_name = _get_subfield_value(brt, "o")
if bibdoc2_id == None:
if bibdoc2_name == None:
raise StandardError("Incorrect relation. Neither name nor identifier of the second obejct has been specified")
else:
# retrieving the ID based on the document name (inside current record)
# The document is attached to current record.
try:
bibdoc2_id = recordDocs.get_docid(bibdoc2_name)
except:
raise StandardError("BibDoc of a name %s does not exist within a record" % (bibdoc2_name, ))
else:
bibdoc2_id = resolve_identifier(tmp_ids, bibdoc2_id)
if bibdoc2_name != None:
write_message("WARNING: both name and id of the first document of a relation have been specified. Ignoring the name", stream=sys.stderr)
bibdoc2_ver = _get_subfield_value(brt, "w")
if not (bibdoc2_ver is None):
bibdoc2_ver = resolve_identifier(tmp_vers, bibdoc2_ver)
bibdoc2_fmt = _get_subfield_value(brt, "g")
control_command = _get_subfield_value(brt, "d")
relation_type = _get_subfield_value(brt, "t")
if not relation_type and not relation_id:
raise StandardError("The relation type must be specified")
more_info = _get_subfield_value(brt, "m")
# the relation id might be specified in the case of updating
# MoreInfo table instead of other fields
rel_obj = None
if not relation_id:
rels = BibRelation.get_relations(rel_type = relation_type,
bibdoc1_id = bibdoc1_id,
bibdoc2_id = bibdoc2_id,
bibdoc1_ver = bibdoc1_ver,
bibdoc2_ver = bibdoc2_ver,
bibdoc1_fmt = bibdoc1_fmt,
bibdoc2_fmt = bibdoc2_fmt)
if len(rels) > 0:
rel_obj = rels[0]
relation_id = rel_obj.id
else:
rel_obj = BibRelation(rel_id=relation_id)
relations_to_create.append((relation_id, bibdoc1_id, bibdoc1_ver,
bibdoc1_fmt, bibdoc2_id, bibdoc2_ver,
bibdoc2_fmt, relation_type, more_info,
rel_obj, control_command))
record_delete_field(record, 'BDR', ' ', ' ')
if mode in ("insert", "replace_or_insert", "append", "correct", "replace"):
# now creating relations between objects based on the data
if not pretend:
for (relation_id, bibdoc1_id, bibdoc1_ver, bibdoc1_fmt,
bibdoc2_id, bibdoc2_ver, bibdoc2_fmt, rel_type,
more_info, rel_obj, control_command) in relations_to_create:
if rel_obj == None:
rel_obj = BibRelation.create(bibdoc1_id = bibdoc1_id,
bibdoc1_ver = bibdoc1_ver,
bibdoc1_fmt = bibdoc1_fmt,
bibdoc2_id = bibdoc2_id,
bibdoc2_ver = bibdoc2_ver,
bibdoc2_fmt = bibdoc2_fmt,
rel_type = rel_type)
relation_id = rel_obj.id
if mode in ("replace"):
# Clearing existing MoreInfo content
rel_obj.get_more_info().delete()
if more_info:
MoreInfo.create_from_serialised(more_info, relation = relation_id)
if control_command == "DELETE":
rel_obj.delete()
else:
write_message("BDR tag is not processed in the %s mode" % (mode, ))
return record
def elaborate_fft_tags(record, rec_id, mode, pretend=False,
tmp_ids = {}, tmp_vers = {}, bibrecdocs=None):
"""
Process FFT tags that should contain $a with file pathes or URLs
to get the fulltext from. This function enriches record with
proper 8564 URL tags, downloads fulltext files and stores them
into var/data structure where appropriate.
CFG_BIBUPLOAD_WGET_SLEEP_TIME defines time to sleep in seconds in
between URL downloads.
Note: if an FFT tag contains multiple $a subfields, we upload them
into different 856 URL tags in the metadata. See regression test
case test_multiple_fft_insert_via_http().
"""
# Let's define some handy sub procedure.
def _add_new_format(bibdoc, url, docformat, docname, doctype, newname, description, comment, flags, modification_date, pretend=False):
"""Adds a new format for a given bibdoc. Returns True when everything's fine."""
write_message('Add new format to %s url: %s, format: %s, docname: %s, doctype: %s, newname: %s, description: %s, comment: %s, flags: %s, modification_date: %s' % (repr(bibdoc), url, docformat, docname, doctype, newname, description, comment, flags, modification_date), verbose=9)
try:
if not url: # Not requesting a new url. Just updating comment & description
return _update_description_and_comment(bibdoc, docname, docformat, description, comment, flags, pretend=pretend)
try:
if not pretend:
bibdoc.add_file_new_format(url, description=description, comment=comment, flags=flags, modification_date=modification_date)
except StandardError as e:
write_message("('%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s') not inserted because format already exists (%s)." % (url, docformat, docname, doctype, newname, description, comment, flags, modification_date, e), stream=sys.stderr)
raise
except Exception as e:
write_message("ERROR: in adding '%s' as a new format because of: %s" % (url, e), stream=sys.stderr)
raise
return True
def _add_new_version(bibdoc, url, docformat, docname, doctype, newname, description, comment, flags, modification_date, pretend=False):
"""Adds a new version for a given bibdoc. Returns True when everything's fine."""
write_message('Add new version to %s url: %s, format: %s, docname: %s, doctype: %s, newname: %s, description: %s, comment: %s, flags: %s' % (repr(bibdoc), url, docformat, docname, doctype, newname, description, comment, flags), verbose=9)
try:
if not url:
return _update_description_and_comment(bibdoc, docname, docformat, description, comment, flags, pretend=pretend)
try:
if not pretend:
bibdoc.add_file_new_version(url, description=description, comment=comment, flags=flags, modification_date=modification_date)
except StandardError as e:
write_message("('%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s') not inserted because '%s'." % (url, docformat, docname, doctype, newname, description, comment, flags, modification_date, e), stream=sys.stderr)
raise
except Exception as e:
write_message("ERROR: in adding '%s' as a new version because of: %s" % (url, e), stream=sys.stderr)
raise
return True
def _update_description_and_comment(bibdoc, docname, docformat, description, comment, flags, pretend=False):
"""Directly update comments and descriptions."""
write_message('Just updating description and comment for %s with format %s with description %s, comment %s and flags %s' % (docname, docformat, description, comment, flags), verbose=9)
try:
if not pretend:
bibdoc.set_description(description, docformat)
bibdoc.set_comment(comment, docformat)
for flag in CFG_BIBDOCFILE_AVAILABLE_FLAGS:
if flag in flags:
bibdoc.set_flag(flag, docformat)
else:
bibdoc.unset_flag(flag, docformat)
except StandardError as e:
write_message("('%s', '%s', '%s', '%s', '%s') description and comment not updated because '%s'." % (docname, docformat, description, comment, flags, e))
raise
return True
def _process_document_moreinfos(more_infos, docname, version, docformat, mode):
if not mode in ('correct', 'append', 'replace_or_insert', 'replace', 'insert'):
#print("exited because the mode is incorrect")
return
docid = None
try:
docid = bibrecdocs.get_docid(docname)
except:
raise StandardError("MoreInfo: No document of a given name associated with the record")
if not version:
# We have to retrieve the most recent version ...
version = bibrecdocs.get_bibdoc(docname).get_latest_version()
doc_moreinfo_s, version_moreinfo_s, version_format_moreinfo_s, format_moreinfo_s = more_infos
if mode in ("replace", "replace_or_insert"):
if doc_moreinfo_s: #only if specified, otherwise do not touch
MoreInfo(docid = docid).delete()
if format_moreinfo_s: #only if specified... otherwise do not touch
MoreInfo(docid = docid, docformat = docformat).delete()
if not doc_moreinfo_s is None:
MoreInfo.create_from_serialised(ser_str = doc_moreinfo_s, docid = docid)
if not version_moreinfo_s is None:
MoreInfo.create_from_serialised(ser_str = version_moreinfo_s,
docid = docid, version = version)
if not version_format_moreinfo_s is None:
MoreInfo.create_from_serialised(ser_str = version_format_moreinfo_s,
docid = docid, version = version,
docformat = docformat)
if not format_moreinfo_s is None:
MoreInfo.create_from_serialised(ser_str = format_moreinfo_s,
docid = docid, docformat = docformat)
if mode == 'delete':
raise StandardError('FFT tag specified but bibupload executed in --delete mode')
tuple_list = extract_tag_from_record(record, 'FFT')
if tuple_list: # FFT Tags analysis
write_message("FFTs: "+str(tuple_list), verbose=9)
docs = {} # docnames and their data
for fft in record_get_field_instances(record, 'FFT', ' ', ' '):
# Very first, we retrieve the potentially temporary odentifiers...
#even if the rest fails, we should include them in teh dictionary
version = _get_subfield_value(fft, 'v', '')
# checking if version is temporary... if so, filling a different varaible
is_tmp_ver, bibdoc_tmpver = parse_identifier(version)
if is_tmp_ver:
version = None
else:
bibdoc_tmpver = None
if not version: #treating cases of empty string etc...
version = None
bibdoc_tmpid = field_get_subfield_values(fft, 'i')
if bibdoc_tmpid:
bibdoc_tmpid = bibdoc_tmpid[0]
else:
bibdoc_tmpid
is_tmp_id, bibdoc_tmpid = parse_identifier(bibdoc_tmpid)
if not is_tmp_id:
bibdoc_tmpid = None
# In the case of having temporary id's, we dont resolve them yet but signaklise that they have been used
# value -1 means that identifier has been declared but not assigned a value yet
if bibdoc_tmpid:
if bibdoc_tmpid in tmp_ids:
write_message("WARNING: the temporary identifier %s has been declared more than once. Ignoring the second occurance" % (bibdoc_tmpid, ), stream=sys.stderr)
else:
tmp_ids[bibdoc_tmpid] = -1
if bibdoc_tmpver:
if bibdoc_tmpver in tmp_vers:
write_message("WARNING: the temporary version identifier %s has been declared more than once. Ignoring the second occurance" % (bibdoc_tmpver, ), stream=sys.stderr)
else:
tmp_vers[bibdoc_tmpver] = -1
# Let's discover the type of the document
# This is a legacy field and will not be enforced any particular
# check on it.
doctype = _get_subfield_value(fft, 't', 'Main') #Default is Main
# Let's discover the url.
url = field_get_subfield_values(fft, 'a')
if url:
url = url[0]
try:
check_valid_url(url)
except StandardError as e:
raise StandardError, "fft '%s' specifies in $a a location ('%s') with problems: %s" % (fft, url, e)
else:
url = ''
#TODO: a lot of code can be compactified using similar syntax ... should be more readable on the longer scale
# maybe right side expressions look a bit cryptic, but the elaborate_fft function would be much clearer
if mode == 'correct' and doctype != 'FIX-MARC':
arg2 = ""
else:
arg2 = KEEP_OLD_VALUE
description = _get_subfield_value(fft, 'd', arg2)
# Let's discover the description
# description = field_get_subfield_values(fft, 'd')
# if description != []:
# description = description[0]
# else:
# if mode == 'correct' and doctype != 'FIX-MARC':
## If the user require to correct, and do not specify
## a description this means she really want to
## modify the description.
# description = ''
# else:
# description = KEEP_OLD_VALUE
# Let's discover the desired docname to be created/altered
name = field_get_subfield_values(fft, 'n')
if name:
## Let's remove undesired extensions
name = file_strip_ext(name[0] + '.pdf')
else:
if url:
name = get_docname_from_url(url)
elif mode != 'correct' and doctype != 'FIX-MARC':
raise StandardError, "WARNING: fft '%s' doesn't specifies either a location in $a or a docname in $n" % str(fft)
else:
continue
# Let's discover the desired new docname in case we want to change it
newname = field_get_subfield_values(fft, 'm')
if newname:
newname = file_strip_ext(newname[0] + '.pdf')
else:
newname = name
# Let's discover the desired format
docformat = field_get_subfield_values(fft, 'f')
if docformat:
docformat = normalize_format(docformat[0])
else:
if url:
docformat = guess_format_from_url(url)
else:
docformat = ""
# Let's discover the icon
icon = field_get_subfield_values(fft, 'x')
if icon != []:
icon = icon[0]
if icon != KEEP_OLD_VALUE:
try:
check_valid_url(icon)
except StandardError as e:
raise StandardError, "fft '%s' specifies in $x an icon ('%s') with problems: %s" % (fft, icon, e)
else:
icon = ''
# Let's discover the comment
comment = field_get_subfield_values(fft, 'z')
if comment != []:
comment = comment[0]
else:
if mode == 'correct' and doctype != 'FIX-MARC':
## See comment on description
comment = ''
else:
comment = KEEP_OLD_VALUE
# Let's discover the restriction
restriction = field_get_subfield_values(fft, 'r')
if restriction != []:
restriction = restriction[0]
else:
if mode == 'correct' and doctype != 'FIX-MARC':
## See comment on description
restriction = ''
else:
restriction = KEEP_OLD_VALUE
document_moreinfo = _get_subfield_value(fft, 'w')
version_moreinfo = _get_subfield_value(fft, 'p')
version_format_moreinfo = _get_subfield_value(fft, 'b')
format_moreinfo = _get_subfield_value(fft, 'u')
# Let's discover the timestamp of the file (if any)
timestamp = field_get_subfield_values(fft, 's')
if timestamp:
try:
timestamp = datetime(*(time.strptime(timestamp[0], "%Y-%m-%d %H:%M:%S")[:6]))
except ValueError:
write_message('WARNING: The timestamp is not in a good format, thus will be ignored. The format should be YYYY-MM-DD HH:MM:SS', stream=sys.stderr)
timestamp = ''
else:
timestamp = ''
flags = field_get_subfield_values(fft, 'o')
for flag in flags:
if flag not in CFG_BIBDOCFILE_AVAILABLE_FLAGS:
raise StandardError, "fft '%s' specifies a non available flag: %s" % (fft, flag)
if name in docs: # new format considered
(doctype2, newname2, restriction2, version2, urls, dummybibdoc_moreinfos2, dummybibdoc_tmpid2, dummybibdoc_tmpver2 ) = docs[name]
if doctype2 != doctype:
raise StandardError, "fft '%s' specifies a different doctype from previous fft with docname '%s'" % (str(fft), name)
if newname2 != newname:
raise StandardError, "fft '%s' specifies a different newname from previous fft with docname '%s'" % (str(fft), name)
if restriction2 != restriction:
raise StandardError, "fft '%s' specifies a different restriction from previous fft with docname '%s'" % (str(fft), name)
if version2 != version:
raise StandardError, "fft '%s' specifies a different version than the previous fft with docname '%s'" % (str(fft), name)
for (dummyurl2, format2, dummydescription2, dummycomment2, dummyflags2, dummytimestamp2) in urls:
if docformat == format2:
raise StandardError, "fft '%s' specifies a second file '%s' with the same format '%s' from previous fft with docname '%s'" % (str(fft), url, docformat, name)
if url or docformat:
urls.append((url, docformat, description, comment, flags, timestamp))
if icon:
urls.append((icon, icon[len(file_strip_ext(icon)):] + ';icon', description, comment, flags, timestamp))
else:
if url or docformat:
docs[name] = (doctype, newname, restriction, version, [(url, docformat, description, comment, flags, timestamp)], [document_moreinfo, version_moreinfo, version_format_moreinfo, format_moreinfo], bibdoc_tmpid, bibdoc_tmpver)
if icon:
docs[name][4].append((icon, icon[len(file_strip_ext(icon)):] + ';icon', description, comment, flags, timestamp))
elif icon:
docs[name] = (doctype, newname, restriction, version, [(icon, icon[len(file_strip_ext(icon)):] + ';icon', description, comment, flags, timestamp)], [document_moreinfo, version_moreinfo, version_format_moreinfo, format_moreinfo], bibdoc_tmpid, bibdoc_tmpver)
else:
docs[name] = (doctype, newname, restriction, version, [], [document_moreinfo, version_moreinfo, version_format_moreinfo, format_moreinfo], bibdoc_tmpid, bibdoc_tmpver)
write_message('Result of FFT analysis:\n\tDocs: %s' % (docs,), verbose=9)
# Let's remove all FFT tags
record_delete_field(record, 'FFT', ' ', ' ')
## Let's pre-download all the URLs to see if, in case of mode 'correct' or 'append'
## we can avoid creating a new revision.
for docname, (doctype, newname, restriction, version, urls, more_infos, bibdoc_tmpid, bibdoc_tmpver ) in docs.items():
downloaded_urls = []
try:
bibdoc = bibrecdocs.get_bibdoc(docname)
except InvenioBibDocFileError:
## A bibdoc with the given docname does not exists.
## So there is no chance we are going to revise an existing
## format with an identical file :-)
bibdoc = None
new_revision_needed = False
for url, docformat, description, comment, flags, timestamp in urls:
if url:
try:
downloaded_url = download_url(url, docformat)
write_message("%s saved into %s" % (url, downloaded_url), verbose=9)
except Exception as err:
write_message("ERROR: in downloading '%s' because of: %s" % (url, err), stream=sys.stderr)
raise
if mode == 'correct' and bibdoc is not None and not new_revision_needed:
downloaded_urls.append((downloaded_url, docformat, description, comment, flags, timestamp))
if not bibrecdocs.check_file_exists(downloaded_url, docformat):
new_revision_needed = True
else:
write_message("WARNING: %s is already attached to bibdoc %s for recid %s" % (url, docname, rec_id), stream=sys.stderr)
elif mode == 'append' and bibdoc is not None:
if not bibrecdocs.check_file_exists(downloaded_url, docformat):
downloaded_urls.append((downloaded_url, docformat, description, comment, flags, timestamp))
else:
write_message("WARNING: %s is already attached to bibdoc %s for recid %s" % (url, docname, rec_id), stream=sys.stderr)
else:
downloaded_urls.append((downloaded_url, docformat, description, comment, flags, timestamp))
else:
downloaded_urls.append(('', docformat, description, comment, flags, timestamp))
if mode == 'correct' and bibdoc is not None and not new_revision_needed:
## Since we don't need a new revision (because all the files
## that are being uploaded are different)
## we can simply remove the urls but keep the other information
write_message("No need to add a new revision for docname %s for recid %s" % (docname, rec_id), verbose=2)
docs[docname] = (doctype, newname, restriction, version, [('', docformat, description, comment, flags, timestamp) for (dummy, docformat, description, comment, flags, timestamp) in downloaded_urls], more_infos, bibdoc_tmpid, bibdoc_tmpver)
for downloaded_url, dummy, dummy, dummy, dummy, dummy in downloaded_urls:
## Let's free up some space :-)
if downloaded_url and os.path.exists(downloaded_url):
os.remove(downloaded_url)
else:
if downloaded_urls or mode != 'append':
docs[docname] = (doctype, newname, restriction, version, downloaded_urls, more_infos, bibdoc_tmpid, bibdoc_tmpver)
else:
## In case we are in append mode and there are no urls to append
## we discard the whole FFT
del docs[docname]
if mode == 'replace': # First we erase previous bibdocs
if not pretend:
for bibdoc in bibrecdocs.list_bibdocs():
bibdoc.delete()
bibrecdocs.dirty = True
for docname, (doctype, newname, restriction, version, urls, more_infos, bibdoc_tmpid, bibdoc_tmpver) in iteritems(docs):
write_message("Elaborating olddocname: '%s', newdocname: '%s', doctype: '%s', restriction: '%s', urls: '%s', mode: '%s'" % (docname, newname, doctype, restriction, urls, mode), verbose=9)
if mode in ('insert', 'replace'): # new bibdocs, new docnames, new marc
if newname in bibrecdocs.get_bibdoc_names():
write_message("('%s', '%s') not inserted because docname already exists." % (newname, urls), stream=sys.stderr)
raise StandardError("('%s', '%s') not inserted because docname already exists." % (newname, urls), stream=sys.stderr)
try:
if not pretend:
bibdoc = bibrecdocs.add_bibdoc(doctype, newname)
bibdoc.set_status(restriction)
else:
bibdoc = None
except Exception as e:
write_message("('%s', '%s', '%s') not inserted because: '%s'." % (doctype, newname, urls, e), stream=sys.stderr)
raise e
for (url, docformat, description, comment, flags, timestamp) in urls:
assert(_add_new_format(bibdoc, url, docformat, docname, doctype, newname, description, comment, flags, timestamp, pretend=pretend))
elif mode == 'replace_or_insert': # to be thought as correct_or_insert
try:
bibdoc = bibrecdocs.get_bibdoc(docname)
found_bibdoc = True
except InvenioBibDocFileError:
found_bibdoc = False
else:
if doctype not in ('PURGE', 'DELETE', 'EXPUNGE', 'REVERT', 'FIX-ALL', 'FIX-MARC', 'DELETE-FILE'):
if newname != docname:
try:
if not pretend:
bibrecdocs.change_name(newname=newname, docid=bibdoc.id)
write_message(lambda: "After renaming: %s" % bibrecdocs, verbose=9)
except StandardError as e:
write_message('ERROR: in renaming %s to %s: %s' % (docname, newname, e), stream=sys.stderr)
raise
try:
bibdoc = bibrecdocs.get_bibdoc(newname)
found_bibdoc = True
except InvenioBibDocFileError:
found_bibdoc = False
else:
if doctype == 'PURGE':
if not pretend:
bibdoc.purge()
bibrecdocs.dirty = True
elif doctype == 'DELETE':
if not pretend:
bibdoc.delete()
bibrecdocs.dirty = True
elif doctype == 'EXPUNGE':
if not pretend:
bibdoc.expunge()
bibrecdocs.dirty = True
elif doctype == 'FIX-ALL':
if not pretend:
bibrecdocs.fix(docname)
elif doctype == 'FIX-MARC':
pass
elif doctype == 'DELETE-FILE':
if urls:
for (url, docformat, description, comment, flags, timestamp) in urls:
if not pretend:
bibdoc.delete_file(docformat, version)
elif doctype == 'REVERT':
try:
if not pretend:
bibdoc.revert(version)
except Exception, e:
write_message('(%s, %s) not correctly reverted: %s' % (newname, version, e), stream=sys.stderr)
raise
else:
if restriction != KEEP_OLD_VALUE:
if not pretend:
bibdoc.set_status(restriction)
# Since the docname already existed we have to first
# bump the version by pushing the first new file
# then pushing the other files.
if urls:
(first_url, first_format, first_description, first_comment, first_flags, first_timestamp) = urls[0]
other_urls = urls[1:]
assert(_add_new_version(bibdoc, first_url, first_format, docname, doctype, newname, first_description, first_comment, first_flags, first_timestamp, pretend=pretend))
for (url, docformat, description, comment, flags, timestamp) in other_urls:
assert(_add_new_format(bibdoc, url, docformat, docname, doctype, newname, description, comment, flags, timestamp, pretend=pretend))
## Let's refresh the list of bibdocs.
if not found_bibdoc:
if not pretend:
bibdoc = bibrecdocs.add_bibdoc(doctype, newname)
bibdoc.set_status(restriction)
for (url, docformat, description, comment, flags, timestamp) in urls:
assert(_add_new_format(bibdoc, url, docformat, docname, doctype, newname, description, comment, flags, timestamp))
elif mode == 'correct':
try:
bibdoc = bibrecdocs.get_bibdoc(docname)
found_bibdoc = True
except InvenioBibDocFileError:
found_bibdoc = False
else:
if doctype not in ('PURGE', 'DELETE', 'EXPUNGE', 'REVERT', 'FIX-ALL', 'FIX-MARC', 'DELETE-FILE'):
if newname != docname:
try:
if not pretend:
bibrecdocs.change_name(newname=newname, docid=bibdoc.id)
write_message(lambda: "After renaming: %s" % bibrecdocs, verbose=9)
except StandardError as e:
write_message('ERROR: in renaming %s to %s: %s' % (docname, newname, e), stream=sys.stderr)
raise
try:
bibdoc = bibrecdocs.get_bibdoc(newname)
found_bibdoc = True
except InvenioBibDocFileError:
found_bibdoc = False
else:
if doctype == 'PURGE':
if not pretend:
bibdoc.purge()
bibrecdocs.dirty = True
elif doctype == 'DELETE':
if not pretend:
bibdoc.delete()
bibrecdocs.dirty = True
elif doctype == 'EXPUNGE':
if not pretend:
bibdoc.expunge()
bibrecdocs.dirty = True
elif doctype == 'FIX-ALL':
if not pretend:
bibrecdocs.fix(newname)
elif doctype == 'FIX-MARC':
pass
elif doctype == 'DELETE-FILE':
if urls:
for (url, docformat, description, comment, flags, timestamp) in urls:
if not pretend:
bibdoc.delete_file(docformat, version)
elif doctype == 'REVERT':
try:
if not pretend:
bibdoc.revert(version)
except Exception, e:
write_message('(%s, %s) not correctly reverted: %s' % (newname, version, e), stream=sys.stderr)
raise
else:
if restriction != KEEP_OLD_VALUE:
if not pretend:
bibdoc.set_status(restriction)
if doctype and doctype != KEEP_OLD_VALUE:
if not pretend:
bibdoc.change_doctype(doctype)
if urls:
(first_url, first_format, first_description, first_comment, first_flags, first_timestamp) = urls[0]
other_urls = urls[1:]
assert(_add_new_version(bibdoc, first_url, first_format, docname, doctype, newname, first_description, first_comment, first_flags, first_timestamp, pretend=pretend))
for (url, docformat, description, comment, flags, timestamp) in other_urls:
assert(_add_new_format(bibdoc, url, docformat, docname, doctype, newname, description, comment, flags, timestamp, pretend=pretend))
if not found_bibdoc:
if doctype in ('PURGE', 'DELETE', 'EXPUNGE', 'FIX-ALL', 'FIX-MARC', 'DELETE-FILE', 'REVERT'):
write_message("('%s', '%s', '%s') not performed because '%s' docname didn't existed." % (doctype, newname, urls, docname), stream=sys.stderr)
raise StandardError
else:
if not pretend:
bibdoc = bibrecdocs.add_bibdoc(doctype, newname)
bibdoc.set_status(restriction)
for (url, docformat, description, comment, flags, timestamp) in urls:
assert(_add_new_format(bibdoc, url, docformat, docname, doctype, newname, description, comment, flags, timestamp))
elif mode == 'append':
found_bibdoc = False
try:
bibdoc = bibrecdocs.get_bibdoc(docname)
found_bibdoc = True
except InvenioBibDocFileError:
found_bibdoc = False
else:
for (url, docformat, description, comment, flags, timestamp) in urls:
assert(_add_new_format(bibdoc, url, docformat, docname, doctype, newname, description, comment, flags, timestamp, pretend=pretend))
if not found_bibdoc:
try:
if not pretend:
bibdoc = bibrecdocs.add_bibdoc(doctype, docname)
bibdoc.set_status(restriction)
for (url, docformat, description, comment, flags, timestamp) in urls:
assert(_add_new_format(bibdoc, url, docformat, docname, doctype, newname, description, comment, flags, timestamp))
except Exception, e:
register_exception()
write_message("('%s', '%s', '%s') not appended because: '%s'." % (doctype, newname, urls, e), stream=sys.stderr)
raise
if not pretend and doctype not in ('PURGE', 'DELETE', 'EXPUNGE'):
_process_document_moreinfos(more_infos, newname, version, urls and urls[0][1], mode)
# resolving temporary version and identifier
if bibdoc_tmpid:
if bibdoc_tmpid in tmp_ids and tmp_ids[bibdoc_tmpid] != -1:
write_message("WARNING: the temporary identifier %s has been declared more than once. Ignoring the second occurance" % (bibdoc_tmpid, ), stream=sys.stderr)
else:
tmp_ids[bibdoc_tmpid] = bibrecdocs.get_docid(docname)
if bibdoc_tmpver:
if bibdoc_tmpver in tmp_vers and tmp_vers[bibdoc_tmpver] != -1:
write_message("WARNING: the temporary version identifier %s has been declared more than once. Ignoring the second occurance" % (bibdoc_tmpver, ), stream=sys.stderr)
else:
if version == None:
if version:
tmp_vers[bibdoc_tmpver] = version
else:
tmp_vers[bibdoc_tmpver] = bibrecdocs.get_bibdoc(docname).get_latest_version()
else:
tmp_vers[bibdoc_tmpver] = version
return record
### Update functions
def update_bibrec_date(now, bibrec_id, insert_mode_p, pretend=False):
"""Update the date of the record in bibrec table """
if insert_mode_p:
query = """UPDATE bibrec SET creation_date=%s, modification_date=%s WHERE id=%s"""
params = (now, now, bibrec_id)
else:
query = """UPDATE bibrec SET modification_date=%s WHERE id=%s"""
params = (now, bibrec_id)
if not pretend:
run_sql(query, params)
write_message(" -Update record creation/modification date: DONE" , verbose=2)
def update_bibfmt_format(id_bibrec, format_value, format_name, modification_date=None, pretend=False):
"""Update the format in the table bibfmt"""
if modification_date is None:
modification_date = time.strftime('%Y-%m-%d %H:%M:%S')
else:
try:
time.strptime(modification_date, "%Y-%m-%d %H:%M:%S")
except ValueError:
modification_date = '1970-01-01 00:00:00'
# We check if the format is already in bibFmt
nb_found = find_record_format(id_bibrec, format_name)
if nb_found == 1:
# we are going to update the format
# compress the format_value value
pickled_format_value = compress(format_value)
# update the format:
query = """UPDATE LOW_PRIORITY bibfmt SET last_updated=%s, value=%s WHERE id_bibrec=%s AND format=%s"""
params = (modification_date, pickled_format_value, id_bibrec, format_name)
if not pretend:
row_id = run_sql(query, params)
if not pretend and row_id is None:
write_message(" ERROR: during update_bibfmt_format function", verbose=1, stream=sys.stderr)
return 1
else:
write_message(" -Update the format %s in bibfmt: DONE" % format_name , verbose=2)
return 0
elif nb_found > 1:
write_message(" Failed: Same format %s found several time in bibfmt for the same record." % format_name, verbose=1, stream=sys.stderr)
return 1
else:
# Insert the format information in BibFMT
res = insert_bibfmt(id_bibrec, format_value, format_name, modification_date, pretend=pretend)
if res is None:
write_message(" ERROR: during insert_bibfmt", verbose=1, stream=sys.stderr)
return 1
else:
write_message(" -Insert the format %s in bibfmt: DONE" % format_name , verbose=2)
return 0
def delete_bibfmt_format(id_bibrec, format_name, pretend=False):
"""
Delete format FORMAT_NAME from bibfmt table fo record ID_BIBREC.
"""
if not pretend:
run_sql("DELETE LOW_PRIORITY FROM bibfmt WHERE id_bibrec=%s and format=%s", (id_bibrec, format_name))
return 0
def archive_marcxml_for_history(recID, affected_fields, pretend=False):
"""
Archive current MARCXML format of record RECID from BIBFMT table
into hstRECORD table. Useful to keep MARCXML history of records.
Return 0 if everything went fine. Return 1 otherwise.
"""
res = run_sql("SELECT id_bibrec, value, last_updated FROM bibfmt WHERE format='xm' AND id_bibrec=%s",
(recID,))
db_affected_fields = ""
if affected_fields:
tmp_affected_fields = {}
for field in affected_fields:
if field.isdigit(): #hack for tags from RevisionVerifier
for ind in affected_fields[field]:
tmp_affected_fields[(field + ind[0] + ind[1] + "%").replace(" ", "_")] = 1
else:
pass #future implementation for fields
tmp_affected_fields = tmp_affected_fields.keys()
tmp_affected_fields.sort()
db_affected_fields = ",".join(tmp_affected_fields)
if res and not pretend:
run_sql("""INSERT INTO hstRECORD (id_bibrec, marcxml, job_id, job_name, job_person, job_date, job_details, affected_fields)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s)""",
(res[0][0], res[0][1], task_get_task_param('task_id', 0), 'bibupload', task_get_task_param('user', 'UNKNOWN'), res[0][2],
'mode: ' + task_get_option('mode', 'UNKNOWN') + '; file: ' + task_get_option('file_path', 'UNKNOWN') + '.',
db_affected_fields))
return 0
def update_database_with_metadata(record, rec_id, oai_rec_id="oai", affected_tags=None, pretend=False):
"""Update the database tables with the record and the record id given in parameter"""
# extract only those tags that have been affected.
# check happens at subfield level. This is to prevent overhead
# associated with inserting already existing field with given ind pair
write_message("update_database_with_metadata: record=%s, rec_id=%s, oai_rec_id=%s, affected_tags=%s" % (record, rec_id, oai_rec_id, affected_tags), verbose=9)
tmp_record = {}
if affected_tags:
for tag in record.keys():
if tag in affected_tags.keys():
write_message(" -Tag %s found to be modified.Setting up for update" % tag, verbose=9)
# initialize new list to hold affected field
new_data_tuple_list = []
for data_tuple in record[tag]:
ind1 = data_tuple[1]
ind2 = data_tuple[2]
if (ind1, ind2) in affected_tags[tag]:
write_message(" -Indicator pair (%s, %s) added to update list" % (ind1, ind2), verbose=9)
new_data_tuple_list.append(data_tuple)
tmp_record[tag] = new_data_tuple_list
write_message(lambda: " -Modified fields: \n%s" % record_xml_output(tmp_record), verbose=2)
else:
tmp_record = record
for tag in tmp_record.keys():
# check if tag is not a special one:
if tag not in CFG_BIBUPLOAD_SPECIAL_TAGS:
# for each tag there is a list of tuples representing datafields
tuple_list = tmp_record[tag]
# this list should contain the elements of a full tag [tag, ind1, ind2, subfield_code]
tag_list = []
tag_list.append(tag)
for single_tuple in tuple_list:
# these are the contents of a single tuple
subfield_list = single_tuple[0]
ind1 = single_tuple[1]
ind2 = single_tuple[2]
# append the ind's to the full tag
if ind1 == '' or ind1 == ' ':
tag_list.append('_')
else:
tag_list.append(ind1)
if ind2 == '' or ind2 == ' ':
tag_list.append('_')
else:
tag_list.append(ind2)
datafield_number = single_tuple[4]
if tag in CFG_BIBUPLOAD_SPECIAL_TAGS:
# nothing to do for special tags (FFT, BDR, BDM)
pass
elif tag in CFG_BIBUPLOAD_CONTROLFIELD_TAGS and tag != "001":
value = single_tuple[3]
# get the full tag
full_tag = ''.join(tag_list)
# update the tables
write_message(" insertion of the tag "+full_tag+" with the value "+value, verbose=9)
# insert the tag and value into into bibxxx
(table_name, bibxxx_row_id) = insert_record_bibxxx(full_tag, value, pretend=pretend)
#print 'tname, bibrow', table_name, bibxxx_row_id;
if table_name is None or bibxxx_row_id is None:
write_message(" Failed: during insert_record_bibxxx", verbose=1, stream=sys.stderr)
# connect bibxxx and bibrec with the table bibrec_bibxxx
res = insert_record_bibrec_bibxxx(table_name, bibxxx_row_id, datafield_number, rec_id, pretend=pretend)
if res is None:
write_message(" Failed: during insert_record_bibrec_bibxxx", verbose=1, stream=sys.stderr)
else:
# get the tag and value from the content of each subfield
for subfield in set(subfield_list):
subtag = subfield[0]
value = subfield[1]
tag_list.append(subtag)
# get the full tag
full_tag = ''.join(tag_list)
# update the tables
write_message(" insertion of the tag "+full_tag+" with the value "+value, verbose=9)
# insert the tag and value into into bibxxx
(table_name, bibxxx_row_id) = insert_record_bibxxx(full_tag, value, pretend=pretend)
if table_name is None or bibxxx_row_id is None:
write_message(" Failed: during insert_record_bibxxx", verbose=1, stream=sys.stderr)
# connect bibxxx and bibrec with the table bibrec_bibxxx
res = insert_record_bibrec_bibxxx(table_name, bibxxx_row_id, datafield_number, rec_id, pretend=pretend)
if res is None:
write_message(" Failed: during insert_record_bibrec_bibxxx", verbose=1, stream=sys.stderr)
# remove the subtag from the list
tag_list.pop()
tag_list.pop()
tag_list.pop()
tag_list.pop()
write_message(" -Update the database with metadata: DONE", verbose=2)
log_record_uploading(oai_rec_id, task_get_task_param('task_id', 0), rec_id, 'P', pretend=pretend)
def append_new_tag_to_old_record(record, rec_old):
"""Append new tags to a old record"""
def _append_tag(tag):
if tag in CFG_BIBUPLOAD_CONTROLFIELD_TAGS:
if tag == '001':
pass
else:
# if it is a controlfield, just access the value
for single_tuple in record[tag]:
controlfield_value = single_tuple[3]
# add the field to the old record
newfield_number = record_add_field(rec_old, tag,
controlfield_value=controlfield_value)
if newfield_number is None:
write_message(" ERROR: when adding the field"+tag, verbose=1, stream=sys.stderr)
else:
# For each tag there is a list of tuples representing datafields
for single_tuple in record[tag]:
# We retrieve the information of the tag
subfield_list = single_tuple[0]
ind1 = single_tuple[1]
ind2 = single_tuple[2]
if '%s%s%s' % (tag, ind1 == ' ' and '_' or ind1, ind2 == ' ' and '_' or ind2) in (CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[:5], CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG[:5]):
## We don't want to append the external identifier
## if it is already existing.
if record_find_field(rec_old, tag, single_tuple)[0] is not None:
write_message(" Not adding tag: %s ind1=%s ind2=%s subfields=%s: it's already there" % (tag, ind1, ind2, subfield_list), verbose=9)
continue
# We add the datafield to the old record
write_message(" Adding tag: %s ind1=%s ind2=%s subfields=%s" % (tag, ind1, ind2, subfield_list), verbose=9)
newfield_number = record_add_field(rec_old, tag, ind1,
ind2, subfields=subfield_list)
if newfield_number is None:
write_message(" ERROR: when adding the field"+tag, verbose=1, stream=sys.stderr)
# Go through each tag in the appended record
for tag in record:
_append_tag(tag)
return rec_old
def copy_strong_tags_from_old_record(record, rec_old):
"""
Look for strong tags in RECORD and REC_OLD. If no strong tags are
found in RECORD, then copy them over from REC_OLD. This function
modifies RECORD structure on the spot.
"""
for strong_tag in CFG_BIBUPLOAD_STRONG_TAGS:
if not record_get_field_instances(record, strong_tag, strong_tag[3:4] or '%', strong_tag[4:5] or '%'):
strong_tag_old_field_instances = record_get_field_instances(rec_old, strong_tag)
if strong_tag_old_field_instances:
for strong_tag_old_field_instance in strong_tag_old_field_instances:
sf_vals, fi_ind1, fi_ind2, controlfield, dummy = strong_tag_old_field_instance
record_add_field(record, strong_tag, fi_ind1, fi_ind2, controlfield, sf_vals)
return
### Delete functions
def delete_tags(record, rec_old):
"""
Returns a record structure with all the fields in rec_old minus the
fields in record.
@param record: The record containing tags to delete.
@type record: record structure
@param rec_old: The original record.
@type rec_old: record structure
@return: The modified record.
@rtype: record structure
"""
returned_record = copy.deepcopy(rec_old)
for tag, fields in iteritems(record):
if tag in ('001', ):
continue
for field in fields:
local_position = record_find_field(returned_record, tag, field)[1]
if local_position is not None:
record_delete_field(returned_record, tag, field_position_local=local_position)
return returned_record
def delete_tags_to_correct(record, rec_old):
"""
Delete tags from REC_OLD which are also existing in RECORD. When
deleting, pay attention not only to tags, but also to indicators,
so that fields with the same tags but different indicators are not
deleted.
"""
## Some fields are controlled via provenance information.
## We should re-add saved fields at the end.
fields_to_readd = {}
for tag in CFG_BIBUPLOAD_CONTROLLED_PROVENANCE_TAGS:
if tag[:3] in record:
tmp_field_instances = record_get_field_instances(record, tag[:3], tag[3], tag[4]) ## Let's discover the provenance that will be updated
provenances_to_update = []
for instance in tmp_field_instances:
for code, value in instance[0]:
if code == tag[5]:
if value not in provenances_to_update:
provenances_to_update.append(value)
break
else:
## The provenance is not specified.
## let's add the special empty provenance.
if '' not in provenances_to_update:
provenances_to_update.append('')
potential_fields_to_readd = record_get_field_instances(rec_old, tag[:3], tag[3], tag[4]) ## Let's take all the field corresponding to tag
## Let's save apart all the fields that should be updated, but
## since they have a different provenance not mentioned in record
## they should be preserved.
fields = []
for sf_vals, ind1, ind2, dummy_cf, dummy_line in potential_fields_to_readd:
for code, value in sf_vals:
if code == tag[5]:
if value not in provenances_to_update:
fields.append(sf_vals)
break
else:
if '' not in provenances_to_update:
## Empty provenance, let's protect in any case
fields.append(sf_vals)
fields_to_readd[tag] = fields
# browse through all the tags from the MARCXML file:
for tag in record:
# check if the tag exists in the old record too:
if tag in rec_old and tag != '001':
# the tag does exist, so delete all record's tag+ind1+ind2 combinations from rec_old
for dummy_sf_vals, ind1, ind2, dummy_cf, dummyfield_number in record[tag]:
write_message(" Delete tag: " + tag + " ind1=" + ind1 + " ind2=" + ind2, verbose=9)
record_delete_field(rec_old, tag, ind1, ind2)
## Ok, we readd necessary fields!
for tag, fields in iteritems(fields_to_readd):
for sf_vals in fields:
write_message(" Adding tag: " + tag[:3] + " ind1=" + tag[3] + " ind2=" + tag[4] + " code=" + str(sf_vals), verbose=9)
record_add_field(rec_old, tag[:3], tag[3], tag[4], subfields=sf_vals)
def delete_bibrec_bibxxx(record, id_bibrec, affected_tags={}, pretend=False):
"""Delete the database record from the table bibxxx given in parameters"""
# we clear all the rows from bibrec_bibxxx from the old record
# clearing only those tags that have been modified.
write_message(lambda: "delete_bibrec_bibxxx(record=%s, id_bibrec=%s, affected_tags=%s)" % (record, id_bibrec, affected_tags), verbose=9)
for tag in affected_tags:
# sanity check with record keys just to make sure its fine.
if tag not in CFG_BIBUPLOAD_SPECIAL_TAGS:
write_message("%s found in record"%tag, verbose=2)
# for each name construct the bibrec_bibxxx table name
table_name = 'bib'+tag[0:2]+'x'
bibrec_table = 'bibrec_'+table_name
# delete all the records with proper id_bibrec. Indicators matter for individual affected tags
tmp_ind_1 = ''
tmp_ind_2 = ''
# construct exact tag value using indicators
for ind_pair in affected_tags[tag]:
if ind_pair[0] == ' ':
tmp_ind_1 = '_'
else:
tmp_ind_1 = ind_pair[0]
if ind_pair[1] == ' ':
tmp_ind_2 = '_'
else:
tmp_ind_2 = ind_pair[1]
# need to escape incase of underscore so that mysql treats it as a char
tag_val = tag+"\\"+tmp_ind_1+"\\"+tmp_ind_2 + '%'
query = """DELETE br.* FROM `%s` br,`%s` b where br.id_bibrec=%%s and br.id_bibxxx=b.id and b.tag like %%s""" % (bibrec_table, table_name)
params = (id_bibrec, tag_val)
write_message(query % params, verbose=9)
if not pretend:
run_sql(query, params)
else:
write_message("%s not found"%tag, verbose=2)
def main():
"""Main that construct all the bibtask."""
task_init(authorization_action='runbibupload',
authorization_msg="BibUpload Task Submission",
description="""Receive MARC XML file and update appropriate database
tables according to options.
Examples:
$ bibupload -i input.xml
""",
help_specific_usage=""" -a, --append\t\tnew fields are appended to the existing record
-c, --correct\t\tfields are replaced by the new ones in the existing record, except
\t\t\twhen overridden by CFG_BIBUPLOAD_CONTROLLED_PROVENANCE_TAGS
-i, --insert\t\tinsert the new record in the database
-r, --replace\t\tthe existing record is entirely replaced by the new one,
\t\t\texcept for fields in CFG_BIBUPLOAD_STRONG_TAGS
-d, --delete\t\tspecified fields are deleted in existing record
-n, --notimechange\tdo not change record last modification date when updating
-o, --holdingpen\tInsert record into holding pen instead of the normal database
--pretend\t\tdo not really insert/append/correct/replace the input file
--force\t\twhen --replace, use provided 001 tag values, even if the matching
\t\t\trecord does not exist (thus allocating it on-the-fly)
--callback-url\tSend via a POST request a JSON-serialized answer (see admin guide), in
\t\t\torder to provide a feedback to an external service about the outcome of the operation.
--nonce\t\twhen used together with --callback add the nonce value in the JSON message.
--special-treatment=MODE\tif "oracle" is specified, when used together with --callback_url,
\t\t\tPOST an application/x-www-form-urlencoded request where the JSON message is encoded
\t\t\tinside a form field called "results".
""",
version=__revision__,
specific_params=("ircazdnoS:",
[
"insert",
"replace",
"correct",
"append",
"reference",
"delete",
"notimechange",
"holdingpen",
"pretend",
"force",
"callback-url=",
"nonce=",
"special-treatment=",
"stage=",
]),
task_submit_elaborate_specific_parameter_fnc=task_submit_elaborate_specific_parameter,
task_run_fnc=task_run_core,
task_submit_check_options_fnc=task_submit_check_options)
def task_submit_elaborate_specific_parameter(key, value, opts, args): # pylint: disable=W0613
""" Given the string key it checks it's meaning, eventually using the
value. Usually it fills some key in the options dict.
It must return True if it has elaborated the key, False, if it doesn't
know that key.
eg:
if key in ['-n', '--number']:
task_get_option(\1) = value
return True
return False
"""
# No time change option
if key in ("-n", "--notimechange"):
task_set_option('notimechange', 1)
# Insert mode option
elif key in ("-i", "--insert"):
if task_get_option('mode') == 'replace':
# if also replace found, then set to replace_or_insert
task_set_option('mode', 'replace_or_insert')
else:
task_set_option('mode', 'insert')
fix_argv_paths([args[0]])
task_set_option('file_path', os.path.abspath(args[0]))
# Replace mode option
elif key in ("-r", "--replace"):
if task_get_option('mode') == 'insert':
# if also insert found, then set to replace_or_insert
task_set_option('mode', 'replace_or_insert')
else:
task_set_option('mode', 'replace')
fix_argv_paths([args[0]])
task_set_option('file_path', os.path.abspath(args[0]))
# Holding pen mode option
elif key in ("-o", "--holdingpen"):
write_message("Holding pen mode", verbose=3)
task_set_option('mode', 'holdingpen')
fix_argv_paths([args[0]])
task_set_option('file_path', os.path.abspath(args[0]))
# Correct mode option
elif key in ("-c", "--correct"):
task_set_option('mode', 'correct')
fix_argv_paths([args[0]])
task_set_option('file_path', os.path.abspath(args[0]))
# Append mode option
elif key in ("-a", "--append"):
task_set_option('mode', 'append')
fix_argv_paths([args[0]])
task_set_option('file_path', os.path.abspath(args[0]))
# Deprecated reference mode option (now correct)
elif key in ("-z", "--reference"):
task_set_option('mode', 'correct')
fix_argv_paths([args[0]])
task_set_option('file_path', os.path.abspath(args[0]))
elif key in ("-d", "--delete"):
task_set_option('mode', 'delete')
fix_argv_paths([args[0]])
task_set_option('file_path', os.path.abspath(args[0]))
elif key in ("--pretend",):
task_set_option('pretend', True)
fix_argv_paths([args[0]])
task_set_option('file_path', os.path.abspath(args[0]))
elif key in ("--force",):
task_set_option('force', True)
fix_argv_paths([args[0]])
task_set_option('file_path', os.path.abspath(args[0]))
elif key in ("--callback-url", ):
task_set_option('callback_url', value)
elif key in ("--nonce", ):
task_set_option('nonce', value)
elif key in ("--special-treatment", ):
if value.lower() in CFG_BIBUPLOAD_ALLOWED_SPECIAL_TREATMENTS:
if value.lower() == 'oracle':
task_set_option('oracle_friendly', True)
else:
print("""The specified value is not in the list of allowed special treatments codes: %s""" % CFG_BIBUPLOAD_ALLOWED_SPECIAL_TREATMENTS, file=sys.stderr)
return False
elif key in ("-S", "--stage"):
print("""WARNING: the --stage parameter is deprecated and ignored.""", file=sys.stderr)
else:
return False
return True
def task_submit_check_options():
""" Reimplement this method for having the possibility to check options
before submitting the task, in order for example to provide default
values. It must return False if there are errors in the options.
"""
if task_get_option('mode') is None:
write_message("Please specify at least one update/insert mode!",
stream=sys.stderr)
return False
file_path = task_get_option('file_path')
if file_path is None:
write_message("Missing filename! -h for help.", stream=sys.stderr)
return False
try:
open(file_path).read().decode('utf-8')
except IOError:
write_message("""File is not accessible: %s""" % file_path,
stream=sys.stderr)
return False
except UnicodeDecodeError:
write_message("""File encoding is not valid utf-8: %s""" % file_path,
stream=sys.stderr)
return False
return True
def writing_rights_p():
"""Return True in case bibupload has the proper rights to write in the
fulltext file folder."""
if _WRITING_RIGHTS is not None:
return _WRITING_RIGHTS
try:
if not os.path.exists(CFG_BIBDOCFILE_FILEDIR):
os.makedirs(CFG_BIBDOCFILE_FILEDIR)
fd, filename = tempfile.mkstemp(suffix='.txt', prefix='test', dir=CFG_BIBDOCFILE_FILEDIR)
test = os.fdopen(fd, 'w')
test.write('TEST')
test.close()
if open(filename).read() != 'TEST':
raise IOError("Can not successfully write and readback %s" % filename)
os.remove(filename)
except:
register_exception(alert_admin=True)
return False
return True
def post_results_to_callback_url(results, callback_url):
write_message("Sending feedback to %s" % callback_url)
if not CFG_JSON_AVAILABLE:
from warnings import warn
warn("--callback-url used but simplejson/json not available")
return
json_results = json.dumps(results)
write_message("Message to send: %s" % json_results, verbose=9)
## <scheme>://<netloc>/<path>?<query>#<fragment>
scheme, dummynetloc, dummypath, dummyquery, dummyfragment = urlparse.urlsplit(callback_url)
## See: http://stackoverflow.com/questions/111945/is-there-any-way-to-do-http-put-in-python
if scheme == 'http':
opener = urllib2.build_opener(urllib2.HTTPHandler)
elif scheme == 'https':
opener = urllib2.build_opener(urllib2.HTTPSHandler)
else:
raise ValueError("Scheme not handled %s for callback_url %s" % (scheme, callback_url))
if task_get_option('oracle_friendly'):
write_message("Oracle friendly mode requested", verbose=9)
request = urllib2.Request(callback_url, data=urllib.urlencode({'results': json_results}))
request.add_header('Content-Type', 'application/x-www-form-urlencoded')
else:
request = urllib2.Request(callback_url, data=json_results)
request.add_header('Content-Type', 'application/json')
request.add_header('User-Agent', make_user_agent_string('BibUpload'))
write_message("Headers about to be sent: %s" % request.headers, verbose=9)
write_message("Data about to be sent: %s" % request.data, verbose=9)
res = opener.open(request)
msg = res.read()
write_message("Result of posting the feedback: %s %s" % (res.code, res.msg), verbose=9)
write_message("Returned message is: %s" % msg, verbose=9)
return res
def bibupload_records(records, opt_mode=None, opt_notimechange=0,
pretend=False, callback_url=None, results_for_callback=None):
"""perform the task of uploading a set of records
returns list of (error_code, recid) tuples for separate records
"""
#Dictionaries maintaining temporary identifiers
# Structure: identifier -> number
tmp_ids = {}
tmp_vers = {}
results = []
# The first phase -> assigning meaning to temporary identifiers
if opt_mode == 'reference':
## NOTE: reference mode has been deprecated in favour of 'correct'
opt_mode = 'correct'
record = None
for record in records:
record_id = record_extract_oai_id(record)
task_sleep_now_if_required(can_stop_too=True)
if opt_mode == "holdingpen":
#inserting into the holding pen
write_message("Inserting into holding pen", verbose=3)
insert_record_into_holding_pen(record, record_id, pretend=pretend)
else:
write_message("Inserting into main database", verbose=3)
error = bibupload(
record,
opt_mode = opt_mode,
opt_notimechange = opt_notimechange,
oai_rec_id = record_id,
pretend = pretend,
tmp_ids = tmp_ids,
tmp_vers = tmp_vers)
results.append(error)
if error[0] == 1:
if record:
write_message(lambda: record_xml_output(record),
stream=sys.stderr)
else:
write_message("Record could not have been parsed",
stream=sys.stderr)
stat['nb_errors'] += 1
if callback_url:
results_for_callback['results'].append({'recid': error[1], 'success': False, 'error_message': error[2]})
elif error[0] == 2:
if record:
write_message(lambda: record_xml_output(record),
stream=sys.stderr)
else:
write_message("Record could not have been parsed",
stream=sys.stderr)
if callback_url:
results_for_callback['results'].append({'recid': error[1], 'success': False, 'error_message': error[2]})
elif error[0] == 0:
if callback_url:
from invenio.legacy.search_engine import print_record
results_for_callback['results'].append({'recid': error[1], 'success': True, "marcxml": print_record(error[1], 'xm'), 'url': "%s/%s/%s" % (CFG_SITE_URL, CFG_SITE_RECORD, error[1])})
else:
if callback_url:
results_for_callback['results'].append({'recid': error[1], 'success': False, 'error_message': error[2]})
# stat us a global variable
task_update_progress("Done %d out of %d." % \
(stat['nb_records_inserted'] + \
stat['nb_records_updated'],
stat['nb_records_to_upload']))
# Second phase -> Now we can process all entries where temporary identifiers might appear (BDR, BDM)
write_message("Identifiers table after processing: %s versions: %s" % (str(tmp_ids), str(tmp_vers)), verbose=2)
write_message("Uploading BDR and BDM fields")
if opt_mode != "holdingpen":
for record in records:
record_id = retrieve_rec_id(record, opt_mode, pretend=pretend, post_phase = True)
bibupload_post_phase(record,
rec_id = record_id,
mode = opt_mode,
pretend = pretend,
tmp_ids = tmp_ids,
tmp_vers = tmp_vers)
return results
def task_run_core():
""" Reimplement to add the body of the task."""
write_message("Input file '%s', input mode '%s'." %
(task_get_option('file_path'), task_get_option('mode')))
write_message("STAGE 0:", verbose=2)
if task_get_option('file_path') is not None:
write_message("start preocessing", verbose=3)
task_update_progress("Reading XML input")
recs = xml_marc_to_records(open_marc_file(task_get_option('file_path')))
stat['nb_records_to_upload'] = len(recs)
write_message(" -Open XML marc: DONE", verbose=2)
task_sleep_now_if_required(can_stop_too=True)
write_message("Entering records loop", verbose=3)
callback_url = task_get_option('callback_url')
results_for_callback = {'results': []}
if recs is not None:
# We proceed each record by record
bibupload_records(records=recs, opt_mode=task_get_option('mode'),
opt_notimechange=task_get_option('notimechange'),
pretend=task_get_option('pretend'),
callback_url=callback_url,
results_for_callback=results_for_callback)
else:
write_message(" ERROR: bibupload failed: No record found",
verbose=1, stream=sys.stderr)
callback_url = task_get_option("callback_url")
if callback_url:
nonce = task_get_option("nonce")
if nonce:
results_for_callback["nonce"] = nonce
post_results_to_callback_url(results_for_callback, callback_url)
if task_get_task_param('verbose') >= 1:
# Print out the statistics
print_out_bibupload_statistics()
# Check if they were errors
return not stat['nb_errors'] >= 1
def log_record_uploading(oai_rec_id, task_id, bibrec_id, insertion_db, pretend=False):
if oai_rec_id != "" and oai_rec_id != None:
query = """UPDATE oaiHARVESTLOG SET date_inserted=NOW(), inserted_to_db=%s, id_bibrec=%s WHERE oai_id = %s AND bibupload_task_id = %s ORDER BY date_harvested LIMIT 1"""
if not pretend:
run_sql(query, (str(insertion_db), str(bibrec_id), str(oai_rec_id), str(task_id), ))
if __name__ == "__main__":
main()
diff --git a/invenio/legacy/inveniocfg.py b/invenio/legacy/inveniocfg.py
index 8895a8b49..3a1b33c11 100644
--- a/invenio/legacy/inveniocfg.py
+++ b/invenio/legacy/inveniocfg.py
@@ -1,1313 +1,1316 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
"""
Invenio configuration and administration CLI tool.
Usage: inveniocfg [options]
General options:
-h, --help print this help
-V, --version print version number
Options to finish your installation:
--create-secret-key generate random CFG_SITE_SECRET_KEY
--create-apache-conf create Apache configuration files
--create-tables create DB tables for Invenio
--load-bibfield-conf load the BibField configuration
--load-webstat-conf load the WebStat configuration
--drop-tables drop DB tables of Invenio
--check-openoffice check for correctly set up of openoffice temporary directory
Options to set up and test a demo site:
--create-demo-site create demo site
--load-demo-records load demo records
--remove-demo-records remove demo records, keeping demo site
--drop-demo-site drop demo site configurations too
--run-unit-tests run unit test suite (needs demo site)
--run-regression-tests run regression test suite (needs demo site)
--run-web-tests run web tests in a browser (needs demo site, Firefox, Selenium IDE)
--run-flask-tests run Flask test suite
Options to update config files in situ:
--update-all perform all the update options
--update-config-py update config.py file from invenio.conf file
--update-dbquery-py update dbquery.py with DB credentials from invenio.conf
--update-dbexec update dbexec with DB credentials from invenio.conf
--update-bibconvert-tpl update bibconvert templates with CFG_SITE_URL from invenio.conf
--update-web-tests update web test cases with CFG_SITE_URL from invenio.conf
Options to update DB tables:
--reset-all perform all the reset options
--reset-sitename reset tables to take account of new CFG_SITE_NAME*
--reset-siteadminemail reset tables to take account of new CFG_SITE_ADMIN_EMAIL
--reset-fieldnames reset tables to take account of new I18N names from PO files
--reset-recstruct-cache reset record structure cache according to CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE
--reset-recjson-cache reset record json cache according to CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE
Options to upgrade your installation:
--upgrade apply all pending upgrades
--upgrade-check run pre-upgrade checks for all pending upgrades
--upgrade-show-pending show pending upgrades ready to be applied
--upgrade-show-applied show history of applied upgrades
--upgrade-create-standard-recipe create a new upgrade recipe (for developers)
--upgrade-create-release-recipe create a new release upgrade recipe (for developers)
Options to help the work:
--list print names and values of all options from conf files
--get <some-opt> get value of a given option from conf files
--conf-dir </some/path> path to directory where invenio*.conf files are [optional]
--detect-system-details print system details such as Apache/Python/MySQL versions
"""
__revision__ = "$Id$"
from ConfigParser import ConfigParser
from optparse import OptionParser, OptionGroup, IndentedHelpFormatter, Option, \
OptionError
import os
import pkg_resources
import random
import re
import shutil
import socket
import string
import sys
import zlib
from six import iteritems
from warnings import warn
def print_usage():
"""Print help."""
print(__doc__)
def get_version():
""" Get running version of Invenio """
from invenio.config import CFG_VERSION
return CFG_VERSION
def print_version():
"""Print version information."""
print(get_version())
def convert_conf_option(option_name, option_value):
"""
Convert conf option into Python config.py line, converting
values to ints or strings as appropriate.
"""
## 1) convert option name to uppercase:
option_name = option_name.upper()
## 1a) adjust renamed variables:
if option_name in ['CFG_WEBSUBMIT_DOCUMENT_FILE_MANAGER_DOCTYPES',
'CFG_WEBSUBMIT_DOCUMENT_FILE_MANAGER_RESTRICTIONS',
'CFG_WEBSUBMIT_DOCUMENT_FILE_MANAGER_MISC',
'CFG_WEBSUBMIT_FILESYSTEM_BIBDOC_GROUP_LIMIT',
'CFG_WEBSUBMIT_ADDITIONAL_KNOWN_FILE_EXTENSIONS',
'CFG_WEBSUBMIT_DESIRED_CONVERSIONS']:
new_option_name = option_name.replace('WEBSUBMIT', 'BIBDOCFILE')
print(("""WARNING: %s has been renamed to %s.
Please, update your invenio-local.conf file accordingly.""" % (option_name, new_option_name)), file=sys.stderr)
option_name = new_option_name
## 2) convert option value to int or string:
if option_name in ['CFG_BIBUPLOAD_REFERENCE_TAG',
'CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG',
'CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG',
'CFG_BIBUPLOAD_EXTERNAL_OAIID_PROVENANCE_TAG',
'CFG_BIBUPLOAD_STRONG_TAGS',
'CFG_BIBFORMAT_HIDDEN_TAGS']:
# some options are supposed be string even when they look like
# numeric
option_value = '"' + option_value + '"'
else:
try:
option_value = int(option_value)
except ValueError:
option_value = '"' + option_value + '"'
## 3a) special cases: chars regexps
if option_name in ['CFG_BIBINDEX_CHARS_ALPHANUMERIC_SEPARATORS',
'CFG_BIBINDEX_CHARS_PUNCTUATION']:
option_value = 'r"[' + option_value[1:-1] + ']"'
## 3abis) special cases: real regexps
if option_name in ['CFG_BIBINDEX_PERFORM_OCR_ON_DOCNAMES',
'CFG_BATCHUPLOADER_WEB_ROBOT_AGENTS']:
option_value = 'r"' + option_value[1:-1] + '"'
## 3b) special cases: True, False, None
if option_value in ['"True"', '"False"', '"None"']:
option_value = option_value[1:-1]
## 3c) special cases: dicts and real pythonic lists
if option_name in ['CFG_WEBSEARCH_FIELDS_CONVERT',
'CFG_BATCHUPLOADER_WEB_ROBOT_RIGHTS',
'CFG_WEBSEARCH_FULLTEXT_SNIPPETS',
'CFG_WEBSEARCH_FULLTEXT_SNIPPETS_CHARS',
'CFG_SITE_EMERGENCY_EMAIL_ADDRESSES',
'CFG_BIBMATCH_FUZZY_WORDLIMITS',
'CFG_BIBMATCH_QUERY_TEMPLATES',
'CFG_WEBSEARCH_SYNONYM_KBRS',
'CFG_BIBINDEX_SYNONYM_KBRS',
'CFG_WEBCOMMENT_EMAIL_REPLIES_TO',
'CFG_WEBCOMMENT_RESTRICTION_DATAFIELD',
'CFG_WEBCOMMENT_ROUND_DATAFIELD',
'CFG_BIBUPLOAD_FFT_ALLOWED_EXTERNAL_URLS',
'CFG_BIBSCHED_NODE_TASKS',
'CFG_BIBEDIT_EXTEND_RECORD_WITH_COLLECTION_TEMPLATE',
'CFG_OAI_METADATA_FORMATS',
'CFG_BIBDOCFILE_DESIRED_CONVERSIONS',
'CFG_BIBDOCFILE_BEST_FORMATS_TO_EXTRACT_TEXT_FROM',
'CFG_WEB_API_KEY_ALLOWED_URL',
'CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_MISC',
'CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_DOCTYPES',
'CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_RESTRICTIONS',
'CFG_DEVEL_TEST_DATABASE_ENGINES',
'CFG_REFEXTRACT_KBS_OVERRIDE',
'CFG_OPENID_CONFIGURATIONS',
'CFG_OAUTH1_CONFIGURATIONS',
'CFG_OAUTH2_CONFIGURATIONS',
'CFG_BIBDOCFILE_ADDITIONAL_KNOWN_MIMETYPES',
'CFG_BIBDOCFILE_PREFERRED_MIMETYPES_MAPPING',
'CFG_BIBSCHED_NON_CONCURRENT_TASKS',
- 'CFG_BIBSCHED_INCOMPATIBLE_TASKS']:
+ 'CFG_REDIS_HOSTS',
+ 'CFG_BIBSCHED_INCOMPATIBLE_TASKS',
+ 'CFG_ICON_CREATION_FORMAT_MAPPINGS']:
try:
option_value = option_value[1:-1]
if option_name == "CFG_BIBEDIT_EXTEND_RECORD_WITH_COLLECTION_TEMPLATE" and option_value.strip().startswith("{"):
print(("""ERROR: CFG_BIBEDIT_EXTEND_RECORD_WITH_COLLECTION_TEMPLATE
now accepts only a list of tuples, not a dictionary. Check invenio.conf for an example.
Please, update your invenio-local.conf file accordingly."""), file=sys.stderr)
sys.exit(1)
except TypeError:
if option_name in ('CFG_WEBSEARCH_FULLTEXT_SNIPPETS',):
print("""WARNING: CFG_WEBSEARCH_FULLTEXT_SNIPPETS
has changed syntax: it can be customised to display different snippets for
different document types. See the corresponding documentation in invenio.conf.
You may want to customise your invenio-local.conf configuration accordingly.""", file=sys.stderr)
option_value = """{'': %s}""" % option_value
else:
print("ERROR: type error in %s value %s." % \
(option_name, option_value), file=sys.stderr)
sys.exit(1)
## 3cbis) very special cases: dicts with backward compatible string
if option_name in ['CFG_BIBINDEX_SPLASH_PAGES']:
if option_value.startswith('"{') and option_value.endswith('}"'):
option_value = option_value[1:-1]
else:
option_value = """{%s: ".*"}""" % option_value
## 3d) special cases: comma-separated lists
if option_name in ['CFG_SITE_LANGS',
'CFG_BIBDOCFILE_ADDITIONAL_KNOWN_FILE_EXTENSIONS',
'CFG_WEBSEARCH_USE_MATHJAX_FOR_FORMATS',
'CFG_BIBUPLOAD_STRONG_TAGS',
'CFG_BIBFORMAT_HIDDEN_TAGS',
'CFG_BIBSCHED_GC_TASKS_TO_REMOVE',
'CFG_BIBSCHED_GC_TASKS_TO_ARCHIVE',
'CFG_BIBUPLOAD_FFT_ALLOWED_LOCAL_PATHS',
'CFG_BIBUPLOAD_CONTROLLED_PROVENANCE_TAGS',
'CFG_BIBUPLOAD_DELETE_FORMATS',
'CFG_WEBSEARCH_ENABLED_SEARCH_INTERFACES',
'CFG_WEBSTYLE_HTTP_STATUS_ALERT_LIST',
'CFG_WEBSEARCH_RSS_I18N_COLLECTIONS',
'CFG_BATCHUPLOADER_FILENAME_MATCHING_POLICY',
'CFG_BIBAUTHORID_EXTERNAL_CLAIMED_RECORDS_KEY',
'CFG_BIBCIRCULATION_ITEM_STATUS_OPTIONAL',
'CFG_PLOTEXTRACTOR_DISALLOWED_TEX',
'CFG_OAI_FRIENDS',
'CFG_WEBSTYLE_REVERSE_PROXY_IPS',
'CFG_BIBEDIT_AUTOCOMPLETE_INSTITUTIONS_FIELDS',
'CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS',
'CFG_BIBFORMAT_HIDDEN_FILE_FORMATS',
'CFG_FLASK_DISABLED_BLUEPRINTS',
'CFG_DEVEL_TOOLS',
'CFG_BIBFIELD_MASTER_FORMATS',
'CFG_OPENID_PROVIDERS',
'CFG_OAUTH1_PROVIDERS',
'CFG_OAUTH2_PROVIDERS',
'CFG_BIBFORMAT_CACHED_FORMATS',
'CFG_BIBEDIT_ADD_TICKET_RT_QUEUES',
'CFG_BIBAUTHORID_ENABLED_REMOTE_LOGIN_SYSTEMS',]:
out = "["
for elem in option_value[1:-1].split(","):
if elem:
elem = elem.strip()
if option_name in ['CFG_WEBSEARCH_ENABLED_SEARCH_INTERFACES']:
# 3d1) integer values
out += "%i, " % int(elem)
else:
# 3d2) string values
out += "'%s', " % elem
out += "]"
option_value = out
## 3e) special cases: multiline
if option_name == 'CFG_OAI_IDENTIFY_DESCRIPTION':
# make triple quotes
option_value = '""' + option_value + '""'
## 3f) ignore some options:
if option_name.startswith('CFG_SITE_NAME_INTL'):
# treated elsewhere
return
## 3g) special cases: float
if option_name in ['CFG_BIBDOCFILE_MD5_CHECK_PROBABILITY',
'CFG_BIBMATCH_LOCAL_SLEEPTIME',
'CFG_BIBMATCH_REMOTE_SLEEPTIME',
'CFG_PLOTEXTRACTOR_DOWNLOAD_TIMEOUT',
'CFG_BIBMATCH_FUZZY_MATCH_VALIDATION_LIMIT']:
option_value = float(option_value[1:-1])
## 3h) special cases: bibmatch validation list
if option_name in ['CFG_BIBMATCH_MATCH_VALIDATION_RULESETS']:
option_value = option_value[1:-1]
## 4a) dropped variables
if option_name in ['CFG_BATCHUPLOADER_WEB_ROBOT_AGENT']:
print(("""ERROR: CFG_BATCHUPLOADER_WEB_ROBOT_AGENT has been dropped in favour of
CFG_BATCHUPLOADER_WEB_ROBOT_AGENTS.
Please, update your invenio-local.conf file accordingly."""), file=sys.stderr)
option_value = option_value[1:-1]
elif option_name in ['CFG_WEBSUBMIT_DOCUMENT_FILE_MANAGER_DOCTYPES',
'CFG_WEBSUBMIT_DOCUMENT_FILE_MANAGER_RESTRICTIONS',
'CFG_WEBSUBMIT_DOCUMENT_FILE_MANAGER_MISC',
'CFG_WEBSUBMIT_FILESYSTEM_BIBDOC_GROUP_LIMIT',
'CFG_WEBSUBMIT_ADDITIONAL_KNOWN_FILE_EXTENSIONS',
'CFG_WEBSUBMIT_DESIRED_CONVERSIONS']:
new_option_name = option_name.replace('WEBSUBMIT', 'BIBDOCFILE')
print(("""ERROR: %s has been renamed to %s.
Please, update your invenio-local.conf file accordingly.""" % (option_name, new_option_name)), file=sys.stderr)
option_name = new_option_name
elif option_name in ['CFG_WEBSTYLE_INSPECT_TEMPLATES']:
print(("""ERROR: CFG_WEBSTYLE_INSPECT_TEMPLATES has been dropped in favour of
CFG_DEVEL_TOOLS.
Please, update your invenio-local.conf file accordingly."""), file=sys.stderr)
return
## 5) finally, return output line:
return '%s = %s' % (option_name, option_value)
def update_config_py(conf):
print('>>> NOT NEEDED!!!')
print('>>> quiting ...')
return
def cli_cmd_update_config_py(conf):
"""
Update new config.py from conf options, keeping previous
config.py in a backup copy.
"""
update_config_py(conf)
# from invenio.base.scripts.config import main
# warn('inveniocfg --update-config-py is deprecated. Using instead: inveniomanage config update')
# sys_argv = sys.argv
# sys.argv = 'config_manager.py update'.split()
# main()
# sys.argv = sys_argv
def cli_cmd_update_dbquery_py(conf):
"""
Update lib/dbquery.py file with DB parameters read from conf file.
Note: this edits dbquery.py in situ, taking a backup first.
Use only when you know what you are doing.
"""
print(">>> Going to update dbquery.py...")
## location where dbquery.py is:
dbqueryconfigpyfile = conf.get("Invenio", "CFG_PYLIBDIR") + \
os.sep + 'invenio' + os.sep + 'dbquery_config.py'
## backup current dbquery.py file:
if os.path.exists(dbqueryconfigpyfile + 'c'):
shutil.copy(dbqueryconfigpyfile + 'c', dbqueryconfigpyfile + 'c.OLD')
out = ["%s = '%s'\n" % (item.upper(), value) \
for item, value in conf.items('Invenio') \
if item.upper().startswith('CFG_DATABASE_')]
fdesc = open(dbqueryconfigpyfile, 'w')
fdesc.write("# -*- coding: utf-8 -*-\n")
fdesc.writelines(out)
fdesc.close()
print("You may want to restart Apache now.")
print(">>> dbquery.py updated successfully.")
def cli_cmd_update_dbexec(conf):
"""
Update bin/dbexec file with DB parameters read from conf file.
Note: this edits dbexec in situ, taking a backup first.
Use only when you know what you are doing.
"""
print(">>> Going to update dbexec...")
## location where dbexec is:
dbexecfile = conf.get("Invenio", "CFG_BINDIR") + \
os.sep + 'dbexec'
## backup current dbexec file:
if os.path.exists(dbexecfile):
shutil.copy(dbexecfile, dbexecfile + '.OLD')
## replace db parameters via sed:
out = ''
for line in open(dbexecfile, 'r').readlines():
match = re.search(r'^CFG_DATABASE_(HOST|PORT|NAME|USER|PASS|SLAVE)(\s*=\s*)\'.*\'$', line)
if match:
dbparam = 'CFG_DATABASE_' + match.group(1)
out += "%s%s'%s'\n" % (dbparam, match.group(2),
conf.get("Invenio", dbparam))
else:
out += line
fdesc = open(dbexecfile, 'w')
fdesc.write(out)
fdesc.close()
print(">>> dbexec updated successfully.")
def cli_cmd_update_bibconvert_tpl(conf):
"""
Update bibconvert/config/*.tpl files looking for 856
http://.../CFG_SITE_RECORD lines, replacing URL with CFG_SITE_URL taken
from conf file. Note: this edits tpl files in situ, taking a
backup first. Use only when you know what you are doing.
"""
from invenio.modules.converter.manage import main
warn('inveniocfg --update-bibconvert-tpl is deprecated. Using instead: inveniomanage converter update')
sys_argv = sys.argv
sys.argv = 'bibconvert_manager.py update'.split()
main()
sys.argv = sys_argv
def cli_cmd_update_web_tests(conf):
"""
Update web test cases lib/webtest/test_*.html looking for
<td>http://.+?[</] strings and replacing them with CFG_SITE_URL
taken from conf file. Note: this edits test files in situ, taking
a backup first. Use only when you know what you are doing.
"""
print(">>> Going to update web tests...")
## location where test_*.html files are:
testdir = conf.get("Invenio", 'CFG_PREFIX') + os.sep + \
'lib' + os.sep + 'webtest' + os.sep + 'invenio'
## find all test_*.html files:
for testfilename in os.listdir(testdir):
if testfilename.startswith("test_") and \
testfilename.endswith(".html"):
## change test file:
testfile = testdir + os.sep + testfilename
shutil.copy(testfile, testfile + '.OLD')
out = ''
for line in open(testfile, 'r').readlines():
match = re.search(r'^(.*<td>)http://.+?([</].*)$', line)
if match:
out += "%s%s%s\n" % (match.group(1),
conf.get("Invenio", 'CFG_SITE_URL'),
match.group(2))
else:
match = re.search(r'^(.*<td>)/opt/invenio(.*)$', line)
if match:
out += "%s%s%s\n" % (match.group(1),
conf.get("Invenio", 'CFG_PREFIX'),
match.group(2))
else:
out += line
fdesc = open(testfile, 'w')
fdesc.write(out)
fdesc.close()
print(">>> web tests updated successfully.")
def cli_cmd_reset_sitename(conf):
"""
Reset collection-related tables with new CFG_SITE_NAME and
CFG_SITE_NAME_INTL* read from conf files.
"""
print(">>> Going to reset CFG_SITE_NAME and CFG_SITE_NAME_INTL...")
from invenio.legacy.dbquery import run_sql, IntegrityError
# reset CFG_SITE_NAME:
sitename = conf.get("Invenio", "CFG_SITE_NAME")
try:
run_sql("""INSERT INTO collection (id, name, dbquery, reclist) VALUES
(1,%s,NULL,NULL)""", (sitename,))
except IntegrityError:
run_sql("""UPDATE collection SET name=%s WHERE id=1""", (sitename,))
# reset CFG_SITE_NAME_INTL:
for lang in conf.get("Invenio", "CFG_SITE_LANGS"):
sitename_lang = conf.get("Invenio", "CFG_SITE_NAME_INTL")[lang]
try:
run_sql("""INSERT INTO collectionname (id_collection, ln, type, value) VALUES
(%s,%s,%s,%s)""", (1, lang, 'ln', sitename_lang))
except IntegrityError:
run_sql("""UPDATE collectionname SET value=%s
WHERE ln=%s AND id_collection=1 AND type='ln'""",
(sitename_lang, lang))
print("You may want to restart Apache now.")
print(">>> CFG_SITE_NAME and CFG_SITE_NAME_INTL* reset successfully.")
def cli_cmd_reset_recstruct_cache(conf):
"""If CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE is changed, this function
will adapt the database to either store or not store the recstruct
format."""
from intbitset import intbitset
from invenio.legacy.dbquery import run_sql, serialize_via_marshal
from invenio.legacy.search_engine import get_record, print_record
from invenio.legacy.bibsched.cli import server_pid, pidfile
enable_recstruct_cache = conf.get("Invenio", "CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE")
enable_recstruct_cache = enable_recstruct_cache in ('True', '1')
pid = server_pid(ping_the_process=False)
if pid:
print("ERROR: bibsched seems to run with pid %d, according to %s." % (pid, pidfile), file=sys.stderr)
print(" Please stop bibsched before running this procedure.", file=sys.stderr)
sys.exit(1)
if enable_recstruct_cache:
print(">>> Searching records which need recstruct cache resetting; this may take a while...")
all_recids = intbitset(run_sql("SELECT id FROM bibrec"))
good_recids = intbitset(run_sql("SELECT bibrec.id FROM bibrec JOIN bibfmt ON bibrec.id = bibfmt.id_bibrec WHERE format='recstruct' AND modification_date < last_updated"))
recids = all_recids - good_recids
print(">>> Generating recstruct cache...")
tot = len(recids)
count = 0
for recid in recids:
try:
value = serialize_via_marshal(get_record(recid))
except zlib.error, err:
print >> sys.stderr, "Looks like XM is corrupted for record %s. Let's recover it from bibxxx" % recid
run_sql("DELETE FROM bibfmt WHERE id_bibrec=%s AND format='xm'", (recid, ))
xm_value = zlib.compress(print_record(recid, 'xm'))
run_sql("INSERT INTO bibfmt(id_bibrec, format, last_updated, value) VALUES(%s, 'xm', NOW(), %s)", (recid, xm_value))
value = serialize_via_marshal(get_record(recid))
run_sql("DELETE FROM bibfmt WHERE id_bibrec=%s AND format='recstruct'", (recid, ))
run_sql("INSERT INTO bibfmt(id_bibrec, format, last_updated, value) VALUES(%s, 'recstruct', NOW(), %s)", (recid, value))
count += 1
if count % 1000 == 0:
print(" ... done records %s/%s" % (count, tot))
if count % 1000 != 0:
print(" ... done records %s/%s" % (count, tot))
print(">>> recstruct cache generated successfully.")
else:
print(">>> Cleaning recstruct cache...")
run_sql("DELETE FROM bibfmt WHERE format='recstruct'")
def cli_cmd_reset_recjson_cache(conf):
"""If CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE is changed, this function
will adapt the database to either store or not store the recjson
format."""
from invenio.legacy.bibfield.bibfield_manager import main
warn('inveniocfg --reset-recjson-cache is deprecated. Using instead: inveniomanage bibfield reset')
sys_argv = sys.argv
sys.argv = 'bibfield_manager.py reset'.split()
main()
sys.argv = sys_argv
-
def cli_cmd_reset_siteadminemail(conf):
"""
Reset user-related tables with new CFG_SITE_ADMIN_EMAIL read from conf files.
"""
print(">>> Going to reset CFG_SITE_ADMIN_EMAIL...")
from invenio.legacy.dbquery import run_sql
siteadminemail = conf.get("Invenio", "CFG_SITE_ADMIN_EMAIL")
run_sql("DELETE FROM user WHERE id=1")
run_sql("""INSERT INTO user (id, email, password, note, nickname) VALUES
(1, %s, AES_ENCRYPT(email, ''), 1, 'admin')""",
(siteadminemail,))
print("You may want to restart Apache now.")
print(">>> CFG_SITE_ADMIN_EMAIL reset successfully.")
def cli_cmd_reset_fieldnames(conf):
"""
Reset I18N field names such as author, title, etc and other I18N
ranking method names such as word similarity. Their translations
are taken from the PO files.
"""
print(">>> Going to reset I18N field names...")
from invenio.base.i18n import gettext_set_language, language_list_long
from invenio.legacy.dbquery import run_sql, IntegrityError
## get field id and name list:
field_id_name_list = run_sql("SELECT id, name FROM field")
## get rankmethod id and name list:
rankmethod_id_name_list = run_sql("SELECT id, name FROM rnkMETHOD")
## update names for every language:
for lang, dummy in language_list_long():
_ = gettext_set_language(lang)
## this list is put here in order for PO system to pick names
## suitable for translation
field_name_names = {"any field": _("any field"),
"title": _("title"),
"author": _("author"),
"abstract": _("abstract"),
"keyword": _("keyword"),
"report number": _("report number"),
"subject": _("subject"),
"reference": _("reference"),
"fulltext": _("fulltext"),
"collection": _("collection"),
"division": _("division"),
"year": _("year"),
"journal": _("journal"),
"experiment": _("experiment"),
"record ID": _("record ID")}
## update I18N names for every language:
for (field_id, field_name) in field_id_name_list:
if field_name in field_name_names:
try:
run_sql("""INSERT INTO fieldname (id_field,ln,type,value) VALUES
(%s,%s,%s,%s)""", (field_id, lang, 'ln',
field_name_names[field_name]))
except IntegrityError:
run_sql("""UPDATE fieldname SET value=%s
WHERE id_field=%s AND ln=%s AND type=%s""",
(field_name_names[field_name], field_id, lang, 'ln',))
## ditto for rank methods:
rankmethod_name_names = {"wrd": _("word similarity"),
"demo_jif": _("journal impact factor"),
"citation": _("times cited"),
"citerank_citation_t": _("time-decay cite count"),
"citerank_pagerank_c": _("all-time-best cite rank"),
"citerank_pagerank_t": _("time-decay cite rank"),}
for (rankmethod_id, rankmethod_name) in rankmethod_id_name_list:
if rankmethod_name in rankmethod_name_names:
try:
run_sql("""INSERT INTO rnkMETHODNAME (id_rnkMETHOD,ln,type,value) VALUES
(%s,%s,%s,%s)""", (rankmethod_id, lang, 'ln',
rankmethod_name_names[rankmethod_name]))
except IntegrityError:
run_sql("""UPDATE rnkMETHODNAME SET value=%s
WHERE id_rnkMETHOD=%s AND ln=%s AND type=%s""",
(rankmethod_name_names[rankmethod_name], rankmethod_id, lang, 'ln',))
print(">>> I18N field names reset successfully.")
def cli_check_openoffice(conf):
"""
If OpenOffice.org integration is enabled, checks whether the system is
properly configured.
"""
from invenio.legacy.bibsched.bibtask import check_running_process_user
from invenio.legacy.websubmit.file_converter import can_unoconv, get_file_converter_logger
logger = get_file_converter_logger()
for handler in logger.handlers:
logger.removeHandler(handler)
check_running_process_user()
print(">>> Checking if Libre/OpenOffice.org is correctly integrated...", end=' ')
sys.stdout.flush()
if can_unoconv(True):
print("ok")
else:
sys.exit(1)
def test_db_connection():
"""
Test DB connection, and if fails, advise user how to set it up.
Useful to be called during table creation.
"""
print("Testing DB connection...", end=' ')
from invenio.utils.text import wrap_text_in_a_box
from invenio.legacy.dbquery import run_sql, Error
## first, test connection to the DB server:
try:
run_sql("SHOW TABLES")
except Error as err:
from invenio.config import CFG_DATABASE_HOST, \
CFG_DATABASE_PORT, CFG_DATABASE_NAME, CFG_DATABASE_USER, \
CFG_DATABASE_PASS
print(wrap_text_in_a_box("""\
DATABASE CONNECTIVITY ERROR %(errno)d: %(errmsg)s.\n
Perhaps you need to set up database and connection rights?
If yes, then please login as MySQL admin user and run the
following commands now:
$ mysql -h %(dbhost)s -P %(dbport)s -u root -p mysql
mysql> CREATE DATABASE %(dbname)s DEFAULT CHARACTER SET utf8;
mysql> GRANT ALL PRIVILEGES ON %(dbname)s.*
TO %(dbuser)s@%(webhost)s IDENTIFIED BY '%(dbpass)s';
mysql> QUIT
The values printed above were detected from your
configuration. If they are not right, then please edit your
invenio-local.conf file and rerun 'inveniocfg --update-all' first.
If the problem is of different nature, then please inspect
the above error message and fix the problem before continuing.""" % \
{'errno': err.args[0],
'errmsg': err.args[1],
'dbname': CFG_DATABASE_NAME,
'dbhost': CFG_DATABASE_HOST,
'dbport': CFG_DATABASE_PORT,
'dbuser': CFG_DATABASE_USER,
'dbpass': CFG_DATABASE_PASS,
'webhost': CFG_DATABASE_HOST == 'localhost' and 'localhost' or os.popen('hostname -f', 'r').read().strip(),
}))
sys.exit(1)
print("ok")
## second, test insert/select of a Unicode string to detect
## possible Python/MySQL/MySQLdb mis-setup:
print("Testing Python/MySQL/MySQLdb UTF-8 chain...", end=' ')
try:
try:
beta_in_utf8 = "β" # Greek beta in UTF-8 is 0xCEB2
run_sql("CREATE TABLE test__invenio__utf8 (x char(1), y varbinary(2)) DEFAULT CHARACTER SET utf8 ENGINE=MyISAM;")
run_sql("INSERT INTO test__invenio__utf8 (x, y) VALUES (%s, %s)", (beta_in_utf8, beta_in_utf8))
res = run_sql("SELECT x,y,HEX(x),HEX(y),LENGTH(x),LENGTH(y),CHAR_LENGTH(x),CHAR_LENGTH(y) FROM test__invenio__utf8")
assert res[0] == ('\xce\xb2', '\xce\xb2', 'CEB2', 'CEB2', 2L, 2L, 1L, 2L)
run_sql("DROP TABLE test__invenio__utf8")
except Exception as err:
print(wrap_text_in_a_box("""\
DATABASE RELATED ERROR %s\n
A problem was detected with the UTF-8 treatment in the chain
between the Python application, the MySQLdb connector, and
the MySQL database. You may perhaps have installed older
versions of some prerequisite packages?\n
Please check the INSTALL file and please fix this problem
before continuing.""" % err))
sys.exit(1)
finally:
run_sql("DROP TABLE IF EXISTS test__invenio__utf8")
print("ok")
def cli_cmd_create_secret_key(conf):
"""Generate and append CFG_SITE_SECRET_KEY to invenio-local.conf.
Useful for the installation process."""
from invenio.base.scripts.config import main
warn('inveniocfg --create-secret-key is deprecated. Using instead: inveniomanage config create secret-key')
sys_argv = sys.argv
sys.argv = 'config_manager.py create secret-key'.split()
main()
sys.argv = sys_argv
def cli_cmd_create_tables(conf):
"""Create and fill Invenio DB tables. Useful for the installation process."""
from invenio.base.scripts.database import main
warn('inveniocfg --create-tables is deprecated. Using instead: inveniomanage database create')
sys_argv = sys.argv
sys.argv = 'database_manager.py create'.split()
main()
sys.argv = sys_argv
def cli_cmd_load_webstat_conf(conf):
print(">>> Going to load WebStat config...")
from invenio.config import CFG_PREFIX
cmd = "%s/bin/webstatadmin --load-config" % CFG_PREFIX
if os.system(cmd):
print("ERROR: failed execution of", cmd)
sys.exit(1)
print(">>> WebStat config load successfully.")
def cli_cmd_load_bibfield_config(conf):
from invenio.legacy.bibfield.bibfield_manager import main
warn('inveniocfg --load-bibfield-conf is deprecated. Using instead: inveniomanage bibfield config load')
sys_argv = sys.argv
sys.argv = 'bibfield_manager.py config load'.split()
main()
sys.argv = sys_argv
def cli_cmd_drop_tables(conf):
"""Drop Invenio DB tables. Useful for the uninstallation process."""
print(">>> Going to drop tables and related data on filesystem ...")
from invenio.base.scripts.database import main
warn('inveniocfg --drop-tables is deprecated. Using instead: inveniomanage database drop')
sys_argv = sys.argv
if '--yes-i-know' in sys_argv:
sys.argv.append('--yes-i-know')
sys.argv = 'database_manager.py drop'.split()
main()
sys.argv = sys_argv
def cli_cmd_create_demo_site(conf):
"""Create demo site. Useful for testing purposes."""
print(">>> Going to create demo site...")
from invenio.config import CFG_PREFIX
from invenio.legacy.dbquery import run_sql
run_sql("TRUNCATE schTASK")
run_sql("TRUNCATE session")
run_sql("DELETE FROM user WHERE email=''")
for cmd in ["%s/bin/dbexec < %s/lib/sql/invenio/democfgdata.sql" % \
(CFG_PREFIX, CFG_PREFIX),]:
if os.system(cmd):
print("ERROR: failed execution of", cmd)
sys.exit(1)
cli_cmd_reset_fieldnames(conf) # needed for I18N demo ranking method names
for cmd in ["%s/bin/webaccessadmin -u admin -c -r -D" % CFG_PREFIX,
"%s/bin/webcoll -u admin" % CFG_PREFIX,
"%s/bin/webcoll 1" % CFG_PREFIX,
"%s/bin/bibsort -u admin --load-config" % CFG_PREFIX,
"%s/bin/bibsort 2" % CFG_PREFIX, ]:
if os.system(cmd):
print("ERROR: failed execution of", cmd)
sys.exit(1)
print(">>> Demo site created successfully.")
def cli_cmd_load_demo_records(conf):
"""Load demo records. Useful for testing purposes."""
from invenio.config import CFG_PREFIX
from invenio.legacy.dbquery import run_sql
print(">>> Going to load demo records...")
run_sql("TRUNCATE schTASK")
for cmd in ["%s/bin/bibupload -u admin -i %s" % (CFG_PREFIX,
pkg_resources.resource_filename('invenio.testsuite', os.path.join('data','demo_record_marc_data.xml'))),
"%s/bin/bibupload 1" % CFG_PREFIX,
"%s/bin/bibdocfile --textify --with-ocr --recid 97" % CFG_PREFIX,
"%s/bin/bibdocfile --textify --all" % CFG_PREFIX,
"%s/bin/bibindex -u admin" % CFG_PREFIX,
"%s/bin/bibindex 2" % CFG_PREFIX,
+ "%s/bin/bibindex -u admin -w global" % CFG_PREFIX,
+ "%s/bin/bibindex 3" % CFG_PREFIX,
"%s/bin/bibreformat -u admin -o HB" % CFG_PREFIX,
- "%s/bin/bibreformat 3" % CFG_PREFIX,
+ "%s/bin/bibreformat 4" % CFG_PREFIX,
"%s/bin/webcoll -u admin" % CFG_PREFIX,
- "%s/bin/webcoll 4" % CFG_PREFIX,
+ "%s/bin/webcoll 5" % CFG_PREFIX,
"%s/bin/bibrank -u admin" % CFG_PREFIX,
- "%s/bin/bibrank 5" % CFG_PREFIX,
+ "%s/bin/bibrank 6" % CFG_PREFIX,
"%s/bin/bibsort -u admin -R" % CFG_PREFIX,
- "%s/bin/bibsort 6" % CFG_PREFIX,
+ "%s/bin/bibsort 7" % CFG_PREFIX,
"%s/bin/oairepositoryupdater -u admin" % CFG_PREFIX,
- "%s/bin/oairepositoryupdater 7" % CFG_PREFIX,
- "%s/bin/bibupload 8" % CFG_PREFIX,]:
+ "%s/bin/oairepositoryupdater 8" % CFG_PREFIX,
+ "%s/bin/bibupload 9" % CFG_PREFIX,]:
if os.system(cmd):
print("ERROR: failed execution of", cmd)
sys.exit(1)
print(">>> Demo records loaded successfully.")
def cli_cmd_remove_demo_records(conf):
"""Remove demo records. Useful when you are finished testing."""
print(">>> Going to remove demo records...")
from invenio.config import CFG_PREFIX
from invenio.legacy.dbquery import run_sql
from invenio.utils.text import wrap_text_in_a_box, wait_for_user
wait_for_user(wrap_text_in_a_box("""WARNING: You are going to destroy
your records and documents!"""))
if os.path.exists(CFG_PREFIX + os.sep + 'var' + os.sep + 'data'):
shutil.rmtree(CFG_PREFIX + os.sep + 'var' + os.sep + 'data')
run_sql("TRUNCATE schTASK")
for cmd in ["%s/bin/dbexec < %s/lib/sql/invenio/tabbibclean.sql" % (CFG_PREFIX, CFG_PREFIX),
"%s/bin/webcoll -u admin" % CFG_PREFIX,
"%s/bin/webcoll 1" % CFG_PREFIX,]:
if os.system(cmd):
print("ERROR: failed execution of", cmd)
sys.exit(1)
print(">>> Demo records removed successfully.")
def cli_cmd_drop_demo_site(conf):
"""Drop demo site completely. Useful when you are finished testing."""
print(">>> Going to drop demo site...")
from invenio.utils.text import wrap_text_in_a_box, wait_for_user
wait_for_user(wrap_text_in_a_box("""WARNING: You are going to destroy
your site and documents!"""))
cli_cmd_drop_tables(conf)
cli_cmd_create_tables(conf)
cli_cmd_remove_demo_records(conf)
print(">>> Demo site dropped successfully.")
def cli_cmd_run_unit_tests(conf):
"""Run unit tests, usually on the working demo site."""
from invenio.testsuite import build_and_run_unit_test_suite
if not build_and_run_unit_test_suite():
sys.exit(1)
def cli_cmd_run_js_unit_tests(conf):
"""Run JavaScript unit tests, usually on the working demo site."""
from invenio.testsuite import build_and_run_js_unit_test_suite
if not build_and_run_js_unit_test_suite():
sys.exit(1)
def cli_cmd_run_regression_tests(conf):
"""Run regression tests, usually on the working demo site."""
from invenio.testsuite import build_and_run_regression_test_suite
if not build_and_run_regression_test_suite():
sys.exit(1)
def cli_cmd_run_web_tests(conf):
"""Run web tests in a browser. Requires Firefox with Selenium."""
from invenio.testsuite import build_and_run_web_test_suite
if not build_and_run_web_test_suite():
sys.exit(1)
def cli_cmd_run_flask_tests(conf):
"""Run flask tests."""
from invenio.testsuite import build_and_run_flask_test_suite
build_and_run_flask_test_suite()
def _detect_ip_address():
"""Detect IP address of this computer. Useful for creating Apache
vhost conf snippet on RHEL like machines.
@return: IP address, or '*' if cannot detect
@rtype: string
@note: creates socket for real in order to detect real IP address,
not the loopback one.
"""
try:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('invenio-software.org', 0))
return s.getsockname()[0]
except:
return '*'
def cli_cmd_create_apache_conf(conf):
"""
Create Apache conf files for this site, keeping previous
files in a backup copy.
"""
from invenio.apache_manager import main
warn('inveniocfg --create-apache-conf is deprecated. Using instead: inveniomanage apache create-config')
sys_argv = sys.argv
sys.argv = 'apache_manager.py create-config'.split()
main()
sys.argv = sys_argv
def cli_cmd_get(conf, varname):
"""
Return value of VARNAME read from CONF files. Useful for
third-party programs to access values of conf options such as
CFG_PREFIX. Return None if VARNAME is not found.
"""
from invenio.base.scripts.config import main
warn('inveniocfg --get="%(varname)s" is deprecated. '
'Using instead: inveniomanage config get "%(varname)s"' % {
'varname': varname
})
sys_argv = sys.argv
sys.argv = 'config_manager.py get'.split()
sys.argv.append(varname)
try:
main()
except SystemExit:
pass
sys.argv = sys_argv
def cli_cmd_list(conf):
"""
Print a list of all conf options and values from CONF.
"""
from invenio.base.scripts.config import main
warn('inveniocfg --list is deprecated. '
'Using instead: inveniomanage config list')
sys_argv = sys.argv
sys.argv = 'config_manager.py list'.split()
main()
sys.argv = sys_argv
def _grep_version_from_executable(path_to_exec, version_regexp):
"""
Try to detect a program version by digging into its binary
PATH_TO_EXEC and looking for VERSION_REGEXP. Return program
version as a string. Return empty string if not succeeded.
"""
from invenio.utils.shell import run_shell_command
exec_version = ""
if os.path.exists(path_to_exec):
dummy1, cmd2_out, dummy2 = run_shell_command("strings %s | grep %s",
(path_to_exec, version_regexp))
if cmd2_out:
for cmd2_out_line in cmd2_out.split("\n"):
if len(cmd2_out_line) > len(exec_version):
# the longest the better
exec_version = cmd2_out_line
return exec_version
def cli_cmd_detect_system_details(conf):
"""
Detect and print system details such as Apache/Python/MySQL
versions etc. Useful for debugging problems on various OS.
"""
from invenio.base.manage import main
warn('inveniocfg --detect-system-name is deprecated. Using instead: inveniomanage detect-system-name')
sys_argv = sys.argv
sys.argv = 'inveniomanage detect-system-name'.split()
main()
sys.argv = sys_argv
def cli_cmd_upgrade(conf):
"""
Command for applying upgrades
"""
from invenio.modules.upgrader.manage import main
warn('inveniocfg --upgrade-check is deprecated. Using instead: inveniomanage upgrade run')
sys_argv = sys.argv
sys.argv = 'modules.upgrader.manage.py run'.split()
main()
sys.argv = sys_argv
def cli_cmd_upgrade_check(conf):
"""
Command for running pre-upgrade checks
"""
from invenio.modules.upgrader.manage import main
warn('inveniocfg --upgrade-check is deprecated. Using instead: inveniomanage upgrade check')
sys_argv = sys.argv
sys.argv = 'modules.upgrader.manage.py check'.split()
main()
sys.argv = sys_argv
def cli_cmd_upgrade_show_pending(conf):
"""
Command for showing upgrades ready to be applied
"""
from invenio.modules.upgrader.manage import main
warn('inveniocfg --upgrade-show-pending is deprecated. Using instead: inveniomanage upgrade show pending')
sys_argv = sys.argv
sys.argv = 'modules.upgrader.manage.py show pending'.split()
main()
sys.argv = sys_argv
def cli_cmd_upgrade_show_applied(conf):
"""
Command for showing all upgrades already applied.
"""
from invenio.modules.upgrader.manage import main
warn('inveniocfg --upgrade-show-applied is deprecated. Using instead: inveniomanage upgrade show applied')
sys_argv = sys.argv
sys.argv = 'modules.upgrader.manage.py show applied'.split()
main()
sys.argv = sys_argv
def prepare_option_parser():
"""Parse the command line options."""
class InvenioOption(Option):
"""
Option class that implements the action 'store_append_const' which will
1) append <const> to list in options.<dest>
2) take a value and store in options.<const>
Useful for e.g. appending a const to an actions list, while also taking
an option value and storing it.
This ensures that we can run actions in the order they are given on the
command-line.
Python 2.4 compatibility note: *append_const* action is not available in
Python 2.4, so it is implemented here, together with the new action
*store_append_const*.
"""
ACTIONS = Option.ACTIONS + ("store_append_const", "append_const")
STORE_ACTIONS = Option.STORE_ACTIONS + ("store_append_const", "append_const")
TYPED_ACTIONS = Option.TYPED_ACTIONS + ("store_append_const", )
ALWAYS_TYPED_ACTIONS = Option.ALWAYS_TYPED_ACTIONS + ("store_append_const", )
CONST_ACTIONS = getattr(Option, 'CONST_ACTIONS', ()) + ("store_append_const", "append_const")
def take_action(self, action, dest, opt, value, values, parser):
if action == "store_append_const":
# Combination of 'store' and 'append_const' actions
values.ensure_value(dest, []).append(self.const)
value_dest = self.const.replace('-', '_')
setattr(values, value_dest, value)
elif action == "append_const" and not hasattr(Option, 'CONST_ACTIONS'):
values.ensure_value(dest, []).append(self.const)
else:
Option.take_action(self, action, dest, opt, value, values, parser)
def _check_const(self):
if self.action not in self.CONST_ACTIONS and self.const is not None:
raise OptionError(
"'const' must not be supplied for action %r" % self.action,
self)
CHECK_METHODS = [
Option._check_action,
Option._check_type,
Option._check_choice,
Option._check_dest,
_check_const,
Option._check_nargs,
Option._check_callback,
]
parser = OptionParser(option_class=InvenioOption, description="Invenio configuration and administration CLI tool", formatter=IndentedHelpFormatter(max_help_position=31))
parser.add_option("-V", "--version", action="store_true", help="print version number")
finish_options = OptionGroup(parser, "Options to finish your installation")
finish_options.add_option("", "--create-secret-key", dest='actions', const='create-secret-key', action="append_const", help="generate random CFG_SITE_SECRET_KEY")
finish_options.add_option("", "--create-apache-conf", dest='actions', const='create-apache-conf', action="append_const", help="create Apache configuration files")
finish_options.add_option("", "--create-tables", dest='actions', const='create-tables', action="append_const", help="create DB tables for Invenio")
finish_options.add_option("", "--load-bibfield-conf", dest='actions', const='load-bibfield-conf', action="append_const", help="load bibfield configuration file")
finish_options.add_option("", "--load-webstat-conf", dest='actions', const='load-webstat-conf', action="append_const", help="load the WebStat configuration")
finish_options.add_option("", "--drop-tables", dest='actions', const='drop-tables', action="append_const", help="drop DB tables of Invenio")
finish_options.add_option("", "--check-openoffice", dest='actions', const='check-openoffice', action="append_const", help="check for correctly set up of openoffice temporary directory")
parser.add_option_group(finish_options)
demotest_options = OptionGroup(parser, "Options to set up and test a demo site")
demotest_options.add_option("", "--create-demo-site", dest='actions', const='create-demo-site', action="append_const", help="create demo site")
demotest_options.add_option("", "--load-demo-records", dest='actions', const='load-demo-records', action="append_const", help="load demo records")
demotest_options.add_option("", "--remove-demo-records", dest='actions', const='remove-demo-records', action="append_const", help="remove demo records, keeping demo site")
demotest_options.add_option("", "--drop-demo-site", dest='actions', const='drop-demo-site', action="append_const", help="drop demo site configurations too")
demotest_options.add_option("", "--run-unit-tests", dest='actions', const='run-unit-tests', action="append_const", help="run unit test suite (needs demo site)")
demotest_options.add_option("", "--run-js-unit-tests", dest='actions', const='run-js-unit-tests', action="append_const", help="run JS unit test suite (needs demo site)")
demotest_options.add_option("", "--run-regression-tests", dest='actions', const='run-regression-tests', action="append_const", help="run regression test suite (needs demo site)")
demotest_options.add_option("", "--run-web-tests", dest='actions', const='run-web-tests', action="append_const", help="run web tests in a browser (needs demo site, Firefox, Selenium IDE)")
demotest_options.add_option("", "--run-flask-tests", dest='actions', const='run-flask-tests', action="append_const", help="run Flask test suite")
parser.add_option_group(demotest_options)
config_options = OptionGroup(parser, "Options to update config files in situ")
config_options.add_option("", "--update-all", dest='actions', const='update-all', action="append_const", help="perform all the update options")
config_options.add_option("", "--update-config-py", dest='actions', const='update-config-py', action="append_const", help="update config.py file from invenio.conf file")
config_options.add_option("", "--update-dbquery-py", dest='actions', const='update-dbquery-py', action="append_const", help="update dbquery.py with DB credentials from invenio.conf")
config_options.add_option("", "--update-dbexec", dest='actions', const='update-dbexec', action="append_const", help="update dbexec with DB credentials from invenio.conf")
config_options.add_option("", "--update-bibconvert-tpl", dest='actions', const='update-bibconvert-tpl', action="append_const", help="update bibconvert templates with CFG_SITE_URL from invenio.conf")
config_options.add_option("", "--update-web-tests", dest='actions', const='update-web-tests', action="append_const", help="update web test cases with CFG_SITE_URL from invenio.conf")
parser.add_option_group(config_options)
reset_options = OptionGroup(parser, "Options to update DB tables")
reset_options.add_option("", "--reset-all", dest='actions', const='reset-all', action="append_const", help="perform all the reset options")
reset_options.add_option("", "--reset-sitename", dest='actions', const='reset-sitename', action="append_const", help="reset tables to take account of new CFG_SITE_NAME*")
reset_options.add_option("", "--reset-siteadminemail", dest='actions', const='reset-siteadminemail', action="append_const", help="reset tables to take account of new CFG_SITE_ADMIN_EMAIL")
reset_options.add_option("", "--reset-fieldnames", dest='actions', const='reset-fieldnames', action="append_const", help="reset tables to take account of new I18N names from PO files")
reset_options.add_option("", "--reset-recstruct-cache", dest='actions', const='reset-recstruct-cache', action="append_const", help="reset record structure cache according to CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE")
reset_options.add_option("", "--reset-recjson-cache", dest='actions', const='reset-recjson-cache', action="append_const", help="reset record json structure cache according to CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE")
parser.add_option_group(reset_options)
upgrade_options = OptionGroup(parser, "Options to upgrade your installation")
upgrade_options.add_option("", "--upgrade", dest='actions', const='upgrade', action="append_const", help="apply all pending upgrades")
upgrade_options.add_option("", "--upgrade-check", dest='actions', const='upgrade-check', action="append_const", help="run pre-upgrade checks for pending upgrades")
upgrade_options.add_option("", "--upgrade-show-pending", dest='actions', const='upgrade-show-pending', action="append_const", help="show pending upgrades")
upgrade_options.add_option("", "--upgrade-show-applied", dest='actions', const='upgrade-show-applied', action="append_const", help="show history of applied upgrades")
upgrade_options.add_option("", "--upgrade-create-standard-recipe", dest='actions', metavar='REPOSITORY[,DIR]', const='upgrade-create-standard-recipe', action="append_const", help="use: inveniomanage upgrade create recipe")
upgrade_options.add_option("", "--upgrade-create-release-recipe", dest='actions', metavar='REPOSITORY[,DIR]', const='upgrade-create-release-recipe', action="append_const", help="use: inveniomanage upgrade create release")
parser.add_option_group(upgrade_options)
helper_options = OptionGroup(parser, "Options to help the work")
helper_options.add_option("", "--list", dest='actions', const='list', action="append_const", help="print names and values of all options from conf files")
helper_options.add_option("", "--get", dest='actions', const='get', action="store_append_const", metavar="OPTION", help="get value of a given option from conf files")
helper_options.add_option("", "--conf-dir", action="store", metavar="PATH", help="path to directory where invenio*.conf files are [optional]")
helper_options.add_option("", "--detect-system-details", dest='actions', const='detect-system-details', action="append_const", help="print system details such as Apache/Python/MySQL versions")
parser.add_option_group(helper_options)
parser.add_option('--yes-i-know', action='store_true', dest='yes-i-know', help='use with care!')
parser.add_option('-x', '--stop', action='store_true', dest='stop_on_error', help='When running tests, stop at first error')
return parser
def prepare_conf(options):
""" Read configuration files """
from flask import current_app
conf = ConfigParser()
conf.add_section('Invenio')
for (k, v) in iteritems(current_app.config):
conf.set('Invenio', k, v)
return conf
def main(*cmd_args):
"""Main entry point."""
# Allow easier testing
if not cmd_args:
cmd_args = sys.argv[1:]
# Parse arguments
parser = prepare_option_parser()
(options, dummy_args) = parser.parse_args(list(cmd_args))
if getattr(options, 'stop_on_error', False):
from invenio.testutils import wrap_failfast
wrap_failfast()
if getattr(options, 'version', False):
from invenio.base import manage
warn('inveniocfg --version is deprecated. Using instead: inveniomanage version')
sys_argv = sys.argv
sys.argv = 'inveniomanage.py version'.split()
manage.main()
sys.argv = sys_argv
else:
# Read configuration
try:
conf = prepare_conf(options)
except Exception as e:
print(e)
sys.exit(1)
## Decide what to do
actions = getattr(options, 'actions', None)
if not actions:
print("""ERROR: Please specify a command. Please see '--help'.""")
sys.exit(1)
if len(actions) > 1:
print("""ERROR: Please specify only one command. Please see '--help'.""")
sys.exit(1)
for action in actions:
if action == 'get':
cli_cmd_get(conf, getattr(options, 'get', None))
elif action == 'list':
cli_cmd_list(conf)
elif action == 'detect-system-details':
cli_cmd_detect_system_details(conf)
elif action == 'create-secret-key':
cli_cmd_create_secret_key(conf)
elif action == 'create-tables':
cli_cmd_create_tables(conf)
elif action == 'load-webstat-conf':
cli_cmd_load_webstat_conf(conf)
elif action == 'drop-tables':
cli_cmd_drop_tables(conf)
elif action == 'check-openoffice':
cli_check_openoffice(conf)
elif action == 'load-bibfield-conf':
cli_cmd_load_bibfield_config(conf)
elif action == 'create-demo-site':
cli_cmd_create_demo_site(conf)
elif action == 'load-demo-records':
cli_cmd_load_demo_records(conf)
elif action == 'remove-demo-records':
cli_cmd_remove_demo_records(conf)
elif action == 'drop-demo-site':
cli_cmd_drop_demo_site(conf)
elif action == 'run-unit-tests':
cli_cmd_run_unit_tests(conf)
elif action == 'run-js-unit-tests':
cli_cmd_run_js_unit_tests(conf)
elif action == 'run-regression-tests':
cli_cmd_run_regression_tests(conf)
elif action == 'run-web-tests':
cli_cmd_run_web_tests(conf)
elif action == 'run-flask-tests':
cli_cmd_run_flask_tests(conf)
elif action == 'update-all':
for f in [cli_cmd_update_config_py,
cli_cmd_update_dbquery_py,
cli_cmd_update_dbexec,
cli_cmd_update_bibconvert_tpl,
cli_cmd_update_web_tests]:
try:
f(conf)
except:
pass
elif action == 'update-config-py':
cli_cmd_update_config_py(conf)
elif action == 'update-dbquery-py':
cli_cmd_update_dbquery_py(conf)
elif action == 'update-dbexec':
cli_cmd_update_dbexec(conf)
elif action == 'update-bibconvert-tpl':
cli_cmd_update_bibconvert_tpl(conf)
elif action == 'update-web-tests':
cli_cmd_update_web_tests(conf)
elif action == 'reset-all':
cli_cmd_reset_sitename(conf)
cli_cmd_reset_siteadminemail(conf)
cli_cmd_reset_fieldnames(conf)
cli_cmd_reset_recstruct_cache(conf)
elif action == 'reset-sitename':
cli_cmd_reset_sitename(conf)
elif action == 'reset-siteadminemail':
cli_cmd_reset_siteadminemail(conf)
elif action == 'reset-fieldnames':
cli_cmd_reset_fieldnames(conf)
elif action == 'reset-recstruct-cache':
cli_cmd_reset_recstruct_cache(conf)
elif action == 'reset-recjson-cache':
cli_cmd_reset_recjson_cache(conf)
elif action == 'create-apache-conf':
cli_cmd_create_apache_conf(conf)
elif action == 'upgrade':
cli_cmd_upgrade(conf)
elif action == 'upgrade-check':
cli_cmd_upgrade_check(conf)
elif action == 'upgrade-show-pending':
cli_cmd_upgrade_show_pending(conf)
elif action == 'upgrade-show-applied':
cli_cmd_upgrade_show_applied(conf)
elif action == 'upgrade-create-standard-recipe':
print('ERROR: inveniocfg --upgrade-create-release-recipe is not supported anymore. Use instead: inveniomanage upgrade create release', file=sys.stderr)
sys.exit(1)
elif action == 'upgrade-create-release-recipe':
print('ERROR: inveniocfg --upgrade-create-standard-recipe is not supported anymore. Use instead: inveniomanage upgrade create recipe', file=sys.stderr)
sys.exit(1)
else:
print("ERROR: Unknown command", action)
sys.exit(1)
if __name__ == '__main__':
main()
diff --git a/invenio/legacy/miscutil/data_cacher.py b/invenio/legacy/miscutil/data_cacher.py
index 446bc272c..c03c3c517 100644
--- a/invenio/legacy/miscutil/data_cacher.py
+++ b/invenio/legacy/miscutil/data_cacher.py
@@ -1,106 +1,106 @@
# -*- coding: utf-8 -*-
## This file is part of Invenio.
## Copyright (C) 2007, 2008, 2009, 2010, 2011 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
Tool for caching important infos, which are slow to rebuild, but that
rarely change.
"""
from invenio.legacy.dbquery import run_sql, get_table_update_time
import time
class InvenioDataCacherError(Exception):
"""Error raised by data cacher."""
pass
-class DataCacher:
+class DataCacher(object):
"""
DataCacher is an abstract cacher system, for caching informations
that are slow to retrieve but that don't change too much during
time.
The .timestamp and .cache objects are exposed to clients. Most
use cases use a dict internal structure for .cache, but some use
lists.
"""
def __init__(self, cache_filler, timestamp_verifier):
""" @param cache_filler: a function that fills the cache dictionary.
@param timestamp_verifier: a function that returns a timestamp for
checking if something has changed after cache creation.
"""
self.timestamp = 0 # WARNING: may be exposed to clients
self.cache = {} # WARNING: may be exposed to clients; lazy
# clients may even alter this object on the fly
if not callable(cache_filler):
raise InvenioDataCacherError, "cache_filler is not callable"
self.cache_filler = cache_filler
if not callable(timestamp_verifier):
raise InvenioDataCacherError, "timestamp_verifier is not callable"
self.timestamp_verifier = timestamp_verifier
self.is_ok_p = True
self.create_cache()
def clear(self):
"""Clear the cache rebuilding it."""
self.create_cache()
def create_cache(self):
"""
Create and populate cache by calling cache filler. Called on
startup and used later during runtime as needed by clients.
"""
self.cache = self.cache_filler()
self.timestamp = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
def recreate_cache_if_needed(self):
"""
Recreate cache if needed, by verifying the cache timestamp
against the timestamp verifier function.
"""
if self.timestamp_verifier() > self.timestamp:
self.create_cache()
class SQLDataCacher(DataCacher):
"""
SQLDataCacher is a cacher system, for caching single queries and
their results.
"""
def __init__(self, query, param=None, affected_tables=()):
""" @param query: the query to cache
@param param: its optional parameters as a tuple
@param affected_tables: the list of tables queried by the query.
"""
self.query = query
self.affected_tables = affected_tables
assert(affected_tables)
def cache_filler():
"""Standard SQL filler, with results from sql query."""
return run_sql(self.query, param)
def timestamp_verifier():
"""The standard timestamp verifier is looking at affected
tables time stamp."""
return max([get_table_update_time(table)
for table in self.affected_tables])
DataCacher.__init__(self, cache_filler, timestamp_verifier)
diff --git a/invenio/legacy/miscutil/sql/tabcreate.sql b/invenio/legacy/miscutil/sql/tabcreate.sql
index ad767a3da..2dc3af1df 100644
--- a/invenio/legacy/miscutil/sql/tabcreate.sql
+++ b/invenio/legacy/miscutil/sql/tabcreate.sql
@@ -1,4907 +1,5041 @@
-- This file is part of Invenio.
--- Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013 CERN.
+-- Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
--
-- Invenio is free software; you can redistribute it and/or
-- modify it under the terms of the GNU General Public License as
-- published by the Free Software Foundation; either version 2 of the
-- License, or (at your option) any later version.
--
-- Invenio is distributed in the hope that it will be useful, but
-- WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-- General Public License for more details.
--
-- You should have received a copy of the GNU General Public License
-- along with Invenio; if not, write to the Free Software Foundation, Inc.,
-- 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-- tables for bibliographic records:
CREATE TABLE IF NOT EXISTS bibrec (
id mediumint(8) unsigned NOT NULL auto_increment,
creation_date datetime NOT NULL default '0000-00-00',
modification_date datetime NOT NULL default '0000-00-00',
master_format varchar(16) NOT NULL default 'marc',
PRIMARY KEY (id),
KEY creation_date (creation_date),
KEY modification_date (modification_date)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib00x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib01x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib02x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib03x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib04x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib05x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib06x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib07x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib08x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib09x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib10x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib11x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib12x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib13x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib14x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib15x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib16x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib17x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib18x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib19x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib20x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib21x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib22x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib23x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib24x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib25x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib26x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib27x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib28x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib29x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib30x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib31x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib32x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib33x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib34x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib35x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib36x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib37x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib38x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib39x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib40x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib41x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib42x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib43x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib44x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib45x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib46x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib47x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib48x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib49x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib50x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib51x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib52x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib53x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib54x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib55x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib56x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib57x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib58x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib59x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib60x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib61x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib62x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib63x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib64x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib65x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib66x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib67x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib68x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib69x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib70x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib71x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib72x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib73x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib74x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib75x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib76x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib77x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib78x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib79x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib80x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib81x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib82x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib83x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib84x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib85x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(100)) -- URLs need usually a larger index for speedy lookups
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib86x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib87x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib88x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib89x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib90x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib91x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib92x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib93x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib94x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib95x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib96x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib97x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib98x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bib99x (
id mediumint(8) unsigned NOT NULL auto_increment,
tag varchar(6) NOT NULL default '',
value text NOT NULL,
PRIMARY KEY (id),
KEY kt (tag),
KEY kv (value(35))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib00x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib01x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib02x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib03x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib04x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib05x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib06x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib07x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib08x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib09x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib10x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib11x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib12x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib13x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib14x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib15x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib16x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib17x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib18x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib19x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib20x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib21x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib22x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib23x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib24x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib25x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib26x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib27x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib28x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib29x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib30x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib31x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib32x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib33x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib34x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib35x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib36x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib37x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib38x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib39x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib40x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib41x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib42x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib43x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib44x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib45x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib46x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib47x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib48x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib49x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib50x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib51x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib52x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib53x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib54x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib55x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib56x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib57x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib58x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib59x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib60x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib61x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib62x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib63x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib64x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib65x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib66x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib67x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib68x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib69x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib70x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib71x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib72x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib73x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib74x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib75x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib76x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib77x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib78x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib79x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib80x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib81x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib82x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib83x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib84x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib85x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib86x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib87x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib88x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib89x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib90x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib91x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib92x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib93x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib94x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib95x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib96x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib97x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib98x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bib99x (
id_bibrec mediumint(8) unsigned NOT NULL default '0',
id_bibxxx mediumint(8) unsigned NOT NULL default '0',
field_number smallint(5) unsigned default NULL,
KEY id_bibxxx (id_bibxxx),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
-- tables for bibliographic records formatted:
CREATE TABLE IF NOT EXISTS bibfmt (
id_bibrec int(8) unsigned NOT NULL default '0',
format varchar(10) NOT NULL default '',
last_updated datetime NOT NULL default '0000-00-00',
value longblob,
needs_2nd_pass TINYINT(1) DEFAULT 0,
PRIMARY KEY (id_bibrec, format),
KEY format (format),
KEY last_updated (last_updated)
) ENGINE=MyISAM;
-- tables for index files:
CREATE TABLE IF NOT EXISTS idxINDEX (
id mediumint(9) unsigned NOT NULL,
name varchar(50) NOT NULL default '',
description varchar(255) NOT NULL default '',
last_updated datetime NOT NULL default '0000-00-00 00:00:00',
stemming_language varchar(10) NOT NULL default '',
indexer varchar(10) NOT NULL default 'native',
synonym_kbrs varchar(255) NOT NULL default '',
remove_stopwords varchar(255) NOT NULL default '',
remove_html_markup varchar(10) NOT NULL default '',
remove_latex_markup varchar(10) NOT NULL default '',
tokenizer varchar(50) NOT NULL default '',
PRIMARY KEY (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxINDEXNAME (
id_idxINDEX mediumint(9) unsigned NOT NULL,
ln char(5) NOT NULL default '',
type char(3) NOT NULL default 'sn',
value varchar(255) NOT NULL,
PRIMARY KEY (id_idxINDEX,ln,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxINDEX_field (
id_idxINDEX mediumint(9) unsigned NOT NULL,
id_field mediumint(9) unsigned NOT NULL,
regexp_punctuation varchar(255) NOT NULL default "[\.\,\:\;\?\!\"]",
regexp_alphanumeric_separators varchar(255) NOT NULL default "[\!\"\#\$\%\&\'\(\)\*\+\,\-\.\/\:\;\<\=\>\?\@\[\\\]\^\_\`\{\|\}\~]",
PRIMARY KEY (id_idxINDEX,id_field)
) ENGINE=MyISAM;
-- this comment line here is just to fix the SQL display mode in Emacs '
CREATE TABLE IF NOT EXISTS idxINDEX_idxINDEX (
id_virtual mediumint(9) unsigned NOT NULL,
id_normal mediumint(9) unsigned NOT NULL,
PRIMARY KEY (id_virtual,id_normal)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD01F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD01R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
+CREATE TABLE IF NOT EXISTS idxWORD01Q (
+ id mediumint(10) unsigned NOT NULL auto_increment,
+ runtime datetime NOT NULL default '0000-00-00 00:00:00',
+ id_bibrec_low mediumint(9) unsigned NOT NULL,
+ id_bibrec_high mediumint(9) unsigned NOT NULL,
+ index_name varchar(50) NOT NULL default '',
+ mode varchar(50) NOT NULL default 'update',
+ PRIMARY KEY (id),
+ INDEX (index_name),
+ INDEX (runtime)
+) ENGINE=MyISAM;
+
CREATE TABLE IF NOT EXISTS idxWORD02F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD02R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD03F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD03R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD04F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD04R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD05F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD05R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD06F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD06R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD07F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD07R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD08F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD08R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD09F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD09R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD10F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD10R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD11F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD11R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD12F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD12R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD13F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD13R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD14F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD14R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD15F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD15R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD16F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD16R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD17F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD17R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD18F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD18R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD19F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD19R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD20F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD20R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD21F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD21R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD22F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD22R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD23F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD23R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD24F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD24R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD25F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD25R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD26F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxWORD26R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
+CREATE TABLE IF NOT EXISTS idxWORD27F (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term varchar(50) default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ UNIQUE KEY term (term)
+) ENGINE=MyISAM;
+
+CREATE TABLE IF NOT EXISTS idxWORD27R (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+) ENGINE=MyISAM;
+
+CREATE TABLE IF NOT EXISTS idxWORD28F (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term varchar(50) default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ UNIQUE KEY term (term)
+) ENGINE=MyISAM;
+
+CREATE TABLE IF NOT EXISTS idxWORD28R (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+) ENGINE=MyISAM;
+
CREATE TABLE IF NOT EXISTS idxPAIR01F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR01R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
+CREATE TABLE IF NOT EXISTS idxPAIR01Q (
+ id mediumint(10) unsigned NOT NULL auto_increment,
+ runtime datetime NOT NULL default '0000-00-00 00:00:00',
+ id_bibrec_low mediumint(9) unsigned NOT NULL,
+ id_bibrec_high mediumint(9) unsigned NOT NULL,
+ index_name varchar(50) NOT NULL default '',
+ mode varchar(50) NOT NULL default 'update',
+ PRIMARY KEY (id),
+ INDEX (index_name),
+ INDEX (runtime)
+) ENGINE=MyISAM;
+
CREATE TABLE IF NOT EXISTS idxPAIR02F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR02R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR03F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR03R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR04F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR04R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR05F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR05R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR06F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR06R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR07F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR07R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR08F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR08R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR09F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR09R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR10F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR10R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR11F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR11R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR12F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR12R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR13F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR13R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR14F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR14R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR15F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR15R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR16F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR16R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR17F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR17R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR18F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR18R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR19F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR19R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR20F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR20R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR21F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR21R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR22F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR22R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR23F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR23R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR24F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR24R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR25F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR25R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR26F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(100) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPAIR26R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
+CREATE TABLE IF NOT EXISTS idxPAIR27F (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term varchar(100) default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ UNIQUE KEY term (term)
+) ENGINE=MyISAM;
+
+CREATE TABLE IF NOT EXISTS idxPAIR27R (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+) ENGINE=MyISAM;
+
+CREATE TABLE IF NOT EXISTS idxPAIR28F (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term varchar(100) default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ UNIQUE KEY term (term)
+) ENGINE=MyISAM;
+
+CREATE TABLE IF NOT EXISTS idxPAIR28R (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+) ENGINE=MyISAM;
+
CREATE TABLE IF NOT EXISTS idxPHRASE01F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE01R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE02F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
+CREATE TABLE IF NOT EXISTS idxPHRASE01Q (
+ id mediumint(10) unsigned NOT NULL auto_increment,
+ runtime datetime NOT NULL default '0000-00-00 00:00:00',
+ id_bibrec_low mediumint(9) unsigned NOT NULL,
+ id_bibrec_high mediumint(9) unsigned NOT NULL,
+ index_name varchar(50) NOT NULL default '',
+ mode varchar(50) NOT NULL default 'update',
+ PRIMARY KEY (id),
+ INDEX (index_name),
+ INDEX (runtime)
+) ENGINE=MyISAM;
+
CREATE TABLE IF NOT EXISTS idxPHRASE02R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE03F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE03R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE04F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE04R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE05F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE05R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE06F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE06R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE07F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE07R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE08F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE08R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE09F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE09R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE10F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE10R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE11F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE11R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE12F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE12R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE13F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE13R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE14F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE14R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE15F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE15R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE16F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE16R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE17F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE17R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE18F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE18R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE19F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE19R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE20F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE20R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE21F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE21R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE22F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE22R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE23F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE23R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE24F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE24R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE25F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE25R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE26F (
id mediumint(9) unsigned NOT NULL auto_increment,
term text default NULL,
hitlist longblob,
PRIMARY KEY (id),
KEY term (term(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS idxPHRASE26R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
+CREATE TABLE IF NOT EXISTS idxPHRASE27F (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term text default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ KEY term (term(50))
+) ENGINE=MyISAM;
+
+CREATE TABLE IF NOT EXISTS idxPHRASE27R (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+) ENGINE=MyISAM;
+
+CREATE TABLE IF NOT EXISTS idxPHRASE28F (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term text default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ KEY term (term(50))
+) ENGINE=MyISAM;
+
+CREATE TABLE IF NOT EXISTS idxPHRASE28R (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+) ENGINE=MyISAM;
+
-- tables for ranking:
CREATE TABLE IF NOT EXISTS rnkMETHOD (
id mediumint(9) unsigned NOT NULL auto_increment,
name varchar(20) NOT NULL default '',
last_updated datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS rnkMETHODNAME (
id_rnkMETHOD mediumint(9) unsigned NOT NULL,
ln char(5) NOT NULL default '',
type char(3) NOT NULL default 'sn',
value varchar(255) NOT NULL,
PRIMARY KEY (id_rnkMETHOD,ln,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS rnkMETHODDATA (
id_rnkMETHOD mediumint(9) unsigned NOT NULL,
relevance_data longblob,
PRIMARY KEY (id_rnkMETHOD)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS collection_rnkMETHOD (
id_collection mediumint(9) unsigned NOT NULL,
id_rnkMETHOD mediumint(9) unsigned NOT NULL,
score tinyint(4) unsigned NOT NULL default '0',
PRIMARY KEY (id_collection,id_rnkMETHOD)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS rnkWORD01F (
id mediumint(9) unsigned NOT NULL auto_increment,
term varchar(50) default NULL,
hitlist longblob,
PRIMARY KEY (id),
UNIQUE KEY term (term)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS rnkWORD01R (
id_bibrec mediumint(9) unsigned NOT NULL,
termlist longblob,
type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
KEY type (type),
PRIMARY KEY (id_bibrec,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS rnkAUTHORDATA (
aterm varchar(50) default NULL,
hitlist longblob,
UNIQUE KEY aterm (aterm)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS rnkPAGEVIEWS (
id_bibrec mediumint(8) unsigned default NULL,
id_user int(15) unsigned default '0',
client_host int(10) unsigned default NULL,
view_time datetime default '0000-00-00 00:00:00',
KEY view_time (view_time),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS rnkDOWNLOADS (
id_bibrec mediumint(8) unsigned default NULL,
download_time datetime default '0000-00-00 00:00:00',
client_host int(10) unsigned default NULL,
id_user int(15) unsigned default NULL,
id_bibdoc mediumint(9) unsigned default NULL,
file_version smallint(2) unsigned default NULL,
file_format varchar(50) NULL default NULL,
KEY download_time (download_time),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
-- a table for citations. record-cites-record
CREATE TABLE IF NOT EXISTS rnkCITATIONDICT (
citee int(10) unsigned NOT NULL,
citer int(10) unsigned NOT NULL,
last_updated datetime NOT NULL,
PRIMARY KEY id (citee, citer),
KEY reverse (citer, citee)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS rnkSELFCITEDICT (
citee int(10) unsigned NOT NULL,
citer int(10) unsigned NOT NULL,
last_updated datetime NOT NULL,
PRIMARY KEY id (citee, citer),
KEY reverse (citer, citee)
) ENGINE=MyISAM;
-- a table for logging changes in the citation dict
CREATE TABLE IF NOT EXISTS rnkCITATIONLOG (
id int(11) unsigned NOT NULL auto_increment,
citee int(10) unsigned NOT NULL,
citer int(10) unsigned NOT NULL,
`type` ENUM('added', 'removed'),
action_date datetime NOT NULL,
PRIMARY KEY (id),
KEY citee (citee),
KEY citer (citer)
) ENGINE=MyISAM;
-- a table for missing citations. This should be scanned by a program
-- occasionally to check if some publication has been cited more than
-- 50 times (or such), and alert cataloguers to create record for that
-- external citation
--
-- id_bibrec is the id of the record. extcitepubinfo is publication info
-- that looks in general like hep-th/0112088
CREATE TABLE IF NOT EXISTS rnkCITATIONDATAEXT (
id_bibrec int(8) unsigned,
extcitepubinfo varchar(255) NOT NULL,
PRIMARY KEY (id_bibrec, extcitepubinfo),
KEY extcitepubinfo (extcitepubinfo)
) ENGINE=MyISAM;
-- tables for self-citations computation
CREATE TABLE IF NOT EXISTS `rnkRECORDSCACHE` (
`id_bibrec` int(10) unsigned NOT NULL,
`authorid` bigint(10) NOT NULL,
PRIMARY KEY (`id_bibrec`,`authorid`)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS `rnkEXTENDEDAUTHORS` (
`id` int(10) unsigned NOT NULL,
`authorid` bigint(10) NOT NULL,
PRIMARY KEY (`id`,`authorid`)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS `rnkSELFCITES` (
`id_bibrec` int(10) unsigned NOT NULL,
`count` int(10) unsigned NOT NULL,
`references` text NOT NULL,
`last_updated` datetime NOT NULL,
PRIMARY KEY (`id_bibrec`)
) ENGINE=MyISAM;
-- a table for storing invalid or ambiguous references encountered
CREATE TABLE IF NOT EXISTS rnkCITATIONDATAERR (
`type` ENUM('multiple-matches', 'not-well-formed'),
citinfo varchar(255) NOT NULL default '',
PRIMARY KEY (`type`, citinfo)
) ENGINE=MyISAM;
-- tables for collections and collection tree:
CREATE TABLE IF NOT EXISTS collection (
id mediumint(9) unsigned NOT NULL auto_increment,
name varchar(255) NOT NULL,
dbquery text,
nbrecs int(10) unsigned default '0',
reclist longblob,
PRIMARY KEY (id),
UNIQUE KEY name (name),
KEY dbquery (dbquery(50))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS collectionname (
id_collection mediumint(9) unsigned NOT NULL,
ln char(5) NOT NULL default '',
type char(3) NOT NULL default 'sn',
value varchar(255) NOT NULL,
PRIMARY KEY (id_collection,ln,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS collection_collection (
id_dad mediumint(9) unsigned NOT NULL,
id_son mediumint(9) unsigned NOT NULL,
type char(1) NOT NULL default 'r',
score tinyint(4) unsigned NOT NULL default '0',
PRIMARY KEY (id_dad,id_son)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS collectionboxname (
id_collection mediumint(9) unsigned NOT NULL,
ln char(5) NOT NULL default '',
type char(1) NOT NULL default 'r',
value varchar(255) NOT NULL,
PRIMARY KEY (id_collection,ln,type)
) ENGINE=MyISAM;
-- tables for OAI sets:
CREATE TABLE IF NOT EXISTS oaiREPOSITORY (
id mediumint(9) unsigned NOT NULL auto_increment,
setName varchar(255) NOT NULL default '',
setSpec varchar(255) NOT NULL default 'GLOBAL_SET',
setCollection varchar(255) NOT NULL default '',
setDescription text NOT NULL default '',
setDefinition text NOT NULL default '',
setRecList longblob,
last_updated datetime NOT NULL default '1970-01-01',
p1 text NOT NULL default '',
f1 text NOT NULL default '',
m1 text NOT NULL default '',
p2 text NOT NULL default '',
f2 text NOT NULL default '',
m2 text NOT NULL default '',
p3 text NOT NULL default '',
f3 text NOT NULL default '',
m3 text NOT NULL default '',
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS oaiHARVEST (
id mediumint(9) unsigned NOT NULL auto_increment,
baseurl varchar(255) NOT NULL default '',
metadataprefix varchar(255) NOT NULL default 'oai_dc',
arguments BLOB NULL default NULL,
comment text,
name varchar(255) NOT NULL,
lastrun datetime,
frequency mediumint(12) NOT NULL default '0',
postprocess varchar(20) NOT NULL default 'h',
setspecs text NOT NULL default '',
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS oaiHARVESTLOG (
id_oaiHARVEST mediumint(9) unsigned NOT NULL REFERENCES oaiHARVEST, -- source we harvest from
id_bibrec mediumint(8) unsigned NOT NULL default '0', -- internal record id ( filled by bibupload )
bibupload_task_id int NOT NULL default 0, -- bib upload task number
oai_id varchar(40) NOT NULL default "", -- OAI record identifier we harvested
date_harvested datetime NOT NULL default '0000-00-00', -- when we harvested
date_inserted datetime NOT NULL default '0000-00-00', -- when it was inserted
inserted_to_db char(1) NOT NULL default 'P', -- where it was inserted (P=prod, H=holding-pen, etc)
PRIMARY KEY (bibupload_task_id, oai_id, date_harvested)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibHOLDINGPEN (
changeset_id INT NOT NULL AUTO_INCREMENT, -- the identifier of the changeset stored in the holding pen
changeset_date datetime NOT NULL DEFAULT '0000:00:00 00:00:00', -- when was the changeset inserted
changeset_xml longblob NOT NULL,
oai_id varchar(40) NOT NULL DEFAULT '', -- OAI identifier of concerned record
id_bibrec mediumint(8) unsigned NOT NULL default '0', -- record ID of concerned record (filled by bibupload)
PRIMARY KEY (changeset_id),
KEY changeset_date (changeset_date),
KEY id_bibrec (id_bibrec)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibARXIVPDF (
id_bibrec mediumint(8) unsigned NOT NULL,
status ENUM('ok', 'missing') NOT NULL,
date_harvested datetime NOT NULL,
version tinyint(2) NOT NULL,
PRIMARY KEY (id_bibrec),
KEY status (status)
) ENGINE=MyISAM;
-- tables for portal elements:
CREATE TABLE IF NOT EXISTS collection_portalbox (
id_collection mediumint(9) unsigned NOT NULL,
id_portalbox mediumint(9) unsigned NOT NULL,
ln char(5) NOT NULL default '',
position char(3) NOT NULL default 'top',
score tinyint(4) unsigned NOT NULL default '0',
PRIMARY KEY (id_collection,id_portalbox,ln)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS portalbox (
id mediumint(9) unsigned NOT NULL auto_increment,
title text NOT NULL,
body text NOT NULL,
UNIQUE KEY id (id)
) ENGINE=MyISAM;
-- tables for search examples:
CREATE TABLE IF NOT EXISTS collection_example (
id_collection mediumint(9) unsigned NOT NULL,
id_example mediumint(9) unsigned NOT NULL,
score tinyint(4) unsigned NOT NULL default '0',
PRIMARY KEY (id_collection,id_example)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS example (
id mediumint(9) unsigned NOT NULL auto_increment,
type text NOT NULL default '',
body text NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
-- tables for collection formats:
CREATE TABLE IF NOT EXISTS collection_format (
id_collection mediumint(9) unsigned NOT NULL,
id_format mediumint(9) unsigned NOT NULL,
score tinyint(4) unsigned NOT NULL default '0',
PRIMARY KEY (id_collection,id_format)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS format (
id mediumint(9) unsigned NOT NULL auto_increment,
name varchar(255) NOT NULL,
code varchar(6) NOT NULL,
description varchar(255) default '',
content_type varchar(255) default '',
visibility tinyint NOT NULL default '1',
last_updated datetime NOT NULL default '0000-00-00',
PRIMARY KEY (id),
UNIQUE KEY code (code)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS formatname (
id_format mediumint(9) unsigned NOT NULL,
ln char(5) NOT NULL default '',
type char(3) NOT NULL default 'sn',
value varchar(255) NOT NULL,
PRIMARY KEY (id_format,ln,type)
) ENGINE=MyISAM;
-- tables for collection detailed page options
CREATE TABLE IF NOT EXISTS collectiondetailedrecordpagetabs (
id_collection mediumint(9) unsigned NOT NULL,
tabs varchar(255) NOT NULL default '',
PRIMARY KEY (id_collection)
) ENGINE=MyISAM;
-- tables for search options and MARC tags:
CREATE TABLE IF NOT EXISTS collection_field_fieldvalue (
id_collection mediumint(9) unsigned NOT NULL,
id_field mediumint(9) unsigned NOT NULL,
id_fieldvalue mediumint(9) unsigned,
type char(3) NOT NULL default 'src',
score tinyint(4) unsigned NOT NULL default '0',
score_fieldvalue tinyint(4) unsigned NOT NULL default '0',
KEY id_collection (id_collection),
KEY id_field (id_field),
KEY id_fieldvalue (id_fieldvalue)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS field (
id mediumint(9) unsigned NOT NULL auto_increment,
name varchar(255) NOT NULL,
code varchar(255) NOT NULL,
PRIMARY KEY (id),
UNIQUE KEY code (code)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS fieldname (
id_field mediumint(9) unsigned NOT NULL,
ln char(5) NOT NULL default '',
type char(3) NOT NULL default 'sn',
value varchar(255) NOT NULL,
PRIMARY KEY (id_field,ln,type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS fieldvalue (
id mediumint(9) unsigned NOT NULL auto_increment,
name varchar(255) NOT NULL,
value text NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS field_tag (
id_field mediumint(9) unsigned NOT NULL,
id_tag mediumint(9) unsigned NOT NULL,
score tinyint(4) unsigned NOT NULL default '0',
PRIMARY KEY (id_field,id_tag)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS tag (
id mediumint(9) unsigned NOT NULL auto_increment,
name varchar(255) NOT NULL,
value char(6) NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
-- tables for file management
CREATE TABLE IF NOT EXISTS bibdoc (
id mediumint(9) unsigned NOT NULL auto_increment,
status text NOT NULL default '',
docname varchar(250) COLLATE utf8_bin default NULL, -- now NULL means that this is new version bibdoc
creation_date datetime NOT NULL default '0000-00-00',
modification_date datetime NOT NULL default '0000-00-00',
text_extraction_date datetime NOT NULL default '0000-00-00',
doctype varchar(255),
PRIMARY KEY (id),
KEY docname (docname),
KEY creation_date (creation_date),
KEY modification_date (modification_date)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibrec_bibdoc (
id_bibrec mediumint(9) unsigned NOT NULL default '0',
id_bibdoc mediumint(9) unsigned NOT NULL default '0',
docname varchar(250) COLLATE utf8_bin NOT NULL default 'file',
type varchar(255),
KEY docname (docname),
KEY (id_bibrec),
KEY (id_bibdoc)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibdoc_bibdoc (
id mediumint(9) unsigned NOT NULL auto_increment,
id_bibdoc1 mediumint(9) unsigned DEFAULT NULL,
version1 tinyint(4) unsigned, -- NULL means all versions
format1 varchar(50),
id_bibdoc2 mediumint(9) unsigned DEFAULT NULL,
version2 tinyint(4) unsigned, -- NULL means all versions
format2 varchar(50),
rel_type varchar(255),
KEY (id_bibdoc1),
KEY (id_bibdoc2),
KEY (id)
) ENGINE=MyISAM;
-- Storage of moreInfo fields
CREATE TABLE IF NOT EXISTS bibdocmoreinfo (
id_bibdoc mediumint(9) unsigned DEFAULT NULL,
version tinyint(4) unsigned DEFAULT NULL, -- NULL means all versions
format VARCHAR(50) DEFAULT NULL,
id_rel mediumint(9) unsigned DEFAULT NULL,
namespace VARCHAR(25) DEFAULT NULL, -- namespace in the moreinfo dictionary
data_key VARCHAR(25), -- key in the moreinfo dictionary
data_value MEDIUMBLOB,
KEY (id_bibdoc, version, format, id_rel, namespace, data_key)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bibdocfsinfo (
id_bibdoc mediumint(9) unsigned NOT NULL,
version tinyint(4) unsigned NOT NULL,
format varchar(50) NOT NULL,
last_version boolean NOT NULL,
cd datetime NOT NULL,
md datetime NOT NULL,
checksum char(32) NOT NULL,
filesize bigint(15) unsigned NOT NULL,
mime varchar(100) NOT NULL,
master_format varchar(50) NULL default NULL,
PRIMARY KEY (id_bibdoc, version, format),
KEY (last_version),
KEY (format),
KEY (cd),
KEY (md),
KEY (filesize),
KEY (mime)
) ENGINE=MyISAM;
-- tables for publication requests:
CREATE TABLE IF NOT EXISTS publreq (
id int(11) NOT NULL auto_increment,
host varchar(255) NOT NULL default '',
date varchar(255) NOT NULL default '',
name varchar(255) NOT NULL default '',
email varchar(255) NOT NULL default '',
address text NOT NULL,
publication text NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
-- table for sessions and users:
CREATE TABLE IF NOT EXISTS session (
session_key varchar(32) NOT NULL default '',
session_expiry datetime NOT NULL default '0000-00-00 00:00:00',
session_object longblob,
uid int(15) unsigned NOT NULL,
UNIQUE KEY session_key (session_key),
KEY uid (uid),
KEY session_expiry (session_expiry)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS user (
id int(15) unsigned NOT NULL auto_increment,
email varchar(255) NOT NULL default '',
password blob NOT NULL,
note varchar(255) default NULL,
settings blob default NULL,
nickname varchar(255) NOT NULL default '',
last_login datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY id (id),
KEY email (email),
KEY nickname (nickname)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS userEXT (
id varbinary(255) NOT NULL,
method varchar(50) NOT NULL,
id_user int(15) unsigned NOT NULL,
PRIMARY KEY (id, method),
UNIQUE KEY (id_user, method)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS oauth1_storage (
token varchar(255) NOT NULL,
secret varchar(255) NOT NULL,
date_creation datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (token)
) ENGINE=MyISAM;
-- tables for usergroups
CREATE TABLE IF NOT EXISTS usergroup (
id int(15) unsigned NOT NULL auto_increment,
name varchar(255) NOT NULL default '',
description text default '',
join_policy char(2) NOT NULL default '',
login_method varchar(255) NOT NULL default 'INTERNAL',
PRIMARY KEY (id),
UNIQUE KEY login_method_name (login_method(70), name),
KEY name (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS user_usergroup (
id_user int(15) unsigned NOT NULL default '0',
id_usergroup int(15) unsigned NOT NULL default '0',
user_status char(1) NOT NULL default '',
user_status_date datetime NOT NULL default '0000-00-00 00:00:00',
KEY id_user (id_user),
KEY id_usergroup (id_usergroup)
) ENGINE=MyISAM;
-- tables for access control engine
CREATE TABLE IF NOT EXISTS accROLE (
id int(15) unsigned NOT NULL auto_increment,
name varchar(32),
description varchar(255),
firerole_def_ser blob NULL,
firerole_def_src text NULL,
PRIMARY KEY (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS user_accROLE (
id_user int(15) unsigned NOT NULL,
id_accROLE int(15) unsigned NOT NULL,
expiration datetime NOT NULL default '9999-12-31 23:59:59',
PRIMARY KEY (id_user, id_accROLE)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS accMAILCOOKIE (
id int(15) unsigned NOT NULL auto_increment,
data blob NOT NULL,
expiration datetime NOT NULL default '9999-12-31 23:59:59',
kind varchar(32) NOT NULL,
onetime boolean NOT NULL default 0,
status char(1) NOT NULL default 'W',
PRIMARY KEY (id),
KEY expiration (expiration)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS accACTION (
id int(15) unsigned NOT NULL auto_increment,
name varchar(32),
description varchar(255),
allowedkeywords varchar(255),
optional ENUM ('yes', 'no') NOT NULL default 'no',
PRIMARY KEY (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS accARGUMENT (
id int(15) unsigned NOT NULL auto_increment,
keyword varchar (32),
value varchar(255),
PRIMARY KEY (id),
KEY KEYVAL (keyword, value)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS accROLE_accACTION_accARGUMENT (
id_accROLE int(15),
id_accACTION int(15),
id_accARGUMENT int(15),
argumentlistid mediumint(8),
KEY id_accROLE (id_accROLE),
KEY id_accACTION (id_accACTION),
KEY id_accARGUMENT (id_accARGUMENT)
) ENGINE=MyISAM;
-- tables for personal/collaborative features (baskets, alerts, searches, messages, usergroups):
CREATE TABLE IF NOT EXISTS user_query (
id_user int(15) unsigned NOT NULL default '0',
id_query int(15) unsigned NOT NULL default '0',
hostname varchar(50) default 'unknown host',
date datetime default NULL,
KEY id_user (id_user,id_query)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS query (
id int(15) unsigned NOT NULL auto_increment,
type char(1) NOT NULL default 'r',
urlargs text NOT NULL,
PRIMARY KEY (id),
KEY urlargs (urlargs(100))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS user_query_basket (
id_user int(15) unsigned NOT NULL default '0',
id_query int(15) unsigned NOT NULL default '0',
id_basket int(15) unsigned NOT NULL default '0',
frequency varchar(5) NOT NULL default '',
date_creation date default NULL,
date_lastrun date default '0000-00-00',
alert_name varchar(30) NOT NULL default '',
alert_desc text default NULL,
alert_recipient text default NULL,
notification char(1) NOT NULL default 'y',
PRIMARY KEY (id_user,id_query,frequency,id_basket),
KEY alert_name (alert_name)
) ENGINE=MyISAM;
-- baskets
CREATE TABLE IF NOT EXISTS bskBASKET (
id int(15) unsigned NOT NULL auto_increment,
id_owner int(15) unsigned NOT NULL default '0',
name varchar(50) NOT NULL default '',
date_modification datetime NOT NULL default '0000-00-00 00:00:00',
nb_views int(15) NOT NULL default '0',
PRIMARY KEY (id),
KEY id_owner (id_owner),
KEY name (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bskREC (
id_bibrec_or_bskEXTREC int(16) NOT NULL default '0',
id_bskBASKET int(15) unsigned NOT NULL default '0',
id_user_who_added_item int(15) NOT NULL default '0',
score int(15) NOT NULL default '0',
date_added datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (id_bibrec_or_bskEXTREC,id_bskBASKET),
KEY id_bibrec_or_bskEXTREC (id_bibrec_or_bskEXTREC),
KEY id_bskBASKET (id_bskBASKET),
KEY score (score),
KEY date_added (date_added)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bskEXTREC (
id int(15) unsigned NOT NULL auto_increment,
external_id int(15) NOT NULL default '0',
collection_id int(15) unsigned NOT NULL default '0',
original_url text,
creation_date datetime NOT NULL default '0000-00-00 00:00:00',
modification_date datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bskEXTFMT (
id int(15) unsigned NOT NULL auto_increment,
id_bskEXTREC int(15) unsigned NOT NULL default '0',
format varchar(10) NOT NULL default '',
last_updated datetime NOT NULL default '0000-00-00 00:00:00',
value longblob,
PRIMARY KEY (id),
KEY id_bskEXTREC (id_bskEXTREC),
KEY format (format)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS user_bskBASKET (
id_user int(15) unsigned NOT NULL default '0',
id_bskBASKET int(15) unsigned NOT NULL default '0',
topic varchar(50) NOT NULL default '',
PRIMARY KEY (id_user,id_bskBASKET),
KEY id_user (id_user),
KEY id_bskBASKET (id_bskBASKET)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS usergroup_bskBASKET (
id_usergroup int(15) unsigned NOT NULL default '0',
id_bskBASKET int(15) unsigned NOT NULL default '0',
topic varchar(50) NOT NULL default '',
date_shared datetime NOT NULL default '0000-00-00 00:00:00',
share_level char(2) NOT NULL default '',
PRIMARY KEY (id_usergroup,id_bskBASKET),
KEY id_usergroup (id_usergroup),
KEY id_bskBASKET (id_bskBASKET)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bskRECORDCOMMENT (
id int(15) unsigned NOT NULL auto_increment,
id_bibrec_or_bskEXTREC int(16) NOT NULL default '0',
id_bskBASKET int(15) unsigned NOT NULL default '0',
id_user int(15) unsigned NOT NULL default '0',
title varchar(255) NOT NULL default '',
body text NOT NULL,
date_creation datetime NOT NULL default '0000-00-00 00:00:00',
priority int(15) NOT NULL default '0',
in_reply_to_id_bskRECORDCOMMENT int(15) unsigned NOT NULL default '0',
reply_order_cached_data blob NULL default NULL,
PRIMARY KEY (id),
KEY id_bskBASKET (id_bskBASKET),
KEY id_bibrec_or_bskEXTREC (id_bibrec_or_bskEXTREC),
KEY date_creation (date_creation),
KEY in_reply_to_id_bskRECORDCOMMENT (in_reply_to_id_bskRECORDCOMMENT),
INDEX (reply_order_cached_data(40))
) ENGINE=MyISAM;
-- tables for messaging system
CREATE TABLE IF NOT EXISTS msgMESSAGE (
id int(15) unsigned NOT NULL auto_increment,
id_user_from int(15) unsigned NOT NULL default '0',
sent_to_user_nicks text NOT NULL default '',
sent_to_group_names text NOT NULL default '',
subject text NOT NULL default '',
body text default NULL,
sent_date datetime NOT NULL default '0000-00-00 00:00:00',
received_date datetime NULL default '0000-00-00 00:00:00',
PRIMARY KEY id (id),
KEY id_user_from (id_user_from)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS user_msgMESSAGE (
id_user_to int(15) unsigned NOT NULL default '0',
id_msgMESSAGE int(15) unsigned NOT NULL default '0',
status char(1) NOT NULL default 'N',
PRIMARY KEY id (id_user_to, id_msgMESSAGE),
KEY id_user_to (id_user_to),
KEY id_msgMESSAGE (id_msgMESSAGE)
) ENGINE=MyISAM;
-- tables for WebComment
CREATE TABLE IF NOT EXISTS cmtRECORDCOMMENT (
id int(15) unsigned NOT NULL auto_increment,
id_bibrec int(15) unsigned NOT NULL default '0',
id_user int(15) unsigned NOT NULL default '0',
title varchar(255) NOT NULL default '',
body text NOT NULL default '',
date_creation datetime NOT NULL default '0000-00-00 00:00:00',
star_score tinyint(5) unsigned NOT NULL default '0',
nb_votes_yes int(10) NOT NULL default '0',
nb_votes_total int(10) unsigned NOT NULL default '0',
nb_abuse_reports int(10) NOT NULL default '0',
status char(2) NOT NULL default 'ok',
round_name varchar(255) NOT NULL default '',
restriction varchar(50) NOT NULL default '',
in_reply_to_id_cmtRECORDCOMMENT int(15) unsigned NOT NULL default '0',
reply_order_cached_data blob NULL default NULL,
PRIMARY KEY (id),
KEY id_bibrec (id_bibrec),
KEY id_user (id_user),
KEY status (status),
KEY in_reply_to_id_cmtRECORDCOMMENT (in_reply_to_id_cmtRECORDCOMMENT),
INDEX (reply_order_cached_data(40))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS cmtACTIONHISTORY (
id_cmtRECORDCOMMENT int(15) unsigned NULL,
id_bibrec int(15) unsigned NULL,
id_user int(15) unsigned NULL default NULL,
client_host int(10) unsigned default NULL,
action_time datetime NOT NULL default '0000-00-00 00:00:00',
action_code char(1) NOT NULL,
KEY id_cmtRECORDCOMMENT (id_cmtRECORDCOMMENT),
KEY client_host (client_host),
KEY id_user (id_user),
KEY action_code (action_code)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS cmtSUBSCRIPTION (
id_bibrec mediumint(8) unsigned NOT NULL,
id_user int(15) unsigned NOT NULL,
creation_time datetime NOT NULL default '0000-00-00 00:00:00',
KEY id_user (id_bibrec, id_user)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS cmtCOLLAPSED (
id_bibrec int(15) unsigned NOT NULL default '0',
id_cmtRECORDCOMMENT int(15) unsigned NULL,
id_user int(15) unsigned NOT NULL,
PRIMARY KEY (id_user, id_bibrec, id_cmtRECORDCOMMENT)
) ENGINE=MyISAM;
-- tables for BibKnowledge:
CREATE TABLE IF NOT EXISTS knwKB (
id mediumint(8) unsigned NOT NULL auto_increment,
name varchar(255) default '',
description text default '',
kbtype char default NULL,
PRIMARY KEY (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS knwKBRVAL (
id mediumint(8) unsigned NOT NULL auto_increment,
m_key varchar(255) NOT NULL default '',
m_value text NOT NULL default '',
id_knwKB mediumint(8) NOT NULL default '0',
PRIMARY KEY (id),
KEY id_knwKB (id_knwKB),
KEY m_key (m_key(30)),
KEY m_value (m_value(30))
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS knwKBDDEF (
id_knwKB mediumint(8) unsigned NOT NULL,
id_collection mediumint(9),
output_tag text default '',
search_expression text default '',
PRIMARY KEY (id_knwKB)
) ENGINE=MyISAM;
-- tables for WebSubmit:
CREATE TABLE IF NOT EXISTS sbmACTION (
lactname text,
sactname char(3) NOT NULL default '',
dir text,
cd date default NULL,
md date default NULL,
actionbutton text,
statustext text,
PRIMARY KEY (sactname)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmALLFUNCDESCR (
function varchar(40) NOT NULL default '',
description tinytext,
PRIMARY KEY (function)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmAPPROVAL (
doctype varchar(10) NOT NULL default '',
categ varchar(50) NOT NULL default '',
rn varchar(50) NOT NULL default '',
status varchar(10) NOT NULL default '',
dFirstReq datetime NOT NULL default '0000-00-00 00:00:00',
dLastReq datetime NOT NULL default '0000-00-00 00:00:00',
dAction datetime NOT NULL default '0000-00-00 00:00:00',
access varchar(20) NOT NULL default '0',
note text NOT NULL default '',
PRIMARY KEY (rn)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmCPLXAPPROVAL (
doctype varchar(10) NOT NULL default '',
categ varchar(50) NOT NULL default '',
rn varchar(50) NOT NULL default '',
type varchar(10) NOT NULL,
status varchar(10) NOT NULL,
id_group int(15) unsigned NOT NULL default '0',
id_bskBASKET int(15) unsigned NOT NULL default '0',
id_EdBoardGroup int(15) unsigned NOT NULL default '0',
dFirstReq datetime NOT NULL default '0000-00-00 00:00:00',
dLastReq datetime NOT NULL default '0000-00-00 00:00:00',
dEdBoardSel datetime NOT NULL default '0000-00-00 00:00:00',
dRefereeSel datetime NOT NULL default '0000-00-00 00:00:00',
dRefereeRecom datetime NOT NULL default '0000-00-00 00:00:00',
dEdBoardRecom datetime NOT NULL default '0000-00-00 00:00:00',
dPubComRecom datetime NOT NULL default '0000-00-00 00:00:00',
dProjectLeaderAction datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (rn, type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmCOLLECTION (
id int(11) NOT NULL auto_increment,
name varchar(100) NOT NULL default '',
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmCOLLECTION_sbmCOLLECTION (
id_father int(11) NOT NULL default '0',
id_son int(11) NOT NULL default '0',
catalogue_order int(11) NOT NULL default '0'
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmCOLLECTION_sbmDOCTYPE (
id_father int(11) NOT NULL default '0',
id_son char(10) NOT NULL default '0',
catalogue_order int(11) NOT NULL default '0'
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmCATEGORIES (
doctype varchar(10) NOT NULL default '',
sname varchar(75) NOT NULL default '',
lname varchar(75) NOT NULL default '',
score tinyint unsigned NOT NULL default 0,
PRIMARY KEY (doctype, sname),
KEY doctype (doctype),
KEY sname (sname)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmCHECKS (
chname varchar(15) NOT NULL default '',
chdesc text,
cd date default NULL,
md date default NULL,
chefi1 text,
chefi2 text,
PRIMARY KEY (chname)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmDOCTYPE (
ldocname text,
sdocname varchar(10) default NULL,
cd date default NULL,
md date default NULL,
description text
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmFIELD (
subname varchar(13) default NULL,
pagenb int(11) default NULL,
fieldnb int(11) default NULL,
fidesc varchar(15) default NULL,
fitext text,
level char(1) default NULL,
sdesc text,
checkn text,
cd date default NULL,
md date default NULL,
fiefi1 text,
fiefi2 text
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmFIELDDESC (
name varchar(15) NOT NULL default '',
alephcode varchar(50) default NULL,
marccode varchar(50) NOT NULL default '',
type char(1) default NULL,
size int(11) default NULL,
rows int(11) default NULL,
cols int(11) default NULL,
maxlength int(11) default NULL,
val text,
fidesc text,
cd date default NULL,
md date default NULL,
modifytext text,
fddfi2 text,
cookie int(11) default '0',
PRIMARY KEY (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmFORMATEXTENSION (
FILE_FORMAT text NOT NULL,
FILE_EXTENSION text NOT NULL
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmFUNCTIONS (
action varchar(10) NOT NULL default '',
doctype varchar(10) NOT NULL default '',
function varchar(40) NOT NULL default '',
score int(11) NOT NULL default '0',
step tinyint(4) NOT NULL default '1'
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmFUNDESC (
function varchar(40) NOT NULL default '',
param varchar(40) default NULL
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmGFILERESULT (
FORMAT text NOT NULL,
RESULT text NOT NULL
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmIMPLEMENT (
docname varchar(10) default NULL,
actname char(3) default NULL,
displayed char(1) default NULL,
subname varchar(13) default NULL,
nbpg int(11) default NULL,
cd date default NULL,
md date default NULL,
buttonorder int(11) default NULL,
statustext text,
level char(1) NOT NULL default '',
score int(11) NOT NULL default '0',
stpage int(11) NOT NULL default '0',
endtxt varchar(100) NOT NULL default ''
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmPARAMETERS (
doctype varchar(10) NOT NULL default '',
name varchar(40) NOT NULL default '',
value text NOT NULL default '',
PRIMARY KEY (doctype,name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmPUBLICATION (
doctype varchar(10) NOT NULL default '',
categ varchar(50) NOT NULL default '',
rn varchar(50) NOT NULL default '',
status varchar(10) NOT NULL default '',
dFirstReq datetime NOT NULL default '0000-00-00 00:00:00',
dLastReq datetime NOT NULL default '0000-00-00 00:00:00',
dAction datetime NOT NULL default '0000-00-00 00:00:00',
accessref varchar(20) NOT NULL default '',
accessedi varchar(20) NOT NULL default '',
access varchar(20) NOT NULL default '',
referees varchar(50) NOT NULL default '',
authoremail varchar(50) NOT NULL default '',
dRefSelection datetime NOT NULL default '0000-00-00 00:00:00',
dRefRec datetime NOT NULL default '0000-00-00 00:00:00',
dEdiRec datetime NOT NULL default '0000-00-00 00:00:00',
accessspo varchar(20) NOT NULL default '',
journal varchar(100) default NULL,
PRIMARY KEY (doctype,categ,rn)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmPUBLICATIONCOMM (
id int(11) NOT NULL auto_increment,
id_parent int(11) default '0',
rn varchar(100) NOT NULL default '',
firstname varchar(100) default NULL,
secondname varchar(100) default NULL,
email varchar(100) default NULL,
date varchar(40) NOT NULL default '',
synopsis varchar(255) NOT NULL default '',
commentfulltext text,
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmPUBLICATIONDATA (
doctype varchar(10) NOT NULL default '',
editoboard varchar(250) NOT NULL default '',
base varchar(10) NOT NULL default '',
logicalbase varchar(10) NOT NULL default '',
spokesperson varchar(50) NOT NULL default '',
PRIMARY KEY (doctype)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmREFEREES (
doctype varchar(10) NOT NULL default '',
categ varchar(10) NOT NULL default '',
name varchar(50) NOT NULL default '',
address varchar(50) NOT NULL default '',
rid int(11) NOT NULL auto_increment,
PRIMARY KEY (rid)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmSUBMISSIONS (
email varchar(50) NOT NULL default '',
doctype varchar(10) NOT NULL default '',
action varchar(10) NOT NULL default '',
status varchar(10) NOT NULL default '',
id varchar(30) NOT NULL default '',
reference varchar(40) NOT NULL default '',
cd datetime NOT NULL default '0000-00-00 00:00:00',
md datetime NOT NULL default '0000-00-00 00:00:00'
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS sbmCOOKIES (
id int(15) unsigned NOT NULL auto_increment,
name varchar(100) NOT NULL,
value text,
uid int(15) NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
-- Scheduler tables
CREATE TABLE IF NOT EXISTS schTASK (
id int(15) unsigned NOT NULL auto_increment,
proc varchar(255) NOT NULL,
host varchar(255) NOT NULL default '',
user varchar(50) NOT NULL,
runtime datetime NOT NULL,
sleeptime varchar(20),
arguments mediumblob,
status varchar(50),
progress varchar(255),
priority tinyint(4) NOT NULL default 0,
sequenceid int(8) NULL default NULL,
PRIMARY KEY (id),
KEY status (status),
KEY runtime (runtime),
KEY priority (priority),
KEY sequenceid (sequenceid)
) ENGINE=MyISAM;
-- FIXME, To be moved to redis when available
CREATE TABLE IF NOT EXISTS schSTATUS (
name varchar(50),
value mediumblob,
PRIMARY KEY (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS hstTASK (
id int(15) unsigned NOT NULL,
proc varchar(255) NOT NULL,
host varchar(255) NOT NULL default '',
user varchar(50) NOT NULL,
runtime datetime NOT NULL,
sleeptime varchar(20),
arguments mediumblob,
status varchar(50),
progress varchar(255),
priority tinyint(4) NOT NULL default 0,
sequenceid int(8) NULL default NULL,
PRIMARY KEY (id),
KEY status (status),
KEY runtime (runtime),
KEY priority (priority),
KEY sequenceid (sequenceid)
) ENGINE=MyISAM;
-- Batch Upload History
CREATE TABLE IF NOT EXISTS hstBATCHUPLOAD (
id int(15) unsigned NOT NULL auto_increment,
user varchar(50) NOT NULL,
submitdate datetime NOT NULL,
filename varchar(255) NOT NULL,
execdate datetime NOT NULL,
id_schTASK int(15) unsigned NOT NULL,
batch_mode varchar(15) NOT NULL,
PRIMARY KEY (id),
KEY user (user)
) ENGINE=MyISAM;
-- External collections
CREATE TABLE IF NOT EXISTS collection_externalcollection (
id_collection mediumint(9) unsigned NOT NULL default '0',
id_externalcollection mediumint(9) unsigned NOT NULL default '0',
type tinyint(4) unsigned NOT NULL default '0',
PRIMARY KEY (id_collection, id_externalcollection)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS externalcollection (
id mediumint(9) unsigned NOT NULL auto_increment,
name varchar(255) NOT NULL default '',
PRIMARY KEY (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;
-- WebStat tables:
CREATE TABLE IF NOT EXISTS staEVENT (
id varchar(255) NOT NULL,
number smallint(2) unsigned ZEROFILL NOT NULL auto_increment,
name varchar(255),
creation_time TIMESTAMP DEFAULT NOW(),
cols varchar(255),
PRIMARY KEY (id),
UNIQUE KEY number (number)
) ENGINE=MyISAM;
-- BibClassify tables:
CREATE TABLE IF NOT EXISTS clsMETHOD (
id mediumint(9) unsigned NOT NULL,
name varchar(50) NOT NULL default '',
location varchar(255) NOT NULL default '',
description varchar(255) NOT NULL default '',
last_updated datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS collection_clsMETHOD (
id_collection mediumint(9) unsigned NOT NULL,
id_clsMETHOD mediumint(9) unsigned NOT NULL,
PRIMARY KEY (id_collection, id_clsMETHOD)
) ENGINE=MyISAM;
-- WebJournal tables:
CREATE TABLE IF NOT EXISTS jrnJOURNAL (
id mediumint(9) unsigned NOT NULL auto_increment,
name varchar(50) NOT NULL default '',
PRIMARY KEY (id),
UNIQUE KEY name (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS jrnISSUE (
id_jrnJOURNAL mediumint(9) unsigned NOT NULL,
issue_number varchar(50) NOT NULL default '',
issue_display varchar(50) NOT NULL default '',
date_released datetime NOT NULL default '0000-00-00 00:00:00',
date_announced datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (id_jrnJOURNAL,issue_number)
) ENGINE=MyISAM;
-- tables recording history of record's metadata and fulltext documents:
CREATE TABLE IF NOT EXISTS hstRECORD (
id_bibrec mediumint(8) unsigned NOT NULL,
marcxml longblob NOT NULL,
job_id mediumint(15) unsigned NOT NULL,
job_name varchar(255) NOT NULL,
job_person varchar(255) NOT NULL,
job_date datetime NOT NULL,
job_details blob NOT NULL,
affected_fields text NOT NULL default '',
KEY (id_bibrec),
KEY (job_id),
KEY (job_name),
KEY (job_person),
KEY (job_date)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS hstDOCUMENT (
id_bibdoc mediumint(9) unsigned NOT NULL,
docname varchar(250) NOT NULL,
docformat varchar(50) NOT NULL,
docversion tinyint(4) unsigned NOT NULL,
docsize bigint(15) unsigned NOT NULL,
docchecksum char(32) NOT NULL,
doctimestamp datetime NOT NULL,
action varchar(50) NOT NULL,
job_id mediumint(15) unsigned NULL default NULL,
job_name varchar(255) NULL default NULL,
job_person varchar(255) NULL default NULL,
job_date datetime NULL default NULL,
job_details blob NULL default NULL,
KEY (action),
KEY (id_bibdoc),
KEY (docname),
KEY (docformat),
KEY (doctimestamp),
KEY (job_id),
KEY (job_name),
KEY (job_person),
KEY (job_date)
) ENGINE=MyISAM;
-- BibCirculation tables:
CREATE TABLE IF NOT EXISTS crcBORROWER (
id int(15) unsigned NOT NULL auto_increment,
ccid int(15) unsigned NULL default NULL,
name varchar(255) NOT NULL default '',
email varchar(255) NOT NULL default '',
phone varchar(60) default NULL,
address varchar(60) default NULL,
mailbox varchar(30) default NULL,
borrower_since datetime NOT NULL default '0000-00-00 00:00:00',
borrower_until datetime NOT NULL default '0000-00-00 00:00:00',
notes text,
PRIMARY KEY (id),
UNIQUE KEY (ccid),
KEY (name),
KEY (email)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS crcILLREQUEST (
id int(15) unsigned NOT NULL auto_increment,
id_crcBORROWER int(15) unsigned NOT NULL default '0',
barcode varchar(30) NOT NULL default '',
period_of_interest_from datetime NOT NULL default '0000-00-00 00:00:00',
period_of_interest_to datetime NOT NULL default '0000-00-00 00:00:00',
id_crcLIBRARY int(15) unsigned NOT NULL default '0',
request_date datetime NOT NULL default '0000-00-00 00:00:00',
expected_date datetime NOT NULL default '0000-00-00 00:00:00',
arrival_date datetime NOT NULL default '0000-00-00 00:00:00',
due_date datetime NOT NULL default '0000-00-00 00:00:00',
return_date datetime NOT NULL default '0000-00-00 00:00:00',
status varchar(20) NOT NULL default '',
cost varchar(30) NOT NULL default '',
budget_code varchar(60) NOT NULL default '',
item_info text,
request_type text,
borrower_comments text,
only_this_edition varchar(10) NOT NULL default '',
library_notes text,
overdue_letter_number int(3) unsigned NOT NULL default '0',
overdue_letter_date datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (id),
KEY id_crcborrower (id_crcBORROWER),
KEY id_crclibrary (id_crcLIBRARY)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS crcITEM (
barcode varchar(30) NOT NULL default '',
id_bibrec int(15) unsigned NOT NULL default '0',
id_crcLIBRARY int(15) unsigned NOT NULL default '0',
collection varchar(60) default NULL,
location varchar(60) default NULL,
description varchar(60) default NULL,
loan_period varchar(30) NOT NULL default '',
status varchar(20) NOT NULL default '',
expected_arrival_date varchar(60) NOT NULL default '',
creation_date datetime NOT NULL default '0000-00-00 00:00:00',
modification_date datetime NOT NULL default '0000-00-00 00:00:00',
number_of_requests int(3) unsigned NOT NULL default '0',
PRIMARY KEY (barcode),
KEY id_bibrec (id_bibrec),
KEY id_crclibrary (id_crcLIBRARY)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS crcLIBRARY (
id int(15) unsigned NOT NULL auto_increment,
name varchar(80) NOT NULL default '',
address varchar(255) NOT NULL default '',
email varchar(255) NOT NULL default '',
phone varchar(30) NOT NULL default '',
type varchar(30) NOT NULL default 'main',
notes text,
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS crcLOAN (
id int(15) unsigned NOT NULL auto_increment,
id_crcBORROWER int(15) unsigned NOT NULL default '0',
id_bibrec int(15) unsigned NOT NULL default '0',
barcode varchar(30) NOT NULL default '',
loaned_on datetime NOT NULL default '0000-00-00 00:00:00',
returned_on date NOT NULL default '0000-00-00',
due_date datetime NOT NULL default '0000-00-00 00:00:00',
number_of_renewals int(3) unsigned NOT NULL default '0',
overdue_letter_number int(3) unsigned NOT NULL default '0',
overdue_letter_date datetime NOT NULL default '0000-00-00 00:00:00',
status varchar(20) NOT NULL default '',
type varchar(20) NOT NULL default '',
notes text,
PRIMARY KEY (id),
KEY id_crcborrower (id_crcBORROWER),
KEY id_bibrec (id_bibrec),
KEY barcode (barcode)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS crcLOANREQUEST (
id int(15) unsigned NOT NULL auto_increment,
id_crcBORROWER int(15) unsigned NOT NULL default '0',
id_bibrec int(15) unsigned NOT NULL default '0',
barcode varchar(30) NOT NULL default '',
period_of_interest_from datetime NOT NULL default '0000-00-00 00:00:00',
period_of_interest_to datetime NOT NULL default '0000-00-00 00:00:00',
status varchar(20) NOT NULL default '',
notes text,
request_date datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (id),
KEY id_crcborrower (id_crcBORROWER),
KEY id_bibrec (id_bibrec),
KEY barcode (barcode)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS crcPURCHASE (
id int(15) unsigned NOT NULL auto_increment,
id_bibrec int(15) unsigned NOT NULL default '0',
id_crcVENDOR int(15) unsigned NOT NULL default '0',
ordered_date datetime NOT NULL default '0000-00-00 00:00:00',
expected_date datetime NOT NULL default '0000-00-00 00:00:00',
price varchar(20) NOT NULL default '0',
status varchar(20) NOT NULL default '',
notes text,
PRIMARY KEY (id),
KEY id_bibrec (id_bibrec),
KEY id_crcVENDOR (id_crcVENDOR)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS crcVENDOR (
id int(15) unsigned NOT NULL auto_increment,
name varchar(80) NOT NULL default '',
address varchar(255) NOT NULL default '',
email varchar(255) NOT NULL default '',
phone varchar(30) NOT NULL default '',
notes text,
PRIMARY KEY (id)
) ENGINE=MyISAM;
-- BibExport tables:
CREATE TABLE IF NOT EXISTS expJOB (
id int(15) unsigned NOT NULL auto_increment,
jobname varchar(50) NOT NULL default '',
jobfreq mediumint(12) NOT NULL default '0',
output_format mediumint(12) NOT NULL default '0',
deleted mediumint(12) NOT NULL default '0',
lastrun datetime NOT NULL default '0000-00-00 00:00:00',
output_directory text,
PRIMARY KEY (id),
UNIQUE KEY jobname (jobname)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS expQUERY (
id int(15) unsigned NOT NULL auto_increment,
name varchar(255) NOT NULL,
search_criteria text NOT NULL,
output_fields text NOT NULL,
notes text,
deleted mediumint(12) NOT NULL default '0',
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS expJOB_expQUERY (
id_expJOB int(15) NOT NULL,
id_expQUERY int(15) NOT NULL,
PRIMARY KEY (id_expJOB,id_expQUERY),
KEY id_expJOB (id_expJOB),
KEY id_expQUERY (id_expQUERY)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS expQUERYRESULT (
id int(15) unsigned NOT NULL auto_increment,
id_expQUERY int(15) NOT NULL,
result text NOT NULL,
status mediumint(12) NOT NULL default '0',
status_message text NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS expJOBRESULT (
id int(15) unsigned NOT NULL auto_increment,
id_expJOB int(15) NOT NULL,
execution_time datetime NOT NULL default '0000-00-00 00:00:00',
status mediumint(12) NOT NULL default '0',
status_message text NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS expJOBRESULT_expQUERYRESULT (
id_expJOBRESULT int(15) NOT NULL,
id_expQUERYRESULT int(15) NOT NULL,
PRIMARY KEY (id_expJOBRESULT, id_expQUERYRESULT),
KEY id_expJOBRESULT (id_expJOBRESULT),
KEY id_expQUERYRESULT (id_expQUERYRESULT)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS user_expJOB (
id_user int(15) NOT NULL,
id_expJOB int(15) NOT NULL,
PRIMARY KEY (id_user, id_expJOB),
KEY id_user (id_user),
KEY id_expJOB (id_expJOB)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS swrREMOTESERVER (
id int(15) unsigned NOT NULL auto_increment,
name varchar(50) unique NOT NULL,
host varchar(50) NOT NULL,
username varchar(50) NOT NULL,
password varchar(50) NOT NULL,
email varchar(50) NOT NULL,
realm varchar(50) NOT NULL,
url_base_record varchar(50) NOT NULL,
url_servicedocument varchar(80) NOT NULL,
xml_servicedocument longblob,
last_update int(15) unsigned NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS swrCLIENTDATA (
id int(15) unsigned NOT NULL auto_increment,
id_swrREMOTESERVER int(15) NOT NULL,
id_record int(15) NOT NULL,
report_no varchar(50) NOT NULL,
id_remote varchar(50) NOT NULL,
id_user int(15) NOT NULL,
user_name varchar(100) NOT NULL,
user_email varchar(100) NOT NULL,
xml_media_deposit longblob NOT NULL,
xml_metadata_submit longblob NOT NULL,
submission_date datetime NOT NULL default '0000-00-00 00:00:00',
publication_date datetime NOT NULL default '0000-00-00 00:00:00',
removal_date datetime NOT NULL default '0000-00-00 00:00:00',
link_medias varchar(150) NOT NULL,
link_metadata varchar(150) NOT NULL,
link_status varchar(150) NOT NULL,
status varchar(150) NOT NULL default 'submitted',
last_update datetime NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
-- tables for exception management
-- This table is used to log exceptions
-- to discover the full details of an exception either check the email
-- that are sent to CFG_SITE_ADMIN_EMAIL or look into invenio.err
CREATE TABLE IF NOT EXISTS hstEXCEPTION (
id int(15) unsigned NOT NULL auto_increment,
name varchar(50) NOT NULL, -- name of the exception
filename varchar(255) NULL, -- file where the exception was raised
line int(9) NULL, -- line at which the exception was raised
last_seen datetime NOT NULL default '0000-00-00 00:00:00', -- last time this exception has been seen
last_notified datetime NOT NULL default '0000-00-00 00:00:00', -- last time this exception has been notified
counter int(15) NOT NULL default 0, -- internal counter to decide when to notify this exception
total int(15) NOT NULL default 0, -- total number of times this exception has been seen
PRIMARY KEY (id),
KEY (last_seen),
KEY (last_notified),
KEY (total),
UNIQUE KEY (name(50), filename(255), line)
) ENGINE=MyISAM;
-- tables for BibAuthorID module:
CREATE TABLE IF NOT EXISTS `aidPERSONIDPAPERS` (
`personid` BIGINT( 16 ) UNSIGNED NOT NULL ,
`bibref_table` ENUM( '100', '700' ) NOT NULL ,
`bibref_value` MEDIUMINT( 8 ) UNSIGNED NOT NULL ,
`bibrec` MEDIUMINT( 8 ) UNSIGNED NOT NULL ,
`name` VARCHAR( 256 ) NOT NULL ,
`flag` SMALLINT( 2 ) NOT NULL DEFAULT '0' ,
`lcul` SMALLINT( 2 ) NOT NULL DEFAULT '0' ,
`last_updated` TIMESTAMP ON UPDATE CURRENT_TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ,
INDEX `personid-b` (`personid`) ,
INDEX `reftable-b` (`bibref_table`) ,
INDEX `refvalue-b` (`bibref_value`) ,
INDEX `rec-b` (`bibrec`) ,
INDEX `name-b` (`name`) ,
INDEX `pn-b` (`personid`, `name`) ,
INDEX `timestamp-b` (`last_updated`) ,
INDEX `flag-b` (`flag`) ,
INDEX `personid-flag-b` (`personid`,`flag`),
INDEX `ptvrf-b` (`personid`, `bibref_table`, `bibref_value`, `bibrec`, `flag`)
) ENGINE=MYISAM;
CREATE TABLE IF NOT EXISTS `aidRESULTS` (
`personid` VARCHAR( 256 ) NOT NULL ,
`bibref_table` ENUM( '100', '700' ) NOT NULL ,
`bibref_value` MEDIUMINT( 8 ) UNSIGNED NOT NULL ,
`bibrec` MEDIUMINT( 8 ) UNSIGNED NOT NULL ,
INDEX `personid-b` (`personid`) ,
INDEX `reftable-b` (`bibref_table`) ,
INDEX `refvalue-b` (`bibref_value`) ,
INDEX `rec-b` (`bibrec`)
) ENGINE=MYISAM;
CREATE TABLE IF NOT EXISTS `aidPERSONIDDATA` (
`personid` BIGINT( 16 ) UNSIGNED NOT NULL ,
`tag` VARCHAR( 64 ) NOT NULL ,
`data` VARCHAR( 256 ) NULL DEFAULT NULL ,
`datablob` LONGBLOB NULL DEFAULT NULL ,
`opt1` MEDIUMINT( 8 ) NULL DEFAULT NULL ,
`opt2` MEDIUMINT( 8 ) NULL DEFAULT NULL ,
`opt3` VARCHAR( 256 ) NULL DEFAULT NULL ,
`last_updated` TIMESTAMP ON UPDATE CURRENT_TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ,
INDEX `personid-b` (`personid`) ,
INDEX `tag-b` (`tag`) ,
INDEX `data-b` (`data`) ,
INDEX `opt1` (`opt1`) ,
INDEX `timestamp-b` (`last_updated`)
) ENGINE=MYISAM;
CREATE TABLE IF NOT EXISTS `aidUSERINPUTLOG` (
`id` bigint(15) NOT NULL AUTO_INCREMENT,
`transactionid` bigint(15) NOT NULL,
`timestamp` datetime NOT NULL,
`userid` int,
`userinfo` varchar(255) NOT NULL,
`personid` bigint(15) NOT NULL,
`action` varchar(50) NOT NULL,
`tag` varchar(50) NOT NULL,
`value` varchar(200) NOT NULL,
`comment` text,
PRIMARY KEY (`id`),
INDEX `transactionid-b` (`transactionid`),
INDEX `timestamp-b` (`timestamp`),
INDEX `userinfo-b` (`userinfo`),
INDEX `userid-b` (`userid`),
INDEX `personid-b` (`personid`),
INDEX `action-b` (`action`),
INDEX `tag-b` (`tag`),
INDEX `value-b` (`value`)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS `aidCACHE` (
`id` int(15) NOT NULL auto_increment,
`object_name` varchar(120) NOT NULL,
`object_key` varchar(120) NOT NULL,
`object_value` text,
`last_updated` datetime NOT NULL,
PRIMARY KEY (`id`),
INDEX `name-b` (`object_name`),
INDEX `key-b` (`object_key`),
INDEX `last_updated-b` (`last_updated`)
) ENGINE=MyISAM;
-- tables for search engine
CREATE TABLE IF NOT EXISTS `aidDENSEINDEX` (
`name_id` INT( 10 ) NOT NULL,
`person_name` VARCHAR( 256 ) NOT NULL,
`personids` LONGBLOB NOT NULL,
PRIMARY KEY (`name_id`)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS `aidINVERTEDLISTS` (
`qgram` VARCHAR( 4 ) NOT NULL,
`inverted_list` LONGBLOB NOT NULL,
`list_cardinality` INT( 10 ) NOT NULL,
PRIMARY KEY (`qgram`)
) ENGINE=MyISAM;
-- refextract tables:
CREATE TABLE IF NOT EXISTS `xtrJOB` (
`id` tinyint(4) NOT NULL AUTO_INCREMENT,
`name` varchar(30) NOT NULL,
`last_updated` datetime NOT NULL,
`last_recid` mediumint(8) unsigned NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM;
-- tables for bibsort module
CREATE TABLE IF NOT EXISTS bsrMETHOD (
id mediumint(8) unsigned NOT NULL auto_increment,
name varchar(20) NOT NULL,
definition varchar(255) NOT NULL,
washer varchar(255) NOT NULL,
PRIMARY KEY (id),
UNIQUE KEY (name)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bsrMETHODNAME (
id_bsrMETHOD mediumint(8) unsigned NOT NULL,
ln char(5) NOT NULL default '',
type char(3) NOT NULL default 'sn',
value varchar(255) NOT NULL,
PRIMARY KEY (id_bsrMETHOD, ln, type)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bsrMETHODDATA (
id_bsrMETHOD mediumint(8) unsigned NOT NULL,
data_dict longblob,
data_dict_ordered longblob,
data_list_sorted longblob,
last_updated datetime,
PRIMARY KEY (id_bsrMETHOD)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS bsrMETHODDATABUCKET (
id_bsrMETHOD mediumint(8) unsigned NOT NULL,
bucket_no tinyint(2) NOT NULL,
bucket_data longblob,
bucket_last_value varchar(255),
last_updated datetime,
PRIMARY KEY (id_bsrMETHOD, bucket_no)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS collection_bsrMETHOD (
id_collection mediumint(9) unsigned NOT NULL,
id_bsrMETHOD mediumint(9) unsigned NOT NULL,
score tinyint(4) unsigned NOT NULL default '0',
PRIMARY KEY (id_collection, id_bsrMETHOD)
) ENGINE=MyISAM;
-- tables for sequence storage
CREATE TABLE IF NOT EXISTS seqSTORE (
id int(15) NOT NULL auto_increment,
seq_name varchar(15),
- seq_value varchar(60),
+ seq_value varchar(255),
PRIMARY KEY (id),
UNIQUE KEY seq_name_value (seq_name, seq_value)
) ENGINE=MyISAM;
-- tables for linkbacks:
CREATE TABLE IF NOT EXISTS lnkENTRY (
id int(15) NOT NULL auto_increment,
origin_url varchar(100) NOT NULL, -- url of the originating resource
id_bibrec mediumint(8) unsigned NOT NULL, -- bibrecord
additional_properties longblob,
type varchar(30) NOT NULL,
status varchar(30) NOT NULL default 'PENDING',
insert_time datetime default '0000-00-00 00:00:00',
PRIMARY KEY (id),
INDEX (id_bibrec),
INDEX (type),
INDEX (status),
INDEX (insert_time)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS lnkENTRYURLTITLE (
id int(15) unsigned NOT NULL auto_increment,
url varchar(100) NOT NULL,
title varchar(100) NOT NULL,
manual_set boolean NOT NULL default 0,
broken_count int(5) default 0,
broken boolean NOT NULL default 0,
PRIMARY KEY (id),
UNIQUE (url),
INDEX (title)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS lnkENTRYLOG (
id_lnkENTRY int(15) unsigned NOT NULL,
id_lnkLOG int(15) unsigned NOT NULL,
FOREIGN KEY (id_lnkENTRY) REFERENCES lnkENTRY(id),
FOREIGN KEY (id_lnkLOG) REFERENCES lnkLOG(id)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS lnkLOG (
id int(15) unsigned NOT NULL auto_increment,
id_user int(15) unsigned,
action varchar(30) NOT NULL,
log_time datetime default '0000-00-00 00:00:00',
PRIMARY KEY (id),
INDEX (id_user),
INDEX (action),
INDEX (log_time)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS lnkADMINURL (
id int(15) unsigned NOT NULL auto_increment,
url varchar(100) NOT NULL,
list varchar(30) NOT NULL,
PRIMARY KEY (id),
UNIQUE (url),
INDEX (list)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS lnkADMINURLLOG (
id_lnkADMINURL int(15) unsigned NOT NULL,
id_lnkLOG int(15) unsigned NOT NULL,
FOREIGN KEY (id_lnkADMINURL) REFERENCES lnkADMINURL(id),
FOREIGN KEY (id_lnkLOG) REFERENCES lnkLOG(id)
) ENGINE=MyISAM;
-- table for API key
CREATE TABLE IF NOT EXISTS webapikey (
id varchar(150) NOT NULL,
secret varchar(150) NOT NULL,
id_user int(15) NOT NULL,
status varchar(25) NOT NULL default 'OK',
description varchar(255) default NULL,
PRIMARY KEY (id),
KEY (id_user),
KEY (status)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS `wapCACHE` (
`object_name` varchar(120) NOT NULL,
`object_key` varchar(120) NOT NULL,
`object_value` longblob,
`object_status` varchar(120),
`last_updated` datetime NOT NULL,
PRIMARY KEY (`object_name`,`object_key`),
INDEX `last_updated-b` (`last_updated`),
INDEX `status-b` (`object_status`)
) ENGINE=MyISAM;
-- table for bibedit cache
CREATE TABLE IF NOT EXISTS `bibEDITCACHE` (
`id_bibrec` mediumint(8) unsigned NOT NULL,
`uid` int(15) unsigned NOT NULL,
`data` LONGBLOB,
`post_date` datetime NOT NULL,
`is_active` tinyint(1) NOT NULL DEFAULT 1,
PRIMARY KEY (`id_bibrec`, `uid`),
INDEX `post_date` (`post_date`)
) ENGINE=MyISAM;
-- tables for goto:
CREATE TABLE IF NOT EXISTS goto (
label varchar(150) NOT NULL,
plugin varchar(150) NOT NULL,
parameters text NOT NULL,
creation_date datetime NOT NULL,
modification_date datetime NOT NULL,
PRIMARY KEY (label),
KEY (creation_date),
KEY (modification_date)
) ENGINE=MyISAM;
-- tables for bibcheck
CREATE TABLE IF NOT EXISTS bibcheck_rules (
name varchar(150) NOT NULL,
last_run datetime NOT NULL default '0000-00-00',
PRIMARY KEY (name)
) ENGINE=MyISAM;
-- tables for author list manager
CREATE TABLE IF NOT EXISTS `aulPAPERS` (
`id` int(15) unsigned NOT NULL auto_increment,
`id_user` int(15) unsigned NOT NULL,
`title` varchar(255) NOT NULL,
`collaboration` varchar(255) NOT NULL,
`experiment_number` varchar(255) NOT NULL,
`last_modified` int unsigned NOT NULL,
PRIMARY KEY (`id`),
INDEX(`id_user`)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS `aulREFERENCES` (
`item` int(15) unsigned NOT NULL,
`reference` varchar(120) NOT NULL,
`paper_id` int(15) unsigned NOT NULL REFERENCES `aulPAPERS(id)`,
PRIMARY KEY (`item`, `paper_id`),
INDEX(`item`),
INDEX(`paper_id`)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS `aulAFFILIATIONS` (
`item` int(15) unsigned NOT NULL,
`acronym` varchar(120) NOT NULL,
`umbrella` varchar(120) NOT NULL,
`name_and_address` varchar(255) NOT NULL,
`domain` varchar(120) NOT NULL,
`member` boolean NOT NULL,
`spires_id` varchar(60) NOT NULL,
`paper_id` int(15) unsigned NOT NULL REFERENCES `aulPAPERS(id)`,
PRIMARY KEY (`item`, `paper_id`),
INDEX(`item`),
INDEX(`paper_id`),
INDEX (`acronym`)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS `aulAUTHORS` (
`item` int(15) unsigned NOT NULL,
`family_name` varchar(255) NOT NULL,
`given_name` varchar(255) NOT NULL,
`name_on_paper` varchar(255) NOT NULL,
`status` varchar(30) NOT NULL,
`paper_id` int(15) unsigned NOT NULL REFERENCES `aulPAPERS(id)`,
PRIMARY KEY (`item`, `paper_id`),
INDEX(`item`),
INDEX(`paper_id`)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS `aulAUTHOR_AFFILIATIONS` (
`item` int(15) unsigned NOT NULL,
`affiliation_acronym` varchar(120) NOT NULL,
`affiliation_status` varchar(120) NOT NULL,
`author_item` int(15) unsigned NOT NULL,
`paper_id` int(15) unsigned NOT NULL REFERENCES `aulPAPERS(id)`,
PRIMARY KEY (`item`, `author_item`, `paper_id`),
INDEX(`item`),
INDEX(`author_item`),
INDEX(`paper_id`)
) ENGINE=MyISAM;
CREATE TABLE IF NOT EXISTS `aulAUTHOR_IDENTIFIERS` (
`item` int(15) unsigned NOT NULL,
`identifier_number` varchar(120) NOT NULL,
`identifier_name` varchar(120) NOT NULL,
`author_item` int(15) unsigned NOT NULL,
`paper_id` int(15) unsigned NOT NULL REFERENCES `aulPAPERS(id)`,
PRIMARY KEY (`item`, `author_item`, `paper_id`),
INDEX(`item`),
INDEX(`author_item`),
INDEX(`paper_id`)
) ENGINE=MyISAM;
-- tables for invenio_upgrader
CREATE TABLE IF NOT EXISTS upgrade (
upgrade varchar(255) NOT NULL,
applied DATETIME NOT NULL,
PRIMARY KEY (upgrade)
) ENGINE=MyISAM;
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_release_1_1_0',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_10_31_tablesorter_location',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_11_01_lower_user_email',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_11_21_aiduserinputlog_userid_check',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_11_15_hstRECORD_marcxml_longblob',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_12_06_new_citation_dict_table',NOW());
+INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_10_25_new_param_websubmit_function',NOW());
+INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_09_10_new_param_websubmit_function',NOW());
+INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_11_12_new_param_websubmit_function',NOW());
-- master upgrade recipes:
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_10_29_idxINDEX_new_indexer_column',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_11_04_circulation_and_linkback_updates',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_11_07_xtrjob_last_recid',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_11_27_new_selfcite_tables',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_12_11_new_citation_errors_table',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_01_08_new_goto_table',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_11_15_bibdocfile_model',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_02_01_oaiREPOSITORY_last_updated',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_07_crcILLREQUEST_overdue_letter',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_01_12_bibrec_master_format',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_06_11_rnkDOWNLOADS_file_format',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_20_idxINDEX_synonym_kb',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_21_idxINDEX_stopwords',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_25_idxINDEX_html_markup',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_28_idxINDEX_tokenizer',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_29_idxINDEX_stopwords_update',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_08_20_bibauthority_updates',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_08_22_new_index_itemcount',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_10_18_crcLIBRARY_type',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_10_18_new_index_filetype',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_10_25_delete_recjson_cache',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_08_22_hstRECORD_affected_fields',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_09_25_virtual_indexes',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_09_30_indexer_interface',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_04_30_new_plotextractor_websubmit_function',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_02_06_new_collectionboxname_table',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_20_new_self_citation_dict_table',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_26_new_citation_log_table',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_28_bibindex_bibrank_type_index',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_04_11_bibformat_2nd_pass',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_06_24_new_bibsched_status_table',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_09_02_new_bibARXIVPDF',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_12_05_oaiHARVEST_arguments_blob',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_09_13_new_bibEDITCACHE',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_09_26_webauthorlist',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_10_11_bibHOLDINGPEN_longblob',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_06_20_new_bibcheck_rules_table',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2012_10_31_WebAuthorProfile_bibformat_dependency_update',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_18_aidPERSONIDDATA_last_updated',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_18_bibauthorid_search_engine_tables',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_03_18_wapCACHE_object_value_longblob',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_09_16_aidPERSONIDDATA_datablob',NOW());
INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_12_04_seqSTORE_larger_value',NOW());
+INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2014_01_22_redis_sessions',NOW());
+INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2014_01_24_seqSTORE_larger_value',NOW());
+INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2014_01_22_queue_table_virtual_index',NOW());
+INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2013_12_05_new_index_doi',NOW());
+INSERT INTO upgrade (upgrade, applied) VALUES ('invenio_2014_03_13_new_index_filename',NOW());
-- end of file
diff --git a/invenio/legacy/miscutil/sql/tabdrop.sql b/invenio/legacy/miscutil/sql/tabdrop.sql
index 6e2fc6c6a..66e16d8e0 100644
--- a/invenio/legacy/miscutil/sql/tabdrop.sql
+++ b/invenio/legacy/miscutil/sql/tabdrop.sql
@@ -1,560 +1,575 @@
-- $Id$
-- This file is part of Invenio.
-- Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012,
-- 2013, 2014 CERN.
--
-- Invenio is free software; you can redistribute it and/or
-- modify it under the terms of the GNU General Public License as
-- published by the Free Software Foundation; either version 2 of the
-- License, or (at your option) any later version.
--
-- Invenio is distributed in the hope that it will be useful, but
-- WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-- General Public License for more details.
--
-- You should have received a copy of the GNU General Public License
-- along with Invenio; if not, write to the Free Software Foundation, Inc.,
-- 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
DROP TABLE IF EXISTS bibrec;
DROP TABLE IF EXISTS bib00x;
DROP TABLE IF EXISTS bib01x;
DROP TABLE IF EXISTS bib02x;
DROP TABLE IF EXISTS bib03x;
DROP TABLE IF EXISTS bib04x;
DROP TABLE IF EXISTS bib05x;
DROP TABLE IF EXISTS bib06x;
DROP TABLE IF EXISTS bib07x;
DROP TABLE IF EXISTS bib08x;
DROP TABLE IF EXISTS bib09x;
DROP TABLE IF EXISTS bib10x;
DROP TABLE IF EXISTS bib11x;
DROP TABLE IF EXISTS bib12x;
DROP TABLE IF EXISTS bib13x;
DROP TABLE IF EXISTS bib14x;
DROP TABLE IF EXISTS bib15x;
DROP TABLE IF EXISTS bib16x;
DROP TABLE IF EXISTS bib17x;
DROP TABLE IF EXISTS bib18x;
DROP TABLE IF EXISTS bib19x;
DROP TABLE IF EXISTS bib20x;
DROP TABLE IF EXISTS bib21x;
DROP TABLE IF EXISTS bib22x;
DROP TABLE IF EXISTS bib23x;
DROP TABLE IF EXISTS bib24x;
DROP TABLE IF EXISTS bib25x;
DROP TABLE IF EXISTS bib26x;
DROP TABLE IF EXISTS bib27x;
DROP TABLE IF EXISTS bib28x;
DROP TABLE IF EXISTS bib29x;
DROP TABLE IF EXISTS bib30x;
DROP TABLE IF EXISTS bib31x;
DROP TABLE IF EXISTS bib32x;
DROP TABLE IF EXISTS bib33x;
DROP TABLE IF EXISTS bib34x;
DROP TABLE IF EXISTS bib35x;
DROP TABLE IF EXISTS bib36x;
DROP TABLE IF EXISTS bib37x;
DROP TABLE IF EXISTS bib38x;
DROP TABLE IF EXISTS bib39x;
DROP TABLE IF EXISTS bib40x;
DROP TABLE IF EXISTS bib41x;
DROP TABLE IF EXISTS bib42x;
DROP TABLE IF EXISTS bib43x;
DROP TABLE IF EXISTS bib44x;
DROP TABLE IF EXISTS bib45x;
DROP TABLE IF EXISTS bib46x;
DROP TABLE IF EXISTS bib47x;
DROP TABLE IF EXISTS bib48x;
DROP TABLE IF EXISTS bib49x;
DROP TABLE IF EXISTS bib50x;
DROP TABLE IF EXISTS bib51x;
DROP TABLE IF EXISTS bib52x;
DROP TABLE IF EXISTS bib53x;
DROP TABLE IF EXISTS bib54x;
DROP TABLE IF EXISTS bib55x;
DROP TABLE IF EXISTS bib56x;
DROP TABLE IF EXISTS bib57x;
DROP TABLE IF EXISTS bib58x;
DROP TABLE IF EXISTS bib59x;
DROP TABLE IF EXISTS bib60x;
DROP TABLE IF EXISTS bib61x;
DROP TABLE IF EXISTS bib62x;
DROP TABLE IF EXISTS bib63x;
DROP TABLE IF EXISTS bib64x;
DROP TABLE IF EXISTS bib65x;
DROP TABLE IF EXISTS bib66x;
DROP TABLE IF EXISTS bib67x;
DROP TABLE IF EXISTS bib68x;
DROP TABLE IF EXISTS bib69x;
DROP TABLE IF EXISTS bib70x;
DROP TABLE IF EXISTS bib71x;
DROP TABLE IF EXISTS bib72x;
DROP TABLE IF EXISTS bib73x;
DROP TABLE IF EXISTS bib74x;
DROP TABLE IF EXISTS bib75x;
DROP TABLE IF EXISTS bib76x;
DROP TABLE IF EXISTS bib77x;
DROP TABLE IF EXISTS bib78x;
DROP TABLE IF EXISTS bib79x;
DROP TABLE IF EXISTS bib80x;
DROP TABLE IF EXISTS bib81x;
DROP TABLE IF EXISTS bib82x;
DROP TABLE IF EXISTS bib83x;
DROP TABLE IF EXISTS bib84x;
DROP TABLE IF EXISTS bib85x;
DROP TABLE IF EXISTS bib86x;
DROP TABLE IF EXISTS bib87x;
DROP TABLE IF EXISTS bib88x;
DROP TABLE IF EXISTS bib89x;
DROP TABLE IF EXISTS bib90x;
DROP TABLE IF EXISTS bib91x;
DROP TABLE IF EXISTS bib92x;
DROP TABLE IF EXISTS bib93x;
DROP TABLE IF EXISTS bib94x;
DROP TABLE IF EXISTS bib95x;
DROP TABLE IF EXISTS bib96x;
DROP TABLE IF EXISTS bib97x;
DROP TABLE IF EXISTS bib98x;
DROP TABLE IF EXISTS bib99x;
DROP TABLE IF EXISTS bibrec_bib00x;
DROP TABLE IF EXISTS bibrec_bib01x;
DROP TABLE IF EXISTS bibrec_bib02x;
DROP TABLE IF EXISTS bibrec_bib03x;
DROP TABLE IF EXISTS bibrec_bib04x;
DROP TABLE IF EXISTS bibrec_bib05x;
DROP TABLE IF EXISTS bibrec_bib06x;
DROP TABLE IF EXISTS bibrec_bib07x;
DROP TABLE IF EXISTS bibrec_bib08x;
DROP TABLE IF EXISTS bibrec_bib09x;
DROP TABLE IF EXISTS bibrec_bib10x;
DROP TABLE IF EXISTS bibrec_bib11x;
DROP TABLE IF EXISTS bibrec_bib12x;
DROP TABLE IF EXISTS bibrec_bib13x;
DROP TABLE IF EXISTS bibrec_bib14x;
DROP TABLE IF EXISTS bibrec_bib15x;
DROP TABLE IF EXISTS bibrec_bib16x;
DROP TABLE IF EXISTS bibrec_bib17x;
DROP TABLE IF EXISTS bibrec_bib18x;
DROP TABLE IF EXISTS bibrec_bib19x;
DROP TABLE IF EXISTS bibrec_bib20x;
DROP TABLE IF EXISTS bibrec_bib21x;
DROP TABLE IF EXISTS bibrec_bib22x;
DROP TABLE IF EXISTS bibrec_bib23x;
DROP TABLE IF EXISTS bibrec_bib24x;
DROP TABLE IF EXISTS bibrec_bib25x;
DROP TABLE IF EXISTS bibrec_bib26x;
DROP TABLE IF EXISTS bibrec_bib27x;
DROP TABLE IF EXISTS bibrec_bib28x;
DROP TABLE IF EXISTS bibrec_bib29x;
DROP TABLE IF EXISTS bibrec_bib30x;
DROP TABLE IF EXISTS bibrec_bib31x;
DROP TABLE IF EXISTS bibrec_bib32x;
DROP TABLE IF EXISTS bibrec_bib33x;
DROP TABLE IF EXISTS bibrec_bib34x;
DROP TABLE IF EXISTS bibrec_bib35x;
DROP TABLE IF EXISTS bibrec_bib36x;
DROP TABLE IF EXISTS bibrec_bib37x;
DROP TABLE IF EXISTS bibrec_bib38x;
DROP TABLE IF EXISTS bibrec_bib39x;
DROP TABLE IF EXISTS bibrec_bib40x;
DROP TABLE IF EXISTS bibrec_bib41x;
DROP TABLE IF EXISTS bibrec_bib42x;
DROP TABLE IF EXISTS bibrec_bib43x;
DROP TABLE IF EXISTS bibrec_bib44x;
DROP TABLE IF EXISTS bibrec_bib45x;
DROP TABLE IF EXISTS bibrec_bib46x;
DROP TABLE IF EXISTS bibrec_bib47x;
DROP TABLE IF EXISTS bibrec_bib48x;
DROP TABLE IF EXISTS bibrec_bib49x;
DROP TABLE IF EXISTS bibrec_bib50x;
DROP TABLE IF EXISTS bibrec_bib51x;
DROP TABLE IF EXISTS bibrec_bib52x;
DROP TABLE IF EXISTS bibrec_bib53x;
DROP TABLE IF EXISTS bibrec_bib54x;
DROP TABLE IF EXISTS bibrec_bib55x;
DROP TABLE IF EXISTS bibrec_bib56x;
DROP TABLE IF EXISTS bibrec_bib57x;
DROP TABLE IF EXISTS bibrec_bib58x;
DROP TABLE IF EXISTS bibrec_bib59x;
DROP TABLE IF EXISTS bibrec_bib60x;
DROP TABLE IF EXISTS bibrec_bib61x;
DROP TABLE IF EXISTS bibrec_bib62x;
DROP TABLE IF EXISTS bibrec_bib63x;
DROP TABLE IF EXISTS bibrec_bib64x;
DROP TABLE IF EXISTS bibrec_bib65x;
DROP TABLE IF EXISTS bibrec_bib66x;
DROP TABLE IF EXISTS bibrec_bib67x;
DROP TABLE IF EXISTS bibrec_bib68x;
DROP TABLE IF EXISTS bibrec_bib69x;
DROP TABLE IF EXISTS bibrec_bib70x;
DROP TABLE IF EXISTS bibrec_bib71x;
DROP TABLE IF EXISTS bibrec_bib72x;
DROP TABLE IF EXISTS bibrec_bib73x;
DROP TABLE IF EXISTS bibrec_bib74x;
DROP TABLE IF EXISTS bibrec_bib75x;
DROP TABLE IF EXISTS bibrec_bib76x;
DROP TABLE IF EXISTS bibrec_bib77x;
DROP TABLE IF EXISTS bibrec_bib78x;
DROP TABLE IF EXISTS bibrec_bib79x;
DROP TABLE IF EXISTS bibrec_bib80x;
DROP TABLE IF EXISTS bibrec_bib81x;
DROP TABLE IF EXISTS bibrec_bib82x;
DROP TABLE IF EXISTS bibrec_bib83x;
DROP TABLE IF EXISTS bibrec_bib84x;
DROP TABLE IF EXISTS bibrec_bib85x;
DROP TABLE IF EXISTS bibrec_bib86x;
DROP TABLE IF EXISTS bibrec_bib87x;
DROP TABLE IF EXISTS bibrec_bib88x;
DROP TABLE IF EXISTS bibrec_bib89x;
DROP TABLE IF EXISTS bibrec_bib90x;
DROP TABLE IF EXISTS bibrec_bib91x;
DROP TABLE IF EXISTS bibrec_bib92x;
DROP TABLE IF EXISTS bibrec_bib93x;
DROP TABLE IF EXISTS bibrec_bib94x;
DROP TABLE IF EXISTS bibrec_bib95x;
DROP TABLE IF EXISTS bibrec_bib96x;
DROP TABLE IF EXISTS bibrec_bib97x;
DROP TABLE IF EXISTS bibrec_bib98x;
DROP TABLE IF EXISTS bibrec_bib99x;
DROP TABLE IF EXISTS bibfmt;
DROP TABLE IF EXISTS idxINDEX;
DROP TABLE IF EXISTS idxINDEXNAME;
DROP TABLE IF EXISTS idxINDEX_field;
DROP TABLE IF EXISTS idxINDEX_idxINDEX;
DROP TABLE IF EXISTS idxWORD01F;
DROP TABLE IF EXISTS idxWORD02F;
DROP TABLE IF EXISTS idxWORD03F;
DROP TABLE IF EXISTS idxWORD04F;
DROP TABLE IF EXISTS idxWORD05F;
DROP TABLE IF EXISTS idxWORD06F;
DROP TABLE IF EXISTS idxWORD07F;
DROP TABLE IF EXISTS idxWORD08F;
DROP TABLE IF EXISTS idxWORD09F;
DROP TABLE IF EXISTS idxWORD10F;
DROP TABLE IF EXISTS idxWORD11F;
DROP TABLE IF EXISTS idxWORD12F;
DROP TABLE IF EXISTS idxWORD13F;
DROP TABLE IF EXISTS idxWORD14F;
DROP TABLE IF EXISTS idxWORD15F;
DROP TABLE IF EXISTS idxWORD16F;
DROP TABLE IF EXISTS idxWORD17F;
DROP TABLE IF EXISTS idxWORD18F;
DROP TABLE IF EXISTS idxWORD19F;
DROP TABLE IF EXISTS idxWORD20F;
DROP TABLE IF EXISTS idxWORD21F;
DROP TABLE IF EXISTS idxWORD22F;
DROP TABLE IF EXISTS idxWORD23F;
DROP TABLE IF EXISTS idxWORD24F;
DROP TABLE IF EXISTS idxWORD25F;
DROP TABLE IF EXISTS idxWORD26F;
+DROP TABLE IF EXISTS idxWORD27F;
+DROP TABLE IF EXISTS idxWORD28F;
DROP TABLE IF EXISTS idxWORD01R;
DROP TABLE IF EXISTS idxWORD02R;
DROP TABLE IF EXISTS idxWORD03R;
DROP TABLE IF EXISTS idxWORD04R;
DROP TABLE IF EXISTS idxWORD05R;
DROP TABLE IF EXISTS idxWORD06R;
DROP TABLE IF EXISTS idxWORD07R;
DROP TABLE IF EXISTS idxWORD08R;
DROP TABLE IF EXISTS idxWORD09R;
DROP TABLE IF EXISTS idxWORD10R;
DROP TABLE IF EXISTS idxWORD11R;
DROP TABLE IF EXISTS idxWORD12R;
DROP TABLE IF EXISTS idxWORD13R;
DROP TABLE IF EXISTS idxWORD14R;
DROP TABLE IF EXISTS idxWORD15R;
DROP TABLE IF EXISTS idxWORD16R;
DROP TABLE IF EXISTS idxWORD17R;
DROP TABLE IF EXISTS idxWORD18R;
DROP TABLE IF EXISTS idxWORD19R;
DROP TABLE IF EXISTS idxWORD20R;
DROP TABLE IF EXISTS idxWORD21R;
DROP TABLE IF EXISTS idxWORD22R;
DROP TABLE IF EXISTS idxWORD23R;
DROP TABLE IF EXISTS idxWORD24R;
DROP TABLE IF EXISTS idxWORD25R;
DROP TABLE IF EXISTS idxWORD26R;
+DROP TABLE IF EXISTS idxWORD27R;
+DROP TABLE IF EXISTS idxWORD28R;
+DROP TABLE IF EXISTS idxWORD01Q;
DROP TABLE IF EXISTS idxPAIR01F;
DROP TABLE IF EXISTS idxPAIR02F;
DROP TABLE IF EXISTS idxPAIR03F;
DROP TABLE IF EXISTS idxPAIR04F;
DROP TABLE IF EXISTS idxPAIR05F;
DROP TABLE IF EXISTS idxPAIR06F;
DROP TABLE IF EXISTS idxPAIR07F;
DROP TABLE IF EXISTS idxPAIR08F;
DROP TABLE IF EXISTS idxPAIR09F;
DROP TABLE IF EXISTS idxPAIR10F;
DROP TABLE IF EXISTS idxPAIR11F;
DROP TABLE IF EXISTS idxPAIR12F;
DROP TABLE IF EXISTS idxPAIR13F;
DROP TABLE IF EXISTS idxPAIR14F;
DROP TABLE IF EXISTS idxPAIR15F;
DROP TABLE IF EXISTS idxPAIR16F;
DROP TABLE IF EXISTS idxPAIR17F;
DROP TABLE IF EXISTS idxPAIR18F;
DROP TABLE IF EXISTS idxPAIR19F;
DROP TABLE IF EXISTS idxPAIR20F;
DROP TABLE IF EXISTS idxPAIR21F;
DROP TABLE IF EXISTS idxPAIR22F;
DROP TABLE IF EXISTS idxPAIR23F;
DROP TABLE IF EXISTS idxPAIR24F;
DROP TABLE IF EXISTS idxPAIR25F;
DROP TABLE IF EXISTS idxPAIR26F;
+DROP TABLE IF EXISTS idxPAIR27F;
+DROP TABLE IF EXISTS idxPAIR28F;
DROP TABLE IF EXISTS idxPAIR01R;
DROP TABLE IF EXISTS idxPAIR02R;
DROP TABLE IF EXISTS idxPAIR03R;
DROP TABLE IF EXISTS idxPAIR04R;
DROP TABLE IF EXISTS idxPAIR05R;
DROP TABLE IF EXISTS idxPAIR06R;
DROP TABLE IF EXISTS idxPAIR07R;
DROP TABLE IF EXISTS idxPAIR08R;
DROP TABLE IF EXISTS idxPAIR09R;
DROP TABLE IF EXISTS idxPAIR10R;
DROP TABLE IF EXISTS idxPAIR11R;
DROP TABLE IF EXISTS idxPAIR12R;
DROP TABLE IF EXISTS idxPAIR13R;
DROP TABLE IF EXISTS idxPAIR14R;
DROP TABLE IF EXISTS idxPAIR15R;
DROP TABLE IF EXISTS idxPAIR16R;
DROP TABLE IF EXISTS idxPAIR17R;
DROP TABLE IF EXISTS idxPAIR18R;
DROP TABLE IF EXISTS idxPAIR19R;
DROP TABLE IF EXISTS idxPAIR20R;
DROP TABLE IF EXISTS idxPAIR21R;
DROP TABLE IF EXISTS idxPAIR22R;
DROP TABLE IF EXISTS idxPAIR23R;
DROP TABLE IF EXISTS idxPAIR24R;
DROP TABLE IF EXISTS idxPAIR25R;
DROP TABLE IF EXISTS idxPAIR26R;
+DROP TABLE IF EXISTS idxPAIR27R;
+DROP TABLE IF EXISTS idxPAIR28R;
+DROP TABLE IF EXISTS idxPAIR01Q;
DROP TABLE IF EXISTS idxPHRASE01F;
DROP TABLE IF EXISTS idxPHRASE02F;
DROP TABLE IF EXISTS idxPHRASE03F;
DROP TABLE IF EXISTS idxPHRASE04F;
DROP TABLE IF EXISTS idxPHRASE05F;
DROP TABLE IF EXISTS idxPHRASE06F;
DROP TABLE IF EXISTS idxPHRASE07F;
DROP TABLE IF EXISTS idxPHRASE08F;
DROP TABLE IF EXISTS idxPHRASE09F;
DROP TABLE IF EXISTS idxPHRASE10F;
DROP TABLE IF EXISTS idxPHRASE11F;
DROP TABLE IF EXISTS idxPHRASE12F;
DROP TABLE IF EXISTS idxPHRASE13F;
DROP TABLE IF EXISTS idxPHRASE14F;
DROP TABLE IF EXISTS idxPHRASE15F;
DROP TABLE IF EXISTS idxPHRASE16F;
DROP TABLE IF EXISTS idxPHRASE17F;
DROP TABLE IF EXISTS idxPHRASE18F;
DROP TABLE IF EXISTS idxPHRASE19F;
DROP TABLE IF EXISTS idxPHRASE20F;
DROP TABLE IF EXISTS idxPHRASE21F;
DROP TABLE IF EXISTS idxPHRASE22F;
DROP TABLE IF EXISTS idxPHRASE23F;
DROP TABLE IF EXISTS idxPHRASE24F;
DROP TABLE IF EXISTS idxPHRASE25F;
DROP TABLE IF EXISTS idxPHRASE26F;
+DROP TABLE IF EXISTS idxPHRASE27F;
+DROP TABLE IF EXISTS idxPHRASE28F;
DROP TABLE IF EXISTS idxPHRASE01R;
DROP TABLE IF EXISTS idxPHRASE02R;
DROP TABLE IF EXISTS idxPHRASE03R;
DROP TABLE IF EXISTS idxPHRASE04R;
DROP TABLE IF EXISTS idxPHRASE05R;
DROP TABLE IF EXISTS idxPHRASE06R;
DROP TABLE IF EXISTS idxPHRASE07R;
DROP TABLE IF EXISTS idxPHRASE08R;
DROP TABLE IF EXISTS idxPHRASE09R;
DROP TABLE IF EXISTS idxPHRASE10R;
DROP TABLE IF EXISTS idxPHRASE11R;
DROP TABLE IF EXISTS idxPHRASE12R;
DROP TABLE IF EXISTS idxPHRASE13R;
DROP TABLE IF EXISTS idxPHRASE14R;
DROP TABLE IF EXISTS idxPHRASE15R;
DROP TABLE IF EXISTS idxPHRASE16R;
DROP TABLE IF EXISTS idxPHRASE17R;
DROP TABLE IF EXISTS idxPHRASE18R;
DROP TABLE IF EXISTS idxPHRASE19R;
DROP TABLE IF EXISTS idxPHRASE20R;
DROP TABLE IF EXISTS idxPHRASE21R;
DROP TABLE IF EXISTS idxPHRASE22R;
DROP TABLE IF EXISTS idxPHRASE23R;
DROP TABLE IF EXISTS idxPHRASE24R;
DROP TABLE IF EXISTS idxPHRASE25R;
DROP TABLE IF EXISTS idxPHRASE26R;
+DROP TABLE IF EXISTS idxPHRASE27R;
+DROP TABLE IF EXISTS idxPHRASE28R;
+DROP TABLE IF EXISTS idxPHRASE01Q;
DROP TABLE IF EXISTS rnkMETHOD;
DROP TABLE IF EXISTS rnkMETHODNAME;
DROP TABLE IF EXISTS rnkMETHODDATA;
DROP TABLE IF EXISTS rnkWORD01F;
DROP TABLE IF EXISTS rnkWORD01R;
DROP TABLE IF EXISTS rnkPAGEVIEWS;
DROP TABLE IF EXISTS rnkDOWNLOADS;
DROP TABLE IF EXISTS rnkCITATIONDATAEXT;
DROP TABLE IF EXISTS rnkCITATIONDATAERR;
DROP TABLE IF EXISTS rnkCITATIONDICT;
DROP TABLE IF EXISTS rnkCITATIONLOG;
DROP TABLE IF EXISTS rnkAUTHORDATA;
DROP TABLE IF EXISTS rnkRECORDSCACHE;
DROP TABLE IF EXISTS rnkEXTENDEDAUTHORS;
DROP TABLE IF EXISTS rnkSELFCITES;
DROP TABLE IF EXISTS collection_rnkMETHOD;
DROP TABLE IF EXISTS collection;
DROP TABLE IF EXISTS collectionname;
DROP TABLE IF EXISTS collectionboxname;
DROP TABLE IF EXISTS oaiREPOSITORY;
DROP TABLE IF EXISTS oaiHARVEST;
DROP TABLE IF EXISTS oaiHARVESTLOG;
DROP TABLE IF EXISTS bibHOLDINGPEN;
DROP TABLE IF EXISTS bibARXIVPDF;
DROP TABLE IF EXISTS collection_collection;
DROP TABLE IF EXISTS collection_portalbox;
DROP TABLE IF EXISTS portalbox;
DROP TABLE IF EXISTS collection_example;
DROP TABLE IF EXISTS example;
DROP TABLE IF EXISTS collection_format;
DROP TABLE IF EXISTS format;
DROP TABLE IF EXISTS formatname;
DROP TABLE IF EXISTS collection_field_fieldvalue;
DROP TABLE IF EXISTS field;
DROP TABLE IF EXISTS fieldname;
DROP TABLE IF EXISTS fieldvalue;
DROP TABLE IF EXISTS field_tag;
DROP TABLE IF EXISTS tag;
DROP TABLE IF EXISTS publreq;
DROP TABLE IF EXISTS session;
DROP TABLE IF EXISTS user;
DROP TABLE IF EXISTS userEXT;
DROP TABLE IF EXISTS accROLE;
DROP TABLE IF EXISTS accMAILCOOKIE;
DROP TABLE IF EXISTS user_accROLE;
DROP TABLE IF EXISTS accACTION;
DROP TABLE IF EXISTS accARGUMENT;
DROP TABLE IF EXISTS accROLE_accACTION_accARGUMENT;
DROP TABLE IF EXISTS user_query;
DROP TABLE IF EXISTS query;
DROP TABLE IF EXISTS user_basket;
DROP TABLE IF EXISTS basket;
DROP TABLE IF EXISTS basket_record;
DROP TABLE IF EXISTS record;
DROP TABLE IF EXISTS user_query_basket;
DROP TABLE IF EXISTS cmtRECORDCOMMENT;
DROP TABLE IF EXISTS cmtCOLLAPSED;
DROP TABLE IF EXISTS cmtNOTECOLLAPSED;
DROP TABLE IF EXISTS knwKB;
DROP TABLE IF EXISTS knwKBRVAL;
DROP TABLE IF EXISTS knwKBDDEF;
DROP TABLE IF EXISTS sbmACTION;
DROP TABLE IF EXISTS sbmALLFUNCDESCR;
DROP TABLE IF EXISTS sbmAPPROVAL;
DROP TABLE IF EXISTS sbmCPLXAPPROVAL;
DROP TABLE IF EXISTS sbmCOLLECTION;
DROP TABLE IF EXISTS sbmCOLLECTION_sbmCOLLECTION;
DROP TABLE IF EXISTS sbmCOLLECTION_sbmDOCTYPE;
DROP TABLE IF EXISTS sbmCATEGORIES;
DROP TABLE IF EXISTS sbmCHECKS;
DROP TABLE IF EXISTS sbmCOOKIES;
DROP TABLE IF EXISTS sbmDOCTYPE;
DROP TABLE IF EXISTS sbmFIELD;
DROP TABLE IF EXISTS sbmFIELDDESC;
DROP TABLE IF EXISTS sbmFORMATEXTENSION;
DROP TABLE IF EXISTS sbmFUNCTIONS;
DROP TABLE IF EXISTS sbmFUNDESC;
DROP TABLE IF EXISTS sbmGFILERESULT;
DROP TABLE IF EXISTS sbmIMPLEMENT;
DROP TABLE IF EXISTS sbmPARAMETERS;
DROP TABLE IF EXISTS sbmPUBLICATION;
DROP TABLE IF EXISTS sbmPUBLICATIONCOMM;
DROP TABLE IF EXISTS sbmPUBLICATIONDATA;
DROP TABLE IF EXISTS sbmREFEREES;
DROP TABLE IF EXISTS sbmSUBMISSIONS;
DROP TABLE IF EXISTS schTASK;
DROP TABLE IF EXISTS bibdoc;
DROP TABLE IF EXISTS bibdoc_bibdoc;
DROP TABLE IF EXISTS bibdocmoreinfo;
DROP TABLE IF EXISTS bibrec_bibdoc;
DROP TABLE IF EXISTS bibdocfsinfo;
DROP TABLE IF EXISTS usergroup;
DROP TABLE IF EXISTS user_usergroup;
DROP TABLE IF EXISTS user_basket;
DROP TABLE IF EXISTS msgMESSAGE;
DROP TABLE IF EXISTS user_msgMESSAGE;
DROP TABLE IF EXISTS bskBASKET;
DROP TABLE IF EXISTS bskEXTREC;
DROP TABLE IF EXISTS bskEXTFMT;
DROP TABLE IF EXISTS bskREC;
DROP TABLE IF EXISTS bskRECORDCOMMENT;
DROP TABLE IF EXISTS cmtACTIONHISTORY;
DROP TABLE IF EXISTS cmtSUBSCRIPTION;
DROP TABLE IF EXISTS user_bskBASKET;
DROP TABLE IF EXISTS usergroup_bskBASKET;
DROP TABLE IF EXISTS collection_externalcollection;
DROP TABLE IF EXISTS externalcollection;
DROP TABLE IF EXISTS collectiondetailedrecordpagetabs;
DROP TABLE IF EXISTS staEVENT;
DROP TABLE IF EXISTS clsMETHOD;
DROP TABLE IF EXISTS collection_clsMETHOD;
DROP TABLE IF EXISTS jrnJOURNAL;
DROP TABLE IF EXISTS jrnISSUE;
DROP TABLE IF EXISTS hstRECORD;
DROP TABLE IF EXISTS hstDOCUMENT;
DROP TABLE IF EXISTS hstTASK;
DROP TABLE IF EXISTS hstBATCHUPLOAD;
DROP TABLE IF EXISTS crcBORROWER;
DROP TABLE IF EXISTS crcILLREQUEST;
DROP TABLE IF EXISTS crcITEM;
DROP TABLE IF EXISTS crcLIBRARY;
DROP TABLE IF EXISTS crcLOAN;
DROP TABLE IF EXISTS crcLOANREQUEST;
DROP TABLE IF EXISTS crcPURCHASE;
DROP TABLE IF EXISTS crcVENDOR;
DROP TABLE IF EXISTS expJOB;
DROP TABLE IF EXISTS expQUERY;
DROP TABLE IF EXISTS expJOB_expQUERY;
DROP TABLE IF EXISTS expQUERYRESULT;
DROP TABLE IF EXISTS expJOBRESULT;
DROP TABLE IF EXISTS expJOBRESULT_expQUERYRESULT;
DROP TABLE IF EXISTS user_expJOB;
DROP TABLE IF EXISTS swrREMOTESERVER;
DROP TABLE IF EXISTS swrCLIENTDATA;
DROP TABLE IF EXISTS hstEXCEPTION;
DROP TABLE IF EXISTS aidUSERINPUTLOG;
DROP TABLE IF EXISTS aidCACHE;
DROP TABLE IF EXISTS aidDENSEINDEX;
DROP TABLE IF EXISTS aidINVERTEDLISTS;
DROP TABLE IF EXISTS aidPERSONIDDATA;
DROP TABLE IF EXISTS aidPERSONIDPAPERS;
DROP TABLE IF EXISTS aidRESULTS;
DROP TABLE IF EXISTS xtrJOB;
DROP TABLE IF EXISTS bsrMETHOD;
DROP TABLE IF EXISTS bsrMETHODNAME;
DROP TABLE IF EXISTS bsrMETHODDATA;
DROP TABLE IF EXISTS bsrMETHODDATABUCKET;
DROP TABLE IF EXISTS collection_bsrMETHOD;
DROP TABLE IF EXISTS lnkENTRY;
DROP TABLE IF EXISTS lnkENTRYURLTITLE;
DROP TABLE IF EXISTS lnkENTRYLOG;
DROP TABLE IF EXISTS lnkLOG;
DROP TABLE IF EXISTS lnkADMINURL;
DROP TABLE IF EXISTS lnkADMINURLLOG;
DROP TABLE IF EXISTS webapikey;
DROP TABLE IF EXISTS wapCACHE;
DROP TABLE IF EXISTS seqSTORE;
DROP TABLE IF EXISTS upgrade;
DROP TABLE IF EXISTS goto;
DROP TABLE IF EXISTS rnkSELFCITEDICT;
DROP TABLE IF EXISTS schSTATUS;
DROP TABLE IF EXISTS oauth1_storage;
DROP TABLE IF EXISTS bibEDITCACHE;
DROP TABLE IF EXISTS aulPAPERS;
DROP TABLE IF EXISTS aulREFERENCES;
DROP TABLE IF EXISTS aulAUTHORS;
DROP TABLE IF EXISTS aulAFFILIATIONS;
DROP TABLE IF EXISTS aulAUTHOR_AFFILIATIONS;
DROP TABLE IF EXISTS aulAUTHOR_IDENTIFIERS;
DROP TABLE IF EXISTS bibcheck_rules;
DROP TABLE IF EXISTS depWORKFLOW;
DROP TABLE IF EXISTS depDRAFT;
DROP TABLE IF EXISTS bwlAUDITLOGGING;
DROP TABLE IF EXISTS bwlOBJECT;
DROP TABLE IF EXISTS bwlTASKLOGGING;
DROP TABLE IF EXISTS bwlWORKFLOW;
DROP TABLE IF EXISTS bwlWORKFLOWLOGGING;
-- end of file
diff --git a/invenio/legacy/miscutil/sql/tabfill.sql b/invenio/legacy/miscutil/sql/tabfill.sql
index 03e315dd5..ba7fd371b 100644
--- a/invenio/legacy/miscutil/sql/tabfill.sql
+++ b/invenio/legacy/miscutil/sql/tabfill.sql
@@ -1,862 +1,868 @@
-- This file is part of Invenio.
-- Copyright (C) 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
--
-- Invenio is free software; you can redistribute it and/or
-- modify it under the terms of the GNU General Public License as
-- published by the Free Software Foundation; either version 2 of the
-- License, or (at your option) any later version.
--
-- Invenio is distributed in the hope that it will be useful, but
-- WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-- General Public License for more details.
--
-- You should have received a copy of the GNU General Public License
-- along with Invenio; if not, write to the Free Software Foundation, Inc.,
-- 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-- Fill Invenio configuration tables with defaults suitable for any site.
INSERT INTO rnkMETHOD (id,name,last_updated) VALUES (1,'wrd','0000-00-00 00:00:00');
INSERT INTO collection_rnkMETHOD (id_collection,id_rnkMETHOD,score) VALUES (1,1,100);
INSERT INTO field VALUES (1,'any field','anyfield');
INSERT INTO field VALUES (2,'title','title');
INSERT INTO field VALUES (3,'author','author');
INSERT INTO field VALUES (4,'abstract','abstract');
INSERT INTO field VALUES (5,'keyword','keyword');
INSERT INTO field VALUES (6,'report number','reportnumber');
INSERT INTO field VALUES (7,'subject','subject');
INSERT INTO field VALUES (8,'reference','reference');
INSERT INTO field VALUES (9,'fulltext','fulltext');
INSERT INTO field VALUES (10,'collection','collection');
INSERT INTO field VALUES (11,'division','division');
INSERT INTO field VALUES (12,'year','year');
INSERT INTO field VALUES (13,'experiment','experiment');
INSERT INTO field VALUES (14,'record ID','recid');
INSERT INTO field VALUES (15,'isbn','isbn');
INSERT INTO field VALUES (16,'issn','issn');
INSERT INTO field VALUES (17,'coden','coden');
--- INSERT INTO field VALUES (18,'doi','doi');
+INSERT INTO field VALUES (18,'doi','doi');
INSERT INTO field VALUES (19,'journal','journal');
INSERT INTO field VALUES (20,'collaboration','collaboration');
INSERT INTO field VALUES (21,'affiliation','affiliation');
INSERT INTO field VALUES (22,'exact author','exactauthor');
INSERT INTO field VALUES (23,'date created','datecreated');
INSERT INTO field VALUES (24,'date modified','datemodified');
INSERT INTO field VALUES (25,'refers to','refersto');
INSERT INTO field VALUES (26,'cited by','citedby');
INSERT INTO field VALUES (27,'caption','caption');
INSERT INTO field VALUES (28,'first author','firstauthor');
INSERT INTO field VALUES (29,'exact first author','exactfirstauthor');
INSERT INTO field VALUES (30,'author count','authorcount');
INSERT INTO field VALUES (31,'reference to','rawref');
INSERT INTO field VALUES (32,'exact title','exacttitle');
INSERT INTO field VALUES (33,'authority author','authorityauthor');
INSERT INTO field VALUES (34,'authority institute','authorityinstitute');
INSERT INTO field VALUES (35,'authority journal','authorityjournal');
INSERT INTO field VALUES (36,'authority subject','authoritysubject');
INSERT INTO field VALUES (37,'item count','itemcount');
INSERT INTO field VALUES (38,'file type','filetype');
INSERT INTO field VALUES (39,'miscellaneous', 'miscellaneous');
INSERT INTO field VALUES (40,'refers to excluding self cites','referstoexcludingselfcites');
INSERT INTO field VALUES (41,'cited by excluding self cites','citedbyexcludingselfcites');
INSERT INTO field VALUES (42,'cataloguer nickname','cataloguer');
-INSERT INTO field VALUES (43,'tag','tag');
+INSERT INTO field VALUES (43,'file name','filename');
+INSERT INTO field VALUES (44,'tag','tag');
INSERT INTO field_tag VALUES (10,11,100);
INSERT INTO field_tag VALUES (11,14,100);
INSERT INTO field_tag VALUES (12,15,10);
INSERT INTO field_tag VALUES (13,116,10);
INSERT INTO field_tag VALUES (2,3,100);
INSERT INTO field_tag VALUES (2,4,90);
INSERT INTO field_tag VALUES (3,1,100);
INSERT INTO field_tag VALUES (3,2,90);
INSERT INTO field_tag VALUES (4,5,100);
INSERT INTO field_tag VALUES (5,6,100);
INSERT INTO field_tag VALUES (6,7,30);
INSERT INTO field_tag VALUES (6,8,10);
INSERT INTO field_tag VALUES (6,9,20);
INSERT INTO field_tag VALUES (7,12,100);
INSERT INTO field_tag VALUES (7,13,90);
INSERT INTO field_tag VALUES (8,10,100);
INSERT INTO field_tag VALUES (9,115,100);
INSERT INTO field_tag VALUES (14,117,100);
INSERT INTO field_tag VALUES (15,118,100);
INSERT INTO field_tag VALUES (16,119,100);
INSERT INTO field_tag VALUES (17,120,100);
--- INSERT INTO field_tag VALUES (18,121,100);
INSERT INTO field_tag VALUES (19,131,100);
INSERT INTO field_tag VALUES (20,132,100);
INSERT INTO field_tag VALUES (21,133,100);
INSERT INTO field_tag VALUES (21,134,90);
INSERT INTO field_tag VALUES (22,1,100);
INSERT INTO field_tag VALUES (22,2,90);
INSERT INTO field_tag VALUES (27,135,100);
INSERT INTO field_tag VALUES (28,1,100);
INSERT INTO field_tag VALUES (29,1,100);
INSERT INTO field_tag VALUES (30,1,100);
INSERT INTO field_tag VALUES (30,2,90);
INSERT INTO field_tag VALUES (32,3,100);
INSERT INTO field_tag VALUES (32,4,90);
-- authority fields
INSERT INTO field_tag VALUES (33,1,100);
INSERT INTO field_tag VALUES (33,146,100);
INSERT INTO field_tag VALUES (33,140,100);
INSERT INTO field_tag VALUES (34,148,100);
INSERT INTO field_tag VALUES (34,149,100);
INSERT INTO field_tag VALUES (34,150,100);
INSERT INTO field_tag VALUES (35,151,100);
INSERT INTO field_tag VALUES (35,152,100);
INSERT INTO field_tag VALUES (35,153,100);
INSERT INTO field_tag VALUES (36,154,100);
INSERT INTO field_tag VALUES (36,155,100);
INSERT INTO field_tag VALUES (36,156,100);
-- misc fields
INSERT INTO field_tag VALUES (39,17,10);
INSERT INTO field_tag VALUES (39,18,10);
INSERT INTO field_tag VALUES (39,157,10);
INSERT INTO field_tag VALUES (39,158,10);
INSERT INTO field_tag VALUES (39,159,10);
INSERT INTO field_tag VALUES (39,160,10);
INSERT INTO field_tag VALUES (39,161,10);
INSERT INTO field_tag VALUES (39,162,10);
INSERT INTO field_tag VALUES (39,163,10);
INSERT INTO field_tag VALUES (39,164,10);
INSERT INTO field_tag VALUES (39,20,10);
INSERT INTO field_tag VALUES (39,21,10);
INSERT INTO field_tag VALUES (39,22,10);
INSERT INTO field_tag VALUES (39,23,10);
INSERT INTO field_tag VALUES (39,165,10);
INSERT INTO field_tag VALUES (39,166,10);
INSERT INTO field_tag VALUES (39,167,10);
INSERT INTO field_tag VALUES (39,168,10);
INSERT INTO field_tag VALUES (39,169,10);
INSERT INTO field_tag VALUES (39,170,10);
INSERT INTO field_tag VALUES (39,25,10);
INSERT INTO field_tag VALUES (39,27,10);
INSERT INTO field_tag VALUES (39,28,10);
INSERT INTO field_tag VALUES (39,29,10);
INSERT INTO field_tag VALUES (39,30,10);
INSERT INTO field_tag VALUES (39,31,10);
INSERT INTO field_tag VALUES (39,32,10);
INSERT INTO field_tag VALUES (39,33,10);
INSERT INTO field_tag VALUES (39,34,10);
INSERT INTO field_tag VALUES (39,35,10);
INSERT INTO field_tag VALUES (39,36,10);
INSERT INTO field_tag VALUES (39,37,10);
INSERT INTO field_tag VALUES (39,38,10);
INSERT INTO field_tag VALUES (39,39,10);
INSERT INTO field_tag VALUES (39,171,10);
INSERT INTO field_tag VALUES (39,172,10);
INSERT INTO field_tag VALUES (39,173,10);
INSERT INTO field_tag VALUES (39,174,10);
INSERT INTO field_tag VALUES (39,175,10);
INSERT INTO field_tag VALUES (39,41,10);
INSERT INTO field_tag VALUES (39,42,10);
INSERT INTO field_tag VALUES (39,43,10);
INSERT INTO field_tag VALUES (39,44,10);
INSERT INTO field_tag VALUES (39,45,10);
INSERT INTO field_tag VALUES (39,46,10);
INSERT INTO field_tag VALUES (39,47,10);
INSERT INTO field_tag VALUES (39,48,10);
INSERT INTO field_tag VALUES (39,49,10);
INSERT INTO field_tag VALUES (39,50,10);
INSERT INTO field_tag VALUES (39,51,10);
INSERT INTO field_tag VALUES (39,52,10);
INSERT INTO field_tag VALUES (39,53,10);
INSERT INTO field_tag VALUES (39,54,10);
INSERT INTO field_tag VALUES (39,55,10);
INSERT INTO field_tag VALUES (39,56,10);
INSERT INTO field_tag VALUES (39,57,10);
INSERT INTO field_tag VALUES (39,58,10);
INSERT INTO field_tag VALUES (39,59,10);
INSERT INTO field_tag VALUES (39,60,10);
INSERT INTO field_tag VALUES (39,61,10);
INSERT INTO field_tag VALUES (39,62,10);
INSERT INTO field_tag VALUES (39,63,10);
INSERT INTO field_tag VALUES (39,64,10);
INSERT INTO field_tag VALUES (39,65,10);
INSERT INTO field_tag VALUES (39,66,10);
INSERT INTO field_tag VALUES (39,67,10);
INSERT INTO field_tag VALUES (39,176,10);
INSERT INTO field_tag VALUES (39,177,10);
INSERT INTO field_tag VALUES (39,178,10);
INSERT INTO field_tag VALUES (39,179,10);
INSERT INTO field_tag VALUES (39,180,10);
INSERT INTO field_tag VALUES (39,69,10);
INSERT INTO field_tag VALUES (39,70,10);
INSERT INTO field_tag VALUES (39,71,10);
INSERT INTO field_tag VALUES (39,72,10);
INSERT INTO field_tag VALUES (39,73,10);
INSERT INTO field_tag VALUES (39,74,10);
INSERT INTO field_tag VALUES (39,75,10);
INSERT INTO field_tag VALUES (39,76,10);
INSERT INTO field_tag VALUES (39,77,10);
INSERT INTO field_tag VALUES (39,78,10);
INSERT INTO field_tag VALUES (39,79,10);
INSERT INTO field_tag VALUES (39,80,10);
INSERT INTO field_tag VALUES (39,181,10);
INSERT INTO field_tag VALUES (39,182,10);
INSERT INTO field_tag VALUES (39,183,10);
INSERT INTO field_tag VALUES (39,184,10);
INSERT INTO field_tag VALUES (39,185,10);
INSERT INTO field_tag VALUES (39,186,10);
INSERT INTO field_tag VALUES (39,82,10);
INSERT INTO field_tag VALUES (39,83,10);
INSERT INTO field_tag VALUES (39,84,10);
INSERT INTO field_tag VALUES (39,85,10);
INSERT INTO field_tag VALUES (39,187,10);
INSERT INTO field_tag VALUES (39,88,10);
INSERT INTO field_tag VALUES (39,89,10);
INSERT INTO field_tag VALUES (39,90,10);
INSERT INTO field_tag VALUES (39,91,10);
INSERT INTO field_tag VALUES (39,92,10);
INSERT INTO field_tag VALUES (39,93,10);
INSERT INTO field_tag VALUES (39,94,10);
INSERT INTO field_tag VALUES (39,95,10);
INSERT INTO field_tag VALUES (39,96,10);
INSERT INTO field_tag VALUES (39,97,10);
INSERT INTO field_tag VALUES (39,98,10);
INSERT INTO field_tag VALUES (39,99,10);
INSERT INTO field_tag VALUES (39,100,10);
INSERT INTO field_tag VALUES (39,102,10);
INSERT INTO field_tag VALUES (39,103,10);
INSERT INTO field_tag VALUES (39,104,10);
INSERT INTO field_tag VALUES (39,105,10);
INSERT INTO field_tag VALUES (39,188,10);
INSERT INTO field_tag VALUES (39,189,10);
INSERT INTO field_tag VALUES (39,190,10);
INSERT INTO field_tag VALUES (39,191,10);
INSERT INTO field_tag VALUES (39,192,10);
INSERT INTO field_tag VALUES (39,193,10);
INSERT INTO field_tag VALUES (39,194,10);
INSERT INTO field_tag VALUES (39,195,10);
INSERT INTO field_tag VALUES (39,196,10);
INSERT INTO field_tag VALUES (39,107,10);
INSERT INTO field_tag VALUES (39,108,10);
INSERT INTO field_tag VALUES (39,109,10);
INSERT INTO field_tag VALUES (39,110,10);
INSERT INTO field_tag VALUES (39,111,10);
INSERT INTO field_tag VALUES (39,112,10);
INSERT INTO field_tag VALUES (39,113,10);
INSERT INTO field_tag VALUES (39,197,10);
INSERT INTO field_tag VALUES (39,198,10);
INSERT INTO field_tag VALUES (39,199,10);
INSERT INTO field_tag VALUES (39,200,10);
INSERT INTO field_tag VALUES (39,201,10);
INSERT INTO field_tag VALUES (39,202,10);
INSERT INTO field_tag VALUES (39,203,10);
INSERT INTO field_tag VALUES (39,204,10);
INSERT INTO field_tag VALUES (39,205,10);
INSERT INTO field_tag VALUES (39,206,10);
INSERT INTO field_tag VALUES (39,207,10);
INSERT INTO field_tag VALUES (39,208,10);
INSERT INTO field_tag VALUES (39,209,10);
INSERT INTO field_tag VALUES (39,210,10);
INSERT INTO field_tag VALUES (39,211,10);
INSERT INTO field_tag VALUES (39,212,10);
INSERT INTO field_tag VALUES (39,213,10);
INSERT INTO field_tag VALUES (39,214,10);
INSERT INTO field_tag VALUES (39,215,10);
INSERT INTO field_tag VALUES (39,122,10);
INSERT INTO field_tag VALUES (39,123,10);
INSERT INTO field_tag VALUES (39,124,10);
INSERT INTO field_tag VALUES (39,125,10);
INSERT INTO field_tag VALUES (39,126,10);
INSERT INTO field_tag VALUES (39,127,10);
INSERT INTO field_tag VALUES (39,128,10);
INSERT INTO field_tag VALUES (39,129,10);
INSERT INTO field_tag VALUES (39,130,10);
INSERT INTO field_tag VALUES (39,1,10);
INSERT INTO field_tag VALUES (39,2,10);
-- misc authority fields
INSERT INTO field_tag VALUES (39,216,10);
INSERT INTO field_tag VALUES (39,217,10);
INSERT INTO field_tag VALUES (39,218,10);
INSERT INTO field_tag VALUES (39,219,10);
INSERT INTO field_tag VALUES (39,220,10);
INSERT INTO field_tag VALUES (39,221,10);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (1,'HTML brief','hb', 'HTML brief output format, used for search results pages.', 'text/html', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (2,'HTML detailed','hd', 'HTML detailed output format, used for Detailed record pages.', 'text/html', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (3,'MARC','hm', 'HTML MARC.', 'text/html', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (4,'Dublin Core','xd', 'XML Dublin Core.', 'text/xml', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (5,'MARCXML','xm', 'XML MARC.', 'text/xml', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (6,'portfolio','hp', 'HTML portfolio-style output format for photos.', 'text/html', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (7,'photo captions only','hc', 'HTML caption-only output format for photos.', 'text/html', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (8,'BibTeX','hx', 'BibTeX.', 'text/html', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (9,'EndNote','xe', 'XML EndNote.', 'text/xml', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (10,'NLM','xn', 'XML NLM.', 'text/xml', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (11,'Excel','excel', 'Excel csv output', 'application/ms-excel', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (12,'HTML similarity','hs', 'Very short HTML output for similarity box (<i>people also viewed..</i>).', 'text/html', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (13,'RSS','xr', 'RSS.', 'text/xml', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (14,'OAI DC','xoaidc', 'OAI DC.', 'text/xml', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (15,'File mini-panel', 'hdfile', 'Used to show fulltext files in mini-panel of detailed record pages.', 'text/html', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (16,'Actions mini-panel', 'hdact', 'Used to display actions in mini-panel of detailed record pages.', 'text/html', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (17,'References tab', 'hdref', 'Display record references in References tab.', 'text/html', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (18,'HTML citesummary','hcs', 'HTML cite summary format, used for search results pages.', 'text/html', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (19,'RefWorks','xw', 'RefWorks.', 'text/xml', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (20,'MODS', 'xo', 'Metadata Object Description Schema', 'application/xml', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (21,'HTML author claiming', 'ha', 'Very brief HTML output format for author/paper claiming facility.', 'text/html', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (22,'Podcast', 'xp', 'Sample format suitable for multimedia feeds, such as podcasts', 'application/rss+xml', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (23,'WebAuthorProfile affiliations helper','wapaff', 'cPickled dicts', 'text', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (24,'EndNote (8-X)','xe8x', 'XML EndNote (8-X).', 'text/xml', 1);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (25,'HTML citesummary extended','hcs2', 'HTML cite summary format, including self-citations counts.', 'text/html', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (26,'DataCite','dcite', 'DataCite XML format.', 'text/xml', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (27,'Mobile brief','mobb', 'Mobile brief format.', 'text/html', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (28,'Mobile detailed','mobd', 'Mobile detailed format.', 'text/html', 0);
INSERT INTO format (id,name,code,description,content_type,visibility) VALUES (29,'WebAuthorProfile data helper','wapdat', 'cPickled dicts', 'text', 0);
INSERT INTO tag VALUES (1,'first author name','100__a');
INSERT INTO tag VALUES (2,'additional author name','700__a');
INSERT INTO tag VALUES (3,'main title','245__%');
INSERT INTO tag VALUES (4,'additional title','246__%');
INSERT INTO tag VALUES (5,'abstract','520__%');
INSERT INTO tag VALUES (6,'keyword','6531_a');
INSERT INTO tag VALUES (7,'primary report number','037__a');
INSERT INTO tag VALUES (8,'additional report number','088__a');
INSERT INTO tag VALUES (9,'added report number','909C0r');
INSERT INTO tag VALUES (10,'reference','999C5%');
INSERT INTO tag VALUES (11,'collection identifier','980__%');
INSERT INTO tag VALUES (12,'main subject','65017a');
INSERT INTO tag VALUES (13,'additional subject','65027a');
INSERT INTO tag VALUES (14,'division','909C0p');
INSERT INTO tag VALUES (15,'year','909C0y');
INSERT INTO tag VALUES (16,'00x','00%');
INSERT INTO tag VALUES (17,'01x','01%');
INSERT INTO tag VALUES (18,'02x','02%');
INSERT INTO tag VALUES (19,'03x','03%');
INSERT INTO tag VALUES (20,'lang','04%');
INSERT INTO tag VALUES (21,'05x','05%');
INSERT INTO tag VALUES (22,'06x','06%');
INSERT INTO tag VALUES (23,'07x','07%');
INSERT INTO tag VALUES (24,'08x','08%');
INSERT INTO tag VALUES (25,'09x','09%');
INSERT INTO tag VALUES (26,'10x','10%');
INSERT INTO tag VALUES (27,'11x','11%');
INSERT INTO tag VALUES (28,'12x','12%');
INSERT INTO tag VALUES (29,'13x','13%');
INSERT INTO tag VALUES (30,'14x','14%');
INSERT INTO tag VALUES (31,'15x','15%');
INSERT INTO tag VALUES (32,'16x','16%');
INSERT INTO tag VALUES (33,'17x','17%');
INSERT INTO tag VALUES (34,'18x','18%');
INSERT INTO tag VALUES (35,'19x','19%');
INSERT INTO tag VALUES (36,'20x','20%');
INSERT INTO tag VALUES (37,'21x','21%');
INSERT INTO tag VALUES (38,'22x','22%');
INSERT INTO tag VALUES (39,'23x','23%');
INSERT INTO tag VALUES (40,'24x','24%');
INSERT INTO tag VALUES (41,'25x','25%');
INSERT INTO tag VALUES (42,'internal','26%');
INSERT INTO tag VALUES (43,'27x','27%');
INSERT INTO tag VALUES (44,'28x','28%');
INSERT INTO tag VALUES (45,'29x','29%');
INSERT INTO tag VALUES (46,'pages','30%');
INSERT INTO tag VALUES (47,'31x','31%');
INSERT INTO tag VALUES (48,'32x','32%');
INSERT INTO tag VALUES (49,'33x','33%');
INSERT INTO tag VALUES (50,'34x','34%');
INSERT INTO tag VALUES (51,'35x','35%');
INSERT INTO tag VALUES (52,'36x','36%');
INSERT INTO tag VALUES (53,'37x','37%');
INSERT INTO tag VALUES (54,'38x','38%');
INSERT INTO tag VALUES (55,'39x','39%');
INSERT INTO tag VALUES (56,'40x','40%');
INSERT INTO tag VALUES (57,'41x','41%');
INSERT INTO tag VALUES (58,'42x','42%');
INSERT INTO tag VALUES (59,'43x','43%');
INSERT INTO tag VALUES (60,'44x','44%');
INSERT INTO tag VALUES (61,'45x','45%');
INSERT INTO tag VALUES (62,'46x','46%');
INSERT INTO tag VALUES (63,'47x','47%');
INSERT INTO tag VALUES (64,'48x','48%');
INSERT INTO tag VALUES (65,'series','49%');
INSERT INTO tag VALUES (66,'50x','50%');
INSERT INTO tag VALUES (67,'51x','51%');
INSERT INTO tag VALUES (68,'52x','52%');
INSERT INTO tag VALUES (69,'53x','53%');
INSERT INTO tag VALUES (70,'54x','54%');
INSERT INTO tag VALUES (71,'55x','55%');
INSERT INTO tag VALUES (72,'56x','56%');
INSERT INTO tag VALUES (73,'57x','57%');
INSERT INTO tag VALUES (74,'58x','58%');
INSERT INTO tag VALUES (75,'summary','59%');
INSERT INTO tag VALUES (76,'60x','60%');
INSERT INTO tag VALUES (77,'61x','61%');
INSERT INTO tag VALUES (78,'62x','62%');
INSERT INTO tag VALUES (79,'63x','63%');
INSERT INTO tag VALUES (80,'64x','64%');
INSERT INTO tag VALUES (81,'65x','65%');
INSERT INTO tag VALUES (82,'66x','66%');
INSERT INTO tag VALUES (83,'67x','67%');
INSERT INTO tag VALUES (84,'68x','68%');
INSERT INTO tag VALUES (85,'subject','69%');
INSERT INTO tag VALUES (86,'70x','70%');
INSERT INTO tag VALUES (87,'71x','71%');
INSERT INTO tag VALUES (88,'author-ad','72%');
INSERT INTO tag VALUES (89,'73x','73%');
INSERT INTO tag VALUES (90,'74x','74%');
INSERT INTO tag VALUES (91,'75x','75%');
INSERT INTO tag VALUES (92,'76x','76%');
INSERT INTO tag VALUES (93,'77x','77%');
INSERT INTO tag VALUES (94,'78x','78%');
INSERT INTO tag VALUES (95,'79x','79%');
INSERT INTO tag VALUES (96,'80x','80%');
INSERT INTO tag VALUES (97,'81x','81%');
INSERT INTO tag VALUES (98,'82x','82%');
INSERT INTO tag VALUES (99,'83x','83%');
INSERT INTO tag VALUES (100,'84x','84%');
INSERT INTO tag VALUES (101,'electr','85%');
INSERT INTO tag VALUES (102,'86x','86%');
INSERT INTO tag VALUES (103,'87x','87%');
INSERT INTO tag VALUES (104,'88x','88%');
INSERT INTO tag VALUES (105,'89x','89%');
INSERT INTO tag VALUES (106,'publication','90%');
INSERT INTO tag VALUES (107,'pub-conf-cit','91%');
INSERT INTO tag VALUES (108,'92x','92%');
INSERT INTO tag VALUES (109,'93x','93%');
INSERT INTO tag VALUES (110,'94x','94%');
INSERT INTO tag VALUES (111,'95x','95%');
INSERT INTO tag VALUES (112,'catinfo','96%');
INSERT INTO tag VALUES (113,'97x','97%');
INSERT INTO tag VALUES (114,'98x','98%');
INSERT INTO tag VALUES (115,'url','8564_u');
INSERT INTO tag VALUES (116,'experiment','909C0e');
INSERT INTO tag VALUES (117,'record ID','001');
INSERT INTO tag VALUES (118,'isbn','020__a');
INSERT INTO tag VALUES (119,'issn','022__a');
INSERT INTO tag VALUES (120,'coden','030__a');
-INSERT INTO tag VALUES (121,'doi','909C4a');
+INSERT INTO tag VALUES (121,'journal doi','909C4a');
INSERT INTO tag VALUES (122,'850x','850%');
INSERT INTO tag VALUES (123,'851x','851%');
INSERT INTO tag VALUES (124,'852x','852%');
INSERT INTO tag VALUES (125,'853x','853%');
INSERT INTO tag VALUES (126,'854x','854%');
INSERT INTO tag VALUES (127,'855x','855%');
INSERT INTO tag VALUES (128,'857x','857%');
INSERT INTO tag VALUES (129,'858x','858%');
INSERT INTO tag VALUES (130,'859x','859%');
INSERT INTO tag VALUES (131,'journal','909C4%');
INSERT INTO tag VALUES (132,'collaboration','710__g');
INSERT INTO tag VALUES (133,'first author affiliation','100__u');
INSERT INTO tag VALUES (134,'additional author affiliation','700__u');
INSERT INTO tag VALUES (135,'caption','8564_y');
INSERT INTO tag VALUES (136,'journal page','909C4c');
INSERT INTO tag VALUES (137,'journal title','909C4p');
INSERT INTO tag VALUES (138,'journal volume','909C4v');
INSERT INTO tag VALUES (139,'journal year','909C4y');
INSERT INTO tag VALUES (140,'comment','500__a');
INSERT INTO tag VALUES (141,'title','245__a');
INSERT INTO tag VALUES (142,'main abstract','245__a');
INSERT INTO tag VALUES (143,'internal notes','595__a');
INSERT INTO tag VALUES (144,'other relationship entry', '787%');
-- INSERT INTO tag VALUES (145,'authority: main personal name','100__a'); -- already exists under a different name ('first author name')
INSERT INTO tag VALUES (146,'authority: alternative personal name','400__a');
-- INSERT INTO tag VALUES (147,'authority: personal name from other record','500__a'); -- already exists under a different name ('comment')
INSERT INTO tag VALUES (148,'authority: organization main name','110__a');
INSERT INTO tag VALUES (149,'organization alternative name','410__a');
INSERT INTO tag VALUES (150,'organization main from other record','510__a');
INSERT INTO tag VALUES (151,'authority: uniform title','130__a');
INSERT INTO tag VALUES (152,'authority: uniform title alternatives','430__a');
INSERT INTO tag VALUES (153,'authority: uniform title from other record','530__a');
INSERT INTO tag VALUES (154,'authority: subject from other record','150__a');
INSERT INTO tag VALUES (155,'authority: subject alternative name','450__a');
INSERT INTO tag VALUES (156,'authority: subject main name','550__a');
-- tags for misc index
INSERT INTO tag VALUES (157,'031x','031%');
INSERT INTO tag VALUES (158,'032x','032%');
INSERT INTO tag VALUES (159,'033x','033%');
INSERT INTO tag VALUES (160,'034x','034%');
INSERT INTO tag VALUES (161,'035x','035%');
INSERT INTO tag VALUES (162,'036x','036%');
INSERT INTO tag VALUES (163,'037x','037%');
INSERT INTO tag VALUES (164,'038x','038%');
INSERT INTO tag VALUES (165,'080x','080%');
INSERT INTO tag VALUES (166,'082x','082%');
INSERT INTO tag VALUES (167,'083x','083%');
INSERT INTO tag VALUES (168,'084x','084%');
INSERT INTO tag VALUES (169,'085x','085%');
INSERT INTO tag VALUES (170,'086x','086%');
INSERT INTO tag VALUES (171,'240x','240%');
INSERT INTO tag VALUES (172,'242x','242%');
INSERT INTO tag VALUES (173,'243x','243%');
INSERT INTO tag VALUES (174,'244x','244%');
INSERT INTO tag VALUES (175,'247x','247%');
INSERT INTO tag VALUES (176,'521x','521%');
INSERT INTO tag VALUES (177,'522x','522%');
INSERT INTO tag VALUES (178,'524x','524%');
INSERT INTO tag VALUES (179,'525x','525%');
INSERT INTO tag VALUES (180,'526x','526%');
INSERT INTO tag VALUES (181,'650x','650%');
INSERT INTO tag VALUES (182,'651x','651%');
INSERT INTO tag VALUES (183,'6531_v','6531_v');
INSERT INTO tag VALUES (184,'6531_y','6531_y');
INSERT INTO tag VALUES (185,'6531_9','6531_9');
INSERT INTO tag VALUES (186,'654x','654%');
INSERT INTO tag VALUES (187,'655x','655%');
INSERT INTO tag VALUES (188,'656x','656%');
INSERT INTO tag VALUES (189,'657x','657%');
INSERT INTO tag VALUES (190,'658x','658%');
INSERT INTO tag VALUES (191,'711x','711%');
INSERT INTO tag VALUES (192,'900x','900%');
INSERT INTO tag VALUES (193,'901x','901%');
INSERT INTO tag VALUES (194,'902x','902%');
INSERT INTO tag VALUES (195,'903x','903%');
INSERT INTO tag VALUES (196,'904x','904%');
INSERT INTO tag VALUES (197,'905x','905%');
INSERT INTO tag VALUES (198,'906x','906%');
INSERT INTO tag VALUES (199,'907x','907%');
INSERT INTO tag VALUES (200,'908x','908%');
INSERT INTO tag VALUES (201,'909C1x','909C1%');
INSERT INTO tag VALUES (202,'909C5x','909C5%');
INSERT INTO tag VALUES (203,'909CSx','909CS%');
INSERT INTO tag VALUES (204,'909COx','909CO%');
INSERT INTO tag VALUES (205,'909CKx','909CK%');
INSERT INTO tag VALUES (206,'909CPx','909CP%');
INSERT INTO tag VALUES (207,'981x','981%');
INSERT INTO tag VALUES (208,'982x','982%');
INSERT INTO tag VALUES (209,'983x','983%');
INSERT INTO tag VALUES (210,'984x','984%');
INSERT INTO tag VALUES (211,'985x','985%');
INSERT INTO tag VALUES (212,'986x','986%');
INSERT INTO tag VALUES (213,'987x','987%');
INSERT INTO tag VALUES (214,'988x','988%');
INSERT INTO tag VALUES (215,'989x','989%');
-- authority controled tags
INSERT INTO tag VALUES (216,'author control','100__0');
INSERT INTO tag VALUES (217,'institute control','110__0');
INSERT INTO tag VALUES (218,'journal control','130__0');
INSERT INTO tag VALUES (219,'subject control','150__0');
INSERT INTO tag VALUES (220,'additional institute control', '260__0');
INSERT INTO tag VALUES (221,'additional author control', '700__0');
INSERT INTO idxINDEX VALUES (1,'global','This index contains words/phrases from global fields.','0000-00-00 00:00:00', '', 'native', 'INDEX-SYNONYM-TITLE,exact','No','No','No','BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (2,'collection','This index contains words/phrases from collection identifiers fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (3,'abstract','This index contains words/phrases from abstract fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (4,'author','This index contains fuzzy words/phrases from author fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexAuthorTokenizer');
INSERT INTO idxINDEX VALUES (5,'keyword','This index contains words/phrases from keyword fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (6,'reference','This index contains words/phrases from references fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (7,'reportnumber','This index contains words/phrases from report numbers fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (8,'title','This index contains words/phrases from title fields.','0000-00-00 00:00:00', '', 'native','INDEX-SYNONYM-TITLE,exact','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (9,'fulltext','This index contains words/phrases from fulltext fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexFulltextTokenizer');
INSERT INTO idxINDEX VALUES (10,'year','This index contains words/phrases from year fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexYearTokenizer');
INSERT INTO idxINDEX VALUES (11,'journal','This index contains words/phrases from journal publication information fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexJournalTokenizer');
INSERT INTO idxINDEX VALUES (12,'collaboration','This index contains words/phrases from collaboration name fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (13,'affiliation','This index contains words/phrases from affiliation fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (14,'exactauthor','This index contains exact words/phrases from author fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexExactAuthorTokenizer');
INSERT INTO idxINDEX VALUES (15,'caption','This index contains exact words/phrases from figure captions.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (16,'firstauthor','This index contains fuzzy words/phrases from first author field.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexAuthorTokenizer');
INSERT INTO idxINDEX VALUES (17,'exactfirstauthor','This index contains exact words/phrases from first author field.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexExactAuthorTokenizer');
INSERT INTO idxINDEX VALUES (18,'authorcount','This index contains number of authors of the record.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexAuthorCountTokenizer');
INSERT INTO idxINDEX VALUES (19,'exacttitle','This index contains exact words/phrases from title fields.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (20,'authorityauthor','This index contains words/phrases from author authority records.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexAuthorTokenizer');
INSERT INTO idxINDEX VALUES (21,'authorityinstitute','This index contains words/phrases from institute authority records.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (22,'authorityjournal','This index contains words/phrases from journal authority records.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (23,'authoritysubject','This index contains words/phrases from subject authority records.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexDefaultTokenizer');
INSERT INTO idxINDEX VALUES (24,'itemcount','This index contains number of copies of items in the library.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexItemCountTokenizer');
INSERT INTO idxINDEX VALUES (25,'filetype','This index contains extensions of files connected to records.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexFiletypeTokenizer');
INSERT INTO idxINDEX VALUES (26,'miscellaneous','This index contains words/phrases from miscellaneous fields','0000-00-00 00:00:00', '', 'native','','No','No','No', 'BibIndexDefaultTokenizer');
+INSERT INTO idxINDEX VALUES (27,'doi','This index contains words/phrases from doi fields','0000-00-00 00:00:00', '', 'native','','No','No','No', 'BibIndexDOITokenizer');
+INSERT INTO idxINDEX VALUES (28,'filename','This index contains file names of files connected to records.','0000-00-00 00:00:00', '', 'native', '','No','No','No', 'BibIndexFilenameTokenizer');
+
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (1,1);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (2,10);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (3,4);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (4,3);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (5,5);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (6,8);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (7,6);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (8,2);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (9,9);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (10,12);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (11,19);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (12,20);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (13,21);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (14,22);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (15,27);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (16,28);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (17,29);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (18,30);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (19,32);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (20,33);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (21,34);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (22,35);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (23,36);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (24,37);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (25,38);
INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (26,39);
+INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (27,18);
+INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (28,43);
INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 2);
INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 3);
INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 5);
INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 7);
INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 8);
INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 10);
INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 11);
INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 12);
INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 13);
INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 19);
INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 26);
+INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, 27);
INSERT INTO sbmACTION VALUES ('Submit New Record','SBI','running','1998-08-17','2001-08-08','','Submit New Record');
INSERT INTO sbmACTION VALUES ('Modify Record','MBI','modify','1998-08-17','2001-11-07','','Modify Record');
INSERT INTO sbmACTION VALUES ('Submit New File','SRV','revise','0000-00-00','2001-11-07','','Submit New File');
INSERT INTO sbmACTION VALUES ('Approve Record','APP','approve','2001-11-08','2002-06-11','','Approve Record');
INSERT INTO sbmALLFUNCDESCR VALUES ('Ask_For_Record_Details_Confirmation','');
INSERT INTO sbmALLFUNCDESCR VALUES ('CaseEDS','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Create_Modify_Interface',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Create_Recid',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Finish_Submission','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Get_Info','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Get_Recid', 'This function gets the recid for a document with a given report-number (as stored in the global variable rn).');
INSERT INTO sbmALLFUNCDESCR VALUES ('Get_Report_Number',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Get_Sysno',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Insert_Modify_Record','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Insert_Record',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Is_Original_Submitter','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Is_Referee','This function checks whether the logged user is a referee for the current document');
INSERT INTO sbmALLFUNCDESCR VALUES ('Mail_Approval_Request_to_Referee',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Mail_Approval_Withdrawn_to_Referee',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Mail_Submitter',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Make_Modify_Record',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Make_Record','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Move_From_Pending','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Move_to_Done',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Move_to_Pending',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Print_Success','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Print_Success_Approval_Request',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Print_Success_APP','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Print_Success_DEL','Prepare a message for the user informing them that their record was successfully deleted.');
INSERT INTO sbmALLFUNCDESCR VALUES ('Print_Success_MBI',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Print_Success_SRV',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Register_Approval_Request',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Register_Referee_Decision',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Withdraw_Approval_Request',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Report_Number_Generation',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Second_Report_Number_Generation','Generate a secondary report number for a document.');
INSERT INTO sbmALLFUNCDESCR VALUES ('Send_Approval_Request',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Send_APP_Mail','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Send_Delete_Mail','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Send_Modify_Mail',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Send_SRV_Mail',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Set_Embargo','Set an embargo on all the documents of a given record.');
INSERT INTO sbmALLFUNCDESCR VALUES ('Stamp_Replace_Single_File_Approval','Stamp a single file when a document is approved.');
INSERT INTO sbmALLFUNCDESCR VALUES ('Stamp_Uploaded_Files','Stamp some of the files that were uploaded during a submission.');
INSERT INTO sbmALLFUNCDESCR VALUES ('Test_Status','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Update_Approval_DB',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('User_is_Record_Owner_or_Curator','Check if user is owner or special editor of a record');
INSERT INTO sbmALLFUNCDESCR VALUES ('Move_Files_to_Storage','Attach files received from chosen file input element(s)');
INSERT INTO sbmALLFUNCDESCR VALUES ('Move_Revised_Files_to_Storage','Revise files initially uploaded with "Move_Files_to_Storage"');
INSERT INTO sbmALLFUNCDESCR VALUES ('Make_Dummy_MARC_XML_Record','');
INSERT INTO sbmALLFUNCDESCR VALUES ('Move_CKEditor_Files_to_Storage','Transfer files attached to the record with the CKEditor');
INSERT INTO sbmALLFUNCDESCR VALUES ('Create_Upload_Files_Interface','Display generic interface to add/revise/delete files. To be used before function "Move_Uploaded_Files_to_Storage"');
INSERT INTO sbmALLFUNCDESCR VALUES ('Move_Uploaded_Files_to_Storage','Attach files uploaded with "Create_Upload_Files_Interface"');
INSERT INTO sbmALLFUNCDESCR VALUES ('Move_Photos_to_Storage','Attach/edit the pictures uploaded with the "create_photos_manager_interface()" function');
INSERT INTO sbmALLFUNCDESCR VALUES ('Link_Records','Link two records toghether via MARC');
INSERT INTO sbmALLFUNCDESCR VALUES ('Video_Processing',NULL);
INSERT INTO sbmALLFUNCDESCR VALUES ('Set_RN_From_Sysno', 'Set the value of global rn variable to the report number identified by sysno (recid)');
INSERT INTO sbmALLFUNCDESCR VALUES ('Notify_URL','Access URL, possibly to post content');
INSERT INTO sbmALLFUNCDESCR VALUES ('Run_PlotExtractor','Run PlotExtractor on the current record');
INSERT INTO sbmFIELDDESC VALUES ('Upload_Photos',NULL,'','R',NULL,NULL,NULL,NULL,NULL,'\"\"\"\r\nThis is an example of element that creates a photos upload interface.\r\nClone it, customize it and integrate it into your submission. Then add function \r\n\'Move_Photos_to_Storage\' to your submission functions list, in order for files \r\nuploaded with this interface to be attached to the record. More information in \r\nthe WebSubmit admin guide.\r\n\"\"\"\r\n\r\nfrom invenio.legacy.websubmit.functions.Shared_Functions import ParamFromFile\r\nfrom invenio.websubmit_functions.Move_Photos_to_Storage import \\\r\n read_param_file, \\\r\n create_photos_manager_interface, \\\r\n get_session_id\r\n\r\n# Retrieve session id\r\ntry:\r\n # User info is defined only in MBI/MPI actions...\r\n session_id = get_session_id(None, uid, user_info) \r\nexcept:\r\n session_id = get_session_id(req, uid, {})\r\n\r\n# Retrieve context\r\nindir = curdir.split(\'/\')[-3]\r\ndoctype = curdir.split(\'/\')[-2]\r\naccess = curdir.split(\'/\')[-1]\r\n\r\n# Get the record ID, if any\r\nsysno = ParamFromFile(\"%s/%s\" % (curdir,\'SN\')).strip()\r\n\r\n\"\"\"\r\nModify below the configuration of the photos manager interface.\r\nNote: `can_reorder_photos\' parameter is not yet fully taken into consideration\r\n\r\nDocumentation of the function is available at <http://localhost/admin/websubmit/websubmitadmin.py/functionedit?funcname=Move_Photos_to_Storage>\r\n\"\"\"\r\ntext += create_photos_manager_interface(sysno, session_id, uid,\r\n doctype, indir, curdir, access,\r\n can_delete_photos=True,\r\n can_reorder_photos=True,\r\n can_upload_photos=True,\r\n editor_width=700,\r\n editor_height=400,\r\n initial_slider_value=100,\r\n max_slider_value=200,\r\n min_slider_value=80)','0000-00-00','0000-00-00',NULL,NULL,0);
INSERT INTO sbmCHECKS VALUES ('AUCheck','function AUCheck(txt) {\r\n var res=1;\r\n tmp=txt.indexOf(\"\\015\");\r\n while (tmp != -1) {\r\n left=txt.substring(0,tmp);\r\n right=txt.substring(tmp+2,txt.length);\r\n txt=left + \"\\012\" + right;\r\n tmp=txt.indexOf(\"\\015\");\r\n }\r\n tmp=txt.indexOf(\"\\012\");\r\n if (tmp==-1){\r\n line=txt;\r\n txt=\'\';}\r\n else{\r\n line=txt.substring(0,tmp);\r\n txt=txt.substring(tmp+1,txt.length);}\r\n while (line != \"\"){\r\n coma=line.indexOf(\",\");\r\n left=line.substring(0,coma);\r\n right=line.substring(coma+1,line.length);\r\n coma2=right.indexOf(\",\");\r\n space=right.indexOf(\" \");\r\n if ((coma==-1)||(left==\"\")||(right==\"\")||(space!=0)||(coma2!=-1)){\r\n res=0;\r\n error_log=line;\r\n }\r\n tmp=txt.indexOf(\"\\012\");\r\n if (tmp==-1){\r\n line=txt;\r\n txt=\'\';}\r\n else{\r\n line=txt.substring(0,tmp-1);\r\n txt=txt.substring(tmp+1,txt.length);}\r\n }\r\n if (res == 0){\r\n alert(\"This author name cannot be managed \\: \\012\\012\" + error_log + \" \\012\\012It is not in the required format!\\012Put one author per line and a comma (,) between the name and the firstname initial letters. \\012The name is going first, followed by the firstname initial letters.\\012Do not forget the whitespace after the comma!!!\\012\\012Example \\: Put\\012\\012Le Meur, J Y \\012Baron, T \\012\\012for\\012\\012Le Meur Jean-Yves & Baron Thomas.\");\r\n return 0;\r\n } \r\n return 1; \r\n}','1998-08-18','0000-00-00','','');
INSERT INTO sbmCHECKS VALUES ('DatCheckNew','function DatCheckNew(txt) {\r\n var res=1;\r\n if (txt.length != 10){res=0;}\r\n if (txt.indexOf(\"/\") != 2){res=0;}\r\n if (txt.lastIndexOf(\"/\") != 5){res=0;}\r\n tmp=parseInt(txt.substring(0,2),10);\r\n if ((tmp > 31)||(tmp < 1)||(isNaN(tmp))){res=0;}\r\n tmp=parseInt(txt.substring(3,5),10);\r\n if ((tmp > 12)||(tmp < 1)||(isNaN(tmp))){res=0;}\r\n tmp=parseInt(txt.substring(6,10),10);\r\n if ((tmp < 1)||(isNaN(tmp))){res=0;}\r\n if (txt.length == 0){res=1;}\r\n if (res == 0){\r\n alert(\"Please enter a correct Date \\012Format: dd/mm/yyyy\");\r\n return 0;\r\n }\r\n return 1; \r\n}','0000-00-00','0000-00-00','','');
INSERT INTO sbmFIELDDESC VALUES ('Upload_Files',NULL,'','R',NULL,NULL,NULL,NULL,NULL,'\"\"\"\r\nThis is an example of element that creates a file upload interface.\r\nClone it, customize it and integrate it into your submission. Then add function \r\n\'Move_Uploaded_Files_to_Storage\' to your submission functions list, in order for files \r\nuploaded with this interface to be attached to the record. More information in \r\nthe WebSubmit admin guide.\r\n\"\"\"\r\nimport os\r\nfrom invenio.legacy.bibdocfile.managedocfiles import create_file_upload_interface\r\nfrom invenio.legacy.websubmit.functions.Shared_Functions import ParamFromFile\r\n\r\nindir = ParamFromFile(os.path.join(curdir, \'indir\'))\r\ndoctype = ParamFromFile(os.path.join(curdir, \'doctype\'))\r\naccess = ParamFromFile(os.path.join(curdir, \'access\'))\r\ntry:\r\n sysno = int(ParamFromFile(os.path.join(curdir, \'SN\')).strip())\r\nexcept:\r\n sysno = -1\r\nln = ParamFromFile(os.path.join(curdir, \'ln\'))\r\n\r\n\"\"\"\r\nRun the following to get the list of parameters of function \'create_file_upload_interface\':\r\necho -e \'from invenio.legacy.bibdocfile.managedocfiles import create_file_upload_interface as f\\nprint f.__doc__\' | python\r\n\"\"\"\r\ntext = create_file_upload_interface(recid=sysno,\r\n print_outside_form_tag=False,\r\n include_headers=True,\r\n ln=ln,\r\n doctypes_and_desc=[(\'main\',\'Main document\'),\r\n (\'additional\',\'Figure, schema, etc.\')],\r\n can_revise_doctypes=[\'*\'],\r\n can_describe_doctypes=[\'main\'],\r\n can_delete_doctypes=[\'additional\'],\r\n can_rename_doctypes=[\'main\'],\r\n sbm_indir=indir, sbm_doctype=doctype, sbm_access=access)[1]\r\n','0000-00-00','0000-00-00',NULL,NULL,0);
INSERT INTO sbmFORMATEXTENSION VALUES ('WORD','.doc');
INSERT INTO sbmFORMATEXTENSION VALUES ('PostScript','.ps');
INSERT INTO sbmFORMATEXTENSION VALUES ('PDF','.pdf');
INSERT INTO sbmFORMATEXTENSION VALUES ('JPEG','.jpg');
INSERT INTO sbmFORMATEXTENSION VALUES ('JPEG','.jpeg');
INSERT INTO sbmFORMATEXTENSION VALUES ('GIF','.gif');
INSERT INTO sbmFORMATEXTENSION VALUES ('PPT','.ppt');
INSERT INTO sbmFORMATEXTENSION VALUES ('HTML','.htm');
INSERT INTO sbmFORMATEXTENSION VALUES ('HTML','.html');
INSERT INTO sbmFORMATEXTENSION VALUES ('Latex','.tex');
INSERT INTO sbmFORMATEXTENSION VALUES ('Compressed PostScript','.ps.gz');
INSERT INTO sbmFORMATEXTENSION VALUES ('Tarred Tex (.tar)','.tar');
INSERT INTO sbmFORMATEXTENSION VALUES ('Text','.txt');
INSERT INTO sbmFUNDESC VALUES ('Get_Recid','record_search_pattern');
INSERT INTO sbmFUNDESC VALUES ('Get_Report_Number','edsrn');
INSERT INTO sbmFUNDESC VALUES ('Send_Modify_Mail','addressesMBI');
INSERT INTO sbmFUNDESC VALUES ('Send_Modify_Mail','sourceDoc');
INSERT INTO sbmFUNDESC VALUES ('Register_Approval_Request','categ_file_appreq');
INSERT INTO sbmFUNDESC VALUES ('Register_Approval_Request','categ_rnseek_appreq');
INSERT INTO sbmFUNDESC VALUES ('Register_Approval_Request','note_file_appreq');
INSERT INTO sbmFUNDESC VALUES ('Register_Referee_Decision','decision_file');
INSERT INTO sbmFUNDESC VALUES ('Withdraw_Approval_Request','categ_file_withd');
INSERT INTO sbmFUNDESC VALUES ('Withdraw_Approval_Request','categ_rnseek_withd');
INSERT INTO sbmFUNDESC VALUES ('Report_Number_Generation','edsrn');
INSERT INTO sbmFUNDESC VALUES ('Report_Number_Generation','autorngen');
INSERT INTO sbmFUNDESC VALUES ('Report_Number_Generation','rnin');
INSERT INTO sbmFUNDESC VALUES ('Report_Number_Generation','counterpath');
INSERT INTO sbmFUNDESC VALUES ('Report_Number_Generation','rnformat');
INSERT INTO sbmFUNDESC VALUES ('Report_Number_Generation','yeargen');
INSERT INTO sbmFUNDESC VALUES ('Report_Number_Generation','nblength');
INSERT INTO sbmFUNDESC VALUES ('Report_Number_Generation','initialvalue');
INSERT INTO sbmFUNDESC VALUES ('Mail_Approval_Request_to_Referee','categ_file_appreq');
INSERT INTO sbmFUNDESC VALUES ('Mail_Approval_Request_to_Referee','categ_rnseek_appreq');
INSERT INTO sbmFUNDESC VALUES ('Mail_Approval_Request_to_Referee','edsrn');
INSERT INTO sbmFUNDESC VALUES ('Mail_Approval_Withdrawn_to_Referee','categ_file_withd');
INSERT INTO sbmFUNDESC VALUES ('Mail_Approval_Withdrawn_to_Referee','categ_rnseek_withd');
INSERT INTO sbmFUNDESC VALUES ('Mail_Submitter','authorfile');
INSERT INTO sbmFUNDESC VALUES ('Mail_Submitter','status');
INSERT INTO sbmFUNDESC VALUES ('Send_Approval_Request','authorfile');
INSERT INTO sbmFUNDESC VALUES ('Create_Modify_Interface','fieldnameMBI');
INSERT INTO sbmFUNDESC VALUES ('Send_Modify_Mail','fieldnameMBI');
INSERT INTO sbmFUNDESC VALUES ('Update_Approval_DB','categformatDAM');
INSERT INTO sbmFUNDESC VALUES ('Update_Approval_DB','decision_file');
INSERT INTO sbmFUNDESC VALUES ('Send_SRV_Mail','categformatDAM');
INSERT INTO sbmFUNDESC VALUES ('Send_SRV_Mail','addressesSRV');
INSERT INTO sbmFUNDESC VALUES ('Send_Approval_Request','directory');
INSERT INTO sbmFUNDESC VALUES ('Send_Approval_Request','categformatDAM');
INSERT INTO sbmFUNDESC VALUES ('Send_Approval_Request','addressesDAM');
INSERT INTO sbmFUNDESC VALUES ('Send_Approval_Request','titleFile');
INSERT INTO sbmFUNDESC VALUES ('Send_APP_Mail','edsrn');
INSERT INTO sbmFUNDESC VALUES ('Mail_Submitter','titleFile');
INSERT INTO sbmFUNDESC VALUES ('Send_Modify_Mail','emailFile');
INSERT INTO sbmFUNDESC VALUES ('Get_Info','authorFile');
INSERT INTO sbmFUNDESC VALUES ('Get_Info','emailFile');
INSERT INTO sbmFUNDESC VALUES ('Get_Info','titleFile');
INSERT INTO sbmFUNDESC VALUES ('Make_Modify_Record','modifyTemplate');
INSERT INTO sbmFUNDESC VALUES ('Send_APP_Mail','addressesAPP');
INSERT INTO sbmFUNDESC VALUES ('Send_APP_Mail','categformatAPP');
INSERT INTO sbmFUNDESC VALUES ('Send_APP_Mail','newrnin');
INSERT INTO sbmFUNDESC VALUES ('Send_APP_Mail','decision_file');
INSERT INTO sbmFUNDESC VALUES ('Send_APP_Mail','comments_file');
INSERT INTO sbmFUNDESC VALUES ('CaseEDS','casevariable');
INSERT INTO sbmFUNDESC VALUES ('CaseEDS','casevalues');
INSERT INTO sbmFUNDESC VALUES ('CaseEDS','casesteps');
INSERT INTO sbmFUNDESC VALUES ('CaseEDS','casedefault');
INSERT INTO sbmFUNDESC VALUES ('Send_SRV_Mail','noteFile');
INSERT INTO sbmFUNDESC VALUES ('Send_SRV_Mail','emailFile');
INSERT INTO sbmFUNDESC VALUES ('Mail_Submitter','emailFile');
INSERT INTO sbmFUNDESC VALUES ('Mail_Submitter','edsrn');
INSERT INTO sbmFUNDESC VALUES ('Mail_Submitter','newrnin');
INSERT INTO sbmFUNDESC VALUES ('Make_Record','sourceTemplate');
INSERT INTO sbmFUNDESC VALUES ('Make_Record','createTemplate');
INSERT INTO sbmFUNDESC VALUES ('Print_Success','edsrn');
INSERT INTO sbmFUNDESC VALUES ('Print_Success','newrnin');
INSERT INTO sbmFUNDESC VALUES ('Print_Success','status');
INSERT INTO sbmFUNDESC VALUES ('Make_Modify_Record','sourceTemplate');
INSERT INTO sbmFUNDESC VALUES ('Move_Files_to_Storage','documenttype');
INSERT INTO sbmFUNDESC VALUES ('Move_Files_to_Storage','iconsize');
INSERT INTO sbmFUNDESC VALUES ('Move_Files_to_Storage','paths_and_suffixes');
INSERT INTO sbmFUNDESC VALUES ('Move_Files_to_Storage','rename');
INSERT INTO sbmFUNDESC VALUES ('Move_Files_to_Storage','paths_and_restrictions');
INSERT INTO sbmFUNDESC VALUES ('Move_Files_to_Storage','paths_and_doctypes');
INSERT INTO sbmFUNDESC VALUES ('Move_Revised_Files_to_Storage','elementNameToDoctype');
INSERT INTO sbmFUNDESC VALUES ('Move_Revised_Files_to_Storage','createIconDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Move_Revised_Files_to_Storage','createRelatedFormats');
INSERT INTO sbmFUNDESC VALUES ('Move_Revised_Files_to_Storage','iconsize');
INSERT INTO sbmFUNDESC VALUES ('Move_Revised_Files_to_Storage','keepPreviousVersionDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Set_Embargo','date_file');
INSERT INTO sbmFUNDESC VALUES ('Set_Embargo','date_format');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Uploaded_Files','files_to_be_stamped');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Uploaded_Files','latex_template');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Uploaded_Files','latex_template_vars');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Uploaded_Files','stamp');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Uploaded_Files','layer');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Uploaded_Files','switch_file');
INSERT INTO sbmFUNDESC VALUES ('Make_Dummy_MARC_XML_Record','dummyrec_source_tpl');
INSERT INTO sbmFUNDESC VALUES ('Make_Dummy_MARC_XML_Record','dummyrec_create_tpl');
INSERT INTO sbmFUNDESC VALUES ('Print_Success_APP','decision_file');
INSERT INTO sbmFUNDESC VALUES ('Print_Success_APP','newrnin');
INSERT INTO sbmFUNDESC VALUES ('Send_Delete_Mail','edsrn');
INSERT INTO sbmFUNDESC VALUES ('Send_Delete_Mail','record_managers');
INSERT INTO sbmFUNDESC VALUES ('Second_Report_Number_Generation','2nd_rn_file');
INSERT INTO sbmFUNDESC VALUES ('Second_Report_Number_Generation','2nd_rn_format');
INSERT INTO sbmFUNDESC VALUES ('Second_Report_Number_Generation','2nd_rn_yeargen');
INSERT INTO sbmFUNDESC VALUES ('Second_Report_Number_Generation','2nd_rncateg_file');
INSERT INTO sbmFUNDESC VALUES ('Second_Report_Number_Generation','2nd_counterpath');
INSERT INTO sbmFUNDESC VALUES ('Second_Report_Number_Generation','2nd_nb_length');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Replace_Single_File_Approval','file_to_be_stamped');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Replace_Single_File_Approval','latex_template');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Replace_Single_File_Approval','latex_template_vars');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Replace_Single_File_Approval','new_file_name');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Replace_Single_File_Approval','stamp');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Replace_Single_File_Approval','layer');
INSERT INTO sbmFUNDESC VALUES ('Stamp_Replace_Single_File_Approval','switch_file');
INSERT INTO sbmFUNDESC VALUES ('Move_CKEditor_Files_to_Storage','input_fields');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','maxsize');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','minsize');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','doctypes');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','restrictions');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canDeleteDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canReviseDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canDescribeDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canCommentDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canKeepDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canAddFormatDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canRestrictDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canRenameDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','canNameNewFiles');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','createRelatedFormats');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','keepDefault');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','showLinks');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','fileLabel');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','filenameLabel');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','descriptionLabel');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','commentLabel');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','restrictionLabel');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','startDoc');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','endDoc');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','defaultFilenameDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','maxFilesDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Move_Uploaded_Files_to_Storage','iconsize');
INSERT INTO sbmFUNDESC VALUES ('Move_Uploaded_Files_to_Storage','createIconDoctypes');
INSERT INTO sbmFUNDESC VALUES ('Move_Uploaded_Files_to_Storage','forceFileRevision');
INSERT INTO sbmFUNDESC VALUES ('Move_Photos_to_Storage','iconsize');
INSERT INTO sbmFUNDESC VALUES ('Move_Photos_to_Storage','iconformat');
INSERT INTO sbmFUNDESC VALUES ('User_is_Record_Owner_or_Curator','curator_role');
INSERT INTO sbmFUNDESC VALUES ('User_is_Record_Owner_or_Curator','curator_flag');
INSERT INTO sbmFUNDESC VALUES ('Link_Records','edsrn');
INSERT INTO sbmFUNDESC VALUES ('Link_Records','edsrn2');
INSERT INTO sbmFUNDESC VALUES ('Link_Records','directRelationship');
INSERT INTO sbmFUNDESC VALUES ('Link_Records','reverseRelationship');
INSERT INTO sbmFUNDESC VALUES ('Link_Records','keep_original_edsrn2');
INSERT INTO sbmFUNDESC VALUES ('Video_Processing','aspect');
INSERT INTO sbmFUNDESC VALUES ('Video_Processing','batch_template');
INSERT INTO sbmFUNDESC VALUES ('Video_Processing','title');
INSERT INTO sbmFUNDESC VALUES ('Set_RN_From_Sysno','edsrn');
INSERT INTO sbmFUNDESC VALUES ('Set_RN_From_Sysno','rep_tags');
INSERT INTO sbmFUNDESC VALUES ('Set_RN_From_Sysno','record_search_pattern');
INSERT INTO sbmFUNDESC VALUES ('Notify_URL','url');
INSERT INTO sbmFUNDESC VALUES ('Notify_URL','data');
INSERT INTO sbmFUNDESC VALUES ('Notify_URL','admin_emails');
INSERT INTO sbmFUNDESC VALUES ('Notify_URL','content_type');
INSERT INTO sbmFUNDESC VALUES ('Notify_URL','attempt_times');
INSERT INTO sbmFUNDESC VALUES ('Notify_URL','attempt_sleeptime');
INSERT INTO sbmFUNDESC VALUES ('Notify_URL','user');
INSERT INTO sbmFUNDESC VALUES ('Run_PlotExtractor','with_docname');
INSERT INTO sbmFUNDESC VALUES ('Run_PlotExtractor','with_doctype');
INSERT INTO sbmFUNDESC VALUES ('Run_PlotExtractor','with_docformat');
INSERT INTO sbmFUNDESC VALUES ('Run_PlotExtractor','extract_plots_switch_file');
INSERT INTO sbmGFILERESULT VALUES ('HTML','HTML document');
INSERT INTO sbmGFILERESULT VALUES ('WORD','data');
INSERT INTO sbmGFILERESULT VALUES ('PDF','PDF document');
INSERT INTO sbmGFILERESULT VALUES ('PostScript','PostScript document');
INSERT INTO sbmGFILERESULT VALUES ('PostScript','data ');
INSERT INTO sbmGFILERESULT VALUES ('PostScript','HP Printer Job Language data');
INSERT INTO sbmGFILERESULT VALUES ('jpg','JPEG image');
INSERT INTO sbmGFILERESULT VALUES ('Compressed PostScript','gzip compressed data');
INSERT INTO sbmGFILERESULT VALUES ('Tarred Tex (.tar)','tar archive');
INSERT INTO sbmGFILERESULT VALUES ('JPEG','JPEG image');
INSERT INTO sbmGFILERESULT VALUES ('GIF','GIF');
INSERT INTO swrREMOTESERVER VALUES (1, 'arXiv', 'arxiv.org', 'CDS_Invenio', 'sword_invenio', 'admin', 'SWORD at arXiv', 'http://arxiv.org/abs', 'https://arxiv.org/sword-app/servicedocument', '', 0);
-- end of file
diff --git a/invenio/legacy/refextract/config.py b/invenio/legacy/refextract/config.py
index 74c8413c7..e3eeee22a 100644
--- a/invenio/legacy/refextract/config.py
+++ b/invenio/legacy/refextract/config.py
@@ -1,127 +1,127 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2005, 2006, 2007, 2008, 2010, 2011 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""RefExtract configuration"""
import pkg_resources
from invenio.config import CFG_VERSION
# pylint: disable=C0301
-CFG_REFEXTRACT_VERSION_NUM = '1.5.39'
+CFG_REFEXTRACT_VERSION_NUM = '1.5.40'
# Version number:
CFG_REFEXTRACT_VERSION = "Invenio/%s refextract/%s" \
% (CFG_VERSION, CFG_REFEXTRACT_VERSION_NUM)
# Module config directory
CFG_CONF_DIR = pkg_resources.resource_filename('invenio.legacy.refextract', 'kbs')
CFG_REFEXTRACT_KBS = {
'journals' : "%s/journal-titles.kb" % CFG_CONF_DIR,
'journals-re' : "%s/journal-titles-re.kb" % CFG_CONF_DIR,
'report-numbers' : "%s/report-numbers.kb" % CFG_CONF_DIR,
'authors' : "%s/authors.kb" % CFG_CONF_DIR,
'collaborations' : "%s/collaborations.kb" % CFG_CONF_DIR,
'books' : "%s/books.kb" % CFG_CONF_DIR,
'conferences' : "%s/conferences.kb" % CFG_CONF_DIR,
'publishers' : "%s/publishers.kb" % CFG_CONF_DIR,
'special-journals': "%s/special-journals.kb" % CFG_CONF_DIR,
}
# Prefix for temp files
CFG_REFEXTRACT_FILENAME = "refextract"
## MARC Fields and subfields used by refextract:
# Reference fields:
CFG_REFEXTRACT_FIELDS = {
'misc': 'm',
'linemarker': 'o',
'doi': 'a',
'reportnumber': 'r',
'journal': 's',
'url': 'u',
'urldesc': 'z',
'author': 'h',
'title': 't',
'isbn': 'i',
'publisher': 'p',
'year': 'y',
'collaboration': 'c',
'recid': '0',
}
CFG_REFEXTRACT_TAG_ID_REFERENCE = "999" # ref field tag
CFG_REFEXTRACT_IND1_REFERENCE = "C" # ref field ind1
CFG_REFEXTRACT_IND2_REFERENCE = "5" # ref field ind2
## refextract statistics fields:
CFG_REFEXTRACT_TAG_ID_EXTRACTION_STATS = "999C6" # ref-stats tag
CFG_REFEXTRACT_SUBFIELD_EXTRACTION_STATS = "a" # ref-stats subfield
CFG_REFEXTRACT_SUBFIELD_EXTRACTION_TIME = "t" # ref-stats time subfield
CFG_REFEXTRACT_SUBFIELD_EXTRACTION_VERSION = "v" # ref-stats version subfield
## Internal tags are used by refextract to mark-up recognised citation
## information.
CFG_REFEXTRACT_MARKER_OPENING_REPORT_NUM = r"<cds.REPORTNUMBER>"
CFG_REFEXTRACT_MARKER_OPENING_TITLE = r"<cds.JOURNAL>"
CFG_REFEXTRACT_MARKER_OPENING_TITLE_IBID = r"<cds.JOURNALibid>"
CFG_REFEXTRACT_MARKER_OPENING_SERIES = r"<cds.SER>"
CFG_REFEXTRACT_MARKER_OPENING_VOLUME = r"<cds.VOL>"
CFG_REFEXTRACT_MARKER_OPENING_YEAR = r"<cds.YR>"
CFG_REFEXTRACT_MARKER_OPENING_PAGE = r"<cds.PG>"
CFG_REFEXTRACT_MARKER_OPENING_QUOTED = r"<cds.QUOTED>"
CFG_REFEXTRACT_MARKER_OPENING_ISBN = r"<cds.ISBN>"
CFG_REFEXTRACT_MARKER_OPENING_PUBLISHER = r"<cds.PUBLISHER>"
CFG_REFEXTRACT_MARKER_OPENING_COLLABORATION = r"<cds.COLLABORATION>"
# These are the "closing tags:
CFG_REFEXTRACT_MARKER_CLOSING_REPORT_NUM = r"</cds.REPORTNUMBER>"
CFG_REFEXTRACT_MARKER_CLOSING_TITLE = r"</cds.JOURNAL>"
CFG_REFEXTRACT_MARKER_CLOSING_TITLE_IBID = r"</cds.JOURNALibid>"
CFG_REFEXTRACT_MARKER_CLOSING_SERIES = r"</cds.SER>"
CFG_REFEXTRACT_MARKER_CLOSING_VOLUME = r"</cds.VOL>"
CFG_REFEXTRACT_MARKER_CLOSING_YEAR = r"</cds.YR>"
CFG_REFEXTRACT_MARKER_CLOSING_PAGE = r"</cds.PG>"
CFG_REFEXTRACT_MARKER_CLOSING_QUOTED = r"</cds.QUOTED>"
CFG_REFEXTRACT_MARKER_CLOSING_ISBN = r"</cds.ISBN>"
CFG_REFEXTRACT_MARKER_CLOSING_PUBLISHER = r"</cds.PUBLISHER>"
CFG_REFEXTRACT_MARKER_CLOSING_COLLABORATION = r"</cds.COLLABORATION>"
## Of the form '</cds.AUTHxxxx>' only
CFG_REFEXTRACT_MARKER_CLOSING_AUTHOR_STND = r"</cds.AUTHstnd>"
CFG_REFEXTRACT_MARKER_CLOSING_AUTHOR_ETAL = r"</cds.AUTHetal>"
CFG_REFEXTRACT_MARKER_CLOSING_AUTHOR_INCL = r"</cds.AUTHincl>"
## The minimum length of a reference's misc text to be deemed insignificant.
## when comparing misc text with semi-colon defined sub-references.
## Values higher than this value reflect meaningful misc text.
## Hence, upon finding a correct semi-colon, but having current misc text
## length less than this value (without other meaningful reference objects:
## report numbers, titles...) then no split will occur.
## (A higher value will increase splitting strictness. i.e. Fewer splits)
CGF_REFEXTRACT_SEMI_COLON_MISC_TEXT_SENSITIVITY = 60
## The length of misc text between two adjacent authors which is
## deemed as insignificant. As such, when misc text of a length less
## than this value is found, then the latter author group is dumped into misc.
## (A higher value will increase splitting strictness. i.e. Fewer splits)
CGF_REFEXTRACT_ADJACENT_AUTH_MISC_SEPARATION = 10
## Maximum number of lines for a citation before it is considered invalid
CFG_REFEXTRACT_MAX_LINES = 25
diff --git a/invenio/legacy/refextract/engine.py b/invenio/legacy/refextract/engine.py
index 428ee67ef..3552bdf39 100644
--- a/invenio/legacy/refextract/engine.py
+++ b/invenio/legacy/refextract/engine.py
@@ -1,1215 +1,1314 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""This is the main body of refextract. It is used to extract references from
fulltext PDF documents.
"""
__revision__ = "$Id$"
import re
import os
import subprocess
from itertools import chain
from .config import (
CFG_REFEXTRACT_MARKER_CLOSING_REPORT_NUM,
CFG_REFEXTRACT_MARKER_CLOSING_AUTHOR_INCL,
CFG_REFEXTRACT_MARKER_CLOSING_AUTHOR_STND,
CFG_REFEXTRACT_MARKER_CLOSING_VOLUME,
CFG_REFEXTRACT_MARKER_CLOSING_YEAR,
CFG_REFEXTRACT_MARKER_CLOSING_PAGE,
CFG_REFEXTRACT_MARKER_CLOSING_TITLE_IBID,
CFG_REFEXTRACT_MARKER_CLOSING_AUTHOR_ETAL,
CFG_REFEXTRACT_MARKER_CLOSING_TITLE,
CFG_REFEXTRACT_MARKER_CLOSING_SERIES)
# make refextract runnable without requiring the full Invenio installation:
from invenio.config import CFG_PATH_GFILE
from .tag import (
tag_reference_line,
sum_2_dictionaries,
identify_and_tag_DOI,
identify_and_tag_URLs,
find_numeration,
extract_series_from_volume)
from .record import build_record, build_references
from invenio.legacy.docextract.pdf import convert_PDF_to_plaintext
from invenio.legacy.docextract.utils import write_message
from .kbs import get_kbs
from .linker import find_referenced_recid
from .regexs import (
get_reference_line_numeration_marker_patterns,
regex_match_list,
re_tagged_citation,
re_numeration_no_ibid_txt,
re_roman_numbers,
re_recognised_numeration_for_title_plus_series)
description = """
Refextract tries to extract the reference section from a full-text document.
Extracted reference lines are processed and any recognised citations are
marked up using MARC XML. Recognises author names, URL's, DOI's, and also
journal titles and report numbers as per the relevant knowledge bases. Results
are output to the standard output stream as default, or instead to an xml file.
"""
# General initiation tasks:
# components relating to the standardisation and
# recognition of citations in reference lines:
def remove_reference_line_marker(line):
"""Trim a reference line's 'marker' from the beginning of the line.
@param line: (string) - the reference line.
@return: (tuple) containing two strings:
+ The reference line's marker (or if there was not one,
a 'space' character.
+ The reference line with it's marker removed from the
beginning.
"""
# Get patterns to identify reference-line marker patterns:
marker_patterns = get_reference_line_numeration_marker_patterns()
line = line.lstrip()
marker_match = regex_match_list(line, marker_patterns)
if marker_match is not None:
# found a marker:
marker_val = marker_match.group(u'mark')
# trim the marker from the start of the line:
line = line[marker_match.end():].lstrip()
else:
marker_val = u" "
return (marker_val, line)
def roman2arabic(num):
"""Convert numbers from roman to arabic
This function expects a string like XXII
and outputs an integer
"""
t = 0
p = 0
for r in num:
n = 10 ** (205558 % ord(r) % 7) % 9995
t += n - 2 * p % n
p = n
return t
## Transformations
def format_volume(citation_elements):
"""format volume number (roman numbers to arabic)
When the volume number is expressed in roman numbers (CXXII),
they are converted to their equivalent in arabic numbers (42)
"""
re_roman = re.compile(re_roman_numbers + u'$', re.UNICODE)
for el in citation_elements:
if el['type'] == 'JOURNAL' and re_roman.match(el['volume']):
el['volume'] = str(roman2arabic(el['volume'].upper()))
return citation_elements
def handle_special_journals(citation_elements, kbs):
"""format special journals (like JHEP) volume number
JHEP needs the volume number prefixed with the year
e.g. JHEP 0301 instead of JHEP 01
"""
for el in citation_elements:
if el['type'] == 'JOURNAL' and el['title'] in kbs['special_journals'] \
and re.match(r'\d{1,2}$', el['volume']):
# Sometimes the page is omitted and the year is written in its place
# We can never be sure but it's very likely that page > 1900 is
# actually a year, so we skip this reference
if el['year'] == '' and re.match(r'(19|20)\d{2}$', el['page']):
el['type'] = 'MISC'
el['misc_txt'] = "%s,%s,%s" \
% (el['title'], el['volume'], el['page'])
el['volume'] = el['year'][-2:] + '%02d' % int(el['volume'])
return citation_elements
def format_report_number(citation_elements):
"""Format report numbers that are missing a dash
e.g. CERN-LCHH2003-01 to CERN-LHCC-2003-01
"""
re_report = re.compile(ur'^(?P<name>[A-Z-]+)(?P<nums>[\d-]+)$', re.UNICODE)
for el in citation_elements:
if el['type'] == 'REPORTNUMBER':
m = re_report.match(el['report_num'])
if m:
name = m.group('name')
if not name.endswith('-'):
el['report_num'] = m.group('name') + '-' + m.group('nums')
return citation_elements
def format_hep(citation_elements):
"""Format hep-th report numbers with a dash
e.g. replaces hep-th-9711200 with hep-th/9711200
"""
prefixes = ('astro-ph-', 'hep-th-', 'hep-ph-', 'hep-ex-', 'hep-lat-',
'math-ph-')
for el in citation_elements:
if el['type'] == 'REPORTNUMBER':
for p in prefixes:
if el['report_num'].startswith(p):
el['report_num'] = el['report_num'][:len(p) - 1] + '/' + \
el['report_num'][len(p):]
return citation_elements
def format_author_ed(citation_elements):
"""Standardise to (ed.) and (eds.)
e.g. Remove extra space in (ed. )
"""
for el in citation_elements:
if el['type'] == 'AUTH':
el['auth_txt'] = el['auth_txt'].replace('(ed. )', '(ed.)')
el['auth_txt'] = el['auth_txt'].replace('(eds. )', '(eds.)')
return citation_elements
def look_for_books(citation_elements, kbs):
"""Look for books in our kb
Create book tags by using the authors and the title to find books
in our knowledge base
"""
- authors = None
title = None
- for el in citation_elements:
- if el['type'] == 'AUTH':
- authors = el
- break
for el in citation_elements:
if el['type'] == 'QUOTED':
title = el
break
- if authors and title:
- if title['title'].upper() in kbs['books']:
- line = kbs['books'][title['title'].upper()]
+ if title:
+ normalized_title = title['title'].upper()
+ if normalized_title in kbs['books']:
+ line = kbs['books'][normalized_title]
el = {'type': 'BOOK',
'misc_txt': '',
'authors': line[0],
'title': line[1],
'year': line[2].strip(';')}
citation_elements.append(el)
citation_elements.remove(title)
return citation_elements
def split_volume_from_journal(citation_elements):
"""Split volume from journal title
We need this because sometimes the volume is attached to the journal title
instead of the volume. In those cases we move it here from the title to the
volume
"""
for el in citation_elements:
if el['type'] == 'JOURNAL' and ';' in el['title']:
el['title'], series = el['title'].rsplit(';', 1)
el['volume'] = series + el['volume']
return citation_elements
def remove_b_for_nucl_phys(citation_elements):
"""Removes b from the volume of some journals
Removes the B from the volume for Nucl.Phys.Proc.Suppl. because in INSPIRE
that journal is handled differently.
"""
for el in citation_elements:
if el['type'] == 'JOURNAL' and el['title'] == 'Nucl.Phys.Proc.Suppl.' \
and 'volume' in el \
and (el['volume'].startswith('b') or el['volume'].startswith('B')):
el['volume'] = el['volume'][1:]
return citation_elements
def mangle_volume(citation_elements):
"""Make sure the volume letter is before the volume number
e.g. transforms 100B to B100
"""
volume_re = re.compile(ur"(\d+)([A-Z])", re.U|re.I)
for el in citation_elements:
if el['type'] == 'JOURNAL':
matches = volume_re.match(el['volume'])
if matches:
el['volume'] = matches.group(2) + matches.group(1)
return citation_elements
def balance_authors(splitted_citations, new_elements):
if not splitted_citations:
return
last_citation = splitted_citations[-1]
current_citation = new_elements
if last_citation[-1]['type'] == 'AUTH' \
and sum([1 for cit in last_citation if cit['type'] == 'AUTH']) > 1:
el = last_citation.pop()
current_citation.insert(0, el)
def associate_recids(citation_elements):
for el in citation_elements:
try:
el['recid'] = find_referenced_recid(el).pop()
except (IndexError, KeyError):
el['recid'] = None
return citation_elements
def associate_recids_catchup(splitted_citations):
for citation_elements in splitted_citations:
associate_recids(citation_elements)
def split_citations(citation_elements):
"""Split a citation line in multiple citations
We handle the case where the author has put 2 citations in the same line
but split with ; or some other method.
"""
splitted_citations = []
new_elements = []
current_recid = None
current_doi = None
def check_ibid(current_elements, trigger_el):
for el in new_elements:
if el['type'] == 'AUTH':
return
# Check for ibid
if trigger_el.get('is_ibid', False):
if splitted_citations:
els = chain(reversed(current_elements),
reversed(splitted_citations[-1]))
else:
els = reversed(current_elements)
for el in els:
if el['type'] == 'AUTH':
new_elements.append(el.copy())
break
def start_new_citation():
"""Start new citation"""
splitted_citations.append(new_elements[:])
del new_elements[:]
for el in citation_elements:
try:
el_recid = el['recid']
except KeyError:
el_recid = None
if current_recid and el_recid and current_recid == el_recid:
# Do not start a new citation
pass
elif current_recid and el_recid and current_recid != el_recid \
or current_doi and el['type'] == 'DOI' and \
current_doi != el['doi_string']:
start_new_citation()
# Some authors may be found in the previous citation
balance_authors(splitted_citations, new_elements)
elif ';' in el['misc_txt']:
misc_txt, el['misc_txt'] = el['misc_txt'].split(';', 1)
if misc_txt:
new_elements.append({'type': 'MISC',
'misc_txt': misc_txt})
start_new_citation()
# In case el['recid'] is None, we want to reset it
# because we are starting a new reference
current_recid = el_recid
while ';' in el['misc_txt']:
misc_txt, el['misc_txt'] = el['misc_txt'].split(';', 1)
if misc_txt:
new_elements.append({'type': 'MISC',
'misc_txt': misc_txt})
start_new_citation()
current_recid = None
if el_recid:
current_recid = el_recid
if el['type'] == 'DOI':
current_doi = el['doi_string']
check_ibid(new_elements, el)
new_elements.append(el)
splitted_citations.append(new_elements)
return [el for el in splitted_citations if not empty_citation(el)]
def empty_citation(citation):
els_to_remove = ('MISC', )
for el in citation:
if el['type'] not in els_to_remove:
return False
if el['misc_txt']:
return False
return True
def valid_citation(citation):
els_to_remove = ('MISC', )
for el in citation:
if el['type'] not in els_to_remove:
return True
return False
def remove_invalid_references(splitted_citations):
def add_misc(el, txt):
if not el.get('misc_txt'):
el['misc_txt'] = txt
else:
el['misc_txt'] += " " + txt
splitted_citations = [citation for citation in splitted_citations
if citation]
# We merge some elements in here which means it only makes sense when
# we have at least 2 elements to merge together
if len(splitted_citations) > 1:
previous_citation = None
for citation in splitted_citations:
if not valid_citation(citation):
# Merge to previous one misc txt
if previous_citation:
citation_to_merge_into = previous_citation
else:
citation_to_merge_into = splitted_citations[1]
for el in citation:
add_misc(citation_to_merge_into[-1], el['misc_txt'])
previous_citation = citation
return [citation for citation in splitted_citations
if valid_citation(citation)]
def merge_invalid_references(splitted_citations):
def add_misc(el, txt):
if not el.get('misc_txt'):
el['misc_txt'] = txt
else:
el['misc_txt'] += " " + txt
splitted_citations = [citation for citation in splitted_citations
if citation]
# We merge some elements in here which means it only makes sense when
# we have at least 2 elements to merge together
if len(splitted_citations) > 1:
previous_citation = None
previous_citation_valid = True
for citation in splitted_citations:
current_citation_valid = valid_citation(citation)
if not current_citation_valid:
# Merge to previous one misc txt
if not previous_citation_valid and not current_citation_valid:
for el in citation:
add_misc(previous_citation[-1], el['misc_txt'])
previous_citation = citation
previous_citation_valid = current_citation_valid
return [citation for citation in splitted_citations
if valid_citation(citation)]
def add_year_elements(splitted_citations):
for citation in splitted_citations:
for el in citation:
if el['type'] == 'YEAR':
continue
year = None
for el in citation:
if el['type'] == 'JOURNAL' or el['type'] == 'BOOK' \
and 'year' in el:
year = el['year']
break
if year:
citation.append({'type': 'YEAR',
'year': year,
'misc_txt': '',
})
return splitted_citations
def look_for_implied_ibids(splitted_citations):
def look_for_journal(els):
for el in els:
if el['type'] == 'JOURNAL':
return True
return False
current_journal = None
for citation in splitted_citations:
if current_journal and not look_for_journal(citation):
for el in citation:
if el['type'] == 'MISC':
numeration = find_numeration(el['misc_txt'])
if numeration:
if not numeration['series']:
numeration['series'] = extract_series_from_volume(current_journal['volume'])
if numeration['series']:
volume = numeration['series'] + numeration['volume']
else:
volume = numeration['volume']
ibid_el = {'type' : 'JOURNAL',
'misc_txt' : '',
'title' : current_journal['title'],
'volume' : volume,
'year' : numeration['year'],
'page' : numeration['page'],
'page_end' : numeration['page_end'],
'is_ibid' : True,
'extra_ibids': []}
citation.append(ibid_el)
el['misc_txt'] = el['misc_txt'][numeration['len']:]
current_journal = None
for el in citation:
if el['type'] == 'JOURNAL':
current_journal = el
return splitted_citations
def remove_duplicated_authors(splitted_citations):
for citation in splitted_citations:
found_author = False
for el in citation:
if el['type'] == 'AUTH':
if found_author:
el['type'] = 'MISC'
el['misc_txt'] = el['misc_txt'] + " " + el['auth_txt']
else:
found_author = True
return splitted_citations
def remove_duplicated_dois(splitted_citations):
for citation in splitted_citations:
found_doi = False
for el in citation[:]:
if el['type'] == 'DOI':
if found_doi:
citation.remove(el)
else:
found_doi = True
return splitted_citations
def add_recid_elements(splitted_citations):
for citation in splitted_citations:
for el in citation:
if el.get('recid', None):
citation.append({'type': 'RECID',
'recid': el['recid'],
'misc_txt': ''})
break
## End of elements transformations
def print_citations(splitted_citations, line_marker):
write_message('* splitted_citations', verbose=9)
write_message(' * line marker %s' % line_marker, verbose=9)
for citation in splitted_citations:
write_message(" * elements", verbose=9)
for el in citation:
write_message(' * %s %s' % (el['type'], repr(el)), verbose=9)
def parse_reference_line(ref_line, kbs, bad_titles_count={}):
"""Parse one reference line
@input a string representing a single reference bullet
@output parsed references (a list of elements objects)
"""
# Strip the 'marker' (e.g. [1]) from this reference line:
line_marker, ref_line = remove_reference_line_marker(ref_line)
# Find DOI sections in citation
ref_line, identified_dois = identify_and_tag_DOI(ref_line)
# Identify and replace URLs in the line:
ref_line, identified_urls = identify_and_tag_URLs(ref_line)
# Tag <cds.JOURNAL>, etc.
tagged_line, bad_titles_count = tag_reference_line(ref_line,
kbs,
bad_titles_count)
# Debug print tagging (authors, titles, volumes, etc.)
write_message('* tags %r' % tagged_line, verbose=9)
# Using the recorded information, create a MARC XML representation
# of the rebuilt line:
# At the same time, get stats of citations found in the reference line
# (titles, urls, etc):
citation_elements, line_marker, counts = \
parse_tagged_reference_line(line_marker,
tagged_line,
identified_dois,
identified_urls)
# Transformations on elements
split_volume_from_journal(citation_elements)
format_volume(citation_elements)
handle_special_journals(citation_elements, kbs)
format_report_number(citation_elements)
format_author_ed(citation_elements)
look_for_books(citation_elements, kbs)
format_hep(citation_elements)
remove_b_for_nucl_phys(citation_elements)
mangle_volume(citation_elements)
associate_recids(citation_elements)
# Split the reference in multiple ones if needed
splitted_citations = split_citations(citation_elements)
+
+ # Look for books in misc field
+ look_for_undetected_books(splitted_citations, kbs)
# Look for implied ibids
look_for_implied_ibids(splitted_citations)
- # Associate recids to the newly added ibids
+ # Associate recids to the newly added ibids/books
associate_recids_catchup(splitted_citations)
# Remove references with only misc text
# splitted_citations = remove_invalid_references(splitted_citations)
# Merge references with only misc text
# splitted_citations = merge_invalid_references(splitted_citations)
# Find year
add_year_elements(splitted_citations)
# Remove duplicate authors
remove_duplicated_authors(splitted_citations)
# Remove duplicate DOIs
remove_duplicated_dois(splitted_citations)
# Add recid elements
add_recid_elements(splitted_citations)
# For debugging puposes
print_citations(splitted_citations, line_marker)
return splitted_citations, line_marker, counts, bad_titles_count
+
+def look_for_undetected_books(splitted_citations, kbs):
+ for citation in splitted_citations:
+ if is_unknown_citation(citation):
+ search_for_book_in_misc(citation, kbs)
+
+
+def search_for_book_in_misc(citation, kbs):
+ """Searches for books in the misc_txt field if the citation is not recognized as anything like a journal, book, etc.
+ """
+ for citation_element in citation:
+ if citation_element['type'] == 'MISC':
+ write_message('* Unknown citation found. Searching for book title in: %s' % citation_element['misc_txt'], verbose=9)
+ for title in kbs['books']:
+ startIndex = find_substring_ignore_special_chars(citation_element['misc_txt'], title)
+ if startIndex != -1:
+ line = kbs['books'][title.upper()]
+ book_year = line[2].strip(';')
+ book_authors = line[0]
+ book_found = False
+ if citation_element['misc_txt'].find(book_year) != -1:
+ # For now consider the citation as valid, we are using
+ # an exact search, we don't need to check the authors
+ # However, the code below will be useful if we decide
+ # to introduce fuzzy matching.
+ book_found = True
+
+ for author in get_possible_author_names(citation):
+ if find_substring_ignore_special_chars(book_authors, author) != -1:
+ book_found = True
+
+ for author in re.findall('[a-zA-Z]{4,}', book_authors):
+ if find_substring_ignore_special_chars(citation_element['misc_txt'], author) != -1:
+ book_found = True
+
+ if book_found is True:
+ write_message('* Book found: %s' % title, verbose=9)
+ book_element = {'type': 'BOOK',
+ 'misc_txt': '',
+ 'authors': book_authors,
+ 'title': line[1],
+ 'year': book_year}
+ citation.append(book_element)
+ citation_element['misc_txt'] = cut_substring_with_special_chars(citation_element['misc_txt'], title, startIndex)
+ return True
+
+ write_message(' * Book not found!', verbose=9)
+ return False
+
+def get_possible_author_names(citation):
+ for citation_element in citation:
+ if citation_element['type'] == 'AUTH':
+ return re.findall('[a-zA-Z]{4,}', citation_element['auth_txt'])
+ return []
+
+def find_substring_ignore_special_chars(s, substr):
+ s = s.upper()
+ substr = substr.upper()
+ clean_s, subs_in_s = re.subn('[^A-Z0-9]', '', s)
+ clean_substr, subs_in_substr = re.subn('[^A-Z0-9]', '', substr)
+ startIndex = clean_s.find(clean_substr)
+ if startIndex != -1:
+ i = 0
+ re_alphanum = re.compile('[A-Z0-9]')
+ for real_index, char in enumerate(s):
+ if re_alphanum.match(char):
+ i += 1
+ if i > startIndex:
+ break
+
+ return real_index
+ else:
+ return -1
+
+def cut_substring_with_special_chars(s, sub, startIndex):
+ counter = 0
+ subPosition = 0
+ s_Upper = s.upper()
+ sub = sub.upper()
+ clean_sub = re.sub('[^A-Z0-9]', '', sub)
+ for char in s_Upper[startIndex:]:
+ if char == clean_sub[subPosition]:
+ subPosition += 1
+ counter += 1
+ #end of substrin reached?
+ if subPosition >= len(clean_sub):
+ #include everything till a space, open bracket or a normal character
+ counter += len(re.split('[ [{(a-zA-Z0-9]',s[startIndex+counter:],1)[0])
+
+ return s[0:startIndex].strip()+ ' ' +s[startIndex+counter:].strip()
+
+def is_unknown_citation(citation):
+ """Checks if the citation got recognized as one of the known types.
+ """
+ knownTypes = ['BOOK', 'JOURNAL', 'DOI', 'ISBN', 'RECID']
+ for citation_element in citation:
+ if citation_element['type'] in knownTypes:
+ return False
+ return True
+
def parse_references_elements(ref_sect, kbs):
"""Passed a complete reference section, process each line and attempt to
## identify and standardise individual citations within the line.
@param ref_sect: (list) of strings - each string in the list is a
reference line.
@param preprint_repnum_search_kb: (dictionary) - keyed by a tuple
containing the line-number of the pattern in the KB and the non-standard
category string. E.g.: (3, 'ASTRO PH'). Value is regexp pattern used to
search for that report-number.
@param preprint_repnum_standardised_categs: (dictionary) - keyed by non-
standard version of institutional report number, value is the
standardised version of that report number.
@param periodical_title_search_kb: (dictionary) - keyed by non-standard
title to search for, value is the compiled regexp pattern used to
search for that title.
@param standardised_periodical_titles: (dictionary) - keyed by non-
standard title to search for, value is the standardised version of that
title.
@param periodical_title_search_keys: (list) - ordered list of non-
standard titles to search for.
@return: (tuple) of 6 components:
( list -> of strings, each string is a MARC XML-ized reference
line.
integer -> number of fields of miscellaneous text found for the
record.
integer -> number of title citations found for the record.
integer -> number of institutional report-number citations found
for the record.
integer -> number of URL citations found for the record.
integer -> number of DOI's found
integer -> number of author groups found
dictionary -> The totals for each 'bad title' found in the reference
section.
)
"""
# a list to contain the processed reference lines:
citations = []
# counters for extraction stats:
counts = {
'misc': 0,
'title': 0,
'reportnum': 0,
'url': 0,
'doi': 0,
'auth_group': 0,
}
# A dictionary to contain the total count of each 'bad title' found
# in the entire reference section:
bad_titles_count = {}
# process references line-by-line:
for ref_line in ref_sect:
citation_elements, line_marker, this_counts, bad_titles_count = \
parse_reference_line(ref_line, kbs, bad_titles_count)
# Accumulate stats
counts = sum_2_dictionaries(counts, this_counts)
citations.append({'elements' : citation_elements,
'line_marker': line_marker})
# Return the list of processed reference lines:
return citations, counts, bad_titles_count
def parse_tagged_reference_line(line_marker,
line,
identified_dois,
identified_urls):
""" Given a single tagged reference line, convert it to its MARC-XML representation.
Try to find all tags and extract their contents and their types into corresponding
dictionary elements. Append each dictionary tag representation onto a list, which
is given to 'build_formatted_xml_citation()' where the correct xml output will be generated.
This method is dumb, with very few heuristics. It simply looks for tags, and makes dictionaries
from the data it finds in a tagged reference line.
@param line_marker: (string) The line marker for this single reference line (e.g. [19])
@param line: (string) The tagged reference line.
@param identified_dois: (list) a list of dois which were found in this line. The ordering of
dois corresponds to the ordering of tags in the line, reading from left to right.
@param identified_urls: (list) a list of urls which were found in this line. The ordering of
urls corresponds to the ordering of tags in the line, reading from left to right.
@param which format to use for references,
roughly "<title> <volume> <page>" or "<title>,<volume>,<page>"
@return xml_line: (string) the MARC-XML representation of the tagged reference line
@return count_*: (integer) the number of * (pieces of info) found in the reference line.
"""
count_misc = count_title = count_reportnum = count_url = count_doi = count_auth_group = 0
processed_line = line
cur_misc_txt = u""
tag_match = re_tagged_citation.search(processed_line)
# contains a list of dictionary entries of previously cited items
citation_elements = []
# the last tag element found when working from left-to-right across the line
identified_citation_element = None
while tag_match is not None:
# While there are tags inside this reference line...
tag_match_start = tag_match.start()
tag_match_end = tag_match.end()
tag_type = tag_match.group(1)
cur_misc_txt += processed_line[0:tag_match_start]
# Catches both standard titles, and ibid's
if tag_type.find("JOURNAL") != -1:
# This tag is an identified journal TITLE. It should be followed
# by VOLUME, YEAR and PAGE tags.
# See if the found title has been tagged as an ibid: <cds.JOURNALibid>
if tag_match.group('ibid'):
is_ibid = True
closing_tag_length = len(CFG_REFEXTRACT_MARKER_CLOSING_TITLE_IBID)
idx_closing_tag = processed_line.find(CFG_REFEXTRACT_MARKER_CLOSING_TITLE_IBID,
tag_match_end)
else:
is_ibid = False
closing_tag_length = len(CFG_REFEXTRACT_MARKER_CLOSING_TITLE)
# extract the title from the line:
idx_closing_tag = processed_line.find(CFG_REFEXTRACT_MARKER_CLOSING_TITLE,
tag_match_end)
if idx_closing_tag == -1:
# no closing TITLE tag found - get rid of the solitary tag
processed_line = processed_line[tag_match_end:]
identified_citation_element = None
else:
# Closing tag was found:
# The title text to be used in the marked-up citation:
title_text = processed_line[tag_match_end:idx_closing_tag]
# Now trim this matched title and its tags from the start of the line:
processed_line = processed_line[idx_closing_tag+closing_tag_length:]
numeration_match = re_recognised_numeration_for_title_plus_series.search(processed_line)
if numeration_match:
# recognised numeration immediately after the title - extract it:
reference_volume = numeration_match.group('vol')
reference_year = numeration_match.group('yr') or ''
reference_page = numeration_match.group('pg')
# This is used on two accounts:
# 1. To get the series char from the title, if no series was found with the numeration
# 2. To always remove any series character from the title match text
# series_from_title = re_series_from_title.search(title_text)
#
if numeration_match.group('series'):
reference_volume = numeration_match.group('series') + reference_volume
# Skip past the matched numeration in the working line:
processed_line = processed_line[numeration_match.end():]
# 'id_ibid' saves whether THIS TITLE is an ibid or not. (True or False)
# 'extra_ibids' are there to hold ibid's without the word 'ibid', which
# come directly after this title
# i.e., they are recognised using title numeration instead of ibid notation
identified_citation_element = {'type' : "JOURNAL",
'misc_txt' : cur_misc_txt,
'title' : title_text,
'volume' : reference_volume,
'year' : reference_year,
'page' : reference_page,
'is_ibid' : is_ibid,
'extra_ibids': []
}
count_title += 1
cur_misc_txt = u""
# Try to find IBID's after this title, on top of previously found titles that were
# denoted with the word 'IBID'. (i.e. look for IBID's without the word 'IBID' by
# looking at extra numeration after this title)
numeration_match = re_numeration_no_ibid_txt.match(processed_line)
while numeration_match is not None:
reference_volume = numeration_match.group('vol')
reference_year = numeration_match.group('yr')
reference_page = numeration_match.group('pg')
if numeration_match.group('series'):
reference_volume = numeration_match.group('series') + reference_volume
# Skip past the matched numeration in the working line:
processed_line = processed_line[numeration_match.end():]
# Takes the just found title text
identified_citation_element['extra_ibids'].append(
{'type' : "JOURNAL",
'misc_txt' : "",
'title' : title_text,
'volume' : reference_volume,
'year' : reference_year,
'page' : reference_page,
})
# Increment the stats counters:
count_title += 1
title_text = ""
reference_volume = ""
reference_year = ""
reference_page = ""
numeration_match = re_numeration_no_ibid_txt.match(processed_line)
else:
# No numeration was recognised after the title. Add the title into a MISC item instead:
cur_misc_txt += "%s" % title_text
identified_citation_element = None
elif tag_type == "REPORTNUMBER":
# This tag is an identified institutional report number:
# extract the institutional report-number from the line:
idx_closing_tag = processed_line.find(CFG_REFEXTRACT_MARKER_CLOSING_REPORT_NUM,
tag_match_end)
# Sanity check - did we find a closing report-number tag?
if idx_closing_tag == -1:
# no closing </cds.REPORTNUMBER> tag found - strip the opening tag and move past this
# recognised reportnumber as it is unreliable:
processed_line = processed_line[tag_match_end:]
identified_citation_element = None
else:
# closing tag was found
report_num = processed_line[tag_match_end:idx_closing_tag]
# now trim this matched institutional report-number
# and its tags from the start of the line:
ending_tag_pos = idx_closing_tag \
+ len(CFG_REFEXTRACT_MARKER_CLOSING_REPORT_NUM)
processed_line = processed_line[ending_tag_pos:]
identified_citation_element = {'type' : "REPORTNUMBER",
'misc_txt' : cur_misc_txt,
'report_num' : report_num}
count_reportnum += 1
cur_misc_txt = u""
elif tag_type == "URL":
# This tag is an identified URL:
# From the "identified_urls" list, get this URL and its
# description string:
url_string = identified_urls[0][0]
url_desc = identified_urls[0][1]
# Now move past this "<cds.URL />"tag in the line:
processed_line = processed_line[tag_match_end:]
# Delete the information for this URL from the start of the list
# of identified URLs:
identified_urls[0:1] = []
# Save the current misc text
identified_citation_element = {
'type' : "URL",
'misc_txt' : "%s" % cur_misc_txt,
'url_string' : "%s" % url_string,
'url_desc' : "%s" % url_desc
}
count_url += 1
cur_misc_txt = u""
elif tag_type == "DOI":
# This tag is an identified DOI:
# From the "identified_dois" list, get this DOI and its
# description string:
doi_string = identified_dois[0]
# Now move past this "<cds.CDS />"tag in the line:
processed_line = processed_line[tag_match_end:]
# Remove DOI from the list of DOI strings
identified_dois[0:1] = []
# SAVE the current misc text
identified_citation_element = {
'type' : "DOI",
'misc_txt' : "%s" % cur_misc_txt,
'doi_string' : "%s" % doi_string
}
# Increment the stats counters:
count_doi += 1
cur_misc_txt = u""
elif tag_type.find("AUTH") != -1:
# This tag is an identified Author:
auth_type = ""
# extract the title from the line:
if tag_type.find("stnd") != -1:
auth_type = "stnd"
idx_closing_tag_nearest = processed_line.find(
CFG_REFEXTRACT_MARKER_CLOSING_AUTHOR_STND, tag_match_end)
elif tag_type.find("etal") != -1:
auth_type = "etal"
idx_closing_tag_nearest = processed_line.find(
CFG_REFEXTRACT_MARKER_CLOSING_AUTHOR_ETAL, tag_match_end)
elif tag_type.find("incl") != -1:
auth_type = "incl"
idx_closing_tag_nearest = processed_line.find(
CFG_REFEXTRACT_MARKER_CLOSING_AUTHOR_INCL, tag_match_end)
if idx_closing_tag_nearest == -1:
# no closing </cds.AUTH****> tag found - strip the opening tag
# and move past it
processed_line = processed_line[tag_match_end:]
identified_citation_element = None
else:
auth_txt = processed_line[tag_match_end:idx_closing_tag_nearest]
# Now move past the ending tag in the line:
processed_line = processed_line[idx_closing_tag_nearest + len("</cds.AUTHxxxx>"):]
#SAVE the current misc text
identified_citation_element = {
'type' : "AUTH",
'misc_txt' : "%s" % cur_misc_txt,
'auth_txt' : "%s" % auth_txt,
'auth_type' : "%s" % auth_type
}
# Increment the stats counters:
count_auth_group += 1
cur_misc_txt = u""
# These following tags may be found separately;
# They are usually found when a "JOURNAL" tag is hit
# (ONLY immediately afterwards, however)
# Sitting by themselves means they do not have
# an associated TITLE tag, and should be MISC
elif tag_type == "SER":
# This tag is a SERIES tag; Since it was not preceeded by a TITLE
# tag, it is useless - strip the tag and put it into miscellaneous:
(cur_misc_txt, processed_line) = \
convert_unusable_tag_to_misc(processed_line, cur_misc_txt,
tag_match_end,
CFG_REFEXTRACT_MARKER_CLOSING_SERIES)
identified_citation_element = None
elif tag_type == "VOL":
# This tag is a VOLUME tag; Since it was not preceeded by a TITLE
# tag, it is useless - strip the tag and put it into miscellaneous:
(cur_misc_txt, processed_line) = \
convert_unusable_tag_to_misc(processed_line, cur_misc_txt,
tag_match_end,
CFG_REFEXTRACT_MARKER_CLOSING_VOLUME)
identified_citation_element = None
elif tag_type == "YR":
# This tag is a YEAR tag; Since it's not preceeded by TITLE and
# VOLUME tags, it is useless - strip the tag and put the contents
# into miscellaneous:
(cur_misc_txt, processed_line) = \
convert_unusable_tag_to_misc(processed_line, cur_misc_txt,
tag_match_end,
CFG_REFEXTRACT_MARKER_CLOSING_YEAR)
identified_citation_element = None
elif tag_type == "PG":
# This tag is a PAGE tag; Since it's not preceeded by TITLE,
# VOLUME and YEAR tags, it is useless - strip the tag and put the
# contents into miscellaneous:
(cur_misc_txt, processed_line) = \
convert_unusable_tag_to_misc(processed_line, cur_misc_txt,
tag_match_end,
CFG_REFEXTRACT_MARKER_CLOSING_PAGE)
identified_citation_element = None
elif tag_type == "QUOTED":
identified_citation_element, processed_line, cur_misc_txt = \
map_tag_to_subfield(tag_type,
processed_line[tag_match_end:],
cur_misc_txt,
'title')
elif tag_type == "ISBN":
identified_citation_element, processed_line, cur_misc_txt = \
map_tag_to_subfield(tag_type,
processed_line[tag_match_end:],
cur_misc_txt,
tag_type)
elif tag_type == "PUBLISHER":
identified_citation_element, processed_line, cur_misc_txt = \
map_tag_to_subfield(tag_type,
processed_line[tag_match_end:],
cur_misc_txt,
'publisher')
elif tag_type == "COLLABORATION":
identified_citation_element, processed_line, cur_misc_txt = \
map_tag_to_subfield(tag_type,
processed_line[tag_match_end:],
cur_misc_txt,
'collaboration')
if identified_citation_element:
# Append the found tagged data and current misc text
citation_elements.append(identified_citation_element)
identified_citation_element = None
# Look for the next tag in the processed line:
tag_match = re_tagged_citation.search(processed_line)
# place any remaining miscellaneous text into the
# appropriate MARC XML fields:
cur_misc_txt += processed_line
# This MISC element will hold the entire citation in the event
# that no tags were found.
if len(cur_misc_txt.strip(" .;,")) > 0:
# Increment the stats counters:
count_misc += 1
identified_citation_element = {
'type' : "MISC",
'misc_txt' : "%s" % cur_misc_txt,
}
citation_elements.append(identified_citation_element)
return (citation_elements, line_marker, {
'misc': count_misc,
'title': count_title,
'reportnum': count_reportnum,
'url': count_url,
'doi': count_doi,
'auth_group': count_auth_group
})
def map_tag_to_subfield(tag_type, line, cur_misc_txt, dest):
"""Create a new reference element"""
closing_tag = '</cds.%s>' % tag_type
# extract the institutional report-number from the line:
idx_closing_tag = line.find(closing_tag)
# Sanity check - did we find a closing tag?
if idx_closing_tag == -1:
# no closing </cds.TAG> tag found - strip the opening tag and move past this
# recognised reportnumber as it is unreliable:
identified_citation_element = None
line = line[len('<cds.%s>' % tag_type):]
else:
tag_content = line[:idx_closing_tag]
identified_citation_element = {'type' : tag_type,
'misc_txt' : cur_misc_txt,
dest : tag_content}
ending_tag_pos = idx_closing_tag + len(closing_tag)
line = line[ending_tag_pos:]
cur_misc_txt = u""
return identified_citation_element, line, cur_misc_txt
def convert_unusable_tag_to_misc(line,
misc_text,
tag_match_end,
closing_tag):
"""Function to remove an unwanted, tagged, citation item from a reference
line. The tagged item itself is put into the miscellaneous text variable;
the data up to the closing tag is then trimmed from the beginning of the
working line. For example, the following working line:
Example, AN. Testing software; <cds.YR>(2001)</cds.YR>, CERN, Geneva.
...would be trimmed down to:
, CERN, Geneva.
...And the Miscellaneous text taken from the start of the line would be:
Example, AN. Testing software; (2001)
...(assuming that the details of <cds.YR> and </cds.YR> were passed to
the function).
@param line: (string) - the reference line.
@param misc_text: (string) - the variable containing the miscellaneous
text recorded so far.
@param tag_match_end: (integer) - the index of the end of the opening tag
in the line.
@param closing_tag: (string) - the closing tag to look for in the line
(e.g. </cds.YR>).
@return: (tuple) - containing misc_text (string) and line (string)
"""
# extract the tagged information:
idx_closing_tag = line.find(closing_tag, tag_match_end)
# Sanity check - did we find a closing tag?
if idx_closing_tag == -1:
# no closing tag found - strip the opening tag and move past this
# recognised item as it is unusable:
line = line[tag_match_end:]
else:
# closing tag was found
misc_text += line[tag_match_end:idx_closing_tag]
# now trim the matched item and its tags from the start of the line:
line = line[idx_closing_tag+len(closing_tag):]
return (misc_text, line)
# Tasks related to extraction of reference section from full-text:
# ----> 1. Removing page-breaks, headers and footers before
# searching for reference section:
# ----> 2. Finding reference section in full-text:
# ----> 3. Found reference section - now take out lines and rebuild them:
def remove_leading_garbage_lines_from_reference_section(ref_sectn):
"""Sometimes, the first lines of the extracted references are completely
blank or email addresses. These must be removed as they are not
references.
@param ref_sectn: (list) of strings - the reference section lines
@return: (list) of strings - the reference section without leading
blank lines or email addresses.
"""
p_email = re.compile(ur'^\s*e\-?mail', re.UNICODE)
while ref_sectn and (ref_sectn[0].isspace() or p_email.match(ref_sectn[0])):
ref_sectn.pop(0)
return ref_sectn
# ----> Glue - logic for finding and extracting reference section:
# Tasks related to conversion of full-text to plain-text:
def get_plaintext_document_body(fpath, keep_layout=False):
"""Given a file-path to a full-text, return a list of unicode strings
whereby each string is a line of the fulltext.
In the case of a plain-text document, this simply means reading the
contents in from the file. In the case of a PDF/PostScript however,
this means converting the document to plaintext.
@param fpath: (string) - the path to the fulltext file
@return: (list) of strings - each string being a line in the document.
"""
textbody = []
status = 0
if os.access(fpath, os.F_OK|os.R_OK):
# filepath OK - attempt to extract references:
# get file type:
cmd_pdftotext = [CFG_PATH_GFILE, fpath]
pipe_pdftotext = subprocess.Popen(cmd_pdftotext, stdout=subprocess.PIPE)
res_gfile = pipe_pdftotext.stdout.read()
if (res_gfile.lower().find("text") != -1) and \
(res_gfile.lower().find("pdf") == -1):
# plain-text file: don't convert - just read in:
f = open(fpath, "r")
try:
textbody = [line.decode("utf-8") for line in f.readlines()]
finally:
f.close()
elif (res_gfile.lower().find("pdf") != -1) or \
(res_gfile.lower().find("pdfa") != -1):
# convert from PDF
(textbody, status) = convert_PDF_to_plaintext(fpath, keep_layout)
else:
# invalid format
status = 1
else:
# filepath not OK
status = 1
return (textbody, status)
def parse_references(reference_lines, recid=None, kbs_files=None):
"""Parse a list of references
Given a list of raw reference lines (list of strings),
output the MARC-XML content extracted version
"""
# RefExtract knowledge bases
kbs = get_kbs(custom_kbs_files=kbs_files)
# Identify journal titles, report numbers, URLs, DOIs, and authors...
processed_references, counts, dummy_bad_titles_count = \
parse_references_elements(reference_lines, kbs)
# Generate marc xml using the elements list
fields = build_references(processed_references)
# Generate the xml string to be outputted
return build_record(counts, fields, recid=recid)
diff --git a/invenio/legacy/refextract/kbs/books.kb b/invenio/legacy/refextract/kbs/books.kb
index 64d44a185..6d14d5651 100644
--- a/invenio/legacy/refextract/kbs/books.kb
+++ b/invenio/legacy/refextract/kbs/books.kb
@@ -1,3283 +1,3284 @@
't Hooft, G., (ed.)|Fifty years of Yang-Mills theory|2005;
't Hooft, Gerard, (Ed.)|50 Years of Yang-Mills Theory|2005;
't Hooft, Gerard|In search of the ultimate building blocks|1992;
't Hooft, Gerard|Under the spell of the gauge principle|1995;
Abad, Carlos, (ed.)|Third International Meeting of Dynamic Astronomy in Latin America (ADeLA 2004), Merida, Venezuela, noviembre 22-26, 2004|2006;
Abbott, L.F., (Ed.)|INFLATIONARY COSMOLOGY|1986;
Abdalla, E.|2-D gravity in noncritical strings: Discrete and continuum approaches|1994;
Abdalla, E.|Nonperturbative methods in two-dimensional quantum field theory|1991;
Abdushukurov, D.A.|Multistep avalanche chambers|2011;
Abe, David K., (ed.)|High Energy Density and High Power RF, 7th Workshop on High Energy Density and High Power RF, Kalamata, Greece, 13-17 June 2005|2006;
Ablamowicz, R., (ed.)|Clifford algebras and spinor structures: A Special volume dedicated to the memory of Albert Crumeyrolle (1919-1992)|1995;
Ablowitz, M.J.|SOLITONS AND THE INVERSE SCATTERING TRANSFORM|1981;
Ablowitz, M.J.|Solitons, nonlinear evolution equations and inverse scattering|1991;
Abramenko, B.|A UNIFIED MODEL OF ELEMENTARY PARTICLES|1982;
Abramyan, E.A.|INDUSTRIAL ELECTRON ACCELERATORS AND APPLICATIONS|1988;
Abreu, Everton M.C.|Topics on the quantum dynamics of chiral bosons|2004;
Accardi, L., (ed.)|Quantum probability and related topics|1995;
Aczel, Amir D.|Present at the creation: The story of CERN and the Large Hadron Collider|2010;
Adair, R.K.|THE GREAT DESIGN: PARTICLES, FIELDS, AND CREATION|1987;
Adams, Jenni, (ed.)|The XXIII Conference on Neutrino Physics and Astrophysics, 25-31 May 2008, Christchurch, New Zealand|2008;
Adams, S.|Frontiers: Twentieth-century physics|2000;
Adams, S.|Relativity: An introduction to space-time physics|1997;
Adem, Alejandro|Orbifolds and stringy topology|2007;
Adler, Stephen L., (ed.)|Quaternionic quantum mechanics and quantum fields|1988;
Adler, Stephen L.|Adventures in theoretical physics: Selected papers with commentaries|2006;
Adler, Stephen L.|Anomalies to all orders|2004;
Adler, Stephen L.|Monopoles and projective representations: Two areas of influence of Yang-Mills theory on mathematics|2004;
Adler, Stephen L.|Quantum theory as an emergent phenomenon: The statistical mechanics of matrix models as the precursor of quantum field theory|2004;
Adler, Stephen L.|Statistical dynamics of global unitary invariant matrix models as pre - quantum mechanics|2002;
Adomian, G.|NONLINEAR STOCHASTIC SYSTEMS THEORY AND APPLICATIONS TO PHYSICS|1989;
Adzhemian, L.Ts.|The field theoretic renormalization group in fully developed turbulence|1999;
Afanasev, G.N.|Vavilov-Cherenkov and synchrotron radiation: Foundations and applications|2004;
Afriat, A.|The Einstein, Podolsky, and Rosen paradox in atomic, nuclear, and particle physics|1999;
Agarwal, A.|Quest for secrets of the universe|2006;
Agarwal, B.K.|Quantum Mechanics and Field Theory|1976;
Aharonian, F.A.|Very high energy cosmic gamma radiation: A crucial window on the extreme universe|2004;
Ahmed, S.N.|Physics and engineering of radiation detection|2007;
Ahrenhoevel, H.|Photodisintegration of the deuteron: A Review of theory and experiment|1991;
Ahrens, T.|From Dirac to neutrino oscillations|2000;
Aichelburg, P.C., (Ed.)|TIME IN THE COURSE OF TIME. (IN GERMAN)|1988;
Aime, Claude, (ed.)|Astronomy with high contrast imaging : From planetary systems to active galactic nuclei, Nice, France, May 13-16, 2002|2003;
Aitchison, I.J.R., (ed.)|GAUGE THEORIES IN PARTICLE PHYSICS. A PRACTICAL INTRODUCTION|1982;
Aitchison, I.J.R.|AN INFORMAL INTRODUCTION TO GAUGE THEORIES|1980;
Aitchison, I.J.R.|GAUGE THEORIES IN PARTICLE PHYSICS: A PRACTICAL INTRODUCTION|1989;
Aitchison, I.J.R.|Gauge theories in particle physics: A practical introduction. Vol. 1: From relativistic quantum mechanics to QED|2003;
Aitchison, I.J.R.|Gauge theories in particle physics: A practical introduction. Vol. 2: Non-Abelian gauge theories: QCD and the electroweak theory|2004;
Aitchison, I.J.R.|Supersymmetry in Particle Physics. An Elementary Introduction|2007;
Ajduk, Z., (ed.)|The Photon, its first hundred years and the future, the centenary of the photon : PHOTON2005, International Conference on the Structure and Interactions of the Photon including the 16th International Workshop
Akhiezer, A.I.|Fields and fundamental interactions|2002;
Akhiezer, A.I.|High-energy electrodynamics in matter|1996;
Akhiezer, A.I.|Nuclear electrodynamics|1994;
Akulin, Vladimir M.|Coherent dynamics of complex quantum systems|2006;
Al-Khalili, J.|Black holes, wormholes and time machines|2001;
Alarcon, Ricardo, (ed.)|VII Latin American Symposium on Nuclear Physics and Applications, Cusco, Peru, 11-16 June 2007|2007;
Albeverio, S., (ed.)|Ideas and methods in quantum and statistical physics. In memory of Raphael Hoeegh-Krohn. Vol. 2|1992;
Albeverio, S.A.|Mathematical Theory of Feynman Path Integrals. Lecture Notes in Mathematics, Vol. 523|1976;
Albeverio, S.|A mathematical introduction to string theory. Variational problems, geometric and probabilistic methods|1997;
Aldaya, Victor, (Ed.)|Symmetries in gravity and field theory|2004;
Aldrovandi, R.|An Introduction to geometrical physics|1996;
Aleksandrov, Yu.A.|Fundamental properties of the neutron|1992;
Alemany-Fernandez, R., (ed.)|LHC Workshop on Experimental Conditions and Beam Induced Detector Backgrounds, CERN, Geneva, Switzerland, 3-4 April 2008|2009;
Alhassid, Y.|Mesoscopic effects in quantum dots, nanoparticles and nuclei|2006;
Ali, A., (Ed.)|HIGH-ENERGY ELECTRON POSITRON PHYSICS|1988;
Ali, A., (ed.)|Selected papers of Abdus Salam: With commentary|1994;
Alkofer, Reinhard|Chiral quark dynamics|1995;
Allday, J.|Quarks, leptons and the big bang|1998;
Allkofer, O.C.|Introduction to Cosmic Radiation|1975;
Allton, C.R.|Quenching effects in the hadron spectrum|2004;
Almeida, Jose B.|Can physics laws be derived from monogenic functions?|2006;
Alonso, M.|QUANTUM PHYSICS. (IN GERMAN)|1988;
Alpher, R.A.|Genesis of the big bang|2001;
Altarelli, G., (Ed.)|PROTON ANTI-PROTON COLLIDER PHYSICS|1989;
Altarelli, G., (ed.)|Neutrino mass|2003;
Altarelli, Guido|Models of neutrino masses and mixings|2004;
Altarelli, Guido|Particle Physics Summer School, Gif-Sur-Yvette, 2-21 September 1974, 2. Photon - Hadron Interactions|1974;
Altarelli, Guido|Summer School of Particle Physics, Gif-Sur-Yvette, 8-19 September 1975. 1. Symmetries and New Particles|1975;
Altarelli, Guido|The Development of perturbative QCD|1995;
Altland, A.|Condensed matter field theory|2006;
Alvarez-Estrada, R.F.|MODELS OF HADRON STRUCTURE BASED ON QUANTUM CHROMODYNAMICS|1986;
Alvarez-Gaume, L., (ed.)|Infinitely CERN: Memories of fifty years of research. 1954-2004|2004;
Alwall, Johan|Quark distributions and charged Higgs boson production - Studies of proton structure and new physics|2005;
Amadei, P.|ARES design study: The Machine|1990;
Amaldi, E.|PION ELECTROPRODUCTION. ELECTROPRODUCTION AT LOW-ENERGY AND HADRON FORM-FACTORS|1979;
Amays, R.K.|Infinite Dimensional Lie Algebras|1974;
Ambjorn, Jan|Quantum geometry. A statistical field theory approach|1997;
Ames, F., (ed.)|The REX-ISOLDE facility, design and commissioning report|2005;
Amit, D.J.|FIELD THEORY, THE RENORMALIZATION GROUP, AND CRITICAL PHENOMENA|1984;
Amit, D.J.|Field Theory, the Renormalization Group, and Critical Phenomena|1978;
Amoroso, Richard L.|The holographic anthropic multiverse: Formalizing the complex geometry of reality|2009;
Amrein, W.O.|Scattering Theory in Quantum Mechanics. Physical Principles and Mathematical Methods|1977;
Amsler, Claude|Nuclear and particle physics|2007;
Anastasovski, P.K.|Quantum mass theory compatible with quantum field theory|1995;
Anastopoulos, Charis|Particle or wave: The evolution of the concept of matter in modern physics|2008;
Anderson, Edward|Leibniz-Mach foundations for GR and fundamental physics|2004;
Anderson, Malcolm R.|The mathematical theory of cosmic strings: Cosmic strings in the wire approximation|2003;
Anderson, R.L.|LIE-BACKLUND TRANSFORMATIONS IN APPLICATIONS|1979;
Andersson, Bo|The Lund model|1998;
Anisovich, V.V.|Mesons and baryons: Systematization and methods of analysis|2008;
Anisovich, V.V.|QUARK MODEL AND HIGH-ENERGY COLLISIONS|1986;
Ankenbrandt, C.|Physics Study Group report on physics potential at FNAL with stronger proton sources|2001;
Annenkov, A.N.|Mass growth of large PbWO-4 single crystals for particle detection in high-energy physics experiments at CERN|2009;
Ansorge, R., (ed.)|Research highlights: On the 75th anniversary of Hamburg University 1994. (In German)|1995;
Antoci, Salvatore|Reinstating Schwarzschild's original manifold and its singularity|2004;
Antonelli, Lucio Angelo, (ed.)|The Multicolored Landscape of Compact Objects and Their Explosive Origins, Cefalu 2006, Cefalu, Sicily, 11-18 June 2006|2007;
Antonelli, P.L., (ed.)|Lagrange and Finsler geometry: Applications to physics and biology|1996;
Antoniou, I., (ed.)|Solitons and chaos|1991;
Antonov, A.N.|Nucleon correlations in nuclei|1994;
Antonov, Dmitri|3D Georgi-Glashow model and confining strings at zero and finite temperatures|2004;
Apparao, K.M.V.|Composition of Cosmic Radiation|1975;
Appelquist, T., (Ed.)|MODERN KALUZA-KLEIN THEORIES|1987;
Arafune, J., (ed.)|A garden of quanta: Essays in honor of Hiroshi Ezawa|2003;
Araki, H.|International Symposium on Mathematical Problems in Theoretical Physics. January 23-29, 1975, Kyoto|1975;
Araki, H.|Mathematical theory of quantum fields|1999;
Aratyn, H., (Ed.)|Integrable hierarchies and modern physical theories|2001;
Arickx, F.|The Modified J-Matrix approach for cluster descriptions of light nuclei|2004;
Arimitsu, A., (ed.)|Selected papers of Hiroomi Umezawa|2001;
Armel, J.|Entropic space-time theory|1996;
Armenteros, R.|Physics from Friends. Papers Dedicated to Ch. Peyrou on His 60th Birthday|1978;
Arminjon, Mayeul|Ether theory of gravitation: Why and how?|2004;
Armoni, A.|From superYang-Mills theory to QCD: Planar equivalence and its implications|2004;
Arnett, D.|Supernovae and nucleosynthesis: An investigation of the history of matter, from the big bang to the present|1996;
Arnoldus, H.F., (ed.)|Theoretical physics 2001|2002;
Arnowitt, Richard L.|The Dynamics of general relativity|1962;
Arteca, G.A.|Large order perturbation theory and summation methods in quantum mechanics|1990;
Asanov, G.S.|FINSLER GEOMETRY, RELATIVITY AND GAUGE THEORIES|1986;
Aschieri, Paolo|Noncommutative spacetimes: Symmetries in noncommutative geometry and field theory|2009;
Ashtekar, A.|ASYMPTOTIC QUANTIZATION: BASED ON 1984 NAPLES LECTURES|1987;
Ashtekar, A.|Lectures on nonperturbative canonical gravity|1991;
Asimov, I.|Atom: Journey across the subatomic cosmos|1991;
Asner, F.M.|High field superconducting magnets|1999;
Asok, Aravind|Yang-Mills theory and Tamagawa numbers: The Fascination of unexpected links in mathematics|2008;
Asselmeyer-Maluga, T.|Exotic smoothness and physics: Differential topology and spacetime models|2007;
Atiyah, M., (ed.)|Fields medallists' lectures|1997;
Atiyah, M.F.|GEOMETRY OF YANG-MILLS FIELDS|1979;
Atiyah, M.F.|THE GEOMETRY AND DYNAMICS OF MAGNETIC MONOPOLES. M.B. PORTER LECTURES|1988;
Atiyah, M.|Michael Atiyah collected works. Vol. 5: Gauge theories|1988;
Atiyah, M.|The Geometry and physics of knots|1990;
Atkinson, D.|Quantum field theory: A self-contained course. Vol. 2|2002;
Attix, F.H.|TOPICS IN RADIATION DOSIMETRY. SUPPLEMENT 1, RADIATION DOSIMETRY|1972;
Aubert, Bernard|WEAK INTERACTIONS. CONFERENCE, VITTEL, 28 MAY - 2 JUNE 1973. (MOSTLY IN FRENCH)|1973;
Audretsch, J., (Ed.)|PHILOSOPHY AND PHYSICS OF SPACE-TIME. (IN GERMAN)|1988;
Audretsch, J., (ed.)|The Beginning of the world: Science, philosophy, religion, myth. (In German)|1990;
Auyang, S.Y.|How is quantum field theory possible?|1995;
Avella, Adolfo, (ed.)|Lectures on the physics of highly correlated electron systems IX, ninth Training Course in the Physics of Correlated Electron Systems and High-Tc Superconductors, Salerno, Italy, 4-15 October 2004|2005;
Avery, J.|Creation and Annihilation Operators|1976;
Avramidi, I.G.|Heat kernel and quantum gravity|2000;
Baadhio, R.A.|Quantum topology and global anomalies|1996;
Baaklini, N.S.|Electronuclear interactions|1990;
Baaklini, N.S.|HIGH-ENERGY ALGEBRAS|1984;
Baaklini, N.S.|High-energy algebras|1990;
Baaklini, N.S.|High-energy thermal theory|1987;
Baaklini, N.S.|INVARIANT FIELD THEORY|1984;
Baaklini, N.S.|Invariant field theory|1990;
Baaklini, N.S.|Prequantum field processes|1990;
Baaklini, N.S.|Quantum field action|1990;
Baaklini, N.S.|Quantum gravidynamics|1990;
Bacciagaluppi, Guido|Quantum theory at the crossroads: Reconsidering the 1927 Solvay conference|2006;
Bacry, H.|LOCALIZABILITY AND SPACE IN QUANTUM PHYSICS|1988;
Baer, H.|Weak scale supersymmetry: From superfields to scattering events|2006;
Baer, Howard|Dark matter and the LHC|2008;
Baeuerle, G.G.A.|Lie algebras. Pt. 1: Finite and infinite dimensional Lie algebras and applications in physics|1990;
Baez, J.C.|Introduction to algebraic and constructive quantum field theory|1992;
Baez, J.|Gauge fields, knots and gravity|1995;
Baez, John C.|Quantum quandaries: A Category theoretic perspective|2004;
Bagchi, B.K.|Supersymmetry in quantum and classical mechanics|2001;
Bagge, E.R.|World and antiworld as physical reality: In the memory of Arnold Sommerfeld and Werner Heisenberg. (In German)|1990;
Bagge, E.R.|World and antiworld as physical reality: Spherical shell elementary particles|1994;
Baggott, Jim|The quantum story: A history in 40 moments|2011;
Bagrov, V.G.|Exact solutions of relativistic wave equations|1990;
Bahcall, J.N., (Ed.)|Solar neutrinos: The first thirty years|1995;
Bahcall, John N.|NEUTRINO ASTROPHYSICS|1989;
Bahder, Thomas B.|Clock synchronization and navigation in the vicinity of the earth|2004;
Baier, V.N.|Electromagnetic processes at high energies in oriented single crystals|1998;
Bailey, T.N., (ed.)|Twistors in mathematics and physics|1990;
Bailin, D.|Cosmology in gauge field theory and string theory|2004;
Bailin, D.|INTRODUCTION TO GAUGE FIELD THEORY|1986;
Bailin, D.|Supersymmetric gauge field theory and string theory|1994;
Bailin, D.|WEAK INTERACTIONS|1982;
Bailin, D.|Weak Interactions|1977;
Bais, F.Alexander|To be or not to be? Magnetic monopoles in non-Abelian gauge theories|2004;
Baker, G.A., (ed.)|PADE APPROXIMANTS. I. BASIC THEORY|1981;
Baker, G.A., (ed.)|PADE APPROXIMANTS. II. EXTENSIONS AND APPLICATIONS|1981;
Balachandran, A.P.|Classical topology and quantum states|1991;
Balachandran, A.P.|GAUGE SYMMETRIES AND FIBER BUNDLES: APPLICATIONS TO PARTICLE DYNAMICS|1983;
Balachandran, A.P.|Hubbard model and anyon superconductivity|1990;
Balachandran, A.P.|LECTURES ON GROUP THEORY FOR PHYSICISTS|1986;
Balakin, Alexander B.|Non-minimal Einstein-Yang-Mills-dilaton theory|2008;
Balakrishnan, A.V.|Applied Functional Analysis. Application of Mathematics. 3.|1976;
Balashov, V.V.|Interaction of particles and radiation with matter|1997;
Baldini, A.|NUMERICAL DATA AND FUNCTIONAL RELATIONSHIPS IN SCIENCE AND TECHNOLOGY. GRP. 1: NUCLEAR AND PARTICLE PHYSICS. VOL. 12: TOTAL CROSS-SECTIONS FOR REACTIONS OF HIGH-ENERGY PARTICLES (INCLUDING ELASTIC, TOPOLOGICAL,
Baldini, A.|NUMERICAL DATA AND FUNCTIONAL RELATIONSHIPS IN SCIENCE AND TECHNOLOGY. GRP. 1: NUCLEAR AND PARTICLE PHYSICS. VOL. 12: TOTAL CROSS-SECTIONS FOR REACTIONS OF HIGH-ENERGY PARTICLES (INCLUDING ELASTIC, TOPOLOGICAL,
Baldo Ceolin, Milla, (ed.)|Third NO-VE International Workshop on Neutrino Oscillations in Venice : Fifty years after the neutrino esperimental discovery : Venezia, February 7-10, 2006, Istituto Veneto di Scienze, Lettere ed
Bali, G.|Two quark potentials|2004;
Balian, R.|Methods in Field Theory. Les Houches Summer School in Theoretical Physics. Session 28, July 28-September 6, 1975|1976;
Balian, R.|Structural Analysis of Collision Amplitudes. Lectures Delivered at Les Houches, Summer School in Theoretical Physics, June 1975|1976;
Bandyopadhyay, P.|Geometry, topology and quantization|1996;
Bandyopadhyay, P.|Geometry, topology and quantum field theory|2003;
Bar, Christian|Wave equations on Lorenzian manifolds and quantization|2007;
Baranger, M.|Advances in Nuclear Physics. 8.|1975;
Baranger, M.|Advances in Nuclear Physics. 9.|1977;
Barbashov, B.M.|Introduction to the relativistic string theory|1990;
Barber, Garth A|Self creation cosmology: An Alternative gravitational theory|2004;
Barboni, E.J.|FUNCTIONAL DIFFERENTIATION AND TECHNOLOGICAL SPECIALIZATION IN A SPECIALITY IN HIGH-ENERGY PHYSICS: THE CASE OF WEAK INTERACTIONS OF ELEMENTARY PARTICLES|1987;
Barbour, J.|The end of time: The next revolution in physics|2000;
Bardin, Dmitri Yu.|The standard model in the making: Precision study of the electroweak interactions|1999;
Barger, A.J., (ed.)|Supermassive black holes in the distant universe|2004;
Barger, Vernon D.|COLLIDER PHYSICS|1987;
Barkas, W.H.|NUCLEAR RESEARCH EMULSIONS. II. PARTICLE BEHAVIOR AND EMULSION APPLICATIONS|1973;
Barker, Gary John|b-quark physics with the LEP collider|2010;
Barklow, T.L., (ed.)|Electroweak symmetry breaking and new physics at the TeV scale|1996;
Barkovich, M.|Neutrinospheres, resonant neutrino oscillations, and pulsar kicks|2005;
Barnabei, O., (ed.)|The origin of the third family|1998;
Barnes, Ken J.|Group theory for the standard model of particle physics and beyond|2010;
Barnes, Peter D., (ed.)|Particles and nuclei : Seventeenth International Conference on Particles and Nuclei, Santa Fe, New Mexico, 23-30 October 2005|2006;
Barnes, Thomas G.|PHYSICS OF THE FUTURE. A CLASSICAL UNIFICATION OF PHYSICS|1987;
Barnett, R.Michael|The charm of strange quarks: Mysteries and revolutions of particle physics|2000;
Barone, V.|High-energy particle diffraction|2002;
Barone, V.|Transverse spin physics|2003;
Barreiro, F., (ed.)|XXXIV International Meeting on Fundamental PHysics, 2-7 April 2006, El Escorial, Madrid, Spain, : From HERA and TEVATRON to LHC|2008;
Barrett, T.W., (ed.)|Advanced electromagnetism: Foundations, theory and applications|1995;
Barriol, J.|Elements of Statistical Mechanics in Strong Interacting Systems. (In French)|1974;
Barron, Katrina|The Moduli space of N=1 superspheres with tubes and the sewing operation|2000;
Barrow, John D., (ed.)|Science and ultimate reality: Quantum theory, cosmology, and complexity|2004;
Barrow, John D.|New theories of everything: The quest for ultimate explanation|2007;
Barrow, John D.|THE LEFTHAND OF CREATION. THE ORIGIN AND EVOLUTION OF THE EXPANDING UNIVERSE|1985;
Barrow, John D.|The World within the world|1988;
Barrow, John D.|The origin of the universe|1998;
Barrow, John D.|Theories of everything: The Quest for ultimate explanation|1991;
Bars, Itzhak|Extra dimensions in space and time|2010;
Bartke, Jerzy|Introduction to relativistic heavy ion physics|2009;
Bartl, W.|Pulses in Border Fields of Physics|1976;
Bartnik, R.|On Maximal hypersurfaces in asymptotically flat space-times|1990;
Barton, A.|States of matter, states of mind|1997;
Bartusiak, M.|Einstein's unfinished symphony: Listening to the sounds of space-time|2000;
Barut, A.O., (Ed.)|QUANTUM THEORY, GROUPS, FIELDS AND PARTICLES|1984;
Barut, A.O., (Ed.)|QUANTUM, SPACE AND TIME - THE QUEST CONTINUES. STUDIES AND ESSAYS IN HONOR OF LOUIS DE BROGLIE, PAUL DIRAC AND EUGENE WIGNER|1985;
Barut, A.O.|DE SITTER AND CONFORMAL GROUPS AND THEIR APPLICATIONS. LECTURES IN THEORETICAL PHYSICS, BOULDER 1970|1971;
Barut, A.O.|Electrodynamics and classical theory of fields and particles|1980;
Barut, A.O.|FOUNDATIONS OF RADIATION THEORY AND QUANTUM ELECTRODYNAMICS|1980;
Barut, A.O.|Group Theory in Nonlinear Problems. Lectures Presented at the Nato Advanced Study Institute on Mathematical Physics Held in Istanbul, August 7-18, 1972|1974;
Barut, A.O.|STUDIES IN MATHEMATICAL PHYSICS. LECTURES PRESENTED AT THE NATO INSTITUTE, ISTANBUL, AUGUST 1970|1973;
Barut, A.O.|THEORY OF GROUP REPRESENTATIONS AND APPLICATIONS|1986;
Barut, A.O.|TOPICS IN STRONG INTERACTIONS. LECTURES IN THEORETICAL PHYSICS, VOL. 14A, BOULDER 1971|1972;
Baryshev, Yu.|Discovery of cosmic fractals|2002;
Basdevant, J.L.|Fundamentals in nuclear physics: From nuclear structure to cosmology|2005;
Bashir, Adnan|Gauge symmetry and its implications for the Schwinger-Dyson equations|2004;
Basov, N.G., (Ed.)|Synchrotron Radiation|1976;
Bass, Steven D.|The Spin structure of the proton|2007;
Bassetto, A.|Yang-Mills theories in algebraic noncovariant gauges: Canonical quantization and renormalization|1991;
Bastian, Ted|The origin of discrete particles|2009;
Bastianelli, F.|Path integrals and anomalies in curved space|2006;
Bastin, T.|Combinatorial physics|1996;
Baston, R.J.|The Penrose transform: Its interaction with representation theory|1989;
Basu, D., (ed.)|Dictionary of material science and high energy physics|2001;
Basu, Debabrata|Introduction to classical and modern analysis and their application to group representation theory|2011;
Batalin, I.A., (Ed.)|QUANTUM FIELD THEORY AND QUANTUM STATISTICS. ESSAYS IN HONOR OF THE SIXTIETH BIRTHDAY OF E.S. FRADKIN. VOL. 1: QUANTUM STATISTICS AND METHODS OF FIELD THEORY|1987;
Batalin, I.A., (Ed.)|QUANTUM FIELD THEORY AND QUANTUM STATISTICS: ESSAYS IN HONOR OF THE SIXTIETH BIRTHDAY OF E.S. FRADKIN. VOL. 2: MODELS OF FIELD THEORY|1987;
Bates, D.R., (ed.)|Advances in Atomic and Molecular Physics. Vol. 16|1980;
Bates, S.|Lectures on the geometry of quantization|1997;
Battaglia, M.|Determination of |V(ub)||2004;
Battle, G.|Wavelets and renormalization|1999;
Baum, Helga|Gauge field theory: An introduction to differential geometry on fibre bundles|2009;
Baumgaertel, H.|Causal nets of operator algebras: Mathematical aspects of algebraic quantum field theory|1992;
Baumgaertel, H.|Operator algebraic methods in quantum field theory: A Series of lectures|1995;
Becchi, C.M.|An introduction to relativistic processes and the standard model of electroweak interactions|2006;
Becher, P.|GAUGE THEORIES OF STRONG AND ELECTROWEAK INTERACTION. (IN GERMAN)|1981;
Becher, P.|GAUGE THEORIES OF STRONG AND ELECTROWEAK INTERACTIONS|1984;
Beck, C.|Spatio-temporal chaos and vacuum fluctuations of quantized fields|2002;
Beck, Douglas H., (ed.)|Advances in the physics of particles and nuclei. Vol. 30|2010;
Becker, K.|Introduction to Solid State Dosimetry|1975;
Becker, K.|SOLID STATE DOSIMETRY|1973;
Becker, K.|String theory and M-theory: A modern introduction|2007;
Beckwith, A.W.|Implications for the Cosmological Landscape: Can Thermal Inputs from a Prior Universe Account for Relic Graviton Production?|2008;
Bederson, B., (ed.)|Advances in atomic, molecular, and optical physics. Vol. 45|2001;
Bederson, B.|Advances in atomic, molecular, and optical physics. Vol. 40|1999;
Beech, Martin|The Large Hadron Collider: Unraveling the mysteries of the universe|2010;
Begelman, M.|Gravity's fatal attraction: Black holes in the universe|1996;
Behrens, H.|DATA COMPILATIONS IN PHYSICS|1986;
Beiser, A.|CONCEPTS OF MODERN PHYSICS|1987;
Belanger, Genevieve (ed.)|Physics at TeV colliders, La physique du TeV aux collisionneurs, Les Houches 2007 : 11-29 June 2007|2007;
Belinfante, F.J.|Measurements and Time Reversal in Objective Quantum Theory|1975;
Belinski, V.|Gravitational solitons|2001;
Bell, J.S.|SPEAKABLE AND UNSPEAKABLE IN QUANTUM MECHANICS. COLLECTED PAPERS ON QUANTUM PHILOSOPHY|1987;
Bell, M., (ed.)|Quantum mechanics, high-energy physics and accelerators: Selected papers of John S. Bell with commentary|1995;
Bellandi, J., (Ed.)|TOPICS ON COSMIC RAYS: 60TH ANNIVERSARY OF C.M.G. LATTES. VOL. 1|1984;
Bellucci, S.|Complete and Consistent Non-Minimal String Corrections to Supergravity|2008;
Bellucci, Stefano, (ed.)|Supersymmetric mechanics. Vol. 1: Supersymmetry, noncommutativity and matrix models|2006;
Bellucci, Stefano, (ed.)|Supersymmetric mechanics. Vol. 3: Attractors and black holes in supersymmetric gravity|2008;
Bellucci, Stefano|Supersymmetric mechanics. Vol. 2: The attractor mechanism and space time singularities|2006;
Belokurov, V.V.|The Theory of particle interactions|1991;
Belusevic, R.|Neutral kaons|1998;
Belusevic, Radoje|Relativity, astrophysics, and cosmology. Vol. 1|2008;
Belusevic, Radoje|Relativity, astrophysics, and cosmology. Vol. 2|2008;
Belyaev, V.B.|Lectures on the theory of few body systems|1990;
Benenson, W., (ed.)|Handbook of physics|2002;
Beneventano, C.G.|Spectral functions of the Dirac operator under local boundary conditions|2004;
Benfatto, G.|Renormalization group|1996;
Benn, I.M.|AN INTRODUCTION TO SPINORS AND GEOMETRY WITH APPLICATIONS IN PHYSICS|1987;
Benton, C.V., (ed.)|Trends in mathematical physics research|2004;
Benucci, Leonardo|Flavour changing neutral currents in top quark decays: A Monte-Carlo study of a new physics interaction in top quark decays, performed with CMS detector at LHC|2009;
Berche, Bertrand|Logarithmic corrections and universal amplitude ratios in the 4-state Potts model|2007;
Berestetsky, V.b.|QUANTUM ELECTRODYNAMICS|1982;
Berezhnoy, Yuri A.|The quantum world of nuclear physics|2005;
Berezin, F.A.|INTRODUCTION TO SUPERANALYSIS|1987;
Berger, C.|Elementary particle physics: From the foundations to the modern experiments|2002;
Berger, Christoph|Particle physics: An Introduction. (In German)|1992;
Bergmann, L.|Constituents of matter: Atoms, molecules, nuclei and particles|1997;
Bergmann, L.|Textbook of experimental physics. Vol. 4: Particles. (In German)|1992;
Bergmann, P.G., (ed.)|Spin in gravity: Is it possible to give an experimental basis to torsion?|1998;
Bergmann, P.G.|THE RIDDLE OF GRAVITATION|1987;
Bergstrom, L., (ed.)|The Oskar Klein memorial lectures. Vol. 3|2001;
Bergstrom, L.|Cosmology and particle astrophysics|1999;
Berkelman, K.|A personal history of CESR and CLEO: The Cornell Electron Storage Ring and its main particle detector|2004;
Bernabeu, J.|ELECTROWEAK THEORY|1981;
Bernstein, J.|KINETIC THEORY IN THE EXPANDING UNIVERSE|1988;
Bernstein, J.|THE TENTH-DIMENSION. AN INFORMAL HISTORY OF HIGH-ENERGY PHYSICS|1989;
Bertlmann, R.A., (ed.)|Quantum unspeakables: From Bell to quantum information|2002;
Bertlmann, R.A.|Anomalies in quantum field theory|1996;
Bertocchi, L.|Summer School of Particle Physics, Gif-Sur-Yvette, 8-19 September 1975. 2. Diffractive Phenomena|1975;
Bertolami, Orfeu|The Mystical formula and the mystery of Khronos|2008;
Bertone, Gianfranco, (Ed.)|Particle Dark Matter: Observations, Models and Searches|2010;
Bertone, Gianfranco, (ed.)|Particle dark matter: Observations, models and searches|2010;
Berz, M.|Modern map methods in particle beam physics|1999;
Bethge, K.|ELEMENTARY PARTICLES AND THEIR INTERACTIONS. (IN GERMAN)|1986;
Bettini, Alessandro|Introduction to elementary particle physics|2008;
Bhaduri, R.K.|MODELS OF THE NUCLEON: FROM QUARKS TO SOLITON|1988;
Bhaumik, Mani|The cosmic detective: Exploring the mysteries of our universe|2009;
Bialynicki-Birula, I.|Quantum Electrodynamics|1975;
Bicak, Jiri|Einstein equations: Exact solutions|2006;
Bick, E., (ed.)|Topology and geometry in physics|2005;
Biedenharn, L.C.|Quantum group symmetry and q tensor algebras|1996;
Biedenharn, L.C.|THE RACAH-WIGNER ALGEBRA IN QUANTUM THEORY|1981;
Bienlein, J.|Introduction to the structure of matter: Nuclei, particles, molecules, solids|2003;
Bigelow, R.|Nuclear and particle physics simulations: The Consortium for Upper level Physics Software|1996;
Bigi, Ikaros I.Y.|CP violation|2000;
Bigi, Ikaros I.|CP violation|2009;
Bilenkii, S.M., (ed.)|Selected scientific works: Recollections on B. Pontecorvo|1997;
Bilenky, Samoil M.|INTRODUCTION TO THE PHYSICS OF ELECTROWEAK INTERACTIONS|1982;
Bilenky, Samoil M.|Introduction to Feynman diagrams and electroweak interactions physics|1995;
Binder, K., (Ed.)|APPLICATIONS OF THE MONTE CARLO METHOD IN STATISTICAL PHYSICS|1984;
Binder, K., (Ed.)|MONTE CARLO METHODS IN STATISTICAL PHYSICS|1986;
Binder, K., (ed.)|The Monte Carlo method in condensed matter physics|1992;
Binetruy, P.|Supersymmetry: Theory, experiment and cosmology|2006;
Binney, J.J.|The Theory of critical phenomena: An Introduction to the renormalization group|1992;
Binz, E.|GEOMETRY OF CLASSICAL FIELDS|1988;
Birks, John B.|The Theory and practice of scintillation counting|1964;
Biro, T.S.|Chaos and gauge field theory|1994;
Birrell, N.D.|QUANTUM FIELDS IN CURVED SPACE|1982;
Biryukov, V.M.|Crystal channeling and its application at high-energy accelerators|1997;
Bisnovatyi-Kogan, G.S.|Stellar physics. Vol. 1: Fundamental concepts and stellar equilibrium|2001;
Bjorken, J.D.|RELATIVISTIC QUANTUM FIELD THEORY. (GERMAN TRANSLATION)|1979;
Bjorken, James D., (ed.)|In Conclusion: A Collection of Summary Talks in High Energy Physics|2003;
Blackadar, Bruce|Operator algebras: Theory of C*-algebras and von Neumann algebras|2006;
Blagojevic, M.|Gravitation and gauge symmetries|2002;
Blaha, S.|A derivation of electroweak theory based on an extension of special relativity: Black hole tachyons and tachyons of any spin|2006;
Blaha, S.|A finite unified quantum field theory of the elementary particle standard model and quantum gravity: Based on new quantum dimensions and a new paradigm in the calculus of variations|2003;
Blaha, S.|Physics beyond the light barrier: The source of parity violation, tachyons, and a derivation of standard model features|2006;
Blaha, S.|Quantum big bang cosmology: Complex space-time general relativity, quantum coordinates, dodecahedral universe, inflation, and new spin 0, 1/2, 1 and 2 tachyons and imagyons|2004;
Blaha, S.|Quantum theory of the third kind: A new type of divergence-free quantum field theory supporting a unified standard model of elementary particles and quantum gravity based on a new method in the calculus of
Blaha, S.|The equivalence of elementary particle theories and computer languages: Quantum computers, Turing machines, standard model, superstring theory, and a proof that Goedel's theorem implies nature must be quantum|2005;
Blaha, S.|The metatheory of physics theories, and the theory of everything as a quantum computer language|2005;
Blaha, Stephen|A complete derivation of the form of the standard model with a new method to generate particle masses|2008;
Blaha, Stephen|A direct derivation of the form of the standard model from GL(16)|2008;
Blaha, Stephen|From asynchronous logic to the standard model to superflight to the stars|2011;
Blaha, Stephen|Relativistic quantum metaphysics: A first principles basis for the standard model of elementary particles|2009;
Blaha, Stephen|The origin of the standard model: The genesis of four quark and lepton species, parity violation, the electroweak sector, colour SU(3), three visible generations of fermions, and one generation of dark matter
Blaha, Stephen|The standard model's form derived from operator logic, superluminal transformations and GL(16). Relativistic quantum metaphysics|2008;
Blair, D.G., (ed.)|The Detection of gravitational waves|1991;
Blair, D.|Ripples on a cosmic sea: The search for gravitational waves|1998;
Blanchard, Alain, (Ed.)|Frontiers of cosmology|2005;
Blanchet, Luc|General relativity and the spiral of compact binary stars|2005;
Blandford, R., (ed.)|Annual review of astronomy and astrophysics. Vol. 44|2006;
Blandford, Roger, (ed.)|Annual review of astronomy and astrophysics. Vol. 45|2007;
Blandford, Roger, (ed.)|Annual review of astronomy and astrophysics. Vol. 46|2008;
Blandford, Roger, (ed.)|Annual review of astronomy and astrophysics. Vol. 47|2009;
Blas, Harold|Generalized sine-Gordon and massive Thirring models|2004;
Blaschke, D.|Separable Dyson-Schwinger model at zero and finite T|2007;
Blasone, M.|A New perspective in the dark energy puzzle from particle mixing phenomenon|2008;
Blasone, M.|Quantum fields with topological defects|2004;
Blatt, F.J.|Modern physics|1992;
Bleecker, D.|GAUGE THEORY AND VARIATIONAL PRINCIPLES|1981;
Blin- Stoyle, R.J.|Nuclear and particle physics|1991;
Blin-Stoyle, R.J.|Eureka! Physics of particles, matter and the universe|1997;
Blinder, S.M.|Foundations of Quantum Dynamics|1974;
Blome, H.J.|The big bang: Beginning and end of the universe|2004;
Blum, W.|Particle detection with drift chambers|1993;
Blum, Walter|Particle detection with drift chambers|2008;
Bobenko, A.I., (ed.)|Discrete integrable geometry and physics|1999;
Bochicchio, Ivana|Computational Approach to Gravitational Waves Forms in Stellar Systems as Complex Structures through Keplerian Parameters|2009;
Bock, R., (ed.)|HEAVY ION COLLISIONS. VOL. 1|1979;
Bock, R., (ed.)|HEAVY ION COLLISIONS. VOL. 2|1979;
Bock, R.K.|Data analysis techniques for high-energy physics experiments|1990;
Bock, R.K.|The particle detector briefbook|1998;
Bock, R.|Heavy ion research: Accelerator, atomic physics, nuclear physics, nuclear chemistry, applications. (In German)|1993;
Boehm, A.|QUANTUM MECHANICS|1979;
Boehm, F.|PHYSICS OF MASSIVE NEUTRINOS|1987;
Boehm, F.|Physics of massive neutrinos|1992;
Boerner, G.|THE EARLY UNIVERSE - FACTS AND FICTION|1983;
Boerner, G.|The early universe: Facts and fiction|2003;
Bogolyubov, N.N.|General principles of quantum field theory|1990;
Bogolyubov, N.N.|INTRODUCTION TO THE THEORY OF QUANTIZED FIELDS|1980;
Bogolyubov, N.N.|Introduction to Axiomatic Quantum Field Theory|1975;
Bogolyubov, N.N.|QUANTUM FIELDS. (IN GERMAN)|1984;
Bogolyubov, N.N.|QUANTUM FIELDS|1983;
Bogoslovsky, G.Yu.|Theory of locally anisotropic space-time. (In Russian)|1992;
Bohm, A.|DYNAMICAL GROUPS AND SPECTRUM GENERATING ALGEBRAS. VOL. 1, 2|1988;
Bohm, G.|Introduction to statistics and measurement analysis for physicists|2005;
Bohm, M.|Gauge theories of the strong and electroweak interaction|2001;
Bojowald, Martin|Back before the big bang: The complete history of the universe|2009;
Bojowald, Martin|Canonical Relativity and the Dimensionality of the World|2008;
Bojowald, Martin|Quantum cosmology|2006;
Bonch-Osmolovsky, A.G.|Physics of new methods of charged particle acceleration: Collective effects in dense charged particle ensembles|1994;
Bondi, H., (ed.)|The universe unfolding|1998;
Bonetti, A., (ed.)|Cosmic ray, particle and astroparticle physics, a conference in honour of Giuseppe Occhialini, Bruno Pontecorvo and Bruno Rossi, Florence, 11-13 september 1995|1997;
Bonometto, S., (ed.)|Modern cosmology|2002;
Boos, E.G., (ed.)|Interactions of particles and nuclei at high-energies and superhigh-energies. Activity Report of the High-Energy Physics Institute, National Academy of Sciences of Kazakhstan Republic, 1993|1994;
Booss, B.|TOPOLOGY AND ANALYSIS: THE ATIYAH-SINGER INDEX FORMULA AND GAUGE THEORETIC PHYSICS|1985;
Bopp, Fritz W.|NUCLEI, HADRONS, AND ELEMENTARY PARTICLES: AN INTRODUCTION. (IN GERMAN)|1989;
Borchers, H.J.|Translation group and particle representations in quantum field theory|1996;
Bordag, M.|Advances in the Casimir effect|2009;
Boresch, A.|Applications of noncovariant gauges in the algebraic renormalization procedure|1988;
Borissova, Larissa|Fields, vacuum, and the mirror universe|2010;
Borner, G.|Cosmology|2004;
Boslough, J.|BEYOND THE BLACK HOLE. STEPHEN HAWKING'S UNIVERSE|1986;
Botelho, Luiz.C.L.|Domains of bosonic Functional integrals and some applications to the mathematical physics of path integrals and String Theory|2009;
Botet, R.|Universal fluctuations: The phenomenology of hadronic matter|2002;
Bothun, G.|Modern cosmological observations and problems|1998;
Boudet, Roger|Quantum mechanics in the geometry of space-time: Elementary theory|2011;
Boudjema, Fawzi|SUSY Tools for Dark Matter and at the Colliders|2010;
Bourdeau, M.F., (Ed.)|Invited talks, 10th European Cosmic Ray Symposium, Bordeaux, August 25-29, 1986|1986;
Bourrely, C.|HADRON PHYSICS AT HIGH-ENERGIES. 5-9 JUNE 1978, MARSEILLE|1979;
Bourrely, Claude, (Ed.)|Structure of the nucleon at large Bjorken x: 2nd international workshop on the structure of the nucleon at large Bjorken x|2005;
Boutet de Monvel, Anne, (ed.)|Rigorous quantum field theory: A Festschrift for Jacques Bros|2007;
Bouwknegt, P., (ed.)|Geometric analysis and applications to quantum field theory|2002;
Bouwknegt, P., (ed.)|W symmetry|1995;
Bouwknegt, Peter|The W(3) algebra: Modules, semiinfinite cohomology and BV algebras|1995;
Bowen, R.|Equilibrium States and the Ergodic Theory of Anosov Diffeomorphisms|1975;
Bowler, M.G.|Femtophysics: a Short course on particle physics|1990;
Bowler, M.G.|Gravitation and Relativity|1976;
Boyarkin, O.M.|Advanced particle physics. Vol. 1: Particles, fields, and quantum electrodynamics|2011;
Boyd, Robert W., (ed.)|Self-focusing: Past and present: Fundamentals and prospects|2009;
Bradamante, F., (ed.)|Anti-proton - nucleon and anti-proton - nucleus interactions|1990;
Brading, Katherine, (ed.)|Symmetries in physics: Philosophical reflections|2003;
Bradt, Hale|Astronomy methods: A physical approach to astronomical observations|2007;
Braginsky, V.B.|Measurement of Weak Forces in Physics Experiments|1977;
Braginsky, V.B.|THE DETECTION OF SMALL ACCELERATIONS, GRAVITATIONAL ANTENNAS, VERIFICATION OF THE PRINCIPLE OF EQUIVALENCE|1972;
Bramon, Albert|LECTURES ON ELECTROMAGNETIC INTERACTIONS OF HADRONS|1979;
Branco, Gustavo C.|CP Violation|1999;
Brandt, D., (ed.)|CERN Accelerator School, beam diagnostics, Dourdan, France, 28 May - 6 June 2008|2009;
Brandt, D., (ed.)|CERN Accelerator School, small accelerators, Zeegse, The Netherlands, 24 May - 2 June 2005|2006;
Brandt, D., (ed.)|CERN Accelerator School, synchrotron radiation and free-electron lasers, Brunnen, Switzerland, 2-9 July 2003|2005;
Brandt, D., (ed.)|CERN Accelerator School, vacuum in accelerators, Platja d'Aro, Spain, 16-24 May 2006|2007;
Brandt, S.|Statistical and Computational Methods in Data Analysis|1976;
Brandt, Siegmund|Data analysis: Using statistical methods and computer programs|1999;
Bratteli, O.|OPERATOR ALGEBRAS AND QUANTUM STATISTICAL MECHANICS. 1. C* AND W* ALGEBRAS, SYMMETRY GROUPS, DECOMPOSITION OF STATES|1979;
Bratteli, O.|Operator algebras and quantum statistical mechanics. Vol. 2: Equilibrium states. Models in quantum statistical mechanics|1996;
Brau, C.A.|Free electron lasers|1990;
Braunbek, W.|SCIENTISTS AT THE ROOTS OF BEING. THE FANTASTIC WORLD OF ELEMENTARY PARTICLES. (IN GERMAN)|1981;
Bray, Hubert L.|The Penrose inequality|2003;
Brecher, K.|High-Energy Astrophysics and Its Relation to Elementary Particle Physics. Advanced Study Institute Held at Erice, June 16 - July 6, 1972|1974;
Brehm, J.J.|INTRODUCTION TO THE STRUCTURE OF MATTER: A COURSE IN MODERN PHYSICS|1989;
Brein, O.|Quantum effects in Higgs physics at high-energy electron positron and hadron colliders|2003;
Breuer, H.P.|The theory of open quantum systems|2002;
Breuer, R.|Always trouble with the big bang: The Cosmological standard model in a crisis. (In German)|1994;
Brevik, I.|A Brief Review of the Singularities in 4D and 5D Viscous Cosmologies Near the Future Singularity|2008;
Brevik, Iver H.|Viscous cosmology, entropy, and the Cardy-Verlinde formula|2004;
Brezin, E., (ed.)|The Large N expansion in quantum field theory and statistical physics: From spin systems to two-dimensional gravity|1994;
Brihaye, Yves|Solitons on nanotubes and fullerenes as solutions of a modified nonlinear Schrodinger equation|2004;
Brink, L., (ed.)|Physics and mathematics of strings: Memorial volume for Vadim Knizhnik|1990;
Brink, L.|PRINCIPLES OF STRING THEORY|1988;
Brink, Lars|From the Nambu-Goto the sigma-model action, memoirs from long ago|2007;
Brittin, W.E.|MATHEMATICAL METHODS IN THEORETICAL PHYSICS. LECTURES IN THEORETICAL PHYSICS, BOULDER 1971, VOL. 14B|1973;
Brock, Ian C., (ed.)|Physics at the Terascale|2011;
Brocker, B.|Dtv ATLAS on Atomic Physics. Tables and Texts|1976;
Brockman, J.|BIRTH OF THE FUTURE. THE BALANCE OF OUR SCIENTIFIC VIEW OF THE WORLD AT THE THRESHOLD TO THE NEXT MILLENNIUM. (IN GERMAN)|1987;
Brodsky, Stanley J.|International Colloquium on Photon - Photon Collisions in electron - Positron Storage Rings, Paris, College de France, 3-4 September 1973|1974;
Broeckhove, J.|On the regularisation in J-matrix methods|2004;
Bromberg, C.|Introduction to nuclear and particle physics: Solutions manual for second edition|2006;
Bronnikov, K.A.|Horizons in matter: Black hole hair versus Null Big Bang|2009;
Brooks, J.O.|Attributes of the unified field and quantum gravity: On defining the Hubble constant|2000;
Brown, G.E., (ed.)|Hans Bethe and his physics|2006;
Brown, G.E., (ed.)|Selected papers, with commentary, of Tony Hilton Royle Skyrme|1995;
Brown, G.E.|THE NUCLEON-NUCLEON INTERACTION. PART I|1973;
Brown, H.R., (Ed.)|PHILOSOPHICAL FOUNDATIONS OF QUANTUM FIELD THEORY|1988;
Brown, J.David|LOWER DIMENSIONAL GRAVITY|1988;
Brown, L.M., (Ed.)|ELEMENTARY PARTICLE THEORY IN JAPAN, 1935 - 1960: JAPAN-USA COLLABORATION, SECOND PHASE|1988;
Brown, L.M., (ed.)|Renormalization: From Lorentz to Landau (and beyond)|1993;
Brown, L.M., (ed.)|Twentieth century physics. Vols. 1-3|1995;
Brown, L.M.|The origin of the concept of nuclear forces|1996;
Brown, L.S.|Quantum field theory|1992;
Brunetti, Federico, (ed.)|The rings of knowledge: The Italian contribution to the world's largest particle research project at CERN, Geneva|2009;
Bryant, P.J.|The Principles of circular accelerators and storage rings|1993;
Brylinski, J.L.|Loop spaces, characteristic classes and geometric quantization|1993;
Buchbinder, I.L.|Effective action in quantum gravity|1992;
Buchbinder, I.L.|Ideas and methods of supersymmetry and supergravity: A Walk through superspace|1995;
Buchbinder, I.L.|Ideas and methods of supersymmetry and supergravity: Or a walk through superspace|1998;
Buchmuller, W., (ed.)|Quarkonia|1992;
Bucka, H.|NUCLEON PHYSICS. (IN GERMAN)|1981;
Buehrke, Thomas|The mysterious cosmos: Astrophysics and cosmology in the 21st century|2009;
Bulik, Tomasz, (ed.)|Astrophysical sources of high-energy particles and radiation, Torun, Poland, 20-24 June 2005|2005;
Bullough, R.K.|SOLITONS|1980;
Buras, A.J., (ed.)|Heavy flavors|1992;
Buras, A.J., (ed.)|Heavy flavours II|1998;
Buras, B.|EUROPEAN SYNCHROTRON RADIATION FACILITY. SUPPLEMENT III: INSTRUMENTATION|1979;
Burbidge, G., (Ed.)|ANNUAL REVIEW OF ASTRONOMY AND ASTROPHYSICS. VOL. 21|1984;
Burbidge, G., (Ed.)|ANNUAL REVIEW OF ASTRONOMY AND ASTROPHYSICS. VOL. 22|1985;
Burbidge, G., (Ed.)|ANNUAL REVIEW OF ASTRONOMY AND ASTROPHYSICS. VOL. 25|1987;
Burbidge, G., (Ed.)|ANNUAL REVIEW OF ASTRONOMY AND ASTROPHYSICS. VOL. 27|1989;
Burbidge, G., (ed.)|Annual review of astronomy and astrophysics. Vol. 29|1991;
Burbidge, G., (ed.)|Annual review of astronomy and astrophysics. Vol. 33|1995;
Burbidge, G., (ed.)|Annual review of astronomy and astrophysics. Vol. 41|2003;
Burcham, W.E.|Nuclear and particle physics|1995;
Burge, E.J.|Atomic Nuclei and their Particles|1977;
Burgess, C.P.|Quantum gravity and precision tests|2006;
Burgess, C.P.|The standard model: A primer|2007;
Burgess, M.|Classical covariant fields|2002;
Burmester, Ralph|Die vier Leben einer Maschine Das 500-MeV Elektronen-Synchrotron der Universitat Bonn = The four lives of a machine, the 500-MeV electron synchrotron of the University of Bonn|2010;
Burnel, Andre, (ed.)|Noncovariant gauges in canonical formalism|2008;
Burt, P.B.|Quantum Mechanics and Nonlinear Waves|1981;
Buts, V.A.|The theory of coherent radiation by intense electron beams|2006;
Butterfield, J., (ed.)|The arguments of time|2006;
Butterfield, Jeremy|On Hamilton-Jacobi theory as a classical root of quantum theory|2003;
Byers, Nina, (ed.)|Out of the shadows, contributions of twentieth-century women to physics|2006;
Byrne, J.|Neutrons, nuclei and matter|1993;
Bystricky, J.|Nucleon Nucleon Scattering Data. 1.|1978;
Bystricky, J.|Nucleon Nucleon Scattering Data. 2.|1978;
Bystricky, J.|Numerical Data and Functional Relationships in Science and Technology. Group I: Nuclear and Particle Physics. Vol. 9: Elastic and Charge Exchange Scattering of Elementary Particles. a: Nucleon Nucleon and Kaon
Bytsenko, A.A.|Analytic aspects of quantum fields|2003;
Cabannes, H.|Pade Approximants Method and Its Applications to Mechanics|1976;
Cabibbo, N., (ed.)|Lepton physics at CERN and Frascati|1995;
Cabibbo, N.|Particle Physics Summer School, Gif-Sur-Yvette, 2-21 September 1974. 1. Weak Interactions|1974;
Cadogan, P.H.|FROM QUARK TO QUASAR. NOTES ON THE SCALE OF THE UNIVERSE|1986;
Caglioti, G.|Symmetry breaking and perception: Examples from experience. (In German)|1990;
Cahill, Reginald T.|'Dark matter' as a quantum foam in-flow effect|2004;
Cahill, Reginald T.|Process physics: From information theory to quantum space and matter|2003;
Cahn, R.N., (Ed.)|E+ E- ANNIHILATION: NEW QUARKS AND LEPTONS|1985;
Cahn, R.N.|SEMISIMPLE LIE ALGEBRAS AND THEIR REPRESENTATIONS|1985;
Cahn, R.N.|THE EXPERIMENTAL FOUNDATIONS OF PARTICLE PHYSICS|1989;
Calabretta, L.|Preliminary Design Study of High-Power $H_2^+$ Cyclotrons for the DAE$\delta\$ALUS Experiment|2011;
Calder, N.|EINSTEIN'S UNIVERSE. (GERMAN TRANSLATION)|1980;
Caldwell, David O., (Ed.)|Current aspects of neutrino physics|2001;
Callender, C., (ed.)|Physics meets philosophy at the Planck scale: Contemporary theories in quantum gravity|2001;
Calogero, F.|SPECTRAL TRANSFORM AND SOLITONS. TOOLS TO SOLVE AND INVESTIGATE NONLINEAR EVOLUTION EQUATIONS. VOL. 1|1982;
Cannata, F.|GIANT RESONANCE PHENOMENA IN INTERMEDIATE-ENERGY NUCLEAR REACTIONS|1980;
Cao, T.Y.|Conceptual developments of 20th century field theories|1997;
Caparthy, J., (ed.)|Muons: New research|2005;
Capri, A.Z.|Relativistic quantum mechanics and introduction to quantum field theory|2002;
Capri, Anton Z.|From quanta to quarks: More anecdotal history of physics|2007;
Cardone, Fabio|A Geometrical meaning to the electron mass from breakdown of Lorentz invariance|2005;
Cardone, Fabio|A New pseudo-Kaluza-Klein scheme for geometrical description of interactions|2005;
Cardone, Fabio|Deformed spacetime. Geometrizing interactions in four and five dimensions|2007;
Cardy, John L., (Ed.)|FINITE SIZE SCALING|1988;
Cardy, John L.|Boundary conformal field theory|2004;
Cardy, John L.|Scaling and renormalization in statistical physics|1996;
Carey, David C.|THE OPTICS OF CHARGED PARTICLE BEAMS|1987;
Carinena, Jose F.|Applications of Lie systems in quantum mechanics and control theory|2003;
Carlip, Steven|Quantum gravity in 2+1 dimensions|1998;
Carmeli, M., (ed.)|Cosmological special relativity: The large scale structure of space, time and velocity|2002;
Carmeli, M., (ed.)|Gravitation: SL(2,C) gauge theory and conservation laws|1990;
Carmeli, M.|CLASSICAL FIELDS: GENERAL RELATIVITY AND GAUGE THEORY|1982;
Carmeli, M.|Classical fields: General relativity and gauge theory|2001;
Carmeli, M.|GAUGE FIELDS: CLASSIFICATION AND EQUATIONS OF MOTION|1989;
Carmeli, M.|Group theory and general relativity: Representations of the Lorentz group and their applications to the gravitational field|2000;
Carmeli, M.|Theory of spinors: An introduction|2000;
Carmeli, Moshe, (ed.)|Relativity: Modern large-scale spacetime structure of the cosmos|2008;
Carmeli, Moshe|Cosmological relativity: The special and general theories for the structure of the universe|2006;
Carr, Bernard J., (Ed.)|Universe or multiverse?|2007;
Carreras, R.|HOW ENERGY BECOMES MATTER... A FIRST LOOK AT THE WORLD OF PARTICLES|1983;
Carrigan, Richard A., (ed.)|Particle physics in the cosmos: Readings from Scientific American Magazine|1989;
Carroll, R.W.|MATHEMATICAL PHYSICS|1988;
Carroll, R.W.|Topics in soliton theory|1991;
Carroll, Robert|Fluctuations, Information, Gravity and the Quantum Potential|2006;
Carroll, Robert|On the Quantum potential|2007;
Carroll, Sean M.|Spacetime and geometry: An introduction to general relativity|2004;
Carroll, Sean|From Eternity to Here: The Quest for the Ultimate Theory of Time|2010;
Carruthers, P., (Ed.)|HADRONIC MULTIPARTICLE PRODUCTION|1988;
Cartas-Fuentevilla, R.|Topological terms and the global symplectic geometry of the phase space in string theory|2004;
Carter, Brandon|Mechanics and equilibrium geometry of black holes, membranes, and strings|2004;
Cartier, Pierre|Supermanifolds: Application to supersymmetry|2002;
Castell, L.|QUANTUM THEORY AND THE STRUCTURES OF TIME AND SPACE. VOL. 3. PAPERS PRESENTED AT A CONFERENCE HELD IN TUTZING, JULY 1978|1979;
Castell, L.|Quantum Theory and the Structures of Time and Space. 2. Feldafing, 1976, in Memoriam Werner Heisenberg|1977;
Castell, L.|Quantum Theory and the Structures of Time and Space. Feldafing Conference, July 1974|1975;
Castell, L.|Time, quantum and information|2004;
Castellani, L.|Supergravity and superstrings: A Geometric perspective. Vol. 1: Mathematical foundations|1991;
Castellani, L.|Supergravity and superstrings: A Geometric perspective. Vol. 2: Supergravity|1991;
Castellani, L.|Supergravity and superstrings: A Geometric perspective. Vol. 3: Superstrings|1991;
Cegla, W., (ed.)|Twentyfive + 1 Karpacz Winter Schools in Theoretical Physics 1964 - 1990: List of lectures|1990;
Celenza, L.S.|Relativistic nuclear physics: Theories of structure and scattering|1986;
Ceolin, Milla Baldo, (ed.)|Eleventh International Workshop on Neutrino Telescopes, Venezia, February 22-25, 2005|2005;
Ceolin, Milla Baldo, (ed.)|Fourth NO-VE International Workshop on Neutrino Oscillations in Venice : Ten years after the neutrino oscillations!! : Venezia, April 15-18, 2008, Istituto Veneto di Scienze, Lettere ed Arti, Campo
Cerny, J.|Nuclear Spectroscopy and Reactions. Part A|1974;
Cerny, J.|Nuclear Spectroscopy and Reactions. Part B|1974;
Chadan, K.|Inverse Problems in Quantum Scattering Theory|1977;
Chaichian, M.|INTRODUCTION TO GAUGE FIELD THEORIES|1984;
Chaichian, M.|Introduction to quantum groups|1996;
Chaichian, M.|Path integrals in physics. Vol. 1: Stochastic processes and quantum mechanics|2001;
Chaichian, M.|Path integrals in physics. Vol. 2: Quantum field theory, statistical physics and other modern applications|2001;
Chaichian, M.|Symmetries in quantum mechanics: From angular momentum to supersymmetry|1998;
Chakravarty, Sudip|Quantum Phase Transition, Dissipation, and Measurement|2009;
Chandrasekhar, Subrahmanyan|Selected papers S. Chandrasekhar. Vol. 5: Relativistic astrophysics|1990;
Chandrasekhar, Subrahmanyan|The mathematical theory of black holes|1985;
Chandrasekharan, K., (Ed.)|HERMANN WEYL 1885 - 1985: CENTENARY LECTURES|1986;
Chang, S.J.|Introduction to quantum field theory|1990;
Chao, A.W., (ed.)|Handbook of accelerator physics and engineering|1999;
Chao, A.W.|Physics of collective beam instabilities in high-energy accelerators|1993;
Chao, Alexander W., (ed.)|Reviews of accelerator science and technology. Vol. 1|2008;
Chao, Alexander W., (ed.)|Reviews of accelerator science and technology. Vol. 2: Medical applications of accelerators|2009;
Charap, J.M.|Explaining the universe: The new age of physics|2002;
Chari, V.|A guide to quantum groups|1994;
Charon, J.E.|COMPLEX RELATIVITY: UNIFYING ALL FOUR PHYSICAL INTERACTIONS|1988;
Charpak, Georges, (ed.)|Research on particle imaging detectors|1995;
Chavanis, Pierre-Henri|Statistical mechanics of two-dimensional vortices and stellar systems|2002;
Chen, Jie-Quan|Group representation theory for physicists|2002;
Chen, Jin-Quan|TABLES OF THE CLEBSCH-GORDAN, RACAH AND SUBDUCTION COEFFICIENTS OF SU(N) GROUPS|1987;
Cheng, D.C.|ELEMENTARY PARTICLE PHYSICS. AN INTRODUCTION|1979;
Cheng, Hung|EXPANDING PROTONS: SCATTERING AT HIGH-ENERGIES|1987;
Cheng, Jingquan|The principles of astronomical telescope design|2009;
Cheng, K.S., (ed.)|Cosmic gamma-ray sources|2004;
Cheng, K.S., (ed.)|Cosmic gamma-ray sources|2004;
Cheng, T.P.|GAUGE THEORY OF ELEMENTARY PARTICLE PHYSICS|1985;
Cheng, T.P.|Gauge theory of elementary particle physics: Problems and solutions|2000;
Cheng, T.P.|Relativity, gravitation, and cosmology: A basic introduction|2005;
Cherednik, I.|Basic methods of soliton theory|1996;
Chern, S.S.|Lectures on differential geometry|1999;
Chester, M.|Particles. An Introduction to Particle Physics|1978;
Choi, Kang-Sin|Quarks and leptons from orbifolded superstring|2006;
Choi, Kiwoon, (ed.)|Particles, strings, and cosmology, 11th International Symposium on Particles, Strings and Cosmology, PASCOS 2005, Gyeongju, Korea, 30 May - 4 June 2005|2005;
Chow, Tai L.|Gravity, black holes, and the very early universe: An introduction to general relativity and cosmology|2008;
Chow, Y.|General Theory of Lie Algebras. 1.|1978;
Chow, Y.|General Theory of Lie Algebras. 2.|1978;
Chow, Y.|Modern Abstract Algebra. 1. Monoids, Groups and Rings|1976;
Chow, Y.|Modern Abstract Algebra. 2. Two-Modules, Linear Endomorphisms and Algebras|1976;
Christe, P.|Introduction to conformal invariance and its applications to critical phenomena|1993;
Christensen, S.M., (Ed.)|QUANTUM THEORY OF GRAVITY. ESSAYS IN HONOR OF THE 60TH BIRTHDAY OF BRYCE S. DEWITT|1984;
Christodoulou, Demetrios|The Formation of Black Holes in General Relativity|2008;
Chrusciel, P.T.|Hamiltonian field theory in the radiating regime|2002;
Chu, Chong-Sun|Non-commutative geometry from strings|2005;
Chubykalo, A.E., (ed.)|Instantaneous action at a distance in modern physics: 'Pro' and 'contra'|1999;
Chupp, E.L.|Gamma-Ray Astronomy. Nuclear Transition Region|1976;
Ciocci, F.|Insertion devices for synchrotron radiation and free electron laser|2000;
Cittolin, S.|The compact muon solenoid experiment at the LHC: Images of assembly and installation|2008;
Ciufolini, I., (ed.)|Gravitation: From the Hubble length to the Planck length|2005;
Ciufolini, I., (ed.)|Gravitational waves|2001;
Ciufolini, Ignazio, (ed.)|General relativity and John Archibald Wheeler|2010;
Ciulli, S., (ed.)|Rigorous methods in particle physics|1990;
Civitarese, Osvaldo, (ed.)|Workshop on Calculation of Double Beta Decay Matrix Elements (MEDEX'07) Prague, Czech Republic, 11-14 June 2007|2007;
Clarke, C.J.S.|The Analysis of space-time singularities|1994;
Clarke, J.A.|The science and technology of undulators and wigglers|2004;
Clay, Roger W.|Cosmic bullets: High energy particles in astrophysics|1998;
Clifton, R., (ed.)|Perspectives on quantum reality: Nonrelativistic, relativistic, and field theoretic|1996;
Cline, D.B., (ed.)|Weak neutral currents: The discovery of the electro-weak force|1997;
Cline, D.B.|Unification of Elementary Forces and Gauge Theories. Papers Presented at the Ben Lee Memorial International Conference on Parity Nonconservation Weak Neutral Currents and Gauge Theories, Fermi National Accelerator
Close, F.E.|AN INTRODUCTION TO QUARKS AND PARTONS|1979;
Close, F.|Lucifer's legacy: The meaning of asymmetry|2000;
Close, F.|Particle physics: A very short introduction|2004;
Close, F.|THE COSMIC ONION. QUARKS AND THE NATURE OF THE UNIVERSE|1984;
Close, F.|THE PARTICLE EXPLOSION|1987;
Close, F.|The new cosmic onion: Quarks and the nature of the universe|2007;
Close, F.|The particle odyssey: A journey to the heart of the matter. The particle explosion|2002;
Close, Frank, (ed.)|Electromagnetic interactions and hadronic structure|2007;
Close, Frank|Antimatter|2009;
Close, Frank|Nothing: A very short indroduction|2009;
Close, Frank|The void|2007;
Coecke, Bob|Introducing categories to the practicing physicist|2008;
Cohen, N.|GRAVITY'S LENS: VIEWS OF THE NEW COSMOLOGY|1988;
Cohen-Tannoudji, C.|PHOTONS AND ATOMS. INTRODUCTION TO QUANTUM ELECTRODYNAMICS. (IN FRENCH)|1987;
Cohen-Tannoudji, G.|MATTER, SPACE, TIME. THE LOGIC OF ELEMENTARY PARTICLES. (IN FRENCH)|1986;
Colangelo, P., (ed.)|IFAE 2009 Incontri di Fisica delle Alte Energie, Bari, Italy 15-17 April 2009|2010;
Colangelo, Pietro, (ed.)|QCD@work 2005, International Workshop on Quantum Chromodynamics Theory and Experiment, Conversano, Bari, Italy, 16-20 June 2005|2006;
Coles, P.|Cosmology: The Origin and evolution of cosmic structure|1995;
Coles, P.|From cosmos to chaos: The science of unpredictability|2006;
Coley, A.A.|Dynamical systems and cosmology|2003;
Collet, Pierre|Concepts and results in chaotic dynamics, a short course|2006;
Collins, H.|Gravity's shadow: The search for gravitational waves|2004;
Collins, John C.|RENORMALIZATION. AN INTRODUCTION TO RENORMALIZATION, THE RENORMALIZATION GROUP, AND THE OPERATOR PRODUCT EXPANSION|1984;
Collins, John|Foundations of perturbative QCD|2011;
Collins, P.D.B.|An Introduction to Regge Theory and High-Energy Physics|1977;
Collins, P.D.B.|HADRON INTERACTIONS|1984;
Collins, P.D.B.|PARTICLE PHYSICS AND COSMOLOGY|1989;
Comay, E.|A Regular theory of magnetic monopoles and its implications|2004;
Commins, E.D.|WEAK INTERACTIONS OF LEPTONS AND QUARKS|1983;
Compton, Chris|Single crystal and large grain niobium research at Michigan State University|2007;
Conn, G.K.T.|Essays in Physics, Vol. 6|1976;
Connes, Alain|Noncommutative geometry|1994;
Constantinescu, F.|Geometric and algebraic methods of physics: Supermanifolds and Virasoro algebras. (In German)|1994;
Conte, M.|An Introduction to the physics of particle accelerators|1991;
Conte, Mario|An introduction to the physics of particle accelerators|2008;
Contopoulos, G.|COSMOLOGY. THE STRUCTURE AND EVOLUTION OF THE UNIVERSE|1987;
Cooley, W.W.|MULTIVARIATE DATA ANALYSIS|1980;
Cooper, F.|Recent Advances in Particle Physics. Conference, New York, 15-17 March 1973|1974;
Cooper, F.|Supersymmetry in quantum mechanics|2001;
Cooper, N.G., (Ed.)|PARTICLE PHYSICS. A LOS ALAMOS PRIMER|1988;
Cooper, P.N.|Introduction to nuclear radiation detectors|2011;
Coquereaux, Robert|Riemannian geometry, fiber bundles, Kaluza-Klein theories and all that|1988;
Cornell, J., (Ed.)|BUBBLES, VOIDS, AND BUMPS IN TIME: THE NEW COSMOLOGY|1989;
Cornell, J., (ed.)|The New cosmology: On dark matter, GUTs and superclusters. (In German) (Bubbles, voids, and bumps in time)|1991;
Cornell, J.C., (ed.)|A handbook on interdisciplinary use of European nuclear physics facilities|2004;
Cornwell, J.F.|GROUP THEORY IN PHYSICS. VOL. 1|1985;
Cornwell, J.F.|GROUP THEORY IN PHYSICS. VOL. 2|1985;
Cornwell, J.F.|GROUP THEORY IN PHYSICS. VOL. 3: SUPERSYMMETRIES AND INFINITE DIMENSIONAL ALGEBRAS|1989;
Cornwell, J.F.|Group theory in physics: An introduction|1997;
Corson, E.M.|INTRODUCTION TO TENSORS, SPINORS, AND RELATIVISTIC WAVE EQUATIONS. RELATION STRUCTURE|1982;
Corsten, C.J.A.|RESONANCE AND COUPLING EFFECTS IN CIRCULAR ACCELERATORS|1982;
Costa De Beauregard, O.|TIME, THE PHYSICAL MAGNITUDE|1987;
Costa, S.|Photonuclear Reactions. 1. International School on Electronuclear and Photonuclear Reactions, Erice, Italy 1976|1976;
Costa, S.|Photonuclear Reactions. 2. International School on Electronuclear and Photonuclear Reactions, Erice, Italy 1976|1977;
Cotaescu, Ion I.|Symmetries and supersymmetries of the Dirac operators in curved spacetimes|2004;
Cottingham, W.N.|AN INTRODUCTION TO NUCLEAR PHYSICS|1986;
Cottingham, W.N.|An introduction to the standard model of particle physics|2007;
Coughlan, G.D.|The Ideas of particle physics: An Introduction for scientists|1994;
Coughlan, G.|Elementary particles: An Introduction for scientists. (In German)|1991;
Cowan, G.|Statistical data analysis|1998;
Cox, D.A.|Mirror symmetry and algebraic geometry|2000;
Craigie, N.S., (Ed.)|THEORY AND DETECTION OF MAGNETIC MONOPOLES IN GAUGE THEORIES. A COLLECTED SET OF LECTURE NOTES|1986;
Crease, R.P., (ed.)|Making physics. A biography of Brookhaven National Laboratory, 1946-1972|1999;
Creighton, Jolien D.E.|Gravitational-wave physics and astronomy: An introduction to theory, experiment and data analysis|2011;
Creswick, R.J.|Introduction to renormalization group methods in physics|1992;
Creutz, M.|QUARKS, GLUONS AND LATTICES|1984;
Creutz, Michael|Yang-Mills fields and the lattice. Chapter 1|2004;
Crowell, L.B.|Quantum fluctuations of spacetime|2005;
Croxton, C.A.|Introductory Eigen Physics. An Approach to the Theory of Fields|1974;
Crozon, M.|Elementary particles. (In French)|1994;
Crozon, M.|The particle universe|1999;
Crumeyrolle, A.|Orthogonal and symplectic Clifford algebras: Spinor structures|1990;
Csanad, M.|Systematics of identified hadron spectra at PHENIX|2005;
Csernai, L.P., (ed.)|Relativistic heavy ion physics|1991;
Csernai, L.P., (ed.)|Relativistic heavy ion physics|1991;
Csernai, L.P.|Introduction to relativistic heavy ion collisions|1994;
Curtis, M.L.|MATRIX GROUPS|1979;
Curtis, W.D.|DIFFERENTIAL MANIFOLDS AND THEORETICAL PHYSICS|1986;
Cushing, J.T.|Theory construction and selection in modern physics: The S matrix|1990;
Cvitanovic, Predrag|CLASSICS ILLUSTRATED: GROUP THEORY. PART 1|1984;
Cvitanovic, Predrag|FIELD THEORY|1983;
Cvitanovic, Predrag|Group theory: Birdtracks, Lie's and exceptional groups|2008;
Cycon, H.L.|SCHRODINGER OPERATORS: WITH APPLICATION TO QUANTUM MECHANICS AND GLOBAL GEOMETRY|1987;
Czyz, Wieslaw|Interactions of High-Energy Particles with Nuclei|1975;
D'Agostini, G.|Bayesian reasoning in data analysis: A critical introduction|2003;
D'Amico, Flavio, (ed.)|The transient Milky Way : A Perspective for MIRAX, Sao Jose dos Campos, Brazil, 7-9 December 2005|2006;
D'Eath, P.D.|Black holes: Gravitational interactions|1996;
D'Eath, P.D.|Supersymmetric quantum cosmology|1996;
D'Souza, I.A.|Preons: Models of leptons, quarks and gauge bosons as composite objects|1992;
Dabrowski, L.|GROUP ACTIONS ON SPINORS: LECTURES AT THE UNIVERSITY OF NAPLES|1988;
Dadhich, N., (Ed.)|A RANDOM WALK IN RELATIVITY AND COSMOLOGY. ESSAYS IN HONOR OF P.C. VAIDYA AND A.K. RAYCHAUDHURI|1985;
Dalarsson, M.|Tensor calculus, relativity, and cosmology: A first course|2005;
Dalitz, R.H., (Ed.)|QUARKS AND LEPTONS: THE NEW ELEMENTARY PARTICLES?|1986;
Damgaard, P.H., (Ed.)|STOCHASTIC QUANTIZATION|1988;
Daniel, R.R., (ed.)|Essays on particles and fields. M.G.K. Menon Festschrift|1989;
Danos, M.|METHODS IN RELATIVISTIC NUCLEAR PHYSICS|1984;
Darling, R.W.R.|Differential forms and connections|1995;
Das, Anadi|The Special theory of relativity: A Mathematical exposition|1993;
Das, Ashok K.|Field theory: A Path integral approach|1993;
Das, Ashok K.|Finite temperature field theory|1997;
Das, Ashok K.|Integrable models|1989;
Das, Ashok K.|Introduction to nuclear and particle physics|1995;
Das, Ashok|Lectures on quantum field theory|2008;
Das, S.|Materia: An elementary treatise on the symmetry and the structure of matter|2004;
Dass, T.|Symmetries, gauge fields, strings and fundamental interactions. Vol. 1: Mathematical techniques in gauge and string theories|1993;
Dauber, P.M.|The three big bangs: Comet crashes, exploding stars, and the creation of the universe|1997;
Dauxois, Thierry|Physics of solitons|2006;
Davidson, R.C.|Physics of intense charged particle beams in high energy accelerators|2001;
Davies, E.B.|Quantum Theory of Open Systems|1976;
Davies, P.C.W., (Ed.)|SUPERSTRINGS: A THEORY OF EVERYTHING?|1988;
Davies, P.C.W., (Ed.)|THE NEW PHYSICS|1989;
Davies, P.C.W.|How to build a time machine|2002;
Davies, P.C.W.|SUPERFORCE: THE SEARCH FOR A GRAND UNIFIED THEORY OF NATURE|1985;
Davies, P.C.W.|THE FORCES OF NATURE|1979;
Davies, P.C.W.|THE SEARCH FOR GRAVITY WAVES|1989;
Davies, Paul C.W.|About time: Einstein's unfinished revolution|1995;
Davies, Paul C.W.|GOD AND THE NEW PHYSICS|1984;
Davies, Paul C.W.|On the way to the world formula: Superstrings, chaos, complexity - and what then? The Matter myth: Dramatic discoveries that challenge our understanding of physical reality. (In German)|1994;
Davies, Paul C.W.|THE ULTIMATE FORCE. LOOKING FOR A UNIFIED THEORY OF NATURE. (IN GERMAN)|1987;
Davies, Paul|The Goldilocks enigma: Why is the universe just right for life?|2007;
Davis, E.A.|J.J. Thomson and the discovery of the electron|1997;
De Groot, S.R.|Relativistic Kinetic Theory. Principles and Applications|1980;
De Sabbata, V.|INTRODUCTION TO GRAVITY|1986;
De Wit, B.|FIELD THEORY IN PARTICLE PHYSICS. VOL. 1|1986;
DeBenedictis, A.|Developments in black hole research: Classical, semi-classical, and quantum|2007;
DeGrand, Thomas|Lattice methods for quantum chromodynamics|2006;
DeWitt, Bryce S.|Dynamical theory of groups and fields|1965;
DeWitt, Bryce S.|SUPERMANIFOLDS|1985;
DeWitt, Bryce S.|Supermanifolds|1992;
DeWitt, Bryce S.|The Everett-Wheeler interpretation of quantum mechanics|1968;
DeWitt, Bryce S.|The global approach to quantum field theory. Vol. 1, 2|2003;
DeWitt-Morette, Cecile|The pursuit of quantum gravity: Memoirs of Bryce DeWitt from 1946 to 2004|2011;
Dean, N.W.|Introduction to the Strong Interactions|1976;
Degrange, B., (ed.)|Towards a network of atmospheric Cherenkov detectors VII, Palaiseau, April 27-29, 2005|2005;
Delaney, C.F.G.|Radiation detectors: Physical principles and applications|1992;
Deligne, P., (Ed.)|Quantum fields and strings: A course for mathematicians. Vol. 1, 2|1999;
Deloff, A.|Fundamentals in hadronic atom theory|2003;
Delphenich, D.H.|Nonlinear optical analogies in quantum electrodynamics|2006;
Delsanto, P.P., (ed.)|New perspectives on problems in classical and quantum physics: A festschrift in honor of Herbert Ueberall. Part I: Radiation and solid state physics, nuclear and high-energy physics, mathematical
Demtroeder, W.|Experimental physics. Vol. 4: Nuclear, particle and astrophysics|1998;
Derdzinski, A.|Geometry of the standard model of elementary particles|1992;
Dermer, Charles D.|High energy radiation from black holes: Gamma rays, cosmic rays and neutrinos|2009;
Desai, Bipin R.|Quantum mechanics with basic field theory|2010;
Desch, Klaus|Model independent determination of the top Yukawa coupling from LHC and LC|2004;
Devenish, R.|Deep inelastic scattering|2004;
Dey, M.|Nuclear and particle physics: The Changing interface|1994;
Di Domenico, Antonio|Handbook on neutral kaon interferometry at a Phi-factory|2007;
Di Francesco, P.|Conformal field theory|1997;
Di Giacomo, A.|Selected problems in theoretical physics (with solutions)|1994;
Diakonov, Dmitri, (ed.)|Subtleties in quantum field theory: Lev Lipatov Festschrift|2010;
Dickey, L.A.|Soliton equations and Hamiltonian systems|1991;
Dikansky, N.|The Physics of intense beams and storage rings|1995;
Dilao, R.|Nonlinear dynamics in particle accelerators.|1996;
Dillenburg, D., (ed.)|Current topics in nuclear physics and quantum field theory: Festschrift for Th.A.J. Maris|1994;
Dimock, Jonathan|Quantum mechanics and quantum field theory: A mathematical primer|2011;
Dine, M., (Ed.)|STRING THEORY IN FOUR-DIMENSIONS|1988;
Dine, M.|Supersymmetry and string theory: Beyond the standard model|2007;
Dineykhan, M.|Oscillator representation in quantum physics|1995;
Dirac, Paul A.M.|Directions in Physics|1978;
Dirac, Paul A.M.|Spinors in Hilbert Space|1974;
Dissertori, G.|High energy experiments and theory|2003;
Dittrich, W.|Classical and quantum dynamics: From classical paths to path integrals|1992;
Dittrich, W.|EFFECTIVE LAGRANGIANS IN QUANTUM ELECTRODYNAMICS|1985;
Dittrich, W.|Probing the quantum vacuum. Perturbative effective action approach in quantum electrodynamics and its application|2000;
Dittrich, Walter|SELECTED TOPICS IN GAUGE THEORIES|1985;
Dobado, A.|Effective lagrangians for the standard model|1997;
Dobrev, V.K.|Harmonic Analysis on the n-Dimensional Lorentz Group and Its Application to Conformal Quantum Field Theory|1977;
Dobrotin, N.A., (ed.)|INELASTIC HADRON HADRON AND HADRON NUCLEUS INTERACTIONS. (IN RUSSIAN)|1980;
Dobrushin, R.L., (ed.)|Topics in statistical and theoretical physics: F.A. Berezin memorial volume|1996;
Dodd, J.E.|THE IDEAS OF PARTICLE PHYSICS. AN INTRODUCTION FOR SCIENTISTS|1984;
Dodd, R.K.|SOLITONS AND NONLINEAR WAVE EQUATIONS|1982;
Dodelson, Scott|Modern cosmology|2003;
Dokshitzer, Yuri L.|Basics of perturbative QCD|1991;
Dolgov, A.D.|Basics of modern cosmology|1991;
Dolling, L.M., (ed.)|The tests of time: Readings in the development of physical theory|2003;
Domb, C., (Ed.)|PHASE TRANSITIONS AND CRITICAL PHENOMENA. VOL. 10|1986;
Domb, C., (Ed.)|PHASE TRANSITIONS AND CRITICAL PHENOMENA. VOL. 11|1987;
Domb, C., (Ed.)|PHASE TRANSITIONS AND CRITICAL PHENOMENA. VOL. 9|1985;
Domb, C.|PHASE TRANSITIONS AND CRITICAL PHENOMENA. VOL 2|1980;
Domb, C.|PHASE TRANSITIONS AND CRITICAL PHENOMENA. VOL. 1|1980;
Domb, C.|Phase Transitions and Critical Phenomena. 3.|1974;
Domb, C.|Phase Transitions and Critical Phenomena. 6.|1976;
Domb, C.|Phase Transitions and Critical Phenomena. Vol. 5a|1976;
Domb, C.|Phase Transitions and Critical Phenomena. Vol. 5b|1976;
Dominguez-Tenreiro, R.|AN INTRODUCTION TO COSMOLOGY AND PARTICLE PHYSICS|1988;
Domokos, G.|The Second Johns Hopkins Workshop on Current Problems in High-Energy Theory, Apr 12-22, 1978|1978;
Donaldson, S.K.|Floer homology groups in Yang-Mills theory|2002;
Donnachie, A.|Electromagnetic Interactions of Hadrons. 1.|1978;
Donnachie, A.|Electromagnetic Interactions of Hadrons. 2.|1978;
Donnachie, S.|Pomeron physics and QCD|2002;
Donoghue, J.F.|Dynamics of the standard model|1992;
Doring, Andreas|`What is a Thing?': Topos Theory in the Foundations of Physics|2008;
Dorman, L.I.|Cosmic ray interactions, propagation, and acceleration in space plasmas|2006;
Dosch, Hans Gunter|Quantum field theory in a semiotic perspective|2005;
Doughty, N.A.|Lagrangian interaction: An Introduction to relativistic symmetry in electrodynamics and gravitation|1996;
Draayer, Jerry P.|Light nuclei in the framework of the symplectic no-core shell model|2007;
Drazin, P.G.|SOLITONS|1984;
Drazin, P.G.|Solitons: An Introduction|1989;
Drechsler, W.|Fiber Bundle Techniques in Gauge Theories. Lectures at the University of Texas, Austin|1977;
Drees, M.|Theory and phenomenology of sparticles: An account of four-dimensional N=1 supersymmetry in high energy physics|2004;
Drees, W.B.|Beyond the big bang: Quantum cosmologies and God|1990;
Droescher, W.|Structures of the physical world and of its non-material side. (In German)|1996;
DuVernois, M.A., (ed.)|Topics in cosmic-ray astrophysics|2000;
Dubin, D.A.|Mathematical aspects of Weyl quantization and phase|2000;
Duck, I.|Pauli and the spin-statistics theorem|1997;
Duff, B.G.|FUNDAMENTAL PARTICLES. AN INTRODUCTION TO QUARKS AND LEPTONS|1986;
Duff, Michael J., (ed.)|The world in eleven-dimensions: Supergravity, supermembranes and M theory|1999;
Duffey, G.H.|Applied group theory: For physicists and chemists|1992;
Duffy, Michael C., (ed.)|Ether space-time and cosmology. Vol. 1: Modern ether concepts relativity and geometry|2008;
Duffy, Michael C., (ed.)|Ether space-time and cosmology. Vol. 2: New insights into a key physical medium|2009;
Duke, P.J.|Synchrotron radiation: Production and properties|2000;
Dumitrache, Cristiana, (ed.)|Fifty years of Romanian astrophysics, Bucharest, Romania 27-30 September 2006|2007;
Dumitrache, Cristiana, (ed.)|Flows, boundaries, interactions, Sinaia, Romania 3-5 May 2007|2007;
Dunajski, Maciej|Solitons, instantons, and twistors|2010;
Dunne, Gerald V.|Heisenberg-Euler effective Lagrangians: Basics and extensions|2004;
Dunne, Gerald V.|Selfdual Chern-Simons theories|1995;
Duplantier, B., (ed.)|Vacuum energy, renormalization: Poincare Seminar 2002|2003;
Duplij, Steven, (Ed.)|Concise Encyclopedia of Supersymmetry: And Noncommutative Structures in Mathematics and Physics|2003;
Dupuy, J.E.|INTRODUCTION TO COSMIC RADIATION PHYSICS. VOL. II. (IN FRENCH) LOW-ENERGY PHYSICS|1972;
Durand, D.|Nuclear dynamics in the nucleonic regime|2001;
Durgaprasad, N., (ed.)|18th International Cosmic Ray Conference Bangalore, India, August 22 to September 3, 1983|1983;
Durrani, S.A.|SOLID STATE NUCLEAR TRACK DETECTION. PRINCIPLES, METHODS AND APPLICATIONS|1987;
Dutra, S.M.|Cavity quantum electrodynamics: The strange theory of light in a box|2005;
Dvoeglazov, V.V., (ed.)|Photon and Poincare group|1999;
Dvoeglazov, V.V., (ed.)|Photon: Old problems in light of new ideas|2000;
Dvoeglazov, V.V., (ed.)|Relativity, gravitation, cosmology|2004;
Dyson, F.J.|Selected papers of Freeman Dyson with commentary|1996;
Earman, J.|Bangs, crunches, whimpers, and shrieks: Singularities and acausalities in relativistic space-times|1995;
Ebert, D.|GAUGE THEORIES: FOUNDATION OF ELEMENTARY PARTICLE PHYSICS. (IN GERMAN)|1989;
Eckhaus, W.|THE INVERSE SCATTERING TRANSFORMATION AND THE THEORY OF SOLITONS. AN INTRODUCTION|1981;
Eddington, A.|SPACE, TIME AND GRAVITATION. AN OUTLINE OF THE GENERAL RELATIVITY THEORY|1987;
Eder, G.|Nuclear matter: Foundations and problems of nuclear physics. (In German)|1995;
Edmonds, J.D.|Relativistic reality|1997;
Edwards, D.A.|An Introduction to the physics of high-energy accelerators|1993;
Efetov, K.|Supersymmetry in disorder and chaos|1997;
Efimov, G.V|The Quark confinement model of hadrons|1993;
Eguchi, T., (ed.)|Broken symmetry: Selected papers of Y. Nambu|1995;
Eichmeier, J., (Ed.)|HANDBOOK OF VACUUM ELECTRONICS. (IN GERMAN)|1989;
Eilenberger, G.|SOLITONS. MATHEMATICAL METHODS FOR PHYSICISTS|1981;
Einhorn, M.B., (ed.)|The Standard Model Higgs boson|1991;
Eisenberg, J.M.|THEORY OF MESON INTERACTIONS WITH NUCLEI|1980;
Ejiri, H., (ed.)|Nucleon hadron many body systems: From hadron meson to quark lepton nuclear physics|1999;
Ekspong, G., (ed.)|Nobel lectures including presentation speeches and laureates' biographies: Physics 1996-2000|2002;
Ekspong, G., (ed.)|Physics 1991-1995: Nobel lectures. Including presentation speeches and laureates' biographies|1997;
Ekspong, G., (ed.)|The Oskar Klein memorial lectures. Vol. 1: Lectures by C.N. Yang and S. Weinberg with translated reprints by O. Klein|1991;
Ekspong, G., (ed.)|The Oskar Klein memorial lectures. Vol. 2: Lectures by Hans A. Bethe and Alan H. Guth with translated reprints by Oskar Klein|1994;
Elbaz, E.|Quantum: The quantum theory of particles, fields, and cosmology|1998;
Eliezer, S.|Fundamentals of equations of state|2002;
Elizalde, E.|Ten physical applications of spectral zeta functions|1996;
Elizalde, E.|Zeta regularization techniques with applications|1994;
Elliott, J.P.|SYMMETRY IN PHYSICS. VOL. 1: PRINCIPLES AND SIMPLE APPLICATIONS|1987;
Elliott, J.P.|SYMMETRY IN PHYSICS. VOL. 2: FURTHER APPLICATIONS|1986;
Ellis, G.F.R.|FLAT AND CURVED SPACE-TIMES|1988;
Ellis, John R., (ed.)|Quantum reflections|2000;
Ellis, N., (ed.)|2003 CERN-CLAF School of High-Energy Physics, San Miguel Regla, Mexico, 1-14 June 2003|2006;
Ellis, N., (ed.)|2007 European School of High-Energy Physics, Trest, Czech Republic, 19 August - 1 September 2007|2008;
Ellis, Nick (ed.)|2008 European School of High-Energy Physics, Herbeumont-sur-Semois, Belgium, 8-21 June 2008|2009;
Ellis, P.J., (ed.)|Trends in theoretical physics. Vol. 1. Based on the 1988 - 1989 distinguished-speaker colloquium series of the Theoretical Physics Institute at the University of Minnesota|1990;
Ellis, P.J., (ed.)|Trends in theoretical physics: Vol. 2. Based on the 1989 - 1990 distinguished speaker colloquium series of the Theoretical Physics Institute at the University of Minnesota|1991;
Ellis, R.Keith|QCD and collider physics|1996;
Ellwanger, Ulrich|From the universe to the elementary particles: A first introduction into cosmology and fundamental interactions|2008;
Emam, Moataz H., (ed.)|Are we there yet? The Search for a theory of everything|2011;
Emch, G.G., (ed.)|On Klauder's path: A Field trip. Essays in honor of John R. Klauder|1995;
Emmanuel-Costa, D., (Ed.)|CP Violation and the Flavour Puzzle: Symposium in Honour of Gustavo C. Branco. GustavoFest 2005, Lisbon, Portugal, July 2005|2005;
Engelke, B.A.|Ptb Calibration Regulations. Radiation Protection Dosimeters for Photon Radiation of Energies Between 5-KeV and 3-MeV|1977;
Englert, Francois|Broken symmetry and Yang-Mills theory|2004;
Enss, C., (ed.)|Cryogenic particle detection|2005;
Enz, C.P., (Ed.)|WOLFGANG PAULI. THE CONSCIENCE OF PHYSICS. (IN GERMAN)|1988;
Erdmenger, Johanna, (ed.)|String cosmology: Modern string theory concepts from the Big Bang to cosmic structure|2009;
Ericson, Torleif Erik Oskar|PIONS AND NUCLEI|1988;
Ericson, Torleif Erik Oskar|QUARK MATTER FORMATION AND HEAVY ION COLLISIONS: A WORKSHOP HELD AT BIELEFELD: SUMMARY REPORT PREPARED BY THE ORGANIZING COMMITTEE|1982;
Ericson, Torleif Erik Oskar|The Meson factories|1991;
Esposito, Giampiero|Complex general relativity|1995;
Esposito, Giampiero|Euclidean quantum gravity on manifolds with boundary|1997;
Esposito, Giampiero|Quantum gravity in four-dimensions|2001;
Esposito, Giampiero|Quantum gravity, quantum cosmology and Lorentzian geometries|1992;
Etingof, P.I.|Lectures on representation theory and Knizhnik-Zamolodchikov equations|1998;
Evans, D.E.|Quantum symmetries on operator algebras|1998;
Evans, Lyndon, (ed.)|The Large Hadron Collider: A marvel technology|2009;
Evans, M.W., (ed.)|The enigmatic photon|1998;
Evans, M.W.|Classical and quantum electrodynamics and the B(3) field|2001;
Evans, M.W.|The enigmatic photon. Vol. 3: Theory and practice of the B(3) field|1996;
Evans, M.W.|The enigmatic photon. Vol. 5: O(3) electrodynamics|1999;
Evans, M.|The Enigmatic photon. Vol. 1: The Field B(3)|1995;
Evans, M.|The Enigmatic photon. Vol. 2: NonAbelian electrodynamics|1995;
Everard, J.|Fundamentals of RF circuit design with low noise oscillators|2001;
Ewen, K.|RADIATION PROTECTION AT ACCELERATORS. (IN GERMAN)|1986;
Exner, P.|OPEN QUANTUM SYSTEMS AND FEYNMAN INTEGRALS|1986;
Exton, H.|Multiple Hypergeometric Functions and Applications|1976;
Ezawa, H., (Ed.)|PROGRESS IN QUANTUM FIELD THEORY|1986;
Ezawa, Z.F.|Quantum Hall effects: Field theoretical approach and related topics|2000;
Ezhela, V.V.|Particle physics: One hundred years of discoveries. An annotated chronological bibliography|1996;
Fabbri, A.|Modeling black hole evaporation|2005;
Fabbri, Alessandro|The Holographic interpretation of Hawking radiation|2007;
Fabian, A.C.|Serendipity in Astronomy|2009;
Faddeev, L.D.|40 years in mathematical physics|1995;
Faddeev, L.D.|GAUGE FIELDS. INTRODUCTION TO QUANTUM THEORY|1980;
Faddeev, L.D.|HAMILTONIAN METHODS IN THE THEORY OF SOLITONS|1987;
Faessler, A., (Ed.)|PROGRESS IN NUCLEAR AND PARTICLE PHYSICS. VOL. 16|1986;
Faessler, A., (Ed.)|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 14|1985;
Faessler, A., (Ed.)|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 18|1987;
Faessler, A., (Ed.)|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 19|1987;
Faessler, A., (Ed.)|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 21|1988;
Faessler, A., (Ed.)|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 22|1989;
Faessler, A., (Ed.)|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 23|1989;
Faessler, A., (ed.)|Progress in particle and nuclear physics, vol. 54, pt. 2|2004;
Faessler, A., (ed.)|Progress in particle and nuclear physics, vol. 56/1|2006;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 25|1990;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 26|1991;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 27|1991;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 29|1992;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 31|1993;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 33|1994;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 35|1995;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 37|1996;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 41|1998;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 45, Suppl. 1|2000;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 45, Suppl. 2|2000;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 45|2000;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 46|2001;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 47, pt. 1|2001;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 47, pt. 2|2001;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 48, Pt. 2|2002;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 49, Pt. 1|2002;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 49, pt. 2|2002;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 50, Pt. 1|2003;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 51, Pt. 2|2003;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 51, pt. 1|2003;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 52, Pt. 1|2004;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 52, Pt. 2|2004;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 53, Pt. 2|2004;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 54, Pt. 1|2004;
Faessler, A., (ed.)|Progress in particle and nuclear physics. Vol. 55, pt. 2|2004;
Faessler, Amand, (ed.)|Progress in particle and nuclear physics. Vol. 57, Pt. 2|2006;
Faessler, Amand, (ed.)|Progress in particle and nuclear physics. Vol. 58, Pt. 2|2007;
Faessler, Amand, (ed.)|Progress in particle and nuclear physics|2007;
Fairall, A.|Cosmology revealed: Living inside the cosmic egg|2001;
Falcitelli, M.|Riemannian submersions and related topics|2004;
Falk, D.|Universe on a T-shirt: The quest for the theory of everything|2004;
Falkenburg, B.|Particle metaphysics: Perception of reality in philosophy of science and microphysics. (In German)|1995;
Falomir, H., (ed.)|J.J. Giambiagi Festschrift|1990;
Fanchi, J.R.|Parametrized relativistic quantum theory|1994;
Fang, L.Z., (ed.)|Wavelets in physics|1998;
Fang, Li-Zhi, (Ed.)|QUANTUM COSMOLOGY|1987;
Fang, Li-Zhi|BASIC CONCEPTS IN RELATIVISTIC ASTROPHYSICS|1984;
Fang, Li-Zhi|COSMOLOGY OF THE EARLY UNIVERSE|1984;
Fang, Li-Zhi|CREATION OF THE UNIVERSE|1989;
Fang, Li-Zhi|FROM NEWTON'S LAWS TO EINSTEIN'S THEORY OF RELATIVITY|1987;
Faraoni, Valerio|Cosmology in scalar tensor gravity|2004;
Farge, Y.|EUROPEAN SYNCHROTRON RADIATION FACILITY. SUPPLEMENT I: THE SCIENTIFIC CASE|1979;
Farge, Y.|EUROPEAN SYNCHROTRON RADIATION FACILITY. THE FEASIBILITY STUDY|1979;
Farhi, E., (ed.)|DYNAMICAL GAUGE SYMMETRY BREAKING. A COLLECTION OF REPRINTS|1982;
Faria, A.J.|The Schrodinger picture and the zero-point radiation|2003;
Farmelo, G., (ed.)|It must be beautiful: Great equations of modern science|2002;
Fasso, A.|Numerical data and functional relationships in science and technology. Group 1: Nuclear and particle physics. Vol. 11: Shielding against high-energy radiation|1990;
Fauser, B., (ed.)|Quantum gravity: Mathematical models and experimental bounds|2007;
Fayngold, M.|Special relativity and motions faster than light|2002;
Fayyazuddin, |A Modern introduction to particle physics|1994;
Fazal-e-Aleem, |Dilepton production by hadrons|1997;
Fazio, G.G., (ed.)|Currents in astrophysics and cosmology: Papers in honor of Maurice M. Shapiro|1994;
Fecko, M.|Differential geometry and Lie groups for physicists|2006;
Fedosov, B.|Deformation quantization and index theory|1996;
Feinberg, G.|What is the World Made of? Atoms, Leptons, Quarks, and Other Tantalizing Particles|1978;
Feldman, Joel S.|QED: A PROOF OF RENORMALIZABILITY|1987;
Felsager, B.|GEOMETRY, PARTICLES AND FIELDS|1981;
Ferbel, T.|EXPERIMENTAL TECHNIQUES IN HIGH-ENERGY PHYSICS|1992;
Fernandez, R.|Random walks, critical phenomena, and triviality in quantum field theory|1992;
Ferrara, S., (Ed.)|SUPERSYMMETRY|1987;
Ferrara, Sergio, (ed.)|Searching for the superworld: A volume in honor of Antonio Zichichi on the occasion of the 6th centenary celebrations of the University of Turin, Italy|2007;
Ferreira, E.|5th Brazilian Symposium on Theoretical Physics, Rio de Janeiro, 1974. 1.|1975;
Ferreira, E.|5th Brazilian Symposium on Theoretical Physics, Rio de Janeiro, 1974. 2.|1975;
Ferreira, E.|5th Brazilian Symposium on Theoretical Physics, Rio de Janeiro, 1974. 3.|1975;
Ferrero, M., (ed.)|Fundamental problems in quantum physics|1996;
Ferris, T.|COMING OF AGE IN THE MILKY WAY. (IN GERMAN)|1989;
Feser, T.|Real-time search for neutrino bursts from supernovae with the AMANDA-II detector|2005;
Feshbach, H.|Theoretical nuclear physics: Nuclear reactions|1992;
Feynman, R.P.|ELEMENTARY PARTICLES AND THE LAWS OF PHYSICS. THE 1986 DIRAC MEMORIAL LECTURES|1987;
Feynman, R.P.|Feynman lectures on gravitation|1996;
Feynman, R.P.|QED. THE STRANGE THEORY OF LIGHT AND MATTER|1986;
Feynman, R.P.|Selected papers of Richard Feynman: With commentary|2000;
Feynman, R.P.|Six not-so-easy pieces: Einstein's relativity, symmetry, and space-time|1997;
Field, R.D.|Applications of Perturbative QCD|1989;
Fierz, M., (Ed.)|THEORETICAL PHYSICS IN THE 20TH CENTURY. A MEMORIAL VOLUME TO WOLFGANG PAULI|1984;
Filippov, A.T.|The versatile soliton|2000;
Finkelstein, D.R.|Quantum relativity: A Synthesis of the ideas of Einstein and Heisenberg|1996;
Finster, F.|The principle of the fermionic projector|2006;
Fiorentini, G.|Exotic Atoms. First Course of the International School of Physics of Exotic Atoms, Erice, 24-30 Apr 1977|1977;
Fischbach, E.|The search for nonNewtonian gravity|1999;
Fischer, W., (ed.)|Beam halo dynamics, diagnostics, and collimation: 29th ICFA Advanced Beam Dynamics Workshop, HALO'03, Workshop on beam-beam interactions, Beam-Beam'03, Montauk, USA, May 19-23, 2003|2003;
Fitzpatrick, G.L.|The family problem: New internal algebraic and geometric regularities|1997;
Flamig, E.|10TH INTERNATIONAL SYMPOSIUM ON NUCLEAR ELECTRONICS. DRESDEN, GDR, 10-16 APRIL, 1980. (ABSTRACTS OF CONTRIBUTED PAPERS)|1980;
Flamm, D.|INTRODUCTION TO THE QUARK MODEL OF ELEMENTARY PARTICLES. VOL. 1. QUANTUM NUMBERS, GAUGE THEORIES AND HADRON SPECTROSCOPY|1982;
Flato, M.|Quantum Mechanics, Determinism, Causality, and Particles. An International Collection of Contributions in Honor of Louis de Broglie on the Occasion of the Jubilee of His Celebrated Thesis|1976;
Flato, M.|SELECTED PAPERS (1937 - 1976) OF JULIAN SCHWINGER|1979;
Fleischer, R., (ed.)|2004 European School of High-Energy Physics, Sant Feliu de Guixols, Spain, 30 May - 12 June 2004|2006;
Fleischer, R.L.|Nuclear Tracks in Solids. Principles and Applications|1975;
Fleischer, R.L.|Tracks to innovation: Nuclear tracks in science and technology|1998;
Fleury, N., (Ed.)|LEITE LOPES FESTSCHRIFT: A PIONEER PHYSICIST IN THE THIRD WORLD|1988;
Fleury, N.|Introduction to the Unitary Symmetry. (In French)|1974;
Fliessbach, T.|GENERAL RELATIVITY THEORY. (IN GERMAN)|1990;
Fliessbach, T.|General relativity theory. (In German)|1995;
Flohr, Michael A.I.|Logarithmic conformal field theory - or - how to compute a torus amplitude on the sphere|2004;
Flood, R., (Ed.)|THE NATURE OF TIME|1989;
Fock, V.A.|Selected works, V. A. Fock: Quantum mechanics and quantum field theory|2004;
Foderaro, A.|The Photon Shielding Manual|1976;
Fokas, A., (ed.)|Mathematical physics 2000|2000;
Folland, Gerald B.|Quantum field theory: A tourist guide for mathematicians|2008;
Fomenko, A.T., (ed.)|Tensor and vector analysis: Geometry, mechanics and physics|1998;
Foot, Robert|Shadowlands: Quest for mirror matter in the universe|2002;
Ford, L.H.|Spacetime in semiclassical gravity|2005;
Forest, E.|Beam dynamics: A new attitude and framework|1998;
Forshaw, Jeffrey R.|Dynamics and relativity|2009;
Forshaw, Jeffrey R.|Quantum chromodynamics and the pomeron|1997;
Forward, R.L.|MIRROR MATTER: PIONEERING ANTIMATTER PHYSICS|1988;
Foster, B., (Ed.)|ELECTRON POSITRON ANNIHILATION PHYSICS|1990;
Foster, J.|A short course in general relativity|1995;
Fradkin, E.S.|Conformal quantum field theory in D-dimensions|1996;
Fradkin, E.S.|Quantum electrodynamics with unstable vacuum|1991;
Fradkin, E.S.|Selected papers on theoretical physics|2007;
Fradkin, Eduardo H.|Field theories of condensed matter systems|1991;
Fraengsmyr, T., (ed.)|Nobel lectures including presentation speeches and laureates' biographies: Physics 1981 - 1990|1993;
Frampton, P.H.|DUAL RESONANCE MODELS AND SUPERSTRINGS|1986;
Frampton, P.H.|GAUGE FIELD THEORIES|1987;
Frampton, Paul .H.|Gauge Field Theories: Third Revised and Improved Edition|2008;
Frampton, Paul H.|Did time begin? Will time end?|2007;
Francaviglia, M., (ed.)|Mechanics, analysis and geometry: 200 years after Lagrange|1991;
Francaviglia, M.|Natural and gauge natural formalism for classical field theories: A geometric perspective including spinors and gauge theories|2003;
Frank, A.|Symmetries in atomic nuclei: From isospin to supersymmetry|2009;
Frank, M.|Radiative corrections to the Higgs sector of the minimal supersymmetric standard model with CP violation|2003;
Frankel, T.|The geometry of physics: An introduction|1997;
Frankfurt, L.|3D parton imaging of the nucleon in high-energy pp and pA collisions|2004;
Franklin, A.|Are there really neutrinos? An evidential history|2000;
Franklin, A.|The Rise and fall of the fifth force: Discovery, pursuit, and justification in modern physics|1993;
Fraser, G., (ed.)|The new physics for the twenty-first century|2006;
Fraser, G., (ed.)|The particle century|1998;
Fraser, G.|Antimatter: The ultimate mirror|2000;
Fraser, G.|The Search for infinity: Solving the mysteries of the universe|1995;
Fraser, G.|The quark machines: How Europe fought the particle physics war|1997;
Fraser, Gordon|Cosmic anger: Abdus Salam - the first Muslim Bobel scientist|2008;
Frauendiener, J., (ed.)|The conformal structure of space-time: Geometry, analysis, numerics|2002;
Frauenfelder, H.|Nuclear and Particle Physics. Background and Symmetries|1975;
Frauenfelder, H.|PARTICLES AND NUCLEI. (IN GERMAN) SUBATOMIC PHYSICS|1979;
Fre, P., (ed.)|Classical and quantum black holes|1999;
Fre, P.|The N=2 wonderland: From Calabi-Yau manifolds to topological field theories|1995;
Freed, D.S., (ed.)|Quantum field theory, supersymmetry, and enumerative geometry|2006;
Freed, D.S.|Five lectures on supersymmetry|1999;
Freed, D.S.|INSTANTONS AND FOUR - MANIFOLDS|1984;
Freedman, M.H.|Selected applications of geometry to low dimensional topology. Marker lectures in the mathematical sciences held at the Pennsylvania State University, February 2-5, 1987|1989;
Freeman, K.|In search of dark matter|2006;
Frenkel, E.|Vertex algebras and algebraic curves|2004;
Frenkel, I.|VERTEX OPERATOR ALGEBRAS AND THE MONSTER|1988;
Freund, P.G.O.|INTRODUCTION TO SUPERSYMMETRY|1986;
Freund, Peter G.O.|Dynamical Spin II|2008;
Frey, H.|LOW TEMPERATURE TECHNOLOGY. (IN GERMAN)|1981;
Fried, H.M.|Basics of functional methods and eikonal models|1990;
Friedman, G.|The Astronomer's universe. (In German)|1991;
Friman, Bengt|General introduction|2011;
Frishman, Yitzhak|Non-perturbative field theory: From two-dimensional conformal field theory to QCD in four dimensions|2010;
Fritzsch, H.|A FORMULA CHANGES THE WORLD: NEWTON, EINSTEIN AND THE THEORY OF RELATIVITY. (IN GERMAN)|1988;
Fritzsch, H.|Deformed space-time: Newton, Einstein, and gravitation. (In German)|1996;
Fritzsch, H.|FROM BIG BANG TO DECAY. THE WORLD BETWEEN BEGINNING AND END. (IN GERMAN)|1983;
Fritzsch, H.|QUARKS - BASIC MATTER OF OUR WORLD. (IN GERMAN)|1981;
Fritzsch, H.|The absolutely invariable: The ultimate riddles of physics|2005;
Fritzsch, Harald|Elementary particles: Building blocks of matter|2005;
Fritzsch, Harald|You are wrong, Mr Einstein! Newton, Einstein, Heisenberg and Feynman discussing quantum mechanics|2011;
Frodesen, A.G.|PROBABILITY AND STATISTICS IN PARTICLE PHYSICS|1979;
Froggatt, C.D.|Origin of symmetries|1991;
Frohlich, J., (Ed.)|SCALING AND SELFSIMILARITY IN PHYSICS. RENORMALIZATION IN STATISTICAL MECHANICS AND DYNAMICS|1984;
Frohlich, J.|Nonperturbative quantum field theory: Mathematical aspects and applications. Selected papers|1992;
Frohlich, Jurg|Quantum groups, quantum categories and quantum field theory|1993;
Frois, B., (ed.)|Modern topics in electron scattering|1991;
Frolov, V.P., (ed.)|Black hole physics: Basic concepts and new developments|1998;
Fronsdal, C., (Ed.)|ESSAYS ON SUPERSYMMETRY|1986;
Fryer, C.L., (ed.)|Stellar collapse|2004;
Fuchs, J.|Affine Lie algebras and quantum groups: An Introduction, with applications in conformal field theory|1992;
Fuchs, J.|Symmetries, Lie algebras and representations: A graduate course for physicists|1997;
Fujii, K., (ed.)|Linear collider physics in the new millennium|2005;
Fujii, Y.|The scalar-tensor theory of gravitation|2003;
Fujikawa, K.|Path integrals and quantum anomalies|2004;
Fujita, S.|The Ta-You Wu Festschrift. Science of Matter|1978;
Fukugita, M., (ed.)|Physics and astrophysics of neutrinos|1994;
Fukugita, M.|Physics of neutrinos and applications to astrophysics|2003;
Fulling, S.A.|ASPECTS OF QUANTUM FIELD THEORY IN CURVED SPACE-TIME|1989;
Funaro, Daniele|Electromagnetism and the structure of matter|2008;
Fushchich, V.I.|SYMMETRIES OF MAXWELL'S EQUATIONS|1987;
Fushchich, W.I.|Symmetry analysis and exact solutions of equations of nonlinear mathematical physics|1993;
Fushchych, Wilhelm|Symmetries and Exact Solutions of Nonlinear Dirac Equations|1997;
Futterman, J.A.H.|SCATTERING FROM BLACK HOLES|1988;
Fyodorov, Y.V.|Resonance Scattering of Waves in Chaotic Systems|2010;
Gaillard, M.K.|Weak Interactions|1977;
Gaisser, T.K.|Cosmic rays and particle physics|1990;
Gal-Or, B.|COSMOLOGY, PHYSICS, AND PHILOSOPHY|1981;
Galishev, V.S.|Problems in the Theory of Multiple Scattering of Particles|1974;
Galison, P.L.|HOW EXPERIMENTS END|1987;
Galison, P.|Image and logic: A material culture of microphysics|1997;
Gallo, A.|DAFNE upgrade at LNF-INFN|2005;
Galperin, A.S.|Harmonic superspace|2001;
Gambini, R.|Loops, knots, gauge theories and quantum gravity|1996;
Gambini, Rodolfo|Discrete space-time|2005;
Gambini, Rodolfo|Modern space-time and undecidability|2008;
Garbaczewski, P.|CLASSICAL AND QUANTUM FIELD THEORY OF EXACTLY SOLUBLE NONLINEAR SYSTEMS|1985;
Garber, D.I.|Neutron Cross-Sections. 2. Curves|1976;
Garcia Canal, C.A.|INTRODUCTION TO HADRON PHYSICS|1980;
Garcia Canal, C.A.|NOTES ON DEEP INELASTIC SCATTERING|1979;
Garcia, A.|THE BETA DECAY OF HYPERONS|1985;
Garcilazo, H.|pi N N systems|1990;
Gardner, M.|The New ambidextrous universe: Symmetry and asymmetry from mirror reflections to superstrings|1990;
Garling, D.J.H.|Clifford algebras: An introduction|2011;
Garsevanishvili, V.R.|Relativistic nuclear physics in the light front formalism|1993;
Gasiorowicz, S.|THE STRUCTURE OF MATTER: AT SURVEY OF MODERN PHYSICS|1979;
Gasperini, Maurizio, (ed.)|String theory and fundamental interactions|2008;
Gasperini, Maurizio|Elements of string cosmology|2007;
Gasperini, Maurizio|The universe before the big bang: Cosmology and string theory|2008;
Gastmans, R.|The Ubiquitous photon: Helicity method for QED and QCD|1990;
Gates, Evalyn|Einstein's telescope: The hunt for dark matter and dark energy in the universe|2009;
Gates, S.J.|Superspace Or One Thousand and One Lessons in Supersymmetry|1983;
Gattringer, Christof|Quantum chromodynamics on the lattice|2010;
Gazeau, Jean-Pierre|Coherent states in quantum physics|2009;
Gelmini, Graciela|DM Production Mechanisms|2010;
Genz, H.|Elementary particles|2003;
Genz, H.|How time came into the world: The creation of an illusion out of order and chaos. (In German)|1996;
Genz, H.|Nothing but nothing: The physics of the vacuum|2004;
Genz, H.|SYMMETRY - CONSTRUCTION DRAWING OF NATURE. (IN GERMAN)|1987;
Genz, H.|Symmetry and symmetry breaking in physics. (In German)|1991;
Genz, H.|The Detection of nothing. Vacuum and fullness in the universe. (In German)|1992;
Genz, H.|Was it a god? Chance, necessity and creativity in the evolution of the universe|2006;
Georgi, H.|LIE ALGEBRAS IN PARTICLE PHYSICS. FROM ISOSPIN TO UNIFIED THEORIES|1982;
Georgi, H.|WEAK INTERACTIONS AND MODERN PARTICLE THEORY|1985;
Gergely, Laszlo A.|Dark energy from gravitational collapse?|2006;
Gervais, J.L., (Ed.)|NONLINEAR AND COLLECTIVE PHENOMENA IN QUANTUM PHYSICS. A REPRINT VOLUME FROM PHYSICS REPORTS|1985;
Geyer, B.|Introduction to the quantum field theory of elementary particles. (In German)|1990;
Giachetta, G.|New Lagrangian and Hamiltonian methods in field theory|1997;
Giannuzzi, Lucille A., (ed.)|Introduction to focused ion beams: Instrumentation, theory, techniques and practice|2005;
Gibbons, G.W., (ed.)|Euclidean quantum gravity|1994;
Gibbs, Philip|Event symmetric space-time|1998;
Gibson, W.M.|Symmetry Principles in Elementary Particle Physics|1976;
Gieres, F.|GEOMETRY OF SUPERSYMMETRIC GAUGE THEORIES: INCLUDING AN INTRODUCTION TO BRS DIFFERENTIAL ALGEBRAS AND ANOMALIES|1988;
Gilmore, R.S.|Single particle detection and measurement|1992;
Gilmore, R.|The wizard of quarks: A fantasy of particle physics|2001;
Gilmore, Robert|Lie groups, physics, and geometry: An introduction for physicists, engineers and chemists|2008;
Gindikin, S., (ed.)|Geometry and physics: Essays in honor of I. M. Gelfand|1991;
Gingrich, D.M.|Particle quantum electrodynamics|2006;
Ginzburg, V.L., (Ed.)|ISSUES IN INTENSE FIELD QUANTUM ELECTRODYNAMICS|1987;
Ginzburg, V.L., (Ed.)|SOLITONS AND INSTANTONS, OPERATOR QUANTIZATION|1987;
Ginzburg, V.L., (ed.)|Astrophysics of cosmic rays|1990;
Ginzburg, V.L.|PHYSICS AND ASTROPHYSICS. A SELECTION OF KEY PROBLEMS|1985;
Giovannini, Alberto, (Ed.)|FESTSCHRIFT IN HONOR OF EDUARDO R. CAIANIELLO|1989;
Giovannini, Alberto, (Ed.)|The legacy of Leon Van Hove|2000;
Giovannini, Massimo|A primer on the physics of the cosmic microwave background|2008;
Gitman, D.M.|Quantization of fields with constraints|1990;
Gitterman, M.|QUALITATIVE ANALYSIS OF PHYSICAL PROBLEMS|1981;
Giudice, G., (ed.)|Fifty years of research at CERN, from past to future|2006;
Giudice, Gian Francesco|A zeptospace odyssey: A journey into the physics of the LHC|2010;
Giulini, D., (ed.)|Quantum gravity: From theory to experimental search|2003;
Giulini, D.|Decoherence and the appearance of a classical world in quantum theory|1996;
Giunti, Carlo|Fundamentals of Neutrino Physics and Astrophysics|2007;
Glashow, S.L.|From alchemy to quarks: The Study of physics as a liberal art|1995;
Glashow, S.L.|The Charm of physics|1991;
Glashow, S.|INTERACTIONS: A JOURNEY THROUGH THE MIND OF A PARTICLE PHYSICIST AND THE MATTER OF THIS WORLD|1988;
Glendenning, N.K.|After the beginning: A cosmic journey through space and time|2004;
Glendenning, N.K.|Compact stars: Nuclear physics, particle physics, and general relativity|1997;
Glimm, J.|COLLECTED PAPERS. VOL. 1: QUANTUM FIELD THEORY AND STATISTICAL MECHANICS. EXPOSITIONS|1986;
Glimm, J.|COLLECTED PAPERS. VOL. 2: CONSTRUCTIVE QUANTUM FIELD THEORY. SELECTED PAPERS|1986;
Glimm, J.|QUANTUM PHYSICS. A FUNCTIONAL INTEGRAL POINT OF VIEW|1981;
Glimm, J.|QUANTUM PHYSICS. A FUNCTIONAL INTEGRAL POINT OF VIEW|1987;
Gockeler, M.|DIFFERENTIAL GEOMETRY, GAUGE THEORIES, AND GRAVITY|1987;
Godbillon, C.|DYNAMICAL SYSTEMS ON SURFACES|1983;
Godbole, Rohini M.|The Heart of Matter|2010;
Goddard, P., (Ed.)|KAC-MOODY AND VIRASORO ALGEBRAS: A REPRINT VOLUME FOR PHYSICISTS|1988;
Goenner, H.|Introduction to special and general relativity theory. (In German)|1996;
Gogolin, A.O.|Bosonization and strongly correlated systems|2004;
Goldansky, V.I.|KINEMATIC METHODS IN HIGH-ENERGY PHYSICS|1989;
Golde, Rudolf Heinrich, (ed.)|Lightning|1977;
Goldenfeld, N.|Lectures on phase transitions and the renormalization group|1992;
Goldhaber, A.S., (ed.)|Magnetic monopoles|1990;
Goldman, J.Terrance, (Ed.)|INTENSE MEDIUM-ENERGY SOURCES OF STRANGENESS|1985;
Goldsmith, D.|Einstein's greatest blunder? The Cosmological constant and other fudge factors in the physics of the universe|1995;
Gomez, C.|Quantum groups in two-dimensional physics|1996;
Gondhalekar, P.|The grip of gravity: The quest to understand the laws of motion and gravitation|2001;
Gonner, H.|Introduction to cosmology. (In German)|1994;
Gonzalez-Diaz, Pedro F.|On the onset of the dark energy era|2008;
Gonzalo, J.A.|Inflationary cosmology revisited: An overview of contemporary scientific cosmology after the inflationary proposal|2005;
Gonzalo, Julio A.|The intelligible universe: An overview of the last thirteen billion years|2008;
Gorbunov, Dmitry S.|Introduction to the theory of the early universe, Cosmological perturbations and inflationalry theory|2011;
Gorbunov, Dmitry S.|Introduction to the theory of the early universe: hot big bang theory|2011;
Gotsman, E., (Ed.)|FROM SU(3) TO GRAVITY. FESTSCHRIFT IN HONOR OF YUVAL NE'EMAN|1985;
Gottfried, K.|CONCEPTS OF PARTICLE PHYSICS. VOL. 1|1984;
Gottfried, K.|CONCEPTS OF PARTICLE PHYSICS. VOL. 2|1986;
Gould, R.J.|Electromagnetic processes|2006;
Gourdin, M.|BASICS OF LIE GROUPS|1982;
Govaerts, J.|Hamiltonian quantization and constrained dynamics|1991;
Govoni, Pietro, (ed.)|IPRD08, 11th Topical Seminar on Innovative Particle and Radiation Detectors, Siena, Italy, 1-4 October 2008|2009;
Gracia-Bondia, Jose M.|Elements of noncommutative geometry|2001;
Grandy, W.T.|Relativistic quantum mechanics of leptons and fields|1991;
Granja, Carlos, (ed.)|Nuclear physics methods and accelerators in biology and medicine. Fourth International Summer School on Nuclear Physics Methods and Accelerators in Biology and Medicine, Prague, Czech Republic, 8-19 July
Grassmann, H.|The top quark, Picasso and Mercedes-Benz or what is physics? (In German)|1997;
Grawe, H.|ATOMIC AND NUCLEAR PHYSICS: FOUNDATIONS, ELEMENTARY PARTICLES, ATOMIC SHELL, ATOMIC NUCLEUS. (IN GERMAN)|1988;
Greco, Mario (Ed.)|Bruno Touschek memorial lectures|2004;
Greco, Mario (Ed.)|Les Rencontres de physique de la Vall'ee d'Aoste, results and perspectives in particle physics, La Thuile, Aoste Valley, 5-11 March, 2006|2006;
Greco, Mario (Ed.)|Les Rencontres de physique de la Vall'ee d'Aoste, results and perspectives in particle physics, La Thuile, Aoste Valley, February 27 - March 5, 2005|2005;
Green, A.M., (ed.)|Hadronic physics from lattice QCD|2004;
Green, A.M.|Bridges from lattice QCD to nuclear physics: Chapter 1|2004;
Green, B.|The fabric of the cosmos: Space, time, and the texture of reality|2004;
Green, D.|Lectures in particle physics|1995;
Green, D.|The physics of particle detectors|2000;
Green, Dan, (Ed.)|At the leading edge: The ATLAS and CMS LHC experiments|2010;
Green, Daniel R.|High P(T) physics at hadron colliders|2005;
Green, H.S.|Information theory and quantum physics: Physical foundations for understanding the conscious process|2000;
Green, J.R.|Statistical Treatment of Experimental Data|1978;
Green, Michael B.|SUPERSTRING THEORY. VOL. 1: INTRODUCTION|1987;
Green, Michael B.|SUPERSTRING THEORY. VOL. 2: LOOP AMPLITUDES, ANOMALIES AND PHENOMENOLOGY|1987;
Greenberg, O.W.|The Parton model|2008;
Greenberger, Daniel, (ed.)|Compendium of quantum physics: Concepts, experiments, history and philosophy|2009;
Greene, A.F.|Fermilab Research Program 1976: Workbook|1976;
Greene, A.F.|Fermilab Research Program 1977: Workbook|1977;
Greene, A.F.|Fermilab Research Program 1978: Workbook|1978;
Greene, B., (ed.)|Mirror symmetry II|1997;
Greene, Brian R.|The elegant universe: Superstrings, hidden dimensions, and the quest of the ultimate theory|1999;
Greeniaus, L.G., (Ed.)|TRIUMF KINEMATICS HANDBOOK|1987;
Greensite, Jeff|An introduction to the confinement problem|2011;
Greenstein, G.|THE SYMBIOTIC UNIVERSE. (IN GERMAN)|1988;
Greiner, W., (Ed.)|Elementary matter, vacuum and fields. The structure of the vacuum and of the building blocks of nature|1986;
Greiner, W.|Field quantization|1996;
Greiner, W.|Gauge theory of weak interactions|1993;
Greiner, W.|QUANTUM ELECTRODYNAMICS OF STRONG FIELDS|1985;
Greiner, W.|Quantum chromodynamics|1995;
Greiner, W.|Quantum chromodynamics|2002;
Greiner, W.|Quantum electrodynamics|1992;
Greiner, W.|Quantum mechanics: Special chapters|1998;
Greiner, W.|Relativistic quantum mechanics: Wave equations|1990;
Greiner, W.|THEORETICAL PHYSICS. (IN GERMAN) 7. QUANTUM ELECTRODYNAMICS|1984;
Greiner, W.|THEORETICAL PHYSICS. (IN GERMAN) A TEXT AND EXERCISE BOOK FOR UNDERGRADUATES. VOL. 3A: SPECIAL THEORY OF RELATIVITY|1984;
Greiner, W.|THEORETICAL PHYSICS. VOL. 10: QUANTUM CHROMODYNAMICS. (IN GERMAN)|1989;
Greiner, W.|THEORETICAL PHYSICS. VOL. 4A: QUANTUM THEORY. (IN GERMAN)|1985;
Greiner, W.|THEORETICAL PHYSICS. VOL. 5: QUANTUM MECHANICS 2. SYMMETRIES. (IN GERMAN)|1985;
Greiner, W.|THEORETICAL PHYSICS. VOL. 6: RELATIVISTIC QUANTUM MECHANICS - WAVE EQUATIONS. (IN GERMAN)|1981;
Greiner, W.|THEORETICAL PHYSICS: A TEXT AND EXERCISE BOOK. VOL. 8: GAUGE THEORY OF THE WEAK INTERACTION. (IN GERMAN)|1986;
Greiner, W.|Theoretical physics. Vol. 2: Quantum mechanics. Symmetries|1989;
Greiner, Walter|Quantum chromodynamics|2007;
Gribbin, J.R.|In search of the big bang: Quantum physics and cosmology|1986;
Gribbin, J.|A Universe made to measure: Conditions of our existence. (In German)|1995;
Gribbin, J.|In search of the big bang: The life and death of the universe|1998;
Gribbin, J.|In the beginning: After COBE and before the big bang. (In German)|1995;
Gribbin, J.|Q is for quantum: An encyclopedia of particle physics|1998;
Gribbin, J.|Schroedinger's kitten and the search for reality|1996;
Gribbin, J.|The Omega point: The search for the missing mass and the ultimate fate of the universe. (In German)|1990;
Gribov, V.N.|Quantum electrodynamics: Gribov lectures on theoretical physics|2001;
Gribov, V.N.|The theory of complex angular momenta: Gribov lectures on theoretical physics|2003;
Gribov, Vladimir N.|Strong interactions of hadrons at high emnergies: Gribov lectures on|2009;
Grieder, P.K.F.|Cosmic rays at earth: Researcher's reference, manual and data book|2001;
Griffin, A., (ed.)|Bose-Einstein condensation|1995;
Griffiths, David J.|INTRODUCTION TO ELEMENTARY PARTICLES|1987;
Griffiths, David|Introduction to elementary particles|2008;
Griffiths, J.B.|Colliding plane waves in general relativity|1991;
Grigorescu, M.|TDHFB-Langevin approach to the nuclear collective dynamics|2009;
Grimm, U., (ed.)|Perspectives on solvable models:|1995;
Grinevald, J.|THE QUADRATURE OF CERN. (IN FRENCH) INTERDISCIPLINARY ESSAY PUBLISHED ON THE OCCASION OF THE 30TH ANNIVERSARY OF CERN, SEPTEMBER 29, 1984|1985;
Gritsev, Vladimir|Universal Dynamics Near Quantum Critical Points|2009;
Groen, Oeyvind|Einstein's general theory of relativity: With modern applications in cosmology|2007;
Groen, Oeyvind|Lecture notes on the general theory of relativity: From Newton's attractive gravity to the repulsive gravity of vacuum energy|2009;
Grosche, C.|Handbook of Feynman Path Integrals|1998;
Grosche, Christian|Path integrals, hyperbolic spaces, and Selberg trace formula|1995;
Gross, F.|Relativistic quantum mechanics and field theory|1993;
Grosse, H.|MODELS IN STATISTICAL PHYSICS AND QUANTUM FIELD THEORY|1988;
Grosse, H.|Particle physics and the Schrodinger equation|1997;
Grossetete, B.|Interactions and particles. (In French)|1991;
Grosshans, F.D.|INVARIANT THEORY AND SUPERALGEBRAS. EXPOSITORY LECTURES FROM CBMS REGIONAL CONFERENCE, WEST CHESTER, USA, AUGUST 19-23, 1985|1987;
Groteluschen, F.|The sound of superstrings: Introduction to the nature of elementary particles|1999;
Grotz, K.|THE WEAK INTERACTION IN NUCLEAR, PARTICLE AND ASTROPHYSICS: AN INTRODUCTION. (IN GERMAN)|1989;
Grotz, K.|The Weak interaction in nuclear, particle and astrophysics|1990;
Grozin, A.G.|Heavy quark effective theory|2004;
Grozin, A.G.|Using REDUCE in high-energy physics|1997;
Grozin, Andrey|Lectures on QED and QCD: Practical calculation and renormalization of one- and multi-loop Feynman diagrams|2007;
Grumiller, Daniel, (ed.)|Fundamental interactions: A memorial volume for Wolfgang Kummer|2010;
Grupen, C.|Astroparticle physics: The universe in the light of cosmic radiation|2000;
Grupen, C.|Astroparticle physics|2005;
Grupen, C.|Particle detectors. (In German)|1994;
Grupen, C.|Particle detectors|1996;
Grupen, Claus|Particle detectors|2008;
Gsponer, Andre|Lanczos-Einstein-Petiau: From Dirac ' $s$ equation to nonlinear wave mechanics|2005;
Gu, Chao-Hao, (ed.)|Soliton theory and its applications|1996;
Guadagnini, E.|The Link invariants of the Chern-Simons field theory: New developments in topological quantum field theory|1994;
Gubser, Steven S.|The little book of string theory|2010;
Guchait, Monoranjan|Using Tau Polarization for Charged Higgs Boson and SUSY Searches at LHC|2008;
Gueller, S.|FRONTIERS OF PHYSICS|1987;
Guidry, M.W.|Gauge field theories: An Introduction with applications|1991;
Guillemin, V.W.|Supersymmetry and equivariant de Rham theory|1999;
Guillemin, V.|Symplectic techniques in physics|1990;
Gunion, John F.|THE HIGGS HUNTER'S GUIDE|1989;
Gupta, Raj K., (ed.)|Physics of particles, nuclei and materials: Recent trends|2002;
Gupta, S.N.|Quantum Electrodynamics|1977;
Gursey, F.|On the role of division, Jordan and related algebras in particle physics|1996;
Gurzadian, V.G., (ed.)|From integrable models to gauge theories: A volume in honor of Sergei Matinyan|2002;
Guth, Alan H.|The inflationary universe: The quest for a new theory of cosmic origins|1997;
Gyoergyi, G., (ed.)|From phase transitions to chaos: Topics in modern statistical physics|1992;
Haag, R.|Local quantum physics: Fields, particles, algebras|1992;
Habfast, C.|Big science with small particles. Deutsches Elektronen-Synchrotron DESY 1956-1970. (in German)|1989;
Hadjimichael, E.|FEW BODY PROBLEMS|1986;
Hadlock, C.R.|Field Theory and Its Classical Problems|1978;
Haensel, H.|Physics. Vol. 3: Atoms, nuclei, elementary particles|1995;
Haensel, P.|Neutron stars 1: Equation of state and structure|2007;
Hafner, C.|NUMERICAL CALCULATION OF ELECTROMAGNETIC FIELDS: FOUNDATIONS, METHODS, APPLICATIONS. (IN GERMAN)|1987;
Haidt, D.|NUMERICAL DATA AND FUNCTIONAL RELATIONSHIPS IN SCIENCE AND TECHNOLOGY. GRP. 1: NUCLEAR AND PARTICLE PHYSICS. VOL. 10: ELECTROWEAK INTERACTIONS: EXPERIMENTAL FACTS AND THEORETICAL FOUNDATION|1988;
Haken, H.|Solid state quantum field theory. (In German)|1993;
Hakim, R.|An introduction to relativistic gravitation|1999;
Halbach, Klaus|The Art and science of magnet design, February 1995, Lawrence Berkeley Laboratory, University of California, Berkeley, CA 94720|1994;
Halliday, David|Fundamentals of physics|2009;
Halpern, P.|The great beyond: Higher dimensions, parallel universes, and the extraordinary search for a theory of everything|2004;
Halpern, Paul|Collider: The search for the world's smallest particles|2009;
Halvorson, Hans|Algebraic quantum field theory|2006;
Halzen, F.|QUARKS AND LEPTONS: AN INTRODUCTORY COURSE IN MODERN PARTICLE PHYSICS|1984;
Hamber, Herbert W.|Quantum gravitation: The Feynman path integral approach|2009;
Hamer, Chris|Series expansion methods for strongly interacting lattice models|2006;
Hamilton, J.|NEW DEVELOPMENTS IN DISPERSION THEORY. VOL. 2A. STRONG INTERACTIONS OF KAONS AND SU(3) PROPERTIES|1978;
Hamilton, J.|New Developments in Dispersion Theory. Vol. 1. Dynamical Singularities and Amplitude Analysis|1975;
Hammond, R.T.|From quarks to black holes: Interviewing the universe|2001;
Han, M.Y.|A short introduction to quantum field theory of quarks and leptons|2005;
Han, M.Y.|Quarks and gluons: A century of particle charges|1999;
Han, M.Y.|The Probable universe: An Owner's guide to quantum physics|1993;
Han, M.Y.|The Secret life of quanta|1990;
Hannabuss, K.|An introduction to quantum theory|1999;
Hansel, H.|Molecules, Nuclei, and Elementary Particles|1977;
Hara, O.|Shoichi Sakata-Scientific Works|1977;
Hardy, Lucien|Formalism Locality in Quantum Theory and Quantum Gravity|2008;
Hargittai, I., (Ed.)|SYMMETRY. UNIFYING HUMAN UNDERSTANDING|1986;
Harland, D.M.|The big bang: A view from the 21st century|2003;
Harmuth, H.F.|Information theory applied to space-time physics|1992;
Harmuth, H.F.|Modified Maxwell equations in quantum electrodynamics|2001;
Harney, H.L.|Bayesian interference: Parameter estimation and decisions|2003;
Harrison, Edward R.|COSMOLOGY: THE SCIENCE OF THE UNIVERSE|1988;
Hartemann, F.V.|High-field electrodynamics|2002;
Hartle, J.B.|An introduction to Einstein's general relativity|2003;
Hartline, Beverly Karplus, (ed.)|Women in physics, 2nd IUPAP International Conference on Women in Physics, Rio de Janeiro, Brazil, May 23-25, 2005|2005;
Hartmann, Frank|Evolution of Silicon Sensor Technology in Particle Physics|2009;
Hashim, Nadir Omar|Measurement of the momentum spectrum of cosmic ray muons at a depth of 320-mwe|2007;
Hasinger, Gunther|The fate of the Universe: Travelling from the beginning to the end|2009;
Hasse, R.W.|GEOMETRICAL RELATIONSHIPS OF MACROSCOPIC NUCLEAR PHYSICS|1988;
Hatfield, B.|Quantum field theory of point particles and strings|1992;
Haug, E.|The elementary process of bremsstrahlung|2004;
Hauptman, John|Particle physics experiments at high energy colliders|2011;
Haussmann, R.|Selfconsistent quantum field theory and bosonization for strongly correlated electron systems|1999;
Hawking, S., (ed.)|Beginning or end? Inaugural lecture. (In German)|1991;
Hawking, S.W., (Ed.)|THREE HUNDRED YEARS OF GRAVITATION|1987;
Hawking, S.W.|A BRIEF HISTORY OF TIME: FROM THE BIG BANG TO BLACK HOLES. (IN GERMAN)|1988;
Hawking, S.W.|Einstein's dream: Expeditions to the frontiers of space-time. Black holes and baby universes and other essays. (In German)|1993;
Hawking, S.W.|GENERAL RELATIVITY. AN EINSTEIN CENTENARY SURVEY|1979;
Hawking, S.W.|The Future of space-time|2002;
Hawking, S.W.|The Large scale structure of space-time|1973;
Hawking, S.W.|The illustrated theory of everything: The origin and fate of the universe|2003;
Hawking, S.|A brief history of time|2005;
Hawking, S.|Black holes and baby universes and other essays|1995;
Hawking, S.|The Nature of space and time|1996;
Hawking, S.|The universe in a nutshell|2001;
Hawkins, M.|Hunting down the universe: The missing mass, primordial black holes and other dark matters|1997;
Hawley, J.F.|Foundations of modern cosmology|1998;
Haxton, W.C., (ed.)|Symmetries and fundamental interactions in nuclei|1995;
Hays, Christopher Paul|Search for the Higgs Boson|2006;
Hayward, Raymond W.|The Dynamics of Fields of Higher Spin|1976;
Haywood, Stephen|Symmetries and conservation laws in particle physics: An introduction to group theory in particle physicsts|2011;
Healey, Richard|Gauging what's real: The conceptual foundations of gauge theories|2005;
Healy, W.P.|NONRELATIVISTIC QUANTUM ELECTRODYNAMICS|1982;
Hecht, K.T.|THE VECTOR COHERENT STATE METHOD AND ITS APPLICATION TO PROBLEMS OF HIGHER SYMMETRIES|1987;
Hedrich, R.|Complex and fundamental structures: Limits of reductionism. (In German)|1990;
Hedrich, Reiner|Raumzeitkonzeptionen in der Quantengravitation (Spacetime in Quantum Gravity)|2011;
Heerikhuisen, Jacob, (ed.)|Physics of the inner heliosheath, Voyager observations, theory, and future prospects, 5th annual IGPP International Astrophysics Conference, Waikiki Beach, Honolulu, Hawaii, 3-9 March 2006|2006;
Heiliger, P.|CP violation in rare K decays. (In German)|1994;
Heim, B.|ELEMENTARY STRUCTURES OF MATTER. UNIFIED STRUCTURAL QUANTUM FIELD THEORY OF MATTER AND GRAVITATION. VOL. 1. (IN GERMAN)|1989;
Heim, B.|ELEMENTARY STRUCTURES OF MATTER. UNIFIED STRUCTURAL QUANTUM FIELD THEORY OF MATTER AND GRAVITATION. VOL. 2. (IN GERMAN)|1984;
Heim, B.|INTRODUCTION TO BURKHARD HEIM: ELEMENTARY STRUCTURES OF MATTER. (IN GERMAN)|1985;
Heim, B.|Unified description of the material world: Informal summary of 'Elementary structures of matter*, vol. 1 and 2. (In German)|1990;
Heinonen, O., (ed.)|Composite fermions: A unified view of the quantum Hall regime|1998;
Held, A., (ed)|GENERAL RELATIVITY AND GRAVITATION. 100-YEARS AFTER THE BIRTH ALBERT EINSTEIN. VOL. 1|1980;
Held, A., (ed)|GENERAL RELATIVITY AND GRAVITATION. 100-YEARS AFTER THE BIRTH OF ALBERT EINSTEIN. VOL. 2|1980;
Hellborg, Ragnar, (ed.)|Electrostatic accelerators: Fundamentals and application|2005;
Heller, M.|Theoretical foundations of cosmology: Introduction to the global structure of space-time|1992;
Hengartner, W.|Introduction to the Monte Carlo Method|1978;
Henkel, M.|Conformal invariance and critical phenomena|1999;
Henneaux, M.|CLASSICAL FOUNDATIONS OF BRST SYMMETRY: LECTURES AT THE UNIVERSITY OF NAPLES|1988;
Henneaux, M.|Quantization of gauge systems|1992;
Hennig, J.D., (ed.)|Differential geometry, group representations, and quantization|1991;
Hermann, A.|HISTORY OF CERN. VOL. 1: LAUNCHING THE EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH|1987;
Hermann, A.|History of CERN. Vol. 2: Building and running the laboratory, 1954 - 1965|1990;
Hermann, R.|Gauge Fields and Cartan-Ehresmann Connections, Part a. Interdisciplinary Mathematics. 10.|1975;
Hermann, R.|Spinors, Clifford and Cayley Algebra. Interdisciplinary Mathematics, Vol. 7|1974;
Hermann, R.|The Geometry of Nonlinear Differential Equations, Backlund Transformations, and Solitons. Part A|1976;
Hermann, R.|The Geometry of Nonlinear Differential Equations, Backlund Transformations, and Solitons. Part B|1977;
Hermann, R.|Toda Lattices, Cosymplectic Manifolds, Backlund Transformations and Kinks. Part A|1977;
Hermann, R.|Toda Lattices, Cosymplectic Manifolds, Backlund Transformations and Kinks. Part B|1977;
Hermann, R.|Yang-Mills, Kaluza-Klein, and the Einstein Program|1978;
Herrmann, D.B.|Antimatter: Searching for the mirror world|1999;
Herrmann, Dieter B.|Big bang in the laboratory: How particle accelerators simulate nature|2010;
Hetznecker, Helmut|Expansion history of the universe: From hot big bang to cold cosmos|2007;
Hewett, JoAnne L., (ed.)|2005 International Linear Collider Workshop : LCWS 2005 : Stanford, California, USA, 18-22 March, 2005|2005;
Hey, A.|THE QUANTUM UNIVERSE|1987;
Hey, T.|The Quantum universe: The World of waves and particles. (In German)|1990;
Hey, T.|The new quantum universe|2003;
Heyde, K.L.G.|Basic ideas and concepts in nuclear physics: An Introductory approach|1995;
Heyde, K.|Basic ideas and concepts in nuclear physics: An introductory approach|1999;
Heyde, K.|From nucleons to the atomic nucleus: Perspectives in nuclear physics|1998;
Heyde, Kris|Basic ideas and concepts in nuclear physics: An introductory approach|2004;
Hicks, H.R.|Two-variable expansions and the k ---> 3pi decays|1972;
Hillman, Jonathan|Four-manifolds, geometries and knots|2002;
Hilscher, H.|Elementary particle physics. (In German)|1996;
Hilscher, H.|Nuclear physics. (In German)|1996;
Hinterberger, F.|The physics of particle accelerators and ion optics. (In German)|1997;
Hladik, J.|Spinors in physics|1999;
Ho- Kim, Q.|Invitation to contemporary physics|1991;
Ho-Kim, Quang, (ed.)|Elementary particles and their interactions|1998;
Hobson, M.P.|General relativity: An introduction for physicists|2006;
Hodgson, P.E.|Introductory nuclear physics|1997;
Hoffstaetter, G.H.|High-energy polarized proton beams: A modern view|2006;
Hofling, O.|THE WORLD OF SMALLEST PARTICLES. DISCOVERING THE STRUCTURE OF MATTER (IN GERMAN)|1984;
Hofmann, Albert|The physics of synchrotron radiation|2004;
Hofmann, C.|Electron positron angular correlation: Electron positron angular correlation in pair conversion of aligned nuclei in heavy ion collisions. (In German)|1995;
Hogan, C.J.|The little book of the big bang: A cosmic primer|1998;
Hogan, C.J.|The little book of the big bang|2000;
Hoh, Fang-Chao|Scalar strong interaction hadron theory|2011;
Hohler, G.|COLLECTIVE ION ACCELERATION|1979;
Hohler, G.|ELASTIC AND CHARGE EXCHANGE SCATTERING OF ELEMENTARY PARTICLES. PION NUCLEON SCATTERING. TABLES OF DATA|1982;
Hohler, G.|HANDBOOK OF PION NUCLEON SCATTERING|1979;
Hohler, G.|NUMERICAL DATA AND FUNCTIONAL RELATIONSHIPS IN SCIENCE AND TECHNOLOGY. GROUP I: NUCLEAR AND PARTICLE PHYSICS. VOL. 9: ELASTIC AND CHARGE EXCHANGE SCATTERING OF ELEMENTARY PARTICLES. B: PION NUCLEON SCATTERING. PT. 2:
Hollik, W.|Electroweak precision tests at LEP|2000;
Holstein, Barry R., (ed.)|Annual review of nuclear and particle science, vol. 59|2009;
Holstein, Barry R., (ed.)|Annual review of nuclear and particle science. Vol. 60|2010;
Holstein, Barry R.|Topics in advanced quantum mechanics|1992;
Holstein, Barry R.|WEAK INTERACTIONS IN NUCLEI|1985;
Holt, Stephen S., (ed.)|Gamma-ray bursts in the Swift era, sixteenth Maryland Astrophysics Conference, Washington, DC, 29 November - 2 December 2005|2006;
Hooper, Dan|Dark cosmos : in search of our universe's missing mass and energy|2006;
Hoppe, J.|Lectures on integrable systems|1992;
Horejsi, J.|Fundamentals of electroweak theory|2002;
Horejsi, J.|Introduction to electroweak unification: Standard model from tree unitarity|1993;
Hori, K.|Mirror symmetry|2003;
Horn, D.|HADRON PHYSICS AT VERY HIGH-ENERGIES|1973;
Horowitz, C.J.|Links between heavy ion and astrophysics|2006;
Horrocks, D.L.|Applications of Liquid Scintillation Counting|1974;
Horuzhii, S.S.|Introduction to algebraic quantum field theory|1990;
Horvath, D.|EXOTIC ATOMS. A BIBLIOGRAPHY 1939 - 1982|1984;
Horvathy, P.A.|INTRODUCTION TO MONOPOLES|1988;
Horwitz, Lawrence P.|Classical gravity as an eikonal approximation to a manifestly Lorentz covariant quantum theory with Brownian interpretation|2004;
Hosaka, A.|Quarks, baryons and chiral symmetry|2001;
Hou, Bo-Yu|Differential geometry for physicists|1999;
Hou, George W.S.|Flavor physics and the TeV scale|2009;
Houben, Pieter Willem Huib|A Measurement of the Mass of the Top Quark Using the Ideogram Technique|2009;
Howard, F.T.|Cyclotrons-1975, AVF and FM|1975;
Hoyle, Fred|Action-At-a-Distance in Physics and Cosmology|1974;
Hoyle, Fred|Lectures on cosmology and action at a distance electrodynamics|1996;
Hoyng, Peter|Relativistic astrophysics and cosmology: A primer|2006;
Hsu, J.P., (ed.)|Lorentz and Poincare invariance: 100 years of relativity|2001;
Hsu, J.P.|Einstein's relativity and beyond: New symmetry approaches|2000;
Hsu, Jong-Ping, (ed.)|100 years of gravity and accelerated frames: The deepest insights of Einstein and Yang-Mills|2005;
Hu, S.|Lecture notes on Chern-Simons-Witten theory|2001;
Huang, Kerson|Fundamental forces of nature: The story of gauge fields|2007;
Huang, Kerson|QUARKS, LEPTONS AND GAUGE FIELDS|1982;
Huang, Kerson|Quantum field theory: From operators to path integrals|1998;
Huang, Yi-Zhi|Two-dimensional conformal geometry and vertex operator algebras|1997;
Huber, M.G.|QUARKS - THE SUBSTANCE THAT BUILDS ATOMIC NUCLEI? DYNAMICAL PROCESSES IN PROTEINS. (IN GERMAN)|1987;
Hubsch, Tristan|Calabi-Yau manifolds: A Bestiary for physicists|1992;
Huehnchen, W.|Graviton as the most elementary mass object: Foundation of an alternative basic theory of our physical reality. (In German)|1991;
Huggett, S., (ed.)|Twistor theory|1995;
Huggett, S.A.|AN INTRODUCTION TO TWISTOR THEORY|1986;
Hughes, I.S.|ELEMENTARY PARTICLES|1986;
Hughes, V.W.|Muon Physics. 1. Electromagnetic Interactions|1977;
Hughes, V.W.|Muon Physics. 2. Weak Interactions|1975;
Hughes, V.W.|Muon Physics. 3. Chemistry and Solids|1975;
Hughston, L.P., (ed.)|ADVANCES IN TWISTOR THEORY|1979;
Hughston, L.P.|TWISTORS AND PARTICLES|1979;
Humpert, B.|Dynamical Concepts on Scaling Violation and the New Resonances in e+ e- Annihilation|1975;
Humphreys, J.E.|Introduction to Lie Algebras and Representation Theory. (3rd Print., Rev.)|1980;
Humphries, S.|Charged particle beams|1990;
Humphries, S.|PRINCIPLES OF CHARGED PARTICLE ACCELERATION|1986;
Hurt, N.E.|GEOMETRIC QUANTIZATION IN ACTION. APPLICATIONS OF HARMONIC ANALYSIS IN QUANTUM STATISTICAL MECHANICS AND QUANTUM FIELD THEORY|1984;
Husa, Sascha|Michele Maggiore: Gravitational waves. Volume 1: Theory and experiments|2009;
Huttemeister, Susanne, (ed.)|The Evolution of starbursts, the 331st Wilhelm and Else Heraeus Seminar, Bad Honnef, Germany, 16-20 August 2004|2005;
Hwa, R.C., (ed.)|Quark - gluon plasma. Vol. 2|1995;
Hwa, R.C., (ed.)|Quark - gluon plasma|1990;
Hwa, R.C., (ed.)|Quark-gluon plasma. Vol. 3|2004;
Iagolnitzer, D.|Scattering in quantum field theories: The Axiomatic and constructive approaches|1994;
Iagolnitzer, D.|The S Matrix|1978;
Icke, V.|The Force of symmetry|1995;
Ikeda, Kiyomi|Di-neutron clustering and deuteron-like tensor correlation in nuclear structure focusing on $^{11}Li$|2010;
Immler, Stefan, (ed.)|Supernova 1987A: 20 years after, Supernovae and gamma ray bursters, Aspen, Colorado, 19-23 February 2007|2007;
Infante, Leopoldo, (ed.)|XI IAU Regional Latin American Meeting on Astronomy, Pucon, Chile, diciembre 12-16, 2005|2006;
Ioffe, B.L.|HARD PROCESSES. VOL. 1: PHENOMENOLOGY, QUARK PARTON MODEL|1985;
Ioffe, Boris L.|Hans Bethe and the global energy problems|2005;
Ioffe, Boris Lazarevich, (ed.)|Quantum chromodynamics: Perturbative and nonperturbative aspects|2010;
Ionescu, Lucian M.|Cohomology of Feynman graphs and perturbative quantum field theory|2005;
Iorio, Lorenzo|Advances in the measurement of the Lense-Thirring effect with Satellite Laser Ranging in the gravitational field of the Earth|2008;
Iorio, Lorenzo|Some Applications of Binary Pulsars to Fundamental Physics|2010;
Irvine, J.M.|Heavy Nuclei, Superheavy Nuclei and Neutron Stars|1975;
Isaev, P.S.|QUANTUM ELECTRODYNAMICS AT HIGH-ENERGIES|1989;
Isayev, A.A.|Spin polarized states in nuclear matter with skyrme effective interaction|2004;
Isham, C.J.|Modern differential geometry for physicists|1999;
Isham, C.J.|Quantum Gravity. Oxford Symposium, Chilton, February 15-16, 1974|1975;
Islam, J.N.|An Introduction to mathematical cosmology|1992;
Israelit, M.|The Weyl-Dirac theory and our universe|1999;
Itzykson, C., (Ed.)|CONFORMAL INVARIANCE AND APPLICATIONS TO STATISTICAL MECHANICS|1988;
Itzykson, C.|QUANTUM FIELD THEORY|1980;
Itzykson, C.|STATISTICAL FIELD THEORY. VOL. 1: FROM BROWNIAN MOTION TO RENORMALIZATION AND LATTICE GAUGE THEORY|1989;
Itzykson, C.|STATISTICAL FIELD THEORY. VOL. 2: STRONG COUPLING, MONTE CARLO METHODS, CONFORMAL FIELD THEORY, AND RANDOM SYSTEMS|1989;
Iucci, N., (ed.)|Invited talks, 8th European Cosmic Ray Symposium, Roma, september 8-16, 1982|1983;
Ivancevic, Vladimir G.|Quantum leap: From Dirac and Feynman, across the universe, to human body and mind|2008;
Iwasawa, K.|LOCAL CLASS FIELD THEORY|1986;
Iyer, Bala R., (Ed.)|Black holes, gravitational radiation and the universe: Essays in honor of C.V. Vishveshwara|1999;
Iyer, Bala R., (Ed.)|GRAVITATION, GAUGE THEORIES AND THE EARLY UNIVERSE|1989;
Jackiw, R.|Diverse topics in theoretical and mathematical physics|1995;
Jackiw, R.|Topological aspects of gauge theories|2005;
Jackson, John David, (Ed.)|ANNUAL REVIEW OF NUCLEAR AND PARTICLE SCIENCE. VOL. 33|1984;
Jackson, John David, (Ed.)|ANNUAL REVIEW OF NUCLEAR AND PARTICLE SCIENCE. VOL. 34, 1984|1985;
Jackson, John David, (Ed.)|ANNUAL REVIEW OF NUCLEAR AND PARTICLE SCIENCE. VOL. 35, 1985|1986;
Jackson, John David, (Ed.)|ANNUAL REVIEW OF NUCLEAR AND PARTICLE SCIENCE. VOL. 36, 1986|1986;
Jackson, John David, (Ed.)|ANNUAL REVIEW OF NUCLEAR AND PARTICLE SCIENCE. VOL. 37, 1987|1987;
Jackson, John David, (Ed.)|ANNUAL REVIEW OF NUCLEAR AND PARTICLE SCIENCE. VOL. 38|1988;
Jackson, John David, (Ed.)|ANNUAL REVIEW OF NUCLEAR AND PARTICLE SCIENCE. VOL. 39|1989;
Jackson, John David, (ed.)|ANNUAL REVIEW OF NUCLEAR AND PARTICLE SCIENCE. VOL. 31|1981;
Jackson, John David, (ed.)|ANNUAL REVIEW OF NUCLEAR AND PARTICLE SCIENCE. VOL. 32, 1982|1982;
Jackson, John David, (ed.)|Annual review of nuclear and particle science. Vol. 40|1990;
Jackson, John David, (ed.)|Annual review of nuclear and particle science. Vol. 41|1991;
Jackson, John David, (ed.)|Annual review of nuclear and particle science. Vol. 42|1992;
Jackson, John David, (ed.)|Annual review of nuclear and particle science. Vol. 43|1994;
Jackson, John David|ANNUAL REVIEW OF NUCLEAR AND PARTICLE SCIENCE. VOL. 29, 1979|1979;
Jackson, John David|ANNUAL REVIEW OF NUCLEAR AND PARTICLE SCIENCE|1980;
Jackson, John David|Annual Review of Nuclear and Particle Science, Vol. 28, 1978|1978;
Jacob, M., (Ed.)|PERTURBATIVE QUANTUM CHROMODYNAMICS|1982;
Jacob, M., (Ed.)|THE ELEMENTARY PARTICLES. (IN FRENCH)|1985;
Jacob, M., (ed.)|CERN. 25-YEARS OF PHYSICS|1981;
Jacob, M.|Gauge Theories and Neutrino Physics|1978;
Jacob, M.|The Quark structure of matter|1992;
Jacobson, Ted|Astrophysical bounds on Planck suppressed Lorentz violation|2004;
Jade, S.|The Unified field theory's principles of dimensional relativity|1991;
Jaeger, R.G.|Dosimetry and Radiation Protection. Practical Data and Methods in Physics and Technology. 2nd Edition. (In German)|1974;
Jaeger, R.G.|Engineering Compendium on Radiation Shielding. 2. Shielding Materials|1975;
Jaffe, A., (Ed.)|QUANTUM FIELD THEORY. A SELECTION OF PAPERS IN MEMORIAM KURT SYMANZIK|1985;
Jaffe, Arthur M.|VORTICES AND MONOPOLES. STRUCTURE OF STATIC GAUGE THEORIES|1980;
Jahn, G.|Sfermion family mixing at one-loop level in the minimal supersymmetric standard model|2003;
James, Frederick|Statistical methods in experimental physics|2006;
Jammer, M.|Concepts of mass in contemporary physics and philosophy|2000;
Jancewicz, B., (Ed.)|QUANTUM THEORY OF PARTICLES AND FIELDS|1984;
Jancewicz, B., (ed.)|From field theory to quantum groups: Birthday volume dedicated to Jerzy Lukierski|1996;
Jancewicz, B.|MULTIVECTORS AND CLIFFORD ALGEBRA IN ELECTRODYNAMICS|1988;
Janke, W., (ed.)|Fluctuating paths and fields: Festschrift dedicated to Hagen Kleinert on the occasion of his 60th birthday|2001;
Janoschek, R., (ed.)|Chirality: From weak bosons to the alpha helix|1992;
Jansen, G.J.|Coulomb interactions in particle beams|1990;
Jantzen, J.C.|Lectures on quantum groups|1995;
Jaranowski, Piotr|Analysis of gravitational-wave data|2009;
Jarlskog, C., (Ed.)|CP Violation|1989;
Jefimenko, Oleg D.|Causality, electromagnetic induction and gravitation: A different approach to the theory of electromagnetic and gravitational fields|2000;
Jegerlehner, Friedrich|The anomalous magnetic moment of the muon|2008;
Jimbo, M., (ed.)|Integrable systems in quantum field theory and statistical mechanics|1989;
Jimbo, M., (ed.)|Yang-Baxter equation in integrable systems|1989;
Joglekar, Satish D.|Absence of nonlocal counterterms in the gauge boson propagator in the Axial type gauges|2003;
Johns, O.D.|Analytical mechanics for relativity and quantum mechanics|2005;
Johnson, C.V.|D-branes|2003;
Johnson, G.W.|The Feynman integral and Feynman's operational calculus|2000;
Johnson, G.|Strange beauty: Murray Gell-Mann and the revolution in twentieth-century physics|1999;
Johnson, W., (ed.)|Relativistic, quantum electrodynamic, and weak interaction effects in atoms. Program at the Institute of Theoretical Physics, Santa Barbara, USA, January - June, 1988|1989;
Joho, W.|7th International Conference on Cyclotrons and their Applications, Zurich, 19-22 August 1975|1975;
Johri, V.B., (ed.)|The early universe|1996;
Jones, H.F.|Groups, representations and physics|1990;
Joos, P.|High-Energy Physics|1977;
Joshi, A.W., (ed.)|Horizons of physics|1989;
Joshi, P.S.|Global aspects in gravitation and cosmology|1987;
Jost, R.|The Fairy tale of the ivory tower: Talks and essays. (In German)|1995;
Junker, G.|Supersymmetric methods in quantum and statistical physics|1996;
Kac, V.G.|BOMBAY LECTURES ON HIGHEST WEIGHT REPRESENTATIONS OF INFINITE DIMENSIONSAL LIE ALGEBRAS|1987;
Kac, V.G.|Infinite dimensional Lie algebras|1990;
Kac, V.|Vertex algebras for beginners|1996;
Kachelriess, Michael|Status of particle physics solutions to the UHECR puzzle|2004;
Kadanoff, L.P.|From order to chaos: Essays: critical, chaotic and otherwise|1994;
Kadanoff, L.P.|Statistical physics: Statics, dynamics and renormalization|2000;
Kadomsev, B.B.|On the pulsar|2010;
Kafatos, M.|The Conscious universe: Part and whole in modern physical theory|1990;
Kagan, Alexander L.|Right-handed currents, CP violation, and B ---> VV|2004;
Kaiser, Gerald|Quantum physics, relativity and complex space-time: Towards a new synthesis|1990;
Kaiser, Ralf I., (ed.)|Astrochemistry, From Laboratory Studies to Astronomical Observations, Honolulu, Hawaii, 18-20 December 2005|2006;
Kaku, M.|Beyond Einstein: The cosmic quest for the theory of the universe|1997;
Kaku, M.|Hyperspace: A Scientific odyssey through parallel universes, time warps, and the tenth-dimension|1994;
Kaku, M.|INTRODUCTION TO SUPERSTRINGS|1988;
Kaku, M.|Introduction to superstrings and M theory|1999;
Kaku, M.|Parallel worlds: A journey through creation, higher dimensions, and the future of the cosmos|2005;
Kaku, M.|Quantum field theory: A Modern introduction|1993;
Kaku, M.|Strings, conformal fields, and M-theory|2000;
Kaku, M.|Strings, conformal fields, and topology: An Introduction|1991;
Kalinovsky, A.N.|PASSAGE OF HIGH-ENERGY PARTICLES THROUGH MATTER|1989;
Kalinowski, M.W.|Nonsymmetric fields: Theory and its applications|1990;
Kalka, H.|Supersymmetry. (In German)|1997;
Kallen, G.|QUANTUM ELECTRODYNAMICS. (ENGLISH TRANSLATION)|1972;
Kalloniatis, A.C., (Ed.)|Lattice hadron physics|2004;
Kalnins, J.R.|BEVALAC EXTRACTION AND EXTERNAL BEAMLINE OPTICS|1990;
Kamae, Tuneyoshi|REPORT OF THE TRISTAN E P (e anti-e) WORKING GROUP|1980;
Kamefuchi, S., (Ed.)|FOUNDATIONS OF QUANTUM MECHANICS IN THE LIGHT OF NEW TECHNOLOGY|1984;
Kaminker, A.M., (ed.)|Physics of neutron stars|1995;
Kampfer, Burkhard|Cosmic phase transitions|1996;
Kane, Gordon L., (ed.)|Perspectives on Higgs physics|1993;
Kane, Gordon L., (ed.)|Perspectives on supersymmetry. Vol.2|2010;
Kane, Gordon L., (ed.)|Perspectives on supersymmetry|1998;
Kane, Gordon L., (ed.)|The supersymmetric world: The beginning of the theory|2000;
Kane, Gordon L.|MODERN ELEMENTARY PARTICLE PHYSICS|1987;
Kane, Gordon L.|Supersymmetry: Squarks, photinos, and the unveiling of the ultimate laws of nature|2000;
Kane, Gordon L.|The Particle garden: Our universe as understood by particle physicists|1995;
Kane, Gordon, (ed.)|Perspectives on LHC physics|2008;
Kanitscheider, B.|THE WORLD VIEW OF ALBERT EINSTEIN. (IN GERMAN)|1988;
Kanzieper, Eugene|Exact replica treatment of nonHermitean complex random matrices|2003;
Kapchinsky, I.M.|THEORY OF RESONANCE LINEAR ACCELERATORS|1985;
Kapitza, S.P.|The Microtron|1978;
Kapusta, J.I.|Finite-temperature field theory: Principles and applications|2006;
Kapusta, Joseph I.|FINITE TEMPERATURE FIELD THEORY|1989;
Kapusta, Joseph I.|FINITE TEMPERATURE FIELD THEORY|1989;
Karamanolis, S.|Creation of the universe: the big bang and its consequences. (in german)|1990;
Karamanolis, S.|IN THE BEGINNING THERE WAS ONLY ENERGY: CREATION PROCESSES OF MICROCOSMOS, MACROCOSMOS, AND BIOCOSMOS. (IN GERMAN)|1987;
Karamanolis, S.|MYSTERIES OF MICROCOSMOS: MATTER - ATOMS - QUARKS - (PREONS?). (IN GERMAN)|1986;
Karpf, A.D.|Structure of Elementary Particle Matter|1976;
Karshenboim, Savely G., (Ed.)|Astrophysics, clocks and fundamental constants|2004;
Karwowski, W.|Recent Development in Relativistic Quantum Field Theory and Its Application. 1. Winter School of Theoretical Physics, Karpacz, February 17-29, 1976|1976;
Karwowski, W.|Recent Development in Relativistic Quantum Field Theory and Its Application. 2. Winter School of Theoretical Physics, Karpacz, February 17-29, 1976|1976;
Kase, K.R., (ed.)|The Dosimetry of ionizing radiation. Vol. 3|1990;
Kashiwa, T.|Path integral methods|1997;
Kassel, C.|Quantum groups|1995;
Kastler, D., (ed.)|Quantum groups, noncommutative geometry and fundamental physical interactions|1999;
Katyshev, Yu.V.|ENGLISH-RUSSIAN HIGH-ENERGY PHYSICS DICTIONARY: ABOUT 23000 TERMS|1984;
Katz, Jonathan I.|HIGH-ENERGY ASTROPHYSICS|1987;
Katz, Sheldon|Enumerative geometry and string theory|2006;
Katz, U.F.|Deep inelastic positron proton scattering in the high-momentum-transfer regime of HERA|2000;
Katzenstein, Larry, (ed.)|A matter of time|2006;
Kauffman, L.H., (ed.)|Knots and applications|1995;
Kauffman, L.H.|Knots and physics|1991;
Kaufmann, W.J., (ed.)|PARTICLES AND FIELDS|1980;
Kawarabayashi, K., (Ed.)|WANDERING IN THE FIELDS: FESTSCHRIFT FOR KAZUHIKO NISHIJIMA ON THE OCCASION OF HIS SIXTIETH BIRTHDAY. SYMPOSIUM, TOKYO, JAPAN, MARCH 23-24, 1987|1987;
Kayser, Boris, (Ed.)|Annual Review of Nuclear and Particle Science, Vol. 57|2007;
Kayser, Boris, (Ed.)|Annual review of nuclear and particle science, vol. 54|2004;
Kayser, Boris, (Ed.)|Annual review of nuclear and particle science|2006;
Kayser, Boris, (ed.)|Annual review of nuclear and particle science. Vol. 58|2008;
Kayser, Boris|The Physics of massive neutrinos|1989;
Kazakov, Kirill A.|Long range behavior in quantum gravity|2004;
Kehrein, S.|The flow equation approach to many-particle systems|2006;
Kennefick, Daniel|Traveling at the speed of thought: Einstein and the quest for gravitational waves|2007;
Kennepohl, P.|Spectroscopy,electronic structure of [fex(4)] (x = c1, sr) complexes: contributions to electron transfer in bioinorganic chemistry|2003;
Kenyon, I.R.|ELEMENTARY PARTICLE PHYSICS|1987;
Kenyon, I.R.|General relativity|1990;
Kerler, T.|Non-semisimple topological quantum field theories for 3-manifolds with corners|2001;
Kessler, J.|Polarized Electrons|1976;
Ketov, S.V.|Conformal field theory|1995;
Ketov, S.V.|Quantum nonlinear sigma models: From quantum field theory to supersymmetry, conformal field theory, black holes and strings|2000;
Keyl, M.|On the geometrical structure of classical fields and their relation to local quantum field theory. (In German)|1994;
Khalatnikov, I.M.(Ed.)|PHYSICS REVIEWS. VOL. 2 (1980)|1990;
Khalatnikov, I.M.(Ed.)|PHYSICS REVIEWS. VOL. 8|1987;
Khalatnikov, I.M., (Ed.)|PHYSICS REVIEWS. VOL. 3 (1981)|1988;
Khalatnikov, I.M., (ed.)|30 years of the Landau Institute: Selected papers|1996;
Khalatnikov, I.M.|PHYSICS REVIEWS. VOL. 4 (1982)|1988;
Khan, S.|Collective phenomena in synchrotron radiation sources: Prediction, diagnostics, countermeasures|2006;
Khandekar, D.C.|Path integral methods and their applications|1993;
Khanna, Faqir C.|Thermal quantum field theory - Algebraic aspects and applications|2009;
Khare, A.|Fractional statistics and quantum theory|1997;
Khlopov, M.Yu.|Cosmological pattern of microphysics in the inflationary universe|2004;
Khlopov, M.Yu.|Cosmoparticle physics|1999;
Khmelnytskaya, Kira V.|Biquaternions for analytic and numerical solution of equations of electrodynamics|2007;
Khriplovich, I.B.|CP violation without strangeness: Electric dipole moments of particles, atoms, and molecules|1997;
Khristiansen, G.B.|COSMIC RAYS OF SUPERHIGH-ENERGIES|1980;
Kiefer, C.|Gravitation|2003;
Kiefer, C.|Quantum gravity|2004;
Kiefer, Claus|Can the Arrow of Time be understood from Quantum Cosmology?|2009;
Kiesling, Christian M.|TESTS OF THE STANDARD THEORY OF ELECTROWEAK INTERACTIONS|1988;
Kijowski, J.|A SYMPLECTIC FRAMEWORK FOR FIELD THEORIES|1979;
Kikkawa, K., (ed.)|A quest for symmetry: Selected works of Bunji Sakita|1999;
Kilian, W.|Electroweak symmetry breaking: The bottom-up approach|2003;
Kim, C.W.|Neutrinos in physics and astrophysics|1994;
Kim, Victor T., (ed.)|First International Workshop, Hadron Structure and QCD, from low to high energies, Repino, St. Petersburg, Russia, 18-22 May 2004|2004;
Kim, Y.D.|TABLES OF ALL IRREDUCIBLE REPRESENTATIONS FOR ALL CLASSICAL GROUP: COMPLETE WEIGHT SYSTEM, BRANCHING MATRICES WITH U(1) AND ANOMALY TABLE|1981;
Kim, Y.S., (ed.)|Quantum systems: New trends and methods|1997;
Kim, Y.S.|Can the quark model be relativistic enough to include the parton model?|2008;
Kim, Y.S.|THEORY AND APPLICATIONS OF THE POINCARE GROUP|1986;
Kinoshita, T., (ed.)|Quantum electrodynamics|1990;
Kirilyuk, Andrei P.|Quantum field mechanics: Complex dynamical completion of fundamental physics and its experimental implications|2004;
Kiritsis, Elias|Introduction to superstring theory|1997;
Kiritsis, Elias|String theory in a nutshell|2007;
Kirkby, Karen J., (ed.)|Ion implantation technology, 16th International Conference on Ion Implantation Technology, IIT 2006, Marseille, France, 11-16 June 2006|2006;
Kirsten, Klaus|Spectral functions in mathematics and physics|2001;
Kiselev, V.G.|Introduction to quantum field theory|2000;
Kittel, W.|Soft multihadron dynamics|2005;
Klapdor, H.V., (Ed.)|NEUTRINOS|1988;
Klapdor-Kleingrothaus, H.V.|Non-accelerator particle physics|1995;
Klapdor-Kleingrothaus, H.V.|Particle astrophysics|1997;
Klapdor-Kleingrothaus, H.V.|Particle physics without accelerators. (In German)|1995;
Klapdor-Kleingrothaus, H.V.|Seventy years of double beta decay: From nuclear physics to beyond-standard-model particle physics|2010;
Klapdor-Kleingrothaus, H.V.|Sixty years of double beta decay: From nuclear physics to beyond standard model particle physics|2001;
Klauder, J.R.|Beyond conventional quantization|2000;
Klauder, J.R.|MAGIC WITHOUT MAGIC - JOHN ARCHIBALD WHEELER. A COLLECTION OF ESSAYS IN HONOR OF HIS 60TH BIRTHDAY|1972;
Kleimenov, V.F.|COORDINATE SENSITIVE DETECTORS. (IN RUSSIAN)|1979;
Klein, E.|Beneath the atom the particles. (In French)|1993;
Klein, E.|The quest for unity: The adventure of physics|1999;
Kleinert, H.|Critical properties of phi**4-theories|2001;
Kleinert, H.|Gauge fields in condensed matter. Vol. 1: Superflow and vortex lines. Disorder fields, phase transitions|1989;
Kleinert, H.|Gauge fields in condensed matter. Vol. 2: Stresses and defects. Differential geometry, crystal melting|1989;
Kleinert, Hagen|Multivalued fields. In condensed matter, electromagnetism, and gravitation|2008;
Kleinknecht, K., (Ed.)|PARTICLES AND DETECTORS. FESTSCHRIFT FOR JACK STEINBERGER|1986;
Kleinknecht, K.|DETECTORS FOR PARTICLE RADIATION. (IN GERMAN)|1984;
Kleinknecht, K.|Uncovering CP violation: Experimental clarification in the neutral $K$ meson and $B$ meson|2003;
Kleinknecht, Konrad|Detectors for particle radiation|1998;
Klimyk, A.|Quantum groups and their representations|1997;
Knizhnik, V.G.|PHYSICS REVIEWS. VOL. 10, PT. 1. MULTILOOP AMPLITUDES IN THE THEORY OF QUANTUM STRINGS AND COMPLEX GEOMETRY|1989;
Knoll, G.F.|RADIATION DETECTION AND MEASUREMENT|2000;
Kobzarev, I.Yu.|ELEMENTARY PARTICLES: MATHEMATICS, PHYSICS AND PHILOSOPHY|1989;
Koch, E.E|Synchrotron Radiation at DESY. A User's Manual|1977;
Kock, A.|Synthetic differential geometry|1981;
Kodama, Hideo|Interplay between high energy physics and cosmophysics: KEK cosmophysics group inaugural conference 'Accelerators in the Universe'. Tsukuba, Japan, March 12-14 2008|2008;
Kodiyalam, V.|Topological quantum field theories from subfactors|2001;
Koebler, Ulrich|Renormalization group theory: Impact on experimental magnetism|2010;
Kogut, J.B.|The phases of quantum chromodynamics: From confinement to extreme environments|2004;
Kohno, T., (ed.)|New developments in the theory of knots|1990;
Kokorelis, Christos|Standard model building from intersecting D-branes|2004;
Koks, Don|Explorations in Mathematical Physics|2006;
Kolanoski, Hermann|TWO PHOTON PHYSICS AT E+ E- STORAGE RINGS|1984;
Kolb, Adrienne W.|Fermilab : physics, the frontier, and megasience|2008;
Kolb, Edward W., (Ed.)|THE EARLY UNIVERSE. REPRINTS|1988;
Kolb, Edward W.|The Early universe|1990;
Komar, A.A., (Ed.)|CLASSICAL AND QUANTUM EFFECTS IN ELECTRODYNAMICS|1988;
Komar, A.A., (Ed.)|GROUP THEORY, GRAVITATION AND ELEMENTARY PARTICLE PHYSICS|1987;
Komar, A.A., (Ed.)|QUANTIZATION, GRAVITATION AND GROUP METHODS IN PHYSICS|1988;
Komarek, P.|High current application of superconductivity|1995;
Konechny, Anatoly|Noncommutative tori, Yang-Mills and string theory|2005;
Kong, Kyoungchul|Extra Dimensions at the LHC|2010;
Konno, Hitoshi|Elliptic Quantum Group U(q,p) affine(sl(2)) and Vertex Operators|2008;
Konopelchenko, B.G.|Introduction to multidimensional integrable equations: The Inverse spectral transform in (2+1)-dimensions|1992;
Konopelchenko, B.G.|NONLINEAR INTEGRABLE EQUATIONS. RECURSION OPERATORS, GROUP THEORETICAL AND HAMILTONIAN STRUCTURES OF SOLITON EQUATIONS|1987;
Konopinski, E.J.|ELECTROMAGNETIC FIELDS AND RELATIVISTIC PARTICLES|1981;
Konopleva, N.P.|Gauge Fields|1981;
Kopaleishvili, T.|Collision theory: A short course|1995;
Kopczynski, W.|Space-time and gravitation|1992;
Kopp, G.|Introduction to quantum electrodynamics. (In German)|1997;
Kopylov, G.I.|ELEMENTARY KINEMATICS OF ELEMENTARY PARTICLES|1985;
Korner, J.G.|Current Induced Reactions. International Summer Institute on Theoretical Particle Physics, Hamburg, 1975|1976;
Koster, L.|Neutron Physics. Springer Tracts in Modern Physics, Volume 80-Neutron Scattering Lengths and Fundamental Neutron Interactions. Very Low-Energy Neutrons|1977;
Kosyakov, B.P.|Introduction to the classical theory of particles and fields|2007;
Kota, V.K.B., (ed.)|Neutrinoless double beta decay|2008;
Kounnas, C.|GRAND UNIFICATION WITH AND WITHOUT SUPERSYMMETRY AND COSMOLOGICAL IMPLICATIONS|1985;
Kovner, Alex|Variational techniques in non-perturbative QCD|2004;
Kovras, O., (ed.)|Focus on quantum field theory|2005;
Kovras, O., (ed.)|Frontiers in field theory|2005;
Kovras, O., (ed.)|Quantum field theory: New research|2005;
Kowalczynski, J.K.|The Tachyon and its fields|1996;
Kox, A.J., (ed.)|The universe of general relativity|2005;
Kragh, H.|Quantum generations: A history of physics in the twentieth century|1999;
Kragh, Helge S.|Conceptions of cosmos: From myths to the accelerating universe. A history of cosmology|2007;
Krainov, V.P.|Approximation Methods of Quantum Mechanics|1977;
Kramer, G.|THEORY OF JETS IN ELECTRON POSITRON ANNIHILATION|1984;
Kramer, Peter D.|GROUPS, SYSTEMS AND MANY BODY PHYSICS|1980;
Kramer, Peter D.|Group Theoretical Methods in Physics. 6th International Colloquium, Tubingen 1977|1978;
Krane, K.S.|INTRODUCTORY NUCLEAR PHYSICS|1987;
Krasinski, Andrzej|Inhomogeneous cosmological models|1997;
Krasnoholovets, Volodymyr V., (Ed.)|New topics in quantum physics research|2006;
Kraus, G.|Has Hawking erred?|1993;
Kraus, John Daniel|Our cosmic universe|1980;
Krause, H.|The Universe: Why it is there: New scientific proofs of creation. (In German)|1992;
Krauss, L.M.|Hiding in the mirror: The mysterious allure of extra dimensions, from Plato to string theory and beyond|2005;
Krauss, L.M.|The Fifth essence: The search for dark matter in the universe|1989;
Krauss, L.|Quintessence: The mystery of the missing mass in the universe|2000;
Kreimer, D.|Knots and Feynman diagrams|2000;
Kriele, M.|Space-time: Foundations of general relativity and differential geometry|1999;
Krige, J., (ed.)|History of CERN|1996;
Kroll, P.|PHENOMENOLOGICAL ANALYSES OF NUCLEON NUCLEON SCATTERING|1981;
Kropp, W.R., (ed.)|Neutrinos and other matters: Selected works of Frederick Reines|1991;
Kruglov, S.I.|Symmetry and electromagnetic interaction of fields with multispin|2001;
Krupchitsky, P.A.|FUNDAMENTAL RESEARCH WITH POLARIZED SLOW NEUTRONS. (TRANSLATION FROM RUSSIAN)|1987;
Kubono, S., (ed.)|Origin of matter and evolution of galaxies, International Symposium on Origin of Matter and Evolution of Galaxies 2005 : new horizon of nuclear astrophysics and cosmology, Tokyo Japan, 8-11 November 2005|2006;
Kubyshin, Yu.A.|DIMENSIONAL REDUCTION OF GAUGE THEORIES, SPONTANEOUS COMPACTIFICATION AND MODEL BUILDING|1989;
Kuhlen, M.|QCD at HERA: The hadronic final state in deep inelastic scattering|1999;
Kuhlman, S.|Physics and technology of the Next Linear Collider: A Report submitted to Snowmass '96|1996;
Kuhlmann, M., (ed.)|Ontological aspects of quantum field theory|2002;
Kuhn, W.|Quantum field theory: Photons and their interpretation. (In German)|1995;
Kuhne, Rainer W.|Cartan's torsion: Necessity and observational evidence|2004;
Kuhne, Rainer W.|Possible observation of a second kind of light: Magnetic photon rays|2004;
Kukulin, V.I.|CLUSTERS AS SUBSYSTEMS IN LIGHT NUCLEI. DIRECT CLUSTER REACTIONS - PROGRESS TOWARDS A UNIFIED THEORY|1983;
Kulikov, D.A.|Regge trajectories of the Klein-Gordon equation with non-minimal interaction|2006;
Kulikov, D.A.|Renormalization of expansions for Regge trajectories of the Schrodinger equation|2006;
Kullander, S.|Out of sight: From quarks to living cells|1994;
Kumar, Manjit|Quantum: Einstein, Bohr, and the great debate about the nature of reality|2008;
Kunz, J.|Dark matter in the universe. (In German)|1990;
Kupershmidt, B.A., (ed.)|Integrable and superintegrable systems|1990;
Kupsc, Andrzej, (ed.)|Meson Physics at COSY-11 and WASA-At-COSY, An International symposium, Krakow, Poland, 17-22 June 2007|2007;
Kursunoglu, B.N., (Ed.)|PAUL ADRIEN MAURICE DIRAC: REMINISCENCES ABOUT A GREAT PHYSICIST|1987;
Kuzelev, M.V., (ed.)|Basics of plasma free electron lasers|1995;
Kuznetsov, A.|Electroweak processes in external electromagnetic fields|2004;
Labastida, Jose|Topological quantum field theory and four manifolds|2005;
Lachieze-Rey, M.|Cosmology: A First course|1996;
Lachieze-Rey, M.|The cosmological background radiation|1999;
Lacki, Jan, (ed.)|E.C.G.Stueckelberg, an unconventional figure of twentieth century physics|2009;
Lahiri, Amitabha|A first book of quantum field theory|2005;
Lai, C.H., (Ed.)|IDEALS AND REALITIES: SELECTED ESSAYS OF ABDUS SALAM|1987;
Lai, C.H., (ed.)|GAUGE THEORY OF WEAK AND ELECTROMAGNETIC INTERACTIONS. (SELECTED PAPERS)|1981;
Lam, Chi-Sing|The Zen in modern cosmology|2008;
Lamb, F.K., (Ed.)|HIGH-ENERGY ASTROPHYSICS|1986;
Lamb, G.L.|ELEMENTS OF SOLITON THEORY|1980;
Lambe, L.A.|Introduction to the quantum Yang-Baxter equation and quantum groups: An algebraic approach|1997;
Landau, L.D.|TEXTBOOK ON THEORETICAL PHYSICS. VOL. 2: CLASSICAL FIELD THEORY. (IN GERMAN)|1987;
Landau, R.H.|Quantum mechanics. Vol. 2: A second course in quantum theory|1990;
Landua, R., (ed.)|Medium-energy anti-protons and the quark - gluon structure of hadrons|1991;
Landua, Rolf|At the edge of the dimensions: Discourse on physics at CERN|2008;
Lang, K.R.|Astrophysical formulae: Vol. 1: Radiation, gas processes and high-energy astrophysics. Vol. 2: Space, time, matter and cosmology|1999;
Langacker, P., (ed.)|Precision tests of the standard electroweak model|1996;
Langacker, Paul|The standard model and beyond|2010;
Langanke, K., (ed.)|Computational nuclear physics. Vol. 2: Nuclear reactions|1993;
Langmann, Edwin|Bosons and fermions in external fields|2005;
Langouche, F.|FUNCTIONAL INTEGRATION AND SEMICLASSICAL EXPANSIONS|1982;
Lanius, K., (ed.)|PHYSICS OF ELEMENTARY PARTICLES. (IN GERMAN)|1981;
Lanius, K.|Elementary Particles 1977|1978;
Lanius, K.|MICROCOSMOS, MACROCOSMOS: THE PHYSICAL VIEW. (IN GERMAN)|1988;
Lansberg, Jean-Philippe, (Ed.)|HLPR 2004: Hadronic Physics: Joint meeting Heidelberg-Liege-Paris-Rostock|2005;
Lapidus, Michel L.|In search of the Riemann zeros: Strings, fractal membranes and noncommutative spacetimes|2008;
Laszlo, E.|The Interconnected universe: Conceptual foundations of transdisciplinary unified theory|1996;
Latham, R.V., (ed.)|High voltage vacuum insulation: Basic concepts and technological practice|1995;
Lattimer, James M.|A Generalized equation of state for hot, dense matter|1991;
Laurent, B.|Introduction to space-time: A First course on relativity|1995;
Lavro, A.S., (ed.)|Neutrinos: A bibliography with indexes|2002;
Lawden, D.F.|Introduction to tensor calculus, relativity and cosmology|2002;
Lawrie, I.D.|A Unified grand tour of theoretical physics|1990;
Lawson, H.B.|Spin geometry|1998;
Lawson, H.Blaine, Jr.|THE THEORY OF GAUGE FIELDS IN FOUR-DIMENSIONS|1985;
Lawson, J.D.|The Physics of Charged Particle Beams|1977;
Lazarides, George|Supersymmetric dark matter, inflation and Yukawa quasi-unification|2004;
Le Bellac, M.|Quantum and statistical field theory|1991;
Le Guillou, J.C., (ed.)|Large order behavior of perturbation theory|1990;
Le Jan, Yves|Markov Paths, loops and fields|2011;
Le Yaouanc, A.|HADRON TRANSITIONS IN THE QUARK MODEL|1988;
LeBrun, Claude|The Einstein-Maxwell Equations, Extremal Kahler Metrics, and Seiberg-Witten Theory|2008;
Leader, E.|AN INTRODUCTION TO GAUGE THEORIES AND THE 'NEW PHYSICS.'|1982;
Leader, E.|An Introduction to gauge theories and modern particle physics. Vol. 1: Electroweak interactions, the new particles and the parton model|1996;
Leader, E.|An Introduction to gauge theories and modern particle physics. Vol. 2: CP violation, QCD and hard processes|1996;
Leader, E.|Spin in particle physics|2001;
Leane, Elizabeth M.|Reading Popular Physics|2007;
Lebellac, M.|FROM CRITICAL PHENOMENA TO GAUGE FIELDS. AN INTRODUCTION TO THE METHODS AND APPLICATIONS OF QUANTUM FIELD THEORY. (IN FRENCH)|1988;
Lechtenfeld, Olaf|WDVV solutions from orthocentric polytopes and Veselov systems|2008;
Lecoq, P.|Inorganic scintillators for detector systems: Physical principles and crystal engineering|2006;
Lederman, L.M., (Ed.)|APPRAISING THE RING: STATEMENTS IN SUPPORT OF THE SUPERCONDUCTING SUPER COLLIDER|1988;
Lederman, L.M.|From quarks to the cosmos: Particle physics as a key to the universe. (In German)|1990;
Lederman, L.|The God particle: If the universe is the answer, what is the question?|1993;
Lederman, Leon M.|Symmetry and the beautiful universe|2004;
Lee, J.Y.|Production of neutral strange particles in central S-32 + Au collisions at 200-GeV/nucleon. (In German)|1995;
Lee, S.Y.|Accelerator physics|1999;
Lee, S.Y.|Spin dynamics and snakes in synchrotrons|1997;
Lee, T.D.|PARTICLE PHYSICS AND INTRODUCTION TO FIELD THEORY|1981;
Lee, T.D.|SYMMETRIES, ASYMMETRIES, AND THE WORLD OF PARTICLES|1988;
Lee, T.D.|T.D. LEE: SELECTED PAPERS. VOL. 1: WEAK INTERACTIONS AND EARLY PAPERS|1986;
Lee, T.D.|T.D. LEE: SELECTED PAPERS. VOL. 2: FIELD THEORY AND SYMMETRY PRINCIPLES|1986;
Lee, T.D.|T.D. LEE: SELECTED PAPERS. VOL. 3: RANDOM LATTICES TO GRAVITY|1986;
Lefschetz, S.|Applications of Algebraic Topology. Graphs and Networks. the Picard-Lefschetz Theory and Feynman Integrals|1975;
Leggett, A.J.|THE PROBLEMS OF PHYSICS|1987;
Lehmann, D.|Mathematical methods of many-body quatum field theory|2004;
Lehnert, Bo|A revised electromagnetic theory with fundamental applications|2008;
Leibbrandt, G.|Noncovariant gauges: Quantization of Yang-Mills and Chern-Simons theory in axial type gauges|1994;
Leite Lopes, J., (ed.)|GAUGE FIELD THEORIES. AN INTRODUCTION|1981;
Lemoine, Martin, (ed.)|Inflationary cosmology|2008;
Lenz, F., (ed.)|Lectures on QCD: Applications|1997;
Lenz, F., (ed.)|Lectures on QCD: Foundations|1997;
Leo, W.R.|TECHNIQUES FOR NUCLEAR AND PARTICLE PHYSICS EXPERIMENTS: A HOW TO APPROACH|1987;
Leon, J.J.P., (Ed.)|NONLINEAR EVOLUTIONS|1988;
Leonardo, Nuno|Matter antimatter fluctuations, search, discovery and analysis of Bs flavor oscillations|2011;
Leplin, J., (ed.)|The Creation of ideas in physics: Studies for a methodology of theory construction|1996;
Lerda, A.|Anyons: Quantum mechanics of particles with fractional statistics|1992;
Lerner, E.J.|The Big Bang never happened|1991;
Lerner, Eric J., (ed.)|1st Crisis in Cosmology Conference, CCC-I, 23-25 June 2005, Moncao, Portugal|2006;
Lesch, Harald|The briefest history of all life: A report on the cycle of growth and decay in 13.7 billion years|2008;
Letessier, Jean|Hadrons and quark - gluon plasma|2002;
Levin, Jana J.|How the universe got its spots: Diary of a finite time in a finite space|2002;
Levy, Joseph|Aether theory and the principle of relativity|2006;
Li, Bing-Ren, (ed.)|Introduction to operator algebras|1992;
Li, Gang, (ed.)|The Physics of collisionless shocks, 4th annual IGPP International Astrophysics Conference, Palm Springs, California, 26 February - 3 March 2005|2005;
Li, Jian-Liang|Performance of the upgraded laser system for the Fermilab-NIU photoinjector|2006;
Li, Miao, (ed.)|Physics in noncommutative world. Vol. 1: Field theories|2002;
Lichtenberg, D.B., (ed.)|DEVELOPMENTS IN THE QUARK THEORY OF HADRONS. VOL. 1. 1964 - 1978|1980;
Lichtenberg, D.B.|Unitary Symmetry and Elementary Particles|1978;
Lichtenberg, Don|The universe and the atom|2007;
Lichtenegger, Herbert|Mach's principle|2004;
Liddle, Andrew R.|An introduction to modern cosmology|1998;
Liddle, Andrew R.|Cosmological inflation and large scale structure|2000;
Liddle, Andrew|An introduction to modern cosmology|2009;
Lidsey, J.E.|The bigger bang|2000;
Lieb, Elliott H.|Studies in Mathematical Physics, Essays in Honor of Valentine Bargmann|1976;
Lieb, Elliott H.|The stability of matter: From atoms to stars. Selecta of Elliott H. Lieb|2001;
Liebscher, D.E.|Cosmology: An Introduction for students of astronomy, physics, and mathematics. (In German)|1995;
Liebscher, D.E.|Cosmology|2005;
Liebscher, D.E.|The geometry of time|2005;
Lilley, John S.|Nuclear physics: Principles and applications|2009;
Lim, Y.K., (ed.)|Problems and solutions on atomic, nuclear and particle physics: Major American universities Ph.D. qualifying questions and solutions|2000;
Limper, M.|Track and Vertex Reconstruction in the ATLAS Inner Detector|2009;
Lin, S.H., (ed.)|Advances in multi-photon processes and spectroscopy|2008;
Lincoln, Donald W.|Understanding the universe: From quarks to the cosmos|2004;
Lincoln, Don|The quantum frontier: The Large Hadron Collider|2009;
Lindberg Christensen, Lars|Hidden universe|2009;
Linde, Andrei D.|Elementary particles and inflationary universe: On present formation of theories. (In German)|1993;
Linde, Andrei D.|Inflation and quantum cosmology|1990;
Linde, Andrei D.|Particle physics and inflationary cosmology|2005;
Lindley, D., (ed.)|Cosmology and particle physics|1991;
Lindley, D.|The End of physics: The Myth of a unified theory|1995;
Lindley, D.|The end of physics: The myth of the grand unified theory. (In German)|1997;
Lindroos, Mats|Beta beams: Neutrino beams|2010;
Lipatov, L.N., (ed.)|The creation of quantum chromodynamics and the effective energy|1998;
Lippitsch, Angelika|A deformation analysis method for the metrological ATLAS cavern network at CERN|2007;
Liu, C.S., (ed.)|Chen Ning Yang: A great physicist of the twentieth century|1995;
Liu, C.S.|Interaction of electromagnetic waves with electron beams and plasmas|1995;
Liu, K.F., (Ed.)|CHIRAL SOLITONS. A REVIEW VOLUME|1987;
Livio, M.|The accelerating universe: Infinite expansion, the cosmological constant, and the beauty of the cosmos|2000;
Lo, S.Y., (Ed.)|GEOMETRICAL PICTURES IN HADRONIC COLLISIONS. A REPRINT VOLUME|1987;
Loaiza, Pia, (ed.)|Topical Workshop on Low Radioactivity Techniques, LRT 2006, Aussois (France), 1-4 October 2006|2007;
Lockwood, M.|The labyrinth of time: Introducing the universe|2005;
Lockyer, T.N.|Vector particle physics|1992;
Loebl, E.M.|Group Theory and Its Applications. 3.|1975;
Loginova, E.A.|Meeting on Programming and Mathematical Methods for Solving the Physical Problems, Dubna, October 30 - November 4, 1973|1974;
Logunov, A.A.|On the articles by Henri Poincare: On the dynamics of the electron|2001;
Logunov, A.A.|Relativistic theory of gravity and the Mach principle|1997;
Logunov, A.A.|Relativistic theory of gravity|1998;
Logunov, A.A.|The Theory of gravity|2001;
Lohmus, J.|Nonassociative algebras in physics|1995;
Lohrmann, E.|HIGH-ENERGY PHYSICS. (IN GERMAN)|2005;
Lohrmann, E.|High-Energy Physics|1978;
Lohrmann, E.|INTRODUCTION TO ELEMENTARY PARTICLE PHYSICS. (IN GERMAN)|1983;
Lohrmann, Erich|On fast particles and intense light|2009;
Longair, M.S., (ed.)|High-energy astrophysics. Vol. 1: Particles, photons and their detection|1992;
Longair, M.S.|Confrontation of Cosmological Theories with Observational Data. Symposium No. 63, Cracow, 10-12 September 1973|1974;
Longair, M.S.|HIGH-ENERGY ASTROPHYSICS. AN INFORMAL INTRODUCTION FOR STUDENTS OF PHYSICS AND ASTRONOMY|1981;
Longair, M.S.|High-energy astrophysics. Vol. 2: Stars, the galaxy and the interstellar medium|1994;
Longair, M.S.|Our evolving universe|1996;
Lonza, M.|Multibunch feedback systems|2009;
Lopez, C.|STRONG INTERACTIONS AT HIGH-ENERGIES. (IN SPANISH)|1979;
Lorente, Miguel|A Realistic interpretation of lattice gauge theories|1996;
Lorente, Miguel|Discrete reflection groups and induced representations of Poincare group on the lattice|2004;
Lorente, Miguel|Modern theories of discrete time|2003;
Lorinczi, Jozsef|Feynman-Kac-type theorems and Gibbs measures on path space: With applications to rigorous quantum field theory|2011;
Low, F.E.|Classical field theory: Electromagnetism and gravitation|1997;
Lubatti, H.J.|Particles and Fields 1975. Meeting of the Division of Particles and Fields of the American Physical Society, Seattle, 27-29 August 1975|1976;
Lucha, W.|Elementary particle physics in theory and experiment. (In German)|1997;
Lucha, W.|Strong interaction: An Introduction to nonrelativistic potential models. (In German)|1989;
Luchini, P.|Undulators and free electron lasers|1990;
Ludwig, W.|SYMMETRIES IN PHYSICS: GROUP THEORY APPLIED TO PHYSICAL PROBLEMS|1988;
Ludwig, W.|The extended unified quantum field theory of Burkhard Heim.|1998;
Lukyanov, Sergei L.|Physics reviews: Additional symmetries and exactly soluble models in two-dimensional conformal field theory|1990;
Luminet, J.P.|Black holes. (In German)|1997;
Luminet, J.P.|Black holes|1992;
Luminet, Jean-Pierre|The wraparound universe|2008;
Luminet, Jean-Pierre|Time, Topology and the Twin Paradox|2009;
Lundqvist, S., (ed.)|Nobel lectures including presentation speeches and laureates' biographies: Physics 1971 - 1980|1992;
Luo, Xiang-Qian|Works of Xiang-Qian Luo. Vol. 1|2008;
Luo, Xiang-Qian|Works of Xiang-Qian Luo. Vol. 2|2008;
Luscher, E.|MODERN PHYSICS. FROM THE MICROSTRUCTURE OF MATTER TO THE BUILDING OF THE UNIVERSE. (IN GERMAN)|1987;
Lust, D.|Lectures on string theory|1989;
Lutz, G.|Semiconductor radiation detectors: Device physics|1999;
Lyons, L.|STATISTICS FOR NUCLEAR AND PARTICLE PHYSICISTS|1986;
Lyre, H.|Local symmetries and reality|2004;
Lyth, David H.|The primordial density perturbation: Cosmology, inflation and the origin of structure|2009;
Ma, Zhong-Qi|Yang-Baxter equation and quantum enveloping algebras|1994;
Maalampi, Jukka|The world line: Albert Einstein and modern physics|2008;
Mac Gregor, M.H.|The power of alpha. Electron elementary particle generation with alpha-quantized lifetimes and masses|2006;
Mac Gregor, Malcolm H.|The Enigmatic electron|1992;
Machner, H.|Introduction to nuclear and elementary particle physics|2005;
Mackey, G.W.|Unitary Group Representations in Physics, Probability and Number Theory|1978;
Madore, J.|An introduction to noncommutative differential geometry and itsphysical applications|2000;
Madriz Aguilar, Jose Edgar|A Scalar field governed cosmological model from noncompact Kaluza-Klein theory|2004;
Maeda, Hideki|Kinematic self-similar solutions in general relativity|2004;
Maggiore, Michele|A Modern introduction to quantum field theory|2005;
Maggiore, Michele|Gravitational Waves. Vol. 1: Theory and Experiments|2007;
Magnon, A.|Arrow of time and reality: In search of a conciliation|1997;
Magris, Gladis, (ed.)|XII IAU Regional Latin American Meeting, Isla Margarita, Venezuela, Octubre 22-26, 2007|2009;
Magueijo, J.|Faster than the speed of light: The story of a scientific speculation|2002;
Mahan, Gerald D.|Quantum mechanics in a nutshell|2009;
Maia, M.D.|Geometry of the fundamental interactions: On Riemann's legacy to high energy physics and cosmology|2011;
Maia, Marcos Duarte|Brane worlds and cosmology|2004;
Maiani, L., (ed.)|The DAPHNE physics handbook. Vol. 1, 2|1992;
Maiani, L., (ed.)|The second DAPHNE physics handbook. Vol. 1, 2|1995;
Maillard, J.M., (ed.)|Yang-Baxter equations.|1994;
Mainzer, K.|SYMMETRIES OF NATURE: A HANDBOOK ON THE PHILOSOPHY OF NATURE AND SCIENCE. (IN GERMAN)|1988;
Majer, U., (ed.)|Semantical aspects of space-time theories|1994;
Majid, S.|Foundations of quantum group theory|1996;
Makeenko, Yu.|Methods of contemporary gauge theory|2002;
Makhankov, V.G.|The Skyrme model: Fundamentals, methods, applications|1993;
Malament, David B.|Classical general relativity|2005;
Mallios, Anastasios|Modern differential geometry in gauge theories. Vol. 1: Maxwell fields|2006;
Mallios, Anastasios|Modern differential geometry in gauge theories. Vol. 2: Yang-Mills fields|2010;
Mallios, Anastasios|Space-time foam dense singularities and de Rham cohomology|2005;
Mandl, F.|QUANTUM FIELD THEORY|1985;
Mangiarotti, L.|Connections in classical and quantum field theory|2000;
Mangiarotti, L.|Gauge mechanics|1998;
Mangiarotti, L.|Geometric and algebraic topological methods in quantum mechanics|2005;
Mania, H., (ed.)|The grand Stephen Hawking reader: Life and work|2004;
Manin, Yu.I.|GAUGE FIELD THEORY AND COMPLEX GEOMETRY|1988;
Manin, Yu.I.|Topics in noncommutative geometry|1991;
Manko, V.I., (ed.)|Research in quantum field theory|1996;
Manko, V.I., (ed.)|Theory of the interaction of multilevel systems with quantized fields|1996;
Mann, A.K., (ed.)|Neutrino interactions with electrons and protons: An Account of an experimental program in particle physics in the 1980s|1994;
Mann, Robert B.|An introduction to particle physics and the standard model|2010;
Mannel, T.|Effective Field Theories in Flavor Physics|2004;
Mannheim, P.D.|Brane-localized gravity|2005;
Mannheim, Philip D.|Brane-localized gravity|2005;
Manohar, Aneesh V.|Heavy quark physics|2000;
Manoukian, E.B.|Quantum theory: A wide spectrum|2006;
Manoukian, Edouard B.|RENORMALIZATION|1984;
Manton, N.S.|Topological solitons|2004;
Manturov, V.O.|Knot theory|2004;
Marathe, K.B.|The Mathematical foundations of gauge theories|1992;
Marcus, M.|Finite Dimensional Multilinear Algebra. 2.|1975;
Margaritondo, G.|Elements of synchrotron light: For biology, chemistry, and medical research|2002;
Margaritondo, G.|INTRODUCTION TO SYNCHROTRON RADIATION|1988;
Marinari, Enzo|Numerical Simulations of Spin Glass Systems|1997;
Marino, M.|Chern-Simons theory, matrix models, and topological strings|2005;
Markopoulou, Fotini|New directions in background independent quantum gravity|2007;
Markov, M.A., (Ed.)|THE PHYSICAL EFFECTS IN THE GRAVITATIONAL FIELD OF BLACK HOLES|1987;
Markov, M.A., (ed.)|Theory of nonstationary quantum oscillators|1992;
Marlow, A.R.|Mathematical Foundations of Quantum Theory. Presentations Made to a Conference at Loyola University, New Orleans, Jun 2-4, 1977|1978;
Marmo, G.|PARTICLE DYNAMICS ON FIBER BUNDLES|1988;
Marmo, Giuseppe|Dynamical systems, a differential geometric approach to symmetry and reduction|1985;
Marshak, R.E.|Conceptual foundations of modern particle physics|1993;
Marshakov, A.|Seiberg-Witten theory and integrable systems|1999;
Martens, T., (ed.)|Science and technology in Europe. (In German)|1990;
Martin, B.R.|Nuclear and particle physics|2006;
Martin, B.R.|Particle physics|1992;
Martin, B.R.|Particle physics|1997;
Martin, B.R.|Pion Pion Interactions in Particle Physics|1976;
Martin, Brian Robert|Nuclear and particle physics|2009;
Martin, Brian Robert|Particle physics|2008;
Martin, P.A.|Many body problems and quantum field theory: An introduction|2002;
Marton, L.|ADVANCES IN ELECTRONICS AND ELECTRON PHYSICS|1980;
Masetti, L.|Measurement of the K+- -> pi+ pi- e+- nu/e form factors and of the pi pi scattering length alpha(00)|2007;
Mason, L.J., (ed.)|Further advances in twistor theory. Vol. 2: Integrable systems, conformal geometry and gravitation|1995;
Mason, L.J., (ed.)|Further advances in twistor theory. Vol. III: Curved twistor spaces|2001;
Mason, L.J.|Integrability, selfduality, and twistor theory|1991;
Massey, H.|A PERSPECTIVE OF PHYSICS. VOL. 3 SELECTIONS FROM 1978 'COMMENTS ON MODERN PHYSICS'|1979;
Mastropietro, Vieri|Non-perturbative renormalization|2008;
Masujima, M.|Path integral quantization and stochastic quantization|2000;
Mathai, A.M.|MODERN PROBLEMS IN NUCLEAR AND NEUTRINO ASTROPHYSICS|1988;
Mathie, E.L., (Ed.)|TRIUMF USERS HANDBOOK|1987;
Mathur, Samir D.|How fast can a black hole release its information?|2009;
Matsuda, S., (Ed.)|PERSPECTIVES ON PARTICLE PHYSICS: FROM MESONS AND RESONANCES TO QUARKS AND STRINGS. SYMPOSIUM IN COMMEMORATION OF THE SIXTIETH BIRTHDAY OF PROFESSOR H. MIYAZAWA, TOKYO, JAPAN, MARCH 29, 1988|1989;
Matthews, J.M., (ed.)|High-energy astrophysics: Models and observations from MeV to EeV|1994;
Mattis, D.C.|The Many body problem: An Encyclopedia of exactly solved models in one-dimension|1993;
Mattuck, R.D.|A Guide to Feynman Diagrams in the Many Body Problem (Second Edition)|1976;
Matute, Ernesto A.|Topological charges, prequarks and presymmetry: A Topological approach to quark fractional charges|2004;
Mauldin, J.H.|PARTICLES IN NATURE: THE CHRONOLOGICAL DISCOVERY OF THE NEW PHYSICS|1986;
Maurer, W.|Development in the Field of High Current Superconductivity in the Kernforschungszentrum Karlsruhe|1976;
Mavromatos, Nick E.|Logarithmic conformal field theories and strings in changing backgrounds|2004;
Mayer-Kuckuk, T.|Nuclear physics: An Introduction. (In German)|1992;
Mayer-Kuckuk, T.|THE BROKEN MIRROR: SYMMETRY, SYMMETRY BREAKING AND ORDER IN NATURE. (IN GERMAN)|1989;
Mazzuchi, Sonia|Mathematical Feynman path integrals and their applications|2009;
McCabe, Gordon|The structure and interpretation of the standard model|2007;
McCarthy, I.E.|Electron - atom collisions|1995;
McComb, W.D.|Renormalization methods: A guide for beginners|2004;
McDonald, Jonathan R.|A Discrete Representation of Einstein's Geometric Theory of Gravitation: The Fundamental Role of Dual Tessellations in Regge Calculus|2008;
McMahon, David|Quantum field theory demystified: A self-teaching guide|2009;
McMahon, David|String theory demystified: A self-teaching guide|2009;
Mccusker, C.B.A.|THE QUEST FOR QUARKS|1989;
Measday, David F.|Nucleon Nucleon Interactions-1977, Vancouver|1978;
Mehra, J.|Climbing the mountain: The Scientific biography of Julian Schwinger|2000;
Mehra, J.|The Beat of a different drum: The Life and science of Richard Feynman|1994;
Mehra, J.|The golden age of theoretical physics. Vol. 2|2001;
Meinel, R.|Solitons: Nonlinear structures. (In German)|1991;
Mele, Salvatore|Physics of W bosons at LEP|2004;
Melia, Fulvio|The edge of infinity: Supermassive black holes in the universe|2003;
Melnikov, K.|Theory of the muon anomalous magnetic moment|2006;
Melrose, R.B.|Geometric scattering theory|1995;
Mensky, M.B.|Continuous quantum measurements and path integrals|1994;
Mersini-Houghton, Laura|Birth of the Universe from the Multiverse|2008;
Mess, K.H.|Superconducting accelerator magnets|1996;
Messiah, A.|QUANTUM MECHANICS. VOL. 2 (GERMAN TRANSLATION)|1979;
Meszaros, P.|High-energy radiation from magnetized neutron stars|1992;
Meyer, C.F.|Relativistically invariant orbits in elementary particles: An electromagnetic particle model|2005;
Meyer, Thomas S., (ed.)|Beam Instrumentation Workshop 2006 : Twelfth Beam Instrumentation Workshop, Batavia, Illinois, 1-4 May 2006|2006;
Meyer-Ortmanns, H.|Principles of phase structures in particle physics|2007;
Michelotti, Leo P.|Intermediate classical dynamics with applications to beam physics|1995;
Mickelsson, J.|CURRENT ALGEBRAS AND GROUPS|1989;
Mielke, E.W.|GEOMETRODYNAMICS OF GAUGE FIELDS. ON THE GEOMETRY OF YANG-MILLS AND GRAVITATIONAL GAUGE THEORIES|1987;
Migdal, Arkady B.|Qualitative Methods in Quantum Theory|1977;
Mikolajewska, Joanna, (ed.)|Stellar astrophysics with the world's largest telescopes, first International Workshop on Stellar Astrophysics With the World's Largest Telescopes, Torun, Poland, 7-10 September 2004|2005;
Miles, J., (ed.)|CERN Accelerator School, radio frequency engineering, Seeheim, Germany, 8-16 May 2000|2005;
Miller, A.I.|Early quantum electrodynamics: A source book|1994;
Miller, R.B.|AN INTRODUCTION TO THE PHYSICS OF INTENSE CHARGED PARTICLE BEAMS|1982;
Mills, R.L.|The grand unified theory of classical quantum mechanics|2000;
Milonni, P.W.|The Quantum vacuum: An Introduction to quantum electrodynamics|1994;
Milton, K.A., (ed.)|A quantum legacy: Seminal papers of Julian Schwinger|2000;
Milton, K.A.|Electromagnetic radiation: Variational methods, waveguides and accelerators|2006;
Milton, K.A.|The Casimir effect: Physical manifestations of zero-point energy|2001;
Minty, M.G.|Measurement and control of charged particle beams|2003;
Mintz, S.L.|Fundamental Theories in Physics. Orbis Scientiae Held at Coral Gables, January 7-12, 1974|1974;
Miransky, V.A.|Dynamical symmetry breaking in quantum field theories|1994;
Mirman, R.|Massless representations of the Poincare group: Electromagnetism, gravitation, quantum mechanics, geometry|1995;
Mirman, R.|Quantum field theory, conformal group theory, conformal field theory: Mathematical and conceptual foundations, physical and geometrical applications|2001;
Mirman, R.|Quantum mechanics, quantum field theory: Geometry, language, logic|2001;
Misra, Aalok, (ed.)|Theoretical high energy physics, International Workshop on Theoretical High Energy Physics, Roorkee, India, 15-20 March 2007|2007;
Misra, S.P.|Introduction to supersymmetry and supergravity|1992;
Mitra, Asoke N., (Ed.)|Quantum field theory: A 20th century profile|2000;
Miura, R.M.|Backlund Transformations, the Inverse Scattering Method, Solitons, and their Applications. NSF Research Workshop on Contact Transformations, Vanderbilt University, Nashville, TN on September 27-29, 1974|1976;
Mohapatra, R.N.|Massive neutrinos in physics and astrophysics. Second edition|1998;
Mohapatra, R.N.|Massive neutrinos in physics and astrophysics|1991;
Mohapatra, R.N.|UNIFICATION AND SUPERSYMMETRY. THE FRONTIERS OF QUARK - LEPTON PHYSICS|1986;
Mohapatra, Rabindra N., (ed.)|GAUGE THEORIES OF FUNDAMENTAL INTERACTIONS. (SELECTED PAPERS)|1981;
Mohr, Peter|Photon induced reactions in stars and in the laboratory: A Critical comparison|2004;
Molokovsky, S.I.|Intense electron and ion beams|2005;
Monastyrsky, M.|RIEMANN, TOPOLOGY, AND PHYSICS|1987;
Monastyrsky, M.|Topology of gauge fields and condensed matter|1993;
Montani, Giovanni|Primordial cosmology|2011;
Month, M., (ed.)|The Physics of particle accelerators: Based in part on USPAS seminars and courses in 1989 and 1990. Vol. 1, 2|1992;
Montvay, I.|Quantum fields on a lattice|1994;
Moore, John H.|Building scientific apparatus|2009;
Morandi, G., (ed.)|Field theories for low dimensional condensed matter systems: Spin systems and strongly correlated electrons|2000;
Morandi, G.|The Role of topology in classical and quantum physics|1992;
Moreno, H.M.|Topics in High-Energy Physics. Lecture Notes: 1977 Advanced School of Physics at the National Polytechnic Institute, Mexico|1978;
Moretti, Valter|Local $\zeta$-functions, stress-energy tensor, field fluctuations, and all that, in curved static spacetime|2010;
Morii, T.|The physics of the standard model and beyond|2004;
Morinari, Takao|Half-Skyrmion theory for high-temperature superconductivity|2009;
Moriyasu, K.|AN ELEMENTARY PRIMER FOR GAUGE THEORY|1984;
Morrison, P.|POWERS OF TEN: DIMENSIONS BETWEEN QUARKS AND GALAXIES|1984;
Mosel, U.|Fields, symmetries, and quarks|1989;
Mosel, U.|Path integrals in field theory: An introduction|2004;
Moshinsky, M.|Canonical Transformations and Quantum Mechanics. Latin American School of Physics 1974|1974;
Moss, I.G.|Quantum theory, black holes and inflation|1996;
Mostepanenko, V.M.|The Casimir effect and its applications|1997;
Motizuki, Y.|Required precision of mass and half-life measurements for r-process nuclei planned at future RI-beam facilities|2004;
Motz, L.|A Festschrift for I.I. Rabi|1977;
Mould, R.A.|Basic relativity|1995;
Mueller, A.H., (Ed.)|PERTURBATIVE QUANTUM CHROMODYNAMICS|1989;
Mukhanov, V.|Physical foundations of cosmology|2005;
Mukhanov, Viatcheslav|Introduction to quantum effects in gravity|2007;
Mukhin, K.N.|Amusing Nuclear Physics. (In German, Transl. From the Russian)|1974;
Mukhin, K.N.|EXPERIMENTAL NUCLEAR PHYSICS. VOL. 1: PHYSICS OF ATOMIC NUCLEUS|1987;
Mukhin, K.N.|EXPERIMENTAL NUCLEAR PHYSICS. VOL. 2: ELEMENTARY PARTICLE PHYSICS|1987;
Mukunda, N.|RELATIVISTIC MODELS OF EXTENDED HADRONS OBEYING A MASS SPIN TRAJECTORY CONSTRAINT|1982;
Muller, Berndt|THE PHYSICS OF THE QUARK - GLUON PLASMA|1983;
Muller, Berndt|THE STRUCTURED VACUUM|1984;
Muller-Kirsten, H.J.W.|An introduction including quantum effects|2004;
Muller-Kirsten, H.J.W.|SUPERSYMMETRY: AN INTRODUCTION WITH CONCEPTUAL AND CALCULATIONAL DETAILS|1986;
Muller-Krumbhaar, H., (ed.)|...and God does play dice: About investigating the very big, the very small and the very many things|2001;
Munowitz, M.|Knowing: The nature of physical law|2005;
Murdin, P.|End in fire: The Supernova in the Large Magellanic Cloud|1990;
Musiol, G.|NUCLEAR AND ELEMENTARY PARTICLE PHYSICS. (IN GERMAN)|1988;
Musiol, G.|NUCLEAR AND ELEMENTARY PARTICLE PHYSICS. VOL. I. (IN GERMAN)|1980;
Musser, George|The complete idiot's guide to string theory|2008;
Musto, Renato|From Heisenberg to Einstein? Recollections and afterthoughts on the birth of string theory|2008;
Muta, T.|Foundations of quantum chromodynamics. Second edition|1998;
Muta, T.|Foundations of quantum chromodynamics: An Introduction to perturbative methods in gauge theories|1987;
Myneni, Ganapati Rao, (ed.)|Hydrogen in materials and vacuum systems, first International Workshop on Hydrogen in Materials and Vacuum Systems : Newport News, Virginia, 11-13 November 2002|2003;
Myung, H.C.|LIE ALGEBRAS AND FLEXIBLE LIE ADMISSIBLE ALGEBRAS|1982;
Naber, G.L.|Topology, geometry, and gauge fields: Foundations|1997;
Naber, G.L.|Topology, geometry, and gauge fields: Interactions|2000;
Nachtmann, O.|ELEMENTARY PARTICLE PHYSICS: CONCEPTS AND PHENOMENA|1990;
Nachtmann, O.|PHENOMENA AND CONCEPTS OF ELEMENTARY PARTICLE PHYSICS. (IN GERMAN)|1986;
Nagaitsev, Sergei, (ed.)|Beam cooling and related topics, International Workshop on Beam Cooling and Related Topics - COOL05, Galena, Illinois, U.S.A. 18-23 September 2005|2006;
Nagamine, K.|Introductory muon science|2003;
Nagaosa, N.|Quantum field theory in condensed matter physics|1999;
Nagaosa, N.|Quantum field theory in strongly correlated electronic systems|1999;
Nagashima, Yorikiyo|Elementary particle physics. Vol. 1: Quantum field theory and particles|2010;
Nagels, M.M|Compilation of Coupling Constants and Low-Energy Parameters. 1978 Edition|1979;
Nagl, A.|Nuclear pion photoproduction|1991;
Nair, V.P.|Quantum field theory: A modern perspective|2005;
Nakagawa, M.|Scientific works of Masami Nakagawa|2001;
Nakahara, M.|Geometry, topology and physics|1990;
Nakahara, M.|Geometry, topology and physics|2003;
Nakai, K., (ed.)|KEK PS 1990 - 1994: A Summary of experimental programs at the KEK PS 1990 - 1994|1995;
Nakamura, T.|Handbook on secondary particle production and transport by high-energy heavy ions|2006;
Nakanishi, N.|Covariant operator formalism of gauge theories and quantum gravity|1990;
Nambu, Y.|QUARKS. FRONTIERS IN ELEMENTARY PARTICLE PHYSICS|1985;
Namiki, M., (ed.)|Quantum physics, chaos theory, and cosmology|1996;
Namiki, M.|Stochastic quantization|1992;
Namsrai, K.|Stochastic and quantum background fluctuations in space-time and matter fields|1999;
Namsrai, Kh., (Ed.)|NONLOCAL QUANTUM FIELD THEORY AND STOCHASTIC QUANTUM MECHANICS|1986;
Namsrai, Khavtgain|Selected problems of contemporary pysics: Nonlocal point of view|2008;
Narison, Stephan|QCD SPECTRAL SUM RULES|1989;
Narison, Stephan|QCD as a theory of hadrons from partons to confinement|2002;
Narison, Stephan|QCD spectral sum rules|1989;
Narlikar, J.V.|GRAVITY, GAUGE THEORIES AND QUANTUM COSMOLOGY|1986;
Narlikar, J.V.|INTRODUCTION TO COSMOLOGY|1986;
Narlikar, J.V.|THE LIGHTER SIDE OF GRAVITY|1988;
Narlikar, J.V.|THE PRIMEVAL UNIVERSE|1988;
Nash, C.|Differential topology and quantum field theory|1991;
Nash, C.|Relativistic Quantum Fields|1978;
Nash, C.|TOPOLOGY AND GEOMETRY FOR PHYSICISTS|1983;
Nataf, R.|INTRODUCTION TO PARTICLE PHYSICS. (IN FRENCH)|1987;
Nath, Pran|APPLIED N=1 SUPERGRAVITY|1983;
Ne'eman, Yuval|Membranes and other extendons: p-branes|1996;
Ne'eman, Yuval|THE PARTICLE HUNTERS|1986;
Nechaev, S.K.|Statistics of knots and entangled random walks|1996;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 11|1979;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 12|1981;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 13|1984;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 14|1984;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 15|1986;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 16|1986;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 17|1986;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 18|1987;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 19|1989;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 20|1991;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 21|1995;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 22|1996;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 23|1996;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 24|1998;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 25|2000;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 26|2001;
Negele, John W., (Ed.)|Advances in nuclear physics. Vol. 27|2003;
Negele, John W.|QUANTUM MANY PARTICLE SYSTEMS|1988;
Nelson, D.|STATISTICAL MECHANICS OF MEMBRANES AND SURFACES|1989;
Neunhoffer, T.|Development of a new procedure to search for cosmic neutrino point sources with the AMANDA neutrino telescope|2004;
Newell, A.C.|SOLITONS IN MATHEMATICS AND PHYSICS|1986;
Newhouse, V.L.|Applied Superconductivity. 2.|1975;
Newton, R.G.|SCATTERING THEORY OF WAVES AND PARTICLES|1982;
Newton, R.G.|Thinking about physics|2000;
Newton, R.G.|What makes nature tick?|1995;
Ng, K.Y.|Physics of intensity dependent beam instabilities|2006;
Ng, Tai-Kai|Introduction to classical and quantum field theory|2009;
Ng, Y.J., (ed.)|Julian Schwinger: The physicist, the teacher, and the man|1996;
Ni, Wei-Tou|Empirical foundations of relativistic gravity|2005;
Nicholls, Jennifer A., (ed.)|From zero to infinity|2003;
Nicolaescu, L.I.|Notes on Seiberg-Witten theory|2000;
Nicolai, H.|Gravitational billiards, dualities and hidden symmetries|2005;
Nicolson, Iain|Dark side of the universe: Dark matter, dark energy, and the fate of the cosmos|2007;
Nikitin, Yu.P.|THEORY OF MULTIPARTICLE PRODUCTION PROCESSES|1988;
Nikolic, M.|KINEMATICS AND SYMMETRIES. VOL. 1. LECTURES PRESENTED AT THE INTERNATIONAL SCHOOL OF ELEMENTARY PARTICLE PHYSICS, BASKO POLJE & KUPARI, YUGOSLAVIA, 1971 - 1978|1979;
Nikolic, M.|KINEMATICS AND SYMMETRIES. VOL. 2. LECTURES PRESENTED AT THE INTERNATIONAL SCHOOL OF ELEMENTARY PARTICLE PHYSICS, BASKO POLJE & KUPARI, YUGOSLAVIA, 1971 - 1978|1979;
Nitsch, J., (ed.)|FUNDAMENTAL PROBLEMS OF MODERN PHYSICS. FESTSCHRIFT FOR PETER MITTELSTADT ON THE OCCASION OF HIS 50TH BIRTHDAY. (IN GERMAN)|1981;
Nojiri, Shin'ichi|Dark energy and modified gravities|2004;
Noldus, Johan|Foundations of a theory of quantum gravity|2011;
Nottale, L.|Fractal space-time and microphysics: Towards a theory of scale relativity|1993;
Novello, M., (ed.)|Artificial black holes|2002;
Novikov, I.D.|Black holes and the universe|1990;
Novikov, I.D.|PHYSICS OF BLACK HOLES|1989;
Novikov, S.P., (Ed.)|MATHEMATICAL PHYSICS REVIEWS. VOL. 2|1981;
Novikov, S.P., (Ed.)|Mathematical Physics Reviews. Vol. 1|1980;
Novikov, S.P.|MATHEMATICAL PHYSICS REVIEWS. VOL. 3|1990;
Novikov, S.P.|Solitons and geometry|1995;
Novikov, S.|THEORY OF SOLITONS. THE INVERSE SCATTERING METHOD|1984;
Novikov, Sergei P.|Modern geometric structures and fields|2006;
Novozhilov, Yu.V.|Introduction to Elementary Particle Theory|1975;
Nowak, Maciej A.|Chiral nuclear dynamics|1996;
Noyes, H.P.|Bit-string physics: A finite and discrete approach to natural philosophy|2001;
Noz, M.E., (Ed.)|SPECIAL RELATIVITY AND QUANTUM THEORY: A COLLECTION OF PAPERS OF THE POINCARE GROUP DEDICATED TO PROFESSOR EUGENE PAUL WIGNER ON THE 50TH ANNIVERSARY OF HIS PAPER ON 'UNITARY REPRESENTATIONS OF THE
Nutku, Y., (ed.)|Conformal field theory|2000;
Nyiri, J., (ed.)|The Gribov theory of quark confinement|2001;
O'Raifeartaigh, L.|GROUP STRUCTURE OF GAUGE THEORIES|1986;
O'Raifeartaigh, L.|The dawning of gauge theory|1997;
Oberhummer, H., (ed.)|Nuclei in the cosmos|1991;
Oberhummer, Heinz|Can all this be an accident? Misterious universe|2009;
Obukhov, V.V.|Conformally-flat Stackel spaces in Brans-Dicke theory|2009;
Oeckl, R.|Discrete gauge theory: From lattices to TQFT|2005;
Oerter, R.|The theory of almost everything: The standard model, the unsung triumph of modern physics|2006;
Ohanian, H.|Gravitation and space-time|1995;
Ohnuki, Y.|QUANTUM FIELD THEORY AND PARASTATISTICS|1982;
Ohnuki, Y.|UNITARY REPRESENTATIONS OF THE POINCARE GROUP AND RELATIVISTIC WAVE EQUATIONS|1988;
Ohsaku, Tadafumi|Chiral symmetry restoration through Hawking-Unruh thermalization effect|2005;
Ohtsuki, T.|Quantum invariants: A study of knots, 3-manifolds, and their sets|2002;
Okada, J.P.|7th Hawaii Topical Conference in Particle Physics, Aug 10-23 1977. Participant Seminars|1977;
Oks, Efim|Plasma cathode electron sources: Physics, technology, applications|2006;
Okubo, Susumu|Introduction to octonion and other nonassociative algebras in physics|1990;
Okun, L.B.|ALPHA, BETA, GAMMA ... Z. A PRIMER IN PARTICLE PHYSICS. (IN GERMAN)|1988;
Okun, L.B.|PARTICLE PHYSICS. THE QUEST FOR THE SUBSTANCE OF SUBSTANCE|1985;
Okun, L.B.|Physics of elementary particles. (In German)|1991;
Okun, L.B.|The Relations of particles|1991;
Okun, L.b.|LEPTONS AND QUARKS|1982;
Okun, Lev B.|Energy and mass in relativity theory|2009;
Oleinik, V.P.|The problem of electron and superluminal signals|2001;
Oloff, R.|Geometry of space-time: A mathematical introduction to relativity theory|1999;
Olshanetsky, M., (ed.)|Multiple facets of quantization and supersymmetry: Michael Marinov memorial volume|2002;
Oneda, S.|Asymptotic symmetry and its implication in elementary particle physics|1991;
Onuki, H., (ed.)|Undulators, wigglers and their applications|2003;
Orear, J.|PHYSICS. (IN GERMAN)|1982;
Oriti, Daniele|Approaches to quantum gravity: Toward a new understanding of space, time and matter|2009;
Ortin, Tomas|Gravity and strings|2004;
Ouseph, P.J.|Introduction to Nuclear Radiation Detectors|1975;
Overbye, D.|The Echo of big bang: Crucial questions of modern cosmology. (In German) (Lonely hearts of the cosmos)|1991;
Overduin, James M.|The light / dark universe|2008;
Oyibo, G.A.|Grand unified theorem: Discovery of the theory of everything and the fundamental building block of quantum theory|2004;
Oyibo, G.|Grand unified theorem|1999;
Paar, Hans P.|An introduction to advanced quantum physics|2010;
Padamsee, H.|RF superconductivity for accelerators|1998;
Padamsee, Hasan|RF superconductivity: Science, technology, and applications|2009;
Padmanabhan, T.|An invitation to astrophysics|2006;
Padmanabhan, T.|Understanding our universe: Current status and open issues|2005;
Padmanabhan, Thanu|Gravitation: Foundations and frontiers|2010;
Page, Don N.|Does God so love the multiverse?|2008;
Page, Don N.|Our place in a vast universe|2008;
Pagels, H.R.|PERFECT SYMMETRY. THE SEARCH FOR THE BEGINNING OF TIME|1985;
Pagels, H.R.|TIME BEFORE THE TIME. THE UNIVERSE UNTIL THE BIG BANG. (IN GERMAN)|1987;
Pais, A.|INWARD BOUND OF MATTER AND FORCES IN THE PHYSICAL WORLD|1986;
Pais, A.|Paul Dirac: The man and his work|1998;
Pakvasa, S., (Ed.)|HAWAII TOPICAL CONFERENCE IN PARTICLE PHYSICS. SELECTED LECTURES. VOL. 1|1982;
Pakvasa, S., (Ed.)|HAWAII TOPICAL CONFERENCE IN PARTICLE PHYSICS. SELECTED LECTURES. VOL. 2|1982;
Panofsky, W.K.H.|Particles and policy|1994;
Panofsky, Wolfgang K.H.|Panofsky on physics, politics, and peace: Pief remembers|2007;
Pantaleo, M.|RELATIVITY, QUANTA, AND COSMOLOGY IN THE DEVELOPMENT OF THE SCIENTIFIC THOUGHT OF ALBERT EINSTEIN. VOL. 1|1979;
Pantaleo, M.|RELATIVITY, QUANTA, AND COSMOLOGY IN THE DEVELOPMENT OF THE SCIENTIFIC THOUGHT OF ALBERT EINSTEIN. VOL. 2|1979;
Papantonopoulos, Eleftherios, (ed.)|From gravity to thermal gauge theories: The AdS/CFT correspondence|2011;
Papapetrou, A.|Lectures on General Relativity|1974;
Parikh, J.C.|Group Symmetries in Nuclear Structure|1978;
Parisi, G.|Field theory, disorder and simulations|1992;
Parisi, G.|STATISTICAL FIELD THEORY|1988;
Park, D.|THE HOW AND THE WHY: AN ESSAY ON THE ORIGINS AND DEVELOPMENT OF PHYSICAL THEORY|1988;
Parker, B.R.|CREATION: THE STORY OF THE ORIGIN AND EVOLUTION OF THE UNIVERSE|1988;
Parker, B.R.|Cosmic time travel: A Scientific odyssey|1991;
Parker, B.R.|SEARCH FOR A SUPERTHEORY: FROM ATOMS TO SUPERSTRINGS|1987;
Parker, B.|EINSTEIN'S DREAM. THE SEARCH FOR A UNIFIED THEORY OF THE UNIVERSE|1987;
Parker, B.|Invisible matter and the fate of the universe|1989;
Parker, B.|The Vindication of the big bang: Breakthroughs and barriers|1993;
Parker, S.P., (Ed.)|NUCLEAR AND PARTICLE PHYSICS SOURCE BOOK|1988;
Parsons, Paul|The big bang: The birth of our universe|2001;
Partridge, R.B.|3-K: The Cosmic microwave background radiation|1995;
Parzen, G.|Magnetic Fields for Transporting Charged Beams|1976;
Paschos, E.A.|Electroweak theory|2007;
Pascual, P.|QCD: RENORMALIZATION FOR THE PRACTITIONER|1984;
Pascual-Sanchez, J.F.|Isotropy of the velocity of light and the Sagnac effect|2003;
Patashinsky, A.Z.|FLUCTUATION THEORY OF PHASE TRANSITIONS. (ENGLISH TRANSLATION)|1979;
Patera, J.|TABLES OF BRANCHING RULES FOR REPRESENTATIONS OF SIMPLE LIE ALGEBRAS|1980;
Patterson, H.W., (ed.)|The History of accelerator radiological protection: Personal and professional memoirs|1995;
Paty, M.|Three Topics on Prospects in Neutrino Physics|1976;
Paul, E.|Elementary Particle Physics. Springer Tracts in Modern Physics, Vol. 79|1976;
Pavsic, Matej|The Landscape of theoretical physics: A Global view. From point particles to the brane world and beyond, in search of a unifying principle|2001;
Peacock, J.A.|Cosmological physics|1999;
Peaslee, D.C., (ed.)|Topics in hadron spectroscopy. V. 3|1995;
Peaslee, D.C., (ed.)|Topics in hadron spectroscopy. Vol. 2|1995;
Peaslee, D.C., (ed.)|Topics in hadron spectroscopy|1991;
Peat, F.D.|SUPERSTRINGS AND THE SEARCH FOR THE THEORY OF EVERYTHING. (IN GERMAN)|1989;
Peccei, R., (ed.)|Particle physics.: Perspectives and opportunities: Report of the DPF Committee on Long Term Planning|1995;
Pedersen, G.K.|C* ALGEBRAS AND THEIR AUTOMORPHISM GROUPS|1979;
Peebles, P.J.E.|Principles of physical cosmology|1994;
Peebles, P.James E.|Finding the big bang|2009;
Peierls, R.E.|More surprises in theoretical physics|1991;
Peierls, R.|A Perspective of Physics. 1. Selections from 1976 'Comments on Modern Physics'|1977;
Pellegrini, C., (ed.)|The development of colliders|1995;
Penner, R., (ed.)|Perspectives in mathematical physics|1994;
Penrose, R.|A complete guide to the laws of the universe|2005;
Penrose, R.|SPINORS AND SPACE-TIME. 1. TWO SPINOR CALCULUS AND RELATIVISTIC FIELDS|1985;
Penrose, R.|SPINORS AND SPACE-TIME. VOL. 2: SPINOR AND TWISTOR METHODS IN SPACE-TIME GEOMETRY|1986;
Penrose, Roger|Cycles of time: An extraordinary new view of the universe|2011;
Pepe, Alberto|Protocols for scholarly communication|2006;
Percacci, R.|GEOMETRY OF NONLINEAR FIELD THEORIES|1986;
Perelomov, A.M.|Generalized coherent states and their applications|1986;
Perez-Garcia, M.Angeles|Binding and Structure properties in Monte Carlo simulation for non-isosymmetric nuclei|2008;
Perkins, D.H.|High-energy physics. (In German)|1987;
Perkins, D.H.|Introduction to high energy physics|1982;
Perkins, D.H.|Particle astrophysics|2003;
Perl, M.L.|Reflections on experimental science|1996;
Perry, P.A.|SCATTERING THEORY BY THE ENSS METHOD|1984;
Peschanski, Robert B.|INTRODUCTION TO THE PHYSICAL PROBLEM OF QUARK CONFINEMENT|1979;
Peskin, Michael E.|An Introduction to quantum field theory|1995;
Pestov, Ivanhoe|Dark matter and potential fields|2004;
Peterson, Winfield H.|The dyon-twist model of fundamental particles: A theory of matter and energy|2005;
Petrov, A.N.|The Field theoretical formulation of general relativity and gravity with non-zero masses of gravitons|2004;
Petz, D.|An Invitation to the algebra of canonical commutation relations|1990;
Pham, F.|Hyperfunctions and Theoretical Physics. Rencontre de Nice, 21-30 May 1973|1975;
Pickering, A.|CONSTRUCTING QUARKS. A SOCIOLOGICAL HISTORY OF PARTICLE PHYSICS|1984;
Pickover, C.A.|Black holes: A traveler's guide|1996;
Pietschmann, H.|WEAK INTERACTIONS - FORMULAE, RESULTS AND DERIVATIONS|1984;
Piguet, O.|Algebraic renormalization: Perturbative renormalization, symmetries and anomalies|1995;
Piguet, O.|RENORMALIZED SUPERSYMMETRY. THE PERTURBATION THEORY OF N=1 SUPERSYMMETRIC THEORIES IN FLAT SPACE-TIME|1986;
Pike, E.R.|The Quantum theory of radiation|1986;
Pike, R., (ed.)|Scattering: Scattering and inverse scattering in pure and applied science. Vol. 1, 2|2002;
Pilkuhn, H.M.|RELATIVISTIC PARTICLE PHYSICS|1979;
Pilkuhn, H.M.|Relativistic quantum mechanics|2003;
Pincock, Stephen|The origins of the universe for dummies|2009;
Pinkau, K.|14th International Cosmic Ray Conference, Munich, August 15-29, 1975. 2. Invited Lectures and Rapporteur Papers|1975;
Pinkau, K.|14th International Cosmic Ray Conference, Munich, August 15-29, 1975. 3. Modulation and Geomagnetic Effect|1975;
Pinkau, K.|14th International Cosmic Ray Conference, Munich, August 15-29, 1975. 4. Modulation and Geomagnetic Effect and Pioneer Symposium|1975;
Pinkau, K.|14th International Cosmic Ray Conference, Munich, August 15-29, 1975. 7. High-Energy Physics|1975;
Pinkau, K.|14th International Cosmic Ray Conference, Munich, August 15-29, 1975. 9. Techniques|1975;
Pinkau, K.|14th International Cosmic Ray Conference, Munich, August 15-29, 1975. Volume 12. Late Papers, Corrections and List of Attendees|1975;
Pirani, F.A.E.|LOCAL JET BUNDLE FORMULATION OF BACKLUND TRANSFORMATIONS, WITH APPLICATIONS TO NONLINEAR EVOLUTION EQUATIONS|1979;
Pisello, D.M.|GRAVITATION, ELECTROMAGNETISM AND THE QUANTIZED CHARGE. THE EINSTEIN INSIGHT|1979;
Pismen, L.M.|Vortices in nonlinear fields: From liquid crystals to superfluids. From nonequilibrium patterns to cosmic strings|1999;
Pittner, L.|Algebraic foundations of noncommutative differential geometry and quantum groups|1996;
Plebanski, J.|An introduction to general relativity and cosmology|2006;
Plendl, H.s., (ed.)|PHILOSOPHICAL PROBLEMS OF MODERN PHYSICS|1982;
Poenaru, D.N., (ed.)|Experimental techniques in nuclear physics|1997;
Pohl, M.|Particles, forces and the vacuum|1998;
Pohl, Martin|Introduction to high energy astrophysics|2002;
Pokorski, S.|GAUGE FIELD THEORIES|1987;
Polchinski, J.|String theory. Vol. 1: An introduction to the bosonic string|1998;
Polchinski, J.|String theory. Vol. 2: Superstring theory and beyond|1998;
Polikarpov, A.|PHILOSOPHICAL PROBLEMS OF ELEMENTARY PARTICLE PHYSICS. (IN GERMAN)|1979;
Polkinghorne, J.C.|MODELS OF HIGH-ENERGY PROCESSES|1980;
Polkinghorne, J.C.|THE PARTICLE PLAY. AN ACCOUNT OF THE ULTIMATE CONSTITUENTS OF MATTER|1979;
Polonsky, Nir|Supersymmetry: Structure and phenomena. Extensions of the standard model|2001;
Polyakov, Alexander M.|GAUGE FIELDS AND STRINGS|1987;
Polyzou, W.N.|Euclidean formulation of relativistic quantum mechanics|2009;
Pool, Robert|YANG-MILLS FIELDS AND EXTENSION THEORY|1987;
Popov, V.N.|FUNCTIONAL INTEGRALS IN QUANTUM FIELD THEORY AND STATISTICAL PHYSICS|1984;
Popovic, Luka C., (ed.)|Spectral line shapes in astrophysics, VI Serbian Conference on Spectral Line Shapes in Astrophysics (VI SCSLSA) Sremski Karlovci, Serbia, 11-15 June 2007|2007;
Poth, H.|COMPILATION OF DATA FROM HADRONIC ATOMS|1979;
Potylitsyn, Alexander Petrovich|Electromagnetic radiation of electrons in periodic structures|2011;
Povh, B.|Particles and nuclei: An Introduction to the physical concepts|1995;
Povh, B.|Particles and nuclei: An introduction to the physical conceptions. (In German)|1993;
Povh, B.|Scattering and structures: Essentials and analogies in quantum physics|2005;
Pradhan, T.|The photon|2001;
Prakash, N.|Mathematical perspectives on theoretical physics: A journey from black holes to superstrings|2001;
Prasad, R.|ELEMENTARY PARTICLES|1985;
Prastaro, A., (ed.)|Geometry in partial differential equations|1994;
Prastaro, A.|Geometry of PDEs and mechanics|1996;
Preparata, G.|An introduction to a realistic quantum physics|2002;
Preparata, G.|QED coherence in matter|1996;
Pressley, A.|LOOP GROUPS|1988;
Preuss, K.H., (Ed.)|FEDERAL REPUBLIC OF GERMANY: A COUNTRY OF TOP QUALITY RESEARCH. (IN GERMAN)|1985;
Preuss, K.H., (Ed.)|STORIES THAT RESEARCH TELLS. A TEXT BOOK OF THE GERMAN RESEARCH SERVICE. VOL. 6: ON MIRACLES OF LIFE AND MIRACULOUS WORLDS. (IN GERMAN)|1987;
Preuss, K.H., (ed.)|Stories that research tells. Vol. 8: Of the old and new image of the world. (In German)|1989;
Preuss, K.H., (ed.)|Stories that research tells. Vol. 9: 60 voyages through science. (In German)|1990;
Preuss, K.H., (ed.)|Stories that research tells. Vol.2: On new elements and antique heads. A Text book of the German Research Service. (In German)|1988;
Price, W.C.|The Uncertainty Principle and Foundations of Quantum Mechanics. A Fifty Year's Survey|1977;
Priester, W.|BIG BANG AND COSMIC EVOLUTION - PROGRESS IN COSMOLOGY. (IN GERMAN)|1985;
Privman, V., (ed.)|Finite size scaling and numerical simulation of statistical systems|1990;
Proca, G.A., (ed.)|Alexandre Proca (1897 - 1955): Scientific publications. (Mostly in French)|1988;
Profumo, Stefano|Multi-Wavelength Searches for Particle Dark Matter|2010;
Pronin, P., (ed.)|Gravity, particles, and space-time|1996;
Pronin, P.I., (ed.)|Modern problems of theoretical physics: Festschrift for Professor D. Ivanenko|1991;
Prugovecki, E.|Principles of quantum general relativity|1995;
Prugovecki, E.|Quantum geometry: A Framework for quantum general relativity|1992;
Prugovecki, E.|STOCHASTIC QUANTUM MECHANICS AND QUANTUM SPACE-TIME. A CONSISTENT UNIFICATION OF RELATIVITY AND QUANTUM THEORY BASED ON STOCHASTIC SPACES|1984;
Prykarpatsky, A.K.|Quantum field theory with application to quantum nonlinear optics|2002;
Quadt, Arnulf|Top quark physics at hadron colliders|2007;
Quang, Ho-Kim|Elementary particles and their interactions: concepts and phenomena|1998;
Queen, N.M.|Dispersion Theory in High-Energy Physics|1974;
Quigg, C., (ed.)|Annual review of nuclear and particle science, vol. 53|2003;
Quigg, C., (ed.)|Annual review of nuclear and particle science. Vol. 44|1994;
Quigg, C., (ed.)|Annual review of nuclear and particle science. Vol. 45|1995;
Quigg, C., (ed.)|Annual review of nuclear and particle science. Vol. 46|1996;
Quigg, C., (ed.)|Annual review of nuclear and particle science. Vol. 49|1999;
Quigg, C., (ed.)|Annual review of nuclear and particle science. Vol. 50|2000;
Quigg, C., (ed.)|Annual review of nuclear and particle science. Vol. 51|2001;
Quigg, C., (ed.)|Annual review of nuclear and particle science. Vol. 52|2002;
Quigg, C.|GAUGE THEORIES OF THE STRONG, WEAK AND ELECTROMAGNETIC INTERACTIONS|1983;
Quigg, C.|Production and Detection of Intermediate Vector Bosons and Heavy Leptons in p p and anti-p p Collisions|1976;
Rabinowitz, Mario|Little black holes as dark matter candidates with feasible cosmic and terrestrial interactions|2005;
Radovanovic, Voja|Problem book in quantum field theory|2008;
Raffelt, G.G.|Stars as laboratories for fundamental physics: The astrophysics of neutrinos, axions, and other weakly interacting particles|1996;
Raine, D.J.|An introduction to the science of cosmology|2001;
Rajaraman, R.|SOLITONS AND INSTANTONS. AN INTRODUCTION TO SOLITONS AND INSTANTONS IN QUANTUM FIELD THEORY|1982;
Raju, C.K.|Time: Towards a consistent theory|1995;
Ramana Murthy, P.V.|GAMMA-RAY ASTRONOMY|1986;
Rammer, Jorgen|Quantum field theory of non-equilibrium states|2007;
Ramond, Pierre|FIELD THEORY. A MODERN PRIMER|1981;
Ramond, Pierre|Journeys beyond the standard model|1999;
Ramond, Pierre|Memoirs of an early string theorist|2007;
Rand, R.E.|RECIRCULATING ELECTRON ACCELERATORS|1984;
Randall, L.|Warped passages: Unraveling the mysteries of the universe's hidden dimensions|2005;
Randjbar-Daemi, S.|Abrikosov vortex and branes|2003;
Ranft, G.|Elementary Particles. An Introduction to High-Energy Physics. 2.|1977;
Ranft, G.|Elementary Particles: An Introduction to High-Energy Physics. 1.|1976;
Ranft, J.|Building blocks of the universe: Quarks and leptons. (In German)|1991;
Rao, M.V.S.|Extensive air showers|1998;
Rasetti, M.G., (ed.)|New problems, methods and techniques in quantum field theory and statistical mechanics|1990;
Ravn, I., (ed.).|Chaos, quarks, and black holes: ABC of the new sciences. (In German)|1996;
Ravndal, F.|Scaling and Renormalization Groups|1976;
Raychaudhuri, A.K.|General relativity, astrophysics, and cosmology|1992;
Rayski, J.|Evolution of physical ideas towards unification|1995;
Rebbi, C., (Ed.)|LATTICE GAUGE THEORIES AND MONTE CARLO SIMULATIONS|1984;
Rebbi, C., (Ed.)|SOLITONS AND PARTICLES|1985;
Rebhan, E.|Theoretical physics. Vol. 2: Quantum mechanics, relativistic quantum mechanics, quantum field theory, elementary particle theory, thermodynamics and statistics|2005;
Reed, M.|METHODS OF MATHEMATICAL PHYSICS. VOL. 3: SCATTERING THEORY|1979;
Reed, M.|Methods of Modern Mathematical Physics. 2. Fourier Analysis, Selfadjointness|1975;
Rees, M.|Before the beginning: Our universe and others|1998;
Rees, M.|Black Holes, Gravitational Waves and Cosmology: An Introduction to Current Research|1974;
Rees, M.|Just six numbers: The deep forces that shape the universe|1999;
Rees, M.|New perspectives in astrophysical cosmology|2000;
Rees, M.|Our cosmic habitat|2003;
Rees, M.|Perspectives in astrophysical cosmology|1995;
Rees, Martin J.|Our final hour, a scientist's warning how terror, error, and environmental disaster threaten humankind's future in this century on earth and beyond|2004;
Reimer, A., (ed.)|New developments in quantum cosmology research|2005;
Reimer, A., (ed.)|Quantum cosmology research trends|2005;
Reineker, Peter|Theoretical physics. Vol. 4: Quantum mechanics 2|2008;
Reiser, M.|Theory and design of charged particle beams|1995;
Ren, Hai-Cang, (ed.)|T.D. Lee: Selected papers, 1985-1996|1998;
Renard, F.M.|BASICS OF ELECTRON POSITRON COLLISIONS|1981;
Renk, B.|Data acquisition in nuclear and particle physics. (In German)|1992;
Renton, P.|ELECTROWEAK INTERACTIONS: AN INTRODUCTION TO THE PHYSICS OF QUARKS AND LEPTONS|1990;
Reynaud, S.|Testing General Relativity with Atomic Clocks|2009;
Rho, Mannque|Chiral nuclear dynamics: From quarks to nuclei to compact stars|2008;
Rho, Mannque|MESONS IN NUCLEI. VOL. III|1979;
Rho, Mannque|MESONS IN NUCLEI. VOL. II|1979;
Rho, Mannque|MESONS IN NUCLEI. VOL. I|1979;
Rich, J.|Fundamentals of cosmology|2001;
Richard, J.-M.|Double charm hadrons revisited|2005;
Rickles, Dean, (ed.)|The structural foundations of quantum gravity|2006;
Ridley, B.K.|TIME, SPACE AND THINGS|1984;
Riffert, H.|Matter at high densities in astrophysics: Compact stars and the equation of state|1996;
Rindler, W., (Ed.)|GRAVITATION AND GEOMETRY: A VOLUME IN HONOR OF IVOR ROBINSON|1987;
Rindler, W.|Relativity: Special, general, and cosmological|2006;
Riordan, M.|The Shadows of creation: Dark matter and the structure of the universe|1991;
Riordan, Michael|The Hunting of the quark. A True story of modern physics|1987;
Ripka, G.|Quarks bound by chiral fields: The quark-structure of the vacuum and of light mesons and baryons|1997;
Rivasseau, V.|From perturbative to constructive renormalization|1991;
Rivers, R.J.|PATH INTEGRAL METHODS IN QUANTUM FIELD THEORY|1987;
Roberts, B.Lee, (ed.)|Lepton dipole moments|2010;
Roberts, R.G.|The Structure of the proton: Deep inelastic scattering|1990;
Robson, B.A.|The Theory of Polarization Phenomena|1974;
Rochester, G.D.|A Discussion on the Origin of the Cosmic Radiation, London, 20-21 February 1974|1974;
Rodrigues, Waldyr Alves|The many faces of Maxwell, Dirac and Einstein equations: A Clifford bundle approach|2007;
Roe, B.P.|Particle physics at the new millennium|1996;
Roepstorff, G.|Path integral approach to quantum physics: An Introduction|1994;
Roepstorff, G.|Path integrals in quantum physics. (In German)|1991;
Rogers, Alice|Supermanifolds: Theory and applications|2007;
Rogers, C.|BACKLUND TRANSFORMATIONS AND THEIR APPLICATIONS|1982;
Rohlf, J.W.|Modern physics from alpha to Z0|1994;
Rohrlich, F.|FROM PARADOX TO REALITY. OUR NEW CONCEPTS OF THE PHYSICAL WORLD|1987;
Roig, Fernando, (ed.)|Graduate school in astronomy, XI Special Courses at the National Observatory of Rio de Janeiro (XI CCE) Rio de Janeiro, Brazil, 16-20 October 2006|2007;
Rollnik, H., (ed.)|IDEAS AND EXPERIMENTS FOR A UNIFIED THEORY OF MATTER. (IN GERMAN)|1979;
Rolnick, W.B.|Remnants of the fall: Revelations of particle secrets|2003;
Rolnick, W.B.|The Fundamental particles and their interactions|1994;
Rompe, R.|FUNDAMENTAL CONSTANTS AND THEIR MEANING. (IN GERMAN)|1988;
Roos, M.|Introduction to cosmology|1994;
Rosales, J.J.|Supersymmetric Cosmology and Dark Energy|2008;
Rosen, J., (Ed.)|SYMMETRY IN PHYSICS. SELECTED REPRINTS|1982;
Rosen, J.|A SYMMETRY PRIMER FOR SCIENTISTS|1984;
Rosen, Joe|Symmetry rules: How science and nature are founded on symmetry|2008;
Rosen, Steven M.|The self-evolving cosmos: A phenomenological approach to Nature's unity-in-diversity|2008;
Rosenthal-Schneider, I.|DISCUSSIONS WITH EINSTEIN, VON LAUE AND PLANCK: REALITY AND SCIENTIFIC TRUTH. (IN GERMAN)|1988;
Rosenzweig, J.B.|Fundamentals of beam physics|2003;
Rosinger, Elemer E.|How to solve smooth nonlinear PDEs in algebras of generalized functions with dense singularities|2004;
Rosner, Jonathan L., (ed.)|NEW PARTICLES. SELECTED REPRINTS|1981;
Ross, Graham G.|GRAND UNIFIED THEORIES|1985;
Ross, L.V., (ed.)|Focus on astrophysics research|2003;
Ross, S., (ed.)|Science|1990;
Rossi, L.|Pixel detectors: From fundamentals to applications|2006;
Rosswog, Stephan|Introduction to high-energy astrophysics|2007;
Roth, Stefan|Precision electroweak physics at electron positron colliders|2007;
Rothe, H.J.|Lattice gauge theories: An Introduction|1992;
Rothe, H.J.|Lattice gauge theories: An Introduction|1997;
Rothe, H.J.|Lattice gauge theories: An Introduction|2005;
Rothlein, B.|The quantum revolution: News from particle physics|2004;
Rovelli, Carlo|Quantum gravity|2004;
Rowan-Robinson, M.|Cosmology|1996;
Rowan-Robinson, M.|The nine numbers of the cosmos|1999;
Rowe, E.G.P.|Geometric physics in Minkowski space-time|2001;
Rowlands, Peter|Zero to infinity: The foundations of physics|2008;
Roy, P.|Theory of Lepton-Hadron Processes at High-Energies. Partons, Scale Invariance and Light Cone Physics|1975;
Rozental, I.L.|BIG BANG, BIG BOUNCE. HOW PARTICLES AND FIELDS DRIVE COSMIC EVOLUTION|1988;
Rubakov, V.A.|Classical theory of gauge fields|2002;
Rubinstein, R., (ed.)|Pan American collaboration in experimental physics|1989;
Rubinstein, R.|Fermilab Research Program 1983: Workbook|1983;
Rubinstein, R.|Fermilab Research Program 1984: Workbook|1984;
Rubinstein, R.|Fermilab Research Program 1987: Workbook|1987;
Ruffini, R., (ed.)|Matter particled: Patterns, structure and dynamics. Selected research papers of Yuval Ne'eman|2006;
Ruggiero, F., (ed.)|CERN Accelerator School, basic course on general accelerator physics (selected contributions), Loutraki, Greece, 2-13 October 2000|2005;
Ruggiero, F., (ed.)|HHH-2004, first CARE-HHH-APD Workshop on Beam Dynamics in Future Hadron Colliders and Rapidly Cycling High-Intensity Synchrotrons, CERN, Geneva, 8-10 November 2004|2005;
Ruggiero, F., (ed.)|Second CARE-HHH-APD Workshop on scenarios for the LHC luminosity upgrade, LHC-LUMI-05, Arcidosso Italy, 31 August-3 September 2005|2006;
Rundle, Bede|Time, space, and metaphysics|2009;
Russ, J.S.|BNL Workshop on Physics with Polarized Targets, Brookhaven, June 3-8, 1974|1975;
Russenschuck, Stephan|Field computation for accelerator magnets: Analytical and numerical methods for electromagnetic design and optimization|2010;
Ryan, M.P.|Homogeneous Relativistic Cosmologies|1975;
Ryden, B.|Introduction to cosmology|2003;
Ryder, L.H., (Ed.)|ELEMENTARY PARTICLES AND SYMMETRIES|1986;
Ryder, L.H.|QUANTUM FIELD THEORY|1985;
Ryder, Lewis|Introduction to general relativity|2009;
Sacchetti, N.|5th International Conference on Magnet Technology, Rome, April 21-25, 1975|1975;
Sachs, M.|GENERAL RELATIVITY AND MATTER. A SPINOR FIELD THEORY FROM FERMIS TO LIGHT YEARS|1982;
Sachs, M.|QUANTUM MECHANICS FROM GENERAL RELATIVITY. AN APPROXIMATION FOR A THEORY OF INERTIA|1986;
Sachs, M.|Quantum mechanics and gravity|2004;
Sachs, Mendel|Concepts of modern physics: The Haifa lectures|2007;
Sachs, R.G.|THE PHYSICS OF TIME REVERSAL|1987;
Sakita, B.|Quantum theory of many variable systems and fields|1986;
Sakurai, Jun John|Modern quantum physics|2011;
Salam, A., (Ed.)|SUPERGRAVITIES IN DIVERSE DIMENSIONS. VOL. 1, 2|1989;
Salam, Abdus|Complex Analysis and Its Applications. 3. Lectures, International Seminar Course, Trieste, 21 May 8 August 1975|1976;
Salam, Abdus|The 6th Trieste Conference on Particle Physics, 26-30 Jun 1978. 1.|1978;
Salam, Abdus|The 6th Trieste Conference on Particle Physics, 26-30 Jun 1978. 2.|1978;
Salam, S.|Unification of fundamental forces: The First of the 1988 Dirac memorial lectures|1990;
Saldin, E.L.|The physics of free electron lasers|2000;
Saleem, M.|Special relativity: Applications to particle physics and the classical theory of fields|1992;
Saller, H.|UNIFIED FIELD THEORIES OF ELEMENTARY PARTICLES. AN INTRODUCTION. (IN GERMAN)|1985;
Saller, Heinrich|Operational quantum theory. Vol. 1: Nonrelativistic structures|2006;
Saller, Heinrich|Operational quantum theory. Vol. 2: Relativistic structure|2006;
Salmhofer, M.|Renormalization: An introduction|1999;
Saltzer, W.G., (ed.)|On the unity of science in past and present. (In German)|1990;
Santilli, R.M.|Elements of hadronic mechanics. Vol. I: Mathematical foundations|1994;
Santilli, R.M.|Elements of hadronic mechanics: Vol. 2: Theoretical foundations|1995;
Santilli, R.M.|Isodual theory of antimatter: With applications to antigravity, grand unification and cosmology|2006;
Santilli, R.M.|LIE ADMISSIBLE APPROACH TO THE HADRONIC STRUCTURE. VOL. 2: COVERING OF THE GALILEI AND EINSTEIN RELATIVITIES?|1982;
Santilli, R.M.|Lie Admissible Approach to the Hadronic Structure. 1. Nonapplicability of the Galilei and Einstein Relativities?|1978;
Santra, A.B., (ed.)|Physics and astrophysics of hadrons and hadronic matter|2008;
Sardanashvily, G.A., (ed.)|New frontiers in gravitation|1996;
Sardanashvily, G.A.|Gauge gravitation theory|1992;
Sardanashvily, G.|Generalized Hamiltonian formalism for field theory: Constraints system|1995;
Sarkar, Utpal, (ed.)|Flavors of research in physics|2010;
Sarlemijn, A., (Ed.)|PHYSICS IN THE MAKING: ESSAYS ON DEVELOPMENTS IN 20TH CENTURY PHYSICS. IN HONOR OF H.B.G. CASIMIR ON THE OCCASION OF HIS 80TH BIRTHDAY|1989;
Satchler, G.R.|DIRECT NUCLEAR REACTIONS|1987;
Sato, H., (ed.)|Search for high-energy gamma-rays from SN1987A|1990;
Sattinger, D.H.|BRANCHING IN THE PRESENCE OF SYMMETRY|1986;
Sauermann, P.F.|Radiation Protection by Shielding. Tables for the Calculation of gamma Radiation Shielding|1976;
Sauli, F., (ed.)|Instrumentation in high-energy physics|1992;
Saulson, Peter R.|Fundamentals of interferometric gravitational wave detectors|1995;
Saunders, S.W., (ed.)|The Philosophy of vacuum|1991;
Savitt, S.F., (ed.)|Time's arrows today: Recent physical and philosophical work on the direction of time|1996;
Scadron, M.D.|ADVANCED QUANTUM THEORY AND ITS APPLICATION THROUGH FEYNMAN DIAGRAMS|1979;
Scadron, M.D.|Advanced quantum theory and its applications through Feynman diagrams|1991;
Scadron, M.D.|Advanced quantum theory|2007;
Scandale, W., (ed.)|CARE HHH APD Workshop on Finalizing the Roadmap for the Upgrade of the CERN and GSI Accelerator Complex (Beam'07), CERN, Geneva, Switzerland, 1-5 October 2007|2008;
Scandale, W., (ed.)|CARE HHH APD Workshop on Interaction Regions for the LHC Upgrade, DAFNE and SuperB (IR'07) Frascati, Italy, 6-9 November 2007|2008;
Scandale, W., (ed.)|Final CARE-HHH Workshop on Scenarios for the LHC Upgrade and FAIR HHH-2008 Chavannes-de-Bogis, Switzerland, 24-25 November 2008|2009;
Scandale, W., (ed.)|Third CARE HHH APD Workshop towards a roadmap for the upgrade of the LHC and GSI accelerator complex, LHC LUMI 06, Valencia, Spain, 16-20 October 2006|2007;
Scarpine, V.E.|Measurements of a newly designed BPM for the Tevatron Electron Lens 2|2006;
Schaechter, L.|Beam-wave interaction in periodic and quasi-periodic structures|1997;
Schakel, Adriaan M.|Boulevard of broken symmetries: Effective field theories of condensed matter|2008;
Scharf, G.|Finite quantum electrodynamics: The Causal approach|1996;
Scharf, G.|Finite quantum electrodynamics|1989;
Scharf, G.|Quantum gauge theories: A true ghost story|2001;
Scharf, J.H., (Ed.)|IN MEMORIAM WERNER HEISENBERG ON THE OCCASION OF HIS 80TH ANNIVERSARY. (IN GERMAN)|1982;
Scharf, J.H.|Basic Questions in Quantum and Relativity Theory. (In German) Symposium of Deutsche Akademie Der Naturforscher Leopoldina Zu Zu Halle, Eisenach, April 15-20, 1972|1974;
Scharf, W.H.|Particle accelerators: Applications in technology and research|1989;
Scharf, W.|PARTICLE ACCELERATORS AND THEIR USES. PT. 1: ACCELERATOR DESIGN|1986;
Scharf, W.|PARTICLE ACCELERATORS AND THEIR USES. PT. 2: APPLICATION OF ACCELERATORS|1986;
Scheck, F.|Electroweak and strong interactions: An introduction to theoretical particle physics|1996;
Scheck, F.|LEPTONS, HADRONS AND NUCLEI|1984;
Scheck, F.|Quantum physics|2007;
Scheck, F.|Theoretical physics. Vol. 4: Quantized fields. From symmetries to quantum electrodynamics|2001;
Scheck, Florian|Theoretical physics. Vol. 3: Classical field theory. From electrodynamics to gauge theories|2006;
Schellekens, A.N.|PERTURBATIVE QCD AND LEPTON PAIR PRODUCTION|1981;
Schellekens, B., (ed.)|Superstring construction|1989;
Scheunert, M.|THE THEORY OF LIE SUPERALGEBRAS. AN INTRODUCTION|1979;
Schewe, P.|PHYSICS NEWS IN 1979|1979;
Schlichenmaier, Martin|An introduction to Riemann surfaces, algebraic curves and moduli spaces|2007;
Schlickeiser, R.|Cosmic ray astrophysics|2002;
Schmid, R., (Ed.)|NOBEL PRICE WINNERS: WORKS, DATES, MEETINGS. NOBEL PRICE WINNERS IN LINDAU 1951 - 1980. FESTSCHRIFT ON THE OCCASION OF THE 60TH BIRTHDAY OF HANS ROTTA|1987;
Schmidt, H.U.|MEASURING ELECTRONICS IN NUCLEAR PHYSICS. (IN GERMAN)|1986;
Schmitt, Andreas|Dense matter in compact stars: A pedagogical introduction|2010;
Schmitz, N.|Neutrino physics. (In German)|1997;
Schmueser, Peter|FEYNMAN GRAPHS AND GAUGE THEORIES FOR EXPERIMENTAL PHYSICISTS. (IN GERMAN)|1988;
Schmuser, Peter|Ultraviolet and soft x-ray free-electron lasers: Introduction to physical principes, experimental results, technological challenges|2008;
Schmutzer, E.|Projective unified field theory with applications in cosmology and astrophysics: New picture of the world without big bang?|2004;
Schneider, C., (ed.)|RESEARCH IN THE FEDERAL REPUBLIC OF GERMANY. EXAMPLES, CRITICISM, SUGGESTIONS. (IN GERMAN)|1983;
Schober, A., (Ed.)|IRREVERSIBILITY AND NONPOTENTIALITY IN STATISTICAL MECHANICS. A REPRINT COLLECTION|1985;
Schommers, W.|The visible and the invisible: Matter and mind in physics|1998;
Schonfelder, V., (ed.)|The Universe in gamma-rays|2001;
Schopper, H., (ed.)|Advances of accelerator physics and technologies|1993;
Schopper, H., (ed.)|Numerical data and functional relationships in science and technology: Group. 1: Nuclear and particle physics. Vol. 14: Electron - positron interactions|1992;
Schopper, H.|MATTER AND ANTIMATTER: PARTICLE ACCELERATORS AND THE ADVANCE TO THE INFINITELY SMALL. (IN GERMAN)|1989;
Schopper, Herwig|LEP: The lord of the collider rings at CERN 1980-2000: The making, operation and legacy of the world's largest scientific instrument|2009;
Schottenloher, M.|A mathematical introduction to conformal field theory: Based on a series of lectures given at the Mathematisches Institut der Universitaet Hamburg|1997;
Schottenloher, M.|Geometry and symmetry in physics: Leading principle of mathematical physics. (In German)|1995;
Schottenloher, Martin, (ed.)|A mathematical introduction to conformal field theory|2008;
Schramm, D.N.|The big bang and other explosions in nuclear and particle astrophysics|1996;
Schrimpf, Ron D.|Radiation effects and soft error in integrated circuits and electronic devices|2004;
Schroder, U.E.|Gravitation: Introduction to general relativity theory|2002;
Schrodinger, E.|Expanding universes|2010;
Schroeder, L.S.|2nd High-Energy Heavy Ion Summer Study, July 15-26, 1974, at the Lawrence Berkeley Laboratory|1974;
Schubert, M.|Quantum theory: Basics and applications. (In German)|1994;
Schucker, Thomas|Noncommutative geometry and the standard model|2003;
Schulman, L.S.|Time's arrows and quantum measurement|1997;
Schulman, L.s.|TECHNIQUES AND APPLICATIONS OF PATH INTEGRATION|1981;
Schumann, R.|Quantum gravity|2006;
Schumm, B.A.|Deep down things: The breathtaking beauty of particle physics|2004;
Schuricht, V.|Radiation Protection Physics|1975;
Schutz, Bernard F.|A FIRST COURSE IN GENERAL RELATIVITY|1985;
Schutz, Bernard F.|Gravity from the ground up|2003;
Schuur, P.C.|Asymptotic analysis of soliton problems: An Inverse scattering approach|1986;
Schwab, A.J.|FIELD THEORY CONCEPTS: ELECTROMAGNETIC FIELDS, MAXWELL'S EQUATIONS, GRAD, CURL, DIV, ETC., FINITE ELEMENT METHOD, FINITE DIFFERENCE METHOD, CHARGE SIMULATION METHOD, MONTE CARLO METHOD|1988;
Schwabl, F.|Advanced quantum mechanics (QM II)|1997;
Schwarz, Albert S.|Quantum field theory and topology|1994;
Schwarz, Albert S.|Topology for physicists|1994;
Schwarz, C.|A Tour of the subatomic zoo: A Guide to particle physics|1992;
Schwarz, J.H., (Ed.)|SUPERSTRINGS. THE FIRST 15-YEARS OF SUPERSTRING THEORY. VOL. 1|1986;
Schwarz, J.H., (Ed.)|SUPERSTRINGS. THE FIRST 15-YEARS OF SUPERSTRING THEORY. VOL. 2|1986;
Schwarz, P.M.|Special relativity: From Einstein to strings|2004;
Schwarze, H.|Teaching materials on the TESLA project: A 33-km long electron positron linear collider with integrated X-ray lasers. Pt. 1: X-ray laser microscope. Pt. 2: Elementary particle physics|2000;
Schweber, S.S.|QED and the men who made it: Dyson, Feynman, Schwinger, and Tomonaga|1994;
Schwinger, Julian S.|PARTICLES, SOURCES AND FIELDS. VOLUME II|1973;
Schwinger, Julian S.|PARTICLES, SOURCES, AND FIELDS. VOL. 3|1989;
Schwinger, Julian S.|Particles, sources, and fields. Vol. 2|1989;
Sciama, D.W.|Modern cosmology and the dark matter problem|1994;
Sciama, D.W.|THE PHYSICAL BASES OF GENERAL RELATIVITY. (IN FRENCH)|1971;
Segre, E.|From x-Rays to Quarks. Modern Physicists and their Discoveries|1980;
Seife, C.|The search for the beginning and the end of the universe|2003;
Seiler, E.|Gauge Theories as a Problem of Constructive Quantum Field Theory and Statistical Mechanics|1982;
Selesnick, S.A.|Quanta, logic and spacetime: Variations on Finkelstein's quantum relativity|1998;
Selleri, F., (Ed.)|QUANTUM MECHANICS VERSUS LOCAL REALISM. THE EINSTEIN-PODOLSKY-ROSEN PARADOX|1988;
Septier, A., (Ed.)|APPLIED CHARGED PARTICLE OPTICS. PART C: VERY HIGH DENSITY BEAMS|1986;
Serber, R.|Serber says: About nuclear physics|1987;
Serot, Brian D.|Covariant effective field theory for nuclear structure and nuclear currents|2004;
Sessler, Andrew|Engines of discovery: A century of particle accelerators|2007;
Setare, M R|Quasinormal modes, reduced phase space and area spectrum of black holes|2005;
Sexl, R.U.|Relativity Groups, Particles. Special Theory of Relativity as the Basis of Field and Elementary Particle Physics|1976;
Sexl, R.U.|Relativity, groups, particles: Special relativity and relativistic symmetry in field and particle physics|2001;
Sexl, R.U.|WHAT BINDS THE WORLD TOGETHER. PHYSICS LOOKING FOR THE PLAN OF NATURE. (IN GERMAN)|1982;
Sexl, R.u.|GRAVITATION AND COSMOLOGY. AN INTRODUCTION TO GENERAL RELATIVITY THEORY. (IN GERMAN, REVISED AND EXTENDED VERSION)|1983;
Sexl, R.|SPACE-TIME RELATIVITY. (IN GERMAN)|1990;
Shabad, A.E.|Polarization of the vacuum and a quantum relativistic gas in an external field|1992;
Shaikh, Dastgeer, (ed.)|Turbulence and nonlinear processes in astrophysical plasmas 6th Annual International Astrophysics Conference, Oahu, Hawaii, 16-22 March 2007|2007;
Shapere, Alfred D., (Ed.)|GEOMETRIC PHASES IN PHYSICS|1989;
Shapiro, S.L.|Black holes, white dwarfs, and neutron stars: The physics of compact objects|1983;
Sheeter, S.|THE UNIFIED MODEL OF THE UNIVERSE. THE GEOMETRICALLY UNIFIED FIELD SOLUTION|1981;
Shifman, M., (ed.)|At the frontier of particle physics. Handbook of QCD. Vol. 1-3|2001;
Shifman, M., (ed.)|From fields to strings: Circumnavigating theoretical physics. Ian Kogan memorial collection (3 volume set)|2005;
Shifman, Mikhail A., (ed.)|Instantons in gauge theories|1994;
Shifman, Mikhail A., (ed.)|The many faces of the superworld: Yuri Golfand memorial volume|2000;
Shifman, Mikhail A., (ed.)|Vacuum structure and QCD sum rules|1992;
Shifman, Mikhail A.|ITEP lectures on particle physics and field theory. Vol. 1, 2|1999;
Shifman, Mikhail|Supersymmetric solitons|2009;
Shiiki, Noriko|Black holes with skyrme hair|2005;
Shiltsev, Vladimir|Accelerator Science and Technology Breakthroughs, Achievements and Lessons from the Tevatron|2011;
Shinmura, S.|Coherent Lambda - Sigma0 mixing in high-density neutron matter|2002;
Shiozawa, T.|Classical relativistic electrodynamics: Theory of light emission and application to free electron lasers|2004;
Shirkov, Dmitry|Large regular QCD coupling at Low Energy?|2008;
Shnir, Ya.M.|Magnetic monopoles|2005;
Shuryak, Edward V.|The QCD vacuum, hadrons and the superdense matter|1988;
Si, Qimiao|Quantum Criticality and the Kondo Lattice|2010;
Sibgatullin, N.R.|Oscillations and waves in strong gravitational and electromagnetic fields|1991;
Sibold, K.|Theory of the elementary particles|2001;
Sick, I.|Elastic electron scattering from light nuclei|2001;
Sidharth, B G|The Interface between classical electrodynamics and quantum theory|2003;
Sidharth, B.G., (ed.)|A century of ideas: Perspectives from leading scientists of the 20th century|2008;
Sidharth, B.G.|The thermodynamic universe: Exploring the limits of physics|2008;
Sidharth, B.G.|The universe of fluctuations: The architecture of spacetime and the universe|2005;
Sidharth, B.G.|When the universe took a u-turn|2010;
Sidharth, Burra Gautam, (ed.)|Frontiers of fundamental physics, eighth international symposium FFP8, Madrid, Spain, 17-19 October 2006|2007;
Sidhu, D.P.|Prospects for Strong Interaction Physics at Isabelle|1977;
Siegel, W.|Introduction to string field theory|1988;
Siegel, Warren|Introduction to string field theory|1988;
Siegfried, T.|Strange matters: Undiscovered ideas at the frontiers of space and time|2002;
Siemens, P.J.|ELEMENTS OF NUCLEI. MANY BODY PHYSICS WITH THE STRONG INTERACTION|1987;
Signore, Monique, (Ed.)|Topological defects in cosmology|1998;
Silk, J.|A short history of the universe|1997;
Silk, J.|The infinite cosmos: Questions from the frontiers of cosmology|2006;
Simeone, C.|Deparametrization and path integral quantization of cosmological models|2001;
Simms, D.J.|Lectures on Geometric Quantization. Lecture Notes in Physics, Vol. 53|1977;
Simon, B.|The p (phi) in Two-Dimensions Euclidean (Quantum) Field Theory|1974;
Singh, S.|Big bang: The most important scientific discovery of all time and why you need to know about it|2004;
Singh, V.|Strings, Lattice Gauge Theory and High Energy Phenomenology|0000;
Sitar, B.|Ionization measurements in high-energy physics|1993;
Sitenko, A.G.|Scattering theory|1991;
Sitenko, A.G.|Theory of nuclear reactions|1990;
Sitenko, A.|Theory of nucleus: Nuclear structure and nuclear interaction|1997;
Sivia, D.S.|Elementary scattering theory: For x-ray and neutron users|2011;
Slobodrian, R.J.|Few Body Problems in Nuclear and Particle Physics. International Conference Held at Laval University, Quebec City, Canada, August 27-31, 1974|1975;
Smaluk, Victor|Particle beam diagnostics for accelerators: Instruments and methods|2009;
Smejkal, J.|Anomalous Lagrangians and the radiative muon capture in hydrogen|2005;
Smilga, Andrei V.|Lectures on quantum chromodynamics|2001;
Smirnov, F.A.|Form-factors in completely integrable models of quantum field theory|1992;
Smirnov, V.A.|Feynman integral calculus|2006;
Smirnov, V.A.|Renormalization and asymptotic expansions|1991;
Smirnov, Vladimir A.|Applied asymptotic expansions in momenta and masses|2002;
Smirnov, Vladimir A.|Evaluating Feynman integrals|2004;
Smit, J.|Introduction to quantum fields on a lattice: A robust mate|2002;
Smith, A., (ed.)|Selected papers on photon-counting detectors|1998;
Smith, Randall K., (ed.)|X-ray diagnostics of astrophysical plasmas : Theory, experiment, and observation, Cambridge, Massachusetts, 15-17 November 2004|2005;
Smith, T.P.|Hunting for quarks in ordinary matter|2003;
Smith, Wesley H., (ed.)|Deep inelastic scattering, 13th International Workshop on Deep Inelastic Scattering, DIS 2005, Madison, Wisconsin, 27 April-1 May 2005|2005;
Smolin, L.|The lifetime of the cosmos|1997;
Smolin, L.|The trouble with physics: The rise of string theory, the fall of a science, and what comes next|2006;
Smolin, L.|Three roads to quantum gravity|2000;
Smolin, Lee|Generic predictions of quantum theories of gravity|2006;
Smolin, Lee|Scientific alternatives to the anthropic principle|2004;
Smoot, George F.|Wrinkles in time. (In German)|1996;
Sniatycki, J.|Geometric quantization and quantum mechanics|1995;
Snygg, J.|Clifford algebra: A computational tool for physicists|1997;
Sobenin, N.P.|Electrodynamic characteristics of accelerating cavities|1999;
Sokolov, A.A.|RADIATION FROM RELATIVISTIC ELECTRONS|1986;
Sokolsky, P.|INTRODUCTION TO ULTRAHIGH-ENERGY COSMIC RAY PHYSICS|1989;
Solomon, A., (ed.)|Theories of matter: A Festschrift for Prof. Joseph L. Birman|1995;
Solomos, Nikolaos H., (ed.)|Recent advances in astronomy and astrophysics, 7th International Conference of the Hellenic Astronomical Society, Lixourion, Kefallinia Island, Greece, 8-11 September 2005|2006;
Song, Xing-Chang, (Ed.)|INTEGRABLE SYSTEMS|1990;
Soper, D.E.|Classical Field Theory|1976;
Sorkin, Rafael D.|Does locality fail at intermediate length-scales|2007;
Sorkin, Rafael D.|Relativity theory does not imply that the future already exists: A Counterexample|2007;
Sozzi, Marco S.|Discrete symmetries and CP violation: From experiment to theory|2008;
Speiser, D.|PARTICLE INTERACTIONS AT VERY HIGH-ENERGIES. PART A. SUMMER INSTITUTE, LOUVAIN 1973|1973;
Speiser, D.|PARTICLE INTERACTIONS AT VERY HIGH-ENERGIES. PART B. SUMMER INSTITUTE, LOUVAIN 1973|1973;
Spieler, H.|Semiconductor detector systems|2005;
Spiering, C.|SEARCHING FOR THE FUNDAMENTAL FORCE. (IN GERMAN)|1986;
Spiering, C.|Searching for the original force. (In German)|1989;
Spinks, J.W.T.|An Introduction to Radiation Chemistry. Second Edition|1976;
Spiridonov, O.P.|UNIVERSAL PHYSICAL CONSTANTS|1986;
Spohn, H.|Dynamics of charged particles and their radiation field|2004;
Squires, E.J.|TO ACKNOWLEDGE THE WONDER. THE STORY OF FUNDAMENTAL PHYSICS|1986;
Srednicki, M.A., (ed.)|Particle physics and cosmology: Dark matter|1990;
Srednicki, M.|Quantum field theory|2007;
Srivastava, P.P.|SUPERSYMMETRY, SUPERFIELDS AND SUPERGRAVITY: AN INTRODUCTION|1986;
Srivastava, S.K.|Aspects of gravitational interactions|1998;
Stahl, A.|Physics with tau leptons|2000;
Staley, K.W.|The evidence for the top quark: Objectivity and bias in collaborative experimentation|2004;
Stamatescu, Ion-Olimpiu, (ed.)|Approaches to fundamental physics: An assessment of current theoretical ideas|2007;
Stancu, F.|Group theory in subnuclear physics|1991;
Stanev, T.|High energy cosmic rays|2004;
Starkl, R.|Matter - field - structure: Revision course on theoretical physics|1998;
Steffen, Frank Daniel|Supersymmetric candidates for dark matter|2007;
Stein, E.|THE OSCILLATION PROPERTIES OF THE SPACE-TIME CONTINUUM. (IN GERMAN)|1988;
Steinberger, J.|Learning about particles: 50 privileged years|2005;
Steinmann, O.|Perturbative quantum electrodynamics and axiomatic field theory|2000;
Stenger, V.J.|Not by design: The origin of the universe|1988;
Stenger, V.J.|Physics and psychics: The search for a world beyond the senses|1990;
Stepanow, Semjon|Relativistic quantum theory: For bachelor: With introduction to the quantum theory of multiparticle systems|2010;
Stephani, Hans|Exact solutions of Einstein's field equations|2003;
Stephani, Hans|GENERAL RELATIVITY. AN INTRODUCTION TO THE THEORY OF THE GRAVITATIONAL FIELD|1982;
Stephani, Hans|Relativity: An introduction to special and general relativity|2004;
Sterman, George F.|An Introduction to quantum field theory|1994;
Sternberg, S.|Group theory and physics|1994;
Stevens, C.F.|The Six core theories of modern physics|1995;
Stewart, I.|Fearful symmetry: Is God a geometer? (In German)|1994;
Stiefel, E.|GROUP THEORETICAL METHODS AND THEIR APPLICATION. (IN GERMAN)|1979;
Stierstadt, K.|PHYSICS OF MATTER. (IN GERMAN)|1989;
Stock, Reinhard, (ed.)|Encyclopedia of applied high energy and particle physics|2009;
Stockler, M.|PHILOSOPHICAL PROBLEMS IN RELATIVISTIC QUANTUM MECHANICS. (IN GERMAN)|1984;
Stockli, Martin P., (ed.)|Production and neutralization of negative ions and beams, 11th international symposium on the Production and Neutralization of Negative Ions and Beams Santa Fe, New Mexico, 13-15 September 2006|2007;
Stoica, S.|Critical view on double-beta decay matrix elements within quasi random phase approximation-based methods|2001;
Stone, M., (ed.)|Bosonization|1995;
Stone, M., (ed.)|Quantum Hall effect|1992;
Stone, M.|The physics of quantum fields|2000;
Stone, S., (ed.)|$B$ decays|1992;
Stork, H., (Ed.)|SYMMETRY. (IN GERMAN)|1985;
Storrie-Lombardi, Lisa J., (ed.)|The Science Opportunities for the Warm Spitzer Mission Workshop, Pasadena, California, 4-5 June 2007|2007;
Straumann, N.|GENERAL RELATIVITY AND RELATIVISTIC ASTROPHYSICS|1984;
Straumann, N.|GENERAL RELATIVITY THEORY AND RELATIVISTIC ASTROPHYSICS. (IN GERMAN)|1981;
Straumann, N.|Relativistic quantum theory: An introduction to quantum field theory|2005;
Streater, R.F.|PCT, spin and statistics, and all that|1989;
Streit, L., (Ed.)|MATHEMATICS + PHYSICS. LECTURES ON RECENT RESULTS. VOL. 1|1985;
Streit, L., (Ed.)|MATHEMATICS + PHYSICS. LECTURES ON RECENT RESULTS. VOL. 2|1986;
Strocchi, F.|ELEMENTS OF QUANTUM MECHANICS OF INFINITE SYSTEMS|1985;
Strocchi, F.|Selected topics on the general properties of quantum field theory|1993;
Strocchi, F.|Symmetry breaking|2005;
Stumpf, H.|Composite particle dynamics in quantum field theory|1994;
Sube, R.|DICTIONARY OF HIGH-ENERGY PHYSICS. ENGLISH, GERMAN, FRENCH, RUSSIAN|1987;
Sudarshan, E.C.G.|A gift of prophecy: Essays in celebration of the life of Robert Eugene Marshak|1994;
Sudbery, Anthony|QUANTUM MECHANICS AND THE PARTICLES OF NATURE. AN OUTLINE FOR MATHEMATICIANS|1986;
Suhendro, Indranu|Spin-curvature and the unification of fields in a twisted space|2008;
Suhonen, Jouni|From nucleons to nucleus, concepts of microscopic nuclear theory|2007;
Sullivan, A.H.|A Guide to radiation and radioactivity levels near high-energy particle accelerators|1992;
Sundaresan, M.K.|Handbook of particle physics|2001;
Sundermeyer, K.|CONSTRAINED DYNAMICS WITH APPLICATIONS TO YANG-MILLS THEORY, GENERAL RELATIVITY, CLASSICAL SPIN, DUAL STRING MODEL|1982;
Sunyaev, R.A., (ed.)|Zeldovich: Reminiscences|2004;
Surrey, Elizabeth, (ed.)|Negative ions beams and sources, 1st International Symposium on Negative Ions, Beams and Sources, Aix-en-Provence, France 9-12 September 2008|2009;
Susskind, L.|An introduction to black holes, information and the string theory revolution: The holographic universe|2005;
Susskind, L.|The Cosmic Landspace: String theory and the illusion of intelligent design|2006;
Susskind, L.|The cosmic landscape: String theory and the illusion of intelligent design|2005;
Sutton, C., (Ed.)|BUILDING THE UNIVERSE|1985;
Sutton, C.|Spaceship neutrino|1992;
Sutton, C.|THE PARTICLE CONNECTION. THE MOST EXCITING SCIENTIFIC CHASE SINCE DNA AND THE DOUBLE HELIX|1985;
Suzuki, Y.|Stochastic variational approach to quantum-mechanical few body problems|1998;
Suzuki, Y.|Structure and reactions of light exotic nuclei|2003;
Svozil, Karl|The Church-Turing thesis as a guiding principle for physics|1997;
Swanson, M.S.|Path integrals and quantum processes|1992;
Swanson, W.P.|RADIOLOGICAL SAFETY ASPECTS OF THE OPERATION OF ELECTRON LINEAR ACCELERATORS|1979;
Synge, J.L., (ed.)|Relativity: The General theory|1960;
Syrovoi, Valery A.|Theory of intense beams of charged particles|2011;
Szabo, Richard J.|An Introduction to String Theory and D-Brane Dynamics|2004;
Szabo, Richard J.|Equivariant Cohomology and Localization of Path Integrals|2000;
Szabo, Richard J.|Perturbation theory and techniques|2005;
Szabo, Richard J.|Strings, gauge fields and membranes|2004;
Szczerba, Ryszard, (ed.)|Planetary neubulae as astronomical tools, International Conference on Planetary Nebulae as Astronomical Tools, Gdansk, Poland, 28 June - 2 July 2005|2005;
Szekeres, Peter|A Course in Modern Mathematical Physics: Groups, Hilbert Space and differential geometry|2004;
Tait, W.H.|RADIATION DETECTION|1980;
Talman, R.|Accelerator X-ray sources|2006;
Tamanoi, H.|Elliptic genera and vertex operator super-algebras|1999;
Tara Prasad Das|RELATIVISTIC QUANTUM MECHANICS OF ELECTRONS|1973;
Tarasov, L.|Symmetry, symmetry: Structural principles in nature and technology. (In German)|1994;
Tassie, L.J.|The Physics of Elementary Particles|1977;
Taube, M.|EVOLUTION OF MATTER AND ENERGY ON A COSMIC AND PLANETARY SCALE|1985;
Tavernier, Stefaan|Experimental techniques in nuclear and particle physics|2010;
Tayler, R.J.|The hidden universe|1995;
Taylor, J.C., (ed.)|Gauge theories in the twentieth century|2001;
Taylor, J.C.|Gauge Theories of Weak Interactions|1976;
Taylor, J.C.|Hidden unity in nature's laws|2001;
Taylor, J.G., (Ed.)|TRIBUTES TO PAUL DIRAC|1987;
Taylor, J.G., (ed.)|THE NEW PHYSICS|1982;
Taylor, J.G.|Finite superstrings|1992;
Taylor, R.J.|The Hidden universe|1991;
Tegmark, Max|Many Worlds in Context|2009;
Teitelboim, C.(ed.)|The black hole: 25 years after|1998;
Teller, P.|An Interpretive introduction to quantum field theory|1995;
Tenchini, Roberto|The physics of the Z and W bosons|2008;
Terazawa, H., (Ed.)|PARTICLES AND NUCLEI. ESSAYS IN HONOR OF THE 60TH BIRTHDAY OF PROFESSOR YOSHITO YAMAGUCHI|1986;
Terning, J.|Modern supersymmetry: Dynamics and duality|2006;
Ternov, I.M.|SYNCHROTRON RADIATION AND ITS APPLICATIONS|1985;
Thaller, B.|The Dirac equation|1992;
Thiemann, Thomas|Modern canonical quantum general relativity|2001;
Thirring, Walter E.|A COURSE IN MATHEMATICAL PHYSICS. VOL. 2. CLASSICAL FIELD THEORY|1979;
Thirring, Walter E.|Selected papers of Walter E. Thirring with commentaries|1998;
Thomas, Anthony W., (ed.)|Modern Three Hadron Physics|1977;
Thomas, Anthony William|The Structure of the Nucleon|2001;
Thomas, E.|From Quarks to Quasars. An Outline of Modern Physics|1977;
Thomas, Jean-Marie, (Ed.)|Journee des Retraites, CERN, Geneve, Suisse, 15 octobre 2004|2005;
Thomas, Jennifer A.|Neutrino oscillations: Present status and future plans|2008;
Thomas, Ralph H.|Radiological Safety Aspects of the Operation of Proton Accelerators|1986;
Thompson, D.J.|EUROPEAN SYNCHROTRON RADIATION FACILITY. SUPPLEMENT II: THE MACHINE|1979;
Thompson, J.M.T., (ed.)|Advances in astronomy: From the big bang to the solar system|2005;
Thorne, Kip S., (Ed.)|BLACK HOLES: THE MEMBRANE PARADIGM|1986;
Thorne, Kip S.|Black holes and time warps: Einstein's outrageous legacy|1994;
Thouless, D.J.|Topological quantum numbers in nonrelativistic physics|1998;
Ticciati, R.|Quantum field theory for mathematicians|1999;
Tipler, F.J., (ed.)|Essays in General Relativity. A Festschrift for Abraham Taub|1980;
Tirapegui, E., (ed.)|FIELD THEORY, QUANTIZATION AND STATISTICAL PHYSICS. IN MEMORY OF BERNARD JOUVET|1981;
Tiwari, S.C.|Superluminal phenomena in modern perspective: Faster-than-light signals. Myth or reality?|2003;
Todorov, I.T.|CONFORMAL DESCRIPTION OF SPINNING PARTICLES|1986;
Todorov, I.T.|Conformal Invariance in Quantum Field Theory|1978;
Todorov, Ivan|Quantum groups and braid group statistics in conformal current algebra models|2010;
Tome, W.|Path integrals on group manifolds: The representation independent propagator for general Lie groups|1998;
Tomonaga, S.I.|The story of spin|1997;
Toomey, David|The new time travelers|2007;
Torres del Castillo, Gerardo F.|Spinors in four-dimensional spaces|2010;
Tourrenc, P.|Relativity and gravitation|1997;
Trautman, A.|DIFFERENTIAL GEOMETRY FOR PHYSICISTS|1985;
Trautman, Andrzej|Einstein-Cartan theory|2006;
Treder, H.J.|Philosophical Problems of the Physical Space. Gravitation, Geometry, Cosmology and Relativity. (In German)|1974;
Trefil, J.S.|FROM ATOMS TO QUARKS. AN INTRODUCTION TO THE STRANGE WORLD OF PARTICLE PHYSICS|1980;
Trefil, J.S.|THE MOMENT OF CREATION. BIG BANG PHYSICS FROM BEFORE THE FIRST MILLISECOND TO THE PRESENT UNIVERSE|1984;
Trefil, J.S.|THE MOMENT OF CREATION. BIG BANG PHYSICS FROM PLANCK TIME TILL TODAY. (IN GERMAN)|1984;
Treichel, M.|Particle physics and cosmology: An introduction into foundations and connections|2000;
Treiman, S.B.|CURRENT ALGEBRA AND ANOMALIES|1986;
Treiman, S.|The odd quantum|1999;
Trigg, G.L., (ed.)|Encyclopedia of applied physics. Vol. 11: Moessbauer effect to nuclear structure|1995;
Trigg, G.L.|Landmark Experiments in 20th Century Physics|1975;
Trimble, V.|Visit to a small universe|1992;
Tromba, A.J., (Ed.)|SEMINAR ON NEW RESULTS IN NONLINEAR PARTIAL DIFFERENTIAL EQUATIONS|1987;
Troshin, S.M.|Spin phenomena in particle interactions|1995;
Trower, W.P., (Ed.)|DISCOVERING ALVAREZ: SELECTED WORKS OF LUIS W. ALVAREZ, WITH COMMENTARY BY HIS STUDENTS AND COLLEAGUES|1987;
Trutnev, Yu.A., (ed.)|In the intermissions ...: Collected works on research into the essentials of theoretical physics in Russian Federal Nuclear Center, Arzamas-16|1998;
Tschernogorova, V.A.|The Mysteries of the Microworld|1976;
Tsipenyuk, Yu.M.|The microtron: Development and applications|2002;
Tsukada, H.|String path integral realization of vertex operator algebras|1991;
Tsvelik, A.M.|Quantum field theory in condensed matter physics|1996;
Tucker, W.H.|Radiation Processes in Astrophysics|1975;
Tucker, W.|THE DARK MATTER: CONTEMPORARY SCIENCE'S QUEST FOR THE MASS HIDDEN IN OUR UNIVERSE|1988;
Tung, W.K.|GROUP THEORY IN PHYSICS|1985;
Turaev, V.G.|Quantum invariants of knots and three manifolds|1994;
Turnbull, R.M.|THE STRUCTURE OF MATTER. AN INTRODUCTION TO ATOMIC NUCLEAR AND PARTICLE PHYSICS|1979;
Turner, R.E.|RELATIVITY PHYSICS|1985;
Tuynman, G.M.|MATHEMATICAL STRUCTURES IN FIELD THEORIES. VOL. 1: GEOMETRIC QUANTIZATION.|1985;
Tzenov, S.I.|Contemporary accelerator physics|2004;
Uesaka, M., (ed.)|Femtosecond beam science|2005;
Uglov, D.B.|Collected papers of Denis B. Uglov|2001;
Umezawa, H.|Advanced field theory: Micro, macro, and thermal physics|1993;
Umezawa, H.|THERMO FIELD DYNAMICS AND CONDENSED STATES|1982;
Urbanowski, K.|Subsystem of neutral mesons beyond the Lee-Oehme-Yang approximation|2004;
Uzan, Jean-Philippe|Dark energy, gravitation and the Copernican principle|2009;
Uzan, Jean-Philippe|The natural laws of the universe: Understanding fundamental constants|2008;
Uzunov, D.I.|Introduction to the theory of critical phenomena: Mean field, fluctuations and renormalization|1993;
Vaas, R.|Tunnel through time and space: Einsteins legacy - black holes, time travel and faster-than-light speed|2005;
Vacaru, S.|Clifford and Riemann-Finsler structures in geometric mechanics and gravity|2005;
Vacaru, Sergiu Ion|Interactions, strings and isotopies in higher order anisotropic superspaces|1998;
Vacaru, Sergiu|Spinors and space-time anisotropy|2001;
Vachaspati, Tanmay|Kinks and domain walls: An introduction to classical and quantum solitons|2006;
Val Blain, J., (ed.)|Progress in dark matter research|2005;
Val Blain, J., (ed.)|Trends in dark matter research|2005;
Valente, V.|ADONE: A milestone on the particle way|1997;
Valentin, L.|SUBATOMIC PHYSICS: NUCLEI AND PARTICLES. VOL. 1|1981;
Valentin, L.|SUBATOMIC PHYSICS: NUCLEI AND PARTICLES. VOL. 2|1981;
Van Der Merwe, A., (ed.)|OLD AND NEW QUESTIONS IN PHYSICS, COSMOLOGY, PHILOSOPHY, AND THEORETICAL BIOLOGY. ESSAYS IN HONOR OF WOLFGANG YOURGRAU|1983;
Van Haeringen, H.|CHARGED PARTICLE INTERACTIONS. THEORY AND FORMULAS|1985;
Vannucci, Francois|The true novel of elementary particles|2010;
Varadarajan, V.S.|Lie Groups, Lie Algebras and their Representations|1974;
Varadarajan, V.S.|Supersymmetry for mathematicians: An introduction|2004;
Vargas Moniz, Paulo|Quantum cosmology: The supersymmetric perspective. Vol. 2|2010;
Varshalovich, D.A.|QUANTUM THEORY OF ANGULAR MOMENTUM: IRREDUCIBLE TENSORS, SPHERICAL HARMONICS, VECTOR COUPLING COEFFICIENTS, 3NJ SYMBOLS|1988;
Vasilev, A.N.|The field theoretic renormalization group in critical behavior theory and stochastic dynamics|2004;
Vasiliev, A.N.|Functional methods in quantum field theory and statistical physics|1998;
Velan, A.K.|The Multiuniverse cosmos: The First complete story of the origin of the universe|1992;
Veltman, M.J.G.|Diagrammatica: The Path to Feynman rules|1994;
Veltman, M.J.G.|Facts and mysteries in elementary particle physics|2003;
Verde, Licia|Statistical methods in cosmology|2009;
Vilasi, G.|Hamiltonian dynamics|2001;
Vilenkin, A.|COSMIC STRINGS AND OTHER TOPOLOGICAL DEFECTS|1986;
Vilenkin, A.|Many worlds in one: The search for other universes|2006;
Visser, Matt|Lorentzian wormholes: From Einstein to Hawking|1995;
Vizgin, V.P.|Unified field theories in the first third of the 20th century|1994;
Vladimirov, V.S.|p-adic analysis and mathematical physics|1994;
Vladimirov, Yu.S.|SPACE, TIME, GRAVITATION|1987;
Vogel, H.|Gerthsen: Physics. (In German)|1997;
Vogt, Ramona|Ultrarelativistic heavy-ion collisions|2007;
Volkel, A.H.|Fields, Particles and Currents|1977;
Volovik, G.E.|The Universe in a helium droplet|2003;
Von Baeyer, H.C.|RAINBOWS, SNOWFLAKES, AND QUARKS. PHYSICS AND THE WORLD AROUND US|1986;
Von Borzeszkowski, H.H.|THE MEANING OF QUANTUM GRAVITY|1988;
Voss, Rudiger, (ed.)|The CERN Large Hadron Collider, accelerator and experiments|2009;
Wadia, Spenta R., (ed.)|The legacy of Albert Einstein: A collection of essays in celebration of the year of physics|2007;
Wagoner, R.V.|COSMIC HORIZONS. UNDERSTANDING THE UNIVERSE|1982;
Wald, Robert M., (Ed.)|Black holes and relativistic stars|1998;
Wald, Robert M.|General Relativity|1984;
Wald, Robert M.|Quantum field theory in curved space-time and black hole thermodynamics|1995;
Wald, Robert M.|Space, time, and gravity: The Theory of the big bang and black holes|1992;
Walecka, J.D.|Theoretical nuclear and subnuclear physics|1995;
Walecka, John D.|Introduction to modern physics: Theoretical foundations|2008;
Walecka, John Dirk|Advanced modern physics: Theoretical foundations|2010;
Walecka, John Dirk|Introduction to general relativity|2008;
Wali, K.C., (ed.)|A quest for perspectives: Selected works of S. Chandrasekhar. With commentary. Vol. 1, 2|2001;
Wali, K.C., (ed.)|S. Chandrasekhar: The man behind the legend|1997;
Wall, E.L.|The physics of tachyons|1995;
Wallace, P.R.|Physics: Imagination and reality|1991;
Waloschek, P., (ed.)|The Infancy of particle accelerators: Life and work of Rolf Wideroe|1994;
Waloschek, P., (ed.)|When the particles learned to walk. (In German)|1993;
Waloschek, P.|KEY TO PHYSICS. (IN GERMAN)|1989;
Waloschek, P.|NEW PARTICLE PHYSICS - SIMPLY PRESENTED. (IN GERMAN)|1989;
Waloschek, P.|Physics dictionary|1998;
Waloschek, P.|Visit to the particle zoo: From crystal to quark. (In German)|1996;
Waloschek, P.|Voyage to the interior of matter: With HERA to the frontiers of knowledge. (In German)|1991;
Wan, Zhe-Xian|Lie Algebras|1975;
Wanas, M.I.|Path deviation equations in AP-geometry|2006;
Wanas, M.I.|Quantum roots in geometry: I|2005;
Wang, Rong|An introduction to differential geometry and topology in mathematical physics|1998;
Wang, Y.|Dark Energy|2010;
Wangler, T.P.|Principles of RF linear accelerators|1998;
Wangler, Thomas P.|RF linear accelerators|2008;
Ward, B.F.L.|Black holes and massive elementary particles in resummed quantum gravity|2005;
Ward, J.P.|Quaternions and Cayley numbers: Algebra and applications|1997;
Ward, R.S.|Twistor geometry and field theory|1990;
Watkins, P.|STORY OF THE W AND Z|1986;
Watson, Andrew|The quantum quark|2004;
Wawrzycki, Jaroslaw|Noncommutative spacetime and quantum mechanics|2004;
Wawrzynczyk, A.|GROUP REPRESENTATIONS AND SPECIAL FUNCTIONS|1984;
Weart, S.R., (Ed.)|HISTORY OF PHYSICS|1986;
Weaver, J.H.|THE WORLD OF PHYSICS. A SMALL LIBRARY OF THE LITERATURE OF PHYSICS FROM ANTIQUITY TO THE PRESENT. VOL. 1: THE ARISTOTELIAN COSMOS AND THE NEWTONIAN SYSTEM|1987;
Weaver, J.H.|THE WORLD OF PHYSICS. A SMALL LIBRARY OF THE LITERATURE OF PHYSICS FROM ANTIQUITY TO THE PRESENT. VOL. 3: THE EVOLUTIONARY COSMOS AND THE LIMITS OF SCIENCE|1987;
Weaver, J.H.|THE WORLD OF PHYSICS: A SMALL LIBRARY OF THE LITERATURE OF PHYSICS FROM ANTIQUITY TO THE PRESENT. VOL. 2: THE EINSTEIN UNIVERSE AND THE BOHR ATOM|1987;
Webb, S.|Out of this world: Colliding universes, branes, strings, and other wild ideas of modern physics|2004;
Weber, F.|Pulsars as astrophysical laboratories for nuclear and particle physics|1999;
Weberruss, V.A.|Survey on quantum physics|1998;
Webre, P.|RISKS AND BENEFITS OF BUILDING THE SUPERCONDUCTING SUPERCOLLIDER|1988;
Weekes, T.|Very high energy gamma-ray astronomy|2003;
Wehr, M.R.|PHYSICS OF THE ATOM|1984;
Weigel, Herbert|Chiral Soliton Models for Baryons|2008;
Weigert, A.|Astronomy and astrophysics. (In German)|1996;
Weihreter, E.|Compact synchrotron light sources|1996;
Weinberg, David H.|From the Big Bang to the Multiverse: Translations in Space and Time|2010;
Weinberg, Steven|Cosmology|2008;
Weinberg, Steven|Dreams of a final theory. (In German)|1993;
Weinberg, Steven|Dreams of a final theory: The Search for the fundamental laws of nature|1992;
Weinberg, Steven|THE DISCOVERY OF SUBATOMIC PARTICLES. (IN GERMAN)|1984;
Weinberg, Steven|THE DISCOVERY OF SUBATOMIC PARTICLES|1984;
Weinberg, Steven|The First Three Minutes. A Modern View of the Origin of the Universe|1977;
Weinberg, Steven|The Quantum theory of fields. Vol. 1: Foundations|1995;
Weinberg, Steven|The quantum theory of fields. Vol. 2: Modern applications|1996;
Weinberg, Steven|The quantum theory of fields. Vol. 3: Supersymmetry|2000;
Weiner, R.M., (ed.)|Bose-Einstein correlations in particle and nuclear physics: A collection of reprints|1997;
Weiner, R.M.|Introduction to Bose-Einstein correlations and subatomic interferometry|2000;
Weise, W., (Ed.)|QUARKS AND NUCLEI|1985;
Weisskopf, V.|My life: A Physicist, contemporary, and humanist remembers our century. (In German)|1991;
Wells, James D.|How to Find a Hidden World at the Large Hadron Collider|2008;
Wen, X.G.|Quantum field theory of many-body systems: From the origin of sound to an origin of light and electrons|2004;
Wendel, N.|Logical foundations of physics: A unified theory of space, time, and matter|2000;
Wess, J.|Supersymmetry and supergravity|1992;
Wess, Julius|SUPERSYMMETRY AND SUPERGRAVITY: NOTES FROM LECTURES GIVEN AT PRINCETON UNIVERSITY. PART 2.|1982;
Wesson, P.S.|Five-dimensional physics: Classical and quantum consequences of Kaluza-Klein cosmology|2006;
Wesson, P.S.|Space - time - matter: Modern Kaluza-Klein theory|1999;
West, Peter C., (Ed.)|SUPERSYMMETRY: A DECADE OF DEVELOPMENT|1986;
West, Peter C.|INTRODUCTION TO SUPERSYMMETRY AND SUPERGRAVITY|1986;
West, Peter C.|Introduction to supersymmetry and supergravity|1990;
Weyl, H.|Space, time, matter: Lectures on general relativity. (In German)|1993;
Wheeler, J.A., (Ed.)|QUANTUM THEORY AND MEASUREMENT|1984;
Wheeler, J.A.|A Journey into gravity and space-time|1990;
Wheeler, J.A.|Geons, black holes, and quantum foam: A life in physics|1998;
Wheeler, J.A.|Gravitation and space-time: The Four-dimensional event space of relativity theory. (A journey into gravity and space-time). (In German)|1991;
Wheeler, J.C.|Cosmic catastrophes: Supernovae, gamma-ray bursts, and adventures in hyperspace|2000;
Wiedemann, H.|Particle accelerator physics: 2: Nonlinear and higher order beam dynamics|1995;
Wiedemann, H.|Particle accelerator physics: Basic principles and linear beam dynamics|1993;
Wiedemann, Helmut|Synchrotron radiation|2003;
Wiegel, F.W.|INTRODUCTION TO PATH INTEGRAL METHODS IN PHYSICS AND POLYMER SCIENCE|1986;
Wienands, H.U., (ed.)|The SSC low-energy booster: Design and component prototypes for the first injector synchrotron|1997;
Wigmans, R.|Calorimetry: Energy measurement in particle physics|2000;
Wigner, E.P.|The collected works of Eugen Paul Wigner. Pt. A: The scientific papers. Vol. 3|1997;
Wiik, B.H., (ed.)|From the preshower to the new technologies for supercolliders: In honour of Antonino Zichichi|2002;
Wiik, B.H.|ELECTRON POSITRON INTERACTIONS|1979;
Wilczek, F.|Fantastic realities: 49 mind journeys and a trip to Stockholm|2006;
Wilczek, Frank, (ed.)|Fractional statistics and anyon superconductivity|1990;
Wilczek, Frank|A Model of anthropic reasoning, addressing the dark to ordinary matter coincidence|2004;
Wilczek, Frank|Diquarks as inspiration and as objects|2004;
Wilczek, Frank|LONGING FOR THE HARMONIES: THEMES AND VARIATIONS FROM MODERN PHYSICS|1988;
Wilczek, Frank|The lightness of being: Mass, ether, and the unification of forces|2008;
Wilczek, Frank|Yang-Mills theory in, beyond, and behind observed reality|2004;
Wilets, Lawrence|Nontopological solitons|1990;
Wilhelm, F.(Ed.)|THE COURSE OF EVOLUTION. HISTORY OF COSMOS, EARTH, AND MAN. (IN GERMAN)|1987;
Wilkes, R.Jeffrey, (Ed.)|NNN06, Next generation nucleon decay and neutrino detectors 2006, Seattle, Washington, 21-23 September 2006|2007;
Wilkinson, D.(ed.)|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 3|1980;
Wilkinson, D., (Ed.)|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 10|1983;
Wilkinson, D., (Ed.)|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 12|1985;
Wilkinson, D., (ed.)|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 5|1981;
Wilkinson, D., (ed.)|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 7|1981;
Wilkinson, D.|PROGRESS IN PARTICLE AND NUCLEAR PHYSICS. VOL. 2|1979;
Will, C.M., (ed.)|THEORY AND EXPERIMENT IN GRAVITATIONAL PHYSICS|1981;
Will, C.M.|Theory and experiment in gravitational physics|1993;
Will, C.M.|WAS EINSTEIN RIGHT? PUTTING GENERAL RELATIVITY TO THE TEST|1986;
Wille, K.|Physics of particle accelerators and synchrotron radiation sources: An Introduction. (In German)|1992;
Wille, K.|The physics of particle accelerators: An introduction|2000;
Williams, W.S.C.|Nuclear and particle physics|1991;
Williams, W.S.C.|Solutions manual for nuclear and particle physics|1995;
Wilson, E.J.N.|An introduction to particle accelerators|2001;
Wiltshire, David L.|The Kerr spacetime: Rotating black holes in general relativity|2009;
Winick, H., (ed.)|Synchrotron radiation sources: A Primer|1995;
Winitzki, Sergei|Eternal inflation|2008;
Winter, K., (ed.)|Neutrino physics (2nd Edition)|2000;
Winter, K., (ed.)|Neutrino physics|1991;
Wissmann, F.|Compton scattering: Investigating the structure of the nucleon with real photons|2004;
Woit, Peter|Not even wrong: The Failure of String Theory & the Continuing Challenge to Unify the Laws of Physics|2006;
Wolf, E., (ed.)|Progress in optics. Vol. 36|1996;
Wolf, E., (ed.)|Progress in optics. Vol. 38|1998;
Wolf, F.A.|Parallel universes: The Search for other worlds|1990;
Wolfenstein, L., (ed.)|CP violation|1989;
Wolfenstein, Lincoln|Exploring fundamental particles|2011;
Wollnik, H.|OPTICS OF CHARGED PARTICLES|1987;
Wolschin, Georg.|Spectrum of physics: Highlights of modern physical and astronomical research. (In German)|1992;
Wolschin, Georg|Facets of physics: Highlights in modern physical and astronomical research|2008;
Wong, C.Y.|Introduction to high-energy heavy ion collisions|1995;
Wong, S.S.M.|Introductory nuclear physics|1998;
Woodhouse, N.M.J.|General relativity|2007;
Woodhouse, N.M.J.|Geometric quantization|1992;
Woodhouse, N.|Geometric Quantization|1980;
Wu, Siye|Mathai-Quillen formalism|2005;
Wu, Ta- You, (ed.)|Relativistic quantum mechanics and quantum fields|1991;
Wu, Zhong-Chao|No boundary universe|1994;
Xing, Zhi-zhong|Neutrinos in particle physics, astronomy and cosmology|2011;
Yagi, K.|Quark-gluon plasma: From big bang to little bang|2005;
Yamazaki, T., (ed.)|Perspectives of meson science|1992;
Yan, Mu-Lin|Quantum horizon and thermodynamics of black hole|2004;
Yan, Y.T., (ed.)|Accelerator physics at the Superconducting Super Collider. Lectures held at the Central Facility auditorium of the SSC Laboratoy, Waxahachie, TX, Oct 1, 1992 to Sep 30, 1993|1995;
Yang, C.N., (Ed.)|BRAID GROUP, KNOT THEORY AND STATISTICAL MECHANICS|1989;
Yang, C.N., (ed.)|Braid group, knot theory and statistical mechanics. Vol. 2|1994;
Yang, C.N.|Selected papers (1945-1980) with commentary|2005;
Yang, Chen-Ning|LECTURES ON FRONTIERS IN PHYSICS|1980;
Yang, Chen-Ning|SELECTED PAPERS 1945 - 1980. WITH COMMENTARY|1987;
Yang, Fujia|Modern atomic and nuclear physics|2010;
Yang, Y.|Solitons in field theory and nonlinear analysis|2001;
Yau, Shing-Tung, (ed.)|Differential geometry inspired by string theory|1999;
Yau, Shing-Tung, (ed.)|Mirror symmetry I|1998;
Yndurain, F.J.|Relativistic quantum mechanics and introduction to field theory|1996;
Yndurain, F.J.|The theory of quark and gluon interactions|1999;
Yoshida, S.|Ultra-high energy particle astrophysics|2003;
Yukalov, V.I.|Lectures on phase transitions|1990;
Yukawa, H.|HIDEKI YUKAWA. 'TABIBITO' (THE TRAVELER)|1982;
Yurov, Artyom V.|Quantum Creation of a Universe with varying speed of light: Lambda-problem and Instantons|2008;
Zakharov, V.E., (ed.)|What is integrability?|1991;
Zakrzewski, W.J.|LOW DIMENSIONAL SIGMA MODELS|1989;
Zamolodchikov, Alexander B.|PHYSICS REVIEWS. VOL. 10, PT. 4: CONFORMAL FIELD THEORY AND CRITICAL PHENOMENA IN TWO-DIMENSIONAL SYSTEMS|1989;
Zampini, Alessandro|Laplacians and gauged Laplacians on a quantum Hopf bundle|2010;
Zavyalov, O.I.|Renormalized quantum field theory|1990;
Zee, A.|FEARFUL SYMMETRY. THE SEARCH FOR BEAUTY IN MODERN PHYSICS|1986;
Zee, A.|Quantum field theory in a nutshell|2003;
Zeh, H.D.|The Physical basis of the direction of time|1992;
Zeidler, E., (ed.)|Quantum field theory. I: Basics in mathematics and physics. A bridge between mathematicians and physicists|2006;
Zeidler, Eberhard|Quantum field theory. II: Quantum electrodynamics. A bridge between mathematicians and physicists|2009;
Zeldovich, Ya.B.|My universe: Selected reviews|1992;
Zeldovich, Ya.B.|RELATIVISTIC ASTROPHYSICS. VOL. 2. THE STRUCTURE AND EVOLUTION OF THE UNIVERSE|1983;
Zelevinsky, Vladimir|Quantum physics, vol. 1: From basics to symmetries and perturbations|2011;
Zelevinsky, Vladimir|Quantum physics, vol. 2: From time-dependent dynamics to many-body physics and quantum chaos|2011;
Zelobenko, D.P.|COMPACT LIE GROUPS AND THEIR REPRESENTATIONS|1981;
Zerlett, G.|Radiation Protection Ordinance. Comments on the German Ordinance on Protection Against Damages Due to Ionizing Radiations|1977;
Zhang, Yuan-Zhong|Special relativity and its experimental foundations|1997;
Zhu, Hong- Yuan, (ed.)|Advances in science of China: Physics. Vol. 1|1986;
Zhu, Hong- Yuan, (ed.)|Advances in science of China: Physics. Vol. 2|1988;
Zhukov, G.P.|8th International Symposium on Nuclear Electronics, Dubna, 24-29 June 1975|1975;
Zichichi, A.|Creativity in science|1999;
Zichichi, A.|HIGHLIGHTS IN PARTICLE PHYSICS. INTERNATIONAL SCHOOL OF SUBNUCLEAR PHYSICS, ERICE, 7-29 JULY 1972|1973;
Zichichi, A.|Laws of Hadronic Matter. 1973 International School of Subnuclear Physics, Erice, July 8-26, 1973. Theoretical Lectures and Seminars|1975;
Zichichi, A.|Lepton and Hadron Structure. 12. International School of Subnuclear Physics, Erice, Trapani, 14-31 July 1974|1975;
Zichichi, A.|Subnuclear physics: The first 50 years: Highlights from Erice to ELN|2000;
Zingl, H.|Few Body Systems and Nuclear Forces. 1. International Conference Held in Graz, Aug 24-30, 1978|1978;
Zinn-Justin, Jean|Phase transitions and renormalization group|2007;
Zinn-Justin, Jean|Quantum field theory and critical phenomena|1989;
Zotter, B.W.|Impedances and wakes in high-energy particle accelerators|1998;
Zuber, K.|Neutrino physics|2004;
Zurek, W.H., (Ed.)|BETWEEN QUANTUM AND COSMOS: STUDIES AND ESSAYS IN HONOR OF JOHN ARCHIBALD WHEELER|1988;
Zwiebach, B.|A first course in string theory|2004;
d'Inverno, R.|Introducing Einstein's relativity|1992;
de Azcarraga, Jose A.|Lie groups, Lie algebras, cohomology and some applications in physics|1995;
de Gouveia Dal Pino, Elisabete M., (ed.)|Magnetic fields in the universe, from laboratory and stars to primordial structures, Angra dos Reis, Brazil, 28 November - 3 December 2004|2005;
de Kerf, E.A.|Lie algebras. Pt. 2: Finite and infinite dimensional Lie algebras and applications in physics|1997;
de Melo, J.P.B.C.|A Light-front quark model for the electromagnetic form-factor of the pion|2005;
de Sabbata, V., (ed.)|New directions in relativity and cosmology. Contributions in honour of Prof. V.B. Johri|1997;
de Sabbata, V., (ed.)|The gravitational constant: Generalized gravitational theories and experiments|2004;
de Sabbata, V.|Spin and torsion in gravitation|1994;
de Sabbata, Venzo|Geometric algebra and applications to physics|2007;
de Souza, Mario E.|The general structure of matter|2004;
van Nieuwenhuizen, P.|Anomalies in quantum field theory: Cancellation of anomalies in d = 10 supergravity|1989;
van Nieuwenhuizen, Peter|Supersymmetry, supergravity, superspace and BRST symmetry in a simple model|2004;
van Putten, Maurice H.P.M.|Gravitational Radiation, Luminous Black Holes and Gamma-Ray Burst Supernovae|2005;
van Rienen, Ursula|Numerical methods in computational electrodynamics: Linear systems in practical applications|2001;
van Suijlekom, Walter D.|Renormalization of gauge fields using Hopf algebras|2008;
van Tilburg, Jeroen Ashwin Niels|Track simulation and reconstruction in LHCb|2005;
von Buttlar, J.|Einstein to the power of 2: The quantum leap of new knowledge|1998;
von Oertzen, Wolfram|Alpha-cluster Condensations in Nuclei and Experimental Approaches for their Studies|2010;
von Westenholz, C.|Differential forms in mathematical physics|1981;
|13TH INTERNATIONAL COSMIC RAY CONFERENCE, DENVER 1973, VOLUME 5 - MISCELLANY, INVITED LECTURES AND MEMORIAL SESSIONS, RAPPORTEUR PAPERS|1973;
|13TH INTERNATIONAL COSMIC RAY CONFERENCE, DENVER, 17-30 AUGUST 1973, CONFERENCE PAPERS, VOL. 1 - OG SESSIONS, (ORIGIN, GALACTIC PHENOMENA)|1973;
|13TH INTERNATIONAL COSMIC RAY CONFERENCE, DENVER, 17-30 AUGUST 1973, CONFERENCE PAPERS, VOL. 3 - MN AND HE SESSIONS (MUONS AND NEUTRINOS, ULTRAHIGH-ENERGY INTERACTIONS)|1973;
|13TH INTERNATIONAL COSMIC RAY CONFERENCE, DENVER, 17-30 AUGUST 1973. CONFERENCE PAPERS, VOL. 2 - MG AND SP SESSIONS (MODULATION, GEOPHYSICAL EFFECTS: ENERGETIC SOLAR PARTICLES AND PHOTONS)|1973;
|13TH INTERNATIONAL COSMIC RAY CONFERENCE. CONFERENCE PAPERS. VOLUME 4, AS AND TI SESSIONS (EXTENSIVE AIR SHOWERS, TECHNIQUES AND INSTRUMENTATION)|1973;
|1976 PEP Conference. Stanford, Jun 23-25, 1976|1976;
|19th International Cosmic Ray Conference, Conference papers, La Jolla, USA, August 11-12, 1985|1985;
|ACCELERATORS IN THE NATIONAL RESEARCH CENTERS. (IN GERMAN)|1987;
|ANNUAL REVIEW OF ASTRONOMY AND ASTROPHYSICS. VOL. 17|1979;
|Annual Review of Nuclear Science. Vol. 25|1975;
|Annual Review of Nuclear Science. Vol. 26, 1976|1976;
|Annual Review of Nuclear Science. Vol. 27, 1977|1977;
|Astronomy and astrophysics in the new millennium: Panel reports|2001;
|Astronomy and astrophysics in the new millennium|2001;
|BRANDEIS UNIVERSITY SUMMER INSTITUTE IN THEORETICAL PHYSICS, 1969. ATOMIC PHYSICS AND ASTROPHYSICS. VOLUME 1|1971;
|Basic Aspects of High-Energy Particle Interactions and Radiation Dosimetry|1978;
|Bulletin of Scientific and Technical Informations. Elementary Particle Physics|1974;
|CERN PS Experimenters Handbook. First Edition, June 1975|1975;
|CMS, tracker technical design report|1998;
|COMPUTING FOR PARTICLE PHYSICS: REPORT OF THE HEPAP SUBPANEL ON COMPUTER NEEDS FOR THE NEXT DECADE|1985;
|CONSTRUCTIVE CIVIL ENGINEERING. (IN GERMAN)|1986;
|COSMOLOGY. STRUCTURE AND EVOLUTION OF THE UNIVERSE. WITH AN INTRODUCTION BY IMMO APPENZELLER. (IN GERMAN)|1985;
|Classical and Quantum Mechanical Aspects of Heavy Ion Collisions. Symposium Held at the Max Planck Institute Fur Kernphysik, Heidelberg, October 2-5, 1974|1975;
|Conceptual design of the SPL II, a high-power superconducting H- linac at CERN|2006;
|Connecting quarks with the cosmos: Eleven science questions for the new century|2003;
|Cosmology and particle physics. (In German)|1990;
|Critical Phenomena. Sitges International School on Statistical Mechanics, June 1976. Lecture Notes in Physics, Vol. 54|1976;
|Discovering the quantum universe : The Role of particle colliders|2005;
|Discussion Meeting on PETRA Experiments, Frascati, Mar 1-5, 1976|1976;
|ELEMENTARY PARTICLES. FIRST SCHOOL OF PHYSICS AT ITEF. VOLUME 3, MOSCOW 1973. (IN RUSSIAN)|1973;
|Elementary-particle physics: Revealing the secrets of energy and matter|1998;
|Encyclopedia of modern physics|1990;
|Final report of the EURISOL design study (2005-2009), a Design study for a European Isotope-Separation-On-Line radioactive ion beam facility|2009;
|GRAVITATION: SPACE-TIME STRUCTURE AND INTERACTION. (IN GERMAN)|1987;
|Gravitational physics: Exploring the structure of space and time|1999;
|Heavy Ion, High Spin States and Nuclear Structure. Lectures Presented at the International Seminar on Nuclear Physics, Trieste, 17 September-21 December 1973, 1. and 2.|1975;
|Helium resources of the United States|2009;
|INTERNATIONAL SCHOOL ON HIGH-ENERGY PHYSICS FOR YOUNG SCIENTISTS, GOMEL, USSR, 25 AUG - 5 SEP 1973. (IN RUSSIAN)|1973;
|KEK PS 1980s: A Summary of experimental programs at the KEK PS in 1980s|1990;
|MATHEMATICS AND PHYSICS. (MOSTLY IN FRENCH) SEMINAR OF THE ECOLE NORMALE SUPERIEURE 1979 - 1982|1984;
|Neutrino Physics at High-Energies. International Colloquium, Paris, 18-20 March 1975|1975;
|Nuclear physics: The core of matter, the fuel of stars|1999;
|PARTICLES, FIELDS, AND SYMMETRIES. (IN GERMAN)|1985;
|PHYSICS THROUGH THE 1990S: ELEMENTARY PARTICLE PHYSICS|1986;
|PHYSICS THROUGH THE 1990S: GRAVITATION, COSMOLOGY, AND COSMIC RAY PHYSICS|1986;
|PHYSICS THROUGH THE 1990S: NUCLEAR PHYSICS|1986;
|Particle physics|1995;
|Physics at TeV colliders, La physique du TeV aux collisionneurs, Les Houches 2 May - 20 May 2005|2006;
|Physics in a new era: An overview|2001;
|Problems of Elementary Particle Physics. Lectures Delivered at the Spring School of Experimental and Theoretical Physics, Erevan, April 8-19, 1|1975;
|Proposal for a Phi factory|1990;
|Quantum physics and condensed matter physics: A Collection of papers dedicated to Wolfgang Weller on the occasion of his 60th birthday. (Partly in German)|1992;
|Recommendations of the International Commission on Radiological Protection. Adopted Jan 17, 1977.|1977;
|Revealing the Hidden Nature of Space and Time: Charting the Course for Elementary Particle Physics|2006;
|SELECTED WORKS OF PENG HUAN-WU. (PARTLY IN CHINESE)|1986;
|SUMMARY OF TALKS GIVEN AT SPSC WORKSHOP ON NEUTRINO PHYSICS, CERN, OCTOBER 30-31, 1978|1978;
|SUMMARY OF THE PRELIMINARY DESIGN OF BEIJING 2.2-GEV / 2.8-GEV ELECTRON POSITRON COLLIDER|1982;
|Some Questions of Theoretical Physics, 1975|1975;
|THE 3RD NORDIC MEETING ON HIGH-ENERGY REACTIONS IN NUCLEI, GEILO, NORWAY, JANUARY 8-12, 1979. (ABSTRACTS ONLY)|1979;
|TOPICS ON COSMIC RAYS: 60TH ANNIVERSARY OF C.M.G. LATTES. VOL. 2|1984;
|Technical design of a detector to be operated at the Superconducting Super Collider|1992;
|The Mirabelle large liquid hydrogen bubble chamber|1974;
|The energy related applications of helium and recommendations concerning the management of the federal helium programs|1975;
|Theoretical Physics: Memorial Book on Occasion of Professor Rzewuski's 60th Birthday|1976;
|Tristan-Design Report|1978;
|University of Timisoara-Seminar of Theoretical Physics 1977|1978;
|XVI International Symposium on Very High Energy Cosmic Ray Interactions, Monday 28 June 2010 - Friday 02 July 2010, Fermilab|2010;
|Zeroth order design report for the Next Linear Collider: Volume 1|1996;
|Arti, Campo Santo Stefano|2006;
+|Electrical breakdown in gases|1973;
diff --git a/invenio/legacy/refextract/kbs/report-numbers.kb b/invenio/legacy/refextract/kbs/report-numbers.kb
index f23dd4893..9505bd8ab 100644
--- a/invenio/legacy/refextract/kbs/report-numbers.kb
+++ b/invenio/legacy/refextract/kbs/report-numbers.kb
@@ -1,304 +1,305 @@
*****LANL*****
<s/syymm999>
<syymm999>
ACC PHYS ---acc-phys
ADAP ORG ---adap-org
ALG GEOM ---alg-geom
AO SCI ---ao-sci
AUTO FMS ---auto-fms
BAYES AN ---bayes-an
CD HG ---cd-hg
CMP LG ---cmp-lg
COMP GAS ---comp-gas
DG GA ---dg-ga
FUNCT AN ---funct-an
GR QC ---gr-qc
ARXIVHEP EX ---hep-ex
ARXIVHEP PH ---hep-ph
ARXIVHEP TH ---hep-th
LC OM ---lc-om
MTRL TH ---mtrl-th
NEURO CEL ---neuro-cel
NEURO DEV ---neuro-dev
NEURO SCI ---neuro-sci
PATT SOL ---patt-sol
*****FermiLab*****
< 9999>
< 999>
< yy 999 [AET ]>
< yyyy 999 [AET ]>
< yyyy 99>
FERMILAB CONF ---FERMILAB-Conf
FERMILAB FN ---FERMILAB-FN
FERMILAB PUB ---FERMILAB-Pub
FERMILAB TM ---FERMILAB-TM
FERMILAB DESIGN ---FERMILAB-Design
FERMILAB THESIS ---FERMILAB-Thesis
FERMILAB MASTERS---FERMILAB-Masters
*****Fermilab DØ notes*****
< 9999>
DØ NOTE---DØ-Note
*****Fermilab CDF*****
< 9999>
CDF ---CDF
CDF-ANAL-ELECTROWEAK-CDFR ---CDF-ANAL-ELECTROWEAK-CDFR
CDF-ANAL-EXOTIC-CDFR ---CDF-ANAL-EXOTIC-CDFR
CDF-ANAL-EXOTIC-PUBLIC ---CDF-ANAL-EXOTIC-PUBLIC
CDF-ANAL-JET-PUBLIC ---CDF-ANAL-JET-PUBLIC
CDF-ANAL-TOP-CDFR ---CDF-ANAL-TOP-CDFR
CDF-ANAL-TOP-PUBLIC ---CDF-ANAL-TOP-PUBLIC
CDF-DOC-CDF-CDFR ---CDF-DOC-CDF-CDFR
CDF-DOC-CDF-PUBLIC ---CDF-DOC-CDF-PUBLIC
CDF-DOC-PLUG-UPGR-CDFR ---CDF-DOC-PLUG-UPGR-CDFR
CDF-NOTE ---CDF-NOTE
CDF-PHYS-BOTTOM-PUBLIC ---CDF-PHYS-BOTTOM-PUBLIC
CDF-PUB ---CDF-PUB
CDF-PUB-BOTTOM-CDFR ---CDF-PUB-BOTTOM-CDFR
CDF-PUB-BOTTOM-PUBLIC ---CDF-PUB-BOTTOM-PUBLIC
CDF-PUB-CDF-PUBLIC ---CDF-PUB-CDF-PUBLIC
CDF-PUB-ELECTROWEAK-CDFR ---CDF-PUB-ELECTROWEAK-CDFR
CDF-PUB-ELECTROWEAK-PUBLIC---CDF-PUB-ELECTROWEAK-PUBLIC
CDF-PUB-EXOTIC-CDFR ---CDF-PUB-EXOTIC-CDFR
CDF-PUB-EXOTIC-PUBLIC ---CDF-PUB-EXOTIC-PUBLIC
CDF-PUB-HEAVYFLAVOR-PUBLIC---CDF-PUB-HEAVYFLAVOR-PUBLIC
CDF-PUB-JET-CDFR ---CDF-PUB-JET-CDFR
CDF-PUB-JET-PUBLIC ---CDF-PUB-JET-PUBLIC
CDF-PUB-MIN-BIAS-PUBLIC ---CDF-PUB-MIN-BIAS-PUBLIC
CDF-PUB-PLUG-UPGR-PUBLIC ---CDF-PUB-PLUG-UPGR-PUBLIC
CDF-PUB-PUBLIC ---CDF-PUB-PUBLIC
CDF-PUB-SEC-VTX-PUBLIC ---CDF-PUB-SEC-VTX-PUBLIC
CDF-PUB-SEC_VTX-PUBLIC ---CDF-PUB-SEC_VTX-PUBLIC
CDF-PUB-TOP-CDFR ---CDF-PUB-TOP-CDFR
CDF-PUB-TOP-PUBLIC ---CDF-PUB-TOP-PUBLIC
CDF-THESIS-BOTTOM-PUBLIC ---CDF-THESIS-BOTTOM-PUBLIC
CDF-THESIS-CDF-PUBLIC ---CDF-THESIS-CDF-PUBLIC
CDF-THESIS-TOP-PUBLIC ---CDF-THESIS-TOP-PUBLIC
CDF-TOP-PUBLIC ---CDF-TOP-PUBLIC
*****CERN*****
< yy 999>
<syyyy 999>
ALEPH ---ALEPH
ALICE ---ALICE
ALICE INT ---ALICE-INT
ALICE NOTE ---ALICE-INT
ATL CAL ---ATL-CAL
ATL COM ---ATL-COM
ATL COM SOFT ---ATL-COM-SOFT
ATL COM PUB ---ATL-COM-DAQ
ATL COM DAQ ---ATL-COM-DAQ
ATL COM MUON ---ATL-COM-MUON
ATL COM PHYS ---ATL-COM-PHYS
TL COM PHYS ---ATL-COM-PHYS
ATL COM TILECAL ---ATL-COM-TILECAL
ATL COM LARG ---ATL-COM-LARG
ATL DAQ ---ATL-DAQ
ATL DAQ CONF ---ATL-DAQ-CONF
ATL GEN ---ATL-GEN
ATL INDET ---ATL-INDET
ATL LARG ---ATL-LARG
ATL MUON ---ATL-MUON
ATL PUB MUON ---ATL-PUB-MUON
ATL PHYS ---ATL-PHYS
ATL PHYS PUB ---ATL-PHYS-PUB
ATL PHYSPUB ---ATL-PHYS-PUB
ATLPHYS PUB ---ATL-PHYS-PUB
ATL PHYS INT ---ATL-PHYS-INT
ATL PHYSINT ---ATL-PHYS-INT
ATLPHYS INT ---ATL-PHYS-INT
ATL TECH ---ATL-TECH
ATL TILECAL ---ATL-TILECAL
ATL SOFT ---ATL-SOFT
ATL SOFT PUB ---ATL-SOFT-PUB
ATL IS EN ---ATL-IS-EN
ATL IS QA ---ATL-IS-QA
ATL LARG PUB ---ATL-LARG-PUB
ATL COM LARG ---ATL-COM-LARG
TL COM LARG ---ATL-COM-LARG
ATLCOM LARG ---ATL-COM-LARG
ATL MAGNET PUB ---ATL-MAGNET-PUB
CERN AB ---CERN-AB
CERN ALEPH ---CERN-ALEPH
CERN ALEPH PHYSIC ---CERN-ALEPH-PHYSIC
CERN ALEPH PUB ---CERN-ALEPH-PUB
CERN ALICE INT ---CERN-ALICE-INT
CERN ALICE PUB ---CERN-ALICE-PUB
CERN ALI ---CERN-ALI
CERN AS ---CERN-AS
CERN AT ---CERN-AT
CERN ATL COM CAL ---CERN-ATL-COM-CAL
CERN ATL COM DAQ ---CERN-ATL-COM-DAQ
CERN ATL COM GEN ---CERN-ATL-COM-GEN
CERN ATL COM INDET ---CERN-ATL-COM-INDET
CERN ATL COM LARG ---CERN-ATL-COM-LARG
CERN ATL COM MUON ---CERN-ATL-COM-MUON
CERN ATL COM PHYS ---CERN-ATL-COM-PHYS
CERN ATL COM TECH ---CERN-ATL-COM
CERN ATL COM TILECAL ---CERN-ATL-COM
CERN ATL DAQ ---CERN-ATL-DAQ
CERN ATL SOFT ---CERN-ATL-SOFT
CERN ATL SOFT INT ---CERN-ATL-SOFT-INT
CERN ATL SOFT PUB ---CERN-ATL-SOFT-PUB
CERN CMS ---CERN-CMS
CERN CMS CR ---CERN-CMS-CR
CERN CMS NOTE ---CERN-CMS-NOTE
CERN CN ---CERN-CN
CERN DD ---CERN-DD
CERN DELPHI ---CERN-DELPHI
CERN ECP ---CERN-ECP
CERN EF ---CERN-EF
CERN ECP ---CERN-EP
CERN EST ---CERN-EST
CERN ETT ---CERN-ETT
CERN IT ---CERN-IT
CERN LHCB ---CERN-LHCB
CERN LHCC ---CERN-LHCC
CERN LHC ---CERN-LHC
CERN LHC PHO ---CERN-LHC-PHO
CERN LHC PROJECT REPORT---CERN-LHC-Project-Report
CERN OPEN ---CERN-OPEN
CERN PPE ---CERN-PPE
CERN PS ---CERN-PS
CERN SL ---CERN-SL
CERN SPSC ---CERN-SPSC
CERN ST ---CERN-ST
CERN TH ---CERN-TH
CERN THESIS ---CERN-THESIS
CERN TIS ---CERN-TIS
CERN ATS ---CERN-ATS
CERN ---CERN
CMS CR ---CMS-CR
CMS NOTE ---CMS-NOTE
CMS EXO ---CMS-EXO
LHCB ---LHCB
SN ATLAS ---SN-ATLAS
PAS SUSY ---CMS-PAS-SUS
CMS PAS EXO ---CMS-PAS-EXO
CMS PAS HIN ---CMS-PAS-HIN
CMS PAS QCD ---CMS-PAS-QCD
CMS PAS TOP ---CMS-PAS-TOP
CMS PAS SUS ---CMS-PAS-SUS
CMS PAS BPH ---CMS-PAS-BPH
CMS PAS SMP ---CMS-PAS-SMP
CMS PAS HIG ---CMS-PAS-HIG
CMS PAS EWK ---CMS-PAS-EWK
CMS PAS BTV ---CMS-PAS-BTV
CMS PAS FWD ---CMS-PAS-FWD
CMS PAS TRK ---CMS-PAS-TRK
CMS PAS SMP ---CMS-PAS-SMP
CMS PAS PFT ---CMS-PAS-PFT
CMS PAS MUO ---CMS-PAS-MUO
CMS PAS JME ---CMS-PAS-JME
CMS PAS EGM ---CMS-PAS-EGM
CMS PAS DIF ---CMS-PAS-DIF
+CMS PAS B2G ---CMS-PAS-B2G
ATLTILECAL PUB ---ATLTILECAL-PUB
ATLAS TECH PUB ---ATLAS-TECH-PUB
TLCOM MAGNET ---TLCOM-MAGNET
ATLLARG ---ATL-LARG
*****CERN MORE*****
< yyyy 999>
< yyyy 99>
< yyyy 9>
< yy 99>
< yy 9>
CERN LHCB ---CERN-LHCB
CERN LHCC ---CERN-LHCC
CERN PHESS ---CERN-PHESS
*****CERN EVEN MORE*****
< 9>
CMS UG TP ---CMS-UG-TP
*****CERN DIFFERENT FORMAT*****
< 9999999>
CERN GE ---CERN-GE
*****LHC*****
< 999>
< 9999>
CERN CLIC NOTE ---CERN-CLIC-Note
LHC PROJECT NOTE ---LHC-Project-Note
CERN LHC PROJECT REPORT ---CERN-LHC-Project-Report
LHC PROJECT REPORT ---CERN-LHC-Project-Report
CLIC NOTE ---CERN-CLIC-Note
ATLAS TDR ---ATL-TDR
CMS TDR ---CMS-TDR
ATC TT ID ---ATC-TT-ID
ATC TT IN ---ATC-TT-IN
LHCCP ---LHCCP
*****KEK*****
< 9999>
< yy 999>
< yyyy 999>
KEK CP ---KEK-CP
KEK INT ---KEK-Internal
KEK INTERNAL ---KEK-Internal
KEK PREPRINT ---KEK-Preprint
KEK TH ---KEK-TH
*****DESY*****
< yy 999>
< yyyy 999>
DESY ---DESY
DESY M ---DESY-M
DESY THESIS ---DESY-THESIS
*****DESY F*****
<99 9>
<9 99 99>
<99 99 99>
DESY F ---DESY-F
*****SLAC*****
< 999>
< 9999>
< yy 99>
SLAC AP ---SLAC-AP
SLAC PUB ---SLAC-PUB
SLAC R ---SLAC-R
SLAC TN ---SLAC-TN
SLAC WP ---SLAC-WP
*****Berkeley Lab*****
< 99999>
LBNL---LBNL
*****Argonne National Laboratory*****
< yy 99>
ANL HEP TR ---ANL-HEP-TR
*****Antares*****
< yyyy 999>
ANTARES SOFT ---ANTARES-SOFT
ANTARES PHYS ---ANTARES-Phys
ANTARES OPMO ---ANTARES-Opmo
diff --git a/invenio/legacy/refextract/linker.py b/invenio/legacy/refextract/linker.py
index 06b0d42ce..a6aee2ed5 100644
--- a/invenio/legacy/refextract/linker.py
+++ b/invenio/legacy/refextract/linker.py
@@ -1,86 +1,113 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from invenio.legacy.bibrank.citation_indexer import \
get_recids_matching_query as bibrank_search, \
standardize_report_number
from invenio.modules.indexer.tokenizers.BibIndexJournalTokenizer import \
CFG_JOURNAL_PUBINFO_STANDARD_FORM
from invenio.legacy.bibrank.tag_based_indexer import load_config
+from invenio.legacy.search_engine import get_collection_reclist, get_fieldvalues
+from intbitset import intbitset
def config_cache(cache={}):
if 'config' not in cache:
cache['config'] = load_config('citation')
return cache['config']
def get_recids_matching_query(p, f, m='e'):
"""Return list of recIDs matching query for pattern and field."""
config = config_cache()
recids = bibrank_search(p=p.encode('utf-8'), f=f, config=config, m=m)
return list(recids)
def format_journal(format_string, mappings):
"""format the publ infostring according to the format"""
def replace(char, data):
return data.get(char, char)
for c in mappings.keys():
format_string = format_string.replace(c, replace(c, mappings))
return format_string
def find_journal(citation_element):
tags_values = {
'773__p': citation_element['title'],
'773__v': citation_element['volume'],
'773__c': citation_element['page'],
'773__y': citation_element['year'],
}
journal_string = format_journal(
CFG_JOURNAL_PUBINFO_STANDARD_FORM, tags_values)
return get_recids_matching_query(journal_string, 'journal')
def find_reportnumber(citation_element):
reportnumber = standardize_report_number(citation_element['report_num'])
return get_recids_matching_query(reportnumber, 'reportnumber')
def find_doi(citation_element):
doi_string = citation_element['doi_string']
return get_recids_matching_query(doi_string, 'doi')
def find_referenced_recid(citation_element):
el_type = citation_element['type']
if el_type in FINDERS:
return FINDERS[el_type](citation_element)
return []
+def find_book(citation_element):
+ books_recids = get_collection_reclist('Books')
+ search_string = citation_element['title']
+ recids = intbitset(get_recids_matching_query(search_string, 'title'))
+ recids &= books_recids
+ if len(recids) == 1:
+ return recids
+
+ if 'year' in citation_element:
+ for recid in recids:
+ year_tags = get_fieldvalues(recid, '269__c')
+ for tag in year_tags:
+ if tag == citation_element['year']:
+ return [recid]
+
+ return []
+
+
+def find_isbn(citation_element):
+ books_recids = get_collection_reclist('Books')
+ recids = intbitset(get_recids_matching_query(citation_element['ISBN'], 'isbn'))
+ return list(recids & books_recids)
+
FINDERS = {
'JOURNAL': find_journal,
'REPORTNUMBER': find_reportnumber,
'DOI': find_doi,
+ 'BOOK': find_book,
+ 'ISBN': find_isbn,
}
diff --git a/invenio/legacy/search_engine/__init__.py b/invenio/legacy/search_engine/__init__.py
index 5afef2828..bed7f33d8 100644
--- a/invenio/legacy/search_engine/__init__.py
+++ b/invenio/legacy/search_engine/__init__.py
@@ -1,7027 +1,7038 @@
# -*- coding: utf-8 -*-
## This file is part of Invenio.
## Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
# pylint: disable=C0301,W0703
"""Invenio Search Engine in mod_python."""
__lastupdated__ = """$Date$"""
__revision__ = "$Id$"
## import general modules:
import cgi
import cStringIO
import copy
import os
import re
import time
import string
import urllib
import urlparse
import zlib
import sys
try:
## import optional module:
import numpy
CFG_NUMPY_IMPORTABLE = True
except ImportError:
CFG_NUMPY_IMPORTABLE = False
if sys.hexversion < 0x2040000:
# pylint: disable=W0622
from sets import Set as set
# pylint: enable=W0622
from six import iteritems
## import Invenio stuff:
from invenio.base.globals import cfg
from invenio.config import \
CFG_CERN_SITE, \
CFG_INSPIRE_SITE, \
+ CFG_SCOAP3_SITE, \
CFG_OAI_ID_FIELD, \
CFG_WEBCOMMENT_ALLOW_REVIEWS, \
CFG_WEBSEARCH_CALL_BIBFORMAT, \
CFG_WEBSEARCH_CREATE_SIMILARLY_NAMED_AUTHORS_LINK_BOX, \
CFG_WEBSEARCH_FIELDS_CONVERT, \
CFG_WEBSEARCH_NB_RECORDS_TO_SORT, \
CFG_WEBSEARCH_SEARCH_CACHE_SIZE, \
CFG_WEBSEARCH_SEARCH_CACHE_TIMEOUT, \
CFG_WEBSEARCH_USE_MATHJAX_FOR_FORMATS, \
CFG_WEBSEARCH_USE_ALEPH_SYSNOS, \
CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS, \
CFG_WEBSEARCH_FULLTEXT_SNIPPETS, \
CFG_WEBSEARCH_DISPLAY_NEAREST_TERMS, \
CFG_WEBSEARCH_WILDCARD_LIMIT, \
CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE, \
CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG, \
CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS, \
CFG_WEBSEARCH_SYNONYM_KBRS, \
CFG_SITE_LANG, \
CFG_SITE_NAME, \
CFG_LOGDIR, \
CFG_SITE_URL, \
CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS, \
CFG_SOLR_URL, \
CFG_WEBSEARCH_DETAILED_META_FORMAT, \
CFG_SITE_RECORD, \
CFG_WEBSEARCH_PREV_NEXT_HIT_LIMIT, \
CFG_WEBSEARCH_VIEWRESTRCOLL_POLICY, \
CFG_BIBSORT_BUCKETS, \
CFG_BIBSORT_ENABLED, \
CFG_XAPIAN_ENABLED, \
CFG_BIBINDEX_CHARS_PUNCTUATION, \
CFG_BASE_URL
from invenio.modules.search.errors import \
InvenioWebSearchUnknownCollectionError, \
InvenioWebSearchWildcardLimitError, \
CFG_WEBSEARCH_IDXPAIRS_FIELDS,\
CFG_WEBSEARCH_IDXPAIRS_EXACT_SEARCH
-from invenio.legacy.bibrecord import get_fieldvalues, get_fieldvalues_alephseq_like
+from invenio.legacy.bibrecord import (get_fieldvalues,
+ get_fieldvalues_alephseq_like,
+ record_exists)
from invenio.legacy.bibrecord import create_record, record_xml_output
-from invenio.legacy.bibrank.record_sorter import get_bibrank_methods, is_method_valid, rank_records as rank_records_bibrank
+from invenio.legacy.bibrank.record_sorter import (
+ get_bibrank_methods,
+ is_method_valid,
+ rank_records as rank_records_bibrank,
+ rank_by_citations)
from invenio.legacy.bibrank.downloads_similarity import register_page_view_event, calculate_reading_similarity_list
from invenio.legacy.bibindex.engine_stemmer import stem
from invenio.modules.indexer.tokenizers.BibIndexDefaultTokenizer import BibIndexDefaultTokenizer
from invenio.modules.indexer.tokenizers.BibIndexCJKTokenizer import BibIndexCJKTokenizer, is_there_any_CJK_character_in_text
from invenio.legacy.bibindex.engine_utils import author_name_requires_phrase_search
from invenio.legacy.bibindex.engine_washer import wash_index_term, lower_index_term, wash_author_name
from invenio.legacy.bibindex.engine_config import CFG_BIBINDEX_SYNONYM_MATCH_TYPE
from invenio.legacy.bibindex.adminlib import get_idx_indexer
from invenio.modules.formatter import format_record, format_records, get_output_format_content_type, create_excel
from invenio.legacy.bibrank.downloads_grapher import create_download_history_graph_and_box
from invenio.modules.knowledge.api import get_kbr_values
from invenio.legacy.miscutil.data_cacher import DataCacher
from invenio.legacy.websearch_external_collections import print_external_results_overview, perform_external_collection_search
from invenio.modules.access.control import acc_get_action_id
from invenio.modules.access.local_config import VIEWRESTRCOLL, \
CFG_ACC_GRANT_AUTHOR_RIGHTS_TO_EMAILS_IN_TAGS, \
CFG_ACC_GRANT_VIEWER_RIGHTS_TO_EMAILS_IN_TAGS
from invenio.legacy.websearch.adminlib import get_detailed_page_tabs, get_detailed_page_tabs_counts
from intbitset import intbitset
from invenio.legacy.dbquery import DatabaseError, deserialize_via_marshal, InvenioDbQueryWildcardLimitError
from invenio.modules.access.engine import acc_authorize_action
from invenio.ext.logging import register_exception
from invenio.ext.cache import cache
from invenio.utils.text import encode_for_xml, wash_for_utf8, strip_accents
from invenio.utils.html import get_mathjax_header
from invenio.utils.html import nmtoken_from_string
from invenio.legacy import bibrecord
import invenio.legacy.template
webstyle_templates = invenio.legacy.template.load('webstyle')
webcomment_templates = invenio.legacy.template.load('webcomment')
from invenio.legacy.bibrank.citation_searcher import calculate_cited_by_list, \
calculate_co_cited_with_list, get_records_with_num_cites, \
get_refersto_hitset, get_citedby_hitset, get_cited_by_list, \
get_refers_to_list, get_citers_log
from invenio.legacy.bibrank.citation_grapher import create_citation_history_graph_and_box
from invenio.legacy.bibrank.selfcites_searcher import get_self_cited_by_list, \
get_self_cited_by, \
get_self_refers_to_list
from invenio.legacy.dbquery import run_sql, run_sql_with_limit, \
wash_table_column_name, get_table_update_time
from invenio.legacy.webuser import getUid, collect_user_info, session_param_set
from invenio.legacy.webpage import pageheaderonly, pagefooteronly, create_error_box, write_warning
from invenio.base.i18n import gettext_set_language
from invenio.legacy.search_engine.query_parser import SearchQueryParenthesisedParser, \
SpiresToInvenioSyntaxConverter
from invenio.utils import apache
from invenio.legacy.miscutil.solrutils_bibindex_searcher import solr_get_bitset
from invenio.legacy.miscutil.xapianutils_bibindex_searcher import xapian_get_bitset
from invenio.modules.search import services
try:
import invenio.legacy.template
websearch_templates = invenio.legacy.template.load('websearch')
except:
pass
from invenio.legacy.websearch_external_collections import calculate_hosted_collections_results, do_calculate_hosted_collections_results
from invenio.legacy.websearch_external_collections.config import CFG_HOSTED_COLLECTION_TIMEOUT_ANTE_SEARCH
from invenio.legacy.websearch_external_collections.config import CFG_HOSTED_COLLECTION_TIMEOUT_POST_SEARCH
from invenio.legacy.websearch_external_collections.config import CFG_EXTERNAL_COLLECTION_MAXRESULTS
from invenio.legacy.bibauthorid.config import LIMIT_TO_COLLECTIONS as BIBAUTHORID_LIMIT_TO_COLLECTIONS
from .utils import record_exists
websearch_templates = invenio.template.load('websearch')
VIEWRESTRCOLL_ID = acc_get_action_id(VIEWRESTRCOLL)
## global vars:
cfg_nb_browse_seen_records = 100 # limit of the number of records to check when browsing certain collection
cfg_nicely_ordered_collection_list = 0 # do we propose collection list nicely ordered or alphabetical?
## precompile some often-used regexp for speed reasons:
re_word = re.compile(r'[\s]')
re_quotes = re.compile('[\'\"]')
re_doublequote = re.compile('\"')
re_logical_and = re.compile(r'\sand\s', re.I)
re_logical_or = re.compile(r'\sor\s', re.I)
re_logical_not = re.compile(r'\snot\s', re.I)
re_operators = re.compile(r'\s([\+\-\|])\s')
re_pattern_wildcards_after_spaces = re.compile(r'(\s)[\*\%]+')
re_pattern_single_quotes = re.compile("'(.*?)'")
re_pattern_double_quotes = re.compile("\"(.*?)\"")
re_pattern_parens_quotes = re.compile(r'[\'\"]{1}[^\'\"]*(\([^\'\"]*\))[^\'\"]*[\'\"]{1}')
re_pattern_regexp_quotes = re.compile(r"\/(.*?)\/")
re_pattern_spaces_after_colon = re.compile(r'(:\s+)')
re_pattern_short_words = re.compile(r'([\s\"]\w{1,3})[\*\%]+')
re_pattern_space = re.compile("__SPACE__")
re_pattern_today = re.compile(r"\$TODAY\$")
re_pattern_parens = re.compile(r'\([^\)]+\s+[^\)]+\)')
re_punctuation_followed_by_space = re.compile(CFG_BIBINDEX_CHARS_PUNCTUATION + r'\s')
## em possible values
EM_REPOSITORY={"body" : "B",
"header" : "H",
"footer" : "F",
"search_box" : "S",
"see_also_box" : "L",
"basket" : "K",
"alert" : "A",
"search_info" : "I",
"overview" : "O",
"all_portalboxes" : "P",
"te_portalbox" : "Pte",
"tp_portalbox" : "Ptp",
"np_portalbox" : "Pnp",
"ne_portalbox" : "Pne",
"lt_portalbox" : "Plt",
"rt_portalbox" : "Prt",
"search_services": "SER"};
class RestrictedCollectionDataCacher(DataCacher):
def __init__(self):
def cache_filler():
ret = []
res = run_sql("""SELECT DISTINCT ar.value
FROM accROLE_accACTION_accARGUMENT raa JOIN accARGUMENT ar ON raa.id_accARGUMENT = ar.id
WHERE ar.keyword = 'collection' AND raa.id_accACTION = %s""", (VIEWRESTRCOLL_ID,), run_on_slave=True)
for coll in res:
ret.append(coll[0])
return ret
def timestamp_verifier():
return max(get_table_update_time('accROLE_accACTION_accARGUMENT'), get_table_update_time('accARGUMENT'))
DataCacher.__init__(self, cache_filler, timestamp_verifier)
def collection_restricted_p(collection, recreate_cache_if_needed=True):
if recreate_cache_if_needed:
restricted_collection_cache.recreate_cache_if_needed()
return collection in restricted_collection_cache.cache
try:
restricted_collection_cache.is_ok_p
except NameError:
restricted_collection_cache = RestrictedCollectionDataCacher()
def ziplist(*lists):
"""Just like zip(), but returns lists of lists instead of lists of tuples
Example:
zip([f1, f2, f3], [p1, p2, p3], [op1, op2, '']) =>
[(f1, p1, op1), (f2, p2, op2), (f3, p3, '')]
ziplist([f1, f2, f3], [p1, p2, p3], [op1, op2, '']) =>
[[f1, p1, op1], [f2, p2, op2], [f3, p3, '']]
FIXME: This is handy to have, and should live somewhere else, like
miscutil.really_useful_functions or something.
XXX: Starting in python 2.6, the same can be achieved (faster) by
using itertools.izip_longest(); when the minimum recommended Python
is bumped, we should use that instead.
"""
def l(*items):
return list(items)
return map(l, *lists)
def get_permitted_restricted_collections(user_info, recreate_cache_if_needed=True):
"""Return a list of collection that are restricted but for which the user
is authorized."""
if recreate_cache_if_needed:
restricted_collection_cache.recreate_cache_if_needed()
ret = []
auths = acc_authorize_action(
user_info,
'viewrestrcoll',
batch_args=True,
collection=restricted_collection_cache.cache
)
for collection, auth in zip(restricted_collection_cache.cache, auths):
if auth[0] == 0:
ret.append(collection)
return ret
def get_all_restricted_recids():
"""
Return the set of all the restricted recids, i.e. the ids of those records
which belong to at least one restricted collection.
"""
ret = intbitset()
for collection in restricted_collection_cache.cache:
ret |= get_collection_reclist(collection)
return ret
def get_restricted_collections_for_recid(recid, recreate_cache_if_needed=True):
"""
Return the list of restricted collection names to which recid belongs.
"""
if recreate_cache_if_needed:
restricted_collection_cache.recreate_cache_if_needed()
collection_reclist_cache.recreate_cache_if_needed()
return [collection for collection in restricted_collection_cache.cache if recid in get_collection_reclist(collection, recreate_cache_if_needed=False)]
def is_user_owner_of_record(user_info, recid):
"""
Check if the user is owner of the record, i.e. he is the submitter
and/or belongs to a owner-like group authorized to 'see' the record.
@param user_info: the user_info dictionary that describe the user.
@type user_info: user_info dictionary
@param recid: the record identifier.
@type recid: positive integer
@return: True if the user is 'owner' of the record; False otherwise
@rtype: bool
"""
authorized_emails_or_group = []
for tag in CFG_ACC_GRANT_AUTHOR_RIGHTS_TO_EMAILS_IN_TAGS:
authorized_emails_or_group.extend(get_fieldvalues(recid, tag))
for email_or_group in authorized_emails_or_group:
if email_or_group in user_info['group']:
return True
email = email_or_group.strip().lower()
if user_info['email'].strip().lower() == email:
return True
return False
###FIXME: This method needs to be refactorized
def is_user_viewer_of_record(user_info, recid):
"""
Check if the user is allow to view the record based in the marc tags
inside CFG_ACC_GRANT_VIEWER_RIGHTS_TO_EMAILS_IN_TAGS
i.e. his email is inside the 506__m tag or he is inside an e-group listed
in the 506__m tag
@param user_info: the user_info dictionary that describe the user.
@type user_info: user_info dictionary
@param recid: the record identifier.
@type recid: positive integer
@return: True if the user is 'allow to view' the record; False otherwise
@rtype: bool
"""
authorized_emails_or_group = []
for tag in CFG_ACC_GRANT_VIEWER_RIGHTS_TO_EMAILS_IN_TAGS:
authorized_emails_or_group.extend(get_fieldvalues(recid, tag))
for email_or_group in authorized_emails_or_group:
if email_or_group in user_info['group']:
return True
email = email_or_group.strip().lower()
if user_info['email'].strip().lower() == email:
return True
return False
def check_user_can_view_record(user_info, recid):
"""
Check if the user is authorized to view the given recid. The function
grants access in two cases: either user has author rights on this
record, or he has view rights to the primary collection this record
belongs to.
@param user_info: the user_info dictionary that describe the user.
@type user_info: user_info dictionary
@param recid: the record identifier.
@type recid: positive integer
@return: (0, ''), when authorization is granted, (>0, 'message') when
authorization is not granted
@rtype: (int, string)
"""
policy = CFG_WEBSEARCH_VIEWRESTRCOLL_POLICY.strip().upper()
if isinstance(recid, str):
recid = int(recid)
## At this point, either webcoll has not yet run or there are some
## restricted collections. Let's see first if the user own the record.
if is_user_owner_of_record(user_info, recid):
## Perfect! It's authorized then!
return (0, '')
if is_user_viewer_of_record(user_info, recid):
## Perfect! It's authorized then!
return (0, '')
restricted_collections = get_restricted_collections_for_recid(recid, recreate_cache_if_needed=False)
if not restricted_collections and record_public_p(recid):
## The record is public and not part of any restricted collection
return (0, '')
if restricted_collections:
## If there are restricted collections the user must be authorized to all/any of them (depending on the policy)
auth_code, auth_msg = 0, ''
for collection in restricted_collections:
(auth_code, auth_msg) = acc_authorize_action(user_info, VIEWRESTRCOLL, collection=collection)
if auth_code and policy != 'ANY':
## Ouch! the user is not authorized to this collection
return (auth_code, auth_msg)
elif auth_code == 0 and policy == 'ANY':
## Good! At least one collection is authorized
return (0, '')
## Depending on the policy, the user will be either authorized or not
return auth_code, auth_msg
if is_record_in_any_collection(recid, recreate_cache_if_needed=False):
## the record is not in any restricted collection
return (0, '')
elif record_exists(recid) > 0:
## We are in the case where webcoll has not run.
## Let's authorize SUPERADMIN
(auth_code, auth_msg) = acc_authorize_action(user_info, VIEWRESTRCOLL, collection=None)
if auth_code == 0:
return (0, '')
else:
## Too bad. Let's print a nice message:
return (1, """The record you are trying to access has just been
submitted to the system and needs to be assigned to the
proper collections. It is currently restricted for security reasons
until the assignment will be fully completed. Please come back later to
properly access this record.""")
else:
## The record either does not exists or has been deleted.
## Let's handle these situations outside of this code.
return (0, '')
class IndexStemmingDataCacher(DataCacher):
"""
Provides cache for stemming information for word/phrase indexes.
This class is not to be used directly; use function
get_index_stemming_language() instead.
"""
def __init__(self):
def cache_filler():
try:
res = run_sql("""SELECT id, stemming_language FROM idxINDEX""")
except DatabaseError:
# database problems, return empty cache
return {}
return dict(res)
def timestamp_verifier():
return get_table_update_time('idxINDEX')
DataCacher.__init__(self, cache_filler, timestamp_verifier)
try:
index_stemming_cache.is_ok_p
except Exception:
index_stemming_cache = IndexStemmingDataCacher()
def get_index_stemming_language(index_id, recreate_cache_if_needed=True):
"""Return stemming langugage for given index."""
if recreate_cache_if_needed:
index_stemming_cache.recreate_cache_if_needed()
return index_stemming_cache.cache[index_id]
class FieldTokenizerDataCacher(DataCacher):
"""
Provides cache for tokenizer information for fields corresponding to indexes.
This class is not to be used directly; use function
get_field_tokenizer_type() instead.
"""
def __init__(self):
def cache_filler():
try:
res = run_sql("""SELECT fld.code, ind.tokenizer FROM idxINDEX AS ind, field AS fld, idxINDEX_field AS indfld WHERE ind.id = indfld.id_idxINDEX AND indfld.id_field = fld.id""")
except DatabaseError:
# database problems, return empty cache
return {}
return dict(res)
def timestamp_verifier():
return get_table_update_time('idxINDEX')
DataCacher.__init__(self, cache_filler, timestamp_verifier)
try:
field_tokenizer_cache.is_ok_p
except Exception:
field_tokenizer_cache = FieldTokenizerDataCacher()
def get_field_tokenizer_type(field_name, recreate_cache_if_needed=True):
"""Return tokenizer type for given field corresponding to an index if applicable."""
if recreate_cache_if_needed:
field_tokenizer_cache.recreate_cache_if_needed()
tokenizer = None
try:
tokenizer = field_tokenizer_cache.cache[field_name]
except KeyError:
return None
return tokenizer
class CollectionRecListDataCacher(DataCacher):
"""
Provides cache for collection reclist hitsets. This class is not
to be used directly; use function get_collection_reclist() instead.
"""
def __init__(self):
def cache_filler():
ret = {}
res = run_sql("SELECT name FROM collection")
for name in res:
ret[name[0]] = None # this will be filled later during runtime by calling get_collection_reclist(coll)
return ret
def timestamp_verifier():
return get_table_update_time('collection')
DataCacher.__init__(self, cache_filler, timestamp_verifier)
try:
if not collection_reclist_cache.is_ok_p:
raise Exception
except Exception:
collection_reclist_cache = CollectionRecListDataCacher()
def get_collection_reclist(coll, recreate_cache_if_needed=True):
"""Return hitset of recIDs that belong to the collection 'coll'."""
if recreate_cache_if_needed:
collection_reclist_cache.recreate_cache_if_needed()
if coll not in collection_reclist_cache.cache:
return intbitset() # collection does not exist; return empty set
if not collection_reclist_cache.cache[coll]:
# collection's reclist not in the cache yet, so calculate it
# and fill the cache:
reclist = intbitset()
query = "SELECT nbrecs,reclist FROM collection WHERE name=%s"
res = run_sql(query, (coll, ), 1)
if res:
try:
reclist = intbitset(res[0][1])
except IndexError:
pass
collection_reclist_cache.cache[coll] = reclist
# finally, return reclist:
return collection_reclist_cache.cache[coll]
def get_available_output_formats(visible_only=False):
"""
Return the list of available output formats. When visible_only is
True, returns only those output formats that have visibility flag
set to 1.
"""
formats = []
query = "SELECT code,name FROM format"
if visible_only:
query += " WHERE visibility='1'"
query += " ORDER BY name ASC"
res = run_sql(query)
if res:
# propose found formats:
for code, name in res:
formats.append({'value': code,
'text': name
})
else:
formats.append({'value': 'hb',
'text': "HTML brief"
})
return formats
# Flask cache for search results.
from invenio.modules.search.cache import search_results_cache, get_search_results_cache_key
class CollectionI18nNameDataCacher(DataCacher):
"""
Provides cache for I18N collection names. This class is not to be
used directly; use function get_coll_i18nname() instead.
"""
def __init__(self):
def cache_filler():
ret = {}
try:
res = run_sql("SELECT c.name,cn.ln,cn.value FROM collectionname AS cn, collection AS c WHERE cn.id_collection=c.id AND cn.type='ln'") # ln=long name
except Exception:
# database problems
return {}
for c, ln, i18nname in res:
if i18nname:
if c not in ret:
ret[c] = {}
ret[c][ln] = i18nname
return ret
def timestamp_verifier():
return get_table_update_time('collectionname')
DataCacher.__init__(self, cache_filler, timestamp_verifier)
try:
if not collection_i18nname_cache.is_ok_p:
raise Exception
except Exception:
collection_i18nname_cache = CollectionI18nNameDataCacher()
def get_coll_i18nname(c, ln=CFG_SITE_LANG, verify_cache_timestamp=True):
"""
Return nicely formatted collection name (of the name type `ln'
(=long name)) for collection C in language LN.
This function uses collection_i18nname_cache, but it verifies
whether the cache is up-to-date first by default. This
verification step is performed by checking the DB table update
time. So, if you call this function 1000 times, it can get very
slow because it will do 1000 table update time verifications, even
though collection names change not that often.
Hence the parameter VERIFY_CACHE_TIMESTAMP which, when set to
False, will assume the cache is already up-to-date. This is
useful namely in the generation of collection lists for the search
results page.
"""
if verify_cache_timestamp:
collection_i18nname_cache.recreate_cache_if_needed()
out = c
try:
out = collection_i18nname_cache.cache[c][ln]
except KeyError:
pass # translation in LN does not exist
return out
class FieldI18nNameDataCacher(DataCacher):
"""
Provides cache for I18N field names. This class is not to be used
directly; use function get_field_i18nname() instead.
"""
def __init__(self):
def cache_filler():
ret = {}
try:
res = run_sql("SELECT f.name,fn.ln,fn.value FROM fieldname AS fn, field AS f WHERE fn.id_field=f.id AND fn.type='ln'") # ln=long name
except Exception:
# database problems, return empty cache
return {}
for f, ln, i18nname in res:
if i18nname:
if f not in ret:
ret[f] = {}
ret[f][ln] = i18nname
return ret
def timestamp_verifier():
return get_table_update_time('fieldname')
DataCacher.__init__(self, cache_filler, timestamp_verifier)
try:
if not field_i18nname_cache.is_ok_p:
raise Exception
except Exception:
field_i18nname_cache = FieldI18nNameDataCacher()
def get_field_i18nname(f, ln=CFG_SITE_LANG, verify_cache_timestamp=True):
"""
Return nicely formatted field name (of type 'ln', 'long name') for
field F in language LN.
If VERIFY_CACHE_TIMESTAMP is set to True, then verify DB timestamp
and field I18N name cache timestamp and refresh cache from the DB
if needed. Otherwise don't bother checking DB timestamp and
return the cached value. (This is useful when get_field_i18nname
is called inside a loop.)
"""
if verify_cache_timestamp:
field_i18nname_cache.recreate_cache_if_needed()
out = f
try:
out = field_i18nname_cache.cache[f][ln]
except KeyError:
pass # translation in LN does not exist
return out
def get_alphabetically_ordered_collection_list(level=0, ln=CFG_SITE_LANG):
"""Returns nicely ordered (score respected) list of collections, more exactly list of tuples
(collection name, printable collection name).
Suitable for create_search_box()."""
out = []
res = run_sql("SELECT name FROM collection ORDER BY name ASC")
for c_name in res:
c_name = c_name[0]
# make a nice printable name (e.g. truncate c_printable for
# long collection names in given language):
c_printable_fullname = get_coll_i18nname(c_name, ln, False)
c_printable = wash_index_term(c_printable_fullname, 30, False)
if c_printable != c_printable_fullname:
c_printable = c_printable + "..."
if level:
c_printable = " " + level * '-' + " " + c_printable
out.append([c_name, c_printable])
return out
def get_nicely_ordered_collection_list(collid=1, level=0, ln=CFG_SITE_LANG):
"""Returns nicely ordered (score respected) list of collections, more exactly list of tuples
(collection name, printable collection name).
Suitable for create_search_box()."""
colls_nicely_ordered = []
res = run_sql("""SELECT c.name,cc.id_son FROM collection_collection AS cc, collection AS c
WHERE c.id=cc.id_son AND cc.id_dad=%s ORDER BY score DESC""", (collid, ))
for c, cid in res:
# make a nice printable name (e.g. truncate c_printable for
# long collection names in given language):
c_printable_fullname = get_coll_i18nname(c, ln, False)
c_printable = wash_index_term(c_printable_fullname, 30, False)
if c_printable != c_printable_fullname:
c_printable = c_printable + "..."
if level:
c_printable = " " + level * '-' + " " + c_printable
colls_nicely_ordered.append([c, c_printable])
colls_nicely_ordered = colls_nicely_ordered + get_nicely_ordered_collection_list(cid, level+1, ln=ln)
return colls_nicely_ordered
def get_index_id_from_field(field):
"""
Return index id with name corresponding to FIELD, or the first
index id where the logical field code named FIELD is indexed.
Return zero in case there is no index defined for this field.
Example: field='author', output=4.
"""
out = 0
if not field:
field = 'global' # empty string field means 'global' index (field 'anyfield')
# first look in the index table:
res = run_sql("""SELECT id FROM idxINDEX WHERE name=%s""", (field,))
if res:
out = res[0][0]
return out
# not found in the index table, now look in the field table:
res = run_sql("""SELECT w.id FROM idxINDEX AS w, idxINDEX_field AS wf, field AS f
WHERE f.code=%s AND wf.id_field=f.id AND w.id=wf.id_idxINDEX
LIMIT 1""", (field,))
if res:
out = res[0][0]
return out
def get_words_from_pattern(pattern):
"""
Returns list of whitespace-separated words from pattern, removing any
trailing punctuation-like signs from words in pattern.
"""
words = {}
# clean trailing punctuation signs inside pattern
pattern = re_punctuation_followed_by_space.sub(' ', pattern)
for word in pattern.split():
if word not in words:
words[word] = 1
return words.keys()
def create_basic_search_units(req, p, f, m=None, of='hb'):
"""Splits search pattern and search field into a list of independently searchable units.
- A search unit consists of '(operator, pattern, field, type, hitset)' tuples where
'operator' is set union (|), set intersection (+) or set exclusion (-);
'pattern' is either a word (e.g. muon*) or a phrase (e.g. 'nuclear physics');
'field' is either a code like 'title' or MARC tag like '100__a';
'type' is the search type ('w' for word file search, 'a' for access file search).
- Optionally, the function accepts the match type argument 'm'.
If it is set (e.g. from advanced search interface), then it
performs this kind of matching. If it is not set, then a guess is made.
'm' can have values: 'a'='all of the words', 'o'='any of the words',
'p'='phrase/substring', 'r'='regular expression',
'e'='exact value'.
- Warnings are printed on req (when not None) in case of HTML output formats."""
opfts = [] # will hold (o,p,f,t,h) units
# FIXME: quick hack for the journal index
if f == 'journal':
opfts.append(['+', p, f, 'w'])
return opfts
## check arguments: is desired matching type set?
if m:
## A - matching type is known; good!
if m == 'e':
# A1 - exact value:
opfts.append(['+', p, f, 'a']) # '+' since we have only one unit
elif m == 'p':
# A2 - phrase/substring:
opfts.append(['+', "%" + p + "%", f, 'a']) # '+' since we have only one unit
elif m == 'r':
# A3 - regular expression:
opfts.append(['+', p, f, 'r']) # '+' since we have only one unit
elif m == 'a' or m == 'w':
# A4 - all of the words:
p = strip_accents(p) # strip accents for 'w' mode, FIXME: delete when not needed
for word in get_words_from_pattern(p):
opfts.append(['+', word, f, 'w']) # '+' in all units
elif m == 'o':
# A5 - any of the words:
p = strip_accents(p) # strip accents for 'w' mode, FIXME: delete when not needed
for word in get_words_from_pattern(p):
if len(opfts)==0:
opfts.append(['+', word, f, 'w']) # '+' in the first unit
else:
opfts.append(['|', word, f, 'w']) # '|' in further units
else:
if of.startswith("h"):
write_warning("Matching type '%s' is not implemented yet." % cgi.escape(m), "Warning", req=req)
opfts.append(['+', "%" + p + "%", f, 'w'])
else:
## B - matching type is not known: let us try to determine it by some heuristics
if f and p[0] == '"' and p[-1] == '"':
## B0 - does 'p' start and end by double quote, and is 'f' defined? => doing ACC search
opfts.append(['+', p[1:-1], f, 'a'])
elif f in ('author', 'firstauthor', 'exactauthor', 'exactfirstauthor', 'authorityauthor') and author_name_requires_phrase_search(p):
## B1 - do we search in author, and does 'p' contain space/comma/dot/etc?
## => doing washed ACC search
opfts.append(['+', p, f, 'a'])
elif f and p[0] == "'" and p[-1] == "'":
## B0bis - does 'p' start and end by single quote, and is 'f' defined? => doing ACC search
opfts.append(['+', '%' + p[1:-1] + '%', f, 'a'])
elif f and p[0] == "/" and p[-1] == "/":
## B0ter - does 'p' start and end by a slash, and is 'f' defined? => doing regexp search
opfts.append(['+', p[1:-1], f, 'r'])
elif f and p.find(',') >= 0:
## B1 - does 'p' contain comma, and is 'f' defined? => doing ACC search
opfts.append(['+', p, f, 'a'])
elif f and str(f[0:2]).isdigit():
## B2 - does 'f' exist and starts by two digits? => doing ACC search
opfts.append(['+', p, f, 'a'])
else:
## B3 - doing WRD search, but maybe ACC too
# search units are separated by spaces unless the space is within single or double quotes
# so, let us replace temporarily any space within quotes by '__SPACE__'
p = re_pattern_single_quotes.sub(lambda x: "'"+x.group(1).replace(' ', '__SPACE__')+"'", p)
p = re_pattern_double_quotes.sub(lambda x: "\""+x.group(1).replace(' ', '__SPACE__')+"\"", p)
p = re_pattern_regexp_quotes.sub(lambda x: "/"+x.group(1).replace(' ', '__SPACE__')+"/", p)
# and spaces after colon as well:
p = re_pattern_spaces_after_colon.sub(lambda x: x.group(1).replace(' ', '__SPACE__'), p)
# wash argument:
p = re_logical_and.sub(" ", p)
p = re_logical_or.sub(" |", p)
p = re_logical_not.sub(" -", p)
p = re_operators.sub(r' \1', p)
for pi in p.split(): # iterate through separated units (or items, as "pi" stands for "p item")
pi = re_pattern_space.sub(" ", pi) # replace back '__SPACE__' by ' '
# firstly, determine set operator
if pi[0] == '+' or pi[0] == '-' or pi[0] == '|':
oi = pi[0]
pi = pi[1:]
else:
# okay, there is no operator, so let us decide what to do by default
oi = '+' # by default we are doing set intersection...
# secondly, determine search pattern and field:
if pi.find(":") > 0:
fi, pi = pi.split(":", 1)
fi = wash_field(fi)
# test whether fi is a real index code or a MARC-tag defined code:
if fi in get_fieldcodes() or '00' <= fi[:2] <= '99':
pass
else:
# it is not, so join it back:
fi, pi = f, fi + ":" + pi
else:
fi, pi = f, pi
# wash 'fi' argument:
fi = wash_field(fi)
# wash 'pi' argument:
pi = pi.strip() # strip eventual spaces
if re_quotes.match(pi):
# B3a - quotes are found => do ACC search (phrase search)
if pi[0] == '"' and pi[-1] == '"':
pi = pi.replace('"', '') # remove quote signs
opfts.append([oi, pi, fi, 'a'])
elif pi[0] == "'" and pi[-1] == "'":
pi = pi.replace("'", "") # remove quote signs
opfts.append([oi, "%" + pi + "%", fi, 'a'])
else: # unbalanced quotes, so fall back to WRD query:
opfts.append([oi, pi, fi, 'w'])
elif pi.startswith('/') and pi.endswith('/'):
# B3b - pi has slashes around => do regexp search
opfts.append([oi, pi[1:-1], fi, 'r'])
elif fi and len(fi) > 1 and str(fi[0]).isdigit() and str(fi[1]).isdigit():
# B3c - fi exists and starts by two digits => do ACC search
opfts.append([oi, pi, fi, 'a'])
elif fi and not get_index_id_from_field(fi) and get_field_name(fi):
# B3d - logical field fi exists but there is no WRD index for fi => try ACC search
opfts.append([oi, pi, fi, 'a'])
else:
# B3e - general case => do WRD search
pi = strip_accents(pi) # strip accents for 'w' mode, FIXME: delete when not needed
for pii in get_words_from_pattern(pi):
opfts.append([oi, pii, fi, 'w'])
## sanity check:
for i in range(0, len(opfts)):
try:
pi = opfts[i][1]
if pi == '*':
if of.startswith("h"):
write_warning("Ignoring standalone wildcard word.", "Warning", req=req)
del opfts[i]
if pi == '' or pi == ' ':
fi = opfts[i][2]
if fi:
if of.startswith("h"):
write_warning("Ignoring empty <em>%s</em> search term." % fi, "Warning", req=req)
del opfts[i]
except:
pass
## replace old logical field names if applicable:
if CFG_WEBSEARCH_FIELDS_CONVERT:
opfts = [[o, p, wash_field(f), t] for o, p, f, t in opfts]
## return search units:
return opfts
def page_start(req, of, cc, aas, ln, uid, title_message=None,
description='', keywords='', recID=-1, tab='', p='', em=''):
"""
Start page according to given output format.
@param title_message: title of the page, not escaped for HTML
@param description: description of the page, not escaped for HTML
@param keywords: keywords of the page, not escaped for HTML
"""
_ = gettext_set_language(ln)
if not req or isinstance(req, cStringIO.OutputType):
return # we were called from CLI
if not title_message:
title_message = _("Search Results")
content_type = get_output_format_content_type(of)
if of.startswith('x'):
if of == 'xr':
# we are doing RSS output
req.content_type = "application/rss+xml"
req.send_http_header()
req.write("""<?xml version="1.0" encoding="UTF-8"?>\n""")
else:
# we are doing XML output:
req.content_type = get_output_format_content_type(of, 'text/xml')
req.send_http_header()
req.write("""<?xml version="1.0" encoding="UTF-8"?>\n""")
elif of.startswith('t') or str(of[0:3]).isdigit():
# we are doing plain text output:
req.content_type = "text/plain"
req.send_http_header()
elif of == "intbitset":
req.content_type = "application/octet-stream"
req.send_http_header()
+ elif of == "recjson":
+ req.content_type = "application/json"
+ req.send_http_header()
elif of == "id":
pass # nothing to do, we shall only return list of recIDs
elif content_type == 'text/html':
# we are doing HTML output:
req.content_type = "text/html"
req.send_http_header()
if not description:
description = "%s %s." % (cc, _("Search Results"))
if not keywords:
keywords = "%s, WebSearch, %s" % (get_coll_i18nname(CFG_SITE_NAME, ln, False), get_coll_i18nname(cc, ln, False))
## generate RSS URL:
argd = {}
if req.args:
argd = cgi.parse_qs(req.args)
rssurl = websearch_templates.build_rss_url(argd)
## add MathJax if displaying single records (FIXME: find
## eventual better place to this code)
if of.lower() in CFG_WEBSEARCH_USE_MATHJAX_FOR_FORMATS:
metaheaderadd = get_mathjax_header(req.is_https())
else:
metaheaderadd = ''
# Add metadata in meta tags for Google scholar-esque harvesting...
# only if we have a detailed meta format and we are looking at a
# single record
- if (recID != -1 and CFG_WEBSEARCH_DETAILED_META_FORMAT):
+ if recID != -1 and CFG_WEBSEARCH_DETAILED_META_FORMAT and \
+ record_exists(recID) == 1:
metaheaderadd += format_record(recID,
CFG_WEBSEARCH_DETAILED_META_FORMAT,
ln=ln)
## generate navtrail:
navtrail = create_navtrail_links(cc, aas, ln)
if navtrail != '':
navtrail += ' &gt; '
if (tab != '' or ((of != '' or of.lower() != 'hd') and of != 'hb')) and \
recID != -1:
# If we are not in information tab in HD format, customize
# the nav. trail to have a link back to main record. (Due
# to the way perform_request_search() works, hb
# (lowercase) is equal to hd)
navtrail += ' <a class="navtrail" href="%s/%s/%s">%s</a>' % \
(CFG_BASE_URL, CFG_SITE_RECORD, recID, cgi.escape(title_message))
if (of != '' or of.lower() != 'hd') and of != 'hb':
# Export
format_name = of
query = "SELECT name FROM format WHERE code=%s"
res = run_sql(query, (of,))
if res:
format_name = res[0][0]
navtrail += ' &gt; ' + format_name
else:
# Discussion, citations, etc. tabs
tab_label = get_detailed_page_tabs(cc, ln=ln)[tab]['label']
navtrail += ' &gt; ' + _(tab_label)
else:
navtrail += cgi.escape(title_message)
if p:
# we are serving search/browse results pages, so insert pattern:
navtrail += ": " + cgi.escape(p)
title_message = p + " - " + title_message
body_css_classes = []
if cc:
# we know the collection, lets allow page styles based on cc
#collection names may not satisfy rules for css classes which
#are something like: -?[_a-zA-Z]+[_a-zA-Z0-9-]*
#however it isn't clear what we should do about cases with
#numbers, so we leave them to fail. Everything else becomes "_"
css = nmtoken_from_string(cc).replace('.', '_').replace('-', '_').replace(':', '_')
body_css_classes.append(css)
## finally, print page header:
if em == '' or EM_REPOSITORY["header"] in em:
req.write(pageheaderonly(req=req, title=title_message,
navtrail=navtrail,
description=description,
keywords=keywords,
metaheaderadd=metaheaderadd,
uid=uid,
language=ln,
navmenuid='search',
navtrail_append_title_p=0,
rssurl=rssurl,
body_css_classes=body_css_classes))
req.write(websearch_templates.tmpl_search_pagestart(ln=ln))
else:
req.content_type = content_type
req.send_http_header()
def page_end(req, of="hb", ln=CFG_SITE_LANG, em=""):
"End page according to given output format: e.g. close XML tags, add HTML footer, etc."
if of == "id":
return [] # empty recID list
if of == "intbitset":
return intbitset()
if not req:
return # we were called from CLI
if of.startswith('h'):
req.write(websearch_templates.tmpl_search_pageend(ln = ln)) # pagebody end
if em == "" or EM_REPOSITORY["footer"] in em:
req.write(pagefooteronly(lastupdated=__lastupdated__, language=ln, req=req))
return
def create_page_title_search_pattern_info(p, p1, p2, p3):
"""Create the search pattern bit for the page <title> web page
HTML header. Basically combine p and (p1,p2,p3) together so that
the page header may be filled whether we are in the Simple Search
or Advanced Search interface contexts."""
out = ""
if p:
out = p
else:
out = p1
if p2:
out += ' ' + p2
if p3:
out += ' ' + p3
return out
def create_inputdate_box(name="d1", selected_year=0, selected_month=0, selected_day=0, ln=CFG_SITE_LANG):
"Produces 'From Date', 'Until Date' kind of selection box. Suitable for search options."
_ = gettext_set_language(ln)
box = ""
# day
box += """<select name="%sd">""" % name
box += """<option value="">%s""" % _("any day")
for day in range(1, 32):
box += """<option value="%02d"%s>%02d""" % (day, is_selected(day, selected_day), day)
box += """</select>"""
# month
box += """<select name="%sm">""" % name
box += """<option value="">%s""" % _("any month")
# trailing space in May distinguishes short/long form of the month name
for mm, month in [(1, _("January")), (2, _("February")), (3, _("March")), (4, _("April")),
(5, _("May ")), (6, _("June")), (7, _("July")), (8, _("August")),
(9, _("September")), (10, _("October")), (11, _("November")), (12, _("December"))]:
box += """<option value="%02d"%s>%s""" % (mm, is_selected(mm, selected_month), month.strip())
box += """</select>"""
# year
box += """<select name="%sy">""" % name
box += """<option value="">%s""" % _("any year")
this_year = int(time.strftime("%Y", time.localtime()))
for year in range(this_year-20, this_year+1):
box += """<option value="%d"%s>%d""" % (year, is_selected(year, selected_year), year)
box += """</select>"""
return box
def create_search_box(cc, colls, p, f, rg, sf, so, sp, rm, of, ot, aas,
ln, p1, f1, m1, op1, p2, f2, m2, op2, p3, f3,
m3, sc, pl, d1y, d1m, d1d, d2y, d2m, d2d, dt, jrec, ec,
action="", em=""):
"""Create search box for 'search again in the results page' functionality."""
if em != "" and EM_REPOSITORY["search_box"] not in em:
if EM_REPOSITORY["body"] in em and cc != CFG_SITE_NAME:
return '''
<h1 class="headline">%(ccname)s</h1>''' % {'ccname' : cgi.escape(cc), }
else:
return ""
# load the right message language
_ = gettext_set_language(ln)
# some computations
cc_intl = get_coll_i18nname(cc, ln, False)
cc_colID = get_colID(cc)
colls_nicely_ordered = []
if cfg_nicely_ordered_collection_list:
colls_nicely_ordered = get_nicely_ordered_collection_list(ln=ln)
else:
colls_nicely_ordered = get_alphabetically_ordered_collection_list(ln=ln)
colls_nice = []
for (cx, cx_printable) in colls_nicely_ordered:
if not cx.startswith("Unnamed collection"):
colls_nice.append({'value': cx,
'text': cx_printable
})
coll_selects = []
if colls and colls[0] != CFG_SITE_NAME:
# some collections are defined, so print these first, and only then print 'add another collection' heading:
for c in colls:
if c:
temp = []
temp.append({'value': CFG_SITE_NAME,
- 'text': '*** %s ***' % _("any public collection")
+ 'text': '*** %s ***' % (CFG_SCOAP3_SITE and _("any publisher or journal") or _("any public collection"))
})
# this field is used to remove the current collection from the ones to be searched.
temp.append({'value': '',
- 'text': '*** %s ***' % _("remove this collection")
+ 'text': '*** %s ***' % (CFG_SCOAP3_SITE and _("remove this publisher or journal") or _("remove this collection"))
})
for val in colls_nice:
# print collection:
if not cx.startswith("Unnamed collection"):
temp.append({'value': val['value'],
'text': val['text'],
'selected' : (c == re.sub(r"^[\s\-]*", "", val['value']))
})
coll_selects.append(temp)
coll_selects.append([{'value': '',
- 'text' : '*** %s ***' % _("add another collection")
+ 'text' : '*** %s ***' % (CFG_SCOAP3_SITE and _("add another publisher or journal") or _("add another collection"))
}] + colls_nice)
else: # we searched in CFG_SITE_NAME, so print 'any public collection' heading
coll_selects.append([{'value': CFG_SITE_NAME,
- 'text' : '*** %s ***' % _("any public collection")
+ 'text' : '*** %s ***' % (CFG_SCOAP3_SITE and _("any publisher or journal") or _("any public collection"))
}] + colls_nice)
## ranking methods
ranks = [{
'value' : '',
'text' : "- %s %s -" % (_("OR").lower(), _("rank by")),
}]
for (code, name) in get_bibrank_methods(cc_colID, ln):
# propose found rank methods:
ranks.append({
'value': code,
'text': name,
})
formats = get_available_output_formats(visible_only=True)
# show collections in the search box? (not if there is only one
# collection defined, and not if we are in light search)
show_colls = True
show_title = True
if len(collection_reclist_cache.cache.keys()) == 1 or \
aas == -1:
show_colls = False
show_title = False
if cc == CFG_SITE_NAME:
show_title = False
if CFG_INSPIRE_SITE:
show_title = False
return websearch_templates.tmpl_search_box(
ln = ln,
aas = aas,
cc_intl = cc_intl,
cc = cc,
ot = ot,
sp = sp,
action = action,
fieldslist = get_searchwithin_fields(ln=ln, colID=cc_colID),
f1 = f1,
f2 = f2,
f3 = f3,
m1 = m1,
m2 = m2,
m3 = m3,
p1 = p1,
p2 = p2,
p3 = p3,
op1 = op1,
op2 = op2,
rm = rm,
p = p,
f = f,
coll_selects = coll_selects,
d1y = d1y, d2y = d2y, d1m = d1m, d2m = d2m, d1d = d1d, d2d = d2d,
dt = dt,
sort_fields = get_sortby_fields(ln=ln, colID=cc_colID),
sf = sf,
so = so,
ranks = ranks,
sc = sc,
rg = rg,
formats = formats,
of = of,
pl = pl,
jrec = jrec,
ec = ec,
show_colls = show_colls,
show_title = show_title and (em=="" or EM_REPOSITORY["body"] in em)
)
def create_exact_author_browse_help_link(p=None, p1=None, p2=None, p3=None, f=None, f1=None, f2=None, f3=None,
rm=None, cc=None, ln=None, jrec=None, rg=None, aas=0, action=""):
"""Creates a link to help switch from author to exact author while browsing"""
if action == 'browse':
search_fields = (f, f1, f2, f3)
if 'author' in search_fields or 'firstauthor' in search_fields:
def add_exact(field):
if field == 'author' or field == 'firstauthor':
return 'exact' + field
return field
fe, f1e, f2e, f3e = [add_exact(field) for field in search_fields]
link_name = f or f1
link_name = (link_name == 'firstauthor' and 'exact first author') or 'exact author'
return websearch_templates.tmpl_exact_author_browse_help_link(p=p, p1=p1, p2=p2, p3=p3, f=fe, f1=f1e, f2=f2e, f3=f3e,
rm=rm, cc=cc, ln=ln, jrec=jrec, rg=rg, aas=aas, action=action,
link_name=link_name)
return ""
def create_navtrail_links(cc=CFG_SITE_NAME, aas=0, ln=CFG_SITE_LANG, self_p=1, tab=''):
"""Creates navigation trail links, i.e. links to collection
ancestors (except Home collection). If aas==1, then links to
Advanced Search interfaces; otherwise Simple Search.
"""
dads = []
for dad in get_coll_ancestors(cc):
if dad != CFG_SITE_NAME: # exclude Home collection
dads.append((dad, get_coll_i18nname(dad, ln, False)))
if self_p and cc != CFG_SITE_NAME:
dads.append((cc, get_coll_i18nname(cc, ln, False)))
return websearch_templates.tmpl_navtrail_links(
aas=aas, ln=ln, dads=dads)
def get_searchwithin_fields(ln='en', colID=None):
"""Retrieves the fields name used in the 'search within' selection box for the collection ID colID."""
res = None
if colID:
res = run_sql("""SELECT f.code,f.name FROM field AS f, collection_field_fieldvalue AS cff
WHERE cff.type='sew' AND cff.id_collection=%s AND cff.id_field=f.id
ORDER BY cff.score DESC, f.name ASC""", (colID,))
if not res:
res = run_sql("SELECT code,name FROM field ORDER BY name ASC")
fields = [{
'value' : '',
'text' : get_field_i18nname("any field", ln, False)
}]
for field_code, field_name in res:
if field_code and field_code != "anyfield":
fields.append({'value': field_code,
'text': get_field_i18nname(field_name, ln, False)
})
return fields
def get_sortby_fields(ln='en', colID=None):
"""Retrieves the fields name used in the 'sort by' selection box for the collection ID colID."""
_ = gettext_set_language(ln)
res = None
if colID:
res = run_sql("""SELECT DISTINCT(f.code),f.name FROM field AS f, collection_field_fieldvalue AS cff
WHERE cff.type='soo' AND cff.id_collection=%s AND cff.id_field=f.id
ORDER BY cff.score DESC, f.name ASC""", (colID,))
if not res:
# no sort fields defined for this colID, try to take Home collection:
res = run_sql("""SELECT DISTINCT(f.code),f.name FROM field AS f, collection_field_fieldvalue AS cff
WHERE cff.type='soo' AND cff.id_collection=%s AND cff.id_field=f.id
ORDER BY cff.score DESC, f.name ASC""", (1,))
if not res:
# no sort fields defined for the Home collection, take all sort fields defined wherever they are:
res = run_sql("""SELECT DISTINCT(f.code),f.name FROM field AS f, collection_field_fieldvalue AS cff
WHERE cff.type='soo' AND cff.id_field=f.id
ORDER BY cff.score DESC, f.name ASC""",)
fields = [{
'value': '',
'text': _("latest first")
}]
for field_code, field_name in res:
if field_code and field_code != "anyfield":
fields.append({'value': field_code,
'text': get_field_i18nname(field_name, ln, False)
})
return fields
def create_andornot_box(name='op', value='', ln='en'):
"Returns HTML code for the AND/OR/NOT selection box."
_ = gettext_set_language(ln)
out = """
<select name="%s">
<option value="a"%s>%s
<option value="o"%s>%s
<option value="n"%s>%s
</select>
""" % (name,
is_selected('a', value), _("AND"),
is_selected('o', value), _("OR"),
is_selected('n', value), _("AND NOT"))
return out
def create_matchtype_box(name='m', value='', ln='en'):
"Returns HTML code for the 'match type' selection box."
_ = gettext_set_language(ln)
out = """
<select name="%s">
<option value="a"%s>%s
<option value="o"%s>%s
<option value="e"%s>%s
<option value="p"%s>%s
<option value="r"%s>%s
</select>
""" % (name,
is_selected('a', value), _("All of the words:"),
is_selected('o', value), _("Any of the words:"),
is_selected('e', value), _("Exact phrase:"),
is_selected('p', value), _("Partial phrase:"),
is_selected('r', value), _("Regular expression:"))
return out
def is_selected(var, fld):
"Checks if the two are equal, and if yes, returns ' selected'. Useful for select boxes."
if type(var) is int and type(fld) is int:
if var == fld:
return " selected"
elif str(var) == str(fld):
return " selected"
elif fld and len(fld)==3 and fld[0] == "w" and var == fld[1:]:
return " selected"
return ""
def wash_colls(cc, c, split_colls=0, verbose=0):
"""Wash collection list by checking whether user has deselected
anything under 'Narrow search'. Checks also if cc is a list or not.
Return list of cc, colls_to_display, colls_to_search since the list
of collections to display is different from that to search in.
This is because users might have chosen 'split by collection'
functionality.
The behaviour of "collections to display" depends solely whether
user has deselected a particular collection: e.g. if it started
from 'Articles and Preprints' page, and deselected 'Preprints',
then collection to display is 'Articles'. If he did not deselect
anything, then collection to display is 'Articles & Preprints'.
The behaviour of "collections to search in" depends on the
'split_colls' parameter:
* if is equal to 1, then we can wash the colls list down
and search solely in the collection the user started from;
* if is equal to 0, then we are splitting to the first level
of collections, i.e. collections as they appear on the page
we started to search from;
The function raises exception
InvenioWebSearchUnknownCollectionError
if cc or one of c collections is not known.
"""
colls_out = []
colls_out_for_display = []
# list to hold the hosted collections to be searched and displayed
hosted_colls_out = []
debug = ""
if verbose:
debug += "<br />"
debug += "<br />1) --- initial parameters ---"
debug += "<br />cc : %s" % cc
debug += "<br />c : %s" % c
debug += "<br />"
# check what type is 'cc':
if type(cc) is list:
for ci in cc:
if ci in collection_reclist_cache.cache:
# yes this collection is real, so use it:
cc = ci
break
else:
# check once if cc is real:
if cc not in collection_reclist_cache.cache:
if cc:
raise InvenioWebSearchUnknownCollectionError(cc)
else:
cc = CFG_SITE_NAME # cc is not set, so replace it with Home collection
# check type of 'c' argument:
if type(c) is list:
colls = c
else:
colls = [c]
if verbose:
debug += "<br />2) --- after check for the integrity of cc and the being or not c a list ---"
debug += "<br />cc : %s" % cc
debug += "<br />c : %s" % c
debug += "<br />"
# remove all 'unreal' collections:
colls_real = []
for coll in colls:
if coll in collection_reclist_cache.cache:
colls_real.append(coll)
else:
if coll:
raise InvenioWebSearchUnknownCollectionError(coll)
colls = colls_real
if verbose:
debug += "<br />3) --- keeping only the real colls of c ---"
debug += "<br />colls : %s" % colls
debug += "<br />"
# check if some real collections remain:
if len(colls)==0:
colls = [cc]
if verbose:
debug += "<br />4) --- in case no colls were left we use cc directly ---"
debug += "<br />colls : %s" % colls
debug += "<br />"
# then let us check the list of non-restricted "real" sons of 'cc' and compare it to 'coll':
res = run_sql("""SELECT c.name FROM collection AS c,
collection_collection AS cc,
collection AS ccc
WHERE c.id=cc.id_son AND cc.id_dad=ccc.id
AND ccc.name=%s AND cc.type='r'""", (cc,))
# list that holds all the non restricted sons of cc that are also not hosted collections
l_cc_nonrestricted_sons_and_nonhosted_colls = []
res_hosted = run_sql("""SELECT c.name FROM collection AS c,
collection_collection AS cc,
collection AS ccc
WHERE c.id=cc.id_son AND cc.id_dad=ccc.id
AND ccc.name=%s AND cc.type='r'
AND (c.dbquery NOT LIKE 'hostedcollection:%%' OR c.dbquery IS NULL)""", (cc,))
for row_hosted in res_hosted:
l_cc_nonrestricted_sons_and_nonhosted_colls.append(row_hosted[0])
l_cc_nonrestricted_sons_and_nonhosted_colls.sort()
l_cc_nonrestricted_sons = []
l_c = colls[:]
for row in res:
if not collection_restricted_p(row[0]):
l_cc_nonrestricted_sons.append(row[0])
l_c.sort()
l_cc_nonrestricted_sons.sort()
if l_cc_nonrestricted_sons == l_c:
colls_out_for_display = [cc] # yep, washing permitted, it is sufficient to display 'cc'
# the following elif is a hack that preserves the above funcionality when we start searching from
# the frontpage with some hosted collections deselected (either by default or manually)
elif set(l_cc_nonrestricted_sons_and_nonhosted_colls).issubset(set(l_c)):
colls_out_for_display = colls
split_colls = 0
else:
colls_out_for_display = colls # nope, we need to display all 'colls' successively
# remove duplicates:
#colls_out_for_display_nondups=filter(lambda x, colls_out_for_display=colls_out_for_display: colls_out_for_display[x-1] not in colls_out_for_display[x:], range(1, len(colls_out_for_display)+1))
#colls_out_for_display = map(lambda x, colls_out_for_display=colls_out_for_display:colls_out_for_display[x-1], colls_out_for_display_nondups)
#colls_out_for_display = list(set(colls_out_for_display))
#remove duplicates while preserving the order
set_out = set()
colls_out_for_display = [coll for coll in colls_out_for_display if coll not in set_out and not set_out.add(coll)]
if verbose:
debug += "<br />5) --- decide whether colls_out_for_diplay should be colls or is it sufficient for it to be cc; remove duplicates ---"
debug += "<br />colls_out_for_display : %s" % colls_out_for_display
debug += "<br />"
# FIXME: The below quoted part of the code has been commented out
# because it prevents searching in individual restricted daughter
# collections when both parent and all its public daughter
# collections were asked for, in addition to some restricted
# daughter collections. The removal was introduced for hosted
# collections, so we may want to double check in this context.
# the following piece of code takes care of removing collections whose ancestors are going to be searched anyway
# list to hold the collections to be removed
#colls_to_be_removed = []
# first calculate the collections that can safely be removed
#for coll in colls_out_for_display:
# for ancestor in get_coll_ancestors(coll):
# #if ancestor in colls_out_for_display: colls_to_be_removed.append(coll)
# if ancestor in colls_out_for_display and not is_hosted_collection(coll): colls_to_be_removed.append(coll)
# secondly remove the collections
#for coll in colls_to_be_removed:
# colls_out_for_display.remove(coll)
if verbose:
debug += "<br />6) --- remove collections that have ancestors about to be search, unless they are hosted ---"
debug += "<br />colls_out_for_display : %s" % colls_out_for_display
debug += "<br />"
# calculate the hosted collections to be searched.
if colls_out_for_display == [cc]:
if is_hosted_collection(cc):
hosted_colls_out.append(cc)
else:
for coll in get_coll_sons(cc):
if is_hosted_collection(coll):
hosted_colls_out.append(coll)
else:
for coll in colls_out_for_display:
if is_hosted_collection(coll):
hosted_colls_out.append(coll)
if verbose:
debug += "<br />7) --- calculate the hosted_colls_out ---"
debug += "<br />hosted_colls_out : %s" % hosted_colls_out
debug += "<br />"
# second, let us decide on collection splitting:
if split_colls == 0:
# type A - no sons are wanted
colls_out = colls_out_for_display
else:
# type B - sons (first-level descendants) are wanted
for coll in colls_out_for_display:
coll_sons = get_coll_sons(coll)
if coll_sons == []:
colls_out.append(coll)
else:
for coll_son in coll_sons:
if not is_hosted_collection(coll_son):
colls_out.append(coll_son)
#else:
# colls_out = colls_out + coll_sons
# remove duplicates:
#colls_out_nondups=filter(lambda x, colls_out=colls_out: colls_out[x-1] not in colls_out[x:], range(1, len(colls_out)+1))
#colls_out = map(lambda x, colls_out=colls_out:colls_out[x-1], colls_out_nondups)
#colls_out = list(set(colls_out))
#remove duplicates while preserving the order
set_out = set()
colls_out = [coll for coll in colls_out if coll not in set_out and not set_out.add(coll)]
if verbose:
debug += "<br />8) --- calculate the colls_out; remove duplicates ---"
debug += "<br />colls_out : %s" % colls_out
debug += "<br />"
# remove the hosted collections from the collections to be searched
if hosted_colls_out:
for coll in hosted_colls_out:
try:
colls_out.remove(coll)
except ValueError:
# in case coll was not found in colls_out
pass
if verbose:
debug += "<br />9) --- remove the hosted_colls from the colls_out ---"
debug += "<br />colls_out : %s" % colls_out
return (cc, colls_out_for_display, colls_out, hosted_colls_out, debug)
def get_synonym_terms(term, kbr_name, match_type, use_memoise=False):
"""
Return list of synonyms for TERM by looking in KBR_NAME in
MATCH_TYPE style.
@param term: search-time term or index-time term
@type term: str
@param kbr_name: knowledge base name
@type kbr_name: str
@param match_type: specifies how the term matches against the KBR
before doing the lookup. Could be `exact' (default),
'leading_to_comma', `leading_to_number'.
@type match_type: str
@param use_memoise: can we memoise while doing lookups?
@type use_memoise: bool
@return: list of term synonyms
@rtype: list of strings
"""
dterms = {}
## exact match is default:
term_for_lookup = term
term_remainder = ''
## but maybe match different term:
if match_type == CFG_BIBINDEX_SYNONYM_MATCH_TYPE['leading_to_comma']:
mmm = re.match(r'^(.*?)(\s*,.*)$', term)
if mmm:
term_for_lookup = mmm.group(1)
term_remainder = mmm.group(2)
elif match_type == CFG_BIBINDEX_SYNONYM_MATCH_TYPE['leading_to_number']:
mmm = re.match(r'^(.*?)(\s*\d.*)$', term)
if mmm:
term_for_lookup = mmm.group(1)
term_remainder = mmm.group(2)
## FIXME: workaround: escaping SQL wild-card signs, since KBR's
## exact search is doing LIKE query, so would match everything:
term_for_lookup = term_for_lookup.replace('%', '\\%')
## OK, now find synonyms:
for kbr_values in get_kbr_values(kbr_name,
searchkey=term_for_lookup,
searchtype='e',
use_memoise=use_memoise):
for kbr_value in kbr_values:
dterms[kbr_value + term_remainder] = 1
## return list of term synonyms:
return dterms.keys()
def wash_output_format(ouput_format):
"""Wash output format FORMAT. Currently only prevents input like
'of=9' for backwards-compatible format that prints certain fields
only. (for this task, 'of=tm' is preferred)"""
if str(ouput_format[0:3]).isdigit() and len(ouput_format) != 6:
# asked to print MARC tags, but not enough digits,
# so let's switch back to HTML brief default
return 'hb'
else:
return ouput_format
def wash_pattern(p):
"""Wash pattern passed by URL. Check for sanity of the wildcard by
removing wildcards if they are appended to extremely short words
(1-3 letters). TODO: instead of this approximative treatment, it
will be much better to introduce a temporal limit, e.g. to kill a
query if it does not finish in 10 seconds."""
# strip accents:
# p = strip_accents(p) # FIXME: when available, strip accents all the time
# add leading/trailing whitespace for the two following wildcard-sanity checking regexps:
p = " " + p + " "
# replace spaces within quotes by __SPACE__ temporarily:
p = re_pattern_single_quotes.sub(lambda x: "'"+x.group(1).replace(' ', '__SPACE__')+"'", p)
p = re_pattern_double_quotes.sub(lambda x: "\""+x.group(1).replace(' ', '__SPACE__')+"\"", p)
p = re_pattern_regexp_quotes.sub(lambda x: "/"+x.group(1).replace(' ', '__SPACE__')+"/", p)
# get rid of unquoted wildcards after spaces:
p = re_pattern_wildcards_after_spaces.sub("\\1", p)
# get rid of extremely short words (1-3 letters with wildcards):
#p = re_pattern_short_words.sub("\\1", p)
# replace back __SPACE__ by spaces:
p = re_pattern_space.sub(" ", p)
# replace special terms:
p = re_pattern_today.sub(time.strftime("%Y-%m-%d", time.localtime()), p)
# remove unnecessary whitespace:
p = p.strip()
# remove potentially wrong UTF-8 characters:
p = wash_for_utf8(p)
return p
def wash_field(f):
"""Wash field passed by URL."""
if f:
# get rid of unnecessary whitespace and make it lowercase
# (e.g. Author -> author) to better suit iPhone etc input
# mode:
f = f.strip().lower()
# wash legacy 'f' field names, e.g. replace 'wau' or `au' by
# 'author', if applicable:
if CFG_WEBSEARCH_FIELDS_CONVERT:
f = CFG_WEBSEARCH_FIELDS_CONVERT.get(f, f)
return f
def wash_dates(d1="", d1y=0, d1m=0, d1d=0, d2="", d2y=0, d2m=0, d2d=0):
"""
Take user-submitted date arguments D1 (full datetime string) or
(D1Y, D1M, D1Y) year, month, day tuple and D2 or (D2Y, D2M, D2Y)
and return (YYY1-M1-D2 H1:M1:S2, YYY2-M2-D2 H2:M2:S2) datetime
strings in the YYYY-MM-DD HH:MM:SS format suitable for time
restricted searching.
Note that when both D1 and (D1Y, D1M, D1D) parameters are present,
the precedence goes to D1. Ditto for D2*.
Note that when (D1Y, D1M, D1D) are taken into account, some values
may be missing and are completed e.g. to 01 or 12 according to
whether it is the starting or the ending date.
"""
datetext1, datetext2 = "", ""
# sanity checking:
if d1 == "" and d1y == 0 and d1m == 0 and d1d == 0 and d2 == "" and d2y == 0 and d2m == 0 and d2d == 0:
return ("", "") # nothing selected, so return empty values
# wash first (starting) date:
if d1:
# full datetime string takes precedence:
datetext1 = d1
else:
# okay, first date passed as (year,month,day):
if d1y:
datetext1 += "%04d" % d1y
else:
datetext1 += "0000"
if d1m:
datetext1 += "-%02d" % d1m
else:
datetext1 += "-01"
if d1d:
datetext1 += "-%02d" % d1d
else:
datetext1 += "-01"
datetext1 += " 00:00:00"
# wash second (ending) date:
if d2:
# full datetime string takes precedence:
datetext2 = d2
else:
# okay, second date passed as (year,month,day):
if d2y:
datetext2 += "%04d" % d2y
else:
datetext2 += "9999"
if d2m:
datetext2 += "-%02d" % d2m
else:
datetext2 += "-12"
if d2d:
datetext2 += "-%02d" % d2d
else:
datetext2 += "-31" # NOTE: perhaps we should add max(datenumber) in
# given month, but for our quering it's not
# needed, 31 will always do
datetext2 += " 00:00:00"
# okay, return constructed YYYY-MM-DD HH:MM:SS datetexts:
return (datetext1, datetext2)
def is_hosted_collection(coll):
"""Check if the given collection is a hosted one; i.e. its dbquery starts with hostedcollection:
Returns True if it is, False if it's not or if the result is empty or if the query failed"""
res = run_sql("SELECT dbquery FROM collection WHERE name=%s", (coll, ))
if not res[0][0]:
return False
try:
return res[0][0].startswith("hostedcollection:")
except IndexError:
return False
def get_colID(c):
"Return collection ID for collection name C. Return None if no match found."
colID = None
res = run_sql("SELECT id FROM collection WHERE name=%s", (c,), 1)
if res:
colID = res[0][0]
return colID
def get_coll_normalised_name(c):
"""Returns normalised collection name (case sensitive) for collection name
C (case insensitive).
Returns None if no match found."""
res = run_sql("SELECT name FROM collection WHERE name=%s", (c,))
if res:
return res[0][0]
else:
return None
def get_coll_ancestors(coll):
"Returns a list of ancestors for collection 'coll'."
coll_ancestors = []
coll_ancestor = coll
while 1:
res = run_sql("""SELECT c.name FROM collection AS c
LEFT JOIN collection_collection AS cc ON c.id=cc.id_dad
LEFT JOIN collection AS ccc ON ccc.id=cc.id_son
WHERE ccc.name=%s ORDER BY cc.id_dad ASC LIMIT 1""",
(coll_ancestor,))
if res:
coll_name = res[0][0]
coll_ancestors.append(coll_name)
coll_ancestor = coll_name
else:
break
# ancestors found, return reversed list:
coll_ancestors.reverse()
return coll_ancestors
def get_coll_sons(coll, coll_type='r', public_only=1):
"""Return a list of sons (first-level descendants) of type 'coll_type' for collection 'coll'.
If public_only, then return only non-restricted son collections.
"""
coll_sons = []
query = "SELECT c.name FROM collection AS c "\
"LEFT JOIN collection_collection AS cc ON c.id=cc.id_son "\
"LEFT JOIN collection AS ccc ON ccc.id=cc.id_dad "\
"WHERE cc.type=%s AND ccc.name=%s"
query += " ORDER BY cc.score DESC"
res = run_sql(query, (coll_type, coll))
for name in res:
if not public_only or not collection_restricted_p(name[0]):
coll_sons.append(name[0])
return coll_sons
class CollectionAllChildrenDataCacher(DataCacher):
"""Cache for all children of a collection (regular & virtual, public & private)"""
def __init__(self):
def cache_filler():
def get_all_children(coll, coll_type='r', public_only=1):
"""Return a list of all children of type 'type' for collection 'coll'.
If public_only, then return only non-restricted child collections.
If type='*', then return both regular and virtual collections.
"""
children = []
if coll_type == '*':
sons = get_coll_sons(coll, 'r', public_only) + get_coll_sons(coll, 'v', public_only)
else:
sons = get_coll_sons(coll, coll_type, public_only)
for child in sons:
children.append(child)
children.extend(get_all_children(child, coll_type, public_only))
return children
ret = {}
collections = collection_reclist_cache.cache.keys()
for collection in collections:
ret[collection] = get_all_children(collection, '*', public_only=0)
return ret
def timestamp_verifier():
return max(get_table_update_time('collection'), get_table_update_time('collection_collection'))
DataCacher.__init__(self, cache_filler, timestamp_verifier)
try:
if not collection_allchildren_cache.is_ok_p:
raise Exception
except Exception:
collection_allchildren_cache = CollectionAllChildrenDataCacher()
def get_collection_allchildren(coll, recreate_cache_if_needed=True):
"""Returns the list of all children of a collection."""
if recreate_cache_if_needed:
collection_allchildren_cache.recreate_cache_if_needed()
if coll not in collection_allchildren_cache.cache:
return [] # collection does not exist; return empty list
return collection_allchildren_cache.cache[coll]
def get_coll_real_descendants(coll, coll_type='_', get_hosted_colls=True):
"""Return a list of all descendants of collection 'coll' that are defined by a 'dbquery'.
IOW, we need to decompose compound collections like "A & B" into "A" and "B" provided
that "A & B" has no associated database query defined.
"""
coll_sons = []
res = run_sql("""SELECT c.name,c.dbquery FROM collection AS c
LEFT JOIN collection_collection AS cc ON c.id=cc.id_son
LEFT JOIN collection AS ccc ON ccc.id=cc.id_dad
WHERE ccc.name=%s AND cc.type LIKE %s ORDER BY cc.score DESC""",
(coll, coll_type,))
for name, dbquery in res:
if dbquery: # this is 'real' collection, so return it:
if get_hosted_colls:
coll_sons.append(name)
else:
if not dbquery.startswith("hostedcollection:"):
coll_sons.append(name)
else: # this is 'composed' collection, so recurse:
coll_sons.extend(get_coll_real_descendants(name))
return coll_sons
def browse_pattern_phrases(req, colls, p, f, rg, ln=CFG_SITE_LANG):
"""Returns either biliographic phrases or words indexes."""
## is p enclosed in quotes? (coming from exact search)
if p.startswith('"') and p.endswith('"'):
p = p[1:-1]
## okay, "real browse" follows:
## FIXME: the maths in the get_nearest_terms_in_bibxxx is just a test
if not f and p.find(":") > 0: # does 'p' contain ':'?
f, p = p.split(":", 1)
## do we search in words indexes?
# FIXME uncomment this
#if not f:
# return browse_in_bibwords(req, p, f)
coll_hitset = intbitset()
for coll_name in colls:
coll_hitset |= get_collection_reclist(coll_name)
index_id = get_index_id_from_field(f)
if index_id != 0:
browsed_phrases_in_colls = get_nearest_terms_in_idxphrase_with_collection(p, index_id, rg/2, rg/2, coll_hitset)
else:
browsed_phrases = get_nearest_terms_in_bibxxx(p, f, (rg+1)/2+1, (rg-1)/2+1)
while not browsed_phrases:
# try again and again with shorter and shorter pattern:
try:
p = p[:-1]
browsed_phrases = get_nearest_terms_in_bibxxx(p, f, (rg+1)/2+1, (rg-1)/2+1)
except:
register_exception(req=req, alert_admin=True)
# probably there are no hits at all:
#req.write(_("No values found."))
return []
## try to check hits in these particular collection selection:
browsed_phrases_in_colls = []
if 0:
for phrase in browsed_phrases:
phrase_hitset = intbitset()
phrase_hitsets = search_pattern("", phrase, f, 'e')
for coll in colls:
phrase_hitset.union_update(phrase_hitsets[coll])
if len(phrase_hitset) > 0:
# okay, this phrase has some hits in colls, so add it:
browsed_phrases_in_colls.append([phrase, len(phrase_hitset)])
## were there hits in collections?
if browsed_phrases_in_colls == []:
if browsed_phrases != []:
#write_warning(req, """<p>No match close to <em>%s</em> found in given collections.
#Please try different term.<p>Displaying matches in any collection...""" % p_orig)
## try to get nbhits for these phrases in any collection:
for phrase in browsed_phrases:
nbhits = get_nbhits_in_bibxxx(phrase, f, coll_hitset)
if nbhits > 0:
browsed_phrases_in_colls.append([phrase, nbhits])
return browsed_phrases_in_colls
def browse_pattern(req, colls, p, f, rg, ln=CFG_SITE_LANG):
"""Displays either biliographic phrases or words indexes."""
# load the right message language
_ = gettext_set_language(ln)
browsed_phrases_in_colls = browse_pattern_phrases(req, colls, p, f, rg, ln)
if len(browsed_phrases_in_colls) == 0:
req.write(_("No values found."))
return
## display results now:
out = websearch_templates.tmpl_browse_pattern(
f=f,
fn=get_field_i18nname(get_field_name(f) or f, ln, False),
ln=ln,
browsed_phrases_in_colls=browsed_phrases_in_colls,
colls=colls,
rg=rg,
)
req.write(out)
return
def browse_in_bibwords(req, p, f, ln=CFG_SITE_LANG):
"""Browse inside words indexes."""
if not p:
return
_ = gettext_set_language(ln)
urlargd = {}
urlargd.update(req.argd)
urlargd['action'] = 'search'
nearest_box = create_nearest_terms_box(urlargd, p, f, 'w', ln=ln, intro_text_p=0)
req.write(websearch_templates.tmpl_search_in_bibwords(
p = p,
f = f,
ln = ln,
nearest_box = nearest_box
))
return
def search_pattern(req=None, p=None, f=None, m=None, ap=0, of="id", verbose=0, ln=CFG_SITE_LANG, display_nearest_terms_box=True, wl=0):
"""Search for complex pattern 'p' within field 'f' according to
matching type 'm'. Return hitset of recIDs.
The function uses multi-stage searching algorithm in case of no
exact match found. See the Search Internals document for
detailed description.
The 'ap' argument governs whether an alternative patterns are to
be used in case there is no direct hit for (p,f,m). For
example, whether to replace non-alphanumeric characters by
spaces if it would give some hits. See the Search Internals
document for detailed description. (ap=0 forbits the
alternative pattern usage, ap=1 permits it.)
'ap' is also internally used for allowing hidden tag search
(for requests coming from webcoll, for example). In this
case ap=-9
The 'of' argument governs whether to print or not some
information to the user in case of no match found. (Usually it
prints the information in case of HTML formats, otherwise it's
silent).
The 'verbose' argument controls the level of debugging information
to be printed (0=least, 9=most).
All the parameters are assumed to have been previously washed.
This function is suitable as a mid-level API.
"""
_ = gettext_set_language(ln)
hitset_empty = intbitset()
# sanity check:
if not p:
hitset_full = intbitset(trailing_bits=1)
hitset_full.discard(0)
# no pattern, so return all universe
return hitset_full
# search stage 1: break up arguments into basic search units:
if verbose and of.startswith("h"):
t1 = os.times()[4]
basic_search_units = create_basic_search_units(req, p, f, m, of)
if verbose and of.startswith("h"):
t2 = os.times()[4]
write_warning("Search stage 1: basic search units are: %s" % cgi.escape(repr(basic_search_units)), req=req)
write_warning("Search stage 1: execution took %.2f seconds." % (t2 - t1), req=req)
# search stage 2: do search for each search unit and verify hit presence:
if verbose and of.startswith("h"):
t1 = os.times()[4]
basic_search_units_hitsets = []
#prepare hiddenfield-related..
myhiddens = cfg['CFG_BIBFORMAT_HIDDEN_TAGS']
can_see_hidden = False
if req:
user_info = collect_user_info(req)
can_see_hidden = user_info.get('precached_canseehiddenmarctags', False)
if not req and ap == -9: # special request, coming from webcoll
can_see_hidden = True
if can_see_hidden:
myhiddens = []
if CFG_INSPIRE_SITE and of.startswith('h'):
# fulltext/caption search warnings for INSPIRE:
fields_to_be_searched = [f for dummy_o, p, f, m in basic_search_units]
if 'fulltext' in fields_to_be_searched:
write_warning(_("Full-text search is currently available for all arXiv papers, many theses, a few report series and some journal articles"), req=req)
elif 'caption' in fields_to_be_searched:
write_warning(_("Warning: figure caption search is only available for a subset of papers mostly from %(x_range_from_year)s-%(x_range_to_year)s.") %
{'x_range_from_year': '2008',
'x_range_to_year': '2012'}, req=req)
for idx_unit in xrange(len(basic_search_units)):
bsu_o, bsu_p, bsu_f, bsu_m = basic_search_units[idx_unit]
if bsu_f and len(bsu_f) < 2:
if of.startswith("h"):
write_warning(_("There is no index %(x_name)s. Searching for %(x_text)s in all fields.", x_name=bsu_f, x_text=bsu_p), req=req)
bsu_f = ''
bsu_m = 'w'
if of.startswith("h") and verbose:
write_warning(_('Instead searching %(x_name)s.', x_name=str([bsu_o, bsu_p, bsu_f, bsu_m])), req=req)
try:
basic_search_unit_hitset = search_unit(bsu_p, bsu_f, bsu_m, wl)
except InvenioWebSearchWildcardLimitError as excp:
basic_search_unit_hitset = excp.res
if of.startswith("h"):
write_warning(_("Search term too generic, displaying only partial results..."), req=req)
# FIXME: print warning if we use native full-text indexing
if bsu_f == 'fulltext' and bsu_m != 'w' and of.startswith('h') and not CFG_SOLR_URL:
write_warning(_("No phrase index available for fulltext yet, looking for word combination..."), req=req)
#check that the user is allowed to search with this tag
#if he/she tries it
if bsu_f and len(bsu_f) > 1 and bsu_f[0].isdigit() and bsu_f[1].isdigit():
for htag in myhiddens:
ltag = len(htag)
samelenfield = bsu_f[0:ltag]
if samelenfield == htag: #user searches by a hidden tag
#we won't show you anything..
basic_search_unit_hitset = intbitset()
if verbose >= 9 and of.startswith("h"):
write_warning("Pattern %s hitlist omitted since \
it queries in a hidden tag %s" %
(cgi.escape(repr(bsu_p)), repr(myhiddens)), req=req)
display_nearest_terms_box = False #..and stop spying, too.
if verbose >= 9 and of.startswith("h"):
write_warning("Search stage 1: pattern %s gave hitlist %s" % (cgi.escape(bsu_p), basic_search_unit_hitset), req=req)
if len(basic_search_unit_hitset) > 0 or \
ap<1 or \
bsu_o in ("|", "-") or \
((idx_unit+1)<len(basic_search_units) and basic_search_units[idx_unit+1][0]=="|"):
# stage 2-1: this basic search unit is retained, since
# either the hitset is non-empty, or the approximate
# pattern treatment is switched off, or the search unit
# was joined by an OR operator to preceding/following
# units so we do not require that it exists
basic_search_units_hitsets.append(basic_search_unit_hitset)
else:
# stage 2-2: no hits found for this search unit, try to replace non-alphanumeric chars inside pattern:
if re.search(r'[^a-zA-Z0-9\s\:]', bsu_p) and bsu_f != 'refersto' and bsu_f != 'citedby':
if bsu_p.startswith('"') and bsu_p.endswith('"'): # is it ACC query?
bsu_pn = re.sub(r'[^a-zA-Z0-9\s\:]+', "*", bsu_p)
else: # it is WRD query
bsu_pn = re.sub(r'[^a-zA-Z0-9\s\:]+', " ", bsu_p)
if verbose and of.startswith('h') and req:
write_warning("Trying (%s,%s,%s)" % (cgi.escape(bsu_pn), cgi.escape(bsu_f), cgi.escape(bsu_m)), req=req)
basic_search_unit_hitset = search_pattern(req=None, p=bsu_pn, f=bsu_f, m=bsu_m, of="id", ln=ln, wl=wl)
if len(basic_search_unit_hitset) > 0:
# we retain the new unit instead
if of.startswith('h'):
write_warning(_("No exact match found for %(x_query1)s, using %(x_query2)s instead...") %
{'x_query1': "<em>" + cgi.escape(bsu_p) + "</em>",
'x_query2': "<em>" + cgi.escape(bsu_pn) + "</em>"}, req=req)
basic_search_units[idx_unit][1] = bsu_pn
basic_search_units_hitsets.append(basic_search_unit_hitset)
else:
# stage 2-3: no hits found either, propose nearest indexed terms:
if of.startswith('h') and display_nearest_terms_box:
if req:
if bsu_f == "recid":
write_warning(_("Requested record does not seem to exist."), req=req)
else:
write_warning(create_nearest_terms_box(req.argd, bsu_p, bsu_f, bsu_m, ln=ln), req=req)
return hitset_empty
else:
# stage 2-3: no hits found either, propose nearest indexed terms:
if of.startswith('h') and display_nearest_terms_box:
if req:
if bsu_f == "recid":
write_warning(_("Requested record does not seem to exist."), req=req)
else:
write_warning(create_nearest_terms_box(req.argd, bsu_p, bsu_f, bsu_m, ln=ln), req=req)
return hitset_empty
if verbose and of.startswith("h"):
t2 = os.times()[4]
for idx_unit in range(0, len(basic_search_units)):
write_warning("Search stage 2: basic search unit %s gave %d hits." %
(basic_search_units[idx_unit][1:], len(basic_search_units_hitsets[idx_unit])), req=req)
write_warning("Search stage 2: execution took %.2f seconds." % (t2 - t1), req=req)
# search stage 3: apply boolean query for each search unit:
if verbose and of.startswith("h"):
t1 = os.times()[4]
# let the initial set be the complete universe:
hitset_in_any_collection = intbitset(trailing_bits=1)
hitset_in_any_collection.discard(0)
for idx_unit in xrange(len(basic_search_units)):
this_unit_operation = basic_search_units[idx_unit][0]
this_unit_hitset = basic_search_units_hitsets[idx_unit]
if this_unit_operation == '+':
hitset_in_any_collection.intersection_update(this_unit_hitset)
elif this_unit_operation == '-':
hitset_in_any_collection.difference_update(this_unit_hitset)
elif this_unit_operation == '|':
hitset_in_any_collection.union_update(this_unit_hitset)
else:
if of.startswith("h"):
write_warning("Invalid set operation %s." % cgi.escape(this_unit_operation), "Error", req=req)
if len(hitset_in_any_collection) == 0:
# no hits found, propose alternative boolean query:
if of.startswith('h') and display_nearest_terms_box:
nearestterms = []
for idx_unit in range(0, len(basic_search_units)):
bsu_o, bsu_p, bsu_f, bsu_m = basic_search_units[idx_unit]
if bsu_p.startswith("%") and bsu_p.endswith("%"):
bsu_p = "'" + bsu_p[1:-1] + "'"
bsu_nbhits = len(basic_search_units_hitsets[idx_unit])
# create a similar query, but with the basic search unit only
argd = {}
argd.update(req.argd)
argd['p'] = bsu_p
argd['f'] = bsu_f
nearestterms.append((bsu_p, bsu_nbhits, argd))
text = websearch_templates.tmpl_search_no_boolean_hits(
ln=ln, nearestterms=nearestterms)
write_warning(text, req=req)
if verbose and of.startswith("h"):
t2 = os.times()[4]
write_warning("Search stage 3: boolean query gave %d hits." % len(hitset_in_any_collection), req=req)
write_warning("Search stage 3: execution took %.2f seconds." % (t2 - t1), req=req)
return hitset_in_any_collection
def search_pattern_parenthesised(req=None, p=None, f=None, m=None, ap=0, of="id", verbose=0, ln=CFG_SITE_LANG, display_nearest_terms_box=True, wl=0):
"""Search for complex pattern 'p' containing parenthesis within field 'f' according to
matching type 'm'. Return hitset of recIDs.
For more details on the parameters see 'search_pattern'
"""
_ = gettext_set_language(ln)
spires_syntax_converter = SpiresToInvenioSyntaxConverter()
spires_syntax_query = False
# if the pattern uses SPIRES search syntax, convert it to Invenio syntax
if spires_syntax_converter.is_applicable(p):
spires_syntax_query = True
p = spires_syntax_converter.convert_query(p)
# sanity check: do not call parenthesised parser for search terms
# like U(1) but still call it for searches like ('U(1)' | 'U(2)'):
if not re_pattern_parens.search(re_pattern_parens_quotes.sub('_', p)):
return search_pattern(req, p, f, m, ap, of, verbose, ln, display_nearest_terms_box=display_nearest_terms_box, wl=wl)
# Try searching with parentheses
try:
parser = SearchQueryParenthesisedParser()
# get a hitset with all recids
result_hitset = intbitset(trailing_bits=1)
# parse the query. The result is list of [op1, expr1, op2, expr2, ..., opN, exprN]
parsing_result = parser.parse_query(p)
if verbose and of.startswith("h"):
write_warning("Search stage 1: search_pattern_parenthesised() searched %s." % repr(p), req=req)
write_warning("Search stage 1: search_pattern_parenthesised() returned %s." % repr(parsing_result), req=req)
# go through every pattern
# calculate hitset for it
# combine pattern's hitset with the result using the corresponding operator
for index in xrange(0, len(parsing_result)-1, 2):
current_operator = parsing_result[index]
current_pattern = parsing_result[index+1]
if CFG_INSPIRE_SITE and spires_syntax_query:
# setting ap=0 to turn off approximate matching for 0 results.
# Doesn't work well in combinations.
# FIXME: The right fix involves collecting statuses for each
# hitset, then showing a nearest terms box exactly once,
# outside this loop.
ap = 0
display_nearest_terms_box = False
# obtain a hitset for the current pattern
current_hitset = search_pattern(req, current_pattern, f, m, ap, of, verbose, ln, display_nearest_terms_box=display_nearest_terms_box, wl=wl)
# combine the current hitset with resulting hitset using the current operator
if current_operator == '+':
result_hitset = result_hitset & current_hitset
elif current_operator == '-':
result_hitset = result_hitset - current_hitset
elif current_operator == '|':
result_hitset = result_hitset | current_hitset
else:
assert False, "Unknown operator in search_pattern_parenthesised()"
return result_hitset
# If searching with parenteses fails, perform search ignoring parentheses
except SyntaxError:
write_warning(_("Search syntax misunderstood. Ignoring all parentheses in the query. If this doesn't help, please check your search and try again."), req=req)
# remove the parentheses in the query. Current implementation removes all the parentheses,
# but it could be improved to romove only these that are not inside quotes
p = p.replace('(', ' ')
p = p.replace(')', ' ')
return search_pattern(req, p, f, m, ap, of, verbose, ln, display_nearest_terms_box=display_nearest_terms_box, wl=wl)
def search_unit(p, f=None, m=None, wl=0, ignore_synonyms=None):
"""Search for basic search unit defined by pattern 'p' and field
'f' and matching type 'm'. Return hitset of recIDs.
All the parameters are assumed to have been previously washed.
'p' is assumed to be already a ``basic search unit'' so that it
is searched as such and is not broken up in any way. Only
wildcard and span queries are being detected inside 'p'.
If CFG_WEBSEARCH_SYNONYM_KBRS is set and we are searching in
one of the indexes that has defined runtime synonym knowledge
base, then look up there and automatically enrich search
results with results for synonyms.
In case the wildcard limit (wl) is greater than 0 and this limit
is reached an InvenioWebSearchWildcardLimitError will be raised.
In case you want to call this function with no limit for the
wildcard queries, wl should be 0.
Parameter 'ignore_synonyms' is a list of terms for which we
should not try to further find a synonym.
This function is suitable as a low-level API.
"""
## create empty output results set:
hitset = intbitset()
if not p: # sanity checking
return hitset
tokenizer = get_field_tokenizer_type(f)
hitset_cjk = intbitset()
if tokenizer == "BibIndexCJKTokenizer":
if is_there_any_CJK_character_in_text(p):
cjk_tok = BibIndexCJKTokenizer()
chars = cjk_tok.tokenize_for_words(p)
for char in chars:
hitset_cjk |= search_unit_in_bibwords(char, f, wl)
## eventually look up runtime synonyms:
hitset_synonyms = intbitset()
if f in CFG_WEBSEARCH_SYNONYM_KBRS:
if ignore_synonyms is None:
ignore_synonyms = []
ignore_synonyms.append(p)
for p_synonym in get_synonym_terms(p,
CFG_WEBSEARCH_SYNONYM_KBRS[f][0],
CFG_WEBSEARCH_SYNONYM_KBRS[f][1]):
if p_synonym != p and \
not p_synonym in ignore_synonyms:
hitset_synonyms |= search_unit(p_synonym, f, m, wl,
ignore_synonyms)
## look up hits:
if f == 'fulltext' and get_idx_indexer('fulltext') == 'SOLR' and CFG_SOLR_URL:
# redirect to Solr
try:
return search_unit_in_solr(p, f, m)
except:
# There were troubles with getting full-text search
# results from Solr. Let us alert the admin of these
# problems and let us simply return empty results to the
# end user.
register_exception()
return hitset
elif f == 'fulltext' and get_idx_indexer('fulltext') == 'XAPIAN' and CFG_XAPIAN_ENABLED:
# redirect to Xapian
try:
return search_unit_in_xapian(p, f, m)
except:
# There were troubles with getting full-text search
# results from Xapian. Let us alert the admin of these
# problems and let us simply return empty results to the
# end user.
register_exception()
return hitset
if f == 'datecreated':
hitset = search_unit_in_bibrec(p, p, 'c')
elif f == 'datemodified':
hitset = search_unit_in_bibrec(p, p, 'm')
elif f == 'refersto':
# we are doing search by the citation count
hitset = search_unit_refersto(p)
elif f == 'referstoexcludingselfcites':
# we are doing search by the citation count
hitset = search_unit_refersto_excluding_selfcites(p)
elif f == 'cataloguer':
# we are doing search by the cataloguer nickname
hitset = search_unit_in_record_history(p)
elif f == 'rawref':
from invenio.legacy.refextract.api import search_from_reference
field, pattern = search_from_reference(p)
return search_unit(pattern, field)
elif f == 'citedby':
# we are doing search by the citation count
hitset = search_unit_citedby(p)
elif f == 'collection':
# we are doing search by the collection name or MARC field
hitset = search_unit_collection(p, m, wl=wl)
elif f == 'tag':
module_found = False
try:
from invenio.modules.tags.search_units import search_unit_in_tags
module_found = True
except:
# WebTag module is disabled, so ignore 'tag' selector
pass
if module_found:
return search_unit_in_tags(p)
elif f == 'citedbyexcludingselfcites':
# we are doing search by the citation count
hitset = search_unit_citedby_excluding_selfcites(p)
elif m == 'a' or m == 'r' or f == 'subject':
# we are doing either phrase search or regexp search
if f == 'fulltext':
# FIXME: workaround for not having phrase index yet
return search_pattern(None, p, f, 'w')
index_id = get_index_id_from_field(f)
if index_id != 0:
if m == 'a' and index_id in get_idxpair_field_ids():
#for exact match on the admin configured fields we are searching in the pair tables
hitset = search_unit_in_idxpairs(p, f, m, wl)
else:
hitset = search_unit_in_idxphrases(p, f, m, wl)
else:
hitset = search_unit_in_bibxxx(p, f, m, wl)
# if not hitset and m == 'a' and (p[0] != '%' and p[-1] != '%'):
# #if we have no results by doing exact matching, do partial matching
# #for removing the distinction between simple and double quotes
# hitset = search_unit_in_bibxxx('%' + p + '%', f, m, wl)
elif p.startswith("cited:"):
# we are doing search by the citation count
hitset = search_unit_by_times_cited(p[6:])
elif p.startswith("citedexcludingselfcites:"):
# we are doing search by the citation count
hitset = search_unit_by_times_cited(p[6:], exclude_selfcites=True)
else:
# we are doing bibwords search by default
hitset = search_unit_in_bibwords(p, f, wl=wl)
## merge synonym results and return total:
hitset |= hitset_synonyms
hitset |= hitset_cjk
return hitset
def get_idxpair_field_ids():
"""Returns the list of ids for the fields that idxPAIRS should be used on"""
index_dict = dict(run_sql("SELECT name, id FROM idxINDEX"))
return [index_dict[field] for field in index_dict if field in cfg['CFG_WEBSEARCH_IDXPAIRS_FIELDS']]
def search_unit_in_bibwords(word, f, decompress=zlib.decompress, wl=0):
"""Searches for 'word' inside bibwordsX table for field 'f' and returns hitset of recIDs."""
hitset = intbitset() # will hold output result set
set_used = 0 # not-yet-used flag, to be able to circumvent set operations
limit_reached = 0 # flag for knowing if the query limit has been reached
# if no field is specified, search in the global index.
f = f or 'anyfield'
index_id = get_index_id_from_field(f)
if index_id:
bibwordsX = "idxWORD%02dF" % index_id
stemming_language = get_index_stemming_language(index_id)
else:
return intbitset() # word index f does not exist
# wash 'word' argument and run query:
if f.endswith('count') and word.endswith('+'):
# field count query of the form N+ so transform N+ to N->99999:
word = word[:-1] + '->99999'
word = word.replace('*', '%') # we now use '*' as the truncation character
words = word.split("->", 1) # check for span query
if len(words) == 2:
word0 = re_word.sub('', words[0])
word1 = re_word.sub('', words[1])
if stemming_language:
word0 = lower_index_term(word0)
word1 = lower_index_term(word1)
word0 = stem(word0, stemming_language)
word1 = stem(word1, stemming_language)
word0_washed = wash_index_term(word0)
word1_washed = wash_index_term(word1)
if f.endswith('count'):
# field count query; convert to integers in order
# to have numerical behaviour for 'BETWEEN n1 AND n2' query
try:
word0_washed = int(word0_washed)
word1_washed = int(word1_washed)
except ValueError:
pass
try:
res = run_sql_with_limit("SELECT term,hitlist FROM %s WHERE term BETWEEN %%s AND %%s" % bibwordsX,
(word0_washed, word1_washed), wildcard_limit=wl)
except InvenioDbQueryWildcardLimitError as excp:
res = excp.res
limit_reached = 1 # set the limit reached flag to true
else:
if f == 'journal':
pass # FIXME: quick hack for the journal index
else:
word = re_word.sub('', word)
if stemming_language:
word = lower_index_term(word)
word = stem(word, stemming_language)
if word.find('%') >= 0: # do we have wildcard in the word?
if f == 'journal':
# FIXME: quick hack for the journal index
# FIXME: we can run a sanity check here for all indexes
res = ()
else:
try:
res = run_sql_with_limit("SELECT term,hitlist FROM %s WHERE term LIKE %%s" % bibwordsX,
(wash_index_term(word),), wildcard_limit = wl)
except InvenioDbQueryWildcardLimitError as excp:
res = excp.res
limit_reached = 1 # set the limit reached flag to true
else:
res = run_sql("SELECT term,hitlist FROM %s WHERE term=%%s" % bibwordsX,
(wash_index_term(word),))
# fill the result set:
for word, hitlist in res:
hitset_bibwrd = intbitset(hitlist)
# add the results:
if set_used:
hitset.union_update(hitset_bibwrd)
else:
hitset = hitset_bibwrd
set_used = 1
#check to see if the query limit was reached
if limit_reached:
#raise an exception, so we can print a nice message to the user
raise InvenioWebSearchWildcardLimitError(hitset)
# okay, return result set:
return hitset
def search_unit_in_idxpairs(p, f, search_type, wl=0):
"""Searches for pair 'p' inside idxPAIR table for field 'f' and
returns hitset of recIDs found."""
limit_reached = 0 # flag for knowing if the query limit has been reached
do_exact_search = True # flag to know when it makes sense to try to do exact matching
result_set = intbitset()
#determine the idxPAIR table to read from
index_id = get_index_id_from_field(f)
if not index_id:
return intbitset()
stemming_language = get_index_stemming_language(index_id)
pairs_tokenizer = BibIndexDefaultTokenizer(stemming_language)
idxpair_table_washed = wash_table_column_name("idxPAIR%02dF" % index_id)
if p.startswith("%") and p.endswith("%"):
p = p[1:-1]
original_pattern = p
p = string.replace(p, '*', '%') # we now use '*' as the truncation character
queries_releated_vars = [] # contains tuples of (query_addons, query_params, use_query_limit)
#is it a span query?
ps = p.split("->", 1)
if len(ps) == 2 and not (ps[0].endswith(' ') or ps[1].startswith(' ')):
#so we are dealing with a span query
pairs_left = pairs_tokenizer.tokenize_for_pairs(ps[0])
pairs_right = pairs_tokenizer.tokenize_for_pairs(ps[1])
if not pairs_left or not pairs_right:
# we are not actually dealing with pairs but with words
return search_unit_in_bibwords(original_pattern, f, wl=wl)
elif len(pairs_left) != len(pairs_right):
# it is kind of hard to know what the user actually wanted
# we have to do: foo bar baz -> qux xyz, so let's swith to phrase
return search_unit_in_idxphrases(original_pattern, f, search_type, wl)
elif len(pairs_left) > 1 and \
len(pairs_right) > 1 and \
pairs_left[:-1] != pairs_right[:-1]:
# again we have something like: foo bar baz -> abc xyz qux
# so we'd better switch to phrase
return search_unit_in_idxphrases(original_pattern, f, search_type, wl)
else:
# finally, we can treat the search using idxPairs
# at this step we have either: foo bar -> abc xyz
# or foo bar abc -> foo bar xyz
queries_releated_vars = [("BETWEEN %s AND %s", (pairs_left[-1], pairs_right[-1]), True)]
for pair in pairs_left[:-1]:# which should be equal with pairs_right[:-1]
queries_releated_vars.append(("= %s", (pair, ), False))
do_exact_search = False # no exact search for span queries
elif p.find('%') > -1:
#tokenizing p will remove the '%', so we have to make sure it stays
replacement = 'xxxxxxxxxx' #hopefuly this will not clash with anything in the future
p = string.replace(p, '%', replacement)
pairs = pairs_tokenizer.tokenize_for_pairs(p)
if not pairs:
# we are not actually dealing with pairs but with words
return search_unit_in_bibwords(original_pattern, f, wl=wl)
queries_releated_vars = []
for pair in pairs:
if string.find(pair, replacement) > -1:
pair = string.replace(pair, replacement, '%') #we replace back the % sign
queries_releated_vars.append(("LIKE %s", (pair, ), True))
else:
queries_releated_vars.append(("= %s", (pair, ), False))
do_exact_search = False
else:
#normal query
pairs = pairs_tokenizer.tokenize_for_pairs(p)
if not pairs:
# we are not actually dealing with pairs but with words
return search_unit_in_bibwords(original_pattern, f, wl=wl)
queries_releated_vars = []
for pair in pairs:
queries_releated_vars.append(("= %s", (pair, ), False))
first_results = 1 # flag to know if it's the first set of results or not
for query_var in queries_releated_vars:
query_addons = query_var[0]
query_params = query_var[1]
use_query_limit = query_var[2]
if use_query_limit:
try:
res = run_sql_with_limit("SELECT term, hitlist FROM %s WHERE term %s"
% (idxpair_table_washed, query_addons), query_params, wildcard_limit=wl) #kwalitee:disable=sql
except InvenioDbQueryWildcardLimitError as excp:
res = excp.res
limit_reached = 1 # set the limit reached flag to true
else:
res = run_sql("SELECT term, hitlist FROM %s WHERE term %s"
% (idxpair_table_washed, query_addons), query_params) #kwalitee:disable=sql
if not res:
return intbitset()
for pair, hitlist in res:
hitset_idxpairs = intbitset(hitlist)
if first_results:
result_set = hitset_idxpairs
first_results = 0
else:
result_set.intersection_update(hitset_idxpairs)
#check to see if the query limit was reached
if limit_reached:
#raise an exception, so we can print a nice message to the user
raise InvenioWebSearchWildcardLimitError(result_set)
# check if we need to eliminate the false positives
if cfg['CFG_WEBSEARCH_IDXPAIRS_EXACT_SEARCH'] and do_exact_search:
# we need to eliminate the false positives
idxphrase_table_washed = wash_table_column_name("idxPHRASE%02dR" % index_id)
not_exact_search = intbitset()
for recid in result_set:
res = run_sql("SELECT termlist FROM %s WHERE id_bibrec %s" %(idxphrase_table_washed, '=%s'), (recid, )) #kwalitee:disable=sql
if res:
termlist = deserialize_via_marshal(res[0][0])
if not [term for term in termlist if term.lower().find(p.lower()) > -1]:
not_exact_search.add(recid)
else:
not_exact_search.add(recid)
# remove the recs that are false positives from the final result
result_set.difference_update(not_exact_search)
return result_set
def search_unit_in_idxphrases(p, f, search_type, wl=0):
"""Searches for phrase 'p' inside idxPHRASE*F table for field 'f' and returns hitset of recIDs found.
The search type is defined by 'type' (e.g. equals to 'r' for a regexp search)."""
# call word search method in some cases:
if f.endswith('count'):
return search_unit_in_bibwords(p, f, wl=wl)
hitset = intbitset() # will hold output result set
set_used = 0 # not-yet-used flag, to be able to circumvent set operations
limit_reached = 0 # flag for knowing if the query limit has been reached
use_query_limit = False # flag for knowing if to limit the query results or not
# deduce in which idxPHRASE table we will search:
idxphraseX = "idxPHRASE%02dF" % get_index_id_from_field("anyfield")
if f:
index_id = get_index_id_from_field(f)
if index_id:
idxphraseX = "idxPHRASE%02dF" % index_id
else:
return intbitset() # phrase index f does not exist
# detect query type (exact phrase, partial phrase, regexp):
if search_type == 'r':
query_addons = "REGEXP %s"
query_params = (p,)
use_query_limit = True
else:
p = p.replace('*', '%') # we now use '*' as the truncation character
ps = p.split("->", 1) # check for span query:
if len(ps) == 2 and not (ps[0].endswith(' ') or ps[1].startswith(' ')):
query_addons = "BETWEEN %s AND %s"
query_params = (ps[0], ps[1])
use_query_limit = True
else:
if p.find('%') > -1:
query_addons = "LIKE %s"
query_params = (p,)
use_query_limit = True
else:
query_addons = "= %s"
query_params = (p,)
# special washing for fuzzy author index:
if f in ('author', 'firstauthor', 'exactauthor', 'exactfirstauthor', 'authorityauthor'):
query_params_washed = ()
for query_param in query_params:
query_params_washed += (wash_author_name(query_param),)
query_params = query_params_washed
# perform search:
if use_query_limit:
try:
res = run_sql_with_limit("SELECT term,hitlist FROM %s WHERE term %s" % (idxphraseX, query_addons),
query_params, wildcard_limit=wl)
except InvenioDbQueryWildcardLimitError as excp:
res = excp.res
limit_reached = 1 # set the limit reached flag to true
else:
res = run_sql("SELECT term,hitlist FROM %s WHERE term %s" % (idxphraseX, query_addons), query_params)
# fill the result set:
for dummy_word, hitlist in res:
hitset_bibphrase = intbitset(hitlist)
# add the results:
if set_used:
hitset.union_update(hitset_bibphrase)
else:
hitset = hitset_bibphrase
set_used = 1
#check to see if the query limit was reached
if limit_reached:
#raise an exception, so we can print a nice message to the user
raise InvenioWebSearchWildcardLimitError(hitset)
# okay, return result set:
return hitset
def search_unit_in_bibxxx(p, f, type, wl=0):
"""Searches for pattern 'p' inside bibxxx tables for field 'f' and returns hitset of recIDs found.
The search type is defined by 'type' (e.g. equals to 'r' for a regexp search)."""
# call word search method in some cases:
if f == 'journal' or f.endswith('count'):
return search_unit_in_bibwords(p, f, wl=wl)
limit_reached = 0 # flag for knowing if the query limit has been reached
use_query_limit = False # flag for knowing if to limit the query results or not
query_addons = "" # will hold additional SQL code for the query
query_params = () # will hold parameters for the query (their number may vary depending on TYPE argument)
# wash arguments:
f = string.replace(f, '*', '%') # replace truncation char '*' in field definition
if type == 'r':
query_addons = "REGEXP %s"
query_params = (p,)
use_query_limit = True
else:
p = string.replace(p, '*', '%') # we now use '*' as the truncation character
ps = string.split(p, "->", 1) # check for span query:
if len(ps) == 2 and not (ps[0].endswith(' ') or ps[1].startswith(' ')):
query_addons = "BETWEEN %s AND %s"
query_params = (ps[0], ps[1])
use_query_limit = True
else:
if string.find(p, '%') > -1:
query_addons = "LIKE %s"
query_params = (p,)
use_query_limit = True
else:
query_addons = "= %s"
query_params = (p,)
# construct 'tl' which defines the tag list (MARC tags) to search in:
tl = []
if len(f) >= 2 and str(f[0]).isdigit() and str(f[1]).isdigit():
tl.append(f) # 'f' seems to be okay as it starts by two digits
else:
# deduce desired MARC tags on the basis of chosen 'f'
tl = get_field_tags(f)
if not tl:
# f index does not exist, nevermind
pass
# okay, start search:
l = [] # will hold list of recID that matched
for t in tl:
# deduce into which bibxxx table we will search:
digit1, digit2 = int(t[0]), int(t[1])
bx = "bib%d%dx" % (digit1, digit2)
bibx = "bibrec_bib%d%dx" % (digit1, digit2)
# construct and run query:
if t == "001":
if query_addons.find('BETWEEN') > -1 or query_addons.find('=') > -1:
# verify that the params are integers (to avoid returning record 123 when searching for 123foo)
try:
query_params = tuple(int(param) for param in query_params)
except ValueError:
return intbitset()
if use_query_limit:
try:
res = run_sql_with_limit("SELECT id FROM bibrec WHERE id %s" % query_addons,
query_params, wildcard_limit=wl)
except InvenioDbQueryWildcardLimitError as excp:
res = excp.res
limit_reached = 1 # set the limit reached flag to true
else:
res = run_sql("SELECT id FROM bibrec WHERE id %s" % query_addons,
query_params)
else:
query = "SELECT bibx.id_bibrec FROM %s AS bx LEFT JOIN %s AS bibx ON bx.id=bibx.id_bibxxx WHERE bx.value %s" % \
(bx, bibx, query_addons)
if len(t) != 6 or t[-1:]=='%':
# wildcard query, or only the beginning of field 't'
# is defined, so add wildcard character:
query += " AND bx.tag LIKE %s"
query_params_and_tag = query_params + (t + '%',)
else:
# exact query for 't':
query += " AND bx.tag=%s"
query_params_and_tag = query_params + (t,)
if use_query_limit:
try:
res = run_sql_with_limit(query, query_params_and_tag, wildcard_limit=wl)
except InvenioDbQueryWildcardLimitError as excp:
res = excp.res
limit_reached = 1 # set the limit reached flag to true
else:
res = run_sql(query, query_params_and_tag)
# fill the result set:
for id_bibrec in res:
if id_bibrec[0]:
l.append(id_bibrec[0])
# check no of hits found:
nb_hits = len(l)
# okay, return result set:
hitset = intbitset(l)
#check to see if the query limit was reached
if limit_reached:
#raise an exception, so we can print a nice message to the user
raise InvenioWebSearchWildcardLimitError(hitset)
return hitset
def search_unit_in_solr(p, f=None, m=None):
"""
Query a Solr index and return an intbitset corresponding
to the result. Parameters (p,f,m) are usual search unit ones.
"""
if m and (m == 'a' or m == 'r'): # phrase/regexp query
if p.startswith('%') and p.endswith('%'):
p = p[1:-1] # fix for partial phrase
p = '"' + p + '"'
return solr_get_bitset(f, p)
def search_unit_in_xapian(p, f=None, m=None):
"""
Query a Xapian index and return an intbitset corresponding
to the result. Parameters (p,f,m) are usual search unit ones.
"""
if m and (m == 'a' or m == 'r'): # phrase/regexp query
if p.startswith('%') and p.endswith('%'):
p = p[1:-1] # fix for partial phrase
p = '"' + p + '"'
return xapian_get_bitset(f, p)
def search_unit_in_bibrec(datetext1, datetext2, search_type='c'):
"""
Return hitset of recIDs found that were either created or modified
(according to 'type' arg being 'c' or 'm') from datetext1 until datetext2, inclusive.
Does not pay attention to pattern, collection, anything. Useful
to intersect later on with the 'real' query.
"""
hitset = intbitset()
if search_type and search_type.startswith("m"):
search_type = "modification_date"
else:
search_type = "creation_date" # by default we are searching for creation dates
parts = datetext1.split('->')
if len(parts) > 1 and datetext1 == datetext2:
datetext1 = parts[0]
datetext2 = parts[1]
if datetext1 == datetext2:
res = run_sql("SELECT id FROM bibrec WHERE %s LIKE %%s" % (search_type,),
(datetext1 + '%',))
else:
res = run_sql("SELECT id FROM bibrec WHERE %s>=%%s AND %s<=%%s" % (search_type, search_type),
(datetext1, datetext2))
for row in res:
hitset += row[0]
return hitset
def search_unit_by_times_cited(p, exclude_selfcites=False):
"""
Return histset of recIDs found that are cited P times.
Usually P looks like '10->23'.
"""
numstr = '"'+p+'"'
#this is sort of stupid but since we may need to
#get the records that do _not_ have cites, we have to
#know the ids of all records, too
#but this is needed only if bsu_p is 0 or 0 or 0->0
allrecs = []
if p == 0 or p == "0" or \
p.startswith("0->") or p.endswith("->0"):
allrecs = intbitset(run_sql("SELECT id FROM bibrec"))
return get_records_with_num_cites(numstr, allrecs,
exclude_selfcites=exclude_selfcites)
def search_unit_refersto(query):
"""
Search for records satisfying the query (e.g. author:ellis) and
return list of records referred to by these records.
"""
if query:
ahitset = search_pattern(p=query)
return get_refersto_hitset(ahitset)
else:
return intbitset([])
def search_unit_refersto_excluding_selfcites(query):
"""
Search for records satisfying the query (e.g. author:ellis) and
return list of records referred to by these records.
"""
if query:
ahitset = search_pattern(p=query)
citers = intbitset()
citations = get_cited_by_list(ahitset)
selfcitations = get_self_cited_by_list(ahitset)
for cites, selfcites in zip(citations, selfcitations):
# cites is in the form [(citee, citers), ...]
citers += cites[1] - selfcites[1]
return citers
else:
return intbitset([])
def search_unit_in_record_history(query):
"""
Return hitset of recIDs that were modified by the given cataloguer
"""
if query:
try:
cataloguer_name, modification_date = query.split(":")
except ValueError:
cataloguer_name = query
modification_date = ""
if modification_date:
spires_syntax_converter = SpiresToInvenioSyntaxConverter()
modification_date = spires_syntax_converter.convert_date(modification_date)
parts = modification_date.split('->', 1)
if len(parts) > 1:
start_date, end_date = parts
res = run_sql("SELECT id_bibrec FROM hstRECORD WHERE job_person=%s AND job_date>=%s AND job_date<=%s",
(cataloguer_name, start_date, end_date))
else:
res = run_sql("SELECT id_bibrec FROM hstRECORD WHERE job_person=%s AND job_date LIKE %s",
(cataloguer_name, modification_date + '%',))
return intbitset(res)
else:
sql = "SELECT id_bibrec FROM hstRECORD WHERE job_person=%s"
res = intbitset(run_sql(sql, (cataloguer_name,)))
return res
else:
return intbitset([])
def search_unit_citedby(query):
"""
Search for records satisfying the query (e.g. author:ellis) and
return list of records cited by these records.
"""
if query:
ahitset = search_pattern(p=query)
if ahitset:
return get_citedby_hitset(ahitset)
else:
return intbitset([])
else:
return intbitset([])
def search_unit_collection(query, m, wl=None):
"""
Search for records satisfying the query (e.g. collection:"BOOK" or
collection:"Books") and return list of records in the collection.
"""
if len(query):
ahitset = get_collection_reclist(query)
if not ahitset:
return search_unit_in_bibwords(query, 'collection', m, wl=wl)
return ahitset
else:
return intbitset([])
def search_unit_citedby_excluding_selfcites(query):
"""
Search for records satisfying the query (e.g. author:ellis) and
return list of records referred to by these records.
"""
if query:
ahitset = search_pattern(p=query)
citees = intbitset()
references = get_refers_to_list(ahitset)
selfreferences = get_self_refers_to_list(ahitset)
for refs, selfrefs in zip(references, selfreferences):
# refs is in the form [(citer, citees), ...]
citees += refs[1] - selfrefs[1]
return citees
else:
return intbitset([])
def get_records_that_can_be_displayed(user_info,
hitset_in_any_collection,
current_coll=CFG_SITE_NAME,
colls=None,
permitted_restricted_collections=None):
"""
Return records that can be displayed.
"""
records_that_can_be_displayed = intbitset()
if colls is None:
colls = [current_coll]
# let's get the restricted collections the user has rights to view
if permitted_restricted_collections is None:
permitted_restricted_collections = user_info.get('precached_permitted_restricted_collections', [])
policy = CFG_WEBSEARCH_VIEWRESTRCOLL_POLICY.strip().upper()
current_coll_children = get_collection_allchildren(current_coll) # real & virtual
# add all restricted collections, that the user has access to, and are under the current collection
# do not use set here, in order to maintain a specific order:
# children of 'cc' (real, virtual, restricted), rest of 'c' that are not cc's children
colls_to_be_displayed = [coll for coll in current_coll_children if coll in colls or coll in permitted_restricted_collections]
colls_to_be_displayed.extend([coll for coll in colls if coll not in colls_to_be_displayed])
if policy == 'ANY':# the user needs to have access to at least one collection that restricts the records
#we need this to be able to remove records that are both in a public and restricted collection
permitted_recids = intbitset()
notpermitted_recids = intbitset()
for collection in restricted_collection_cache.cache:
if collection in permitted_restricted_collections:
permitted_recids |= get_collection_reclist(collection)
else:
notpermitted_recids |= get_collection_reclist(collection)
records_that_can_be_displayed = hitset_in_any_collection - (notpermitted_recids - permitted_recids)
else:# the user needs to have access to all collections that restrict a records
notpermitted_recids = intbitset()
for collection in restricted_collection_cache.cache:
if collection not in permitted_restricted_collections:
notpermitted_recids |= get_collection_reclist(collection)
records_that_can_be_displayed = hitset_in_any_collection - notpermitted_recids
if records_that_can_be_displayed.is_infinite():
# We should not return infinite results for user.
records_that_can_be_displayed = intbitset()
for coll in colls_to_be_displayed:
records_that_can_be_displayed |= get_collection_reclist(coll)
return records_that_can_be_displayed
def intersect_results_with_collrecs(req, hitset_in_any_collection, colls, of="hb", verbose=0, ln=CFG_SITE_LANG, display_nearest_terms_box=True):
"""Return dict of hitsets given by intersection of hitset with the collection universes."""
_ = gettext_set_language(ln)
# search stage 4: intersect with the collection universe
if verbose and of.startswith("h"):
t1 = os.times()[4]
results = {} # all final results
results_nbhits = 0
# calculate the list of recids (restricted or not) that the user has rights to access and we should display (only those)
if not req or isinstance(req, cStringIO.OutputType): # called from CLI
user_info = {}
for coll in colls:
results[coll] = hitset_in_any_collection & get_collection_reclist(coll)
results_nbhits += len(results[coll])
records_that_can_be_displayed = hitset_in_any_collection
permitted_restricted_collections = []
else:
user_info = collect_user_info(req)
# let's get the restricted collections the user has rights to view
if user_info['guest'] == '1':
## For guest users that are actually authorized to some restricted
## collection (by virtue of the IP address in a FireRole rule)
## we explicitly build the list of permitted_restricted_collections
permitted_restricted_collections = get_permitted_restricted_collections(user_info)
else:
permitted_restricted_collections = user_info.get('precached_permitted_restricted_collections', [])
# let's build the list of the both public and restricted
# child collections of the collection from which the user
# started his/her search. This list of children colls will be
# used in the warning proposing a search in that collections
try:
current_coll = req.argd['cc'] # current_coll: coll from which user started his/her search
except:
from flask import request
current_coll = request.args.get('cc', CFG_SITE_NAME) # current_coll: coll from which user started his/her search
current_coll_children = get_collection_allchildren(current_coll) # real & virtual
# add all restricted collections, that the user has access to, and are under the current collection
# do not use set here, in order to maintain a specific order:
# children of 'cc' (real, virtual, restricted), rest of 'c' that are not cc's children
colls_to_be_displayed = [coll for coll in current_coll_children if coll in colls or coll in permitted_restricted_collections]
colls_to_be_displayed.extend([coll for coll in colls if coll not in colls_to_be_displayed])
records_that_can_be_displayed = get_records_that_can_be_displayed(
user_info,
hitset_in_any_collection,
current_coll,
colls,
permitted_restricted_collections)
for coll in colls_to_be_displayed:
results[coll] = results.get(coll, intbitset()) | (records_that_can_be_displayed & get_collection_reclist(coll))
results_nbhits += len(results[coll])
if results_nbhits == 0:
# no hits found, try to search in Home and restricted and/or hidden collections:
results = {}
results_in_Home = records_that_can_be_displayed & get_collection_reclist(CFG_SITE_NAME)
results_in_restricted_collections = intbitset()
results_in_hidden_collections = intbitset()
for coll in permitted_restricted_collections:
if not get_coll_ancestors(coll): # hidden collection
results_in_hidden_collections.union_update(records_that_can_be_displayed & get_collection_reclist(coll))
else:
results_in_restricted_collections.union_update(records_that_can_be_displayed & get_collection_reclist(coll))
# in this way, we do not count twice, records that are both in Home collection and in a restricted collection
total_results = len(results_in_Home.union(results_in_restricted_collections))
if total_results > 0:
# some hits found in Home and/or restricted collections, so propose this search:
if of.startswith("h") and display_nearest_terms_box:
url = websearch_templates.build_search_url(req.argd, cc=CFG_SITE_NAME, c=[])
len_colls_to_display = len(colls_to_be_displayed)
# trim the list of collections to first two, since it might get very large
write_warning(_("No match found in collection %(x_collection)s. Other collections gave %(x_url_open)s%(x_nb_hits)d hits%(x_url_close)s.") %
{'x_collection': '<em>' +
string.join([get_coll_i18nname(coll, ln, False) for coll in colls_to_be_displayed[:2]], ', ') +
(len_colls_to_display > 2 and ' et al' or '') + '</em>',
'x_url_open': '<a class="nearestterms" href="%s">' % (url),
'x_nb_hits': total_results,
'x_url_close': '</a>'}, req=req)
# display the hole list of collections in a comment
if len_colls_to_display > 2:
write_warning("<!--No match found in collection <em>%(x_collection)s</em>.-->" %
{'x_collection': string.join([get_coll_i18nname(coll, ln, False) for coll in colls_to_be_displayed], ', ')},
req=req)
else:
# no hits found, either user is looking for a document and he/she has not rights
# or user is looking for a hidden document:
if of.startswith("h") and display_nearest_terms_box:
if len(results_in_hidden_collections) > 0:
write_warning(_("No public collection matched your query. "
"If you were looking for a hidden document, please type "
"the correct URL for this record."), req=req)
else:
write_warning(_("No public collection matched your query. "
"If you were looking for a non-public document, please choose "
"the desired restricted collection first."), req=req)
if verbose and of.startswith("h"):
t2 = os.times()[4]
write_warning("Search stage 4: intersecting with collection universe gave %d hits." % results_nbhits, req=req)
write_warning("Search stage 4: execution took %.2f seconds." % (t2 - t1), req=req)
return results
def intersect_results_with_hitset(req, results, hitset, ap=0, aptext="", of="hb"):
"""Return intersection of search 'results' (a dict of hitsets
with collection as key) with the 'hitset', i.e. apply
'hitset' intersection to each collection within search
'results'.
If the final set is to be empty, and 'ap'
(approximate pattern) is true, and then print the `warningtext'
and return the original 'results' set unchanged. If 'ap' is
false, then return empty results set.
"""
if ap:
results_ap = copy.deepcopy(results)
else:
results_ap = {} # will return empty dict in case of no hits found
nb_total = 0
final_results = {}
for coll in results.keys():
final_results[coll] = results[coll].intersection(hitset)
nb_total += len(final_results[coll])
if nb_total == 0:
if of.startswith("h"):
write_warning(aptext, req=req)
final_results = results_ap
return final_results
def create_similarly_named_authors_link_box(author_name, ln=CFG_SITE_LANG):
"""Return a box similar to ``Not satisfied...'' one by proposing
author searches for similar names. Namely, take AUTHOR_NAME
and the first initial of the firstame (after comma) and look
into author index whether authors with e.g. middle names exist.
Useful mainly for CERN Library that sometimes contains name
forms like Ellis-N, Ellis-Nick, Ellis-Nicolas all denoting the
same person. The box isn't proposed if no similarly named
authors are found to exist.
"""
# return nothing if not configured:
if CFG_WEBSEARCH_CREATE_SIMILARLY_NAMED_AUTHORS_LINK_BOX == 0:
return ""
# return empty box if there is no initial:
if re.match(r'[^ ,]+, [^ ]', author_name) is None:
return ""
# firstly find name comma initial:
author_name_to_search = re.sub(r'^([^ ,]+, +[^ ,]).*$', '\\1', author_name)
# secondly search for similar name forms:
similar_author_names = {}
for name in author_name_to_search, strip_accents(author_name_to_search):
for tag in get_field_tags("author"):
# deduce into which bibxxx table we will search:
digit1, digit2 = int(tag[0]), int(tag[1])
bx = "bib%d%dx" % (digit1, digit2)
if len(tag) != 6 or tag[-1:] == '%':
# only the beginning of field 't' is defined, so add wildcard character:
res = run_sql("""SELECT bx.value FROM %s AS bx
WHERE bx.value LIKE %%s AND bx.tag LIKE %%s""" % bx,
(name + "%", tag + "%"))
else:
res = run_sql("""SELECT bx.value FROM %s AS bx
WHERE bx.value LIKE %%s AND bx.tag=%%s""" % bx,
(name + "%", tag))
for row in res:
similar_author_names[row[0]] = 1
# remove the original name and sort the list:
try:
del similar_author_names[author_name]
except KeyError:
pass
# thirdly print the box:
out = ""
if similar_author_names:
out_authors = similar_author_names.keys()
out_authors.sort()
tmp_authors = []
for out_author in out_authors:
nbhits = get_nbhits_in_bibxxx(out_author, "author")
if nbhits:
tmp_authors.append((out_author, nbhits))
out += websearch_templates.tmpl_similar_author_names(
authors=tmp_authors, ln=ln)
return out
def create_nearest_terms_box(urlargd, p, f, t='w', n=5, ln=CFG_SITE_LANG, intro_text_p=True):
"""Return text box containing list of 'n' nearest terms above/below 'p'
for the field 'f' for matching type 't' (words/phrases) in
language 'ln'.
Propose new searches according to `urlargs' with the new words.
If `intro_text_p' is true, then display the introductory message,
otherwise print only the nearest terms in the box content.
"""
# load the right message language
_ = gettext_set_language(ln)
if not CFG_WEBSEARCH_DISPLAY_NEAREST_TERMS:
return _("Your search did not match any records. Please try again.")
nearest_terms = []
if not p: # sanity check
p = "."
if p.startswith('%') and p.endswith('%'):
p = p[1:-1] # fix for partial phrase
index_id = get_index_id_from_field(f)
if f == 'fulltext':
if CFG_SOLR_URL:
return _("No match found, please enter different search terms.")
else:
# FIXME: workaround for not having native phrase index yet
t = 'w'
# special indexes:
if f == 'refersto' or f == 'referstoexcludingselfcites':
return _("There are no records referring to %(x_rec)s.", x_rec=cgi.escape(p))
if f == 'cataloguer':
return _("There are no records modified by %(x_rec)s.", x_rec=cgi.escape(p))
if f == 'citedby' or f == 'citedbyexcludingselfcites':
return _("There are no records cited by %(x_rec)s.", x_rec=cgi.escape(p))
# look for nearest terms:
if t == 'w':
nearest_terms = get_nearest_terms_in_bibwords(p, f, n, n)
if not nearest_terms:
return _("No word index is available for %(x_name)s.",
x_name=('<em>' + cgi.escape(get_field_i18nname(get_field_name(f) or f, ln, False)) + '</em>'))
else:
nearest_terms = []
if index_id:
nearest_terms = get_nearest_terms_in_idxphrase(p, index_id, n, n)
if f == 'datecreated' or f == 'datemodified':
nearest_terms = get_nearest_terms_in_bibrec(p, f, n, n)
if not nearest_terms:
nearest_terms = get_nearest_terms_in_bibxxx(p, f, n, n)
if not nearest_terms:
return _("No phrase index is available for %(x_name)s.",
x_name=('<em>' + cgi.escape(get_field_i18nname(get_field_name(f) or f, ln, False)) + '</em>'))
terminfo = []
for term in nearest_terms:
if t == 'w':
hits = get_nbhits_in_bibwords(term, f)
else:
if index_id:
hits = get_nbhits_in_idxphrases(term, f)
elif f == 'datecreated' or f == 'datemodified':
hits = get_nbhits_in_bibrec(term, f)
else:
hits = get_nbhits_in_bibxxx(term, f)
argd = {}
argd.update(urlargd)
# check which fields contained the requested parameter, and replace it.
for px, dummy_fx in ('p', 'f'), ('p1', 'f1'), ('p2', 'f2'), ('p3', 'f3'):
if px in argd:
argd_px = argd[px]
if t == 'w':
# p was stripped of accents, to do the same:
argd_px = strip_accents(argd_px)
#argd[px] = string.replace(argd_px, p, term, 1)
#we need something similar, but case insensitive
pattern_index = string.find(argd_px.lower(), p.lower())
if pattern_index > -1:
argd[px] = argd_px[:pattern_index] + term + argd_px[pattern_index+len(p):]
break
#this is doing exactly the same as:
#argd[px] = re.sub('(?i)' + re.escape(p), term, argd_px, 1)
#but is ~4x faster (2us vs. 8.25us)
terminfo.append((term, hits, argd))
intro = ""
if intro_text_p: # add full leading introductory text
if f:
intro = _("Search term %(x_term)s inside index %(x_index)s did not match any record. Nearest terms in any collection are:") % \
{'x_term': "<em>" + cgi.escape(p.startswith("%") and p.endswith("%") and p[1:-1] or p) + "</em>",
'x_index': "<em>" + cgi.escape(get_field_i18nname(get_field_name(f) or f, ln, False)) + "</em>"}
else:
intro = _("Search term %(x_name)s did not match any record. Nearest terms in any collection are:",
x_name=("<em>" + cgi.escape(p.startswith("%") and p.endswith("%") and p[1:-1] or p) + "</em>"))
return websearch_templates.tmpl_nearest_term_box(p=p, ln=ln, f=f, terminfo=terminfo,
intro=intro)
def get_nearest_terms_in_bibwords(p, f, n_below, n_above):
"""Return list of +n -n nearest terms to word `p' in index for field `f'."""
nearest_words = [] # will hold the (sorted) list of nearest words to return
# deduce into which bibwordsX table we will search:
bibwordsX = "idxWORD%02dF" % get_index_id_from_field("anyfield")
if f:
index_id = get_index_id_from_field(f)
if index_id:
bibwordsX = "idxWORD%02dF" % index_id
else:
return nearest_words
# firstly try to get `n' closest words above `p':
res = run_sql("SELECT term FROM %s WHERE term<%%s ORDER BY term DESC LIMIT %%s" % bibwordsX,
(p, n_above))
for row in res:
nearest_words.append(row[0])
nearest_words.reverse()
# secondly insert given word `p':
nearest_words.append(p)
# finally try to get `n' closest words below `p':
res = run_sql("SELECT term FROM %s WHERE term>%%s ORDER BY term ASC LIMIT %%s" % bibwordsX,
(p, n_below))
for row in res:
nearest_words.append(row[0])
return nearest_words
def get_nearest_terms_in_idxphrase(p, index_id, n_below, n_above):
"""Browse (-n_above, +n_below) closest bibliographic phrases
for the given pattern p in the given field idxPHRASE table,
regardless of collection.
Return list of [phrase1, phrase2, ... , phrase_n]."""
if CFG_INSPIRE_SITE and index_id in (3, 15): # FIXME: workaround due to new fuzzy index
return [p]
idxphraseX = "idxPHRASE%02dF" % index_id
res_above = run_sql("SELECT term FROM %s WHERE term<%%s ORDER BY term DESC LIMIT %%s" % idxphraseX, (p, n_above))
res_above = [x[0] for x in res_above]
res_above.reverse()
res_below = run_sql("SELECT term FROM %s WHERE term>=%%s ORDER BY term ASC LIMIT %%s" % idxphraseX, (p, n_below))
res_below = [x[0] for x in res_below]
return res_above + res_below
def get_nearest_terms_in_idxphrase_with_collection(p, index_id, n_below, n_above, collection):
"""Browse (-n_above, +n_below) closest bibliographic phrases
for the given pattern p in the given field idxPHRASE table,
considering the collection (intbitset).
Return list of [(phrase1, hitset), (phrase2, hitset), ... , (phrase_n, hitset)]."""
idxphraseX = "idxPHRASE%02dF" % index_id
res_above = run_sql("SELECT term,hitlist FROM %s WHERE term<%%s ORDER BY term DESC LIMIT %%s" % idxphraseX, (p, n_above * 3))
res_above = [(term, intbitset(hitlist) & collection) for term, hitlist in res_above]
res_above = [(term, len(hitlist)) for term, hitlist in res_above if hitlist]
res_below = run_sql("SELECT term,hitlist FROM %s WHERE term>=%%s ORDER BY term ASC LIMIT %%s" % idxphraseX, (p, n_below * 3))
res_below = [(term, intbitset(hitlist) & collection) for term, hitlist in res_below]
res_below = [(term, len(hitlist)) for term, hitlist in res_below if hitlist]
res_above.reverse()
return res_above[-n_above:] + res_below[:n_below]
def get_nearest_terms_in_bibxxx(p, f, n_below, n_above):
"""Browse (-n_above, +n_below) closest bibliographic phrases
for the given pattern p in the given field f, regardless
of collection.
Return list of [phrase1, phrase2, ... , phrase_n]."""
## determine browse field:
if not f and string.find(p, ":") > 0: # does 'p' contain ':'?
f, p = string.split(p, ":", 1)
# FIXME: quick hack for the journal index
if f == 'journal':
return get_nearest_terms_in_bibwords(p, f, n_below, n_above)
## We are going to take max(n_below, n_above) as the number of
## values to ferch from bibXXx. This is needed to work around
## MySQL UTF-8 sorting troubles in 4.0.x. Proper solution is to
## use MySQL 4.1.x or our own idxPHRASE in the future.
index_id = get_index_id_from_field(f)
if index_id:
return get_nearest_terms_in_idxphrase(p, index_id, n_below, n_above)
n_fetch = 2*max(n_below, n_above)
## construct 'tl' which defines the tag list (MARC tags) to search in:
tl = []
if str(f[0]).isdigit() and str(f[1]).isdigit():
tl.append(f) # 'f' seems to be okay as it starts by two digits
else:
# deduce desired MARC tags on the basis of chosen 'f'
tl = get_field_tags(f)
## start browsing to fetch list of hits:
browsed_phrases = {} # will hold {phrase1: 1, phrase2: 1, ..., phraseN: 1} dict of browsed phrases (to make them unique)
# always add self to the results set:
browsed_phrases[p.startswith("%") and p.endswith("%") and p[1:-1] or p] = 1
for t in tl:
# deduce into which bibxxx table we will search:
digit1, digit2 = int(t[0]), int(t[1])
bx = "bib%d%dx" % (digit1, digit2)
# firstly try to get `n' closest phrases above `p':
if len(t) != 6 or t[-1:] == '%': # only the beginning of field 't' is defined, so add wildcard character:
res = run_sql("""SELECT bx.value FROM %s AS bx
WHERE bx.value<%%s AND bx.tag LIKE %%s
ORDER BY bx.value DESC LIMIT %%s""" % bx,
(p, t + "%", n_fetch))
else:
res = run_sql("""SELECT bx.value FROM %s AS bx
WHERE bx.value<%%s AND bx.tag=%%s
ORDER BY bx.value DESC LIMIT %%s""" % bx,
(p, t, n_fetch))
for row in res:
browsed_phrases[row[0]] = 1
# secondly try to get `n' closest phrases equal to or below `p':
if len(t) != 6 or t[-1:]=='%': # only the beginning of field 't' is defined, so add wildcard character:
res = run_sql("""SELECT bx.value FROM %s AS bx
WHERE bx.value>=%%s AND bx.tag LIKE %%s
ORDER BY bx.value ASC LIMIT %%s""" % bx,
(p, t + "%", n_fetch))
else:
res = run_sql("""SELECT bx.value FROM %s AS bx
WHERE bx.value>=%%s AND bx.tag=%%s
ORDER BY bx.value ASC LIMIT %%s""" % bx,
(p, t, n_fetch))
for row in res:
browsed_phrases[row[0]] = 1
# select first n words only: (this is needed as we were searching
# in many different tables and so aren't sure we have more than n
# words right; this of course won't be needed when we shall have
# one ACC table only for given field):
phrases_out = browsed_phrases.keys()
phrases_out.sort(lambda x, y: cmp(string.lower(strip_accents(x)),
string.lower(strip_accents(y))))
# find position of self:
try:
idx_p = phrases_out.index(p)
except ValueError:
idx_p = len(phrases_out)/2
# return n_above and n_below:
return phrases_out[max(0, idx_p-n_above):idx_p+n_below]
def get_nearest_terms_in_bibrec(p, f, n_below, n_above):
"""Return list of nearest terms and counts from bibrec table.
p is usually a date, and f either datecreated or datemodified.
Note: below/above count is very approximative, not really respected.
"""
col = 'creation_date'
if f == 'datemodified':
col = 'modification_date'
res_above = run_sql("""SELECT DATE_FORMAT(%s,'%%%%Y-%%%%m-%%%%d %%%%H:%%%%i:%%%%s')
FROM bibrec WHERE %s < %%s
ORDER BY %s DESC LIMIT %%s""" % (col, col, col),
(p, n_above))
res_below = run_sql("""SELECT DATE_FORMAT(%s,'%%%%Y-%%%%m-%%%%d %%%%H:%%%%i:%%%%s')
FROM bibrec WHERE %s > %%s
ORDER BY %s ASC LIMIT %%s""" % (col, col, col),
(p, n_below))
out = set([])
for row in res_above:
out.add(row[0])
for row in res_below:
out.add(row[0])
out_list = list(out)
out_list.sort()
return list(out_list)
def get_nbhits_in_bibrec(term, f):
"""Return number of hits in bibrec table. term is usually a date,
and f is either 'datecreated' or 'datemodified'."""
col = 'creation_date'
if f == 'datemodified':
col = 'modification_date'
res = run_sql("SELECT COUNT(*) FROM bibrec WHERE %s LIKE %%s" % (col,),
(term + '%',))
return res[0][0]
def get_nbhits_in_bibwords(word, f):
"""Return number of hits for word 'word' inside words index for field 'f'."""
out = 0
# deduce into which bibwordsX table we will search:
bibwordsX = "idxWORD%02dF" % get_index_id_from_field("anyfield")
if f:
index_id = get_index_id_from_field(f)
if index_id:
bibwordsX = "idxWORD%02dF" % index_id
else:
return 0
if word:
res = run_sql("SELECT hitlist FROM %s WHERE term=%%s" % bibwordsX,
(word,))
for hitlist in res:
out += len(intbitset(hitlist[0]))
return out
def get_nbhits_in_idxphrases(word, f):
"""Return number of hits for word 'word' inside phrase index for field 'f'."""
out = 0
# deduce into which bibwordsX table we will search:
idxphraseX = "idxPHRASE%02dF" % get_index_id_from_field("anyfield")
if f:
index_id = get_index_id_from_field(f)
if index_id:
idxphraseX = "idxPHRASE%02dF" % index_id
else:
return 0
if word:
res = run_sql("SELECT hitlist FROM %s WHERE term=%%s" % idxphraseX,
(word,))
for hitlist in res:
out += len(intbitset(hitlist[0]))
return out
def get_nbhits_in_bibxxx(p, f, in_hitset=None):
"""Return number of hits for word 'word' inside words index for field 'f'."""
## determine browse field:
if not f and string.find(p, ":") > 0: # does 'p' contain ':'?
f, p = string.split(p, ":", 1)
# FIXME: quick hack for the journal index
if f == 'journal':
return get_nbhits_in_bibwords(p, f)
## construct 'tl' which defines the tag list (MARC tags) to search in:
tl = []
if str(f[0]).isdigit() and str(f[1]).isdigit():
tl.append(f) # 'f' seems to be okay as it starts by two digits
else:
# deduce desired MARC tags on the basis of chosen 'f'
tl = get_field_tags(f)
# start searching:
recIDs = {} # will hold dict of {recID1: 1, recID2: 1, ..., } (unique recIDs, therefore)
for t in tl:
# deduce into which bibxxx table we will search:
digit1, digit2 = int(t[0]), int(t[1])
bx = "bib%d%dx" % (digit1, digit2)
bibx = "bibrec_bib%d%dx" % (digit1, digit2)
if len(t) != 6 or t[-1:]=='%': # only the beginning of field 't' is defined, so add wildcard character:
res = run_sql("""SELECT bibx.id_bibrec FROM %s AS bibx, %s AS bx
WHERE bx.value=%%s AND bx.tag LIKE %%s
AND bibx.id_bibxxx=bx.id""" % (bibx, bx),
(p, t + "%"))
else:
res = run_sql("""SELECT bibx.id_bibrec FROM %s AS bibx, %s AS bx
WHERE bx.value=%%s AND bx.tag=%%s
AND bibx.id_bibxxx=bx.id""" % (bibx, bx),
(p, t))
for row in res:
recIDs[row[0]] = 1
if in_hitset is None:
nbhits = len(recIDs)
else:
nbhits = len(intbitset(recIDs.keys()).intersection(in_hitset))
return nbhits
def get_mysql_recid_from_aleph_sysno(sysno):
"""Returns DB's recID for ALEPH sysno passed in the argument (e.g. "002379334CER").
Returns None in case of failure."""
out = None
res = run_sql("""SELECT bb.id_bibrec FROM bibrec_bib97x AS bb, bib97x AS b
WHERE b.value=%s AND b.tag='970__a' AND bb.id_bibxxx=b.id""",
(sysno,))
if res:
out = res[0][0]
return out
def guess_primary_collection_of_a_record(recID):
"""Return primary collection name a record recid belongs to, by
testing 980 identifier.
May lead to bad guesses when a collection is defined dynamically
via dbquery.
In that case, return 'CFG_SITE_NAME'."""
out = CFG_SITE_NAME
dbcollids = get_fieldvalues(recID, "980__a")
for dbcollid in dbcollids:
variants = ("collection:" + dbcollid,
'collection:"' + dbcollid + '"',
"980__a:" + dbcollid,
'980__a:"' + dbcollid + '"',
'980:' + dbcollid ,
'980:"' + dbcollid + '"')
res = run_sql("SELECT name FROM collection WHERE dbquery IN (%s,%s,%s,%s,%s,%s)", variants)
if res:
out = res[0][0]
break
if CFG_CERN_SITE:
recID = int(recID)
# dirty hack for ATLAS collections at CERN:
if out in ('ATLAS Communications', 'ATLAS Internal Notes'):
for alternative_collection in ('ATLAS Communications Physics',
'ATLAS Communications General',
'ATLAS Internal Notes Physics',
'ATLAS Internal Notes General',):
if recID in get_collection_reclist(alternative_collection):
return alternative_collection
# dirty hack for FP
FP_collections = {'DO': ['Current Price Enquiries', 'Archived Price Enquiries'],
'IT': ['Current Invitation for Tenders', 'Archived Invitation for Tenders'],
'MS': ['Current Market Surveys', 'Archived Market Surveys']}
fp_coll_ids = [coll for coll in dbcollids if coll in FP_collections]
for coll in fp_coll_ids:
for coll_name in FP_collections[coll]:
if recID in get_collection_reclist(coll_name):
return coll_name
return out
_re_collection_url = re.compile('/collection/(.+)')
def guess_collection_of_a_record(recID, referer=None, recreate_cache_if_needed=True):
"""Return collection name a record recid belongs to, by first testing
the referer URL if provided and otherwise returning the
primary collection."""
if referer:
dummy, hostname, path, dummy, query, dummy = urlparse.urlparse(referer)
#requests can come from different invenio installations, with different collections
if CFG_SITE_URL.find(hostname) < 0:
return guess_primary_collection_of_a_record(recID)
g = _re_collection_url.match(path)
if g:
name = urllib.unquote_plus(g.group(1))
#check if this collection actually exist (also normalize the name if case-insensitive)
name = get_coll_normalised_name(name)
if name and recID in get_collection_reclist(name):
return name
elif path.startswith('/search'):
if recreate_cache_if_needed:
collection_reclist_cache.recreate_cache_if_needed()
query = cgi.parse_qs(query)
for name in query.get('cc', []) + query.get('c', []):
name = get_coll_normalised_name(name)
if name and recID in get_collection_reclist(name, recreate_cache_if_needed=False):
return name
return guess_primary_collection_of_a_record(recID)
def is_record_in_any_collection(recID, recreate_cache_if_needed=True):
"""Return True if the record belongs to at least one collection. This is a
good, although not perfect, indicator to guess if webcoll has already run
after this record has been entered into the system.
"""
if recreate_cache_if_needed:
collection_reclist_cache.recreate_cache_if_needed()
for name in collection_reclist_cache.cache.keys():
if recID in get_collection_reclist(name, recreate_cache_if_needed=False):
return True
return False
def get_all_collections_of_a_record(recID, recreate_cache_if_needed=True):
"""Return all the collection names a record belongs to.
Note this function is O(n_collections)."""
ret = []
if recreate_cache_if_needed:
collection_reclist_cache.recreate_cache_if_needed()
for name in collection_reclist_cache.cache.keys():
if recID in get_collection_reclist(name, recreate_cache_if_needed=False):
ret.append(name)
return ret
def get_tag_name(tag_value, prolog="", epilog=""):
"""Return tag name from the known tag value, by looking up the 'tag' table.
Return empty string in case of failure.
Example: input='100__%', output=first author'."""
out = ""
res = run_sql("SELECT name FROM tag WHERE value=%s", (tag_value,))
if res:
out = prolog + res[0][0] + epilog
return out
def get_fieldcodes():
"""Returns a list of field codes that may have been passed as 'search options' in URL.
Example: output=['subject','division']."""
out = []
res = run_sql("SELECT DISTINCT(code) FROM field")
for row in res:
out.append(row[0])
return out
def get_field_name(code):
"""Return the corresponding field_name given the field code.
e.g. reportnumber -> report number."""
res = run_sql("SELECT name FROM field WHERE code=%s", (code, ))
if res:
return res[0][0]
else:
return ""
def get_field_tags(field):
"""Returns a list of MARC tags for the field code 'field'.
Returns empty list in case of error.
Example: field='author', output=['100__%','700__%']."""
out = []
query = """SELECT t.value FROM tag AS t, field_tag AS ft, field AS f
WHERE f.code=%s AND ft.id_field=f.id AND t.id=ft.id_tag
ORDER BY ft.score DESC"""
res = run_sql(query, (field, ))
for val in res:
out.append(val[0])
return out
def get_fieldvalues_alephseq_like(recID, tags_in, can_see_hidden=False):
"""Return buffer of ALEPH sequential-like textual format with fields found
in the list TAGS_IN for record RECID.
If can_see_hidden is True, just print everything. Otherwise hide fields
from CFG_BIBFORMAT_HIDDEN_TAGS.
"""
out = ""
if type(tags_in) is not list:
tags_in = [tags_in]
if len(tags_in) == 1 and len(tags_in[0]) == 6:
## case A: one concrete subfield asked, so print its value if found
## (use with care: can mislead if field has multiple occurrences)
out += string.join(get_fieldvalues(recID, tags_in[0]), "\n")
else:
## case B: print our "text MARC" format; works safely all the time
# find out which tags to output:
dict_of_tags_out = {}
if not tags_in:
for i in range(0, 10):
for j in range(0, 10):
dict_of_tags_out["%d%d%%" % (i, j)] = 1
else:
for tag in tags_in:
if len(tag) == 0:
for i in range(0, 10):
for j in range(0, 10):
dict_of_tags_out["%d%d%%" % (i, j)] = 1
elif len(tag) == 1:
for j in range(0, 10):
dict_of_tags_out["%s%d%%" % (tag, j)] = 1
elif len(tag) < 5:
dict_of_tags_out["%s%%" % tag] = 1
elif tag >= 6:
dict_of_tags_out[tag[0:5]] = 1
tags_out = dict_of_tags_out.keys()
tags_out.sort()
# search all bibXXx tables as needed:
for tag in tags_out:
digits = tag[0:2]
try:
intdigits = int(digits)
if intdigits < 0 or intdigits > 99:
raise ValueError
except ValueError:
# invalid tag value asked for
continue
if tag.startswith("001") or tag.startswith("00%"):
if out:
out += "\n"
out += "%09d %s %d" % (recID, "001__", recID)
bx = "bib%sx" % digits
bibx = "bibrec_bib%sx" % digits
query = "SELECT b.tag,b.value,bb.field_number FROM %s AS b, %s AS bb "\
"WHERE bb.id_bibrec=%%s AND b.id=bb.id_bibxxx AND b.tag LIKE %%s"\
"ORDER BY bb.field_number, b.tag ASC" % (bx, bibx)
res = run_sql(query, (recID, str(tag)+'%'))
# go through fields:
field_number_old = -999
field_old = ""
for row in res:
field, value, field_number = row[0], row[1], row[2]
ind1, ind2 = field[3], field[4]
printme = True
#check the stuff in hiddenfields
if not can_see_hidden:
for htag in CFG_BIBFORMAT_HIDDEN_TAGS:
ltag = len(htag)
samelenfield = field[0:ltag]
if samelenfield == htag:
printme = False
if ind1 == "_":
ind1 = ""
if ind2 == "_":
ind2 = ""
# print field tag
if printme:
if field_number != field_number_old or field[:-1] != field_old[:-1]:
if out:
out += "\n"
out += "%09d %s " % (recID, field[:5])
field_number_old = field_number
field_old = field
# print subfield value
if field[0:2] == "00" and field[-1:] == "_":
out += value
else:
out += "$$%s%s" % (field[-1:], value)
return out
def get_merged_recid(recID):
""" Return the record ID of the record with
which the given record has been merged.
@param recID: deleted record recID
@type recID: int
@return: merged record recID
@rtype: int or None
"""
merged_recid = None
for val in get_fieldvalues(recID, "970__d"):
try:
merged_recid = int(val)
break
except ValueError:
pass
return merged_recid
def record_empty(recID):
"""
Is this record empty, e.g. has only 001, waiting for integration?
@param recID: the record identifier.
@type recID: int
@return: 1 if the record is empty, 0 otherwise.
@rtype: int
"""
return bibrecord.record_empty(get_record(recID))
def record_public_p(recID, recreate_cache_if_needed=True):
"""Return 1 if the record is public, i.e. if it can be found in the Home collection.
Return 0 otherwise.
"""
return recID in get_collection_reclist(CFG_SITE_NAME, recreate_cache_if_needed=recreate_cache_if_needed)
def get_creation_date(recID, fmt="%Y-%m-%d"):
"Returns the creation date of the record 'recID'."
out = ""
res = run_sql("SELECT DATE_FORMAT(creation_date,%s) FROM bibrec WHERE id=%s", (fmt, recID), 1)
if res:
out = res[0][0]
return out
def get_modification_date(recID, fmt="%Y-%m-%d"):
"Returns the date of last modification for the record 'recID'."
out = ""
res = run_sql("SELECT DATE_FORMAT(modification_date,%s) FROM bibrec WHERE id=%s", (fmt, recID), 1)
if res:
out = res[0][0]
return out
def print_search_info(p, f, sf, so, sp, rm, of, ot, collection=CFG_SITE_NAME, nb_found=-1, jrec=1, rg=CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS,
aas=0, ln=CFG_SITE_LANG, p1="", p2="", p3="", f1="", f2="", f3="", m1="", m2="", m3="", op1="", op2="",
sc=1, pl_in_url="",
d1y=0, d1m=0, d1d=0, d2y=0, d2m=0, d2d=0, dt="",
cpu_time=-1, middle_only=0, em=""):
"""Prints stripe with the information on 'collection' and 'nb_found' results and CPU time.
Also, prints navigation links (beg/next/prev/end) inside the results set.
If middle_only is set to 1, it will only print the middle box information (beg/netx/prev/end/etc) links.
This is suitable for displaying navigation links at the bottom of the search results page."""
if em != '' and EM_REPOSITORY["search_info"] not in em:
return ""
# sanity check:
if jrec < 1:
jrec = 1
if jrec > nb_found:
jrec = max(nb_found-rg+1, 1)
return websearch_templates.tmpl_print_search_info(
ln = ln,
collection = collection,
aas = aas,
collection_name = get_coll_i18nname(collection, ln, False),
collection_id = get_colID(collection),
middle_only = middle_only,
rg = rg,
nb_found = nb_found,
sf = sf,
so = so,
rm = rm,
of = of,
ot = ot,
p = p,
f = f,
p1 = p1,
p2 = p2,
p3 = p3,
f1 = f1,
f2 = f2,
f3 = f3,
m1 = m1,
m2 = m2,
m3 = m3,
op1 = op1,
op2 = op2,
pl_in_url = pl_in_url,
d1y = d1y,
d1m = d1m,
d1d = d1d,
d2y = d2y,
d2m = d2m,
d2d = d2d,
dt = dt,
jrec = jrec,
sc = sc,
sp = sp,
all_fieldcodes = get_fieldcodes(),
cpu_time = cpu_time,
)
def print_hosted_search_info(p, f, sf, so, sp, rm, of, ot, collection=CFG_SITE_NAME, nb_found=-1, jrec=1, rg=CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS,
aas=0, ln=CFG_SITE_LANG, p1="", p2="", p3="", f1="", f2="", f3="", m1="", m2="", m3="", op1="", op2="",
sc=1, pl_in_url="",
d1y=0, d1m=0, d1d=0, d2y=0, d2m=0, d2d=0, dt="",
cpu_time=-1, middle_only=0, em=""):
"""Prints stripe with the information on 'collection' and 'nb_found' results and CPU time.
Also, prints navigation links (beg/next/prev/end) inside the results set.
If middle_only is set to 1, it will only print the middle box information (beg/netx/prev/end/etc) links.
This is suitable for displaying navigation links at the bottom of the search results page."""
if em != '' and EM_REPOSITORY["search_info"] not in em:
return ""
# sanity check:
if jrec < 1:
jrec = 1
if jrec > nb_found:
jrec = max(nb_found-rg+1, 1)
return websearch_templates.tmpl_print_hosted_search_info(
ln = ln,
collection = collection,
aas = aas,
collection_name = get_coll_i18nname(collection, ln, False),
collection_id = get_colID(collection),
middle_only = middle_only,
rg = rg,
nb_found = nb_found,
sf = sf,
so = so,
rm = rm,
of = of,
ot = ot,
p = p,
f = f,
p1 = p1,
p2 = p2,
p3 = p3,
f1 = f1,
f2 = f2,
f3 = f3,
m1 = m1,
m2 = m2,
m3 = m3,
op1 = op1,
op2 = op2,
pl_in_url = pl_in_url,
d1y = d1y,
d1m = d1m,
d1d = d1d,
d2y = d2y,
d2m = d2m,
d2d = d2d,
dt = dt,
jrec = jrec,
sc = sc,
sp = sp,
all_fieldcodes = get_fieldcodes(),
cpu_time = cpu_time,
)
def print_results_overview(colls, results_final_nb_total, results_final_nb, cpu_time, ln=CFG_SITE_LANG, ec=[], hosted_colls_potential_results_p=False, em=""):
"""Prints results overview box with links to particular collections below."""
if em != "" and EM_REPOSITORY["overview"] not in em:
return ""
new_colls = []
for coll in colls:
new_colls.append({
'id': get_colID(coll),
'code': coll,
'name': get_coll_i18nname(coll, ln, False),
})
return websearch_templates.tmpl_print_results_overview(
ln = ln,
results_final_nb_total = results_final_nb_total,
results_final_nb = results_final_nb,
cpu_time = cpu_time,
colls = new_colls,
ec = ec,
hosted_colls_potential_results_p = hosted_colls_potential_results_p,
)
def print_hosted_results(url_and_engine, ln=CFG_SITE_LANG, of=None, req=None, no_records_found=False, search_timed_out=False, limit=CFG_EXTERNAL_COLLECTION_MAXRESULTS, em = ""):
"""Prints the full results of a hosted collection"""
if of.startswith("h"):
if no_records_found:
return "<br />No results found."
if search_timed_out:
return "<br />The search engine did not respond in time."
return websearch_templates.tmpl_print_hosted_results(
url_and_engine=url_and_engine,
ln=ln,
of=of,
req=req,
limit=limit,
display_body = em == "" or EM_REPOSITORY["body"] in em,
display_add_to_basket = em == "" or EM_REPOSITORY["basket"] in em)
class BibSortDataCacher(DataCacher):
"""
Cache holding all structures created by bibsort
( _data, data_dict).
"""
def __init__(self, method_name):
self.method_name = method_name
self.method_id = 0
res = run_sql("""SELECT id from bsrMETHOD where name = %s""", (self.method_name,))
if res and res[0]:
self.method_id = res[0][0]
else:
self.method_id = 0
def cache_filler():
method_id = self.method_id
alldicts = {}
if self.method_id == 0:
return {}
try:
res_data = run_sql("""SELECT data_dict_ordered from bsrMETHODDATA \
where id_bsrMETHOD = %s""", (method_id,))
res_buckets = run_sql("""SELECT bucket_no, bucket_data from bsrMETHODDATABUCKET\
where id_bsrMETHOD = %s""", (method_id,))
except Exception:
# database problems, return empty cache
return {}
try:
data_dict_ordered = deserialize_via_marshal(res_data[0][0])
except IndexError:
data_dict_ordered = {}
alldicts['data_dict_ordered'] = data_dict_ordered # recid: weight
if not res_buckets:
alldicts['bucket_data'] = {}
return alldicts
for row in res_buckets:
bucket_no = row[0]
try:
bucket_data = intbitset(row[1])
except IndexError:
bucket_data = intbitset([])
alldicts.setdefault('bucket_data', {})[bucket_no] = bucket_data
return alldicts
def timestamp_verifier():
method_id = self.method_id
res = run_sql("""SELECT last_updated from bsrMETHODDATA where id_bsrMETHOD = %s""", (method_id,))
try:
update_time_methoddata = str(res[0][0])
except IndexError:
update_time_methoddata = '1970-01-01 00:00:00'
res = run_sql("""SELECT max(last_updated) from bsrMETHODDATABUCKET where id_bsrMETHOD = %s""", (method_id,))
try:
update_time_buckets = str(res[0][0])
except IndexError:
update_time_buckets = '1970-01-01 00:00:00'
return max(update_time_methoddata, update_time_buckets)
DataCacher.__init__(self, cache_filler, timestamp_verifier)
def get_sorting_methods():
res = run_sql("""SELECT m.name, m.definition
FROM bsrMETHOD m, bsrMETHODDATA md
WHERE m.id = md.id_bsrMETHOD""")
return dict(res)
SORTING_METHODS = get_sorting_methods()
CACHE_SORTED_DATA = {}
for sorting_method in SORTING_METHODS:
try:
CACHE_SORTED_DATA[sorting_method].is_ok_p
except KeyError:
CACHE_SORTED_DATA[sorting_method] = BibSortDataCacher(sorting_method)
def get_tags_from_sort_fields(sort_fields):
"""Given a list of sort_fields, return the tags associated with it and
also the name of the field that has no tags associated, to be able to
display a message to the user."""
tags = []
if not sort_fields:
return [], ''
for sort_field in sort_fields:
if sort_field and (len(sort_field) > 1 and str(sort_field[0:2]).isdigit()):
# sort_field starts by two digits, so this is probably a MARC tag already
tags.append(sort_field)
else:
# let us check the 'field' table
field_tags = get_field_tags(sort_field)
if field_tags:
tags.extend(field_tags)
else:
return [], sort_field
return tags, ''
def rank_records(req, rank_method_code, rank_limit_relevance, hitset_global, pattern=None, verbose=0, sort_order='d', of='hb', ln=CFG_SITE_LANG, rg=None, jrec=None, field='', sorting_methods=SORTING_METHODS):
"""Initial entry point for ranking records, acts like a dispatcher.
(i) rank_method_code is in bsrMETHOD, bibsort buckets can be used;
(ii)rank_method_code is not in bsrMETHOD, use bibrank;
"""
# Special case: sorting by citations is fast because we store the
# ranking dictionary in memory, so we do not use bibsort buckets.
if CFG_BIBSORT_ENABLED and sorting_methods and rank_method_code != 'citation':
for sort_method in sorting_methods:
definition = sorting_methods[sort_method]
if definition.startswith('RNK') and \
definition.replace('RNK:', '').strip().lower() == rank_method_code.lower():
solution_recs, solution_scores = \
sort_records_bibsort(req, hitset_global, sort_method,
'', sort_order, verbose, of, ln,
rg, jrec, 'r')
comment = ''
if verbose > 0:
comment = 'find_citations retlist %s' % [[solution_recs[i], solution_scores[i]] for i in range(len(solution_recs))]
return solution_recs, solution_scores, '(', ')', comment
if rank_method_code.lower() == 'citation':
related_to = []
else:
related_to = pattern
solution_recs, solution_scores, prefix, suffix, comment = \
rank_records_bibrank(rank_method_code=rank_method_code,
rank_limit_relevance=rank_limit_relevance,
hitset=hitset_global,
verbose=verbose,
field=field,
related_to=related_to,
rg=rg,
jrec=jrec)
# Solution recs can be None, in case of error or other cases
# which should be all be changed to return an empty list.
if solution_recs and sort_order == 'd':
solution_recs.reverse()
solution_scores.reverse()
return solution_recs, solution_scores, prefix, suffix, comment
+def sort_records_latest(recIDs, jrec, rg, sort_order):
+ if sort_order == 'd':
+ recIDs.reverse()
+ return slice_records(recIDs, jrec, rg)
+
def sort_records(req, recIDs, sort_field='', sort_order='d', sort_pattern='', verbose=0, of='hb', ln=CFG_SITE_LANG, rg=None, jrec=None, sorting_methods=SORTING_METHODS):
"""Initial entry point for sorting records, acts like a dispatcher.
(i) sort_field is in the bsrMETHOD, and thus, the BibSort has sorted the data for this field, so we can use the cache;
(ii)sort_field is not in bsrMETHOD, and thus, the cache does not contain any information regarding this sorting method"""
_ = gettext_set_language(ln)
- # Calculate the min index on the reverted list
- index_min = jrec
-
#bibsort does not handle sort_pattern for now, use bibxxx
if sort_pattern:
return sort_records_bibxxx(req, recIDs, None, sort_field, sort_order, sort_pattern, verbose, of, ln, rg, jrec)
#ignore the use of buckets, use old fashion sorting
use_sorting_buckets = CFG_BIBSORT_ENABLED and sorting_methods
if not sort_field:
if use_sorting_buckets:
return sort_records_bibsort(req, recIDs, 'latest first', sort_field, sort_order, verbose, of, ln, rg, jrec)
else:
- return recIDs[index_min:]
+ return sort_records_latest(recIDs, jrec, rg, sort_order)
- sort_fields = string.split(sort_field, ",")
+ sort_fields = sort_field.split(",")
if len(sort_fields) == 1:
# we have only one sorting_field, check if it is treated by BibSort
for sort_method in sorting_methods:
definition = sorting_methods[sort_method]
if use_sorting_buckets and \
((definition.startswith('FIELD') and
definition.replace('FIELD:', '').strip().lower() == sort_fields[0].lower()) or
sort_method == sort_fields[0]):
#use BibSort
return sort_records_bibsort(req, recIDs, sort_method, sort_field, sort_order, verbose, of, ln, rg, jrec)
#deduce sorting MARC tag out of the 'sort_field' argument:
tags, error_field = get_tags_from_sort_fields(sort_fields)
if error_field:
if use_sorting_buckets:
return sort_records_bibsort(req, recIDs, 'latest first', sort_field, sort_order, verbose, of, ln, rg, jrec)
else:
if of.startswith('h'):
- write_warning(_("Sorry, %(x_name)s does not seem to be a valid sort option. The records will not be sorted.", x_name=cgi.escape(error_field)), "Error", req=req)
- return recIDs[index_min:]
- if tags:
+ write_warning(_("Sorry, %s does not seem to be a valid sort option. The records will not be sorted.") % cgi.escape(error_field), "Error", req=req)
+ return slice_records(recIDs, jrec, rg)
+ elif tags:
for sort_method in sorting_methods:
definition = sorting_methods[sort_method]
if definition.startswith('MARC') \
and definition.replace('MARC:', '').strip().split(',') == tags \
and use_sorting_buckets:
#this list of tags have a designated method in BibSort, so use it
return sort_records_bibsort(req, recIDs, sort_method, sort_field, sort_order, verbose, of, ln, rg, jrec)
#we do not have this sort_field in BibSort tables -> do the old fashion sorting
return sort_records_bibxxx(req, recIDs, tags, sort_field, sort_order, sort_pattern, verbose, of, ln, rg, jrec)
-
- return recIDs[index_min:]
+ else:
+ return slice_records(recIDs, jrec, rg)
def sort_records_bibsort(req, recIDs, sort_method, sort_field='', sort_order='d', verbose=0, of='hb', ln=CFG_SITE_LANG, rg=None, jrec=1, sort_or_rank='s', sorting_methods=SORTING_METHODS):
"""This function orders the recIDs list, based on a sorting method(sort_field) using the BibSortDataCacher for speed"""
_ = gettext_set_language(ln)
if not jrec:
jrec = 1
#sanity check
if sort_method not in sorting_methods:
if sort_or_rank == 'r':
return rank_records_bibrank(rank_method_code=sort_method,
rank_limit_relevance=0,
hitset=recIDs,
verbose=verbose)
else:
return sort_records_bibxxx(req, recIDs, None, sort_field, sort_order, '', verbose, of, ln, rg, jrec)
if verbose >= 3 and of.startswith('h'):
write_warning("Sorting (using BibSort cache) by method %s (definition %s)."
% (cgi.escape(repr(sort_method)), cgi.escape(repr(sorting_methods[sort_method]))), req=req)
#we should return sorted records up to irec_max(exclusive)
dummy, irec_max = get_interval_for_records_to_sort(len(recIDs), jrec, rg)
solution = intbitset()
input_recids = intbitset(recIDs)
CACHE_SORTED_DATA[sort_method].recreate_cache_if_needed()
sort_cache = CACHE_SORTED_DATA[sort_method].cache
bucket_numbers = sort_cache['bucket_data'].keys()
#check if all buckets have been constructed
if len(bucket_numbers) != CFG_BIBSORT_BUCKETS:
if verbose > 3 and of.startswith('h'):
write_warning("Not all buckets have been constructed.. switching to old fashion sorting.", req=req)
if sort_or_rank == 'r':
return rank_records_bibrank(rank_method_code=sort_method,
rank_limit_relevance=0,
hitset=recIDs,
verbose=verbose)
else:
return sort_records_bibxxx(req, recIDs, None, sort_field,
sort_order, '', verbose, of, ln, rg,
jrec)
if sort_order == 'd':
bucket_numbers.reverse()
for bucket_no in bucket_numbers:
solution.union_update(input_recids & sort_cache['bucket_data'][bucket_no])
if len(solution) >= irec_max:
break
dict_solution = {}
missing_records = intbitset()
for recid in solution:
try:
dict_solution[recid] = sort_cache['data_dict_ordered'][recid]
except KeyError:
#recid is in buckets, but not in the bsrMETHODDATA,
#maybe because the value has been deleted, but the change has not yet been propagated to the buckets
missing_records.add(recid)
#check if there are recids that are not in any bucket -> to be added at the end/top, ordered by insertion date
if len(solution) < irec_max:
#some records have not been yet inserted in the bibsort structures
#or, some records have no value for the sort_method
missing_records += input_recids - solution
#the records need to be sorted in reverse order for the print record function
#the return statement should be equivalent with the following statements
#(these are clearer, but less efficient, since they revert the same list twice)
#sorted_solution = (missing_records + sorted(dict_solution, key=dict_solution.__getitem__, reverse=sort_order=='d'))[:irec_max]
#sorted_solution.reverse()
#return sorted_solution
reverse = sort_order == 'd'
if sort_method.strip().lower().startswith('latest') and reverse:
# If we want to sort the records on their insertion date, add the missing records at the top
solution = sorted(dict_solution, key=dict_solution.__getitem__, reverse=True) + sorted(missing_records, reverse=True)
else:
solution = sorted(missing_records) + sorted(dict_solution, key=dict_solution.__getitem__, reverse=reverse)
# Only keep records, we are going to display
index_min = jrec - 1
if rg:
index_max = index_min + rg
solution = solution[index_min:index_max]
else:
solution = solution[index_min:]
if sort_or_rank == 'r':
# We need the recids, with their ranking score
return solution, [dict_solution.get(record, 0) for record in solution]
else:
return solution
+def slice_records(recIDs, jrec, rg):
+ if not jrec:
+ jrec = 1
+ if rg:
+ recIDs = recIDs[jrec-1:jrec-1+rg]
+ else:
+ recIDs = recIDs[jrec-1:]
+ return recIDs
+
def sort_records_bibxxx(req, recIDs, tags, sort_field='', sort_order='d', sort_pattern='', verbose=0, of='hb', ln=CFG_SITE_LANG, rg=None, jrec=None):
"""OLD FASHION SORTING WITH NO CACHE, for sort fields that are not run in BibSort
Sort records in 'recIDs' list according sort field 'sort_field' in order 'sort_order'.
If more than one instance of 'sort_field' is found for a given record, try to choose that that is given by
'sort pattern', for example "sort by report number that starts by CERN-PS".
Note that 'sort_field' can be field code like 'author' or MARC tag like '100__a' directly."""
_ = gettext_set_language(ln)
- #we should return sorted records up to irec_max(exclusive)
- dummy, irec_max = get_interval_for_records_to_sort(len(recIDs), jrec, rg)
- #calculate the min index on the reverted list
- index_min = max(len(recIDs) - irec_max, 0) #just to be sure that the min index is not negative
-
## check arguments:
if not sort_field:
- return recIDs[index_min:]
+ return slice_records(recIDs, jrec, rg)
+
if len(recIDs) > CFG_WEBSEARCH_NB_RECORDS_TO_SORT:
if of.startswith('h'):
write_warning(_("Sorry, sorting is allowed on sets of up to %(x_name)d records only. Using default sort order.", x_name=CFG_WEBSEARCH_NB_RECORDS_TO_SORT), "Warning", req=req)
- return recIDs[index_min:]
+ return slice_records(recIDs, jrec, rg)
+
recIDs_dict = {}
recIDs_out = []
if not tags:
# tags have not been camputed yet
- sort_fields = string.split(sort_field, ",")
+ sort_fields = sort_field.split(',')
tags, error_field = get_tags_from_sort_fields(sort_fields)
if error_field:
if of.startswith('h'):
write_warning(_("Sorry, %(x_name)s does not seem to be a valid sort option. The records will not be sorted.", x_name=cgi.escape(error_field)), "Error", req=req)
- return recIDs[index_min:]
+ return slice_records(recIDs, jrec, rg)
+
if verbose >= 3 and of.startswith('h'):
write_warning("Sorting by tags %s." % cgi.escape(repr(tags)), req=req)
if sort_pattern:
write_warning("Sorting preferentially by %s." % cgi.escape(sort_pattern), req=req)
- ## check if we have sorting tag defined:
+
+ ## check if we have sorting tag defined:
if tags:
# fetch the necessary field values:
for recID in recIDs:
val = "" # will hold value for recID according to which sort
vals = [] # will hold all values found in sorting tag for recID
for tag in tags:
if CFG_CERN_SITE and tag == '773__c':
# CERN hack: journal sorting
# 773__c contains page numbers, e.g. 3-13, and we want to sort by 3, and numerically:
vals.extend(["%050s" % x.split("-", 1)[0] for x in get_fieldvalues(recID, tag)])
else:
vals.extend(get_fieldvalues(recID, tag))
if sort_pattern:
# try to pick that tag value that corresponds to sort pattern
bingo = 0
for v in vals:
if v.lower().startswith(sort_pattern.lower()): # bingo!
bingo = 1
val = v
break
if not bingo: # sort_pattern not present, so add other vals after spaces
- val = sort_pattern + " " + string.join(vals)
+ val = sort_pattern + " " + ''.join(vals)
else:
# no sort pattern defined, so join them all together
- val = string.join(vals)
+ val = ''.join(vals)
val = strip_accents(val.lower()) # sort values regardless of accents and case
if val in recIDs_dict:
recIDs_dict[val].append(recID)
else:
recIDs_dict[val] = [recID]
- # sort them:
- recIDs_dict_keys = recIDs_dict.keys()
- recIDs_dict_keys.sort()
- # now that keys are sorted, create output array:
- for k in recIDs_dict_keys:
- for s in recIDs_dict[k]:
- recIDs_out.append(s)
+
+ # create output array:
+ for k in sorted(recIDs_dict.keys()):
+ recIDs_out.extend(recIDs_dict[k])
+
# ascending or descending?
- if sort_order == 'a':
+ if sort_order == 'd':
recIDs_out.reverse()
- # okay, we are done
- # return only up to the maximum that we need to sort
- if len(recIDs_out) != len(recIDs):
- dummy, irec_max = get_interval_for_records_to_sort(len(recIDs_out), jrec, rg)
- index_min = max(len(recIDs_out) - irec_max, 0) #just to be sure that the min index is not negative
- return recIDs_out[index_min:]
- else:
- # good, no sort needed
- return recIDs[index_min:]
+
+ recIDs = recIDs_out
+
+ # return only up to the maximum that we need
+ return slice_records(recIDs, jrec, rg)
def get_interval_for_records_to_sort(nb_found, jrec=None, rg=None):
"""calculates in which interval should the sorted records be
a value of 'rg=-9999' means to print all records: to be used with care."""
if not jrec:
jrec = 1
if not rg:
#return all
return jrec-1, nb_found
if rg == -9999: # print all records
rg = nb_found
else:
rg = abs(rg)
if jrec < 1: # sanity checks
jrec = 1
if jrec > nb_found:
jrec = max(nb_found-rg+1, 1)
# will sort records from irec_min to irec_max excluded
irec_min = jrec - 1
irec_max = irec_min + rg
if irec_min < 0:
irec_min = 0
if irec_max > nb_found:
irec_max = nb_found
return irec_min, irec_max
def print_records(req, recIDs, jrec=1, rg=CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS, format='hb', ot='', ln=CFG_SITE_LANG,
relevances=[], relevances_prologue="(", relevances_epilogue="%%)",
decompress=zlib.decompress, search_pattern='', print_records_prologue_p=True,
print_records_epilogue_p=True, verbose=0, tab='', sf='', so='d', sp='',
rm='', em=''):
"""
Prints list of records 'recIDs' formatted according to 'format' in
groups of 'rg' starting from 'jrec'.
Assumes that the input list 'recIDs' is sorted in reverse order,
so it counts records from tail to head.
A value of 'rg=-9999' means to print all records: to be used with care.
Print also list of RELEVANCES for each record (if defined), in
between RELEVANCE_PROLOGUE and RELEVANCE_EPILOGUE.
Print prologue and/or epilogue specific to 'format' if
'print_records_prologue_p' and/or print_records_epilogue_p' are
True.
'sf' is sort field and 'rm' is ranking method that are passed here
only for proper linking purposes: e.g. when a certain ranking
method or a certain sort field was selected, keep it selected in
any dynamic search links that may be printed.
"""
if em != "" and EM_REPOSITORY["body"] not in em:
return
# load the right message language
_ = gettext_set_language(ln)
# sanity checking:
if req is None:
return
# get user_info (for formatting based on user)
if isinstance(req, cStringIO.OutputType):
user_info = {}
else:
user_info = collect_user_info(req)
if len(recIDs):
nb_found = len(recIDs)
if not rg or rg == -9999: # print all records
rg = nb_found
else:
rg = abs(rg)
if jrec < 1: # sanity checks
jrec = 1
if jrec > nb_found:
jrec = max(nb_found-rg+1, 1)
# will print records from irec_max to irec_min excluded:
irec_max = nb_found - jrec
irec_min = nb_found - jrec - rg
if irec_min < 0:
irec_min = -1
if irec_max >= nb_found:
irec_max = nb_found - 1
#req.write("%s:%d-%d" % (recIDs, irec_min, irec_max))
if format.startswith('x'):
# print header if needed
if print_records_prologue_p:
print_records_prologue(req, format)
if ot:
# asked to print some filtered fields only, so call print_record() on the fly:
for recid in recIDs:
x = print_record(recid,
format,
ot=ot,
ln=ln,
search_pattern=search_pattern,
user_info=user_info,
verbose=verbose,
sf=sf,
so=so,
sp=sp,
rm=rm)
req.write(x)
if x:
req.write('\n')
else:
format_records(recIDs,
format,
ln=ln,
search_pattern=search_pattern,
record_separator="\n",
user_info=user_info,
req=req)
# print footer if needed
if print_records_epilogue_p:
print_records_epilogue(req, format)
elif format.startswith('t') or str(format[0:3]).isdigit():
# we are doing plain text output:
for recid in recIDs:
x = print_record(recid, format, ot, ln, search_pattern=search_pattern,
user_info=user_info, verbose=verbose, sf=sf, so=so, sp=sp, rm=rm)
req.write(x)
if x:
req.write('\n')
+ elif format.startswith('recjson'):
+ # we are doing recjson output:
+ req.write('[')
+ for idx, recid in enumerate(recIDs):
+ if idx > 0:
+ req.write(',')
+ req.write(print_record(recid, format, ot, ln,
+ search_pattern=search_pattern,
+ user_info=user_info, verbose=verbose,
+ sf=sf, so=so, sp=sp, rm=rm))
+ req.write(']')
elif format == 'excel':
create_excel(recIDs=recIDs, req=req, ot=ot, user_info=user_info)
else:
# we are doing HTML output:
if format == 'hp' or format.startswith("hb_") or format.startswith("hd_"):
# portfolio and on-the-fly formats:
for recid in recIDs:
req.write(print_record(recid,
format,
ot=ot,
ln=ln,
search_pattern=search_pattern,
user_info=user_info,
verbose=verbose,
sf=sf,
so=so,
sp=sp,
rm=rm))
elif format.startswith("hb"):
# HTML brief format:
display_add_to_basket = True
if user_info:
if user_info['email'] == 'guest':
if CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS > 4:
display_add_to_basket = False
else:
if not user_info['precached_usebaskets']:
display_add_to_basket = False
if em != "" and EM_REPOSITORY["basket"] not in em:
display_add_to_basket = False
req.write(websearch_templates.tmpl_record_format_htmlbrief_header(ln=ln))
for irec, recid in enumerate(recIDs):
row_number = jrec+irec
if relevances and relevances[irec]:
relevance = relevances[irec]
else:
relevance = ''
record = print_record(recid,
format,
ot=ot,
ln=ln,
search_pattern=search_pattern,
user_info=user_info,
verbose=verbose,
sf=sf,
so=so,
sp=sp,
rm=rm)
req.write(websearch_templates.tmpl_record_format_htmlbrief_body(
ln=ln,
recid=recid,
row_number=row_number,
relevance=relevance,
record=record,
relevances_prologue=relevances_prologue,
relevances_epilogue=relevances_epilogue,
display_add_to_basket=display_add_to_basket
))
req.write(websearch_templates.tmpl_record_format_htmlbrief_footer(
ln=ln,
display_add_to_basket=display_add_to_basket))
elif format.startswith("hd"):
# HTML detailed format:
for recid in recIDs:
if record_exists(recid) == -1:
write_warning(_("The record has been deleted."), req=req)
merged_recid = get_merged_recid(recid)
if merged_recid:
write_warning(_("The record %(x_rec)d replaces it.", x_rec=merged_recid), req=req)
continue
unordered_tabs = get_detailed_page_tabs(get_colID(guess_primary_collection_of_a_record(recid)),
recid, ln=ln)
ordered_tabs_id = [(tab_id, values['order']) for (tab_id, values) in iteritems(unordered_tabs)]
ordered_tabs_id.sort(lambda x, y: cmp(x[1], y[1]))
link_ln = ''
if ln != CFG_SITE_LANG:
link_ln = '?ln=%s' % ln
recid_to_display = recid # Record ID used to build the URL.
if CFG_WEBSEARCH_USE_ALEPH_SYSNOS:
try:
recid_to_display = get_fieldvalues(recid,
CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG)[0]
except IndexError:
# No external sysno is available, keep using
# internal recid.
pass
tabs = [(unordered_tabs[tab_id]['label'],
'%s/%s/%s/%s%s' % (CFG_BASE_URL, CFG_SITE_RECORD, recid_to_display, tab_id, link_ln),
tab_id == tab,
unordered_tabs[tab_id]['enabled'])
for (tab_id, dummy_order) in ordered_tabs_id
if unordered_tabs[tab_id]['visible'] is True]
tabs_counts = get_detailed_page_tabs_counts(recid)
citedbynum = tabs_counts['Citations']
references = tabs_counts['References']
discussions = tabs_counts['Discussions']
# load content
if tab == 'usage':
- req.write(webstyle_templates.detailed_record_container_top(recIDs[irec],
+ req.write(webstyle_templates.detailed_record_container_top(recid,
tabs,
ln,
citationnum=citedbynum,
referencenum=references,
discussionnum=discussions))
r = calculate_reading_similarity_list(recid, "downloads")
downloadsimilarity = None
downloadhistory = None
#if r:
# downloadsimilarity = r
if CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS:
downloadhistory = create_download_history_graph_and_box(recid, ln)
r = calculate_reading_similarity_list(recid, "pageviews")
viewsimilarity = None
if r:
viewsimilarity = r
content = websearch_templates.tmpl_detailed_record_statistics(recid,
ln,
downloadsimilarity=downloadsimilarity,
downloadhistory=downloadhistory,
viewsimilarity=viewsimilarity)
req.write(content)
req.write(webstyle_templates.detailed_record_container_bottom(recid,
tabs,
ln))
elif tab == 'citations':
req.write(webstyle_templates.detailed_record_container_top(recid,
tabs,
ln,
citationnum=citedbynum,
referencenum=references,
discussionnum=discussions))
req.write(websearch_templates.tmpl_detailed_record_citations_prologue(recid, ln))
# Citing
citinglist = calculate_cited_by_list(recid)
req.write(websearch_templates.tmpl_detailed_record_citations_citing_list(recid,
ln,
citinglist,
sf=sf,
so=so,
sp=sp,
rm=rm))
# Self-cited
selfcited = get_self_cited_by(recid)
+ selfcited = rank_by_citations(get_self_cited_by(recid), verbose=verbose)
+ selfcited = reversed(selfcited[0])
+ selfcited = [recid for recid, dummy in selfcited]
req.write(websearch_templates.tmpl_detailed_record_citations_self_cited(recid,
ln, selfcited=selfcited, citinglist=citinglist))
# Co-cited
s = calculate_co_cited_with_list(recid)
cociting = None
if s:
cociting = s
req.write(websearch_templates.tmpl_detailed_record_citations_co_citing(recid,
ln,
cociting=cociting))
# Citation history, if needed
citationhistory = None
if citinglist:
citationhistory = create_citation_history_graph_and_box(recid, ln)
#debug
if verbose > 3:
write_warning("Citation graph debug: " +
str(len(citationhistory)), req=req)
req.write(websearch_templates.tmpl_detailed_record_citations_citation_history(ln, citationhistory))
# Citation log
entries = get_citers_log(recid)
req.write(websearch_templates.tmpl_detailed_record_citations_citation_log(ln, entries))
req.write(websearch_templates.tmpl_detailed_record_citations_epilogue(recid, ln))
req.write(webstyle_templates.detailed_record_container_bottom(recid,
tabs,
ln))
elif tab == 'references':
req.write(webstyle_templates.detailed_record_container_top(recid,
tabs,
ln,
citationnum=citedbynum,
referencenum=references,
discussionnum=discussions))
req.write(format_record(recid, 'HDREF', ln=ln, user_info=user_info, verbose=verbose, force_2nd_pass=True))
req.write(webstyle_templates.detailed_record_container_bottom(recid,
tabs,
ln))
elif tab == 'keywords':
- from invenio.bibclassify_webinterface import \
- record_get_keywords, write_keywords_body, \
- generate_keywords
- from invenio.webinterface_handler import wash_urlargd
- form = req.form
- argd = wash_urlargd(form, {
- 'generate': (str, 'no'),
- 'sort': (str, 'occurrences'),
- 'type': (str, 'tagcloud'),
- 'numbering': (str, 'off'),
- })
- recid = recIDs[irec]
-
- req.write(webstyle_templates.detailed_record_container_top(recid,
- tabs,
- ln))
- content = websearch_templates.tmpl_record_plots(recID=recid,
- ln=ln)
- req.write(content)
- req.write(webstyle_templates.detailed_record_container_bottom(recid,
- tabs,
- ln))
-
- req.write(webstyle_templates.detailed_record_container_top(recid,
- tabs, ln, citationnum=citedbynum, referencenum=references))
-
- if argd['generate'] == 'yes':
- # The user asked to generate the keywords.
- keywords = generate_keywords(req, recid, argd)
- else:
- # Get the keywords contained in the MARC.
- keywords = record_get_keywords(recid, argd)
-
- if argd['sort'] == 'related' and not keywords:
- req.write('You may want to run BibIndex.')
-
- # Output the keywords or the generate button.
- write_keywords_body(keywords, req, recid, argd)
-
- req.write(webstyle_templates.detailed_record_container_bottom(recid,
- tabs, ln))
+ from invenio.bibclassify_webinterface import main_page
+ main_page(req, recid, tabs, ln,
+ webstyle_templates)
elif tab == 'plots':
req.write(webstyle_templates.detailed_record_container_top(recid,
tabs,
ln))
content = websearch_templates.tmpl_record_plots(recID=recid,
ln=ln)
req.write(content)
req.write(webstyle_templates.detailed_record_container_bottom(recid,
tabs,
ln))
elif tab == 'hepdata':
req.write(webstyle_templates.detailed_record_container_top(recid,
tabs,
ln,
include_jquery=True,
include_mathjax=True))
from invenio import hepdatautils
from invenio import hepdatadisplayutils
data = hepdatautils.retrieve_data_for_record(recid)
if data:
content = websearch_templates.tmpl_record_hepdata(data, recid, True)
else:
content = websearch_templates.tmpl_record_no_hepdata()
req.write(content)
req.write(webstyle_templates.detailed_record_container_bottom(recid,
tabs,
ln))
else:
# Metadata tab
req.write(webstyle_templates.detailed_record_container_top(
recid,
tabs,
ln,
show_short_rec_p=False,
citationnum=citedbynum,
referencenum=references,
discussionnum=discussions))
creationdate = None
modificationdate = None
if record_exists(recid) == 1:
creationdate = get_creation_date(recid)
modificationdate = get_modification_date(recid)
content = print_record(recid, format, ot, ln,
search_pattern=search_pattern,
user_info=user_info, verbose=verbose,
sf=sf, so=so, sp=sp, rm=rm)
content = websearch_templates.tmpl_detailed_record_metadata(
recID=recid,
ln=ln,
format=format,
creationdate=creationdate,
modificationdate=modificationdate,
content=content)
# display of the next-hit/previous-hit/back-to-search links
# on the detailed record pages
content += websearch_templates.tmpl_display_back_to_search(req,
recid,
ln)
req.write(content)
req.write(webstyle_templates.detailed_record_container_bottom(recid,
tabs,
ln,
creationdate=creationdate,
modificationdate=modificationdate,
show_short_rec_p=False))
if len(tabs) > 0:
# Add the mini box at bottom of the page
if CFG_WEBCOMMENT_ALLOW_REVIEWS:
from invenio.modules.comments.api import get_mini_reviews
reviews = get_mini_reviews(recid=recid, ln=ln)
else:
reviews = ''
actions = format_record(recid, 'HDACT', ln=ln, user_info=user_info, verbose=verbose)
files = format_record(recid, 'HDFILE', ln=ln, user_info=user_info, verbose=verbose)
req.write(webstyle_templates.detailed_record_mini_panel(recid,
ln,
format,
files=files,
reviews=reviews,
actions=actions))
else:
# Other formats
for recid in recIDs:
req.write(print_record(recid, format, ot, ln,
search_pattern=search_pattern,
user_info=user_info, verbose=verbose,
sf=sf, so=so, sp=sp, rm=rm))
else:
write_warning(_("Use different search terms."), req=req)
def print_records_prologue(req, format, cc=None):
"""
Print the appropriate prologue for list of records in the given
format.
"""
prologue = "" # no prologue needed for HTML or Text formats
if format.startswith('xm'):
prologue = websearch_templates.tmpl_xml_marc_prologue()
elif format.startswith('xn'):
prologue = websearch_templates.tmpl_xml_nlm_prologue()
elif format.startswith('xw'):
prologue = websearch_templates.tmpl_xml_refworks_prologue()
elif format.startswith('xr'):
prologue = websearch_templates.tmpl_xml_rss_prologue(cc=cc)
elif format.startswith('xe8x'):
prologue = websearch_templates.tmpl_xml_endnote_8x_prologue()
elif format.startswith('xe'):
prologue = websearch_templates.tmpl_xml_endnote_prologue()
elif format.startswith('xo'):
prologue = websearch_templates.tmpl_xml_mods_prologue()
elif format.startswith('xp'):
prologue = websearch_templates.tmpl_xml_podcast_prologue(cc=cc)
elif format.startswith('x'):
prologue = websearch_templates.tmpl_xml_default_prologue()
req.write(prologue)
def print_records_epilogue(req, format):
"""
Print the appropriate epilogue for list of records in the given
format.
"""
epilogue = "" # no epilogue needed for HTML or Text formats
if format.startswith('xm'):
epilogue = websearch_templates.tmpl_xml_marc_epilogue()
elif format.startswith('xn'):
epilogue = websearch_templates.tmpl_xml_nlm_epilogue()
elif format.startswith('xw'):
epilogue = websearch_templates.tmpl_xml_refworks_epilogue()
elif format.startswith('xr'):
epilogue = websearch_templates.tmpl_xml_rss_epilogue()
elif format.startswith('xe8x'):
epilogue = websearch_templates.tmpl_xml_endnote_8x_epilogue()
elif format.startswith('xe'):
epilogue = websearch_templates.tmpl_xml_endnote_epilogue()
elif format.startswith('xo'):
epilogue = websearch_templates.tmpl_xml_mods_epilogue()
elif format.startswith('xp'):
epilogue = websearch_templates.tmpl_xml_podcast_epilogue()
elif format.startswith('x'):
epilogue = websearch_templates.tmpl_xml_default_epilogue()
req.write(epilogue)
def get_record(recid):
"""Directly the record object corresponding to the recid."""
if CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE:
value = run_sql("SELECT value FROM bibfmt WHERE id_bibrec=%s AND FORMAT='recstruct'", (recid, ))
if value:
try:
val = value[0][0]
except IndexError:
### In case it does not exist, let's build it!
pass
else:
return deserialize_via_marshal(val)
return create_record(print_record(recid, 'xm'))[0]
def print_record(recID, format='hb', ot='', ln=CFG_SITE_LANG, decompress=zlib.decompress,
search_pattern=None, user_info=None, verbose=0, sf='', so='d',
sp='', rm='', brief_links=True):
"""
Prints record 'recID' formatted according to 'format'.
'sf' is sort field and 'rm' is ranking method that are passed here
only for proper linking purposes: e.g. when a certain ranking
method or a certain sort field was selected, keep it selected in
any dynamic search links that may be printed.
"""
if format == 'recstruct':
return get_record(recID)
+ if format.startswith('recjson'):
+ import json
+ from invenio.bibfield import get_record as get_recjson
+ if ot:
+ ot = list(set(ot) - set(CFG_BIBFORMAT_HIDDEN_TAGS))
+ return json.dumps(dict(get_recjson(recID, fields=ot)))
+ else:
+ return json.dumps(get_recjson(recID).dumps())
+
_ = gettext_set_language(ln)
# The 'attribute this paper' link is shown only if the session states it should and
# the record is included in the collections to which bibauthorid is limited.
if user_info:
display_claim_this_paper = (user_info.get("precached_viewclaimlink", False) and
recID in intbitset.union(*[get_collection_reclist(x)
for x in BIBAUTHORID_LIMIT_TO_COLLECTIONS]))
else:
display_claim_this_paper = False
#check from user information if the user has the right to see hidden fields/tags in the
#records as well
can_see_hidden = False
if user_info:
can_see_hidden = user_info.get('precached_canseehiddenmarctags', False)
can_edit_record = False
if check_user_can_edit_record(user_info, recID):
can_edit_record = True
out = ""
# sanity check:
record_exist_p = record_exists(recID)
if record_exist_p == 0: # doesn't exist
return out
# We must still check some special formats, but these
# should disappear when BibFormat improves.
if not (format.lower().startswith('t')
or format.lower().startswith('hm')
or str(format[0:3]).isdigit()
or ot):
# Unspecified format is hd
if format == '':
format = 'hd'
if record_exist_p == -1 and get_output_format_content_type(format) == 'text/html':
# HTML output displays a default value for deleted records.
# Other format have to deal with it.
out += _("The record has been deleted.")
# was record deleted-but-merged ?
merged_recid = get_merged_recid(recID)
if merged_recid:
out += ' ' + _("The record %(x_rec)d replaces it.", x_rec=merged_recid)
else:
out += call_bibformat(recID, format, ln, search_pattern=search_pattern,
user_info=user_info, verbose=verbose)
# at the end of HTML brief mode, print the "Detailed record" functionality:
if brief_links and format.lower().startswith('hb') and \
format.lower() != 'hb_p':
out += websearch_templates.tmpl_print_record_brief_links(ln=ln,
recID=recID,
sf=sf,
so=so,
sp=sp,
rm=rm,
display_claim_link=display_claim_this_paper,
display_edit_link=can_edit_record)
return out
if format == "marcxml" or format == "oai_dc":
out += " <record>\n"
out += " <header>\n"
for oai_id in get_fieldvalues(recID, CFG_OAI_ID_FIELD):
out += " <identifier>%s</identifier>\n" % oai_id
out += " <datestamp>%s</datestamp>\n" % get_modification_date(recID)
out += " </header>\n"
out += " <metadata>\n"
if format.startswith("xm") or format == "marcxml":
# look for detailed format existence:
query = "SELECT value FROM bibfmt WHERE id_bibrec=%s AND format=%s"
res = run_sql(query, (recID, format), 1)
if res and record_exist_p == 1 and not ot:
# record 'recID' is formatted in 'format', and we are not
# asking for field-filtered output; so print it:
out += "%s" % decompress(res[0][0])
elif ot:
# field-filtered output was asked for; print only some fields
if not can_see_hidden:
ot = list(set(ot) - set(cfg['CFG_BIBFORMAT_HIDDEN_TAGS']))
out += record_xml_output(get_record(recID), ot)
else:
# record 'recID' is not formatted in 'format' or we ask
# for field-filtered output -- they are not in "bibfmt"
# table; so fetch all the data from "bibXXx" tables:
if format == "marcxml":
out += """ <record xmlns="http://www.loc.gov/MARC21/slim">\n"""
out += " <controlfield tag=\"001\">%d</controlfield>\n" % int(recID)
elif format.startswith("xm"):
out += """ <record>\n"""
out += " <controlfield tag=\"001\">%d</controlfield>\n" % int(recID)
if record_exist_p == -1:
# deleted record, so display only OAI ID and 980:
oai_ids = get_fieldvalues(recID, CFG_OAI_ID_FIELD)
if oai_ids:
out += "<datafield tag=\"%s\" ind1=\"%s\" ind2=\"%s\"><subfield code=\"%s\">%s</subfield></datafield>\n" % \
(CFG_OAI_ID_FIELD[0:3], CFG_OAI_ID_FIELD[3:4], CFG_OAI_ID_FIELD[4:5], CFG_OAI_ID_FIELD[5:6], oai_ids[0])
out += "<datafield tag=\"980\" ind1=\"\" ind2=\"\"><subfield code=\"c\">DELETED</subfield></datafield>\n"
else:
# controlfields
query = "SELECT b.tag,b.value,bb.field_number FROM bib00x AS b, bibrec_bib00x AS bb "\
"WHERE bb.id_bibrec=%s AND b.id=bb.id_bibxxx AND b.tag LIKE '00%%' "\
"ORDER BY bb.field_number, b.tag ASC"
res = run_sql(query, (recID, ))
for row in res:
field, value = row[0], row[1]
value = encode_for_xml(value)
out += """ <controlfield tag="%s">%s</controlfield>\n""" % \
(encode_for_xml(field[0:3]), value)
# datafields
i = 1 # Do not process bib00x and bibrec_bib00x, as
# they are controlfields. So start at bib01x and
# bibrec_bib00x (and set i = 0 at the end of
# first loop)
for digit1 in range(0, 10):
for digit2 in range(i, 10):
bx = "bib%d%dx" % (digit1, digit2)
bibx = "bibrec_bib%d%dx" % (digit1, digit2)
query = "SELECT b.tag,b.value,bb.field_number FROM %s AS b, %s AS bb "\
"WHERE bb.id_bibrec=%%s AND b.id=bb.id_bibxxx AND b.tag LIKE %%s"\
"ORDER BY bb.field_number, b.tag ASC" % (bx, bibx)
res = run_sql(query, (recID, str(digit1)+str(digit2)+'%'))
field_number_old = -999
field_old = ""
for row in res:
field, value, field_number = row[0], row[1], row[2]
ind1, ind2 = field[3], field[4]
if ind1 == "_" or ind1 == "":
ind1 = " "
if ind2 == "_" or ind2 == "":
ind2 = " "
# print field tag, unless hidden
printme = True
if not can_see_hidden:
for htag in cfg['CFG_BIBFORMAT_HIDDEN_TAGS']:
ltag = len(htag)
samelenfield = field[0:ltag]
if samelenfield == htag:
printme = False
if printme:
if field_number != field_number_old or field[:-1] != field_old[:-1]:
if field_number_old != -999:
out += """ </datafield>\n"""
out += """ <datafield tag="%s" ind1="%s" ind2="%s">\n""" % \
(encode_for_xml(field[0:3]), encode_for_xml(ind1), encode_for_xml(ind2))
field_number_old = field_number
field_old = field
# print subfield value
value = encode_for_xml(value)
out += """ <subfield code="%s">%s</subfield>\n""" % \
(encode_for_xml(field[-1:]), value)
# all fields/subfields printed in this run, so close the tag:
if field_number_old != -999:
out += """ </datafield>\n"""
i = 0 # Next loop should start looking at bib%0 and bibrec_bib00x
# we are at the end of printing the record:
out += " </record>\n"
elif format == "xd" or format == "oai_dc":
# XML Dublin Core format, possibly OAI -- select only some bibXXx fields:
out += """ <dc xmlns="http://purl.org/dc/elements/1.1/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://purl.org/dc/elements/1.1/
http://www.openarchives.org/OAI/1.1/dc.xsd">\n"""
if record_exist_p == -1:
out += ""
else:
for f in get_fieldvalues(recID, "041__a"):
out += " <language>%s</language>\n" % f
for f in get_fieldvalues(recID, "100__a"):
out += " <creator>%s</creator>\n" % encode_for_xml(f)
for f in get_fieldvalues(recID, "700__a"):
out += " <creator>%s</creator>\n" % encode_for_xml(f)
for f in get_fieldvalues(recID, "245__a"):
out += " <title>%s</title>\n" % encode_for_xml(f)
for f in get_fieldvalues(recID, "65017a"):
out += " <subject>%s</subject>\n" % encode_for_xml(f)
for f in get_fieldvalues(recID, "8564_u"):
if f.split('.') == 'png':
continue
out += " <identifier>%s</identifier>\n" % encode_for_xml(f)
for f in get_fieldvalues(recID, "520__a"):
out += " <description>%s</description>\n" % encode_for_xml(f)
out += " <date>%s</date>\n" % get_creation_date(recID)
out += " </dc>\n"
elif len(format) == 6 and str(format[0:3]).isdigit():
# user has asked to print some fields only
if format == "001":
out += "<!--%s-begin-->%s<!--%s-end-->\n" % (format, recID, format)
else:
vals = get_fieldvalues(recID, format)
for val in vals:
out += "<!--%s-begin-->%s<!--%s-end-->\n" % (format, val, format)
elif format.startswith('t'):
## user directly asked for some tags to be displayed only
if record_exist_p == -1:
out += get_fieldvalues_alephseq_like(recID, ["001", CFG_OAI_ID_FIELD, "980"], can_see_hidden)
else:
out += get_fieldvalues_alephseq_like(recID, ot, can_see_hidden)
elif format == "hm":
if record_exist_p == -1:
out += "\n<pre style=\"margin: 1em 0px;\">" + cgi.escape(get_fieldvalues_alephseq_like(recID, ["001", CFG_OAI_ID_FIELD, "980"], can_see_hidden)) + "</pre>"
else:
out += "\n<pre style=\"margin: 1em 0px;\">" + cgi.escape(get_fieldvalues_alephseq_like(recID, ot, can_see_hidden)) + "</pre>"
elif format.startswith("h") and ot:
## user directly asked for some tags to be displayed only
if record_exist_p == -1:
out += "\n<pre>" + get_fieldvalues_alephseq_like(recID, ["001", CFG_OAI_ID_FIELD, "980"], can_see_hidden) + "</pre>"
else:
out += "\n<pre>" + get_fieldvalues_alephseq_like(recID, ot, can_see_hidden) + "</pre>"
elif format == "hd":
# HTML detailed format
if record_exist_p == -1:
out += _("The record has been deleted.")
else:
# look for detailed format existence:
query = "SELECT value FROM bibfmt WHERE id_bibrec=%s AND format=%s"
res = run_sql(query, (recID, format), 1)
if res:
# record 'recID' is formatted in 'format', so print it
out += "%s" % decompress(res[0][0])
else:
# record 'recID' is not formatted in 'format', so try to call BibFormat on the fly or use default format:
out_record_in_format = call_bibformat(recID, format, ln, search_pattern=search_pattern,
user_info=user_info, verbose=verbose)
if out_record_in_format:
out += out_record_in_format
else:
out += websearch_templates.tmpl_print_record_detailed(
ln = ln,
recID = recID,
)
elif format.startswith("hb_") or format.startswith("hd_"):
# underscore means that HTML brief/detailed formats should be called on-the-fly; suitable for testing formats
if record_exist_p == -1:
out += _("The record has been deleted.")
else:
out += call_bibformat(recID, format, ln, search_pattern=search_pattern,
user_info=user_info, verbose=verbose)
elif format.startswith("hx"):
# BibTeX format, called on the fly:
if record_exist_p == -1:
out += _("The record has been deleted.")
else:
out += call_bibformat(recID, format, ln, search_pattern=search_pattern,
user_info=user_info, verbose=verbose)
elif format.startswith("hs"):
# for citation/download similarity navigation links:
if record_exist_p == -1:
out += _("The record has been deleted.")
else:
out += '<a href="%s">' % websearch_templates.build_search_url(recid=recID, ln=ln)
# firstly, title:
titles = get_fieldvalues(recID, "245__a")
if titles:
for title in titles:
out += "<strong>%s</strong>" % title
else:
# usual title not found, try conference title:
titles = get_fieldvalues(recID, "111__a")
if titles:
for title in titles:
out += "<strong>%s</strong>" % title
else:
# just print record ID:
out += "<strong>%s %d</strong>" % (get_field_i18nname("record ID", ln, False), recID)
out += "</a>"
# secondly, authors:
authors = get_fieldvalues(recID, "100__a") + get_fieldvalues(recID, "700__a")
if authors:
out += " - %s" % authors[0]
if len(authors) > 1:
out += " <em>et al</em>"
# thirdly publication info:
publinfos = get_fieldvalues(recID, "773__s")
if not publinfos:
publinfos = get_fieldvalues(recID, "909C4s")
if not publinfos:
publinfos = get_fieldvalues(recID, "037__a")
if not publinfos:
publinfos = get_fieldvalues(recID, "088__a")
if publinfos:
out += " - %s" % publinfos[0]
else:
# fourthly publication year (if not publication info):
years = get_fieldvalues(recID, "773__y")
if not years:
years = get_fieldvalues(recID, "909C4y")
if not years:
years = get_fieldvalues(recID, "260__c")
if years:
out += " (%s)" % years[0]
else:
# HTML brief format by default
if record_exist_p == -1:
out += _("The record has been deleted.")
else:
query = "SELECT value FROM bibfmt WHERE id_bibrec=%s AND format=%s"
res = run_sql(query, (recID, format))
if res:
# record 'recID' is formatted in 'format', so print it
out += "%s" % decompress(res[0][0])
else:
# record 'recID' is not formatted in 'format', so try to call BibFormat on the fly: or use default format:
if CFG_WEBSEARCH_CALL_BIBFORMAT:
out_record_in_format = call_bibformat(recID, format, ln, search_pattern=search_pattern,
user_info=user_info, verbose=verbose)
if out_record_in_format:
out += out_record_in_format
else:
out += websearch_templates.tmpl_print_record_brief(
ln = ln,
recID = recID,
)
else:
out += websearch_templates.tmpl_print_record_brief(
ln = ln,
recID = recID,
)
# at the end of HTML brief mode, print the "Detailed record" functionality:
if format == 'hp' or format.startswith("hb_") or format.startswith("hd_"):
pass # do nothing for portfolio and on-the-fly formats
else:
out += websearch_templates.tmpl_print_record_brief_links(ln=ln,
recID=recID,
sf=sf,
so=so,
sp=sp,
rm=rm,
display_claim_link=display_claim_this_paper,
display_edit_link=can_edit_record)
# print record closing tags, if needed:
if format == "marcxml" or format == "oai_dc":
out += " </metadata>\n"
out += " </record>\n"
return out
def call_bibformat(recID, format="HD", ln=CFG_SITE_LANG, search_pattern=None, user_info=None, verbose=0):
"""
Calls BibFormat and returns formatted record.
BibFormat will decide by itself if old or new BibFormat must be used.
"""
from invenio.modules.formatter.utils import get_pdf_snippets
keywords = []
if search_pattern is not None:
for unit in create_basic_search_units(None, str(search_pattern), None):
bsu_o, bsu_p, bsu_f, bsu_m = unit[0], unit[1], unit[2], unit[3]
if (bsu_o != '-' and bsu_f in [None, 'fulltext']):
if bsu_m == 'a' and bsu_p.startswith('%') and bsu_p.endswith('%'):
# remove leading and training `%' representing partial phrase search
keywords.append(bsu_p[1:-1])
else:
keywords.append(bsu_p)
out = format_record(recID,
of=format,
ln=ln,
search_pattern=keywords,
user_info=user_info,
verbose=verbose)
if CFG_WEBSEARCH_FULLTEXT_SNIPPETS and user_info and \
'fulltext' in user_info['uri'].lower():
# check snippets only if URL contains fulltext
# FIXME: make it work for CLI too, via new function arg
if keywords:
snippets = ''
try:
snippets = get_pdf_snippets(recID, keywords, user_info)
except:
register_exception()
if snippets:
out += snippets
return out
def log_query(hostname, query_args, uid=-1):
"""
Log query into the query and user_query tables.
Return id_query or None in case of problems.
"""
id_query = None
if uid >= 0:
# log the query only if uid is reasonable
res = run_sql("SELECT id FROM query WHERE urlargs=%s", (query_args,), 1)
try:
id_query = res[0][0]
except IndexError:
id_query = run_sql("INSERT INTO query (type, urlargs) VALUES ('r', %s)", (query_args,))
if id_query:
run_sql("INSERT INTO user_query (id_user, id_query, hostname, date) VALUES (%s, %s, %s, %s)",
(uid, id_query, hostname,
time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())))
return id_query
def log_query_info(action, p, f, colls, nb_records_found_total=-1):
"""Write some info to the log file for later analysis."""
try:
log = open(CFG_LOGDIR + "/search.log", "a")
log.write(time.strftime("%Y%m%d%H%M%S#", time.localtime()))
log.write(action+"#")
log.write(p+"#")
log.write(f+"#")
for coll in colls[:-1]:
log.write("%s," % coll)
log.write("%s#" % colls[-1])
log.write("%d" % nb_records_found_total)
log.write("\n")
log.close()
except:
pass
return
def clean_dictionary(dictionary, list_of_items):
"""Returns a copy of the dictionary with all the items
in the list_of_items as empty strings"""
out_dictionary = dictionary.copy()
out_dictionary.update((item, '') for item in list_of_items)
return out_dictionary
### CALLABLES
def perform_request_search(req=None, cc=CFG_SITE_NAME, c=None, p="", f="", rg=None, sf="", so="a", sp="", rm="", of="id", ot="", aas=0,
p1="", f1="", m1="", op1="", p2="", f2="", m2="", op2="", p3="", f3="", m3="", sc=0, jrec=0,
recid=-1, recidb=-1, sysno="", id=-1, idb=-1, sysnb="", action="", d1="",
d1y=0, d1m=0, d1d=0, d2="", d2y=0, d2m=0, d2d=0, dt="", verbose=0, ap=0, ln=CFG_SITE_LANG, ec=None, tab="",
wl=0, em=""):
"""Perform search or browse request, without checking for
authentication. Return list of recIDs found, if of=id.
Otherwise create web page.
The arguments are as follows:
req - mod_python Request class instance.
cc - current collection (e.g. "ATLAS"). The collection the
user started to search/browse from.
c - collection list (e.g. ["Theses", "Books"]). The
collections user may have selected/deselected when
starting to search from 'cc'.
p - pattern to search for (e.g. "ellis and muon or kaon").
f - field to search within (e.g. "author").
rg - records in groups of (e.g. "10"). Defines how many hits
per collection in the search results page are
displayed. (Note that `rg' is ignored in case of `of=id'.)
sf - sort field (e.g. "title").
so - sort order ("a"=ascending, "d"=descending).
sp - sort pattern (e.g. "CERN-") -- in case there are more
values in a sort field, this argument tells which one
to prefer
rm - ranking method (e.g. "jif"). Defines whether results
should be ranked by some known ranking method.
of - output format (e.g. "hb"). Usually starting "h" means
HTML output (and "hb" for HTML brief, "hd" for HTML
detailed), "x" means XML output, "t" means plain text
output, "id" means no output at all but to return list
of recIDs found, "intbitset" means to return an intbitset
representation of the recIDs found (no sorting or ranking
will be performed). (Suitable for high-level API.)
ot - output only these MARC tags (e.g. "100,700,909C0b").
Useful if only some fields are to be shown in the
output, e.g. for library to control some fields.
em - output only part of the page.
aas - advanced search ("0" means no, "1" means yes). Whether
search was called from within the advanced search
interface.
p1 - first pattern to search for in the advanced search
interface. Much like 'p'.
f1 - first field to search within in the advanced search
interface. Much like 'f'.
m1 - first matching type in the advanced search interface.
("a" all of the words, "o" any of the words, "e" exact
phrase, "p" partial phrase, "r" regular expression).
op1 - first operator, to join the first and the second unit
in the advanced search interface. ("a" add, "o" or,
"n" not).
p2 - second pattern to search for in the advanced search
interface. Much like 'p'.
f2 - second field to search within in the advanced search
interface. Much like 'f'.
m2 - second matching type in the advanced search interface.
("a" all of the words, "o" any of the words, "e" exact
phrase, "p" partial phrase, "r" regular expression).
op2 - second operator, to join the second and the third unit
in the advanced search interface. ("a" add, "o" or,
"n" not).
p3 - third pattern to search for in the advanced search
interface. Much like 'p'.
f3 - third field to search within in the advanced search
interface. Much like 'f'.
m3 - third matching type in the advanced search interface.
("a" all of the words, "o" any of the words, "e" exact
phrase, "p" partial phrase, "r" regular expression).
sc - split by collection ("0" no, "1" yes). Governs whether
we want to present the results in a single huge list,
or splitted by collection.
jrec - jump to record (e.g. "234"). Used for navigation
inside the search results. (Note that `jrec' is ignored
in case of `of=id'.)
recid - display record ID (e.g. "20000"). Do not
search/browse but go straight away to the Detailed
record page for the given recID.
recidb - display record ID bis (e.g. "20010"). If greater than
'recid', then display records from recid to recidb.
Useful for example for dumping records from the
database for reformatting.
sysno - display old system SYS number (e.g. ""). If you
migrate to Invenio from another system, and store your
old SYS call numbers, you can use them instead of recid
if you wish so.
id - the same as recid, in case recid is not set. For
backwards compatibility.
idb - the same as recid, in case recidb is not set. For
backwards compatibility.
sysnb - the same as sysno, in case sysno is not set. For
backwards compatibility.
action - action to do. "SEARCH" for searching, "Browse" for
browsing. Default is to search.
d1 - first datetime in full YYYY-mm-dd HH:MM:DD format
(e.g. "1998-08-23 12:34:56"). Useful for search limits
on creation/modification date (see 'dt' argument
below). Note that 'd1' takes precedence over d1y, d1m,
d1d if these are defined.
d1y - first date's year (e.g. "1998"). Useful for search
limits on creation/modification date.
d1m - first date's month (e.g. "08"). Useful for search
limits on creation/modification date.
d1d - first date's day (e.g. "23"). Useful for search
limits on creation/modification date.
d2 - second datetime in full YYYY-mm-dd HH:MM:DD format
(e.g. "1998-09-02 12:34:56"). Useful for search limits
on creation/modification date (see 'dt' argument
below). Note that 'd2' takes precedence over d2y, d2m,
d2d if these are defined.
d2y - second date's year (e.g. "1998"). Useful for search
limits on creation/modification date.
d2m - second date's month (e.g. "09"). Useful for search
limits on creation/modification date.
d2d - second date's day (e.g. "02"). Useful for search
limits on creation/modification date.
dt - first and second date's type (e.g. "c"). Specifies
whether to search in creation dates ("c") or in
modification dates ("m"). When dt is not set and d1*
and d2* are set, the default is "c".
verbose - verbose level (0=min, 9=max). Useful to print some
internal information on the searching process in case
something goes wrong.
ap - alternative patterns (0=no, 1=yes). In case no exact
match is found, the search engine can try alternative
patterns e.g. to replace non-alphanumeric characters by
a boolean query. ap defines if this is wanted.
ln - language of the search interface (e.g. "en"). Useful
for internationalization.
ec - list of external search engines to search as well
(e.g. "SPIRES HEP").
wl - wildcard limit (ex: 100) the wildcard queries will be
limited at 100 results
"""
kwargs = prs_wash_arguments(req=req, cc=cc, c=c, p=p, f=f, rg=rg, sf=sf, so=so, sp=sp, rm=rm, of=of, ot=ot, aas=aas,
p1=p1, f1=f1, m1=m1, op1=op1, p2=p2, f2=f2, m2=m2, op2=op2, p3=p3, f3=f3, m3=m3, sc=sc, jrec=jrec,
recid=recid, recidb=recidb, sysno=sysno, id=id, idb=idb, sysnb=sysnb, action=action, d1=d1,
d1y=d1y, d1m=d1m, d1d=d1d, d2=d2, d2y=d2y, d2m=d2m, d2d=d2d, dt=dt, verbose=verbose, ap=ap, ln=ln, ec=ec,
tab=tab, wl=wl, em=em)
return prs_perform_search(kwargs=kwargs, **kwargs)
def prs_perform_search(kwargs=None, **dummy):
"""Internal call which does the search, it is calling standard Invenio;
Unless you know what you are doing, don't use this call as an API
"""
# separately because we can call it independently
out = prs_wash_arguments_colls(kwargs=kwargs, **kwargs)
if not out:
return out
return prs_search(kwargs=kwargs, **kwargs)
def prs_wash_arguments_colls(kwargs=None, of=None, req=None, cc=None, c=None, sc=None, verbose=None,
aas=None, ln=None, em="", **dummy):
"""
Check and wash collection list argument before we start searching.
If there are troubles, e.g. a collection is not defined, print
warning to the browser.
@return: True if collection list is OK, and various False values
(empty string, empty list) if there was an error.
"""
# raise an exception when trying to print out html from the cli
if of.startswith("h"):
assert req
# for every search engine request asking for an HTML output, we
# first regenerate cache of collection and field I18N names if
# needed; so that later we won't bother checking timestamps for
# I18N names at all:
if of.startswith("h"):
collection_i18nname_cache.recreate_cache_if_needed()
field_i18nname_cache.recreate_cache_if_needed()
try:
(cc, colls_to_display, colls_to_search, hosted_colls, wash_colls_debug) = wash_colls(cc, c, sc, verbose) # which colls to search and to display?
kwargs['colls_to_display'] = colls_to_display
kwargs['colls_to_search'] = colls_to_search
kwargs['hosted_colls'] = hosted_colls
kwargs['wash_colls_debug'] = wash_colls_debug
except InvenioWebSearchUnknownCollectionError as exc:
colname = exc.colname
if of.startswith("h"):
page_start(req, of, cc, aas, ln, getUid(req),
websearch_templates.tmpl_collection_not_found_page_title(colname, ln))
req.write(websearch_templates.tmpl_collection_not_found_page_body(colname, ln))
page_end(req, of, ln, em)
return ''
elif of == "id":
return []
elif of == "intbitset":
return intbitset()
+ elif of == "recjson":
+ return []
elif of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
page_end(req, of, ln, em)
return ''
else:
page_end(req, of, ln, em)
return ''
return True
def prs_wash_arguments(req=None, cc=CFG_SITE_NAME, c=None, p="", f="", rg=CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS,
sf="", so="d", sp="", rm="", of="id", ot="", aas=0,
p1="", f1="", m1="", op1="", p2="", f2="", m2="", op2="", p3="", f3="", m3="",
sc=0, jrec=0, recid=-1, recidb=-1, sysno="", id=-1, idb=-1, sysnb="", action="", d1="",
d1y=0, d1m=0, d1d=0, d2="", d2y=0, d2m=0, d2d=0, dt="", verbose=0, ap=0, ln=CFG_SITE_LANG,
ec=None, tab="", uid=None, wl=0, em="", **dummy):
"""
Sets the (default) values and checks others for the PRS call
"""
# wash output format:
of = wash_output_format(of)
# wash all arguments requiring special care
p = wash_pattern(p)
f = wash_field(f)
p1 = wash_pattern(p1)
f1 = wash_field(f1)
p2 = wash_pattern(p2)
f2 = wash_field(f2)
p3 = wash_pattern(p3)
f3 = wash_field(f3)
(d1y, d1m, d1d, d2y, d2m, d2d) = map(int, (d1y, d1m, d1d, d2y, d2m, d2d))
datetext1, datetext2 = wash_dates(d1, d1y, d1m, d1d, d2, d2y, d2m, d2d)
# wash ranking method:
if not is_method_valid(None, rm):
rm = ""
# backwards compatibility: id, idb, sysnb -> recid, recidb, sysno (if applicable)
if sysnb != "" and sysno == "":
sysno = sysnb
if id > 0 and recid == -1:
recid = id
if idb > 0 and recidb == -1:
recidb = idb
# TODO deduce passed search limiting criterias (if applicable)
pl, pl_in_url = "", "" # no limits by default
if action != "browse" and req and not isinstance(req, cStringIO.OutputType) \
and req.args and not isinstance(req.args, dict): # we do not want to add options while browsing or while calling via command-line
fieldargs = cgi.parse_qs(req.args)
for fieldcode in get_fieldcodes():
if fieldcode in fieldargs:
for val in fieldargs[fieldcode]:
pl += "+%s:\"%s\" " % (fieldcode, val)
pl_in_url += "&amp;%s=%s" % (urllib.quote(fieldcode), urllib.quote(val))
# deduce recid from sysno argument (if applicable):
if sysno: # ALEPH SYS number was passed, so deduce DB recID for the record:
recid = get_mysql_recid_from_aleph_sysno(sysno)
if recid is None:
recid = 0 # use recid 0 to indicate that this sysno does not exist
# deduce collection we are in (if applicable):
if recid > 0:
referer = None
if req:
referer = req.headers_in.get('Referer')
cc = guess_collection_of_a_record(recid, referer)
# deduce user id (if applicable):
if uid is None:
try:
uid = getUid(req)
except:
uid = 0
_ = gettext_set_language(ln)
kwargs = {'req': req, 'cc': cc, 'c': c, 'p': p, 'f': f, 'rg': rg, 'sf': sf,
'so': so, 'sp': sp, 'rm': rm, 'of': of, 'ot': ot, 'aas': aas,
'p1': p1, 'f1': f1, 'm1': m1, 'op1': op1, 'p2': p2, 'f2': f2,
'm2': m2, 'op2': op2, 'p3': p3, 'f3': f3, 'm3': m3, 'sc': sc,
'jrec': jrec, 'recid': recid, 'recidb': recidb, 'sysno': sysno,
'id': id, 'idb': idb, 'sysnb': sysnb, 'action': action, 'd1': d1,
'd1y': d1y, 'd1m': d1m, 'd1d': d1d, 'd2': d2, 'd2y': d2y,
'd2m': d2m, 'd2d': d2d, 'dt': dt, 'verbose': verbose, 'ap': ap,
'ln': ln, 'ec': ec, 'tab': tab, 'wl': wl, 'em': em,
'datetext1': datetext1, 'datetext2': datetext2, 'uid': uid,
'pl': pl, 'pl_in_url': pl_in_url, '_': _,
'selected_external_collections_infos': None,
}
kwargs.update(**dummy)
return kwargs
def prs_search(kwargs=None, recid=0, req=None, cc=None, p=None, p1=None, p2=None, p3=None,
f=None, ec=None, verbose=None, ln=None, selected_external_collections_infos=None,
action=None, rm=None, of=None, em=None,
**dummy):
"""
This function write various bits into the req object as the search
proceeds (so that pieces of a page are rendered even before the
search ended)
"""
## 0 - start output
if recid >= 0: # recid can be 0 if deduced from sysno and if such sysno does not exist
output = prs_detailed_record(kwargs=kwargs, **kwargs)
if output is not None:
return output
elif action == "browse":
## 2 - browse needed
of = 'hb'
output = prs_browse(kwargs=kwargs, **kwargs)
if output is not None:
return output
elif rm and p.startswith("recid:"):
## 3-ter - similarity search (or old-style citation search) needed
output = prs_search_similar_records(kwargs=kwargs, **kwargs)
if output is not None:
return output
elif p.startswith("cocitedwith:"): #WAS EXPERIMENTAL
## 3-terter - cited by search needed
output = prs_search_cocitedwith(kwargs=kwargs, **kwargs)
if output is not None:
return output
else:
## 3 - common search needed
output = prs_search_common(kwargs=kwargs, **kwargs)
if output is not None:
return output
# External searches
if of.startswith("h"):
if not of in ['hcs', 'hcs2']:
perform_external_collection_search_with_em(req, cc, [p, p1, p2, p3], f, ec, verbose,
ln, selected_external_collections_infos, em=em)
return page_end(req, of, ln, em)
def prs_detailed_record(kwargs=None, req=None, of=None, cc=None, aas=None, ln=None, uid=None, recid=None, recidb=None,
p=None, verbose=None, tab=None, sf=None, so=None, sp=None, rm=None, ot=None, _=None, em=None,
**dummy):
"""Formats and prints one record"""
## 1 - detailed record display
title, description, keywords = \
websearch_templates.tmpl_record_page_header_content(req, recid, ln)
if req is not None and req.method != 'HEAD':
page_start(req, of, cc, aas, ln, uid, title, description, keywords, recid, tab, em)
# Default format is hb but we are in detailed -> change 'of'
if of == "hb":
of = "hd"
if record_exists(recid):
if recidb <= recid: # sanity check
recidb = recid + 1
if of in ["id", "intbitset"]:
result = [recidx for recidx in range(recid, recidb) if record_exists(recidx)]
if of == "intbitset":
return intbitset(result)
else:
return result
else:
print_records(req, range(recid, recidb), -1, -9999, of, ot, ln, search_pattern=p, verbose=verbose,
tab=tab, sf=sf, so=so, sp=sp, rm=rm, em=em)
if req and of.startswith("h"): # register detailed record page view event
client_ip_address = str(req.remote_ip)
register_page_view_event(recid, uid, client_ip_address)
else: # record does not exist
if of == "id":
return []
elif of == "intbitset":
return intbitset()
+ elif of == "recjson":
+ return []
elif of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
elif of.startswith("h"):
if req.header_only:
raise apache.SERVER_RETURN(apache.HTTP_NOT_FOUND)
else:
write_warning(_("Requested record does not seem to exist."), req=req)
def prs_browse(kwargs=None, req=None, of=None, cc=None, aas=None, ln=None, uid=None, _=None, p=None,
p1=None, p2=None, p3=None, colls_to_display=None, f=None, rg=None, sf=None,
so=None, sp=None, rm=None, ot=None, f1=None, m1=None, op1=None,
f2=None, m2=None, op2=None, f3=None, m3=None, sc=None, pl=None,
d1y=None, d1m=None, d1d=None, d2y=None, d2m=None, d2d=None,
dt=None, jrec=None, ec=None, action=None,
colls_to_search=None, verbose=None, em=None, **dummy):
page_start(req, of, cc, aas, ln, uid, _("Browse"), p=create_page_title_search_pattern_info(p, p1, p2, p3), em=em)
req.write(create_search_box(cc, colls_to_display, p, f, rg, sf, so, sp, rm, of, ot, aas, ln, p1, f1, m1, op1,
p2, f2, m2, op2, p3, f3, m3, sc, pl, d1y, d1m, d1d, d2y, d2m, d2d, dt, jrec, ec, action,
em
))
write_warning(create_exact_author_browse_help_link(p, p1, p2, p3, f, f1, f2, f3,
rm, cc, ln, jrec, rg, aas, action),
req=req)
try:
if aas == 1 or (p1 or p2 or p3):
browse_pattern(req, colls_to_search, p1, f1, rg, ln)
browse_pattern(req, colls_to_search, p2, f2, rg, ln)
browse_pattern(req, colls_to_search, p3, f3, rg, ln)
else:
browse_pattern(req, colls_to_search, p, f, rg, ln)
except KeyboardInterrupt:
# This happens usually from the command line
# The error handling we want is different
raise
except:
register_exception(req=req, alert_admin=True)
if of.startswith("h"):
req.write(create_error_box(req, verbose=verbose, ln=ln))
elif of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
return page_end(req, of, ln, em)
def prs_search_similar_records(kwargs=None, req=None, of=None, cc=None, pl_in_url=None, ln=None, uid=None, _=None, p=None,
p1=None, p2=None, p3=None, colls_to_display=None, f=None, rg=None, sf=None,
so=None, sp=None, rm=None, ot=None, aas=None, f1=None, m1=None, op1=None,
f2=None, m2=None, op2=None, f3=None, m3=None, sc=None, pl=None,
d1y=None, d1m=None, d1d=None, d2y=None, d2m=None, d2d=None,
dt=None, jrec=None, ec=None, action=None, em=None,
verbose=None, **dummy):
if req and req.method != 'HEAD':
page_start(req, of, cc, aas, ln, uid, _("Search Results"), p=create_page_title_search_pattern_info(p, p1, p2, p3),
em=em)
if of.startswith("h"):
req.write(create_search_box(cc, colls_to_display, p, f, rg, sf, so, sp, rm, of, ot, aas, ln, p1, f1, m1, op1,
p2, f2, m2, op2, p3, f3, m3, sc, pl, d1y, d1m, d1d, d2y, d2m, d2d, dt, jrec, ec, action,
em
))
recid = p[6:]
if record_exists(recid) != 1:
# record does not exist
if of.startswith("h"):
if req.header_only:
raise apache.SERVER_RETURN(apache.HTTP_NOT_FOUND)
else:
write_warning(_("Requested record does not seem to exist."), req=req)
if of == "id":
return []
if of == "intbitset":
return intbitset()
+ elif of == "recjson":
+ return []
elif of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
else:
# record well exists, so find similar ones to it
t1 = os.times()[4]
(results_similar_recIDs,
results_similar_relevances,
results_similar_relevances_prologue,
results_similar_relevances_epilogue,
results_similar_comments) = \
rank_records_bibrank(rank_method_code=rm,
rank_limit_relevance=0,
hitset=get_collection_reclist(cc),
related_to=[p],
verbose=verbose,
field=f,
rg=rg,
jrec=jrec)
if results_similar_recIDs:
t2 = os.times()[4]
cpu_time = t2 - t1
if of.startswith("h"):
req.write(print_search_info(p, f, sf, so, sp, rm, of, ot, cc, len(results_similar_recIDs),
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time, em=em))
write_warning(results_similar_comments, req=req)
print_records(req, results_similar_recIDs, jrec, rg, of, ot, ln,
results_similar_relevances, results_similar_relevances_prologue,
results_similar_relevances_epilogue,
search_pattern=p, verbose=verbose, sf=sf, so=so, sp=sp, rm=rm, em=em)
elif of == "id":
return results_similar_recIDs
elif of == "intbitset":
return intbitset(results_similar_recIDs)
elif of.startswith("x"):
print_records(req, results_similar_recIDs, jrec, rg, of, ot, ln,
results_similar_relevances, results_similar_relevances_prologue,
results_similar_relevances_epilogue, search_pattern=p, verbose=verbose,
sf=sf, so=so, sp=sp, rm=rm, em=em)
else:
# rank_records failed and returned some error message to display:
if of.startswith("h"):
write_warning(results_similar_relevances_prologue, req=req)
write_warning(results_similar_relevances_epilogue, req=req)
write_warning(results_similar_comments, req=req)
if of == "id":
return []
elif of == "intbitset":
return intbitset()
+ elif of == "recjson":
+ return []
elif of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
def prs_search_cocitedwith(kwargs=None, req=None, of=None, cc=None, pl_in_url=None, ln=None, uid=None, _=None, p=None,
p1=None, p2=None, p3=None, colls_to_display=None, f=None, rg=None, sf=None,
so=None, sp=None, rm=None, ot=None, aas=None, f1=None, m1=None, op1=None,
f2=None, m2=None, op2=None, f3=None, m3=None, sc=None, pl=None,
d1y=None, d1m=None, d1d=None, d2y=None, d2m=None, d2d=None,
dt=None, jrec=None, ec=None, action=None,
verbose=None, em=None, **dummy):
page_start(req, of, cc, aas, ln, uid, _("Search Results"), p=create_page_title_search_pattern_info(p, p1, p2, p3),
em=em)
if of.startswith("h"):
req.write(create_search_box(cc, colls_to_display, p, f, rg, sf, so, sp, rm, of, ot, aas, ln, p1, f1, m1, op1,
p2, f2, m2, op2, p3, f3, m3, sc, pl, d1y, d1m, d1d, d2y, d2m, d2d, dt, jrec, ec, action,
em
))
recID = p[12:]
if record_exists(recID) != 1:
# record does not exist
if of.startswith("h"):
write_warning(_("Requested record does not seem to exist."), req=req)
if of == "id":
return []
elif of == "intbitset":
return intbitset()
+ elif of == "recjson":
+ return []
elif of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
else:
# record well exists, so find co-cited ones:
t1 = os.times()[4]
results_cocited_recIDs = [x[0] for x in calculate_co_cited_with_list(int(recID))]
if results_cocited_recIDs:
t2 = os.times()[4]
cpu_time = t2 - t1
if of.startswith("h"):
req.write(print_search_info(p, f, sf, so, sp, rm, of, ot, CFG_SITE_NAME, len(results_cocited_recIDs),
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time, em=em))
print_records(req, results_cocited_recIDs, jrec, rg, of, ot, ln, search_pattern=p, verbose=verbose,
sf=sf, so=so, sp=sp, rm=rm, em=em)
elif of == "id":
return results_cocited_recIDs
elif of == "intbitset":
return intbitset(results_cocited_recIDs)
elif of.startswith("x"):
print_records(req, results_cocited_recIDs, jrec, rg, of, ot, ln, search_pattern=p, verbose=verbose,
sf=sf, so=so, sp=sp, rm=rm, em=em)
else:
# cited rank_records failed and returned some error message to display:
if of.startswith("h"):
write_warning("nothing found", req=req)
if of == "id":
return []
elif of == "intbitset":
return intbitset()
+ elif of == "recjson":
+ return []
elif of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
def prs_search_hosted_collections(kwargs=None, req=None, of=None, ln=None, _=None, p=None,
p1=None, p2=None, p3=None, hosted_colls=None, f=None,
colls_to_search=None, hosted_colls_actual_or_potential_results_p=None,
verbose=None, **dummy):
hosted_colls_results = hosted_colls_timeouts = hosted_colls_true_results = None
# search into the hosted collections only if the output format is html or xml
if hosted_colls and (of.startswith("h") or of.startswith("x")) and not p.startswith("recid:"):
# hosted_colls_results : the hosted collections' searches that did not timeout
# hosted_colls_timeouts : the hosted collections' searches that timed out and will be searched later on again
(hosted_colls_results, hosted_colls_timeouts) = calculate_hosted_collections_results(req, [p, p1, p2, p3], f, hosted_colls, verbose, ln, CFG_HOSTED_COLLECTION_TIMEOUT_ANTE_SEARCH)
# successful searches
if hosted_colls_results:
hosted_colls_true_results = []
for result in hosted_colls_results:
# if the number of results is None or 0 (or False) then just do nothing
if result[1] is None or result[1] is False:
# these are the searches the returned no or zero results
if verbose:
write_warning("Hosted collections (perform_search_request): %s returned no results" % result[0][1].name, req=req)
else:
# these are the searches that actually returned results on time
hosted_colls_true_results.append(result)
if verbose:
write_warning("Hosted collections (perform_search_request): %s returned %s results in %s seconds" % (result[0][1].name, result[1], result[2]), req=req)
else:
if verbose:
write_warning("Hosted collections (perform_search_request): there were no hosted collections results to be printed at this time", req=req)
if hosted_colls_timeouts:
if verbose:
for timeout in hosted_colls_timeouts:
write_warning("Hosted collections (perform_search_request): %s timed out and will be searched again later" % timeout[0][1].name, req=req)
# we need to know for later use if there were any hosted collections to be searched even if they weren't in the end
elif hosted_colls and ((not (of.startswith("h") or of.startswith("x"))) or p.startswith("recid:")):
(hosted_colls_results, hosted_colls_timeouts) = (None, None)
else:
if verbose:
write_warning("Hosted collections (perform_search_request): there were no hosted collections to be searched", req=req)
## let's define some useful boolean variables:
# True means there are actual or potential hosted collections results to be printed
kwargs['hosted_colls_actual_or_potential_results_p'] = not (not hosted_colls or not ((hosted_colls_results and hosted_colls_true_results) or hosted_colls_timeouts))
# True means there are hosted collections timeouts to take care of later
# (useful for more accurate printing of results later)
kwargs['hosted_colls_potential_results_p'] = not (not hosted_colls or not hosted_colls_timeouts)
# True means we only have hosted collections to deal with
kwargs['only_hosted_colls_actual_or_potential_results_p'] = not colls_to_search and hosted_colls_actual_or_potential_results_p
kwargs['hosted_colls_results'] = hosted_colls_results
kwargs['hosted_colls_timeouts'] = hosted_colls_timeouts
kwargs['hosted_colls_true_results'] = hosted_colls_true_results
def prs_advanced_search(results_in_any_collection, kwargs=None, req=None, of=None,
cc=None, ln=None, _=None, p=None, p1=None, p2=None, p3=None,
f=None, f1=None, m1=None, op1=None, f2=None, m2=None,
op2=None, f3=None, m3=None, ap=None, ec=None,
selected_external_collections_infos=None, verbose=None,
wl=None, em=None, **dummy):
len_results_p1 = 0
len_results_p2 = 0
len_results_p3 = 0
try:
results_in_any_collection.union_update(search_pattern_parenthesised(req, p1, f1, m1, ap=ap, of=of, verbose=verbose, ln=ln, wl=wl))
len_results_p1 = len(results_in_any_collection)
if len_results_p1 == 0:
if of.startswith("h"):
perform_external_collection_search_with_em(req, cc, [p, p1, p2, p3], f, ec,
verbose, ln, selected_external_collections_infos, em=em)
elif of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
return page_end(req, of, ln, em)
if p2:
results_tmp = search_pattern_parenthesised(req, p2, f2, m2, ap=ap, of=of, verbose=verbose, ln=ln, wl=wl)
len_results_p2 = len(results_tmp)
if op1 == "a": # add
results_in_any_collection.intersection_update(results_tmp)
elif op1 == "o": # or
results_in_any_collection.union_update(results_tmp)
elif op1 == "n": # not
results_in_any_collection.difference_update(results_tmp)
else:
if of.startswith("h"):
write_warning("Invalid set operation %s." % cgi.escape(op1), "Error", req=req)
if len(results_in_any_collection) == 0:
if of.startswith("h"):
if len_results_p2:
#each individual query returned results, but the boolean operation did not
nearestterms = []
nearest_search_args = req.argd.copy()
if p1:
nearestterms.append((p1, len_results_p1, clean_dictionary(nearest_search_args, ['p2', 'f2', 'm2', 'p3', 'f3', 'm3'])))
nearestterms.append((p2, len_results_p2, clean_dictionary(nearest_search_args, ['p1', 'f1', 'm1', 'p3', 'f3', 'm3'])))
write_warning(websearch_templates.tmpl_search_no_boolean_hits(ln=ln, nearestterms=nearestterms), req=req)
perform_external_collection_search_with_em(req, cc, [p, p1, p2, p3], f, ec, verbose,
ln, selected_external_collections_infos, em=em)
elif of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
if p3:
results_tmp = search_pattern_parenthesised(req, p3, f3, m3, ap=ap, of=of, verbose=verbose, ln=ln, wl=wl)
len_results_p3 = len(results_tmp)
if op2 == "a": # add
results_in_any_collection.intersection_update(results_tmp)
elif op2 == "o": # or
results_in_any_collection.union_update(results_tmp)
elif op2 == "n": # not
results_in_any_collection.difference_update(results_tmp)
else:
if of.startswith("h"):
write_warning("Invalid set operation %s." % cgi.escape(op2), "Error", req=req)
if len(results_in_any_collection) == 0 and len_results_p3 and of.startswith("h"):
#each individual query returned results but the boolean operation did not
nearestterms = []
nearest_search_args = req.argd.copy()
if p1:
nearestterms.append((p1, len_results_p1, clean_dictionary(nearest_search_args, ['p2', 'f2', 'm2', 'p3', 'f3', 'm3'])))
if p2:
nearestterms.append((p2, len_results_p2, clean_dictionary(nearest_search_args, ['p1', 'f1', 'm1', 'p3', 'f3', 'm3'])))
nearestterms.append((p3, len_results_p3, clean_dictionary(nearest_search_args, ['p1', 'f1', 'm1', 'p2', 'f2', 'm2'])))
write_warning(websearch_templates.tmpl_search_no_boolean_hits(ln=ln, nearestterms=nearestterms), req=req)
except KeyboardInterrupt:
# This happens usually from the command line
# The error handling we want is different
raise
except:
register_exception(req=req, alert_admin=True)
if of.startswith("h"):
req.write(create_error_box(req, verbose=verbose, ln=ln))
perform_external_collection_search_with_em(req, cc, [p, p1, p2, p3], f, ec, verbose,
ln, selected_external_collections_infos, em=em)
elif of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
return page_end(req, of, ln, em)
def prs_simple_search(results_in_any_collection, kwargs=None, req=None, of=None, cc=None, ln=None, p=None, f=None,
p1=None, p2=None, p3=None, ec=None, verbose=None, selected_external_collections_infos=None,
only_hosted_colls_actual_or_potential_results_p=None, query_representation_in_cache=None,
ap=None, hosted_colls_actual_or_potential_results_p=None, wl=None, em=None,
**dummy):
try:
results_in_cache = intbitset().fastload(
search_results_cache.get(query_representation_in_cache))
except:
results_in_cache = None
if results_in_cache is not None:
# query is not in the cache already, so reuse it:
results_in_any_collection.union_update(results_in_cache)
if verbose and of.startswith("h"):
write_warning("Search stage 0: query found in cache, reusing cached results.", req=req)
else:
try:
# added the display_nearest_terms_box parameter to avoid printing out the "Nearest terms in any collection"
# recommendations when there are results only in the hosted collections. Also added the if clause to avoid
# searching in case we know we only have actual or potential hosted collections results
if not only_hosted_colls_actual_or_potential_results_p:
results_in_any_collection.union_update(search_pattern_parenthesised(req, p, f, ap=ap, of=of, verbose=verbose, ln=ln,
display_nearest_terms_box=not hosted_colls_actual_or_potential_results_p,
wl=wl))
except:
register_exception(req=req, alert_admin=True)
if of.startswith("h"):
req.write(create_error_box(req, verbose=verbose, ln=ln))
perform_external_collection_search_with_em(req, cc, [p, p1, p2, p3], f, ec, verbose,
ln, selected_external_collections_infos, em=em)
return page_end(req, of, ln, em)
def prs_intersect_results_with_collrecs(results_final, results_in_any_collection,
kwargs=None, colls_to_search=None,
req=None, of=None, ln=None,
cc=None, p=None, p1=None, p2=None, p3=None, f=None,
ec=None, verbose=None, selected_external_collections_infos=None,
em=None, **dummy):
display_nearest_terms_box=not kwargs['hosted_colls_actual_or_potential_results_p']
try:
# added the display_nearest_terms_box parameter to avoid printing out the "Nearest terms in any collection"
# recommendations when there results only in the hosted collections. Also added the if clause to avoid
# searching in case we know since the last stage that we have no results in any collection
if len(results_in_any_collection) != 0:
results_final.update(intersect_results_with_collrecs(req, results_in_any_collection, colls_to_search, of,
verbose, ln, display_nearest_terms_box=display_nearest_terms_box))
except:
register_exception(req=req, alert_admin=True)
if of.startswith("h"):
req.write(create_error_box(req, verbose=verbose, ln=ln))
perform_external_collection_search_with_em(req, cc, [p, p1, p2, p3], f, ec, verbose,
ln, selected_external_collections_infos, em=em)
return page_end(req, of, ln, em)
def prs_store_results_in_cache(query_representation_in_cache, results_in_any_collection, req=None, verbose=None, of=None, **dummy):
if CFG_WEBSEARCH_SEARCH_CACHE_SIZE > 0:
search_results_cache.set(query_representation_in_cache,
results_in_any_collection.fastdump(),
timeout=CFG_WEBSEARCH_SEARCH_CACHE_TIMEOUT)
search_results_cache.set(query_representation_in_cache + '::cc',
dummy.get('cc', CFG_SITE_NAME),
timeout=CFG_WEBSEARCH_SEARCH_CACHE_TIMEOUT)
if req:
from flask import request
req = request
search_results_cache.set(query_representation_in_cache + '::p',
req.values.get('p', ''),
timeout=CFG_WEBSEARCH_SEARCH_CACHE_TIMEOUT)
if verbose and of.startswith("h"):
write_warning(req, "Search stage 3: storing query results in cache.", req=req)
def prs_apply_search_limits(results_final, kwargs=None, req=None, of=None, cc=None, ln=None, _=None,
p=None, p1=None, p2=None, p3=None, f=None, pl=None, ap=None, dt=None,
ec=None, selected_external_collections_infos=None,
hosted_colls_actual_or_potential_results_p=None,
datetext1=None, datetext2=None, verbose=None, wl=None, em=None,
**dummy):
if datetext1 != "" and results_final != {}:
if verbose and of.startswith("h"):
write_warning("Search stage 5: applying time etc limits, from %s until %s..." % (datetext1, datetext2), req=req)
try:
results_final = intersect_results_with_hitset(req,
results_final,
search_unit_in_bibrec(datetext1, datetext2, dt),
ap,
aptext= _("No match within your time limits, "
"discarding this condition..."),
of=of)
except:
register_exception(req=req, alert_admin=True)
if of.startswith("h"):
req.write(create_error_box(req, verbose=verbose, ln=ln))
perform_external_collection_search_with_em(req, cc, [p, p1, p2, p3], f, ec, verbose,
ln, selected_external_collections_infos, em=em)
return page_end(req, of, ln, em)
if results_final == {} and not hosted_colls_actual_or_potential_results_p:
if of.startswith("h"):
perform_external_collection_search_with_em(req, cc, [p, p1, p2, p3], f, ec, verbose,
ln, selected_external_collections_infos, em=em)
#if of.startswith("x"):
# # Print empty, but valid XML
# print_records_prologue(req, of)
# print_records_epilogue(req, of)
return page_end(req, of, ln, em)
if pl and results_final != {}:
pl = wash_pattern(pl)
if verbose and of.startswith("h"):
write_warning("Search stage 5: applying search pattern limit %s..." % cgi.escape(pl), req=req)
try:
results_final = intersect_results_with_hitset(req,
results_final,
search_pattern_parenthesised(req, pl, ap=0, ln=ln, wl=wl),
ap,
aptext=_("No match within your search limits, "
"discarding this condition..."),
of=of)
except:
register_exception(req=req, alert_admin=True)
if of.startswith("h"):
req.write(create_error_box(req, verbose=verbose, ln=ln))
perform_external_collection_search_with_em(req, cc, [p, p1, p2, p3], f, ec, verbose,
ln, selected_external_collections_infos, em=em)
return page_end(req, of, ln, em)
if results_final == {} and not hosted_colls_actual_or_potential_results_p:
if of.startswith("h"):
perform_external_collection_search_with_em(req, cc, [p, p1, p2, p3], f, ec, verbose,
ln, selected_external_collections_infos, em=em)
if of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
return page_end(req, of, ln, em)
def prs_split_into_collections(kwargs=None, results_final=None, colls_to_search=None, hosted_colls_results=None,
cpu_time=0, results_final_nb_total=None, hosted_colls_actual_or_potential_results_p=None,
hosted_colls_true_results=None, hosted_colls_timeouts=None, **dummy):
results_final_nb_total = 0
results_final_nb = {} # will hold number of records found in each collection
# (in simple dict to display overview more easily)
for coll in results_final.keys():
results_final_nb[coll] = len(results_final[coll])
#results_final_nb_total += results_final_nb[coll]
# Now let us calculate results_final_nb_total more precisely,
# in order to get the total number of "distinct" hits across
# searched collections; this is useful because a record might
# have been attributed to more than one primary collection; so
# we have to avoid counting it multiple times. The price to
# pay for this accuracy of results_final_nb_total is somewhat
# increased CPU time.
if results_final.keys() == 1:
# only one collection; no need to union them
results_final_for_all_selected_colls = results_final.values()[0]
results_final_nb_total = results_final_nb.values()[0]
else:
# okay, some work ahead to union hits across collections:
results_final_for_all_selected_colls = intbitset()
for coll in results_final.keys():
results_final_for_all_selected_colls.union_update(results_final[coll])
results_final_nb_total = len(results_final_for_all_selected_colls)
#if hosted_colls and (of.startswith("h") or of.startswith("x")):
if hosted_colls_actual_or_potential_results_p:
if hosted_colls_results:
for result in hosted_colls_true_results:
colls_to_search.append(result[0][1].name)
results_final_nb[result[0][1].name] = result[1]
results_final_nb_total += result[1]
cpu_time += result[2]
if hosted_colls_timeouts:
for timeout in hosted_colls_timeouts:
colls_to_search.append(timeout[1].name)
# use -963 as a special number to identify the collections that timed out
results_final_nb[timeout[1].name] = -963
kwargs['results_final_nb'] = results_final_nb
kwargs['results_final_nb_total'] = results_final_nb_total
kwargs['results_final_for_all_selected_colls'] = results_final_for_all_selected_colls
kwargs['cpu_time'] = cpu_time #rca TODO: check where the cpu_time is used, this line was missing
return (results_final_nb, results_final_nb_total, results_final_for_all_selected_colls)
def prs_summarize_records(kwargs=None, req=None, p=None, f=None, aas=None,
p1=None, p2=None, p3=None, f1=None, f2=None, f3=None, op1=None, op2=None,
ln=None, results_final_for_all_selected_colls=None, of='hcs', **dummy):
# feed the current search to be summarized:
from invenio.legacy.search_engine.summarizer import summarize_records
search_p = p
search_f = f
if not p and (aas == 1 or p1 or p2 or p3):
op_d = {'n': ' and not ', 'a': ' and ', 'o': ' or ', '': ''}
triples = ziplist([f1, f2, f3], [p1, p2, p3], [op1, op2, ''])
triples_len = len(triples)
for i in range(triples_len):
fi, pi, oi = triples[i] # e.g.:
if i < triples_len-1 and not triples[i+1][1]: # if p2 empty
triples[i+1][0] = '' # f2 must be too
oi = '' # and o1
if ' ' in pi:
pi = '"'+pi+'"'
if fi:
fi = fi + ':'
search_p += fi + pi + op_d[oi]
search_f = ''
summarize_records(results_final_for_all_selected_colls, of, ln, search_p, search_f, req)
def prs_print_records(kwargs=None, results_final=None, req=None, of=None, cc=None, pl_in_url=None,
ln=None, _=None, p=None, p1=None, p2=None, p3=None, f=None, rg=None, sf=None,
so=None, sp=None, rm=None, ot=None, aas=None, f1=None, m1=None, op1=None,
f2=None, m2=None, op2=None, f3=None, m3=None, sc=None, d1y=None, d1m=None,
d1d=None, d2y=None, d2m=None, d2d=None, dt=None, jrec=None, colls_to_search=None,
hosted_colls_actual_or_potential_results_p=None, hosted_colls_results=None,
hosted_colls_true_results=None, hosted_colls_timeouts=None, results_final_nb=None,
cpu_time=None, verbose=None, em=None, **dummy):
if len(colls_to_search) > 1:
cpu_time = -1 # we do not want to have search time printed on each collection
print_records_prologue(req, of, cc=cc)
results_final_colls = []
wlqh_results_overlimit = 0
for coll in colls_to_search:
if coll in results_final and len(results_final[coll]):
if of.startswith("h"):
req.write(print_search_info(p, f, sf, so, sp, rm, of, ot, coll, results_final_nb[coll],
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time, em=em))
results_final_recIDs = list(results_final[coll])
results_final_relevances = []
results_final_relevances_prologue = ""
results_final_relevances_epilogue = ""
-
if rm: # do we have to rank?
results_final_recIDs_ranked, results_final_relevances, results_final_relevances_prologue, results_final_relevances_epilogue, results_final_comments = \
rank_records(req, rm, 0, results_final[coll],
string.split(p) + string.split(p1) +
string.split(p2) + string.split(p3), verbose, so, of, ln, rg, jrec, kwargs['f'])
if of.startswith("h"):
write_warning(results_final_comments, req=req)
if results_final_recIDs_ranked:
results_final_recIDs = results_final_recIDs_ranked
else:
# rank_records failed and returned some error message to display:
write_warning(results_final_relevances_prologue, req=req)
write_warning(results_final_relevances_epilogue, req=req)
- elif sf or (CFG_BIBSORT_ENABLED and SORTING_METHODS): # do we have to sort?
+ else:
results_final_recIDs = sort_records(req, results_final_recIDs, sf, so, sp, verbose, of, ln, rg, jrec)
if len(results_final_recIDs) < CFG_WEBSEARCH_PREV_NEXT_HIT_LIMIT:
results_final_colls.append(results_final_recIDs)
else:
wlqh_results_overlimit = 1
print_records(req, results_final_recIDs, jrec, rg, of, ot, ln,
results_final_relevances,
results_final_relevances_prologue,
results_final_relevances_epilogue,
search_pattern=p,
print_records_prologue_p=False,
print_records_epilogue_p=False,
verbose=verbose,
sf=sf,
so=so,
sp=sp,
rm=rm,
em=em)
if of.startswith("h"):
req.write(print_search_info(p, f, sf, so, sp, rm, of, ot, coll, results_final_nb[coll],
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time, 1, em=em))
if req and not isinstance(req, cStringIO.OutputType):
# store the last search results page
session_param_set(req, 'websearch-last-query', req.unparsed_uri)
if wlqh_results_overlimit:
results_final_colls = None
# store list of results if user wants to display hits
# in a single list, or store list of collections of records
# if user displays hits split by collections:
session_param_set(req, 'websearch-last-query-hits', results_final_colls)
#if hosted_colls and (of.startswith("h") or of.startswith("x")):
if hosted_colls_actual_or_potential_results_p:
if hosted_colls_results:
# TODO: add a verbose message here
for result in hosted_colls_true_results:
if of.startswith("h"):
req.write(print_hosted_search_info(p, f, sf, so, sp, rm, of, ot, result[0][1].name, results_final_nb[result[0][1].name],
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time, em=em))
req.write(print_hosted_results(url_and_engine=result[0], ln=ln, of=of, req=req, limit=rg, em=em))
if of.startswith("h"):
req.write(print_hosted_search_info(p, f, sf, so, sp, rm, of, ot, result[0][1].name, results_final_nb[result[0][1].name],
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time, 1))
if hosted_colls_timeouts:
# TODO: add a verbose message here
# TODO: check if verbose messages still work when dealing with (re)calculations of timeouts
(hosted_colls_timeouts_results, hosted_colls_timeouts_timeouts) = do_calculate_hosted_collections_results(req, ln, None, verbose, None, hosted_colls_timeouts, CFG_HOSTED_COLLECTION_TIMEOUT_POST_SEARCH)
if hosted_colls_timeouts_results:
for result in hosted_colls_timeouts_results:
if result[1] is None or result[1] is False:
## these are the searches the returned no or zero results
## also print a nearest terms box, in case this is the only
## collection being searched and it returns no results?
if of.startswith("h"):
req.write(print_hosted_search_info(p, f, sf, so, sp, rm, of, ot, result[0][1].name, -963,
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time))
req.write(print_hosted_results(url_and_engine=result[0], ln=ln, of=of, req=req, no_records_found=True, limit=rg, em=em))
req.write(print_hosted_search_info(p, f, sf, so, sp, rm, of, ot, result[0][1].name, -963,
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time, 1))
else:
# these are the searches that actually returned results on time
if of.startswith("h"):
req.write(print_hosted_search_info(p, f, sf, so, sp, rm, of, ot, result[0][1].name, result[1],
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time))
req.write(print_hosted_results(url_and_engine=result[0], ln=ln, of=of, req=req, limit=rg, em=em))
if of.startswith("h"):
req.write(print_hosted_search_info(p, f, sf, so, sp, rm, of, ot, result[0][1].name, result[1],
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time, 1))
if hosted_colls_timeouts_timeouts:
for timeout in hosted_colls_timeouts_timeouts:
if of.startswith("h"):
req.write(print_hosted_search_info(p, f, sf, so, sp, rm, of, ot, timeout[1].name, -963,
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time))
req.write(print_hosted_results(url_and_engine=timeout[0], ln=ln, of=of, req=req, search_timed_out=True, limit=rg, em=em))
req.write(print_hosted_search_info(p, f, sf, so, sp, rm, of, ot, timeout[1].name, -963,
jrec, rg, aas, ln, p1, p2, p3, f1, f2, f3, m1, m2, m3, op1, op2,
sc, pl_in_url,
d1y, d1m, d1d, d2y, d2m, d2d, dt, cpu_time, 1))
print_records_epilogue(req, of)
if f == "author" and of.startswith("h"):
req.write(create_similarly_named_authors_link_box(p, ln))
def prs_log_query(kwargs=None, req=None, uid=None, of=None, ln=None, p=None, f=None,
colls_to_search=None, results_final_nb_total=None, em=None, **dummy):
# FIXME move query logging to signal receiver
# log query:
try:
from flask.ext.login import current_user
if req:
from flask import request
req = request
id_query = log_query(req.host,
'&'.join(map(lambda (k,v): k+'='+v, request.values.iteritems(multi=True))),
uid)
#id_query = log_query(req.remote_host, req.args, uid)
#of = request.values.get('of', 'hb')
if of.startswith("h") and id_query and (em == '' or EM_REPOSITORY["alert"] in em):
if not of in ['hcs', 'hcs2']:
# display alert/RSS teaser for non-summary formats:
display_email_alert_part = True
if current_user:
if current_user['email'] == 'guest':
if CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS > 4:
display_email_alert_part = False
else:
if not current_user['precached_usealerts']:
display_email_alert_part = False
from flask import flash
flash(websearch_templates.tmpl_alert_rss_teaser_box_for_query(id_query, \
ln=ln, display_email_alert_part=display_email_alert_part), 'search-results-after')
except:
# do not log query if req is None (used by CLI interface)
pass
log_query_info("ss", p, f, colls_to_search, results_final_nb_total)
def prs_search_common(kwargs=None, req=None, of=None, cc=None, ln=None, uid=None, _=None, p=None,
p1=None, p2=None, p3=None, colls_to_display=None, f=None, rg=None, sf=None,
so=None, sp=None, rm=None, ot=None, aas=None, f1=None, m1=None, op1=None,
f2=None, m2=None, op2=None, f3=None, m3=None, sc=None, pl=None,
d1y=None, d1m=None, d1d=None, d2y=None, d2m=None, d2d=None,
dt=None, jrec=None, ec=None, action=None, colls_to_search=None, wash_colls_debug=None,
verbose=None, wl=None, em=None, **dummy):
query_representation_in_cache = get_search_results_cache_key(**kwargs)
page_start(req, of, cc, aas, ln, uid, p=create_page_title_search_pattern_info(p, p1, p2, p3), em=em)
if of.startswith("h") and verbose and wash_colls_debug:
write_warning("wash_colls debugging info : %s" % wash_colls_debug, req=req)
prs_search_hosted_collections(kwargs=kwargs, **kwargs)
if of.startswith("h"):
req.write(create_search_box(cc, colls_to_display, p, f, rg, sf, so, sp, rm, of, ot, aas, ln, p1, f1, m1, op1,
p2, f2, m2, op2, p3, f3, m3, sc, pl, d1y, d1m, d1d, d2y, d2m, d2d, dt, jrec, ec, action,
em
))
# WebSearch services
if jrec <= 1 and \
(em == "" and True or (EM_REPOSITORY["search_services"] in em)):
user_info = collect_user_info(req)
# display only on first search page, and only if wanted
# when 'em' param set.
for answer_relevance, answer_html in services.get_answers(
req, user_info, of, cc, colls_to_search, p, f, ln):
req.write('<div class="searchservicebox">')
req.write(answer_html)
if verbose > 8:
write_warning("Service relevance: %i" % answer_relevance, req=req)
req.write('</div>')
t1 = os.times()[4]
results_in_any_collection = intbitset()
if aas == 1 or (p1 or p2 or p3):
## 3A - advanced search
output = prs_advanced_search(results_in_any_collection, kwargs=kwargs, **kwargs)
if output is not None:
return output
else:
## 3B - simple search
output = prs_simple_search(results_in_any_collection, kwargs=kwargs, **kwargs)
if output is not None:
return output
if len(results_in_any_collection) == 0 and not kwargs['hosted_colls_actual_or_potential_results_p']:
if of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
return None
# store this search query results into search results cache if needed:
prs_store_results_in_cache(query_representation_in_cache, results_in_any_collection, **kwargs)
# search stage 4 and 5: intersection with collection universe and sorting/limiting
try:
output = prs_intersect_with_colls_and_apply_search_limits(results_in_any_collection, kwargs=kwargs, **kwargs)
if output is not None:
return output
except KeyboardInterrupt:
# This happens usually from the command line
# The error handling we want is different
raise
except: # no results to display
return None
t2 = os.times()[4]
cpu_time = t2 - t1
kwargs['cpu_time'] = cpu_time
## search stage 6: display results:
return prs_display_results(kwargs=kwargs, **kwargs)
def prs_intersect_with_colls_and_apply_search_limits(results_in_any_collection,
kwargs=None, req=None, of=None,
**dummy):
# search stage 4: intersection with collection universe:
results_final = {}
output = prs_intersect_results_with_collrecs(results_final, results_in_any_collection, kwargs, **kwargs)
if output is not None:
return output
# another external search if we still don't have something
if results_final == {} and not kwargs['hosted_colls_actual_or_potential_results_p']:
if of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
kwargs['results_final'] = results_final
raise Exception
# search stage 5: apply search option limits and restrictions:
output = prs_apply_search_limits(results_final, kwargs=kwargs, **kwargs)
kwargs['results_final'] = results_final
if output is not None:
return output
def prs_display_results(kwargs=None, results_final=None, req=None, of=None, sf=None,
so=None, sp=None, verbose=None, p=None, p1=None, p2=None, p3=None,
cc=None, ln=None, _=None, ec=None, colls_to_search=None, rm=None, cpu_time=None,
f=None, em=None, jrec=0, rg=None, **dummy
):
## search stage 6: display results:
# split result set into collections
(results_final_nb, results_final_nb_total, results_final_for_all_selected_colls) = prs_split_into_collections(kwargs=kwargs, **kwargs)
# we continue past this point only if there is a hosted collection that has timed out and might offer potential results
if results_final_nb_total == 0 and not kwargs['hosted_colls_potential_results_p']:
if of.startswith("h"):
write_warning("No match found, please enter different search terms.", req=req)
elif of.startswith("x"):
# Print empty, but valid XML
print_records_prologue(req, of)
print_records_epilogue(req, of)
else:
prs_log_query(kwargs=kwargs, **kwargs)
# yes, some hits found: good!
# collection list may have changed due to not-exact-match-found policy so check it out:
for coll in results_final.keys():
if coll not in colls_to_search:
colls_to_search.append(coll)
# print results overview:
if of == "intbitset":
#return the result as an intbitset
return results_final_for_all_selected_colls
elif of == "id":
# we have been asked to return list of recIDs
recIDs = list(results_final_for_all_selected_colls)
if rm: # do we have to rank?
results_final_for_all_colls_rank_records_output = rank_records(req, rm, 0, results_final_for_all_selected_colls,
p.split() + p1.split() +
p2.split() + p3.split(), verbose, so, of, ln, kwargs['rg'], kwargs['jrec'], kwargs['f'])
if results_final_for_all_colls_rank_records_output[0]:
recIDs = results_final_for_all_colls_rank_records_output[0]
elif sf or (CFG_BIBSORT_ENABLED and SORTING_METHODS): # do we have to sort?
recIDs = sort_records(req, recIDs, sf, so, sp, verbose, of, ln)
if rg:
return recIDs[jrec:jrec+rg]
else:
return recIDs[jrec:]
elif of.startswith("h"):
if of not in ['hcs', 'hcs2', 'hcv', 'htcv', 'tlcv']:
# added the hosted_colls_potential_results_p parameter to help print out the overview more accurately
req.write(print_results_overview(colls_to_search, results_final_nb_total, results_final_nb, cpu_time,
ln, ec, hosted_colls_potential_results_p=kwargs['hosted_colls_potential_results_p'], em=em))
kwargs['selected_external_collections_infos'] = print_external_results_overview(req, cc, [p, p1, p2, p3],
f, ec, verbose, ln, print_overview=em == "" or EM_REPOSITORY["overview"] in em)
# print number of hits found for XML outputs:
if of.startswith("x") or of == 'mobb':
req.write("<!-- Search-Engine-Total-Number-Of-Results: %s -->\n" % kwargs['results_final_nb_total'])
# print records:
if of in ['hcs', 'hcs2']:
prs_summarize_records(kwargs=kwargs, **kwargs)
elif of in ['hcv', 'htcv', 'tlcv'] and CFG_INSPIRE_SITE:
from invenio.search_engine_cvifier import cvify_records
cvify_records(results_final_for_all_selected_colls, of, req, so)
else:
prs_print_records(kwargs=kwargs, **kwargs)
# this is a copy of the prs_display_results with output parts removed, needed for external modules
def prs_rank_results(kwargs=None, results_final=None, req=None, colls_to_search=None,
sf=None, so=None, sp=None, of=None, rm=None, p=None, p1=None, p2=None, p3=None,
verbose=None, **dummy
):
## search stage 6: display results:
# split result set into collections
dummy_results_final_nb, dummy_results_final_nb_total, results_final_for_all_selected_colls = prs_split_into_collections(kwargs=kwargs, **kwargs)
# yes, some hits found: good!
# collection list may have changed due to not-exact-match-found policy so check it out:
for coll in results_final.keys():
if coll not in colls_to_search:
colls_to_search.append(coll)
# we have been asked to return list of recIDs
recIDs = list(results_final_for_all_selected_colls)
if rm: # do we have to rank?
results_final_for_all_colls_rank_records_output = rank_records(req, rm, 0, results_final_for_all_selected_colls,
p.split() + p1.split() +
p2.split() + p3.split(), verbose, so, of, field=kwargs['f'])
if results_final_for_all_colls_rank_records_output[0]:
recIDs = results_final_for_all_colls_rank_records_output[0]
elif sf or (CFG_BIBSORT_ENABLED and SORTING_METHODS): # do we have to sort?
recIDs = sort_records(req, recIDs, sf, so, sp, verbose, of)
return recIDs
def perform_request_cache(req, action="show"):
"""Manipulates the search engine cache."""
req.content_type = "text/html"
req.send_http_header()
req.write("<html>")
out = ""
out += "<h1>Search Cache</h1>"
req.write(out)
# show collection reclist cache:
out = "<h3>Collection reclist cache</h3>"
out += "- collection table last updated: %s" % get_table_update_time('collection')
out += "<br />- reclist cache timestamp: %s" % collection_reclist_cache.timestamp
out += "<br />- reclist cache contents:"
out += "<blockquote>"
for coll in collection_reclist_cache.cache.keys():
if collection_reclist_cache.cache[coll]:
out += "%s (%d)<br />" % (coll, len(collection_reclist_cache.cache[coll]))
out += "</blockquote>"
req.write(out)
# show field i18nname cache:
out = "<h3>Field I18N names cache</h3>"
out += "- fieldname table last updated: %s" % get_table_update_time('fieldname')
out += "<br />- i18nname cache timestamp: %s" % field_i18nname_cache.timestamp
out += "<br />- i18nname cache contents:"
out += "<blockquote>"
for field in field_i18nname_cache.cache.keys():
for ln in field_i18nname_cache.cache[field].keys():
out += "%s, %s = %s<br />" % (field, ln, field_i18nname_cache.cache[field][ln])
out += "</blockquote>"
req.write(out)
# show collection i18nname cache:
out = "<h3>Collection I18N names cache</h3>"
out += "- collectionname table last updated: %s" % get_table_update_time('collectionname')
out += "<br />- i18nname cache timestamp: %s" % collection_i18nname_cache.timestamp
out += "<br />- i18nname cache contents:"
out += "<blockquote>"
for coll in collection_i18nname_cache.cache.keys():
for ln in collection_i18nname_cache.cache[coll].keys():
out += "%s, %s = %s<br />" % (coll, ln, collection_i18nname_cache.cache[coll][ln])
out += "</blockquote>"
req.write(out)
req.write("</html>")
return "\n"
def perform_request_log(req, date=""):
"""Display search log information for given date."""
req.content_type = "text/html"
req.send_http_header()
req.write("<html>")
req.write("<h1>Search Log</h1>")
if date: # case A: display stats for a day
yyyymmdd = string.atoi(date)
req.write("<p><big><strong>Date: %d</strong></big><p>" % yyyymmdd)
req.write("""<table border="1">""")
req.write("<tr><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td><td><strong>%s</strong></td></tr>" % ("No.", "Time", "Pattern", "Field", "Collection", "Number of Hits"))
# read file:
p = os.popen("grep ^%d %s/search.log" % (yyyymmdd, CFG_LOGDIR), 'r')
lines = p.readlines()
p.close()
# process lines:
i = 0
for line in lines:
try:
datetime, dummy_aas, p, f, c, nbhits = line.split("#")
i += 1
req.write("<tr><td align=\"right\">#%d</td><td>%s:%s:%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td></tr>"
% (i, datetime[8:10], datetime[10:12], datetime[12:], p, f, c, nbhits))
except:
pass # ignore eventual wrong log lines
req.write("</table>")
else: # case B: display summary stats per day
yyyymm01 = int(time.strftime("%Y%m01", time.localtime()))
yyyymmdd = int(time.strftime("%Y%m%d", time.localtime()))
req.write("""<table border="1">""")
req.write("<tr><td><strong>%s</strong></td><td><strong>%s</strong></tr>" % ("Day", "Number of Queries"))
for day in range(yyyymm01, yyyymmdd + 1):
p = os.popen("grep -c ^%d %s/search.log" % (day, CFG_LOGDIR), 'r')
for line in p.readlines():
req.write("""<tr><td>%s</td><td align="right"><a href="%s/search/log?date=%d">%s</a></td></tr>""" %
(day, CFG_SITE_URL, day, line))
p.close()
req.write("</table>")
req.write("</html>")
return "\n"
def get_all_field_values(tag):
"""
Return all existing values stored for a given tag.
@param tag: the full tag, e.g. 909C0b
@type tag: string
@return: the list of values
@rtype: list of strings
"""
table = 'bib%02dx' % int(tag[:2])
return [row[0] for row in run_sql("SELECT DISTINCT(value) FROM %s WHERE tag=%%s" % table, (tag, ))]
def get_most_popular_field_values(recids, tags, exclude_values=None, count_repetitive_values=True, split_by=0):
"""
Analyze RECIDS and look for TAGS and return most popular values
and the frequency with which they occur sorted according to
descending frequency.
If a value is found in EXCLUDE_VALUES, then do not count it.
If COUNT_REPETITIVE_VALUES is True, then we count every occurrence
of value in the tags. If False, then we count the value only once
regardless of the number of times it may appear in a record.
(But, if the same value occurs in another record, we count it, of
course.)
@return: list of tuples containing tag and its frequency
Example:
>>> get_most_popular_field_values(range(11,20), '980__a')
[('PREPRINT', 10), ('THESIS', 7), ...]
>>> get_most_popular_field_values(range(11,20), ('100__a', '700__a'))
[('Ellis, J', 10), ('Ellis, N', 7), ...]
>>> get_most_popular_field_values(range(11,20), ('100__a', '700__a'), ('Ellis, J'))
[('Ellis, N', 7), ...]
"""
def _get_most_popular_field_values_helper_sorter(val1, val2):
"""Compare VAL1 and VAL2 according to, firstly, frequency, then
secondly, alphabetically."""
compared_via_frequencies = cmp(valuefreqdict[val2],
valuefreqdict[val1])
if compared_via_frequencies == 0:
return cmp(val1.lower(), val2.lower())
else:
return compared_via_frequencies
valuefreqdict = {}
## sanity check:
if not exclude_values:
exclude_values = []
if isinstance(tags, str):
tags = (tags,)
## find values to count:
vals_to_count = []
displaytmp = {}
if count_repetitive_values:
# counting technique A: can look up many records at once: (very fast)
for tag in tags:
vals_to_count.extend(get_fieldvalues(recids, tag, sort=False,
split_by=split_by))
else:
# counting technique B: must count record-by-record: (slow)
for recid in recids:
vals_in_rec = []
for tag in tags:
for val in get_fieldvalues(recid, tag, False):
vals_in_rec.append(val)
# do not count repetitive values within this record
# (even across various tags, so need to unify again):
dtmp = {}
for val in vals_in_rec:
dtmp[val.lower()] = 1
displaytmp[val.lower()] = val
vals_in_rec = dtmp.keys()
vals_to_count.extend(vals_in_rec)
## are we to exclude some of found values?
for val in vals_to_count:
if val not in exclude_values:
if val in valuefreqdict:
valuefreqdict[val] += 1
else:
valuefreqdict[val] = 1
## sort by descending frequency of values:
if not CFG_NUMPY_IMPORTABLE:
## original version
out = []
vals = valuefreqdict.keys()
vals.sort(_get_most_popular_field_values_helper_sorter)
for val in vals:
tmpdisplv = ''
if val in displaytmp:
tmpdisplv = displaytmp[val]
else:
tmpdisplv = val
out.append((tmpdisplv, valuefreqdict[val]))
return out
else:
f = [] # frequencies
n = [] # original names
ln = [] # lowercased names
## build lists within one iteration
for (val, freq) in iteritems(valuefreqdict):
f.append(-1 * freq)
if val in displaytmp:
n.append(displaytmp[val])
else:
n.append(val)
ln.append(val.lower())
## sort by frequency (desc) and then by lowercased name.
return [(n[i], -1 * f[i]) for i in numpy.lexsort([ln, f])]
def profile(p="", f="", c=CFG_SITE_NAME):
"""Profile search time."""
import profile as pyprofile
import pstats
pyprofile.run("perform_request_search(p='%s',f='%s', c='%s')" % (p, f, c), "perform_request_search_profile")
p = pstats.Stats("perform_request_search_profile")
p.strip_dirs().sort_stats("cumulative").print_stats()
return 0
def perform_external_collection_search_with_em(req, current_collection, pattern_list, field,
external_collection, verbosity_level=0, lang=CFG_SITE_LANG,
selected_external_collections_infos=None, em=""):
perform_external_collection_search(req, current_collection, pattern_list, field, external_collection,
verbosity_level, lang, selected_external_collections_infos,
print_overview=em == "" or EM_REPOSITORY["overview"] in em,
print_search_info=em == "" or EM_REPOSITORY["search_info"] in em,
print_see_also_box=em == "" or EM_REPOSITORY["see_also_box"] in em,
print_body=em == "" or EM_REPOSITORY["body"] in em)
@cache.memoize(timeout=5)
def get_fulltext_terms_from_search_pattern(search_pattern):
keywords = []
if search_pattern is not None:
for unit in create_basic_search_units(None, search_pattern.encode('utf-8'), None):
bsu_o, bsu_p, bsu_f, bsu_m = unit[0], unit[1], unit[2], unit[3]
if (bsu_o != '-' and bsu_f in [None, 'fulltext']):
if bsu_m == 'a' and bsu_p.startswith('%') and bsu_p.endswith('%'):
# remove leading and training `%' representing partial phrase search
keywords.append(bsu_p[1:-1])
else:
keywords.append(bsu_p)
return keywords
def check_user_can_edit_record(req, recid):
""" Check if user has authorization to modify a collection
the recid belongs to
"""
record_collections = get_all_collections_of_a_record(recid)
if not record_collections:
# Check if user has access to all collections
auth_code, auth_message = acc_authorize_action(req, 'runbibedit',
collection='')
if auth_code == 0:
return True
else:
for collection in record_collections:
auth_code, auth_message = acc_authorize_action(req, 'runbibedit',
collection=collection)
if auth_code == 0:
return True
return False
diff --git a/invenio/legacy/webcomment/templates.py b/invenio/legacy/webcomment/templates.py
index 5c5216c57..ec43b443b 100644
--- a/invenio/legacy/webcomment/templates.py
+++ b/invenio/legacy/webcomment/templates.py
@@ -1,2541 +1,2549 @@
# -*- coding: utf-8 -*-
## Comments and reviews for records.
## This file is part of Invenio.
-## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013 CERN.
+## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""HTML Templates for commenting features """
__revision__ = "$Id$"
import cgi
# Invenio imports
from invenio.utils.url import create_html_link, create_url
from invenio.legacy.webuser import get_user_info, collect_user_info, isGuestUser, get_email
from invenio.utils.date import convert_datetext_to_dategui
from invenio.utils.mail import email_quoted_txt2html
from invenio.config import CFG_SITE_URL, \
CFG_SITE_SECURE_URL, \
CFG_BASE_URL, \
CFG_SITE_LANG, \
CFG_SITE_NAME, \
CFG_SITE_NAME_INTL,\
CFG_SITE_SUPPORT_EMAIL,\
CFG_WEBCOMMENT_ALLOW_REVIEWS, \
CFG_WEBCOMMENT_ALLOW_COMMENTS, \
CFG_WEBCOMMENT_USE_RICH_TEXT_EDITOR, \
CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN, \
CFG_WEBCOMMENT_AUTHOR_DELETE_COMMENT_OPTION, \
CFG_CERN_SITE, \
CFG_SITE_RECORD, \
CFG_WEBCOMMENT_MAX_ATTACHED_FILES, \
CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE
from invenio.utils.html import get_html_text_editor, create_html_select
from invenio.base.i18n import gettext_set_language
from invenio.modules.formatter import format_record
from invenio.modules.access.engine import acc_authorize_action
from invenio.modules.access.control import acc_get_user_roles_from_user_info, acc_get_role_id
from invenio.legacy.bibrecord import get_fieldvalues
class Template:
"""templating class, refer to webcomment.py for examples of call"""
def tmpl_get_first_comments_without_ranking(self, recID, ln, comments, nb_comments_total, warnings):
"""
@param recID: record id
@param ln: language
@param comments: tuple as returned from webcomment.py/query_retrieve_comments_or_remarks
@param nb_comments_total: total number of comments for this record
@param warnings: list of warning tuples (warning_text, warning_color)
@return: html of comments
"""
# load the right message language
_ = gettext_set_language(ln)
# naming data fields of comments
c_nickname = 0
c_user_id = 1
c_date_creation = 2
c_body = 3
c_id = 6
warnings = self.tmpl_warnings(warnings, ln)
# write button
write_button_label = _("Write a comment")
write_button_link = '%s/%s/%s/comments/add' % (CFG_SITE_URL, CFG_SITE_RECORD, recID)
write_button_form = '<input type="hidden" name="ln" value="%s"/>' % ln
write_button_form = self.createhiddenform(action=write_button_link, method="get", text=write_button_form, button=write_button_label)
# comments
comment_rows = ''
last_comment_round_name = None
comment_round_names = [comment[0] for comment in comments]
if comment_round_names:
last_comment_round_name = comment_round_names[-1]
for comment_round_name, comments_list in comments:
comment_rows += '<div id="cmtRound%s" class="cmtRound">' % (comment_round_name)
if comment_round_name:
comment_rows += '<div class="webcomment_comment_round_header">' + \
_('%(x_nb)i Comments for round "%(x_name)s"') % {'x_nb': len(comments_list), 'x_name': comment_round_name} + "</div>"
else:
comment_rows += '<div class="webcomment_comment_round_header">' + \
_('%(x_nb)i Comments') % {'x_nb': len(comments_list),} + "</div>"
for comment in comments_list:
if comment[c_nickname]:
nickname = comment[c_nickname]
display = nickname
else:
(uid, nickname, display) = get_user_info(comment[c_user_id])
messaging_link = self.create_messaging_link(nickname, display, ln)
comment_rows += """
<tr>
<td>"""
report_link = '%s/%s/%s/comments/report?ln=%s&amp;comid=%s' % (CFG_SITE_URL, CFG_SITE_RECORD, recID, ln, comment[c_id])
reply_link = '%s/%s/%s/comments/add?ln=%s&amp;comid=%s&amp;action=REPLY' % (CFG_SITE_URL, CFG_SITE_RECORD, recID, ln, comment[c_id])
comment_rows += self.tmpl_get_comment_without_ranking(req=None, ln=ln, nickname=messaging_link, comment_uid=comment[c_user_id],
date_creation=comment[c_date_creation],
body=comment[c_body], status='', nb_reports=0,
report_link=report_link, reply_link=reply_link, recID=recID)
comment_rows += """
<br />
<br />
</td>
</tr>"""
# Close comment round
comment_rows += '</div>'
# output
if nb_comments_total > 0:
out = warnings
comments_label = len(comments) > 1 and _("Showing the latest %(x_num)i comments:", x_num=len(comments)) \
or ""
out += """
<div class="video_content_clear"></div>
<table class="webcomment_header_comments">
<tr>
<td class="blocknote">%(comment_title)s</td>
</tr>
</table>
<div class="websomment_header_comments_label">%(comments_label)s</div>
%(comment_rows)s
%(view_all_comments_link)s
%(write_button_form)s<br />""" % \
{'comment_title': _("Discuss this document"),
'comments_label': comments_label,
'nb_comments_total' : nb_comments_total,
'recID': recID,
'comment_rows': comment_rows,
'tab': '&nbsp;'*4,
'siteurl': CFG_SITE_URL,
's': nb_comments_total>1 and 's' or "",
'view_all_comments_link': nb_comments_total>0 and '''<a class="webcomment_view_all_comments" href="%s/%s/%s/comments/display">View all %s comments</a>''' \
% (CFG_SITE_URL, CFG_SITE_RECORD, recID, nb_comments_total) or "",
'write_button_form': write_button_form,
'nb_comments': len(comments)
}
if not comments:
out = """
<!-- comments title table -->
<table class="webcomment_header_comments">
<tr>
<td class="blocknote">%(discuss_label)s:</td>
</tr>
</table>
<div class="webcomment_header_details">%(detailed_info)s
<br />
</div>
%(form)s
""" % {'form': write_button_form,
'discuss_label': _("Discuss this document"),
'detailed_info': _("Start a discussion about any aspect of this document.")
}
return out
def tmpl_record_not_found(self, status='missing', recID="", ln=CFG_SITE_LANG):
"""
Displays a page when bad or missing record ID was given.
@param status: 'missing' : no recID was given
'inexistant': recID doesn't have an entry in the database
+ 'deleted': : recID has been deleted
'nan' : recID is not a number
'invalid' : recID is an error code, i.e. in the interval [-99,-1]
@param return: body of the page
"""
_ = gettext_set_language(ln)
if status == 'inexistant':
body = _("Sorry, the record %(x_rec)s does not seem to exist.", x_rec=(recID,))
+ elif status in ('deleted'):
+ body = _("The record has been deleted.")
elif status in ('nan', 'invalid'):
body = _("Sorry, %(x_rec)s is not a valid ID value.", x_rec=(recID,))
else:
body = _("Sorry, no record ID was provided.")
body += "<br /><br />"
link = "<a href=\"%s?ln=%s\">%s</a>." % (CFG_SITE_URL, ln, CFG_SITE_NAME_INTL.get(ln, CFG_SITE_NAME))
body += _("You may want to start browsing from %(x_link)s", x_link=link)
return body
def tmpl_get_first_comments_with_ranking(self, recID, ln, comments=None, nb_comments_total=None, avg_score=None, warnings=[]):
"""
@param recID: record id
@param ln: language
@param comments: tuple as returned from webcomment.py/query_retrieve_comments_or_remarks
@param nb_comments_total: total number of comments for this record
@param avg_score: average score of all reviews
@param warnings: list of warning tuples (warning_text, warning_color)
@return: html of comments
"""
# load the right message language
_ = gettext_set_language(ln)
# naming data fields of comments
c_nickname = 0
c_user_id = 1
c_date_creation = 2
c_body = 3
c_nb_votes_yes = 4
c_nb_votes_total = 5
c_star_score = 6
c_title = 7
c_id = 8
warnings = self.tmpl_warnings(warnings, ln)
#stars
if avg_score > 0:
avg_score_img = 'stars-' + str(avg_score).split('.')[0] + '-' + str(avg_score).split('.')[1] + '.png'
else:
avg_score_img = "stars-0-0.png"
# voting links
useful_dict = { 'siteurl' : CFG_SITE_URL,
'CFG_SITE_RECORD' : CFG_SITE_RECORD,
'recID' : recID,
'ln' : ln,
'yes_img' : 'smchk_gr.gif', #'yes.gif',
'no_img' : 'iconcross.gif' #'no.gif'
}
link = '<a href="%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/reviews/vote?ln=%(ln)s&amp;comid=%%(comid)s' % useful_dict
useful_yes = link + '&amp;com_value=1">' + _("Yes") + '</a>'
useful_no = link + '&amp;com_value=-1">' + _("No") + '</a>'
#comment row
comment_rows = ' '
last_comment_round_name = None
comment_round_names = [comment[0] for comment in comments]
if comment_round_names:
last_comment_round_name = comment_round_names[-1]
for comment_round_name, comments_list in comments:
comment_rows += '<div id="cmtRound%s" class="cmtRound">' % (comment_round_name)
comment_rows += _('%(x_nb)i comments for round "%(x_name)s"') % {'x_nb': len(comments_list), 'x_name': comment_round_name} + "<br/>"
for comment in comments_list:
if comment[c_nickname]:
nickname = comment[c_nickname]
display = nickname
else:
(uid, nickname, display) = get_user_info(comment[c_user_id])
messaging_link = self.create_messaging_link(nickname, display, ln)
comment_rows += '''
<tr>
<td>'''
report_link = '%s/%s/%s/reviews/report?ln=%s&amp;comid=%s' % (CFG_SITE_URL, CFG_SITE_RECORD, recID, ln, comment[c_id])
comment_rows += self.tmpl_get_comment_with_ranking(None, ln=ln, nickname=messaging_link,
comment_uid=comment[c_user_id],
date_creation=comment[c_date_creation],
body=comment[c_body],
status='', nb_reports=0,
nb_votes_total=comment[c_nb_votes_total],
nb_votes_yes=comment[c_nb_votes_yes],
star_score=comment[c_star_score],
title=comment[c_title], report_link=report_link, recID=recID)
comment_rows += '''
%s %s / %s<br />''' % (_("Was this review helpful?"), useful_yes % {'comid':comment[c_id]}, useful_no % {'comid':comment[c_id]})
comment_rows += '''
<br />
</td>
</tr>'''
# Close comment round
comment_rows += '</div>'
# write button
write_button_link = '''%s/%s/%s/reviews/add''' % (CFG_SITE_URL, CFG_SITE_RECORD, recID)
write_button_form = ' <input type="hidden" name="ln" value="%s"/>' % ln
write_button_form = self.createhiddenform(action=write_button_link, method="get", text=write_button_form, button=_("Write a review"))
if nb_comments_total > 0:
avg_score_img = str(avg_score_img)
avg_score = str(avg_score)
nb_comments_total = str(nb_comments_total)
score = '<b>'
score += _("Average review score: %(x_nb_score)s based on %(x_nb_reviews)s reviews") % \
{'x_nb_score': '</b><img src="' + CFG_SITE_URL + '/img/' + avg_score_img + '" alt="' + avg_score + '" />',
'x_nb_reviews': nb_comments_total}
useful_label = _("Readers found the following %(x_name)s reviews to be most helpful.", x_name=len(comments) if len(comments) > 0 else '')
view_all_comments_link ='<a class"webcomment_view_all_reviews" href="%s/%s/%s/reviews/display?ln=%s&amp;do=hh">' % (CFG_SITE_URL, CFG_SITE_RECORD, recID, ln)
view_all_comments_link += _("View all %(x_name)s reviews", x_name=nb_comments_total)
view_all_comments_link += '</a><br />'
out = warnings + """
<!-- review title table -->
<table class="webcomment_header_ratings">
<tr>
<td class="blocknote">%(comment_title)s:</td>
</tr>
</table>
%(score_label)s<br />
%(useful_label)s
<!-- review table -->
<table class="webcomment_review_title_table">
%(comment_rows)s
</table>
%(view_all_comments_link)s
%(write_button_form)s<br />
""" % \
{ 'comment_title' : _("Rate this document"),
'score_label' : score,
'useful_label' : useful_label,
'recID' : recID,
'view_all_comments' : _("View all %(x_name)s reviews", x_name=(nb_comments_total,)),
'write_comment' : _("Write a review"),
'comment_rows' : comment_rows,
'tab' : '&nbsp;'*4,
'siteurl' : CFG_SITE_URL,
'view_all_comments_link': nb_comments_total>0 and view_all_comments_link or "",
'write_button_form' : write_button_form
}
else:
out = '''
<!-- review title table -->
<table class="webcomment_header_ratings">
<tr>
<td class="blocknote"><div class="webcomment_review_first_introduction">%s:</td>
</tr>
</table>
%s<br />
%s
<br />''' % (_("Rate this document"),
_('Be the first to review this document.</div>'),
write_button_form)
return out
def tmpl_get_comment_without_ranking(self, req, ln, nickname, comment_uid, date_creation, body, status, nb_reports, reply_link=None, report_link=None, undelete_link=None, delete_links=None, unreport_link=None, recID=-1, com_id='', attached_files=None, collapsed_p=False):
"""
private function
@param req: request object to fetch user info
@param ln: language
@param nickname: nickname
@param date_creation: date comment was written
@param body: comment body
@param status: status of the comment:
da: deleted by author
dm: deleted by moderator
ok: active
@param nb_reports: number of reports the comment has
@param reply_link: if want reply and report, give the http links
@param report_link: if want reply and report, give the http links
@param undelete_link: http link to delete the message
@param delete_links: http links to delete the message
@param unreport_link: http link to unreport the comment
@param recID: recID where the comment is posted
@param com_id: ID of the comment displayed
@param attached_files: list of attached files
@param collapsed_p: if the comment should be collapsed or not
@return: html table of comment
"""
from invenio.legacy.search_engine import guess_primary_collection_of_a_record
# load the right message language
_ = gettext_set_language(ln)
user_info = collect_user_info(req)
date_creation = convert_datetext_to_dategui(date_creation, ln=ln)
if attached_files is None:
attached_files = []
out = ''
final_body = email_quoted_txt2html(body)
title = nickname
title += '<a name="C%s" id="C%s"></a>' % (com_id, com_id)
links = ''
if not isGuestUser(user_info['uid']):
# Add link to toggle comment visibility
links += create_html_link(CFG_SITE_URL + '/' + CFG_SITE_RECORD + '/' + str(recID) + '/comments/toggle',
{'comid': com_id, 'ln': ln, 'collapse': collapsed_p and '0' or '1', 'referer': user_info['uri']},
_("Close"),
{'onclick': "return toggle_visibility(this, %s, 'fast');" % com_id},
escape_linkattrd=False)
moderator_links = ''
if reply_link:
links += '<a class="webcomment_comment_reply" href="' + reply_link +'">' + _("Reply") +'</a>'
if report_link and status != 'ap':
links += '<a class="webcomment_comment_report" href="' + report_link +'">' + _("Report abuse") + '</a>'
# Check if user is a comment moderator
record_primary_collection = guess_primary_collection_of_a_record(recID)
(auth_code, auth_msg) = acc_authorize_action(user_info, 'moderatecomments', collection=record_primary_collection)
if status in ['dm', 'da'] and req:
if not auth_code:
if status == 'dm':
final_body = '<div class="webcomment_deleted_comment_message">(Comment deleted by the moderator) - not visible for users<br /><br />' +\
final_body + '</div>'
else:
final_body = '<div class="webcomment_deleted_comment_message">(Comment deleted by the author) - not visible for users<br /><br />' +\
final_body + '</div>'
links = ''
moderator_links += '<a class="webcomment_deleted_comment_undelete" href="' + undelete_link + '">' + _("Undelete comment") + '</a>'
else:
if status == 'dm':
final_body = '<div class="webcomment_deleted_comment_message">Comment deleted by the moderator</div>'
else:
final_body = '<div class="webcomment_deleted_comment_message">Comment deleted by the author</div>'
links = ''
else:
if not auth_code:
moderator_links += '<a class="webcomment_comment_delete" href="' + delete_links['mod'] +'">' + _("Delete comment") + '</a>'
elif (user_info['uid'] == comment_uid) and CFG_WEBCOMMENT_AUTHOR_DELETE_COMMENT_OPTION:
moderator_links += '<a class="webcomment_comment_delete" href="' + delete_links['auth'] +'">' + _("Delete comment") + '</a>'
if nb_reports >= CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN:
if not auth_code:
final_body = '<div class="webcomment_reported_comment_message">(Comment reported. Pending approval) - not visible for users<br /><br />' + final_body + '</div>'
links = ''
moderator_links += '<a class="webcomment_reported_comment_unreport" href="' + unreport_link +'">' + _("Unreport comment") + '</a>'
else:
final_body = '<div class="webcomment_comment_pending_approval_message">This comment is pending approval due to user reports</div>'
links = ''
if links and moderator_links:
links = links + moderator_links
elif not links:
links = moderator_links
attached_files_html = ''
if attached_files:
attached_files_html = '<div class="cmtfilesblock"><b>%s:</b><br/>' % (len(attached_files) == 1 and _("Attached file") or _("Attached files"))
for (filename, filepath, fileurl) in attached_files:
attached_files_html += create_html_link(urlbase=fileurl, urlargd={},
link_label=cgi.escape(filename)) + '<br />'
attached_files_html += '</div>'
toggle_visibility_block = ''
if not isGuestUser(user_info['uid']):
toggle_visibility_block = """<div class="webcomment_toggle_visibility"><a id="collapsible_ctr_%(comid)s" class="%(collapse_ctr_class)s" href="%(toggle_url)s" onclick="return toggle_visibility(this, %(comid)i);" title="%(collapse_label)s"><span style="display:none">%(collapse_label)s</span></a></div>""" % \
{'comid': com_id,
'toggle_url': create_url(CFG_SITE_URL + '/' + CFG_SITE_RECORD + '/' + str(recID) + '/comments/toggle', {'comid': com_id, 'ln': ln, 'collapse': collapsed_p and '0' or '1', 'referer': user_info['uri']}),
'collapse_ctr_class': collapsed_p and 'webcomment_collapse_ctr_right' or 'webcomment_collapse_ctr_down',
'collapse_label': collapsed_p and _("Open") or _("Close")}
out += """
<div class="webcomment_comment_box">
%(toggle_visibility_block)s
<div class="webcomment_comment_avatar"><img class="webcomment_comment_avatar_default" src="%(site_url)s/img/user-icon-1-24x24.gif" alt="avatar" /></div>
<div class="webcomment_comment_content">
<div class="webcomment_comment_title">
%(title)s
<div class="webcomment_comment_date">%(date)s</div>
<a class="webcomment_permalink" title="Permalink to this comment" href="#C%(comid)i">¶</a>
</div>
<div class="collapsible_content" id="collapsible_content_%(comid)i" style="%(collapsible_content_style)s">
<blockquote>
%(body)s
</blockquote>
%(attached_files_html)s
<div class="webcomment_comment_options">%(links)s</div>
</div>
<div class="clearer"></div>
</div>
<div class="clearer"></div>
</div>""" % \
{'title' : title,
'body' : final_body,
'links' : links,
'attached_files_html': attached_files_html,
'date': date_creation,
'site_url': CFG_SITE_URL,
'comid': com_id,
'collapsible_content_style': collapsed_p and 'display:none' or '',
'toggle_visibility_block': toggle_visibility_block,
}
return out
def tmpl_get_comment_with_ranking(self, req, ln, nickname, comment_uid, date_creation, body, status, nb_reports, nb_votes_total, nb_votes_yes, star_score, title, report_link=None, delete_links=None, undelete_link=None, unreport_link=None, recID=-1):
"""
private function
@param req: request object to fetch user info
@param ln: language
@param nickname: nickname
@param date_creation: date comment was written
@param body: comment body
@param status: status of the comment
@param nb_reports: number of reports the comment has
@param nb_votes_total: total number of votes for this review
@param nb_votes_yes: number of positive votes for this record
@param star_score: star score for this record
@param title: title of review
@param report_link: if want reply and report, give the http links
@param undelete_link: http link to delete the message
@param delete_link: http link to delete the message
@param unreport_link: http link to unreport the comment
@param recID: recID where the comment is posted
@return: html table of review
"""
from invenio.legacy.search_engine import guess_primary_collection_of_a_record
# load the right message language
_ = gettext_set_language(ln)
if star_score > 0:
star_score_img = 'stars-' + str(star_score) + '-0.png'
else:
star_score_img = 'stars-0-0.png'
out = ""
date_creation = convert_datetext_to_dategui(date_creation, ln=ln)
reviewed_label = _("Reviewed by %(x_nickname)s on %(x_date)s") % {'x_nickname': nickname, 'x_date':date_creation}
## FIX
nb_votes_yes = str(nb_votes_yes)
nb_votes_total = str(nb_votes_total)
useful_label = _("%(x_nb_people)s out of %(x_nb_total)s people found this review useful") % {'x_nb_people': nb_votes_yes,
'x_nb_total': nb_votes_total}
links = ''
_body = ''
if body != '':
_body = '''
<blockquote>
%s
</blockquote>''' % email_quoted_txt2html(body, linebreak_html='')
# Check if user is a comment moderator
record_primary_collection = guess_primary_collection_of_a_record(recID)
user_info = collect_user_info(req)
(auth_code, auth_msg) = acc_authorize_action(user_info, 'moderatecomments', collection=record_primary_collection)
if status in ['dm', 'da'] and req:
if not auth_code:
if status == 'dm':
_body = '<div class="webcomment_deleted_review_message">(Review deleted by moderator) - not visible for users<br /><br />' +\
_body + '</div>'
else:
_body = '<div class="webcomment_deleted_review_message">(Review deleted by author) - not visible for users<br /><br />' +\
_body + '</div>'
links = '<a class="webcomment_deleted_review_undelete" href="' + undelete_link + '">' + _("Undelete review") + '</a>'
else:
if status == 'dm':
_body = '<div class="webcomment_deleted_review_message">Review deleted by moderator</div>'
else:
_body = '<div class="webcomment_deleted_review_message">Review deleted by author</div>'
links = ''
else:
if not auth_code:
links += '<a class="webcomment_review_delete" href="' + delete_links['mod'] +'">' + _("Delete review") + '</a>'
if nb_reports >= CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN:
if not auth_code:
_body = '<div class="webcomment_review_pending_approval_message">(Review reported. Pending approval) - not visible for users<br /><br />' + _body + '</div>'
links += ' | '
links += '<a class="webcomment_reported_review_unreport" href="' + unreport_link +'">' + _("Unreport review") + '</a>'
else:
_body = '<div class="webcomment_review_pending_approval_message">This review is pending approval due to user reports.</div>'
links = ''
out += '''
<div class="webcomment_review_box">
<div class="webcomment_review_box_inner">
<img src="%(baseurl)s/img/%(star_score_img)s" alt="%(star_score)s/>
<div class="webcomment_review_title">%(title)s</div>
<div class="webcomment_review_label_reviewed">%(reviewed_label)s</div>
<div class="webcomment_review_label_useful">%(useful_label)s</div>
%(body)s
</div>
</div>
%(abuse)s''' % {'baseurl' : CFG_BASE_URL,
'star_score_img': star_score_img,
'star_score' : star_score,
'title' : cgi.escape(title),
'reviewed_label': reviewed_label,
'useful_label' : useful_label,
'body' : _body,
'abuse' : links
}
return out
def tmpl_get_comments(self, req, recID, ln,
nb_per_page, page, nb_pages,
display_order, display_since,
CFG_WEBCOMMENT_ALLOW_REVIEWS,
comments, total_nb_comments,
avg_score,
warnings,
border=0, reviews=0,
total_nb_reviews=0,
nickname='', uid=-1, note='',score=5,
can_send_comments=False,
can_attach_files=False,
user_is_subscribed_to_discussion=False,
user_can_unsubscribe_from_discussion=False,
display_comment_rounds=None):
"""
Get table of all comments
@param recID: record id
@param ln: language
@param nb_per_page: number of results per page
@param page: page number
@param display_order: hh = highest helpful score, review only
lh = lowest helpful score, review only
hs = highest star score, review only
ls = lowest star score, review only
od = oldest date
nd = newest date
@param display_since: all= no filtering by date
nd = n days ago
nw = n weeks ago
nm = n months ago
ny = n years ago
where n is a single digit integer between 0 and 9
@param CFG_WEBCOMMENT_ALLOW_REVIEWS: is ranking enable, get from config.py/CFG_WEBCOMMENT_ALLOW_REVIEWS
@param comments: tuple as returned from webcomment.py/query_retrieve_comments_or_remarks
@param total_nb_comments: total number of comments for this record
@param avg_score: average score of reviews for this record
@param warnings: list of warning tuples (warning_text, warning_color)
@param border: boolean, active if want to show border around each comment/review
@param reviews: boolean, enabled for reviews, disabled for comments
@param can_send_comments: boolean, if user can send comments or not
@param can_attach_files: boolean, if user can attach file to comment or not
@param user_is_subscribed_to_discussion: True if user already receives new comments by email
@param user_can_unsubscribe_from_discussion: True is user is allowed to unsubscribe from discussion
"""
# load the right message language
_ = gettext_set_language(ln)
# CERN hack begins: display full ATLAS user name. Check further below too.
current_user_fullname = ""
override_nickname_p = False
if CFG_CERN_SITE:
from invenio.legacy.search_engine import get_all_collections_of_a_record
user_info = collect_user_info(uid)
if 'atlas-readaccess-active-members [CERN]' in user_info['group']:
# An ATLAS member is never anonymous to its colleagues
# when commenting inside ATLAS collections
recid_collections = get_all_collections_of_a_record(recID)
if 'ATLAS' in str(recid_collections):
override_nickname_p = True
current_user_fullname = user_info.get('external_fullname', '')
# CERN hack ends
# naming data fields of comments
if reviews:
c_nickname = 0
c_user_id = 1
c_date_creation = 2
c_body = 3
c_status = 4
c_nb_reports = 5
c_nb_votes_yes = 6
c_nb_votes_total = 7
c_star_score = 8
c_title = 9
c_id = 10
c_round_name = 11
c_restriction = 12
reply_to = 13
c_visibility = 14
discussion = 'reviews'
comments_link = '<a href="%s/%s/%s/comments/">%s</a> (%i)' % (CFG_SITE_URL, CFG_SITE_RECORD, recID, _('Comments'), total_nb_comments)
reviews_link = '<b>%s (%i)</b>' % (_('Reviews'), total_nb_reviews)
add_comment_or_review = self.tmpl_add_comment_form_with_ranking(recID, uid, current_user_fullname or nickname, ln, '', score, note, warnings, show_title_p=True, can_attach_files=can_attach_files)
else:
c_nickname = 0
c_user_id = 1
c_date_creation = 2
c_body = 3
c_status = 4
c_nb_reports = 5
c_id = 6
c_round_name = 7
c_restriction = 8
reply_to = 9
c_visibility = 10
discussion = 'comments'
comments_link = '<b>%s (%i)</b>' % (_('Comments'), total_nb_comments)
reviews_link = '<a href="%s/%s/%s/reviews/">%s</a> (%i)' % (CFG_SITE_URL, CFG_SITE_RECORD, recID, _('Reviews'), total_nb_reviews)
add_comment_or_review = self.tmpl_add_comment_form(recID, uid, nickname, ln, note, warnings, can_attach_files=can_attach_files, user_is_subscribed_to_discussion=user_is_subscribed_to_discussion)
# voting links
useful_dict = { 'siteurl' : CFG_SITE_URL,
'CFG_SITE_RECORD' : CFG_SITE_RECORD,
'recID' : recID,
'ln' : ln,
'do' : display_order,
'ds' : display_since,
'nb' : nb_per_page,
'p' : page,
'reviews' : reviews,
'discussion' : discussion
}
useful_yes = '<a href="%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/%(discussion)s/vote?ln=%(ln)s&amp;comid=%%(comid)s&amp;com_value=1&amp;do=%(do)s&amp;ds=%(ds)s&amp;nb=%(nb)s&amp;p=%(p)s&amp;referer=%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/%(discussion)s/display">' + _("Yes") + '</a>'
useful_yes %= useful_dict
useful_no = '<a href="%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/%(discussion)s/vote?ln=%(ln)s&amp;comid=%%(comid)s&amp;com_value=-1&amp;do=%(do)s&amp;ds=%(ds)s&amp;nb=%(nb)s&amp;p=%(p)s&amp;referer=%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/%(discussion)s/display">' + _("No") + '</a>'
useful_no %= useful_dict
warnings = self.tmpl_warnings(warnings, ln)
link_dic = { 'siteurl' : CFG_SITE_URL,
'CFG_SITE_RECORD' : CFG_SITE_RECORD,
'module' : 'comments',
'function' : 'index',
'discussion': discussion,
'arguments' : 'do=%s&amp;ds=%s&amp;nb=%s' % (display_order, display_since, nb_per_page),
'arg_page' : '&amp;p=%s' % page,
'page' : page,
'rec_id' : recID}
if not req:
req = None
## comments table
comments_rows = ''
last_comment_round_name = None
comment_round_names = [comment[0] for comment in comments]
if comment_round_names:
last_comment_round_name = comment_round_names[-1]
for comment_round_name, comments_list in comments:
comment_round_style = "display:none;"
comment_round_is_open = False
if comment_round_name in display_comment_rounds:
comment_round_is_open = True
comment_round_style = ""
comments_rows += '<div id="cmtRound%s" class="cmtround">' % (comment_round_name)
if not comment_round_is_open and \
(comment_round_name or len(comment_round_names) > 1):
new_cmtgrp = list(display_comment_rounds)
new_cmtgrp.append(comment_round_name)
comments_rows += '''<img src="/img/right-trans.gif" id="cmtarrowiconright%(grp_id)s" alt="Open group" /><img src="/img/down-trans.gif" id="cmtarrowicondown%(grp_id)s" alt="Close group" style="display:none" />
<a class="cmtgrpswitch" name="cmtgrpLink%(grp_id)s" onclick="var cmtarrowicondown=document.getElementById('cmtarrowicondown%(grp_id)s');var cmtarrowiconright=document.getElementById('cmtarrowiconright%(grp_id)s');var subgrp=document.getElementById('cmtSubRound%(grp_id)s');if (subgrp.style.display==''){subgrp.style.display='none';cmtarrowiconright.style.display='';cmtarrowicondown.style.display='none';}else{subgrp.style.display='';cmtarrowiconright.style.display='none';cmtarrowicondown.style.display='';};return false;"''' % {'grp_id': comment_round_name}
comments_rows += 'href=\"%(siteurl)s/%(CFG_SITE_RECORD)s/%(rec_id)s/%(discussion)s/%(function)s?%(arguments)s&amp;%(arg_page)s' % link_dic
comments_rows += '&amp;' + '&amp;'.join(["cmtgrp=" + grp for grp in new_cmtgrp if grp != 'none']) + \
'#cmtgrpLink%s' % (comment_round_name) + '\">'
comments_rows += _('%(x_nb)i comments for round "%(x_name)s"') % {'x_nb': len(comments_list), 'x_name': comment_round_name} + "</a><br/>"
elif comment_round_name or len(comment_round_names) > 1:
new_cmtgrp = list(display_comment_rounds)
new_cmtgrp.remove(comment_round_name)
comments_rows += '''<img src="/img/right-trans.gif" id="cmtarrowiconright%(grp_id)s" alt="Open group" style="display:none" /><img src="/img/down-trans.gif" id="cmtarrowicondown%(grp_id)s" alt="Close group" />
<a class="cmtgrpswitch" name="cmtgrpLink%(grp_id)s" onclick="var cmtarrowicondown=document.getElementById('cmtarrowicondown%(grp_id)s');var cmtarrowiconright=document.getElementById('cmtarrowiconright%(grp_id)s');var subgrp=document.getElementById('cmtSubRound%(grp_id)s');if (subgrp.style.display==''){subgrp.style.display='none';cmtarrowiconright.style.display='';cmtarrowicondown.style.display='none';}else{subgrp.style.display='';cmtarrowiconright.style.display='none';cmtarrowicondown.style.display='';};return false;"''' % {'grp_id': comment_round_name}
comments_rows += 'href=\"%(siteurl)s/%(CFG_SITE_RECORD)s/%(rec_id)s/%(discussion)s/%(function)s?%(arguments)s&amp;%(arg_page)s' % link_dic
comments_rows += '&amp;' + ('&amp;'.join(["cmtgrp=" + grp for grp in new_cmtgrp if grp != 'none']) or 'cmtgrp=none' ) + \
'#cmtgrpLink%s' % (comment_round_name) + '\">'
comments_rows += _('%(x_nb)i comments for round "%(x_name)s"') % {'x_nb': len(comments_list), 'x_name': comment_round_name}+ "</a><br/>"
comments_rows += '<div id="cmtSubRound%s" class="cmtsubround" style="%s">' % (comment_round_name,
comment_round_style)
comments_rows += '''
<script type='text/javascript'>//<![CDATA[
function toggle_visibility(this_link, comid, duration) {
if (duration == null) duration = 0;
var isVisible = $('#collapsible_content_' + comid).is(':visible');
$('#collapsible_content_' + comid).toggle(duration);
$('#collapsible_ctr_' + comid).toggleClass('webcomment_collapse_ctr_down');
$('#collapsible_ctr_' + comid).toggleClass('webcomment_collapse_ctr_right');
if (isVisible){
$('#collapsible_ctr_' + comid).attr('title', '%(open_label)s');
$('#collapsible_ctr_' + comid + ' > span').html('%(open_label)s');
} else {
$('#collapsible_ctr_' + comid).attr('title', '%(close_label)s');
$('#collapsible_ctr_' + comid + ' > span').html('%(close_label)s');
}
$.ajax({
type: 'POST',
url: '%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/comments/toggle',
data: {'comid': comid, 'ln': '%(ln)s', 'collapse': isVisible && 1 || 0}
});
/* Replace our link with a jump to the adequate, in case needed
(default link is for non-Javascript user) */
this_link.href = "#C" + comid
/* Find out if after closing comment we shall scroll a bit to the top,
i.e. go back to main anchor of the comment that we have just set */
var top = $(window).scrollTop();
if ($(window).scrollTop() >= $("#C" + comid).offset().top) {
// Our comment is now above the window: scroll to it
return true;
}
return false;
}
//]]></script>
''' % {'siteurl': CFG_SITE_URL,
'recID': recID,
'ln': ln,
'CFG_SITE_RECORD': CFG_SITE_RECORD,
'open_label': _("Open"),
'close_label': _("Close")}
thread_history = [0]
previous_depth = 0
for comment in comments_list:
if comment[reply_to] not in thread_history:
# Going one level down in the thread
thread_history.append(comment[reply_to])
depth = thread_history.index(comment[reply_to])
else:
depth = thread_history.index(comment[reply_to])
thread_history = thread_history[:depth + 1]
if previous_depth > depth:
comments_rows += ("""</div>""" * (previous_depth-depth))
if previous_depth < depth:
comments_rows += ("""<div class="webcomment_thread_block">""" * (depth-previous_depth))
previous_depth = depth
# CERN hack begins: display full ATLAS user name.
comment_user_fullname = ""
if CFG_CERN_SITE and override_nickname_p:
comment_user_fullname = get_email(comment[c_user_id])
# CERN hack ends
if comment[c_nickname]:
_nickname = comment[c_nickname]
display = _nickname
else:
(uid, _nickname, display) = get_user_info(comment[c_user_id])
messaging_link = self.create_messaging_link(_nickname, comment_user_fullname or display, ln)
from invenio.modules.comments.api import get_attached_files # FIXME
files = get_attached_files(recID, comment[c_id])
# do NOT delete the HTML comment below. It is used for parsing... (I plead unguilty!)
comments_rows += """
<!-- start comment row -->
<div>"""
delete_links = {}
if not reviews:
report_link = '%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/comments/report?ln=%(ln)s&amp;comid=%%(comid)s&amp;do=%(do)s&amp;ds=%(ds)s&amp;nb=%(nb)s&amp;p=%(p)s&amp;referer=%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/comments/display' % useful_dict % {'comid':comment[c_id]}
reply_link = '%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/comments/add?ln=%(ln)s&amp;action=REPLY&amp;comid=%%(comid)s' % useful_dict % {'comid':comment[c_id]}
delete_links['mod'] = "%s/admin/webcomment/webcommentadmin.py/del_single_com_mod?ln=%s&amp;id=%s" % (CFG_SITE_URL, ln, comment[c_id])
delete_links['auth'] = "%s/admin/webcomment/webcommentadmin.py/del_single_com_auth?ln=%s&amp;id=%s" % (CFG_SITE_URL, ln, comment[c_id])
undelete_link = "%s/admin/webcomment/webcommentadmin.py/undel_com?ln=%s&amp;id=%s" % (CFG_SITE_URL, ln, comment[c_id])
unreport_link = "%s/admin/webcomment/webcommentadmin.py/unreport_com?ln=%s&amp;id=%s" % (CFG_SITE_URL, ln, comment[c_id])
comments_rows += self.tmpl_get_comment_without_ranking(req, ln, messaging_link, comment[c_user_id], comment[c_date_creation], comment[c_body], comment[c_status], comment[c_nb_reports], reply_link, report_link, undelete_link, delete_links, unreport_link, recID, comment[c_id], files, comment[c_visibility])
else:
report_link = '%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/reviews/report?ln=%(ln)s&amp;comid=%%(comid)s&amp;do=%(do)s&amp;ds=%(ds)s&amp;nb=%(nb)s&amp;p=%(p)s&amp;referer=%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/reviews/display' % useful_dict % {'comid': comment[c_id]}
delete_links['mod'] = "%s/admin/webcomment/webcommentadmin.py/del_single_com_mod?ln=%s&amp;id=%s" % (CFG_SITE_URL, ln, comment[c_id])
delete_links['auth'] = "%s/admin/webcomment/webcommentadmin.py/del_single_com_auth?ln=%s&amp;id=%s" % (CFG_SITE_URL, ln, comment[c_id])
undelete_link = "%s/admin/webcomment/webcommentadmin.py/undel_com?ln=%s&amp;id=%s" % (CFG_SITE_URL, ln, comment[c_id])
unreport_link = "%s/admin/webcomment/webcommentadmin.py/unreport_com?ln=%s&amp;id=%s" % (CFG_SITE_URL, ln, comment[c_id])
comments_rows += self.tmpl_get_comment_with_ranking(req, ln, messaging_link, comment[c_user_id], comment[c_date_creation], comment[c_body], comment[c_status], comment[c_nb_reports], comment[c_nb_votes_total], comment[c_nb_votes_yes], comment[c_star_score], comment[c_title], report_link, delete_links, undelete_link, unreport_link, recID)
helpful_label = _("Was this review helpful?")
report_abuse_label = "(" + _("Report abuse") + ")"
yes_no_separator = '<td> / </td>'
if comment[c_nb_reports] >= CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN or comment[c_status] in ['dm', 'da']:
report_abuse_label = ""
helpful_label = ""
useful_yes = ""
useful_no = ""
yes_no_separator = ""
comments_rows += """
<table>
<tr>
<td>%(helpful_label)s %(tab)s</td>
<td> %(yes)s </td>
%(yes_no_separator)s
<td> %(no)s </td>
<td class="reportabuse">%(tab)s%(tab)s<a href="%(report)s">%(report_abuse_label)s</a></td>
</tr>
</table>""" \
% {'helpful_label': helpful_label,
'yes' : useful_yes % {'comid':comment[c_id]},
'yes_no_separator': yes_no_separator,
'no' : useful_no % {'comid':comment[c_id]},
'report' : report_link % {'comid':comment[c_id]},
'report_abuse_label': comment[c_nb_reports] >= CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN and '' or report_abuse_label,
'tab' : '&nbsp;'*2}
# do NOT remove HTML comment below. It is used for parsing...
comments_rows += """
</div>
<!-- end comment row -->"""
comments_rows += '</div></div>'
## page links
page_links = ''
# Previous
if page != 1:
link_dic['arg_page'] = 'p=%s' % (page - 1)
page_links += '<a href=\"%(siteurl)s/%(CFG_SITE_RECORD)s/%(rec_id)s/%(discussion)s/%(function)s?%(arguments)s&amp;%(arg_page)s\">&lt;&lt;</a> ' % link_dic
else:
page_links += ' %s ' % ('&nbsp;'*(len(_('Previous'))+7))
# Page Numbers
for i in range(1, nb_pages+1):
link_dic['arg_page'] = 'p=%s' % i
link_dic['page'] = '%s' % i
if i != page:
page_links += '''
<a href=\"%(siteurl)s/%(CFG_SITE_RECORD)s/%(rec_id)s/%(discussion)s/%(function)s?%(arguments)s&amp;%(arg_page)s\">%(page)s</a> ''' % link_dic
else:
page_links += ''' <b>%s</b> ''' % i
# Next
if page != nb_pages:
link_dic['arg_page'] = 'p=%s' % (page + 1)
page_links += '''
<a href=\"%(siteurl)s/%(CFG_SITE_RECORD)s/%(rec_id)s/%(discussion)s/%(function)s?%(arguments)s&amp;%(arg_page)s\">&gt;&gt;</a> ''' % link_dic
else:
page_links += '%s' % ('&nbsp;'*(len(_('Next'))+7))
## stuff for ranking if enabled
if reviews:
if avg_score > 0:
avg_score_img = 'stars-' + str(avg_score).split('.')[0] + '-' + str(avg_score).split('.')[1] + '.png'
else:
avg_score_img = "stars-0-0.png"
ranking_average = '<br /><b>'
ranking_average += _("Average review score: %(x_nb_score)s based on %(x_nb_reviews)s reviews") % \
{'x_nb_score': '</b><img src="' + CFG_SITE_URL + '/img/' + avg_score_img + '" alt="' + str(avg_score) + '" />',
'x_nb_reviews': str(total_nb_reviews)}
ranking_average += '<br />'
else:
ranking_average = ""
write_button_link = '''%s/%s/%s/%s/add''' % (CFG_SITE_URL, CFG_SITE_RECORD, recID, discussion)
write_button_form = '<input type="hidden" name="ln" value="%s"/>'
write_button_form = self.createhiddenform(action=write_button_link,
method="get",
text=write_button_form,
button = reviews and _('Write a review') or _('Write a comment'))
if reviews:
total_label = _("There is a total of %(x_num)s reviews", x_num=total_nb_comments)
else:
total_label = _("There is a total of %(x_num)s comments", x_num=total_nb_comments)
#total_label %= total_nb_comments
review_or_comment_first = ''
if reviews == 0 and total_nb_comments == 0 and can_send_comments:
review_or_comment_first = _("Start a discussion about any aspect of this document.") + '<br />'
elif reviews == 1 and total_nb_reviews == 0 and can_send_comments:
review_or_comment_first = _("Be the first to review this document.") + '<br />'
# do NOT remove the HTML comments below. Used for parsing
body = '''
%(comments_and_review_tabs)s
<!-- start comments table -->
<div class="webcomment_comment_table">
%(comments_rows)s
</div>
<!-- end comments table -->
%(review_or_comment_first)s
<br />''' % \
{ 'record_label': _("Record"),
'back_label': _("Back to search results"),
'total_label': total_label,
'write_button_form' : write_button_form,
'write_button_form_again' : total_nb_comments>3 and write_button_form or "",
'comments_rows' : comments_rows,
'total_nb_comments' : total_nb_comments,
'comments_or_reviews' : reviews and _('review') or _('comment'),
'comments_or_reviews_title' : reviews and _('Review') or _('Comment'),
'siteurl' : CFG_SITE_URL,
'module' : "comments",
'recid' : recID,
'ln' : ln,
#'border' : border,
'ranking_avg' : ranking_average,
'comments_and_review_tabs' : CFG_WEBCOMMENT_ALLOW_REVIEWS and \
CFG_WEBCOMMENT_ALLOW_COMMENTS and \
'%s | %s <br />' % \
(comments_link, reviews_link) or '',
'review_or_comment_first' : review_or_comment_first
}
# form is not currently used. reserved for an eventual purpose
#form = """
# Display <select name="nb" size="1"> per page
# <option value="all">All</option>
# <option value="10">10</option>
# <option value="25">20</option>
# <option value="50">50</option>
# <option value="100" selected="selected">100</option>
# </select>
# comments per page that are <select name="ds" size="1">
# <option value="all" selected="selected">Any age</option>
# <option value="1d">1 day old</option>
# <option value="3d">3 days old</option>
# <option value="1w">1 week old</option>
# <option value="2w">2 weeks old</option>
# <option value="1m">1 month old</option>
# <option value="3m">3 months old</option>
# <option value="6m">6 months old</option>
# <option value="1y">1 year old</option>
# </select>
# and sorted by <select name="do" size="1">
# <option value="od" selected="selected">Oldest first</option>
# <option value="nd">Newest first</option>
# %s
# </select>
# """ % \
# (reviews==1 and '''
# <option value=\"hh\">most helpful</option>
# <option value=\"lh\">least helpful</option>
# <option value=\"hs\">highest star ranking</option>
# <option value=\"ls\">lowest star ranking</option>
# </select>''' or '''
# </select>''')
#
#form_link = "%(siteurl)s/%(module)s/%(function)s" % link_dic
#form = self.createhiddenform(action=form_link, method="get", text=form, button='Go', recid=recID, p=1)
pages = """
<div>
%(v_label)s %(comments_or_reviews)s %(results_nb_lower)s-%(results_nb_higher)s <br />
%(page_links)s
</div>
""" % \
{'v_label': _("Viewing"),
'page_links': _("Page:") + page_links ,
'comments_or_reviews': reviews and _('review') or _('comment'),
'results_nb_lower': len(comments)>0 and ((page-1) * nb_per_page)+1 or 0,
'results_nb_higher': page == nb_pages and (((page-1) * nb_per_page) + len(comments)) or (page * nb_per_page)}
if nb_pages > 1:
#body = warnings + body + form + pages
body = warnings + body + pages
else:
body = warnings + body
if reviews == 0:
if not user_is_subscribed_to_discussion:
body += '<div class="comment-subscribe">' + '<img src="%s/img/mail-icon-12x8.gif" border="0" alt="" />' % CFG_SITE_URL + \
'&nbsp;' + '<b>' + create_html_link(urlbase=CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + \
str(recID) + '/comments/subscribe',
urlargd={},
link_label=_('Subscribe')) + \
'</b>' + ' to this discussion. You will then receive all new comments by email.' + '</div>'
body += '<br />'
elif user_can_unsubscribe_from_discussion:
body += '<div class="comment-subscribe">' + '<img src="%s/img/mail-icon-12x8.gif" border="0" alt="" />' % CFG_SITE_URL + \
'&nbsp;' + '<b>' + create_html_link(urlbase=CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + \
str(recID) + '/comments/unsubscribe',
urlargd={},
link_label=_('Unsubscribe')) + \
'</b>' + ' from this discussion. You will no longer receive emails about new comments.' + '</div>'
body += '<br />'
if can_send_comments:
body += add_comment_or_review
else:
body += '<br/><em>' + _("You are not authorized to comment or review.") + '</em>'
return '<div class="webcomment_container">' + body + '</div>'
def create_messaging_link(self, to, display_name, ln=CFG_SITE_LANG):
"""prints a link to the messaging system"""
link = "%s/yourmessages/write?msg_to=%s&amp;ln=%s" % (CFG_SITE_URL, to, ln)
if to:
return '<a href="%s" class="maillink">%s</a>' % (link, display_name)
else:
return display_name
def createhiddenform(self, action="", method="get", text="", button="confirm", cnfrm='', **hidden):
"""
create select with hidden values and submit button
@param action: name of the action to perform on submit
@param method: 'get' or 'post'
@param text: additional text, can also be used to add non hidden input
@param button: value/caption on the submit button
@param cnfrm: if given, must check checkbox to confirm
@param **hidden: dictionary with name=value pairs for hidden input
@return: html form
"""
output = """
<form action="%s" method="%s">""" % (action, method.lower().strip() in ['get', 'post'] and method or 'get')
output += """
<table style="width:90%">
<tr>
<td style="vertical-align: top">
"""
output += text + '\n'
if cnfrm:
output += """
<input type="checkbox" name="confirm" value="1" />"""
for key in hidden.keys():
if type(hidden[key]) is list:
for value in hidden[key]:
output += """
<input type="hidden" name="%s" value="%s" />""" % (key, value)
else:
output += """
<input type="hidden" name="%s" value="%s" />""" % (key, hidden[key])
output += """
</td>
</tr>
<tr>
<td>"""
output += """
<input class="adminbutton" type="submit" value="%s" />""" % (button, )
output += """
</td>
</tr>
</table>
</form>"""
return output
def create_write_comment_hiddenform(self, action="", method="get", text="", button="confirm", cnfrm='',
enctype='', form_id=None, form_name=None, **hidden):
"""
create select with hidden values and submit button
@param action: name of the action to perform on submit
@param method: 'get' or 'post'
@param text: additional text, can also be used to add non hidden input
@param button: value/caption on the submit button
@param cnfrm: if given, must check checkbox to confirm
@param form_id: HTML 'id' attribute of the form tag
@param form_name: HTML 'name' attribute of the form tag
@param **hidden: dictionary with name=value pairs for hidden input
@return: html form
"""
enctype_attr = ''
if enctype:
enctype_attr = 'enctype="%s"' % enctype
output = """
<form action="%s" method="%s" %s%s%s>""" % \
(action, method.lower().strip() in ['get', 'post'] and method or 'get',
enctype_attr, form_name and ' name="%s"' % form_name or '',
form_id and ' id="%s"' % form_id or '')
if cnfrm:
output += """
<input type="checkbox" name="confirm" value="1" />"""
for key in hidden.keys():
if type(hidden[key]) is list:
for value in hidden[key]:
output += """
<input type="hidden" name="%s" value="%s" />""" % (key, value)
else:
output += """
<input type="hidden" name="%s" value="%s" />""" % (key, hidden[key])
output += text + '\n'
output += """
</form>"""
return output
def tmpl_warnings(self, warnings=[], ln=CFG_SITE_LANG):
"""
Display len(warnings) warning fields
@param warnings: list of warning tuples (warning_text, warning_color)
@param ln=language
@return: html output
"""
if type(warnings) is not list:
warnings = [warnings]
warningbox = ""
if warnings:
for i in range(len(warnings)):
warning_text = warnings[i][0]
warning_color = warnings[i][1]
if warning_color == 'green':
span_class = 'exampleleader'
else:
span_class = 'important'
warningbox += '''
<span class="%(span_class)s">%(warning)s</span><br />''' % \
{ 'span_class' : span_class,
'warning' : warning_text }
return warningbox
else:
return ""
def tmpl_error(self, error, ln=CFG_SITE_LANG):
"""
Display error
@param error: string
@param ln=language
@return: html output
"""
_ = gettext_set_language(ln)
errorbox = ""
if error != "":
errorbox = "<div class=\"errorbox\">\n <b>Error:</b>\n"
errorbox += " <p>"
errorbox += error + " </p>"
errorbox += "</div><br />\n"
return errorbox
def tmpl_add_comment_form(self, recID, uid, nickname, ln, msg,
warnings, textual_msg=None, can_attach_files=False,
user_is_subscribed_to_discussion=False, reply_to=None):
"""
Add form for comments
@param recID: record id
@param uid: user id
@param ln: language
@param msg: comment body contents for when refreshing due to
warning, or when replying to a comment
@param textual_msg: same as 'msg', but contains the textual
version in case user cannot display CKeditor
@param warnings: list of warning tuples (warning_text, warning_color)
@param can_attach_files: if user can upload attach file to record or not
@param user_is_subscribed_to_discussion: True if user already receives new comments by email
@param reply_to: the ID of the comment we are replying to. None if not replying
@return html add comment form
"""
_ = gettext_set_language(ln)
link_dic = { 'siteurl' : CFG_SITE_URL,
'CFG_SITE_RECORD' : CFG_SITE_RECORD,
'module' : 'comments',
'function' : 'add',
'arguments' : 'ln=%s&amp;action=%s' % (ln, 'SUBMIT'),
'recID' : recID}
if textual_msg is None:
textual_msg = msg
# FIXME a cleaner handling of nicknames is needed.
if not nickname:
(uid, nickname, display) = get_user_info(uid)
if nickname:
note = _("Note: Your nickname, %(nick)s, will be displayed as author of this comment.",
nick='<i>' + nickname + '</i>')
else:
(uid, nickname, display) = get_user_info(uid)
link = '<a href="%s/youraccount/edit">' % CFG_SITE_SECURE_URL
note = _("Note: you have not %(x_url_open)sdefined your nickname%(x_url_close)s. %(x_nickname)s will be displayed as the author of this comment.") % \
{'x_url_open': link,
'x_url_close': '</a>',
'x_nickname': ' <br /><i>' + display + '</i>'}
if not CFG_WEBCOMMENT_USE_RICH_TEXT_EDITOR:
note += '<br />' + '&nbsp;'*10 + cgi.escape('You can use some HTML tags: <a href>, <strong>, <blockquote>, <br />, <p>, <em>, <ul>, <li>, <b>, <i>')
#from invenio.legacy.search_engine import print_record
#record_details = print_record(recID=recID, format='hb', ln=ln)
warnings = self.tmpl_warnings(warnings, ln)
# Prepare file upload settings. We must enable file upload in
# the ckeditor + a simple file upload interface (independant from editor)
file_upload_url = None
simple_attach_file_interface = ''
if isGuestUser(uid):
simple_attach_file_interface = "<small><em>%s</em></small><br/>" % _("Once logged in, authorized users can also attach files.")
if can_attach_files:
# Note that files can be uploaded only when user is logged in
#file_upload_url = '%s/%s/%i/comments/attachments/put' % \
# (CFG_SITE_URL, CFG_SITE_RECORD, recID)
simple_attach_file_interface = '''
<div id="uploadcommentattachmentsinterface">
<small>%(attach_msg)s: <em>(%(nb_files_limit_msg)s. %(file_size_limit_msg)s)</em></small><br />
<input class="multi max-%(CFG_WEBCOMMENT_MAX_ATTACHED_FILES)s" type="file" name="commentattachment[]"/><br />
<noscript>
<input type="file" name="commentattachment[]" /><br />
</noscript>
</div>
''' % \
{'CFG_WEBCOMMENT_MAX_ATTACHED_FILES': CFG_WEBCOMMENT_MAX_ATTACHED_FILES,
'attach_msg': CFG_WEBCOMMENT_MAX_ATTACHED_FILES == 1 and _("Optionally, attach a file to this comment") or \
_("Optionally, attach files to this comment"),
'nb_files_limit_msg': _("Max one file") and CFG_WEBCOMMENT_MAX_ATTACHED_FILES == 1 or \
_("Max %(maxfiles)i files", maxfiles=CFG_WEBCOMMENT_MAX_ATTACHED_FILES),
'file_size_limit_msg': CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE > 0 and _("Max %(x_nb_bytes)s per file") % {'x_nb_bytes': (CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE < 1024*1024 and (str(CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE/1024) + 'KB') or (str(CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE/(1024*1024)) + 'MB'))} or ''}
editor = get_html_text_editor(name='msg',
content=msg,
textual_content=textual_msg,
width='100%',
height='400px',
enabled=CFG_WEBCOMMENT_USE_RICH_TEXT_EDITOR,
file_upload_url=file_upload_url,
toolbar_set = "WebComment",
ln=ln)
subscribe_to_discussion = ''
if not user_is_subscribed_to_discussion:
# Offer to subscribe to discussion
subscribe_to_discussion = '<small><input type="checkbox" name="subscribe" id="subscribe"/><label for="subscribe">%s</label></small>' % _("Send me an email when a new comment is posted")
form = """<div id="comment-write"><h2>%(add_comment)s</h2>
%(editor)s
<br />
%(simple_attach_file_interface)s
<span class="reportabuse">%(note)s</span>
<div class="submit-area">
%(subscribe_to_discussion)s<br />
<input class="adminbutton" type="submit" value="Add comment" onclick="user_must_confirm_before_leaving_page = false;return true;"/>
%(reply_to)s
</div>
</div>
""" % {'note': note,
'record_label': _("Article") + ":",
'comment_label': _("Comment") + ":",
'add_comment': _('Add comment'),
'editor': editor,
'subscribe_to_discussion': subscribe_to_discussion,
'reply_to': reply_to and '<input type="hidden" name="comid" value="%s"/>' % reply_to or '',
'simple_attach_file_interface': simple_attach_file_interface}
form_link = "%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/comments/%(function)s?%(arguments)s" % link_dic
form = self.create_write_comment_hiddenform(action=form_link, method="post", text=form, button='Add comment',
enctype='multipart/form-data', form_id='cmtForm',
form_name='cmtForm')
return warnings + form + self.tmpl_page_do_not_leave_comment_page_js(ln=ln)
def tmpl_add_comment_form_with_ranking(self, recID, uid, nickname, ln, msg, score, note,
warnings, textual_msg=None, show_title_p=False,
can_attach_files=False):
"""
Add form for reviews
@param recID: record id
@param uid: user id
@param ln: language
@param msg: comment body contents for when refreshing due to warning
@param textual_msg: the textual version of 'msg' when user cannot display Ckeditor
@param score: review score
@param note: review title
@param warnings: list of warning tuples (warning_text, warning_color)
@param show_title_p: if True, prefix the form with "Add Review" as title
@param can_attach_files: if user can upload attach file to record or not
@return: html add review form
"""
_ = gettext_set_language(ln)
link_dic = { 'siteurl' : CFG_SITE_URL,
'CFG_SITE_RECORD' : CFG_SITE_RECORD,
'module' : 'comments',
'function' : 'add',
'arguments' : 'ln=%s&amp;action=%s' % (ln, 'SUBMIT'),
'recID' : recID}
warnings = self.tmpl_warnings(warnings, ln)
if textual_msg is None:
textual_msg = msg
#from search_engine import print_record
#record_details = print_record(recID=recID, format='hb', ln=ln)
if nickname:
note_label = _("Note: Your nickname, %(x_name)s, will be displayed as the author of this review.",
x_name=('<i>' + nickname + '</i>'))
else:
(uid, nickname, display) = get_user_info(uid)
link = '<a href="%s/youraccount/edit">' % CFG_SITE_SECURE_URL
note_label = _("Note: you have not %(x_url_open)sdefined your nickname%(x_url_close)s. %(x_nickname)s will be displayed as the author of this comment.") % \
{'x_url_open': link,
'x_url_close': '</a>',
'x_nickname': ' <br /><i>' + display + '</i>'}
selected0 = ''
selected1 = ''
selected2 = ''
selected3 = ''
selected4 = ''
selected5 = ''
if score == 0:
selected0 = ' selected="selected"'
elif score == 1:
selected1 = ' selected="selected"'
elif score == 2:
selected2 = ' selected="selected"'
elif score == 3:
selected3 = ' selected="selected"'
elif score == 4:
selected4 = ' selected="selected"'
elif score == 5:
selected5 = ' selected="selected"'
## file_upload_url = None
## if can_attach_files:
## file_upload_url = '%s/%s/%i/comments/attachments/put' % \
## (CFG_SITE_URL, CFG_SITE_RECORD, recID)
editor = get_html_text_editor(name='msg',
content=msg,
textual_content=msg,
width='90%',
height='400px',
enabled=CFG_WEBCOMMENT_USE_RICH_TEXT_EDITOR,
# file_upload_url=file_upload_url,
toolbar_set = "WebComment",
ln=ln)
form = """%(add_review)s
<table style="width: 100%%">
<tr>
<td style="padding-bottom: 10px;">%(rate_label)s:
<select name=\"score\" size=\"1\">
<option value=\"0\"%(selected0)s>-%(select_label)s-</option>
<option value=\"5\"%(selected5)s>***** (best)</option>
<option value=\"4\"%(selected4)s>****</option>
<option value=\"3\"%(selected3)s>***</option>
<option value=\"2\"%(selected2)s>**</option>
<option value=\"1\"%(selected1)s>* (worst)</option>
</select>
</td>
</tr>
<tr>
<td>%(title_label)s:</td>
</tr>
<tr>
<td style="padding-bottom: 10px;">
<input type="text" name="note" maxlength="250" style="width:90%%" value="%(note)s" />
</td>
</tr>
<tr>
<td>%(write_label)s:</td>
</tr>
<tr>
<td>
%(editor)s
</td>
</tr>
<tr>
<td class="reportabuse">%(note_label)s</td></tr>
</table>
""" % {'article_label': _('Article'),
'rate_label': _("Rate this article"),
'select_label': _("Select a score"),
'title_label': _("Give a title to your review"),
'write_label': _("Write your review"),
'note_label': note_label,
'note' : note!='' and cgi.escape(note, quote=True) or "",
'msg' : msg!='' and msg or "",
#'record' : record_details
'add_review': show_title_p and ('<h2>'+_('Add review')+'</h2>') or '',
'selected0': selected0,
'selected1': selected1,
'selected2': selected2,
'selected3': selected3,
'selected4': selected4,
'selected5': selected5,
'editor': editor,
}
form_link = "%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/reviews/%(function)s?%(arguments)s" % link_dic
form = self.createhiddenform(action=form_link, method="post", text=form, button=_('Add Review'))
return warnings + form
def tmpl_add_comment_successful(self, recID, ln, reviews, warnings, success):
"""
@param recID: record id
@param ln: language
@return: html page of successfully added comment/review
"""
_ = gettext_set_language(ln)
link_dic = { 'siteurl' : CFG_SITE_URL,
'CFG_SITE_RECORD' : CFG_SITE_RECORD,
'module' : 'comments',
'function' : 'display',
'arguments' : 'ln=%s&amp;do=od' % ln,
'recID' : recID,
'discussion': reviews == 1 and 'reviews' or 'comments'}
link = "%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/%(discussion)s/%(function)s?%(arguments)s" % link_dic
if warnings:
out = self.tmpl_warnings(warnings, ln) + '<br /><br />'
else:
if reviews:
out = _("Your review was successfully added.") + '<br /><br />'
else:
out = _("Your comment was successfully added.") + '<br /><br />'
link += "#C%s" % success
out += '<a href="%s">' % link
out += _('Back to record') + '</a>'
out += '<br/><br/>' \
+ _('You can also view all the comments you have submitted so far on "%(x_url_open)sYour Comments%(x_url_close)s" page.') % \
{'x_url_open': '<a target="_blank" href="%(CFG_SITE_URL)s/yourcomments?ln=%(ln)s">' % {'CFG_SITE_URL': CFG_SITE_URL, 'ln': ln},
'x_url_close': '</a>'}
return out
def tmpl_create_multiple_actions_form(self,
form_name="",
form_action="",
method="get",
action_display={},
action_field_name="",
button_label="",
button_name="",
content="",
**hidden):
""" Creates an HTML form with a multiple choice of actions and a button to select it.
@param form_action: link to the receiver of the formular
@param form_name: name of the HTML formular
@param method: either 'GET' or 'POST'
@param action_display: dictionary of actions.
action is HTML name (name of action)
display is the string provided in the popup
@param action_field_name: html name of action field
@param button_label: what's written on the button
@param button_name: html name of the button
@param content: what's inside te formular
@param **hidden: dictionary of name/value pairs of hidden fields.
"""
output = """
<form action="%s" method="%s">""" % (form_action, method)
output += """
<table>
<tr>
<td style="vertical-align: top" colspan="2">
"""
output += content + '\n'
for key in hidden.keys():
if type(hidden[key]) is list:
for value in hidden[key]:
output += """
<input type="hidden" name="%s" value="%s" />""" % (key, value)
else:
output += """
<input type="hidden" name="%s" value="%s" />""" % (key, hidden[key])
output += """
</td>
</tr>
<tr>
<td style="text-align:right;">"""
if type(action_display) is dict and len(action_display.keys()):
output += """
<select name="%s">""" % action_field_name
for (key, value) in action_display.items():
output += """
<option value="%s">%s</option>""" % (key, value)
output += """
</select>"""
output += """
</td>
<td style="text-align:left;">
<input class="adminbutton" type="submit" value="%s" name="%s"/>""" % (button_label, button_name)
output += """
</td>
</tr>
</table>
</form>"""
return output
def tmpl_admin_index(self, ln):
"""
Index page
"""
# load the right message language
_ = gettext_set_language(ln)
out = '<ol>'
if CFG_WEBCOMMENT_ALLOW_COMMENTS or CFG_WEBCOMMENT_ALLOW_REVIEWS:
if CFG_WEBCOMMENT_ALLOW_COMMENTS:
out += '<h3>Comments status</h3>'
out += '<li><a href="%(siteurl)s/admin/webcomment/webcommentadmin.py/hot?ln=%(ln)s&amp;comments=1">%(hot_cmt_label)s</a></li>' % \
{'siteurl': CFG_SITE_URL, 'ln': ln, 'hot_cmt_label': _("View most commented records")}
out += '<li><a href="%(siteurl)s/admin/webcomment/webcommentadmin.py/latest?ln=%(ln)s&amp;comments=1">%(latest_cmt_label)s</a></li>' % \
{'siteurl': CFG_SITE_URL, 'ln': ln, 'latest_cmt_label': _("View latest commented records")}
out += '<li><a href="%(siteurl)s/admin/webcomment/webcommentadmin.py/comments?ln=%(ln)s&amp;reviews=0">%(reported_cmt_label)s</a></li>' % \
{'siteurl': CFG_SITE_URL, 'ln': ln, 'reported_cmt_label': _("View all comments reported as abuse")}
if CFG_WEBCOMMENT_ALLOW_REVIEWS:
out += '<h3>Reviews status</h3>'
out += '<li><a href="%(siteurl)s/admin/webcomment/webcommentadmin.py/hot?ln=%(ln)s&amp;comments=0">%(hot_rev_label)s</a></li>' % \
{'siteurl': CFG_SITE_URL, 'ln': ln, 'hot_rev_label': _("View most reviewed records")}
out += '<li><a href="%(siteurl)s/admin/webcomment/webcommentadmin.py/latest?ln=%(ln)s&amp;comments=0">%(latest_rev_label)s</a></li>' % \
{'siteurl': CFG_SITE_URL, 'ln': ln, 'latest_rev_label': _("View latest reviewed records")}
out += '<li><a href="%(siteurl)s/admin/webcomment/webcommentadmin.py/comments?ln=%(ln)s&amp;reviews=1">%(reported_rev_label)s</a></li>' % \
{'siteurl': CFG_SITE_URL, 'ln': ln, 'reported_rev_label': _("View all reviews reported as abuse")}
#<li><a href="%(siteurl)s/admin/webcomment/webcommentadmin.py/delete?ln=%(ln)s&amp;comid=-1">%(delete_label)s</a></li>
out +="""
<h3>General</h3>
<li><a href="%(siteurl)s/admin/webcomment/webcommentadmin.py/users?ln=%(ln)s">%(view_users)s</a></li>
<li><a href="%(siteurl)s/help/admin/webcomment-admin-guide">%(guide)s</a></li>
""" % {'siteurl' : CFG_SITE_URL,
#'delete_label': _("Delete/Undelete comment(s) or suppress abuse report(s)"),
'view_users': _("View all users who have been reported"),
'ln' : ln,
'guide' : _("Guide")}
else:
out += _("Comments and reviews are disabled") + '<br />'
out += '</ol>'
from invenio.legacy.bibrank.adminlib import addadminbox
return addadminbox('<b>%s</b>'% _("Menu"), [out])
def tmpl_admin_delete_form(self, ln, warnings):
"""
Display admin interface to fetch list of records to delete
@param warnings: list of warning tuples (warning_text, warning_color)
see tmpl_warnings, warning_color is optional
"""
# load the right message language
_ = gettext_set_language(ln)
warnings = self.tmpl_warnings(warnings, ln)
out = '''
<br />
%s<br />
<br />'''% _("Please enter the ID of the comment/review so that you can view it before deciding whether to delete it or not")
form = '''
<table>
<tr>
<td>%s</td>
<td><input type=text name="comid" size="10" maxlength="10" value="" /></td>
</tr>
<tr>
<td><br /></td>
<tr>
</table>
<br />
%s <br/>
<br />
<table>
<tr>
<td>%s</td>
<td><input type=text name="recid" size="10" maxlength="10" value="" /></td>
</tr>
<tr>
<td><br /></td>
<tr>
</table>
<br />
''' % (_("Comment ID:"),
_("Or enter a record ID to list all the associated comments/reviews:"),
_("Record ID:"))
form_link = "%s/admin/webcomment/webcommentadmin.py/delete?ln=%s" % (CFG_SITE_URL, ln)
form = self.createhiddenform(action=form_link, method="get", text=form, button=_('View Comment'))
return warnings + out + form
def tmpl_admin_users(self, ln, users_data):
"""
@param users_data: tuple of ct, i.e. (ct, ct, ...)
where ct is a tuple (total_number_reported, total_comments_reported, total_reviews_reported, total_nb_votes_yes_of_reported,
total_nb_votes_total_of_reported, user_id, user_email, user_nickname)
sorted by order of ct having highest total_number_reported
"""
_ = gettext_set_language(ln)
u_reports = 0
u_comment_reports = 1
u_reviews_reports = 2
u_nb_votes_yes = 3
u_nb_votes_total = 4
u_uid = 5
u_email = 6
u_nickname = 7
if not users_data:
return self.tmpl_warnings([(_("There have been no reports so far."), 'green')])
user_rows = ""
for utuple in users_data:
com_label = _("View all %(x_name)s reported comments", x_name=utuple[u_comment_reports])
com_link = '''<a href="%s/admin/webcomment/webcommentadmin.py/comments?ln=%s&amp;uid=%s&amp;reviews=0">%s</a><br />''' % \
(CFG_SITE_URL, ln, utuple[u_uid], com_label)
rev_label = _("View all %(x_name)s reported reviews", x_name=utuple[u_reviews_reports])
rev_link = '''<a href="%s/admin/webcomment/webcommentadmin.py/comments?ln=%s&amp;uid=%s&amp;reviews=1">%s</a>''' % \
(CFG_SITE_URL, ln, utuple[u_uid], rev_label)
if not utuple[u_nickname]:
user_info = get_user_info(utuple[u_uid])
nickname = user_info[2]
else:
nickname = utuple[u_nickname]
if CFG_WEBCOMMENT_ALLOW_REVIEWS:
review_row = """
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>"""
review_row %= (utuple[u_nb_votes_yes],
utuple[u_nb_votes_total] - utuple[u_nb_votes_yes],
utuple[u_nb_votes_total])
else:
review_row = ''
user_rows += """
<tr>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%(nickname)s</td>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%(email)s</td>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%(uid)s</td>%(review_row)s
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray; font-weight: bold;">%(reports)s</td>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%(com_link)s%(rev_link)s</td>
</tr>""" % { 'nickname' : nickname,
'email' : utuple[u_email],
'uid' : utuple[u_uid],
'reports' : utuple[u_reports],
'review_row': review_row,
'siteurl' : CFG_SITE_URL,
'ln' : ln,
'com_link' : CFG_WEBCOMMENT_ALLOW_COMMENTS and com_link or "",
'rev_link' : CFG_WEBCOMMENT_ALLOW_REVIEWS and rev_link or ""
}
out = "<br />"
out += _("Here is a list, sorted by total number of reports, of all users who have had a comment reported at least once.")
out += """
<br />
<br />
<table class="admin_wvar" style="width: 100%%;">
<thead>
<tr class="adminheaderleft">
<th>"""
out += _("Nickname") + '</th>\n'
out += '<th>' + _("Email") + '</th>\n'
out += '<th>' + _("User ID") + '</th>\n'
if CFG_WEBCOMMENT_ALLOW_REVIEWS > 0:
out += '<th>' + _("Number positive votes") + '</th>\n'
out += '<th>' + _("Number negative votes") + '</th>\n'
out += '<th>' + _("Total number votes") + '</th>\n'
out += '<th>' + _("Total number of reports") + '</th>\n'
out += '<th>' + _("View all user's reported comments/reviews") + '</th>\n'
out += """
</tr>
</thead>
<tbody>%s
</tbody>
</table>
""" % user_rows
return out
def tmpl_admin_select_comment_checkbox(self, cmt_id):
""" outputs a checkbox named "comidXX" where XX is cmt_id """
return '<input type="checkbox" name="comid%i" />' % int(cmt_id)
def tmpl_admin_user_info(self, ln, nickname, uid, email):
""" prepares informations about a user"""
_ = gettext_set_language(ln)
out = """
%(nickname_label)s: %(messaging)s<br />
%(uid_label)s: %(uid)i<br />
%(email_label)s: <a href="mailto:%(email)s">%(email)s</a>"""
out %= {'nickname_label': _("Nickname"),
'messaging': self.create_messaging_link(uid, nickname, ln),
'uid_label': _("User ID"),
'uid': int(uid),
'email_label': _("Email"),
'email': email}
return out
def tmpl_admin_review_info(self, ln, reviews, nb_reports, cmt_id, rec_id, status):
""" outputs information about a review """
_ = gettext_set_language(ln)
if reviews:
reported_label = _("This review has been reported %(x_num)i times", x_num=int(nb_reports))
else:
reported_label = _("This comment has been reported %(x_num)i times", x_num=int(nb_reports))
# reported_label %= int(nb_reports)
out = """
%(reported_label)s<br />
<a href="%(siteurl)s/%(CFG_SITE_RECORD)s/%(rec_id)i?ln=%(ln)s">%(rec_id_label)s</a><br />
%(cmt_id_label)s"""
out %= {'reported_label': reported_label,
'rec_id_label': _("Record") + ' #' + str(rec_id),
'siteurl': CFG_SITE_URL,
'CFG_SITE_RECORD' : CFG_SITE_RECORD,
'rec_id': int(rec_id),
'cmt_id_label': _("Comment") + ' #' + str(cmt_id),
'ln': ln}
if status in ['dm', 'da']:
out += '<br /><div style="color:red;">Marked as deleted</div>'
return out
def tmpl_admin_latest(self, ln, comment_data, comments, error, user_collections, collection):
"""
@param comment_data: same type of tuple as that
which is return by webcommentadminlib.py/query_get_latest i.e.
tuple (nickname, uid, date_creation, body, id) if latest comments or
tuple (nickname, uid, date_creation, body, star_score, id) if latest reviews
"""
_ = gettext_set_language(ln)
out = """
<script type='text/javascript'>
function collectionChange()
{
document.collection_form.submit();
}
</script>
"""
out += '<form method="get" name="collection_form" action="%s/admin/webcomment/webcommentadmin.py/latest?ln=%s&comments=%s">' % (CFG_SITE_URL, ln, comments)
out += '<input type="hidden" name="ln" value=%s>' % ln
out += '<input type="hidden" name="comments" value=%s>' % comments
out += '<div> Filter by collection: <select name="collection" onchange="javascript:collectionChange();">'
for collection_name in user_collections:
if collection_name == collection:
out += '<option "SELECTED" value="%(collection_name)s">%(collection_name)s</option>' % {'collection_name': cgi.escape(collection_name)}
else:
out += '<option value="%(collection_name)s">%(collection_name)s</option>' % {'collection_name': cgi.escape(collection_name)}
out += '</select></div></form><br />'
if error == 1:
out += "<i>User is not authorized to view such collection.</i><br />"
return out
elif error == 2:
out += "<i>There are no %s for this collection.</i><br />" % (comments and 'comments' or 'reviews')
return out
out += """
<ol>
"""
for (cmt_tuple, meta_data) in comment_data:
bibrec_id = meta_data[3]
content = format_record(bibrec_id, "hs")
if not comments:
out += """
<li> %(content)s <br/> <span class="moreinfo"> <a class="moreinfo" href=%(comment_url)s> reviewed by %(user)s</a>
(%(stars)s) \"%(body)s\" on <i> %(date)s </i></li> </span> <br/>
""" % {'content': content,
'comment_url': CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + str(bibrec_id) + '/reviews',
'user':cmt_tuple[0] ,
'stars': '*' * int(cmt_tuple[4]) ,
'body': cmt_tuple[3][:20] + '...',
'date': cmt_tuple[2]}
else:
out += """
<li> %(content)s <br/> <span class="moreinfo"> <a class="moreinfo" href=%(comment_url)s> commented by %(user)s</a>,
\"%(body)s\" on <i> %(date)s </i></li> </span> <br/>
""" % {'content': content,
'comment_url': CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + str(bibrec_id) + '/comments',
'user':cmt_tuple[0] ,
'body': cmt_tuple[3][:20] + '...',
'date': cmt_tuple[2]}
out += """</ol>"""
return out
def tmpl_admin_hot(self, ln, comment_data, comments, error, user_collections, collection):
"""
@param comment_data: same type of tuple as that
which is return by webcommentadminlib.py/query_get_hot i.e.
tuple (id_bibrec, date_last_comment, users, count)
"""
_ = gettext_set_language(ln)
out = """
<script type='text/javascript'>
function collectionChange()
{
document.collection_form.submit();
}
</script>
"""
out += '<form method="get" name="collection_form" action="%s/admin/webcomment/webcommentadmin.py/hot?ln=%s&comments=%s">' % (CFG_SITE_URL, ln, comments)
out += '<input type="hidden" name="ln" value=%s>' % ln
out += '<input type="hidden" name="comments" value=%s>' % comments
out += '<div> Filter by collection: <select name="collection" onchange="javascript:collectionChange();">'
for collection_name in user_collections:
if collection_name == collection:
out += '<option "SELECTED" value="%(collection_name)s">%(collection_name)s</option>' % {'collection_name': cgi.escape(collection_name)}
else:
out += '<option value="%(collection_name)s">%(collection_name)s</option>' % {'collection_name': cgi.escape(collection_name)}
out += '</select></div></form><br />'
if error == 1:
out += "<i>User is not authorized to view such collection.</i><br />"
return out
elif error == 2:
out += "<i>There are no %s for this collection.</i><br />" % (comments and 'comments' or 'reviews')
return out
for cmt_tuple in comment_data:
bibrec_id = cmt_tuple[0]
content = format_record(bibrec_id, "hs")
last_comment_date = cmt_tuple[1]
total_users = cmt_tuple[2]
total_comments = cmt_tuple[3]
if comments:
comment_url = CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + str(bibrec_id) + '/comments'
str_comment = int(total_comments) > 1 and 'comments' or 'comment'
else:
comment_url = CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + str(bibrec_id) + '/reviews'
str_comment = int(total_comments) > 1 and 'reviews' or 'review'
out += """
<li> %(content)s <br/> <span class="moreinfo"> <a class="moreinfo" href=%(comment_url)s> %(total_comments)s
%(str_comment)s</a>
(%(total_users)s %(user)s), latest on <i> %(last_comment_date)s </i></li> </span> <br/>
""" % {'content': content,
'comment_url': comment_url ,
'total_comments': total_comments,
'str_comment': str_comment,
'total_users': total_users,
'user': int(total_users) > 1 and 'users' or 'user',
'last_comment_date': last_comment_date}
out += """</ol>"""
return out
def tmpl_admin_comments(self, ln, uid, comID, recID, comment_data, reviews, error, user_collections, collection):
"""
@param comment_data: same type of tuple as that
which is returned by webcomment.py/query_retrieve_comments_or_remarks i.e.
tuple of comment where comment is
tuple (nickname,
date_creation,
body,
id) if ranking disabled or
tuple (nickname,
date_creation,
body,
nb_votes_yes,
nb_votes_total,
star_score,
title,
id)
"""
_ = gettext_set_language(ln)
coll_form = """
<script type='text/javascript'>
function collectionChange()
{
document.collection_form.submit();
}
</script>
"""
coll_form += '<form method="get" name="collection_form" action="%s/admin/webcomment/webcommentadmin.py/comments?ln=%s&reviews=%s">' % (CFG_SITE_URL, ln, reviews)
coll_form += '<input type="hidden" name="ln" value=%s>' % ln
coll_form += '<input type="hidden" name="reviews" value=%s>' % reviews
coll_form += '<div> Filter by collection: <select name="collection" onchange="javascript:collectionChange();">'
for collection_name in user_collections:
if collection_name == collection:
coll_form += '<option "SELECTED" value="%(collection_name)s">%(collection_name)s</option>' % {'collection_name': cgi.escape(collection_name)}
else:
coll_form += '<option value="%(collection_name)s">%(collection_name)s</option>' % {'collection_name': cgi.escape(collection_name)}
coll_form += '</select></div></form><br />'
if error == 1:
coll_form += "<i>User is not authorized to view such collection.</i><br />"
return coll_form
elif error == 2:
coll_form += "<i>There are no %s for this collection.</i><br />" % (reviews and 'reviews' or 'comments')
return coll_form
comments = []
comments_info = []
checkboxes = []
users = []
for (cmt_tuple, meta_data) in comment_data:
if reviews:
comments.append(self.tmpl_get_comment_with_ranking(None,#request object
ln,
cmt_tuple[0],#nickname
cmt_tuple[1],#userid
cmt_tuple[2],#date_creation
cmt_tuple[3],#body
cmt_tuple[9],#status
0,
cmt_tuple[5],#nb_votes_total
cmt_tuple[4],#nb_votes_yes
cmt_tuple[6],#star_score
cmt_tuple[7]))#title
else:
comments.append(self.tmpl_get_comment_without_ranking(None,#request object
ln,
cmt_tuple[0],#nickname
cmt_tuple[1],#userid
cmt_tuple[2],#date_creation
cmt_tuple[3],#body
cmt_tuple[5],#status
0,
None, #reply_link
None, #report_link
None, #undelete_link
None, #delete_links
None, #unreport_link
-1, # recid
cmt_tuple[4] # com_id
))
users.append(self.tmpl_admin_user_info(ln,
meta_data[0], #nickname
meta_data[1], #uid
meta_data[2]))#email
if reviews:
status = cmt_tuple[9]
else:
status = cmt_tuple[5]
comments_info.append(self.tmpl_admin_review_info(ln,
reviews,
meta_data[5], # nb abuse reports
meta_data[3], # cmt_id
meta_data[4], # rec_id
status)) # status
checkboxes.append(self.tmpl_admin_select_comment_checkbox(meta_data[3]))
form_link = "%s/admin/webcomment/webcommentadmin.py/del_com?ln=%s" % (CFG_SITE_URL, ln)
out = """
<table class="admin_wvar" style="width:100%%;">
<thead>
<tr class="adminheaderleft">
<th>%(review_label)s</th>
<th>%(written_by_label)s</th>
<th>%(review_info_label)s</th>
<th>%(select_label)s</th>
</tr>
</thead>
<tbody>""" % {'review_label': reviews and _("Review") or _("Comment"),
'written_by_label': _("Written by"),
'review_info_label': _("General informations"),
'select_label': _("Select")}
for i in range (0, len(comments)):
out += """
<tr>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
<td class="admintd" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
</tr>""" % (comments[i], users[i], comments_info[i], checkboxes[i])
out += """
</tbody>
</table>"""
if reviews:
action_display = {
'delete': _('Delete selected reviews'),
'unreport': _('Suppress selected abuse report'),
'undelete': _('Undelete selected reviews')
}
else:
action_display = {
'undelete': _('Undelete selected comments'),
'delete': _('Delete selected comments'),
'unreport': _('Suppress selected abuse report')
}
form = self.tmpl_create_multiple_actions_form(form_name="admin_comment",
form_action=form_link,
method="post",
action_display=action_display,
action_field_name='action',
button_label=_("OK"),
button_name="okbutton",
content=out)
if uid > 0:
header = '<br />'
if reviews:
header += _("Here are the reported reviews of user %(x_name)s", x_name=uid)
else:
header += _("Here are the reported comments of user %(x_name)s", x_name=uid)
header += '<br /><br />'
if comID > 0 and recID <= 0 and uid <= 0:
if reviews:
header = '<br />' +_("Here is review %(x_name)s", x_name=comID) + '<br /><br />'
else:
header = '<br />' +_("Here is comment %(x_name)s", x_name=comID) + '<br /><br />'
if uid > 0 and comID > 0 and recID <= 0:
if reviews:
header = '<br />' + _("Here is review %(x_cmtID)s written by user %(x_user)s") % {'x_cmtID': comID, 'x_user': uid}
else:
header = '<br />' + _("Here is comment %(x_cmtID)s written by user %(x_user)s") % {'x_cmtID': comID, 'x_user': uid}
header += '<br/ ><br />'
if comID <= 0 and recID <= 0 and uid <= 0:
header = '<br />'
if reviews:
header += _("Here are all reported reviews sorted by the most reported")
else:
header += _("Here are all reported comments sorted by the most reported")
header += "<br /><br />"
elif recID > 0:
header = '<br />'
if reviews:
header += _("Here are all reviews for record %(x_num)i, sorted by the most reported", x_num=recID)
header += '<br /><a href="%s/admin/webcomment/webcommentadmin.py/delete?comid=&recid=%s&amp;reviews=0">%s</a>' % (CFG_SITE_URL, recID, _("Show comments"))
else:
header += _("Here are all comments for record %(x_num)i, sorted by the most reported", x_num=recID)
header += '<br /><a href="%s/admin/webcomment/webcommentadmin.py/delete?comid=&recid=%s&amp;reviews=1">%s</a>' % (CFG_SITE_URL, recID, _("Show reviews"))
header += "<br /><br />"
return coll_form + header + form
def tmpl_admin_del_com(self, del_res, ln=CFG_SITE_LANG):
"""
@param del_res: list of the following tuple (comment_id, was_successfully_deleted),
was_successfully_deleted is boolean (0=false, >0=true
"""
_ = gettext_set_language(ln)
table_rows = ''
for deltuple in del_res:
table_rows += """
<tr>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
</tr>""" % (deltuple[0], deltuple[1]>0 and _("Yes") or "<span class=\"important\">" +_("No") + "</span>")
out = """
<table class="admin_wvar">
<tr class="adminheaderleft">
<td style="padding-right:10px;">%s</td>
<td>%s</td>
</tr>%s
<table>""" % (_("comment ID"), _("successfully deleted"), table_rows)
return out
def tmpl_admin_undel_com(self, del_res, ln=CFG_SITE_LANG):
"""
@param del_res: list of the following tuple (comment_id, was_successfully_undeleted),
was_successfully_undeleted is boolean (0=false, >0=true
"""
_ = gettext_set_language(ln)
table_rows = ''
for deltuple in del_res:
table_rows += """
<tr>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
</tr>""" % (deltuple[0], deltuple[1]>0 and _("Yes") or "<span class=\"important\">" +_("No") + "</span>")
out = """
<table class="admin_wvar">
<tr class="adminheaderleft">
<td style="padding-right:10px;">%s</td>
<td>%s</td>
</tr>%s
<table>""" % (_("comment ID"), _("successfully undeleted"), table_rows)
return out
def tmpl_admin_suppress_abuse_report(self, del_res, ln=CFG_SITE_LANG):
"""
@param del_res: list of the following tuple (comment_id, was_successfully_deleted),
was_successfully_deleted is boolean (0=false, >0=true
"""
_ = gettext_set_language(ln)
table_rows = ''
for deltuple in del_res:
table_rows += """
<tr>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
<td class="admintdleft" style="padding: 5px; border-bottom: 1px solid lightgray;">%s</td>
</tr>""" % (deltuple[0], deltuple[1]>0 and _("Yes") or "<span class=\"important\">" +_("No") + "</span>")
out = """
<table class="admin_wvar">
<tr class="adminheaderleft">
<td style ="padding-right: 10px;">%s</td>
<td>%s</td>
</tr>%s
<table>""" % (_("comment ID"), _("successfully suppressed abuse report"), table_rows)
return out
def tmpl_mini_review(self, recID, ln=CFG_SITE_LANG, action='SUBMIT',
avg_score=0, nb_comments_total=0):
"""Display the mini version of reviews (only the grading part)"""
_ = gettext_set_language(ln)
url = '%s/%s/%s/reviews/add?ln=%s&amp;action=%s' % (CFG_BASE_URL, CFG_SITE_RECORD, recID, ln, action)
if avg_score > 0:
score = _("Average review score: %(x_nb_score)s based on %(x_nb_reviews)s reviews") % \
{'x_nb_score': '<b>%.1f</b>' % avg_score,
'x_nb_reviews': nb_comments_total}
else:
score = '(' +_("Not yet reviewed") + ')'
if avg_score == 5:
s1, s2, s3, s4, s5 = 'full', 'full', 'full', 'full', 'full'
elif avg_score >= 4.5:
s1, s2, s3, s4, s5 = 'full', 'full', 'full', 'full', 'half'
elif avg_score >= 4:
s1, s2, s3, s4, s5 = 'full', 'full', 'full', 'full', ''
elif avg_score >= 3.5:
s1, s2, s3, s4, s5 = 'full', 'full', 'full', 'half', ''
elif avg_score >= 3:
s1, s2, s3, s4, s5 = 'full', 'full', 'full', '', ''
elif avg_score >= 2.5:
s1, s2, s3, s4, s5 = 'full', 'full', 'half', '', ''
elif avg_score >= 2:
s1, s2, s3, s4, s5 = 'full', 'full', '', '', ''
elif avg_score >= 1.5:
s1, s2, s3, s4, s5 = 'full', 'half', '', '', ''
elif avg_score == 1:
s1, s2, s3, s4, s5 = 'full', '', '', '', ''
else:
s1, s2, s3, s4, s5 = '', '', '', '', ''
out = '''
<small class="detailedRecordActions">%(rate)s:</small><br /><br />
<div style="margin:auto;width:160px;">
<span style="display:none;">Rate this document:</span>
<div class="star %(s1)s" ><a href="%(url)s&amp;score=1">1</a>
<div class="star %(s2)s" ><a href="%(url)s&amp;score=2">2</a>
<div class="star %(s3)s" ><a href="%(url)s&amp;score=3">3</a>
<div class="star %(s4)s" ><a href="%(url)s&amp;score=4">4</a>
<div class="star %(s5)s" ><a href="%(url)s&amp;score=5">5</a></div></div></div></div></div>
<div style="clear:both">&nbsp;</div>
</div>
<small>%(score)s</small>
''' % {'url': url,
'score': score,
'rate': _("Rate this document"),
's1': s1,
's2': s2,
's3': s3,
's4': s4,
's5': s5
}
return out
def tmpl_email_new_comment_header(self, recID, title, reviews,
comID, report_numbers,
can_unsubscribe=True,
ln=CFG_SITE_LANG, uid=-1):
"""
Prints the email header used to notify subscribers that a new
comment/review was added.
@param recid: the ID of the commented/reviewed record
@param title: the title of the commented/reviewed record
@param reviews: True if it is a review, else if a comment
@param comID: the comment ID
@param report_numbers: the report number(s) of the record
@param can_unsubscribe: True if user can unsubscribe from alert
@param ln: language
"""
# load the right message language
_ = gettext_set_language(ln)
user_info = collect_user_info(uid)
out = _("Hello:") + '\n\n' + \
(reviews and _("The following review was sent to %(CFG_SITE_NAME)s by %(user_nickname)s:") or \
_("The following comment was sent to %(CFG_SITE_NAME)s by %(user_nickname)s:")) % \
{'CFG_SITE_NAME': CFG_SITE_NAME,
'user_nickname': user_info['nickname']}
out += '\n(<%s>)' % (CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + str(recID))
out += '\n\n\n'
return out
def tmpl_email_new_comment_footer(self, recID, title, reviews,
comID, report_numbers,
can_unsubscribe=True,
ln=CFG_SITE_LANG):
"""
Prints the email footer used to notify subscribers that a new
comment/review was added.
@param recid: the ID of the commented/reviewed record
@param title: the title of the commented/reviewed record
@param reviews: True if it is a review, else if a comment
@param comID: the comment ID
@param report_numbers: the report number(s) of the record
@param can_unsubscribe: True if user can unsubscribe from alert
@param ln: language
"""
# load the right message language
_ = gettext_set_language(ln)
out = '\n\n-- \n'
out += _("This is an automatic message, please don't reply to it.")
out += '\n'
out += _("To post another comment, go to <%(x_url)s> instead.") % \
{'x_url': CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + str(recID) + \
(reviews and '/reviews' or '/comments') + '/add'}
out += '\n'
if not reviews:
out += _("To specifically reply to this comment, go to <%(x_url)s>") % \
{'x_url': CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + str(recID) + \
'/comments/add?action=REPLY&comid=' + str(comID)}
out += '\n'
if can_unsubscribe:
out += _("To unsubscribe from this discussion, go to <%(x_url)s>") % \
{'x_url': CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + str(recID) + \
'/comments/unsubscribe'}
out += '\n'
out += _("For any question, please use <%(CFG_SITE_SUPPORT_EMAIL)s>") % \
{'CFG_SITE_SUPPORT_EMAIL': CFG_SITE_SUPPORT_EMAIL}
return out
def tmpl_email_new_comment_admin(self, recID):
"""
Prints the record information used in the email to notify the
system administrator that a new comment has been posted.
@param recID: the ID of the commented/reviewed record
"""
out = ""
title = get_fieldvalues(recID, "245__a")
authors = ', '.join(get_fieldvalues(recID, "100__a") + get_fieldvalues(recID, "700__a"))
#res_author = ""
#res_rep_num = ""
#for author in authors:
# res_author = res_author + ' ' + author
dates = get_fieldvalues(recID, "260__c")
report_nums = get_fieldvalues(recID, "037__a")
report_nums += get_fieldvalues(recID, "088__a")
report_nums = ', '.join(report_nums)
#for rep_num in report_nums:
# res_rep_num = res_rep_num + ', ' + rep_num
out += " Title = %s \n" % (title and title[0] or "No Title")
out += " Authors = %s \n" % authors
if dates:
out += " Date = %s \n" % dates[0]
out += " Report number = %s" % report_nums
return out
def tmpl_page_do_not_leave_comment_page_js(self, ln):
"""
Code to ask user confirmation when leaving the page, so that the
comment is not lost if clicking by mistake on links.
@param ln: the user language
"""
# load the right message language
_ = gettext_set_language(ln)
out = '''
<script type="text/javascript" language="JavaScript">//<![CDATA[
var initial_comment_value = document.forms.cmtForm.msg.value;
var user_must_confirm_before_leaving_page = true;
window.onbeforeunload = confirmExit;
function confirmExit() {
var editor_type_field = document.getElementById('%(name)seditortype');
if (editor_type_field && editor_type_field.value == 'ckeditor') {
var oEditor = CKEDITOR.instances.%(name)s;
if (user_must_confirm_before_leaving_page && oEditor.checkDirty()) {
/* Might give false positives, when editor pre-loaded
with content. But is better than the opposite */
return "%(message)s";
}
} else {
if (user_must_confirm_before_leaving_page && document.forms.cmtForm.msg.value != initial_comment_value){
return "%(message)s";
}
}
}
//]]></script>
''' % {'message': _('Your comment will be lost.').replace('"', '\\"'),
'name': 'msg'}
return out
def tmpl_your_comments(self, user_info, comments, page_number=1, selected_order_by_option="lcf", selected_display_number_option="all", selected_display_format_option="rc", nb_total_results=0, nb_total_pages=0, ln=CFG_SITE_LANG):
"""
Display all submitted comments by the user
@param user_info: standard user info object.
@param comments: ordered list of tuples (id_bibrec, comid, date_creation, body, status, in_reply_to_id_cmtRECORDCOMMENT)
@param page_number: page on which the user is.
@type page_number: integer
@param selected_order_by_option: seleccted ordering option. Can be one of:
- ocf: Oldest comment first
- lcf: Latest comment first
- grof: Group by record, oldest commented first
- grlf: Group by record, latest commented first
@type selected_order_by_option: string
@param selected_display_number_option: number of results to show per page. Can be a string-digit or 'all'.
@type selected_display_number_option: string
@param selected_display_format_option: how to show records. Can be one of:
- rc: Records and comments
- ro: Records only
- co: Comments only
@type selected_display_format_option: string
@param nb_total_results: total number of items to display.
@type nb_total_results: integer
@param nb_total_pages: total number of pages.
@type nb_total_pages: integer
@ln: language
@type ln: string
"""
# load the right message language
_ = gettext_set_language(ln)
+ from invenio.search_engine import record_exists
+
your_comments_order_by_options = (('ocf', _("Oldest comment first")),
('lcf', _("Latest comment first")),
('grof', _("Group by record, oldest commented first")),
('grlf', _("Group by record, latest commented first")),
)
your_comments_display_format_options = (('rc', _("Records and comments")),
('ro', _('Records only')),
('co', _('Comments only')),
)
your_comments_display_number_options = (('20', _("%(x_i)s items", x_i = 20)),
('50', _("%(x_i)s items", x_i = 50)),
('100', _("%(x_i)s items", x_i = 100)),
('500',_("%(x_i)s items", x_i = 500)),
('all', _('All items')),
)
out = ""
out += _("Below is the list of the comments you have submitted so far.") + "<br/>"
if CFG_CERN_SITE:
if nb_total_results == 0:
out = _('You have not yet submitted any comment in the document "discussion" tab.') + "<br/>"
user_roles = acc_get_user_roles_from_user_info(user_info)
if acc_get_role_id('ATLASDraftPublication') in user_roles:
out += _('You might find other comments here: ')
out += create_html_link(urlbase=CFG_SITE_URL + '/search',
urlargd={'ln': ln,
'cc': 'ATLAS Publication Drafts Comments',
'p': user_info['email'],
'f': '859__f'},
link_label='ATLAS Publication Drafts Comments')
elif acc_get_role_id('cmsphysicsmembers') in user_roles:
out += _('You might find other comments here: ')
out += create_html_link(urlbase=CFG_SITE_URL + '/search',
urlargd={'ln': ln,
'cc': '',
'p': user_info['email'],
'f': '859__f'},
link_label='CMS Publication Drafts Comments')
elif acc_get_role_id('LHCbDraftPublication') in user_roles:
out += _('You might find other comments here: ')
out += create_html_link(urlbase=CFG_SITE_URL + '/search',
urlargd={'ln': ln,
'cc': '',
'p': user_info['email'],
'f': '859__f'},
link_label='LHCb Publication Drafts Comments')
out += '<br/>'
if nb_total_results == 0:
return out
else:
if nb_total_results == 0:
return _("You have not yet submitted any comment. Browse documents from the search interface and take part to discussions!")
# Show controls
format_selection = create_html_select(your_comments_display_format_options,
name="format", selected=selected_display_format_option,
attrs={'id': 'format',
'onchange': 'this.form.submit();'})
order_by_selection = create_html_select(your_comments_order_by_options,
name="order_by", selected=selected_order_by_option,
attrs={'id': 'order_by',
'onchange': 'this.form.submit();'})
nb_per_page_selection = create_html_select(your_comments_display_number_options,
name="per_page", selected=selected_display_number_option,
attrs={'id': 'per_page',
'onchange': 'this.form.submit();'})
out += '''
<form method="get" class="yourcommentsdisplayoptionsform">
<fieldset id="yourcommentsdisplayoptions">
<legend>%(display_option_label)s:</legend>
<label for="format">%(format_selection_label)s :</label> %(format_selection)s
<label for="order_by">%(order_selection_label)s :</label> %(order_by_selection)s
<label for="per_page">%(per_page_selection_label)s :</label> %(nb_per_page_selection)s
<noscript><input type="submit" value="%(refresh_label)s" class="formbutton"/></noscript>
</fieldset>
</form>
''' % {'format_selection_label': _("Display"),
'order_selection_label': _("Order by"),
'per_page_selection_label': _("Per page"),
'format_selection': format_selection,
'order_by_selection': order_by_selection,
'nb_per_page_selection': nb_per_page_selection,
'display_option_label': _("Display options"),
'refresh_label': _("Refresh"),
}
# Show comments
last_id_bibrec = None
nb_record_groups = 0
out += '<div id="yourcommentsmaincontent">'
for id_bibrec, comid, date_creation, body, status, in_reply_to_id_cmtRECORDCOMMENT in comments:
if last_id_bibrec != id_bibrec and selected_display_format_option in ('rc', 'ro'):
# We moved to another record. Show some info about
# current record.
if last_id_bibrec:
# Close previous group
out += "</div></div>"
nb_record_groups += 1
- #You might want to hide this information if user does
- #not have access, though it would make sense that he
- #can at least know on which page is comment appears..
+ # You might want to hide this information if user does
+ # not have access, though it would make sense that he
+ # can at least know on which page his comment appears..
+ if record_exists(id_bibrec) == -1:
+ record_info_html = '<em>%s</em>' % _("The record has been deleted.")
+ else:
+ record_info_html = format_record(id_bibrec, of="HS")
out += '''<div class="yourcommentsrecordgroup" id="yourcomments-record-group-%(recid)s">
<div class="yourcommentsrecordgroup%(recid)sheader">&#149; ''' % {'recid': id_bibrec} + \
- format_record(id_bibrec, of="HS") + '</div><div style="padding-left: 20px;">'
+ record_info_html + '</div><div style="padding-left: 20px;">'
if selected_display_format_option != 'ro':
-
final_body = email_quoted_txt2html(body)
title = '<a name="C%s" id="C%s"></a>' % (comid, comid)
if status == "dm":
final_body = '<div class="webcomment_deleted_comment_message">%s</div>' % _("Comment deleted by the moderator")
elif status == "da":
final_body = ('<div class="webcomment_deleted_comment_message">%s<br /><br />' % _("You have deleted this comment: it is not visible by other users")) +\
final_body + '</div>'
links = []
if in_reply_to_id_cmtRECORDCOMMENT:
links.append(create_html_link(urlbase=CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + \
str(id_bibrec) + '/comments/',
urlargd={'ln': ln},
link_label=_('(in reply to a comment)'),
urlhash=str(in_reply_to_id_cmtRECORDCOMMENT)))
links.append(create_html_link(urlbase=CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + \
str(id_bibrec) + '/comments/',
urlargd={'ln': ln},
link_label=_('See comment on discussion page'),
urlhash='C' + str(comid)))
out += '''
<div class="webcomment_comment_box">
<div class="webcomment_comment_avatar"><img class="webcomment_comment_avatar_default" src="%(site_url)s/img/user-icon-1-24x24.gif" alt="avatar" /></div>
<div class="webcomment_comment_content">
<div class="webcomment_comment_title">
%(title)s
<div class="webcomment_comment_date">%(date)s</div>
<a class="webcomment_permalink" title="Permalink to this comment" href="#C%(comid)i">¶</a>
</div>
<div class="collapsible_content">
<blockquote>
%(body)s
</blockquote>
<div class="webcomment_comment_options">%(links)s</div>
</div>
<div class="clearer"></div>
</div>
<div class="clearer"></div>
</div>''' % \
{'title' : title,
'body' : final_body,
'links' : " ".join(links),
'date' : date_creation,
'site_url' : CFG_SITE_URL,
'comid' : comid,
}
last_id_bibrec = id_bibrec
out += '</div>' # end 'yourcommentsmaincontent'
# Show page navigation
page_links = ''
if selected_display_format_option == 'ro' and \
selected_order_by_option in ('ocf', 'lcf'):
# We just have an approximation here (we count by
# comments, not record...)
page_links += (_("%(x_num)i comments found in total (not shown on this page)", x_num=nb_total_results)) + '&nbsp;'
else:
page_links += (_("%(x_num)i items found in total", x_num=nb_total_results)) + '&nbsp;'
if selected_display_number_option != 'all':
# Previous
if page_number != 1:
page_links += create_html_link(CFG_SITE_URL + '/yourcomments/',
{'page': page_number - 1,
'order_by': selected_order_by_option,
'per_page': selected_display_number_option,
'format': selected_display_format_option,
'ln': ln},
_("Previous"))
# Page Numbers
for i in range(1, nb_total_pages + 1):
if i != page_number:
page_links += '&nbsp;&nbsp;' + \
create_html_link(CFG_SITE_URL + '/yourcomments/',
{'page': i,
'order_by': selected_order_by_option,
'per_page': selected_display_number_option,
'format': selected_display_format_option,
'ln': ln},
str(i)) + \
'&nbsp;&nbsp;'
elif nb_total_pages > 1:
page_links += '''&nbsp;&nbsp;<b>%s</b>&nbsp;&nbsp;''' % i
# Next
if page_number != nb_total_pages:
page_links += create_html_link(CFG_SITE_URL + '/yourcomments/',
{'page': page_number + 1,
'order_by': selected_order_by_option,
'per_page': selected_display_number_option,
'format': selected_display_format_option,
'ln': ln},
_("Next"))
out += '<br/><div id="yourcommentsnavigationlinks">' + page_links + '</div>'
return out
diff --git a/invenio/legacy/webcomment/webinterface.py b/invenio/legacy/webcomment/webinterface.py
index adaa2e953..6dc14ce58 100644
--- a/invenio/legacy/webcomment/webinterface.py
+++ b/invenio/legacy/webcomment/webinterface.py
@@ -1,934 +1,934 @@
# -*- coding: utf-8 -*-
## Comments and reviews for records.
## This file is part of Invenio.
## Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2012, 2013 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
""" Comments and reviews for records: web interface """
__lastupdated__ = """$Date$"""
__revision__ = """$Id$"""
import cgi
from six import iteritems
from invenio.modules.comments.api import check_recID_is_in_range, \
perform_request_display_comments_or_remarks, \
perform_request_add_comment_or_remark, \
perform_request_vote, \
perform_request_report, \
subscribe_user_to_discussion, \
unsubscribe_user_from_discussion, \
get_user_subscription_to_discussion, \
check_user_can_attach_file_to_comments, \
check_user_can_view_comments, \
check_user_can_send_comments, \
check_user_can_view_comment, \
query_get_comment, \
toggle_comment_visibility, \
check_comment_belongs_to_record, \
is_comment_deleted, \
perform_display_your_comments
from invenio.config import \
- CFG_TMPDIR, \
+ CFG_TMPSHAREDDIR, \
CFG_SITE_LANG, \
CFG_SITE_URL, \
CFG_SITE_SECURE_URL, \
CFG_PREFIX, \
CFG_SITE_NAME, \
CFG_SITE_NAME_INTL, \
CFG_WEBCOMMENT_ALLOW_COMMENTS,\
CFG_WEBCOMMENT_ALLOW_REVIEWS, \
CFG_WEBCOMMENT_USE_MATHJAX_IN_COMMENTS, \
CFG_SITE_RECORD, \
CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE, \
CFG_WEBCOMMENT_MAX_ATTACHED_FILES, \
CFG_ACCESS_CONTROL_LEVEL_SITE
from invenio.legacy.webuser import getUid, page_not_authorized, isGuestUser, collect_user_info
from invenio.legacy.webpage import page, pageheaderonly, pagefooteronly
from invenio.legacy.search_engine import create_navtrail_links, \
guess_primary_collection_of_a_record, \
get_colID
from invenio.utils.url import redirect_to_url, \
make_canonical_urlargd
from invenio.utils.html import get_mathjax_header
from invenio.ext.logging import register_exception
from invenio.base.i18n import gettext_set_language
from invenio.ext.legacy.handler import wash_urlargd, WebInterfaceDirectory
from invenio.legacy.websearch.adminlib import get_detailed_page_tabs, get_detailed_page_tabs_counts
from invenio.modules.access.local_config import VIEWRESTRCOLL
from invenio.modules.access.mailcookie import \
mail_cookie_create_authorize_action, \
mail_cookie_create_common, \
mail_cookie_check_common, \
InvenioWebAccessMailCookieDeletedError, \
InvenioWebAccessMailCookieError
from invenio.modules.comments.config import \
InvenioWebCommentError, \
InvenioWebCommentWarning
import invenio.legacy.template
webstyle_templates = invenio.legacy.template.load('webstyle')
websearch_templates = invenio.legacy.template.load('websearch')
import os
from invenio.utils import apache
from invenio.legacy.bibdocfile.api import \
stream_file, \
decompose_file, \
propose_next_docname
class WebInterfaceCommentsPages(WebInterfaceDirectory):
"""Defines the set of /comments pages."""
_exports = ['', 'display', 'add', 'vote', 'report', 'index', 'attachments',
'subscribe', 'unsubscribe', 'toggle']
def __init__(self, recid=-1, reviews=0):
self.recid = recid
self.discussion = reviews # 0:comments, 1:reviews
self.attachments = WebInterfaceCommentsFiles(recid, reviews)
def index(self, req, form):
"""
Redirects to display function
"""
return self.display(req, form)
def display(self, req, form):
"""
Display comments (reviews if enabled) associated with record having id recid where recid>0.
This function can also be used to display remarks associated with basket having id recid where recid<-99.
@param ln: language
@param recid: record id, integer
@param do: display order hh = highest helpful score, review only
lh = lowest helpful score, review only
hs = highest star score, review only
ls = lowest star score, review only
od = oldest date
nd = newest date
@param ds: display since all= no filtering by date
nd = n days ago
nw = n weeks ago
nm = n months ago
ny = n years ago
where n is a single digit integer between 0 and 9
@param nb: number of results per page
@param p: results page
@param voted: boolean, active if user voted for a review, see vote function
@param reported: int, active if user reported a certain comment/review, see report function
@param reviews: boolean, enabled for reviews, disabled for comments
@param subscribed: int, 1 if user just subscribed to discussion, -1 if unsubscribed
@return the full html page.
"""
argd = wash_urlargd(form, {'do': (str, "od"),
'ds': (str, "all"),
'nb': (int, 100),
'p': (int, 1),
'voted': (int, -1),
'reported': (int, -1),
'subscribed': (int, 0),
'cmtgrp': (list, ["latest"]) # 'latest' is now a reserved group/round name
})
_ = gettext_set_language(argd['ln'])
uid = getUid(req)
user_info = collect_user_info(req)
(auth_code, auth_msg) = check_user_can_view_comments(user_info, self.recid)
if auth_code and user_info['email'] == 'guest':
cookie = mail_cookie_create_authorize_action(VIEWRESTRCOLL, {'collection' : guess_primary_collection_of_a_record(self.recid)})
target = CFG_SITE_SECURE_URL + '/youraccount/login' + \
make_canonical_urlargd({'action': cookie, 'ln' : argd['ln'], 'referer' : \
CFG_SITE_SECURE_URL + user_info['uri']}, {})
return redirect_to_url(req, target, norobot=True)
elif auth_code:
return page_not_authorized(req, "../", \
text = auth_msg)
can_send_comments = False
(auth_code, auth_msg) = check_user_can_send_comments(user_info, self.recid)
if not auth_code:
can_send_comments = True
can_attach_files = False
(auth_code, auth_msg) = check_user_can_attach_file_to_comments(user_info, self.recid)
if not auth_code and (user_info['email'] != 'guest'):
can_attach_files = True
subscription = get_user_subscription_to_discussion(self.recid, uid)
if subscription == 1:
user_is_subscribed_to_discussion = True
user_can_unsubscribe_from_discussion = True
elif subscription == 2:
user_is_subscribed_to_discussion = True
user_can_unsubscribe_from_discussion = False
else:
user_is_subscribed_to_discussion = False
user_can_unsubscribe_from_discussion = False
unordered_tabs = get_detailed_page_tabs(get_colID(guess_primary_collection_of_a_record(self.recid)),
self.recid,
ln=argd['ln'])
ordered_tabs_id = [(tab_id, values['order']) for (tab_id, values) in iteritems(unordered_tabs)]
ordered_tabs_id.sort(lambda x, y: cmp(x[1], y[1]))
link_ln = ''
if argd['ln'] != CFG_SITE_LANG:
link_ln = '?ln=%s' % argd['ln']
tabs = [(unordered_tabs[tab_id]['label'], \
'%s/record/%s/%s%s' % (CFG_SITE_URL, self.recid, tab_id, link_ln), \
tab_id in ['comments', 'reviews'],
unordered_tabs[tab_id]['enabled']) \
for (tab_id, order) in ordered_tabs_id
if unordered_tabs[tab_id]['visible'] == True]
tabs_counts = get_detailed_page_tabs_counts(self.recid)
citedbynum = tabs_counts['Citations']
references = tabs_counts['References']
discussions = tabs_counts['Discussions']
top = webstyle_templates.detailed_record_container_top(self.recid,
tabs,
argd['ln'],
citationnum=citedbynum,
referencenum=references,
discussionnum=discussions)
bottom = webstyle_templates.detailed_record_container_bottom(self.recid,
tabs,
argd['ln'])
#display_comment_rounds = [cmtgrp for cmtgrp in argd['cmtgrp'] if cmtgrp.isdigit() or cmtgrp == "all" or cmtgrp == "-1"]
display_comment_rounds = argd['cmtgrp']
check_warnings = []
(ok, problem) = check_recID_is_in_range(self.recid, check_warnings, argd['ln'])
if ok:
body = perform_request_display_comments_or_remarks(req=req, recID=self.recid,
display_order=argd['do'],
display_since=argd['ds'],
nb_per_page=argd['nb'],
page=argd['p'],
ln=argd['ln'],
voted=argd['voted'],
reported=argd['reported'],
subscribed=argd['subscribed'],
reviews=self.discussion,
uid=uid,
can_send_comments=can_send_comments,
can_attach_files=can_attach_files,
user_is_subscribed_to_discussion=user_is_subscribed_to_discussion,
user_can_unsubscribe_from_discussion=user_can_unsubscribe_from_discussion,
display_comment_rounds=display_comment_rounds
)
title, description, keywords = websearch_templates.tmpl_record_page_header_content(req, self.recid, argd['ln'])
navtrail = create_navtrail_links(cc=guess_primary_collection_of_a_record(self.recid), ln=argd['ln'])
if navtrail:
navtrail += ' &gt; '
navtrail += '<a class="navtrail" href="%s/%s/%s?ln=%s">'% (CFG_SITE_URL, CFG_SITE_RECORD, self.recid, argd['ln'])
navtrail += cgi.escape(title)
navtrail += '</a>'
navtrail += ' &gt; <a class="navtrail">%s</a>' % (self.discussion==1 and _("Reviews") or _("Comments"))
mathjaxheader = ''
if CFG_WEBCOMMENT_USE_MATHJAX_IN_COMMENTS:
mathjaxheader = get_mathjax_header(req.is_https())
jqueryheader = '''
<script src="%(CFG_SITE_URL)s/js/jquery.MultiFile.pack.js" type="text/javascript" language="javascript"></script>
''' % {'CFG_SITE_URL': CFG_SITE_URL}
return pageheaderonly(title=title,
navtrail=navtrail,
uid=uid,
verbose=1,
metaheaderadd = mathjaxheader + jqueryheader,
req=req,
language=argd['ln'],
navmenuid='search',
navtrail_append_title_p=0) + \
websearch_templates.tmpl_search_pagestart(argd['ln']) + \
top + body + bottom + \
websearch_templates.tmpl_search_pageend(argd['ln']) + \
pagefooteronly(lastupdated=__lastupdated__, language=argd['ln'], req=req)
else:
return page(title=_("Record Not Found"),
body=problem,
uid=uid,
verbose=1,
req=req,
language=argd['ln'],
navmenuid='search')
# Return the same page wether we ask for /CFG_SITE_RECORD/123 or /CFG_SITE_RECORD/123/
__call__ = index
def add(self, req, form):
"""
Add a comment (review) to record with id recid where recid>0
Also works for adding a remark to basket with id recid where recid<-99
@param ln: languange
@param recid: record id
@param action: 'DISPLAY' to display add form
'SUBMIT' to submit comment once form is filled
'REPLY' to reply to an already existing comment
@param msg: the body of the comment/review or remark
@param score: star score of the review
@param note: title of the review
@param comid: comment id, needed for replying
@param editor_type: the type of editor used for submitting the
comment: 'textarea', 'ckeditor'.
@param subscribe: if set, subscribe user to receive email
notifications when new comment are added to
this discussion
@return the full html page.
"""
argd = wash_urlargd(form, {'action': (str, "DISPLAY"),
'msg': (str, ""),
'note': (str, ''),
'score': (int, 0),
'comid': (int, 0),
'editor_type': (str, ""),
'subscribe': (str, ""),
'cookie': (str, "")
})
_ = gettext_set_language(argd['ln'])
actions = ['DISPLAY', 'REPLY', 'SUBMIT']
uid = getUid(req)
# Is site ready to accept comments?
if uid == -1 or (not CFG_WEBCOMMENT_ALLOW_COMMENTS and not CFG_WEBCOMMENT_ALLOW_REVIEWS):
return page_not_authorized(req, "../comments/add",
navmenuid='search')
# Is user allowed to post comment?
user_info = collect_user_info(req)
(auth_code_1, auth_msg_1) = check_user_can_view_comments(user_info, self.recid)
(auth_code_2, auth_msg_2) = check_user_can_send_comments(user_info, self.recid)
if isGuestUser(uid):
cookie = mail_cookie_create_authorize_action(VIEWRESTRCOLL, {'collection' : guess_primary_collection_of_a_record(self.recid)})
# Save user's value in cookie, so that these "POST"
# parameters are not lost during login process
msg_cookie = mail_cookie_create_common('comment_msg',
{'msg': argd['msg'],
'note': argd['note'],
'score': argd['score'],
'editor_type': argd['editor_type'],
'subscribe': argd['subscribe']},
onetime=True)
target = CFG_SITE_SECURE_URL + '/youraccount/login' + \
make_canonical_urlargd({'action': cookie, 'ln' : argd['ln'], 'referer' : \
CFG_SITE_SECURE_URL + user_info['uri'] + '&cookie=' + msg_cookie}, {})
return redirect_to_url(req, target, norobot=True)
elif (auth_code_1 or auth_code_2):
return page_not_authorized(req, "../", \
text = auth_msg_1 + auth_msg_2)
if argd['comid']:
# If replying to a comment, are we on a record that
# matches the original comment user is replying to?
if not check_comment_belongs_to_record(argd['comid'], self.recid):
return page_not_authorized(req, "../", \
text = _("Specified comment does not belong to this record"))
# Is user trying to reply to a restricted comment? Make
# sure user has access to it. We will then inherit its
# restriction for the new comment
(auth_code, auth_msg) = check_user_can_view_comment(user_info, argd['comid'])
if auth_code:
return page_not_authorized(req, "../", \
text = _("You do not have access to the specified comment"))
# Is user trying to reply to a deleted comment? If so, we
# let submitted comment go (to not lose possibly submitted
# content, if comment is submitted while original is
# deleted), but we "reset" comid to make sure that for
# action 'REPLY' the original comment is not included in
# the reply
if is_comment_deleted(argd['comid']):
argd['comid'] = 0
user_info = collect_user_info(req)
can_attach_files = False
(auth_code, auth_msg) = check_user_can_attach_file_to_comments(user_info, self.recid)
if not auth_code and (user_info['email'] != 'guest'):
can_attach_files = True
warning_msgs = [] # list of warning tuples (warning_text, warning_color)
added_files = {}
if can_attach_files:
# User is allowed to attach files. Process the files
file_too_big = False
formfields = form.get('commentattachment[]', [])
if not hasattr(formfields, "__getitem__"): # A single file was uploaded
formfields = [formfields]
for formfield in formfields[:CFG_WEBCOMMENT_MAX_ATTACHED_FILES]:
if hasattr(formfield, "filename") and formfield.filename:
filename = formfield.filename
- dir_to_open = os.path.join(CFG_TMPDIR, 'webcomment', str(uid))
+ dir_to_open = os.path.join(CFG_TMPSHAREDDIR, 'webcomment', str(uid))
try:
- assert(dir_to_open.startswith(CFG_TMPDIR))
+ assert(dir_to_open.startswith(CFG_TMPSHAREDDIR))
except AssertionError:
register_exception(req=req,
prefix='User #%s tried to upload file to forbidden location: %s' \
% (uid, dir_to_open))
if not os.path.exists(dir_to_open):
try:
os.makedirs(dir_to_open)
except:
register_exception(req=req, alert_admin=True)
## Before saving the file to disc, wash the filename (in particular
## washing away UNIX and Windows (e.g. DFS) paths):
filename = os.path.basename(filename.split('\\')[-1])
filename = filename.strip()
if filename != "":
# Check that file does not already exist
n = 1
while os.path.exists(os.path.join(dir_to_open, filename)):
basedir, name, extension = decompose_file(filename)
new_name = propose_next_docname(name)
filename = new_name + extension
fp = open(os.path.join(dir_to_open, filename), "w")
# FIXME: temporary, waiting for wsgi handler to be
# fixed. Once done, read chunk by chunk
## while formfield.file:
## fp.write(formfield.file.read(10240))
fp.write(formfield.file.read())
fp.close()
# Isn't this file too big?
file_size = os.path.getsize(os.path.join(dir_to_open, filename))
if CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE > 0 and \
file_size > CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE:
os.remove(os.path.join(dir_to_open, filename))
# One file is too big: record that,
# dismiss all uploaded files and re-ask to
# upload again
file_too_big = True
try:
raise InvenioWebCommentWarning(_('The size of file \\"%(x_file)s\\" (%(x_size)s) is larger than maximum allowed file size (%(x_max)s). Select files again.',
x_file=cgi.escape(filename), x_size=str(file_size/1024) + 'KB', x_max=str(CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE/1024) + 'KB'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning')
warning_msgs.append((exc.message, ''))
#warning_msgs.append(('WRN_WEBCOMMENT_MAX_FILE_SIZE_REACHED', cgi.escape(filename), str(file_size/1024) + 'KB', str(CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE/1024) + 'KB'))
else:
added_files[filename] = os.path.join(dir_to_open, filename)
if file_too_big:
# One file was too big. Removed all uploaded filed
for filepath in added_files.items():
try:
os.remove(filepath)
except:
# File was already removed or does not exist?
pass
client_ip_address = req.remote_ip
check_warnings = []
(ok, problem) = check_recID_is_in_range(self.recid, check_warnings, argd['ln'])
if ok:
title, description, keywords = websearch_templates.tmpl_record_page_header_content(req,
self.recid,
argd['ln'])
navtrail = create_navtrail_links(cc=guess_primary_collection_of_a_record(self.recid))
if navtrail:
navtrail += ' &gt; '
navtrail += '<a class="navtrail" href="%s/%s/%s?ln=%s">'% (CFG_SITE_URL, CFG_SITE_RECORD, self.recid, argd['ln'])
navtrail += cgi.escape(title)
navtrail += '</a>'
navtrail += '&gt; <a class="navtrail" href="%s/%s/%s/%s/?ln=%s">%s</a>' % (CFG_SITE_URL,
CFG_SITE_RECORD,
self.recid,
self.discussion==1 and 'reviews' or 'comments',
argd['ln'],
self.discussion==1 and _('Reviews') or _('Comments'))
if argd['action'] not in actions:
argd['action'] = 'DISPLAY'
if not argd['msg']:
# User had to login in-between, so retrieve msg
# from cookie
try:
(kind, cookie_argd) = mail_cookie_check_common(argd['cookie'],
delete=True)
argd.update(cookie_argd)
except InvenioWebAccessMailCookieDeletedError as e:
return redirect_to_url(req, CFG_SITE_SECURE_URL + '/'+ CFG_SITE_RECORD +'/' + \
str(self.recid) + (self.discussion==1 and \
'/reviews' or '/comments'))
except InvenioWebAccessMailCookieError as e:
# Invalid or empty cookie: continue
pass
subscribe = False
if argd['subscribe'] and \
get_user_subscription_to_discussion(self.recid, uid) == 0:
# User is not already subscribed, and asked to subscribe
subscribe = True
body = perform_request_add_comment_or_remark(recID=self.recid,
ln=argd['ln'],
uid=uid,
action=argd['action'],
msg=argd['msg'],
note=argd['note'],
score=argd['score'],
reviews=self.discussion,
comID=argd['comid'],
client_ip_address=client_ip_address,
editor_type=argd['editor_type'],
can_attach_files=can_attach_files,
subscribe=subscribe,
req=req,
attached_files=added_files,
warnings=warning_msgs)
if self.discussion:
title = _("Add Review")
else:
title = _("Add Comment")
jqueryheader = '''
<script src="%(CFG_SITE_URL)s/js/jquery.MultiFile.pack.js" type="text/javascript" language="javascript"></script>
''' % {'CFG_SITE_URL': CFG_SITE_URL}
return page(title=title,
body=body,
navtrail=navtrail,
uid=uid,
language=CFG_SITE_LANG,
verbose=1,
req=req,
navmenuid='search',
metaheaderadd=jqueryheader)
# id not in range
else:
return page(title=_("Record Not Found"),
body=problem,
uid=uid,
verbose=1,
req=req,
navmenuid='search')
def vote(self, req, form):
"""
Vote positively or negatively for a comment/review.
@param comid: comment/review id
@param com_value: +1 to vote positively
-1 to vote negatively
@param recid: the id of the record the comment/review is associated with
@param ln: language
@param do: display order hh = highest helpful score, review only
lh = lowest helpful score, review only
hs = highest star score, review only
ls = lowest star score, review only
od = oldest date
nd = newest date
@param ds: display since all= no filtering by date
nd = n days ago
nw = n weeks ago
nm = n months ago
ny = n years ago
where n is a single digit integer between 0 and 9
@param nb: number of results per page
@param p: results page
@param referer: http address of the calling function to redirect to (refresh)
@param reviews: boolean, enabled for reviews, disabled for comments
"""
argd = wash_urlargd(form, {'comid': (int, -1),
'com_value': (int, 0),
'recid': (int, -1),
'do': (str, "od"),
'ds': (str, "all"),
'nb': (int, 100),
'p': (int, 1),
'referer': (str, None)
})
_ = gettext_set_language(argd['ln'])
client_ip_address = req.remote_ip
uid = getUid(req)
user_info = collect_user_info(req)
(auth_code, auth_msg) = check_user_can_view_comments(user_info, self.recid)
if auth_code and user_info['email'] == 'guest':
cookie = mail_cookie_create_authorize_action(VIEWRESTRCOLL, {'collection' : guess_primary_collection_of_a_record(self.recid)})
target = CFG_SITE_SECURE_URL + '/youraccount/login' + \
make_canonical_urlargd({'action': cookie, 'ln' : argd['ln'], 'referer' : \
CFG_SITE_SECURE_URL + user_info['uri']}, {})
return redirect_to_url(req, target, norobot=True)
elif auth_code:
return page_not_authorized(req, "../", \
text = auth_msg)
# Check that comment belongs to this recid
if not check_comment_belongs_to_record(argd['comid'], self.recid):
return page_not_authorized(req, "../", \
text = _("Specified comment does not belong to this record"))
# Check that user can access the record
(auth_code, auth_msg) = check_user_can_view_comment(user_info, argd['comid'])
if auth_code:
return page_not_authorized(req, "../", \
text = _("You do not have access to the specified comment"))
# Check that comment is not currently deleted
if is_comment_deleted(argd['comid']):
return page_not_authorized(req, "../", \
text = _("You cannot vote for a deleted comment"),
ln=argd['ln'])
success = perform_request_vote(argd['comid'], client_ip_address, argd['com_value'], uid)
if argd['referer']:
argd['referer'] += "?ln=%s&do=%s&ds=%s&nb=%s&p=%s&voted=%s&" % (
argd['ln'], argd['do'], argd['ds'], argd['nb'], argd['p'], success)
redirect_to_url(req, argd['referer'])
else:
#Note: sent to comments display
referer = "%s/%s/%s/%s?&ln=%s&voted=1"
referer %= (CFG_SITE_SECURE_URL, CFG_SITE_RECORD, self.recid, self.discussion == 1 and 'reviews' or 'comments', argd['ln'])
redirect_to_url(req, referer)
def report(self, req, form):
"""
Report a comment/review for inappropriate content
@param comid: comment/review id
@param recid: the id of the record the comment/review is associated with
@param ln: language
@param do: display order hh = highest helpful score, review only
lh = lowest helpful score, review only
hs = highest star score, review only
ls = lowest star score, review only
od = oldest date
nd = newest date
@param ds: display since all= no filtering by date
nd = n days ago
nw = n weeks ago
nm = n months ago
ny = n years ago
where n is a single digit integer between 0 and 9
@param nb: number of results per page
@param p: results page
@param referer: http address of the calling function to redirect to (refresh)
@param reviews: boolean, enabled for reviews, disabled for comments
"""
argd = wash_urlargd(form, {'comid': (int, -1),
'recid': (int, -1),
'do': (str, "od"),
'ds': (str, "all"),
'nb': (int, 100),
'p': (int, 1),
'referer': (str, None)
})
_ = gettext_set_language(argd['ln'])
client_ip_address = req.remote_ip
uid = getUid(req)
user_info = collect_user_info(req)
(auth_code, auth_msg) = check_user_can_view_comments(user_info, self.recid)
if isGuestUser(uid):
cookie = mail_cookie_create_authorize_action(VIEWRESTRCOLL, {'collection' : guess_primary_collection_of_a_record(self.recid)})
target = CFG_SITE_SECURE_URL + '/youraccount/login' + \
make_canonical_urlargd({'action': cookie, 'ln' : argd['ln'], 'referer' : \
CFG_SITE_SECURE_URL + user_info['uri']}, {})
return redirect_to_url(req, target, norobot=True)
elif auth_code:
return page_not_authorized(req, "../", \
text = auth_msg)
# Check that comment belongs to this recid
if not check_comment_belongs_to_record(argd['comid'], self.recid):
return page_not_authorized(req, "../", \
text = _("Specified comment does not belong to this record"))
# Check that user can access the record
(auth_code, auth_msg) = check_user_can_view_comment(user_info, argd['comid'])
if auth_code:
return page_not_authorized(req, "../", \
text = _("You do not have access to the specified comment"))
# Check that comment is not currently deleted
if is_comment_deleted(argd['comid']):
return page_not_authorized(req, "../", \
text = _("You cannot report a deleted comment"),
ln=argd['ln'])
success = perform_request_report(argd['comid'], client_ip_address, uid)
if argd['referer']:
argd['referer'] += "?ln=%s&do=%s&ds=%s&nb=%s&p=%s&reported=%s&" % (argd['ln'], argd['do'], argd['ds'], argd['nb'], argd['p'], str(success))
redirect_to_url(req, argd['referer'])
else:
#Note: sent to comments display
referer = "%s/%s/%s/%s/display?ln=%s&voted=1"
referer %= (CFG_SITE_SECURE_URL, CFG_SITE_RECORD, self.recid, self.discussion==1 and 'reviews' or 'comments', argd['ln'])
redirect_to_url(req, referer)
def subscribe(self, req, form):
"""
Subscribe current user to receive email notification when new
comments are added to current discussion.
"""
argd = wash_urlargd(form, {'referer': (str, None)})
uid = getUid(req)
user_info = collect_user_info(req)
(auth_code, auth_msg) = check_user_can_view_comments(user_info, self.recid)
if isGuestUser(uid):
cookie = mail_cookie_create_authorize_action(VIEWRESTRCOLL, {'collection' : guess_primary_collection_of_a_record(self.recid)})
target = CFG_SITE_SECURE_URL + '/youraccount/login' + \
make_canonical_urlargd({'action': cookie, 'ln' : argd['ln'], 'referer' : \
CFG_SITE_SECURE_URL + user_info['uri']}, {})
return redirect_to_url(req, target, norobot=True)
elif auth_code:
return page_not_authorized(req, "../", \
text = auth_msg)
success = subscribe_user_to_discussion(self.recid, uid)
display_url = "%s/%s/%s/comments/display?subscribed=%s&ln=%s" % \
(CFG_SITE_SECURE_URL, CFG_SITE_RECORD, self.recid, str(success), argd['ln'])
redirect_to_url(req, display_url)
def unsubscribe(self, req, form):
"""
Unsubscribe current user from current discussion.
"""
argd = wash_urlargd(form, {'referer': (str, None)})
user_info = collect_user_info(req)
uid = getUid(req)
if isGuestUser(uid):
cookie = mail_cookie_create_authorize_action(VIEWRESTRCOLL, {'collection' : guess_primary_collection_of_a_record(self.recid)})
target = CFG_SITE_SECURE_URL + '/youraccount/login' + \
make_canonical_urlargd({'action': cookie, 'ln' : argd['ln'], 'referer' : \
CFG_SITE_SECURE_URL + user_info['uri']}, {})
return redirect_to_url(req, target, norobot=True)
success = unsubscribe_user_from_discussion(self.recid, uid)
display_url = "%s/%s/%s/comments/display?subscribed=%s&ln=%s" % \
(CFG_SITE_SECURE_URL, CFG_SITE_RECORD, self.recid, str(-success), argd['ln'])
redirect_to_url(req, display_url)
def toggle(self, req, form):
"""
Store the visibility of a comment for current user
"""
argd = wash_urlargd(form, {'comid': (int, -1),
'referer': (str, None),
'collapse': (int, 1)})
uid = getUid(req)
if isGuestUser(uid):
# We do not store information for guests
return ''
toggle_comment_visibility(uid, argd['comid'], argd['collapse'], self.recid)
if argd['referer']:
return redirect_to_url(req, CFG_SITE_SECURE_URL + \
(not argd['referer'].startswith('/') and '/' or '') + \
argd['referer'] + '#' + str(argd['comid']))
class WebInterfaceCommentsFiles(WebInterfaceDirectory):
"""Handle <strike>upload and </strike> access to files for comments.
<strike>The upload is currently only available through the Ckeditor.</strike>
"""
#_exports = ['put'] # 'get' is handled by _lookup(..)
def __init__(self, recid=-1, reviews=0):
self.recid = recid
self.discussion = reviews # 0:comments, 1:reviews
def _lookup(self, component, path):
""" This handler is invoked for the dynamic URLs (for getting
<strike>and putting attachments</strike>) Eg:
CFG_SITE_URL/CFG_SITE_RECORD/5953/comments/attachments/get/652/myfile.pdf
"""
if component == 'get' and len(path) > 1:
comid = path[0] # comment ID
file_name = '/'.join(path[1:]) # the filename
def answer_get(req, form):
"""Accessing files attached to comments."""
form['file'] = file_name
form['comid'] = comid
return self._get(req, form)
return answer_get, []
# All other cases: file not found
return None, []
def _get(self, req, form):
"""
Returns a file attached to a comment.
Example:
CFG_SITE_URL/CFG_SITE_RECORD/5953/comments/attachments/get/652/myfile.pdf
where 652 is the comment ID
"""
argd = wash_urlargd(form, {'file': (str, None),
'comid': (int, 0)})
_ = gettext_set_language(argd['ln'])
# Can user view this record, i.e. can user access its
# attachments?
uid = getUid(req)
user_info = collect_user_info(req)
# Check that user can view record, and its comments (protected
# with action "viewcomment")
(auth_code, auth_msg) = check_user_can_view_comments(user_info, self.recid)
if auth_code and user_info['email'] == 'guest':
cookie = mail_cookie_create_authorize_action(VIEWRESTRCOLL, {'collection' : guess_primary_collection_of_a_record(self.recid)})
target = CFG_SITE_SECURE_URL + '/youraccount/login' + \
make_canonical_urlargd({'action': cookie, 'ln' : argd['ln'], 'referer' : \
CFG_SITE_SECURE_URL + user_info['uri']}, {})
return redirect_to_url(req, target, norobot=True)
elif auth_code:
return page_not_authorized(req, "../", \
text = auth_msg)
# Does comment exist?
if not query_get_comment(argd['comid']):
req.status = apache.HTTP_NOT_FOUND
return page(title=_("Page Not Found"),
body=_('The requested comment could not be found'),
req=req)
# Check that user can view this particular comment, protected
# using its own restriction
(auth_code, auth_msg) = check_user_can_view_comment(user_info, argd['comid'])
if auth_code and user_info['email'] == 'guest':
cookie = mail_cookie_create_authorize_action(VIEWRESTRCOLL, {'collection' : guess_primary_collection_of_a_record(self.recid)})
target = CFG_SITE_SECURE_URL + '/youraccount/login' + \
make_canonical_urlargd({'action': cookie, 'ln' : argd['ln'], 'referer' : \
CFG_SITE_SECURE_URL + user_info['uri']}, {})
return redirect_to_url(req, target)
elif auth_code:
return page_not_authorized(req, "../", \
text = auth_msg,
ln=argd['ln'])
# Check that comment is not currently deleted
if is_comment_deleted(argd['comid']):
return page_not_authorized(req, "../", \
text = _("You cannot access files of a deleted comment"),
ln=argd['ln'])
if not argd['file'] is None:
# Prepare path to file on disk. Normalize the path so that
# ../ and other dangerous components are removed.
path = os.path.abspath(CFG_PREFIX + '/var/data/comments/' + \
str(self.recid) + '/' + str(argd['comid']) + \
'/' + argd['file'])
# Check that we are really accessing attachements
# directory, for the declared record.
if path.startswith(CFG_PREFIX + '/var/data/comments/' + \
str(self.recid)) and \
os.path.exists(path):
return stream_file(req, path)
# Send error 404 in all other cases
req.status = apache.HTTP_NOT_FOUND
return page(title=_("Page Not Found"),
body=_('The requested file could not be found'),
req=req,
language=argd['ln'])
class WebInterfaceYourCommentsPages(WebInterfaceDirectory):
"""Defines the set of /yourcomments pages."""
_exports = ['', ]
def index(self, req, form):
"""Index page."""
argd = wash_urlargd(form, {'page': (int, 1),
'format': (str, "rc"),
'order_by': (str, "lcf"),
'per_page': (str, "all"),
})
# TODO: support also "reviews", by adding new option to show/hide them if needed
uid = getUid(req)
# load the right language
_ = gettext_set_language(argd['ln'])
# Is site ready to accept comments?
if not CFG_WEBCOMMENT_ALLOW_COMMENTS or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return page_not_authorized(req, "%s/yourcomments" % \
(CFG_SITE_SECURE_URL,),
text="Comments are currently disabled on this site",
navmenuid="yourcomments")
elif uid == -1 or isGuestUser(uid):
return redirect_to_url(req, "%s/youraccount/login%s" % (
CFG_SITE_SECURE_URL,
make_canonical_urlargd({
'referer' : "%s/yourcomments%s" % (
CFG_SITE_SECURE_URL,
make_canonical_urlargd(argd, {})),
"ln" : argd['ln']}, {})))
user_info = collect_user_info(req)
if not user_info['precached_sendcomments']:
# Maybe we should still authorize if user submitted
# comments in the past?
return page_not_authorized(req, "../", \
text = _("You are not authorized to use comments."))
return page(title=_("Your Comments"),
body=perform_display_your_comments(user_info,
page_number=argd['page'],
selected_order_by_option=argd['order_by'],
selected_display_number_option=argd['per_page'],
selected_display_format_option=argd['format'],
ln=argd['ln']),
navtrail= """<a class="navtrail" href="%(sitesecureurl)s/youraccount/display?ln=%(ln)s">%(account)s</a>""" % {
'sitesecureurl' : CFG_SITE_SECURE_URL,
'ln': argd['ln'],
'account' : _("Your Account"),
},
description=_("%(x_name)s View your previously submitted comments", x_name=CFG_SITE_NAME_INTL.get(argd['ln'], CFG_SITE_NAME)),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(argd['ln'], CFG_SITE_NAME)),
uid=uid,
language=argd['ln'],
req=req,
lastupdated=__lastupdated__,
navmenuid='youralerts',
secure_page_p=1)
# Return the same page wether we ask for /CFG_SITE_RECORD/123 or /CFG_SITE_RECORD/123/
__call__ = index
diff --git a/invenio/legacy/webhelp/web/admin/howto/howto-run.webdoc b/invenio/legacy/webhelp/web/admin/howto/howto-run.webdoc
index 2e10f1a71..412a8b4da 100644
--- a/invenio/legacy/webhelp/web/admin/howto/howto-run.webdoc
+++ b/invenio/legacy/webhelp/web/admin/howto/howto-run.webdoc
@@ -1,341 +1,351 @@
## This file is part of Invenio.
-## Copyright (C) 2007, 2008, 2009, 2010, 2011, 2013 CERN.
+## Copyright (C) 2007, 2008, 2009, 2010, 2011, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
<!-- WebDoc-Page-Title: HOWTO Run Your Invenio Installation -->
<!-- WebDoc-Page-Navtrail: <a class="navtrail" href="<CFG_SITE_URL>/help/admin<lang:link/>">Admin Area</a> &gt; <a class="navtrail" href="howto">Admin HOWTOs</a> -->
<!-- WebDoc-Page-Revision: $Id$ -->
<h2>Overview</h2>
<p>This HOWTO guide intends to give you ideas on how to run your CDS
Invenio installation and how to take care of its normal day-to-day
operation.
</p>
<h2>Setting up periodical daemon tasks</h2>
<p>Many tasks that manipulate the bibliographic record database can be
set to run in a periodical mode. For example, we want to have the
indexing engine to scan periodically for newly arrived documents to
index them as soon as they enter into the system. It is the role of
the BibSched system to take care of the task scheduling and the task
execution.
</p>
<p>Periodical tasks (such as regular metadata indexing) as well as
one-time tasks (such as a batch upload of newly acquired metadata
file) are not executed straight away but are stored in the BibSched
task queue. BibSched daemon looks periodically in the queue and
launches the tasks according to their order or the date of programmed
runtime. You can consider BibSched to be a kind of cron daemon for
bibliographic tasks.
</p>
<p>This means that after having installed Invenio you will want to
have the BibSched daemon running permanently. To launch BibSched
daemon, do:
<blockquote>
<pre>
$ bibsched start
</pre>
</blockquote>
</p>
<p>To setup indexing, ranking, sorting, formatting, and collection cache updating
daemons to run periodically with a sleeping period of, say, 5 minutes:
<blockquote>
<pre>
$ bibindex -f50000 -s5m
$ bibreformat -oHB -s5m
$ webcoll -v0 -s5m
$ bibrank -f50000 -s5m
$ bibsort -s5m
</pre>
</blockquote>
</p>
-<p>It is imperative to have the above five tasks permanently in your
+<p>Note that if you are using virtual index facility, such as for
+the <em>global</em> index, then you should schedule them apart:
+
+<blockquote>
+<pre>
+$ bibindex -w global -f50000 -s5m
+</pre>
+</blockquote>
+</p>
+
+<p>It is imperative to have the above tasks run permanently in your
BibSched queue so that newly submitted documents will be processed
automatically.
</p>
<p>You may also want to set up some periodical housekeeping tasks:
<blockquote>
<pre>
$ bibrank -f50000 -R -wwrd -s14d -LSunday
$ bibsort -R -s7d -L 'Sunday 01:00-05:00'
$ inveniogc -a -s7d -L 'Sunday 01:00-05:00'
$ dbdump -s20h -L '22:00-06:00' -o/opt2/mysql-backups -n10
</pre>
</blockquote>
</p>
<p>Please consult the sections below for more details about these
housekeeping tasks.</p>
<p>There is also the possibility to setup the batch uploader daemon
to run periodically, looking for new documents or metadata files
to upload:
<blockquote>
<pre>
$ batchuploader --documents -s20m
$ batchuploader --metadata -s20m
</pre>
</blockquote>
</p>
<p>Additionally you might want to automatically generate <tt>sitemap.xml</tt>
files for your installation. For this just schedule:
<blockquote>
<pre>
$ bibexport -w sitemap -s1d
</pre>
</blockquote>
You will then need to add a line such as:
<blockquote>
<pre>
Sitemap: <CFG_SITE_URL>/sitemap-index.xml.gz
</pre>
</blockquote>
to your <tt>robots.txt</tt> file.</p>
<p>If you are using the WebLinkback module, you may want to run some of the following tasklets:
<blockquote>
<pre>
# Delete rejected, broken and pending linkbacks whose URLs is on blacklist
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbackupdaterdeleteurlsonblacklist \
-T bst_weblinkback_updater \
-a "mode=1" \
-u admin -s1d -L '22:00-05:00'
# Update page titles of new linkbacks
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbackupdaternewpages \
-T bst_weblinkback_updater \
-a "mode=2" \
-u admin -s1d -L '22:00-05:00'
# Update page titles of old linkbacks
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbackupdateroldpages \
-T bst_weblinkback_updater \
-a "mode=3" \
-u admin -s7d -L '22:00-05:00'
# Update manually set page titles
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbackupdatermanuallysettitles \
-T bst_weblinkback_updater \
-a "mode=4" \
-u admin -s7d -L '22:00-05:00'
# Detect and disable broken linkbacks
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbackupdaterdetectbrokenlinkbacks \
-T bst_weblinkback_updater \
-a "mode=5" \
-u admin -s7d -L 'Sunday 01:00-05:00'
# Send notification email for all pending linkbacks
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbacknotifications \
-T bst_weblinkback_updater \
-a "mode=6" \
-u admin -s1d
</pre>
</blockquote>
</p>
<h2>Monitoring periodical daemon tasks</h2>
<p>Note that the BibSched automated daemon stops as soon as some of
its tasks end with an error. You will be informed by email about this
incident. Nevertheless, it is a good idea to inspect the BibSched
queue from time to time anyway, say several times per day, to see what
is going on. This can be done by running the BibSched command-line
monitor:
<blockquote>
<pre>
$ bibsched
</pre>
</blockquote>
</p>
<p>The monitor will permit you to stop/start the automated mode, to
delete the tasks that were wrongly submitted, to run some of the tasks
manually, etc. Note also that the BibSched daemon writes its log and
error files about its own operation, as well as on the operation of
its tasks, to the <code>/opt/invenio/var/log</code> directory.</p>
<h2>Running alert engine</h2>
<p>Invenio users may set up an automatic notification email alerts
so that they are automatically alerted about documents of their
interest by email, either daily, weekly, or monthly. It is the job of
the alert engine to do this. The alert engine has to be run every
day:
<blockquote>
<pre>
$ alertengine
</pre>
</blockquote>
</p>
<p>You may want to set up an external cron job to call
<code>alertengine</code> each day, for example:
<blockquote>
<pre>
$ crontab -l
&#35; invenio: periodically restart Apache:
59 23 * * * /usr/sbin/apachectl restart
&#35; invenio: run alert engine:
30 14 * * * /usr/bin/sudo -u apache /opt/invenio/bin/alertengine
</pre>
</blockquote>
</p>
<h2>Housekeeping task details</h2>
<h4>Housekeeping: recalculating ranking weights</h4>
<p>When you are adding new records to the system, the word frequency
ranking weights for old records aren't recalculated by default in
order to speed up the insertion of new records. This may influence a
bit the precision of word similarity searches. It is therefore
advised to expressely run bibrank in the recalculating mode once in a
while during a relatively quiet site operation day, by doing:
<blockquote>
<pre>
$ bibrank -R -w wrd -s 14d -L Sunday
</pre>
</blockquote>
You may want to do this either (i) periodically, say once per month
(see the previous section), or (ii) depending on the frequency of new
additions to the record database, say when the size grows by 2-3
percent.
</p>
<h4>Housekeeping: recalculating sorting weights</h4>
<p>It is advised to run from time to time the rebalancing of the
sorting buckets. In order to speed up the process of insertion of
new records, the sorting buckets are not being recalculated,
but new records are being added at the end of the corresponding
bucket. This might create differences in the size of each bucket
which might have a small impact on the speed of sorting.
<blockquote>
<pre>
$ bibsort -R -s 7d -L 'Sunday 01:00-05:00'
</pre>
</blockquote>
The rebalancing might be run weekly or even daily.
</p>
<h4>Housekeeping: cleaning up the garbage</h4>
<p>The tool <code>inveniogc</code> provides a garbage collector for
the database, temporary files, and the like.</p>
<p>If you choose to differentiate between guest users
(see <code>CFG_WEBSESSION_DIFFERENTIATE_BETWEEN_GUESTS</code>
in <code>invenio.conf</code>), then guest users can create a lot of
entries in Invenio tables that are related to their web sessions,
their search history, personal baskets, etc. This data has to be
garbage-collected periodically. You can run this, say every Sunday
between 01:00 and 05:00, via:
<blockquote>
<pre>
$ inveniogc -s 7d -L 'Sunday 01:00-05:00'
</pre>
</blockquote>
</p>
<p>Different temporary log and err files are created
in <code>/opt/invenio/var/log</code>
and <code>/opt/invenio/var/tmp</code> directory that is good to
clean up from time to time. The previous command could be used to
clean those files, too, via:
<blockquote>
<pre>
$ inveniogc -s 7d -d -L 'Sunday 01:00-05:00'
</pre>
</blockquote>
</p>
<p>The <code>inveniogc</code> tool can run other cleaning actions;
please refer to its help (<code>inveniogc --help</code>) for more
details.</p>
<p>Note that in the above section "Setting up periodical daemon
tasks", we have set up <code>inveniogc</code> with
argument <code>-a</code> in that example, meaning that it would run
all possible cleaning actions. Please modify this if it is not what
you want.</p>
<h4>Housekeeping: backing up the database</h4>
<p>You can launch a bibsched task called <code>dbdump</code> in order
to take regular snapshot of your database content into SQL dump files.
For example, to back up the database content
into <code>/opt2/mysql-backups</code> directory every night, keeping
at most 10 latest copies of the backup file, you would launch:</p>
<blockquote>
<pre>
$ dbdump -s 20h -L '22:00-06:00' -o /opt2/mysql-backups -n 10
</pre>
</blockquote>
<p>This will create files named
like <code>invenio-dbdump-2009-03-10_22:10:28.sql</code> in this
folder.</p>
<p>Note that you may use Invenio-independent MySQL backuping tools
like <code>mysqldump</code>, but these might generally lock all tables
during backup for consistency, hence it could happen that your site
might not be accessible during backuping time due to the user session
table being locked as well. The <code>dbdump</code> tool does not
lock all tables, therefore the site remains accessible to users while
the dump files are being created. Note that the dump files are kept
consistent with respect to the data, since <code>dbdump</code> runs
via <code>bibsched</code>, hence not allowing any other important
bibliographic task to run during the backup.</p>
<p>To load a dump file produced by <code>dbdump</code> into a running
Invenio instance later, you can use:</p>
<blockquote>
<pre>
$ bibsched stop
$ cat /opt2/mysql-backups/invenio-dbdump-2009-03-10_22\:10\:28.sql | dbexec
$ bibsched start
</pre>
</blockquote>
diff --git a/invenio/legacy/websearch/templates.py b/invenio/legacy/websearch/templates.py
index 54b5e3ec9..11d66d4e6 100644
--- a/invenio/legacy/websearch/templates.py
+++ b/invenio/legacy/websearch/templates.py
@@ -1,4793 +1,4793 @@
# -*- coding: utf-8 -*-
## This file is part of Invenio.
## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
# pylint: disable=C0301
__revision__ = "$Id$"
import time
import cgi
import string
import re
import locale
from six import iteritems
from urllib import quote, urlencode
from xml.sax.saxutils import escape as xml_escape
from invenio.config import \
CFG_WEBSEARCH_LIGHTSEARCH_PATTERN_BOX_WIDTH, \
CFG_WEBSEARCH_SIMPLESEARCH_PATTERN_BOX_WIDTH, \
CFG_WEBSEARCH_ADVANCEDSEARCH_PATTERN_BOX_WIDTH, \
CFG_WEBSEARCH_AUTHOR_ET_AL_THRESHOLD, \
CFG_WEBSEARCH_USE_ALEPH_SYSNOS, \
CFG_WEBSEARCH_SPLIT_BY_COLLECTION, \
CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS, \
CFG_BIBRANK_SHOW_READING_STATS, \
CFG_BIBRANK_SHOW_DOWNLOAD_STATS, \
CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS, \
CFG_BIBRANK_SHOW_CITATION_LINKS, \
CFG_BIBRANK_SHOW_CITATION_STATS, \
CFG_BIBRANK_SHOW_CITATION_GRAPHS, \
CFG_WEBSEARCH_RSS_TTL, \
CFG_SITE_LANG, \
CFG_SITE_NAME, \
CFG_SITE_NAME_INTL, \
CFG_VERSION, \
CFG_SITE_SUPPORT_EMAIL, \
CFG_SITE_ADMIN_EMAIL, \
CFG_CERN_SITE, \
CFG_INSPIRE_SITE, \
CFG_WEBSEARCH_DEFAULT_SEARCH_INTERFACE, \
CFG_WEBSEARCH_ENABLED_SEARCH_INTERFACES, \
CFG_WEBSEARCH_MAX_RECORDS_IN_GROUPS, \
CFG_BIBINDEX_CHARS_PUNCTUATION, \
CFG_WEBCOMMENT_ALLOW_COMMENTS, \
CFG_WEBCOMMENT_ALLOW_REVIEWS, \
CFG_WEBSEARCH_WILDCARD_LIMIT, \
CFG_WEBSEARCH_SHOW_COMMENT_COUNT, \
CFG_WEBSEARCH_SHOW_REVIEW_COUNT, \
CFG_SITE_RECORD, \
CFG_WEBSEARCH_PREV_NEXT_HIT_LIMIT, \
CFG_HEPDATA_URL, \
CFG_HEPDATA_PLOTSIZE, \
CFG_BASE_URL, \
CFG_SITE_URL, \
CFG_WEBSEARCH_PREV_NEXT_HIT_FOR_GUESTS
from invenio.legacy.search_engine.config import CFG_WEBSEARCH_RESULTS_OVERVIEW_MAX_COLLS_TO_PRINT
from invenio.modules.search.services import \
CFG_WEBSEARCH_MAX_SEARCH_COLL_RESULTS_TO_PRINT
from invenio.legacy.dbquery import run_sql
from invenio.base.i18n import gettext_set_language
from invenio.base.globals import cfg
from invenio.utils.url import make_canonical_urlargd, drop_default_urlargd, create_html_link, create_url
from invenio.utils.html import nmtoken_from_string
from invenio.ext.legacy.handler import wash_urlargd
from invenio.legacy.bibrank.citation_searcher import get_cited_by_count
from invenio.legacy.webuser import session_param_get
from invenio.modules.search.services import \
CFG_WEBSEARCH_MAX_SEARCH_COLL_RESULTS_TO_PRINT
from intbitset import intbitset
from invenio.legacy.websearch_external_collections import external_collection_get_state, get_external_collection_engine
from invenio.legacy.websearch_external_collections.utils import get_collection_id
from invenio.legacy.websearch_external_collections.config import CFG_EXTERNAL_COLLECTION_MAXRESULTS
from invenio.legacy.bibrecord import get_fieldvalues
from invenio.modules.formatter import format_record
from invenio.legacy.search_engine.utils import record_exists
from invenio import hepdatadisplayutils
_RE_PUNCTUATION = re.compile(CFG_BIBINDEX_CHARS_PUNCTUATION)
_RE_SPACES = re.compile(r"\s+")
class Template:
# This dictionary maps Invenio language code to locale codes (ISO 639)
tmpl_localemap = {
'bg': 'bg_BG',
'ar': 'ar_AR',
'ca': 'ca_ES',
'de': 'de_DE',
'el': 'el_GR',
'en': 'en_US',
'es': 'es_ES',
'pt': 'pt_BR',
'fa': 'fa_IR',
'fr': 'fr_FR',
'it': 'it_IT',
'ka': 'ka_GE',
'lt': 'lt_LT',
'ro': 'ro_RO',
'ru': 'ru_RU',
'rw': 'rw_RW',
'sk': 'sk_SK',
'cs': 'cs_CZ',
'no': 'no_NO',
'sv': 'sv_SE',
'uk': 'uk_UA',
'ja': 'ja_JA',
'pl': 'pl_PL',
'hr': 'hr_HR',
'zh_CN': 'zh_CN',
'zh_TW': 'zh_TW',
'hu': 'hu_HU',
'af': 'af_ZA',
'gl': 'gl_ES'
}
tmpl_default_locale = "en_US" # which locale to use by default, useful in case of failure
# Type of the allowed parameters for the web interface for search results
@property
def search_results_default_urlargd(self):
from invenio.modules.search.washers import \
search_results_default_urlargd
return search_results_default_urlargd
# ...and for search interfaces
search_interface_default_urlargd = {
'aas': (int, CFG_WEBSEARCH_DEFAULT_SEARCH_INTERFACE),
'as': (int, CFG_WEBSEARCH_DEFAULT_SEARCH_INTERFACE),
'verbose': (int, 0),
'em' : (str, "")}
# ...and for RSS feeds
rss_default_urlargd = {'c' : (list, []),
'cc' : (str, ""),
'p' : (str, ""),
'f' : (str, ""),
'p1' : (str, ""),
'f1' : (str, ""),
'm1' : (str, ""),
'op1': (str, ""),
'p2' : (str, ""),
'f2' : (str, ""),
'm2' : (str, ""),
'op2': (str, ""),
'p3' : (str, ""),
'f3' : (str, ""),
'm3' : (str, ""),
'wl' : (int, CFG_WEBSEARCH_WILDCARD_LIMIT)}
tmpl_openurl_accepted_args = {
'id' : (list, []),
'genre' : (str, ''),
'aulast' : (str, ''),
'aufirst' : (str, ''),
'auinit' : (str, ''),
'auinit1' : (str, ''),
'auinitm' : (str, ''),
'issn' : (str, ''),
'eissn' : (str, ''),
'coden' : (str, ''),
'isbn' : (str, ''),
'sici' : (str, ''),
'bici' : (str, ''),
'title' : (str, ''),
'stitle' : (str, ''),
'atitle' : (str, ''),
'volume' : (str, ''),
'part' : (str, ''),
'issue' : (str, ''),
'spage' : (str, ''),
'epage' : (str, ''),
'pages' : (str, ''),
'artnum' : (str, ''),
'date' : (str, ''),
'ssn' : (str, ''),
'quarter' : (str, ''),
'url_ver' : (str, ''),
'ctx_ver' : (str, ''),
'rft_val_fmt' : (str, ''),
'rft_id' : (list, []),
'rft.atitle' : (str, ''),
'rft.title' : (str, ''),
'rft.jtitle' : (str, ''),
'rft.stitle' : (str, ''),
'rft.date' : (str, ''),
'rft.volume' : (str, ''),
'rft.issue' : (str, ''),
'rft.spage' : (str, ''),
'rft.epage' : (str, ''),
'rft.pages' : (str, ''),
'rft.artnumber' : (str, ''),
'rft.issn' : (str, ''),
'rft.eissn' : (str, ''),
'rft.aulast' : (str, ''),
'rft.aufirst' : (str, ''),
'rft.auinit' : (str, ''),
'rft.auinit1' : (str, ''),
'rft.auinitm' : (str, ''),
'rft.ausuffix' : (str, ''),
'rft.au' : (list, []),
'rft.aucorp' : (str, ''),
'rft.isbn' : (str, ''),
'rft.coden' : (str, ''),
'rft.sici' : (str, ''),
'rft.genre' : (str, 'unknown'),
'rft.chron' : (str, ''),
'rft.ssn' : (str, ''),
'rft.quarter' : (int, ''),
'rft.part' : (str, ''),
'rft.btitle' : (str, ''),
'rft.isbn' : (str, ''),
'rft.atitle' : (str, ''),
'rft.place' : (str, ''),
'rft.pub' : (str, ''),
'rft.edition' : (str, ''),
'rft.tpages' : (str, ''),
'rft.series' : (str, ''),
}
tmpl_opensearch_rss_url_syntax = "%(CFG_BASE_URL)s/rss?p={searchTerms}&amp;jrec={startIndex}&amp;rg={count}&amp;ln={language}" % {'CFG_BASE_URL': CFG_BASE_URL}
tmpl_opensearch_html_url_syntax = "%(CFG_BASE_URL)s/search?p={searchTerms}&amp;jrec={startIndex}&amp;rg={count}&amp;ln={language}" % {'CFG_BASE_URL': CFG_BASE_URL}
def tmpl_openurl2invenio(self, openurl_data):
""" Return an Invenio url corresponding to a search with the data
included in the openurl form map.
"""
def isbn_to_isbn13_isbn10(isbn):
isbn = isbn.replace(' ', '').replace('-', '')
if len(isbn) == 10 and isbn.isdigit():
## We already have isbn10
return ('', isbn)
if len(isbn) != 13 and isbn.isdigit():
return ('', '')
isbn13, isbn10 = isbn, isbn[3:-1]
checksum = 0
weight = 10
for char in isbn10:
checksum += int(char) * weight
weight -= 1
checksum = 11 - (checksum % 11)
if checksum == 10:
isbn10 += 'X'
if checksum == 11:
isbn10 += '0'
else:
isbn10 += str(checksum)
return (isbn13, isbn10)
from invenio.legacy.search_engine import perform_request_search
doi = ''
pmid = ''
bibcode = ''
oai = ''
issn = ''
isbn = ''
for elem in openurl_data['id']:
if elem.startswith('doi:'):
doi = elem[len('doi:'):]
elif elem.startswith('pmid:'):
pmid = elem[len('pmid:'):]
elif elem.startswith('bibcode:'):
bibcode = elem[len('bibcode:'):]
elif elem.startswith('oai:'):
oai = elem[len('oai:'):]
for elem in openurl_data['rft_id']:
if elem.startswith('info:doi/'):
doi = elem[len('info:doi/'):]
elif elem.startswith('info:pmid/'):
pmid = elem[len('info:pmid/'):]
elif elem.startswith('info:bibcode/'):
bibcode = elem[len('info:bibcode/'):]
elif elem.startswith('info:oai/'):
oai = elem[len('info:oai/')]
elif elem.startswith('urn:ISBN:'):
isbn = elem[len('urn:ISBN:'):]
elif elem.startswith('urn:ISSN:'):
issn = elem[len('urn:ISSN:'):]
## Building author query
aulast = openurl_data['rft.aulast'] or openurl_data['aulast']
aufirst = openurl_data['rft.aufirst'] or openurl_data['aufirst']
auinit = openurl_data['rft.auinit'] or \
openurl_data['auinit'] or \
openurl_data['rft.auinit1'] + ' ' + openurl_data['rft.auinitm'] or \
openurl_data['auinit1'] + ' ' + openurl_data['auinitm'] or aufirst[:1]
auinit = auinit.upper()
if aulast and aufirst:
author_query = 'author:"%s, %s" or author:"%s, %s"' % (aulast, aufirst, aulast, auinit)
elif aulast and auinit:
author_query = 'author:"%s, %s"' % (aulast, auinit)
else:
author_query = ''
## Building title query
title = openurl_data['rft.atitle'] or \
openurl_data['atitle'] or \
openurl_data['rft.btitle'] or \
openurl_data['rft.title'] or \
openurl_data['title']
if title:
title_query = 'title:"%s"' % title
title_query_cleaned = 'title:"%s"' % _RE_SPACES.sub(' ', _RE_PUNCTUATION.sub(' ', title))
else:
title_query = ''
## Building journal query
jtitle = openurl_data['rft.stitle'] or \
openurl_data['stitle'] or \
openurl_data['rft.jtitle'] or \
openurl_data['title']
if jtitle:
journal_query = 'journal:"%s"' % jtitle
else:
journal_query = ''
## Building isbn query
isbn = isbn or openurl_data['rft.isbn'] or \
openurl_data['isbn']
isbn13, isbn10 = isbn_to_isbn13_isbn10(isbn)
if isbn13:
isbn_query = 'isbn:"%s" or isbn:"%s"' % (isbn13, isbn10)
elif isbn10:
isbn_query = 'isbn:"%s"' % isbn10
else:
isbn_query = ''
## Building issn query
issn = issn or openurl_data['rft.eissn'] or \
openurl_data['eissn'] or \
openurl_data['rft.issn'] or \
openurl_data['issn']
if issn:
issn_query = 'issn:"%s"' % issn
else:
issn_query = ''
## Building coden query
coden = openurl_data['rft.coden'] or openurl_data['coden']
if coden:
coden_query = 'coden:"%s"' % coden
else:
coden_query = ''
## Building doi query
if False: #doi: #FIXME Temporaly disabled until doi field is properly setup
doi_query = 'doi:"%s"' % doi
else:
doi_query = ''
## Trying possible searches
if doi_query:
if perform_request_search(p=doi_query):
return '%s/search?%s' % (CFG_BASE_URL, urlencode({
'p' : doi_query,
'sc' : CFG_WEBSEARCH_SPLIT_BY_COLLECTION,
'of' : 'hd'}))
if isbn_query:
if perform_request_search(p=isbn_query):
return '%s/search?%s' % (CFG_BASE_URL, urlencode({
'p' : isbn_query,
'sc' : CFG_WEBSEARCH_SPLIT_BY_COLLECTION,
'of' : 'hd'}))
if coden_query:
if perform_request_search(p=coden_query):
return '%s/search?%s' % (CFG_BASE_URL, urlencode({
'p' : coden_query,
'sc' : CFG_WEBSEARCH_SPLIT_BY_COLLECTION,
'of' : 'hd'}))
if author_query and title_query:
if perform_request_search(p='%s and %s' % (title_query, author_query)):
return '%s/search?%s' % (CFG_BASE_URL, urlencode({
'p' : '%s and %s' % (title_query, author_query),
'sc' : CFG_WEBSEARCH_SPLIT_BY_COLLECTION,
'of' : 'hd'}))
if title_query:
result = len(perform_request_search(p=title_query))
if result == 1:
return '%s/search?%s' % (CFG_BASE_URL, urlencode({
'p' : title_query,
'sc' : CFG_WEBSEARCH_SPLIT_BY_COLLECTION,
'of' : 'hd'}))
elif result > 1:
return '%s/search?%s' % (CFG_BASE_URL, urlencode({
'p' : title_query,
'sc' : CFG_WEBSEARCH_SPLIT_BY_COLLECTION,
'of' : 'hb'}))
## Nothing worked, let's return a search that the user can improve
if author_query and title_query:
return '%s/search%s' % (CFG_BASE_URL, make_canonical_urlargd({
'p' : '%s and %s' % (title_query_cleaned, author_query),
'sc' : CFG_WEBSEARCH_SPLIT_BY_COLLECTION,
'of' : 'hb'}, {}))
elif title_query:
return '%s/search%s' % (CFG_BASE_URL, make_canonical_urlargd({
'p' : title_query_cleaned,
'sc' : CFG_WEBSEARCH_SPLIT_BY_COLLECTION,
'of' : 'hb'}, {}))
else:
## Mmh. Too few information provided.
return '%s/search%s' % (CFG_BASE_URL, make_canonical_urlargd({
'p' : 'recid:-1',
'sc' : CFG_WEBSEARCH_SPLIT_BY_COLLECTION,
'of' : 'hb'}, {}))
def tmpl_opensearch_description(self, ln):
""" Returns the OpenSearch description file of this site.
"""
_ = gettext_set_language(ln)
return """<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/"
xmlns:moz="http://www.mozilla.org/2006/browser/search/">
<ShortName>%(short_name)s</ShortName>
<LongName>%(long_name)s</LongName>
<Description>%(description)s</Description>
<InputEncoding>UTF-8</InputEncoding>
<OutputEncoding>UTF-8</OutputEncoding>
<Language>*</Language>
<Contact>%(CFG_SITE_ADMIN_EMAIL)s</Contact>
<Query role="example" searchTerms="a" />
<Developer>Powered by Invenio</Developer>
<Url type="text/html" indexOffset="1" rel="results" template="%(html_search_syntax)s" />
<Url type="application/rss+xml" indexOffset="1" rel="results" template="%(rss_search_syntax)s" />
<Url type="application/opensearchdescription+xml" rel="self" template="%(CFG_BASE_URL)s/opensearchdescription" />
<moz:SearchForm>%(CFG_BASE_URL)s</moz:SearchForm>
</OpenSearchDescription>""" % \
{'CFG_BASE_URL': CFG_BASE_URL,
'short_name': CFG_SITE_NAME_INTL.get(ln, CFG_SITE_NAME)[:16],
'long_name': CFG_SITE_NAME_INTL.get(ln, CFG_SITE_NAME),
'description': _("Search on %(x_CFG_SITE_NAME_INTL)s",
x_CFG_SITE_NAME_INTL=CFG_SITE_NAME_INTL.get(ln, CFG_SITE_NAME))[:1024],
'CFG_SITE_ADMIN_EMAIL': CFG_SITE_ADMIN_EMAIL,
'rss_search_syntax': self.tmpl_opensearch_rss_url_syntax,
'html_search_syntax': self.tmpl_opensearch_html_url_syntax
}
def build_search_url(self, known_parameters={}, **kargs):
""" Helper for generating a canonical search
url. 'known_parameters' is the list of query parameters you
inherit from your current query. You can then pass keyword
arguments to modify this query.
build_search_url(known_parameters, of="xm")
The generated URL is absolute.
"""
parameters = {}
parameters.update(known_parameters)
parameters.update(kargs)
# Now, we only have the arguments which have _not_ their default value
parameters = drop_default_urlargd(parameters, self.search_results_default_urlargd)
# Treat `as' argument specially:
if 'aas' in parameters:
parameters['as'] = parameters['aas']
del parameters['aas']
# Asking for a recid? Return a /CFG_SITE_RECORD/<recid> URL
if 'recid' in parameters:
target = "%s/%s/%s" % (CFG_BASE_URL, CFG_SITE_RECORD, parameters['recid'])
del parameters['recid']
target += make_canonical_urlargd(parameters, self.search_results_default_urlargd)
return target
return "%s/search%s" % (CFG_BASE_URL, make_canonical_urlargd(parameters, self.search_results_default_urlargd))
def build_search_interface_url(self, known_parameters={}, **kargs):
""" Helper for generating a canonical search interface URL."""
parameters = {}
parameters.update(known_parameters)
parameters.update(kargs)
c = parameters['c']
del parameters['c']
# Now, we only have the arguments which have _not_ their default value
parameters = drop_default_urlargd(parameters, self.search_results_default_urlargd)
# Treat `as' argument specially:
if 'aas' in parameters:
parameters['as'] = parameters['aas']
del parameters['aas']
if c and c != CFG_SITE_NAME:
base = CFG_BASE_URL + '/collection/' + quote(c)
else:
base = CFG_BASE_URL
return create_url(base, parameters)
def build_rss_url(self, known_parameters, **kargs):
"""Helper for generating a canonical RSS URL"""
parameters = {}
parameters.update(known_parameters)
parameters.update(kargs)
# Keep only interesting parameters
argd = wash_urlargd(parameters, self.rss_default_urlargd)
if argd:
# Handle 'c' differently since it is a list
c = argd.get('c', [])
del argd['c']
# Create query, and drop empty params
args = make_canonical_urlargd(argd, self.rss_default_urlargd)
if c != []:
# Add collections
c = [quote(coll) for coll in c]
if args == '':
args += '?'
else:
args += '&amp;'
args += 'c=' + '&amp;c='.join(c)
return CFG_BASE_URL + '/rss' + args
def tmpl_record_page_header_content(self, req, recid, ln):
"""
Provide extra information in the header of /CFG_SITE_RECORD pages
Return (title, description, keywords), not escaped for HTML
"""
_ = gettext_set_language(ln)
title = get_fieldvalues(recid, "245__a") or \
get_fieldvalues(recid, "111__a")
if title:
title = title[0]
else:
title = _("Record") + ' #%d' % recid
keywords = ', '.join(get_fieldvalues(recid, "6531_a"))
description = ' '.join(get_fieldvalues(recid, "520__a"))
description += "\n"
description += '; '.join(get_fieldvalues(recid, "100__a") + get_fieldvalues(recid, "700__a"))
return (title, description, keywords)
def tmpl_exact_author_browse_help_link(self, p, p1, p2, p3, f, f1, f2, f3, rm, cc, ln, jrec, rg, aas, action, link_name):
"""
Creates the 'exact author' help link for browsing.
"""
_ = gettext_set_language(ln)
url = create_html_link(self.build_search_url(p=p,
p1=p1,
p2=p2,
p3=p3,
f=f,
f1=f1,
f2=f2,
f3=f3,
rm=rm,
cc=cc,
ln=ln,
jrec=jrec,
rg=rg,
aas=aas,
action=action),
{}, _(link_name), {'class': 'nearestterms'})
return "Did you mean to browse in %s index?" % url
def tmpl_navtrail_links(self, aas, ln, dads):
"""
Creates the navigation bar at top of each search page (*Home > Root collection > subcollection > ...*)
Parameters:
- 'aas' *int* - Should we display an advanced search box?
- 'ln' *string* - The language to display
- 'separator' *string* - The separator between two consecutive collections
- 'dads' *list* - A list of parent links, eachone being a dictionary of ('name', 'longname')
"""
out = []
for url, name in dads:
args = {'c': url, 'as': aas, 'ln': ln}
out.append(create_html_link(self.build_search_interface_url(**args), {}, cgi.escape(name), {'class': 'navtrail'}))
return ' &gt; '.join(out)
def tmpl_webcoll_body(self, ln, collection, te_portalbox,
searchfor, np_portalbox, narrowsearch,
focuson, instantbrowse, ne_portalbox, show_body=True):
""" Creates the body of the main search page.
Parameters:
- 'ln' *string* - language of the page being generated
- 'collection' - collection id of the page being generated
- 'te_portalbox' *string* - The HTML code for the portalbox on top of search
- 'searchfor' *string* - The HTML code for the search for box
- 'np_portalbox' *string* - The HTML code for the portalbox on bottom of search
- 'narrowsearch' *string* - The HTML code for the search categories (left bottom of page)
- 'focuson' *string* - The HTML code for the "focuson" categories (right bottom of page)
- 'ne_portalbox' *string* - The HTML code for the bottom of the page
"""
if not narrowsearch:
narrowsearch = instantbrowse
body = '''
<form name="search" action="%(siteurl)s/search" method="get">
%(searchfor)s
%(np_portalbox)s
''' % {
'siteurl': CFG_BASE_URL,
'searchfor': searchfor,
'np_portalbox': np_portalbox
}
if show_body:
body += '''
<table cellspacing="0" cellpadding="0" border="0" class="narrowandfocusonsearchbox">
<tr>
<td valign="top">%(narrowsearch)s</td>
''' % { 'narrowsearch' : narrowsearch }
if focuson:
body += """<td valign="top">""" + focuson + """</td>"""
body += """</tr></table>"""
elif focuson:
body += focuson
body += """%(ne_portalbox)s
</form>""" % {'ne_portalbox' : ne_portalbox}
return body
def tmpl_portalbox(self, title, body):
"""Creates portalboxes based on the parameters
Parameters:
- 'title' *string* - The title of the box
- 'body' *string* - The HTML code for the body of the box
"""
out = """<div class="portalbox">
<div class="portalboxheader">%(title)s</div>
<div class="portalboxbody">%(body)s</div>
</div>""" % {'title' : cgi.escape(title), 'body' : body}
return out
def tmpl_searchfor_light(self, ln, collection_id, collection_name, record_count,
example_search_queries): # EXPERIMENTAL
"""Produces light *Search for* box for the current collection.
Parameters:
- 'ln' *string* - *str* The language to display
- 'collection_id' - *str* The collection id
- 'collection_name' - *str* The collection name in current language
- 'example_search_queries' - *list* List of search queries given as example for this collection
"""
# load the right message language
_ = gettext_set_language(ln)
out = '''
<!--create_searchfor_light()-->
'''
argd = drop_default_urlargd({'ln': ln, 'sc': CFG_WEBSEARCH_SPLIT_BY_COLLECTION},
self.search_results_default_urlargd)
# Only add non-default hidden values
for field, value in argd.items():
out += self.tmpl_input_hidden(field, value)
header = _("Search %(x_name)s records for:",
x_name=self.tmpl_nbrecs_info(record_count, "", ""))
asearchurl = self.build_search_interface_url(c=collection_id,
aas=max(CFG_WEBSEARCH_ENABLED_SEARCH_INTERFACES),
ln=ln)
# Build example of queries for this collection
example_search_queries_links = [create_html_link(self.build_search_url(p=example_query,
ln=ln,
aas= -1,
c=collection_id),
{},
cgi.escape(example_query),
{'class': 'examplequery'}) \
for example_query in example_search_queries]
example_query_html = ''
if len(example_search_queries) > 0:
example_query_link = example_search_queries_links[0]
# offers more examples if possible
more = ''
if len(example_search_queries_links) > 1:
more = '''
<script type="text/javascript">
function toggle_more_example_queries_visibility(){
var more = document.getElementById('more_example_queries');
var link = document.getElementById('link_example_queries');
var sep = document.getElementById('more_example_sep');
if (more.style.display=='none'){
more.style.display = '';
link.innerHTML = "%(show_less)s"
link.style.color = "rgb(204,0,0)";
sep.style.display = 'none';
} else {
more.style.display = 'none';
link.innerHTML = "%(show_more)s"
link.style.color = "rgb(0,0,204)";
sep.style.display = '';
}
return false;
}
</script>
<span id="more_example_queries" style="display:none;text-align:right"><br/>%(more_example_queries)s<br/></span>
<a id="link_example_queries" href="#" onclick="toggle_more_example_queries_visibility()" style="display:none"></a>
<script type="text/javascript">
var link = document.getElementById('link_example_queries');
var sep = document.getElementById('more_example_sep');
link.style.display = '';
link.innerHTML = "%(show_more)s";
sep.style.display = '';
</script>
''' % {'more_example_queries': '<br/>'.join(example_search_queries_links[1:]),
'show_less':_("less"),
'show_more':_("more")}
example_query_html += '''<p style="text-align:right;margin:0px;">
%(example)s<span id="more_example_sep" style="display:none;">&nbsp;&nbsp;::&nbsp;</span>%(more)s
</p>
''' % {'example': _("Example: %(x_sample_search_query)s") % \
{'x_sample_search_query': example_query_link},
'more': more}
# display options to search in current collection or everywhere
search_in = ''
if collection_name != CFG_SITE_NAME_INTL.get(ln, CFG_SITE_NAME):
search_in += '''
<input type="radio" name="cc" value="%(collection_id)s" id="searchCollection" checked="checked"/>
<label for="searchCollection">%(search_in_collection_name)s</label>
<input type="radio" name="cc" value="%(root_collection_name)s" id="searchEverywhere" />
<label for="searchEverywhere">%(search_everywhere)s</label>
''' % {'search_in_collection_name': _("Search in %(x_collection_name)s") % \
{'x_collection_name': collection_name},
'collection_id': collection_id,
'root_collection_name': CFG_SITE_NAME,
'search_everywhere': _("Search everywhere")}
# print commentary start:
out += '''
<table class="searchbox lightsearch">
<tbody>
<tr valign="baseline">
<td class="searchboxbody" align="right"><input type="text" name="p" size="%(sizepattern)d" value="" class="lightsearchfield"/><br/>
<small><small>%(example_query_html)s</small></small>
</td>
<td class="searchboxbody" align="left">
<input class="formbutton" type="submit" name="action_search" value="%(msg_search)s" />
</td>
<td class="searchboxbody" align="left" rowspan="2" valign="top">
<small><small>
<a href="%(siteurl)s/help/search-tips%(langlink)s">%(msg_search_tips)s</a><br/>
%(asearch)s
</small></small>
</td>
</tr></table>
<!--<tr valign="baseline">
<td class="searchboxbody" colspan="2" align="left">
<small>
--><small>%(search_in)s</small><!--
</small>
</td>
</tr>
</tbody>
</table>-->
<!--/create_searchfor_light()-->
''' % {'ln' : ln,
'sizepattern' : CFG_WEBSEARCH_LIGHTSEARCH_PATTERN_BOX_WIDTH,
'langlink': '?ln=' + ln,
'siteurl' : CFG_BASE_URL,
'asearch' : create_html_link(asearchurl, {}, _('Advanced Search')),
'header' : header,
'msg_search' : _('Search'),
'msg_browse' : _('Browse'),
'msg_search_tips' : _('Search Tips'),
'search_in': search_in,
'example_query_html': example_query_html}
return out
def tmpl_searchfor_simple(self, ln, collection_id, collection_name, record_count, middle_option):
"""Produces simple *Search for* box for the current collection.
Parameters:
- 'ln' *string* - *str* The language to display
- 'collection_id' - *str* The collection id
- 'collection_name' - *str* The collection name in current language
- 'record_count' - *str* Number of records in this collection
- 'middle_option' *string* - HTML code for the options (any field, specific fields ...)
"""
# load the right message language
_ = gettext_set_language(ln)
out = '''
<!--create_searchfor_simple()-->
'''
argd = drop_default_urlargd({'ln': ln, 'cc': collection_id, 'sc': CFG_WEBSEARCH_SPLIT_BY_COLLECTION},
self.search_results_default_urlargd)
# Only add non-default hidden values
for field, value in argd.items():
out += self.tmpl_input_hidden(field, value)
header = _("Search %(x_name)s records for:",
x_name=self.tmpl_nbrecs_info(record_count, "", ""))
asearchurl = self.build_search_interface_url(c=collection_id,
aas=max(CFG_WEBSEARCH_ENABLED_SEARCH_INTERFACES),
ln=ln)
# print commentary start:
out += '''
<table class="searchbox simplesearch">
<thead>
<tr align="left">
<th colspan="3" class="searchboxheader">%(header)s</th>
</tr>
</thead>
<tbody>
<tr valign="baseline">
<td class="searchboxbody" align="left"><input type="text" name="p" size="%(sizepattern)d" value="" class="simplesearchfield"/></td>
<td class="searchboxbody" align="left">%(middle_option)s</td>
<td class="searchboxbody" align="left">
<input class="formbutton" type="submit" name="action_search" value="%(msg_search)s" />
<input class="formbutton" type="submit" name="action_browse" value="%(msg_browse)s" /></td>
</tr>
<tr valign="baseline">
<td class="searchboxbody" colspan="3" align="right">
<small>
<a href="%(siteurl)s/help/search-tips%(langlink)s">%(msg_search_tips)s</a> ::
%(asearch)s
</small>
</td>
</tr>
</tbody>
</table>
<!--/create_searchfor_simple()-->
''' % {'ln' : ln,
'sizepattern' : CFG_WEBSEARCH_SIMPLESEARCH_PATTERN_BOX_WIDTH,
'langlink': '?ln=' + ln,
'siteurl' : CFG_BASE_URL,
'asearch' : create_html_link(asearchurl, {}, _('Advanced Search')),
'header' : header,
'middle_option' : middle_option,
'msg_search' : _('Search'),
'msg_browse' : _('Browse'),
'msg_search_tips' : _('Search Tips')}
return out
def tmpl_searchfor_advanced(self,
ln, # current language
collection_id,
collection_name,
record_count,
middle_option_1, middle_option_2, middle_option_3,
searchoptions,
sortoptions,
rankoptions,
displayoptions,
formatoptions
):
"""
Produces advanced *Search for* box for the current collection.
Parameters:
- 'ln' *string* - The language to display
- 'middle_option_1' *string* - HTML code for the first row of options (any field, specific fields ...)
- 'middle_option_2' *string* - HTML code for the second row of options (any field, specific fields ...)
- 'middle_option_3' *string* - HTML code for the third row of options (any field, specific fields ...)
- 'searchoptions' *string* - HTML code for the search options
- 'sortoptions' *string* - HTML code for the sort options
- 'rankoptions' *string* - HTML code for the rank options
- 'displayoptions' *string* - HTML code for the display options
- 'formatoptions' *string* - HTML code for the format options
"""
# load the right message language
_ = gettext_set_language(ln)
out = '''
<!--create_searchfor_advanced()-->
'''
argd = drop_default_urlargd({'ln': ln, 'aas': 1, 'cc': collection_id, 'sc': CFG_WEBSEARCH_SPLIT_BY_COLLECTION},
self.search_results_default_urlargd)
# Only add non-default hidden values
for field, value in argd.items():
out += self.tmpl_input_hidden(field, value)
header = _("Search %(x_rec)s records for",
x_rec=self.tmpl_nbrecs_info(record_count, "", ""))
header += ':'
ssearchurl = self.build_search_interface_url(c=collection_id, aas=min(CFG_WEBSEARCH_ENABLED_SEARCH_INTERFACES), ln=ln)
out += '''
<table class="searchbox advancedsearch">
<thead>
<tr>
<th class="searchboxheader" colspan="3">%(header)s</th>
</tr>
</thead>
<tbody>
<tr valign="bottom">
<td class="searchboxbody" style="white-space: nowrap;">
%(matchbox_m1)s<input type="text" name="p1" size="%(sizepattern)d" value="" class="advancedsearchfield"/>
</td>
<td class="searchboxbody" style="white-space: nowrap;">%(middle_option_1)s</td>
<td class="searchboxbody">%(andornot_op1)s</td>
</tr>
<tr valign="bottom">
<td class="searchboxbody" style="white-space: nowrap;">
%(matchbox_m2)s<input type="text" name="p2" size="%(sizepattern)d" value="" class="advancedsearchfield"/>
</td>
<td class="searchboxbody">%(middle_option_2)s</td>
<td class="searchboxbody">%(andornot_op2)s</td>
</tr>
<tr valign="bottom">
<td class="searchboxbody" style="white-space: nowrap;">
%(matchbox_m3)s<input type="text" name="p3" size="%(sizepattern)d" value="" class="advancedsearchfield"/>
</td>
<td class="searchboxbody">%(middle_option_3)s</td>
<td class="searchboxbody" style="white-space: nowrap;">
<input class="formbutton" type="submit" name="action_search" value="%(msg_search)s" />
<input class="formbutton" type="submit" name="action_browse" value="%(msg_browse)s" /></td>
</tr>
<tr valign="bottom">
<td colspan="3" class="searchboxbody" align="right">
<small>
<a href="%(siteurl)s/help/search-tips%(langlink)s">%(msg_search_tips)s</a> ::
%(ssearch)s
</small>
</td>
</tr>
</tbody>
</table>
<!-- @todo - more imports -->
''' % {'ln' : ln,
'sizepattern' : CFG_WEBSEARCH_ADVANCEDSEARCH_PATTERN_BOX_WIDTH,
'langlink': '?ln=' + ln,
'siteurl' : CFG_BASE_URL,
'ssearch' : create_html_link(ssearchurl, {}, _("Simple Search")),
'header' : header,
'matchbox_m1' : self.tmpl_matchtype_box('m1', ln=ln),
'middle_option_1' : middle_option_1,
'andornot_op1' : self.tmpl_andornot_box('op1', ln=ln),
'matchbox_m2' : self.tmpl_matchtype_box('m2', ln=ln),
'middle_option_2' : middle_option_2,
'andornot_op2' : self.tmpl_andornot_box('op2', ln=ln),
'matchbox_m3' : self.tmpl_matchtype_box('m3', ln=ln),
'middle_option_3' : middle_option_3,
'msg_search' : _("Search"),
'msg_browse' : _("Browse"),
'msg_search_tips' : _("Search Tips")}
if (searchoptions):
out += """<table class="searchbox">
<thead>
<tr>
<th class="searchboxheader">
%(searchheader)s
</th>
</tr>
</thead>
<tbody>
<tr valign="bottom">
<td class="searchboxbody">%(searchoptions)s</td>
</tr>
</tbody>
</table>""" % {
'searchheader' : _("Search options:"),
'searchoptions' : searchoptions
}
out += """<table class="searchbox">
<thead>
<tr>
<th class="searchboxheader">
%(added)s
</th>
<th class="searchboxheader">
%(until)s
</th>
</tr>
</thead>
<tbody>
<tr valign="bottom">
<td class="searchboxbody">%(added_or_modified)s %(date_added)s</td>
<td class="searchboxbody">%(date_until)s</td>
</tr>
</tbody>
</table>
<table class="searchbox">
<thead>
<tr>
<th class="searchboxheader">
%(msg_sort)s
</th>
<th class="searchboxheader">
%(msg_display)s
</th>
<th class="searchboxheader">
%(msg_format)s
</th>
</tr>
</thead>
<tbody>
<tr valign="bottom">
<td class="searchboxbody">%(sortoptions)s %(rankoptions)s</td>
<td class="searchboxbody">%(displayoptions)s</td>
<td class="searchboxbody">%(formatoptions)s</td>
</tr>
</tbody>
</table>
<!--/create_searchfor_advanced()-->
""" % {
'added' : _("Added/modified since:"),
'until' : _("until:"),
'added_or_modified': self.tmpl_inputdatetype(ln=ln),
'date_added' : self.tmpl_inputdate("d1", ln=ln),
'date_until' : self.tmpl_inputdate("d2", ln=ln),
'msg_sort' : _("Sort by:"),
'msg_display' : _("Display results:"),
'msg_format' : _("Output format:"),
'sortoptions' : sortoptions,
'rankoptions' : rankoptions,
'displayoptions' : displayoptions,
'formatoptions' : formatoptions
}
return out
def tmpl_matchtype_box(self, name='m', value='', ln='en'):
"""Returns HTML code for the 'match type' selection box.
Parameters:
- 'name' *string* - The name of the produced select
- 'value' *string* - The selected value (if any value is already selected)
- 'ln' *string* - the language to display
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<select name="%(name)s">
<option value="a"%(sela)s>%(opta)s</option>
<option value="o"%(selo)s>%(opto)s</option>
<option value="e"%(sele)s>%(opte)s</option>
<option value="p"%(selp)s>%(optp)s</option>
<option value="r"%(selr)s>%(optr)s</option>
</select>
""" % {'name' : name,
'sela' : self.tmpl_is_selected('a', value),
'opta' : _("All of the words:"),
'selo' : self.tmpl_is_selected('o', value),
'opto' : _("Any of the words:"),
'sele' : self.tmpl_is_selected('e', value),
'opte' : _("Exact phrase:"),
'selp' : self.tmpl_is_selected('p', value),
'optp' : _("Partial phrase:"),
'selr' : self.tmpl_is_selected('r', value),
'optr' : _("Regular expression:")
}
return out
def tmpl_is_selected(self, var, fld):
"""
Checks if *var* and *fld* are equal, and if yes, returns ' selected="selected"'. Useful for select boxes.
Parameters:
- 'var' *string* - First value to compare
- 'fld' *string* - Second value to compare
"""
if var == fld:
return ' selected="selected"'
else:
return ""
def tmpl_andornot_box(self, name='op', value='', ln='en'):
"""
Returns HTML code for the AND/OR/NOT selection box.
Parameters:
- 'name' *string* - The name of the produced select
- 'value' *string* - The selected value (if any value is already selected)
- 'ln' *string* - the language to display
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<select name="%(name)s">
<option value="a"%(sela)s>%(opta)s</option>
<option value="o"%(selo)s>%(opto)s</option>
<option value="n"%(seln)s>%(optn)s</option>
</select>
""" % {'name' : name,
'sela' : self.tmpl_is_selected('a', value), 'opta' : _("AND"),
'selo' : self.tmpl_is_selected('o', value), 'opto' : _("OR"),
'seln' : self.tmpl_is_selected('n', value), 'optn' : _("AND NOT")
}
return out
def tmpl_inputdate(self, name, ln, sy=0, sm=0, sd=0):
"""
Produces *From Date*, *Until Date* kind of selection box. Suitable for search options.
Parameters:
- 'name' *string* - The base name of the produced selects
- 'ln' *string* - the language to display
"""
# load the right message language
_ = gettext_set_language(ln)
box = """
<select name="%(name)sd">
<option value=""%(sel)s>%(any)s</option>
""" % {
'name' : name,
'any' : _("any day"),
'sel' : self.tmpl_is_selected(sd, 0)
}
for day in range(1, 32):
box += """<option value="%02d"%s>%02d</option>""" % (day, self.tmpl_is_selected(sd, day), day)
box += """</select>"""
# month
box += """
<select name="%(name)sm">
<option value=""%(sel)s>%(any)s</option>
""" % {
'name' : name,
'any' : _("any month"),
'sel' : self.tmpl_is_selected(sm, 0)
}
# trailing space in May distinguishes short/long form of the month name
for mm, month in [(1, _("January")), (2, _("February")), (3, _("March")), (4, _("April")), \
(5, _("May ")), (6, _("June")), (7, _("July")), (8, _("August")), \
(9, _("September")), (10, _("October")), (11, _("November")), (12, _("December"))]:
box += """<option value="%02d"%s>%s</option>""" % (mm, self.tmpl_is_selected(sm, mm), month.strip())
box += """</select>"""
# year
box += """
<select name="%(name)sy">
<option value=""%(sel)s>%(any)s</option>
""" % {
'name' : name,
'any' : _("any year"),
'sel' : self.tmpl_is_selected(sy, 0)
}
this_year = int(time.strftime("%Y", time.localtime()))
for year in range(this_year - 20, this_year + 1):
box += """<option value="%d"%s>%d</option>""" % (year, self.tmpl_is_selected(sy, year), year)
box += """</select>"""
return box
def tmpl_inputdatetype(self, dt='', ln=CFG_SITE_LANG):
"""
Produces input date type selection box to choose
added-or-modified date search option.
Parameters:
- 'dt' *string - date type (c=created, m=modified)
- 'ln' *string* - the language to display
"""
# load the right message language
_ = gettext_set_language(ln)
box = """<select name="dt">
<option value="">%(added)s </option>
<option value="m"%(sel)s>%(modified)s </option>
</select>
""" % { 'added': _("Added since:"),
'modified': _("Modified since:"),
'sel': self.tmpl_is_selected(dt, 'm'),
}
return box
def tmpl_narrowsearch(self, aas, ln, type, father,
has_grandchildren, sons, display_grandsons,
grandsons):
"""
Creates list of collection descendants of type *type* under title *title*.
If aas==1, then links to Advanced Search interfaces; otherwise Simple Search.
Suitable for 'Narrow search' and 'Focus on' boxes.
Parameters:
- 'aas' *bool* - Should we display an advanced search box?
- 'ln' *string* - The language to display
- 'type' *string* - The type of the produced box (virtual collections or normal collections)
- 'father' *collection* - The current collection
- 'has_grandchildren' *bool* - If the current collection has grand children
- 'sons' *list* - The list of the sub-collections (first level)
- 'display_grandsons' *bool* - If the grand children collections should be displayed (2 level deep display)
- 'grandsons' *list* - The list of sub-collections (second level)
"""
# load the right message language
_ = gettext_set_language(ln)
title = father.get_collectionbox_name(ln, type)
if has_grandchildren:
- style_prolog = "<strong>"
- style_epilog = "</strong>"
+ style_class = 'collection-first-level collection-father-has-grandchildren'
else:
- style_prolog = ""
- style_epilog = ""
+ style_class = 'collection-first-level'
out = """<table class="%(narrowsearchbox)s">
<thead>
<tr>
<th colspan="2" align="left" class="%(narrowsearchbox)sheader">
%(title)s
</th>
</tr>
</thead>
<tbody>""" % {'title' : title,
'narrowsearchbox': {'r': 'narrowsearchbox',
'v': 'focusonsearchbox'}[type]}
# iterate through sons:
i = 0
for son in sons:
out += """<tr><td class="%(narrowsearchbox)sbody" valign="top">""" % \
{ 'narrowsearchbox': {'r': 'narrowsearchbox',
'v': 'focusonsearchbox'}[type]}
if type == 'r':
if son.restricted_p() and son.restricted_p() != father.restricted_p():
out += """<input type="checkbox" name="c" value="%(name)s" /></td>""" % {'name' : cgi.escape(son.name) }
# hosted collections are checked by default only when configured so
elif str(son.dbquery).startswith("hostedcollection:"):
external_collection_engine = get_external_collection_engine(str(son.name))
if external_collection_engine and external_collection_engine.selected_by_default:
out += """<input type="checkbox" name="c" value="%(name)s" checked="checked" /></td>""" % {'name' : cgi.escape(son.name) }
elif external_collection_engine and not external_collection_engine.selected_by_default:
out += """<input type="checkbox" name="c" value="%(name)s" /></td>""" % {'name' : cgi.escape(son.name) }
else:
# strangely, the external collection engine was never found. In that case,
# why was the hosted collection here in the first place?
out += """<input type="checkbox" name="c" value="%(name)s" /></td>""" % {'name' : cgi.escape(son.name) }
else:
out += """<input type="checkbox" name="c" value="%(name)s" checked="checked" /></td>""" % {'name' : cgi.escape(son.name) }
else:
out += '</td>'
- out += """<td valign="top">%(link)s%(recs)s """ % {
+ out += """<td valign="top"><span class="%(style_class)s">%(link)s%(recs)s</span> """ % {
'link': create_html_link(self.build_search_interface_url(c=son.name, ln=ln, aas=aas),
- {}, style_prolog + cgi.escape(son.get_name(ln)) + style_epilog),
- 'recs' : self.tmpl_nbrecs_info(son.nbrecs, ln=ln)}
+ {}, cgi.escape(son.get_name(ln))),
+ 'recs' : self.tmpl_nbrecs_info(son.nbrecs, ln=ln),
+ 'style_class': style_class}
# the following prints the "external collection" arrow just after the name and
# number of records of the hosted collection
# 1) we might want to make the arrow work as an anchor to the hosted collection as well.
# That would probably require a new separate function under invenio.utils.url
# 2) we might want to place the arrow between the name and the number of records of the hosted collection
# That would require to edit/separate the above out += ...
if type == 'r':
if str(son.dbquery).startswith("hostedcollection:"):
out += """<img src="%(siteurl)s/img/external-icon-light-8x8.gif" border="0" alt="%(name)s"/>""" % \
{ 'siteurl' : CFG_BASE_URL, 'name' : cgi.escape(son.name), }
if son.restricted_p():
out += """ <small class="warning">[%(msg)s]</small> """ % { 'msg' : _("restricted") }
if display_grandsons and len(grandsons[i]):
# iterate trough grandsons:
- out += """<br />"""
+ out += """<ul class="collection-second-level">"""
for grandson in grandsons[i]:
- out += """ <small>%(link)s%(nbrec)s</small> """ % {
+ out += """ <li>%(link)s%(nbrec)s</li> """ % {
'link': create_html_link(self.build_search_interface_url(c=grandson.name, ln=ln, aas=aas),
{},
cgi.escape(grandson.get_name(ln))),
'nbrec' : self.tmpl_nbrecs_info(grandson.nbrecs, ln=ln)}
# the following prints the "external collection" arrow just after the name and
# number of records of the hosted collection
# Some relatives comments have been made just above
if type == 'r':
if str(grandson.dbquery).startswith("hostedcollection:"):
out += """<img src="%(siteurl)s/img/external-icon-light-8x8.gif" border="0" alt="%(name)s"/>""" % \
{ 'siteurl' : CFG_BASE_URL, 'name' : cgi.escape(grandson.name), }
+ out += """</ul>"""
out += """</td></tr>"""
i += 1
out += "</tbody></table>"
return out
def tmpl_searchalso(self, ln, engines_list, collection_id):
_ = gettext_set_language(ln)
box_name = _("Search also:")
html = """<table cellspacing="0" cellpadding="0" border="0">
<tr><td valign="top"><table class="searchalsosearchbox">
<thead><tr><th colspan="2" align="left" class="searchalsosearchboxheader">%(box_name)s
</th></tr></thead><tbody>
""" % locals()
for engine in engines_list:
internal_name = engine.name
name = _(internal_name)
base_url = engine.base_url
if external_collection_get_state(engine, collection_id) == 3:
checked = ' checked="checked"'
else:
checked = ''
html += """<tr><td class="searchalsosearchboxbody" valign="top">
<input type="checkbox" name="ec" id="%(id)s" value="%(internal_name)s" %(checked)s /></td>
<td valign="top" class="searchalsosearchboxbody">
<div style="white-space: nowrap"><label for="%(id)s">%(name)s</label>
<a href="%(base_url)s">
<img src="%(siteurl)s/img/external-icon-light-8x8.gif" border="0" alt="%(name)s"/></a>
</div></td></tr>""" % \
{ 'checked': checked,
'base_url': base_url,
'internal_name': internal_name,
'name': cgi.escape(name),
'id': "extSearch" + nmtoken_from_string(name),
'siteurl': CFG_BASE_URL, }
html += """</tbody></table></td></tr></table>"""
return html
def tmpl_nbrecs_info(self, number, prolog=None, epilog=None, ln=CFG_SITE_LANG):
"""
Return information on the number of records.
Parameters:
- 'number' *string* - The number of records
- 'prolog' *string* (optional) - An HTML code to prefix the number (if **None**, will be
'<small class="nbdoccoll">(')
- 'epilog' *string* (optional) - An HTML code to append to the number (if **None**, will be
')</small>')
"""
if number is None:
number = 0
if prolog is None:
prolog = '''&nbsp;<small class="nbdoccoll">('''
if epilog is None:
epilog = ''')</small>'''
return prolog + self.tmpl_nice_number(number, ln) + epilog
def tmpl_box_restricted_content(self, ln):
"""
Displays a box containing a *restricted content* message
Parameters:
- 'ln' *string* - The language to display
"""
# load the right message language
_ = gettext_set_language(ln)
return _("This collection is restricted. If you are authorized to access it, please click on the Search button.")
def tmpl_box_hosted_collection(self, ln):
"""
Displays a box containing a *hosted collection* message
Parameters:
- 'ln' *string* - The language to display
"""
# load the right message language
_ = gettext_set_language(ln)
return _("This is a hosted external collection. Please click on the Search button to see its content.")
def tmpl_box_no_records(self, ln):
"""
Displays a box containing a *no content* message
Parameters:
- 'ln' *string* - The language to display
"""
# load the right message language
_ = gettext_set_language(ln)
return _("This collection does not contain any document yet.")
def tmpl_instant_browse(self, aas, ln, recids, more_link=None, grid_layout=False, father=None):
"""
Formats a list of records (given in the recids list) from the database.
Parameters:
- 'aas' *int* - Advanced Search interface or not (0 or 1)
- 'ln' *string* - The language to display
- 'recids' *list* - the list of records from the database
- 'more_link' *string* - the "More..." link for the record. If not given, will not be displayed
- 'father' *collection* - The current collection
"""
# load the right message language
_ = gettext_set_language(ln)
body = '''<table class="latestadditionsbox">'''
if grid_layout:
body += '<tr><td><div>'
for recid in recids:
if grid_layout:
body += '''
<abbr class="unapi-id" title="%(recid)s"></abbr>
%(body)s
''' % {
'recid': recid['id'],
'body': recid['body']}
else:
body += '''
<tr>
<td class="latestadditionsboxtimebody">%(date)s</td>
<td class="latestadditionsboxrecordbody">
<abbr class="unapi-id" title="%(recid)s"></abbr>
%(body)s
</td>
</tr>''' % {
'recid': recid['id'],
'date': recid['date'],
'body': recid['body']
}
if grid_layout:
body += '''<div style="clear:both"></div>'''
body += '''</div></td></tr>'''
body += "</table>"
if more_link:
body += '<div align="right"><small>' + \
create_html_link(more_link, {}, '[&gt;&gt; %s]' % _("more")) + \
'</small></div>'
return '''
<table class="narrowsearchbox">
<thead>
<tr>
<th class="narrowsearchboxheader">%(header)s</th>
</tr>
</thead>
<tbody>
<tr>
<td class="narrowsearchboxbody">%(body)s</td>
</tr>
</tbody>
</table>''' % {'header' : father.get_collectionbox_name(ln, 'l') ,
'body' : body,
}
def tmpl_searchwithin_select(self, ln, fieldname, selected, values):
"""
Produces 'search within' selection box for the current collection.
Parameters:
- 'ln' *string* - The language to display
- 'fieldname' *string* - the name of the select box produced
- 'selected' *string* - which of the values is selected
- 'values' *list* - the list of values in the select
"""
out = '<select name="%(fieldname)s">' % {'fieldname': fieldname}
if values:
for pair in values:
out += """<option value="%(value)s"%(selected)s>%(text)s</option>""" % {
'value' : cgi.escape(pair['value']),
'selected' : self.tmpl_is_selected(pair['value'], selected),
'text' : cgi.escape(pair['text'])
}
out += """</select>"""
return out
def tmpl_select(self, fieldname, values, selected=None, css_class=''):
"""
Produces a generic select box
Parameters:
- 'css_class' *string* - optional, a css class to display this select with
- 'fieldname' *list* - the name of the select box produced
- 'selected' *string* - which of the values is selected
- 'values' *list* - the list of values in the select
"""
if css_class != '':
class_field = ' class="%s"' % css_class
else:
class_field = ''
out = '<select name="%(fieldname)s"%(class)s>' % {
'fieldname' : fieldname,
'class' : class_field
}
for pair in values:
if pair.get('selected', False) or pair['value'] == selected:
flag = ' selected="selected"'
else:
flag = ''
out += '<option value="%(value)s"%(selected)s>%(text)s</option>' % {
'value' : cgi.escape(str(pair['value'])),
'selected' : flag,
'text' : cgi.escape(pair['text'])
}
out += """</select>"""
return out
def tmpl_record_links(self, recid, ln, sf='', so='d', sp='', rm=''):
"""
Displays the *More info* and *Find similar* links for a record
Parameters:
- 'ln' *string* - The language to display
- 'recid' *string* - the id of the displayed record
"""
# load the right message language
_ = gettext_set_language(ln)
out = '''<br /><span class="moreinfo">%(detailed)s - %(similar)s</span>''' % {
'detailed': create_html_link(self.build_search_url(recid=recid, ln=ln),
{},
_("Detailed record"), {'class': "moreinfo"}),
'similar': create_html_link(self.build_search_url(p="recid:%d" % recid, rm='wrd', ln=ln),
{},
_("Similar records"),
{'class': "moreinfo"})}
if CFG_BIBRANK_SHOW_CITATION_LINKS:
num_timescited = get_cited_by_count(recid)
if num_timescited:
out += '''<span class="moreinfo"> - %s </span>''' % \
create_html_link(self.build_search_url(p='refersto:recid:%d' % recid,
sf=sf,
so=so,
sp=sp,
rm=rm,
ln=ln),
{}, _("Cited by %(x_num)i records", x_num=num_timescited), {'class': "moreinfo"})
return out
def tmpl_record_body(self, titles, authors, dates, rns, abstracts, urls_u, urls_z, ln):
"""
Displays the "HTML basic" format of a record
Parameters:
- 'authors' *list* - the authors (as strings)
- 'dates' *list* - the dates of publication
- 'rns' *list* - the quicknotes for the record
- 'abstracts' *list* - the abstracts for the record
- 'urls_u' *list* - URLs to the original versions of the record
- 'urls_z' *list* - Not used
"""
out = ""
for title in titles:
out += "<strong>%(title)s</strong> " % {
'title' : cgi.escape(title)
}
if authors:
out += " / "
for author in authors[:CFG_WEBSEARCH_AUTHOR_ET_AL_THRESHOLD]:
out += '%s ' % \
create_html_link(self.build_search_url(p=author, f='author', ln=ln),
{}, cgi.escape(author))
if len(authors) > CFG_WEBSEARCH_AUTHOR_ET_AL_THRESHOLD:
out += "<em>et al</em>"
for date in dates:
out += " %s." % cgi.escape(date)
for rn in rns:
out += """ <small class="quicknote">[%(rn)s]</small>""" % {'rn' : cgi.escape(rn)}
for abstract in abstracts:
out += "<br /><small>%(abstract)s [...]</small>" % {'abstract' : cgi.escape(abstract[:1 + string.find(abstract, '.')]) }
for idx in range(0, len(urls_u)):
out += """<br /><small class="note"><a class="note" href="%(url)s">%(name)s</a></small>""" % {
'url' : urls_u[idx],
'name' : urls_u[idx]
}
return out
def tmpl_search_in_bibwords(self, p, f, ln, nearest_box):
"""
Displays the *Words like current ones* links for a search
Parameters:
- 'p' *string* - Current search words
- 'f' *string* - the fields in which the search was done
- 'nearest_box' *string* - the HTML code for the "nearest_terms" box - most probably from a create_nearest_terms_box call
"""
# load the right message language
_ = gettext_set_language(ln)
out = '<p>'
if f:
out += _("Words nearest to %(x_word)s inside %(x_field)s in any collection are:") % {'x_word': '<em>' + cgi.escape(p) + '</em>',
'x_field': '<em>' + cgi.escape(f) + '</em>'}
else:
out += _("Words nearest to %(x_word)s in any collection are:") % {'x_word': '<em>' + cgi.escape(p) + '</em>'}
out += '<br />' + nearest_box + '</p>'
return out
def tmpl_nearest_term_box(self, p, ln, f, terminfo, intro):
"""
Displays the *Nearest search terms* box
Parameters:
- 'p' *string* - Current search words
- 'f' *string* - a collection description (if the search has been completed in a collection)
- 'ln' *string* - The language to display
- 'terminfo': tuple (term, hits, argd) for each near term
- 'intro' *string* - the intro HTML to prefix the box with
"""
out = '''<table class="nearesttermsbox" cellpadding="0" cellspacing="0" border="0">'''
for term, hits, argd in terminfo:
if hits:
hitsinfo = str(hits)
else:
hitsinfo = '-'
argd['f'] = f
argd['p'] = term
term = cgi.escape(term)
# FIXME this is hack to get correct links to nearest terms
from flask import has_request_context, request
if has_request_context() and request.values.get('of', '') != argd.get('of', ''):
if 'of' in request.values:
argd['of'] = request.values.get('of')
else:
del argd['of']
if term == p: # print search word for orientation:
nearesttermsboxbody_class = "nearesttermsboxbodyselected"
if hits > 0:
term = create_html_link(self.build_search_url(argd), {},
term, {'class': "nearesttermsselected"})
else:
nearesttermsboxbody_class = "nearesttermsboxbody"
term = create_html_link(self.build_search_url(argd), {},
term, {'class': "nearestterms"})
out += '''\
<tr>
<td class="%(nearesttermsboxbody_class)s" align="right">%(hits)s</td>
<td class="%(nearesttermsboxbody_class)s" width="15">&nbsp;</td>
<td class="%(nearesttermsboxbody_class)s" align="left">%(term)s</td>
</tr>
''' % {'hits': hitsinfo,
'nearesttermsboxbody_class': nearesttermsboxbody_class,
'term': term}
out += "</table>"
return intro + "<blockquote>" + out + "</blockquote>"
def tmpl_browse_pattern(self, f, fn, ln, browsed_phrases_in_colls, colls, rg):
"""
Displays the *Nearest search terms* box
Parameters:
- 'f' *string* - field (*not* i18nized)
- 'fn' *string* - field name (i18nized)
- 'ln' *string* - The language to display
- 'browsed_phrases_in_colls' *array* - the phrases to display
- 'colls' *array* - the list of collection parameters of the search (c's)
- 'rg' *int* - the number of records
"""
# load the right message language
_ = gettext_set_language(ln)
out = """<table class="searchresultsbox">
<thead>
<tr>
<th class="searchresultsboxheader" style="text-align: right;" width="15">
%(hits)s
</th>
<th class="searchresultsboxheader" width="15">
&nbsp;
</th>
<th class="searchresultsboxheader" style="text-align: left;">
%(fn)s
</th>
</tr>
</thead>
<tbody>""" % {
'hits' : _("Hits"),
'fn' : cgi.escape(fn)
}
if len(browsed_phrases_in_colls) == 1:
# one hit only found:
phrase, nbhits = browsed_phrases_in_colls[0][0], browsed_phrases_in_colls[0][1]
query = {'c': colls,
'ln': ln,
'p': '"%s"' % phrase.replace('"', '\\"'),
'f': f,
'rg' : rg}
out += """<tr>
<td class="searchresultsboxbody" style="text-align: right;">
%(nbhits)s
</td>
<td class="searchresultsboxbody" width="15">
&nbsp;
</td>
<td class="searchresultsboxbody" style="text-align: left;">
%(link)s
</td>
</tr>""" % {'nbhits': nbhits,
'link': create_html_link(self.build_search_url(query),
{}, cgi.escape(phrase))}
elif len(browsed_phrases_in_colls) > 1:
# first display what was found but the last one:
for phrase, nbhits in browsed_phrases_in_colls[:-1]:
query = {'c': colls,
'ln': ln,
'p': '"%s"' % phrase.replace('"', '\\"'),
'f': f,
'rg' : rg}
out += """<tr>
<td class="searchresultsboxbody" style="text-align: right;">
%(nbhits)s
</td>
<td class="searchresultsboxbody" width="15">
&nbsp;
</td>
<td class="searchresultsboxbody" style="text-align: left;">
%(link)s
</td>
</tr>""" % {'nbhits' : nbhits,
'link': create_html_link(self.build_search_url(query),
{},
cgi.escape(phrase))}
# now display last hit as "previous term":
phrase, nbhits = browsed_phrases_in_colls[0]
query_previous = {'c': colls,
'ln': ln,
'p': '"%s"' % phrase.replace('"', '\\"'),
'f': f,
'rg' : rg}
# now display last hit as "next term":
phrase, nbhits = browsed_phrases_in_colls[-1]
query_next = {'c': colls,
'ln': ln,
'p': '"%s"' % phrase.replace('"', '\\"'),
'f': f,
'rg' : rg}
out += """<tr><td colspan="2" class="normal">
&nbsp;
</td>
<td class="normal">
%(link_previous)s
<img src="%(siteurl)s/img/sp.gif" alt="" border="0" />
<img src="%(siteurl)s/img/sn.gif" alt="" border="0" />
%(link_next)s
</td>
</tr>""" % {'link_previous': create_html_link(self.build_search_url(query_previous, action='browse'), {}, _("Previous")),
'link_next': create_html_link(self.build_search_url(query_next, action='browse'),
{}, _("next")),
'siteurl' : CFG_BASE_URL}
out += """</tbody>
</table>"""
return out
def tmpl_search_box(self, ln, aas, cc, cc_intl, ot, sp,
action, fieldslist, f1, f2, f3, m1, m2, m3,
p1, p2, p3, op1, op2, rm, p, f, coll_selects,
d1y, d2y, d1m, d2m, d1d, d2d, dt, sort_fields,
sf, so, ranks, sc, rg, formats, of, pl, jrec, ec,
show_colls=True, show_title=True):
"""
Displays the *Nearest search terms* box
Parameters:
- 'ln' *string* - The language to display
- 'aas' *bool* - Should we display an advanced search box? -1 -> 1, from simpler to more advanced
- 'cc_intl' *string* - the i18nized current collection name, used for display
- 'cc' *string* - the internal current collection name
- 'ot', 'sp' *string* - hidden values
- 'action' *string* - the action demanded by the user
- 'fieldslist' *list* - the list of all fields available, for use in select within boxes in advanced search
- 'p, f, f1, f2, f3, m1, m2, m3, p1, p2, p3, op1, op2, op3, rm' *strings* - the search parameters
- 'coll_selects' *array* - a list of lists, each containing the collections selects to display
- 'd1y, d2y, d1m, d2m, d1d, d2d' *int* - the search between dates
- 'dt' *string* - the dates' types (creation dates, modification dates)
- 'sort_fields' *array* - the select information for the sort fields
- 'sf' *string* - the currently selected sort field
- 'so' *string* - the currently selected sort order ("a" or "d")
- 'ranks' *array* - ranking methods
- 'rm' *string* - selected ranking method
- 'sc' *string* - split by collection or not
- 'rg' *string* - selected results/page
- 'formats' *array* - available output formats
- 'of' *string* - the selected output format
- 'pl' *string* - `limit to' search pattern
- show_colls *bool* - propose coll selection box?
- show_title *bool* show cc_intl in page title?
"""
# load the right message language
_ = gettext_set_language(ln)
# These are hidden fields the user does not manipulate
# directly
if aas == -1:
argd = drop_default_urlargd({
'ln': ln, 'aas': aas,
'ot': ot, 'sp': sp, 'ec': ec,
}, self.search_results_default_urlargd)
else:
argd = drop_default_urlargd({
'cc': cc, 'ln': ln, 'aas': aas,
'ot': ot, 'sp': sp, 'ec': ec,
}, self.search_results_default_urlargd)
out = ""
if show_title:
# display cc name if asked for
out += '''
<h1 class="headline">%(ccname)s</h1>''' % {'ccname' : cgi.escape(cc_intl), }
out += '''
<form name="search" action="%(siteurl)s/search" method="get">
''' % {'siteurl' : CFG_BASE_URL}
# Only add non-default hidden values
for field, value in argd.items():
out += self.tmpl_input_hidden(field, value)
leadingtext = _("Search")
if action == 'browse':
leadingtext = _("Browse")
if aas == 1:
# print Advanced Search form:
# define search box elements:
out += '''
<table class="searchbox advancedsearch">
<thead>
<tr>
<th colspan="3" class="searchboxheader">
%(leading)s:
</th>
</tr>
</thead>
<tbody>
<tr valign="top" style="white-space:nowrap;">
<td class="searchboxbody">%(matchbox1)s
<input type="text" name="p1" size="%(sizepattern)d" value="%(p1)s" class="advancedsearchfield"/>
</td>
<td class="searchboxbody">%(searchwithin1)s</td>
<td class="searchboxbody">%(andornot1)s</td>
</tr>
<tr valign="top">
<td class="searchboxbody">%(matchbox2)s
<input type="text" name="p2" size="%(sizepattern)d" value="%(p2)s" class="advancedsearchfield"/>
</td>
<td class="searchboxbody">%(searchwithin2)s</td>
<td class="searchboxbody">%(andornot2)s</td>
</tr>
<tr valign="top">
<td class="searchboxbody">%(matchbox3)s
<input type="text" name="p3" size="%(sizepattern)d" value="%(p3)s" class="advancedsearchfield"/>
</td>
<td class="searchboxbody">%(searchwithin3)s</td>
<td class="searchboxbody" style="white-space:nowrap;">
<input class="formbutton" type="submit" name="action_search" value="%(search)s" />
<input class="formbutton" type="submit" name="action_browse" value="%(browse)s" />&nbsp;
</td>
</tr>
<tr valign="bottom">
<td colspan="3" align="right" class="searchboxbody">
<small>
<a href="%(siteurl)s/help/search-tips%(langlink)s">%(search_tips)s</a> ::
%(simple_search)s
</small>
</td>
</tr>
</tbody>
</table>
''' % {
'simple_search': create_html_link(self.build_search_url(p=p1, f=f1, rm=rm, cc=cc, ln=ln, jrec=jrec, rg=rg),
{}, _("Simple Search")),
'leading' : leadingtext,
'sizepattern' : CFG_WEBSEARCH_ADVANCEDSEARCH_PATTERN_BOX_WIDTH,
'matchbox1' : self.tmpl_matchtype_box('m1', m1, ln=ln),
'p1' : cgi.escape(p1, 1),
'searchwithin1' : self.tmpl_searchwithin_select(
ln=ln,
fieldname='f1',
selected=f1,
values=self._add_mark_to_field(value=f1, fields=fieldslist, ln=ln)
),
'andornot1' : self.tmpl_andornot_box(
name='op1',
value=op1,
ln=ln
),
'matchbox2' : self.tmpl_matchtype_box('m2', m2, ln=ln),
'p2' : cgi.escape(p2, 1),
'searchwithin2' : self.tmpl_searchwithin_select(
ln=ln,
fieldname='f2',
selected=f2,
values=self._add_mark_to_field(value=f2, fields=fieldslist, ln=ln)
),
'andornot2' : self.tmpl_andornot_box(
name='op2',
value=op2,
ln=ln
),
'matchbox3' : self.tmpl_matchtype_box('m3', m3, ln=ln),
'p3' : cgi.escape(p3, 1),
'searchwithin3' : self.tmpl_searchwithin_select(
ln=ln,
fieldname='f3',
selected=f3,
values=self._add_mark_to_field(value=f3, fields=fieldslist, ln=ln)
),
'search' : _("Search"),
'browse' : _("Browse"),
'siteurl' : CFG_BASE_URL,
'ln' : ln,
'langlink': '?ln=' + ln,
'search_tips': _("Search Tips")
}
elif aas == 0:
# print Simple Search form:
out += '''
<table class="searchbox simplesearch">
<thead>
<tr>
<th colspan="3" class="searchboxheader">
%(leading)s:
</th>
</tr>
</thead>
<tbody>
<tr valign="top">
<td class="searchboxbody"><input type="text" name="p" size="%(sizepattern)d" value="%(p)s" class="simplesearchfield"/></td>
<td class="searchboxbody">%(searchwithin)s</td>
<td class="searchboxbody">
<input class="formbutton" type="submit" name="action_search" value="%(search)s" />
<input class="formbutton" type="submit" name="action_browse" value="%(browse)s" />&nbsp;
</td>
</tr>
<tr valign="bottom">
<td colspan="3" align="right" class="searchboxbody">
<small>
<a href="%(siteurl)s/help/search-tips%(langlink)s">%(search_tips)s</a> ::
%(advanced_search)s
</small>
</td>
</tr>
</tbody>
</table>
''' % {
'advanced_search': create_html_link(self.build_search_url(p1=p,
f1=f,
rm=rm,
aas=max(CFG_WEBSEARCH_ENABLED_SEARCH_INTERFACES),
cc=cc,
jrec=jrec,
ln=ln,
rg=rg),
{}, _("Advanced Search")),
'leading' : leadingtext,
'sizepattern' : CFG_WEBSEARCH_SIMPLESEARCH_PATTERN_BOX_WIDTH,
'p' : cgi.escape(p, 1),
'searchwithin' : self.tmpl_searchwithin_select(
ln=ln,
fieldname='f',
selected=f,
values=self._add_mark_to_field(value=f, fields=fieldslist, ln=ln)
),
'search' : _("Search"),
'browse' : _("Browse"),
'siteurl' : CFG_BASE_URL,
'ln' : ln,
'langlink': '?ln=' + ln,
'search_tips': _("Search Tips")
}
else:
# EXPERIMENTAL
# print light search form:
search_in = ''
if cc_intl != CFG_SITE_NAME_INTL.get(ln, CFG_SITE_NAME):
search_in = '''
<input type="radio" name="cc" value="%(collection_id)s" id="searchCollection" checked="checked"/>
<label for="searchCollection">%(search_in_collection_name)s</label>
<input type="radio" name="cc" value="%(root_collection_name)s" id="searchEverywhere" />
<label for="searchEverywhere">%(search_everywhere)s</label>
''' % {'search_in_collection_name': _("Search in %(x_collection_name)s") % \
{'x_collection_name': cgi.escape(cc_intl)},
'collection_id': cc,
'root_collection_name': CFG_SITE_NAME,
'search_everywhere': _("Search everywhere")}
out += '''
<table class="searchbox lightsearch">
<tr valign="top">
<td class="searchboxbody"><input type="text" name="p" size="%(sizepattern)d" value="%(p)s" class="lightsearchfield"/></td>
<td class="searchboxbody">
<input class="formbutton" type="submit" name="action_search" value="%(search)s" />
</td>
<td class="searchboxbody" align="left" rowspan="2" valign="top">
<small><small>
<a href="%(siteurl)s/help/search-tips%(langlink)s">%(search_tips)s</a><br/>
%(advanced_search)s
</td>
</tr>
</table>
<small>%(search_in)s</small>
''' % {
'sizepattern' : CFG_WEBSEARCH_LIGHTSEARCH_PATTERN_BOX_WIDTH,
'advanced_search': create_html_link(self.build_search_url(p1=p,
f1=f,
rm=rm,
aas=max(CFG_WEBSEARCH_ENABLED_SEARCH_INTERFACES),
cc=cc,
jrec=jrec,
ln=ln,
rg=rg),
{}, _("Advanced Search")),
'leading' : leadingtext,
'p' : cgi.escape(p, 1),
'searchwithin' : self.tmpl_searchwithin_select(
ln=ln,
fieldname='f',
selected=f,
values=self._add_mark_to_field(value=f, fields=fieldslist, ln=ln)
),
'search' : _("Search"),
'browse' : _("Browse"),
'siteurl' : CFG_BASE_URL,
'ln' : ln,
'langlink': '?ln=' + ln,
'search_tips': _("Search Tips"),
'search_in': search_in
}
## secondly, print Collection(s) box:
if show_colls and aas > -1:
# display collections only if there is more than one
selects = ''
for sel in coll_selects:
selects += self.tmpl_select(fieldname='c', values=sel)
out += """
<table class="searchbox">
<thead>
<tr>
<th colspan="3" class="searchboxheader">
%(leading)s %(msg_coll)s:
</th>
</tr>
</thead>
<tbody>
<tr valign="bottom">
<td valign="top" class="searchboxbody">
%(colls)s
</td>
</tr>
</tbody>
</table>
""" % {
'leading' : leadingtext,
'msg_coll' : _("collections"),
'colls' : selects,
}
## thirdly, print search limits, if applicable:
if action != _("Browse") and pl:
out += """<table class="searchbox">
<thead>
<tr>
<th class="searchboxheader">
%(limitto)s
</th>
</tr>
</thead>
<tbody>
<tr valign="bottom">
<td class="searchboxbody">
<input type="text" name="pl" size="%(sizepattern)d" value="%(pl)s" />
</td>
</tr>
</tbody>
</table>""" % {
'limitto' : _("Limit to:"),
'sizepattern' : CFG_WEBSEARCH_ADVANCEDSEARCH_PATTERN_BOX_WIDTH,
'pl' : cgi.escape(pl, 1),
}
## fourthly, print from/until date boxen, if applicable:
if action == _("Browse") or (d1y == 0 and d1m == 0 and d1d == 0 and d2y == 0 and d2m == 0 and d2d == 0):
pass # do not need it
else:
cell_6_a = self.tmpl_inputdatetype(dt, ln) + self.tmpl_inputdate("d1", ln, d1y, d1m, d1d)
cell_6_b = self.tmpl_inputdate("d2", ln, d2y, d2m, d2d)
out += """<table class="searchbox">
<thead>
<tr>
<th class="searchboxheader">
%(added)s
</th>
<th class="searchboxheader">
%(until)s
</th>
</tr>
</thead>
<tbody>
<tr valign="bottom">
<td class="searchboxbody">%(added_or_modified)s %(date1)s</td>
<td class="searchboxbody">%(date2)s</td>
</tr>
</tbody>
</table>""" % {
'added' : _("Added/modified since:"),
'until' : _("until:"),
'added_or_modified': self.tmpl_inputdatetype(dt, ln),
'date1' : self.tmpl_inputdate("d1", ln, d1y, d1m, d1d),
'date2' : self.tmpl_inputdate("d2", ln, d2y, d2m, d2d),
}
## fifthly, print Display results box, including sort/rank, formats, etc:
if action != _("Browse") and aas > -1:
rgs = []
for i in [10, 25, 50, 100, 250, 500]:
if i <= CFG_WEBSEARCH_MAX_RECORDS_IN_GROUPS:
rgs.append({ 'value' : i, 'text' : "%d %s" % (i, _("results"))})
# enrich sort fields list if we are sorting by some MARC tag:
sort_fields = self._add_mark_to_field(value=sf, fields=sort_fields, ln=ln)
# create sort by HTML box:
out += """<table class="searchbox">
<thead>
<tr>
<th class="searchboxheader">
%(sort_by)s
</th>
<th class="searchboxheader">
%(display_res)s
</th>
<th class="searchboxheader">
%(out_format)s
</th>
</tr>
</thead>
<tbody>
<tr valign="bottom">
<td class="searchboxbody">
%(select_sf)s %(select_so)s %(select_rm)s
</td>
<td class="searchboxbody">
%(select_rg)s %(select_sc)s
</td>
<td class="searchboxbody">%(select_of)s</td>
</tr>
</tbody>
</table>""" % {
'sort_by' : _("Sort by:"),
'display_res' : _("Display results:"),
'out_format' : _("Output format:"),
'select_sf' : self.tmpl_select(fieldname='sf', values=sort_fields, selected=sf, css_class='address'),
'select_so' : self.tmpl_select(fieldname='so', values=[{
'value' : 'a',
'text' : _("asc.")
}, {
'value' : 'd',
'text' : _("desc.")
}], selected=so, css_class='address'),
'select_rm' : self.tmpl_select(fieldname='rm', values=ranks, selected=rm, css_class='address'),
'select_rg' : self.tmpl_select(fieldname='rg', values=rgs, selected=rg, css_class='address'),
'select_sc' : self.tmpl_select(fieldname='sc', values=[{
'value' : 0,
'text' : _("single list")
}, {
'value' : 1,
'text' : _("split by collection")
}], selected=sc, css_class='address'),
'select_of' : self.tmpl_select(
fieldname='of',
selected=of,
values=self._add_mark_to_field(value=of, fields=formats, chars=3, ln=ln),
css_class='address'),
}
## last but not least, print end of search box:
out += """</form>"""
return out
def tmpl_input_hidden(self, name, value):
"Produces the HTML code for a hidden field "
if isinstance(value, list):
list_input = [self.tmpl_input_hidden(name, val) for val in value]
return "\n".join(list_input)
# # Treat `as', `aas' arguments specially:
if name == 'aas':
name = 'as'
return """<input type="hidden" name="%(name)s" value="%(value)s" />""" % {
'name' : cgi.escape(str(name), 1),
'value' : cgi.escape(str(value), 1),
}
def _add_mark_to_field(self, value, fields, ln, chars=1):
"""Adds the current value as a MARC tag in the fields array
Useful for advanced search"""
# load the right message language
_ = gettext_set_language(ln)
out = fields
if value and str(value[0:chars]).isdigit():
out.append({'value' : value,
'text' : str(value) + " " + _("MARC tag")
})
return out
def tmpl_search_pagestart(self, ln) :
"page start for search page. Will display after the page header"
return """<div class="pagebody"><div class="pagebodystripemiddle">"""
def tmpl_search_pageend(self, ln) :
"page end for search page. Will display just before the page footer"
return """</div></div>"""
def tmpl_print_search_info(self, ln, middle_only,
collection, collection_name, collection_id,
aas, sf, so, rm, rg, nb_found, of, ot, p, f, f1,
f2, f3, m1, m2, m3, op1, op2, p1, p2,
p3, d1y, d1m, d1d, d2y, d2m, d2d, dt,
all_fieldcodes, cpu_time, pl_in_url,
jrec, sc, sp):
"""Prints stripe with the information on 'collection' and 'nb_found' results and CPU time.
Also, prints navigation links (beg/next/prev/end) inside the results set.
If middle_only is set to 1, it will only print the middle box information (beg/netx/prev/end/etc) links.
This is suitable for displaying navigation links at the bottom of the search results page.
Parameters:
- 'ln' *string* - The language to display
- 'middle_only' *bool* - Only display parts of the interface
- 'collection' *string* - the collection name
- 'collection_name' *string* - the i18nized current collection name
- 'aas' *bool* - if we display the advanced search interface
- 'sf' *string* - the currently selected sort format
- 'so' *string* - the currently selected sort order ("a" or "d")
- 'rm' *string* - selected ranking method
- 'rg' *int* - selected results/page
- 'nb_found' *int* - number of results found
- 'of' *string* - the selected output format
- 'ot' *string* - hidden values
- 'p' *string* - Current search words
- 'f' *string* - the fields in which the search was done
- 'f1, f2, f3, m1, m2, m3, p1, p2, p3, op1, op2' *strings* - the search parameters
- 'jrec' *int* - number of first record on this page
- 'd1y, d2y, d1m, d2m, d1d, d2d' *int* - the search between dates
- 'dt' *string* the dates' type (creation date, modification date)
- 'all_fieldcodes' *array* - all the available fields
- 'cpu_time' *float* - the time of the query in seconds
"""
# load the right message language
_ = gettext_set_language(ln)
out = ""
# left table cells: print collection name
if not middle_only:
out += '''
<a name="%(collection_id)s"></a>
<form action="%(siteurl)s/search" method="get">
<table class="searchresultsbox"><tr><td class="searchresultsboxheader" align="left">
<strong><big>%(collection_link)s</big></strong></td>
''' % {
'collection_id': collection_id,
'siteurl' : CFG_BASE_URL,
'collection_link': create_html_link(self.build_search_interface_url(c=collection, aas=aas, ln=ln),
{}, cgi.escape(collection_name))
}
else:
out += """
<div style="clear:both"></div>
<form action="%(siteurl)s/search" method="get"><div align="center">
""" % { 'siteurl' : CFG_BASE_URL }
# middle table cell: print beg/next/prev/end arrows:
if not middle_only:
out += """<td class="searchresultsboxheader" align="center">
%(recs_found)s &nbsp;""" % {
'recs_found' : _("%(x_rec)s records found", x_rec=('<strong>' + self.tmpl_nice_number(nb_found, ln) + '</strong>'))
}
else:
out += "<small>"
if nb_found > rg:
out += "" + cgi.escape(collection_name) + " : " + _("%(x_rec)s records found", x_rec=('<strong>' + self.tmpl_nice_number(nb_found, ln) + '</strong>')) + " &nbsp; "
if nb_found > rg: # navig.arrows are needed, since we have many hits
query = {'p': p, 'f': f,
'cc': collection,
'sf': sf, 'so': so,
'sp': sp, 'rm': rm,
'of': of, 'ot': ot,
'aas': aas, 'ln': ln,
'p1': p1, 'p2': p2, 'p3': p3,
'f1': f1, 'f2': f2, 'f3': f3,
'm1': m1, 'm2': m2, 'm3': m3,
'op1': op1, 'op2': op2,
'sc': 0,
'd1y': d1y, 'd1m': d1m, 'd1d': d1d,
'd2y': d2y, 'd2m': d2m, 'd2d': d2d,
'dt': dt,
}
# @todo here
def img(gif, txt):
return '<img src="%(siteurl)s/img/%(gif)s.gif" alt="%(txt)s" border="0" />' % {
'txt': txt, 'gif': gif, 'siteurl': CFG_BASE_URL}
if jrec - rg > 1:
out += create_html_link(self.build_search_url(query, jrec=1, rg=rg),
{}, img('sb', _("begin")),
{'class': 'img'})
if jrec > 1:
out += create_html_link(self.build_search_url(query, jrec=max(jrec - rg, 1), rg=rg),
{}, img('sp', _("previous")),
{'class': 'img'})
if jrec + rg - 1 < nb_found:
out += "%d - %d" % (jrec, jrec + rg - 1)
else:
out += "%d - %d" % (jrec, nb_found)
if nb_found >= jrec + rg:
out += create_html_link(self.build_search_url(query,
jrec=jrec + rg,
rg=rg),
{}, img('sn', _("next")),
{'class':'img'})
if nb_found >= jrec + rg + rg:
out += create_html_link(self.build_search_url(query,
jrec=nb_found - rg + 1,
rg=rg),
{}, img('se', _("end")),
{'class': 'img'})
# still in the navigation part
cc = collection
sc = 0
for var in ['p', 'cc', 'f', 'sf', 'so', 'of', 'rg', 'aas', 'ln', 'p1', 'p2', 'p3', 'f1', 'f2', 'f3', 'm1', 'm2', 'm3', 'op1', 'op2', 'sc', 'd1y', 'd1m', 'd1d', 'd2y', 'd2m', 'd2d', 'dt']:
out += self.tmpl_input_hidden(name=var, value=vars()[var])
for var in ['ot', 'sp', 'rm']:
if vars()[var]:
out += self.tmpl_input_hidden(name=var, value=vars()[var])
if pl_in_url:
fieldargs = cgi.parse_qs(pl_in_url)
for fieldcode in all_fieldcodes:
# get_fieldcodes():
if fieldcode in fieldargs:
for val in fieldargs[fieldcode]:
out += self.tmpl_input_hidden(name=fieldcode, value=val)
out += """&nbsp; %(jump)s <input type="text" name="jrec" size="4" value="%(jrec)d" />""" % {
'jump' : _("jump to record:"),
'jrec' : jrec,
}
if not middle_only:
out += "</td>"
else:
out += "</small>"
# right table cell: cpu time info
if not middle_only:
if cpu_time > -1:
out += """<td class="searchresultsboxheader" align="right"><small>%(time)s</small>&nbsp;</td>""" % {
'time' : _("Search took %(x_sec)s seconds.", x_sec=('%.2f' % cpu_time)),
}
out += "</tr></table>"
else:
out += "</div>"
out += "</form>"
return out
def tmpl_print_hosted_search_info(self, ln, middle_only,
collection, collection_name, collection_id,
aas, sf, so, rm, rg, nb_found, of, ot, p, f, f1,
f2, f3, m1, m2, m3, op1, op2, p1, p2,
p3, d1y, d1m, d1d, d2y, d2m, d2d, dt,
all_fieldcodes, cpu_time, pl_in_url,
jrec, sc, sp):
"""Prints stripe with the information on 'collection' and 'nb_found' results and CPU time.
Also, prints navigation links (beg/next/prev/end) inside the results set.
If middle_only is set to 1, it will only print the middle box information (beg/netx/prev/end/etc) links.
This is suitable for displaying navigation links at the bottom of the search results page.
Parameters:
- 'ln' *string* - The language to display
- 'middle_only' *bool* - Only display parts of the interface
- 'collection' *string* - the collection name
- 'collection_name' *string* - the i18nized current collection name
- 'aas' *bool* - if we display the advanced search interface
- 'sf' *string* - the currently selected sort format
- 'so' *string* - the currently selected sort order ("a" or "d")
- 'rm' *string* - selected ranking method
- 'rg' *int* - selected results/page
- 'nb_found' *int* - number of results found
- 'of' *string* - the selected output format
- 'ot' *string* - hidden values
- 'p' *string* - Current search words
- 'f' *string* - the fields in which the search was done
- 'f1, f2, f3, m1, m2, m3, p1, p2, p3, op1, op2' *strings* - the search parameters
- 'jrec' *int* - number of first record on this page
- 'd1y, d2y, d1m, d2m, d1d, d2d' *int* - the search between dates
- 'dt' *string* the dates' type (creation date, modification date)
- 'all_fieldcodes' *array* - all the available fields
- 'cpu_time' *float* - the time of the query in seconds
"""
# load the right message language
_ = gettext_set_language(ln)
out = ""
# left table cells: print collection name
if not middle_only:
out += '''
<a name="%(collection_id)s"></a>
<form action="%(siteurl)s/search" method="get">
<table class="searchresultsbox"><tr><td class="searchresultsboxheader" align="left">
<strong><big>%(collection_link)s</big></strong></td>
''' % {
'collection_id': collection_id,
'siteurl' : CFG_BASE_URL,
'collection_link': create_html_link(self.build_search_interface_url(c=collection, aas=aas, ln=ln),
{}, cgi.escape(collection_name))
}
else:
out += """
<form action="%(siteurl)s/search" method="get"><div align="center">
""" % { 'siteurl' : CFG_BASE_URL }
# middle table cell: print beg/next/prev/end arrows:
if not middle_only:
# in case we have a hosted collection that timed out do not print its number of records, as it is yet unknown
if nb_found != -963:
out += """<td class="searchresultsboxheader" align="center">
%(recs_found)s &nbsp;""" % {
'recs_found' : _("%(x_rec)s records found", x_rec=('<strong>' + self.tmpl_nice_number(nb_found, ln) + '</strong>'))
}
#elif nb_found = -963:
# out += """<td class="searchresultsboxheader" align="center">
# %(recs_found)s &nbsp;""" % {
# 'recs_found' : _("%s records found") % ('<strong>' + self.tmpl_nice_number(nb_found, ln) + '</strong>')
# }
else:
out += "<small>"
# we do not care about timed out hosted collections here, because the bumber of records found will never be bigger
# than rg anyway, since it's negative
if nb_found > rg:
out += "" + cgi.escape(collection_name) + " : " + _("%(x_rec)s records found", x_rec=('<strong>' + self.tmpl_nice_number(nb_found, ln) + '</strong>')) + " &nbsp; "
if nb_found > rg: # navig.arrows are needed, since we have many hits
query = {'p': p, 'f': f,
'cc': collection,
'sf': sf, 'so': so,
'sp': sp, 'rm': rm,
'of': of, 'ot': ot,
'aas': aas, 'ln': ln,
'p1': p1, 'p2': p2, 'p3': p3,
'f1': f1, 'f2': f2, 'f3': f3,
'm1': m1, 'm2': m2, 'm3': m3,
'op1': op1, 'op2': op2,
'sc': 0,
'd1y': d1y, 'd1m': d1m, 'd1d': d1d,
'd2y': d2y, 'd2m': d2m, 'd2d': d2d,
'dt': dt,
}
# @todo here
def img(gif, txt):
return '<img src="%(siteurl)s/img/%(gif)s.gif" alt="%(txt)s" border="0" />' % {
'txt': txt, 'gif': gif, 'siteurl': CFG_BASE_URL}
if jrec - rg > 1:
out += create_html_link(self.build_search_url(query, jrec=1, rg=rg),
{}, img('sb', _("begin")),
{'class': 'img'})
if jrec > 1:
out += create_html_link(self.build_search_url(query, jrec=max(jrec - rg, 1), rg=rg),
{}, img('sp', _("previous")),
{'class': 'img'})
if jrec + rg - 1 < nb_found:
out += "%d - %d" % (jrec, jrec + rg - 1)
else:
out += "%d - %d" % (jrec, nb_found)
if nb_found >= jrec + rg:
out += create_html_link(self.build_search_url(query,
jrec=jrec + rg,
rg=rg),
{}, img('sn', _("next")),
{'class':'img'})
if nb_found >= jrec + rg + rg:
out += create_html_link(self.build_search_url(query,
jrec=nb_found - rg + 1,
rg=rg),
{}, img('se', _("end")),
{'class': 'img'})
# still in the navigation part
cc = collection
sc = 0
for var in ['p', 'cc', 'f', 'sf', 'so', 'of', 'rg', 'aas', 'ln', 'p1', 'p2', 'p3', 'f1', 'f2', 'f3', 'm1', 'm2', 'm3', 'op1', 'op2', 'sc', 'd1y', 'd1m', 'd1d', 'd2y', 'd2m', 'd2d', 'dt']:
out += self.tmpl_input_hidden(name=var, value=vars()[var])
for var in ['ot', 'sp', 'rm']:
if vars()[var]:
out += self.tmpl_input_hidden(name=var, value=vars()[var])
if pl_in_url:
fieldargs = cgi.parse_qs(pl_in_url)
for fieldcode in all_fieldcodes:
# get_fieldcodes():
if fieldcode in fieldargs:
for val in fieldargs[fieldcode]:
out += self.tmpl_input_hidden(name=fieldcode, value=val)
out += """&nbsp; %(jump)s <input type="text" name="jrec" size="4" value="%(jrec)d" />""" % {
'jump' : _("jump to record:"),
'jrec' : jrec,
}
if not middle_only:
out += "</td>"
else:
out += "</small>"
# right table cell: cpu time info
if not middle_only:
if cpu_time > -1:
out += """<td class="searchresultsboxheader" align="right"><small>%(time)s</small>&nbsp;</td>""" % {
'time' : _("Search took %(x_sec)s seconds.", x_sec=('%.2f' % cpu_time)),
}
out += "</tr></table>"
else:
out += "</div>"
out += "</form>"
return out
def tmpl_nice_number(self, number, ln=CFG_SITE_LANG, thousands_separator=',', max_ndigits_after_dot=None):
"""
Return nicely printed number NUMBER in language LN using
given THOUSANDS_SEPARATOR character.
If max_ndigits_after_dot is specified and the number is float, the
number is rounded by taking in consideration up to max_ndigits_after_dot
digit after the dot.
This version does not pay attention to locale. See
tmpl_nice_number_via_locale().
"""
if type(number) is float:
if max_ndigits_after_dot is not None:
number = round(number, max_ndigits_after_dot)
int_part, frac_part = str(number).split('.')
return '%s.%s' % (self.tmpl_nice_number(int(int_part), ln, thousands_separator), frac_part)
else:
chars_in = list(str(number))
number = len(chars_in)
chars_out = []
for i in range(0, number):
if i % 3 == 0 and i != 0:
chars_out.append(thousands_separator)
chars_out.append(chars_in[number - i - 1])
chars_out.reverse()
return ''.join(chars_out)
def tmpl_nice_number_via_locale(self, number, ln=CFG_SITE_LANG):
"""
Return nicely printed number NUM in language LN using the locale.
See also version tmpl_nice_number().
"""
if number is None:
return None
# Temporarily switch the numeric locale to the requested one, and format the number
# In case the system has no locale definition, use the vanilla form
ol = locale.getlocale(locale.LC_NUMERIC)
try:
locale.setlocale(locale.LC_NUMERIC, self.tmpl_localemap.get(ln, self.tmpl_default_locale))
except locale.Error:
return str(number)
try:
number = locale.format('%d', number, True)
except TypeError:
return str(number)
locale.setlocale(locale.LC_NUMERIC, ol)
return number
def tmpl_record_format_htmlbrief_header(self, ln):
"""Returns the header of the search results list when output
is html brief. Note that this function is called for each collection
results when 'split by collection' is enabled.
See also: tmpl_record_format_htmlbrief_footer,
tmpl_record_format_htmlbrief_body
Parameters:
- 'ln' *string* - The language to display
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<form action="%(siteurl)s/yourbaskets/add" method="post">
<table>
""" % {
'siteurl' : CFG_BASE_URL,
}
return out
def tmpl_record_format_htmlbrief_footer(self, ln, display_add_to_basket=True):
"""Returns the footer of the search results list when output
is html brief. Note that this function is called for each collection
results when 'split by collection' is enabled.
See also: tmpl_record_format_htmlbrief_header(..),
tmpl_record_format_htmlbrief_body(..)
Parameters:
- 'ln' *string* - The language to display
- 'display_add_to_basket' *bool* - whether to display Add-to-basket button
"""
# load the right message language
_ = gettext_set_language(ln)
out = """</table>
<br />
<input type="hidden" name="colid" value="0" />
%(add_to_basket)s
</form>""" % {
'add_to_basket': display_add_to_basket and """<input class="formbutton" type="submit" name="action" value="%s" />""" % _("Add to basket") or "",
}
return out
def tmpl_record_format_htmlbrief_body(self, ln, recid,
row_number, relevance,
record, relevances_prologue,
relevances_epilogue,
display_add_to_basket=True):
"""Returns the html brief format of one record. Used in the
search results list for each record.
See also: tmpl_record_format_htmlbrief_header(..),
tmpl_record_format_htmlbrief_footer(..)
Parameters:
- 'ln' *string* - The language to display
- 'row_number' *int* - The position of this record in the list
- 'recid' *int* - The recID
- 'relevance' *string* - The relevance of the record
- 'record' *string* - The formatted record
- 'relevances_prologue' *string* - HTML code to prepend the relevance indicator
- 'relevances_epilogue' *string* - HTML code to append to the relevance indicator (used mostly for formatting)
"""
# load the right message language
_ = gettext_set_language(ln)
checkbox_for_baskets = """<input name="recid" type="checkbox" value="%(recid)s" />""" % \
{'recid': recid, }
if not display_add_to_basket:
checkbox_for_baskets = ''
out = """
<tr><td valign="top" align="right" style="white-space: nowrap;">
%(checkbox_for_baskets)s
<abbr class="unapi-id" title="%(recid)s"></abbr>
%(number)s.
""" % {'recid': recid,
'number': row_number,
'checkbox_for_baskets': checkbox_for_baskets}
if relevance:
out += """<br /><div class="rankscoreinfo"><a title="rank score">%(prologue)s%(relevance)s%(epilogue)s</a></div>""" % {
'prologue' : relevances_prologue,
'epilogue' : relevances_epilogue,
'relevance' : relevance
}
out += """</td><td valign="top">%s</td></tr>""" % record
return out
def tmpl_print_results_overview(self, ln, results_final_nb_total, cpu_time, results_final_nb, colls, ec, hosted_colls_potential_results_p=False):
"""Prints results overview box with links to particular collections below.
Parameters:
- 'ln' *string* - The language to display
- 'results_final_nb_total' *int* - The total number of hits for the query
- 'colls' *array* - The collections with hits, in the format:
- 'coll[code]' *string* - The code of the collection (canonical name)
- 'coll[name]' *string* - The display name of the collection
- 'results_final_nb' *array* - The number of hits, indexed by the collection codes:
- 'cpu_time' *string* - The time the query took
- 'url_args' *string* - The rest of the search query
- 'ec' *array* - selected external collections
- 'hosted_colls_potential_results_p' *boolean* - check if there are any hosted collections searches
that timed out during the pre-search
"""
if len(colls) == 1 and not ec:
# if one collection only and no external collections, print nothing:
return ""
# load the right message language
_ = gettext_set_language(ln)
# first find total number of hits:
# if there were no hosted collections that timed out during the pre-search print out the exact number of records found
if not hosted_colls_potential_results_p:
out = """<table class="searchresultsbox">
<thead><tr><th class="searchresultsboxheader">%(founds)s</th></tr></thead>
<tbody><tr><td class="searchresultsboxbody"> """ % {
'founds' : _("%(x_fmt_open)sResults overview:%(x_fmt_close)s Found %(x_nb_records)s records in %(x_nb_seconds)s seconds.") % \
{'x_fmt_open': '<strong>',
'x_fmt_close': '</strong>',
'x_nb_records': '<strong>' + self.tmpl_nice_number(results_final_nb_total, ln) + '</strong>',
'x_nb_seconds': '%.2f' % cpu_time}
}
# if there were (only) hosted_collections that timed out during the pre-search print out a fuzzier message
else:
if results_final_nb_total == 0:
out = """<table class="searchresultsbox">
<thead><tr><th class="searchresultsboxheader">%(founds)s</th></tr></thead>
<tbody><tr><td class="searchresultsboxbody"> """ % {
'founds' : _("%(x_fmt_open)sResults overview%(x_fmt_close)s") % \
{'x_fmt_open': '<strong>',
'x_fmt_close': '</strong>'}
}
elif results_final_nb_total > 0:
out = """<table class="searchresultsbox">
<thead><tr><th class="searchresultsboxheader">%(founds)s</th></tr></thead>
<tbody><tr><td class="searchresultsboxbody"> """ % {
'founds' : _("%(x_fmt_open)sResults overview:%(x_fmt_close)s Found at least %(x_nb_records)s records in %(x_nb_seconds)s seconds.") % \
{'x_fmt_open': '<strong>',
'x_fmt_close': '</strong>',
'x_nb_records': '<strong>' + self.tmpl_nice_number(results_final_nb_total, ln) + '</strong>',
'x_nb_seconds': '%.2f' % cpu_time}
}
# then print hits per collection:
out += """<script type="text/javascript">
$(document).ready(function() {
$('a.morecolls').click(function() {
$('.morecollslist').show();
$(this).hide();
$('.lesscolls').show();
return false;
});
$('a.lesscolls').click(function() {
$('.morecollslist').hide();
$(this).hide();
$('.morecolls').show();
return false;
});
});
</script>"""
count = 0
for coll in colls:
if coll['code'] in results_final_nb and results_final_nb[coll['code']] > 0:
count += 1
out += """
<span %(collclass)s><strong><a href="#%(coll)s">%(coll_name)s</a></strong>, <a href="#%(coll)s">%(number)s</a><br /></span>""" % \
{'collclass' : count > cfg['CFG_WEBSEARCH_RESULTS_OVERVIEW_MAX_COLLS_TO_PRINT'] and 'class="morecollslist" style="display:none"' or '',
'coll' : coll['id'],
'coll_name' : cgi.escape(coll['name']),
'number' : _("%(x_rec)s records found", x_rec=('<strong>' + self.tmpl_nice_number(results_final_nb[coll['code']], ln) + '</strong>'))}
# the following is used for hosted collections that have timed out,
# i.e. for which we don't know the exact number of results yet.
elif coll['code'] in results_final_nb and results_final_nb[coll['code']] == -963:
count += 1
out += """
<span %(collclass)s><strong><a href="#%(coll)s">%(coll_name)s</a></strong><br /></span>""" % \
{'collclass' : count > cfg['CFG_WEBSEARCH_RESULTS_OVERVIEW_MAX_COLLS_TO_PRINT'] and 'class="morecollslist" style="display:none"' or '',
'coll' : coll['id'],
'coll_name' : cgi.escape(coll['name']),
'number' : _("%(x_rec)s records found", x_rec=('<strong>' + self.tmpl_nice_number(results_final_nb[coll['code']], ln) + '</strong>'))}
if count > cfg['CFG_WEBSEARCH_RESULTS_OVERVIEW_MAX_COLLS_TO_PRINT']:
out += """<a class="lesscolls" style="display:none; color:red; font-size:small" href="#"><i>%s</i></a>""" % _("Show less collections")
out += """<a class="morecolls" style="color:red; font-size:small" href="#"><i>%s</i></a>""" % _("Show all collections")
out += "</td></tr></tbody></table>"
return out
def tmpl_print_hosted_results(self, url_and_engine, ln, of=None, req=None, limit=CFG_EXTERNAL_COLLECTION_MAXRESULTS, display_body=True, display_add_to_basket = True):
"""Print results of a given search engine.
"""
if display_body:
_ = gettext_set_language(ln)
#url = url_and_engine[0]
engine = url_and_engine[1]
#name = _(engine.name)
db_id = get_collection_id(engine.name)
#base_url = engine.base_url
out = ""
results = engine.parser.parse_and_get_results(None, of=of, req=req, limit=limit, parseonly=True)
if len(results) != 0:
if of == 'hb':
out += """
<form action="%(siteurl)s/yourbaskets/add" method="post">
<input type="hidden" name="colid" value="%(col_db_id)s" />
<table>
""" % {
'siteurl' : CFG_BASE_URL,
'col_db_id' : db_id,
}
else:
if of == 'hb':
out += """
<table>
"""
for result in results:
out += result.html.replace('>Detailed record<', '>External record<').replace('>Similar records<', '>Similar external records<')
if len(results) != 0:
if of == 'hb':
out += """</table>
<br />"""
if display_add_to_basket:
out += """<input class="formbutton" type="submit" name="action" value="%(basket)s" />
""" % {'basket' : _("Add to basket")}
out += """</form>"""
else:
if of == 'hb':
out += """
</table>
"""
# we have already checked if there are results or no, maybe the following if should be removed?
if not results:
if of.startswith("h"):
out = _('No results found...') + '<br />'
return out
else:
return ""
def tmpl_print_service_list_links(self, label, labels_and_urls, ln=CFG_SITE_URL):
"""
Prints service results as list
@param label: the label to display before the list of links
@type label: string
@param labels_and_urls: list of tuples (label, url), already translated, not escaped
@type labels_and_urls: list(string, string)
@param ln: language
"""
# load the right message language
_ = gettext_set_language(ln)
out = '''
<span class="searchservicelabel">%s</span> ''' % cgi.escape(label)
out += """<script type="text/javascript">
$(document).ready(function() {
$('a.moreserviceitemslink').click(function() {
$('.moreserviceitemslist', $(this).parent()).show();
$(this).hide();
$('.lessserviceitemslink', $(this).parent()).show();
return false;
});
$('a.lessserviceitemslink').click(function() {
$('.moreserviceitemslist', $(this).parent()).hide();
$(this).hide();
$('.moreserviceitemslink', $(this).parent()).show();
return false;
});
});
</script>"""
count = 0
for link_label, link_url in labels_and_urls:
count += 1
out += """<span %(itemclass)s>%(separator)s <a class="searchserviceitem" href="%(url)s">%(link_label)s</a></span>""" % \
{'itemclass' : count > CFG_WEBSEARCH_MAX_SEARCH_COLL_RESULTS_TO_PRINT and 'class="moreserviceitemslist" style="display:none"' or '',
'separator': count > 1 and ', ' or '',
'url' : link_url,
'link_label' : cgi.escape(link_label)}
if count > CFG_WEBSEARCH_MAX_SEARCH_COLL_RESULTS_TO_PRINT:
out += """ <a class="lessserviceitemslink" style="display:none;" href="#">%s</a>""" % _("Less suggestions")
out += """ <a class="moreserviceitemslink" style="" href="#">%s</a>""" % _("More suggestions")
return out
def tmpl_print_searchresultbox(self, header, body):
"""print a nicely formatted box for search results """
#_ = gettext_set_language(ln)
# first find total number of hits:
out = '<table class="searchresultsbox"><thead><tr><th class="searchresultsboxheader">' + header + '</th></tr></thead><tbody><tr><td class="searchresultsboxbody">' + body + '</td></tr></tbody></table>'
return out
def tmpl_search_no_boolean_hits(self, ln, nearestterms):
"""No hits found, proposes alternative boolean queries
Parameters:
- 'ln' *string* - The language to display
- 'nearestterms' *array* - Parts of the interface to display, in the format:
- 'nearestterms[nbhits]' *int* - The resulting number of hits
- 'nearestterms[url_args]' *string* - The search parameters
- 'nearestterms[p]' *string* - The search terms
"""
# load the right message language
_ = gettext_set_language(ln)
out = _("Boolean query returned no hits. Please combine your search terms differently.")
out += '''<blockquote><table class="nearesttermsbox" cellpadding="0" cellspacing="0" border="0">'''
for term, hits, argd in nearestterms:
out += '''\
<tr>
<td class="nearesttermsboxbody" align="right">%(hits)s</td>
<td class="nearesttermsboxbody" width="15">&nbsp;</td>
<td class="nearesttermsboxbody" align="left">
%(link)s
</td>
</tr>''' % {'hits' : hits,
'link': create_html_link(self.build_search_url(argd),
{}, cgi.escape(term),
{'class': "nearestterms"})}
out += """</table></blockquote>"""
return out
def tmpl_similar_author_names(self, authors, ln):
"""No hits found, proposes alternative boolean queries
Parameters:
- 'authors': a list of (name, hits) tuples
- 'ln' *string* - The language to display
"""
# load the right message language
_ = gettext_set_language(ln)
out = '''<a name="googlebox"></a>
<table class="googlebox"><tr><th colspan="2" class="googleboxheader">%(similar)s</th></tr>''' % {
'similar' : _("See also: similar author names")
}
for author, hits in authors:
out += '''\
<tr>
<td class="googleboxbody">%(nb)d</td>
<td class="googleboxbody">%(link)s</td>
</tr>''' % {'link': create_html_link(
self.build_search_url(p=author,
f='author',
ln=ln),
{}, cgi.escape(author), {'class':"google"}),
'nb' : hits}
out += """</table>"""
return out
def tmpl_print_record_detailed(self, recID, ln):
"""Displays a detailed on-the-fly record
Parameters:
- 'ln' *string* - The language to display
- 'recID' *int* - The record id
"""
# okay, need to construct a simple "Detailed record" format of our own:
out = "<p>&nbsp;"
# secondly, title:
titles = get_fieldvalues(recID, "245__a") or \
get_fieldvalues(recID, "111__a")
for title in titles:
out += "<p><center><big><strong>%s</strong></big></center></p>" % cgi.escape(title)
# thirdly, authors:
authors = get_fieldvalues(recID, "100__a") + get_fieldvalues(recID, "700__a")
if authors:
out += "<p><center>"
for author in authors:
out += '%s; ' % create_html_link(self.build_search_url(
ln=ln,
p=author,
f='author'),
{}, cgi.escape(author))
out += "</center></p>"
# fourthly, date of creation:
dates = get_fieldvalues(recID, "260__c")
for date in dates:
out += "<p><center><small>%s</small></center></p>" % date
# fifthly, abstract:
abstracts = get_fieldvalues(recID, "520__a")
for abstract in abstracts:
out += """<p style="margin-left: 15%%; width: 70%%">
<small><strong>Abstract:</strong> %s</small></p>""" % abstract
# fifthly bis, keywords:
keywords = get_fieldvalues(recID, "6531_a")
if len(keywords):
out += """<p style="margin-left: 15%%; width: 70%%">
<small><strong>Keyword(s):</strong>"""
for keyword in keywords:
out += '%s; ' % create_html_link(
self.build_search_url(ln=ln,
p=keyword,
f='keyword'),
{}, cgi.escape(keyword))
out += '</small></p>'
# fifthly bis bis, published in:
prs_p = get_fieldvalues(recID, "909C4p")
prs_v = get_fieldvalues(recID, "909C4v")
prs_y = get_fieldvalues(recID, "909C4y")
prs_n = get_fieldvalues(recID, "909C4n")
prs_c = get_fieldvalues(recID, "909C4c")
for idx in range(0, len(prs_p)):
out += """<p style="margin-left: 15%%; width: 70%%">
<small><strong>Publ. in:</strong> %s""" % prs_p[idx]
if prs_v and prs_v[idx]:
out += """<strong>%s</strong>""" % prs_v[idx]
if prs_y and prs_y[idx]:
out += """(%s)""" % prs_y[idx]
if prs_n and prs_n[idx]:
out += """, no.%s""" % prs_n[idx]
if prs_c and prs_c[idx]:
out += """, p.%s""" % prs_c[idx]
out += """.</small></p>"""
# sixthly, fulltext link:
urls_z = get_fieldvalues(recID, "8564_z")
urls_u = get_fieldvalues(recID, "8564_u")
# we separate the fulltext links and image links
for url_u in urls_u:
if url_u.endswith('.png'):
continue
else:
link_text = "URL"
try:
if urls_z[idx]:
link_text = urls_z[idx]
except IndexError:
pass
out += """<p style="margin-left: 15%%; width: 70%%">
<small><strong>%s:</strong> <a href="%s">%s</a></small></p>""" % (link_text, urls_u[idx], urls_u[idx])
# print some white space at the end:
out += "<br /><br />"
return out
def tmpl_print_record_list_for_similarity_boxen(self, title, recID_score_list, ln=CFG_SITE_LANG):
"""Print list of records in the "hs" (HTML Similarity) format for similarity boxes.
RECID_SCORE_LIST is a list of (recID1, score1), (recID2, score2), etc.
"""
from invenio.legacy.search_engine import print_record, record_public_p
recID_score_list_to_be_printed = []
# firstly find 5 first public records to print:
nb_records_to_be_printed = 0
nb_records_seen = 0
while nb_records_to_be_printed < 5 and nb_records_seen < len(recID_score_list) and nb_records_seen < 50:
# looking through first 50 records only, picking first 5 public ones
(recID, score) = recID_score_list[nb_records_seen]
nb_records_seen += 1
if record_public_p(recID):
nb_records_to_be_printed += 1
recID_score_list_to_be_printed.append([recID, score])
# secondly print them:
out = '''
<table><tr>
<td>
<table><tr><td class="blocknote">%(title)s</td></tr></table>
</td>
</tr>
<tr>
<td><table>
''' % { 'title': cgi.escape(title) }
for recid, score in recID_score_list_to_be_printed:
out += '''
<tr><td><font class="rankscoreinfo"><a>(%(score)s)&nbsp;</a></font><small>&nbsp;%(info)s</small></td></tr>''' % {
'score': score,
'info' : print_record(recid, format="hs", ln=ln),
}
out += """</table></td></tr></table> """
return out
def tmpl_print_record_brief(self, ln, recID):
"""Displays a brief record on-the-fly
Parameters:
- 'ln' *string* - The language to display
- 'recID' *int* - The record id
"""
out = ""
# record 'recID' does not exist in format 'format', so print some default format:
# firstly, title:
titles = get_fieldvalues(recID, "245__a") or \
get_fieldvalues(recID, "111__a")
# secondly, authors:
authors = get_fieldvalues(recID, "100__a") + get_fieldvalues(recID, "700__a")
# thirdly, date of creation:
dates = get_fieldvalues(recID, "260__c")
# thirdly bis, report numbers:
rns = get_fieldvalues(recID, "037__a")
rns = get_fieldvalues(recID, "088__a")
# fourthly, beginning of abstract:
abstracts = get_fieldvalues(recID, "520__a")
# fifthly, fulltext link:
urls_z = get_fieldvalues(recID, "8564_z")
urls_u = get_fieldvalues(recID, "8564_u")
# get rid of images
images = []
non_image_urls_u = []
for url_u in urls_u:
if url_u.endswith('.png'):
images.append(url_u)
else:
non_image_urls_u.append(url_u)
## unAPI identifier
out = '<abbr class="unapi-id" title="%s"></abbr>\n' % recID
out += self.tmpl_record_body(
titles=titles,
authors=authors,
dates=dates,
rns=rns,
abstracts=abstracts,
urls_u=non_image_urls_u,
urls_z=urls_z,
ln=ln)
return out
def tmpl_print_record_brief_links(self, ln, recID, sf='', so='d', sp='', rm='', display_claim_link=False, display_edit_link=False):
"""Displays links for brief record on-the-fly
Parameters:
- 'ln' *string* - The language to display
- 'recID' *int* - The record id
"""
from invenio.ext.template import render_template_to_string
tpl = """{%- from "search/helpers.html" import record_brief_links with context -%}
{{ record_brief_links(get_record(recid)) }}"""
return render_template_to_string(tpl, recid=recID, _from_string=True).encode('utf-8')
def tmpl_xml_rss_prologue(self, current_url=None,
previous_url=None, next_url=None,
first_url=None, last_url=None,
nb_found=None, jrec=None, rg=None, cc=None):
"""Creates XML RSS 2.0 prologue."""
title = CFG_SITE_NAME
description = '%s latest documents' % CFG_SITE_NAME
if cc and cc != CFG_SITE_NAME:
title += ': ' + cgi.escape(cc)
description += ' in ' + cgi.escape(cc)
out = """<rss version="2.0"
xmlns:media="http://search.yahoo.com/mrss/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:dcterms="http://purl.org/dc/terms/"
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
<channel>
<title>%(rss_title)s</title>
<link>%(siteurl)s</link>
<description>%(rss_description)s</description>
<language>%(sitelang)s</language>
<pubDate>%(timestamp)s</pubDate>
<category></category>
<generator>Invenio %(version)s</generator>
<webMaster>%(sitesupportemail)s</webMaster>
<ttl>%(timetolive)s</ttl>%(previous_link)s%(next_link)s%(current_link)s%(total_results)s%(start_index)s%(items_per_page)s
<image>
<url>%(siteurl)s/img/site_logo_rss.png</url>
<title>%(sitename)s</title>
<link>%(siteurl)s</link>
</image>
<atom:link rel="search" href="%(siteurl)s/opensearchdescription" type="application/opensearchdescription+xml" title="Content Search" />
<textInput>
<title>Search </title>
<description>Search this site:</description>
<name>p</name>
<link>%(siteurl)s/search</link>
</textInput>
""" % {'sitename': CFG_SITE_NAME,
'siteurl': CFG_SITE_URL,
'sitelang': CFG_SITE_LANG,
'search_syntax': self.tmpl_opensearch_rss_url_syntax,
'timestamp': time.strftime("%a, %d %b %Y %H:%M:%S GMT", time.gmtime()),
'version': CFG_VERSION,
'sitesupportemail': CFG_SITE_SUPPORT_EMAIL,
'timetolive': CFG_WEBSEARCH_RSS_TTL,
'current_link': (current_url and \
'\n<atom:link rel="self" href="%s" />\n' % current_url) or '',
'previous_link': (previous_url and \
'\n<atom:link rel="previous" href="%s" />' % previous_url) or '',
'next_link': (next_url and \
'\n<atom:link rel="next" href="%s" />' % next_url) or '',
'first_link': (first_url and \
'\n<atom:link rel="first" href="%s" />' % first_url) or '',
'last_link': (last_url and \
'\n<atom:link rel="last" href="%s" />' % last_url) or '',
'total_results': (nb_found and \
'\n<opensearch:totalResults>%i</opensearch:totalResults>' % nb_found) or '',
'start_index': (jrec and \
'\n<opensearch:startIndex>%i</opensearch:startIndex>' % jrec) or '',
'items_per_page': (rg and \
'\n<opensearch:itemsPerPage>%i</opensearch:itemsPerPage>' % rg) or '',
'rss_title': title,
'rss_description': description
}
return out
def tmpl_xml_rss_epilogue(self):
"""Creates XML RSS 2.0 epilogue."""
out = """\
</channel>
</rss>\n"""
return out
def tmpl_xml_podcast_prologue(self, current_url=None,
previous_url=None, next_url=None,
first_url=None, last_url=None,
nb_found=None, jrec=None, rg=None, cc=None):
"""Creates XML podcast prologue."""
title = CFG_SITE_NAME
description = '%s latest documents' % CFG_SITE_NAME
if CFG_CERN_SITE:
title = 'CERN'
description = 'CERN latest documents'
if cc and cc != CFG_SITE_NAME:
title += ': ' + cgi.escape(cc)
description += ' in ' + cgi.escape(cc)
out = """<rss xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" version="2.0">
<channel>
<title>%(podcast_title)s</title>
<link>%(siteurl)s</link>
<description>%(podcast_description)s</description>
<language>%(sitelang)s</language>
<pubDate>%(timestamp)s</pubDate>
<category></category>
<generator>Invenio %(version)s</generator>
<webMaster>%(siteadminemail)s</webMaster>
<ttl>%(timetolive)s</ttl>%(previous_link)s%(next_link)s%(current_link)s
<image>
<url>%(siteurl)s/img/site_logo_rss.png</url>
<title>%(sitename)s</title>
<link>%(siteurl)s</link>
</image>
<itunes:owner>
<itunes:email>%(siteadminemail)s</itunes:email>
</itunes:owner>
""" % {'sitename': CFG_SITE_NAME,
'siteurl': CFG_SITE_URL,
'sitelang': CFG_SITE_LANG,
'siteadminemail': CFG_SITE_ADMIN_EMAIL,
'timestamp': time.strftime("%a, %d %b %Y %H:%M:%S GMT", time.gmtime()),
'version': CFG_VERSION,
'sitesupportemail': CFG_SITE_SUPPORT_EMAIL,
'timetolive': CFG_WEBSEARCH_RSS_TTL,
'current_link': (current_url and \
'\n<atom:link rel="self" href="%s" />\n' % current_url) or '',
'previous_link': (previous_url and \
'\n<atom:link rel="previous" href="%s" />' % previous_url) or '',
'next_link': (next_url and \
'\n<atom:link rel="next" href="%s" />' % next_url) or '',
'first_link': (first_url and \
'\n<atom:link rel="first" href="%s" />' % first_url) or '',
'last_link': (last_url and \
'\n<atom:link rel="last" href="%s" />' % last_url) or '',
'podcast_title': title,
'podcast_description': description
}
return out
def tmpl_xml_podcast_epilogue(self):
"""Creates XML podcast epilogue."""
out = """\n</channel>
</rss>\n"""
return out
def tmpl_xml_nlm_prologue(self):
"""Creates XML NLM prologue."""
out = """<articles>\n"""
return out
def tmpl_xml_nlm_epilogue(self):
"""Creates XML NLM epilogue."""
out = """\n</articles>"""
return out
def tmpl_xml_refworks_prologue(self):
"""Creates XML RefWorks prologue."""
out = """<references>\n"""
return out
def tmpl_xml_refworks_epilogue(self):
"""Creates XML RefWorks epilogue."""
out = """\n</references>"""
return out
def tmpl_xml_endnote_prologue(self):
"""Creates XML EndNote prologue."""
out = """<xml>\n<records>\n"""
return out
def tmpl_xml_endnote_8x_prologue(self):
"""Creates XML EndNote prologue."""
out = """<records>\n"""
return out
def tmpl_xml_endnote_epilogue(self):
"""Creates XML EndNote epilogue."""
out = """\n</records>\n</xml>"""
return out
def tmpl_xml_endnote_8x_epilogue(self):
"""Creates XML EndNote epilogue."""
out = """\n</records>"""
return out
def tmpl_xml_marc_prologue(self):
"""Creates XML MARC prologue."""
out = """<collection xmlns="http://www.loc.gov/MARC21/slim">\n"""
return out
def tmpl_xml_marc_epilogue(self):
"""Creates XML MARC epilogue."""
out = """\n</collection>"""
return out
def tmpl_xml_mods_prologue(self):
"""Creates XML MODS prologue."""
out = """<modsCollection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"\n
xsi:schemaLocation="http://www.loc.gov/mods/v3\n
http://www.loc.gov/standards/mods/v3/mods-3-3.xsd">\n"""
return out
def tmpl_xml_mods_epilogue(self):
"""Creates XML MODS epilogue."""
out = """\n</modsCollection>"""
return out
def tmpl_xml_default_prologue(self):
"""Creates XML default format prologue. (Sanity calls only.)"""
out = """<collection>\n"""
return out
def tmpl_xml_default_epilogue(self):
"""Creates XML default format epilogue. (Sanity calls only.)"""
out = """\n</collection>"""
return out
def tmpl_collection_not_found_page_title(self, colname, ln=CFG_SITE_LANG):
"""
Create page title for cases when unexisting collection was asked for.
"""
_ = gettext_set_language(ln)
out = _("Collection %(x_name)s Not Found", x_name=cgi.escape(colname))
return out
def tmpl_collection_not_found_page_body(self, colname, ln=CFG_SITE_LANG):
"""
Create page body for cases when unexisting collection was asked for.
"""
_ = gettext_set_language(ln)
out = """<h1>%(title)s</h1>
<p>%(sorry)s</p>
<p>%(you_may_want)s</p>
""" % { 'title': self.tmpl_collection_not_found_page_title(colname, ln),
'sorry': _("Sorry, collection %(x_name)s does not seem to exist.",
x_name=('<strong>' + cgi.escape(colname) + '</strong>')),
'you_may_want': _("You may want to start browsing from %(x_name)s.",
x_name=('<a href="' + CFG_SITE_URL + '?ln=' + ln + '">' + cgi.escape(CFG_SITE_NAME_INTL.get(ln, CFG_SITE_NAME)) + '</a>'))}
return out
def tmpl_alert_rss_teaser_box_for_query(self, id_query, ln, display_email_alert_part=True):
"""Propose teaser for setting up this query as alert or RSS feed.
Parameters:
- 'id_query' *int* - ID of the query we make teaser for
- 'ln' *string* - The language to display
- 'display_email_alert_part' *bool* - whether to display email alert part
"""
# load the right message language
_ = gettext_set_language(ln)
# get query arguments:
res = run_sql("SELECT urlargs FROM query WHERE id=%s", (id_query,))
argd = {}
if res:
argd = cgi.parse_qs(res[0][0])
rssurl = self.build_rss_url(argd)
alerturl = CFG_BASE_URL + '/youralerts/input?ln=%s&amp;idq=%s' % (ln, id_query)
if display_email_alert_part:
msg_alert = _("""Set up a personal %(x_url1_open)semail alert%(x_url1_close)s
or subscribe to the %(x_url2_open)sRSS feed%(x_url2_close)s.""") % \
{'x_url1_open': '<a href="%s"><img src="%s/img/mail-icon-12x8.gif" border="0" alt="" /></a> ' % (alerturl, CFG_BASE_URL) + ' <a class="google" href="%s">' % (alerturl),
'x_url1_close': '</a>',
'x_url2_open': '<a href="%s"><img src="%s/img/feed-icon-12x12.gif" border="0" alt="" /></a> ' % (rssurl, CFG_BASE_URL) + ' <a class="google" href="%s">' % rssurl,
'x_url2_close': '</a>', }
else:
msg_alert = _("""Subscribe to the %(x_url2_open)sRSS feed%(x_url2_close)s.""") % \
{'x_url2_open': '<a href="%s"><img src="%s/img/feed-icon-12x12.gif" border="0" alt="" /></a> ' % (rssurl, CFG_BASE_URL) + ' <a class="google" href="%s">' % rssurl,
'x_url2_close': '</a>', }
out = '''<a name="googlebox"></a>
<table class="googlebox"><tr><th class="googleboxheader">%(similar)s</th></tr>
<tr><td class="googleboxbody">%(msg_alert)s</td></tr>
</table>
''' % {
'similar' : _("Interested in being notified about new results for this query?"),
'msg_alert': msg_alert, }
return out
def tmpl_detailed_record_metadata(self, recID, ln, format,
content,
creationdate=None,
modificationdate=None):
"""Returns the main detailed page of a record
Parameters:
- 'recID' *int* - The ID of the printed record
- 'ln' *string* - The language to display
- 'format' *string* - The format in used to print the record
- 'content' *string* - The main content of the page
- 'creationdate' *string* - The creation date of the printed record
- 'modificationdate' *string* - The last modification date of the printed record
"""
_ = gettext_set_language(ln)
## unAPI identifier
out = '<abbr class="unapi-id" title="%s"></abbr>\n' % recID
out += content
return out
def tmpl_display_back_to_search(self, req, recID, ln):
"""
Displays next-hit/previous-hit/back-to-search links
on the detailed record pages in order to be able to quickly
flip between detailed record pages
@param req: Apache request object
@type req: Apache request object
@param recID: detailed record ID
@type recID: int
@param ln: language of the page
@type ln: string
@return: html output
@rtype: html
"""
_ = gettext_set_language(ln)
# this variable is set to zero and then, nothing is displayed
if not CFG_WEBSEARCH_PREV_NEXT_HIT_LIMIT:
return ''
# this variable is set to zero and then nothing is saved in the previous session
if not CFG_WEBSEARCH_PREV_NEXT_HIT_FOR_GUESTS:
return ''
# search for a specific record having not done any search before
wlq = session_param_get(req, 'websearch-last-query', '')
wlqh = session_param_get(req, 'websearch-last-query-hits')
out = '''<br/><br/><div align="right">'''
# excedeed limit CFG_WEBSEARCH_PREV_NEXT_HIT_LIMIT,
# then will be displayed only the back to search link
if wlqh is None:
out += '''<div style="padding-bottom:2px;padding-top:30px;"><span class="moreinfo" style="margin-right:10px;">
%(back)s </span></div></div>''' % \
{'back': create_html_link(wlq, {}, _("Back to search"), {'class': "moreinfo"})}
return out
# let's look for the recID's collection
record_found = False
for coll in wlqh:
if recID in coll:
record_found = True
coll_recID = coll
break
# let's calculate lenght of recID's collection
if record_found:
recIDs = coll_recID[::-1]
totalrec = len(recIDs)
# search for a specific record having not done any search before
else:
return ''
# if there is only one hit,
# to show only the "back to search" link
if totalrec == 1:
# to go back to the last search results page
out += '''<div style="padding-bottom:2px;padding-top:30px;"><span class="moreinfo" style="margin-right:10px;">
%(back)s </span></div></div>''' % \
{'back': create_html_link(wlq, {}, _("Back to search"), {'class': "moreinfo"})}
elif totalrec > 1:
pos = recIDs.index(recID)
numrec = pos + 1
if pos == 0:
recIDnext = recIDs[pos + 1]
recIDlast = recIDs[totalrec - 1]
# to display only next and last links
out += '''<div><span class="moreinfo" style="margin-right:10px;">
%(numrec)s %(totalrec)s %(next)s %(last)s </span></div> ''' % {
'numrec': _("%(x_name)s of", x_name=('<strong>' + self.tmpl_nice_number(numrec, ln) + '</strong>')),
'totalrec': ("%s") % ('<strong>' + self.tmpl_nice_number(totalrec, ln) + '</strong>'),
'next': create_html_link(self.build_search_url(recid=recIDnext, ln=ln),
{}, ('<font size="4">&rsaquo;</font>'), {'class': "moreinfo"}),
'last': create_html_link(self.build_search_url(recid=recIDlast, ln=ln),
{}, ('<font size="4">&raquo;</font>'), {'class': "moreinfo"})}
elif pos == totalrec - 1:
recIDfirst = recIDs[0]
recIDprev = recIDs[pos - 1]
# to display only first and previous links
out += '''<div style="padding-top:30px;"><span class="moreinfo" style="margin-right:10px;">
%(first)s %(previous)s %(numrec)s %(totalrec)s</span></div>''' % {
'first': create_html_link(self.build_search_url(recid=recIDfirst, ln=ln),
{}, ('<font size="4">&laquo;</font>'), {'class': "moreinfo"}),
'previous': create_html_link(self.build_search_url(recid=recIDprev, ln=ln),
{}, ('<font size="4">&lsaquo;</font>'), {'class': "moreinfo"}),
'numrec': _("%(x_name)s of", x_name=('<strong>' + self.tmpl_nice_number(numrec, ln) + '</strong>')),
'totalrec': ("%s") % ('<strong>' + self.tmpl_nice_number(totalrec, ln) + '</strong>')}
else:
# to display all links
recIDfirst = recIDs[0]
recIDprev = recIDs[pos - 1]
recIDnext = recIDs[pos + 1]
recIDlast = recIDs[len(recIDs) - 1]
out += '''<div style="padding-top:30px;"><span class="moreinfo" style="margin-right:10px;">
%(first)s %(previous)s
%(numrec)s %(totalrec)s %(next)s %(last)s </span></div>''' % {
'first': create_html_link(self.build_search_url(recid=recIDfirst, ln=ln),
{}, ('<font size="4">&laquo;</font>'),
{'class': "moreinfo"}),
'previous': create_html_link(self.build_search_url(recid=recIDprev, ln=ln),
{}, ('<font size="4">&lsaquo;</font>'), {'class': "moreinfo"}),
'numrec': _("%(x_name)s of", x_name=('<strong>' + self.tmpl_nice_number(numrec, ln) + '</strong>')),
'totalrec': ("%s") % ('<strong>' + self.tmpl_nice_number(totalrec, ln) + '</strong>'),
'next': create_html_link(self.build_search_url(recid=recIDnext, ln=ln),
{}, ('<font size="4">&rsaquo;</font>'), {'class': "moreinfo"}),
'last': create_html_link(self.build_search_url(recid=recIDlast, ln=ln),
{}, ('<font size="4">&raquo;</font>'), {'class': "moreinfo"})}
out += '''<div style="padding-bottom:2px;"><span class="moreinfo" style="margin-right:10px;">
%(back)s </span></div></div>''' % {
'back': create_html_link(wlq, {}, _("Back to search"), {'class': "moreinfo"})}
return out
def tmpl_record_hepdata(self, data, recid, isLong=True):
""" Generate a page for HepData records
"""
from invenio import hepdatautils
from invenio.search_engine import get_fieldvalues
c = []
c.append("<div style=\"background-color: #ececec;\">")
flag_hepdata = 0
flag_dataverse = 0
for dataset in data.datasets:
try:
publisher = get_fieldvalues(dataset.recid, '520__9')[0]
except IndexError:
from invenio.hepdatautils import create_hepdata_ticket
create_hepdata_ticket(dataset.recid, 'Data missing in 520__9')
continue
if publisher == "HEPDATA" and flag_hepdata == 0:
flag_hepdata = 1
elif publisher == "Dataverse":
flag_dataverse = 1
if flag_hepdata == 1 or flag_dataverse == 1:
c.append("<h3> This data comes from ")
if flag_hepdata == 1:
c.append('<a href="http://hepdata.cedar.ac.uk/view/ins%s" target="_blank"> Durham HepData project </a>' % (recid))
if flag_hepdata == 1 and flag_dataverse == 1:
c.append(' and ')
if flag_dataverse == 1:
c.append('<a href="http://thedata.harvard.edu/"> Dataverse </a>')
c.append('</h3>')
c.append("<div style=\"background-color: #ececec;\">")
if data.comment:
c.append("<h3> Summary:</h3>")
c.append("""<div class="hepdataSummary">%s</div>""" % (data.comment, ))
if data.systematics and data.systematics.strip() != "":
c.append("<h3>Systematic data: </h3>")
c.append(data.systematics)
c.append("</div>")
if data.additional_data_links:
c.append("<h3>Additional data:</h3>")
for link in data.additional_data_links:
if "href" in link and "description" in link:
c.append("<a href=\"%s/%s\">%s</a><br>" % (CFG_HEPDATA_URL, link["href"], link["description"]))
c.append("<h3> Datasets:</h3>")
seq = 0
for dataset in data.datasets:
seq += 1
try:
publisher = get_fieldvalues(dataset.recid, '520__9')[0]
except IndexError:
from invenio.hepdatautils import create_hepdata_ticket
create_hepdata_ticket(dataset.recid, 'Data missing in 520__9')
continue
if publisher == "HEPDATA":
c.append(hepdatadisplayutils.render_hepdata_dataset_html(dataset, recid, seq))
elif publisher == "Dataverse":
c.append(hepdatadisplayutils.render_dataverse_dataset_html(dataset.recid))
elif publisher == "INSPIRE":
c.append(hepdatadisplayutils.render_inspire_dataset_html(dataset.recid))
else:
c.append(hepdatadisplayutils.render_other_dataset_html(dataset.recid))
c.append("</div>")
return "\n".join(c)
def tmpl_record_no_hepdata(self):
return "This record does not have HEP data associated"
def tmpl_record_plots(self, recID, ln):
"""
Displays little tables containing the images and captions contained in the specified document.
Parameters:
- 'recID' *int* - The ID of the printed record
- 'ln' *string* - The language to display
"""
from invenio.legacy.search_engine import get_record
from invenio.legacy.bibrecord import field_get_subfield_values
from invenio.legacy.bibrecord import record_get_field_instances
_ = gettext_set_language(ln)
out = ''
rec = get_record(recID)
flds = record_get_field_instances(rec, '856', '4')
images = []
for fld in flds:
image = field_get_subfield_values(fld, 'u')
caption = field_get_subfield_values(fld, 'y')
data_urls = field_get_subfield_values(fld, 'z')
if type(data_urls) == list and len(data_urls) > 0:
data_urls = str(data_urls[0])
if data_urls.startswith("HEPDATA:"):
data_urls = data_urls[8:].split(";")
else:
data_urls = []
if type(image) == list and len(image) > 0:
image = image[0]
else:
continue
if type(caption) == list and len(caption) > 0:
caption = caption[0]
else:
continue
if not image.endswith('.png'):
# huh?
continue
if len(caption) >= 5:
images.append((int(caption[:5]), image, caption[5:], data_urls))
else:
# we don't have any idea of the order... just put it on
images.append(99999, image, caption, data_urls)
images = sorted(images, key=lambda x: x[0])
for (index, image, caption, data_urls) in images:
# let's put everything in nice little subtables with the image
# next to the caption
data_string_list = []
seq_num = 1
for data_url in data_urls:
val = ""
if len(data_urls) > 1:
val = " %i" % seq_num
data_string_list.append("<br><a href=\"%s\">Data%s</a>" % (str(data_url), val))
seq_num += 1
data_string = "".join(data_string_list)
out = out + '<table width="95%" style="display: inline;">' + \
'<tr><td width="66%"><a name="' + str(index) + '" ' + \
'href="' + image + '">' + \
'<img src="' + image + '" width="95%"/></a></td>' + \
'<td width="33%">' + caption + data_string + '</td></tr>' + \
'</table>'
out = out + '<br /><br />'
return out
def tmpl_detailed_record_statistics(self, recID, ln,
downloadsimilarity,
downloadhistory, viewsimilarity):
"""Returns the statistics page of a record
Parameters:
- 'recID' *int* - The ID of the printed record
- 'ln' *string* - The language to display
- downloadsimilarity *string* - downloadsimilarity box
- downloadhistory *string* - downloadhistory box
- viewsimilarity *string* - viewsimilarity box
"""
# load the right message language
_ = gettext_set_language(ln)
out = ''
if CFG_BIBRANK_SHOW_DOWNLOAD_STATS and downloadsimilarity is not None:
similar = self.tmpl_print_record_list_for_similarity_boxen (
_("People who downloaded this document also downloaded:"), downloadsimilarity, ln)
out = '<table>'
out += '''
<tr><td>%(graph)s</td></tr>
<tr><td>%(similar)s</td></tr>
''' % { 'siteurl': CFG_BASE_URL, 'recid': recID, 'ln': ln,
'similar': similar, 'more': _("more"),
'graph': downloadsimilarity
}
out += '</table>'
out += '<br />'
if CFG_BIBRANK_SHOW_READING_STATS and viewsimilarity is not None:
out += self.tmpl_print_record_list_for_similarity_boxen (
_("People who viewed this page also viewed:"), viewsimilarity, ln)
if CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS and downloadhistory is not None:
out += downloadhistory + '<br />'
return out
def tmpl_detailed_record_citations_prologue(self, recID, ln):
"""Returns the prologue of the citations page of a record
Parameters:
- 'recID' *int* - The ID of the printed record
- 'ln' *string* - The language to display
"""
return '<table>'
def tmpl_detailed_record_citations_epilogue(self, recID, ln):
"""Returns the epilogue of the citations page of a record
Parameters:
- 'recID' *int* - The ID of the printed record
- 'ln' *string* - The language to display
"""
return '</table>'
def tmpl_detailed_record_citations_citing_list(self, recID, ln,
citinglist,
sf='', so='d', sp='', rm=''):
"""Returns the list of record citing this one
Parameters:
- 'recID' *int* - The ID of the printed record
- 'ln' *string* - The language to display
- citinglist *list* - a list of tuples [(x1,y1),(x2,y2),..] where x is doc id and y is number of citations
"""
# load the right message language
_ = gettext_set_language(ln)
out = ''
if CFG_BIBRANK_SHOW_CITATION_STATS and citinglist is not None:
similar = self.tmpl_print_record_list_for_similarity_boxen(
_("Cited by: %(x_num)s records", x_num=len(citinglist)), citinglist, ln)
out += '''
<tr><td>
%(similar)s&nbsp;%(more)s
<br /><br />
</td></tr>''' % {
'more': create_html_link(
self.build_search_url(p='refersto:recid:%d' % recID, #XXXX
sf=sf,
so=so,
sp=sp,
rm=rm,
ln=ln),
{}, _("more")),
'similar': similar}
return out
def tmpl_detailed_record_citations_citation_history(self, ln,
citationhistory):
"""Returns the citations history graph of this record
Parameters:
- 'recID' *int* - The ID of the printed record
- 'ln' *string* - The language to display
- citationhistory *string* - citationhistory box
"""
# load the right message language
_ = gettext_set_language(ln)
out = ''
if CFG_BIBRANK_SHOW_CITATION_GRAPHS and citationhistory is not None:
out = '<!--citation history--><tr><td>%s</td></tr>' % citationhistory
else:
out = "<!--not showing citation history. CFG_BIBRANK_SHOW_CITATION_GRAPHS:"
out += str(CFG_BIBRANK_SHOW_CITATION_GRAPHS) + " citationhistory "
if citationhistory:
out += str(len(citationhistory)) + "-->"
else:
out += "no citationhistory -->"
return out
def tmpl_detailed_record_citations_citation_log(self, ln, log_entries):
"""Returns the citations history graph of this record
Parameters:
- 'recID' *int* - The ID of the printed record
- 'ln' *string* - The language to display
- citationhistory *string* - citationhistory box
"""
# load the right message language
_ = gettext_set_language(ln)
out = []
if log_entries:
out.append('<style>td.citationlogdate { width: 5.4em; }</style>')
out.append('<table><tr><td class="blocknote">Citation Log: </td></tr><tr><td><a id="citationlogshow" class="moreinfo" style="text-decoration: underline; " onclick="$(\'#citationlog\').show(); $(\'#citationlogshow\').hide();">show</a></td></tr></table>')
out.append('<table id="citationlog" style="display: none;">')
for recid, action_type, action_date in log_entries:
if record_exists(recid) == 1:
record_str = format_record(recid, 'HS2')
else:
record_str = 'The record with id %s was deleted' % recid
out.append("""<tr>
<td>%s</td>
<td class="citationlogdate">%s</td>
<td>%s</td>
</tr>""" % (action_type, action_date.strftime('%Y-%m-%d'), record_str))
out.append('</table>')
return '\n'.join(out)
def tmpl_detailed_record_citations_co_citing(self, recID, ln,
cociting):
"""Returns the list of cocited records
Parameters:
- 'recID' *int* - The ID of the printed record
- 'ln' *string* - The language to display
- cociting *string* - cociting box
"""
# load the right message language
_ = gettext_set_language(ln)
out = ''
if CFG_BIBRANK_SHOW_CITATION_STATS and cociting is not None:
similar = self.tmpl_print_record_list_for_similarity_boxen (
_("Co-cited with: %(x_num)s records", x_num=len (cociting)), cociting, ln)
out = '''
<tr><td>
%(similar)s&nbsp;%(more)s
<br />
</td></tr>''' % { 'more': create_html_link(self.build_search_url(p='cocitedwith:%d' % recID, ln=ln),
{}, _("more")),
'similar': similar }
return out
def tmpl_detailed_record_citations_self_cited(self, recID, ln,
selfcited, citinglist):
"""Returns the list of self-citations for this record
Parameters:
- 'recID' *int* - The ID of the printed record
- 'ln' *string* - The language to display
- selfcited list - a list of self-citations for recID
"""
# load the right message language
_ = gettext_set_language(ln)
out = ''
if CFG_BIBRANK_SHOW_CITATION_GRAPHS and selfcited is not None:
sc_scorelist = [] #a score list for print..
for s in selfcited:
#copy weight from citations
weight = 0
for c in citinglist:
(crec, score) = c
if crec == s:
weight = score
tmp = [s, weight]
sc_scorelist.append(tmp)
scite = self.tmpl_print_record_list_for_similarity_boxen (
_(".. of which self-citations: %(x_rec)s records", x_rec=len (selfcited)), sc_scorelist, ln)
out = '<tr><td>' + scite + '</td></tr>'
return out
def tmpl_author_information(self, req, pubs, authorname, num_downloads,
aff_pubdict, citedbylist, kwtuples, authors,
vtuples, names_dict, person_link,
bibauthorid_data, ln, return_html=False):
"""Prints stuff about the author given as authorname.
1. Author name + his/her institutes. Each institute I has a link
to papers where the auhtor has I as institute.
2. Publications, number: link to search by author.
3. Keywords
4. Author collabs
5. Publication venues like journals
The parameters are data structures needed to produce 1-6, as follows:
req - request
pubs - list of recids, probably the records that have the author as an author
authorname - evident
num_downloads - evident
aff_pubdict - a dictionary where keys are inst names and values lists of recordids
citedbylist - list of recs that cite pubs
kwtuples - keyword tuples like ('HIGGS BOSON',[3,4]) where 3 and 4 are recids
authors - a list of authors that have collaborated with authorname
names_dict - a dict of {name: frequency}
"""
from invenio.legacy.search_engine import perform_request_search
from operator import itemgetter
_ = gettext_set_language(ln)
ib_pubs = intbitset(pubs)
html = []
# construct an extended search as an interim solution for author id
# searches. Will build "(exactauthor:v1 OR exactauthor:v2)" strings
# extended_author_search_str = ""
# if bibauthorid_data["is_baid"]:
# if len(names_dict.keys()) > 1:
# extended_author_search_str = '('
#
# for name_index, name_query in enumerate(names_dict.keys()):
# if name_index > 0:
# extended_author_search_str += " OR "
#
# extended_author_search_str += 'exactauthor:"' + name_query + '"'
#
# if len(names_dict.keys()) > 1:
# extended_author_search_str += ')'
# rec_query = 'exactauthor:"' + authorname + '"'
#
# if bibauthorid_data["is_baid"] and extended_author_search_str:
# rec_query = extended_author_search_str
baid_query = ""
extended_author_search_str = ""
if 'is_baid' in bibauthorid_data and bibauthorid_data['is_baid']:
if bibauthorid_data["cid"]:
baid_query = 'author:%s' % bibauthorid_data["cid"]
elif bibauthorid_data["pid"] > -1:
baid_query = 'author:%s' % bibauthorid_data["pid"]
## todo: figure out if the author index is filled with pids/cids.
## if not: fall back to exactauthor search.
# if not index:
# baid_query = ""
if not baid_query:
baid_query = 'exactauthor:"' + authorname + '"'
if bibauthorid_data['is_baid']:
if len(names_dict.keys()) > 1:
extended_author_search_str = '('
for name_index, name_query in enumerate(names_dict.keys()):
if name_index > 0:
extended_author_search_str += " OR "
extended_author_search_str += 'exactauthor:"' + name_query + '"'
if len(names_dict.keys()) > 1:
extended_author_search_str += ')'
if bibauthorid_data['is_baid'] and extended_author_search_str:
baid_query = extended_author_search_str
baid_query = baid_query + " "
sorted_names_list = sorted(iteritems(names_dict), key=itemgetter(1),
reverse=True)
# Prepare data for display
# construct names box
header = "<strong>" + _("Name variants") + "</strong>"
content = []
for name, frequency in sorted_names_list:
prquery = baid_query + ' exactauthor:"' + name + '"'
name_lnk = create_html_link(self.build_search_url(p=prquery),
{},
str(frequency),)
content.append("%s (%s)" % (name, name_lnk))
if not content:
content = [_("No Name Variants")]
names_box = self.tmpl_print_searchresultbox(header, "<br />\n".join(content))
# construct papers box
rec_query = baid_query
searchstr = create_html_link(self.build_search_url(p=rec_query),
{}, "<strong>" + "All papers (" + str(len(pubs)) + ")" + "</strong>",)
line1 = "<strong>" + _("Papers") + "</strong>"
line2 = searchstr
if CFG_BIBRANK_SHOW_DOWNLOAD_STATS and num_downloads:
line2 += " (" + _("downloaded") + " "
line2 += str(num_downloads) + " " + _("times") + ")"
if CFG_INSPIRE_SITE:
CFG_COLLS = ['Book',
'Conference',
'Introductory',
'Lectures',
'Preprint',
'Published',
'Review',
'Thesis']
else:
CFG_COLLS = ['Article',
'Book',
'Preprint', ]
collsd = {}
for coll in CFG_COLLS:
coll_papers = list(ib_pubs & intbitset(perform_request_search(f="collection", p=coll)))
if coll_papers:
collsd[coll] = coll_papers
colls = collsd.keys()
colls.sort(lambda x, y: cmp(len(collsd[y]), len(collsd[x]))) # sort by number of papers
for coll in colls:
rec_query = baid_query + 'collection:' + coll
line2 += "<br />" + create_html_link(self.build_search_url(p=rec_query),
{}, coll + " (" + str(len(collsd[coll])) + ")",)
if not pubs:
line2 = _("No Papers")
papers_box = self.tmpl_print_searchresultbox(line1, line2)
#make a authoraff string that looks like CERN (1), Caltech (2) etc
authoraff = ""
aff_pubdict_keys = aff_pubdict.keys()
aff_pubdict_keys.sort(lambda x, y: cmp(len(aff_pubdict[y]), len(aff_pubdict[x])))
if aff_pubdict_keys:
for a in aff_pubdict_keys:
print_a = a
if (print_a == ' '):
print_a = _("unknown affiliation")
if authoraff:
authoraff += '<br>'
authoraff += create_html_link(self.build_search_url(p=' or '.join(["%s" % x for x in aff_pubdict[a]]),
f='recid'),
{}, print_a + ' (' + str(len(aff_pubdict[a])) + ')',)
else:
authoraff = _("No Affiliations")
line1 = "<strong>" + _("Affiliations") + "</strong>"
line2 = authoraff
affiliations_box = self.tmpl_print_searchresultbox(line1, line2)
# print frequent keywords:
keywstr = ""
if (kwtuples):
for (kw, freq) in kwtuples:
if keywstr:
keywstr += '<br>'
rec_query = baid_query + 'keyword:"' + kw + '"'
searchstr = create_html_link(self.build_search_url(p=rec_query),
{}, kw + " (" + str(freq) + ")",)
keywstr = keywstr + " " + searchstr
else:
keywstr += _('No Keywords')
line1 = "<strong>" + _("Frequent keywords") + "</strong>"
line2 = keywstr
keyword_box = self.tmpl_print_searchresultbox(line1, line2)
header = "<strong>" + _("Frequent co-authors") + "</strong>"
content = []
sorted_coauthors = sorted(sorted(iteritems(authors), key=itemgetter(0)),
key=itemgetter(1), reverse=True)
for name, frequency in sorted_coauthors:
rec_query = baid_query + 'exactauthor:"' + name + '"'
lnk = create_html_link(self.build_search_url(p=rec_query), {}, "%s (%s)" % (name, frequency),)
content.append("%s" % lnk)
if not content:
content = [_("No Frequent Co-authors")]
coauthor_box = self.tmpl_print_searchresultbox(header, "<br />\n".join(content))
pubs_to_papers_link = create_html_link(self.build_search_url(p=baid_query), {}, str(len(pubs)))
display_name = ""
try:
display_name = sorted_names_list[0][0]
except IndexError:
display_name = "&nbsp;"
headertext = ('<h1>%s <span style="font-size:50%%;">(%s papers)</span></h1>'
% (display_name, pubs_to_papers_link))
if return_html:
html.append(headertext)
else:
req.write(headertext)
#req.write("<h1>%s</h1>" % (authorname))
if person_link:
cmp_link = ('<div><a href="%s/author/claim/claimstub?person=%s">%s</a></div>'
% (CFG_SITE_URL, person_link,
_("This is me. Verify my publication list.")))
if return_html:
html.append(cmp_link)
else:
req.write(cmp_link)
if return_html:
html.append("<table width=80%><tr valign=top><td>")
html.append(names_box)
html.append("<br />")
html.append(papers_box)
html.append("<br />")
html.append(keyword_box)
html.append("</td>")
html.append("<td>&nbsp;</td>")
html.append("<td>")
html.append(affiliations_box)
html.append("<br />")
html.append(coauthor_box)
html.append("</td></tr></table>")
else:
req.write("<table width=80%><tr valign=top><td>")
req.write(names_box)
req.write("<br />")
req.write(papers_box)
req.write("<br />")
req.write(keyword_box)
req.write("</td>")
req.write("<td>&nbsp;</td>")
req.write("<td>")
req.write(affiliations_box)
req.write("<br />")
req.write(coauthor_box)
req.write("</td></tr></table>")
# print citations:
rec_query = baid_query
if len(citedbylist):
line1 = "<strong>" + _("Citations:") + "</strong>"
line2 = ""
if not pubs:
line2 = _("No Citation Information available")
sr_box = self.tmpl_print_searchresultbox(line1, line2)
if return_html:
html.append(sr_box)
else:
req.write(sr_box)
if return_html:
return "\n".join(html)
# print frequent co-authors:
# collabstr = ""
# if (authors):
# for c in authors:
# c = c.strip()
# if collabstr:
# collabstr += '<br>'
# #do not add this person him/herself in the list
# cUP = c.upper()
# authornameUP = authorname.upper()
# if not cUP == authornameUP:
# commpubs = intbitset(pubs) & intbitset(perform_request_search(p="exactauthor:\"%s\" exactauthor:\"%s\"" % (authorname, c)))
# collabstr = collabstr + create_html_link(self.build_search_url(p='exactauthor:"' + authorname + '" exactauthor:"' + c + '"'),
# {}, c + " (" + str(len(commpubs)) + ")",)
# else: collabstr += 'None'
# banner = self.tmpl_print_searchresultbox("<strong>" + _("Frequent co-authors:") + "</strong>", collabstr)
# print frequently publishes in journals:
#if (vtuples):
# pubinfo = ""
# for t in vtuples:
# (journal, num) = t
# pubinfo += create_html_link(self.build_search_url(p='exactauthor:"' + authorname + '" ' + \
# 'journal:"' + journal + '"'),
# {}, journal + " ("+str(num)+")<br/>")
# banner = self.tmpl_print_searchresultbox("<strong>" + _("Frequently publishes in:") + "<strong>", pubinfo)
# req.write(banner)
def tmpl_detailed_record_references(self, recID, ln, content):
"""Returns the discussion page of a record
Parameters:
- 'recID' *int* - The ID of the printed record
- 'ln' *string* - The language to display
- 'content' *string* - The main content of the page
"""
# load the right message language
out = ''
if content is not None:
out += content
return out
def tmpl_citesummary_title(self, ln=CFG_SITE_LANG):
"""HTML citesummary title and breadcrumbs
A part of HCS format suite."""
return ''
def tmpl_citesummary2_title(self, searchpattern, ln=CFG_SITE_LANG):
"""HTML citesummary title and breadcrumbs
A part of HCS2 format suite."""
return ''
def tmpl_citesummary_back_link(self, searchpattern, ln=CFG_SITE_LANG):
"""HTML back to citesummary link
A part of HCS2 format suite."""
_ = gettext_set_language(ln)
out = ''
params = {'ln': 'en',
'p': quote(searchpattern),
'of': 'hcs'}
msg = _('Back to citesummary')
url = CFG_SITE_URL + '/search?' + \
'&'.join(['='.join(i) for i in iteritems(params)])
out += '<p><a href="%(url)s">%(msg)s</a></p>' % {'url': url, 'msg': msg}
return out
def tmpl_citesummary_more_links(self, searchpattern, ln=CFG_SITE_LANG):
_ = gettext_set_language(ln)
out = ''
msg = '<p><a href="%(url)s">%(msg)s</a></p>'
params = {'ln': ln,
'p': quote(searchpattern),
'of': 'hcs2'}
url = CFG_SITE_URL + '/search?' + \
'&amp;'.join(['='.join(i) for i in iteritems(params)])
out += msg % {'url': url,
'msg': _('Exclude self-citations')}
return out
def tmpl_citesummary_prologue(self, coll_recids, collections, search_patterns,
searchfield, citable_recids, total_count,
ln=CFG_SITE_LANG):
"""HTML citesummary format, prologue. A part of HCS format suite."""
_ = gettext_set_language(ln)
out = """<table id="citesummary">
<tr>
<td>
<strong class="headline">%(msg_title)s</strong>
</td>""" % \
{'msg_title': _("Citation summary results"), }
for coll, dummy in collections:
out += '<td align="right">%s</td>' % _(coll)
out += '</tr>'
out += """<tr><td><strong>%(msg_recs)s</strong></td>""" % \
{'msg_recs': _("Total number of papers analyzed:"), }
for coll, colldef in collections:
link_url = CFG_BASE_URL + '/search?p='
if search_patterns[coll]:
p = search_patterns[coll]
if searchfield:
if " " in p:
p = searchfield + ':"' + p + '"'
else:
p = searchfield + ':' + p
link_url += quote(p)
if colldef:
link_url += '%20AND%20' + quote(colldef)
link_text = self.tmpl_nice_number(len(coll_recids[coll]), ln)
out += '<td align="right"><a href="%s">%s</a></td>' % (link_url,
link_text)
out += '</tr>'
return out
def tmpl_citesummary_overview(self, collections, d_total_cites,
d_avg_cites, ln=CFG_SITE_LANG):
"""HTML citesummary format, overview. A part of HCS format suite."""
_ = gettext_set_language(ln)
out = """<tr><td><strong>%(msg_cites)s</strong></td>""" % \
{'msg_cites': _("Total number of citations:"), }
for coll, dummy in collections:
total_cites = d_total_cites[coll]
out += '<td align="right">%s</td>' % \
self.tmpl_nice_number(total_cites, ln)
out += '</tr>'
out += """<tr><td><strong>%(msg_avgcit)s</strong></td>""" % \
{'msg_avgcit': _("Average citations per paper:"), }
for coll, dummy in collections:
avg_cites = d_avg_cites[coll]
out += '<td align="right">%.1f</td>' % avg_cites
out += '</tr>'
return out
def tmpl_citesummary_minus_self_cites(self, d_total_cites, d_avg_cites,
ln=CFG_SITE_LANG):
"""HTML citesummary format, overview. A part of HCS format suite."""
_ = gettext_set_language(ln)
msg = _("Total number of citations excluding self-citations")
out = """<tr><td><strong>%(msg_cites)s</strong>""" % \
{'msg_cites': msg, }
# use ? help linking in the style of oai_repository_admin.py
msg = ' <small><small>[<a href="%s%s">?</a>]</small></small></td>'
out += msg % (CFG_BASE_URL,
'/help/citation-metrics#citesummary_self-cites')
for total_cites in d_total_cites.values():
out += '<td align="right">%s</td>' % \
self.tmpl_nice_number(total_cites, ln)
out += '</tr>'
msg = _("Average citations per paper excluding self-citations")
out += """<tr><td><strong>%(msg_avgcit)s</strong>""" % \
{'msg_avgcit': msg, }
# use ? help linking in the style of oai_repository_admin.py
msg = ' <small><small>[<a href="%s%s">?</a>]</small></small></td>'
out += msg % (CFG_BASE_URL,
'/help/citation-metrics#citesummary_self-cites')
for avg_cites in d_avg_cites.itervalues():
out += '<td align="right">%.1f</td>' % avg_cites
out += '</tr>'
return out
def tmpl_citesummary_footer(self):
return ''
def tmpl_citesummary_breakdown_header(self, ln=CFG_SITE_LANG):
_ = gettext_set_language(ln)
return """<tr><td><strong>%(msg_breakdown)s</strong></td></tr>""" % \
{'msg_breakdown': _("Breakdown of papers by citations:"), }
def tmpl_citesummary_breakdown_by_fame(self, d_cites, low, high, fame,
l_colls, searchpatterns,
searchfield, ln=CFG_SITE_LANG):
"""HTML citesummary format, breakdown by fame.
A part of HCS format suite."""
_ = gettext_set_language(ln)
out = """<tr><td>%(fame)s</td>""" % \
{'fame': _(fame), }
for coll, colldef in l_colls:
if 'excluding self cites' in coll:
keyword = 'citedexcludingselfcites'
else:
keyword = 'cited'
link_url = CFG_BASE_URL + '/search?p='
if searchpatterns.get(coll, None):
p = searchpatterns.get(coll, None)
if searchfield:
if " " in p:
p = searchfield + ':"' + p + '"'
else:
p = searchfield + ':' + p
link_url += quote(p) + '%20AND%20'
if colldef:
link_url += quote(colldef) + '%20AND%20'
if low == 0 and high == 0:
link_url += quote('%s:0' % keyword)
else:
link_url += quote('%s:%i->%i' % (keyword, low, high))
link_text = self.tmpl_nice_number(d_cites[coll], ln)
out += '<td align="right"><a href="%s">%s</a></td>' % (link_url,
link_text)
out += '</tr>'
return out
def tmpl_citesummary_h_index(self, collections,
d_h_factors, ln=CFG_SITE_LANG):
"""HTML citesummary format, h factor output. A part of the HCS suite."""
_ = gettext_set_language(ln)
out = "<tr><td></td></tr><tr><td><strong>%(msg_metrics)s</strong> <small><small>[<a href=\"%(help_url)s\">?</a>]</small></small></td></tr>" % \
{'msg_metrics': _("Citation metrics"),
'help_url': CFG_SITE_URL + '/help/citation-metrics', }
out += '<tr><td>h-index'
# use ? help linking in the style of oai_repository_admin.py
msg = ' <small><small>[<a href="%s%s">?</a>]</small></small></td>'
out += msg % (CFG_BASE_URL,
'/help/citation-metrics#citesummary_h-index')
for coll, dummy in collections:
h_factors = d_h_factors[coll]
out += '<td align="right">%s</td>' % \
self.tmpl_nice_number(h_factors, ln)
out += '</tr>'
return out
def tmpl_citesummary_epilogue(self):
"""HTML citesummary format, epilogue. A part of HCS format suite."""
out = "</table>"
return out
def tmpl_unapi(self, formats, identifier=None):
"""
Provide a list of object format available from the unAPI service
for the object identified by IDENTIFIER
"""
out = '<?xml version="1.0" encoding="UTF-8" ?>\n'
if identifier:
out += '<formats id="%i">\n' % (identifier)
else:
out += "<formats>\n"
for format_name, format_type in iteritems(formats):
docs = ''
if format_name == 'xn':
docs = 'http://www.nlm.nih.gov/databases/dtd/'
format_type = 'application/xml'
format_name = 'nlm'
elif format_name == 'xm':
docs = 'http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd'
format_type = 'application/xml'
format_name = 'marcxml'
elif format_name == 'xr':
format_type = 'application/rss+xml'
docs = 'http://www.rssboard.org/rss-2-0/'
elif format_name == 'xw':
format_type = 'application/xml'
docs = 'http://www.refworks.com/RefWorks/help/RefWorks_Tagged_Format.htm'
elif format_name == 'xoaidc':
format_type = 'application/xml'
docs = 'http://www.openarchives.org/OAI/2.0/oai_dc.xsd'
elif format_name == 'xe':
format_type = 'application/xml'
docs = 'http://www.endnote.com/support/'
format_name = 'endnote'
elif format_name == 'xd':
format_type = 'application/xml'
docs = 'http://dublincore.org/schemas/'
format_name = 'dc'
elif format_name == 'xo':
format_type = 'application/xml'
docs = 'http://www.loc.gov/standards/mods/v3/mods-3-3.xsd'
format_name = 'mods'
if docs:
out += '<format name="%s" type="%s" docs="%s" />\n' % (xml_escape(format_name), xml_escape(format_type), xml_escape(docs))
else:
out += '<format name="%s" type="%s" />\n' % (xml_escape(format_name), xml_escape(format_type))
out += "</formats>"
return out
diff --git a/invenio/legacy/websearch/webcoll.py b/invenio/legacy/websearch/webcoll.py
index 51e919693..d2c2389f8 100644
--- a/invenio/legacy/websearch/webcoll.py
+++ b/invenio/legacy/websearch/webcoll.py
@@ -1,1179 +1,1185 @@
# -*- coding: utf-8 -*-
## This file is part of Invenio.
-## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013 CERN.
+## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
"""Create Invenio collection cache."""
__revision__ = "$Id$"
import calendar
import copy
import sys
import cgi
import re
import os
import string
import time
from six.moves import cPickle
from invenio.config import \
CFG_CERN_SITE, \
CFG_WEBSEARCH_INSTANT_BROWSE, \
CFG_WEBSEARCH_NARROW_SEARCH_SHOW_GRANDSONS, \
CFG_WEBSEARCH_I18N_LATEST_ADDITIONS, \
CFG_CACHEDIR, \
CFG_SITE_LANG, \
CFG_SITE_NAME, \
CFG_SITE_LANGS, \
CFG_WEBSEARCH_ENABLED_SEARCH_INTERFACES, \
CFG_WEBSEARCH_DEFAULT_SEARCH_INTERFACE, \
- CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS
+ CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS, \
+ CFG_SCOAP3_SITE
from invenio.base.i18n import gettext_set_language, language_list_long
from invenio.legacy.search_engine import search_pattern_parenthesised, get_creation_date, get_field_i18nname, collection_restricted_p, sort_records, EM_REPOSITORY
from invenio.legacy.dbquery import run_sql, Error, get_table_update_time
from invenio.legacy.bibrank.record_sorter import get_bibrank_methods
from invenio.utils.date import convert_datestruct_to_dategui, strftime
from invenio.modules.formatter import format_record
from invenio.utils.shell import mymkdir
from intbitset import intbitset
from invenio.legacy.websearch_external_collections import \
external_collection_load_states, \
dico_collection_external_searches, \
external_collection_sort_engine_by_name
from invenio.legacy.bibsched.bibtask import task_init, task_get_option, task_set_option, \
write_message, task_has_option, task_update_progress, \
task_sleep_now_if_required
import invenio.legacy.template
websearch_templates = invenio.legacy.template.load('websearch')
from invenio.legacy.websearch_external_collections.searcher import external_collections_dictionary
from invenio.legacy.websearch_external_collections.config import CFG_EXTERNAL_COLLECTION_TIMEOUT
from invenio.legacy.websearch_external_collections.config import CFG_HOSTED_COLLECTION_TIMEOUT_NBRECS
from invenio.base.signals import webcoll_after_webpage_cache_update, \
webcoll_after_reclist_cache_update
## global vars
COLLECTION_HOUSE = {} # will hold collections we treat in this run of the program; a dict of {collname2, collobject1}, ...
# CFG_CACHE_LAST_UPDATED_TIMESTAMP_TOLERANCE -- cache timestamp
# tolerance (in seconds), to account for the fact that an admin might
# accidentally happen to edit the collection definitions at exactly
# the same second when some webcoll process was about to be started.
# In order to be safe, let's put an exaggerated timestamp tolerance
# value such as 20 seconds:
CFG_CACHE_LAST_UPDATED_TIMESTAMP_TOLERANCE = 20
# CFG_CACHE_LAST_UPDATED_TIMESTAMP_FILE -- location of the cache
# timestamp file:
CFG_CACHE_LAST_UPDATED_TIMESTAMP_FILE = "%s/collections/last_updated" % CFG_CACHEDIR
# CFG_CACHE_LAST_FAST_UPDATED_TIMESTAMP_FILE -- location of the cache
# timestamp file usef when running webcoll in the fast-mode.
CFG_CACHE_LAST_FAST_UPDATED_TIMESTAMP_FILE = "%s/collections/last_fast_updated" % CFG_CACHEDIR
def get_collection(colname):
"""Return collection object from the collection house for given colname.
If does not exist, then create it."""
if colname not in COLLECTION_HOUSE:
colobject = Collection(colname)
COLLECTION_HOUSE[colname] = colobject
return COLLECTION_HOUSE[colname]
## auxiliary functions:
def is_selected(var, fld):
"Checks if the two are equal, and if yes, returns ' selected'. Useful for select boxes."
if var == fld:
return ' selected="selected"'
else:
return ""
def get_field(recID, tag):
"Gets list of field 'tag' for the record with 'recID' system number."
out = []
digit = tag[0:2]
bx = "bib%sx" % digit
bibx = "bibrec_bib%sx" % digit
query = "SELECT bx.value FROM %s AS bx, %s AS bibx WHERE bibx.id_bibrec='%s' AND bx.id=bibx.id_bibxxx AND bx.tag='%s'" \
% (bx, bibx, recID, tag)
res = run_sql(query)
for row in res:
out.append(row[0])
return out
def check_nbrecs_for_all_external_collections():
"""Check if any of the external collections have changed their total number of records, aka nbrecs.
Return True if any of the total numbers of records have changed and False if they're all the same."""
res = run_sql("SELECT name FROM collection WHERE dbquery LIKE 'hostedcollection:%';")
for row in res:
coll_name = row[0]
if (get_collection(coll_name)).check_nbrecs_for_external_collection():
return True
return False
class Collection:
"Holds the information on collections (id,name,dbquery)."
def __init__(self, name=""):
"Creates collection instance by querying the DB configuration database about 'name'."
self.calculate_reclist_run_already = 0 # to speed things up without much refactoring
self.update_reclist_run_already = 0 # to speed things up without much refactoring
self.reclist_updated_since_start = 0 # to check if webpage cache need rebuilding
self.reclist_with_nonpublic_subcolls = intbitset()
# temporary counters for the number of records in hosted collections
self.nbrecs_tmp = None # number of records in a hosted collection
self.nbrecs_from_hosted_collections = 0 # total number of records from
# descendant hosted collections
if not name:
self.name = CFG_SITE_NAME # by default we are working on the home page
self.id = 1
self.dbquery = None
self.nbrecs = None
self.reclist = intbitset()
self.old_reclist = intbitset()
self.reclist_updated_since_start = 1
else:
self.name = name
try:
res = run_sql("""SELECT id,name,dbquery,nbrecs,reclist FROM collection
WHERE name=%s""", (name,))
if res:
self.id = res[0][0]
self.name = res[0][1]
self.dbquery = res[0][2]
self.nbrecs = res[0][3]
try:
self.reclist = intbitset(res[0][4])
except:
self.reclist = intbitset()
self.reclist_updated_since_start = 1
else: # collection does not exist!
self.id = None
self.dbquery = None
self.nbrecs = None
self.reclist = intbitset()
self.reclist_updated_since_start = 1
self.old_reclist = intbitset(self.reclist)
except Error as e:
print("Error %d: %s" % (e.args[0], e.args[1]))
sys.exit(1)
def get_example_search_queries(self):
"""Returns list of sample search queries for this collection.
"""
res = run_sql("""SELECT example.body FROM example
LEFT JOIN collection_example on example.id=collection_example.id_example
WHERE collection_example.id_collection=%s ORDER BY collection_example.score""", (self.id,))
return [query[0] for query in res]
def get_name(self, ln=CFG_SITE_LANG, name_type="ln", prolog="", epilog="", prolog_suffix=" ", epilog_suffix=""):
"""Return nicely formatted collection name for language LN.
The NAME_TYPE may be 'ln' (=long name), 'sn' (=short name), etc."""
out = prolog
i18name = ""
res = run_sql("SELECT value FROM collectionname WHERE id_collection=%s AND ln=%s AND type=%s", (self.id, ln, name_type))
try:
i18name += res[0][0]
except IndexError:
pass
if i18name:
out += i18name
else:
out += self.name
out += epilog
return out
def get_collectionbox_name(self, ln=CFG_SITE_LANG, box_type="r"):
"""
Return collection-specific labelling of 'Focus on' (regular
collection), 'Narrow by' (virtual collection) and 'Latest
addition' boxes.
If translation for given language does not exist, use label
for CFG_SITE_LANG. If no custom label is defined for
CFG_SITE_LANG, return default label for the box.
@param ln: the language of the label
@param box_type: can be 'r' (=Narrow by), 'v' (=Focus on), 'l' (=Latest additions)
"""
i18name = ""
res = run_sql("SELECT value FROM collectionboxname WHERE id_collection=%s AND ln=%s AND type=%s", (self.id, ln, box_type))
try:
i18name = res[0][0]
except IndexError:
res = run_sql("SELECT value FROM collectionboxname WHERE id_collection=%s AND ln=%s AND type=%s", (self.id, CFG_SITE_LANG, box_type))
try:
i18name = res[0][0]
except IndexError:
pass
if not i18name:
# load the right message language
_ = gettext_set_language(ln)
if box_type == "v":
i18name = _('Focus on:')
elif box_type == "r":
- i18name = _('Narrow by collection:')
+ if CFG_SCOAP3_SITE:
+ i18name = _('Narrow by publisher/journal:')
+ else:
+ i18name = _('Narrow by collection:')
elif box_type == "l":
i18name = _('Latest additions:')
return i18name
def get_ancestors(self):
"Returns list of ancestors of the current collection."
ancestors = []
ancestors_ids = intbitset()
id_son = self.id
while 1:
query = "SELECT cc.id_dad,c.name FROM collection_collection AS cc, collection AS c "\
"WHERE cc.id_son=%d AND c.id=cc.id_dad" % int(id_son)
res = run_sql(query, None, 1)
if res:
col_ancestor = get_collection(res[0][1])
# looking for loops
if self.id in ancestors_ids:
write_message("Loop found in collection %s" % self.name, stream=sys.stderr)
raise OverflowError("Loop found in collection %s" % self.name)
else:
ancestors.append(col_ancestor)
ancestors_ids.add(col_ancestor.id)
id_son = res[0][0]
else:
break
ancestors.reverse()
return ancestors
def restricted_p(self):
"""Predicate to test if the collection is restricted or not. Return the contect of the
`restrited' column of the collection table (typically Apache group). Otherwise return
None if the collection is public."""
if collection_restricted_p(self.name):
return 1
return None
def get_sons(self, type='r'):
"Returns list of direct sons of type 'type' for the current collection."
sons = []
id_dad = self.id
query = "SELECT cc.id_son,c.name FROM collection_collection AS cc, collection AS c "\
"WHERE cc.id_dad=%d AND cc.type='%s' AND c.id=cc.id_son ORDER BY score DESC, c.name ASC" % (int(id_dad), type)
res = run_sql(query)
for row in res:
sons.append(get_collection(row[1]))
return sons
def get_descendants(self, type='r'):
"Returns list of all descendants of type 'type' for the current collection."
descendants = []
descendant_ids = intbitset()
id_dad = self.id
query = "SELECT cc.id_son,c.name FROM collection_collection AS cc, collection AS c "\
"WHERE cc.id_dad=%d AND cc.type='%s' AND c.id=cc.id_son ORDER BY score DESC" % (int(id_dad), type)
res = run_sql(query)
for row in res:
col_desc = get_collection(row[1])
# looking for loops
if self.id in descendant_ids:
write_message("Loop found in collection %s" % self.name, stream=sys.stderr)
raise OverflowError("Loop found in collection %s" % self.name)
else:
descendants.append(col_desc)
descendant_ids.add(col_desc.id)
tmp_descendants = col_desc.get_descendants()
for descendant in tmp_descendants:
descendant_ids.add(descendant.id)
descendants += tmp_descendants
return descendants
def write_cache_file(self, filename='', filebody={}):
"Write a file inside collection cache."
# open file:
dirname = "%s/collections" % (CFG_CACHEDIR)
mymkdir(dirname)
fullfilename = dirname + "/%s.html" % filename
try:
os.umask(0o022)
f = open(fullfilename, "wb")
except IOError as v:
try:
(code, message) = v
except:
code = 0
message = v
print("I/O Error: " + str(message) + " (" + str(code) + ")")
sys.exit(1)
# print user info:
write_message("... creating %s" % fullfilename, verbose=6)
# print page body:
cPickle.dump(filebody, f, cPickle.HIGHEST_PROTOCOL)
# close file:
f.close()
def update_webpage_cache(self, lang):
"""Create collection page header, navtrail, body (including left and right stripes) and footer, and
call write_cache_file() afterwards to update the collection webpage cache."""
return {} ## webpage cache update is not really needed in
## Invenio-on-Flask, so let's return quickly here
## for great speed-up benefit
## precalculate latest additions for non-aggregate
## collections (the info is ln and as independent)
if self.dbquery:
if CFG_WEBSEARCH_I18N_LATEST_ADDITIONS:
self.create_latest_additions_info(ln=lang)
else:
self.create_latest_additions_info()
# load the right message language
_ = gettext_set_language(lang)
# create dictionary with data
cache = {"te_portalbox" : self.create_portalbox(lang, 'te'),
"np_portalbox" : self.create_portalbox(lang, 'np'),
"ne_portalbox" : self.create_portalbox(lang, 'ne'),
"tp_portalbox" : self.create_portalbox(lang, "tp"),
"lt_portalbox" : self.create_portalbox(lang, "lt"),
"rt_portalbox" : self.create_portalbox(lang, "rt"),
"last_updated" : convert_datestruct_to_dategui(time.localtime(),
ln=lang)}
for aas in CFG_WEBSEARCH_ENABLED_SEARCH_INTERFACES: # do light, simple and advanced search pages:
cache["navtrail_%s" % aas] = self.create_navtrail_links(aas, lang)
cache["searchfor_%s" % aas] = self.create_searchfor(aas, lang)
cache["narrowsearch_%s" % aas] = self.create_narrowsearch(aas, lang, 'r')
cache["focuson_%s" % aas] = self.create_narrowsearch(aas, lang, "v")+ \
self.create_external_collections_box(lang)
cache["instantbrowse_%s" % aas] = self.create_instant_browse(aas=aas, ln=lang)
# write cache file
self.write_cache_file("%s-ln=%s"%(self.name, lang), cache)
return cache
def create_navtrail_links(self, aas=CFG_WEBSEARCH_DEFAULT_SEARCH_INTERFACE, ln=CFG_SITE_LANG):
"""Creates navigation trail links, i.e. links to collection
ancestors (except Home collection). If aas==1, then links to
Advanced Search interfaces; otherwise Simple Search.
"""
dads = []
for dad in self.get_ancestors():
if dad.name != CFG_SITE_NAME: # exclude Home collection
dads.append((dad.name, dad.get_name(ln)))
return websearch_templates.tmpl_navtrail_links(
aas=aas, ln=ln, dads=dads)
def create_portalbox(self, lang=CFG_SITE_LANG, position="rt"):
"""Creates portalboxes of language CFG_SITE_LANG of the position POSITION by consulting DB configuration database.
The position may be: 'lt'='left top', 'rt'='right top', etc."""
out = ""
query = "SELECT p.title,p.body FROM portalbox AS p, collection_portalbox AS cp "\
" WHERE cp.id_collection=%d AND p.id=cp.id_portalbox AND cp.ln='%s' AND cp.position='%s' "\
" ORDER BY cp.score DESC" % (self.id, lang, position)
res = run_sql(query)
for row in res:
title, body = row[0], row[1]
if title:
out += websearch_templates.tmpl_portalbox(title = title,
body = body)
else:
# no title specified, so print body ``as is'' only:
out += body
return out
def create_narrowsearch(self, aas=CFG_WEBSEARCH_DEFAULT_SEARCH_INTERFACE, ln=CFG_SITE_LANG, type="r"):
"""Creates list of collection descendants of type 'type' under title 'title'.
If aas==1, then links to Advanced Search interfaces; otherwise Simple Search.
Suitable for 'Narrow search' and 'Focus on' boxes."""
# get list of sons and analyse it
sons = self.get_sons(type)
if not sons:
return ''
# get descendents
descendants = self.get_descendants(type)
grandsons = []
if CFG_WEBSEARCH_NARROW_SEARCH_SHOW_GRANDSONS:
# load grandsons for each son
for son in sons:
grandsons.append(son.get_sons())
# return ""
return websearch_templates.tmpl_narrowsearch(
aas = aas,
ln = ln,
type = type,
father = self,
has_grandchildren = len(descendants)>len(sons),
sons = sons,
display_grandsons = CFG_WEBSEARCH_NARROW_SEARCH_SHOW_GRANDSONS,
grandsons = grandsons
)
def create_external_collections_box(self, ln=CFG_SITE_LANG):
external_collection_load_states()
if self.id not in dico_collection_external_searches:
return ""
engines_list = external_collection_sort_engine_by_name(dico_collection_external_searches[self.id])
return websearch_templates.tmpl_searchalso(ln, engines_list, self.id)
def create_latest_additions_info(self, rg=CFG_WEBSEARCH_INSTANT_BROWSE, ln=CFG_SITE_LANG):
"""
Create info about latest additions that will be used for
create_instant_browse() later.
"""
self.latest_additions_info = []
if self.nbrecs and self.reclist:
# firstly, get last 'rg' records:
recIDs = list(self.reclist)
of = 'hb'
# CERN hack begins: tweak latest additions for selected collections:
if CFG_CERN_SITE:
# alter recIDs list for some CERN collections:
this_year = time.strftime("%Y", time.localtime())
if self.name in ['CERN Yellow Reports','Videos']:
last_year = str(int(this_year) - 1)
# detect recIDs only from this and past year:
recIDs = list(self.reclist & \
search_pattern_parenthesised(p='year:%s or year:%s' % \
(this_year, last_year)))
elif self.name in ['VideosXXX']:
# detect recIDs only from this year:
recIDs = list(self.reclist & \
search_pattern_parenthesised(p='year:%s' % this_year))
elif self.name == 'CMS Physics Analysis Summaries' and \
1281585 in self.reclist:
# REALLY, REALLY temporary hack
recIDs = list(self.reclist)
recIDs.remove(1281585)
# apply special filters:
if self.name in ['Videos']:
# select only videos with movies:
recIDs = list(intbitset(recIDs) & \
search_pattern_parenthesised(p='collection:"PUBLVIDEOMOVIE"'))
of = 'hvp'
# sort some CERN collections specially:
if self.name in ['Videos',
'Video Clips',
'Video Movies',
'Video News',
'Video Rushes',
'Webcast',
'ATLAS Videos',
'Restricted Video Movies',
'Restricted Video Rushes',
'LHC First Beam Videos',
'CERN openlab Videos']:
recIDs = sort_records(None, recIDs, '269__c')
elif self.name in ['LHCb Talks']:
recIDs = sort_records(None, recIDs, 'reportnumber')
+ elif self.name in ['CERN Yellow Reports']:
+ recIDs = sort_records(None, recIDs, '084__a')
# CERN hack ends.
total = len(recIDs)
to_display = min(rg, total)
for idx in range(total-1, total-to_display-1, -1):
recid = recIDs[idx]
self.latest_additions_info.append({'id': recid,
'format': format_record(recid, of, ln=ln),
'date': get_creation_date(recid, fmt="%Y-%m-%d<br />%H:%i")})
return
def create_instant_browse(self, rg=CFG_WEBSEARCH_INSTANT_BROWSE, aas=CFG_WEBSEARCH_DEFAULT_SEARCH_INTERFACE, ln=CFG_SITE_LANG):
"Searches database and produces list of last 'rg' records."
if self.restricted_p():
return websearch_templates.tmpl_box_restricted_content(ln = ln)
if str(self.dbquery).startswith("hostedcollection:"):
return websearch_templates.tmpl_box_hosted_collection(ln = ln)
if rg == 0:
# do not show latest additions box
return ""
# CERN hack: do not display latest additions for some CERN collections:
if CFG_CERN_SITE and self.name in ['Periodicals', 'Electronic Journals',
'Press Office Photo Selection',
'Press Office Video Selection']:
return ""
try:
self.latest_additions_info
latest_additions_info_p = True
except:
latest_additions_info_p = False
if latest_additions_info_p:
passIDs = []
for idx in range(0, min(len(self.latest_additions_info), rg)):
# CERN hack: display the records in a grid layout, so do not show the related links
if CFG_CERN_SITE and self.name in ['Videos']:
passIDs.append({'id': self.latest_additions_info[idx]['id'],
'body': self.latest_additions_info[idx]['format'],
'date': self.latest_additions_info[idx]['date']})
else:
passIDs.append({'id': self.latest_additions_info[idx]['id'],
'body': self.latest_additions_info[idx]['format'] + \
websearch_templates.tmpl_record_links(recid=self.latest_additions_info[idx]['id'],
rm='citation',
ln=ln),
'date': self.latest_additions_info[idx]['date']})
if self.nbrecs > rg:
url = websearch_templates.build_search_url(
cc=self.name, jrec=rg+1, ln=ln, aas=aas)
else:
url = ""
# CERN hack: display the records in a grid layout
if CFG_CERN_SITE and self.name in ['Videos']:
return websearch_templates.tmpl_instant_browse(
aas=aas, ln=ln, recids=passIDs, more_link=url, grid_layout=True, father=self)
return websearch_templates.tmpl_instant_browse(
aas=aas, ln=ln, recids=passIDs, more_link=url, father=self)
return websearch_templates.tmpl_box_no_records(ln=ln)
def create_searchoptions(self):
"Produces 'Search options' portal box."
box = ""
query = """SELECT DISTINCT(cff.id_field),f.code,f.name FROM collection_field_fieldvalue AS cff, field AS f
WHERE cff.id_collection=%d AND cff.id_fieldvalue IS NOT NULL AND cff.id_field=f.id
ORDER BY cff.score DESC""" % self.id
res = run_sql(query)
if res:
for row in res:
field_id = row[0]
field_code = row[1]
field_name = row[2]
query_bis = """SELECT fv.value,fv.name FROM fieldvalue AS fv, collection_field_fieldvalue AS cff
WHERE cff.id_collection=%d AND cff.type='seo' AND cff.id_field=%d AND fv.id=cff.id_fieldvalue
ORDER BY cff.score_fieldvalue DESC, cff.score DESC, fv.name ASC""" % (self.id, field_id)
res_bis = run_sql(query_bis)
if res_bis:
values = [{'value' : '', 'text' : 'any' + ' ' + field_name}] # FIXME: internationalisation of "any"
for row_bis in res_bis:
values.append({'value' : cgi.escape(row_bis[0], 1), 'text' : row_bis[1]})
box += websearch_templates.tmpl_select(
fieldname = field_code,
values = values
)
return box
def create_sortoptions(self, ln=CFG_SITE_LANG):
"""Produces 'Sort options' portal box."""
# load the right message language
_ = gettext_set_language(ln)
box = ""
query = """SELECT f.code,f.name FROM field AS f, collection_field_fieldvalue AS cff
WHERE id_collection=%d AND cff.type='soo' AND cff.id_field=f.id
ORDER BY cff.score DESC, f.name ASC""" % self.id
values = [{'value' : '', 'text': "- %s -" % _("latest first")}]
res = run_sql(query)
if res:
for row in res:
values.append({'value' : row[0], 'text': get_field_i18nname(row[1], ln)})
else:
for tmp in ('title', 'author', 'report number', 'year'):
values.append({'value' : tmp.replace(' ', ''), 'text' : get_field_i18nname(tmp, ln)})
box = websearch_templates.tmpl_select(
fieldname = 'sf',
css_class = 'address',
values = values
)
box += websearch_templates.tmpl_select(
fieldname = 'so',
css_class = 'address',
values = [
{'value' : 'a' , 'text' : _("asc.")},
{'value' : 'd' , 'text' : _("desc.")}
]
)
return box
def create_rankoptions(self, ln=CFG_SITE_LANG):
"Produces 'Rank options' portal box."
# load the right message language
_ = gettext_set_language(ln)
values = [{'value' : '', 'text': "- %s %s -" % (string.lower(_("OR")), _("rank by"))}]
for (code, name) in get_bibrank_methods(self.id, ln):
values.append({'value' : code, 'text': name})
box = websearch_templates.tmpl_select(
fieldname = 'rm',
css_class = 'address',
values = values
)
return box
def create_displayoptions(self, ln=CFG_SITE_LANG):
"Produces 'Display options' portal box."
# load the right message language
_ = gettext_set_language(ln)
values = []
for i in ['10', '25', '50', '100', '250', '500']:
values.append({'value' : i, 'text' : i + ' ' + _("results")})
box = websearch_templates.tmpl_select(
fieldname = 'rg',
selected = str(CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS),
css_class = 'address',
values = values
)
if self.get_sons():
box += websearch_templates.tmpl_select(
fieldname = 'sc',
css_class = 'address',
values = [
- {'value' : '1' , 'text' : _("split by collection")},
+ {'value' : '1' , 'text' : CFG_SCOAP3_SITE and _("split by publisher/journal") or _("split by collection")},
{'value' : '0' , 'text' : _("single list")}
]
)
return box
def create_formatoptions(self, ln=CFG_SITE_LANG):
"Produces 'Output format options' portal box."
# load the right message language
_ = gettext_set_language(ln)
box = ""
values = []
query = """SELECT f.code,f.name FROM format AS f, collection_format AS cf
WHERE cf.id_collection=%d AND cf.id_format=f.id AND f.visibility='1'
ORDER BY cf.score DESC, f.name ASC""" % self.id
res = run_sql(query)
if res:
for row in res:
values.append({'value' : row[0], 'text': row[1]})
else:
values.append({'value' : 'hb', 'text' : "HTML %s" % _("brief")})
box = websearch_templates.tmpl_select(
fieldname = 'of',
css_class = 'address',
values = values
)
return box
def create_searchwithin_selection_box(self, fieldname='f', value='', ln='en'):
"""Produces 'search within' selection box for the current collection."""
# get values
query = """SELECT f.code,f.name FROM field AS f, collection_field_fieldvalue AS cff
WHERE cff.type='sew' AND cff.id_collection=%d AND cff.id_field=f.id
ORDER BY cff.score DESC, f.name ASC""" % self.id
res = run_sql(query)
values = [{'value' : '', 'text' : get_field_i18nname("any field", ln)}]
if res:
for row in res:
values.append({'value' : row[0], 'text' : get_field_i18nname(row[1], ln)})
else:
if CFG_CERN_SITE:
for tmp in ['title', 'author', 'abstract', 'report number', 'year']:
values.append({'value' : tmp.replace(' ', ''), 'text' : get_field_i18nname(tmp, ln)})
else:
for tmp in ['title', 'author', 'abstract', 'keyword', 'report number', 'journal', 'year', 'fulltext', 'reference']:
values.append({'value' : tmp.replace(' ', ''), 'text' : get_field_i18nname(tmp, ln)})
return websearch_templates.tmpl_searchwithin_select(
fieldname = fieldname,
ln = ln,
selected = value,
values = values
)
def create_searchexample(self):
"Produces search example(s) for the current collection."
out = "$collSearchExamples = getSearchExample(%d, $se);" % self.id
return out
def create_searchfor(self, aas=CFG_WEBSEARCH_DEFAULT_SEARCH_INTERFACE, ln=CFG_SITE_LANG):
"Produces either Simple or Advanced 'Search for' box for the current collection."
if aas == 1:
return self.create_searchfor_advanced(ln)
elif aas == 0:
return self.create_searchfor_simple(ln)
else:
return self.create_searchfor_light(ln)
def create_searchfor_light(self, ln=CFG_SITE_LANG):
"Produces light 'Search for' box for the current collection."
return websearch_templates.tmpl_searchfor_light(
ln=ln,
collection_id = self.name,
collection_name=self.get_name(ln=ln),
record_count=self.nbrecs,
example_search_queries=self.get_example_search_queries(),
)
def create_searchfor_simple(self, ln=CFG_SITE_LANG):
"Produces simple 'Search for' box for the current collection."
return websearch_templates.tmpl_searchfor_simple(
ln=ln,
collection_id = self.name,
collection_name=self.get_name(ln=ln),
record_count=self.nbrecs,
middle_option = self.create_searchwithin_selection_box(ln=ln),
)
def create_searchfor_advanced(self, ln=CFG_SITE_LANG):
"Produces advanced 'Search for' box for the current collection."
return websearch_templates.tmpl_searchfor_advanced(
ln = ln,
collection_id = self.name,
collection_name=self.get_name(ln=ln),
record_count=self.nbrecs,
middle_option_1 = self.create_searchwithin_selection_box('f1', ln=ln),
middle_option_2 = self.create_searchwithin_selection_box('f2', ln=ln),
middle_option_3 = self.create_searchwithin_selection_box('f3', ln=ln),
searchoptions = self.create_searchoptions(),
sortoptions = self.create_sortoptions(ln),
rankoptions = self.create_rankoptions(ln),
displayoptions = self.create_displayoptions(ln),
formatoptions = self.create_formatoptions(ln)
)
def calculate_reclist(self):
"""
Calculate, set and return the (reclist,
reclist_with_nonpublic_subcolls,
nbrecs_from_hosted_collections)
tuple for the given collection."""
if str(self.dbquery).startswith("hostedcollection:"):
# we don't normally use this function to calculate the reclist
# for hosted collections. In case we do, recursively for a regular
# ancestor collection, then quickly return the object attributes.
return (self.reclist,
self.reclist_with_nonpublic_subcolls,
self.nbrecs)
if self.calculate_reclist_run_already:
# do we really have to recalculate? If not,
# then return the object attributes
return (self.reclist,
self.reclist_with_nonpublic_subcolls,
self.nbrecs_from_hosted_collections)
write_message("... calculating reclist of %s" % self.name, verbose=6)
reclist = intbitset() # will hold results for public sons only; good for storing into DB
reclist_with_nonpublic_subcolls = intbitset() # will hold results for both public and nonpublic sons; good for deducing total
# number of documents
nbrecs_from_hosted_collections = 0 # will hold the total number of records from descendant hosted collections
if not self.dbquery:
# A - collection does not have dbquery, so query recursively all its sons
# that are either non-restricted or that have the same restriction rules
for coll in self.get_sons():
coll_reclist,\
coll_reclist_with_nonpublic_subcolls,\
coll_nbrecs_from_hosted_collection = coll.calculate_reclist()
if ((coll.restricted_p() is None) or
(coll.restricted_p() == self.restricted_p())):
# add this reclist ``for real'' only if it is public
reclist.union_update(coll_reclist)
reclist_with_nonpublic_subcolls.union_update(coll_reclist_with_nonpublic_subcolls)
# increment the total number of records from descendant hosted collections
nbrecs_from_hosted_collections += coll_nbrecs_from_hosted_collection
else:
# B - collection does have dbquery, so compute it:
# (note: explicitly remove DELETED records)
if CFG_CERN_SITE:
reclist = search_pattern_parenthesised(None, self.dbquery + \
' -980__:"DELETED" -980__:"DUMMY"', ap=-9) #ap=-9 for allow queries containing hidden tags
else:
reclist = search_pattern_parenthesised(None, self.dbquery + ' -980__:"DELETED"', ap=-9) #ap=-9 allow queries containing hidden tags
reclist_with_nonpublic_subcolls = copy.deepcopy(reclist)
# store the results:
self.nbrecs_from_hosted_collections = nbrecs_from_hosted_collections
self.nbrecs = len(reclist_with_nonpublic_subcolls) + \
nbrecs_from_hosted_collections
self.reclist = reclist
self.reclist_with_nonpublic_subcolls = reclist_with_nonpublic_subcolls
# last but not least, update the speed-up flag:
self.calculate_reclist_run_already = 1
# return the two sets, as well as
# the total number of records from descendant hosted collections:
return (self.reclist,
self.reclist_with_nonpublic_subcolls,
self.nbrecs_from_hosted_collections)
def calculate_nbrecs_for_external_collection(self, timeout=CFG_EXTERNAL_COLLECTION_TIMEOUT):
"""Calculate the total number of records, aka nbrecs, for given external collection."""
#if self.calculate_reclist_run_already:
# do we have to recalculate?
#return self.nbrecs
#write_message("... calculating nbrecs of external collection %s" % self.name, verbose=6)
if self.name in external_collections_dictionary:
engine = external_collections_dictionary[self.name]
if engine.parser:
self.nbrecs_tmp = engine.parser.parse_nbrecs(timeout)
if self.nbrecs_tmp >= 0: return self.nbrecs_tmp
# the parse_nbrecs() function returns negative values for some specific cases
# maybe we can handle these specific cases, some warnings or something
# for now the total number of records remains silently the same
else: return self.nbrecs
else: write_message("External collection %s does not have a parser!" % self.name, verbose=6)
else: write_message("External collection %s not found!" % self.name, verbose=6)
return 0
# last but not least, update the speed-up flag:
#self.calculate_reclist_run_already = 1
def check_nbrecs_for_external_collection(self):
"""Check if the external collections has changed its total number of records, aka nbrecs.
Rerurns True if the total number of records has changed and False if it's the same"""
write_message("*** self.nbrecs = %s / self.cal...ion = %s ***" % (str(self.nbrecs), str(self.calculate_nbrecs_for_external_collection())), verbose=6)
write_message("*** self.nbrecs != self.cal...ion = %s ***" % (str(self.nbrecs != self.calculate_nbrecs_for_external_collection()),), verbose=6)
return self.nbrecs != self.calculate_nbrecs_for_external_collection(CFG_HOSTED_COLLECTION_TIMEOUT_NBRECS)
def set_nbrecs_for_external_collection(self):
"""Set this external collection's total number of records, aka nbrecs"""
if self.calculate_reclist_run_already:
# do we have to recalculate?
return
write_message("... calculating nbrecs of external collection %s" % self.name, verbose=6)
if self.nbrecs_tmp:
self.nbrecs = self.nbrecs_tmp
else:
self.nbrecs = self.calculate_nbrecs_for_external_collection(CFG_HOSTED_COLLECTION_TIMEOUT_NBRECS)
# last but not least, update the speed-up flag:
self.calculate_reclist_run_already = 1
def update_reclist(self):
"Update the record universe for given collection; nbrecs, reclist of the collection table."
if self.update_reclist_run_already:
# do we have to reupdate?
return 0
write_message("... updating reclist of %s (%s recs)" % (self.name, self.nbrecs), verbose=6)
sys.stdout.flush()
try:
## In principle we could skip this update if old_reclist==reclist
## however we just update it here in case of race-conditions.
run_sql("UPDATE collection SET nbrecs=%s, reclist=%s WHERE id=%s",
(self.nbrecs, self.reclist.fastdump(), self.id))
if self.old_reclist != self.reclist:
self.reclist_updated_since_start = 1
else:
write_message("... no changes in reclist detected", verbose=6)
except Error as e:
print("Database Query Error %d: %s." % (e.args[0], e.args[1]))
sys.exit(1)
# last but not least, update the speed-up flag:
self.update_reclist_run_already = 1
return 0
def perform_display_collection(colID, colname, aas, ln, em, show_help_boxes):
"""Returns the data needed to display a collection page
The arguments are as follows:
colID - id of the collection to display
colname - name of the collection to display
aas - 0 if simple search, 1 if advanced search
ln - language of the page
em - code to display just part of the page
show_help_boxes - whether to show the help boxes or not"""
# check and update cache if necessary
cachedfile = open("%s/collections/%s-ln=%s.html" %
(CFG_CACHEDIR, colname, ln), "rb")
try:
data = cPickle.load(cachedfile)
except ValueError:
data = get_collection(colname).update_webpage_cache(ln)
cachedfile.close()
# check em value to return just part of the page
if em != "":
if EM_REPOSITORY["search_box"] not in em:
data["searchfor_%s" % aas] = ""
if EM_REPOSITORY["see_also_box"] not in em:
data["focuson_%s" % aas] = ""
if EM_REPOSITORY["all_portalboxes"] not in em:
if EM_REPOSITORY["te_portalbox"] not in em:
data["te_portalbox"] = ""
if EM_REPOSITORY["np_portalbox"] not in em:
data["np_portalbox"] = ""
if EM_REPOSITORY["ne_portalbox"] not in em:
data["ne_portalbox"] = ""
if EM_REPOSITORY["tp_portalbox"] not in em:
data["tp_portalbox"] = ""
if EM_REPOSITORY["lt_portalbox"] not in em:
data["lt_portalbox"] = ""
if EM_REPOSITORY["rt_portalbox"] not in em:
data["rt_portalbox"] = ""
c_body = websearch_templates.tmpl_webcoll_body(ln, colID, data.get("te_portalbox", ""),
data.get("searchfor_%s"%aas,''), data.get("np_portalbox", ''), data.get("narrowsearch_%s"%aas, ''),
data.get("focuson_%s"%aas, ''), data.get("instantbrowse_%s"%aas, ''), data.get("ne_portalbox", ''),
em=="" or EM_REPOSITORY["body"] in em)
if show_help_boxes <= 0:
data["rt_portalbox"] = ""
return (c_body, data.get("navtrail_%s"%aas, ''), data.get("lt_portalbox", ''), data.get("rt_portalbox", ''),
data.get("tp_portalbox", ''), data.get("te_portalbox", ''), data.get("last_updated", ''))
def get_datetime(var, format_string="%Y-%m-%d %H:%M:%S"):
"""Returns a date string according to the format string.
It can handle normal date strings and shifts with respect
to now."""
date = time.time()
shift_re = re.compile("([-\+]{0,1})([\d]+)([dhms])")
factors = {"d":24*3600, "h":3600, "m":60, "s":1}
m = shift_re.match(var)
if m:
sign = m.groups()[0] == "-" and -1 or 1
factor = factors[m.groups()[2]]
value = float(m.groups()[1])
date = time.localtime(date + sign * factor * value)
date = strftime(format_string, date)
else:
date = time.strptime(var, format_string)
date = strftime(format_string, date)
return date
def get_current_time_timestamp():
"""Return timestamp corresponding to the current time."""
return time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
def compare_timestamps_with_tolerance(timestamp1,
timestamp2,
tolerance=0):
"""Compare two timestamps TIMESTAMP1 and TIMESTAMP2, of the form
'2005-03-31 17:37:26'. Optionally receives a TOLERANCE argument
(in seconds). Return -1 if TIMESTAMP1 is less than TIMESTAMP2
minus TOLERANCE, 0 if they are equal within TOLERANCE limit,
and 1 if TIMESTAMP1 is greater than TIMESTAMP2 plus TOLERANCE.
"""
# remove any trailing .00 in timestamps:
timestamp1 = re.sub(r'\.[0-9]+$', '', timestamp1)
timestamp2 = re.sub(r'\.[0-9]+$', '', timestamp2)
# first convert timestamps to Unix epoch seconds:
timestamp1_seconds = calendar.timegm(time.strptime(timestamp1, "%Y-%m-%d %H:%M:%S"))
timestamp2_seconds = calendar.timegm(time.strptime(timestamp2, "%Y-%m-%d %H:%M:%S"))
# now compare them:
if timestamp1_seconds < timestamp2_seconds - tolerance:
return -1
elif timestamp1_seconds > timestamp2_seconds + tolerance:
return 1
else:
return 0
def get_database_last_updated_timestamp():
"""Return last updated timestamp for collection-related and
record-related database tables.
"""
database_tables_timestamps = []
database_tables_timestamps.append(get_table_update_time('bibrec'))
## In INSPIRE bibfmt is on innodb and there is not such configuration
bibfmt_last_update = run_sql("SELECT max(last_updated) FROM bibfmt")
- if bibfmt_last_update:
+ if bibfmt_last_update and bibfmt_last_update[0][0]:
database_tables_timestamps.append(str(bibfmt_last_update[0][0]))
try:
database_tables_timestamps.append(get_table_update_time('idxWORD%'))
except ValueError:
# There are no indexes in the database. That's OK.
pass
database_tables_timestamps.append(get_table_update_time('collection%'))
database_tables_timestamps.append(get_table_update_time('portalbox'))
database_tables_timestamps.append(get_table_update_time('field%'))
database_tables_timestamps.append(get_table_update_time('format%'))
database_tables_timestamps.append(get_table_update_time('rnkMETHODNAME'))
database_tables_timestamps.append(get_table_update_time('accROLE_accACTION_accARGUMENT', run_on_slave=True))
return max(database_tables_timestamps)
def get_cache_last_updated_timestamp():
"""Return last updated cache timestamp."""
try:
f = open(CFG_CACHE_LAST_UPDATED_TIMESTAMP_FILE, "r")
except:
return "1970-01-01 00:00:00"
timestamp = f.read()
f.close()
return timestamp
def set_cache_last_updated_timestamp(timestamp):
"""Set last updated cache timestamp to TIMESTAMP."""
try:
with open(CFG_CACHE_LAST_UPDATED_TIMESTAMP_FILE, "w") as f:
f.write(timestamp)
except:
# FIXME: do something here
pass
return timestamp
def task_submit_elaborate_specific_parameter(key, value, opts, args):
""" Given the string key it checks it's meaning, eventually using the value.
Usually it fills some key in the options dict.
It must return True if it has elaborated the key, False, if it doesn't
know that key.
eg:
if key in ['-n', '--number']:
self.options['number'] = value
return True
return False
"""
if key in ("-c", "--collection"):
task_set_option("collection", value)
elif key in ("-r", "--recursive"):
task_set_option("recursive", 1)
elif key in ("-f", "--force"):
task_set_option("force", 1)
elif key in ("-q", "--quick"):
task_set_option("quick", 1)
elif key in ("-p", "--part"):
task_set_option("part", int(value))
elif key in ("-l", "--language"):
languages = task_get_option("language", [])
languages += value.split(',')
for ln in languages:
if ln not in CFG_SITE_LANGS:
print('ERROR: "%s" is not a recognized language code' % ln)
return False
task_set_option("language", languages)
else:
return False
return True
def task_submit_check_options():
if task_has_option('collection'):
coll = get_collection(task_get_option("collection"))
if coll.id is None:
print('ERROR: Collection "%s" does not exist' % coll.name)
return False
return True
def task_run_core():
""" Reimplement to add the body of the task."""
##
## ------->--->time--->------>
## (-1) | ( 0) | ( 1)
## | | |
## [T.db] | [T.fc] | [T.db]
## | | |
## |<-tol|tol->|
##
## the above is the compare_timestamps_with_tolerance result "diagram"
## [T.db] stands fore the database timestamp and [T.fc] for the file cache timestamp
## ( -1, 0, 1) stand for the returned value
## tol stands for the tolerance in seconds
##
## When a record has been added or deleted from one of the collections the T.db becomes greater that the T.fc
## and when webcoll runs it is fully ran. It recalculates the reclists and nbrecs, and since it updates the
## collections db table it also updates the T.db. The T.fc is set as the moment the task started running thus
## slightly before the T.db (practically the time distance between the start of the task and the last call of
## update_reclist). Therefore when webcoll runs again, and even if no database changes have taken place in the
## meanwhile, it fully runs (because compare_timestamps_with_tolerance returns 0). This time though, and if
## no databases changes have taken place, the T.db remains the same while T.fc is updated and as a result if
## webcoll runs again it will not be fully ran
##
task_run_start_timestamp = get_current_time_timestamp()
colls = []
# decide whether we need to run or not, by comparing last updated timestamps:
write_message("Database timestamp is %s." % get_database_last_updated_timestamp(), verbose=3)
write_message("Collection cache timestamp is %s." % get_cache_last_updated_timestamp(), verbose=3)
if task_has_option("part"):
write_message("Running cache update part %s only." % task_get_option("part"), verbose=3)
if check_nbrecs_for_all_external_collections() or task_has_option("force") or \
compare_timestamps_with_tolerance(get_database_last_updated_timestamp(),
get_cache_last_updated_timestamp(),
CFG_CACHE_LAST_UPDATED_TIMESTAMP_TOLERANCE) >= 0:
## either forced update was requested or cache is not up to date, so recreate it:
# firstly, decide which collections to do:
if task_has_option("collection"):
coll = get_collection(task_get_option("collection"))
colls.append(coll)
if task_has_option("recursive"):
r_type_descendants = coll.get_descendants(type='r')
colls += r_type_descendants
v_type_descendants = coll.get_descendants(type='v')
colls += v_type_descendants
else:
res = run_sql("SELECT name FROM collection ORDER BY id")
for row in res:
colls.append(get_collection(row[0]))
# secondly, update collection reclist cache:
if task_get_option('part', 1) == 1:
i = 0
for coll in colls:
i += 1
write_message("%s / reclist cache update" % coll.name)
if str(coll.dbquery).startswith("hostedcollection:"):
coll.set_nbrecs_for_external_collection()
else:
coll.calculate_reclist()
coll.update_reclist()
task_update_progress("Part 1/2: done %d/%d" % (i, len(colls)))
task_sleep_now_if_required(can_stop_too=True)
webcoll_after_reclist_cache_update.send('webcoll', collections=colls)
# thirdly, update collection webpage cache:
if task_get_option("part", 2) == 2:
i = 0
for coll in colls:
i += 1
if coll.reclist_updated_since_start or task_has_option("collection") or task_get_option("force") or not task_get_option("quick"):
write_message("%s / webpage cache update" % coll.name)
for lang in CFG_SITE_LANGS:
coll.update_webpage_cache(lang)
webcoll_after_webpage_cache_update.send(coll.name, collection=coll, lang=lang)
else:
write_message("%s / webpage cache seems not to need an update and --quick was used" % coll.name, verbose=2)
task_update_progress("Part 2/2: done %d/%d" % (i, len(colls)))
task_sleep_now_if_required(can_stop_too=True)
# finally update the cache last updated timestamp:
# (but only when all collections were updated, not when only
# some of them were forced-updated as per admin's demand)
if not task_has_option("collection"):
set_cache_last_updated_timestamp(task_run_start_timestamp)
write_message("Collection cache timestamp is set to %s." % get_cache_last_updated_timestamp(), verbose=3)
else:
## cache up to date, we don't have to run
write_message("Collection cache is up to date, no need to run.")
## we are done:
return True
### okay, here we go:
if __name__ == '__main__':
main()
diff --git a/invenio/legacy/websearch_external_collections/__init__.py b/invenio/legacy/websearch_external_collections/__init__.py
index d91380268..98afa02ab 100644
--- a/invenio/legacy/websearch_external_collections/__init__.py
+++ b/invenio/legacy/websearch_external_collections/__init__.py
@@ -1,483 +1,483 @@
# -*- coding: utf-8 -*-
## This file is part of Invenio.
## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""External collection 'core' file.
Perform search, database access."""
__revision__ = "$Id$"
import cgi
import sys
from copy import copy
if sys.hexversion < 0x2040000:
# pylint: disable=W0622
from sets import Set as set
# pylint: enable=W0622
from invenio.config import CFG_SITE_LANG
from invenio.legacy.dbquery import run_sql, OperationalError, ProgrammingError
from invenio.base.i18n import gettext_set_language
from .config import CFG_EXTERNAL_COLLECTION_TIMEOUT
from .searcher import external_collections_dictionary
from .getter import HTTPAsyncPageGetter, async_download
from .templates import print_results, print_timeout
from .utils import get_collection_id, get_collection_descendants, \
warning, get_verbose_print
import invenio.legacy.template
# Global variables
template = invenio.legacy.template.load('websearch_external_collections')
external_collections_state = None
dico_collection_external_searches = None
dico_collection_seealso = None
#dico_collection_external_searches = {}
#dico_collection_seealso = {}
def print_external_results_overview(req, current_collection, pattern_list, field,
external_collection, verbosity_level=0, lang=CFG_SITE_LANG, print_overview=True):
"""Print the external collection overview box. Return the selected external collections and parsed query"""
from invenio.legacy.search_engine import create_basic_search_units
assert req
vprint = get_verbose_print(req, 'External collection (print_external_results_overview): ', verbosity_level)
pattern = bind_patterns(pattern_list)
vprint(3, 'pattern = %s' % cgi.escape(pattern))
if not pattern:
return (None, None, None, None)
basic_search_units = create_basic_search_units(None, pattern, field)
vprint(3, 'basic_search_units = %s' % cgi.escape(repr(basic_search_units)))
(search_engines, seealso_engines) = select_external_engines(current_collection, external_collection)
vprint(3, 'search_engines = ' + str(search_engines))
vprint(3, 'seealso_engines = ' + str(seealso_engines))
search_engines_list = external_collection_sort_engine_by_name(search_engines)
vprint(3, 'search_engines_list (sorted) : ' + str(search_engines_list))
if print_overview:
html = template.external_collection_overview(lang, search_engines_list)
req.write(html)
return (search_engines, seealso_engines, pattern, basic_search_units)
def perform_external_collection_search(req, current_collection, pattern_list, field,
external_collection, verbosity_level=0, lang=CFG_SITE_LANG,
selected_external_collections_infos=None, print_overview=True,
print_search_info=True, print_see_also_box=True, print_body=True):
"""Search external collection and print the seealso box."""
vprint = get_verbose_print(req, 'External collection: ', verbosity_level)
if selected_external_collections_infos:
(search_engines, seealso_engines, pattern, basic_search_units) = selected_external_collections_infos
else:
(search_engines, seealso_engines, pattern, basic_search_units) = print_external_results_overview(req,
current_collection, pattern_list, field, external_collection, verbosity_level, lang, print_overview=print_overview)
if not pattern:
return
do_external_search(req, lang, vprint, basic_search_units, search_engines, print_search_info, print_body)
if print_see_also_box:
create_seealso_box(req, lang, vprint, basic_search_units, seealso_engines, pattern)
vprint(3, 'end')
def bind_patterns(pattern_list):
"""Combine a list of patterns in an unique pattern.
pattern_list[0] should be the standart search pattern,
pattern_list[1:] are advanced search patterns."""
# just in case an empty list is fed to this function
try:
if pattern_list[0]:
return pattern_list[0]
except IndexError:
return None
pattern = ""
for pattern_part in pattern_list[1:]:
if pattern_part:
pattern += " " + pattern_part
return pattern.strip()
# See also box
def create_seealso_box(req, lang, vprint, basic_search_units=None, seealso_engines=None, query=''):
"Create the box that proposes links to other useful search engines like Google."
vprint(3, 'Create seealso box')
seealso_engines_list = external_collection_sort_engine_by_name(seealso_engines)
vprint(3, 'seealso_engines_list = ' + str(seealso_engines_list))
links = build_seealso_links(basic_search_units, seealso_engines_list, req, lang, query)
html = template.external_collection_seealso_box(lang, links)
req.write(html)
def build_seealso_links(basic_search_units, seealso_engines, req, lang, query):
"""Build the links for the see also box."""
_ = gettext_set_language(lang)
links = []
for engine in seealso_engines:
url = engine.build_search_url(basic_search_units, req.args, lang)
user_url = engine.build_user_search_url(basic_search_units, req.args, lang)
url = user_url or url
if url:
links.append('<a class="google" href="%(url)s">%(query)s %(text_in)s %(name)s</a>' % \
{'url': cgi.escape(url),
'query': cgi.escape(query),
'text_in': _('in'),
'name': _(engine.name)})
return links
# Selection
def select_external_engines(collection_name, selected_external_searches):
"""Build a tuple of two sets. The first one is the list of engine to use for an external search and the
second one is for the seealso box."""
collection_id = get_collection_id(collection_name)
if not collection_id:
return (None, None)
if not type(selected_external_searches) is list:
selected_external_searches = [selected_external_searches]
seealso_engines = set()
search_engines = set()
if collection_id in dico_collection_seealso:
seealso_engines = copy(dico_collection_seealso[collection_id])
if collection_id in dico_collection_external_searches:
seealso_engines = seealso_engines.union(dico_collection_external_searches[collection_id])
for ext_search_name in selected_external_searches:
if ext_search_name in external_collections_dictionary:
engine = external_collections_dictionary[ext_search_name]
if engine.parser:
search_engines.add(engine)
else:
warning('select_external_engines: %(ext_search_name)s unknown.' % locals())
seealso_engines = seealso_engines.difference(search_engines)
return (search_engines, seealso_engines)
# Search
def do_external_search(req, lang, vprint, basic_search_units, search_engines, print_search_info=True, print_body=True):
"""Make the external search."""
_ = gettext_set_language(lang)
vprint(3, 'beginning external search')
engines_list = []
for engine in search_engines:
url = engine.build_search_url(basic_search_units, req.args, lang)
user_url = engine.build_user_search_url(basic_search_units, req.args, lang)
if url:
engines_list.append([url, engine, user_url])
pagegetters_list = [HTTPAsyncPageGetter(engine[0]) for engine in engines_list]
def finished(pagegetter, data, current_time, print_search_info=True, print_body=True):
"""Function called, each time the download of a web page finish.
Will parse and print the results of this page."""
print_results(req, lang, pagegetter, data, current_time, print_search_info, print_body)
finished_list = async_download(pagegetters_list, finished, engines_list, CFG_EXTERNAL_COLLECTION_TIMEOUT, print_search_info, print_body)
for (finished, engine) in zip(finished_list, engines_list):
if not finished:
- url = engine[0]
+ url = engine[2] or engine[0]
name = engine[1].name
print_timeout(req, lang, engine[1], name, url)
# Database management
def external_collection_load_states():
global external_collections_state, dico_collection_external_searches, dico_collection_seealso
external_collections_state = {}
dico_collection_external_searches = {}
dico_collection_seealso = {}
query = "SELECT collection_externalcollection.id_collection, collection_externalcollection.type, externalcollection.name FROM collection_externalcollection, externalcollection WHERE collection_externalcollection.id_externalcollection = externalcollection.id;"
try:
results = run_sql(query)
except (OperationalError, ProgrammingError):
results = None
if results:
for result in results:
collection_id = int(result[0])
search_type = int(result[1])
engine_name = result[2]
if engine_name not in external_collections_dictionary:
warning("No search engine : " + engine_name)
continue
engine = external_collections_dictionary[engine_name]
if collection_id not in external_collections_state:
external_collections_state[collection_id] = {}
col_states = external_collections_state[collection_id]
col_states[engine] = search_type
dictionary = None
if search_type == 1:
dictionary = dico_collection_seealso
if search_type in [2, 3]:
dictionary = dico_collection_external_searches
if dictionary is None:
continue
if collection_id not in dictionary:
dictionary[collection_id] = set()
engine_set = dictionary[collection_id]
engine_set.add(engine)
def external_collection_get_state(external_collection, collection_id):
external_collection_load_states()
if collection_id not in external_collections_state:
return 0
col_states = external_collections_state[collection_id]
if external_collection not in col_states:
return 0
return col_states[external_collection]
def external_collection_get_update_state_list(external_collection, collection_id, state, recurse=False):
changes = []
if external_collection_get_state(external_collection, collection_id) != state:
changes = ['(%(collection_id)d, %(id_externalcollection)d, %(state)d)' %
{'collection_id': collection_id, 'id_externalcollection': external_collection_getid(external_collection), 'state': state}]
if not recurse:
return changes
for descendant_id in get_collection_descendants(collection_id):
changes += external_collection_get_update_state_list(external_collection, descendant_id, state)
return changes
def external_collection_apply_changes(changes_list):
if not changes_list:
return
sql_values = ", ".join(changes_list)
sql = 'INSERT INTO collection_externalcollection (id_collection, id_externalcollection, type) VALUES ' + sql_values + 'ON DUPLICATE KEY UPDATE type=VALUES(type);'
run_sql(sql)
# Misc functions
def external_collection_sort_engine_by_name(engines_set):
"""Return a list of sorted (by name) search engines."""
if not engines_set:
return []
engines_list = [engine for engine in engines_set]
engines_list.sort(lambda x, y: cmp(x.name, y.name))
return engines_list
# External search ID
def external_collection_getid(external_collection):
"""Return the id of an external_collection. Will create a new entry in DB if needed."""
if 'id' in external_collection.__dict__:
return external_collection.id
query = 'SELECT id FROM externalcollection WHERE name="%(name)s";' % {'name': external_collection.name}
results = run_sql(query)
if not results:
query = 'INSERT INTO externalcollection (name) VALUES ("%(name)s");' % {'name': external_collection.name}
run_sql(query)
return external_collection_getid(external_collection)
external_collection.id = results[0][0]
return external_collection.id
def get_external_collection_engine(external_collection_name):
"""Return the external collection engine given its name"""
if external_collection_name in external_collections_dictionary:
return external_collections_dictionary[external_collection_name]
else:
return None
# Load db infos if it's not already done.
if external_collections_state is None:
external_collection_load_states()
# Hosted Collections related functions (the following functions should eventually be regrouped as above)
# These functions could eventually be placed into there own file, ex. websearch_hosted_collections.py
def calculate_hosted_collections_results(req, pattern_list, field, hosted_collections, verbosity_level=0,
lang=CFG_SITE_LANG, timeout=CFG_EXTERNAL_COLLECTION_TIMEOUT):
"""Ruturn a list of the various results for a every hosted collection organized in tuples"""
# normally, the following should be checked before even running this function so the following line could be removed
if not hosted_collections: return (None, None)
vprint = get_verbose_print(req, 'Hosted collections: ', verbosity_level)
vprint(3, 'pattern_list = %s, field = %s' % (cgi.escape(repr(pattern_list)), cgi.escape(field)))
# firstly we calculate the search parameters, i.e. the actual hosted search engines and the basic search units
(hosted_search_engines, basic_search_units) = \
calculate_hosted_collections_search_params(req,
pattern_list,
field,
hosted_collections,
verbosity_level)
# in case something went wrong with the above calculation just return None
# however, once we run this function no fail should be expected here
# UPDATE : let search go on even there are no basic search units (an empty pattern_list and field)
#if basic_search_units == None or len(hosted_search_engines) == 0: return (None, None)
if len(hosted_search_engines) == 0: return (None, None)
# finally return the list of tuples with the results
return do_calculate_hosted_collections_results(req, lang, vprint, verbosity_level, basic_search_units, hosted_search_engines, timeout)
vprint(3, 'end')
def calculate_hosted_collections_search_params(req,
pattern_list,
field,
hosted_collections,
verbosity_level=0):
"""Calculate the searching parameters for the selected hosted collections
i.e. the actual hosted search engines and the basic search units"""
from invenio.legacy.search_engine import create_basic_search_units
assert req
vprint = get_verbose_print(req, 'Hosted collections (calculate_hosted_collections_search_params): ', verbosity_level)
pattern = bind_patterns(pattern_list)
vprint(3, 'pattern = %s' % cgi.escape(pattern))
# if for any strange reason there is no pattern, just return
# UPDATE : let search go on even there is no pattern (an empty pattern_list and field)
#if not pattern: return (None, None)
# calculate the basic search units
basic_search_units = create_basic_search_units(None, pattern, field)
vprint(3, 'basic_search_units = %s' % cgi.escape(repr(basic_search_units)))
# calculate the set of hosted search engines
hosted_search_engines = select_hosted_search_engines(hosted_collections)
vprint(3, 'hosted_search_engines = ' + str(hosted_search_engines))
# no need really to print out a sorted list of the hosted search engines, is there? I'll leave this commented out
#hosted_search_engines_list = external_collection_sort_engine_by_name(hosted_search_engines)
#vprint(3, 'hosted_search_engines_list (sorted) : ' + str(hosted_search_engines_list))
return (hosted_search_engines, basic_search_units)
def select_hosted_search_engines(selected_hosted_collections):
"""Build the set of engines to be used for the hosted collections"""
if not type(selected_hosted_collections) is list:
selected_hosted_collections = [selected_hosted_collections]
hosted_search_engines = set()
for hosted_collection_name in selected_hosted_collections:
if hosted_collection_name in external_collections_dictionary:
engine = external_collections_dictionary[hosted_collection_name]
# the hosted collection cannot present its results unless it has a parser implemented
if engine.parser:
hosted_search_engines.add(engine)
else:
warning('select_hosted_search_engines: %(hosted_collection_name)s unknown.' % locals())
return hosted_search_engines
def do_calculate_hosted_collections_results(req, lang, vprint, verbosity_level, basic_search_units, hosted_search_engines,
timeout=CFG_EXTERNAL_COLLECTION_TIMEOUT):
"""Actually search the hosted collections and return their results and information in a list of tuples.
One tuple for each hosted collection. Handles timeouts"""
_ = gettext_set_language(lang)
if not vprint:
vprint = get_verbose_print(req, 'Hosted collections (calculate_hosted_collections_search_params): ', verbosity_level)
# defining vprint at this moment probably means we'll just run this one function at this time, therefore the "verbose"
# end hosted search string will not be printed (it is normally printed by the initial calculate function)
# Therefore, either define a flag here to print it by the end of this function or redefine the whole "verbose"
# printing logic of the above functions
vprint(3, 'beginning hosted search')
# list to hold the hosted search engines and their respective search urls
engines_list = []
# list to hold the non timed out results
results_list = []
# list to hold all the results
full_results_list = []
# list to hold all the timeouts
timeout_list = []
# in case this is an engine-only list
if type(hosted_search_engines) is set:
for engine in hosted_search_engines:
url = engine.build_search_url(basic_search_units, req.args, lang)
user_url = engine.build_user_search_url(basic_search_units, req.args, lang)
if url:
engines_list.append([url, engine, user_url])
# in case we are iterating a pre calculated url+engine list
elif type(hosted_search_engines) is list:
for engine in hosted_search_engines:
engines_list.append(engine)
# in both the above cases we end up with a [[search url], [engine]] kind of list
# create the list of search urls to be handed to the asynchronous getter
pagegetters_list = [HTTPAsyncPageGetter(engine[0]) for engine in engines_list]
# function to be run on every result
def finished(pagegetter, data, current_time):
"""Function called, each time the download of a web page finish.
Will parse and print the results of this page."""
# each pagegetter that didn't timeout is added to this list
results_list.append((pagegetter, data, current_time))
# run the asynchronous getter
finished_list = async_download(pagegetters_list, finished, engines_list, timeout)
# create the complete list of tuples, one for each hosted collection, with the results and other information,
# including those that timed out
for (finished, engine) in zip(finished_list, engines_list): #finished_and_engines_list:
if finished:
for result in results_list:
if result[1] == engine:
# the engine is fed the results, it will be parsed later, at printing time
engine[1].parser.parse_and_get_results(result[0].data, feedonly=True)
## the list contains:
## * the engine itself: [ search url], [engine]
## * the parsed number of found results
## * the fetching time
full_results_list.append(
(engine, engine[1].parser.parse_num_results(), result[2])
)
break
elif not finished:
## the list contains:
## * the engine itself: [search url], [engine]
timeout_list.append(engine)
return (full_results_list, timeout_list)
diff --git a/invenio/legacy/websession/session.py b/invenio/legacy/websession/session.py
index 7d7491ae8..b9c5329e3 100644
--- a/invenio/legacy/websession/session.py
+++ b/invenio/legacy/websession/session.py
@@ -1,502 +1,552 @@
# -*- coding: utf-8 -*-
## This file is part of Invenio.
-## Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
+## Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
Session management adapted from mod_python Session class.
Just use L{get_session} to obtain a session object (with a dictionary
interface, which will let you store permanent information).
"""
from invenio.legacy.wsgi.utils import add_cookies, Cookie, get_cookie
import random
import zlib
from six.moves import cPickle
import re
import sys
import os
import time
from datetime import datetime, timedelta
from uuid import uuid4
from invenio.utils.date import convert_datestruct_to_datetext
from invenio.legacy.dbquery import run_sql, blob_to_string
-from invenio.config import CFG_WEBSESSION_EXPIRY_LIMIT_REMEMBER, \
- CFG_WEBSESSION_EXPIRY_LIMIT_DEFAULT, CFG_SITE_URL, CFG_SITE_SECURE_URL, \
- CFG_WEBSESSION_IPADDR_CHECK_SKIP_BITS, \
- CFG_WEBSEARCH_PREV_NEXT_HIT_FOR_GUESTS
-from invenio.legacy.websession.websession_config import CFG_WEBSESSION_COOKIE_NAME, \
- CFG_WEBSESSION_ONE_DAY, CFG_WEBSESSION_CLEANUP_CHANCE, \
- CFG_WEBSESSION_ENABLE_LOCKING
-#from invenio.session_flask import InvenioSession as FlaskInvenioSession
+from invenio.config import (CFG_WEBSESSION_EXPIRY_LIMIT_REMEMBER,
+ CFG_WEBSESSION_EXPIRY_LIMIT_DEFAULT,
+ CFG_SITE_URL,
+ CFG_SITE_SECURE_URL,
+ CFG_WEBSESSION_IPADDR_CHECK_SKIP_BITS,
+ CFG_WEBSEARCH_PREV_NEXT_HIT_FOR_GUESTS,
+ CFG_WEBSESSION_STORAGE)
+from invenio.legacy.websession.websession_config import (CFG_WEBSESSION_COOKIE_NAME,
+ CFG_WEBSESSION_ONE_DAY,
+ CFG_WEBSESSION_CLEANUP_CHANCE)
+from invenio.utils.redis import get_redis
from invenio.utils.hash import md5
+
CFG_FULL_HTTPS = CFG_SITE_URL.lower().startswith("https://")
if CFG_WEBSEARCH_PREV_NEXT_HIT_FOR_GUESTS:
_CFG_SESSION_NON_USEFUL_KEYS = ('uid', 'user_info')
else:
_CFG_SESSION_NON_USEFUL_KEYS = ('uid', 'user_info', 'websearch-last-query', 'websearch-last-query-hits')
def get_session(req, sid=None):
"""
Obtain a session.
If the session has already been created for the current request,
returns the already existing session.
@param req: the mod_python request object.
@type req: mod_python request object
@param sid: the session identifier of an already existing session.
@type sid: 32 hexadecimal string
@return: the session.
@rtype: InvenioSession
@raise ValueError: if C{sid} is provided and it doesn't correspond to a
valid session.
"""
from flask import session
if sid is not None:
req._session = session
return req._session
if not hasattr(req, '_session'):
req._session = session
return req._session
-class InvenioSession(dict):
+
+class InvenioSessionBase(dict):
"""
This class implements a Session handling based on MySQL.
@param req: the mod_python request object.
@type req: mod_python request object
@param sid: the session identifier if already known
@type sid: 32 hexadecimal string
@ivar _remember_me: if the session cookie should last one day or until
the browser is closed.
@type _remember_me: bool
@note: The code is heavily based on ModPython 3.3.1 DBMSession
implementation.
@note: This class implements IP verification to prevent basic cookie
stealing.
@raise ValueError: if C{sid} is provided and correspond to a broken
session.
"""
def __init__(self, req, sid=None):
self._remember_me = False
self._req, self._sid, self._secret = req, sid, None
self._lock = CFG_WEBSESSION_ENABLE_LOCKING
self._new = 1
self._locked = 0
self._invalid = 0
self._dirty = False
self._http_ip = None
self._https_ip = None
self.__need_https = False
+ self._cleanup_function = None
dict.__init__(self)
if not self._sid:
# check to see if cookie exists
cookie = get_cookie(req, CFG_WEBSESSION_COOKIE_NAME)
if cookie:
self._sid = cookie.value
else:
stub_cookie = get_cookie(req, CFG_WEBSESSION_COOKIE_NAME + 'stub')
self.__need_https = stub_cookie and stub_cookie.value == 'HTTPS'
if self._sid:
if not _check_sid(self._sid):
if sid:
# Supplied explicitly by user of the class,
# raise an exception and make the user code
# deal with it.
raise ValueError("Invalid Session ID: sid=%s" % sid)
else:
# Derived from the cookie sent by browser,
# wipe it out so it gets replaced with a
# correct value.
self._sid = None
if self._sid:
# attempt to load ourselves
self.lock()
if self.load():
self._new = 0
if self._new:
# make a new session
if self._sid:
self.unlock() # unlock old sid
self._sid = _new_sid(self._req)
self.lock() # lock new sid
remote_ip = self._req.remote_ip
if self._req.is_https():
self._https_ip = remote_ip
else:
self._http_ip = remote_ip
# need cleanup?
if random.randint(1, CFG_WEBSESSION_CLEANUP_CHANCE) == 1:
self.cleanup()
def get_dirty(self):
"""
Is this session dirty?
"""
return self._dirty
def set_dirty(self, dummy=True):
"""
Flag this session as dirty. It takes a parameter, just in order
to be used within a property
"""
self._dirty = True
dirty = property(get_dirty, set_dirty)
def __setitem__(self, key, value):
if self.get(key) != value:
dict.__setitem__(self, key, value)
self._dirty = True
def __delitem__(self, key):
if key in self:
dict.__delitem__(self, key)
self._dirty = True
def set_remember_me(self, remember_me=True):
"""
Set/Unset the L{_remember_me} flag.
@param remember_me: True if the session cookie should last one day or
until the browser is closed.
@type remember_me: bool
"""
self._remember_me = remember_me
self['_permanent'] = remember_me
add_cookies(self._req, self.make_cookies())
def load(self):
"""
Load the session from the database.
@return: 1 in case of success, 0 otherwise.
@rtype: integer
"""
session_dict = None
invalid = False
- res = run_sql("SELECT session_object FROM session "
- "WHERE session_key=%s AND session_expiry>=UTC_TIMESTAMP()", (self._sid, ))
+ res = self.load_from_storage(self._sid)
if res:
- session_dict = cPickle.loads(zlib.decompress(blob_to_string(res[0][0])))
+ session_dict = cPickle.loads(blob_to_string(res))
remote_ip = self._req.remote_ip
if self._req.is_https():
if session_dict['_https_ip'] is not None:
if ':' in remote_ip:
## IPV6 address, we don't skip bits
if session_dict['_https_ip'] != remote_ip:
invalid = True
else:
if _mkip(session_dict['_https_ip']) >> \
CFG_WEBSESSION_IPADDR_CHECK_SKIP_BITS != \
_mkip(remote_ip) >> \
CFG_WEBSESSION_IPADDR_CHECK_SKIP_BITS:
invalid = True
else:
session_dict['_https_ip'] = remote_ip
else:
if session_dict['_http_ip'] is not None:
if ':' in remote_ip:
## IPV6 address, we don't skip bits
if session_dict['_http_ip'] != remote_ip:
invalid = True
else:
if _mkip(session_dict['_http_ip']) >> \
CFG_WEBSESSION_IPADDR_CHECK_SKIP_BITS != \
_mkip(remote_ip) >> \
CFG_WEBSESSION_IPADDR_CHECK_SKIP_BITS:
invalid = True
else:
session_dict['_http_ip'] = remote_ip
if session_dict is None:
return 0
if invalid:
return 0
self.update(session_dict)
self._remember_me = session_dict.get("_permanent", False)
return 1
def is_useful(self):
"""
Return True if the session contains some key considered
useful (i.e. that deserve being preserved)
"""
for key in self:
if key not in _CFG_SESSION_NON_USEFUL_KEYS:
return True
return False
def save(self):
"""
Save the session to the database.
"""
uid = self.get('uid', -1)
if not self._invalid and self._sid and self._dirty and (uid > 0 or self.is_useful()):
## We store something only for real users or useful sessions.
session_dict = {"_data" : self.copy(),
"_created" : self._created,
"_accessed": self._accessed,
"_timeout" : self._timeout,
"_http_ip" : self._http_ip,
"_https_ip" : self._https_ip,
- "_permanent" : self._remember_me
- }
- session_key = self._sid
+ "_remember_me" : self._remember_me
+ }
session_object = cPickle.dumps(session_dict, -1)
- session_expiry = time.time() + self._timeout + CFG_WEBSESSION_ONE_DAY
- session_expiry = convert_datestruct_to_datetext(time.gmtime(session_expiry))
-
- run_sql("""
- INSERT session(
- session_key,
- session_expiry,
- session_object,
- uid
- ) VALUE(%s,
- %s,
- %s,
- %s
- ) ON DUPLICATE KEY UPDATE
- session_expiry=%s,
- session_object=%s,
- uid=%s
- """, (session_key, session_expiry, session_object, uid,
- session_expiry, session_object, uid))
- add_cookies(self._req, self.make_cookies())
+
+ self.save_in_storage(self._sid,
+ session_object,
+ self._timeout,
+ uid)
+
+ for cookie in self.make_cookies():
+ self._req.set_cookie(cookie)
## No more dirty :-)
self._dirty = False
def delete(self):
"""
Delete the session.
"""
- run_sql("DELETE LOW_PRIORITY FROM session WHERE session_key=%s", (self._sid, ))
+ self.delete_from_storage(self._sid)
self.clear()
def invalidate(self):
"""
Declare the session as invalid.
"""
cookies = self.make_cookies()
for cookie in cookies:
cookie.expires = 0
add_cookies(self._req, cookies)
self.delete()
self._invalid = 1
if hasattr(self._req, '_session'):
delattr(self._req, '_session')
def make_cookies(self):
"""
Create the necessary cookies to implement secure session handling
(possibly over HTTPS).
@return: a list of cookies.
"""
cookies = []
uid = self.get('_uid', -1)
if uid > 0 and CFG_SITE_SECURE_URL.startswith("https://"):
stub_cookie = Cookie(CFG_WEBSESSION_COOKIE_NAME + 'stub', 'HTTPS')
else:
stub_cookie = Cookie(CFG_WEBSESSION_COOKIE_NAME + 'stub', 'NO')
cookies.append(stub_cookie)
if self._req.is_https() or not CFG_SITE_SECURE_URL.startswith("https://") or uid <= 0:
cookie = Cookie(CFG_WEBSESSION_COOKIE_NAME, self._sid)
if CFG_SITE_SECURE_URL.startswith("https://") and uid > 0:
cookie.secure = True
cookie.httponly = True
cookies.append(cookie)
for cookie in cookies:
cookie.path = '/'
if self._remember_me:
cookie.expires = time.time() + CFG_WEBSESSION_ONE_DAY * CFG_WEBSESSION_EXPIRY_LIMIT_REMEMBER
cookie.max_age = CFG_WEBSESSION_ONE_DAY * CFG_WEBSESSION_EXPIRY_LIMIT_REMEMBER
return cookies
def initial_http_ip(self):
"""
@return: the initial ip addressed for the HTTP protocol for which this
session was issued.
@rtype: string
@note: it returns None if this session has always been used through
HTTPS requests.
"""
return self._http_ip
def initial_https_ip(self):
"""
@return: the initial ip addressed for the HTTPS protocol for which this
session was issued.
@rtype: string
@note: it returns None if this session has always been used through
HTTP requests.
"""
return self._https_ip
def lock(self):
"""
Lock the session.
"""
if self._lock:
self._locked = 1
def unlock(self):
"""
Unlock the session.
"""
if self._lock and self._locked:
self._locked = 0
def is_new(self):
"""
@return: True if the session has just been created.
@rtype: bool
"""
return not not self._new
def sid(self):
"""
@return: the session identifier.
@rtype: 32 hexadecimal string
"""
return self._sid
def cleanup(self):
"""
Perform the database session cleanup.
"""
- def cb_session_cleanup(data=None):
- """
- Session cleanup procedure which to be executed at the end
- of the request handling.
- """
- run_sql("""
- DELETE LOW_PRIORITY FROM session
- WHERE session_expiry<=UTC_TIMESTAMP()
- """)
- self._req.register_cleanup(cb_session_cleanup)
+ if self._cleanup_function:
+ self._req.register_cleanup(self._cleanup_function)
self._req.log_error("InvenioSession: registered database cleanup.")
def __del__(self):
self.save()
self.unlock()
def get_need_https(self):
return self.__need_https
## This property will be True if the connection need to be set to HTTPS
## in order for the session to be successfully read. This can actually
## be checked by not having a cookie, but just having the stub_cookie.
## The default cookie is only sent via HTTPS, while the stub_cookie
## is also sent via HTTP and contains the uid, of the user. So if there
## is actually a stub cookie and its value is different than -1 this
## property will be True, meaning the server should redirect the client
## to an HTTPS connection if she really wants to access authenticated
## resources.
need_https = property(get_need_https)
def _unlock_session_cleanup(session):
"""
Auxliary function to unlock a session.
"""
session.unlock()
_RE_VALIDATE_SID = re.compile('[0-9a-f]{32}$')
def _check_sid(sid):
"""
Check the validity of the session identifier.
The sid must be 32 characters long, and consisting of the characters
0-9 and a-f.
The sid may be passed in a cookie from the client and as such
should not be trusted. This is particularly important in
FileSession, where the session filename is derived from the sid.
A sid containing '/' or '.' characters could result in a directory
traversal attack
@param sid: the session identifier.
@type sid: string
@return: True if the session identifier is valid.
@rtype: bool
"""
return not not _RE_VALIDATE_SID.match(sid)
def _new_sid(req):
"""
Make a number based on current time, pid, remote ip
and two random ints, then hash with md5. This should
be fairly unique and very difficult to guess.
@param req: the mod_python request object.
@type req: mod_python request object.
@return: the session identifier.
@rtype: 32 hexadecimal string
@warning: The current implementation of _new_sid returns an
md5 hexdigest string. To avoid a possible directory traversal
attack in FileSession the sid is validated using
the _check_sid() method and the compiled regex
validate_sid_re. The sid will be accepted only if len(sid) == 32
and it only contains the characters 0-9 and a-f.
If you change this implementation of _new_sid, make sure to also
change the validation scheme, as well as the test_Session_illegal_sid()
unit test in test/test.py.
"""
return uuid4().hex
the_time = long(time.time()*10000)
pid = os.getpid()
random_generator = _get_generator()
rnd1 = random_generator.randint(0, 999999999)
rnd2 = random_generator.randint(0, 999999999)
remote_ip = req.remote_ip
return md5("%d%d%d%d%s" % (
the_time,
pid,
rnd1,
rnd2,
remote_ip)
).hexdigest()
def _mkip(ip):
"""
Compute a numerical value for a dotted IP
"""
num = 0L
- for i in map (int, ip.split ('.')):
- num = (num << 8) + i
+ for i in ip.split('.'):
+ num = (num << 8) + int(i)
return num
+
+
+
+class InvenioSessionMySQL(InvenioSessionBase):
+ def __init__(self, req, sid=None):
+
+ def cb_session_cleanup(data=None):
+ """
+ Session cleanup procedure which to be executed at the end
+ of the request handling.
+ """
+ run_sql("""DELETE LOW_PRIORITY FROM session
+ WHERE session_expiry <= UTC_TIMESTAMP()""")
+
+ self.cleanup_function = cb_session_cleanup
+ super(InvenioSessionMySQL, self).__init__(req, sid)
+
+ def load_from_storage(self, sid):
+ ret = run_sql("""SELECT session_object FROM session
+ WHERE session_key = %s""", [sid])
+ if ret:
+ return ret[0][0]
+
+ def delete_from_storage(self, sid):
+ return run_sql("""DELETE LOW_PRIORITY FROM session
+ WHERE session_key=%s""", [sid])
+
+ def save_in_storage(self, sid, session_object, timeout, uid):
+ session_key = sid
+ session_expiry = time.time() + timeout + CFG_WEBSESSION_ONE_DAY
+ session_expiry = convert_datestruct_to_datetext(time.gmtime(session_expiry))
+
+ run_sql("""INSERT INTO session(
+ session_key,
+ session_expiry,
+ session_object,
+ uid
+ ) VALUES (%s, %s, %s, %s)
+ ON DUPLICATE KEY UPDATE
+ session_expiry=%s,
+ session_object=%s,
+ uid=%s
+ """, (session_key, session_expiry, session_object, uid,
+ session_expiry, session_object, uid))
+
+
+class InvenioSessionRedis(InvenioSessionBase):
+
+ def generate_key(self, sid):
+ return 'session_%s' % sid
+
+ def load_from_storage(self, sid):
+ return get_redis().get(self.generate_key(sid))
+
+ def delete_from_storage(self, sid):
+ return get_redis().delete(self.generate_key(sid))
+
+ def save_in_storage(self, sid, session_object, timeout, uid): # pylint: disable=W0613
+ return get_redis().setex(self.generate_key(sid),
+ session_object,
+ timeout)
+
+if CFG_WEBSESSION_STORAGE == 'mysql':
+ InvenioSession = InvenioSessionMySQL
+elif CFG_WEBSESSION_STORAGE == 'redis':
+ InvenioSession = InvenioSessionRedis
diff --git a/invenio/legacy/websession/templates.py b/invenio/legacy/websession/templates.py
index 535048959..b09600d70 100644
--- a/invenio/legacy/websession/templates.py
+++ b/invenio/legacy/websession/templates.py
@@ -1,2850 +1,2877 @@
## This file is part of Invenio.
-## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
+## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
__revision__ = "$Id$"
import urllib
import cgi
from invenio.base.wrappers import lazy_import
from invenio.config import \
CFG_CERN_SITE, \
CFG_SITE_LANG, \
CFG_SITE_NAME, \
CFG_SITE_NAME_INTL, \
CFG_SITE_SUPPORT_EMAIL, \
CFG_SITE_SECURE_URL, \
CFG_SITE_URL, \
CFG_WEBSESSION_RESET_PASSWORD_EXPIRE_IN_DAYS, \
CFG_WEBSESSION_ADDRESS_ACTIVATION_EXPIRE_IN_DAYS, \
CFG_WEBSESSION_DIFFERENTIATE_BETWEEN_GUESTS, \
CFG_WEBSEARCH_MAX_RECORDS_IN_GROUPS, \
CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS, \
CFG_SITE_RECORD
CFG_EXTERNAL_AUTH_USING_SSO = lazy_import('invenio.modules.access.local_config:CFG_EXTERNAL_AUTH_USING_SSO')
CFG_EXTERNAL_AUTH_USING_SSO = lazy_import('invenio.modules.access.local_config:CFG_EXTERNAL_AUTH_LOGOUT_SSO')
CFG_OPENID_PROVIDERS = lazy_import('invenio.modules.access.local_config:CFG_OPENID_PROVIDERS')
CFG_OAUTH2_PROVIDERS = lazy_import('invenio.modules.access.local_config:CFG_OAUTH2_PROVIDERS')
CFG_OAUTH1_PROVIDERS = lazy_import('invenio.modules.access.local_config:CFG_OAUTH1_PROVIDERS')
CFG_OPENID_AUTHENTICATION = lazy_import('invenio.modules.access.local_config:CFG_OPENID_AUTHENTICATION')
CFG_OAUTH2_AUTHENTICATION = lazy_import('invenio.modules.access.local_config:CFG_OAUTH2_AUTHENTICATION')
CFG_OAUTH1_AUTHENTICATION = lazy_import('invenio.modules.access.local_config:CFG_OAUTH1_AUTHENTICATION')
from invenio.utils.url import make_canonical_urlargd, create_url, create_html_link
from invenio.utils.html import escape_html, nmtoken_from_string
from invenio.base.i18n import gettext_set_language, language_list_long
from invenio.modules.apikeys.models import WebAPIKey
from invenio.legacy.websession.websession_config import CFG_WEBSESSION_GROUP_JOIN_POLICY
class Template:
def tmpl_back_form(self, ln, message, url, link):
"""
A standard one-message-go-back-link page.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'message' *string* - The message to display
- 'url' *string* - The url to go back to
- 'link' *string* - The link text
"""
out = """
<table>
<tr>
<td align="left">%(message)s
<a href="%(url)s">%(link)s</a></td>
</tr>
</table>
"""% {
'message' : message,
'url' : url,
'link' : link,
'ln' : ln
}
return out
def tmpl_external_setting(self, ln, key, value):
_ = gettext_set_language(ln)
out = """
<tr>
<td align="right"><strong>%s:</strong></td>
<td><i>%s</i></td>
</tr>""" % (key, value)
return out
def tmpl_external_user_settings(self, ln, html_settings):
_ = gettext_set_language(ln)
out = """
<p><big><strong class="headline">%(external_user_settings)s</strong></big></p>
<table>
%(html_settings)s
</table>
<p><big><strong class="headline">%(external_user_groups)s</strong></big></p>
<p>%(consult_external_groups)s</p>
""" % {
'external_user_settings' : _('External account settings'),
'html_settings' : html_settings,
'consult_external_groups' : _('You can consult the list of your external groups directly in the %(x_url_open)sgroups page%(x_url_close)s.', **{
'x_url_open' : '<a href="../yourgroups/display?ln=%s#external_groups">' % ln,
'x_url_close' : '</a>'
}),
'external_user_groups' : _('External user groups'),
}
return out
def tmpl_user_api_key(self, ln=CFG_SITE_LANG, keys_info=None):
"""
Displays all the API key that the user owns the user
Parameters:
- 'ln' *string* - The language to display the interface in
- 'key_info' *tuples* - Contains the tuples with the key data (id, desciption, status)
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<script type="text/javascript">
$(document).ready(function(){
$(".key_value").hide();
$(".key_label").click(function(){
$(this).next(".key_value").slideToggle("slow");
});
});
</script>
<p><big><strong class="headline">%(user_api_key)s</strong></big></p>
""" % {
'user_api_key' : _("API keys")
}
if keys_info and len(keys_info) != 0:
out += "<p>%(user_keys)s</p>" % {'user_keys': _("These are your current API keys")}
out += """
<table>
"""
+
for key_info in keys_info:
out += """
<tr><td>%(key_description)s</td>
<td>%(key_status)s</td>
</tr><tr>
<td class = "key_label">
<a name="#%(index)s" href="#%(index)s"> %(key_label)s</a>
</td>
<td class="key_value"><code/>%(key_id)s</code></td>
</tr><tr>
<td></td>
<td align="left">
<form method="post" action="%(sitesecureurl)s/youraccount/apikey" name="api_key_remove">
<input type="hidden" name="key_id" value="%(key_id)s" />
<code class="blocknote"><input class="formbutton" type="%(input_type)s" value="%(remove_key)s" /></code>
</form>
</td>
</tr>
""" % {
'key_description': _("Description: " + cgi.escape(key_info[1])),
'key_status': _("Status: " + key_info[2]),
'key_id': _(key_info[0]),
'index': keys_info.index(key_info),
'key_label': _("API key"),
'remove_key' : _("Delete key"),
'sitesecureurl': CFG_SITE_SECURE_URL,
'input_type': ("submit", "hidden")[key_info[2] == WebAPIKey.CFG_WEB_API_KEY_STATUS['REVOKED']]
}
out += "</table>"
out += """
<form method="post" action="%(sitesecureurl)s/youraccount/apikey" name="api_key_create">
<p>%(create_new_key)s</p>
<table>
<tr><td align="right" valign="top"><strong>
<label for="new_key_description">%(new_key_description_label)s:</label></strong><br />
<small class="important">(%(mandatory)s)</small>
</td><td valign="top">
<input type="text" size="50" name="key_description" id="key_description" value=""/><br />
<small><span class="quicknote">%(note)s:</span>
%(new_key_description_note)s
</small>
</td>
</tr>
<tr><td></td><td align="left">
<code class="blocknote"><input class="formbutton" type="submit" value="%(create_new_key_button)s" /></code>
</td></tr>
</table>
</form>
""" % {
'create_new_key' : _("If you want to create a new API key, please enter a description for it"),
'new_key_description_label' : _("Description for the new API key"),
'mandatory' : _("mandatory"),
'note' : _("Note"),
'new_key_description_note': _("The description should be something meaningful for you to recognize the API key"),
'create_new_key_button' : _("Create new key"),
'sitesecureurl': CFG_SITE_SECURE_URL
}
return out
def tmpl_user_preferences(self, ln, email, email_disabled, password_disabled, nickname):
"""
Displays a form for the user to change his email/password.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'email' *string* - The email of the user
- 'email_disabled' *boolean* - If the user has the right to edit his email
- 'password_disabled' *boolean* - If the user has the right to edit his password
- 'nickname' *string* - The nickname of the user (empty string if user does not have it)
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<p><big><strong class="headline">%(edit_params)s</strong></big></p>
<form method="post" action="%(sitesecureurl)s/youraccount/change" name="edit_logins_settings">
<p>%(change_user)s</p>
<table>
<tr><td align="right" valign="top"><strong>
<label for="nickname">%(nickname_label)s:</label></strong><br />
<small class="important">(%(mandatory)s)</small>
</td><td valign="top">
%(nickname_prefix)s%(nickname)s%(nickname_suffix)s<br />
<small><span class="quicknote">%(note)s:</span>
%(fixed_nickname_note)s
</small>
</td>
</tr>
<tr><td align="right"><strong>
<label for="email">%(new_email)s:</label></strong><br />
<small class="important">(%(mandatory)s)</small>
</td><td>
<input type="text" size="25" name="email" id="email" %(email_disabled)s value="%(email)s" /><br />
<small><span class="quicknote">%(example)s:</span>
<span class="example">john.doe@example.com</span>
</small>
</td>
</tr>
<tr><td></td><td align="left">
<input class="formbutton" type="submit" value="%(set_values)s" />&nbsp;&nbsp;&nbsp;
</td></tr>
</table>
<input type="hidden" name="action" value="edit" />
</form>
""" % {
'change_user' : _("If you want to change your email or set for the first time your nickname, please set new values in the form below."),
'edit_params' : _("Edit login credentials"),
'nickname_label' : _("Nickname"),
'nickname' : nickname,
'nickname_prefix' : nickname=='' and '<input type="text" size="25" name="nickname" id="nickname" value=""' or '',
'nickname_suffix' : nickname=='' and '" /><br /><small><span class="quicknote">'+_("Example")+':</span><span class="example">johnd</span></small>' or '',
'new_email' : _("New email address"),
'mandatory' : _("mandatory"),
'example' : _("Example"),
'note' : _("Note"),
'set_values' : _("Set new values"),
'email' : email,
'email_disabled' : email_disabled and "readonly" or "",
'sitesecureurl': CFG_SITE_SECURE_URL,
'fixed_nickname_note' : _('Since this is considered as a signature for comments and reviews, once set it can not be changed.')
}
if not password_disabled and not CFG_EXTERNAL_AUTH_USING_SSO:
out += """
<form method="post" action="%(sitesecureurl)s/youraccount/change" name="edit_password">
<p>%(change_pass)s</p>
<table>
<tr>
<td align="right"><strong><label for="old_password">%(old_password)s:</label></strong><br />
</td><td align="left">
<input type="password" size="25" name="old_password" id="old_password" %(password_disabled)s /><br />
<small><span class="quicknote">%(note)s:</span>
%(old_password_note)s
</small>
</td>
</tr>
<tr>
<td align="right"><strong><label for="new_password">%(new_password)s:</label></strong><br />
</td><td align="left">
<input type="password" size="25" name="password" id="new_password" %(password_disabled)s /><br />
<small><span class="quicknote">%(note)s:</span>
%(password_note)s
</small>
</td>
</tr>
<tr>
<td align="right"><strong><label for="new_password2">%(retype_password)s:</label></strong></td>
<td align="left">
<input type="password" size="25" name="password2" id="new_password2" %(password_disabled)s value="" />
</td>
</tr>
<tr><td></td><td align="left">
<input class="formbutton" type="submit" value="%(set_values)s" />&nbsp;&nbsp;&nbsp;
</td></tr>
</table>
<input type="hidden" name="action" value="edit" />
</form>
""" % {
'change_pass' : _("If you want to change your password, please enter the old one and set the new value in the form below."),
'mandatory' : _("mandatory"),
'old_password' : _("Old password"),
'new_password' : _("New password"),
'optional' : _("optional"),
'note' : _("Note"),
'password_note' : _("The password phrase may contain punctuation, spaces, etc."),
'old_password_note' : _("You must fill the old password in order to set a new one."),
'retype_password' : _("Retype password"),
'set_values' : _("Set new password"),
'password_disabled' : password_disabled and "disabled" or "",
'sitesecureurl': CFG_SITE_SECURE_URL,
}
elif not CFG_EXTERNAL_AUTH_USING_SSO and CFG_CERN_SITE:
out += "<p>" + _("""If you are using a lightweight CERN account you can %(x_url_open)sreset the password%(x_url_close)s.""",
{'x_url_open' : '<a href="http://cern.ch/LightweightRegistration/ResetPassword.aspx%s">'
% (make_canonical_urlargd({'email': email,
'returnurl': CFG_SITE_SECURE_URL + '/youraccount/edit' + make_canonical_urlargd({'lang' : ln}, {})}, {})),
'x_url_close' : '</a>'}) + "</p>"
elif CFG_EXTERNAL_AUTH_USING_SSO and CFG_CERN_SITE:
out += "<p>" + _("""You can change or reset your CERN account password by means of the %(x_url_open)sCERN account system%(x_url_close)s.""") % \
{'x_url_open' : '<a href="https://cern.ch/login/password.aspx">', 'x_url_close' : '</a>'} + "</p>"
return out
def tmpl_user_bibcatalog_auth(self, bibcatalog_username="", bibcatalog_password="", ln=CFG_SITE_LANG):
"""template for setting username and pw for bibcatalog backend"""
_ = gettext_set_language(ln)
out = """
<form method="post" action="%(sitesecureurl)s/youraccount/change" name="edit_bibcatalog_settings">
<p><big><strong class="headline">%(edit_bibcatalog_settings)s</strong></big></p>
<table>
<tr>
<td> %(username)s: <input type="text" size="25" name="bibcatalog_username" value="%(bibcatalog_username)s" id="bibcatuid"></td>
<td> %(password)s: <input type="password" size="25" name="bibcatalog_password" value="%(bibcatalog_password)s" id="bibcatpw"></td>
</tr>
<tr>
<td><input class="formbutton" type="submit" value="%(update_settings)s" /></td>
</tr>
- </table>
+ </table></form>
""" % {
'sitesecureurl' : CFG_SITE_SECURE_URL,
'bibcatalog_username' : bibcatalog_username,
'bibcatalog_password' : bibcatalog_password,
'edit_bibcatalog_settings' : _("Edit cataloging interface settings"),
'username' : _("Username"),
'password' : _("Password"),
'update_settings' : _('Update settings')
}
return out
def tmpl_user_lang_edit(self, ln, preferred_lang):
_ = gettext_set_language(ln)
out = """
<form method="post" action="%(sitesecureurl)s/youraccount/change" name="edit_lang_settings">
<p><big><strong class="headline">%(edit_lang_settings)s</strong></big></p>
<table>
<tr><td align="right"><select name="lang" id="lang">
""" % {
'sitesecureurl' : CFG_SITE_SECURE_URL,
'edit_lang_settings' : _("Edit language-related settings"),
}
for short_ln, long_ln in language_list_long():
out += """<option %(selected)s value="%(short_ln)s">%(long_ln)s</option>""" % {
'selected' : preferred_lang == short_ln and 'selected="selected"' or '',
'short_ln' : short_ln,
'long_ln' : escape_html(long_ln)
}
out += """</select></td><td valign="top"><strong><label for="lang">%(select_lang)s</label></strong></td></tr>
<tr><td></td><td><input class="formbutton" type="submit" value="%(update_settings)s" /></td></tr>
</table></form>""" % {
'select_lang' : _('Select desired language of the web interface.'),
'update_settings' : _('Update settings')
}
return out
+ def tmpl_user_profiling_settings(self, ln, enable_profiling):
+ _ = gettext_set_language(ln)
+ out = """
+ <form method="post" action="%(sitesecureurl)s/youraccount/change" name="edit_profiling_settings">
+ <p><big><strong class="headline">%(edit_settings)s</strong></big></p>
+ <table>
+ <tr><td align="right"><select name="profiling">
+ """ % {
+ 'sitesecureurl' : CFG_SITE_SECURE_URL,
+ 'edit_settings' : _("Edit profiling settings"),
+ }
+ out += """<option %(selected)s value="0">%(desc)s</option>""" % {
+ 'selected' : 'selected="selected"' if enable_profiling is False else '',
+ 'desc' : _("Disabled")
+ }
+ out += """<option %(selected)s value="1">%(desc)s</option>""" % {
+ 'selected' : 'selected="selected"' if enable_profiling is True else '',
+ 'desc' : _("Enabled")
+ }
+ out += """</select></td><td valign="top"></td></tr>
+ <tr><td></td><td><input class="formbutton" type="submit" value="%(update_settings)s" /></td></tr>
+ </table></form>""" % {
+ 'update_settings' : _('Update settings')
+ }
+ return out
+
def tmpl_user_websearch_edit(self, ln, current = 10, show_latestbox = True, show_helpbox = True):
_ = gettext_set_language(ln)
out = """
<form method="post" action="%(sitesecureurl)s/youraccount/change" name="edit_websearch_settings">
<p><big><strong class="headline">%(edit_websearch_settings)s</strong></big></p>
<table>
<tr><td align="right"><input type="checkbox" %(checked_latestbox)s value="1" name="latestbox" id="latestbox"/></td>
<td valign="top"><b><label for="latestbox">%(show_latestbox)s</label></b></td></tr>
<tr><td align="right"><input type="checkbox" %(checked_helpbox)s value="1" name="helpbox" id="helpbox"/></td>
<td valign="top"><b><label for="helpbox">%(show_helpbox)s</label></b></td></tr>
<tr><td align="right"><select name="group_records" id="group_records">
""" % {
'sitesecureurl' : CFG_SITE_SECURE_URL,
'edit_websearch_settings' : _("Edit search-related settings"),
'show_latestbox' : _("Show the latest additions box"),
'checked_latestbox' : show_latestbox and 'checked="checked"' or '',
'show_helpbox' : _("Show collection help boxes"),
'checked_helpbox' : show_helpbox and 'checked="checked"' or '',
}
for i in 10, 25, 50, 100, 250, 500:
if i <= CFG_WEBSEARCH_MAX_RECORDS_IN_GROUPS:
out += """<option %(selected)s>%(i)s</option>
""" % {
'selected' : current == i and 'selected="selected"' or '',
'i' : i
}
out += """</select></td><td valign="top"><strong><label for="group_records">%(select_group_records)s</label></strong></td></tr>
<tr><td></td><td><input class="formbutton" type="submit" value="%(update_settings)s" /></td></tr>
</table>
</form>""" % {
'update_settings' : _("Update settings"),
'select_group_records' : _("Number of search results per page"),
}
return out
def tmpl_user_external_auth(self, ln, methods, current, method_disabled):
"""
Displays a form for the user to change his authentication method.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'methods' *array* - The methods of authentication
- 'method_disabled' *boolean* - If the user has the right to change this
- 'current' *string* - The currently selected method
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<form method="post" action="%(sitesecureurl)s/youraccount/change">
<big><strong class="headline">%(edit_method)s</strong></big>
<p>%(explain_method)s:</p>
<table>
<tr><td valign="top"><b>%(select_method)s:</b></td><td>
""" % {
'edit_method' : _("Edit login method"),
'explain_method' : _("Please select which login method you would like to use to authenticate yourself"),
'select_method' : _("Select method"),
'sitesecureurl': CFG_SITE_SECURE_URL,
}
for system in methods:
out += """<input type="radio" name="login_method" value="%(system)s" id="%(id)s" %(disabled)s %(selected)s /><label for="%(id)s">%(system)s</label><br />""" % {
'system' : system,
'disabled' : method_disabled and 'disabled="disabled"' or "",
'selected' : current == system and 'checked="checked"' or "",
'id' : nmtoken_from_string(system),
}
out += """ </td></tr>
<tr><td>&nbsp;</td>
<td><input class="formbutton" type="submit" value="%(select_method)s" /></td></tr></table>
</form>""" % {
'select_method' : _("Select method"),
}
return out
def tmpl_lost_password_form(self, ln):
"""
Displays a form for the user to ask for his password sent by email.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'msg' *string* - Explicative message on top of the form.
"""
# load the right message language
_ = gettext_set_language(ln)
out = "<p>" + _("If you have lost the password for your %(sitename)s %(x_fmt_open)sinternal account%(x_fmt_close)s, then please enter your email address in the following form in order to have a password reset link emailed to you.", **{'x_fmt_open' : '<em>', 'x_fmt_close' : '</em>', 'sitename' : CFG_SITE_NAME_INTL[ln]}) + "</p>"
out += """
<blockquote>
<form method="post" action="../youraccount/send_email">
<table>
<tr>
<td align="right"><strong><label for="p_email">%(email)s:</label></strong></td>
<td><input type="text" size="25" name="p_email" id="p_email" value="" />
<input type="hidden" name="ln" value="%(ln)s" />
<input type="hidden" name="action" value="lost" />
</td>
</tr>
<tr><td>&nbsp;</td>
<td><input class="formbutton" type="submit" value="%(send)s" /></td>
</tr>
</table>
</form>
</blockquote>
""" % {
'ln': ln,
'email' : _("Email address"),
'send' : _("Send password reset link"),
}
if CFG_CERN_SITE:
out += "<p>" + _("If you have been using the %(x_fmt_open)sCERN login system%(x_fmt_close)s, then you can recover your password through the %(x_url_open)sCERN authentication system%(x_url_close)s.",
**{'x_fmt_open' : '<em>',
'x_fmt_close' : '</em>',
'x_url_open' : '<a href="https://cern.ch/lightweightregistration/ResetPassword.aspx%s">' % make_canonical_urlargd(
{'lf': 'auth', 'returnURL': CFG_SITE_SECURE_URL + '/youraccount/login?ln='+ln}, {}),
'x_url_close' : '</a>'}) + " "
else:
out += "<p>" + _("Note that if you have been using an external login system, then we cannot do anything and you have to ask there.") + " "
out += _("Alternatively, you can ask %(x_name)s to change your login system from external to internal.",
x_name=("""<a href="mailto:%(email)s">%(email)s</a>""" % { 'email' : CFG_SITE_SUPPORT_EMAIL })) + "</p>"
return out
def tmpl_account_info(self, ln, uid, guest, CFG_CERN_SITE):
"""
Displays the account information
Parameters:
- 'ln' *string* - The language to display the interface in
- 'uid' *string* - The user id
- 'guest' *boolean* - If the user is guest
- 'CFG_CERN_SITE' *boolean* - If the site is a CERN site
"""
# load the right message language
_ = gettext_set_language(ln)
out = """<p>%(account_offer)s</p>
<blockquote>
<dl>
""" % {
'account_offer' : _("%(x_name)s offers you the possibility to personalize the interface, to set up your own personal library of documents, or to set up an automatic alert query that would run periodically and would notify you of search results by email.",
x_name=CFG_SITE_NAME_INTL[ln]),
}
if not guest:
out += """
<dt>
<a href="./edit?ln=%(ln)s">%(your_settings)s</a>
</dt>
<dd>%(change_account)s</dd>""" % {
'ln' : ln,
'your_settings' : _("Your Settings"),
'change_account' : _("Set or change your account email address or password. Specify your preferences about the look and feel of the interface.")
}
out += """
<dt><a href="../youralerts/display?ln=%(ln)s">%(your_searches)s</a></dt>
<dd>%(search_explain)s</dd>""" % {
'ln' : ln,
'your_searches' : _("Your Searches"),
'search_explain' : _("View all the searches you performed during the last 30 days."),
}
out += """
<dt><a href="../yourbaskets/display?ln=%(ln)s">%(your_baskets)s</a></dt>
<dd>%(basket_explain)s""" % {
'ln' : ln,
'your_baskets' : _("Your Baskets"),
'basket_explain' : _("With baskets you can define specific collections of items, store interesting records you want to access later or share with others."),
}
if not guest:
out += """
<dt><a href="../yourcomments/?ln=%(ln)s">%(your_comments)s</a></dt>
<dd>%(comments_explain)s""" % {
'ln' : ln,
'your_comments' : _("Your Comments"),
'comments_explain' : _("Display all the comments you have submitted so far."),
}
if guest and CFG_WEBSESSION_DIFFERENTIATE_BETWEEN_GUESTS:
out += self.tmpl_warning_guest_user(ln = ln, type = "baskets")
out += """</dd>
<dt><a href="../youralerts/list?ln=%(ln)s">%(your_alerts)s</a></dt>
<dd>%(explain_alerts)s""" % {
'ln' : ln,
'your_alerts' : _("Your Alerts"),
'explain_alerts' : _("Subscribe to a search which will be run periodically by our service. The result can be sent to you via Email or stored in one of your baskets."),
}
if guest and CFG_WEBSESSION_DIFFERENTIATE_BETWEEN_GUESTS:
out += self.tmpl_warning_guest_user(type="alerts", ln = ln)
out += "</dd>"
if CFG_CERN_SITE:
out += """</dd>
<dt><a href="%(CFG_SITE_SECURE_URL)s/yourloans/display?ln=%(ln)s">%(your_loans)s</a></dt>
<dd>%(explain_loans)s</dd>""" % {
'your_loans' : _("Your Loans"),
'explain_loans' : _("Check out book you have on loan, submit borrowing requests, etc. Requires CERN ID."),
'ln': ln,
'CFG_SITE_SECURE_URL': CFG_SITE_SECURE_URL
}
out += """
</dl>
</blockquote>"""
return out
def tmpl_warning_guest_user(self, ln, type):
"""
Displays a warning message about the specified type
Parameters:
- 'ln' *string* - The language to display the interface in
- 'type' *string* - The type of data that will get lost in case of guest account (for the moment: 'alerts' or 'baskets')
"""
# load the right message language
_ = gettext_set_language(ln)
if (type=='baskets'):
msg = _("You are logged in as a guest user, so your baskets will disappear at the end of the current session.") + ' '
elif (type=='alerts'):
msg = _("You are logged in as a guest user, so your alerts will disappear at the end of the current session.") + ' '
msg += _("If you wish you can %(x_url_open)slogin or register here%(x_url_close)s.", **{'x_url_open': '<a href="' + CFG_SITE_SECURE_URL + '/youraccount/login?ln=' + ln + '">',
'x_url_close': '</a>'})
return """<table class="errorbox" summary="">
<tr>
<th class="errorboxheader">%s</th>
</tr>
</table>""" % msg
def tmpl_account_body(self, ln, user):
"""
Displays the body of the actions of the user
Parameters:
- 'ln' *string* - The language to display the interface in
- 'user' *string* - The username (nickname or email)
"""
# load the right message language
_ = gettext_set_language(ln)
out = _("You are logged in as %(x_user)s. You may want to a) %(x_url1_open)slogout%(x_url1_close)s; b) edit your %(x_url2_open)saccount settings%(x_url2_close)s.") %\
{'x_user': user,
'x_url1_open': '<a href="' + CFG_SITE_SECURE_URL + '/youraccount/logout?ln=' + ln + '">',
'x_url1_close': '</a>',
'x_url2_open': '<a href="' + CFG_SITE_SECURE_URL + '/youraccount/edit?ln=' + ln + '">',
'x_url2_close': '</a>',
}
return out + "<br /><br />"
def tmpl_account_template(self, title, body, ln, url):
"""
Displays a block of the your account page
Parameters:
- 'ln' *string* - The language to display the interface in
- 'title' *string* - The title of the block
- 'body' *string* - The body of the block
- 'url' *string* - The URL to go to the proper section
"""
out ="""
<table class="youraccountbox" width="90%%" summary="" >
<tr>
<th class="youraccountheader"><a href="%s">%s</a></th>
</tr>
<tr>
<td class="youraccountbody">%s</td>
</tr>
</table>""" % (url, title, body)
return out
def tmpl_account_page(self, ln, warnings, warning_list, accBody, baskets, alerts, searches, messages, loans, groups, submissions, approvals, tickets, administrative, comments):
"""
Displays the your account page
Parameters:
- 'ln' *string* - The language to display the interface in
- 'accBody' *string* - The body of the heading block
- 'baskets' *string* - The body of the baskets block
- 'alerts' *string* - The body of the alerts block
- 'searches' *string* - The body of the searches block
- 'messages' *string* - The body of the messages block
- 'groups' *string* - The body of the groups block
- 'submissions' *string* - The body of the submission block
- 'approvals' *string* - The body of the approvals block
- 'administrative' *string* - The body of the administrative block
- 'comments' *string* - The body of the comments block
"""
# load the right message language
_ = gettext_set_language(ln)
out = ""
if warnings == "1":
out += self.tmpl_general_warnings(warning_list)
out += self.tmpl_account_template(_("Your Account"), accBody, ln, '/youraccount/edit?ln=%s' % ln)
if messages:
out += self.tmpl_account_template(_("Your Messages"), messages, ln, '/yourmessages/display?ln=%s' % ln)
if loans:
out += self.tmpl_account_template(_("Your Loans"), loans, ln, '/yourloans/display?ln=%s' % ln)
if baskets:
out += self.tmpl_account_template(_("Your Baskets"), baskets, ln, '/yourbaskets/display?ln=%s' % ln)
if comments:
comments_description = _("You can consult the list of %(x_url_open)syour comments%(x_url_close)s submitted so far.")
comments_description %= {'x_url_open': '<a href="' + CFG_SITE_URL + '/yourcomments/?ln=' + ln + '">',
'x_url_close': '</a>'}
out += self.tmpl_account_template(_("Your Comments"), comments_description, ln, '/yourcomments/?ln=%s' % ln)
if alerts:
out += self.tmpl_account_template(_("Your Alert Searches"), alerts, ln, '/youralerts/list?ln=%s' % ln)
if searches:
out += self.tmpl_account_template(_("Your Searches"), searches, ln, '/youralerts/display?ln=%s' % ln)
if groups:
groups_description = _("You can consult the list of %(x_url_open)syour groups%(x_url_close)s you are administering or are a member of.")
groups_description %= {'x_url_open': '<a href="' + CFG_SITE_URL + '/yourgroups/display?ln=' + ln + '">',
'x_url_close': '</a>'}
out += self.tmpl_account_template(_("Your Groups"), groups_description, ln, '/yourgroups/display?ln=%s' % ln)
if submissions:
submission_description = _("You can consult the list of %(x_url_open)syour submissions%(x_url_close)s and inquire about their status.")
submission_description %= {'x_url_open': '<a href="' + CFG_SITE_URL + '/yoursubmissions.py?ln=' + ln + '">',
'x_url_close': '</a>'}
out += self.tmpl_account_template(_("Your Submissions"), submission_description, ln, '/yoursubmissions.py?ln=%s' % ln)
if approvals:
approval_description = _("You can consult the list of %(x_url_open)syour approvals%(x_url_close)s with the documents you approved or refereed.")
approval_description %= {'x_url_open': '<a href="' + CFG_SITE_URL + '/yourapprovals.py?ln=' + ln + '">',
'x_url_close': '</a>'}
out += self.tmpl_account_template(_("Your Approvals"), approval_description, ln, '/yourapprovals.py?ln=%s' % ln)
#check if this user might have tickets
if tickets:
ticket_description = _("You can consult the list of %(x_url_open)syour tickets%(x_url_close)s.")
ticket_description %= {'x_url_open': '<a href="' + CFG_SITE_URL + '/yourtickets?ln=' + ln + '">',
'x_url_close': '</a>'}
out += self.tmpl_account_template(_("Your Tickets"), ticket_description, ln, '/yourtickets?ln=%s' % ln)
if administrative:
out += self.tmpl_account_template(_("Your Administrative Activities"), administrative, ln, '/admin')
return out
def tmpl_account_emailMessage(self, ln, msg):
"""
Displays a link to retrieve the lost password
Parameters:
- 'ln' *string* - The language to display the interface in
- 'msg' *string* - Explicative message on top of the form.
"""
# load the right message language
_ = gettext_set_language(ln)
out =""
out +="""
<body>
%(msg)s <a href="../youraccount/lost?ln=%(ln)s">%(try_again)s</a>
</body>
""" % {
'ln' : ln,
'msg' : msg,
'try_again' : _("Try again")
}
return out
def tmpl_account_reset_password_email_body(self, email, reset_key, ip_address, ln=CFG_SITE_LANG):
"""
The body of the email that sends lost internal account
passwords to users.
"""
_ = gettext_set_language(ln)
out = """
%(intro)s
%(intro2)s
<%(link)s>
%(outro)s
%(outro2)s""" % {
'intro': _("Somebody (possibly you) coming from %(x_ip_address)s "
"has asked\nfor a password reset at %(x_sitename)s\nfor "
"the account \"%(x_email)s\"." % {
'x_sitename' :CFG_SITE_NAME_INTL.get(ln, CFG_SITE_NAME),
'x_email' : email,
'x_ip_address' : ip_address,
}
),
'intro2' : _("If you want to reset the password for this account, please go to:"),
'link' : "%s/youraccount/resetpassword%s" %
(CFG_SITE_SECURE_URL, make_canonical_urlargd({
'ln' : ln,
'k' : reset_key
}, {})),
'outro' : _("in order to confirm the validity of this request."),
'outro2' : _("Please note that this URL will remain valid for about %(days)s days only.", days=CFG_WEBSESSION_RESET_PASSWORD_EXPIRE_IN_DAYS),
}
return out
def tmpl_account_address_activation_email_body(self, email, address_activation_key, ip_address, ln=CFG_SITE_LANG):
"""
The body of the email that sends email address activation cookie
passwords to users.
"""
_ = gettext_set_language(ln)
out = """
%(intro)s
%(intro2)s
<%(link)s>
%(outro)s
%(outro2)s""" % {
'intro': _("Somebody (possibly you) coming from %(x_ip_address)s "
"has asked\nto register a new account at %(x_sitename)s\nfor the "
"email address \"%(x_email)s\"." % {
'x_sitename' :CFG_SITE_NAME_INTL.get(ln, CFG_SITE_NAME),
'x_email' : email,
'x_ip_address' : ip_address,
}
),
'intro2' : _("If you want to complete this account registration, please go to:"),
'link' : "%s/youraccount/access%s" %
(CFG_SITE_SECURE_URL, make_canonical_urlargd({
'ln' : ln,
'mailcookie' : address_activation_key
}, {})),
'outro' : _("in order to confirm the validity of this request."),
'outro2' : _("Please note that this URL will remain valid for about %(days)s days only.", days=CFG_WEBSESSION_ADDRESS_ACTIVATION_EXPIRE_IN_DAYS),
}
return out
def tmpl_account_emailSent(self, ln, email):
"""
Displays a confirmation message for an email sent
Parameters:
- 'ln' *string* - The language to display the interface in
- 'email' *string* - The email to which the message has been sent
"""
# load the right message language
_ = gettext_set_language(ln)
out =""
out += _("Okay, a password reset link has been emailed to %(x_email)s.", x_email=email)
return out
def tmpl_account_delete(self, ln):
"""
Displays a confirmation message about deleting the account
Parameters:
- 'ln' *string* - The language to display the interface in
"""
# load the right message language
_ = gettext_set_language(ln)
out = "<p>" + _("""Deleting your account""") + '</p>'
return out
def tmpl_account_logout(self, ln):
"""
Displays a confirmation message about logging out
Parameters:
- 'ln' *string* - The language to display the interface in
"""
# load the right message language
_ = gettext_set_language(ln)
out = _("You are no longer recognized by our system.") + ' '
if CFG_EXTERNAL_AUTH_USING_SSO and CFG_EXTERNAL_AUTH_LOGOUT_SSO:
out += _("""You are still recognized by the centralized
%(x_fmt_open)sSSO%(x_fmt_close)s system. You can
%(x_url_open)slogout from SSO%(x_url_close)s, too.""") % \
{'x_fmt_open' : '<strong>', 'x_fmt_close' : '</strong>',
'x_url_open' : '<a href="%s">' % CFG_EXTERNAL_AUTH_LOGOUT_SSO,
'x_url_close' : '</a>'}
out += '<br />'
out += _("If you wish you can %(x_url_open)slogin here%(x_url_close)s.") % \
{'x_url_open': '<a href="./login?ln=' + ln + '">',
'x_url_close': '</a>'}
return out
def tmpl_login_form(self, ln, referer, internal, register_available, methods, selected_method, msg=None):
"""
Displays a login form
Parameters:
- 'ln' *string* - The language to display the interface in
- 'referer' *string* - The referer URL - will be redirected upon after login
- 'internal' *boolean* - If we are producing an internal authentication
- 'register_available' *boolean* - If users can register freely in the system
- 'methods' *array* - The available authentication methods
- 'selected_method' *string* - The default authentication method
- 'msg' *string* - The message to print before the form, if needed
"""
# load the right message language
_ = gettext_set_language(ln)
out = "<div style='float:left'>"
if msg is "":
out += "<p>%(please_login)s</p>" % {
'please_login' : cgi.escape(_("If you already have an account, please login using the form below."))
}
if CFG_CERN_SITE:
out += "<p>" + _("If you don't own a CERN account yet, you can register a %(x_url_open)snew CERN lightweight account%(x_url_close)s.", **{'x_url_open' : '<a href="https://www.cern.ch/lightweightregistration/RegisterAccount.aspx">', 'x_url_close' : '</a>'}) + "</p>"
else:
if register_available:
out += "<p>"+_("If you don't own an account yet, please %(x_url_open)sregister%(x_url_close)s an internal account.") %\
{'x_url_open': '<a href="../youraccount/register?ln=' + ln + '">',
'x_url_close': '</a>'} + "</p>"
else:
# users cannot register accounts, so advise them
# how to get one, or be silent about register
# facility if account level is more than 4:
if CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS < 5:
out += "<p>" + _("If you don't own an account yet, please contact %(x_name)s.",
x_name=('<a href="mailto:%s">%s</a>' % (cgi.escape(CFG_SITE_SUPPORT_EMAIL, True), cgi.escape(CFG_SITE_SUPPORT_EMAIL)))) + "</p>"
else:
out += "<p>%s</p>" % msg
out += """<form method="post" action="%(CFG_SITE_SECURE_URL)s/youraccount/login">
<table>
""" % {'CFG_SITE_SECURE_URL': CFG_SITE_SECURE_URL}
if len(methods) - CFG_OPENID_AUTHENTICATION - CFG_OAUTH2_AUTHENTICATION - CFG_OAUTH1_AUTHENTICATION > 1:
# more than one method, must make a select
login_select = """<select name="login_method" id="login_method">"""
for method in methods:
# OpenID/OAuth shouldn't be shown in this list.
if not method in ['openid', 'oauth1', 'oauth2']:
login_select += """<option value="%(method)s" %(selected)s>%(method)s</option>""" % {
'method' : cgi.escape(method, True),
'selected' : (method == selected_method and 'selected="selected"' or "")
}
login_select += "</select>"
out += """
<tr>
<td align="right"><strong><label for="login_method">%(login_title)s</label></strong></td>
<td>%(login_select)s</td>
</tr>""" % {
'login_title' : cgi.escape(_("Login method:")),
'login_select' : login_select,
}
else:
# only one login method available
out += """<input type="hidden" name="login_method" value="%s" />""" % cgi.escape(methods[0], True)
out += """<tr>
<td align="right">
<input type="hidden" name="ln" value="%(ln)s" />
<input type="hidden" name="referer" value="%(referer)s" />
<strong><label for="p_un">%(username)s:</label></strong>
</td>
<td><input type="text" size="25" name="p_un" id="p_un" value="" /></td>
</tr>
<tr>
<td align="right"><strong><label for="p_pw">%(password)s:</label></strong></td>
<td align="left"><input type="password" size="25" name="p_pw" id="p_pw" value="" /></td>
</tr>
<tr>
<td></td>
<td align="left"><input type="checkbox" name="remember_me" id="remember_me"/><em><label for="remember_me">%(remember_me)s</label></em></td>
<tr>
<td></td>
<td align="center" colspan="3"><input class="formbutton" type="submit" name="action" value="%(login)s" />""" % {
'ln': cgi.escape(ln, True),
'referer' : cgi.escape(referer, True),
'username' : cgi.escape(_("Username")),
'password' : cgi.escape(_("Password")),
'remember_me' : cgi.escape(_("Remember login on this computer.")),
'login' : cgi.escape(_("login")),
}
if internal:
out += """&nbsp;&nbsp;&nbsp;(<a href="./lost?ln=%(ln)s">%(lost_pass)s</a>)""" % {
'ln' : cgi.escape(ln, True),
'lost_pass' : cgi.escape(_("Lost your password?"))
}
out += """</td>
</tr>
</table></form>"""
out += """<p><strong>%(note)s:</strong> %(note_text)s</p>""" % {
'note' : cgi.escape(_("Note")),
'note_text': cgi.escape(_("You can use your nickname or your email address to login."))}
out += "</div>"
if CFG_OPENID_AUTHENTICATION or \
CFG_OAUTH2_AUTHENTICATION or \
CFG_OAUTH1_AUTHENTICATION:
# If OpenID or OAuth authentication is enabled, we put the login
# forms of providers.
out += self.tmpl_external_login_panel(ln, referer)
return out
def tmpl_lost_your_password_teaser(self, ln=CFG_SITE_LANG):
"""Displays a short sentence to attract user to the fact that
maybe he lost his password. Used by the registration page.
"""
_ = gettext_set_language(ln)
out = ""
out += """<a href="./lost?ln=%(ln)s">%(maybe_lost_pass)s</a>""" % {
'ln' : ln,
'maybe_lost_pass': ("Maybe you have lost your password?")
}
return out
def tmpl_reset_password_form(self, ln, email, reset_key, msg=''):
"""Display a form to reset the password."""
_ = gettext_set_language(ln)
out = ""
out = "<p>%s</p>" % _("Your request is valid. Please set the new "
"desired password in the following form.")
if msg:
out += """<p class='warning'>%s</p>""" % msg
out += """
<form method="post" action="../youraccount/resetpassword?ln=%(ln)s">
<input type="hidden" name="k" value="%(reset_key)s" />
<input type="hidden" name="e" value="%(email)s" />
<input type="hidden" name="reset" value="1" />
<table>
<tr><td align="right"><strong>%(set_password_for)s</strong>:</td><td><em>%(email)s</em></td></tr>
<tr><td align="right"><strong><label for="password">%(type_new_password)s:</label></strong></td>
<td><input type="password" name="password" id="password" value="123" /></td></tr>
<tr><td align="right"><strong><label for="password2">%(type_it_again)s:</label></strong></td>
<td><input type="password" name="password2" id="password2" value="" /></td></tr>
<tr><td align="center" colspan="2">
<input class="formbutton" type="submit" name="action" value="%(set_new_password)s" />
</td></tr>
</table>
</form>""" % {
'ln' : ln,
'reset_key' : reset_key,
'email' : email,
'set_password_for' : _('Set a new password for'),
'type_new_password' : _('Type the new password'),
'type_it_again' : _('Type again the new password'),
'set_new_password' : _('Set the new password')
}
return out
def tmpl_register_page(self, ln, referer, level):
"""
Displays a login form
Parameters:
- 'ln' *string* - The language to display the interface in
- 'referer' *string* - The referer URL - will be redirected upon after login
- 'level' *int* - Login level (0 - all access, 1 - accounts activated, 2+ - no self-registration)
"""
# load the right message language
_ = gettext_set_language(ln)
out = ""
if level <= 1:
out += _("Please enter your email address and desired nickname and password:")
if level == 1:
out += _("It will not be possible to use the account before it has been verified and activated.")
out += """
<form method="post" action="../youraccount/register">
<input type="hidden" name="referer" value="%(referer)s" />
<input type="hidden" name="ln" value="%(ln)s" />
<table>
<tr>
<td align="right"><strong><label for="p_email">%(email_address)s:</label></strong><br /><small class="important">(%(mandatory)s)</small></td>
<td><input type="text" size="25" name="p_email" id="p_email" value="" /><br />
<small><span class="quicknote">%(example)s:</span>
<span class="example">john.doe@example.com</span></small>
</td>
<td></td>
</tr>
<tr>
<td align="right"><strong><label for="p_nickname">%(nickname)s:</label></strong><br /><small class="important">(%(mandatory)s)</small></td>
<td><input type="text" size="25" name="p_nickname" id="p_nickname" value="" /><br />
<small><span class="quicknote">%(example)s:</span>
<span class="example">johnd</span></small>
</td>
<td></td>
</tr>
<tr>
<td align="right"><strong><label for="p_pw">%(password)s:</label></strong><br /><small class="quicknote">(%(optional)s)</small></td>
<td align="left"><input type="password" size="25" name="p_pw" id="p_pw" value="" /><br />
<small><span class="quicknote">%(note)s:</span> %(password_contain)s</small>
</td>
<td></td>
</tr>
<tr>
<td align="right"><strong><label for="p_pw2">%(retype)s:</label></strong></td>
<td align="left"><input type="password" size="25" name="p_pw2" id="p_pw2" value="" /></td>
<td></td>
</tr>
<tr>
<td></td>
<td align="left" colspan="3"><input class="formbutton" type="submit" name="action" value="%(register)s" /></td>
</tr>
</table>
</form>
<p><strong>%(note)s:</strong> %(explain_acc)s""" % {
'referer' : cgi.escape(referer),
'ln' : cgi.escape(ln),
'email_address' : _("Email address"),
'nickname' : _("Nickname"),
'password' : _("Password"),
'mandatory' : _("mandatory"),
'optional' : _("optional"),
'example' : _("Example"),
'note' : _("Note"),
'password_contain' : _("The password phrase may contain punctuation, spaces, etc."),
'retype' : _("Retype Password"),
'register' : _("register"),
'explain_acc' : _("Please do not use valuable passwords such as your Unix, AFS or NICE passwords with this service. Your email address will stay strictly confidential and will not be disclosed to any third party. It will be used to identify you for personal services of %(x_name)s. For example, you may set up an automatic alert search that will look for new preprints and will notify you daily of new arrivals by email.", x_name=CFG_SITE_NAME),
}
else:
# level >=2, so users cannot register accounts
out += "<p>" + _("It is not possible to create an account yourself. Contact %(x_name)s if you want an account.",
x_name=('<a href="mailto:%s">%s</a>' % (CFG_SITE_SUPPORT_EMAIL, CFG_SITE_SUPPORT_EMAIL))) + "</p>"
return out
def tmpl_account_adminactivities(self, ln, uid, guest, roles, activities):
"""
Displays the admin activities block for this user
Parameters:
- 'ln' *string* - The language to display the interface in
- 'uid' *string* - The used id
- 'guest' *boolean* - If the user is guest
- 'roles' *array* - The current user roles
- 'activities' *array* - The user allowed activities
"""
# load the right message language
_ = gettext_set_language(ln)
out = ""
# guest condition
if guest:
return _("You seem to be a guest user. You have to %(x_url_open)slogin%(x_url_close)s first.",
x_url_open='<a href="' + CFG_SITE_SECURE_URL + '/youraccount/login?ln=' + ln + '">',
x_url_close='<a/>')
# no rights condition
if not roles:
return "<p>" + _("You are not authorized to access administrative functions.") + "</p>"
# displaying form
out += "<p>" + _("You are enabled to the following roles: %(x_role)s.",
x_role=('<em>' + ", ".join(roles) + "</em>")) + '</p>'
if activities:
# print proposed links:
activities.sort(lambda x, y: cmp(x.lower(), y.lower()))
tmp_out = ''
for action in activities:
if action == "runbibedit":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/%s/edit/">%s</a>""" % (CFG_SITE_URL, CFG_SITE_RECORD, _("Run Record Editor"))
if action == "runbibeditmulti":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/%s/multiedit/">%s</a>""" % (CFG_SITE_URL, CFG_SITE_RECORD, _("Run Multi-Record Editor"))
if action == "runauthorlist":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/authorlist/">%s</a>""" % (CFG_SITE_URL, _("Run Author List Manager"))
if action == "runbibcirculation":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/bibcirculation/bibcirculationadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Run BibCirculation"))
if action == "runbibmerge":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/%s/merge/">%s</a>""" % (CFG_SITE_URL, CFG_SITE_RECORD, _("Run Record Merger"))
if action == "runbibswordclient":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/%s/bibsword/">%s</a>""" % (CFG_SITE_URL, CFG_SITE_RECORD, _("Run BibSword Client"))
if action == "runbatchuploader":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/batchuploader/metadata?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Run Batch Uploader"))
if action == "cfgbibformat":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/bibformat/bibformatadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure BibFormat"))
if action == "cfgbibknowledge":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/kb?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure BibKnowledge"))
if action == "cfgoaiharvest":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/oaiharvest/oaiharvestadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure OAI Harvest"))
if action == "cfgoairepository":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/oairepository/oairepositoryadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure OAI Repository"))
if action == "cfgbibindex":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/bibindex/bibindexadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure BibIndex"))
if action == "cfgbibrank":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/bibrank/bibrankadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure BibRank"))
if action == "cfgwebaccess":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/webaccess/webaccessadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure WebAccess"))
if action == "cfgwebcomment":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/webcomment/webcommentadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure WebComment"))
if action == "cfgweblinkback":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/weblinkback/weblinkbackadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure WebLinkback"))
if action == "cfgwebjournal":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/webjournal/webjournaladmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure WebJournal"))
if action == "cfgwebsearch":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/websearch/websearchadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure WebSearch"))
if action == "cfgwebsubmit":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/websubmit/websubmitadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure WebSubmit"))
if action == "runbibdocfile":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/%s/managedocfiles?ln=%s">%s</a>""" % (CFG_SITE_URL, CFG_SITE_RECORD, ln, _("Run Document File Manager"))
if action == "cfgbibsort":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/admin/bibsort/bibsortadmin.py?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Configure BibSort"))
if action == "runinfomanager":
tmp_out += """<br />&nbsp;&nbsp;&nbsp; <a href="%s/info/manage?ln=%s">%s</a>""" % (CFG_SITE_URL, ln, _("Run Info Space Manager"))
if tmp_out:
out += _("Here are some interesting web admin links for you:") + tmp_out
out += "<br />" + _("For more admin-level activities, see the complete %(x_url_open)sAdmin Area%(x_url_close)s.",
x_url_open='<a href="' + CFG_SITE_URL + '/help/admin?ln=' + ln + '">',
x_url_close='</a>')
return out
def tmpl_create_userinfobox(self, ln, url_referer, guest, username, submitter, referee, admin, usebaskets, usemessages, usealerts, usegroups, useloans, usestats):
"""
Displays the user block
Parameters:
- 'ln' *string* - The language to display the interface in
- 'url_referer' *string* - URL of the page being displayed
- 'guest' *boolean* - If the user is guest
- 'username' *string* - The username (nickname or email)
- 'submitter' *boolean* - If the user is submitter
- 'referee' *boolean* - If the user is referee
- 'admin' *boolean* - If the user is admin
- 'usebaskets' *boolean* - If baskets are enabled for the user
- 'usemessages' *boolean* - If messages are enabled for the user
- 'usealerts' *boolean* - If alerts are enabled for the user
- 'usegroups' *boolean* - If groups are enabled for the user
- 'useloans' *boolean* - If loans are enabled for the user
- 'usestats' *boolean* - If stats are enabled for the user
@note: with the update of CSS classes (cds.cds ->
invenio.css), the variables useloans etc are not used in
this function, since they are in the menus. But we keep
them in the function signature for backwards
compatibility.
"""
# load the right message language
_ = gettext_set_language(ln)
out = """<img src="%s/img/user-icon-1-20x20.gif" border="0" alt=""/> """ % CFG_SITE_URL
if guest:
out += """%(guest_msg)s ::
<a class="userinfo" href="%(sitesecureurl)s/youraccount/login?ln=%(ln)s%(referer)s">%(login)s</a>""" % {
'sitesecureurl': CFG_SITE_SECURE_URL,
'ln' : ln,
'guest_msg' : _("guest"),
'referer' : url_referer and ('&amp;referer=%s' % urllib.quote(url_referer)) or '',
'login' : _('login')
}
else:
out += """
<a class="userinfo" href="%(sitesecureurl)s/youraccount/display?ln=%(ln)s">%(username)s</a> :: """ % {
'sitesecureurl' : CFG_SITE_SECURE_URL,
'ln' : ln,
'username' : username
}
out += """<a class="userinfo" href="%(sitesecureurl)s/youraccount/logout?ln=%(ln)s">%(logout)s</a>""" % {
'sitesecureurl' : CFG_SITE_SECURE_URL,
'ln' : ln,
'logout' : _("logout"),
}
return out
def tmpl_create_useractivities_menu(self, ln, selected, url_referer, guest, username, submitter, referee, admin, usebaskets, usemessages, usealerts, usegroups, useloans, usestats, usecomments):
"""
Returns the main navigation menu with actions based on user's
priviledges
@param ln: The language to display the interface in
@type ln: string
@param selected: If the menu is currently selected
@type selected: boolean
@param url_referer: URL of the page being displayed
@type url_referer: string
@param guest: If the user is guest
@type guest: string
@param username: The username (nickname or email)
@type username: string
@param submitter: If the user is submitter
@type submitter: boolean
@param referee: If the user is referee
@type referee: boolean
@param admin: If the user is admin
@type admin: boolean
@param usebaskets: If baskets are enabled for the user
@type usebaskets: boolean
@param usemessages: If messages are enabled for the user
@type usemessages: boolean
@param usealerts: If alerts are enabled for the user
@type usealerts: boolean
@param usegroups: If groups are enabled for the user
@type usegroups: boolean
@param useloans: If loans are enabled for the user
@type useloans: boolean
@param usestats: If stats are enabled for the user
@type usestats: boolean
@param usecomments: If comments are enabled for the user
@type usecomments: boolean
@return: html menu of the user activities
@rtype: string
"""
# load the right message language
_ = gettext_set_language(ln)
out = '''<div class="hassubmenu%(on)s">
<a hreflang="en" class="header%(selected)s" href="%(CFG_SITE_SECURE_URL)s/youraccount/display?ln=%(ln)s">%(personalize)s</a>
<ul class="subsubmenu">''' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'personalize': _("Personalize"),
'on': selected and " on" or '',
'selected': selected and "selected" or ''
}
if not guest:
out += '<li><a href="%(CFG_SITE_SECURE_URL)s/youraccount/display?ln=%(ln)s">%(account)s</a></li>' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'account' : _('Your account')
}
if usealerts or guest:
out += '<li><a href="%(CFG_SITE_SECURE_URL)s/youralerts/list?ln=%(ln)s">%(alerts)s</a></li>' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'alerts' : _('Your alerts')
}
if referee:
out += '<li><a href="%(CFG_SITE_SECURE_URL)s/yourapprovals.py?ln=%(ln)s">%(approvals)s</a></li>' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'approvals' : _('Your approvals')
}
if usebaskets or guest:
out += '<li><a href="%(CFG_SITE_SECURE_URL)s/yourbaskets/display?ln=%(ln)s">%(baskets)s</a></li>' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'baskets' : _('Your baskets')
}
if usecomments:
out += '<li><a href="%(CFG_SITE_SECURE_URL)s/yourcomments?ln=%(ln)s">%(comments)s</a></li>' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'comments' : _('Your comments')
}
if usegroups:
out += '<li><a href="%(CFG_SITE_SECURE_URL)s/yourgroups/display?ln=%(ln)s">%(groups)s</a></li>' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'groups' : _('Your groups')
}
if useloans:
out += '<li><a href="%(CFG_SITE_SECURE_URL)s/yourloans/display?ln=%(ln)s">%(loans)s</a></li>' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'loans' : _('Your loans')
}
if usemessages:
out += '<li><a href="%(CFG_SITE_SECURE_URL)s/yourmessages/display?ln=%(ln)s">%(messages)s</a></li>' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'messages' : _('Your messages')
}
if submitter:
out += '<li><a href="%(CFG_SITE_SECURE_URL)s/yoursubmissions.py?ln=%(ln)s">%(submissions)s</a></li>' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'submissions' : _('Your submissions')
}
if usealerts or guest:
out += '<li><a href="%(CFG_SITE_SECURE_URL)s/youralerts/display?ln=%(ln)s">%(searches)s</a></li>' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'searches' : _('Your searches')
}
out += '</ul></div>'
return out
def tmpl_create_adminactivities_menu(self, ln, selected, url_referer, guest, username, submitter, referee, admin, usebaskets, usemessages, usealerts, usegroups, useloans, usestats, activities):
"""
Returns the main navigation menu with actions based on user's
priviledges
@param ln: The language to display the interface in
@type ln: string
@param selected: If the menu is currently selected
@type selected: boolean
@param url_referer: URL of the page being displayed
@type url_referer: string
@param guest: If the user is guest
@type guest: string
@param username: The username (nickname or email)
@type username: string
@param submitter: If the user is submitter
@type submitter: boolean
@param referee: If the user is referee
@type referee: boolean
@param admin: If the user is admin
@type admin: boolean
@param usebaskets: If baskets are enabled for the user
@type usebaskets: boolean
@param usemessages: If messages are enabled for the user
@type usemessages: boolean
@param usealerts: If alerts are enabled for the user
@type usealerts: boolean
@param usegroups: If groups are enabled for the user
@type usegroups: boolean
@param useloans: If loans are enabled for the user
@type useloans: boolean
@param usestats: If stats are enabled for the user
@type usestats: boolean
@param activities: dictionary of admin activities
@rtype activities: dict
@return: html menu of the user activities
@rtype: string
"""
# load the right message language
_ = gettext_set_language(ln)
out = ''
if activities:
out += '''<div class="hassubmenu%(on)s">
<a hreflang="en" class="header%(selected)s" href="%(CFG_SITE_SECURE_URL)s/youraccount/youradminactivities?ln=%(ln)s">%(admin)s</a>
<ul class="subsubmenu">''' % {
'CFG_SITE_SECURE_URL' : CFG_SITE_SECURE_URL,
'ln' : ln,
'admin': _("Administration"),
'on': selected and " on" or '',
'selected': selected and "selected" or ''
}
for name in sorted(activities.iterkeys()):
url = activities[name]
out += '<li><a href="%(url)s">%(name)s</a></li>' % {
'url': url,
'name': name
}
if usestats:
out += """<li><a href="%(CFG_SITE_URL)s/stats/?ln=%(ln)s">%(stats)s</a></li>""" % {
'CFG_SITE_URL' : CFG_SITE_URL,
'ln' : ln,
'stats' : _("Statistics"),
}
out += '</ul></div>'
return out
def tmpl_warning(self, warnings, ln=CFG_SITE_LANG):
"""
Display len(warnings) warning fields
@param infos: list of strings
@param ln=language
@return: html output
"""
if not((type(warnings) is list) or (type(warnings) is tuple)):
warnings = [warnings]
warningbox = ""
if warnings != []:
warningbox = "<div class=\"warningbox\">\n <b>Warning:</b>\n"
for warning in warnings:
lines = warning.split("\n")
warningbox += " <p>"
for line in lines[0:-1]:
warningbox += line + " <br />\n"
warningbox += lines[-1] + " </p>"
warningbox += "</div><br />\n"
return warningbox
def tmpl_error(self, error, ln=CFG_SITE_LANG):
"""
Display error
@param error: string
@param ln=language
@return: html output
"""
_ = gettext_set_language(ln)
errorbox = ""
if error != "":
errorbox = "<div class=\"errorbox\">\n <b>Error:</b>\n"
errorbox += " <p>"
errorbox += error + " </p>"
errorbox += "</div><br />\n"
return errorbox
def tmpl_display_all_groups(self,
infos,
admin_group_html,
member_group_html,
external_group_html = None,
warnings=[],
ln=CFG_SITE_LANG):
"""
Displays the 3 tables of groups: admin, member and external
Parameters:
- 'ln' *string* - The language to display the interface in
- 'admin_group_html' *string* - HTML code for displaying all the groups
the user is the administrator of
- 'member_group_html' *string* - HTML code for displaying all the groups
the user is member of
- 'external_group_html' *string* - HTML code for displaying all the
external groups the user is member of
"""
_ = gettext_set_language(ln)
group_text = self.tmpl_infobox(infos)
group_text += self.tmpl_warning(warnings)
if external_group_html:
group_text += """
<table>
<tr>
<td>%s</td>
</tr>
<tr>
<td><br />%s</td>
</tr>
<tr>
<td><br /><a name='external_groups'></a>%s</td>
</tr>
</table>""" %(admin_group_html, member_group_html, external_group_html)
else:
group_text += """
<table>
<tr>
<td>%s</td>
</tr>
<tr>
<td><br />%s</td>
</tr>
</table>""" %(admin_group_html, member_group_html)
return group_text
def tmpl_display_admin_groups(self, groups, ln=CFG_SITE_LANG):
"""
Display the groups the user is admin of.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'groups' *list* - All the group the user is admin of
- 'infos' *list* - Display infos on top of admin group table
"""
_ = gettext_set_language(ln)
img_link = """
<a href="%(siteurl)s/yourgroups/%(action)s?grpID=%(grpID)s&amp;ln=%(ln)s">
<img src="%(siteurl)s/img/%(img)s" alt="%(text)s" style="border:0" width="25"
height="25" /><br /><small>%(text)s</small>
</a>"""
out = self.tmpl_group_table_title(img="/img/group_admin.png",
text=_("You are an administrator of the following groups:") )
out += """
<table class="mailbox">
<thead class="mailboxheader">
<tr class="inboxheader">
<td>%s</td>
<td>%s</td>
<td style="width: 20px;" >&nbsp;</td>
<td style="width: 20px;">&nbsp;</td>
</tr>
</thead>
<tfoot>
<tr style="height:0px;">
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
</tfoot>
<tbody class="mailboxbody">""" %(_("Group"), _("Description"))
if len(groups) == 0:
out += """
<tr class="mailboxrecord" style="height: 100px;">
<td colspan="4" style="text-align: center;">
<small>%s</small>
</td>
</tr>""" %(_("You are not an administrator of any groups."),)
for group_data in groups:
(grpID, name, description) = group_data
edit_link = img_link % {'siteurl' : CFG_SITE_URL,
'grpID' : grpID,
'ln': ln,
'img':"webbasket_create_small.png",
'text':_("Edit group"),
'action':"edit"
}
members_link = img_link % {'siteurl' : CFG_SITE_URL,
'grpID' : grpID,
'ln': ln,
'img':"webbasket_usergroup.png",
'text':_("Edit %(x_num)s members", x_num=''),
'action':"members"
}
out += """
<tr class="mailboxrecord">
<td>%s</td>
<td>%s</td>
<td style="text-align: center;" >%s</td>
<td style="text-align: center;" >%s</td>
</tr>""" % (cgi.escape(name), cgi.escape(description), edit_link, members_link)
out += """
<tr class="mailboxfooter">
<td colspan="2">
<form name="newGroup" action="create?ln=%(ln)s" method="post">
<input type="submit" name="create_group" value="%(write_label)s" class="formbutton" />
</form>
</td>
<td>&nbsp;</td>
<td>&nbsp;</td>
<td>&nbsp;</td>
</tr>
</tbody>
</table>""" % {'ln': ln,
'write_label': _("Create new group"),
}
return out
def tmpl_display_member_groups(self, groups, ln=CFG_SITE_LANG):
"""
Display the groups the user is member of.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'groups' *list* - All the group the user is member of
"""
_ = gettext_set_language(ln)
group_text = self.tmpl_group_table_title(img="/img/webbasket_us.png", text=_("You are a member of the following groups:"))
group_text += """
<table class="mailbox">
<thead class="mailboxheader">
<tr class="inboxheader">
<td>%s</td>
<td>%s</td>
</tr>
</thead>
<tfoot>
<tr style="height:0px;">
<td></td>
<td></td>
</tr>
</tfoot>
<tbody class="mailboxbody">""" % (_("Group"), _("Description"))
if len(groups) == 0:
group_text += """
<tr class="mailboxrecord" style="height: 100px;">
<td colspan="2" style="text-align: center;">
<small>%s</small>
</td>
</tr>""" %(_("You are not a member of any groups."),)
for group_data in groups:
(id, name, description) = group_data
group_text += """
<tr class="mailboxrecord">
<td>%s</td>
<td>%s</td>
</tr>""" % (cgi.escape(name), cgi.escape(description))
group_text += """
<tr class="mailboxfooter">
<td>
<form name="newGroup" action="join?ln=%(ln)s" method="post">
<input type="submit" name="join_group" value="%(join_label)s" class="formbutton" />
</form>
</td>
<td>
<form name="newGroup" action="leave?ln=%(ln)s" method="post">
<input type="submit" name="leave" value="%(leave_label)s" class="formbutton" />
</form>
</td>
</tr>
</tbody>
</table>
""" % {'ln': ln,
'join_label': _("Join new group"),
'leave_label':_("Leave group")
}
return group_text
def tmpl_display_external_groups(self, groups, ln=CFG_SITE_LANG):
"""
Display the external groups the user is member of.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'groups' *list* - All the group the user is member of
"""
_ = gettext_set_language(ln)
group_text = self.tmpl_group_table_title(img="/img/webbasket_us.png", text=_("You are a member of the following external groups:"))
group_text += """
<table class="mailbox">
<thead class="mailboxheader">
<tr class="inboxheader">
<td>%s</td>
<td>%s</td>
</tr>
</thead>
<tfoot>
<tr style="height:0px;">
<td></td>
<td></td>
</tr>
</tfoot>
<tbody class="mailboxbody">""" % (_("Group"), _("Description"))
if len(groups) == 0:
group_text += """
<tr class="mailboxrecord" style="height: 100px;">
<td colspan="2" style="text-align: center;">
<small>%s</small>
</td>
</tr>""" %(_("You are not a member of any external groups."),)
for group_data in groups:
(id, name, description) = group_data
group_text += """
<tr class="mailboxrecord">
<td>%s</td>
<td>%s</td>
</tr>""" % (cgi.escape(name), cgi.escape(description))
group_text += """
</tbody>
</table>
"""
return group_text
def tmpl_display_input_group_info(self,
group_name,
group_description,
join_policy,
act_type="create",
grpID=None,
warnings=[],
ln=CFG_SITE_LANG):
"""
Display group data when creating or updating a group:
Name, description, join_policy.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'group_name' *string* - name of the group
- 'group_description' *string* - description of the group
- 'join_policy' *string* - join policy
- 'act_type' *string* - info about action : create or edit(update)
- 'grpID' *int* - ID of the group(not None in case of group editing)
- 'warnings' *list* - Display warning if values are not correct
"""
_ = gettext_set_language(ln)
#default
hidden_id =""
form_name = "create_group"
action = CFG_SITE_URL + '/yourgroups/create'
button_label = _("Create new group")
button_name = "create_button"
label = _("Create new group")
delete_text = ""
if act_type == "update":
form_name = "update_group"
action = CFG_SITE_URL + '/yourgroups/edit'
button_label = _("Update group")
button_name = "update"
label = _('Edit group %(x_name)s', x_name=cgi.escape(group_name))
delete_text = """<input type="submit" value="%s" class="formbutton" name="%s" />"""
delete_text %= (_("Delete group"),"delete")
if grpID is not None:
hidden_id = """<input type="hidden" name="grpID" value="%s" />"""
hidden_id %= grpID
out = self.tmpl_warning(warnings)
out += """
<form name="%(form_name)s" action="%(action)s" method="post">
<input type="hidden" name="ln" value="%(ln)s" />
<div style="padding:10px;">
<table class="bskbasket">
<thead class="bskbasketheader">
<tr>
<td class="bskactions">
<img src="%(logo)s" alt="%(label)s" />
</td>
<td class="bsktitle">
<b>%(label)s</b><br />
</td>
</tr>
</thead>
<tfoot>
<tr><td colspan="2"></td></tr>
</tfoot>
<tbody>
<tr>
<td colspan="2">
<table>
<tr>
<td><label for="group_name">%(name_label)s</label></td>
<td>
<input type="text" name="group_name" id="group_name" value="%(group_name)s" />
</td>
</tr>
<tr>
<td><label for="group_description">%(description_label)s</label></td>
<td>
<input type="text" name="group_description" id="group_description" value="%(group_description)s" />
</td>
</tr>
<tr>
<td>%(join_policy_label)s</td>
<td>
%(join_policy)s
</td>
</tr>
</table>
</td>
</tr>
</tbody>
</table>
%(hidden_id)s
<table>
<tr>
<td>
<input type="submit" value="%(button_label)s" class="formbutton" name="%(button_name)s" />
</td>
<td>
%(delete_text)s
</td>
<td>
<input type="submit" value="%(cancel_label)s" class="formbutton" name="cancel" />
</td>
</tr>
</table>
</div>
</form>
"""
out %= {'action' : action,
'logo': CFG_SITE_URL + '/img/webbasket_create.png',
'label': label,
'form_name' : form_name,
'name_label': _("Group name:"),
'delete_text': delete_text,
'description_label': _("Group description:"),
'join_policy_label': _("Group join policy:"),
'group_name': cgi.escape(group_name, 1),
'group_description': cgi.escape(group_description, 1),
'button_label': button_label,
'button_name':button_name,
'cancel_label':_("Cancel"),
'hidden_id':hidden_id,
'ln': ln,
'join_policy' :self.__create_join_policy_selection_menu("join_policy",
join_policy,
ln)
}
return out
def tmpl_display_input_join_group(self,
group_list,
group_name,
group_from_search,
search,
warnings=[],
ln=CFG_SITE_LANG):
"""
Display the groups the user can join.
He can use default select list or the search box
Parameters:
- 'ln' *string* - The language to display the interface in
- 'group_list' *list* - All the group the user can join
- 'group_name' *string* - Name of the group the user is looking for
- 'group_from search' *list* - List of the group the user can join matching group_name
- 'search' *int* - User is looking for group using group_name
- 'warnings' *list* - Display warning if two group are selected
"""
_ = gettext_set_language(ln)
out = self.tmpl_warning(warnings)
search_content = ""
if search:
search_content = """<tr><td>&nbsp;</td><td>"""
if group_from_search != []:
search_content += self.__create_select_menu('grpID', group_from_search, _("Please select:"))
else:
search_content += _("No matching group")
search_content += """</td><td>&nbsp;</td></tr>"""
out += """
<form name="join_group" action="%(action)s" method="post">
<input type="hidden" name="ln" value="%(ln)s" />
<div style="padding:10px;">
<table class="bskbasket">
<thead class="bskbasketheader">
<tr>
<td class="bskactions">
<img src="%(logo)s" alt="%(label)s" />
</td>
<td class="bsktitle">
<b>%(label)s</b><br />
</td>
</tr>
</thead>
<tfoot>
<tr><td colspan="2"></td></tr>
</tfoot>
<tbody>
<tr>
<td colspan="2">
<table>
<tr>
<td>%(list_label)s</td>
<td>
%(group_list)s
</td>
<td>
&nbsp;
</td>
</tr>
<tr>
<td><br /><label for="group_name">%(label2)s</label></td>
<td><br /><input type="text" name="group_name" id="group_name" value="%(group_name)s" /></td>
<td><br />
<input type="submit" name="find_button" value="%(find_label)s" class="nonsubmitbutton" />
</td>
</tr>
%(search_content)s
</table>
</td>
</tr>
</tbody>
</table>
<table>
<tr>
<td>
<input type="submit" name="join_button" value="%(label)s" class="formbutton" />
</td>
<td>
<input type="submit" value="%(cancel_label)s" class="formbutton" name="cancel" />
</td>
</tr>
</table>
</div>
</form>
"""
out %= {'action' : CFG_SITE_URL + '/yourgroups/join',
'logo': CFG_SITE_URL + '/img/webbasket_create.png',
'label': _("Join group"),
'group_name': cgi.escape(group_name, 1),
'label2':_("or find it") + ': ',
'list_label':_("Choose group:"),
'ln': ln,
'find_label': _("Find group"),
'cancel_label':_("Cancel"),
'group_list' :self.__create_select_menu("grpID",group_list, _("Please select:")),
'search_content' : search_content
}
return out
def tmpl_display_manage_member(self,
grpID,
group_name,
members,
pending_members,
infos=[],
warnings=[],
ln=CFG_SITE_LANG):
"""Display current members and waiting members of a group.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'grpID *int* - ID of the group
- 'group_name' *string* - Name of the group
- 'members' *list* - List of the current members
- 'pending_members' *list* - List of the waiting members
- 'infos' *tuple of 2 lists* - Message to inform user about his last action
- 'warnings' *list* - Display warning if two group are selected
"""
_ = gettext_set_language(ln)
out = self.tmpl_warning(warnings)
out += self.tmpl_infobox(infos)
out += """
<form name="member" action="%(action)s" method="post">
<p>%(title)s</p>
<input type="hidden" name="ln" value="%(ln)s" />
<input type="hidden" name="grpID" value="%(grpID)s"/>
<table>
<tr>
<td>
<table class="bskbasket">
<thead class="bskbasketheader">
<tr>
<td class="bskactions">
<img src="%(imgurl)s/webbasket_usergroup.png" alt="%(img_alt_header1)s" />
</td>
<td class="bsktitle">
%(header1)s<br />
&nbsp;
</td>
</tr>
</thead>
<tfoot>
<tr><td colspan="2"></td></tr>
</tfoot>
<tbody>
<tr>
<td colspan="2">
<table>
<tr>
%(member_text)s
</tr>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td>
<table class="bskbasket">
<thead class="bskbasketheader">
<tr>
<td class="bskactions">
<img src="%(imgurl)s/webbasket_usergroup_gray.png" alt="%(img_alt_header2)s" />
</td>
<td class="bsktitle">
%(header2)s<br />
&nbsp;
</td>
</tr>
</thead>
<tfoot>
<tr><td colspan="2"></td></tr>
</tfoot>
<tbody>
<tr>
<td colspan="2">
<table>
<tr>
%(pending_text)s
</tr>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td>
<table class="bskbasket" style="width: 400px">
<thead class="bskbasketheader">
<tr>
<td class="bskactions">
<img src="%(imgurl)s/iconpen.gif" alt="%(img_alt_header3)s" />
</td>
<td class="bsktitle">
<b>%(header3)s</b><br />
&nbsp;
</td>
</tr>
</thead>
<tfoot>
<tr><td colspan="2"></td></tr>
</tfoot>
<tbody>
<tr>
<td colspan="2">
<table>
<tr>
<td colspan="2" style="padding: 0 5 10 5;">%(invite_text)s</td>
</tr>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td>
<input type="submit" value="%(cancel_label)s" class="formbutton" name="cancel" />
</td>
</tr>
</table>
</form>
"""
if members :
member_list = self.__create_select_menu("member_id", members, _("Please select:"))
member_text = """
<td style="padding: 0 5 10 5;">%s</td>
<td style="padding: 0 5 10 5;">
<input type="submit" name="remove_member" value="%s" class="nonsubmitbutton"/>
</td>""" % (member_list,_("Remove member"))
else :
member_text = """<td style="padding: 0 5 10 5;" colspan="2">%s</td>""" % _("No members.")
if pending_members :
pending_list = self.__create_select_menu("pending_member_id", pending_members, _("Please select:"))
pending_text = """
<td style="padding: 0 5 10 5;">%s</td>
<td style="padding: 0 5 10 5;">
<input type="submit" name="add_member" value="%s" class="nonsubmitbutton"/>
</td>
<td style="padding: 0 5 10 5;">
<input type="submit" name="reject_member" value="%s" class="nonsubmitbutton"/>
</td>""" % (pending_list,_("Accept member"), _("Reject member"))
else :
pending_text = """<td style="padding: 0 5 10 5;" colspan="2">%s</td>""" % _("No members awaiting approval.")
header1 = self.tmpl_group_table_title(text=_("Current members"))
header2 = self.tmpl_group_table_title(text=_("Members awaiting approval"))
header3 = _("Invite new members")
write_a_message_url = create_url(
"%s/yourmessages/write" % CFG_SITE_URL,
{
'ln' : ln,
'msg_subject' : _('Invitation to join "%(x_name)s" group', x_name=escape_html(group_name)),
'msg_body' : _("""\
Hello:
I think you might be interested in joining the group "%(x_name)s".
You can join by clicking here: %(x_url)s.
Best regards.
""", **{'x_name': group_name,
'x_url': create_html_link("%s/yourgroups/join" % CFG_SITE_URL, { 'grpID' : grpID,
'join_button' : "1",
},
link_label=group_name, escape_urlargd=True, escape_linkattrd=True)})})
link_open = '<a href="%s">' % escape_html(write_a_message_url)
invite_text = _("If you want to invite new members to join your group, please use the %(x_url_open)sweb message%(x_url_close)s system.",
**{'x_url_open': link_open, 'x_url_close': '</a>'})
action = CFG_SITE_URL + '/yourgroups/members?ln=' + ln
out %= {'title':_('Group: %(x_name)s', x_name=escape_html(group_name)),
'member_text' : member_text,
'pending_text' :pending_text,
'action':action,
'grpID':grpID,
'header1': header1,
'header2': header2,
'header3': header3,
'img_alt_header1': _("Current members"),
'img_alt_header2': _("Members awaiting approval"),
'img_alt_header3': _("Invite new members"),
'invite_text': invite_text,
'imgurl': CFG_SITE_URL + '/img',
'cancel_label':_("Cancel"),
'ln':ln
}
return out
def tmpl_display_input_leave_group(self,
groups,
warnings=[],
ln=CFG_SITE_LANG):
"""Display groups the user can leave.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'groups' *list* - List of groups the user is currently member of
- 'warnings' *list* - Display warning if no group is selected
"""
_ = gettext_set_language(ln)
out = self.tmpl_warning(warnings)
out += """
<form name="leave" action="%(action)s" method="post">
<input type="hidden" name="ln" value="%(ln)s" />
<div style="padding:10px;">
<table class="bskbasket">
<thead class="bskbasketheader">
<tr>
<td class="bskactions">
<img src="%(logo)s" alt="%(label)s" />
</td>
<td class="bsktitle">
<b>%(label)s</b><br />
</td>
</tr>
</thead>
<tfoot>
<tr><td colspan="2"></td></tr>
</tfoot>
<tbody>
<tr>
<td colspan="2">
<table>
<tr>
<td>%(list_label)s</td>
<td>
%(groups)s
</td>
<td>
&nbsp;
</td>
</tr>
</table>
</td>
</tr>
</tbody>
</table>
<table>
<tr>
<td>
%(submit)s
</td>
<td>
<input type="submit" value="%(cancel_label)s" class="formbutton" name="cancel" />
</td>
</tr>
</table>
</div>
</form>
"""
if groups:
groups = self.__create_select_menu("grpID", groups, _("Please select:"))
list_label = _("Group list")
submit = """<input type="submit" name="leave_button" value="%s" class="formbutton"/>""" % _("Leave group")
else :
groups = _("You are not member of any group.")
list_label = ""
submit = ""
action = CFG_SITE_URL + '/yourgroups/leave?ln=%s'
action %= (ln)
out %= {'groups' : groups,
'list_label' : list_label,
'action':action,
'logo': CFG_SITE_URL + '/img/webbasket_create.png',
'label' : _("Leave group"),
'cancel_label':_("Cancel"),
'ln' :ln,
'submit' : submit
}
return out
def tmpl_confirm_delete(self, grpID, ln=CFG_SITE_LANG):
"""
display a confirm message when deleting a group
@param grpID *int* - ID of the group
@param ln: language
@return: html output
"""
_ = gettext_set_language(ln)
action = CFG_SITE_URL + '/yourgroups/edit'
out = """
<form name="delete_group" action="%(action)s" method="post">
<table class="confirmoperation">
<tr>
<td colspan="2" class="confirmmessage">
%(message)s
</td>
</tr>
<tr>
<td>
<input type="hidden" name="confirmed" value="1" />
<input type="hidden" name="ln" value="%(ln)s" />
<input type="hidden" name="grpID" value="%(grpID)s" />
<input type="submit" name="delete" value="%(yes_label)s" class="formbutton" />
</td>
<td>
<input type="hidden" name="ln" value="%(ln)s" />
<input type="hidden" name="grpID" value="%(grpID)s" />
<input type="submit" value="%(no_label)s" class="formbutton" />
</td>
</tr>
</table>
</form>"""% {'message': _("Are you sure you want to delete this group?"),
'ln':ln,
'yes_label': _("Yes"),
'no_label': _("No"),
'grpID':grpID,
'action': action
}
return out
def tmpl_confirm_leave(self, uid, grpID, ln=CFG_SITE_LANG):
"""
display a confirm message
@param grpID *int* - ID of the group
@param ln: language
@return: html output
"""
_ = gettext_set_language(ln)
action = CFG_SITE_URL + '/yourgroups/leave'
out = """
<form name="leave_group" action="%(action)s" method="post">
<table class="confirmoperation">
<tr>
<td colspan="2" class="confirmmessage">
%(message)s
</td>
</tr>
<tr>
<td>
<input type="hidden" name="confirmed" value="1" />
<input type="hidden" name="ln" value="%(ln)s" />
<input type="hidden" name="grpID" value="%(grpID)s" />
<input type="submit" name="leave_button" value="%(yes_label)s" class="formbutton" />
</td>
<td>
<input type="hidden" name="ln" value="%(ln)s" />
<input type="hidden" name="grpID" value="%(grpID)s" />
<input type="submit" value="%(no_label)s" class="formbutton" />
</td>
</tr>
</table>
</form>"""% {'message': _("Are you sure you want to leave this group?"),
'ln':ln,
'yes_label': _("Yes"),
'no_label': _("No"),
'grpID':grpID,
'action': action
}
return out
def __create_join_policy_selection_menu(self, name, current_join_policy, ln=CFG_SITE_LANG):
"""Private function. create a drop down menu for selection of join policy
@param current_join_policy: join policy as defined in CFG_WEBSESSION_GROUP_JOIN_POLICY
@param ln: language
"""
_ = gettext_set_language(ln)
elements = [(CFG_WEBSESSION_GROUP_JOIN_POLICY['VISIBLEOPEN'],
_("Visible and open for new members")),
(CFG_WEBSESSION_GROUP_JOIN_POLICY['VISIBLEMAIL'],
_("Visible but new members need approval"))
]
select_text = _("Please select:")
return self.__create_select_menu(name, elements, select_text, selected_key=current_join_policy)
def __create_select_menu(self, name, elements, select_text, multiple=0, selected_key=None):
""" private function, returns a popup menu
@param name: name of HTML control
@param elements: list of (key, value)
"""
if multiple :
out = """
<select name="%s" multiple="multiple" style="width:100%%">"""% (name)
else :
out = """<select name="%s" style="width:100%%">""" % name
out += '<option value="-1">%s</option>' % (select_text)
for (key, label) in elements:
selected = ''
if key == selected_key:
selected = ' selected="selected"'
out += '<option value="%s"%s>%s</option>'% (key, selected, label)
out += '</select>'
return out
def tmpl_infobox(self, infos, ln=CFG_SITE_LANG):
"""Display len(infos) information fields
@param infos: list of strings
@param ln=language
@return: html output
"""
_ = gettext_set_language(ln)
if not((type(infos) is list) or (type(infos) is tuple)):
infos = [infos]
infobox = ""
for info in infos:
infobox += '<div><span class="info">'
lines = info.split("\n")
for line in lines[0:-1]:
infobox += line + "<br />\n"
infobox += lines[-1] + "</span></div>\n"
return infobox
def tmpl_navtrail(self, ln=CFG_SITE_LANG, title=""):
"""
display the navtrail, e.g.:
Your account > Your group > title
@param title: the last part of the navtrail. Is not a link
@param ln: language
return html formatted navtrail
"""
_ = gettext_set_language(ln)
nav_h1 = '<a class="navtrail" href="%s/youraccount/display">%s</a>'
nav_h2 = ""
if (title != ""):
nav_h2 = ' &gt; <a class="navtrail" href="%s/yourgroups/display">%s</a>'
nav_h2 = nav_h2 % (CFG_SITE_URL, _("Your Groups"))
return nav_h1 % (CFG_SITE_URL, _("Your Account")) + nav_h2
def tmpl_group_table_title(self, img="", text="", ln=CFG_SITE_LANG):
"""
display the title of a table:
- 'img' *string* - img path
- 'text' *string* - title
- 'ln' *string* - The language to display the interface in
"""
out = "<div>"
if img:
out += """
<img src="%s" alt="" />
""" % (CFG_SITE_URL + img)
out += """
<b>%s</b>
</div>""" % text
return out
def tmpl_admin_msg(self, group_name, grpID, ln=CFG_SITE_LANG):
"""
return message content for joining group
- 'group_name' *string* - name of the group
- 'grpID' *int* - ID of the group
- 'ln' *string* - The language to display the interface in
"""
_ = gettext_set_language(ln)
subject = _("Group %(x_name)s: New membership request", x_name=group_name)
url = CFG_SITE_URL + "/yourgroups/members?grpID=%s&ln=%s"
url %= (grpID, ln)
# FIXME: which user? We should show his nickname.
body = (_("A user wants to join the group %(x_name)s.", x_name=group_name)) + '<br />'
body += _("Please %(x_url_open)saccept or reject%(x_url_close)s this user's request.",
x_url_open='<a href="' + url + '">',
x_url_close='</a>')
body += '<br />'
return subject, body
def tmpl_member_msg(self,
group_name,
accepted=0,
ln=CFG_SITE_LANG):
"""
return message content when new member is accepted/rejected
- 'group_name' *string* - name of the group
- 'accepted' *int* - 1 if new membership has been accepted, 0 if it has been rejected
- 'ln' *string* - The language to display the interface in
"""
_ = gettext_set_language(ln)
if accepted:
subject = _("Group %(x_name)s: Join request has been accepted", x_name=group_name)
body = _("Your request for joining group %(x_name)s has been accepted.", x_name=group_name)
else:
subject = _("Group %(x_name)s: Join request has been rejected", x_name=group_name)
body = _("Your request for joining group %(x_name)s has been rejected.", x_name=group_name)
url = CFG_SITE_URL + "/yourgroups/display?ln=" + ln
body += '<br />'
body += _("You can consult the list of %(x_url_open)syour groups%(x_url_close)s.",
x_url_open='<a href="' + url + '">',
x_url_close='</a>')
body += '<br />'
return subject, body
def tmpl_delete_msg(self,
group_name,
ln=CFG_SITE_LANG):
"""
return message content when new member is accepted/rejected
- 'group_name' *string* - name of the group
- 'ln' *string* - The language to display the interface in
"""
_ = gettext_set_language(ln)
subject = _("Group %(x_name)s has been deleted", x_name=group_name)
url = CFG_SITE_URL + "/yourgroups/display?ln=" + ln
body = _("Group %(x_name)s has been deleted by its administrator.", x_name=group_name)
body += '<br />'
body += _("You can consult the list of %(x_url_open)syour groups%(x_url_close)s.", **{'x_url_open': '<a href="' + url + '">',
'x_url_close': '</a>'})
body += '<br />'
return subject, body
def tmpl_group_info(self, nb_admin_groups=0, nb_member_groups=0, nb_total_groups=0, ln=CFG_SITE_LANG):
"""
display infos about groups (used by myaccount.py)
@param nb_admin_group: number of groups the user is admin of
@param nb_member_group: number of groups the user is member of
@param total_group: number of groups the user belongs to
@param ln: language
return: html output.
"""
_ = gettext_set_language(ln)
out = _("You can consult the list of %(x_url_open)s%(x_nb_total)i groups%(x_url_close)s you are subscribed to (%(x_nb_member)i) or administering (%(x_nb_admin)i).")
out %= {'x_url_open': '<a href="' + CFG_SITE_URL + '/yourgroups/display?ln=' + ln + '">',
'x_nb_total': nb_total_groups,
'x_url_close': '</a>',
'x_nb_admin': nb_admin_groups,
'x_nb_member': nb_member_groups}
return out
def tmpl_general_warnings(self, warning_list, ln=CFG_SITE_LANG):
"""
display information to the admin user about possible
ssecurity problems in the system.
"""
message = ""
_ = gettext_set_language(ln)
#Try and connect to the mysql database with the default invenio password
if "warning_mysql_password_equal_to_invenio_password" in warning_list:
message += "<p><font color=red>"
message += _("Warning: The password set for MySQL root user is the same as the default Invenio password. For security purposes, you may want to change the password.")
message += "</font></p>"
#Try and connect to the invenio database with the default invenio password
if "warning_invenio_password_equal_to_default" in warning_list:
message += "<p><font color=red>"
message += _("Warning: The password set for the Invenio MySQL user is the same as the shipped default. For security purposes, you may want to change the password.")
message += "</font></p>"
#Check if the admin password is empty
if "warning_empty_admin_password" in warning_list:
message += "<p><font color=red>"
message += _("Warning: The password set for the Invenio admin user is currently empty. For security purposes, it is strongly recommended that you add a password.")
message += "</font></p>"
#Check if the admin email has been changed from the default
if "warning_site_support_email_equal_to_default" in warning_list:
message += "<p><font color=red>"
message += _("Warning: The email address set for support email is currently set to info@invenio-software.org. It is recommended that you change this to your own address.")
message += "</font></p>"
#Check for a new release
if "note_new_release_available" in warning_list:
message += "<p><font color=red>"
message += _("A newer version of Invenio is available for download. You may want to visit ")
message += "<a href=\"http://invenio-software.org/wiki/Installation/Download\">http://invenio-software.org/wiki/Installation/Download</a>"
message += "</font></p>"
#Error downloading release notes
if "error_cannot_download_release_notes" in warning_list:
message += "<p><font color=red>"
message += _("Cannot download or parse release notes from http://invenio-software.org/repo/invenio/tree/RELEASE-NOTES")
message += "</font></p>"
if "email_auto_generated" in warning_list:
message += "<p><font color=red>"
message += _("Your e-mail is auto-generated by the system. Please change your e-mail from <a href='%(x_site)s/youraccount/edit?ln=%(x_link)s'>account settings</a>.",
x_site=CFG_SITE_SECURE_URL, x_link=ln)
message += "</font></p>"
return message
def tmpl_external_login_button(self, provider, referer = '', icon_size = 48,
classes = ""):
"""
Template of the login button for providers which don't need username.
@param provider: The name of the provider
@type provider: str
@param referer: The referer URL - will be redirected upon after login
@type referer: str
@param icon_size: The size of the icon of the provider
@type icon_size: int
@param classes: Additional classes for the login form
@type classes: str
@rtype: str
"""
login_url = CFG_SITE_SECURE_URL + "/youraccount/"
if provider in CFG_OPENID_PROVIDERS:
login_url += 'openid'
elif provider in CFG_OAUTH2_PROVIDERS:
login_url += 'oauth2'
elif provider in CFG_OAUTH1_PROVIDERS:
login_url += 'oauth1'
login_url += '?'
if referer:
if not 'youraccount/login' in referer:
login_url += "referer=" + referer + "&"
out = ""
out += """
<div class="login_button %(class)s" id="%(provider)s_login_button">
<div class="provider_img" id="%(provider)s_img">
<a class="openid_url" id="%(provider)s_login" href="%(loginurl)s\
provider=%(provider)s">
<img class="external_provider %(class)s" src="%(imgurl)s/\
%(provider)s_icon_%(icon_size)s.png"></img>
</a>
</div>
</div>""" % {
'loginurl': login_url,
'imgurl': CFG_SITE_SECURE_URL + "/img",
'provider': provider,
'class': classes,
'icon_size': icon_size
}
return out
def tmpl_external_login_form(self, provider, referer = '', icon_size = 48,
classes = "", label = "%(provider)s username"):
"""
Template of the login form for providers which need an username for
verification.
@param provider: The name of the provider
@type provider: str
@param referer: The referer URL - will be redirected upon after login
@type referer: str
@param icon_size: The size of the icon of the provider
@type icon_size: int
@param classes: Additional classes for the login form
@type classes: str
@param label: The label for text input.
@param label: str
@rtype: str
"""
login_url = CFG_SITE_SECURE_URL + "/youraccount/"
if provider in CFG_OPENID_PROVIDERS:
login_url += 'openid'
elif provider in CFG_OAUTH2_PROVIDERS:
login_url += 'oauth2'
elif provider in CFG_OAUTH1_PROVIDERS:
login_url += 'oauth1'
label %= {'provider': provider}
out = ""
out += """
<div class="login_button %(class)s login_form" id="%(provider)s_verify_form">
<div class="provider_img with_login_form" id="%(provider)s_login_img" \
onclick="show_username_form(this)">
<img class="external_provider %(class)s" src="%(imgurl)s/\
%(provider)s_icon_%(icon_size)s.png"></img>
</div>
<div class="login_content with_label" id="%(provider)s_verifier" hidden=\
"hidden">
<form method="get" accept-charset="UTF-8" action="%(loginurl)s">
<input type="hidden" name="provider" value="%(provider)s">
<input type="hidden" name="referer" value="%(referer)s">
<label class="openid_label" for="%(provider)s">%(label)s:</label>
</br>
<input class="openid_input" id="%(provider)s_username_field" \
type="text" name="identifier" value="" >
<input type="submit" value=" Login ">
</form>
</div>
</div>
""" % {
'loginurl': login_url,
'imgurl': CFG_SITE_SECURE_URL + "/img",
'provider': provider,
'label': label,
'referer': referer,
'class': classes,
'icon_size': icon_size
}
return out
def tmpl_external_login_panel(self, ln, referer):
"""
Template for external login buttons
"""
from invenio.legacy.websession.websession_config import CFG_EXTERNAL_LOGIN_LARGE
from invenio.legacy.websession.websession_config import CFG_EXTERNAL_LOGIN_BUTTON_ORDER
from invenio.legacy.websession.websession_config import CFG_EXTERNAL_LOGIN_FORM_LABELS
from invenio.modules.access.local_config import CFG_OPENID_CONFIGURATIONS
def construct_button(provider, size, button_class):
"""
Constructs a button for given provider.
@param provider: the name of the provider.
@type provider: str
@param size: the size of the login button
@type size: int
@param button_class: the additional class for the login button
@type button_class: str
@rtype str
"""
_ = gettext_set_language(ln)
# Look if the login button needs a form.
config = CFG_OPENID_CONFIGURATIONS.get(provider, {})
identifier = config.get('identifier', '')
if "{0}" in identifier:
label = CFG_EXTERNAL_LOGIN_FORM_LABELS.get(provider,
"%(provider)s username")
return self.tmpl_external_login_form(provider,
referer = referer,
icon_size = size,
classes = button_class,
label = _(label))
else:
return self.tmpl_external_login_button(provider,
referer = referer,
icon_size = size,
classes = button_class)
activated_providers = CFG_OPENID_PROVIDERS * CFG_OPENID_AUTHENTICATION \
+ CFG_OAUTH1_PROVIDERS * CFG_OAUTH1_AUTHENTICATION \
+ CFG_OAUTH2_PROVIDERS * CFG_OAUTH2_AUTHENTICATION
if not len(activated_providers):
return ""
out = ""
out += "<div id='buttons'>"
out += "<strong>You may login with:</strong>"
out += "<div id='big_buttons'>"
for provider in CFG_EXTERNAL_LOGIN_LARGE:
if provider in activated_providers:
out += construct_button(provider, 48, "login_button_big")
out += "</div>"
out += "<div id='small_buttons'>"
providers = CFG_EXTERNAL_LOGIN_BUTTON_ORDER
if (len(activated_providers) - len(CFG_EXTERNAL_LOGIN_LARGE)) != \
len(providers):
# Not all the providers ordered. Add the unsorted ones to the end.
for provider in sorted(activated_providers):
if not provider in providers:
providers.append(provider)
for provider in providers:
if not provider in CFG_EXTERNAL_LOGIN_LARGE:
out += construct_button(provider, 24, "login_button_small")
out += "</div>"
out += "<div id='form_field'>"
out += "</div>"
out += "</div>"
out += """
<script type="text/javascript">
function show_username_form(element) {
form_field = document.getElementById('form_field');
form_field.innerHTML = element.nextSibling.nextSibling.innerHTML;
}
</script>"""
return out
diff --git a/invenio/legacy/websession/webaccount.py b/invenio/legacy/websession/webaccount.py
index b471c20c8..b994fc785 100644
--- a/invenio/legacy/websession/webaccount.py
+++ b/invenio/legacy/websession/webaccount.py
@@ -1,484 +1,489 @@
## This file is part of Invenio.
-## Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
+## Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
__revision__ = "$Id$"
import re
import MySQLdb
import urllib
from six import iteritems
from invenio.base.globals import cfg
from invenio.config import \
CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS, \
CFG_CERN_SITE, \
CFG_SITE_LANG, \
CFG_SITE_SUPPORT_EMAIL, \
CFG_SITE_ADMIN_EMAIL, \
CFG_SITE_SECURE_URL, \
CFG_VERSION, \
CFG_SITE_RECORD
from invenio.modules.access.engine import acc_authorize_action
from invenio.modules.access.local_config import CFG_EXTERNAL_AUTHENTICATION, \
SUPERADMINROLE, CFG_EXTERNAL_AUTH_DEFAULT
from invenio.legacy.dbquery import run_sql
from invenio.legacy.webuser import getUid, get_user_preferences, \
collect_user_info
from invenio.modules.access.control import acc_find_user_role_actions
from invenio.base.i18n import gettext_set_language
from invenio.legacy.external_authentication import InvenioWebAccessExternalAuthError
import invenio.legacy.template
websession_templates = invenio.legacy.template.load('websession')
from invenio.modules import apikeys as web_api_key
def perform_info(req, ln):
"""Display the main features of CDS personalize"""
uid = getUid(req)
user_info = collect_user_info(req)
return websession_templates.tmpl_account_info(
ln = ln,
uid = uid,
guest = int(user_info['guest']),
CFG_CERN_SITE = CFG_CERN_SITE,
)
def perform_display_external_user_settings(settings, ln):
"""show external user settings which is a dictionary."""
_ = gettext_set_language(ln)
html_settings = ""
print_settings = False
settings_keys = settings.keys()
settings_keys.sort()
for key in settings_keys:
value = settings[key]
if key.startswith("EXTERNAL_") and not "HIDDEN_" in key:
print_settings = True
key = key[9:].capitalize()
html_settings += websession_templates.tmpl_external_setting(ln, key, value)
return print_settings and websession_templates.tmpl_external_user_settings(ln, html_settings) or ""
def perform_youradminactivities(user_info, ln):
"""Return text for the `Your Admin Activities' box. Analyze
whether user UID has some admin roles, and if yes, then print
suitable links for the actions he can do. If he's not admin,
print a simple non-authorized message."""
your_role_actions = acc_find_user_role_actions(user_info)
your_roles = []
your_admin_activities = []
guest = int(user_info['guest'])
for (role, action) in your_role_actions:
if role not in your_roles:
your_roles.append(role)
if action not in your_admin_activities:
your_admin_activities.append(action)
if SUPERADMINROLE in your_roles:
for action in ("runbibedit", "cfgbibformat", "cfgoaiharvest", "cfgoairepository", "cfgbibrank", "cfgbibindex", "cfgwebaccess", "cfgwebcomment", "cfgwebsearch", "cfgwebsubmiit", "cfgbibknowledge", "runbatchuploader"):
if action not in your_admin_activities:
your_admin_activities.append(action)
return websession_templates.tmpl_account_adminactivities(
ln = ln,
uid = user_info['uid'],
guest = guest,
roles = your_roles,
activities = your_admin_activities,
)
def perform_display_account(req, username, bask, aler, sear, msgs, loan, grps, sbms, appr, admn, ln, comments):
"""Display a dynamic page that shows the user's account."""
# load the right message language
_ = gettext_set_language(ln)
uid = getUid(req)
user_info = collect_user_info(req)
#your account
if int(user_info['guest']):
user = "guest"
login = "%s/youraccount/login?ln=%s" % (CFG_SITE_SECURE_URL, ln)
accBody = _("You are logged in as guest. You may want to %(x_url_open)slogin%(x_url_close)s as a regular user.") %\
{'x_url_open': '<a href="' + login + '">',
'x_url_close': '</a>'}
accBody += "<br /><br />"
bask=aler=msgs=comments= _("The %(x_fmt_open)sguest%(x_fmt_close)s users need to %(x_url_open)sregister%(x_url_close)s first") %\
{'x_fmt_open': '<strong class="headline">',
'x_fmt_close': '</strong>',
'x_url_open': '<a href="' + login + '">',
'x_url_close': '</a>'}
sear= _("No queries found")
else:
user = username
accBody = websession_templates.tmpl_account_body(
ln = ln,
user = user,
)
#Display warnings if user is superuser
roles = acc_find_user_role_actions(user_info)
warnings = "0"
warning_list = []
for role in roles:
if "superadmin" in role:
warnings = "1"
break
if warnings == "1":
warning_list.extend(superuser_account_warnings())
# Display the warning if the email of the user is autogenerated
email_autogenerated_warning = external_user_warning(uid)
if email_autogenerated_warning:
warnings = "1"
warning_list.append(email_autogenerated_warning)
#check if tickets ok
tickets = (acc_authorize_action(user_info, 'runbibedit')[0] == 0)
return websession_templates.tmpl_account_page(
ln = ln,
warnings = warnings,
warning_list = warning_list,
accBody = accBody,
baskets = bask,
alerts = aler,
searches = sear,
messages = msgs,
loans = loan,
groups = grps,
submissions = sbms,
approvals = appr,
tickets = tickets,
administrative = admn,
comments = comments,
)
def superuser_account_warnings():
"""Check to see whether admin accounts have default / blank password etc. Returns a list"""
warning_array = []
#Try and connect to the mysql database with the default invenio password
try:
conn = MySQLdb.connect (host = cfg['CFG_DATABASE_HOST'],
user = "root",
passwd = "my123p$ss",
db = "mysql")
conn.close()
warning_array.append("warning_mysql_password_equal_to_invenio_password")
except:
pass
#Try and connect to the invenio database with the default invenio password
try:
conn = MySQLdb.connect (host = cfg['CFG_DATABASE_HOST'],
user = "invenio",
passwd = "my123p$ss",
db = cfg['CFG_DATABASE_NAME'])
conn.close ()
warning_array.append("warning_invenio_password_equal_to_default")
except:
pass
#Check if the admin password is empty
res = run_sql("SELECT password, email from user where nickname = 'admin'")
if res:
res1 = run_sql("SELECT email from user where nickname = 'admin' and password = AES_ENCRYPT(%s,'')", (res[0][1], ))
else:
# no account nick-named `admin' exists; keep on going
res1 = []
for user in res1:
warning_array.append("warning_empty_admin_password")
#Check if the admin email has been changed from the default
if (CFG_SITE_ADMIN_EMAIL == "info@invenio-software.org" or CFG_SITE_SUPPORT_EMAIL == "info@invenio-software.org") and CFG_CERN_SITE == 0:
warning_array.append("warning_site_support_email_equal_to_default")
#Check for a new release of Invenio
try:
find = re.compile('Invenio v[0-9]+.[0-9]+.[0-9]+(\-rc[0-9])? is released')
webFile = urllib.urlopen("http://invenio-software.org/repo/invenio/tree/RELEASE-NOTES")
temp = ""
version = ""
version1 = ""
while 1:
temp = webFile.readline()
match1 = find.match(temp)
try:
version = match1.group()
break
except:
pass
if not temp:
break
webFile.close()
submatch = re.compile('[0-9]+.[0-9]+.[0-9]+(\-rc[0-9])?')
version1 = submatch.search(version)
web_version = version1.group().split(".")
local_version = CFG_VERSION.split(".")
if web_version[0] > local_version[0]:
warning_array.append("note_new_release_available")
elif web_version[0] == local_version[0] and web_version[1] > local_version[1]:
warning_array.append("note_new_release_available")
elif web_version[0] == local_version[0] and web_version[1] == local_version[1] and web_version[2] > local_version[2]:
warning_array.append("note_new_release_available")
except:
warning_array.append("error_cannot_download_release_notes")
return warning_array
def template_account(title, body, ln):
"""It is a template for print each of the options from the user's account."""
return websession_templates.tmpl_account_template(
ln = ln,
title = title,
body = body
)
def warning_guest_user(type, ln=CFG_SITE_LANG):
"""It returns an alert message,showing that the user is a guest user and should log into the system."""
# load the right message language
_ = gettext_set_language(ln)
return websession_templates.tmpl_warning_guest_user(
ln = ln,
type = type,
)
def perform_delete(ln):
"""Delete the account of the user, not implement yet."""
# TODO
return websession_templates.tmpl_account_delete(ln = ln)
-def perform_set(email, ln, can_config_bibcatalog = False, verbose = 0):
+def perform_set(email, ln, can_config_bibcatalog=False,
+ can_config_profiling=False, verbose=0):
"""Perform_set(email,password): edit your account parameters, email and
password.
If can_config_bibcatalog is True, show the bibcatalog dialog (if configured).
"""
try:
res = run_sql("SELECT id, nickname FROM user WHERE email=%s", (email,))
uid = res[0][0]
nickname = res[0][1]
- except:
+ except IndexError:
uid = 0
nickname = ""
CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS_LOCAL = CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS
prefs = get_user_preferences(uid)
if prefs['login_method'] in CFG_EXTERNAL_AUTHENTICATION and CFG_EXTERNAL_AUTHENTICATION[prefs['login_method']] is not None:
CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS_LOCAL = 3
out = websession_templates.tmpl_user_preferences(
ln = ln,
email = email,
email_disabled = (CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS_LOCAL >= 2),
password_disabled = (CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS_LOCAL >= 3),
nickname = nickname,
)
if len(CFG_EXTERNAL_AUTHENTICATION) > 1:
try:
uid = run_sql("SELECT id FROM user where email=%s", (email,))
uid = uid[0][0]
- except:
+ except IndexError:
uid = 0
current_login_method = prefs['login_method']
methods = CFG_EXTERNAL_AUTHENTICATION.keys()
# Filtering out methods that don't provide user_exists to check if
# a user exists in the external auth method before letting him/her
# to switch.
for method in methods:
if CFG_EXTERNAL_AUTHENTICATION[method] is not None:
try:
if not CFG_EXTERNAL_AUTHENTICATION[method].user_exists(email):
methods.remove(method)
except (AttributeError, InvenioWebAccessExternalAuthError, NotImplementedError):
methods.remove(method)
methods.sort()
if len(methods) > 1:
out += websession_templates.tmpl_user_external_auth(
ln = ln,
methods = methods,
current = current_login_method,
method_disabled = (CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS >= 4)
)
current_group_records = prefs.get('websearch_group_records', 10)
show_latestbox = prefs.get('websearch_latestbox', True)
show_helpbox = prefs.get('websearch_helpbox', True)
out += websession_templates.tmpl_user_websearch_edit(
ln = ln,
current = current_group_records,
show_latestbox = show_latestbox,
show_helpbox = show_helpbox,
)
preferred_lang = prefs.get('language', ln)
out += websession_templates.tmpl_user_lang_edit(
ln = ln,
preferred_lang = preferred_lang
)
keys_info = web_api_key.show_web_api_keys(uid=uid)
out+=websession_templates.tmpl_user_api_key(
ln = ln,
keys_info = keys_info
)
#show this dialog only if the system has been configured to use a ticket system
from invenio.config import CFG_BIBCATALOG_SYSTEM
if CFG_BIBCATALOG_SYSTEM and can_config_bibcatalog:
bibcatalog_username = prefs.get('bibcatalog_username', "")
bibcatalog_password = prefs.get('bibcatalog_password', "")
- out += websession_templates.tmpl_user_bibcatalog_auth(bibcatalog_username, \
+ out += websession_templates.tmpl_user_bibcatalog_auth(bibcatalog_username,
bibcatalog_password, ln=ln)
+ if can_config_profiling:
+ out += websession_templates.tmpl_user_profiling_settings(ln=ln,
+ enable_profiling=prefs.get('enable_profiling'))
+
if verbose >= 9:
for key, value in prefs.items():
out += "<b>%s</b>:%s<br />" % (key, value)
out += perform_display_external_user_settings(prefs, ln)
return out
def create_register_page_box(referer='', ln=CFG_SITE_LANG):
"""Register a new account."""
return websession_templates.tmpl_register_page(
referer = referer,
ln = ln,
level = CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS,
)
## create_login_page_box(): ask for the user's email and password, for login into the system
def create_login_page_box(referer='', ln=CFG_SITE_LANG):
# List of referer regexep and message to print
_ = gettext_set_language(ln)
login_referrer2msg = (
(re.compile(r"/search"), "<p>" + _("This collection is restricted. If you think you have right to access it, please authenticate yourself.") + "</p>"),
(re.compile(r"/%s/\d+/files/.+" % CFG_SITE_RECORD), "<p>" + _("This file is restricted. If you think you have right to access it, please authenticate yourself.") + "</p>"),
(re.compile(r"openid-invalid"), "<p>" + _("The OpenID identifier is invalid") + "</p>"),
(re.compile(r"openid-python"), "<p>%s</p><p>%s</p>" % (_("python-openid package must be installed: run make install-openid-package or download manually from https://github.com/openid/python-openid/"), _("Please inform the <a href='mailto%(x_email)s'>administator</a>", x_email=CFG_SITE_ADMIN_EMAIL))),
(re.compile(r"oauth-rauth"), "<p>%s</p><p>%s</p>" % (_("rauth package must be installed: run make install-oauth-package or download manually from https://github.com/litl/rauth/"), _("Please inform the <a href='mailto%(x_email)s'>administator</a>", x_email=CFG_SITE_ADMIN_EMAIL))),
(re.compile(r"oauth-config"), "<p>%s</p><p>%s</p>" % (_("The configuration isn't set properly"), _("Please inform the <a href='mailto%(x_email)s'>administator</a>", x_email=CFG_SITE_ADMIN_EMAIL))),
(re.compile(r"connection-error"), "<p>%s</p>" % (_("Cannot connect the provider. Please try again later."))),
)
msg = ""
for regexp, txt in login_referrer2msg:
if regexp.search(referer):
msg = txt
break
internal = None
for system in CFG_EXTERNAL_AUTHENTICATION.keys():
if CFG_EXTERNAL_AUTHENTICATION[system] is None:
internal = system
break
register_available = CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS <= 1 and internal
## Let's retrieve all the login method that are not dedicated to robots
methods = [method[0] for method in iteritems(CFG_EXTERNAL_AUTHENTICATION) if not method[1] or not method[1].robot_login_method_p()]
methods.sort()
return websession_templates.tmpl_login_form(
ln = ln,
referer = referer,
internal = internal,
register_available = register_available,
methods = methods,
selected_method = CFG_EXTERNAL_AUTH_DEFAULT,
msg = msg,
)
# perform_logout: display the message of not longer authorized,
def perform_logout(req, ln):
return websession_templates.tmpl_account_logout(ln = ln)
#def perform_lost: ask the user for his email, in order to send him the lost password
def perform_lost(ln):
return websession_templates.tmpl_lost_password_form(ln)
#def perform_reset_password: ask the user for a new password to reset the lost one
def perform_reset_password(ln, email, reset_key, msg=''):
return websession_templates.tmpl_reset_password_form(ln, email, reset_key, msg)
# perform_emailSent(email): confirm that the password has been emailed to 'email' address
def perform_emailSent(email, ln):
return websession_templates.tmpl_account_emailSent(ln = ln, email = email)
# peform_emailMessage : display a error message when the email introduced is not correct, and sugest to try again
def perform_emailMessage(eMsg, ln):
return websession_templates.tmpl_account_emailMessage( ln = ln,
msg = eMsg
)
# perform_back(): template for return to a previous page, used for login,register and setting
def perform_back(mess, url, linkname, ln='en'):
return websession_templates.tmpl_back_form(
ln = ln,
message = mess,
url = url,
link = linkname,
)
def external_user_warning(uid):
"""
Returns 'email_auto_generated' if the email of the user is auto-generated.
@param uid: user id
@type uid: int
@rtype: ''|'email_auto_generated'
"""
from invenio.modules.access.local_config import CFG_TEMP_EMAIL_ADDRESS
query = """
SELECT email
FROM user
WHERE id=%s
"""
params = (uid, )
email = run_sql(query, params)[0][0]
regexp = re.compile(CFG_TEMP_EMAIL_ADDRESS % "\w+", re.IGNORECASE)
query = """
SELECT *
FROM userEXT
WHERE id_user=%s
"""
if run_sql(query, params) and re.match(regexp, email):
return 'email_auto_generated'
return ''
diff --git a/invenio/legacy/websession/webinterface.py b/invenio/legacy/websession/webinterface.py
index d836207ec..acc6b7829 100644
--- a/invenio/legacy/websession/webinterface.py
+++ b/invenio/legacy/websession/webinterface.py
@@ -1,1798 +1,1810 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from invenio.legacy.webstat.api import register_customevent
"""Invenio ACCOUNT HANDLING"""
__revision__ = "$Id$"
__lastupdated__ = """$Date$"""
import cgi
from datetime import timedelta
import os
import re
from invenio.config import \
CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS, \
CFG_ACCESS_CONTROL_LEVEL_SITE, \
CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_NEW_ACCOUNT, \
CFG_SITE_NAME, \
CFG_SITE_NAME_INTL, \
CFG_SITE_SUPPORT_EMAIL, \
CFG_SITE_SECURE_URL, \
CFG_SITE_URL, \
CFG_CERN_SITE, \
CFG_WEBSESSION_RESET_PASSWORD_EXPIRE_IN_DAYS
from invenio.legacy import webuser
from invenio.legacy.webpage import page
from invenio.legacy.websession import webaccount
from invenio.legacy.webbasket import api as webbasket
from invenio.legacy.webalert import api as webalert
from invenio.legacy.dbquery import run_sql
from invenio.legacy.webmessage.api import account_new_mail
from invenio.modules.access.engine import acc_authorize_action
from invenio.ext.legacy.handler import wash_urlargd, WebInterfaceDirectory
from invenio.utils.apache import SERVER_RETURN, HTTP_NOT_FOUND
from invenio.utils.url import redirect_to_url, make_canonical_urlargd
from invenio.legacy.websession import webgroup
from invenio.legacy.websession import dblayer as webgroup_dblayer
from invenio.base.i18n import gettext_set_language, wash_language
from invenio.ext.email import send_email
from invenio.ext.logging import register_exception
from invenio.modules.access.mailcookie import mail_cookie_retrieve_kind, \
mail_cookie_check_pw_reset, mail_cookie_delete_cookie, \
mail_cookie_create_pw_reset, mail_cookie_check_role, \
mail_cookie_check_mail_activation, InvenioWebAccessMailCookieError, \
InvenioWebAccessMailCookieDeletedError, mail_cookie_check_authorize_action
from invenio.modules.access.local_config import CFG_WEBACCESS_WARNING_MSGS, \
CFG_EXTERNAL_AUTH_USING_SSO, CFG_EXTERNAL_AUTH_LOGOUT_SSO, \
CFG_EXTERNAL_AUTHENTICATION, CFG_EXTERNAL_AUTH_SSO_REFRESH, \
CFG_OPENID_CONFIGURATIONS, CFG_OAUTH2_CONFIGURATIONS, \
CFG_OAUTH1_CONFIGURATIONS, CFG_OAUTH2_PROVIDERS, CFG_OAUTH1_PROVIDERS, \
CFG_OPENID_PROVIDERS, CFG_OPENID_AUTHENTICATION, \
CFG_OAUTH1_AUTHENTICATION, CFG_OAUTH2_AUTHENTICATION
from invenio.legacy.websession.session import get_session
from invenio.modules import apikeys as web_api_key
import invenio.legacy.template
websession_templates = invenio.legacy.template.load('websession')
bibcatalog_templates = invenio.legacy.template.load('bibcatalog')
class WebInterfaceYourAccountPages(WebInterfaceDirectory):
_exports = ['', 'edit', 'change', 'lost', 'display',
'send_email', 'youradminactivities', 'access',
'delete', 'logout', 'login', 'register', 'resetpassword',
'robotlogin', 'robotlogout', 'apikey', 'openid',
'oauth1', 'oauth2']
_force_https = True
def index(self, req, form):
redirect_to_url(req, '%s/youraccount/display' % CFG_SITE_SECURE_URL)
def access(self, req, form):
args = wash_urlargd(form, {'mailcookie' : (str, '')})
_ = gettext_set_language(args['ln'])
title = _("Mail Cookie Service")
try:
kind = mail_cookie_retrieve_kind(args['mailcookie'])
if kind == 'pw_reset':
redirect_to_url(req, '%s/youraccount/resetpassword?k=%s&ln=%s' % (CFG_SITE_SECURE_URL, args['mailcookie'], args['ln']))
elif kind == 'role':
uid = webuser.getUid(req)
try:
(role_name, expiration) = mail_cookie_check_role(args['mailcookie'], uid)
except InvenioWebAccessMailCookieDeletedError:
return page(title=_("Role authorization request"), req=req, body=_("This request for an authorization has already been authorized."), uid=webuser.getUid(req), navmenuid='youraccount', language=args['ln'], secure_page_p=1)
return page(title=title,
body=webaccount.perform_back(
_("You have successfully obtained an authorization as %(x_role)s! "
"This authorization will last until %(x_expiration)s and until "
"you close your browser if you are a guest user.") %
{'x_role' : '<strong>%s</strong>' % role_name,
'x_expiration' : '<em>%s</em>' % expiration.strftime("%Y-%m-%d %H:%M:%S")},
'/youraccount/display?ln=%s' % args['ln'], _('login'), args['ln']),
req=req,
uid=webuser.getUid(req),
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount',
secure_page_p=1)
elif kind == 'mail_activation':
try:
email = mail_cookie_check_mail_activation(args['mailcookie'])
if not email:
raise StandardError
webuser.confirm_email(email)
body = "<p>" + _("You have confirmed the validity of your email"
" address!") + "</p>"
if CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS == 1:
body += "<p>" + _("Please, wait for the administrator to "
"enable your account.") + "</p>"
else:
uid = webuser.update_Uid(req, email)
body += "<p>" + _("You can now go to %(x_url_open)syour account page%(x_url_close)s.") % {'x_url_open' : '<a href="/youraccount/display?ln=%s">' % args['ln'], 'x_url_close' : '</a>'} + "</p>"
return page(title=_("Email address successfully activated"),
body=body, req=req, language=args['ln'], uid=webuser.getUid(req), lastupdated=__lastupdated__, navmenuid='youraccount', secure_page_p=1)
except InvenioWebAccessMailCookieDeletedError as e:
body = "<p>" + _("You have already confirmed the validity of your email address!") + "</p>"
if CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS == 1:
body += "<p>" + _("Please, wait for the administrator to "
"enable your account.") + "</p>"
else:
body += "<p>" + _("You can now go to %(x_url_open)syour account page%(x_url_close)s.") % {'x_url_open' : '<a href="/youraccount/display?ln=%s">' % args['ln'], 'x_url_close' : '</a>'} + "</p>"
return page(title=_("Email address successfully activated"),
body=body, req=req, language=args['ln'], uid=webuser.getUid(req), lastupdated=__lastupdated__, navmenuid='youraccount', secure_page_p=1)
return webuser.page_not_authorized(req, "../youraccount/access",
text=_("This request for confirmation of an email "
"address is not valid or"
" is expired."), navmenuid='youraccount')
except InvenioWebAccessMailCookieError:
return webuser.page_not_authorized(req, "../youraccount/access",
text=_("This request for an authorization is not valid or"
" is expired."), navmenuid='youraccount')
def resetpassword(self, req, form):
args = wash_urlargd(form, {
'k' : (str, ''),
'reset' : (int, 0),
'password' : (str, ''),
'password2' : (str, '')
})
_ = gettext_set_language(args['ln'])
title = _('Reset password')
reset_key = args['k']
try:
email = mail_cookie_check_pw_reset(reset_key)
except InvenioWebAccessMailCookieDeletedError:
return page(title=title, req=req, body=_("This request for resetting a password has already been used."), uid=webuser.getUid(req), navmenuid='youraccount', language=args['ln'], secure_page_p=1)
except InvenioWebAccessMailCookieError:
return webuser.page_not_authorized(req, "../youraccount/access",
text=_("This request for resetting a password is not valid or"
" is expired."), navmenuid='youraccount')
if email is None or CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS >= 3:
return webuser.page_not_authorized(req, "../youraccount/resetpassword",
text=_("This request for resetting the password is not valid or"
" is expired."), navmenuid='youraccount')
if not args['reset']:
return page(title=title,
body=webaccount.perform_reset_password(args['ln'], email, reset_key),
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
elif args['password'] != args['password2']:
msg = _('The two provided passwords aren\'t equal.')
return page(title=title,
body=webaccount.perform_reset_password(args['ln'], email, reset_key, msg),
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
run_sql('UPDATE user SET password=AES_ENCRYPT(email,%s) WHERE email=%s', (args['password'], email))
mail_cookie_delete_cookie(reset_key)
return page(title=title,
body=webaccount.perform_back(
_("The password was successfully set! "
"You can now proceed with the login."),
CFG_SITE_SECURE_URL + '/youraccount/login?ln=%s' % args['ln'], _('login'), args['ln']),
req=req,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount', secure_page_p=1)
def display(self, req, form):
args = wash_urlargd(form, {})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(args['ln'])
if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../youraccount/display",
navmenuid='youraccount')
if webuser.isGuestUser(uid):
return page(title=_("Your Account"),
body=webaccount.perform_info(req, args['ln']),
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
username = webuser.get_nickname_or_email(uid)
user_info = webuser.collect_user_info(req)
bask = user_info['precached_usebaskets'] and webbasket.account_list_baskets(uid, ln=args['ln']) or ''
aler = user_info['precached_usealerts'] and webalert.account_list_alerts(uid, ln=args['ln']) or ''
sear = webalert.account_list_searches(uid, ln=args['ln'])
msgs = user_info['precached_usemessages'] and account_new_mail(uid, ln=args['ln']) or ''
grps = user_info['precached_usegroups'] and webgroup.account_group(uid, ln=args['ln']) or ''
appr = user_info['precached_useapprove']
sbms = user_info['precached_viewsubmissions']
comments = user_info['precached_sendcomments']
loan = ''
admn = webaccount.perform_youradminactivities(user_info, args['ln'])
return page(title=_("Your Account"),
body=webaccount.perform_display_account(req, username, bask, aler, sear, msgs, loan, grps, sbms, appr, admn, args['ln'], comments),
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
def apikey(self, req, form):
args = wash_urlargd(form, {
'key_description' : (str, None),
'key_id' : (str, None),
'referer': (str, '')
})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(args['ln'])
if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../youraccount/edit",
navmenuid='youraccount')
if webuser.isGuestUser(uid):
return webuser.page_not_authorized(req, "../youraccount/edit",
text=_("This functionality is forbidden to guest users."),
navmenuid='youraccount')
if args['key_id']:
web_api_key.mark_web_api_key_as_removed(args['key_id'])
else:
uid = webuser.getUid(req)
web_api_key.create_new_web_api_key(uid, args['key_description'])
if args['referer']:
redirect_to_url(req, args['referer'])
else:
redirect_to_url(req, '%s/youraccount/edit?ln=%s' % (CFG_SITE_SECURE_URL, args['ln']))
def edit(self, req, form):
args = wash_urlargd(form, {"verbose" : (int, 0)})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(args['ln'])
if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../youraccount/edit",
navmenuid='youraccount')
if webuser.isGuestUser(uid):
return webuser.page_not_authorized(req, "../youraccount/edit",
text=_("This functionality is forbidden to guest users."),
navmenuid='youraccount')
body = ''
user_info = webuser.collect_user_info(req)
if args['verbose'] == 9:
keys = user_info.keys()
keys.sort()
for key in keys:
body += "<b>%s</b>:%s<br />" % (key, user_info[key])
#check if the user should see bibcatalog user name / passwd in the settings
can_config_bibcatalog = (acc_authorize_action(user_info, 'runbibedit')[0] == 0)
+ can_config_profiling = (acc_authorize_action(user_info, 'profiling')[0] == 0)
return page(title= _("Your Settings"),
body=body+webaccount.perform_set(webuser.get_email(uid),
- args['ln'], can_config_bibcatalog,
+ args['ln'],
+ can_config_bibcatalog,
+ can_config_profiling,
verbose=args['verbose']),
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description=_("%(x_name)s Personalize, Your Settings", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
def change(self, req, form):
args = wash_urlargd(form, {
'nickname': (str, None),
'email': (str, None),
'old_password': (str, None),
'password': (str, None),
'password2': (str, None),
'login_method': (str, ""),
'group_records' : (int, None),
'latestbox' : (int, None),
'helpbox' : (int, None),
'lang' : (str, None),
'bibcatalog_username' : (str, None),
'bibcatalog_password' : (str, None),
+ 'profiling' : (int, 0),
})
## Wash arguments:
args['login_method'] = wash_login_method(args['login_method'])
if args['email']:
args['email'] = args['email'].lower()
## Load the right message language:
_ = gettext_set_language(args['ln'])
## Identify user and load old preferences:
uid = webuser.getUid(req)
prefs = webuser.get_user_preferences(uid)
## Check rights:
if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../youraccount/change",
navmenuid='youraccount')
# FIXME: the branching below is far from optimal. Should be
# based on the submitted form name ids, to know precisely on
# which form the user clicked. Not on the passed values, as
# is the case now. The function body is too big and in bad
# need of refactoring anyway.
## Will hold the output messages:
mess = ''
## Would hold link to previous page and title for the link:
act = None
linkname = None
title = None
## Change login method if needed:
if args['login_method'] and CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS < 4 \
and args['login_method'] in CFG_EXTERNAL_AUTHENTICATION:
title = _("Settings edited")
act = "/youraccount/display?ln=%s" % args['ln']
linkname = _("Show account")
if prefs['login_method'] != args['login_method']:
if CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS >= 4:
mess += '<p>' + _("Unable to change login method.")
elif not CFG_EXTERNAL_AUTHENTICATION[args['login_method']]:
# Switching to internal authentication: we drop any external datas
p_email = webuser.get_email(uid)
webuser.drop_external_settings(uid)
webgroup_dblayer.drop_external_groups(uid)
prefs['login_method'] = args['login_method']
webuser.set_user_preferences(uid, prefs)
mess += "<p>" + _("Switched to internal login method.") + " "
mess += _("Please note that if this is the first time that you are using this account "
"with the internal login method then the system has set for you "
"a randomly generated password. Please click the "
"following button to obtain a password reset request "
"link sent to you via email:") + '</p>'
mess += """<p><form method="post" action="../youraccount/send_email">
<input type="hidden" name="p_email" value="%s">
<input class="formbutton" type="submit" value="%s">
</form></p>""" % (p_email, _("Send Password"))
else:
res = run_sql("SELECT email FROM user WHERE id=%s", (uid,))
if res:
email = res[0][0]
else:
email = None
if not email:
mess += '<p>' + _("Unable to switch to external login method %(x_name)s, because your email address is unknown.",
x_name=cgi.escape(args['login_method']))
else:
try:
if not CFG_EXTERNAL_AUTHENTICATION[args['login_method']].user_exists(email):
mess += '<p>' + _("Unable to switch to external login method %(x_meth)s, because your email address is unknown to the external login system.",
x_meth=cgi.escape(args['login_method']))
else:
prefs['login_method'] = args['login_method']
webuser.set_user_preferences(uid, prefs)
mess += '<p>' + _("Login method successfully selected.")
except AttributeError:
mess += '<p>' + _("The external login method %(x_name)s does not support email address based logins. Please contact the site administrators.",
x_name=cgi.escape(args['login_method']))
## Change email or nickname:
if args['email'] or args['nickname']:
uid2 = webuser.emailUnique(args['email'])
uid_with_the_same_nickname = webuser.nicknameUnique(args['nickname'])
current_nickname = webuser.get_nickname(uid)
if current_nickname and args['nickname'] and \
current_nickname != args['nickname']:
# User tried to set nickname while one is already
# defined (policy is that nickname is not to be
# changed)
mess += '<p>' + _("Your nickname has not been updated")
elif (CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS >= 2 or (CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS <= 1 and \
webuser.email_valid_p(args['email']))) \
and (args['nickname'] is None or webuser.nickname_valid_p(args['nickname'])) \
and uid2 != -1 and (uid2 == uid or uid2 == 0) \
and uid_with_the_same_nickname != -1 and (uid_with_the_same_nickname == uid or uid_with_the_same_nickname == 0):
if CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS < 3:
change = webuser.updateDataUser(uid,
args['email'],
args['nickname'])
else:
return webuser.page_not_authorized(req, "../youraccount/change",
navmenuid='youraccount')
if change:
mess += '<p>' + _("Settings successfully edited.")
mess += '<p>' + _("Note that if you have changed your email address, "
"you will have to %(x_url_open)sreset your password%(x_url_close)s anew.") % \
{'x_url_open': '<a href="%s">' % (CFG_SITE_SECURE_URL + '/youraccount/lost?ln=%s' % args['ln']),
'x_url_close': '</a>'}
act = "/youraccount/display?ln=%s" % args['ln']
linkname = _("Show account")
title = _("Settings edited")
elif args['nickname'] is not None and not webuser.nickname_valid_p(args['nickname']):
mess += '<p>' + _("Desired nickname %(x_name)s is invalid.", x_name=cgi.escape(args['nickname']))
mess += " " + _("Please try again.")
act = "/youraccount/edit?ln=%s" % args['ln']
linkname = _("Edit settings")
title = _("Editing settings failed")
elif not webuser.email_valid_p(args['email']):
mess += '<p>' + _("Supplied email address %(x_name)s is invalid.", x_name=cgi.escape(args['email']))
mess += " " + _("Please try again.")
act = "/youraccount/edit?ln=%s" % args['ln']
linkname = _("Edit settings")
title = _("Editing settings failed")
elif uid2 == -1 or uid2 != uid and not uid2 == 0:
mess += '<p>' + _("Supplied email address %(x_email)s already exists in the database.", x_email=cgi.escape(args['email']))
mess += " " + websession_templates.tmpl_lost_your_password_teaser(args['ln'])
mess += " " + _("Or please try again.")
act = "/youraccount/edit?ln=%s" % args['ln']
linkname = _("Edit settings")
title = _("Editing settings failed")
elif uid_with_the_same_nickname == -1 or uid_with_the_same_nickname != uid and not uid_with_the_same_nickname == 0:
mess += '<p>' + _("Desired nickname %(x_name)s is already in use.", x_name=cgi.escape(args['nickname']))
mess += " " + _("Please try again.")
act = "/youraccount/edit?ln=%s" % args['ln']
linkname = _("Edit settings")
title = _("Editing settings failed")
## Change passwords:
if args['old_password'] or args['password'] or args['password2']:
if CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS >= 3:
mess += '<p>' + _("Users cannot edit passwords on this site.")
else:
res = run_sql("SELECT id FROM user "
"WHERE AES_ENCRYPT(email,%s)=password AND id=%s",
(args['old_password'], uid))
if res:
if args['password'] == args['password2']:
webuser.updatePasswordUser(uid, args['password'])
mess += '<p>' + _("Password successfully edited.")
act = "/youraccount/display?ln=%s" % args['ln']
linkname = _("Show account")
title = _("Password edited")
else:
mess += '<p>' + _("Both passwords must match.")
mess += " " + _("Please try again.")
act = "/youraccount/edit?ln=%s" % args['ln']
linkname = _("Edit settings")
title = _("Editing password failed")
else:
mess += '<p>' + _("Wrong old password inserted.")
mess += " " + _("Please try again.")
act = "/youraccount/edit?ln=%s" % args['ln']
linkname = _("Edit settings")
title = _("Editing password failed")
## Change search-related settings:
if args['group_records']:
prefs = webuser.get_user_preferences(uid)
prefs['websearch_group_records'] = args['group_records']
prefs['websearch_latestbox'] = args['latestbox']
prefs['websearch_helpbox'] = args['helpbox']
webuser.set_user_preferences(uid, prefs)
title = _("Settings edited")
act = "/youraccount/display?ln=%s" % args['ln']
linkname = _("Show account")
mess += '<p>' + _("User settings saved correctly.")
## Change language-related settings:
if args['lang']:
lang = wash_language(args['lang'])
prefs = webuser.get_user_preferences(uid)
prefs['language'] = lang
args['ln'] = lang
_ = gettext_set_language(lang)
webuser.set_user_preferences(uid, prefs)
title = _("Settings edited")
act = "/youraccount/display?ln=%s" % args['ln']
linkname = _("Show account")
mess += '<p>' + _("User settings saved correctly.")
## Edit cataloging-related settings:
if args['bibcatalog_username'] or args['bibcatalog_password']:
act = "/youraccount/display?ln=%s" % args['ln']
linkname = _("Show account")
- if ((len(args['bibcatalog_username']) == 0) or (len(args['bibcatalog_password']) == 0)):
+ if len(args['bibcatalog_username']) == 0 or len(args['bibcatalog_password']) == 0:
title = _("Editing bibcatalog authorization failed")
mess += '<p>' + _("Empty username or password")
else:
title = _("Settings edited")
prefs['bibcatalog_username'] = args['bibcatalog_username']
prefs['bibcatalog_password'] = args['bibcatalog_password']
webuser.set_user_preferences(uid, prefs)
mess += '<p>' + _("User settings saved correctly.")
+ if 'profiling' in args:
+ user_info = webuser.collect_user_info(req)
+ can_config_profiling = (acc_authorize_action(user_info, 'profiling')[0] == 0)
+ if can_config_profiling:
+ prefs['enable_profiling'] = bool(args['profiling'])
+ webuser.set_user_preferences(uid, prefs)
+ mess += '<p>' + _("User settings saved correctly.")
+
if not mess:
mess = _("Unable to update settings.")
if not act:
act = "/youraccount/edit?ln=%s" % args['ln']
if not linkname:
linkname = _("Edit settings")
if not title:
title = _("Editing settings failed")
## Finally, output the results:
return page(title=title,
body=webaccount.perform_back(mess, act, linkname, args['ln']),
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
def lost(self, req, form):
args = wash_urlargd(form, {})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(args['ln'])
if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../youraccount/lost",
navmenuid='youraccount')
return page(title=_("Lost your password?"),
body=webaccount.perform_lost(args['ln']),
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
def send_email(self, req, form):
# set all the declared query fields as local variables
args = wash_urlargd(form, {'p_email': (str, None)})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(args['ln'])
if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../youraccount/send_email",
navmenuid='youraccount')
user_prefs = webuser.get_user_preferences(webuser.emailUnique(args['p_email']))
if user_prefs:
if user_prefs['login_method'] in CFG_EXTERNAL_AUTHENTICATION and \
CFG_EXTERNAL_AUTHENTICATION[user_prefs['login_method']] is not None:
eMsg = _("Cannot send password reset request since you are using external authentication system.")
return page(title=_("Your Account"),
body=webaccount.perform_emailMessage(eMsg, args['ln']),
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid, req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
try:
reset_key = mail_cookie_create_pw_reset(args['p_email'], cookie_timeout=timedelta(days=CFG_WEBSESSION_RESET_PASSWORD_EXPIRE_IN_DAYS))
except InvenioWebAccessMailCookieError:
reset_key = None
if reset_key is None:
eMsg = _("The entered email address does not exist in the database.")
return page(title=_("Your Account"),
body=webaccount.perform_emailMessage(eMsg, args['ln']),
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid, req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
ip_address = req.remote_host or req.remote_ip
if not send_email(CFG_SITE_SUPPORT_EMAIL, args['p_email'], "%s %s"
% (_("Password reset request for"),
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
websession_templates.tmpl_account_reset_password_email_body(
args['p_email'],reset_key, ip_address, args['ln'])):
eMsg = _("The entered email address is incorrect, please check that it is written correctly (e.g. johndoe@example.com).")
return page(title=_("Incorrect email address"),
body=webaccount.perform_emailMessage(eMsg, args['ln']),
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
return page(title=_("Reset password link sent"),
body=webaccount.perform_emailSent(args['p_email'], args['ln']),
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid, req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
def youradminactivities(self, req, form):
args = wash_urlargd(form, {})
uid = webuser.getUid(req)
user_info = webuser.collect_user_info(req)
# load the right message language
_ = gettext_set_language(args['ln'])
if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../youraccount/youradminactivities",
navmenuid='admin')
return page(title=_("Your Administrative Activities"),
body=webaccount.perform_youradminactivities(user_info, args['ln']),
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='admin')
def delete(self, req, form):
args = wash_urlargd(form, {})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(args['ln'])
if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../youraccount/delete",
navmenuid='youraccount')
return page(title=_("Delete Account"),
body=webaccount.perform_delete(args['ln']),
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
def logout(self, req, form):
args = wash_urlargd(form, {})
uid = webuser.logoutUser(req)
# load the right message language
_ = gettext_set_language(args['ln'])
if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../youraccount/logout",
navmenuid='youraccount')
if CFG_EXTERNAL_AUTH_USING_SSO:
return redirect_to_url(req, CFG_EXTERNAL_AUTH_LOGOUT_SSO)
return page(title=_("Logout"),
body=webaccount.perform_logout(req, args['ln']),
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords=_("%(x_name)s, personalize", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
def robotlogout(self, req, form):
"""
Implement logout method for external service providers.
"""
webuser.logoutUser(req)
redirect_to_url(req, "%s/img/pix.png" % CFG_SITE_SECURE_URL)
def robotlogin(self, req, form):
"""
Implement authentication method for external service providers.
"""
from invenio.legacy.external_authentication import InvenioWebAccessExternalAuthError
args = wash_urlargd(form, {
'login_method': (str, None),
'remember_me' : (str, ''),
'referer': (str, ''),
'p_un': (str, ''),
'p_pw': (str, '')
})
# sanity checks:
args['login_method'] = wash_login_method(args['login_method'])
args['remember_me'] = args['remember_me'] != ''
locals().update(args)
if CFG_ACCESS_CONTROL_LEVEL_SITE > 0:
return webuser.page_not_authorized(req, CFG_SITE_SECURE_URL + "/youraccount/login?ln=%s" % args['ln'],
navmenuid='youraccount')
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(args['ln'])
try:
(iden, args['p_un'], args['p_pw'], msgcode) = webuser.loginUser(req, args['p_un'], args['p_pw'], args['login_method'])
except InvenioWebAccessExternalAuthError as err:
return page("Error", body=str(err), req=req)
if iden:
uid = webuser.update_Uid(req, args['p_un'], args['remember_me'])
uid2 = webuser.getUid(req)
if uid2 == -1:
webuser.logoutUser(req)
return webuser.page_not_authorized(req, CFG_SITE_SECURE_URL + "/youraccount/login?ln=%s" % args['ln'], uid=uid,
navmenuid='youraccount')
# login successful!
if args['referer']:
redirect_to_url(req, args['referer'])
else:
return self.display(req, form)
else:
mess = CFG_WEBACCESS_WARNING_MSGS[msgcode] % cgi.escape(args['login_method'])
if msgcode == 14:
if webuser.username_exists_p(args['p_un']):
mess = CFG_WEBACCESS_WARNING_MSGS[15] % cgi.escape(args['login_method'])
act = CFG_SITE_SECURE_URL + '/youraccount/login%s' % make_canonical_urlargd({'ln' : args['ln'], 'referer' : args['referer']}, {})
return page(title=_("Login"),
body=webaccount.perform_back(mess, act, _("login"), args['ln']),
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords="%s , personalize" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
def login(self, req, form):
args = wash_urlargd(form, {
'p_un': (str, None),
'p_pw': (str, None),
'login_method': (str, None),
'provider': (str, None),
'action': (str, ''),
'remember_me' : (str, ''),
'referer': (str, '')})
# sanity checks:
args['login_method'] = wash_login_method(args['login_method'])
if args['p_un']:
args['p_un'] = args['p_un'].strip()
args['remember_me'] = args['remember_me'] != ''
locals().update(args)
if CFG_ACCESS_CONTROL_LEVEL_SITE > 0:
return webuser.page_not_authorized(req, CFG_SITE_SECURE_URL + "/youraccount/login?ln=%s" % args['ln'],
navmenuid='youraccount')
uid = webuser.getUid(req)
# If user has logged in to ORCID through oauth2, store his ORCID id
if uid > 0 and args['login_method'] == 'oauth2' and args['provider'] == 'orcid':
from invenio.bibauthorid_webapi import get_pid_from_uid, add_orcid_to_pid
CFG_EXTERNAL_AUTHENTICATION['oauth2'].auth_user(None, None, req)
pid = get_pid_from_uid(uid)
try:
orcid = str(req.g['oauth2_orcid'])
except KeyError:
return redirect_to_url(req, '%s/author/manage_profile/%s' % (CFG_SITE_SECURE_URL, pid))
add_orcid_to_pid(pid, orcid)
return redirect_to_url(req, '%s/author/manage_profile/%s' % (CFG_SITE_SECURE_URL, pid))
# If user is already logged in, redirect it to referer or your account
# page
if uid > 0:
redirect_to_url(req, args['referer'] or '%s/youraccount/display?ln=%s' % (CFG_SITE_SECURE_URL, args['ln']))
# load the right message language
_ = gettext_set_language(args['ln'])
if args['action']:
cookie = args['action']
try:
action, arguments = mail_cookie_check_authorize_action(cookie)
except InvenioWebAccessMailCookieError:
pass
if not CFG_EXTERNAL_AUTH_USING_SSO:
if (args['p_un'] is None or not args['login_method']) and (not args['login_method'] in ['openid', 'oauth1', 'oauth2']):
return page(title=_("Login"),
body=webaccount.create_login_page_box(args['referer'], args['ln']),
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords="%s , personalize" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid=uid,
req=req,
secure_page_p=1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
(iden, args['p_un'], args['p_pw'], msgcode) = webuser.loginUser(req, args['p_un'], args['p_pw'], args['login_method'])
else:
# Fake parameters for p_un & p_pw because SSO takes them from the environment
(iden, args['p_un'], args['p_pw'], msgcode) = webuser.loginUser(req, '', '', CFG_EXTERNAL_AUTH_USING_SSO)
args['remember_me'] = False
if iden:
uid = webuser.update_Uid(req, args['p_un'], args['remember_me'])
uid2 = webuser.getUid(req)
if uid2 == -1:
webuser.logoutUser(req)
return webuser.page_not_authorized(req, CFG_SITE_SECURE_URL + "/youraccount/login?ln=%s" % args['ln'], uid=uid,
navmenuid='youraccount')
# login successful!
try:
register_customevent("login", [req.remote_host or req.remote_ip, uid, args['p_un']])
except:
register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'")
if args['referer']:
redirect_to_url(req, args['referer'].replace(CFG_SITE_URL, CFG_SITE_SECURE_URL))
else:
return self.display(req, form)
else:
mess = None
if isinstance(msgcode, (str, unicode)):
# if msgcode is string, show it.
mess = msgcode
elif msgcode in [21, 22, 23]:
mess = CFG_WEBACCESS_WARNING_MSGS[msgcode]
elif msgcode == 14:
if webuser.username_exists_p(args['p_un']):
mess = CFG_WEBACCESS_WARNING_MSGS[15] % cgi.escape(args['login_method'])
if not mess:
mess = CFG_WEBACCESS_WARNING_MSGS[msgcode] % cgi.escape(args['login_method'])
act = CFG_SITE_SECURE_URL + '/youraccount/login%s' % make_canonical_urlargd({'ln' : args['ln'], 'referer' : args['referer']}, {})
return page(title=_("Login"),
body=webaccount.perform_back(mess, act, _("login"), args['ln']),
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description="%s Personalize, Main page" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords="%s , personalize" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
def register(self, req, form):
args = wash_urlargd(form, {
'p_nickname': (str, None),
'p_email': (str, None),
'p_pw': (str, None),
'p_pw2': (str, None),
'action': (str, "login"),
'referer': (str, "")})
if CFG_ACCESS_CONTROL_LEVEL_SITE > 0:
return webuser.page_not_authorized(req, "../youraccount/register?ln=%s" % args['ln'],
navmenuid='youraccount')
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(args['ln'])
if args['p_nickname'] is None or args['p_email'] is None:
return page(title=_("Register"),
body=webaccount.create_register_page_box(args['referer'], args['ln']),
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description=_("%(x_name)s Personalize, Main page", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
keywords="%s , personalize" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
mess = ""
act = ""
if args['p_pw'] == args['p_pw2']:
ruid = webuser.registerUser(req, args['p_email'], args['p_pw'],
args['p_nickname'], ln=args['ln'])
else:
ruid = -2
if ruid == 0:
mess = _("Your account has been successfully created.")
title = _("Account created")
if CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_NEW_ACCOUNT == 1:
mess += " " + _("In order to confirm its validity, an email message containing an account activation key has been sent to the given email address.")
mess += " " + _("Please follow instructions presented there in order to complete the account registration process.")
if CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS >= 1:
mess += " " + _("A second email will be sent when the account has been activated and can be used.")
elif CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_NEW_ACCOUNT != 1:
uid = webuser.update_Uid(req, args['p_email'])
mess += " " + _("You can now access your %(x_url_open)saccount%(x_url_close)s.") %\
{'x_url_open': '<a href="' + CFG_SITE_SECURE_URL + '/youraccount/display?ln=' + args['ln'] + '">',
'x_url_close': '</a>'}
elif ruid == -2:
mess = _("Both passwords must match.")
mess += " " + _("Please try again.")
act = "/youraccount/register?ln=%s" % args['ln']
title = _("Registration failure")
elif ruid == 1:
mess = _("Supplied email address %(x_addr)s is invalid.", x_addr=cgi.escape(args['p_email']))
mess += " " + _("Please try again.")
act = "/youraccount/register?ln=%s" % args['ln']
title = _("Registration failure")
elif ruid == 2:
mess = _("Desired nickname %(x_name)s is invalid.", x_name=cgi.escape(args['p_nickname']))
mess += " " + _("Please try again.")
act = "/youraccount/register?ln=%s" % args['ln']
title = _("Registration failure")
elif ruid == 3:
mess = _("Supplied email address %(x_addr)s already exists in the database.", x_addr=cgi.escape(args['p_email']))
mess += " " + websession_templates.tmpl_lost_your_password_teaser(args['ln'])
mess += " " + _("Or please try again.")
act = "/youraccount/register?ln=%s" % args['ln']
title = _("Registration failure")
elif ruid == 4:
mess = _("Desired nickname %(x_name)s already exists in the database.", x_name=cgi.escape(args['p_nickname']))
mess += " " + _("Please try again.")
act = "/youraccount/register?ln=%s" % args['ln']
title = _("Registration failure")
elif ruid == 5:
mess = _("Users cannot register themselves, only admin can register them.")
act = "/youraccount/register?ln=%s" % args['ln']
title = _("Registration failure")
elif ruid == 6:
mess = _("The site is having troubles in sending you an email for confirming your email address.") + _("The error has been logged and will be taken in consideration as soon as possible.")
act = "/youraccount/register?ln=%s" % args['ln']
title = _("Registration failure")
else:
# this should never happen
mess = _("Internal Error")
act = "/youraccount/register?ln=%s" % args['ln']
title = _("Registration failure")
return page(title=title,
body=webaccount.perform_back(mess,act, _("register"), args['ln']),
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description=_("%(x_name)s Personalize, Main page", x_name=CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME)),
keywords="%s , personalize" % CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid=uid,
req=req,
secure_page_p = 1,
language=args['ln'],
lastupdated=__lastupdated__,
navmenuid='youraccount')
def openid(self, req, form):
"""
Constructs the URL of the login page of the OpenID provider and
redirects or constructs it.
"""
def get_consumer(req):
"""
Returns a consumer without a memory.
"""
return consumer.Consumer({"id": get_session(req)}, None)
def request_registration_data(request, provider):
"""
Adds simple registration (sreg) and attribute exchage (ax) extension
to given OpenID request.
@param request: OpenID request
@type request: openid.consumer.consumer.AuthRequest
@param provider: OpenID provider
@type provider: str
"""
# We ask the user nickname if the provider accepts sreg request.
sreg_request = sreg.SRegRequest(required = ['nickname'])
request.addExtension(sreg_request)
# If the provider is trusted, we may ask the email of the user, too.
ax_request = ax.FetchRequest()
if CFG_OPENID_CONFIGURATIONS[provider].get('trust_email', False):
ax_request.add(ax.AttrInfo(
'http://axschema.org/contact/email',
required = True))
ax_request.add(ax.AttrInfo(
'http://axschema.org/namePerson/friendly',
required = True))
request.addExtension(ax_request)
# All arguements must be extracted
content = {
'provider': (str, ''),
'identifier': (str, ''),
'referer': (str, '')
}
for key in CFG_OPENID_CONFIGURATIONS.keys():
content[key] = (str, '')
args = wash_urlargd(form, content)
# Load the right message language
_ = gettext_set_language(args['ln'])
try:
from openid.consumer import consumer
from openid.extensions import ax
from openid.extensions import sreg
except:
# Return login page with 'Need to install python-openid' error
return page(title = _("Login"),
body = webaccount.create_login_page_box(
'%s/youraccount/login?error=openid-python' % \
CFG_SITE_SECURE_URL,
args['ln']
),
navtrail = """
<a class="navtrail" href="%s/youraccount/display?ln=%s">
""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description = "%s Personalize, Main page" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords = "%s , personalize" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid = 0,
req = req,
secure_page_p = 1,
language = args['ln'],
lastupdated = __lastupdated__,
navmenuid = 'youraccount')
# If either provider isn't activated or OpenID authentication is
# disabled, redirect to login page.
if not (args['provider'] in CFG_OPENID_PROVIDERS and
CFG_OPENID_AUTHENTICATION):
redirect_to_url(req, CFG_SITE_SECURE_URL + "/youraccount/login")
# Load the right message language
_ = gettext_set_language(args['ln'])
# Construct the OpenID identifier url according to given template in the
# configuration.
openid_url = CFG_OPENID_CONFIGURATIONS[args['provider']]['identifier'].\
format(args['identifier'])
oidconsumer = get_consumer(req)
try:
request = oidconsumer.begin(openid_url)
except consumer.DiscoveryFailure:
# If the identifier is invalid, then display login form with error
# message.
return page(title = _("Login"),
body = webaccount.create_login_page_box(
'%s/youraccount/login?error=openid-invalid' % \
CFG_SITE_SECURE_URL,
args['ln']
),
navtrail = """
<a class="navtrail" href="%s/youraccount/display?ln=%s">
""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description = "%s Personalize, Main page" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords = "%s , personalize" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid = 0,
req = req,
secure_page_p = 1,
language = args['ln'],
lastupdated = __lastupdated__,
navmenuid = 'youraccount')
else:
trust_root = CFG_SITE_SECURE_URL + "/"
return_to = CFG_SITE_SECURE_URL + "/youraccount/login?"
if args['provider'] == 'openid':
# Look if the identifier is defined.
for key in CFG_OPENID_CONFIGURATIONS.keys():
if CFG_OPENID_CONFIGURATIONS[key]['identifier']!='{0}':
regexp = re.compile(CFG_OPENID_CONFIGURATIONS[key]\
['identifier'].\
format("\w+"), re.IGNORECASE)
if openid_url in CFG_OPENID_CONFIGURATIONS[key]\
['identifier'] or \
regexp.match(openid_url):
args['provider'] = key
break
return_to += "login_method=openid&provider=%s" % (
args['provider']
)
request_registration_data(request, args['provider'])
if args['referer']:
return_to += "&referer=%s" % args['referer']
if request.shouldSendRedirect():
redirect_url = request.redirectURL(
trust_root,
return_to,
immediate = False)
redirect_to_url(req, redirect_url)
else:
form_html = request.htmlMarkup(trust_root,
return_to,
form_tag_attrs = {
'id':'openid_message'
},
immediate = False)
return form_html
def oauth2(self, req, form):
args = wash_urlargd(form, {'provider': (str, '')})
# If either provider isn't activated or OAuth2 authentication is
# disabled, redirect to login page.
if not (args['provider'] in CFG_OAUTH2_PROVIDERS and
CFG_OAUTH2_AUTHENTICATION):
redirect_to_url(req, CFG_SITE_SECURE_URL + "/youraccount/login")
# Load the right message language
_ = gettext_set_language(args['ln'])
try:
from rauth.service import OAuth2Service
except:
# Return login page with 'Need to install rauth' error
return page(title = _("Login"),
body = webaccount.create_login_page_box(
'%s/youraccount/login?error=oauth-rauth' % \
CFG_SITE_SECURE_URL,
args['ln']
),
navtrail = """
<a class="navtrail" href="%s/youraccount/display?ln=%s">
""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description = "%s Personalize, Main page" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords = "%s , personalize" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid = 0,
req = req,
secure_page_p = 1,
language = args['ln'],
lastupdated = __lastupdated__,
navmenuid = 'youraccount')
provider_name = args['provider']
# Load the configurations of the OAuth2 provider
config = CFG_OAUTH2_CONFIGURATIONS[provider_name]
try:
if not (config['consumer_key'] and config['consumer_secret']):
raise Exception
provider = OAuth2Service(
name = provider_name,
client_id = config['consumer_key'],
client_secret = config['consumer_secret'],
access_token_url = config['access_token_url'],
authorize_url = config['authorize_url']
)
except:
# Return login page with 'OAuth service isn't configurated' error
return page(title = _("Login"),
body = webaccount.create_login_page_box(
'%s/youraccount/login?error=oauth-config' % \
CFG_SITE_SECURE_URL,
args['ln']
),
navtrail = """
<a class="navtrail" href="%s/youraccount/display?ln=%s">
""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description = "%s Personalize, Main page" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords = "%s , personalize" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid = 0,
req = req,
secure_page_p = 1,
language = args['ln'],
lastupdated = __lastupdated__,
navmenuid = 'youraccount')
# Construct the authorization url
params = config.get('authorize_parameters', {})
params['redirect_uri'] = '%s/youraccount/login?login_method=oauth2\
&provider=%s' % (CFG_SITE_URL, args['provider'])
url = provider.get_authorize_url(**params)
redirect_to_url(req, url)
def oauth1(self, req, form):
args = wash_urlargd(form, {'provider': (str, '')})
# If either provider isn't activated or OAuth1 authentication is
# disabled, redirect to login page.
if not (args['provider'] in CFG_OAUTH1_PROVIDERS and
CFG_OAUTH1_AUTHENTICATION):
redirect_to_url(req, CFG_SITE_SECURE_URL + "/youraccount/login")
# Load the right message language
_ = gettext_set_language(args['ln'])
try:
from rauth.service import OAuth1Service
except:
# Return login page with 'Need to install rauth' error
return page(title = _("Login"),
body = webaccount.create_login_page_box(
'%s/youraccount/login?error=oauth-rauth' % \
CFG_SITE_SECURE_URL,
args['ln']
),
navtrail = """
<a class="navtrail" href="%s/youraccount/display?ln=%s">
""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description = "%s Personalize, Main page" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords = "%s , personalize" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid = 0,
req = req,
secure_page_p = 1,
language = args['ln'],
lastupdated = __lastupdated__,
navmenuid = 'youraccount')
# Load the configurations of the OAuth1 provider
config = CFG_OAUTH1_CONFIGURATIONS[args['provider']]
try:
if not (config['consumer_key'] and config['consumer_secret']):
raise Exception
provider = OAuth1Service(
name = args['provider'],
consumer_key = config['consumer_key'],
consumer_secret = config['consumer_secret'],
request_token_url = config['request_token_url'],
access_token_url = config['access_token_url'],
authorize_url = config['authorize_url'],
header_auth = True
)
except:
# Return login page with 'OAuth service isn't configurated' error
return page(title = _("Login"),
body = webaccount.create_login_page_box(
'%s/youraccount/login?error=oauth-config' % \
CFG_SITE_SECURE_URL,
args['ln']
),
navtrail = """
<a class="navtrail" href="%s/youraccount/display?ln=%s">
""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description = "%s Personalize, Main page" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords = "%s , personalize" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid = 0,
req = req,
secure_page_p = 1,
language = args['ln'],
lastupdated = __lastupdated__,
navmenuid = 'youraccount')
try:
# Obtain request token and its secret.
request_token, request_token_secret = \
provider.get_request_token(
method = 'GET',
data = {
'oauth_callback': \
"%s/youraccount/login?login_method=oauth1&provider=%s" % (
CFG_SITE_SECURE_URL,
args['provider']
)
}
)
except:
# Return login page with 'Cannot connect the provider' error
return page(title = _("Login"),
body = webaccount.create_login_page_box(
'%s/youraccount/login?error=connection-error' % \
CFG_SITE_SECURE_URL,
args['ln']
),
navtrail = """
<a class="navtrail" href="%s/youraccount/display?ln=%s">
""" % (CFG_SITE_SECURE_URL, args['ln']) + _("Your Account") + """</a>""",
description = "%s Personalize, Main page" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
keywords = "%s , personalize" % \
CFG_SITE_NAME_INTL.get(args['ln'], CFG_SITE_NAME),
uid = 0,
req = req,
secure_page_p = 1,
language = args['ln'],
lastupdated = __lastupdated__,
navmenuid = 'youraccount')
# Construct the authorization url.
authorize_parameters = config.get('authorize_parameters', {})
authorize_url = provider.get_authorize_url(request_token,
**authorize_parameters)
# Save request token into database since it will be used in
# authentication
query = """INSERT INTO oauth1_storage VALUES(%s, %s, NOW())"""
params = (request_token, request_token_secret)
run_sql(query, params)
redirect_to_url(req, authorize_url)
class WebInterfaceYourTicketsPages(WebInterfaceDirectory):
#support for /yourtickets url
_exports = ['', 'display']
def __call__(self, req, form):
#if there is no trailing slash
self.index(req, form)
def index(self, req, form):
#take all the parameters..
unparsed_uri = req.unparsed_uri
qstr = ""
if unparsed_uri.count('?') > 0:
dummy, qstr = unparsed_uri.split('?')
qstr = '?'+qstr
redirect_to_url(req, '/yourtickets/display'+qstr)
def display(self, req, form):
#show tickets for this user
argd = wash_urlargd(form, {'ln': (str, ''), 'start': (int, 1) })
uid = webuser.getUid(req)
ln = argd['ln']
start = argd['start']
_ = gettext_set_language(ln)
body = bibcatalog_templates.tmpl_your_tickets(uid, ln, start)
return page(title=_("Your tickets"),
body=body,
navtrail="""<a class="navtrail" href="%s/youraccount/display?ln=%s">""" % (CFG_SITE_SECURE_URL, argd['ln']) + _("Your Account") + """</a>""",
uid=uid,
req=req,
language=argd['ln'],
lastupdated=__lastupdated__,
secure_page_p=1)
class WebInterfaceYourGroupsPages(WebInterfaceDirectory):
_exports = ['', 'display', 'create', 'join', 'leave', 'edit', 'members']
def index(self, req, form):
redirect_to_url(req, '/yourgroups/display')
def display(self, req, form):
"""
Displays groups the user is admin of
and the groups the user is member of(but not admin)
@param ln: language
@return: the page for all the groups
"""
argd = wash_urlargd(form, {})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(argd['ln'])
if uid == -1 or webuser.isGuestUser(uid) or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../yourgroups/display",
navmenuid='yourgroups')
user_info = webuser.collect_user_info(req)
if not user_info['precached_usegroups']:
return webuser.page_not_authorized(req, "../", \
text = _("You are not authorized to use groups."))
body = webgroup.perform_request_groups_display(uid=uid,
ln=argd['ln'])
return page(title = _("Your Groups"),
body = body,
navtrail = webgroup.get_navtrail(argd['ln']),
uid = uid,
req = req,
language = argd['ln'],
lastupdated = __lastupdated__,
navmenuid = 'yourgroups',
secure_page_p = 1)
def create(self, req, form):
"""create(): interface for creating a new group
@param group_name: : name of the new webgroup.Must be filled
@param group_description: : description of the new webgroup.(optionnal)
@param join_policy: : join policy of the new webgroup.Must be chosen
@param *button: which button was pressed
@param ln: language
@return: the compose page Create group
"""
argd = wash_urlargd(form, {'group_name': (str, ""),
'group_description': (str, ""),
'join_policy': (str, ""),
'create_button':(str, ""),
'cancel':(str, "")
})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(argd['ln'])
if uid == -1 or webuser.isGuestUser(uid) or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../yourgroups/create",
navmenuid='yourgroups')
user_info = webuser.collect_user_info(req)
if not user_info['precached_usegroups']:
return webuser.page_not_authorized(req, "../", \
text = _("You are not authorized to use groups."))
if argd['cancel']:
url = CFG_SITE_SECURE_URL + '/yourgroups/display?ln=%s'
url %= argd['ln']
redirect_to_url(req, url)
if argd['create_button'] :
body= webgroup.perform_request_create_group(uid=uid,
group_name=argd['group_name'],
group_description=argd['group_description'],
join_policy=argd['join_policy'],
ln = argd['ln'])
else:
body = webgroup.perform_request_input_create_group(group_name=argd['group_name'],
group_description=argd['group_description'],
join_policy=argd['join_policy'],
ln=argd['ln'])
title = _("Create new group")
return page(title = title,
body = body,
navtrail = webgroup.get_navtrail(argd['ln'], title),
uid = uid,
req = req,
language = argd['ln'],
lastupdated = __lastupdated__,
navmenuid = 'yourgroups',
secure_page_p = 1)
def join(self, req, form):
"""join(): interface for joining a new group
@param grpID: : list of the group the user wants to become a member.
The user must select only one group.
@param group_name: : will search for groups matching group_name
@param *button: which button was pressed
@param ln: language
@return: the compose page Join group
"""
argd = wash_urlargd(form, {'grpID':(list, []),
'group_name':(str, ""),
'find_button':(str, ""),
'join_button':(str, ""),
'cancel':(str, "")
})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(argd['ln'])
if uid == -1 or webuser.isGuestUser(uid) or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../yourgroups/join",
navmenuid='yourgroups')
user_info = webuser.collect_user_info(req)
if not user_info['precached_usegroups']:
return webuser.page_not_authorized(req, "../", \
text = _("You are not authorized to use groups."))
if argd['cancel']:
url = CFG_SITE_SECURE_URL + '/yourgroups/display?ln=%s'
url %= argd['ln']
redirect_to_url(req, url)
if argd['join_button']:
search = 0
if argd['group_name']:
search = 1
body = webgroup.perform_request_join_group(uid,
argd['grpID'],
argd['group_name'],
search,
argd['ln'])
else:
search = 0
if argd['find_button']:
search = 1
body = webgroup.perform_request_input_join_group(uid,
argd['group_name'],
search,
ln=argd['ln'])
title = _("Join New Group")
return page(title = title,
body = body,
navtrail = webgroup.get_navtrail(argd['ln'], title),
uid = uid,
req = req,
language = argd['ln'],
lastupdated = __lastupdated__,
navmenuid = 'yourgroups',
secure_page_p = 1)
def leave(self, req, form):
"""leave(): interface for leaving a group
@param grpID: : group the user wants to leave.
@param group_name: : name of the group the user wants to leave
@param *button: which button was pressed
@param confirmed: : the user is first asked to confirm
@param ln: language
@return: the compose page Leave group
"""
argd = wash_urlargd(form, {'grpID':(int, 0),
'group_name':(str, ""),
'leave_button':(str, ""),
'cancel':(str, ""),
'confirmed': (int, 0)
})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(argd['ln'])
if uid == -1 or webuser.isGuestUser(uid) or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../yourgroups/leave",
navmenuid='yourgroups')
user_info = webuser.collect_user_info(req)
if not user_info['precached_usegroups']:
return webuser.page_not_authorized(req, "../", \
text = _("You are not authorized to use groups."))
if argd['cancel']:
url = CFG_SITE_SECURE_URL + '/yourgroups/display?ln=%s'
url %= argd['ln']
redirect_to_url(req, url)
if argd['leave_button']:
body = webgroup.perform_request_leave_group(uid,
argd['grpID'],
argd['confirmed'],
argd['ln'])
else:
body = webgroup.perform_request_input_leave_group(uid=uid,
ln=argd['ln'])
title = _("Leave Group")
return page(title = title,
body = body,
navtrail = webgroup.get_navtrail(argd['ln'], title),
uid = uid,
req = req,
language = argd['ln'],
lastupdated = __lastupdated__,
navmenuid = 'yourgroups',
secure_page_p = 1)
def edit(self, req, form):
"""edit(): interface for editing group
@param grpID: : group ID
@param group_name: : name of the new webgroup.Must be filled
@param group_description: : description of the new webgroup.(optionnal)
@param join_policy: : join policy of the new webgroup.Must be chosen
@param update: button update group pressed
@param delete: button delete group pressed
@param cancel: button cancel pressed
@param confirmed: : the user is first asked to confirm before deleting
@param ln: language
@return: the main page displaying all the groups
"""
argd = wash_urlargd(form, {'grpID': (int, 0),
'update': (str, ""),
'cancel': (str, ""),
'delete': (str, ""),
'group_name': (str, ""),
'group_description': (str, ""),
'join_policy': (str, ""),
'confirmed': (int, 0)
})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(argd['ln'])
if uid == -1 or webuser.isGuestUser(uid) or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../yourgroups/display",
navmenuid='yourgroups')
user_info = webuser.collect_user_info(req)
if not user_info['precached_usegroups']:
return webuser.page_not_authorized(req, "../", \
text = _("You are not authorized to use groups."))
if argd['cancel']:
url = CFG_SITE_SECURE_URL + '/yourgroups/display?ln=%s'
url %= argd['ln']
redirect_to_url(req, url)
elif argd['delete']:
body = webgroup.perform_request_delete_group(uid=uid,
grpID=argd['grpID'],
confirmed=argd['confirmed'])
elif argd['update']:
body = webgroup.perform_request_update_group(uid= uid,
grpID=argd['grpID'],
group_name=argd['group_name'],
group_description=argd['group_description'],
join_policy=argd['join_policy'],
ln=argd['ln'])
else :
body= webgroup.perform_request_edit_group(uid=uid,
grpID=argd['grpID'],
ln=argd['ln'])
title = _("Edit Group")
return page(title = title,
body = body,
navtrail = webgroup.get_navtrail(argd['ln'], title),
uid = uid,
req = req,
language = argd['ln'],
lastupdated = __lastupdated__,
navmenuid = 'yourgroups',
secure_page_p = 1)
def members(self, req, form):
"""member(): interface for managing members of a group
@param grpID: : group ID
@param add_member: button add_member pressed
@param remove_member: button remove_member pressed
@param reject_member: button reject__member pressed
@param delete: button delete group pressed
@param member_id: : ID of the existing member selected
@param pending_member_id: : ID of the pending member selected
@param cancel: button cancel pressed
@param info: : info about last user action
@param ln: language
@return: the same page with data updated
"""
argd = wash_urlargd(form, {'grpID': (int, 0),
'cancel': (str, ""),
'add_member': (str, ""),
'remove_member': (str, ""),
'reject_member': (str, ""),
'member_id': (int, 0),
'pending_member_id': (int, 0)
})
uid = webuser.getUid(req)
# load the right message language
_ = gettext_set_language(argd['ln'])
if uid == -1 or webuser.isGuestUser(uid) or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1:
return webuser.page_not_authorized(req, "../yourgroups/display",
navmenuid='yourgroups')
user_info = webuser.collect_user_info(req)
if not user_info['precached_usegroups']:
return webuser.page_not_authorized(req, "../", \
text = _("You are not authorized to use groups."))
if argd['cancel']:
url = CFG_SITE_SECURE_URL + '/yourgroups/display?ln=%s'
url %= argd['ln']
redirect_to_url(req, url)
if argd['remove_member']:
body = webgroup.perform_request_remove_member(uid=uid,
grpID=argd['grpID'],
member_id=argd['member_id'],
ln=argd['ln'])
elif argd['reject_member']:
body = webgroup.perform_request_reject_member(uid=uid,
grpID=argd['grpID'],
user_id=argd['pending_member_id'],
ln=argd['ln'])
elif argd['add_member']:
body = webgroup.perform_request_add_member(uid=uid,
grpID=argd['grpID'],
user_id=argd['pending_member_id'],
ln=argd['ln'])
else:
body= webgroup.perform_request_manage_member(uid=uid,
grpID=argd['grpID'],
ln=argd['ln'])
title = _("Edit group members")
return page(title = title,
body = body,
navtrail = webgroup.get_navtrail(argd['ln'], title),
uid = uid,
req = req,
language = argd['ln'],
lastupdated = __lastupdated__,
navmenuid = 'yourgroups',
secure_page_p = 1)
def wash_login_method(login_method):
"""
Wash the login_method parameter that came from the web input form.
@param login_method: Wanted login_method value as it came from the
web input form.
@type login_method: string
@return: Washed version of login_method. If the login_method
value is valid, then return it. If it is not valid, then
return `Local' (the default login method).
@rtype: string
@warning: Beware, 'Local' is hardcoded here!
"""
if login_method in CFG_EXTERNAL_AUTHENTICATION:
return login_method
else:
return 'Local'
diff --git a/invenio/legacy/webstat/api.py b/invenio/legacy/webstat/api.py
index bb46008f2..b4852dd98 100644
--- a/invenio/legacy/webstat/api.py
+++ b/invenio/legacy/webstat/api.py
@@ -1,1992 +1,1983 @@
## This file is part of Invenio.
-## Copyright (C) 2007, 2008, 2009, 2010, 2011, 2013 CERN.
+## Copyright (C) 2007, 2008, 2009, 2010, 2011, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
__revision__ = "$Id$"
__lastupdated__ = "$Date$"
import os
import time
import re
import datetime
from six.moves import cPickle
import calendar
from datetime import timedelta
from urllib import quote
from invenio.legacy import template
from invenio.config import \
CFG_WEBDIR, \
CFG_TMPDIR, \
CFG_SITE_URL, \
CFG_SITE_LANG, \
CFG_WEBSTAT_BIBCIRCULATION_START_YEAR
from invenio.legacy.webstat.config import CFG_WEBSTAT_CONFIG_PATH
+from invenio.legacy.bibindex.engine_utils import get_all_indexes
from invenio.modules.indexer.tokenizers.BibIndexJournalTokenizer import CFG_JOURNAL_TAG
from invenio.legacy.search_engine import get_coll_i18nname, \
wash_index_term
from invenio.legacy.dbquery import run_sql, wash_table_column_name, ProgrammingError
from invenio.legacy.bibsched.cli import is_task_scheduled, \
get_task_ids_by_descending_date, \
get_task_options
# Imports handling key events and error log
from invenio.legacy.webstat.engine import get_keyevent_trend_collection_population, \
get_keyevent_trend_new_records, \
get_keyevent_trend_search_frequency, \
get_keyevent_trend_search_type_distribution, \
get_keyevent_trend_download_frequency, \
get_keyevent_trend_comments_frequency, \
get_keyevent_trend_number_of_loans, \
get_keyevent_trend_web_submissions, \
get_keyevent_snapshot_apache_processes, \
get_keyevent_snapshot_bibsched_status, \
get_keyevent_snapshot_uptime_cmd, \
get_keyevent_snapshot_sessions, \
get_keyevent_bibcirculation_report, \
get_keyevent_loan_statistics, \
get_keyevent_loan_lists, \
get_keyevent_renewals_lists, \
get_keyevent_returns_table, \
get_keyevent_trend_returns_percentage, \
get_keyevent_ill_requests_statistics, \
get_keyevent_ill_requests_lists, \
get_keyevent_trend_satisfied_ill_requests_percentage, \
get_keyevent_items_statistics, \
get_keyevent_items_lists, \
get_keyevent_loan_request_statistics, \
get_keyevent_loan_request_lists, \
get_keyevent_user_statistics, \
get_keyevent_user_lists, \
_get_doctypes, \
_get_item_statuses, \
_get_item_doctype, \
_get_request_statuses, \
_get_libraries, \
_get_loan_periods, \
get_invenio_error_log_ranking, \
get_invenio_last_n_errors, \
update_error_log_analyzer, \
get_apache_error_log_ranking, \
get_last_updates, \
get_list_link, \
get_general_status, \
get_ingestion_matching_records, \
get_record_ingestion_status, \
get_specific_ingestion_status, \
get_title_ingestion, \
get_record_last_modification
# Imports handling custom events
from invenio.legacy.webstat.engine import get_customevent_table, \
get_customevent_trend, \
get_customevent_dump
# Imports handling custom report
from invenio.legacy.webstat.engine import get_custom_summary_data, \
_get_tag_name, \
create_custom_summary_graph
# Imports for handling outputting
from invenio.legacy.webstat.engine import create_graph_trend, \
create_graph_dump, \
create_graph_table, \
get_numeric_stats
# Imports for handling exports
from invenio.legacy.webstat.engine import export_to_python, \
export_to_csv, \
export_to_file
TEMPLATES = template.load('webstat')
# Constants
WEBSTAT_CACHE_INTERVAL = 600 # Seconds, cache_* functions not affected by this.
# Also not taking into account if BibSched has
# webstatadmin process.
WEBSTAT_RAWDATA_DIRECTORY = CFG_TMPDIR + "/"
WEBSTAT_GRAPH_DIRECTORY = CFG_WEBDIR + "/img/"
TYPE_REPOSITORY = [('gnuplot', 'Image - Gnuplot'),
('asciiart', 'Image - ASCII art'),
('flot', 'Image - Flot'),
('asciidump', 'Image - ASCII dump'),
('python', 'Data - Python code', export_to_python),
('csv', 'Data - CSV', export_to_csv)]
def get_collection_list_plus_all():
""" Return all the collection names plus the name All"""
coll = [('All', 'All')]
res = run_sql("SELECT name FROM collection WHERE (dbquery IS NULL OR dbquery \
NOT LIKE 'hostedcollection:%') ORDER BY name ASC")
for c_name in res:
# make a nice printable name (e.g. truncate c_printable for
# long collection names in given language):
c_printable_fullname = get_coll_i18nname(c_name[0], CFG_SITE_LANG, False)
c_printable = wash_index_term(c_printable_fullname, 30, False)
if c_printable != c_printable_fullname:
c_printable = c_printable + "..."
coll.append([c_name[0], c_printable])
return coll
# Key event repository, add an entry here to support new key measures.
KEYEVENT_REPOSITORY = {'collection population':
{'fullname': 'Collection population',
'specificname':
'Population in collection "%(collection)s"',
'description':
('The collection population is the number of \
documents existing in the selected collection.', ),
'gatherer':
get_keyevent_trend_collection_population,
'extraparams': {'collection': ('combobox', 'Collection',
get_collection_list_plus_all)},
'cachefilename':
'webstat_%(event_id)s_%(collection)s_%(timespan)s',
'ylabel': 'Number of records',
'multiple': None,
'output': 'Graph'},
'new records':
{'fullname': 'New records',
'specificname':
'New records in collection "%(collection)s"',
'description':
('The graph shows the new documents created in \
the selected collection and time span.', ),
'gatherer':
get_keyevent_trend_new_records,
'extraparams': {'collection': ('combobox', 'Collection',
get_collection_list_plus_all)},
'cachefilename':
'webstat_%(event_id)s_%(collection)s_%(timespan)s',
'ylabel': 'Number of records',
'multiple': None,
'output': 'Graph'},
'search frequency':
{'fullname': 'Search frequency',
'specificname': 'Search frequency',
'description':
('The search frequency is the number of searches \
performed in a specific time span.', ),
'gatherer': get_keyevent_trend_search_frequency,
'extraparams': {},
'cachefilename':
'webstat_%(event_id)s_%(timespan)s',
'ylabel': 'Number of searches',
'multiple': None,
'output': 'Graph'},
'search type distribution':
{'fullname': 'Search type distribution',
'specificname': 'Search type distribution',
'description':
('The search type distribution shows both the \
number of simple searches and the number of advanced searches in the same graph.', ),
'gatherer':
get_keyevent_trend_search_type_distribution,
'extraparams': {},
'cachefilename':
'webstat_%(event_id)s_%(timespan)s',
'ylabel': 'Number of searches',
'multiple': ['Simple searches',
'Advanced searches'],
'output': 'Graph'},
'download frequency':
{'fullname': 'Download frequency',
'specificname': 'Download frequency in collection "%(collection)s"',
'description':
('The download frequency is the number of fulltext \
downloads of the documents.', ),
'gatherer': get_keyevent_trend_download_frequency,
'extraparams': {'collection': ('combobox', 'Collection',
get_collection_list_plus_all)},
'cachefilename': 'webstat_%(event_id)s_%(collection)s_%(timespan)s',
'ylabel': 'Number of downloads',
'multiple': None,
'output': 'Graph'},
'comments frequency':
{'fullname': 'Comments frequency',
'specificname': 'Comments frequency in collection "%(collection)s"',
'description':
('The comments frequency is the amount of comments written \
for all the documents.', ),
'gatherer': get_keyevent_trend_comments_frequency,
'extraparams': {'collection': ('combobox', 'Collection',
get_collection_list_plus_all)},
'cachefilename': 'webstat_%(event_id)s_%(collection)s_%(timespan)s',
'ylabel': 'Number of comments',
'multiple': None,
'output': 'Graph'},
'number of loans':
{'fullname': 'Number of circulation loans',
'specificname': 'Number of circulation loans',
'description':
('The number of loans shows the total number of records loaned \
over a time span', ),
'gatherer': get_keyevent_trend_number_of_loans,
'extraparams': {},
'cachefilename':
'webstat_%(event_id)s_%(timespan)s',
'ylabel': 'Number of loans',
'multiple': None,
'output': 'Graph',
'type': 'bibcirculation'},
'web submissions':
{'fullname': 'Number of web submissions',
'specificname':
'Number of web submissions of "%(doctype)s"',
'description':
("The web submissions are the number of submitted \
documents using the web form.", ),
'gatherer': get_keyevent_trend_web_submissions,
'extraparams': {
'doctype': ('combobox', 'Type of document', _get_doctypes)},
'cachefilename':
'webstat_%(event_id)s_%(doctype)s_%(timespan)s',
'ylabel': 'Web submissions',
'multiple': None,
'output': 'Graph'},
'loans statistics':
{'fullname': 'Circulation loans statistics',
'specificname': 'Circulation loans statistics',
'description':
('The loan statistics consist on different numbers \
related to the records loaned. It is important to see the difference between document \
and item. The item is the physical representation of a document (like every copy of a \
book). There may be more items than documents, but never the opposite.', ),
'gatherer':
get_keyevent_loan_statistics,
'extraparams': {
'udc': ('textbox', 'UDC'),
'item_status': ('combobox', 'Item status', _get_item_statuses),
'publication_date': ('textbox', 'Publication date'),
'creation_date': ('textbox', 'Creation date')},
'cachefilename':
'webstat_%(event_id)s_%(udc)s_%(item_status)s_%(publication_date)s' + \
'_%(creation_date)s_%(timespan)s',
'rows': ['Number of documents loaned',
'Number of items loaned on the total number of items (%)',
'Number of items never loaned on the \
total number of items (%)',
'Average time between the date of \
the record creation and the date of the first loan (in days)'],
'output': 'Table',
'type': 'bibcirculation'},
'loans lists':
{'fullname': 'Circulation loans lists',
'specificname': 'Circulation loans lists',
'description':
('The loan lists show the most loaned and the never loaned \
records in a time span. The most loaned record are calculated as the number of loans by copy.', ),
'gatherer':
get_keyevent_loan_lists,
'extraparams': {
'udc': ('textbox', 'UDC'),
'loan_period': ('combobox', 'Loan period', _get_loan_periods),
'max_loans': ('textbox', 'Maximum number of loans'),
'min_loans': ('textbox', 'Minimum number of loans'),
'publication_date': ('textbox', 'Publication date'),
'creation_date': ('textbox', 'Creation date')},
'cachefilename':
'webstat_%(event_id)s_%(udc)s_%(loan_period)s' + \
'_%(min_loans)s_%(max_loans)s_%(publication_date)s_' + \
'%(creation_date)s_%(timespan)s',
'rows': [],
'output': 'List',
'type': 'bibcirculation'},
'renewals':
{'fullname': 'Circulation renewals',
'specificname': 'Circulation renewals',
'description':
('Here the list of most renewed items stored is shown \
by decreasing order', ),
'gatherer':
get_keyevent_renewals_lists,
'extraparams': {
'udc': ('textbox', 'UDC')},
'cachefilename':
'webstat_%(event_id)s_%(udc)s_%(timespan)s',
'rows': [],
'output': 'List',
'type': 'bibcirculation'},
'number returns':
{'fullname': 'Number of circulation overdue returns',
'specificname': 'Number of circulation overdue returns',
'description':
('The number of overdue returns is the number of loans \
that has not been returned by the due date (they may have been returned after or never).', ),
'gatherer':
get_keyevent_returns_table,
'extraparams': {},
'cachefilename':
'webstat_%(event_id)s_%(timespan)s',
'rows': ['Number of overdue returns'],
'output': 'Table',
'type': 'bibcirculation'},
'percentage returns':
{'fullname': 'Percentage of circulation overdue returns',
'specificname': 'Percentage of overdue returns',
'description':
('This graphs shows both the overdue returns and the total \
of returns.', ),
'gatherer':
get_keyevent_trend_returns_percentage,
'extraparams': {},
'cachefilename':
'webstat_%(event_id)s_%(timespan)s',
'ylabel': 'Percentage of overdue returns',
'multiple': ['Overdue returns',
'Total returns'],
'output': 'Graph',
'type': 'bibcirculation'},
'ill requests statistics':
{'fullname': 'Circulation ILL Requests statistics',
'specificname': 'Circulation ILL Requests statistics',
'description':
('The ILL requests statistics are different numbers \
related to the requests to other libraries.', ),
'gatherer':
get_keyevent_ill_requests_statistics,
'extraparams': {
'doctype': ('combobox', 'Type of document', _get_item_doctype),
'status': ('combobox', 'Status of request', _get_request_statuses),
'supplier': ('combobox', 'Supplier', _get_libraries)},
'cachefilename':
'webstat_%(event_id)s_%(doctype)s_%(status)s_%(supplier)s_%(timespan)s',
'rows': ['Number of ILL requests',
'Number of satisfied ILL requests 2 weeks \
after the date of request creation',
'Average time between the day \
of the ILL request date and day \
of the delivery item to the user (in days)',
'Average time between the day \
the ILL request was sent to the supplier and \
the day of the delivery item (in days)'],
'output': 'Table',
'type': 'bibcirculation'},
'ill requests list':
{'fullname': 'Circulation ILL Requests list',
'specificname': 'Circulation ILL Requests list',
'description':
('The ILL requests list shows 50 requests to other \
libraries on the selected time span.', ),
'gatherer':
get_keyevent_ill_requests_lists,
'extraparams': {
'doctype': ('combobox', 'Type of document', _get_item_doctype),
'supplier': ('combobox', 'Supplier', _get_libraries)},
'cachefilename':
'webstat_%(event_id)s_%(doctype)s_%(supplier)s_%(timespan)s',
'rows': [],
'output': 'List',
'type': 'bibcirculation'},
'percentage satisfied ill requests':
{'fullname': 'Percentage of circulation satisfied ILL requests',
'specificname': 'Percentage of circulation satisfied ILL requests',
'description':
('This graph shows both the satisfied ILL requests and \
the total number of requests in the selected time span.', ),
'gatherer':
get_keyevent_trend_satisfied_ill_requests_percentage,
'extraparams': {
'doctype': ('combobox', 'Type of document', _get_item_doctype),
'status': ('combobox', 'Status of request', _get_request_statuses),
'supplier': ('combobox', 'Supplier', _get_libraries)},
'cachefilename':
'webstat_%(event_id)s_%(doctype)s_%(status)s_%(supplier)s_%(timespan)s',
'ylabel': 'Percentage of satisfied ILL requests',
'multiple': ['Satisfied ILL requests',
'Total requests'],
'output': 'Graph',
'type': 'bibcirculation'},
'items stats':
{'fullname': 'Circulation items statistics',
'specificname': 'Circulation items statistics',
'description':
('The items statistics show the total number of items at \
the moment and the number of new items in the selected time span.', ),
'gatherer':
get_keyevent_items_statistics,
'extraparams': {
'udc': ('textbox', 'UDC'),
},
'cachefilename':
'webstat_%(event_id)s_%(udc)s_%(timespan)s',
'rows': ['The total number of items', 'Total number of new items'],
'output': 'Table',
'type': 'bibcirculation'},
'items list':
{'fullname': 'Circulation items list',
'specificname': 'Circulation items list',
'description':
('The item list shows data about the existing items.', ),
'gatherer':
get_keyevent_items_lists,
'extraparams': {
'library': ('combobox', 'Library', _get_libraries),
'status': ('combobox', 'Status', _get_item_statuses)},
'cachefilename':
'webstat_%(event_id)s_%(library)s_%(status)s',
'rows': [],
'output': 'List',
'type': 'bibcirculation'},
'loan request statistics':
{'fullname': 'Circulation hold requests statistics',
'specificname': 'Circulation hold requests statistics',
'description':
('The hold requests statistics show numbers about the \
requests for documents. For the numbers to be correct, there must be data in the loanrequest \
custom event.', ),
'gatherer':
get_keyevent_loan_request_statistics,
'extraparams': {
'item_status': ('combobox', 'Item status', _get_item_statuses)},
'cachefilename':
'webstat_%(event_id)s_%(item_status)s_%(timespan)s',
'rows': ['Number of hold requests, one week after the date of \
request creation',
'Number of successful hold requests transactions',
'Average time between the hold request date and \
the date of delivery document in a year'],
'output': 'Table',
'type': 'bibcirculation'},
'loan request lists':
{'fullname': 'Circulation hold requests lists',
'specificname': 'Circulation hold requests lists',
'description':
('The hold requests list shows the most requested items.', ),
'gatherer':
get_keyevent_loan_request_lists,
'extraparams': {
'udc': ('textbox', 'UDC')},
'cachefilename':
'webstat_%(event_id)s_%(udc)s_%(timespan)s',
'rows': [],
'output': 'List',
'type': 'bibcirculation'},
'user statistics':
{'fullname': 'Circulation users statistics',
'specificname': 'Circulation users statistics',
'description':
('The user statistics show the number of active users \
(at least one transaction) in the selected timespan.', ),
'gatherer':
get_keyevent_user_statistics,
'extraparams': {},
'cachefilename':
'webstat_%(event_id)s_%(timespan)s',
'rows': ['Number of active users'],
'output': 'Table',
'type': 'bibcirculation'},
'user lists':
{'fullname': 'Circulation users lists',
'specificname': 'Circulation users lists',
'description':
('The user list shows the most intensive users \
(ILL requests + Loans)', ),
'gatherer':
get_keyevent_user_lists,
'extraparams': {},
'cachefilename':
'webstat_%(event_id)s_%(timespan)s',
'rows': [],
'output': 'List',
'type': 'bibcirculation'}
}
# CLI
def create_customevent(event_id=None, name=None, cols=[]):
"""
Creates a new custom event by setting up the necessary MySQL tables.
@param event_id: Proposed human-readable id of the new event.
@type event_id: str
@param name: Optionally, a descriptive name.
@type name: str
@param cols: Optionally, the name of the additional columns.
@type cols: [str]
@return: A status message
@type: str
"""
if event_id is None:
return "Please specify a human-readable ID for the event."
# Only accept id and name with standard characters
if not re.search("[^\w]", str(event_id) + str(name)) is None:
return "Please note that both event id and event name needs to be " + \
"written without any non-standard characters."
# Make sure the chosen id is not already taken
if len(run_sql("SELECT NULL FROM staEVENT WHERE id = %s",
(event_id, ))) != 0:
return "Event id [%s] already exists! Aborted." % event_id
# Check if the cols are valid titles
for argument in cols:
if (argument == "creation_time") or (argument == "id"):
return "Invalid column title: %s! Aborted." % argument
# Insert a new row into the events table describing the new event
sql_param = [event_id]
if name is not None:
sql_name = "%s"
sql_param.append(name)
else:
sql_name = "NULL"
if len(cols) != 0:
sql_cols = "%s"
sql_param.append(cPickle.dumps(cols))
else:
sql_cols = "NULL"
run_sql("INSERT INTO staEVENT (id, name, cols) VALUES (%s, " + \
sql_name + ", " + sql_cols + ")", tuple(sql_param))
tbl_name = get_customevent_table(event_id)
# Create a table for the new event
sql_query = ["CREATE TABLE %s (" % wash_table_column_name(tbl_name)]
sql_query.append("id MEDIUMINT unsigned NOT NULL auto_increment,")
sql_query.append("creation_time TIMESTAMP DEFAULT NOW(),")
for argument in cols:
arg = wash_table_column_name(argument)
sql_query.append("`%s` MEDIUMTEXT NULL," % arg)
sql_query.append("INDEX `%s` (`%s` (50))," % (arg, arg))
sql_query.append("PRIMARY KEY (id))")
sql_str = ' '.join(sql_query)
run_sql(sql_str)
# We're done! Print notice containing the name of the event.
return ("Event table [%s] successfully created.\n" +
"Please use event id [%s] when registering an event.") \
% (tbl_name, event_id)
def modify_customevent(event_id=None, name=None, cols=[]):
"""
Modify a custom event. It can modify the columns definition
or/and the descriptive name
@param event_id: Human-readable id of the event.
@type event_id: str
@param name: Optionally, a descriptive name.
@type name: str
@param cols: Optionally, the name of the additional columns.
@type cols: [str]
@return: A status message
@type: str
"""
if event_id is None:
return "Please specify a human-readable ID for the event."
# Only accept name with standard characters
if not re.search("[^\w]", str(name)) is None:
return "Please note that event name needs to be written " + \
"without any non-standard characters."
# Check if the cols are valid titles
for argument in cols:
if (argument == "creation_time") or (argument == "id"):
return "Invalid column title: %s! Aborted." % argument
res = run_sql("SELECT CONCAT('staEVENT', number), cols " + \
"FROM staEVENT WHERE id = %s", (event_id, ))
if not res:
return "Invalid event id: %s! Aborted" % event_id
if not run_sql("SHOW TABLES LIKE %s", res[0][0]):
run_sql("DELETE FROM staEVENT WHERE id=%s", (event_id, ))
create_customevent(event_id, event_id, cols)
return
cols_orig = cPickle.loads(res[0][1])
# add new cols
cols_add = []
for col in cols:
if not col in cols_orig:
cols_add.append(col)
# del old cols
cols_del = []
for col in cols_orig:
if not col in cols:
cols_del.append(col)
#modify event table
if cols_del or cols_add:
sql_query = ["ALTER TABLE %s " % wash_table_column_name(res[0][0])]
# check if a column was renamed
for col_del in cols_del:
result = -1
while result < 1 or result > len(cols_add) + 1:
print("""What do you want to do with the column %s in event %s?:
1.- Delete it""" % (col_del, event_id))
for i in range(len(cols_add)):
print("%d.- Rename it to %s" % (i + 2, cols_add[i]))
result = int(raw_input("\n"))
if result == 1:
sql_query.append("DROP COLUMN `%s`" % col_del)
sql_query.append(", ")
else:
col_add = cols_add[result-2]
sql_query.append("CHANGE `%s` `%s` MEDIUMTEXT NULL"%(col_del, col_add))
sql_query.append(", ")
cols_add.remove(col_add)
# add the rest of the columns
for col_add in cols_add:
sql_query.append("ADD COLUMN `%s` MEDIUMTEXT NULL, " % col_add)
sql_query.append("ADD INDEX `%s` (`%s`(50))" % (col_add, col_add))
sql_query.append(", ")
sql_query[-1] = ";"
run_sql("".join(sql_query))
#modify event definition
sql_query = ["UPDATE staEVENT SET"]
sql_param = []
if cols_del or cols_add:
sql_query.append("cols = %s")
sql_query.append(",")
sql_param.append(cPickle.dumps(cols))
if name:
sql_query.append("name = %s")
sql_query.append(",")
sql_param.append(name)
if sql_param:
sql_query[-1] = "WHERE id = %s"
sql_param.append(event_id)
sql_str = ' '.join(sql_query)
run_sql(sql_str, sql_param)
# We're done! Print notice containing the name of the event.
return ("Event table [%s] successfully modified." % (event_id, ))
def destroy_customevent(event_id=None):
"""
Removes an existing custom event by destroying the MySQL tables and
the event data that might be around. Use with caution!
@param event_id: Human-readable id of the event to be removed.
@type event_id: str
@return: A status message
@type: str
"""
if event_id is None:
return "Please specify an existing event id."
# Check if the specified id exists
if len(run_sql("SELECT NULL FROM staEVENT WHERE id = %s",
(event_id, ))) == 0:
return "Custom event ID '%s' doesn't exist! Aborted." % event_id
else:
tbl_name = get_customevent_table(event_id)
run_sql("DROP TABLE %s" % wash_table_column_name(tbl_name)) # kwalitee: disable=sql
run_sql("DELETE FROM staEVENT WHERE id = %s", (event_id, ))
return ("Custom event ID '%s' table '%s' was successfully destroyed.\n") \
% (event_id, tbl_name)
def destroy_customevents():
"""
Removes all existing custom events by destroying the MySQL tables and
the events data that might be around. Use with caution!
@return: A status message
@type: str
"""
msg = ''
try:
res = run_sql("SELECT id FROM staEVENT")
except ProgrammingError:
return msg
for event in res:
msg += destroy_customevent(event[0])
return msg
def register_customevent(event_id, *arguments):
"""
Registers a custom event. Will add to the database's event tables
as created by create_customevent().
This function constitutes the "function hook" that should be
called throughout Invenio where one wants to register a
custom event! Refer to the help section on the admin web page.
@param event_id: Human-readable id of the event to be registered
@type event_id: str
@param *arguments: The rest of the parameters of the function call
@type *arguments: [params]
"""
res = run_sql("SELECT CONCAT('staEVENT', number),cols " + \
"FROM staEVENT WHERE id = %s", (event_id, ))
if not res:
return # the id does not exist
tbl_name = res[0][0]
if res[0][1]:
col_titles = cPickle.loads(res[0][1])
else:
col_titles = []
if len(col_titles) != len(arguments[0]):
return # there is different number of arguments than cols
# Make sql query
if len(arguments[0]) != 0:
sql_param = []
sql_query = ["INSERT INTO %s (" % wash_table_column_name(tbl_name)]
for title in col_titles:
sql_query.append("`%s`" % title)
sql_query.append(",")
sql_query.pop() # del the last ','
sql_query.append(") VALUES (")
for argument in arguments[0]:
sql_query.append("%s")
sql_query.append(",")
sql_param.append(argument)
sql_query.pop() # del the last ','
sql_query.append(")")
sql_str = ''.join(sql_query)
run_sql(sql_str, tuple(sql_param))
else:
run_sql("INSERT INTO %s () VALUES ()" % wash_table_column_name(tbl_name)) # kwalitee: disable=sql
def cache_keyevent_trend(ids=[]):
"""
Runs the rawdata gatherer for the specific key events.
Intended to be run mainly but the BibSched daemon interface.
For a specific id, all possible timespans' rawdata is gathered.
@param ids: The key event ids that are subject to caching.
@type ids: []
"""
args = {}
for event_id in ids:
args['event_id'] = event_id
if 'type' in KEYEVENT_REPOSITORY[event_id] and \
KEYEVENT_REPOSITORY[event_id]['type'] == 'bibcirculation':
timespans = _get_timespans(bibcirculation_stat=True)[:-1]
else:
timespans = _get_timespans()[:-1]
extraparams = KEYEVENT_REPOSITORY[event_id]['extraparams']
# Construct all combinations of extraparams and store as
# [{param name: arg value}] so as we can loop over them and just
# pattern-replace the each dictionary against
# the KEYEVENT_REPOSITORY['event_id']['cachefilename'].
combos = [[]]
for extra in [[(param, extra[0]) for extra in extraparams[param][1]()]
for param in extraparams]:
combos = [i + [y] for y in extra for i in combos]
combos = [dict(extra) for extra in combos]
for i in range(len(timespans)):
# Get timespans parameters
args['timespan'] = timespans[i][0]
args.update({'t_start': timespans[i][2], 't_end': timespans[i][3],
'granularity': timespans[i][4],
't_format': timespans[i][5],
'xtic_format': timespans[i][6]})
for combo in combos:
args.update(combo)
# Create unique filename for this combination of parameters
filename = KEYEVENT_REPOSITORY[event_id]['cachefilename'] \
% dict([(param, re.subn("[^\w]", "_",
args[param])[0]) for param in args])
# Create closure of gatherer function in case cache
# needs to be refreshed
gatherer = lambda: KEYEVENT_REPOSITORY[event_id] \
['gatherer'](args)
# Get data file from cache, ALWAYS REFRESH DATA!
_get_file_using_cache(filename, gatherer, True).read()
return True
def cache_customevent_trend(ids=[]):
"""
Runs the rawdata gatherer for the specific custom events.
Intended to be run mainly but the BibSched daemon interface.
For a specific id, all possible timespans' rawdata is gathered.
@param ids: The custom event ids that are subject to caching.
@type ids: []
"""
args = {}
timespans = _get_timespans()
for event_id in ids:
args['event_id'] = event_id
args['cols'] = []
for i in range(len(timespans)):
# Get timespans parameters
args['timespan'] = timespans[i][0]
args.update({'t_start': timespans[i][2], 't_end': timespans[i][3],
'granularity': timespans[i][4],
't_format': timespans[i][5],
'xtic_format': timespans[i][6]})
# Create unique filename for this combination of parameters
filename = "webstat_customevent_%(event_id)s_%(timespan)s" \
% {'event_id': re.subn("[^\w]", "_", event_id)[0],
'timespan': re.subn("[^\w]", "_", args['timespan'])[0]}
# Create closure of gatherer function in case cache
# needs to be refreshed
gatherer = lambda: get_customevent_trend(args)
# Get data file from cache, ALWAYS REFRESH DATA!
_get_file_using_cache(filename, gatherer, True).read()
return True
def basket_display():
"""
Display basket statistics.
"""
tbl_name = get_customevent_table("baskets")
if not tbl_name:
# custom event baskets not defined, so return empty output:
return []
try:
res = run_sql("SELECT creation_time FROM %s ORDER BY creation_time" % wash_table_column_name(tbl_name)) # kwalitee: disable=sql
days = (res[-1][0] - res[0][0]).days + 1
public = run_sql("SELECT COUNT(*) FROM %s " % wash_table_column_name(tbl_name) + " WHERE action = 'display_public'")[0][0] # kwalitee: disable=sql
users = run_sql("SELECT COUNT(DISTINCT user) FROM %s" % wash_table_column_name(tbl_name))[0][0] # kwalitee: disable=sql
adds = run_sql("SELECT COUNT(*) FROM %s WHERE action = 'add'" % wash_table_column_name(tbl_name))[0][0] # kwalitee: disable=sql
displays = run_sql("SELECT COUNT(*) FROM %s " % wash_table_column_name(tbl_name) + " WHERE action = 'display' OR action = 'display_public'")[0][0] # kwalitee: disable=sql
hits = adds + displays
average = hits / days
res = [("Basket page hits", hits)]
res.append((" Average per day", average))
res.append((" Unique users", users))
res.append((" Additions", adds))
res.append((" Public", public))
except IndexError:
res = []
return res
def alert_display():
"""
Display alert statistics.
"""
tbl_name = get_customevent_table("alerts")
if not tbl_name:
# custom event alerts not defined, so return empty output:
return []
try:
res = run_sql("SELECT creation_time FROM %s ORDER BY creation_time"
% wash_table_column_name(tbl_name))
days = (res[-1][0] - res[0][0]).days + 1
res = run_sql("SELECT COUNT(DISTINCT user),COUNT(*) FROM %s" % wash_table_column_name(tbl_name)) # kwalitee: disable=sql
users = res[0][0]
hits = res[0][1]
displays = run_sql("SELECT COUNT(*) FROM %s WHERE action = 'list'"
% wash_table_column_name(tbl_name))[0][0]
search = run_sql("SELECT COUNT(*) FROM %s WHERE action = 'display'"
% wash_table_column_name(tbl_name))[0][0]
average = hits / days
res = [("Alerts page hits", hits)]
res.append((" Average per day", average))
res.append((" Unique users", users))
res.append((" Displays", displays))
res.append((" Searches history display", search))
except IndexError:
res = []
return res
def loan_display():
"""
Display loan statistics.
"""
try:
loans, renewals, returns, illrequests, holdrequests = \
get_keyevent_bibcirculation_report()
res = [("Yearly report", '')]
res.append((" Loans", loans))
res.append((" Renewals", renewals))
res.append((" Returns", returns))
res.append((" ILL requests", illrequests))
res.append((" Hold requests", holdrequests))
return res
except IndexError:
return []
def get_url_customevent(url_dest, event_id, *arguments):
"""
Get an url for registers a custom event. Every time is load the
url will register a customevent as register_customevent().
@param url_dest: url to redirect after register the event
@type url_dest: str
@param event_id: Human-readable id of the event to be registered
@type event_id: str
@param *arguments: The rest of the parameters of the function call
the param "WEBSTAT_IP" will tell webstat that here
should be the IP who request the url
@type *arguments: [params]
@return: url for register event
@type: str
"""
return "%s/stats/customevent_register?event_id=%s&arg=%s&url=%s" % \
(CFG_SITE_URL, event_id, ','.join(arguments[0]), quote(url_dest))
# WEB
def perform_request_index(ln=CFG_SITE_LANG):
"""
Displays some informative text, the health box, and a the list of
key/custom events.
"""
out = TEMPLATES.tmpl_welcome(ln=ln)
# Display the health box
out += TEMPLATES.tmpl_system_health_list(get_general_status(), ln=ln)
# Produce a list of the key statistics
out += TEMPLATES.tmpl_keyevent_list(ln=ln)
# Display the custom statistics
out += TEMPLATES.tmpl_customevent_list(_get_customevents(), ln=ln)
# Display error log analyzer
out += TEMPLATES.tmpl_error_log_statistics_list(ln=ln)
# Display annual report
out += TEMPLATES.tmpl_custom_summary(ln=ln)
out += TEMPLATES.tmpl_yearly_report_list(ln=ln)
# Display test for collections
out += TEMPLATES.tmpl_collection_stats_main_list(ln=ln)
return out
def perform_display_current_system_health(ln=CFG_SITE_LANG):
"""
Display the current general system health:
- Uptime/load average
- Apache status
- Session information
- Searches recount
- New records
- Bibsched queue
- New/modified records
- Indexing, ranking, sorting and collecting methods
- Baskets
- Alerts
"""
from ConfigParser import ConfigParser
conf = ConfigParser()
conf.read(CFG_WEBSTAT_CONFIG_PATH)
# Prepare the health base data
health_indicators = []
now = datetime.datetime.now()
yesterday = (now - datetime.timedelta(days=1)).strftime("%Y-%m-%d")
today = now.strftime("%Y-%m-%d")
tomorrow = (now + datetime.timedelta(days=1)).strftime("%Y-%m-%d")
# Append uptime and load average to the health box
if conf.get("general", "uptime_box") == "True":
health_indicators.append(("Uptime cmd",
get_keyevent_snapshot_uptime_cmd()))
# Append number of Apache processes to the health box
if conf.get("general", "apache_box") == "True":
health_indicators.append(("Apache processes",
get_keyevent_snapshot_apache_processes()))
health_indicators.append(None)
# Append session information to the health box
if conf.get("general", "visitors_box") == "True":
sess = get_keyevent_snapshot_sessions()
health_indicators.append(("Total active visitors", sum(sess)))
health_indicators.append((" Logged in", sess[1]))
health_indicators.append(None)
# Append searches information to the health box
if conf.get("general", "search_box") == "True":
args = {'t_start': today, 't_end': tomorrow,
'granularity': "day", 't_format': "%Y-%m-%d"}
searches = get_keyevent_trend_search_type_distribution(args)
health_indicators.append(("Searches since midnight",
sum(searches[0][1])))
health_indicators.append((" Simple", searches[0][1][0]))
health_indicators.append((" Advanced", searches[0][1][1]))
health_indicators.append(None)
# Append new records information to the health box
if conf.get("general", "record_box") == "True":
args = {'collection': "All", 't_start': today,
't_end': tomorrow, 'granularity': "day",
't_format': "%Y-%m-%d"}
try:
tot_records = get_keyevent_trend_collection_population(args)[0][1]
except IndexError:
tot_records = 0
args = {'collection': "All", 't_start': yesterday,
't_end': today, 'granularity': "day", 't_format': "%Y-%m-%d"}
try:
new_records = tot_records - \
get_keyevent_trend_collection_population(args)[0][1]
except IndexError:
new_records = 0
health_indicators.append(("Total records", tot_records))
health_indicators.append((" New records since midnight",
new_records))
health_indicators.append(None)
# Append status of BibSched queue to the health box
if conf.get("general", "bibsched_box") == "True":
bibsched = get_keyevent_snapshot_bibsched_status()
health_indicators.append(("BibSched queue",
sum([x[1] for x in bibsched])))
for item in bibsched:
health_indicators.append((" " + item[0], str(item[1])))
health_indicators.append(None)
# Append records pending
if conf.get("general", "waiting_box") == "True":
last_index, last_rank, last_sort, last_coll=get_last_updates()
- index_categories = ('global', 'collection', 'abstract',
- 'author', 'keyword', 'reference',
- 'reportnumber', 'title', 'fulltext',
- 'year', 'journal', 'collaboration',
- 'affiliation', 'exactauthor', 'caption',
- 'firstauthor', 'exactfirstauthor',
- 'authorcount')
+ index_categories = zip(*get_all_indexes(with_ids=True))[1]
rank_categories = ('wrd', 'demo_jif', 'citation',
'citerank_citation_t',
'citerank_pagerank_c',
'citerank_pagerank_t')
sort_categories = ('latest first', 'title', 'author', 'report number',
'most cited')
health_indicators.append(("Records pending per indexing method since", last_index))
for ic in index_categories:
health_indicators.append((" - " + str(ic), get_list_link('index', ic)))
health_indicators.append(None)
health_indicators.append(("Records pending per ranking method since", last_rank))
for rc in rank_categories:
health_indicators.append((" - " + str(rc), get_list_link('rank', rc)))
health_indicators.append(None)
health_indicators.append(("Records pending per sorting method since", last_sort))
for sc in sort_categories:
health_indicators.append((" - " + str(sc), get_list_link('sort', sc)))
health_indicators.append(None)
health_indicators.append(("Records pending for webcolling since", last_coll))
health_indicators.append((" - webcoll", get_list_link('collect')))
health_indicators.append(None)
# Append basket stats to the health box
if conf.get("general", "basket_box") == "True":
health_indicators += basket_display()
health_indicators.append(None)
# Append alerts stats to the health box
if conf.get("general", "alert_box") == "True":
health_indicators += alert_display()
health_indicators.append(None)
# Display the health box
return TEMPLATES.tmpl_system_health(health_indicators, ln=ln)
def perform_display_ingestion_status(req_ingestion, ln=CFG_SITE_LANG):
"""
Display the updating status for the records matching a
given request.
@param req_ingestion: Search pattern request
@type req_ingestion: str
"""
# preconfigured values
- index_methods = ('global', 'collection', 'abstract', 'author', 'keyword',
- 'reference', 'reportnumber', 'title', 'fulltext',
- 'year', 'journal', 'collaboration', 'affiliation',
- 'exactauthor', 'caption', 'firstauthor',
- 'exactfirstauthor', 'authorcount')
+ index_methods = zip(*get_all_indexes(with_ids=True))[1]
rank_methods = ('wrd', 'demo_jif', 'citation', 'citerank_citation_t',
'citerank_pagerank_c', 'citerank_pagerank_t')
sort_methods = ('latest first', 'title', 'author', 'report number',
'most cited')
from ConfigParser import ConfigParser
conf = ConfigParser()
conf.read(CFG_WEBSTAT_CONFIG_PATH)
general = get_general_status()
flag = 0 # match with pending records
stats = []
list_records = get_ingestion_matching_records(req_ingestion, \
int(conf.get("general", "max_ingestion_health")))
if list_records == []:
stats.append(("No matches for your query!", " "*60))
return TEMPLATES.tmpl_ingestion_health(general, req_ingestion, stats, \
ln=ln)
else:
for record in list_records:
if record == 0:
return TEMPLATES.tmpl_ingestion_health(general, None, \
None, ln=ln)
elif record == -1:
stats.append(("Invalid pattern! Please retry", " "*60))
return TEMPLATES.tmpl_ingestion_health(general, None, \
stats, ln=ln)
else:
stat = get_record_ingestion_status(record)
last_mod = get_record_last_modification(record)
if stat != 0:
flag = 1 # match
# Indexing
stats.append((get_title_ingestion(record, last_mod)," "*90))
stats.append(("Pending for indexing methods:", " "*80))
for im in index_methods:
last = get_specific_ingestion_status(record,"index", im)
if last != None:
stats.append((" - %s"%im, "last: " + last))
# Ranking
stats.append(("Pending for ranking methods:", " "*80))
for rm in rank_methods:
last = get_specific_ingestion_status(record, "rank", rm)
if last != None:
stats.append((" - %s"%rm, "last: " + last))
# Sorting
stats.append(("Pending for sorting methods:", " "*80))
for sm in sort_methods:
last = get_specific_ingestion_status(record, "sort", sm)
if last != None:
stats.append((" - %s"%sm, "last: " + last))
# Collecting
stats.append(("Pending for webcolling:", " "*80))
last = get_specific_ingestion_status(record, "collect", )
if last != None:
stats.append((" - webcoll", "last: " + last))
# if there was no match
if flag == 0:
stats.append(("All matching records up to date!", " "*60))
return TEMPLATES.tmpl_ingestion_health(general, req_ingestion, stats, ln=ln)
def perform_display_yearly_report(ln=CFG_SITE_LANG):
"""
Display the year recount
"""
# Append loans stats to the box
year_report = []
year_report += loan_display()
year_report.append(None)
return TEMPLATES.tmpl_yearly_report(year_report, ln=ln)
def perform_display_keyevent(event_id=None, args={},
req=None, ln=CFG_SITE_LANG):
"""
Display key events using a certain output type over the given time span.
@param event_id: The ids for the custom events that are to be displayed.
@type event_id: [str]
@param args: { param name: argument value }
@type args: { str: str }
@param req: The Apache request object, necessary for export redirect.
@type req:
"""
# Get all the option lists:
# { parameter name: [(argument internal name, argument full name)]}
options = dict()
order = []
for param in KEYEVENT_REPOSITORY[event_id]['extraparams']:
# Order of options
order.append(param)
if KEYEVENT_REPOSITORY[event_id]['extraparams'][param][0] == 'combobox':
options[param] = ('combobox',
KEYEVENT_REPOSITORY[event_id]['extraparams'][param][1],
KEYEVENT_REPOSITORY[event_id]['extraparams'][param][2]())
else:
options[param] = (KEYEVENT_REPOSITORY[event_id]['extraparams'][param][0],
(KEYEVENT_REPOSITORY[event_id]['extraparams'][param][1]))
# Build a dictionary for the selected parameters:
# { parameter name: argument internal name }
choosed = dict([(param, args[param]) for param in KEYEVENT_REPOSITORY
[event_id]['extraparams']])
if KEYEVENT_REPOSITORY[event_id]['output'] == 'Graph':
options['format'] = ('combobox', 'Output format', _get_formats())
choosed['format'] = args['format']
order += ['format']
if event_id != 'items list':
if 'type' in KEYEVENT_REPOSITORY[event_id] and \
KEYEVENT_REPOSITORY[event_id]['type'] == 'bibcirculation':
options['timespan'] = ('combobox', 'Time span', _get_timespans(bibcirculation_stat=True))
else:
options['timespan'] = ('combobox', 'Time span', _get_timespans())
choosed['timespan'] = args['timespan']
order += ['timespan']
choosed['s_date'] = args['s_date']
choosed['f_date'] = args['f_date']
# Send to template to prepare event customization FORM box
list = KEYEVENT_REPOSITORY[event_id]['output'] == 'List'
out = "\n".join(["<p>%s</p>" % parr for parr in KEYEVENT_REPOSITORY[event_id]['description']]) \
+ TEMPLATES.tmpl_keyevent_box(options, order, choosed, ln=ln, list=list)
# Arguments OK?
# Check for existance. If nothing, only show FORM box from above.
if len(choosed) == 0:
return out
# Make sure extraparams are valid, if any
if KEYEVENT_REPOSITORY[event_id]['output'] == 'Graph' and \
event_id != 'percentage satisfied ill requests':
for param in choosed:
if param in options and options[param] == 'combobox' and \
not choosed[param] in [x[0] for x in options[param][2]]:
return out + TEMPLATES.tmpl_error(
'Please specify a valid value for parameter "%s".'
% options[param][0], ln=ln)
# Arguments OK beyond this point!
# Get unique name for caching purposes (make sure that the params used
# in the filename are safe!)
filename = KEYEVENT_REPOSITORY[event_id]['cachefilename'] \
% dict([(param, re.subn("[^\w]", "_", choosed[param])[0])
for param in choosed] +
[('event_id', re.subn("[^\w]", "_", event_id)[0])])
# Get time parameters from repository
if 'timespan' in choosed:
if choosed['timespan'] == "select date":
t_args = _get_time_parameters_select_date(args["s_date"], args["f_date"])
else:
t_args = _get_time_parameters(options, choosed['timespan'])
else:
t_args = args
for param in KEYEVENT_REPOSITORY[event_id]['extraparams']:
t_args[param] = choosed[param]
if 'format' in args and args['format'] == 'Full list':
gatherer = lambda: KEYEVENT_REPOSITORY[event_id]['gatherer'](t_args, limit=-1)
export_to_file(gatherer(), req)
return out
# Create closure of frequency function in case cache needs to be refreshed
gatherer = lambda return_sql: KEYEVENT_REPOSITORY[event_id]['gatherer'](t_args, return_sql=return_sql)
# Determine if this particular file is due for scheduling cacheing,
# in that case we must not allow refreshing of the rawdata.
allow_refresh = not _is_scheduled_for_cacheing(event_id)
# Get data file from cache (refresh if necessary)
force = 'timespan' in choosed and choosed['timespan'] == "select date"
data = eval(_get_file_using_cache(filename, gatherer, force,
allow_refresh=allow_refresh).read())
if KEYEVENT_REPOSITORY[event_id]['output'] == 'Graph':
# If type indicates an export, run the export function and we're done
if _is_type_export(choosed['format']):
_get_export_closure(choosed['format'])(data, req)
return out
# Prepare the graph settings that are being passed on to grapher
settings = {"title": KEYEVENT_REPOSITORY[event_id]['specificname']\
% choosed,
"xlabel": t_args['t_fullname'] + ' (' + \
t_args['granularity'] + ')',
"ylabel": KEYEVENT_REPOSITORY[event_id]['ylabel'],
"xtic_format": t_args['xtic_format'],
"format": choosed['format'],
"multiple": KEYEVENT_REPOSITORY[event_id]['multiple']}
else:
settings = {"title": KEYEVENT_REPOSITORY[event_id]['specificname']\
% choosed, "format": 'Table',
"rows": KEYEVENT_REPOSITORY[event_id]['rows']}
if args['sql']:
sql = gatherer(True)
else:
sql = ''
return out + _perform_display_event(data,
os.path.basename(filename), settings, ln=ln) + sql
def perform_display_customevent(ids=[], args={}, req=None, ln=CFG_SITE_LANG):
"""
Display custom events using a certain output type over the given time span.
@param ids: The ids for the custom events that are to be displayed.
@type ids: [str]
@param args: { param name: argument value }
@type args: { str: str }
@param req: The Apache request object, necessary for export redirect.
@type req:
"""
# Get all the option lists:
# { parameter name: [(argument internal name, argument full name)]}
cols_dict = _get_customevent_cols()
cols_dict['__header'] = 'Argument'
cols_dict['__none'] = []
options = {'ids': ('Custom event', _get_customevents()),
'timespan': ('Time span', _get_timespans()),
'format': ('Output format', _get_formats(True)),
'cols': cols_dict}
# Build a dictionary for the selected parameters:
# { parameter name: argument internal name }
choosed = {'ids': args['ids'], 'timespan': args['timespan'],
'format': args['format'], 's_date': args['s_date'],
'f_date': args['f_date']}
# Calculate cols
index = []
for key in args.keys():
if key[:4] == 'cols':
index.append(key[4:])
index.sort()
choosed['cols'] = [zip([""] + args['bool' + i], args['cols' + i],
args['col_value' + i]) for i in index]
# Send to template to prepare event customization FORM box
out = TEMPLATES.tmpl_customevent_box(options, choosed, ln=ln)
# Arguments OK?
# Make sure extraparams are valid, if any
for param in ['ids', 'timespan', 'format']:
legalvalues = [x[0] for x in options[param][1]]
if type(args[param]) is list:
# If the argument is a list, like the content of 'ids'
# every value has to be checked
if len(args[param]) == 0:
return out + TEMPLATES.tmpl_error(
'Please specify a valid value for parameter "%s".'
% options[param][0], ln=ln)
for arg in args[param]:
if not arg in legalvalues:
return out + TEMPLATES.tmpl_error(
'Please specify a valid value for parameter "%s".'
% options[param][0], ln=ln)
else:
if not args[param] in legalvalues:
return out + TEMPLATES.tmpl_error(
'Please specify a valid value for parameter "%s".'
% options[param][0], ln=ln)
# Fetch time parameters from repository
if choosed['timespan'] == "select date":
args_req = _get_time_parameters_select_date(args["s_date"],
args["f_date"])
else:
args_req = _get_time_parameters(options, choosed['timespan'])
# ASCII dump data is different from the standard formats
if choosed['format'] == 'asciidump':
data = perform_display_customevent_data_ascii_dump(ids, args,
args_req, choosed)
else:
data = perform_display_customevent_data(ids, args_req, choosed)
# If type indicates an export, run the export function and we're done
if _is_type_export(args['format']):
_get_export_closure(args['format'])(data, req)
return out
# Get full names, for those that have them
names = []
events = _get_customevents()
for event_id in ids:
temp = events[[x[0] for x in events].index(event_id)]
if temp[1] != None:
names.append(temp[1])
else:
names.append(temp[0])
# Generate a filename for the graph
filename = "tmp_webstat_customevent_" + ''.join([re.subn("[^\w]", "",
event_id)[0] for event_id in ids]) + "_"
if choosed['timespan'] == "select date":
filename += args_req['t_start'] + "_" + args_req['t_end']
else:
filename += choosed['timespan']
settings = {"title": 'Custom event',
"xlabel": args_req['t_fullname'] + ' (' + \
args_req['granularity'] + ')',
"ylabel": "Action quantity",
"xtic_format": args_req['xtic_format'],
"format": choosed['format'],
"multiple": (type(ids) is list) and names or []}
return out + _perform_display_event(data, os.path.basename(filename),
settings, ln=ln)
def perform_display_customevent_data(ids, args_req, choosed):
"""Returns the trend data"""
data_unmerged = []
for event_id, i in [(ids[i], str(i)) for i in range(len(ids))]:
# Calculate cols
args_req['cols'] = choosed['cols'][int(i)]
# Get unique name for the rawdata file (wash arguments!)
filename = "webstat_customevent_" + re.subn("[^\w]", "", event_id + \
"_" + choosed['timespan'] + "_" + '-'.join([':'.join(col)
for col in args_req['cols']]))[0]
# Add the current id to the gatherer's arguments
args_req['event_id'] = event_id
# Prepare raw data gatherer, if cache needs refreshing.
gatherer = lambda x: get_customevent_trend(args_req)
# Determine if this particular file is due for scheduling cacheing,
# in that case we must not allow refreshing of the rawdata.
allow_refresh = not _is_scheduled_for_cacheing(event_id)
# Get file from cache, and evaluate it to trend data
force = choosed['timespan'] == "select date"
data_unmerged.append(eval(_get_file_using_cache(filename, gatherer,
force, allow_refresh=allow_refresh).read()))
# Merge data from the unmerged trends into the final destination
return [(x[0][0], tuple([y[1] for y in x])) for x in zip(*data_unmerged)]
def perform_display_customevent_data_ascii_dump(ids, args, args_req, choosed):
"""Returns the trend data"""
for i in [str(j) for j in range(len(ids))]:
args['bool' + i].insert(0, "")
args_req['cols' + i] = zip(args['bool' + i], args['cols' + i],
args['col_value' + i])
filename = "webstat_customevent_" + re.subn("[^\w]", "", ''.join(ids) +
"_" + choosed['timespan'] + "_" + '-'.join([':'.join(col) for
col in [args['cols' + str(i)] for i in range(len(ids))]]) +
"_asciidump")[0]
args_req['ids'] = ids
gatherer = lambda: get_customevent_dump(args_req)
force = choosed['timespan'] == "select date"
return eval(_get_file_using_cache(filename, gatherer, force).read())
def perform_display_coll_list(req=None, ln=CFG_SITE_LANG):
"""
Display list of collections
@param req: The Apache request object, necessary for export redirect.
@type req:
"""
return TEMPLATES.tmpl_collection_stats_complete_list(get_collection_list_plus_all())
def perform_display_stats_per_coll(args={}, req=None, ln=CFG_SITE_LANG):
"""
Display general statistics for a given collection
@param args: { param name: argument value }
@type args: { str: str }
@param req: The Apache request object, necessary for export redirect.
@type req:
"""
events_id = ('collection population', 'download frequency', 'comments frequency')
# Get all the option lists:
# Make sure extraparams are valid, if any
if not args['collection'] in [x[0] for x in get_collection_list_plus_all()]:
return TEMPLATES.tmpl_error('Please specify a valid value for parameter "Collection".')
# { parameter name: [(argument internal name, argument full name)]}
options = {'collection': ('combobox', 'Collection', get_collection_list_plus_all()),
'timespan': ('combobox', 'Time span', _get_timespans()),
'format': ('combobox', 'Output format', _get_formats())}
order = options.keys()
# Arguments OK beyond this point!
# Get unique name for caching purposes (make sure that the params
# used in the filename are safe!)
out = TEMPLATES.tmpl_keyevent_box(options, order, args, ln=ln)
out += "<table>"
pair = False
for event_id in events_id:
# Get unique name for caching purposes (make sure that the params used
# in the filename are safe!)
filename = KEYEVENT_REPOSITORY[event_id]['cachefilename'] \
% dict([(param, re.subn("[^\w]", "_", args[param])[0])
for param in args] +
[('event_id', re.subn("[^\w]", "_", event_id)[0])])
# Get time parameters from repository
if args['timespan'] == "select date":
t_args = _get_time_parameters_select_date(args["s_date"], args["f_date"])
else:
t_args = _get_time_parameters(options, args['timespan'])
for param in KEYEVENT_REPOSITORY[event_id]['extraparams']:
t_args[param] = args[param]
# Create closure of frequency function in case cache needs to be refreshed
gatherer = lambda return_sql: KEYEVENT_REPOSITORY[event_id]['gatherer'](t_args, return_sql=return_sql)
# Determine if this particular file is due for scheduling cacheing,
# in that case we must not allow refreshing of the rawdata.
allow_refresh = not _is_scheduled_for_cacheing(event_id)
# Get data file from cache (refresh if necessary)
data = eval(_get_file_using_cache(filename, gatherer, allow_refresh=allow_refresh).read())
# Prepare the graph settings that are being passed on to grapher
settings = {"title": KEYEVENT_REPOSITORY[event_id]['specificname'] % t_args,
"xlabel": t_args['t_fullname'] + ' (' + \
t_args['granularity'] + ')',
"ylabel": KEYEVENT_REPOSITORY[event_id]['ylabel'],
"xtic_format": t_args['xtic_format'],
"format": args['format'],
"multiple": KEYEVENT_REPOSITORY[event_id]['multiple'],
"size": '360,270'}
if not pair:
out += '<tr>'
out += '<td>%s</td>' % _perform_display_event(data,
os.path.basename(filename), settings, ln=ln)
if pair:
out += '</tr>'
pair = not pair
return out + "</table>"
def perform_display_customevent_help(ln=CFG_SITE_LANG):
"""Display the custom event help"""
return TEMPLATES.tmpl_customevent_help(ln=ln)
def perform_display_error_log_analyzer(ln=CFG_SITE_LANG):
"""Display the error log analyzer"""
update_error_log_analyzer()
return TEMPLATES.tmpl_error_log_analyzer(get_invenio_error_log_ranking(),
get_invenio_last_n_errors(5),
get_apache_error_log_ranking())
def perform_display_custom_summary(args, ln=CFG_SITE_LANG):
"""Display the custom summary (annual report)
@param args: { param name: argument value } (chart title, search query and output tag)
@type args: { str: str }
"""
if args['tag'] == '':
args['tag'] = CFG_JOURNAL_TAG.replace("%", "p")
data = get_custom_summary_data(args['query'], args['tag'])
tag_name = _get_tag_name(args['tag'])
if tag_name == '':
tag_name = args['tag']
path = WEBSTAT_GRAPH_DIRECTORY + os.path.basename("tmp_webstat_custom_summary_"
+ args['query'] + args['tag'])
create_custom_summary_graph(data[:-1], path, args['title'])
return TEMPLATES.tmpl_display_custom_summary(tag_name, data, args['title'],
args['query'], args['tag'], path, ln=ln)
# INTERNALS
def _perform_display_event(data, name, settings, ln=CFG_SITE_LANG):
"""
Retrieves a graph or a table.
@param data: The trend/dump data
@type data: [(str, str|int|(str|int,...))] | [(str|int,...)]
@param name: The name of the trend (to be used as basename of graph file)
@type name: str
@param settings: Dictionary of graph parameters
@type settings: dict
@return: The URL of the graph (ASCII or image)
@type: str
"""
path = WEBSTAT_GRAPH_DIRECTORY + "tmp_" + name
# Generate, and insert using the appropriate template
if settings["format"] == "asciidump":
path += "_asciidump"
create_graph_dump(data, path)
out = TEMPLATES.tmpl_display_event_trend_ascii(settings["title"],
path, ln=ln)
if settings["format"] == "Table":
create_graph_table(data, path, settings)
return TEMPLATES.tmpl_display_event_trend_text(settings["title"], path, ln=ln)
create_graph_trend(data, path, settings)
if settings["format"] == "asciiart":
out = TEMPLATES.tmpl_display_event_trend_ascii(
settings["title"], path, ln=ln)
else:
if settings["format"] == "gnuplot":
try:
import Gnuplot
except ImportError:
out = 'Gnuplot is not installed. Returning ASCII art.' + \
TEMPLATES.tmpl_display_event_trend_ascii(
settings["title"], path, ln=ln)
out = TEMPLATES.tmpl_display_event_trend_image(
settings["title"], path, ln=ln)
elif settings["format"] == "flot":
out = TEMPLATES.tmpl_display_event_trend_text(
settings["title"], path, ln=ln)
else:
out = TEMPLATES.tmpl_display_event_trend_ascii(
settings["title"], path, ln=ln)
avgs, maxs, mins = get_numeric_stats(data, settings["multiple"] is not None)
return out + TEMPLATES.tmpl_display_numeric_stats(settings["multiple"],
avgs, maxs, mins)
def _get_customevents():
"""
Retrieves registered custom events from the database.
@return: [(internal name, readable name)]
@type: [(str, str)]
"""
return [(x[0], x[1]) for x in run_sql("SELECT id, name FROM staEVENT")]
def _get_timespans(dttime=None, bibcirculation_stat=False):
"""
Helper function that generates possible time spans to be put in the
drop-down in the generation box. Computes possible years, and also some
pre-defined simpler values. Some items in the list returned also tweaks the
output graph, if any, since such values are closely related to the nature
of the time span.
@param dttime: A datetime object indicating the current date and time
@type dttime: datetime.datetime
@return: [(Internal name, Readable name, t_start, t_end, granularity, format, xtic_format)]
@type [(str, str, str, str, str, str, str)]
"""
if dttime is None:
dttime = datetime.datetime.now()
dtformat = "%Y-%m-%d"
# Helper function to return a timediff object reflecting a diff of x days
d_diff = lambda x: datetime.timedelta(days=x)
# Helper function to return the number of days in the month x months ago
d_in_m = lambda x: calendar.monthrange(
((dttime.month - x < 1) and dttime.year - 1 or dttime.year),
(((dttime.month - 1) - x) % 12 + 1))[1]
to_str = lambda x: x.strftime(dtformat)
dt_str = to_str(dttime)
spans = [("today", "Today",
dt_str,
to_str(dttime + d_diff(1)),
"hour", dtformat, "%H"),
("this week", "This week",
to_str(dttime - d_diff(dttime.weekday())),
to_str(dttime + d_diff(1)),
"day", dtformat, "%a"),
("last week", "Last week",
to_str(dttime - d_diff(dttime.weekday() + 7)),
to_str(dttime - d_diff(dttime.weekday())),
"day", dtformat, "%a"),
("this month", "This month",
to_str(dttime - d_diff(dttime.day) + d_diff(1)),
to_str(dttime + d_diff(1)),
"day", dtformat, "%d"),
("last month", "Last month",
to_str(dttime - d_diff(d_in_m(1)) - d_diff(dttime.day) + d_diff(1)),
to_str(dttime - d_diff(dttime.day) + d_diff(1)),
"day", dtformat, "%d"),
("last three months", "Last three months",
to_str(dttime - d_diff(d_in_m(1)) - d_diff(d_in_m(2)) -
d_diff(dttime.day) + d_diff(1)),
dt_str,
"month", dtformat, "%b"),
("last year", "Last year",
to_str((dttime - datetime.timedelta(days=365)).replace(day=1)),
to_str((dttime + datetime.timedelta(days=31)).replace(day=1)),
"month", dtformat, "%b")]
# Get first year as indicated by the content's in bibrec or
# CFG_WEBSTAT_BIBCIRCULATION_START_YEAR
try:
if bibcirculation_stat and CFG_WEBSTAT_BIBCIRCULATION_START_YEAR:
year1 = int(CFG_WEBSTAT_BIBCIRCULATION_START_YEAR)
else:
year1 = run_sql("SELECT creation_date FROM bibrec ORDER BY \
creation_date LIMIT 1")[0][0].year
except:
year1 = dttime.year
year2 = time.localtime()[0]
diff_year = year2 - year1
if diff_year >= 2:
spans.append(("last 2 years", "Last 2 years",
to_str((dttime - datetime.timedelta(days=365 * 2)).replace(day=1)),
to_str((dttime + datetime.timedelta(days=31)).replace(day=1)),
"month", dtformat, "%b"))
if diff_year >= 5:
spans.append(("last 5 years", "Last 5 years",
to_str((dttime - datetime.timedelta(days=365 * 5)).replace(day=1)),
to_str((dttime + datetime.timedelta(days=31)).replace(day=1)),
"year", dtformat, "%Y"))
if diff_year >= 10:
spans.append(("last 10 years", "Last 10 years",
to_str((dttime - datetime.timedelta(days=365 * 10)).replace(day=1)),
to_str((dttime + datetime.timedelta(days=31)).replace(day=1)),
"year", dtformat, "%Y"))
spans.append(("full history", "Full history", str(year1), str(year2 + 1),
"year", "%Y", "%Y"))
spans.extend([(str(x), str(x), str(x), str(x + 1), "month", "%Y", "%b")
for x in range(year2, year1 - 1, -1)])
spans.append(("select date", "Select date...", "", "",
"hour", dtformat, "%H"))
return spans
def _get_time_parameters(options, timespan):
"""
Returns the time parameters from the repository when it is a default timespan
@param options: A dictionary with the option lists
@type options: { parameter name: [(argument internal name, argument full name)]}
@param timespan: name of the chosen timespan
@type timespan: str
@return: [(Full name, t_start, t_end, granularity, format, xtic_format)]
@type [(str, str, str, str, str, str, str)]
"""
if len(options['timespan']) == 2:
i = 1
else:
i = 2
_, t_fullname, t_start, t_end, granularity, t_format, xtic_format = \
options['timespan'][i][[x[0]
for x in options['timespan'][i]].index(timespan)]
return {'t_fullname': t_fullname, 't_start': t_start, 't_end': t_end,
'granularity': granularity, 't_format': t_format,
'xtic_format': xtic_format}
def _get_time_parameters_select_date(s_date, f_date):
"""
Returns the time parameters from the repository when it is a custom timespan
@param s_date: start date for the graph
@type s_date: str %m/%d/%Y %H:%M
@param f_date: finish date for the graph
@type f_date: str %m/%d/%Y %H:%M
@return: [(Full name, t_start, t_end, granularity, format, xtic_format)]
@type [(str, str, str, str, str, str, str)]
"""
t_fullname = "%s-%s" % (s_date, f_date)
dt_start = datetime.datetime(*(time.strptime(s_date, "%m/%d/%Y %H:%M")[0:6]))
dt_end = datetime.datetime(*(time.strptime(f_date, "%m/%d/%Y %H:%M")[0:6]))
if dt_end - dt_start <= timedelta(hours=1):
xtic_format = "%m:%s"
granularity = 'second'
elif dt_end - dt_start <= timedelta(days=1):
xtic_format = "%H:%m"
granularity = 'minute'
elif dt_end - dt_start <= timedelta(days=7):
xtic_format = "%H"
granularity = 'hour'
elif dt_end - dt_start <= timedelta(days=60):
xtic_format = "%a"
granularity = 'day'
elif dt_end - dt_start <= timedelta(days=730):
xtic_format = "%d"
granularity = 'month'
else:
xtic_format = "%H"
granularity = 'hour'
t_format = "%Y-%m-%d %H:%M:%S"
t_start = dt_start.strftime("%Y-%m-%d %H:%M:%S")
t_end = dt_end.strftime("%Y-%m-%d %H:%M:%S")
return {'t_fullname': t_fullname, 't_start': t_start, 't_end': t_end,
'granularity': granularity, 't_format': t_format,
'xtic_format': xtic_format}
def _get_formats(with_dump=False):
"""
Helper function to retrieve a Invenio friendly list of all possible
output types (displaying and exporting) from the central repository as
stored in the variable self.types at the top of this module.
@param with_dump: Optionally displays the custom-event only type 'asciidump'
@type with_dump: bool
@return: [(Internal name, Readable name)]
@type [(str, str)]
"""
# The third tuple value is internal
if with_dump:
return [(x[0], x[1]) for x in TYPE_REPOSITORY]
else:
return [(x[0], x[1]) for x in TYPE_REPOSITORY if x[0] != 'asciidump']
def _get_customevent_cols(event_id=""):
"""
List of all the diferent name of columns in customevents.
@return: {id: [(internal name, readable name)]}
@type: {str: [(str, str)]}
"""
sql_str = "SELECT id,cols FROM staEVENT"
sql_param = []
if event_id:
sql_str += "WHERE id = %s"
sql_param.append(event_id)
cols = {}
for event in run_sql(sql_str, sql_param):
if event[0]:
if event[1]:
cols[event[0]] = [(name, name) for name
in cPickle.loads(event[1])]
else:
cols[event[0]] = []
return cols
def _is_type_export(typename):
"""
Helper function that consults the central repository of types to determine
whether the input parameter represents an export type.
@param typename: Internal type name
@type typename: str
@return: Information whether a certain type exports data
@type: bool
"""
return len(TYPE_REPOSITORY[[x[0] for x in
TYPE_REPOSITORY].index(typename)]) == 3
def _get_export_closure(typename):
"""
Helper function that for a certain type, gives back the corresponding export
closure.
@param typename: Internal type name
@type typename: str
@return: Closure that exports data to the type's format
@type: function
"""
return TYPE_REPOSITORY[[x[0] for x in TYPE_REPOSITORY].index(typename)][2]
def _get_file_using_cache(filename, closure, force=False, allow_refresh=True):
"""
Uses the Invenio cache, i.e. the tempdir, to see if there's a recent
cached version of the sought-after file in there. If not, use the closure to
compute a new, and return that instead. Relies on Invenio configuration
parameter WEBSTAT_CACHE_INTERVAL.
@param filename: The name of the file that might be cached
@type filename: str
@param closure: A function, that executed will return data to be cached. The
function should return either a string, or something that
makes sense after being interpreted with str().
@type closure: function
@param force: Override cache default value.
@type force: bool
"""
# Absolute path to cached files, might not exist.
filename = os.path.normpath(WEBSTAT_RAWDATA_DIRECTORY + filename)
# Get the modification time of the cached file (if any).
try:
mtime = os.path.getmtime(filename)
except OSError:
# No cached version of this particular file exists, thus the
# modification time is set to 0 for easy logic below.
mtime = 0
# Consider refreshing cache if FORCE or NO CACHE AT ALL,
# or CACHE EXIST AND REFRESH IS ALLOWED.
if force or mtime == 0 or (mtime > 0 and allow_refresh):
# Is the file modification time recent enough?
if force or (time.time() - mtime > WEBSTAT_CACHE_INTERVAL):
# No! Use closure to compute new content
content = closure(False)
# Cache the data
open(filename, 'w').write(str(content))
# Return the (perhaps just) cached file
return open(filename, 'r')
def _is_scheduled_for_cacheing(event_id):
"""
@param event_id: The event id
@type event_id: str
@return: Indication of if the event id is scheduling for BibSched execution.
@type: bool
"""
if not is_task_scheduled('webstatadmin'):
return False
# Get the task id
try:
task_id = get_task_ids_by_descending_date('webstatadmin',
['RUNNING', 'WAITING'])[0]
except IndexError:
return False
else:
args = get_task_options(task_id)
return event_id in (args['keyevents'] + args['customevents'])
diff --git a/invenio/legacy/webstat/engine.py b/invenio/legacy/webstat/engine.py
index 2819e036c..4ee15d0a0 100644
--- a/invenio/legacy/webstat/engine.py
+++ b/invenio/legacy/webstat/engine.py
@@ -1,2867 +1,2867 @@
## This file is part of Invenio.
## Copyright (C) 2007, 2008, 2010, 2011, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
__revision__ = "$Id$"
__lastupdated__ = "$Date$"
import calendar, commands, datetime, time, os, cPickle, random, cgi
from operator import itemgetter
from invenio.config import CFG_TMPDIR, \
CFG_SITE_URL, \
CFG_SITE_NAME, \
CFG_BINDIR, \
CFG_CERN_SITE, \
CFG_BIBCIRCULATION_ITEM_STATUS_CANCELLED, \
CFG_BIBCIRCULATION_ITEM_STATUS_CLAIMED, \
CFG_BIBCIRCULATION_ITEM_STATUS_IN_PROCESS, \
CFG_BIBCIRCULATION_ITEM_STATUS_NOT_ARRIVED, \
CFG_BIBCIRCULATION_ITEM_STATUS_ON_LOAN, \
CFG_BIBCIRCULATION_ITEM_STATUS_ON_ORDER, \
CFG_BIBCIRCULATION_ITEM_STATUS_ON_SHELF, \
CFG_BIBCIRCULATION_ITEM_STATUS_OPTIONAL, \
CFG_BIBCIRCULATION_REQUEST_STATUS_DONE, \
CFG_BIBCIRCULATION_ILL_STATUS_CANCELLED
from invenio.modules.indexer.tokenizers.BibIndexJournalTokenizer import CFG_JOURNAL_TAG
from invenio.utils.url import redirect_to_url
from invenio.legacy.search_engine import perform_request_search, \
get_collection_reclist, \
get_most_popular_field_values, \
search_pattern
from invenio.legacy.bibrecord import get_fieldvalues
from invenio.legacy.dbquery import run_sql, \
wash_table_column_name
from invenio.legacy.websubmit.admin_dblayer import get_docid_docname_alldoctypes
from invenio.legacy.bibcirculation.utils import book_title_from_MARC, \
book_information_from_MARC
from invenio.legacy.bibcirculation.db_layer import get_id_bibrec, \
get_borrower_data
from invenio.legacy.websearch.webcoll import CFG_CACHE_LAST_UPDATED_TIMESTAMP_FILE
from invenio.utils.date import convert_datetext_to_datestruct, convert_datestruct_to_dategui
from invenio.legacy.bibsched.bibtask import get_modified_records_since
WEBSTAT_SESSION_LENGTH = 48 * 60 * 60 # seconds
WEBSTAT_GRAPH_TOKENS = '-=#+@$%&XOSKEHBC'
# KEY EVENT TREND SECTION
def get_keyevent_trend_collection_population(args, return_sql=False):
"""
Returns the quantity of documents in Invenio for
the given timestamp range.
@param args['collection']: A collection name
@type args['collection']: str
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
# collect action dates
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
if args.get('collection', 'All') == 'All':
sql_query_g = _get_sql_query("creation_date", args['granularity'],
"bibrec")
sql_query_i = "SELECT COUNT(id) FROM bibrec WHERE creation_date < %s"
initial_quantity = run_sql(sql_query_i, (lower, ))[0][0]
return _get_keyevent_trend(args, sql_query_g, initial_quantity=initial_quantity,
return_sql=return_sql, sql_text=
"Previous count: %s<br />Current count: %%s" % (sql_query_i),
acumulative=True)
else:
ids = get_collection_reclist(args['collection'])
if len(ids) == 0:
return []
g = get_keyevent_trend_new_records(args, return_sql, True)
sql_query_i = "SELECT id FROM bibrec WHERE creation_date < %s"
if return_sql:
return "Previous count: %s<br />Current count: %s" % (sql_query_i % lower, g)
initial_quantity = len(filter(lambda x: x[0] in ids, run_sql(sql_query_i, (lower, ))))
return _get_trend_from_actions(g, initial_quantity, args['t_start'],
args['t_end'], args['granularity'], args['t_format'], acumulative=True)
def get_keyevent_trend_new_records(args, return_sql=False, only_action=False):
"""
Returns the number of new records uploaded during the given timestamp range.
@param args['collection']: A collection name
@type args['collection']: str
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
if args.get('collection', 'All') == 'All':
return _get_keyevent_trend(args, _get_sql_query("creation_date", args['granularity'],
"bibrec"),
return_sql=return_sql)
else:
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
ids = get_collection_reclist(args['collection'])
if len(ids) == 0:
return []
sql = _get_sql_query("creation_date", args["granularity"], "bibrec",
extra_select=", id", group_by=False, count=False)
if return_sql:
return sql % (lower, upper)
recs = run_sql(sql, (lower, upper))
if recs:
def add_count(i_list, element):
""" Reduce function to create a dictionary with the count of ids
for each date """
if i_list and element == i_list[-1][0]:
i_list[-1][1] += 1
else:
i_list.append([element, 1])
return i_list
action_dates = reduce(add_count,
map(lambda x: x[0], filter(lambda x: x[1] in ids, recs)),
[])
else:
action_dates = []
if only_action:
return action_dates
return _get_trend_from_actions(action_dates, 0, args['t_start'],
args['t_end'], args['granularity'], args['t_format'])
def get_keyevent_trend_search_frequency(args, return_sql=False):
"""
Returns the number of searches (of any kind) carried out
during the given timestamp range.
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
return _get_keyevent_trend(args, _get_sql_query("date", args["granularity"],
"query INNER JOIN user_query ON id=id_query"),
return_sql=return_sql)
def get_keyevent_trend_comments_frequency(args, return_sql=False):
"""
Returns the number of comments (of any kind) carried out
during the given timestamp range.
@param args['collection']: A collection name
@type args['collection']: str
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
if args.get('collection', 'All') == 'All':
sql = _get_sql_query("date_creation", args["granularity"],
"cmtRECORDCOMMENT")
else:
sql = _get_sql_query("date_creation", args["granularity"],
"cmtRECORDCOMMENT", conditions=
_get_collection_recids_for_sql_query(args['collection']))
return _get_keyevent_trend(args, sql, return_sql=return_sql)
def get_keyevent_trend_search_type_distribution(args, return_sql=False):
"""
Returns the number of searches carried out during the given
timestamp range, but also partion them by type Simple and
Advanced.
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
# SQL to determine all simple searches:
simple = _get_sql_query("date", args["granularity"],
"query INNER JOIN user_query ON id=id_query",
conditions="urlargs LIKE '%%p=%%'")
# SQL to determine all advanced searches:
advanced = _get_sql_query("date", args["granularity"],
"query INNER JOIN user_query ON id=id_query",
conditions="urlargs LIKE '%%as=1%%'")
# Compute the trend for both types
s_trend = _get_keyevent_trend(args, simple,
return_sql=return_sql, sql_text="Simple: %s")
a_trend = _get_keyevent_trend(args, advanced,
return_sql=return_sql, sql_text="Advanced: %s")
# Assemble, according to return type
if return_sql:
return "%s <br /> %s" % (s_trend, a_trend)
return [(s_trend[i][0], (s_trend[i][1], a_trend[i][1]))
for i in range(len(s_trend))]
def get_keyevent_trend_download_frequency(args, return_sql=False):
"""
Returns the number of full text downloads carried out
during the given timestamp range.
@param args['collection']: A collection name
@type args['collection']: str
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
# Collect list of timestamps of insertion in the specific collection
if args.get('collection', 'All') == 'All':
return _get_keyevent_trend(args, _get_sql_query("download_time",
args["granularity"], "rnkDOWNLOADS"), return_sql=return_sql)
else:
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
ids = get_collection_reclist(args['collection'])
if len(ids) == 0:
return []
sql = _get_sql_query("download_time", args["granularity"], "rnkDOWNLOADS",
extra_select=", GROUP_CONCAT(id_bibrec)")
if return_sql:
return sql % (lower, upper)
action_dates = []
for result in run_sql(sql, (lower, upper)):
count = result[1]
for id in result[2].split(","):
if id == '' or not int(id) in ids:
count -= 1
action_dates.append((result[0], count))
return _get_trend_from_actions(action_dates, 0, args['t_start'],
args['t_end'], args['granularity'], args['t_format'])
def get_keyevent_trend_number_of_loans(args, return_sql=False):
"""
Returns the number of loans carried out
during the given timestamp range.
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
return _get_keyevent_trend(args, _get_sql_query("loaned_on",
args["granularity"], "crcLOAN"), return_sql=return_sql)
def get_keyevent_trend_web_submissions(args, return_sql=False):
"""
Returns the quantity of websubmissions in Invenio for
the given timestamp range.
@param args['doctype']: A doctype name
@type args['doctype']: str
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
if args['doctype'] == 'all':
sql = _get_sql_query("cd", args["granularity"], "sbmSUBMISSIONS",
conditions="action='SBI' AND status='finished'")
res = _get_keyevent_trend(args, sql, return_sql=return_sql)
else:
sql = _get_sql_query("cd", args["granularity"], "sbmSUBMISSIONS",
conditions="doctype=%s AND action='SBI' AND status='finished'")
res = _get_keyevent_trend(args, sql, extra_param=[args['doctype']],
return_sql=return_sql)
return res
def get_keyevent_loan_statistics(args, return_sql=False):
"""
Data:
- Number of documents (=records) loaned
- Number of items loaned on the total number of items
- Number of items never loaned on the total number of items
- Average time between the date of the record creation and the date of the first loan
Filter by
- in a specified time span
- by UDC (see MARC field 080__a - list to be submitted)
- by item status (available, missing)
- by date of publication (MARC field 260__c)
- by date of the record creation in the database
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['udc']: MARC field 080__a
@type args['udc']: str
@param args['item_status']: available, missing...
@type args['item_status']: str
@param args['publication_date']: MARC field 260__c
@type args['publication_date']: str
@param args['creation_date']: date of the record creation in the database
@type args['creation_date']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
# collect action dates
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
sql_from = "FROM crcLOAN l "
sql_where = "WHERE loaned_on > %s AND loaned_on < %s "
param = [lower, upper]
if 'udc' in args and args['udc'] != '':
sql_where += "AND l." + _check_udc_value_where()
param.append(_get_udc_truncated(args['udc']))
if 'item_status' in args and args['item_status'] != '':
sql_from += ", crcITEM i "
sql_where += "AND l.barcode = i.barcode AND i.status = %s "
param.append(args['item_status'])
if 'publication_date' in args and args['publication_date'] != '':
sql_where += "AND l.id_bibrec IN ( SELECT brb.id_bibrec \
FROM bibrec_bib26x brb, bib26x b WHERE brb.id_bibxxx = b.id AND tag='260__c' \
AND value LIKE %s)"
param.append('%%%s%%' % args['publication_date'])
if 'creation_date' in args and args['creation_date'] != '':
sql_from += ", bibrec br "
sql_where += "AND br.id=l.id_bibrec AND br.creation_date LIKE %s "
param.append('%%%s%%' % args['creation_date'])
param = tuple(param)
# Number of loans:
loans_sql = "SELECT COUNT(DISTINCT l.id_bibrec) " + sql_from + sql_where
items_loaned_sql = "SELECT COUNT(DISTINCT l.barcode) " + sql_from + sql_where
# Only the CERN site wants the items of the collection "Books & Proceedings"
if CFG_CERN_SITE:
items_in_book_coll = _get_collection_recids_for_sql_query("Books & Proceedings")
if items_in_book_coll == "":
total_items_sql = 0
else:
total_items_sql = "SELECT COUNT(*) FROM crcITEM WHERE %s" % \
items_in_book_coll
else: # The rest take all the items
total_items_sql = "SELECT COUNT(*) FROM crcITEM"
# Average time between the date of the record creation and the date of the first loan
avg_sql = "SELECT AVG(DATEDIFF(loaned_on, br.creation_date)) " + sql_from
if not ('creation_date' in args and args['creation_date'] != ''):
avg_sql += ", bibrec br "
avg_sql += sql_where
if not ('creation_date' in args and args['creation_date'] != ''):
avg_sql += "AND br.id=l.id_bibrec "
if return_sql:
return "<ol><li>%s</li><li>Items loaned * 100 / Number of items <ul><li>\
Items loaned: %s </li><li>Number of items: %s</li></ul></li><li>100 - Items \
loaned on total number of items</li><li>%s</li></ol>" % \
(loans_sql % param, items_loaned_sql % param, total_items_sql, avg_sql % param)
loans = run_sql(loans_sql, param)[0][0]
items_loaned = run_sql(items_loaned_sql, param)[0][0]
if total_items_sql:
total_items = run_sql(total_items_sql)[0][0]
else:
total_items = 0
if total_items == 0:
loaned_on_total = 0
never_loaned_on_total = 0
else:
# Number of items loaned on the total number of items:
loaned_on_total = float(items_loaned) * 100 / float(total_items)
# Number of items never loaned on the total number of items:
never_loaned_on_total = 100L - loaned_on_total
avg = run_sql(avg_sql, param)[0][0]
if avg:
avg = float(avg)
else:
avg = 0L
return ((loans, ), (loaned_on_total, ), (never_loaned_on_total, ), (avg, ))
def get_keyevent_loan_lists(args, return_sql=False, limit=50):
"""
Lists:
- List of documents (= records) never loaned
- List of most loaned documents (columns: number of loans,
number of copies and the creation date of the record, in
order to calculate the number of loans by copy), sorted
by decreasing order (50 items)
Filter by
- in a specified time span
- by UDC (see MARC field 080__a - list to be submitted)
- by loan period (4 week loan, one week loan...)
- by a certain number of loans
- by date of publication (MARC field 260__c)
- by date of the record creation in the database
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['udc']: MARC field 080__a
@type args['udc']: str
@param args['loan_period']: 4 week loan, one week loan...
@type args['loan_period']: str
@param args['min_loan']: minimum number of loans
@type args['min_loan']: int
@param args['max_loan']: maximum number of loans
@type args['max_loan']: int
@param args['publication_date']: MARC field 260__c
@type args['publication_date']: str
@param args['creation_date']: date of the record creation in the database
@type args['creation_date']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
sql_where = []
param = []
sql_from = ""
if 'udc' in args and args['udc'] != '':
sql_where.append("i." + _check_udc_value_where())
param.append(_get_udc_truncated(args['udc']))
if 'loan_period' in args and args['loan_period'] != '':
sql_where.append("loan_period = %s")
param.append(args['loan_period'])
if 'publication_date' in args and args['publication_date'] != '':
sql_where.append("i.id_bibrec IN ( SELECT brb.id_bibrec \
FROM bibrec_bib26x brb, bib26x b WHERE brb.id_bibxxx = b.id AND tag='260__c' \
AND value LIKE %s)")
param.append('%%%s%%' % args['publication_date'])
if 'creation_date' in args and args['creation_date'] != '':
sql_from += ", bibrec br"
sql_where.append("br.id=i.id_bibrec AND br.creation_date LIKE %s")
param.append('%%%s%%' % args['creation_date'])
if sql_where:
sql_where = "WHERE %s AND" % " AND ".join(sql_where)
else:
sql_where = "WHERE"
param = tuple(param + [lower, upper])
# SQL for both queries
check_num_loans = "HAVING "
if 'min_loans' in args and args['min_loans'] != '':
check_num_loans += "COUNT(*) >= %s" % args['min_loans']
if 'max_loans' in args and args['max_loans'] != '' and args['max_loans'] != 0:
if check_num_loans != "HAVING ":
check_num_loans += " AND "
check_num_loans += "COUNT(*) <= %s" % args['max_loans']
# Optimized to get all the data in only one query (not call get_fieldvalues several times)
mldocs_sql = "SELECT i.id_bibrec, COUNT(*) \
FROM crcLOAN l, crcITEM i%s %s l.barcode=i.barcode AND type = 'normal' AND \
loaned_on > %%s AND loaned_on < %%s GROUP BY i.id_bibrec %s" % \
(sql_from, sql_where, check_num_loans)
limit_n = ""
if limit > 0:
limit_n = "LIMIT %d" % limit
nldocs_sql = "SELECT id_bibrec, COUNT(*) FROM crcITEM i%s %s \
barcode NOT IN (SELECT id_bibrec FROM crcLOAN WHERE loaned_on > %%s AND \
loaned_on < %%s AND type = 'normal') GROUP BY id_bibrec ORDER BY COUNT(*) DESC %s" % \
(sql_from, sql_where, limit_n)
items_sql = "SELECT id_bibrec, COUNT(*) items FROM crcITEM GROUP BY id_bibrec"
creation_date_sql = "SELECT creation_date FROM bibrec WHERE id=%s"
authors_sql = "SELECT bx.value FROM bib10x bx, bibrec_bib10x bibx \
WHERE bx.id = bibx.id_bibxxx AND bx.tag LIKE '100__a' AND bibx.id_bibrec=%s"
title_sql = "SELECT GROUP_CONCAT(bx.value SEPARATOR ' ') value FROM bib24x bx, bibrec_bib24x bibx \
WHERE bx.id = bibx.id_bibxxx AND bx.tag LIKE %s AND bibx.id_bibrec=%s GROUP BY bibx.id_bibrec"
edition_sql = "SELECT bx.value FROM bib25x bx, bibrec_bib25x AS bibx \
WHERE bx.id = bibx.id_bibxxx AND bx.tag LIKE '250__a' AND bibx.id_bibrec=%s"
if return_sql:
return "Most loaned: %s<br \>Never loaned: %s" % \
(mldocs_sql % param, nldocs_sql % param)
mldocs = run_sql(mldocs_sql, param)
items = dict(run_sql(items_sql))
order_m = []
for mldoc in mldocs:
order_m.append([mldoc[0], mldoc[1], items[mldoc[0]], \
float(mldoc[1]) / float(items[mldoc[0]])])
order_m = sorted(order_m, key=itemgetter(3))
order_m.reverse()
# Check limit values
if limit > 0:
order_m = order_m[:limit]
res = [("", "Title", "Author", "Edition", "Number of loans",
"Number of copies", "Date of creation of the record")]
for mldoc in order_m:
res.append(("Most loaned documents",
_check_empty_value(run_sql(title_sql, ('245__%%', mldoc[0], ))),
_check_empty_value(run_sql(authors_sql, (mldoc[0], ))),
_check_empty_value(run_sql(edition_sql, (mldoc[0], ))),
mldoc[1], mldoc[2],
_check_empty_value(run_sql(creation_date_sql, (mldoc[0], )))))
nldocs = run_sql(nldocs_sql, param)
for nldoc in nldocs:
res.append(("Not loaned documents",
_check_empty_value(run_sql(title_sql, ('245__%%', nldoc[0], ))),
_check_empty_value(run_sql(authors_sql, (nldoc[0], ))),
_check_empty_value(run_sql(edition_sql, (nldoc[0], ))),
0, items[nldoc[0]],
_check_empty_value(run_sql(creation_date_sql, (nldoc[0], )))))
# nldocs = run_sql(nldocs_sql, param_n)
return (res)
def get_keyevent_renewals_lists(args, return_sql=False, limit=50):
"""
Lists:
- List of most renewed items stored by decreasing order (50 items)
Filter by
- in a specified time span
- by UDC (see MARC field 080__a - list to be submitted)
- by collection
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['udc']: MARC field 080__a
@type args['udc']: str
@param args['collection']: collection of the record
@type args['collection']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
sql_from = "FROM crcLOAN l, crcITEM i "
sql_where = "WHERE loaned_on > %s AND loaned_on < %s AND i.barcode = l.barcode "
param = [lower, upper]
if 'udc' in args and args['udc'] != '':
sql_where += "AND l." + _check_udc_value_where()
param.append(_get_udc_truncated(args['udc']))
filter_coll = False
if 'collection' in args and args['collection'] != '':
filter_coll = True
recid_list = get_collection_reclist(args['collection'])
param = tuple(param)
if limit > 0:
limit = "LIMIT %d" % limit
else:
limit = ""
sql = "SELECT i.id_bibrec, SUM(number_of_renewals) %s %s \
GROUP BY i.id_bibrec ORDER BY SUM(number_of_renewals) DESC %s" \
% (sql_from, sql_where, limit)
if return_sql:
return sql % param
# Results:
res = [("Title", "Author", "Edition", "Number of renewals")]
for rec, renewals in run_sql(sql, param):
if filter_coll and rec not in recid_list:
continue
author = get_fieldvalues(rec, "100__a")
if len(author) > 0:
author = author[0]
else:
author = ""
edition = get_fieldvalues(rec, "250__a")
if len(edition) > 0:
edition = edition[0]
else:
edition = ""
res.append((book_title_from_MARC(rec), author, edition, int(renewals)))
return (res)
def get_keyevent_returns_table(args, return_sql=False):
"""
Data:
- Number of overdue returns in a timespan
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
# Overdue returns:
sql = "SELECT COUNT(*) FROM crcLOAN l WHERE loaned_on > %s AND loaned_on < %s AND \
due_date < NOW() AND (returned_on IS NULL OR returned_on > due_date)"
if return_sql:
return sql % (lower, upper)
return ((run_sql(sql, (lower, upper))[0][0], ), )
def get_keyevent_trend_returns_percentage(args, return_sql=False):
"""
Returns the number of overdue returns and the total number of returns
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
# SQL to determine overdue returns:
overdue = _get_sql_query("due_date", args["granularity"], "crcLOAN",
conditions="due_date < NOW() AND due_date IS NOT NULL \
AND (returned_on IS NULL OR returned_on > due_date)",
dates_range_param="loaned_on")
# SQL to determine all returns:
total = _get_sql_query("due_date", args["granularity"], "crcLOAN",
conditions="due_date < NOW() AND due_date IS NOT NULL",
dates_range_param="loaned_on")
# Compute the trend for both types
o_trend = _get_keyevent_trend(args, overdue,
return_sql=return_sql, sql_text="Overdue: %s")
t_trend = _get_keyevent_trend(args, total,
return_sql=return_sql, sql_text="Total: %s")
# Assemble, according to return type
if return_sql:
return "%s <br /> %s" % (o_trend, t_trend)
return [(o_trend[i][0], (o_trend[i][1], t_trend[i][1]))
for i in range(len(o_trend))]
def get_keyevent_ill_requests_statistics(args, return_sql=False):
"""
Data:
- Number of ILL requests
- Number of satisfied ILL requests 2 weeks after the date of request
creation on a timespan
- Average time between the date and the hour of the ill request
date and the date and the hour of the delivery item to the user
on a timespan
- Average time between the date and the hour the ILL request
was sent to the supplier and the date and hour of the
delivery item on a timespan
Filter by
- in a specified time span
- by type of document (book or article)
- by status of the request (= new, sent, etc.)
- by supplier
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['doctype']: type of document (book or article)
@type args['doctype']: str
@param args['status']: status of the request (= new, sent, etc.)
@type args['status']: str
@param args['supplier']: supplier
@type args['supplier']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
sql_from = "FROM crcILLREQUEST ill "
sql_where = "WHERE period_of_interest_from > %s AND period_of_interest_from < %s "
param = [lower, upper]
if 'doctype' in args and args['doctype'] != '':
sql_where += "AND ill.request_type=%s"
param.append(args['doctype'])
if 'status' in args and args['status'] != '':
sql_where += "AND ill.status = %s "
param.append(args['status'])
else:
sql_where += "AND ill.status != %s "
param.append(CFG_BIBCIRCULATION_ILL_STATUS_CANCELLED)
if 'supplier' in args and args['supplier'] != '':
sql_from += ", crcLIBRARY lib "
sql_where += "AND lib.id=ill.id_crcLIBRARY AND lib.name=%s "
param.append(args['supplier'])
param = tuple(param)
requests_sql = "SELECT COUNT(*) %s %s" % (sql_from, sql_where)
satrequests_sql = "SELECT COUNT(*) %s %s \
AND arrival_date IS NOT NULL AND \
DATEDIFF(arrival_date, period_of_interest_from) < 14 " % (sql_from, sql_where)
avgdel_sql = "SELECT AVG(TIMESTAMPDIFF(DAY, period_of_interest_from, arrival_date)) %s %s \
AND arrival_date IS NOT NULL" % (sql_from, sql_where)
avgsup_sql = "SELECT AVG(TIMESTAMPDIFF(DAY, request_date, arrival_date)) %s %s \
AND arrival_date IS NOT NULL \
AND request_date IS NOT NULL" % (sql_from, sql_where)
if return_sql:
return "<ol><li>%s</li><li>%s</li><li>%s</li><li>%s</li></ol>" % \
(requests_sql % param, satrequests_sql % param,
avgdel_sql % param, avgsup_sql % param)
# Number of requests:
requests = run_sql(requests_sql, param)[0][0]
# Number of satisfied ILL requests 2 weeks after the date of request creation:
satrequests = run_sql(satrequests_sql, param)[0][0]
# Average time between the date and the hour of the ill request date and
# the date and the hour of the delivery item to the user
avgdel = run_sql(avgdel_sql, param)[0][0]
if avgdel:
avgdel = float(avgdel)
else:
avgdel = 0
# Average time between the date and the hour the ILL request was sent to
# the supplier and the date and hour of the delivery item
avgsup = run_sql(avgsup_sql, param)[0][0]
if avgsup:
avgsup = float(avgsup)
else:
avgsup = 0
return ((requests, ), (satrequests, ), (avgdel, ), (avgsup, ))
def get_keyevent_ill_requests_lists(args, return_sql=False, limit=50):
"""
Lists:
- List of ILL requests
Filter by
- in a specified time span
- by type of request (article or book)
- by supplier
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['doctype']: type of request (article or book)
@type args['doctype']: str
@param args['supplier']: supplier
@type args['supplier']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
sql_from = "FROM crcILLREQUEST ill "
sql_where = "WHERE status != '%s' AND request_date > %%s AND request_date < %%s " \
% CFG_BIBCIRCULATION_ITEM_STATUS_CANCELLED
param = [lower, upper]
if 'doctype' in args and args['doctype'] != '':
sql_where += "AND ill.request_type=%s "
param.append(args['doctype'])
if 'supplier' in args and args['supplier'] != '':
sql_from += ", crcLIBRARY lib "
sql_where += "AND lib.id=ill.id_crcLIBRARY AND lib.name=%s "
param.append(args['supplier'])
param = tuple(param)
if limit > 0:
limit = "LIMIT %d" % limit
else:
limit = ""
sql = "SELECT ill.id, item_info %s %s %s" % (sql_from, sql_where, limit)
if return_sql:
return sql % param
# Results:
res = [("Id", "Title", "Author", "Edition")]
for req_id, item_info in run_sql(sql, param):
item_info = eval(item_info)
try:
res.append((req_id, item_info['title'], item_info['authors'], item_info['edition']))
except KeyError:
pass
return (res)
def get_keyevent_trend_satisfied_ill_requests_percentage(args, return_sql=False):
"""
Returns the number of satisfied ILL requests 2 weeks after the date of request
creation and the total number of ILL requests
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['doctype']: type of document (book or article)
@type args['doctype']: str
@param args['status']: status of the request (= new, sent, etc.)
@type args['status']: str
@param args['supplier']: supplier
@type args['supplier']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
sql_from = "crcILLREQUEST ill "
sql_where = ""
param = []
if 'doctype' in args and args['doctype'] != '':
sql_where += "AND ill.request_type=%s"
param.append(args['doctype'])
if 'status' in args and args['status'] != '':
sql_where += "AND ill.status = %s "
param.append(args['status'])
else:
sql_where += "AND ill.status != %s "
param.append(CFG_BIBCIRCULATION_ILL_STATUS_CANCELLED)
if 'supplier' in args and args['supplier'] != '':
sql_from += ", crcLIBRARY lib "
sql_where += "AND lib.id=ill.id_crcLIBRARY AND lib.name=%s "
param.append(args['supplier'])
# SQL to determine satisfied ILL requests:
satisfied = _get_sql_query("request_date", args["granularity"], sql_from,
conditions="ADDDATE(request_date, 14) < NOW() AND \
(arrival_date IS NULL OR arrival_date < ADDDATE(request_date, 14)) " + sql_where)
# SQL to determine all ILL requests:
total = _get_sql_query("request_date", args["granularity"], sql_from,
conditions="ADDDATE(request_date, 14) < NOW() "+ sql_where)
# Compute the trend for both types
s_trend = _get_keyevent_trend(args, satisfied, extra_param=param,
return_sql=return_sql, sql_text="Satisfied: %s")
t_trend = _get_keyevent_trend(args, total, extra_param=param,
return_sql=return_sql, sql_text="Total: %s")
# Assemble, according to return type
if return_sql:
return "%s <br /> %s" % (s_trend, t_trend)
return [(s_trend[i][0], (s_trend[i][1], t_trend[i][1]))
for i in range(len(s_trend))]
def get_keyevent_items_statistics(args, return_sql=False):
"""
Data:
- The total number of items
- Total number of new items added in last year
Filter by
- in a specified time span
- by collection
- by UDC (see MARC field 080__a - list to be submitted)
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['udc']: MARC field 080__a
@type args['udc']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
sql_from = "FROM crcITEM i "
sql_where = "WHERE "
param = []
if 'udc' in args and args['udc'] != '':
sql_where += "i." + _check_udc_value_where()
param.append(_get_udc_truncated(args['udc']))
# Number of items:
if sql_where == "WHERE ":
sql_where = ""
items_sql = "SELECT COUNT(i.id_bibrec) %s %s" % (sql_from, sql_where)
# Number of new items:
if sql_where == "":
sql_where = "WHERE creation_date > %s AND creation_date < %s "
else:
sql_where += " AND creation_date > %s AND creation_date < %s "
new_items_sql = "SELECT COUNT(i.id_bibrec) %s %s" % (sql_from, sql_where)
if return_sql:
return "Total: %s <br />New: %s" % (items_sql % tuple(param), new_items_sql % tuple(param + [lower, upper]))
return ((run_sql(items_sql, tuple(param))[0][0], ), (run_sql(new_items_sql, tuple(param + [lower, upper]))[0][0], ))
def get_keyevent_items_lists(args, return_sql=False, limit=50):
"""
Lists:
- The list of items
Filter by
- by library (=physical location of the item)
- by status (=on loan, available, requested, missing...)
@param args['library']: physical location of the item
@type args[library'']: str
@param args['status']: on loan, available, requested, missing...
@type args['status']: str
"""
sql_from = "FROM crcITEM i "
sql_where = "WHERE "
param = []
if 'library' in args and args['library'] != '':
sql_from += ", crcLIBRARY li "
sql_where += "li.id=i.id_crcLIBRARY AND li.name=%s "
param.append(args['library'])
if 'status' in args and args['status'] != '':
if sql_where != "WHERE ":
sql_where += "AND "
sql_where += "i.status = %s "
param.append(args['status'])
param = tuple(param)
# Results:
res = [("Title", "Author", "Edition", "Barcode", "Publication date")]
if sql_where == "WHERE ":
sql_where = ""
if limit > 0:
limit = "LIMIT %d" % limit
else:
limit = ""
sql = "SELECT i.barcode, i.id_bibrec %s %s %s" % (sql_from, sql_where, limit)
if len(param) == 0:
sqlres = run_sql(sql)
else:
sqlres = run_sql(sql, tuple(param))
sql = sql % param
if return_sql:
return sql
for barcode, rec in sqlres:
author = get_fieldvalues(rec, "100__a")
if len(author) > 0:
author = author[0]
else:
author = ""
edition = get_fieldvalues(rec, "250__a")
if len(edition) > 0:
edition = edition[0]
else:
edition = ""
res.append((book_title_from_MARC(rec),
author, edition, barcode,
book_information_from_MARC(int(rec))[1]))
return (res)
def get_keyevent_loan_request_statistics(args, return_sql=False):
"""
Data:
- Number of hold requests, one week after the date of request creation
- Number of successful hold requests transactions
- Average time between the hold request date and the date of delivery document in a year
Filter by
- in a specified time span
- by item status (available, missing)
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['item_status']: available, missing...
@type args['item_status']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
sql_from = "FROM crcLOANREQUEST lr "
sql_where = "WHERE request_date > %s AND request_date < %s "
param = [lower, upper]
if 'item_status' in args and args['item_status'] != '':
sql_from += ", crcITEM i "
sql_where += "AND lr.barcode = i.barcode AND i.status = %s "
param.append(args['item_status'])
param = tuple(param)
custom_table = get_customevent_table("loanrequest")
# Number of hold requests, one week after the date of request creation:
holds = "SELECT COUNT(*) %s, %s ws %s AND ws.request_id=lr.id AND \
DATEDIFF(ws.creation_time, lr.request_date) >= 7" % (sql_from, custom_table, sql_where)
# Number of successful hold requests transactions
succesful_holds = "SELECT COUNT(*) %s %s AND lr.status='%s'" % (sql_from, sql_where,
CFG_BIBCIRCULATION_REQUEST_STATUS_DONE)
# Average time between the hold request date and the date of delivery document in a year
avg_sql = "SELECT AVG(DATEDIFF(ws.creation_time, lr.request_date)) \
%s, %s ws %s AND ws.request_id=lr.id" % (sql_from, custom_table, sql_where)
if return_sql:
return "<ol><li>%s</li><li>%s</li><li>%s</li></ol>" % \
(holds % param, succesful_holds % param, avg_sql % param)
avg = run_sql(avg_sql, param)[0][0]
if avg is int:
avg = int(avg)
else:
avg = 0
return ((run_sql(holds, param)[0][0], ),
(run_sql(succesful_holds, param)[0][0], ), (avg, ))
def get_keyevent_loan_request_lists(args, return_sql=False, limit=50):
"""
Lists:
- List of the most requested items
Filter by
- in a specified time span
- by UDC (see MARC field 080__a - list to be submitted)
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['udc']: MARC field 080__a
@type args['udc']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
sql_from = "FROM crcLOANREQUEST lr "
sql_where = "WHERE request_date > %s AND request_date < %s "
param = [lower, upper]
if 'udc' in args and args['udc'] != '':
sql_where += "AND lr." + _check_udc_value_where()
param.append(_get_udc_truncated(args['udc']))
if limit > 0:
limit = "LIMIT %d" % limit
else:
limit = ""
sql = "SELECT lr.barcode %s %s GROUP BY barcode \
ORDER BY COUNT(*) DESC %s" % (sql_from, sql_where, limit)
if return_sql:
return sql
res = [("Title", "Author", "Edition", "Barcode")]
# Most requested items:
for barcode in run_sql(sql, param):
rec = get_id_bibrec(barcode[0])
author = get_fieldvalues(rec, "100__a")
if len(author) > 0:
author = author[0]
else:
author = ""
edition = get_fieldvalues(rec, "250__a")
if len(edition) > 0:
edition = edition[0]
else:
edition = ""
res.append((book_title_from_MARC(rec), author, edition, barcode[0]))
return (res)
def get_keyevent_user_statistics(args, return_sql=False):
"""
Data:
- Total number of active users (to be defined = at least one transaction in the past year)
Filter by
- in a specified time span
- by registration date
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
sql_from_ill = "FROM crcILLREQUEST ill "
sql_from_loan = "FROM crcLOAN l "
sql_where_ill = "WHERE request_date > %s AND request_date < %s "
sql_where_loan = "WHERE loaned_on > %s AND loaned_on < %s "
param = (lower, upper, lower, upper)
# Total number of active users:
users = "SELECT COUNT(DISTINCT user) FROM ((SELECT id_crcBORROWER user %s %s) \
UNION (SELECT id_crcBORROWER user %s %s)) res" % \
(sql_from_ill, sql_where_ill, sql_from_loan, sql_where_loan)
if return_sql:
return users % param
return ((run_sql(users, param)[0][0], ), )
def get_keyevent_user_lists(args, return_sql=False, limit=50):
"""
Lists:
- List of most intensive users (ILL requests + Loan)
Filter by
- in a specified time span
- by registration date
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
param = (lower, upper, lower, upper)
if limit > 0:
limit = "LIMIT %d" % limit
else:
limit = ""
sql = "SELECT user, SUM(trans) FROM \
((SELECT id_crcBORROWER user, COUNT(*) trans FROM crcILLREQUEST ill \
WHERE request_date > %%s AND request_date < %%s GROUP BY id_crcBORROWER) UNION \
(SELECT id_crcBORROWER user, COUNT(*) trans FROM crcLOAN l WHERE loaned_on > %%s AND \
loaned_on < %%s GROUP BY id_crcBORROWER)) res GROUP BY user ORDER BY SUM(trans) DESC \
%s" % (limit)
if return_sql:
return sql % param
res = [("Name", "Address", "Mailbox", "E-mail", "Number of transactions")]
# List of most intensive users (ILL requests + Loan):
for borrower_id, trans in run_sql(sql, param):
name, address, mailbox, email = get_borrower_data(borrower_id)
res.append((name, address, mailbox, email, int(trans)))
return (res)
# KEY EVENT SNAPSHOT SECTION
def get_keyevent_snapshot_uptime_cmd():
"""
A specific implementation of get_current_event().
@return: The std-out from the UNIX command 'uptime'.
@type: str
"""
return _run_cmd('uptime').strip().replace(' ', ' ')
def get_keyevent_snapshot_apache_processes():
"""
A specific implementation of get_current_event().
@return: The std-out from the UNIX command 'uptime'.
@type: str
"""
# The number of Apache processes (root+children)
return _run_cmd('ps -e | grep apache2 | grep -v grep | wc -l')
def get_keyevent_snapshot_bibsched_status():
"""
A specific implementation of get_current_event().
@return: Information about the number of tasks in the different status modes.
@type: [(str, int)]
"""
sql = "SELECT status, COUNT(status) FROM schTASK GROUP BY status"
return [(x[0], int(x[1])) for x in run_sql(sql)]
def get_keyevent_snapshot_sessions():
"""
A specific implementation of get_current_event().
@return: The current number of website visitors (guests, logged in)
@type: (int, int)
"""
# SQL to retrieve sessions in the Guests
sql = "SELECT COUNT(session_expiry) " + \
"FROM session INNER JOIN user ON uid=id " + \
"WHERE email = '' AND " + \
"session_expiry-%d < unix_timestamp() AND " \
% WEBSTAT_SESSION_LENGTH + \
"unix_timestamp() < session_expiry"
guests = run_sql(sql)[0][0]
# SQL to retrieve sessions in the Logged in users
sql = "SELECT COUNT(session_expiry) " + \
"FROM session INNER JOIN user ON uid=id " + \
"WHERE email <> '' AND " + \
"session_expiry-%d < unix_timestamp() AND " \
% WEBSTAT_SESSION_LENGTH + \
"unix_timestamp() < session_expiry"
logged_ins = run_sql(sql)[0][0]
# Assemble, according to return type
return (guests, logged_ins)
def get_keyevent_bibcirculation_report(freq='yearly'):
"""
Monthly and yearly report with the total number of circulation
transactions (loans, renewals, returns, ILL requests, hold request).
@param freq: yearly or monthly
@type freq: str
@return: loans, renewals, returns, ILL requests, hold request
@type: (int, int, int, int, int)
"""
if freq == 'monthly':
datefrom = datetime.date.today().strftime("%Y-%m-01 00:00:00")
else: #yearly
datefrom = datetime.date.today().strftime("%Y-01-01 00:00:00")
loans, renewals = run_sql("SELECT COUNT(*), \
SUM(number_of_renewals) \
FROM crcLOAN WHERE loaned_on > %s", (datefrom, ))[0]
returns = run_sql("SELECT COUNT(*) FROM crcLOAN \
WHERE returned_on!='0000-00-00 00:00:00' and loaned_on > %s", (datefrom, ))[0][0]
illrequests = run_sql("SELECT COUNT(*) FROM crcILLREQUEST WHERE request_date > %s",
(datefrom, ))[0][0]
holdrequest = run_sql("SELECT COUNT(*) FROM crcLOANREQUEST WHERE request_date > %s",
(datefrom, ))[0][0]
return (loans, renewals, returns, illrequests, holdrequest)
def get_last_updates():
"""
List date/time when the last updates where done (easy reading format).
@return: last indexing, last ranking, last sorting, last webcolling
@type: (datetime, datetime, datetime, datetime)
"""
try:
last_index = convert_datestruct_to_dategui(convert_datetext_to_datestruct \
(str(run_sql('SELECT last_updated FROM idxINDEX WHERE \
name="global"')[0][0])))
last_rank = convert_datestruct_to_dategui(convert_datetext_to_datestruct \
(str(run_sql('SELECT last_updated FROM rnkMETHOD ORDER BY \
last_updated DESC LIMIT 1')[0][0])))
last_sort = convert_datestruct_to_dategui(convert_datetext_to_datestruct \
(str(run_sql('SELECT last_updated FROM bsrMETHODDATA ORDER BY \
last_updated DESC LIMIT 1')[0][0])))
file_coll_last_update = open(CFG_CACHE_LAST_UPDATED_TIMESTAMP_FILE, 'r')
last_coll = convert_datestruct_to_dategui(convert_datetext_to_datestruct \
(str(file_coll_last_update.read())))
file_coll_last_update.close()
# database not filled
except IndexError:
return ("", "", "", "")
return (last_index, last_rank, last_sort, last_coll)
def get_list_link(process, category=None):
"""
Builds the link for the list of records not indexed, ranked, sorted or
collected.
@param process: kind of process the records are waiting for (index, rank,
sort, collect)
@type process: str
@param category: specific sub-category of the process.
Index: global, collection, abstract, author, keyword,
reference, reportnumber, title, fulltext, year,
journal, collaboration, affiliation, exactauthor,
caption, firstauthor, exactfirstauthor, authorcount)
Rank: wrd, demo_jif, citation, citerank_citation_t,
citerank_pagerank_c, citerank_pagerank_t
Sort: latest first, title, author, report number,
most cited
Collect: Empty / None
@type category: str
@return: link text
@type: string
"""
if process == "index":
list_registers = run_sql('SELECT id FROM bibrec WHERE \
modification_date > (SELECT last_updated FROM \
idxINDEX WHERE name=%s)', (category,))
elif process == "rank":
list_registers = run_sql('SELECT id FROM bibrec WHERE \
modification_date > (SELECT last_updated FROM \
rnkMETHOD WHERE name=%s)', (category,))
elif process == "sort":
list_registers = run_sql('SELECT id FROM bibrec WHERE \
modification_date > (SELECT last_updated FROM \
bsrMETHODDATA WHERE id_bsrMETHOD=(SELECT id \
FROM bsrMETHOD WHERE name=%s))', (category,))
elif process == "collect":
file_coll_last_update = open(CFG_CACHE_LAST_UPDATED_TIMESTAMP_FILE, 'r')
coll_last_update = file_coll_last_update.read()
file_coll_last_update.close()
- list_registers = get_modified_records_since(coll_last_update)
+ list_registers = zip(get_modified_records_since(coll_last_update).tolist())
# build the link
- if list_registers == ():
+ if len(list_registers) == 0:
return "Up to date"
link = '<a href="' + CFG_SITE_URL + '/search?p='
for register in list_registers:
link += 'recid%3A' + str(register[0]) + '+or+'
# delete the last '+or+'
link = link[:len(link)-4]
link += '">' + str(len(list_registers)) + '</a>'
return link
def get_search_link(record_id):
"""
Auxiliar, builds the direct link for a given record.
@param record_id: record's id number
@type record_id: int
@return: link text
@type: string
"""
link = '<a href="' + CFG_SITE_URL + '/record/' + \
str(record_id) + '">Record [' + str(record_id) + ']</a>'
return link
def get_ingestion_matching_records(request=None, limit=25):
"""
Fetches all the records matching a given pattern, arranges them by last
modificaton date and returns a list.
@param request: requested pattern to match
@type request: str
@return: list of records matching a pattern,
(0,) if no request,
(-1,) if the request was invalid
@type: list
"""
if request==None or request=="":
return (0,)
try:
records = list(search_pattern(p=request))
except:
return (-1,)
if records == []:
return records
# order by most recent modification date
query = 'SELECT id FROM bibrec WHERE '
for r in records:
query += 'id="' + str(r) + '" OR '
query = query[:len(query)-4]
query += ' ORDER BY modification_date DESC LIMIT %s'
list_records = run_sql(query, (limit,))
final_list = []
for lr in list_records:
final_list.append(lr[0])
return final_list
def get_record_ingestion_status(record_id):
"""
Returns the amount of ingestion methods not updated yet to a given record.
If 0, the record is up to date.
@param record_id: record id number
@type record_id: int
@return: number of methods not updated for the record
@type: int
"""
counter = 0
counter += run_sql('SELECT COUNT(*) FROM bibrec WHERE \
id=%s AND modification_date > (SELECT last_updated FROM \
idxINDEX WHERE name="global")', (record_id, ))[0][0]
counter += run_sql('SELECT COUNT(*) FROM bibrec WHERE \
id=%s AND modification_date > (SELECT last_updated FROM \
rnkMETHOD ORDER BY last_updated DESC LIMIT 1)', \
(record_id, ))[0][0]
counter = run_sql('SELECT COUNT(*) FROM bibrec WHERE \
id=%s AND modification_date > (SELECT last_updated FROM \
bsrMETHODDATA ORDER BY last_updated DESC LIMIT 1)', \
(record_id, ))[0][0]
file_coll_last_update = open(CFG_CACHE_LAST_UPDATED_TIMESTAMP_FILE, 'r')
last_coll = file_coll_last_update.read()
file_coll_last_update.close()
counter += run_sql('SELECT COUNT(*) FROM bibrec WHERE \
id=%s AND \
modification_date >\
%s', (record_id, last_coll,))[0][0]
return counter
def get_specific_ingestion_status(record_id, process, method=None):
"""
Returns whether a record is or not up to date for a given
process and method.
@param record_id: identification number of the record
@type record_id: int
@param process: kind of process the records may be waiting for (index,
rank, sort, collect)
@type process: str
@param method: specific sub-method of the process.
Index: global, collection, abstract, author, keyword,
reference, reportnumber, title, fulltext, year,
journal, collaboration, affiliation, exactauthor,
caption, firstauthor, exactfirstauthor, authorcount
Rank: wrd, demo_jif, citation, citerank_citation_t,
citerank_pagerank_c, citerank_pagerank_t
Sort: latest first, title, author, report number,
most cited
Collect: Empty / None
@type category: str
@return: text: None if the record is up to date
Last time the method was updated if it is waiting
@type: date/time string
"""
exist = run_sql('SELECT COUNT(*) FROM bibrec WHERE id=%s', (record_id, ))
if exist[0][0] == 0:
return "REG not in DB"
if process == "index":
list_registers = run_sql('SELECT COUNT(*) FROM bibrec WHERE \
id=%s AND modification_date > (SELECT \
last_updated FROM idxINDEX WHERE name=%s)',
(record_id, method,))
last_time = run_sql ('SELECT last_updated FROM idxINDEX WHERE \
name=%s', (method,))[0][0]
elif process == "rank":
list_registers = run_sql('SELECT COUNT(*) FROM bibrec WHERE \
id=%s AND modification_date > (SELECT \
last_updated FROM rnkMETHOD WHERE name=%s)',
(record_id, method,))
last_time = run_sql ('SELECT last_updated FROM rnkMETHOD WHERE \
name=%s', (method,))[0][0]
elif process == "sort":
list_registers = run_sql('SELECT COUNT(*) FROM bibrec WHERE \
id=%s AND modification_date > (SELECT \
last_updated FROM bsrMETHODDATA WHERE \
id_bsrMETHOD=(SELECT id FROM bsrMETHOD \
WHERE name=%s))', (record_id, method,))
last_time = run_sql ('SELECT last_updated FROM bsrMETHODDATA WHERE \
id_bsrMETHOD=(SELECT id FROM bsrMETHOD \
WHERE name=%s)', (method,))[0][0]
elif process == "collect":
file_coll_last_update = open(CFG_CACHE_LAST_UPDATED_TIMESTAMP_FILE, 'r')
last_time = file_coll_last_update.read()
file_coll_last_update.close()
list_registers = run_sql('SELECT COUNT(*) FROM bibrec WHERE id=%s \
AND modification_date > %s',
(record_id, last_time,))
# no results means the register is up to date
if list_registers[0][0] == 0:
return None
else:
return convert_datestruct_to_dategui(convert_datetext_to_datestruct \
(str(last_time)))
def get_title_ingestion(record_id, last_modification):
"""
Auxiliar, builds a direct link for a given record, with its last
modification date.
@param record_id: id number of the record
@type record_id: string
@param last_modification: date/time of the last modification
@type last_modification: string
@return: link text
@type: string
"""
return '<h3><a href="%s/record/%s">Record [%s] last modification: %s</a></h3>' \
% (CFG_SITE_URL, record_id, record_id, last_modification)
def get_record_last_modification (record_id):
"""
Returns the date/time of the last modification made to a given record.
@param record_id: id number of the record
@type record_id: int
@return: date/time of the last modification
@type: string
"""
return convert_datestruct_to_dategui(convert_datetext_to_datestruct \
(str(run_sql('SELECT modification_date FROM bibrec \
WHERE id=%s', (record_id,))[0][0])))
def get_general_status():
"""
Returns an aproximate amount of ingestions processes not aplied to new or
updated records, using the "global" category.
@return: number of processes not updated
@type: int
"""
return run_sql('SELECT COUNT(*) FROM bibrec WHERE \
modification_date > (SELECT last_updated FROM \
idxINDEX WHERE name="global")')[0][0]
# ERROR LOG STATS
def update_error_log_analyzer():
"""Creates splitted files for today's errors"""
_run_cmd('bash %s/webstat -e -is' % CFG_BINDIR)
def get_invenio_error_log_ranking():
""" Returns the ranking of the errors in the invenio log"""
return _run_cmd('bash %s/webstat -e -ir' % CFG_BINDIR)
def get_invenio_last_n_errors(nerr):
"""Returns the last nerr errors in the invenio log (without details)"""
return _run_cmd('bash %s/webstat -e -il %d' % (CFG_BINDIR, nerr))
def get_invenio_error_details(error):
"""Returns the complete text of the invenio error."""
out = _run_cmd('bash %s/webstat -e -id %s' % (CFG_BINDIR, error))
return out
def get_apache_error_log_ranking():
""" Returns the ranking of the errors in the apache log"""
return _run_cmd('bash %s/webstat -e -ar' % CFG_BINDIR)
# CUSTOM EVENT SECTION
def get_customevent_trend(args):
"""
Returns trend data for a custom event over a given
timestamp range.
@param args['event_id']: The event id
@type args['event_id']: str
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
@param args['cols']: Columns and it's content that will be include
if don't exist or it's empty it will include all cols
@type args['cols']: [ [ str, str ], ]
"""
# Get a MySQL friendly date
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
tbl_name = get_customevent_table(args['event_id'])
col_names = get_customevent_args(args['event_id'])
where = []
sql_param = [lower, upper]
for col_bool, col_title, col_content in args['cols']:
if not col_title in col_names:
continue
if col_content:
if col_bool == "" or not where:
where.append(wash_table_column_name(col_title))
elif col_bool == "and":
where.append("AND %s"
% wash_table_column_name(col_title))
elif col_bool == "or":
where.append("OR %s"
% wash_table_column_name(col_title))
elif col_bool == "and_not":
where.append("AND NOT %s"
% wash_table_column_name(col_title))
else:
continue
where.append(" LIKE %s")
sql_param.append("%" + col_content + "%")
sql = _get_sql_query("creation_time", args['granularity'], tbl_name, " ".join(where))
return _get_trend_from_actions(run_sql(sql, tuple(sql_param)), 0,
args['t_start'], args['t_end'],
args['granularity'], args['t_format'])
def get_customevent_dump(args):
"""
Similar to a get_event_trend implemention, but NO refining aka frequency
handling is carried out what so ever. This is just a dump. A dump!
@param args['event_id']: The event id
@type args['event_id']: str
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
@param args['cols']: Columns and it's content that will be include
if don't exist or it's empty it will include all cols
@type args['cols']: [ [ str, str ], ]
"""
# Get a MySQL friendly date
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
# Get customevents
# events_list = [(creation_time, event, [arg1, arg2, ...]), ...]
event_list = []
event_cols = {}
for event_id, i in [(args['ids'][i], str(i))
for i in range(len(args['ids']))]:
# Get all the event arguments and creation times
tbl_name = get_customevent_table(event_id)
col_names = get_customevent_args(event_id)
sql_query = ["SELECT * FROM %s WHERE creation_time > '%%s'" % wash_table_column_name(tbl_name), (lower,)] # kwalitee: disable=sql
sql_query.append("AND creation_time < '%s'" % upper)
sql_param = []
for col_bool, col_title, col_content in args['cols' + i]:
if not col_title in col_names:
continue
if col_content:
if col_bool == "and" or col_bool == "":
sql_query.append("AND %s" % \
wash_table_column_name(col_title))
elif col_bool == "or":
sql_query.append("OR %s" % \
wash_table_column_name(col_title))
elif col_bool == "and_not":
sql_query.append("AND NOT %s" % \
wash_table_column_name(col_title))
else:
continue
sql_query.append(" LIKE %s")
sql_param.append("%" + col_content + "%")
sql_query.append("ORDER BY creation_time DESC")
sql = ' '.join(sql_query)
res = run_sql(sql, tuple(sql_param))
for row in res:
event_list.append((row[1], event_id, row[2:]))
# Get the event col names
try:
event_cols[event_id] = cPickle.loads(run_sql(
"SELECT cols FROM staEVENT WHERE id = %s",
(event_id, ))[0][0])
except TypeError:
event_cols[event_id] = ["Unnamed"]
event_list.sort()
output = []
for row in event_list:
temp = [row[1], row[0].strftime('%Y-%m-%d %H:%M:%S')]
arguments = ["%s: %s" % (event_cols[row[1]][i],
row[2][i]) for i in range(len(row[2]))]
temp.extend(arguments)
output.append(tuple(temp))
return output
def get_customevent_table(event_id):
"""
Helper function that for a certain event id retrives the corresponding
event table name.
"""
res = run_sql(
"SELECT CONCAT('staEVENT', number) FROM staEVENT WHERE id = %s", (event_id, ))
try:
return res[0][0]
except IndexError:
# No such event table
return None
def get_customevent_args(event_id):
"""
Helper function that for a certain event id retrives the corresponding
event argument (column) names.
"""
res = run_sql("SELECT cols FROM staEVENT WHERE id = %s", (event_id, ))
try:
if res[0][0]:
return cPickle.loads(res[0][0])
else:
return []
except IndexError:
# No such event table
return None
# CUSTOM SUMMARY SECTION
def get_custom_summary_data(query, tag):
"""Returns the annual report data for the specified year
@param query: Search query to make customized report
@type query: str
@param tag: MARC tag for the output
@type tag: str
"""
# Check arguments
if tag == '':
tag = CFG_JOURNAL_TAG.replace("%", "p")
# First get records of the year
recids = perform_request_search(p=query, of="id", wl=0)
# Then return list by tag
pub = get_most_popular_field_values(recids, tag)
if len(pub) == 0:
return []
if CFG_CERN_SITE:
total = sum([x[1] for x in pub])
else:
others = 0
total = 0
first_other = -1
for elem in pub:
total += elem[1]
if elem[1] < 2:
if first_other == -1:
first_other = pub.index(elem)
others += elem[1]
del pub[first_other:]
if others != 0:
pub.append(('Others', others))
pub.append(('TOTAL', total))
return pub
def create_custom_summary_graph(data, path, title):
"""
Creates a pie chart with the information from the custom summary and
saves it in the file specified by the path argument
"""
# If no input, we don't bother about anything
if len(data) == 0:
return
os.environ['HOME'] = CFG_TMPDIR
try:
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
except ImportError:
return
# make a square figure and axes
matplotlib.rcParams['font.size'] = 8
labels = [x[0] for x in data]
numb_elem = len(labels)
width = 6 + float(numb_elem) / 7
gfile = plt.figure(1, figsize=(width, 6))
plt.axes([0.1, 0.1, 4.2 / width, 0.7])
numb = [x[1] for x in data]
total = sum(numb)
fracs = [x * 100 / total for x in numb]
colors = []
random.seed()
for i in range(numb_elem):
col = 0.5 + float(i) / (float(numb_elem) * 2.0)
rand = random.random() / 2.0
if i % 3 == 0:
red = col
green = col + rand
blue = col - rand
if green > 1.0:
green = 1
elif i % 3 == 1:
red = col - rand
green = col
blue = col + rand
if blue > 1.0:
blue = 1
elif i % 3 == 2:
red = col + rand
green = col - rand
blue = col
if red > 1.0:
red = 1
colors.append((red, green, blue))
patches = plt.pie(fracs, colors=tuple(colors), labels=labels,
autopct='%1i%%', pctdistance=0.8, shadow=True)[0]
ttext = plt.title(title)
plt.setp(ttext, size='xx-large', color='b', family='monospace', weight='extra bold')
legend_keywords = {"prop": {"size": "small"}}
plt.figlegend(patches, labels, 'lower right', **legend_keywords)
plt.savefig(path)
plt.close(gfile)
# GRAPHER
def create_graph_trend(trend, path, settings):
"""
Creates a graph representation out of data produced from get_event_trend.
@param trend: The trend data
@type trend: [(str, str|int|(str|int,...))]
@param path: Where to store the graph
@type path: str
@param settings: Dictionary of graph parameters
@type settings: dict
"""
# If no input, we don't bother about anything
if not trend or len(trend) == 0:
return
# If no filename is given, we'll assume STD-out format and ASCII.
if path == '':
settings["format"] = 'asciiart'
if settings["format"] == 'asciiart':
create_graph_trend_ascii_art(trend, path, settings)
elif settings["format"] == 'gnuplot':
create_graph_trend_gnu_plot(trend, path, settings)
elif settings["format"] == "flot":
create_graph_trend_flot(trend, path, settings)
def create_graph_trend_ascii_art(trend, path, settings):
"""Creates the graph trend using ASCII art"""
out = ""
if settings["multiple"] is not None:
# Tokens that will represent the different data sets (maximum 16 sets)
# Set index (=100) to the biggest of the histogram sums
index = max([sum(x[1]) for x in trend])
# Print legend box
out += "Legend: %s\n\n" % ", ".join(["%s (%s)" % x
for x in zip(settings["multiple"], WEBSTAT_GRAPH_TOKENS)])
else:
index = max([x[1] for x in trend])
width = 82
# Figure out the max length of the xtics, in order to left align
xtic_max_len = max([len(_to_datetime(x[0]).strftime(
settings["xtic_format"])) for x in trend])
for row in trend:
# Print the xtic
xtic = _to_datetime(row[0]).strftime(settings["xtic_format"])
out_row = xtic + ': ' + ' ' * (xtic_max_len - len(xtic)) + '|'
try:
col_width = (1.0 * width / index)
except ZeroDivisionError:
col_width = 0
if settings["multiple"] is not None:
# The second value of the row-tuple, represents the n values from
# the n data sets. Each set, will be represented by a different
# ASCII character, chosen from the randomized string
# 'WEBSTAT_GRAPH_TOKENS'.
# NOTE: Only up to 16 (len(WEBSTAT_GRAPH_TOKENS)) data
# sets are supported.
total = sum(row[1])
for i in range(len(row[1])):
col = row[1][i]
try:
out_row += WEBSTAT_GRAPH_TOKENS[i] * int(1.0 * col * col_width)
except ZeroDivisionError:
break
if len([i for i in row[1] if type(i) is int and i > 0]) - 1 > 0:
out_row += out_row[-1]
else:
total = row[1]
try:
out_row += '-' * int(1.0 * total * col_width)
except ZeroDivisionError:
break
# Print sentinel, and the total
out += out_row + '>' + ' ' * (xtic_max_len + 4 +
width - len(out_row)) + str(total) + '\n'
# Write to destination file
if path == '':
print(out)
else:
open(path, 'w').write(out)
def create_graph_trend_gnu_plot(trend, path, settings):
"""Creates the graph trend using the GNU plot library"""
try:
import Gnuplot
except ImportError:
return
gnup = Gnuplot.Gnuplot()
gnup('set style data steps')
if 'size' in settings:
gnup('set terminal png tiny size %s' % settings['size'])
else:
gnup('set terminal png tiny')
gnup('set output "%s"' % path)
if settings["title"] != '':
gnup.title(settings["title"].replace("\"", ""))
if settings["xlabel"] != '':
gnup.xlabel(settings["xlabel"])
if settings["ylabel"] != '':
gnup.ylabel(settings["ylabel"])
if settings["xtic_format"] != '':
xtics = 'set xtics ('
xtics += ', '.join(['"%s" %d' %
(_to_datetime(trend[i][0], '%Y-%m-%d \
%H:%M:%S').strftime(settings["xtic_format"]), i)
for i in range(len(trend))]) + ')'
gnup(xtics)
gnup('set format y "%.0f"')
# If we have multiple data sets, we need to do
# some magic to make Gnuplot eat it,
# This is basically a matrix transposition,
# and the addition of index numbers.
if settings["multiple"] is not None:
cols = len(trend[0][1])
rows = len(trend)
plot_items = []
y_max = 0
y_min = 0
for col in range(cols):
data = []
for row in range(rows):
data.append([row, trend[row][1][col]])
data.append([rows, trend[-1][1][col]])
plot_items.append(Gnuplot.PlotItems
.Data(data, title=settings["multiple"][col]))
tmp_max = max([x[col] for x in data])
tmp_min = min([x[col] for x in data])
if tmp_max > y_max:
y_max = tmp_max
if tmp_min < y_min:
y_min = tmp_min
if y_max - y_min < 5 and y_min != 0:
gnup('set ytic %d, 1, %d' % (y_min - 1, y_max + 2))
elif y_max < 5:
gnup('set ytic 1')
gnup.plot(*plot_items)
else:
data = [x[1] for x in trend]
data.append(trend[-1][1])
y_max = max(data)
y_min = min(data)
if y_max - y_min < 5 and y_min != 0:
gnup('set ytic %d, 1, %d' % (y_min - 1, y_max + 2))
elif y_max < 5:
gnup('set ytic 1')
gnup.plot(data)
def create_graph_trend_flot(trend, path, settings):
"""Creates the graph trend using the flot library"""
size = settings.get("size", "500,400").split(",")
title = cgi.escape(settings["title"].replace(" ", "")[:10])
out = """<!--[if IE]><script language="javascript" type="text/javascript"
src="%(site)s/js/excanvas.min.js"></script><![endif]-->
<script language="javascript" type="text/javascript" src="%(site)s/js/jquery.flot.min.js"></script>
<script language="javascript" type="text/javascript" src="%(site)s/js/jquery.flot.selection.min.js"></script>
<script id="source" language="javascript" type="text/javascript">
document.write('<div style="float:left"><div id="placeholder%(title)s" style="width:%(width)spx;height:%(height)spx"></div></div>'+
'<div id="miniature%(title)s" style="float:left;margin-left:20px;margin-top:50px">' +
'<div id="overview%(title)s" style="width:%(hwidth)dpx;height:%(hheigth)dpx"></div>' +
'<p id="overviewLegend%(title)s" style="margin-left:10px"></p>' +
'</div>');
$(function () {
function parseDate%(title)s(sdate){
var div1 = sdate.split(' ');
var day = div1[0].split('-');
var hour = div1[1].split(':');
return new Date(day[0], day[1]-1, day[2], hour[0], hour[1], hour[2]).getTime() - (new Date().getTimezoneOffset() * 60 * 1000) ;
}
function getData%(title)s() {""" % \
{'site': CFG_SITE_URL, 'width': size[0], 'height': size[1], 'hwidth': int(size[0]) / 2,
'hheigth': int(size[1]) / 2, 'title': title}
if(len(trend) > 1):
granularity_td = (_to_datetime(trend[1][0], '%Y-%m-%d %H:%M:%S') -
_to_datetime(trend[0][0], '%Y-%m-%d %H:%M:%S'))
else:
granularity_td = datetime.timedelta()
# Create variables with the format dn = [[x1,y1], [x2,y2]]
minx = trend[0][0]
maxx = trend[0][0]
if settings["multiple"] is not None:
cols = len(trend[0][1])
rows = len(trend)
first = 0
for col in range(cols):
out += """var d%d = [""" % (col)
for row in range(rows):
if(first == 0):
first = 1
else:
out += ", "
if trend[row][0] < minx:
minx = trend[row][0]
if trend[row][0] > maxx:
maxx = trend[row][0]
out += '[parseDate%s("%s"),%d]' % \
(title, _to_datetime(trend[row][0], '%Y-%m-%d \
%H:%M:%S'), trend[row][1][col])
out += ", [parseDate%s('%s'), %d]];\n" % (title,
_to_datetime(maxx, '%Y-%m-%d %H:%M:%S')+ granularity_td,
trend[-1][1][col])
out += "return [\n"
first = 0
for col in range(cols):
if first == 0:
first = 1
else:
out += ", "
out += '{data : d%d, label : "%s"}' % \
(col, settings["multiple"][col])
out += "];\n}\n"
else:
out += """var d1 = ["""
rows = len(trend)
first = 0
for row in range(rows):
if trend[row][0] < minx:
minx = trend[row][0]
if trend[row][0] > maxx:
maxx = trend[row][0]
if first == 0:
first = 1
else:
out += ', '
out += '[parseDate%s("%s"),%d]' % \
(title, _to_datetime(trend[row][0], '%Y-%m-%d %H:%M:%S'),
trend[row][1])
out += """, [parseDate%s("%s"), %d]];
return [d1];
}
""" % (title, _to_datetime(maxx, '%Y-%m-%d %H:%M:%S') +
granularity_td, trend[-1][1])
# Set options
tics = """yaxis: {
tickDecimals : 0
},"""
if settings["xtic_format"] != '':
current = _to_datetime(maxx, '%Y-%m-%d %H:%M:%S')
next = current + granularity_td
if (granularity_td.seconds + granularity_td.days * 24 * 3600) > 2592000:
next = current.replace(day=31)
tics += 'xaxis: { mode:"time",min:parseDate%s("%s"),max:parseDate%s("%s")},'\
% (title, _to_datetime(minx, '%Y-%m-%d %H:%M:%S'), title, next)
out += """var options%s ={
series: {
lines: { steps: true, fill: true},
points: { show: false }
},
legend: {show: false},
%s
grid: { hoverable: true, clickable: true },
selection: { mode: "xy" }
};
""" % (title, tics, )
# Write the plot method in javascript
out += """var startData%(title)s = getData%(title)s();
var plot%(title)s = $.plot($("#placeholder%(title)s"), startData%(title)s, options%(title)s);
// setup overview
var overview%(title)s = $.plot($("#overview%(title)s"), startData%(title)s, {
legend: { show: true, container: $("#overviewLegend%(title)s") },
series: {
lines: { steps: true, fill: true, lineWidth: 1},
shadowSize: 0
},
%(tics)s
grid: { color: "#999" },
selection: { mode: "xy" }
});
""" % {"title": title, "tics": tics}
# Tooltip and zoom
out += """
function showTooltip%(title)s(x, y, contents) {
$('<div id="tooltip%(title)s">' + contents + '</div>').css( {
position: 'absolute',
display: 'none',
top: y - 5,
left: x + 10,
border: '1px solid #fdd',
padding: '2px',
'background-color': '#fee',
opacity: 0.80
}).appendTo("body").fadeIn(200);
}
var previousPoint%(title)s = null;
$("#placeholder%(title)s").bind("plothover", function (event, pos, item) {
if (item) {
if (previousPoint%(title)s != item.datapoint) {
previousPoint%(title)s = item.datapoint;
$("#tooltip%(title)s").remove();
var y = item.datapoint[1];
showTooltip%(title)s(item.pageX, item.pageY, y);
}
}
else {
$("#tooltip%(title)s").remove();
previousPoint%(title)s = null;
}
});
$("#placeholder%(title)s").bind("plotclick", function (event, pos, item) {
if (item) {
plot%(title)s.highlight(item.series, item.datapoint);
}
});
// now connect the two
$("#placeholder%(title)s").bind("plotselected", function (event, ranges) {
// clamp the zooming to prevent eternal zoom
if (ranges.xaxis.to - ranges.xaxis.from < 0.00001){
ranges.xaxis.to = ranges.xaxis.from + 0.00001;}
if (ranges.yaxis.to - ranges.yaxis.from < 0.00001){
ranges.yaxis.to = ranges.yaxis.from + 0.00001;}
// do the zooming
plot%(title)s = $.plot($("#placeholder%(title)s"), getData%(title)s(ranges.xaxis.from, ranges.xaxis.to),
$.extend(true, {}, options%(title)s, {
xaxis: { min: ranges.xaxis.from, max: ranges.xaxis.to },
yaxis: { min: ranges.yaxis.from, max: ranges.yaxis.to }
}));
// don't fire event on the overview to prevent eternal loop
overview%(title)s.setSelection(ranges, true);
});
$("#overview%(title)s").bind("plotselected", function (event, ranges) {
plot%(title)s.setSelection(ranges);
});
});
</script>
<noscript>Your browser does not support JavaScript!
Please, select another output format</noscript>""" % {'title' : title}
open(path, 'w').write(out)
def get_numeric_stats(data, multiple):
""" Returns average, max and min values for data """
data = [x[1] for x in data]
if data == []:
return (0, 0, 0)
if multiple:
lists = []
for i in range(len(data[0])):
lists.append([x[i] for x in data])
return ([float(sum(x)) / len(x) for x in lists], [max(x) for x in lists],
[min(x) for x in lists])
else:
return (float(sum(data)) / len(data), max(data), min(data))
def create_graph_table(data, path, settings):
"""
Creates a html table representation out of data.
@param data: The data
@type data: (str,...)
@param path: Where to store the graph
@type path: str
@param settings: Dictionary of table parameters
@type settings: dict
"""
out = """<table border="1">
"""
if settings['rows'] == []:
for row in data:
out += """<tr>
"""
for value in row:
out += """<td>%s</td>
""" % value
out += "</tr>"
else:
for dta, value in zip(settings['rows'], data):
out += """<tr>
<td>%s</td>
<td>
""" % dta
for vrow in value:
out += """%s<br />
""" % vrow
out = out[:-6] + "</td></tr>"
out += "</table>"
open(path, 'w').write(out)
def create_graph_dump(dump, path):
"""
Creates a graph representation out of data produced from get_event_trend.
@param dump: The dump data
@type dump: [(str|int,...)]
@param path: Where to store the graph
@type path: str
"""
out = ""
if len(dump) == 0:
out += "No actions for this custom event " + \
"are registered in the given time range."
else:
# Make every row in dump equally long, insert None if appropriate.
max_len = max([len(x) for x in dump])
events = [tuple(list(x) + [None] * (max_len - len(x))) for x in dump]
cols = ["Event", "Date and time"] + ["Argument %d" % i
for i in range(max_len - 2)]
column_widths = [max([len(str(x[i])) \
for x in events + [cols]]) + 3 for i in range(len(events[0]))]
for i in range(len(cols)):
out += cols[i] + ' ' * (column_widths[i] - len(cols[i]))
out += "\n"
for i in range(len(cols)):
out += '=' * (len(cols[i])) + ' ' * (column_widths[i] - len(cols[i]))
out += "\n\n"
for action in dump:
for i in range(len(action)):
if action[i] is None:
temp = ''
else:
temp = action[i]
out += str(temp) + ' ' * (column_widths[i] - len(str(temp)))
out += "\n"
# Write to destination file
if path == '':
print(out)
else:
open(path, 'w').write(out)
# EXPORT DATA TO SLS
def get_search_frequency(day=datetime.datetime.now().date()):
"""Returns the number of searches performed in the chosen day"""
searches = get_keyevent_trend_search_type_distribution(get_args(day))
return sum(searches[0][1])
def get_total_records(day=datetime.datetime.now().date()):
"""Returns the total number of records which existed in the chosen day"""
tomorrow = (datetime.datetime.now() +
datetime.timedelta(days=1)).strftime("%Y-%m-%d")
args = {'collection': CFG_SITE_NAME, 't_start': day.strftime("%Y-%m-%d"),
't_end': tomorrow, 'granularity': "day", 't_format': "%Y-%m-%d"}
try:
return get_keyevent_trend_collection_population(args)[0][1]
except IndexError:
return 0
def get_new_records(day=datetime.datetime.now().date()):
"""Returns the number of new records submitted in the chosen day"""
args = {'collection': CFG_SITE_NAME,
't_start': (day - datetime.timedelta(days=1)).strftime("%Y-%m-%d"),
't_end': day.strftime("%Y-%m-%d"), 'granularity': "day",
't_format': "%Y-%m-%d"}
try:
return (get_total_records(day) -
get_keyevent_trend_collection_population(args)[0][1])
except IndexError:
return 0
def get_download_frequency(day=datetime.datetime.now().date()):
"""Returns the number of downloads during the chosen day"""
return get_keyevent_trend_download_frequency(get_args(day))[0][1]
def get_comments_frequency(day=datetime.datetime.now().date()):
"""Returns the number of comments during the chosen day"""
return get_keyevent_trend_comments_frequency(get_args(day))[0][1]
def get_loans_frequency(day=datetime.datetime.now().date()):
"""Returns the number of comments during the chosen day"""
return get_keyevent_trend_number_of_loans(get_args(day))[0][1]
def get_web_submissions(day=datetime.datetime.now().date()):
"""Returns the number of web submissions during the chosen day"""
args = get_args(day)
args['doctype'] = 'all'
return get_keyevent_trend_web_submissions(args)[0][1]
def get_alerts(day=datetime.datetime.now().date()):
"""Returns the number of alerts during the chosen day"""
args = get_args(day)
args['cols'] = [('', '', '')]
args['event_id'] = 'alerts'
return get_customevent_trend(args)[0][1]
def get_journal_views(day=datetime.datetime.now().date()):
"""Returns the number of journal displays during the chosen day"""
args = get_args(day)
args['cols'] = [('', '', '')]
args['event_id'] = 'journals'
return get_customevent_trend(args)[0][1]
def get_basket_views(day=datetime.datetime.now().date()):
"""Returns the number of basket displays during the chosen day"""
args = get_args(day)
args['cols'] = [('', '', '')]
args['event_id'] = 'baskets'
return get_customevent_trend(args)[0][1]
def get_args(day):
"""Returns the most common arguments for the exporting to SLS methods"""
return {'t_start': day.strftime("%Y-%m-%d"),
't_end': (day + datetime.timedelta(days=1)).strftime("%Y-%m-%d"),
'granularity': "day", 't_format': "%Y-%m-%d"}
# EXPORTER
def export_to_python(data, req):
"""
Exports the data to Python code.
@param data: The Python data that should be exported
@type data: []
@param req: The Apache request object
@type req:
"""
_export("text/x-python", str(data), req)
def export_to_csv(data, req):
"""
Exports the data to CSV.
@param data: The Python data that should be exported
@type data: []
@param req: The Apache request object
@type req:
"""
csv_list = [""""%s",%s""" % (x[0], ",".join([str(y) for y in \
((type(x[1]) is tuple) and x[1] or (x[1], ))])) for x in data]
_export('text/csv', '\n'.join(csv_list), req)
def export_to_file(data, req):
"""
Exports the data to a file.
@param data: The Python data that should be exported
@type data: []
@param req: The Apache request object
@type req:
"""
try:
import xlwt
book = xlwt.Workbook(encoding="utf-8")
sheet1 = book.add_sheet('Sheet 1')
for row in range(0, len(data)):
for col in range(0, len(data[row])):
sheet1.write(row, col, "%s" % data[row][col])
filename = CFG_TMPDIR + "/webstat_export_" + \
str(time.time()).replace('.', '') + '.xls'
book.save(filename)
redirect_to_url(req, '%s/stats/export?filename=%s&mime=%s' \
% (CFG_SITE_URL, os.path.basename(filename), 'application/vnd.ms-excel'))
except ImportError:
csv_list = []
for row in data:
row = ['"%s"' % str(col) for col in row]
csv_list.append(",".join(row))
_export('text/csv', '\n'.join(csv_list), req)
# INTERNAL
def _export(mime, content, req):
"""
Helper function to pass on the export call. Create a
temporary file in which the content is stored, then let
redirect to the export web interface.
"""
filename = CFG_TMPDIR + "/webstat_export_" + \
str(time.time()).replace('.', '')
open(filename, 'w').write(content)
redirect_to_url(req, '%s/stats/export?filename=%s&mime=%s' \
% (CFG_SITE_URL, os.path.basename(filename), mime))
def _get_trend_from_actions(action_dates, initial_value,
t_start, t_end, granularity, dt_format, acumulative=False):
"""
Given a list of dates reflecting some sort of action/event, and some additional parameters,
an internal data format is returned. 'initial_value' set to zero, means that the frequency
will not be accumulative, but rather non-causal.
@param action_dates: A list of dates, indicating some sort of action/event.
@type action_dates: [datetime.datetime]
@param initial_value: The numerical offset the first action's value should make use of.
@type initial_value: int
@param t_start: Start time for the time domain in dt_format
@type t_start: str
@param t_end: End time for the time domain in dt_format
@type t_end: str
@param granularity: The granularity of the time domain, span between values.
Possible values are [year,month,day,hour,minute,second].
@type granularity: str
@param dt_format: Format of the 't_start' and 't_stop' parameters
@type dt_format: str
@return: A list of tuples zipping a time-domain and a value-domain
@type: [(str, int)]
"""
# Append the maximum date as a sentinel indicating we're done
action_dates = list(action_dates)
# Construct the datetime tuple for the stop time
stop_at = _to_datetime(t_end, dt_format) - datetime.timedelta(seconds=1)
vector = [(None, initial_value)]
try:
upcoming_action = action_dates.pop()
#Do not count null values (when year, month or day is 0)
if granularity in ("year", "month", "day") and upcoming_action[0] == 0:
upcoming_action = action_dates.pop()
except IndexError:
upcoming_action = (datetime.datetime.max, 0)
# Create an iterator running from the first day of activity
for current in _get_datetime_iter(t_start, granularity, dt_format):
# Counter of action_dates in the current span, set the initial value to
# zero to avoid accumlation.
if acumulative:
actions_here = vector[-1][1]
else:
actions_here = 0
# Check to see if there's an action date in the current span
if upcoming_action[0] == {"year": current.year,
"month": current.month,
"day": current.day,
"hour": current.hour,
"minute": current.minute,
"second": current.second
}[granularity]:
actions_here += upcoming_action[1]
try:
upcoming_action = action_dates.pop()
except IndexError:
upcoming_action = (datetime.datetime.max, 0)
vector.append((current.strftime('%Y-%m-%d %H:%M:%S'), actions_here))
# Make sure to stop the iteration at the end time
if {"year": current.year >= stop_at.year,
"month": current.month >= stop_at.month and current.year == stop_at.year,
"day": current.day >= stop_at.day and current.month == stop_at.month,
"hour": current.hour >= stop_at.hour and current.day == stop_at.day,
"minute": current.minute >= stop_at.minute and current.hour == stop_at.hour,
"second": current.second >= stop_at.second and current.minute == stop_at.minute
}[granularity]:
break
# Remove the first bogus tuple, and return
return vector[1:]
def _get_keyevent_trend(args, sql, initial_quantity=0, extra_param=[],
return_sql=False, sql_text='%s', acumulative=False):
"""
Returns the trend for the sql passed in the given timestamp range.
@param args['t_start']: Date and time of start point
@type args['t_start']: str
@param args['t_end']: Date and time of end point
@type args['t_end']: str
@param args['granularity']: Granularity of date and time
@type args['granularity']: str
@param args['t_format']: Date and time formatting string
@type args['t_format']: str
"""
# collect action dates
lower = _to_datetime(args['t_start'], args['t_format']).isoformat()
upper = _to_datetime(args['t_end'], args['t_format']).isoformat()
param = tuple([lower, upper] + extra_param)
if return_sql:
sql = sql % param
return sql_text % sql
return _get_trend_from_actions(run_sql(sql, param), initial_quantity, args['t_start'],
args['t_end'], args['granularity'], args['t_format'], acumulative)
def _get_datetime_iter(t_start, granularity='day',
dt_format='%Y-%m-%d %H:%M:%S'):
"""
Returns an iterator over datetime elements starting at an arbitrary time,
with granularity of a [year,month,day,hour,minute,second].
@param t_start: An arbitrary starting time in format %Y-%m-%d %H:%M:%S
@type t_start: str
@param granularity: The span between iterable elements, default is 'days'.
Possible values are [year,month,day,hour,minute,second].
@type granularity: str
@param dt_format: Format of the 't_start' parameter
@type dt_format: str
@return: An iterator of points in time
@type: iterator over datetime elements
"""
tim = _to_datetime(t_start, dt_format)
# Make a time increment depending on the granularity and the current time
# (the length of years and months vary over time)
span = ""
while True:
yield tim
if granularity == "year":
span = (calendar.isleap(tim.year) and ["days=366"] or ["days=365"])[0]
elif granularity == "month":
span = "days=" + str(calendar.monthrange(tim.year, tim.month)[1])
elif granularity == "day":
span = "days=1"
elif granularity == "hour":
span = "hours=1"
elif granularity == "minute":
span = "minutes=1"
elif granularity == "second":
span = "seconds=1"
else:
# Default just in case
span = "days=1"
tim += eval("datetime.timedelta(" + span + ")")
def _to_datetime(dttime, dt_format='%Y-%m-%d %H:%M:%S'):
"""
Transforms a string into a datetime
"""
return datetime.datetime(*time.strptime(dttime, dt_format)[:6])
def _run_cmd(command):
"""
Runs a certain command and returns the string output. If the command is
not found a string saying so will be returned. Use with caution!
@param command: The UNIX command to execute.
@type command: str
@return: The std-out from the command.
@type: str
"""
return commands.getoutput(command)
def _get_doctypes():
"""Returns all the possible doctypes of a new submission"""
doctypes = [("all", "All")]
for doctype in get_docid_docname_alldoctypes():
doctypes.append(doctype)
return doctypes
def _get_item_statuses():
"""Returns all the possible status of an item"""
return [(CFG_BIBCIRCULATION_ITEM_STATUS_CANCELLED, "Cancelled"),
(CFG_BIBCIRCULATION_ITEM_STATUS_CLAIMED, "Claimed"),
(CFG_BIBCIRCULATION_ITEM_STATUS_IN_PROCESS, "In process"),
(CFG_BIBCIRCULATION_ITEM_STATUS_NOT_ARRIVED, "Not arrived"),
(CFG_BIBCIRCULATION_ITEM_STATUS_ON_LOAN, "On loan"),
(CFG_BIBCIRCULATION_ITEM_STATUS_ON_ORDER, "On order"),
(CFG_BIBCIRCULATION_ITEM_STATUS_ON_SHELF, "On shelf")] + \
[(status, status) for status in CFG_BIBCIRCULATION_ITEM_STATUS_OPTIONAL]
def _get_item_doctype():
"""Returns all the possible types of document for an item"""
dts = []
for dat in run_sql("""SELECT DISTINCT(request_type)
FROM crcILLREQUEST ORDER BY request_type ASC"""):
dts.append((dat[0], dat[0]))
return dts
def _get_request_statuses():
"""Returns all the possible statuses for an ILL request"""
dts = []
for dat in run_sql("SELECT DISTINCT(status) FROM crcILLREQUEST ORDER BY status ASC"):
dts.append((dat[0], dat[0]))
return dts
def _get_libraries():
"""Returns all the possible libraries"""
dts = []
for dat in run_sql("SELECT name FROM crcLIBRARY ORDER BY name ASC"):
if not CFG_CERN_SITE or not "CERN" in dat[0]: # do not add internal libraries for CERN site
dts.append((dat[0], dat[0]))
return dts
def _get_loan_periods():
"""Returns all the possible loan periods for an item"""
dts = []
for dat in run_sql("SELECT DISTINCT(loan_period) FROM crcITEM ORDER BY loan_period ASC"):
dts.append((dat[0], dat[0]))
return dts
def _get_tag_name(tag):
"""
For a specific MARC tag, it returns the human-readable name
"""
res = run_sql("SELECT name FROM tag WHERE value LIKE %s", ('%' + tag + '%',))
if res:
return res[0][0]
res = run_sql("SELECT name FROM tag WHERE value LIKE %s", ('%' + tag[:-1] + '%',))
if res:
return res[0][0]
return ''
def _get_collection_recids_for_sql_query(coll):
ids = get_collection_reclist(coll).tolist()
if len(ids) == 0:
return ""
return "id_bibrec IN %s" % str(ids).replace('[', '(').replace(']', ')')
def _check_udc_value_where():
return "id_bibrec IN (SELECT brb.id_bibrec \
FROM bibrec_bib08x brb, bib08x b WHERE brb.id_bibxxx = b.id AND tag='080__a' \
AND value LIKE %s) "
def _get_udc_truncated(udc):
if udc[-1] == '*':
return "%s%%" % udc[:-1]
if udc[0] == '*':
return "%%%s" % udc[1:]
return "%s" % udc
def _check_empty_value(value):
if len(value) == 0:
return ""
else:
return value[0][0]
def _get_granularity_sql_functions(granularity):
try:
return {
"year": ("YEAR",),
"month": ("YEAR", "MONTH",),
"day": ("MONTH", "DAY",),
"hour": ("DAY", "HOUR",),
"minute": ("HOUR", "MINUTE",),
"second": ("MINUTE", "SECOND")
}[granularity]
except KeyError:
return ("MONTH", "DAY",)
def _get_sql_query(creation_time_name, granularity, tables_from, conditions="",
extra_select="", dates_range_param="", group_by=True, count=True):
if len(dates_range_param) == 0:
dates_range_param = creation_time_name
conditions = "%s > %%s AND %s < %%s %s" % (dates_range_param, dates_range_param,
len(conditions) > 0 and "AND %s" % conditions or "")
values = {'creation_time_name': creation_time_name,
'granularity_sql_function': _get_granularity_sql_functions(granularity)[-1],
'count': count and ", COUNT(*)" or "",
'tables_from': tables_from,
'conditions': conditions,
'extra_select': extra_select,
'group_by': ""}
if group_by:
values['group_by'] = "GROUP BY "
for fun in _get_granularity_sql_functions(granularity):
values['group_by'] += "%s(%s), " % (fun, creation_time_name)
values['group_by'] = values['group_by'][:-2]
return "SELECT %(granularity_sql_function)s(%(creation_time_name)s) %(count)s %(extra_select)s \
FROM %(tables_from)s WHERE %(conditions)s \
%(group_by)s \
ORDER BY %(creation_time_name)s DESC" % values
diff --git a/invenio/legacy/webstyle/templates.py b/invenio/legacy/webstyle/templates.py
index 94e55eaa1..e72f1427a 100644
--- a/invenio/legacy/webstyle/templates.py
+++ b/invenio/legacy/webstyle/templates.py
@@ -1,751 +1,749 @@
## This file is part of Invenio.
## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
WebStyle templates. Customize the look of pages of Invenio
"""
__revision__ = \
"$Id$"
import time
import cgi
import traceback
import urllib
import sys
import string
from bs4 import BeautifulSoup
-from flask import g, current_app
from invenio.ext.template import render_template_to_string
from invenio.config import \
CFG_SITE_RECORD, \
CFG_SITE_LANG, \
CFG_SITE_NAME, \
CFG_SITE_NAME_INTL, \
CFG_SITE_SUPPORT_EMAIL, \
CFG_SITE_SECURE_URL, \
CFG_BASE_URL, \
CFG_SITE_URL, \
CFG_VERSION, \
CFG_WEBSTYLE_TEMPLATE_SKIN, \
CFG_INSPIRE_SITE, \
CFG_WEBLINKBACK_TRACKBACK_ENABLED
from invenio.base.i18n import gettext_set_language, language_list_long, is_language_rtl
from invenio.utils.url import make_canonical_urlargd, create_html_link, \
get_canonical_and_alternates_urls
from invenio.utils.date import convert_datecvs_to_datestruct, \
convert_datestruct_to_dategui
from invenio.modules.formatter import format_record
from invenio.legacy import template
from invenio.utils.html import get_mathjax_header
websearch_templates = template.load('websearch')
class Template:
def tmpl_navtrailbox_body(self, ln, title, previous_links, separator,
prolog, epilog):
"""Bootstrap friendly-Create navigation trail box body
Parameters:
- 'ln' *string* - The language to display
- 'title' *string* - page title;
- 'previous_links' *string* - the trail content from site title until current page (both ends exclusive)
- 'prolog' *string* - HTML code to prefix the navtrail item with
- 'epilog' *string* - HTML code to suffix the navtrail item with
- 'separator' *string* - HTML code that separates two navtrail items
Output:
- text containing the navtrail
Note: returns empty string for Home page. (guessed by title).
"""
# load the right message language
_ = gettext_set_language(ln)
if title == CFG_SITE_NAME_INTL.get(ln, CFG_SITE_NAME):
return ""
# Breadcrumbs
# breadcrumb objects should provide properties 'text' and 'url'
# First element
breadcrumbs = [dict(text=_("Home"), url=CFG_SITE_URL), ]
# Decode previous elements
if previous_links:
soup = BeautifulSoup(previous_links)
for link in soup.find_all('a'):
breadcrumbs.append(dict(
text=unicode(' '.join(link.contents)),
url=link.get('href')))
# Add head
if title:
breadcrumbs.append(dict(text=title, url='#'))
return render_template_to_string("breadcrumbs.html",
breadcrumbs=breadcrumbs).encode('utf8')
def tmpl_page(self, req, **kwargs):
"""Creates a complete page
Parameters:
- 'ln' *string* - The language to display
- 'description' *string* - description goes to the metadata in the header of the HTML page,
not yet escaped for HTML
- 'keywords' *string* - keywords goes to the metadata in the header of the HTML page,
not yet escaped for HTML
- 'userinfobox' *string* - the HTML code for the user information box
- 'useractivities_menu' *string* - the HTML code for the user activities menu
- 'adminactivities_menu' *string* - the HTML code for the admin activities menu
- 'navtrailbox' *string* - the HTML code for the navigation trail box
- 'pageheaderadd' *string* - additional page header HTML code
- 'boxlefttop' *string* - left-top box HTML code
- 'boxlefttopadd' *string* - additional left-top box HTML code
- 'boxleftbottom' *string* - left-bottom box HTML code
- 'boxleftbottomadd' *string* - additional left-bottom box HTML code
- 'boxrighttop' *string* - right-top box HTML code
- 'boxrighttopadd' *string* - additional right-top box HTML code
- 'boxrightbottom' *string* - right-bottom box HTML code
- 'boxrightbottomadd' *string* - additional right-bottom box HTML code
- 'title' *string* - the title of the page, not yet escaped for HTML
- 'titleprologue' *string* - what to print before page title
- 'titleepilogue' *string* - what to print after page title
- 'body' *string* - the body of the page
- 'lastupdated' *string* - when the page was last updated
- 'uid' *int* - user ID
- 'pagefooteradd' *string* - additional page footer HTML code
- 'secure_page_p' *int* (0 or 1) - are we to use HTTPS friendly page elements or not?
- 'navmenuid' *string* - the id of the navigation item to highlight for this page
- 'metaheaderadd' *string* - list of further tags to add to the <HEAD></HEAD> part of the page
- 'rssurl' *string* - the url of the RSS feed for this page
- 'show_title_p' *int* (0 or 1) - do we display the page title in the body of the page?
- 'body_css_classes' *list* - list of classes to add to the body tag
- 'show_header' *boolean* - tells whether page header should be displayed or not
- 'show_footer' *boolean* - tells whether page footer should be displayed or not
Output:
- HTML code of the page
"""
ctx = dict(ln=CFG_SITE_LANG, description="",
keywords="", userinfobox="", useractivities_menu="",
adminactivities_menu="", navtrailbox="",
pageheaderadd="", boxlefttop="", boxlefttopadd="",
boxleftbottom="", boxleftbottomadd="",
boxrighttop="", boxrighttopadd="",
boxrightbottom="", boxrightbottomadd="",
titleprologue="", title="", titleepilogue="",
body="", lastupdated=None, pagefooteradd="", uid=0,
secure_page_p=0, navmenuid="", metaheaderadd="",
rssurl=CFG_SITE_URL+"/rss",
show_title_p=True, body_css_classes=None,
show_header=True, show_footer=True)
ctx.update(kwargs)
return render_template_to_string("legacy_page.html", **ctx).encode('utf8')
def tmpl_pageheader(self, req, **kwargs):
"""Creates a page header
Parameters:
- 'ln' *string* - The language to display
- 'headertitle' *string* - the title of the HTML page, not yet escaped for HTML
- 'description' *string* - description goes to the metadata in the header of the HTML page,
not yet escaped for HTML
- 'keywords' *string* - keywords goes to the metadata in the header of the HTML page,
not yet escaped for HTML
- 'userinfobox' *string* - the HTML code for the user information box
- 'useractivities_menu' *string* - the HTML code for the user activities menu
- 'adminactivities_menu' *string* - the HTML code for the admin activities menu
- 'navtrailbox' *string* - the HTML code for the navigation trail box
- 'pageheaderadd' *string* - additional page header HTML code
- 'uid' *int* - user ID
- 'secure_page_p' *int* (0 or 1) - are we to use HTTPS friendly page elements or not?
- 'navmenuid' *string* - the id of the navigation item to highlight for this page
- 'metaheaderadd' *string* - list of further tags to add to the <HEAD></HEAD> part of the page
- 'rssurl' *string* - the url of the RSS feed for this page
- 'body_css_classes' *list* - list of classes to add to the body tag
Output:
- HTML code of the page headers
"""
ctx = dict(ln=CFG_SITE_LANG, headertitle="",
description="", keywords="", userinfobox="",
useractivities_menu="", adminactivities_menu="",
navtrailbox="", pageheaderadd="", uid=0,
secure_page_p=0, navmenuid="admin", metaheaderadd="",
rssurl=CFG_SITE_URL+"/rss", body_css_classes=None)
ctx.update(kwargs)
if ctx['body_css_classes'] is None:
ctx['body_css_classes'] = [ctx.get('navmenuid', '')]
else:
ctx['body_css_classes'].append([ctx.get('navmenuid', '')])
return render_template_to_string(
"legacy_page.html",
no_pagebody=True,
no_pagefooter=True,
**ctx
).encode('utf8')
- return out
def tmpl_pagefooter(self, req, **kwargs):
"""Creates a page footer
Parameters:
- 'ln' *string* - The language to display
- 'lastupdated' *string* - when the page was last updated
- 'pagefooteradd' *string* - additional page footer HTML code
Output:
- HTML code of the page headers
"""
ctx = dict(ln=CFG_SITE_LANG, lastupdated=None, pagefooteradd=None)
ctx.update(kwargs)
lastupdated = ctx.get('lastupdated')
if lastupdated and lastupdated != '$Date$':
if lastupdated.startswith("$Date: ") or lastupdated.startswith("$Id: "):
ctx['lastupdated'] = convert_datecvs_to_datestruct(lastupdated)
return render_template_to_string(
"legacy_page.html",
no_pagebody=True,
no_pageheader=True,
**ctx
).encode('utf8')
def tmpl_language_selection_box(self, req, language=CFG_SITE_LANG):
"""Take URLARGS and LANGUAGE and return textual language
selection box for the given page.
Parameters:
- 'req' - The mod_python request object
- 'language' *string* - The selected language
"""
# load the right message language
_ = gettext_set_language(language)
# Work on a copy in order not to bork the arguments of the caller
argd = {}
if req and req.args:
argd.update(cgi.parse_qs(req.args))
parts = []
for (lang, lang_namelong) in language_list_long():
if lang == language:
parts.append('<span class="langinfo">%s</span>' % lang_namelong)
else:
# Update the 'ln' argument in the initial request
argd['ln'] = lang
if req and req.uri:
args = urllib.quote(req.uri, '/:?') + make_canonical_urlargd(argd, {})
else:
args = ""
parts.append(create_html_link(args,
{}, lang_namelong,
{'class': "langinfo"}))
if len(parts) > 1:
return _("This site is also available in the following languages:") + \
"<br />" + ' &nbsp;'.join(parts)
else:
## There is only one (or zero?) languages configured,
## so there so need to display language alternatives.
return ""
def tmpl_error_box(self, ln, title, verbose, req, errors):
"""Produces an error box.
Parameters:
- 'title' *string* - The title of the error box
- 'ln' *string* - The selected language
- 'verbose' *bool* - If lots of information should be displayed
- 'req' *object* - the request object
- 'errors' list of tuples (error_code, error_message)
"""
# load the right message language
_ = gettext_set_language(ln)
info_not_available = _("N/A")
if title is None:
if errors:
title = _("Error") + ': %s' % errors[0][1]
else:
title = _("Internal Error")
browser_s = _("Browser")
if req:
try:
if 'User-Agent' in req.headers_in:
browser_s += ': ' + req.headers_in['User-Agent']
else:
browser_s += ': ' + info_not_available
host_s = req.hostname
page_s = req.unparsed_uri
client_s = req.remote_ip
except: # FIXME: bad except
browser_s += ': ' + info_not_available
host_s = page_s = client_s = info_not_available
else:
browser_s += ': ' + info_not_available
host_s = page_s = client_s = info_not_available
error_s = ''
sys_error_s = ''
traceback_s = ''
if verbose >= 1:
if sys.exc_info()[0]:
sys_error_s = '\n' + _("System Error") + ': %s %s\n' % \
(sys.exc_info()[0], sys.exc_info()[1])
if errors:
errs = ''
for error_tuple in errors:
try:
errs += "%s%s : %s\n " % (' '*6, error_tuple[0],
error_tuple[1])
except:
errs += "%s%s\n" % (' '*6, error_tuple)
errs = errs[6:-2] # get rid of trainling ','
error_s = _("Error") + ': %s")' % errs + "\n"
else:
error_s = _("Error") + ': ' + info_not_available
if verbose >= 9:
traceback_s = '\n' + _("Traceback") + ': \n%s' % \
string.join(traceback.format_tb(sys.exc_info()[2]),
"\n")
out = """
<table class="errorbox">
<thead>
<tr>
<th class="errorboxheader">
<p> %(title)s %(sys1)s %(sys2)s</p>
</th>
</tr>
</thead>
<tbody>
<tr>
<td class="errorboxbody">
<p>%(contact)s</p>
<blockquote><pre>
URI: http://%(host)s%(page)s
%(time_label)s: %(time)s
%(browser)s
%(client_label)s: %(client)s
%(error)s%(sys_error)s%(traceback)s
</pre></blockquote>
</td>
</tr>
<tr>
<td>
<form action="%(siteurl)s/error/send" method="post">
%(send_error_label)s
<input class="adminbutton" type="submit" value="%(send_label)s" />
<input type="hidden" name="header" value="%(title)s %(sys1)s %(sys2)s" />
<input type="hidden" name="url" value="URI: http://%(host)s%(page)s" />
<input type="hidden" name="time" value="Time: %(time)s" />
<input type="hidden" name="browser" value="%(browser)s" />
<input type="hidden" name="client" value="Client: %(client)s" />
<input type="hidden" name="error" value="%(error)s" />
<input type="hidden" name="sys_error" value="%(sys_error)s" />
<input type="hidden" name="traceback" value="%(traceback)s" />
<input type="hidden" name="referer" value="%(referer)s" />
</form>
</td>
</tr>
</tbody>
</table>
""" % {
'title' : cgi.escape(title).replace('"', '&quot;'),
'time_label': _("Time"),
'client_label': _("Client"),
'send_error_label': \
_("Please send an error report to the administrator."),
'send_label': _("Send error report"),
'sys1' : cgi.escape(str((sys.exc_info()[0] or ''))).replace('"', '&quot;'),
'sys2' : cgi.escape(str((sys.exc_info()[1] or ''))).replace('"', '&quot;'),
'contact' : \
_("Please contact %(x_name)s quoting the following information:",
x_name=('<a href="mailto:' + urllib.quote(CFG_SITE_SUPPORT_EMAIL) +'">' + CFG_SITE_SUPPORT_EMAIL + '</a>')),
'host' : cgi.escape(host_s),
'page' : cgi.escape(page_s),
'time' : time.strftime("%d/%b/%Y:%H:%M:%S %z"),
'browser' : cgi.escape(browser_s).replace('"', '&quot;'),
'client' : cgi.escape(client_s).replace('"', '&quot;'),
'error' : cgi.escape(error_s).replace('"', '&quot;'),
'traceback' : cgi.escape(traceback_s).replace('"', '&quot;'),
'sys_error' : cgi.escape(sys_error_s).replace('"', '&quot;'),
'siteurl' : CFG_BASE_URL,
'referer' : page_s!=info_not_available and \
("http://" + host_s + page_s) or \
info_not_available
}
return out
def detailed_record_container_top(self, recid, tabs, ln=CFG_SITE_LANG,
show_similar_rec_p=True,
creationdate=None,
modificationdate=None, show_short_rec_p=True,
citationnum=-1, referencenum=-1, discussionnum=-1,
include_jquery = False, include_mathjax = False):
"""Prints the box displayed in detailed records pages, with tabs at the top.
Returns content as it is if the number of tabs for this record
is smaller than 2
Parameters:
@param recid: int - the id of the displayed record
@param tabs: ** - the tabs displayed at the top of the box.
@param ln: *string* - the language of the page in which the box is displayed
@param show_similar_rec_p: *bool* print 'similar records' link in the box
@param creationdate: *string* - the creation date of the displayed record
@param modificationdate: *string* - the last modification date of the displayed record
@param show_short_rec_p: *boolean* - prints a very short version of the record as reminder.
@param citationnum: show (this) number of citations in the citations tab
@param referencenum: show (this) number of references in the references tab
@param discussionnum: show (this) number of comments/reviews in the discussion tab
"""
from invenio.legacy.search_engine import \
get_restricted_collections_for_recid, \
is_record_in_any_collection
# load the right message language
_ = gettext_set_language(ln)
# Prepare restriction flag
restriction_flag = ''
if get_restricted_collections_for_recid(recid, recreate_cache_if_needed=False):
restriction_flag = '<div class="restrictedflag"><span>%s</span></div>' % _("Restricted")
elif not is_record_in_any_collection(recid, recreate_cache_if_needed=False):
restriction_flag = '<div class="restrictedflag restrictedflag-pending"><span>%s</span></div>' % _("Restricted (Processing Record)")
# If no tabs, returns nothing (excepted if restricted)
if len(tabs) <= 1:
return restriction_flag
# Build the tabs at the top of the page
out_tabs = ''
if len(tabs) > 1:
first_tab = True
for (label, url, selected, enabled) in tabs:
addnum = ""
if (citationnum > -1) and url.count("/citation") == 1:
addnum = "(" + str(citationnum) + ")"
if (referencenum > -1) and url.count("/references") == 1:
addnum = "(" + str(referencenum) + ")"
if (discussionnum > -1) and url.count("/comments") == 1:
addnum = "(" + str(discussionnum) + ")"
css_class = []
if selected:
css_class.append('on')
if first_tab:
css_class.append('first')
first_tab = False
if not enabled:
css_class.append('disabled')
css_class = ' class="%s"' % ' '.join(css_class)
if not enabled:
out_tabs += '<li%(class)s><a>%(label)s %(addnum)s</a></li>' % \
{'class':css_class,
'label':label,
'addnum':addnum}
else:
out_tabs += '<li%(class)s><a href="%(url)s">%(label)s %(addnum)s </a></li>' % \
{'class':css_class,
'url':url,
'label':label,
'addnum':addnum}
if out_tabs != '':
out_tabs = ''' <div class="detailedrecordtabs">
<div>
<ul class="detailedrecordtabs">%s</ul>
<div id="tabsSpacer" style="clear:both;height:0px">&nbsp;</div></div>
</div>''' % out_tabs
# Add the clip icon and the brief record reminder if necessary
record_brief = ''
if show_short_rec_p:
record_brief = format_record(recID=recid, of='hs', ln=ln)
record_brief = '''<div id="detailedrecordshortreminder">
<div id="clip">&nbsp;</div>
<div id="HB">
%(record_brief)s
</div>
</div>
<div style="clear:both;height:1px">&nbsp;</div>
''' % {'record_brief': record_brief}
additional_scripts = ""
if include_jquery:
additional_scripts += """<script type="text/javascript" src="%s/js/jquery.min.js">' \
'</script>\n""" % (CFG_BASE_URL, )
if include_mathjax:
additional_scripts += get_mathjax_header()
# Print the content
out = """
%(additional_scripts)s<div class="detailedrecordbox">
%(tabs)s
<div class="detailedrecordboxcontent">
<div class="top-left-folded"></div>
<div class="top-right-folded"></div>
<div class="inside">
<!--<div style="height:0.1em;">&nbsp;</div>
<p class="notopgap">&nbsp;</p>-->
%(record_brief)s
""" % {'additional_scripts': additional_scripts,
'tabs':out_tabs,
'record_brief':record_brief}
out = restriction_flag + out
return out
def detailed_record_container_bottom(self, recid, tabs, ln=CFG_SITE_LANG,
show_similar_rec_p=True,
creationdate=None,
modificationdate=None, show_short_rec_p=True):
"""Prints the box displayed in detailed records pages, with tabs at the top.
Returns content as it is if the number of tabs for this record
is smaller than 2
Parameters:
- recid *int* - the id of the displayed record
- tabs ** - the tabs displayed at the top of the box.
- ln *string* - the language of the page in which the box is displayed
- show_similar_rec_p *bool* print 'similar records' link in the box
- creationdate *string* - the creation date of the displayed record
- modificationdate *string* - the last modification date of the displayed record
- show_short_rec_p *boolean* - prints a very short version of the record as reminder.
"""
# If no tabs, returns nothing
if len(tabs) <= 1:
return ''
# load the right message language
_ = gettext_set_language(ln)
similar = ""
if show_similar_rec_p and not CFG_INSPIRE_SITE:
similar = create_html_link(
websearch_templates.build_search_url(p='recid:%d' % \
recid,
rm='wrd',
ln=ln),
{}, _("Similar records"),{'class': "moreinfo"})
out = """
<div class="bottom-left-folded">%(dates)s</div>
<div class="bottom-right-folded" style="text-align:right;padding-bottom:2px;">
<span class="moreinfo" style="margin-right:10px;">%(similar)s</span></div>
</div>
</div>
</div>
<br/>
""" % {'similar' : similar,
'dates' : creationdate and '<div class="recordlastmodifiedbox" style="position:relative;margin-left:1px">&nbsp;%(dates)s</div>' % {
'dates': _("Record created %(x_date_creation)s, last modified %(x_date_modification)s",
x_date_creation=creationdate,
x_date_modification=modificationdate),
} or ''
}
return out
def detailed_record_mini_panel(self, recid, ln=CFG_SITE_LANG,
format='hd',
files='',
reviews='',
actions=''):
"""Displays the actions dock at the bottom of the detailed record
pages.
Parameters:
- recid *int* - the id of the displayed record
- ln *string* - interface language code
- format *string* - the format used to display the record
- files *string* - the small panel representing the attached files
- reviews *string* - the small panel representing the reviews
- actions *string* - the small panel representing the possible user's action
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<br />
<div class="detailedrecordminipanel">
<div class="top-left"></div><div class="top-right"></div>
<div class="inside">
<div id="detailedrecordminipanelfile" style="width:33%%;float:left;text-align:center;margin-top:0">
%(files)s
</div>
<div id="detailedrecordminipanelreview" style="width:30%%;float:left;text-align:center">
%(reviews)s
</div>
<div id="detailedrecordminipanelactions" style="width:36%%;float:right;text-align:right;">
%(actions)s
</div>
<div style="clear:both;margin-bottom: 0;"></div>
</div>
<div class="bottom-left"></div><div class="bottom-right"></div>
</div>
""" % {
'siteurl': CFG_BASE_URL,
'ln':ln,
'recid':recid,
'files': files,
'reviews':reviews,
'actions': actions,
}
return out
def tmpl_error_page(self, ln=CFG_SITE_LANG, status="", admin_was_alerted=True):
"""
Display an error page.
- status *string* - the HTTP status.
"""
_ = gettext_set_language(ln)
out = """
<p>%(message)s</p>
<p>%(alerted)s</p>
<p>%(doubts)s</p>""" % {
'status' : status,
'message' : _("The server encountered an error while dealing with your request."),
'alerted' : admin_was_alerted and _("The system administrators have been alerted.") or '',
'doubts' : _("In case of doubt, please contact %(x_admin_email)s.",
x_admin_email='<a href="mailto:%(admin)s">%(admin)s</a>' % {'admin' : CFG_SITE_SUPPORT_EMAIL})
}
return out
def tmpl_warning_message(self, ln, msg):
"""
Produces a warning message for the specified text
Parameters:
- 'ln' *string* - The language to display the interface in
- 'msg' *string* - The message to display
"""
# load the right message language
_ = gettext_set_language(ln)
return """<center><font color="red">%s</font></center>""" % msg
def tmpl_write_warning(self, msg, type='', prologue='', epilogue=''):
"""
Returns formatted warning message.
Parameters:
- 'msg' *string* - The message string
- 'type' *string* - the warning type
- 'prologue' *string* - HTML code to display before the warning
- 'epilogue' *string* - HTML code to display after the warning
"""
out = '\n%s<span class="quicknote">' % (prologue)
if type:
out += '%s: ' % type
out += '%s</span>%s' % (msg, epilogue)
return out
diff --git a/invenio/legacy/websubmit/doc/hacking/websubmit-file-stamper.webdoc b/invenio/legacy/websubmit/doc/hacking/websubmit-file-stamper.webdoc
index 6f6d38055..2ccb1ba3f 100644
--- a/invenio/legacy/websubmit/doc/hacking/websubmit-file-stamper.webdoc
+++ b/invenio/legacy/websubmit/doc/hacking/websubmit-file-stamper.webdoc
@@ -1,77 +1,80 @@
## -*- mode: html; coding: utf-8; -*-
## This file is part of Invenio.
## Copyright (C) 2010, 2011 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
<!-- WebDoc-Page-Title: Stamping fulltextes -->
<!-- WebDoc-Page-Navtrail: <a class="navtrail" href="<CFG_SITE_URL>/help/hacking">Hacking Invenio</a> &gt; <a class="navtrail" href="websubmit-internals">WebSubmit Internals</a> -->
<p>The WebSubmit File Stamper library (<tt>websubmit_file_stamper.py</tt>) let you stamps your PDFs.</p>
<h2>Python API</h2>
<protect><pre>
def stamp_file(options):
"""The driver for the stamping process. This is effectively the function
that is responsible for coordinating the stamping of a file.
@param options: (dictionary) - a dictionary of options that are required
by the function in order to carry out the stamping process.
The dictionary must have the following structure:
+ latex-template: (string) - the path to the LaTeX template to be
used for the creation of the stamp itself;
+ latex-template-var: (dictionary) - This dictionary contains
variables that should be sought in the LaTeX template file, and
the values that should be substituted in their place. E.g.:
{ "TITLE" : "An Introduction to Invenio" }
+ input-file: (string) - the path to the input file (i.e. that
which is to be stamped;
+ output-file: (string) - the name of the stamped file that should
be created by the program. This is optional - if not provided,
a default name will be applied to a file instead;
+ stamp: (string) - the type of stamp that is to be applied to the
input file. It must take one of 3 values:
- "first": Stamp only the first page of the document;
- "all": Apply the stamp to all pages of the document;
- "coverpage": Add a "cover page" to the document;
+ layer: (string) - the position of the stamp in the layers of the
file. Is one of the following values:
- "background": stamp applied to the background layer;
- "foreground": stamp applied to the foreground layer;
+ verbosity: (integer) - the verbosity level under which the program
is to run;
+ + skip-metadata: (boolean) - whether to skip copying the metadata
+ or not;
So, an example of the returned dictionary would be something like:
{ 'latex-template' : "demo-stamp-left.tex",
'latex-template-var' : { "REPORTNUMBER" : "TEST-2008-001",
"DATE" : "15/02/2008",
},
'input-file' : "test-doc.pdf",
'output-file' : "",
'stamp' : "first",
'layer' : "background",
'verbosity' : 0,
+ 'skip-metadata' : False,
}
@return: (tuple) - consisting of two strings:
1. the path to the working directory in which all stamping-related
files are stored;
2. The name of the "stamped" file;
@Exceptions raised: (InvenioWebSubmitFileStamperError) exceptions may
be raised or propagated by this function when the stamping process
fails for one reason or another.
"""
</pre></protect>
<p>See <a href="http://invenio-software.org/code-browser/invenio.websubmit_file_stamper-module.html">websubmit_file_stamper API</a> for a complete API description.</p>
diff --git a/invenio/legacy/websubmit/engine.py b/invenio/legacy/websubmit/engine.py
index ac2ad0c84..791a228df 100644
--- a/invenio/legacy/websubmit/engine.py
+++ b/invenio/legacy/websubmit/engine.py
@@ -1,1861 +1,1858 @@
## This file is part of Invenio.
## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""WebSubmit: the mechanism for the submission of new records into Invenio
via a Web interface.
"""
__revision__ = "$Id$"
## import interesting modules:
import traceback
import string
import os
import sys
import time
import types
import re
import pprint
from urllib import quote_plus
from cgi import escape
from invenio.config import \
CFG_SITE_LANG, \
CFG_SITE_NAME, \
CFG_SITE_URL, \
CFG_WEBSUBMIT_STORAGEDIR, \
CFG_DEVEL_SITE, \
CFG_SITE_SECURE_URL, \
CFG_WEBSUBMIT_USE_MATHJAX
from invenio.legacy.dbquery import Error
from invenio.modules.access.engine import acc_authorize_action
from invenio.legacy.webpage import page, error_page, warning_page
from invenio.legacy.webuser import getUid, get_email, collect_user_info, isGuestUser, \
page_not_authorized
from invenio.legacy.websubmit.config import CFG_RESERVED_SUBMISSION_FILENAMES, \
InvenioWebSubmitFunctionError, InvenioWebSubmitFunctionStop, \
InvenioWebSubmitFunctionWarning
from invenio.base.i18n import gettext_set_language, wash_language
from invenio.legacy.webstat.api import register_customevent
from invenio.ext.logging import register_exception
from invenio.utils.url import make_canonical_urlargd, redirect_to_url
from invenio.legacy.websubmit.admin_engine import string_is_alphanumeric_including_underscore
from invenio.utils.html import get_mathjax_header
from invenio.legacy.websubmit.db_layer import \
get_storage_directory_of_action, \
get_longname_of_doctype, \
get_longname_of_action, \
get_num_pages_of_submission, \
get_parameter_value_for_doctype, \
submission_exists_in_log, \
log_new_pending_submission, \
log_new_completed_submission, \
update_submission_modified_date_in_log, \
update_submission_reference_in_log, \
update_submission_reference_and_status_in_log, \
get_form_fields_on_submission_page, \
get_element_description, \
get_element_check_description, \
get_form_fields_not_on_submission_page, \
function_step_is_last, \
get_collection_children_of_submission_collection, \
get_submission_collection_name, \
get_doctype_children_of_submission_collection, \
get_categories_of_doctype, \
get_doctype_details, \
get_actions_on_submission_page_for_doctype, \
get_action_details, \
get_parameters_of_function, \
get_details_of_submission, \
get_functions_for_submission_step, \
get_submissions_at_level_X_with_score_above_N, \
submission_is_finished
import invenio.legacy.template
websubmit_templates = invenio.legacy.template.load('websubmit')
def interface(req,
c=CFG_SITE_NAME,
ln=CFG_SITE_LANG,
doctype="",
act="",
startPg=1,
access="",
mainmenu="",
fromdir="",
nextPg="",
nbPg="",
curpage=1):
"""This function is called after a user has visited a document type's
"homepage" and selected the type of "action" to perform. Having
clicked an action-button (e.g. "Submit a New Record"), this function
will be called . It performs the task of initialising a new submission
session (retrieving information about the submission, creating a
working submission-directory, etc), and "drawing" a submission page
containing the WebSubmit form that the user uses to input the metadata
to be submitted.
When a user moves between pages in the submission interface, this
function is recalled so that it can save the metadata entered into the
previous page by the user, and draw the current submission-page.
Note: During a submission, for each page refresh, this function will be
called while the variable "step" (a form variable, seen by
websubmit_webinterface, which calls this function) is 0 (ZERO).
In other words, this function handles the FRONT-END phase of a
submission, BEFORE the WebSubmit functions are called.
@param req: (apache request object) *** NOTE: Added into this object, is
a variable called "form" (req.form). This is added into the object in
the index function of websubmit_webinterface. It contains a
"mod_python.util.FieldStorage" instance, that contains the form-fields
found on the previous submission page.
@param c: (string), defaulted to CFG_SITE_NAME. The name of the Invenio
installation.
@param ln: (string), defaulted to CFG_SITE_LANG. The language in which to
display the pages.
@param doctype: (string) - the doctype ID of the doctype for which the
submission is being made.
@param act: (string) - The ID of the action being performed (e.g.
submission of bibliographic information; modification of bibliographic
information, etc).
@param startPg: (integer) - Starting page for the submission? Defaults
to 1.
@param indir: (string) - the directory used to store all submissions
of the given "type" of this submission. For example, if the submission
is of the type "modify bibliographic information", this variable would
contain "modify".
@param access: (string) - the "access" number for the submission
(e.g. 1174062451_7010). This number is also used as the name for the
current working submission directory.
@param mainmenu: (string) - contains the URL (minus the Invenio
home stem) for the submission's home-page. (E.g. If this submission
is "PICT", the "mainmenu" file would contain "/submit?doctype=PICT".
@param fromdir: (integer)
@param nextPg: (string)
@param nbPg: (string)
@param curpage: (integer) - the current submission page number. Defaults
to 1.
"""
ln = wash_language(ln)
# load the right message language
_ = gettext_set_language(ln)
sys.stdout = req
# get user ID:
user_info = collect_user_info(req)
uid = user_info['uid']
uid_email = user_info['email']
# variable initialisation
t = ""
field = []
fieldhtml = []
level = []
fullDesc = []
text = ''
check = []
select = []
radio = []
upload = []
txt = []
noPage = []
# Preliminary tasks
if not access:
# In some cases we want to take the users directly to the submit-form.
# This fix makes this possible - as it generates the required access
# parameter if it is not present.
pid = os.getpid()
now = time.time()
access = "%i_%s" % (now, pid)
# check we have minimum fields
if not doctype or not act or not access:
## We don't have all the necessary information to go ahead
## with this submission:
return warning_page(_("Not enough information to go ahead with the submission."), req, ln)
try:
assert(not access or re.match('\d+_\d+', access))
except AssertionError:
register_exception(req=req, prefix='doctype="%s", access="%s"' % (doctype, access))
return warning_page(_("Invalid parameters"), req, ln)
if doctype and act:
## Let's clean the input
details = get_details_of_submission(doctype, act)
if not details:
return warning_page(_("Invalid doctype and act parameters"), req, ln)
doctype = details[0]
act = details[1]
## Before continuing to display the submission form interface,
## verify that this submission has not already been completed:
if submission_is_finished(doctype, act, access, uid_email):
## This submission has already been completed.
## This situation can arise when, having completed a submission,
## the user uses the browser's back-button to go back to the form
## stage of the submission and then tries to submit once more.
## This is unsafe and should not be allowed. Instead of re-displaying
## the submission forms, display an error message to the user:
wrnmsg = """<b>This submission has been completed. Please go to the""" \
""" <a href="/submit?doctype=%(doctype)s&amp;ln=%(ln)s">""" \
"""main menu</a> to start a new submission.</b>""" \
% { 'doctype' : quote_plus(doctype), 'ln' : ln }
return warning_page(wrnmsg, req, ln)
## retrieve the action and doctype data:
## Concatenate action ID and doctype ID to make the submission ID:
subname = "%s%s" % (act, doctype)
## Get the submission storage directory from the DB:
submission_dir = get_storage_directory_of_action(act)
if submission_dir:
indir = submission_dir
else:
## Unable to determine the submission-directory:
return warning_page(_("Unable to find the submission directory for the action: %(x_dir)s", x_dir=escape(str(act))), req, ln)
## get the document type's long-name:
doctype_lname = get_longname_of_doctype(doctype)
if doctype_lname is not None:
## Got the doctype long-name: replace spaces with HTML chars:
docname = doctype_lname.replace(" ", "&nbsp;")
else:
## Unknown document type:
return warning_page(_("Unknown document type"), req, ln)
## get the action's long-name:
actname = get_longname_of_action(act)
if actname is None:
## Unknown action:
return warning_page(_("Unknown action"), req, ln)
## Get the number of pages for this submission:
num_submission_pages = get_num_pages_of_submission(subname)
if num_submission_pages is not None:
nbpages = num_submission_pages
else:
## Unable to determine the number of pages for this submission:
return warning_page(_("Unable to determine the number of submission pages."), req, ln)
## If unknown, get the current page of submission:
if startPg != "" and curpage in ("", 0):
curpage = startPg
## retrieve the name of the file in which the reference of
## the submitted document will be stored
rn_filename = get_parameter_value_for_doctype(doctype, "edsrn")
if rn_filename is not None:
edsrn = rn_filename
else:
## Unknown value for edsrn - set it to an empty string:
edsrn = ""
## This defines the path to the directory containing the action data
curdir = os.path.join(CFG_WEBSUBMIT_STORAGEDIR, indir, doctype, access)
try:
assert(curdir == os.path.abspath(curdir))
except AssertionError:
register_exception(req=req, prefix='indir="%s", doctype="%s", access="%s"' % (indir, doctype, access))
return warning_page(_("Invalid parameters"), req, ln)
## if this submission comes from another one (fromdir is then set)
## We retrieve the previous submission directory and put it in the proper one
if fromdir != "":
olddir = os.path.join(CFG_WEBSUBMIT_STORAGEDIR, fromdir, doctype, access)
try:
assert(olddir == os.path.abspath(olddir))
except AssertionError:
register_exception(req=req, prefix='fromdir="%s", doctype="%s", access="%s"' % (fromdir, doctype, access))
return warning_page(_("Invalid parameters"), req, ln)
if os.path.exists(olddir):
os.rename(olddir, curdir)
## If the submission directory still does not exist, we create it
if not os.path.exists(curdir):
try:
os.makedirs(curdir)
except Exception as e:
register_exception(req=req, alert_admin=True)
return warning_page(_("Unable to create a directory for this submission. The administrator has been alerted."), req, ln)
## Retrieve the previous page, as submitted to curdir (before we
## overwrite it with our curpage as declared from the incoming
## form)
try:
fp = open(os.path.join(curdir, "curpage"))
previous_page_from_disk = fp.read()
fp.close()
except:
previous_page_from_disk = "1"
# retrieve the original main menu url and save it in the "mainmenu" file
if mainmenu != "":
fp = open(os.path.join(curdir, "mainmenu"), "w")
fp.write(mainmenu)
fp.close()
# and if the file containing the URL to the main menu exists
# we retrieve it and store it in the $mainmenu variable
if os.path.exists(os.path.join(curdir, "mainmenu")):
fp = open(os.path.join(curdir, "mainmenu"), "r");
mainmenu = fp.read()
fp.close()
else:
mainmenu = "%s/submit" % (CFG_SITE_URL,)
# various authentication related tasks...
if uid_email != "guest" and uid_email != "":
#First save the username (email address) in the SuE file. This way bibconvert will be able to use it if needed
fp = open(os.path.join(curdir, "SuE"), "w")
fp.write(uid_email)
fp.close()
if os.path.exists(os.path.join(curdir, "combo%s" % doctype)):
fp = open(os.path.join(curdir, "combo%s" % doctype), "r");
categ = fp.read()
fp.close()
else:
categ = req.form.get('combo%s' % doctype, '*')
# is user authorized to perform this action?
(auth_code, auth_message) = acc_authorize_action(req, 'submit', \
authorized_if_no_roles=not isGuestUser(uid), \
verbose=0, \
doctype=doctype, \
act=act, \
categ=categ)
if not auth_code == 0:
return warning_page("""<center><font color="red">%s</font></center>""" % auth_message, req, ln)
## update the "journal of submission":
## Does the submission already exist in the log?
submission_exists = \
submission_exists_in_log(doctype, act, access, uid_email)
if submission_exists == 1:
## update the modification-date of this submission in the log:
update_submission_modified_date_in_log(doctype, act, access, uid_email)
else:
## Submission doesn't exist in log - create it:
log_new_pending_submission(doctype, act, access, uid_email)
## Let's write in curdir file under curdir the curdir value
## in case e.g. it is needed in FFT.
fp = open(os.path.join(curdir, "curdir"), "w")
fp.write(curdir)
fp.close()
## Let's write in ln file the current language
fp = open(os.path.join(curdir, "ln"), "w")
fp.write(ln)
fp.close()
# Save the form fields entered in the previous submission page
# If the form was sent with the GET method
form = dict(req.form)
value = ""
# we parse all the form variables
for key, formfields in form.items():
filename = key.replace("[]", "")
file_to_open = os.path.join(curdir, filename)
try:
assert(file_to_open == os.path.abspath(file_to_open))
except AssertionError:
register_exception(req=req, prefix='curdir="%s", filename="%s"' % (curdir, filename))
return warning_page(_("Invalid parameters"), req, ln)
# Do not write reserved filenames to disk
if filename in CFG_RESERVED_SUBMISSION_FILENAMES:
# Unless there is really an element with that name on this
# page or previous one (either visited, or declared to be
# visited), which means that admin authorized it.
if not ((str(curpage).isdigit() and \
filename in [submission_field[3] for submission_field in \
get_form_fields_on_submission_page(subname, curpage)]) or \
(str(curpage).isdigit() and int(curpage) > 1 and \
filename in [submission_field[3] for submission_field in \
get_form_fields_on_submission_page(subname, int(curpage) - 1)]) or \
(previous_page_from_disk.isdigit() and \
filename in [submission_field[3] for submission_field in \
get_form_fields_on_submission_page(subname, int(previous_page_from_disk))])):
# Still this will filter out reserved field names that
# might have been called by functions such as
# Create_Modify_Interface function in MBI step, or
# dynamic fields in response elements, but that is
# unlikely to be a problem.
continue
# Skip variables containing characters that are not allowed in
# WebSubmit elements
if not string_is_alphanumeric_including_underscore(filename):
continue
# the field is an array
if isinstance(formfields, types.ListType):
fp = open(file_to_open, "w")
for formfield in formfields:
#stripslashes(value)
value = specialchars(formfield)
fp.write(value+"\n")
fp.close()
# the field is a normal string
elif isinstance(formfields, types.StringTypes) and formfields != "":
value = formfields
fp = open(file_to_open, "w")
fp.write(specialchars(value))
fp.close()
# the field is a file
elif hasattr(formfields,"filename") and formfields.filename:
dir_to_open = os.path.join(curdir, 'files', key)
try:
assert(dir_to_open == os.path.abspath(dir_to_open))
assert(dir_to_open.startswith(CFG_WEBSUBMIT_STORAGEDIR))
except AssertionError:
register_exception(req=req, prefix='curdir="%s", key="%s"' % (curdir, key))
return warning_page(_("Invalid parameters"), req, ln)
if not os.path.exists(dir_to_open):
try:
os.makedirs(dir_to_open)
except:
register_exception(req=req, alert_admin=True)
return warning_page(_("Cannot create submission directory. The administrator has been alerted."), req, ln)
filename = formfields.filename
## Before saving the file to disc, wash the filename (in particular
## washing away UNIX and Windows (e.g. DFS) paths):
filename = os.path.basename(filename.split('\\')[-1])
filename = filename.strip()
if filename != "":
fp = open(os.path.join(dir_to_open, filename), "w")
while True:
buf = formfields.read(10240)
if buf:
fp.write(buf)
else:
break
fp.close()
fp = open(os.path.join(curdir, "lastuploadedfile"), "w")
fp.write(filename)
fp.close()
fp = open(file_to_open, "w")
fp.write(filename)
fp.close()
else:
return warning_page(_("No file uploaded?"), req, ln)
## if the found field is the reference of the document,
## save this value in the "journal of submissions":
if uid_email != "" and uid_email != "guest":
if key == edsrn:
update_submission_reference_in_log(doctype, access, uid_email, value)
## create the interface:
subname = "%s%s" % (act, doctype)
## Get all of the form fields that appear on this page, ordered by fieldnum:
form_fields = get_form_fields_on_submission_page(subname, curpage)
full_fields = []
values = []
the_globals = {
'doctype' : doctype,
'action' : action,
'access' : access,
'ln' : ln,
'curdir' : curdir,
'uid' : uid,
'uid_email' : uid_email,
'form' : form,
'act' : act,
'action' : act, ## for backward compatibility
'req' : req,
'user_info' : user_info,
'InvenioWebSubmitFunctionError' : InvenioWebSubmitFunctionError,
'__websubmit_in_jail__' : True,
'__builtins__' : globals()['__builtins__']
}
for field_instance in form_fields:
full_field = {}
## Retrieve the field's description:
element_descr = get_element_description(field_instance[3])
try:
assert(element_descr is not None)
except AssertionError:
msg = _("Unknown form field found on submission page.")
register_exception(req=req, alert_admin=True, prefix=msg)
## The form field doesn't seem to exist - return with error message:
return warning_page(_("Unknown form field found on submission page."), req, ln)
if element_descr[8] is None:
val = ""
else:
val = element_descr[8]
## we also retrieve and add the javascript code of the checking function, if needed
## Set it to empty string to begin with:
full_field['javascript'] = ''
if field_instance[7] != '':
check_descr = get_element_check_description(field_instance[7])
if check_descr is not None:
## Retrieved the check description:
full_field['javascript'] = check_descr
full_field['type'] = element_descr[3]
full_field['name'] = field_instance[3]
full_field['rows'] = element_descr[5]
full_field['cols'] = element_descr[6]
full_field['val'] = val
full_field['size'] = element_descr[4]
full_field['maxlength'] = element_descr[7]
full_field['htmlcode'] = element_descr[9]
full_field['typename'] = field_instance[1] ## TODO: Investigate this, Not used?
## It also seems to refer to pagenum.
# The 'R' fields must be executed in the engine's environment,
# as the runtime functions access some global and local
# variables.
if full_field ['type'] == 'R':
try:
co = compile (full_field ['htmlcode'].replace("\r\n","\n"), "<string>", "exec")
the_globals['text'] = ''
exec co in the_globals
text = the_globals['text']
except:
register_exception(req=req, alert_admin=True, prefix="Error in evaluating response element %s with globals %s" % (pprint.pformat(full_field), pprint.pformat(the_globals)))
raise
else:
text = websubmit_templates.tmpl_submit_field (ln = ln, field = full_field)
# we now determine the exact type of the created field
if full_field['type'] not in [ 'D','R']:
field.append(full_field['name'])
level.append(field_instance[5])
fullDesc.append(field_instance[4])
txt.append(field_instance[6])
check.append(field_instance[7])
# If the field is not user-defined, we try to determine its type
# (select, radio, file upload...)
# check whether it is a select field or not
if re.search("SELECT", text, re.IGNORECASE) is not None:
select.append(1)
else:
select.append(0)
# checks whether it is a radio field or not
if re.search(r"TYPE=[\"']?radio", text, re.IGNORECASE) is not None:
radio.append(1)
else:
radio.append(0)
# checks whether it is a file upload or not
if re.search(r"TYPE=[\"']?file", text, re.IGNORECASE) is not None:
upload.append(1)
else:
upload.append(0)
# if the field description contains the "<COMBO>" string, replace
# it by the category selected on the document page submission page
combofile = "combo%s" % doctype
if os.path.exists("%s/%s" % (curdir, combofile)):
f = open("%s/%s" % (curdir, combofile), "r")
combo = f.read()
f.close()
else:
combo = ""
text = text.replace("<COMBO>", combo)
# if there is a <YYYY> tag in it, replace it by the current year
year = time.strftime("%Y");
text = text.replace("<YYYY>", year)
# if there is a <TODAY> tag in it, replace it by the current year
today = time.strftime("%d/%m/%Y");
text = text.replace("<TODAY>", today)
fieldhtml.append(text)
else:
select.append(0)
radio.append(0)
upload.append(0)
# field.append(value) - initial version, not working with JS, taking a submitted value
field.append(field_instance[3])
level.append(field_instance[5])
txt.append(field_instance[6])
fullDesc.append(field_instance[4])
check.append(field_instance[7])
fieldhtml.append(text)
full_field['fullDesc'] = field_instance[4]
full_field['text'] = text
# If a file exists with the name of the field we extract the saved value
text = ''
if os.path.exists(os.path.join(curdir, full_field['name'])):
file = open(os.path.join(curdir, full_field['name']), "r");
text = file.read()
- text = re.compile("[\n\r]*$").sub("", text)
- text = re.compile("\n").sub("\\n", text)
- text = re.compile("\r").sub("", text)
file.close()
values.append(text)
full_fields.append(full_field)
returnto = {}
if int(curpage) == int(nbpages):
subname = "%s%s" % (act, doctype)
other_form_fields = \
get_form_fields_not_on_submission_page(subname, curpage)
nbFields = 0
message = ""
fullcheck_select = []
fullcheck_radio = []
fullcheck_upload = []
fullcheck_field = []
fullcheck_level = []
fullcheck_txt = []
fullcheck_noPage = []
fullcheck_check = []
for field_instance in other_form_fields:
if field_instance[5] == "M":
## If this field is mandatory, get its description:
element_descr = get_element_description(field_instance[3])
try:
assert(element_descr is not None)
except AssertionError:
msg = _("Unknown form field found on submission page.")
register_exception(req=req, alert_admin=True, prefix=msg)
## The form field doesn't seem to exist - return with error message:
return warning_page(_("Unknown form field found on submission page."), req, ln)
if element_descr[3] in ['D', 'R']:
if element_descr[3] == "D":
text = element_descr[9]
else:
text = eval(element_descr[9])
formfields = text.split(">")
for formfield in formfields:
match = re.match("name=([^ <>]+)", formfield, re.IGNORECASE)
if match is not None:
names = match.groups
for value in names:
if value != "":
value = re.compile("[\"']+").sub("", value)
fullcheck_field.append(value)
fullcheck_level.append(field_instance[5])
fullcheck_txt.append(field_instance[6])
fullcheck_noPage.append(field_instance[1])
fullcheck_check.append(field_instance[7])
nbFields = nbFields + 1
else:
fullcheck_noPage.append(field_instance[1])
fullcheck_field.append(field_instance[3])
fullcheck_level.append(field_instance[5])
fullcheck_txt.append(field_instance[6])
fullcheck_check.append(field_instance[7])
nbFields = nbFields+1
# tests each mandatory field
fld = 0
res = 1
for i in xrange(nbFields):
res = 1
if not os.path.exists(os.path.join(curdir, fullcheck_field[i])):
res = 0
else:
file = open(os.path.join(curdir, fullcheck_field[i]), "r")
text = file.read()
if text == '':
res = 0
else:
if text == "Select:":
res = 0
if res == 0:
fld = i
break
if not res:
returnto = {
'field' : fullcheck_txt[fld],
'page' : fullcheck_noPage[fld],
}
t += websubmit_templates.tmpl_page_interface(
ln = ln,
docname = docname,
actname = actname,
curpage = curpage,
nbpages = nbpages,
nextPg = nextPg,
access = access,
nbPg = nbPg,
doctype = doctype,
act = act,
fields = full_fields,
javascript = websubmit_templates.tmpl_page_interface_js(
ln = ln,
upload = upload,
field = field,
fieldhtml = fieldhtml,
txt = txt,
check = check,
level = level,
curdir = curdir,
values = values,
select = select,
radio = radio,
curpage = curpage,
nbpages = nbpages,
returnto = returnto,
),
mainmenu = mainmenu,
)
t += websubmit_templates.tmpl_page_do_not_leave_submission_js(ln)
# start display:
req.content_type = "text/html"
req.send_http_header()
p_navtrail = """<a href="/submit?ln=%(ln)s" class="navtrail">%(submit)s</a>&nbsp;>&nbsp;<a href="/submit?doctype=%(doctype)s&amp;ln=%(ln)s" class="navtrail">%(docname)s</a>&nbsp;""" % {
'submit' : _("Submit"),
'doctype' : quote_plus(doctype),
'docname' : docname,
'ln' : ln
}
## add MathJax if wanted
if CFG_WEBSUBMIT_USE_MATHJAX:
metaheaderadd = get_mathjax_header(req.is_https())
metaheaderadd += websubmit_templates.tmpl_mathpreview_header(ln, req.is_https())
else:
metaheaderadd = ''
return page(title= actname,
body = t,
navtrail = p_navtrail,
description = "submit documents",
keywords = "submit",
uid = uid,
language = ln,
req = req,
navmenuid='submit',
metaheaderadd=metaheaderadd)
def endaction(req,
c=CFG_SITE_NAME,
ln=CFG_SITE_LANG,
doctype="",
act="",
startPg=1,
access="",
mainmenu="",
fromdir="",
nextPg="",
nbPg="",
curpage=1,
step=1,
mode="U"):
"""Having filled-in the WebSubmit form created for metadata by the interface
function, the user clicks a button to either "finish the submission" or
to "proceed" to the next stage of the submission. At this point, a
variable called "step" will be given a value of 1 or above, which means
that this function is called by websubmit_webinterface.
So, during all non-zero steps of the submission, this function is called.
In other words, this function is called during the BACK-END phase of a
submission, in which WebSubmit *functions* are being called.
The function first ensures that all of the WebSubmit form field values
have been saved in the current working submission directory, in text-
files with the same name as the field elements have. It then determines
the functions to be called for the given step of the submission, and
executes them.
Following this, if this is the last step of the submission, it logs the
submission as "finished" in the journal of submissions.
@param req: (apache request object) *** NOTE: Added into this object, is
a variable called "form" (req.form). This is added into the object in
the index function of websubmit_webinterface. It contains a
"mod_python.util.FieldStorage" instance, that contains the form-fields
found on the previous submission page.
@param c: (string), defaulted to CFG_SITE_NAME. The name of the Invenio
installation.
@param ln: (string), defaulted to CFG_SITE_LANG. The language in which to
display the pages.
@param doctype: (string) - the doctype ID of the doctype for which the
submission is being made.
@param act: (string) - The ID of the action being performed (e.g.
submission of bibliographic information; modification of bibliographic
information, etc).
@param startPg: (integer) - Starting page for the submission? Defaults
to 1.
@param indir: (string) - the directory used to store all submissions
of the given "type" of this submission. For example, if the submission
is of the type "modify bibliographic information", this variable would
contain "modify".
@param access: (string) - the "access" number for the submission
(e.g. 1174062451_7010). This number is also used as the name for the
current working submission directory.
@param mainmenu: (string) - contains the URL (minus the Invenio
home stem) for the submission's home-page. (E.g. If this submission
is "PICT", the "mainmenu" file would contain "/submit?doctype=PICT".
@param fromdir:
@param nextPg:
@param nbPg:
@param curpage: (integer) - the current submission page number. Defaults
to 1.
@param step: (integer) - the current step of the submission. Defaults to
1.
@param mode:
"""
# load the right message language
_ = gettext_set_language(ln)
dismode = mode
ln = wash_language(ln)
sys.stdout = req
rn = ""
t = ""
# get user ID:
uid = getUid(req)
uid_email = get_email(uid)
## Get the submission storage directory from the DB:
submission_dir = get_storage_directory_of_action(act)
if submission_dir:
indir = submission_dir
else:
## Unable to determine the submission-directory:
return warning_page(_("Unable to find the submission directory for the action: %(x_dir)s", x_dir=escape(str(act))), req, ln)
curdir = os.path.join(CFG_WEBSUBMIT_STORAGEDIR, indir, doctype, access)
if os.path.exists(os.path.join(curdir, "combo%s" % doctype)):
fp = open(os.path.join(curdir, "combo%s" % doctype), "r");
categ = fp.read()
fp.close()
else:
categ = req.form.get('combo%s' % doctype, '*')
# is user authorized to perform this action?
(auth_code, auth_message) = acc_authorize_action(req, 'submit', \
authorized_if_no_roles=not isGuestUser(uid), \
verbose=0, \
doctype=doctype, \
act=act, \
categ=categ)
if not auth_code == 0:
return warning_page("""<center><font color="red">%s</font></center>""" % auth_message, req, ln)
# Preliminary tasks
## check we have minimum fields
if not doctype or not act or not access:
## We don't have all the necessary information to go ahead
## with this submission:
return warning_page(_("Not enough information to go ahead with the submission."), req, ln)
if doctype and act:
## Let's clean the input
details = get_details_of_submission(doctype, act)
if not details:
return warning_page(_("Invalid doctype and act parameters"), req, ln)
doctype = details[0]
act = details[1]
try:
assert(not access or re.match('\d+_\d+', access))
except AssertionError:
register_exception(req=req, prefix='doctype="%s", access="%s"' % (doctype, access))
return warning_page(_("Invalid parameters"), req, ln)
## Before continuing to process the submitted data, verify that
## this submission has not already been completed:
if submission_is_finished(doctype, act, access, uid_email):
## This submission has already been completed.
## This situation can arise when, having completed a submission,
## the user uses the browser's back-button to go back to the form
## stage of the submission and then tries to submit once more.
## This is unsafe and should not be allowed. Instead of re-processing
## the submitted data, display an error message to the user:
wrnmsg = """<b>This submission has been completed. Please go to the""" \
""" <a href="/submit?doctype=%(doctype)s&amp;ln=%(ln)s">""" \
"""main menu</a> to start a new submission.</b>""" \
% { 'doctype' : quote_plus(doctype), 'ln' : ln }
return warning_page(wrnmsg, req, ln)
## Get the number of pages for this submission:
subname = "%s%s" % (act, doctype)
## retrieve the action and doctype data
## Get the submission storage directory from the DB:
submission_dir = get_storage_directory_of_action(act)
if submission_dir:
indir = submission_dir
else:
## Unable to determine the submission-directory:
return warning_page(_("Unable to find the submission directory for the action: %(x_dir)s", x_dir=escape(str(act))), req, ln)
# The following words are reserved and should not be used as field names
reserved_words = ["stop", "file", "nextPg", "startPg", "access", "curpage", "nbPg", "act", \
"indir", "doctype", "mode", "step", "deleted", "file_path", "userfile_name"]
# This defines the path to the directory containing the action data
curdir = os.path.join(CFG_WEBSUBMIT_STORAGEDIR, indir, doctype, access)
try:
assert(curdir == os.path.abspath(curdir))
except AssertionError:
register_exception(req=req, prefix='indir="%s", doctype=%s, access=%s' % (indir, doctype, access))
return warning_page(_("Invalid parameters"), req, ln)
## If the submission directory still does not exist, we create it
if not os.path.exists(curdir):
try:
os.makedirs(curdir)
except Exception as e:
register_exception(req=req, alert_admin=True)
return warning_page(_("Unable to create a directory for this submission. The administrator has been alerted."), req, ln)
# retrieve the original main menu url ans save it in the "mainmenu" file
if mainmenu != "":
fp = open(os.path.join(curdir, "mainmenu"), "w")
fp.write(mainmenu)
fp.close()
# and if the file containing the URL to the main menu exists
# we retrieve it and store it in the $mainmenu variable
if os.path.exists(os.path.join(curdir, "mainmenu")):
fp = open(os.path.join(curdir, "mainmenu"), "r");
mainmenu = fp.read()
fp.close()
else:
mainmenu = "%s/submit" % (CFG_SITE_URL,)
num_submission_pages = get_num_pages_of_submission(subname)
if num_submission_pages is not None:
nbpages = num_submission_pages
else:
## Unable to determine the number of pages for this submission:
return warning_page(_("Unable to determine the number of submission pages."), \
req, ln)
## Retrieve the previous page, as submitted to curdir (before we
## overwrite it with our curpage as declared from the incoming
## form)
try:
fp = open(os.path.join(curdir, "curpage"))
previous_page_from_disk = fp.read()
fp.close()
except:
previous_page_from_disk = str(num_submission_pages)
## retrieve the name of the file in which the reference of
## the submitted document will be stored
rn_filename = get_parameter_value_for_doctype(doctype, "edsrn")
if rn_filename is not None:
edsrn = rn_filename
else:
## Unknown value for edsrn - set it to an empty string:
edsrn = ""
## Determine whether the action is finished
## (ie there are no other steps after the current one):
finished = function_step_is_last(doctype, act, step)
## Let's write in curdir file under curdir the curdir value
## in case e.g. it is needed in FFT.
fp = open(os.path.join(curdir, "curdir"), "w")
fp.write(curdir)
fp.close()
## Let's write in ln file the current language
fp = open(os.path.join(curdir, "ln"), "w")
fp.write(ln)
fp.close()
# Save the form fields entered in the previous submission page
# If the form was sent with the GET method
form = req.form
value = ""
# we parse all the form variables
for key in form.keys():
formfields = form[key]
filename = key.replace("[]", "")
file_to_open = os.path.join(curdir, filename)
try:
assert(file_to_open == os.path.abspath(file_to_open))
assert(file_to_open.startswith(CFG_WEBSUBMIT_STORAGEDIR))
except AssertionError:
register_exception(req=req, prefix='curdir="%s", filename="%s"' % (curdir, filename))
return warning_page(_("Invalid parameters"), req, ln)
# Do not write reserved filenames to disk
if filename in CFG_RESERVED_SUBMISSION_FILENAMES:
# Unless there is really an element with that name on this
# page, or on the previously visited one, which means that
# admin authorized it. Note that in endaction() curpage is
# equivalent to the "previous" page value
if not ((previous_page_from_disk.isdigit() and \
filename in [submission_field[3] for submission_field in \
get_form_fields_on_submission_page(subname, int(previous_page_from_disk))]) or \
(str(curpage).isdigit() and int(curpage) > 1 and \
filename in [submission_field[3] for submission_field in \
get_form_fields_on_submission_page(subname, int(curpage) - 1)])):
# might have been called by functions such as
# Create_Modify_Interface function in MBI step, or
# dynamic fields in response elements, but that is
# unlikely to be a problem.
continue
# Skip variables containing characters that are not allowed in
# WebSubmit elements
if not string_is_alphanumeric_including_underscore(filename):
continue
# the field is an array
if isinstance(formfields, types.ListType):
fp = open(file_to_open, "w")
for formfield in formfields:
#stripslashes(value)
value = specialchars(formfield)
fp.write(value+"\n")
fp.close()
# the field is a normal string
elif isinstance(formfields, types.StringTypes) and formfields != "":
value = formfields
fp = open(file_to_open, "w")
fp.write(specialchars(value))
fp.close()
# the field is a file
elif hasattr(formfields, "filename") and formfields.filename:
dir_to_open = os.path.join(curdir, 'files', key)
try:
assert(dir_to_open == os.path.abspath(dir_to_open))
assert(dir_to_open.startswith(CFG_WEBSUBMIT_STORAGEDIR))
except AssertionError:
register_exception(req=req, prefix='curdir="%s", key="%s"' % (curdir, key))
return warning_page(_("Invalid parameters"), req, ln)
if not os.path.exists(dir_to_open):
try:
os.makedirs(dir_to_open)
except:
register_exception(req=req, alert_admin=True)
return warning_page(_("Cannot create submission directory. The administrator has been alerted."), req, ln)
filename = formfields.filename
## Before saving the file to disc, wash the filename (in particular
## washing away UNIX and Windows (e.g. DFS) paths):
filename = os.path.basename(filename.split('\\')[-1])
filename = filename.strip()
if filename != "":
fp = open(os.path.join(dir_to_open, filename), "w")
while True:
buf = formfields.file.read(10240)
if buf:
fp.write(buf)
else:
break
fp.close()
fp = open(os.path.join(curdir, "lastuploadedfile"), "w")
fp.write(filename)
fp.close()
fp = open(file_to_open, "w")
fp.write(filename)
fp.close()
else:
return warning_page(_("No file uploaded?"), req, ln)
## if the found field is the reference of the document
## we save this value in the "journal of submissions"
if uid_email != "" and uid_email != "guest":
if key == edsrn:
update_submission_reference_in_log(doctype, access, uid_email, value)
## get the document type's long-name:
doctype_lname = get_longname_of_doctype(doctype)
if doctype_lname is not None:
## Got the doctype long-name: replace spaces with HTML chars:
docname = doctype_lname.replace(" ", "&nbsp;")
else:
## Unknown document type:
return warning_page(_("Unknown document type"), req, ln)
## get the action's long-name:
actname = get_longname_of_action(act)
if actname is None:
## Unknown action:
return warning_page(_("Unknown action"), req, ln)
## Determine whether the action is finished
## (ie there are no other steps after the current one):
last_step = function_step_is_last(doctype, act, step)
next_action = '' ## The next action to be proposed to the user
# Prints the action details, returning the mandatory score
action_score = action_details(doctype, act)
current_level = get_level(doctype, act)
# Calls all the function's actions
function_content = ''
try:
## Handle the execution of the functions for this
## submission/step:
start_time = time.time()
(function_content, last_step, action_score, rn) = \
print_function_calls(req=req,
doctype=doctype,
action=act,
step=step,
form=form,
start_time=start_time,
access=access,
curdir=curdir,
dismode=mode,
rn=rn,
last_step=last_step,
action_score=action_score,
ln=ln)
except InvenioWebSubmitFunctionError as e:
register_exception(req=req, alert_admin=True, prefix='doctype="%s", action="%s", step="%s", form="%s", start_time="%s"' % (doctype, act, step, form, start_time))
## There was a serious function-error. Execution ends.
if CFG_DEVEL_SITE:
raise
else:
return warning_page(_("A serious function-error has been encountered. Adminstrators have been alerted. <br /><em>Please not that this might be due to wrong characters inserted into the form</em> (e.g. by copy and pasting some text from a PDF file)."), req, ln)
except InvenioWebSubmitFunctionStop as e:
## For one reason or another, one of the functions has determined that
## the data-processing phase (i.e. the functions execution) should be
## halted and the user should be returned to the form interface once
## more. (NOTE: Redirecting the user to the Web-form interface is
## currently done using JavaScript. The "InvenioWebSubmitFunctionStop"
## exception contains a "value" string, which is effectively JavaScript
## - probably an alert box and a form that is submitted). **THIS WILL
## CHANGE IN THE FUTURE WHEN JavaScript IS REMOVED!**
if e.value is not None:
function_content = e.value
else:
function_content = e
else:
## No function exceptions (InvenioWebSubmitFunctionStop,
## InvenioWebSubmitFunctionError) were raised by the functions. Propose
## the next action (if applicable), and log the submission as finished:
## If the action was mandatory we propose the next
## mandatory action (if any)
if action_score != -1 and last_step == 1:
next_action = Propose_Next_Action(doctype, \
action_score, \
access, \
current_level, \
indir)
## If we are in the last step of an action, we can update
## the "journal of submissions"
if last_step == 1:
if uid_email != "" and uid_email != "guest":
## update the "journal of submission":
## Does the submission already exist in the log?
submission_exists = \
submission_exists_in_log(doctype, act, access, uid_email)
if submission_exists == 1:
## update the rn and status to finished for this submission
## in the log:
update_submission_reference_and_status_in_log(doctype, \
act, \
access, \
uid_email, \
rn, \
"finished")
else:
## Submission doesn't exist in log - create it:
log_new_completed_submission(doctype, \
act, \
access, \
uid_email, \
rn)
## Having executed the functions, create the page that will be displayed
## to the user:
t = websubmit_templates.tmpl_page_endaction(
ln = ln,
# these fields are necessary for the navigation
nextPg = nextPg,
startPg = startPg,
access = access,
curpage = curpage,
nbPg = nbPg,
nbpages = nbpages,
doctype = doctype,
act = act,
docname = docname,
actname = actname,
mainmenu = mainmenu,
finished = finished,
function_content = function_content,
next_action = next_action,
)
if finished:
# register event in webstat
try:
register_customevent("websubmissions", [get_longname_of_doctype(doctype)])
except:
register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'")
else:
t += websubmit_templates.tmpl_page_do_not_leave_submission_js(ln)
# start display:
req.content_type = "text/html"
req.send_http_header()
p_navtrail = '<a href="/submit?ln='+ln+'" class="navtrail">' + _("Submit") +\
"""</a>&nbsp;>&nbsp;<a href="/submit?doctype=%(doctype)s&amp;ln=%(ln)s" class="navtrail">%(docname)s</a>""" % {
'doctype' : quote_plus(doctype),
'docname' : docname,
'ln' : ln,
}
## add MathJax if wanted
if CFG_WEBSUBMIT_USE_MATHJAX:
metaheaderadd = get_mathjax_header(req.is_https())
metaheaderadd += websubmit_templates.tmpl_mathpreview_header(ln, req.is_https())
else:
metaheaderadd = ''
return page(title= actname,
body = t,
navtrail = p_navtrail,
description="submit documents",
keywords="submit",
uid = uid,
language = ln,
req = req,
navmenuid='submit',
metaheaderadd=metaheaderadd)
def home(req, catalogues_text, c=CFG_SITE_NAME, ln=CFG_SITE_LANG):
"""This function generates the WebSubmit "home page".
Basically, this page contains a list of submission-collections
in WebSubmit, and gives links to the various document-type
submissions.
Document-types only appear on this page when they have been
connected to a submission-collection in WebSubmit.
@param req: (apache request object)
@param catalogues_text (string): the computed catalogues tree
@param c: (string) - defaults to CFG_SITE_NAME
@param ln: (string) - The Invenio interface language of choice.
Defaults to CFG_SITE_LANG (the default language of the installation).
@return: (string) - the Web page to be displayed.
"""
ln = wash_language(ln)
# get user ID:
try:
uid = getUid(req)
user_info = collect_user_info(req)
except Error as e:
return error_page(e, req, ln)
# load the right message language
_ = gettext_set_language(ln)
finaltext = websubmit_templates.tmpl_submit_home_page(
ln = ln,
catalogues = catalogues_text,
user_info = user_info,
)
return page(title=_("Submit"),
body=finaltext,
navtrail=[],
description="submit documents",
keywords="submit",
uid=uid,
language=ln,
req=req,
navmenuid='submit'
)
def makeCataloguesTable(req, ln=CFG_SITE_LANG):
"""Build the 'catalogues' (submission-collections) tree for
the WebSubmit home-page. This tree contains the links to
the various document types in WebSubmit.
@param req: (dict) - the user request object
in order to decide whether to display a submission.
@param ln: (string) - the language of the interface.
(defaults to 'CFG_SITE_LANG').
@return: (string, bool, bool) - the submission-collections tree.
True if there is at least one submission authorized for the user
True if there is at least one submission
"""
def is_at_least_one_submission_authorized(cats):
for cat in cats:
if cat['docs']:
return True
if is_at_least_one_submission_authorized(cat['sons']):
return True
return False
text = ""
catalogues = []
## Get the submission-collections attached at the top level
## of the submission-collection tree:
top_level_collctns = get_collection_children_of_submission_collection(0)
if len(top_level_collctns) != 0:
## There are submission-collections attatched to the top level.
## retrieve their details for displaying:
for child_collctn in top_level_collctns:
catalogues.append(getCatalogueBranch(child_collctn[0], 1, req))
text = websubmit_templates.tmpl_submit_home_catalogs(
ln=ln,
catalogs=catalogues)
submissions_exist = True
at_least_one_submission_authorized = is_at_least_one_submission_authorized(catalogues)
else:
text = websubmit_templates.tmpl_submit_home_catalog_no_content(ln=ln)
submissions_exist = False
at_least_one_submission_authorized = False
return text, at_least_one_submission_authorized, submissions_exist
def getCatalogueBranch(id_father, level, req):
"""Build up a given branch of the submission-collection
tree. I.e. given a parent submission-collection ID,
build up the tree below it. This tree will include
doctype-children, as well as other submission-
collections and their children.
Finally, return the branch as a dictionary.
@param id_father: (integer) - the ID of the submission-collection
from which to begin building the branch.
@param level: (integer) - the level of the current submission-
collection branch.
@param req: (dict) - the user request object in order to decide
whether to display a submission.
@return: (dictionary) - the branch and its sub-branches.
"""
elem = {} ## The dictionary to contain this branch of the tree.
## First, get the submission-collection-details:
collctn_name = get_submission_collection_name(id_father)
if collctn_name is not None:
## Got the submission-collection's name:
elem['name'] = collctn_name
else:
## The submission-collection is unknown to the DB
## set its name as empty:
elem['name'] = ""
elem['id'] = id_father
elem['level'] = level
## Now get details of the doctype-children of this
## submission-collection:
elem['docs'] = [] ## List to hold the doctype-children
## of the submission-collection
doctype_children = \
get_doctype_children_of_submission_collection(id_father)
user_info = collect_user_info(req)
for child_doctype in doctype_children:
## To get access to a submission pipeline for a logged in user,
## it is decided by any authorization. If none are defined for the action
## then a logged in user will get access.
## If user is not logged in, a specific rule to allow the action is needed
if acc_authorize_action(req, 'submit', \
authorized_if_no_roles=not isGuestUser(user_info['uid']), \
doctype=child_doctype[0])[0] == 0:
elem['docs'].append(getDoctypeBranch(child_doctype[0]))
## Now, get the collection-children of this submission-collection:
elem['sons'] = []
collctn_children = \
get_collection_children_of_submission_collection(id_father)
for child_collctn in collctn_children:
elem['sons'].append(getCatalogueBranch(child_collctn[0], level + 1, req))
## Now return this branch of the built-up 'collection-tree':
return elem
def getDoctypeBranch(doctype):
"""Create a document-type 'leaf-node' for the submission-collections
tree. Basically, this leaf is a dictionary containing the name
and ID of the document-type submission to which it links.
@param doctype: (string) - the ID of the document type.
@return: (dictionary) - the document-type 'leaf node'. Contains
the following values:
+ id: (string) - the document-type ID.
+ name: (string) - the (long) name of the document-type.
"""
ldocname = get_longname_of_doctype(doctype)
if ldocname is None:
ldocname = "Unknown Document Type"
return { 'id' : doctype, 'name' : ldocname, }
def displayCatalogueBranch(id_father, level, catalogues):
text = ""
collctn_name = get_submission_collection_name(id_father)
if collctn_name is None:
## If this submission-collection wasn't known in the DB,
## give it the name "Unknown Submission-Collection" to
## avoid errors:
collctn_name = "Unknown Submission-Collection"
## Now, create the display for this submission-collection:
if level == 1:
text = "<LI><font size=\"+1\"><strong>%s</strong></font>\n" \
% collctn_name
else:
## TODO: These are the same (and the if is ugly.) Why?
if level == 2:
text = "<LI>%s\n" % collctn_name
else:
if level > 2:
text = "<LI>%s\n" % collctn_name
## Now display the children document-types that are attached
## to this submission-collection:
## First, get the children:
doctype_children = get_doctype_children_of_submission_collection(id_father)
collctn_children = get_collection_children_of_submission_collection(id_father)
if len(doctype_children) > 0 or len(collctn_children) > 0:
## There is something to display, so open a list:
text = text + "<UL>\n"
## First, add the doctype leaves of this branch:
for child_doctype in doctype_children:
## Add the doctype 'leaf-node':
text = text + displayDoctypeBranch(child_doctype[0], catalogues)
## Now add the submission-collection sub-branches:
for child_collctn in collctn_children:
catalogues.append(child_collctn[0])
text = text + displayCatalogueBranch(child_collctn[0], level+1, catalogues)
## Finally, close up the list if there were nodes to display
## at this branch:
if len(doctype_children) > 0 or len(collctn_children) > 0:
text = text + "</UL>\n"
return text
def displayDoctypeBranch(doctype, catalogues):
text = ""
ldocname = get_longname_of_doctype(doctype)
if ldocname is None:
ldocname = "Unknown Document Type"
text = "<LI><a href=\"\" onmouseover=\"javascript:" \
"popUpTextWindow('%s',true,event);\" onmouseout" \
"=\"javascript:popUpTextWindow('%s',false,event);\" " \
"onClick=\"document.forms[0].doctype.value='%s';" \
"document.forms[0].submit();return false;\">%s</a>\n" \
% (doctype, doctype, doctype, ldocname)
return text
def action(req, c=CFG_SITE_NAME, ln=CFG_SITE_LANG, doctype=""):
# load the right message language
_ = gettext_set_language(ln)
nbCateg = 0
snameCateg = []
lnameCateg = []
actionShortDesc = []
indir = []
actionbutton = []
statustext = []
t = ""
ln = wash_language(ln)
# get user ID:
try:
uid = getUid(req)
except Error as e:
return error_page(e, req, ln)
#parses database to get all data
## first, get the list of categories
doctype_categs = get_categories_of_doctype(doctype)
for doctype_categ in doctype_categs:
if not acc_authorize_action(req, 'submit', \
authorized_if_no_roles=not isGuestUser(uid), \
verbose=0, \
doctype=doctype, \
categ=doctype_categ[0])[0] == 0:
# This category is restricted for this user, move on to the next categories.
continue
nbCateg = nbCateg+1
snameCateg.append(doctype_categ[0])
lnameCateg.append(doctype_categ[1])
## Now get the details of the document type:
doctype_details = get_doctype_details(doctype)
if doctype_details is None:
## Doctype doesn't exist - raise error:
return warning_page(_("Unable to find document type: %(doctype)s",
doctype=escape(str(doctype))), req, ln)
else:
docFullDesc = doctype_details[0]
# Also update the doctype as returned by the database, since
# it might have a differnent case (eg. DemOJrN->demoJRN)
doctype = docShortDesc = doctype_details[1]
description = doctype_details[4]
## Get the details of the actions supported by this document-type:
doctype_actions = get_actions_on_submission_page_for_doctype(doctype)
for doctype_action in doctype_actions:
if not acc_authorize_action(req, 'submit', \
authorized_if_no_roles=not isGuestUser(uid), \
doctype=doctype, \
act=doctype_action[0])[0] == 0:
# This action is not authorized for this user, move on to the next actions.
continue
## Get the details of this action:
action_details = get_action_details(doctype_action[0])
if action_details is not None:
actionShortDesc.append(doctype_action[0])
indir.append(action_details[1])
actionbutton.append(action_details[4])
statustext.append(action_details[5])
if not snameCateg and not actionShortDesc:
if isGuestUser(uid):
# If user is guest and does not have access to any of the
# categories, offer to login.
return redirect_to_url(req, "%s/youraccount/login%s" % (
CFG_SITE_SECURE_URL,
make_canonical_urlargd({'referer' : CFG_SITE_SECURE_URL + req.unparsed_uri, 'ln' : ln}, {})),
norobot=True)
else:
return page_not_authorized(req, "../submit",
uid=uid,
text=_("You are not authorized to access this submission interface."),
navmenuid='submit')
## Send the gathered information to the template so that the doctype's
## home-page can be displayed:
t = websubmit_templates.tmpl_action_page(
ln=ln,
uid=uid,
pid = os.getpid(),
now = time.time(),
doctype = doctype,
description = description,
docfulldesc = docFullDesc,
snameCateg = snameCateg,
lnameCateg = lnameCateg,
actionShortDesc = actionShortDesc,
indir = indir,
# actionbutton = actionbutton,
statustext = statustext,
)
p_navtrail = """<a href="/submit?ln=%(ln)s" class="navtrail">%(submit)s</a>""" % {'submit' : _("Submit"),
'ln' : ln}
return page(title = docFullDesc,
body=t,
navtrail=p_navtrail,
description="submit documents",
keywords="submit",
uid=uid,
language=ln,
req=req,
navmenuid='submit'
)
def Request_Print(m, txt):
"""The argumemts to this function are the display mode (m) and the text
to be displayed (txt).
"""
return txt
def Evaluate_Parameter (field, doctype):
# Returns the literal value of the parameter. Assumes that the value is
# uniquely determined by the doctype, i.e. doctype is the primary key in
# the table
# If the table name is not null, evaluate the parameter
## TODO: The above comment looks like nonesense? This
## function only seems to get the values of parameters
## from the db...
## Get the value for the parameter:
param_val = get_parameter_value_for_doctype(doctype, field)
if param_val is None:
## Couldn't find a value for this parameter for this doctype.
## Instead, try with the default doctype (DEF):
param_val = get_parameter_value_for_doctype("DEF", field)
if param_val is None:
## There was no value for the parameter for the default doctype.
## Nothing can be done about it - return an empty string:
return ""
else:
## There was some kind of value for the parameter; return it:
return param_val
def Get_Parameters (function, doctype):
"""For a given function of a given document type, a dictionary
of the parameter names and values are returned.
@param function: (string) - the name of the function for which the
parameters are to be retrieved.
@param doctype: (string) - the ID of the document type.
@return: (dictionary) - of the parameters of the function.
Keyed by the parameter name, values are of course the parameter
values.
"""
parray = {}
## Get the names of the parameters expected by this function:
func_params = get_parameters_of_function(function)
for func_param in func_params:
## For each of the parameters, get its value for this document-
## type and add it into the dictionary of parameters:
parameter = func_param[0]
parray[parameter] = Evaluate_Parameter (parameter, doctype)
return parray
def get_level(doctype, action):
"""Get the level of a given submission. If unknown, return 0
as the level.
@param doctype: (string) - the ID of the document type.
@param action: (string) - the ID of the action.
@return: (integer) - the level of the submission; 0 otherwise.
"""
subm_details = get_details_of_submission(doctype, action)
if subm_details is not None:
## Return the level of this action
subm_level = subm_details[9]
try:
int(subm_level)
except ValueError:
return 0
else:
return subm_level
else:
return 0
def action_details (doctype, action):
# Prints whether the action is mandatory or optional. The score of the
# action is returned (-1 if the action was optional)
subm_details = get_details_of_submission(doctype, action)
if subm_details is not None:
if subm_details[9] != "0":
## This action is mandatory; return the score:
return subm_details[10]
else:
return -1
else:
return -1
def print_function_calls(req, doctype, action, step, form, start_time,
access, curdir, dismode, rn, last_step, action_score,
ln=CFG_SITE_LANG):
""" Calls the functions required by an 'action'
action on a 'doctype' document In supervisor mode, a table of the
function calls is produced
@return: (function_output_string, last_step, action_score, rn)
"""
user_info = collect_user_info(req)
# load the right message language
_ = gettext_set_language(ln)
t = ""
## Here follows the global protect environment.
the_globals = {
'doctype' : doctype,
'action' : action,
'act' : action, ## for backward compatibility
'step' : step,
'access' : access,
'ln' : ln,
'curdir' : curdir,
'uid' : user_info['uid'],
'uid_email' : user_info['email'],
'rn' : rn,
'last_step' : last_step,
'action_score' : action_score,
'__websubmit_in_jail__' : True,
'form' : form,
'user_info' : user_info,
'__builtins__' : globals()['__builtins__'],
'Request_Print': Request_Print
}
## Get the list of functions to be called
funcs_to_call = get_functions_for_submission_step(doctype, action, step)
## If no functions are found at this step for this doctype,
## get the functions for the DEF(ault) doctype:
if len(funcs_to_call) == 0:
funcs_to_call = get_functions_for_submission_step("DEF", action, step)
if len(funcs_to_call) > 0:
# while there are functions left...
functions = []
for function in funcs_to_call:
try:
function_name = function[0]
function_score = function[1]
currfunction = {
'name' : function_name,
'score' : function_score,
'error' : 0,
'text' : '',
}
#FIXME: deprecated
from invenio.legacy.websubmit import functions as legacy_functions
function_path = os.path.join(legacy_functions.__path__[0],
function_name + '.py')
if os.path.exists(function_path):
# import the function itself
#function = getattr(invenio.legacy.websubmit.functions, function_name)
execfile(function_path, the_globals)
if function_name not in the_globals:
currfunction['error'] = 1
else:
the_globals['function'] = the_globals[function_name]
# Evaluate the parameters, and place them in an array
the_globals['parameters'] = Get_Parameters(function_name, doctype)
# Call function:
log_function(curdir, "Start %s" % function_name, start_time)
try:
try:
## Attempt to call the function with 4 arguments:
## ("parameters", "curdir" and "form" as usual),
## and "user_info" - the dictionary of user
## information:
##
## Note: The function should always be called with
## these keyword arguments because the "TypeError"
## except clause checks for a specific mention of
## the 'user_info' keyword argument when a legacy
## function (one that accepts only 'parameters',
## 'curdir' and 'form') has been called and if
## the error string doesn't contain this,
## the TypeError will be considered as a something
## that was incorrectly handled in the function and
## will be propagated as an
## InvenioWebSubmitFunctionError instead of the
## function being called again with the legacy 3
## arguments.
func_returnval = eval("function(parameters=parameters, curdir=curdir, form=form, user_info=user_info)", the_globals)
except TypeError as err:
## If the error contains the string "got an
## unexpected keyword argument", it means that the
## function doesn't accept the "user_info"
## argument. Test for this:
if "got an unexpected keyword argument 'user_info'" in \
str(err).lower():
## As expected, the function doesn't accept
## the user_info keyword argument. Call it
## again with the legacy 3 arguments
## (parameters, curdir, form):
func_returnval = eval("function(parameters=parameters, curdir=curdir, form=form)", the_globals)
else:
## An unexpected "TypeError" was caught.
## It looks as though the function itself didn't
## handle something correctly.
## Convert this error into an
## InvenioWebSubmitFunctionError and raise it:
msg = "Unhandled TypeError caught when " \
"calling [%s] WebSubmit function: " \
"[%s]: \n%s" % (function_name, str(err), traceback.format_exc())
raise InvenioWebSubmitFunctionError(msg)
except InvenioWebSubmitFunctionWarning as err:
## There was an unexpected behaviour during the
## execution. Log the message into function's log
## and go to next function
log_function(curdir, "***Warning*** from %s: %s" \
% (function_name, str(err)), start_time)
## Reset "func_returnval" to None:
func_returnval = None
register_exception(req=req, alert_admin=True, prefix="Warning in executing function %s with globals %s" % (pprint.pformat(currfunction), pprint.pformat(the_globals)))
log_function(curdir, "End %s" % function_name, start_time)
if func_returnval is not None:
## Append the returned value as a string:
currfunction['text'] = str(func_returnval)
else:
## The function the NoneType. Don't keep that value as
## the currfunction->text. Replace it with the empty
## string.
currfunction['text'] = ""
else:
currfunction['error'] = 1
functions.append(currfunction)
except InvenioWebSubmitFunctionStop as err:
## The submission asked to stop execution. This is
## ok. Do not alert admin, and raise exception further
log_function(curdir, "***Stop*** from %s: %s" \
% (function_name, str(err)), start_time)
raise
except:
register_exception(req=req, alert_admin=True, prefix="Error in executing function %s with globals %s" % (pprint.pformat(currfunction), pprint.pformat(the_globals)))
raise
t = websubmit_templates.tmpl_function_output(
ln = ln,
display_on = (dismode == 'S'),
action = action,
doctype = doctype,
step = step,
functions = functions,
)
else :
if dismode == 'S':
t = "<br /><br /><b>" + _("The chosen action is not supported by the document type.") + "</b>"
return (t, the_globals['last_step'], the_globals['action_score'], the_globals['rn'])
def Propose_Next_Action (doctype, action_score, access, currentlevel, indir, ln=CFG_SITE_LANG):
t = ""
next_submissions = \
get_submissions_at_level_X_with_score_above_N(doctype, currentlevel, action_score)
if len(next_submissions) > 0:
actions = []
first_score = next_submissions[0][10]
for action in next_submissions:
if action[10] == first_score:
## Get the submission directory of this action:
nextdir = get_storage_directory_of_action(action[1])
if nextdir is None:
nextdir = ""
curraction = {
'page' : action[11],
'action' : action[1],
'doctype' : doctype,
'nextdir' : nextdir,
'access' : access,
'indir' : indir,
'name' : action[12],
}
actions.append(curraction)
t = websubmit_templates.tmpl_next_action(
ln = ln,
actions = actions,
)
return t
def specialchars(text):
text = string.replace(text, "&#147;", "\042");
text = string.replace(text, "&#148;", "\042");
text = string.replace(text, "&#146;", "\047");
text = string.replace(text, "&#151;", "\055");
text = string.replace(text, "&#133;", "\056\056\056");
return text
def log_function(curdir, message, start_time, filename="function_log"):
"""Write into file the message and the difference of time
between starttime and current time
@param curdir:(string) path to the destination dir
@param message: (string) message to write into the file
@param starttime: (float) time to compute from
@param filname: (string) name of log file
"""
time_lap = "%.3f" % (time.time() - start_time)
if os.access(curdir, os.F_OK|os.W_OK):
fd = open("%s/%s" % (curdir, filename), "a+")
fd.write("""%s --- %s\n""" % (message, time_lap))
fd.close()
diff --git a/invenio/legacy/websubmit/file_converter.py b/invenio/legacy/websubmit/file_converter.py
index f1c31244b..2d42ae58d 100644
--- a/invenio/legacy/websubmit/file_converter.py
+++ b/invenio/legacy/websubmit/file_converter.py
@@ -1,1485 +1,1484 @@
# -*- coding: utf-8 -*-
## This file is part of Invenio.
## Copyright (C) 2009, 2010, 2011, 2012 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
"""
This module implement fulltext conversion between many different file formats.
"""
import HTMLParser
import atexit
import os
import pkg_resources
import re
import shutil
import signal
import stat
import subprocess
import sys
import tempfile
import threading
import time
from logging import DEBUG, getLogger
from six.moves.html_entities import entitydefs
from optparse import OptionParser
from six import iteritems
try:
from invenio.legacy.websubmit.hocrlib import create_pdf, extract_hocr, CFG_PPM_RESOLUTION
try:
from PyPDF2 import PdfFileReader, PdfFileWriter
except ImportError:
from pyPdf import PdfFileReader, PdfFileWriter
CFG_CAN_DO_OCR = True
except ImportError:
CFG_CAN_DO_OCR = False
from invenio.utils.text import wrap_text_in_a_box
from invenio.utils.shell import run_process_with_timeout, run_shell_command
from invenio.config import CFG_TMPDIR, CFG_ETCDIR, CFG_PYLIBDIR, \
CFG_PATH_ANY2DJVU, \
CFG_PATH_PDFINFO, \
CFG_PATH_GS, \
CFG_PATH_PDFOPT, \
CFG_PATH_PDFTOPS, \
CFG_PATH_GZIP, \
CFG_PATH_GUNZIP, \
CFG_PATH_PDFTOTEXT, \
CFG_PATH_PDFTOPPM, \
CFG_PATH_OCROSCRIPT, \
CFG_PATH_DJVUPS, \
CFG_PATH_DJVUTXT, \
CFG_PATH_OPENOFFICE_PYTHON, \
CFG_PATH_PSTOTEXT, \
CFG_PATH_TIFF2PDF, \
CFG_PATH_PS2PDF, \
CFG_OPENOFFICE_SERVER_HOST, \
CFG_OPENOFFICE_SERVER_PORT, \
CFG_OPENOFFICE_USER, \
CFG_PATH_CONVERT, \
CFG_PATH_PAMFILE, \
CFG_BINDIR, \
CFG_LOGDIR, \
CFG_BIBSCHED_PROCESS_USER, \
CFG_BIBDOCFILE_BEST_FORMATS_TO_EXTRACT_TEXT_FROM, \
CFG_BIBDOCFILE_DESIRED_CONVERSIONS
from invenio.ext.logging import register_exception
def get_file_converter_logger():
return getLogger("InvenioWebSubmitFileConverterLogger")
CFG_TWO2THREE_LANG_CODES = {
'en': 'eng',
'nl': 'nld',
'es': 'spa',
'de': 'deu',
'it': 'ita',
'fr': 'fra',
}
CFG_OPENOFFICE_TMPDIR = os.path.join(CFG_TMPDIR, 'ooffice-tmp-files')
CFG_GS_MINIMAL_VERSION_FOR_PDFA = "8.65"
CFG_GS_MINIMAL_VERSION_FOR_PDFX = "8.52"
#FIXME: pu don't know where is this file
CFG_ICC_PATH = os.path.join(CFG_ETCDIR, 'websubmit', 'file_converter_templates', 'ISOCoatedsb.icc')
CFG_PDFA_DEF_PATH = pkg_resources.resource_filename('invenio.legacy.websubmit', os.path.join('file_converter_template', 'PDFA_def.ps'))
CFG_PDFX_DEF_PATH = pkg_resources.resource_filename('invenio.legacy.websubmit', os.path.join('file_converter_template', 'PDFX_def.ps'))
CFG_UNOCONV_LOG_PATH = os.path.join(CFG_LOGDIR, 'unoconv.log')
_RE_CLEAN_SPACES = re.compile(r'\s+')
class InvenioWebSubmitFileConverterError(Exception):
pass
def get_conversion_map():
"""Return a dictionary of the form:
'.pdf' : {'.ps.gz' : ('pdf2ps', {param1 : value1...})
"""
ret = {
'.csv': {},
'.djvu': {},
'.doc': {},
'.docx': {},
'.sxw': {},
'.htm': {},
'.html': {},
'.odp': {},
'.ods': {},
'.odt': {},
'.pdf': {},
'.ppt': {},
'.pptx': {},
'.sxi': {},
'.ps': {},
'.ps.gz': {},
'.rtf': {},
'.tif': {},
'.tiff': {},
'.txt': {},
'.xls': {},
'.xlsx': {},
'.sxc': {},
'.xml': {},
'.hocr': {},
'.pdf;pdfa': {},
'.asc': {},
}
if CFG_PATH_GZIP:
ret['.ps']['.ps.gz'] = (gzip, {})
if CFG_PATH_GUNZIP:
ret['.ps.gz']['.ps'] = (gunzip, {})
if CFG_PATH_ANY2DJVU:
ret['.pdf']['.djvu'] = (any2djvu, {})
ret['.ps']['.djvu'] = (any2djvu, {})
if CFG_PATH_DJVUPS:
ret['.djvu']['.ps'] = (djvu2ps, {'compress': False})
if CFG_PATH_GZIP:
ret['.djvu']['.ps.gz'] = (djvu2ps, {'compress': True})
if CFG_PATH_DJVUTXT:
ret['.djvu']['.txt'] = (djvu2text, {})
if CFG_PATH_PSTOTEXT:
ret['.ps']['.txt'] = (pstotext, {})
if CFG_PATH_GUNZIP:
ret['.ps.gz']['.txt'] = (pstotext, {})
if can_pdfa():
ret['.ps']['.pdf;pdfa'] = (ps2pdfa, {})
ret['.pdf']['.pdf;pdfa'] = (pdf2pdfa, {})
if CFG_PATH_GUNZIP:
ret['.ps.gz']['.pdf;pdfa'] = (ps2pdfa, {})
else:
if CFG_PATH_PS2PDF:
ret['.ps']['.pdf;pdfa'] = (ps2pdf, {})
if CFG_PATH_GUNZIP:
ret['.ps.gz']['.pdf'] = (ps2pdf, {})
if can_pdfx():
ret['.ps']['.pdf;pdfx'] = (ps2pdfx, {})
ret['.pdf']['.pdf;pdfx'] = (pdf2pdfx, {})
if CFG_PATH_GUNZIP:
ret['.ps.gz']['.pdf;pdfx'] = (ps2pdfx, {})
if CFG_PATH_PDFTOPS:
ret['.pdf']['.ps'] = (pdf2ps, {'compress': False})
ret['.pdf;pdfa']['.ps'] = (pdf2ps, {'compress': False})
if CFG_PATH_GZIP:
ret['.pdf']['.ps.gz'] = (pdf2ps, {'compress': True})
ret['.pdf;pdfa']['.ps.gz'] = (pdf2ps, {'compress': True})
if CFG_PATH_PDFTOTEXT:
ret['.pdf']['.txt'] = (pdf2text, {})
ret['.pdf;pdfa']['.txt'] = (pdf2text, {})
ret['.asc']['.txt'] = (txt2text, {})
ret['.txt']['.txt'] = (txt2text, {})
ret['.csv']['.txt'] = (txt2text, {})
ret['.html']['.txt'] = (html2text, {})
ret['.htm']['.txt'] = (html2text, {})
ret['.xml']['.txt'] = (html2text, {})
if CFG_PATH_TIFF2PDF:
ret['.tiff']['.pdf'] = (tiff2pdf, {})
ret['.tif']['.pdf'] = (tiff2pdf, {})
if CFG_PATH_OPENOFFICE_PYTHON and CFG_OPENOFFICE_SERVER_HOST:
ret['.rtf']['.odt'] = (unoconv, {'output_format': 'odt'})
ret['.rtf']['.pdf;pdfa'] = (unoconv, {'output_format': 'pdf'})
ret['.rtf']['.txt'] = (unoconv, {'output_format': 'txt'})
ret['.rtf']['.docx'] = (unoconv, {'output_format': 'docx'})
ret['.doc']['.odt'] = (unoconv, {'output_format': 'odt'})
ret['.doc']['.pdf;pdfa'] = (unoconv, {'output_format': 'pdf'})
ret['.doc']['.txt'] = (unoconv, {'output_format': 'txt'})
ret['.doc']['.docx'] = (unoconv, {'output_format': 'docx'})
ret['.docx']['.odt'] = (unoconv, {'output_format': 'odt'})
ret['.docx']['.pdf;pdfa'] = (unoconv, {'output_format': 'pdf'})
ret['.docx']['.txt'] = (unoconv, {'output_format': 'txt'})
ret['.sxw']['.odt'] = (unoconv, {'output_format': 'odt'})
ret['.sxw']['.pdf;pdfa'] = (unoconv, {'output_format': 'pdf'})
ret['.sxw']['.txt'] = (unoconv, {'output_format': 'txt'})
ret['.docx']['.docx'] = (unoconv, {'output_format': 'docx'})
ret['.odt']['.doc'] = (unoconv, {'output_format': 'doc'})
ret['.odt']['.pdf;pdfa'] = (unoconv, {'output_format': 'pdf'})
ret['.odt']['.txt'] = (unoconv, {'output_format': 'txt'})
ret['.odt']['.docx'] = (unoconv, {'output_format': 'docx'})
ret['.ppt']['.odp'] = (unoconv, {'output_format': 'odp'})
ret['.ppt']['.pdf;pdfa'] = (unoconv, {'output_format': 'pdf'})
ret['.ppt']['.txt'] = (unoconv, {'output_format': 'txt'})
ret['.ppt']['.pptx'] = (unoconv, {'output_format': 'pptx'})
ret['.pptx']['.odp'] = (unoconv, {'output_format': 'odp'})
ret['.pptx']['.pdf;pdfa'] = (unoconv, {'output_format': 'pdf'})
ret['.pptx']['.txt'] = (unoconv, {'output_format': 'txt'})
ret['.sxi']['.odp'] = (unoconv, {'output_format': 'odp'})
ret['.sxi']['.pdf;pdfa'] = (unoconv, {'output_format': 'pdf'})
ret['.sxi']['.txt'] = (unoconv, {'output_format': 'txt'})
ret['.sxi']['.pptx'] = (unoconv, {'output_format': 'pptx'})
ret['.odp']['.ppt'] = (unoconv, {'output_format': 'ppt'})
ret['.odp']['.pptx'] = (unoconv, {'output_format': 'pptx'})
ret['.odp']['.pdf;pdfa'] = (unoconv, {'output_format': 'pdf'})
ret['.odp']['.txt'] = (unoconv, {'output_format': 'txt'})
ret['.odp']['.pptx'] = (unoconv, {'output_format': 'pptx'})
ret['.xls']['.ods'] = (unoconv, {'output_format': 'ods'})
ret['.xls']['.xlsx'] = (unoconv, {'output_format': 'xslx'})
ret['.xlsx']['.ods'] = (unoconv, {'output_format': 'ods'})
ret['.sxc']['.ods'] = (unoconv, {'output_format': 'ods'})
ret['.sxc']['.xlsx'] = (unoconv, {'output_format': 'xslx'})
ret['.ods']['.xls'] = (unoconv, {'output_format': 'xls'})
ret['.ods']['.pdf;pdfa'] = (unoconv, {'output_format': 'pdf'})
ret['.ods']['.csv'] = (unoconv, {'output_format': 'csv'})
ret['.ods']['.xlsx'] = (unoconv, {'output_format': 'xslx'})
ret['.csv']['.txt'] = (txt2text, {})
## Let's add all the existing output formats as potential input formats.
for value in ret.values():
for key in value.keys():
if key not in ret:
ret[key] = {}
return ret
def get_best_format_to_extract_text_from(filelist, best_formats=CFG_BIBDOCFILE_BEST_FORMATS_TO_EXTRACT_TEXT_FROM):
"""
Return among the filelist the best file whose format is best suited for
extracting text.
"""
from invenio.legacy.bibdocfile.api import decompose_file, normalize_format
best_formats = [normalize_format(aformat) for aformat in best_formats if can_convert(aformat, '.txt')]
for aformat in best_formats:
for filename in filelist:
if decompose_file(filename, skip_version=True)[2].endswith(aformat):
return filename
raise InvenioWebSubmitFileConverterError("It's not possible to extract valuable text from any of the proposed files.")
def get_missing_formats(filelist, desired_conversion=None):
"""Given a list of files it will return a dictionary of the form:
file1 : missing formats to generate from it...
"""
from invenio.legacy.bibdocfile.api import normalize_format, decompose_file
def normalize_desired_conversion():
ret = {}
for key, value in iteritems(desired_conversion):
ret[normalize_format(key)] = [normalize_format(aformat) for aformat in value]
return ret
if desired_conversion is None:
desired_conversion = CFG_BIBDOCFILE_DESIRED_CONVERSIONS
available_formats = [decompose_file(filename, skip_version=True)[2] for filename in filelist]
missing_formats = []
desired_conversion = normalize_desired_conversion()
ret = {}
for filename in filelist:
aformat = decompose_file(filename, skip_version=True)[2]
if aformat in desired_conversion:
for desired_format in desired_conversion[aformat]:
if desired_format not in available_formats and desired_format not in missing_formats:
missing_formats.append(desired_format)
if filename not in ret:
ret[filename] = []
ret[filename].append(desired_format)
return ret
def can_convert(input_format, output_format, max_intermediate_conversions=4):
"""Return the chain of conversion to transform input_format into output_format, if any."""
from invenio.legacy.bibdocfile.api import normalize_format
if max_intermediate_conversions <= 0:
return []
input_format = normalize_format(input_format)
output_format = normalize_format(output_format)
if input_format in __CONVERSION_MAP:
if output_format in __CONVERSION_MAP[input_format]:
return [__CONVERSION_MAP[input_format][output_format]]
best_res = []
best_intermediate = ''
for intermediate_format in __CONVERSION_MAP[input_format]:
res = can_convert(intermediate_format, output_format, max_intermediate_conversions-1)
if res and (len(res) < best_res or not best_res):
best_res = res
best_intermediate = intermediate_format
if best_res:
return [__CONVERSION_MAP[input_format][best_intermediate]] + best_res
return []
def can_pdfopt(verbose=False):
"""Return True if it's possible to optimize PDFs."""
if CFG_PATH_PDFOPT:
return True
elif verbose:
print("PDF linearization is not supported because the pdfopt executable is not available", file=sys.stderr)
return False
def can_pdfx(verbose=False):
"""Return True if it's possible to generate PDF/Xs."""
if not CFG_PATH_PDFTOPS:
if verbose:
print("Conversion of PS or PDF to PDF/X is not possible because the pdftops executable is not available", file=sys.stderr)
return False
if not CFG_PATH_GS:
if verbose:
print("Conversion of PS or PDF to PDF/X is not possible because the gs executable is not available", file=sys.stderr)
return False
else:
try:
output = run_shell_command("%s --version" % CFG_PATH_GS)[1].strip()
if not output:
raise ValueError("No version information returned")
if [int(number) for number in output.split('.')] < [int(number) for number in CFG_GS_MINIMAL_VERSION_FOR_PDFX.split('.')]:
print("Conversion of PS or PDF to PDF/X is not possible because the minimal gs version for the executable %s is not met: it should be %s but %s has been found" % (CFG_PATH_GS, CFG_GS_MINIMAL_VERSION_FOR_PDFX, output), file=sys.stderr)
return False
except Exception as err:
print("Conversion of PS or PDF to PDF/X is not possible because it's not possible to retrieve the gs version using the executable %s: %s" % (CFG_PATH_GS, err), file=sys.stderr)
return False
if not CFG_PATH_PDFINFO:
if verbose:
print("Conversion of PS or PDF to PDF/X is not possible because the pdfinfo executable is not available", file=sys.stderr)
return False
if not os.path.exists(CFG_ICC_PATH):
if verbose:
print("Conversion of PS or PDF to PDF/X is not possible because %s does not exists. Have you run make install-pdfa-helper-files?" % CFG_ICC_PATH, file=sys.stderr)
return False
return True
def can_pdfa(verbose=False):
"""Return True if it's possible to generate PDF/As."""
if not CFG_PATH_PDFTOPS:
if verbose:
print("Conversion of PS or PDF to PDF/A is not possible because the pdftops executable is not available", file=sys.stderr)
return False
if not CFG_PATH_GS:
if verbose:
print("Conversion of PS or PDF to PDF/A is not possible because the gs executable is not available", file=sys.stderr)
return False
else:
try:
output = run_shell_command("%s --version" % CFG_PATH_GS)[1].strip()
if not output:
raise ValueError("No version information returned")
if [int(number) for number in output.split('.')] < [int(number) for number in CFG_GS_MINIMAL_VERSION_FOR_PDFA.split('.')]:
print("Conversion of PS or PDF to PDF/A is not possible because the minimal gs version for the executable %s is not met: it should be %s but %s has been found" % (CFG_PATH_GS, CFG_GS_MINIMAL_VERSION_FOR_PDFA, output), file=sys.stderr)
return False
except Exception as err:
print("Conversion of PS or PDF to PDF/A is not possible because it's not possible to retrieve the gs version using the executable %s: %s" % (CFG_PATH_GS, err), file=sys.stderr)
return False
if not CFG_PATH_PDFINFO:
if verbose:
print("Conversion of PS or PDF to PDF/A is not possible because the pdfinfo executable is not available", file=sys.stderr)
return False
if not os.path.exists(CFG_ICC_PATH):
if verbose:
print("Conversion of PS or PDF to PDF/A is not possible because %s does not exists. Have you run make install-pdfa-helper-files?" % CFG_ICC_PATH, file=sys.stderr)
return False
return True
def can_perform_ocr(verbose=False):
"""Return True if it's possible to perform OCR."""
if not CFG_CAN_DO_OCR:
if verbose:
print("OCR is not supported because either the pyPdf of ReportLab Python libraries are missing", file=sys.stderr)
return False
if not CFG_PATH_OCROSCRIPT:
if verbose:
print("OCR is not supported because the ocroscript executable is not available", file=sys.stderr)
return False
if not CFG_PATH_PDFTOPPM:
if verbose:
print("OCR is not supported because the pdftoppm executable is not available", file=sys.stderr)
return False
return True
def guess_ocropus_produced_garbage(input_file, hocr_p):
"""Return True if the output produced by OCROpus in hocr format contains
only garbage instead of text. This is implemented via an heuristic:
if the most common length for sentences encoded in UTF-8 is 1 then
this is Garbage (tm).
"""
def _get_words_from_text():
ret = []
for row in open(input_file):
for word in row.strip().split(' '):
ret.append(word.strip())
return ret
def _get_words_from_hocr():
ret = []
hocr = extract_hocr(open(input_file).read())
for dummy, dummy, lines in hocr:
for dummy, line in lines:
for word in line.split():
ret.append(word.strip())
return ret
if hocr_p:
words = _get_words_from_hocr()
else:
words = _get_words_from_text()
#stats = {}
#most_common_len = 0
#most_common_how_many = 0
#for word in words:
#if word:
#word_length = len(word.decode('utf-8'))
#stats[word_length] = stats.get(word_length, 0) + 1
#if stats[word_length] > most_common_how_many:
#most_common_len = word_length
#most_common_how_many = stats[word_length]
goods = 0
bads = 0
for word in words:
for char in word.decode('utf-8'):
if (u'a' <= char <= u'z') or (u'A' <= char <= u'Z'):
goods += 1
else:
bads += 1
if bads > goods:
get_file_converter_logger().debug('OCROpus produced garbage')
return True
else:
return False
def guess_is_OCR_needed(input_file, ln='en'):
"""
Tries to see if enough text is retrievable from input_file.
Return True if OCR is needed, False if it's already
possible to retrieve information from the document.
"""
## FIXME: a way to understand if pdftotext has returned garbage
## shuould be found. E.g. 1.0*len(text)/len(zlib.compress(text)) < 2.1
## could be a good hint for garbage being found.
return True
def convert_file(input_file, output_file=None, output_format=None, **params):
"""
Convert files from one format to another.
@param input_file [string] the path to an existing file
@param output_file [string] the path to the desired ouput. (if None a
temporary file is generated)
@param output_format [string] the desired format (if None it is taken from
output_file)
@param params other paramaters to pass to the particular converter
@return [string] the final output_file
"""
from invenio.legacy.bibdocfile.api import decompose_file, normalize_format
if output_format is None:
if output_file is None:
raise ValueError("At least output_file or format should be specified.")
else:
output_ext = decompose_file(output_file, skip_version=True)[2]
else:
output_ext = normalize_format(output_format)
input_ext = decompose_file(input_file, skip_version=True)[2]
conversion_chain = can_convert(input_ext, output_ext)
if conversion_chain:
get_file_converter_logger().debug("Conversion chain from %s to %s: %s" % (input_ext, output_ext, conversion_chain))
current_input = input_file
for i, (converter, final_params) in enumerate(conversion_chain):
current_output = None
if i == (len(conversion_chain) - 1):
current_output = output_file
final_params = dict(final_params)
final_params.update(params)
try:
get_file_converter_logger().debug("Converting from %s to %s using %s with params %s" % (current_input, current_output, converter, final_params))
current_output = converter(current_input, current_output, **final_params)
get_file_converter_logger().debug("... current_output %s" % (current_output, ))
except InvenioWebSubmitFileConverterError as err:
raise InvenioWebSubmitFileConverterError("Error when converting from %s to %s: %s" % (input_file, output_ext, err))
except Exception as err:
register_exception(alert_admin=True)
raise InvenioWebSubmitFileConverterError("Unexpected error when converting from %s to %s (%s): %s" % (input_file, output_ext, type(err), err))
if current_input != input_file:
os.remove(current_input)
current_input = current_output
return current_output
else:
raise InvenioWebSubmitFileConverterError("It's impossible to convert from %s to %s" % (input_ext, output_ext))
try:
_UNOCONV_DAEMON
except NameError:
_UNOCONV_DAEMON = None
_UNOCONV_DAEMON_LOCK = threading.Lock()
def _register_unoconv():
global _UNOCONV_DAEMON
if CFG_OPENOFFICE_SERVER_HOST != 'localhost':
return
_UNOCONV_DAEMON_LOCK.acquire()
try:
if not _UNOCONV_DAEMON:
output_log = open(CFG_UNOCONV_LOG_PATH, 'a')
_UNOCONV_DAEMON = subprocess.Popen(['sudo', '-S', '-u', CFG_OPENOFFICE_USER, os.path.join(CFG_BINDIR, 'inveniounoconv'), '-vvv', '-s', CFG_OPENOFFICE_SERVER_HOST, '-p', str(CFG_OPENOFFICE_SERVER_PORT), '-l'], stdin=open('/dev/null', 'r'), stdout=output_log, stderr=output_log)
time.sleep(3)
finally:
_UNOCONV_DAEMON_LOCK.release()
def _unregister_unoconv():
global _UNOCONV_DAEMON
if CFG_OPENOFFICE_SERVER_HOST != 'localhost':
return
_UNOCONV_DAEMON_LOCK.acquire()
try:
if _UNOCONV_DAEMON:
output_log = open(CFG_UNOCONV_LOG_PATH, 'a')
subprocess.call(['sudo', '-S', '-u', CFG_OPENOFFICE_USER, os.path.join(CFG_BINDIR, 'inveniounoconv'), '-k', '-vvv'], stdin=open('/dev/null', 'r'), stdout=output_log, stderr=output_log)
time.sleep(1)
if _UNOCONV_DAEMON.poll():
try:
os.kill(_UNOCONV_DAEMON.pid, signal.SIGTERM)
except OSError:
pass
if _UNOCONV_DAEMON.poll():
try:
os.kill(_UNOCONV_DAEMON.pid, signal.SIGKILL)
except OSError:
pass
finally:
_UNOCONV_DAEMON_LOCK.release()
## NOTE: in case we switch back keeping LibreOffice running, uncomment
## the following line.
#atexit.register(_unregister_unoconv)
def unoconv(input_file, output_file=None, output_format='txt', pdfopt=True, **dummy):
"""Use unconv to convert among OpenOffice understood documents."""
from invenio.legacy.bibdocfile.api import normalize_format
## NOTE: in case we switch back keeping LibreOffice running, uncomment
## the following line.
#_register_unoconv()
input_file, output_file, dummy = prepare_io(input_file, output_file, output_format, need_working_dir=False)
if output_format == 'txt':
unoconv_format = 'text'
else:
unoconv_format = output_format
try:
try:
## We copy the input file and we make it available to OpenOffice
## with the user nobody
from invenio.legacy.bibdocfile.api import decompose_file
input_format = decompose_file(input_file, skip_version=True)[2]
fd, tmpinputfile = tempfile.mkstemp(dir=CFG_TMPDIR, suffix=normalize_format(input_format))
os.close(fd)
shutil.copy(input_file, tmpinputfile)
get_file_converter_logger().debug("Prepared input file %s" % tmpinputfile)
os.chmod(tmpinputfile, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH)
tmpoutputfile = tempfile.mktemp(dir=CFG_OPENOFFICE_TMPDIR, suffix=normalize_format(output_format))
get_file_converter_logger().debug("Prepared output file %s" % tmpoutputfile)
try:
execute_command(os.path.join(CFG_BINDIR, 'inveniounoconv'), '-vvv', '-s', CFG_OPENOFFICE_SERVER_HOST, '-p', str(CFG_OPENOFFICE_SERVER_PORT), '--output', tmpoutputfile, '-f', unoconv_format, tmpinputfile, sudo=CFG_OPENOFFICE_USER)
except:
register_exception(alert_admin=True)
raise
except InvenioWebSubmitFileConverterError:
## Ok maybe OpenOffice hanged. Let's better kill it and restarted!
if CFG_OPENOFFICE_SERVER_HOST != 'localhost':
## There's not that much that we can do. Let's bail out
if not os.path.exists(tmpoutputfile) or not os.path.getsize(tmpoutputfile):
raise
else:
## Sometimes OpenOffice crashes but we don't care :-)
## it still have created a nice file.
pass
else:
execute_command(os.path.join(CFG_BINDIR, 'inveniounoconv'), '-vvv', '-k', sudo=CFG_OPENOFFICE_USER)
## NOTE: in case we switch back keeping LibreOffice running, uncomment
## the following lines.
#_unregister_unoconv()
#_register_unoconv()
time.sleep(5)
try:
execute_command(os.path.join(CFG_BINDIR, 'inveniounoconv'), '-vvv', '-s', CFG_OPENOFFICE_SERVER_HOST, '-p', str(CFG_OPENOFFICE_SERVER_PORT), '--output', tmpoutputfile, '-f', unoconv_format, tmpinputfile, sudo=CFG_OPENOFFICE_USER)
except InvenioWebSubmitFileConverterError:
execute_command(os.path.join(CFG_BINDIR, 'inveniounoconv'), '-vvv', '-k', sudo=CFG_OPENOFFICE_USER)
if not os.path.exists(tmpoutputfile) or not os.path.getsize(tmpoutputfile):
raise InvenioWebSubmitFileConverterError('No output was generated by OpenOffice')
else:
## Sometimes OpenOffice crashes but we don't care :-)
## it still have created a nice file.
pass
except Exception as err:
raise InvenioWebSubmitFileConverterError(get_unoconv_installation_guideline(err))
output_format = normalize_format(output_format)
if output_format == '.pdf' and pdfopt:
pdf2pdfopt(tmpoutputfile, output_file)
else:
shutil.copy(tmpoutputfile, output_file)
execute_command(os.path.join(CFG_BINDIR, 'inveniounoconv'), '-r', tmpoutputfile, sudo=CFG_OPENOFFICE_USER)
os.remove(tmpinputfile)
return output_file
def get_unoconv_installation_guideline(err):
"""Return the Libre/OpenOffice installation guideline (embedding the
current error message).
"""
from invenio.legacy.bibsched.bibtask import guess_apache_process_user
return wrap_text_in_a_box("""\
OpenOffice.org can't properly create files in the OpenOffice.org temporary
directory %(tmpdir)s, as the user %(nobody)s (as configured in
CFG_OPENOFFICE_USER invenio(-local).conf variable): %(err)s.
In your /etc/sudoers file, you should authorize the %(apache)s user to run
%(unoconv)s as %(nobody)s user as in:
%(apache)s ALL=(%(nobody)s) NOPASSWD: %(unoconv)s
You should then run the following commands:
$ sudo mkdir -p %(tmpdir)s
$ sudo chown -R %(nobody)s %(tmpdir)s
$ sudo chmod -R 755 %(tmpdir)s""" % {
'tmpdir' : CFG_OPENOFFICE_TMPDIR,
'nobody' : CFG_OPENOFFICE_USER,
'err' : err,
'apache' : CFG_BIBSCHED_PROCESS_USER or guess_apache_process_user(),
'python' : CFG_PATH_OPENOFFICE_PYTHON,
'unoconv' : os.path.join(CFG_BINDIR, 'inveniounoconv')
})
def can_unoconv(verbose=False):
"""
If OpenOffice.org integration is enabled, checks whether the system is
properly configured.
"""
if CFG_PATH_OPENOFFICE_PYTHON and CFG_OPENOFFICE_SERVER_HOST:
try:
test = os.path.join(CFG_TMPDIR, 'test.txt')
open(test, 'w').write('test')
output = unoconv(test, output_format='pdf')
output2 = convert_file(output, output_format='.txt')
if 'test' not in open(output2).read():
raise Exception("Coulnd't produce a valid PDF with Libre/OpenOffice.org")
os.remove(output2)
os.remove(output)
os.remove(test)
return True
except Exception as err:
if verbose:
print(get_unoconv_installation_guideline(err), file=sys.stderr)
return False
else:
if verbose:
print("Libre/OpenOffice.org integration not enabled", file=sys.stderr)
return False
def any2djvu(input_file, output_file=None, resolution=400, ocr=True, input_format=5, **dummy):
"""
Transform input_file into a .djvu file.
@param input_file [string] the input file name
@param output_file [string] the output_file file name, None for temporary generated
@param resolution [int] the resolution of the output_file
@param input_format [int] [1-9]:
1 - DjVu Document (for verification or OCR)
2 - PS/PS.GZ/PDF Document (default)
3 - Photo/Picture/Icon
4 - Scanned Document - B&W - <200 dpi
5 - Scanned Document - B&W - 200-400 dpi
6 - Scanned Document - B&W - >400 dpi
7 - Scanned Document - Color/Mixed - <200 dpi
8 - Scanned Document - Color/Mixed - 200-400 dpi
9 - Scanned Document - Color/Mixed - >400 dpi
@return [string] output_file input_file.
raise InvenioWebSubmitFileConverterError in case of errors.
Note: due to the bottleneck of using a centralized server, it is very
slow and is not suitable for interactive usage (e.g. WebSubmit functions)
"""
from invenio.legacy.bibdocfile.api import decompose_file
input_file, output_file, working_dir = prepare_io(input_file, output_file, '.djvu')
ocr = ocr and "1" or "0"
## Any2djvu expect to find the file in the current directory.
execute_command(CFG_PATH_ANY2DJVU, '-a', '-c', '-r', resolution, '-o', ocr, '-f', input_format, os.path.basename(input_file), cwd=working_dir)
## Any2djvu doesn't let you choose the output_file file name.
djvu_output = os.path.join(working_dir, decompose_file(input_file)[1] + '.djvu')
shutil.move(djvu_output, output_file)
clean_working_dir(working_dir)
return output_file
_RE_FIND_TITLE = re.compile(r'^Title:\s*(.*?)\s*$')
def pdf2pdfx(input_file, output_file=None, title=None, pdfopt=False, profile="pdf/x-3:2002", **dummy):
"""
Transform any PDF into a PDF/X (see: <http://en.wikipedia.org/wiki/PDF/X>)
@param input_file [string] the input file name
@param output_file [string] the output_file file name, None for temporary generated
@param title [string] the title of the document. None for autodiscovery.
@param pdfopt [bool] whether to linearize the pdf, too.
@param profile: [string] the PDFX profile to use. Supports: 'pdf/x-1a:2001', 'pdf/x-1a:2003', 'pdf/x-3:2002'
@return [string] output_file input_file
raise InvenioWebSubmitFileConverterError in case of errors.
"""
input_file, output_file, working_dir = prepare_io(input_file, output_file, '.pdf')
if title is None:
stdout = execute_command(CFG_PATH_PDFINFO, input_file)
for line in stdout.split('\n'):
g = _RE_FIND_TITLE.match(line)
if g:
title = g.group(1)
break
if not title:
title = 'No title'
get_file_converter_logger().debug("Extracted title is %s" % title)
if os.path.exists(CFG_ICC_PATH):
shutil.copy(CFG_ICC_PATH, working_dir)
else:
raise InvenioWebSubmitFileConverterError('ERROR: ISOCoatedsb.icc file missing. Have you run "make install-pdfa-helper-files" as part of your Invenio deployment?')
pdfx_header = open(CFG_PDFX_DEF_PATH).read()
pdfx_header = pdfx_header.replace('<<<<TITLEMARKER>>>>', title)
icc_iso_profile_def = ''
if profile == 'pdf/x-1a:2001':
pdfx_version = 'PDF/X-1a:2001'
pdfx_conformance = 'PDF/X-1a:2001'
elif profile == 'pdf/x-1a:2003':
pdfx_version = 'PDF/X-1a:2003'
pdfx_conformance = 'PDF/X-1a:2003'
elif profile == 'pdf/x-3:2002':
icc_iso_profile_def = '/ICCProfile (ISOCoatedsb.icc)'
pdfx_version = 'PDF/X-3:2002'
pdfx_conformance = 'PDF/X-3:2002'
pdfx_header = pdfx_header.replace('<<<<ICCPROFILEDEF>>>>', icc_iso_profile_def)
pdfx_header = pdfx_header.replace('<<<<GTS_PDFXVersion>>>>', pdfx_version)
pdfx_header = pdfx_header.replace('<<<<GTS_PDFXConformance>>>>', pdfx_conformance)
outputpdf = os.path.join(working_dir, 'output_file.pdf')
open(os.path.join(working_dir, 'PDFX_def.ps'), 'w').write(pdfx_header)
if profile in ['pdf/x-3:2002']:
execute_command(CFG_PATH_GS, '-sProcessColorModel=DeviceCMYK', '-dPDFX', '-dBATCH', '-dNOPAUSE', '-dNOOUTERSAVE', '-dUseCIEColor', '-sDEVICE=pdfwrite', '-dAutoRotatePages=/None', '-sOutputFile=output_file.pdf', os.path.join(working_dir, 'PDFX_def.ps'), input_file, cwd=working_dir)
elif profile in ['pdf/x-1a:2001', 'pdf/x-1a:2003']:
execute_command(CFG_PATH_GS, '-sProcessColorModel=DeviceCMYK', '-dPDFX', '-dBATCH', '-dNOPAUSE', '-dNOOUTERSAVE', '-sColorConversionStrategy=CMYK', '-sDEVICE=pdfwrite', '-dAutoRotatePages=/None', '-sOutputFile=output_file.pdf', os.path.join(working_dir, 'PDFX_def.ps'), input_file, cwd=working_dir)
if pdfopt:
execute_command(CFG_PATH_PDFOPT, outputpdf, output_file)
else:
shutil.move(outputpdf, output_file)
clean_working_dir(working_dir)
return output_file
def pdf2pdfa(input_file, output_file=None, title=None, pdfopt=True, **dummy):
"""
Transform any PDF into a PDF/A (see: <http://www.pdfa.org/>)
@param input_file [string] the input file name
@param output_file [string] the output_file file name, None for temporary generated
@param title [string] the title of the document. None for autodiscovery.
@param pdfopt [bool] whether to linearize the pdf, too.
@return [string] output_file input_file
raise InvenioWebSubmitFileConverterError in case of errors.
"""
-
- input_file, output_file, working_dir = prepare_io(input_file, output_file, '.pdf')
+ input_file, output_file, working_dir = prepare_io(input_file, output_file, '.pdf;pdfa')
if title is None:
stdout = execute_command(CFG_PATH_PDFINFO, input_file)
for line in stdout.split('\n'):
g = _RE_FIND_TITLE.match(line)
if g:
title = g.group(1)
break
if not title:
title = 'No title'
get_file_converter_logger().debug("Extracted title is %s" % title)
if os.path.exists(CFG_ICC_PATH):
shutil.copy(CFG_ICC_PATH, working_dir)
else:
raise InvenioWebSubmitFileConverterError('ERROR: ISOCoatedsb.icc file missing. Have you run "make install-pdfa-helper-files" as part of your Invenio deployment?')
pdfa_header = open(CFG_PDFA_DEF_PATH).read()
pdfa_header = pdfa_header.replace('<<<<TITLEMARKER>>>>', title)
inputps = os.path.join(working_dir, 'input.ps')
outputpdf = os.path.join(working_dir, 'output_file.pdf')
open(os.path.join(working_dir, 'PDFA_def.ps'), 'w').write(pdfa_header)
execute_command(CFG_PATH_PDFTOPS, '-level3', input_file, inputps)
execute_command(CFG_PATH_GS, '-sProcessColorModel=DeviceCMYK', '-dPDFA', '-dBATCH', '-dNOPAUSE', '-dNOOUTERSAVE', '-dUseCIEColor', '-sDEVICE=pdfwrite', '-dAutoRotatePages=/None', '-sOutputFile=output_file.pdf', os.path.join(working_dir, 'PDFA_def.ps'), 'input.ps', cwd=working_dir)
if pdfopt:
execute_command(CFG_PATH_PDFOPT, outputpdf, output_file)
else:
shutil.move(outputpdf, output_file)
clean_working_dir(working_dir)
return output_file
def pdf2pdfopt(input_file, output_file=None, **dummy):
"""
Linearize the input PDF in order to improve the web-experience when
visualizing the document through the web.
@param input_file [string] the input input_file
@param output_file [string] the output_file file name, None for temporary generated
@return [string] output_file input_file
raise InvenioWebSubmitFileConverterError in case of errors.
"""
input_file, output_file, dummy = prepare_io(input_file, output_file, '.pdf', need_working_dir=False)
execute_command(CFG_PATH_PDFOPT, input_file, output_file)
return output_file
def pdf2ps(input_file, output_file=None, level=2, compress=True, **dummy):
"""
Convert from Pdf to Postscript.
"""
if compress:
suffix = '.ps.gz'
else:
suffix = '.ps'
input_file, output_file, working_dir = prepare_io(input_file, output_file, suffix)
execute_command(CFG_PATH_PDFTOPS, '-level%i' % level, input_file, os.path.join(working_dir, 'output.ps'))
if compress:
execute_command(CFG_PATH_GZIP, '-c', os.path.join(working_dir, 'output.ps'), filename_out=output_file)
else:
shutil.move(os.path.join(working_dir, 'output.ps'), output_file)
clean_working_dir(working_dir)
return output_file
def ps2pdfx(input_file, output_file=None, title=None, pdfopt=False, profile="pdf/x-3:2002", **dummy):
"""
Transform any PS into a PDF/X (see: <http://en.wikipedia.org/wiki/PDF/X>)
@param input_file [string] the input file name
@param output_file [string] the output_file file name, None for temporary generated
@param title [string] the title of the document. None for autodiscovery.
@param pdfopt [bool] whether to linearize the pdf, too.
@param profile: [string] the PDFX profile to use. Supports: 'pdf/x-1a:2001', 'pdf/x-1a:2003', 'pdf/x-3:2002'
@return [string] output_file input_file
raise InvenioWebSubmitFileConverterError in case of errors.
"""
input_file, output_file, working_dir = prepare_io(input_file, output_file, '.pdf')
if input_file.endswith('.gz'):
new_input_file = os.path.join(working_dir, 'input.ps')
execute_command(CFG_PATH_GUNZIP, '-c', input_file, filename_out=new_input_file)
input_file = new_input_file
if not title:
title = 'No title'
shutil.copy(CFG_ICC_PATH, working_dir)
pdfx_header = open(CFG_PDFX_DEF_PATH).read()
pdfx_header = pdfx_header.replace('<<<<TITLEMARKER>>>>', title)
icc_iso_profile_def = ''
if profile == 'pdf/x-1a:2001':
pdfx_version = 'PDF/X-1a:2001'
pdfx_conformance = 'PDF/X-1a:2001'
elif profile == 'pdf/x-1a:2003':
pdfx_version = 'PDF/X-1a:2003'
pdfx_conformance = 'PDF/X-1a:2003'
elif profile == 'pdf/x-3:2002':
icc_iso_profile_def = '/ICCProfile (ISOCoatedsb.icc)'
pdfx_version = 'PDF/X-3:2002'
pdfx_conformance = 'PDF/X-3:2002'
pdfx_header = pdfx_header.replace('<<<<ICCPROFILEDEF>>>>', icc_iso_profile_def)
pdfx_header = pdfx_header.replace('<<<<GTS_PDFXVersion>>>>', pdfx_version)
pdfx_header = pdfx_header.replace('<<<<TITLEMARKER>>>>', title)
outputpdf = os.path.join(working_dir, 'output_file.pdf')
open(os.path.join(working_dir, 'PDFX_def.ps'), 'w').write(pdfx_header)
if profile in ['pdf/x-3:2002']:
execute_command(CFG_PATH_GS, '-sProcessColorModel=DeviceCMYK', '-dPDFX', '-dBATCH', '-dNOPAUSE', '-dNOOUTERSAVE', '-dUseCIEColor', '-sDEVICE=pdfwrite', '-dAutoRotatePages=/None', '-sOutputFile=output_file.pdf', os.path.join(working_dir, 'PDFX_def.ps'), 'input.ps', cwd=working_dir)
elif profile in ['pdf/x-1a:2001', 'pdf/x-1a:2003']:
execute_command(CFG_PATH_GS, '-sProcessColorModel=DeviceCMYK', '-dPDFX', '-dBATCH', '-dNOPAUSE', '-dNOOUTERSAVE', '-sColorConversionStrategy=CMYK', '-dAutoRotatePages=/None', '-sDEVICE=pdfwrite', '-sOutputFile=output_file.pdf', os.path.join(working_dir, 'PDFX_def.ps'), 'input.ps', cwd=working_dir)
if pdfopt:
execute_command(CFG_PATH_PDFOPT, outputpdf, output_file)
else:
shutil.move(outputpdf, output_file)
clean_working_dir(working_dir)
return output_file
def ps2pdfa(input_file, output_file=None, title=None, pdfopt=True, **dummy):
"""
Transform any PS into a PDF/A (see: <http://www.pdfa.org/>)
@param input_file [string] the input file name
@param output_file [string] the output_file file name, None for temporary generated
@param title [string] the title of the document. None for autodiscovery.
@param pdfopt [bool] whether to linearize the pdf, too.
@return [string] output_file input_file
raise InvenioWebSubmitFileConverterError in case of errors.
"""
- input_file, output_file, working_dir = prepare_io(input_file, output_file, '.pdf')
+ input_file, output_file, working_dir = prepare_io(input_file, output_file, '.pdf;pdfa')
if input_file.endswith('.gz'):
new_input_file = os.path.join(working_dir, 'input.ps')
execute_command(CFG_PATH_GUNZIP, '-c', input_file, filename_out=new_input_file)
input_file = new_input_file
if not title:
title = 'No title'
shutil.copy(CFG_ICC_PATH, working_dir)
pdfa_header = open(CFG_PDFA_DEF_PATH).read()
pdfa_header = pdfa_header.replace('<<<<TITLEMARKER>>>>', title)
outputpdf = os.path.join(working_dir, 'output_file.pdf')
open(os.path.join(working_dir, 'PDFA_def.ps'), 'w').write(pdfa_header)
execute_command(CFG_PATH_GS, '-sProcessColorModel=DeviceCMYK', '-dPDFA', '-dBATCH', '-dNOPAUSE', '-dNOOUTERSAVE', '-dUseCIEColor', '-sDEVICE=pdfwrite', '-dAutoRotatePages=/None', '-sOutputFile=output_file.pdf', os.path.join(working_dir, 'PDFA_def.ps'), input_file, cwd=working_dir)
if pdfopt:
execute_command(CFG_PATH_PDFOPT, outputpdf, output_file)
else:
shutil.move(outputpdf, output_file)
clean_working_dir(working_dir)
return output_file
def ps2pdf(input_file, output_file=None, pdfopt=None, **dummy):
"""
Transform any PS into a PDF
@param input_file [string] the input file name
@param output_file [string] the output_file file name, None for temporary generated
@param pdfopt [bool] whether to linearize the pdf, too.
@return [string] output_file input_file
raise InvenioWebSubmitFileConverterError in case of errors.
"""
if pdfopt is None:
pdfopt = bool(CFG_PATH_PDFOPT)
input_file, output_file, working_dir = prepare_io(input_file, output_file, '.pdf')
if input_file.endswith('.gz'):
new_input_file = os.path.join(working_dir, 'input.ps')
execute_command(CFG_PATH_GUNZIP, '-c', input_file, filename_out=new_input_file)
input_file = new_input_file
outputpdf = os.path.join(working_dir, 'output_file.pdf')
execute_command(CFG_PATH_PS2PDF, input_file, outputpdf, cwd=working_dir)
if pdfopt:
execute_command(CFG_PATH_PDFOPT, outputpdf, output_file)
else:
shutil.move(outputpdf, output_file)
clean_working_dir(working_dir)
return output_file
def pdf2pdfhocr(input_pdf, text_hocr, output_pdf, rotations=None, font='Courier', draft=False):
"""
Adds the OCRed text to the original pdf.
@param rotations: a list of angles by which pages should be rotated
"""
def _get_page_rotation(i):
if len(rotations) > i:
return rotations[i]
return 0
if rotations is None:
rotations = []
input_pdf, hocr_pdf, dummy = prepare_io(input_pdf, output_ext='.pdf', need_working_dir=False)
create_pdf(extract_hocr(open(text_hocr).read()), hocr_pdf, font, draft)
input1 = PdfFileReader(file(input_pdf, "rb"))
input2 = PdfFileReader(file(hocr_pdf, "rb"))
output = PdfFileWriter()
info = input1.getDocumentInfo()
if info:
infoDict = output._info.getObject()
infoDict.update(info)
for i in range(0, input1.getNumPages()):
orig_page = input1.getPage(i)
text_page = input2.getPage(i)
angle = _get_page_rotation(i)
if angle != 0:
print("Rotating page %d by %d degrees." % (i, angle), file=sys.stderr)
text_page = text_page.rotateClockwise(angle)
if draft:
below, above = orig_page, text_page
else:
below, above = text_page, orig_page
below.mergePage(above)
if angle != 0 and not draft:
print("Rotating back page %d by %d degrees." % (i, angle), file=sys.stderr)
below.rotateCounterClockwise(angle)
output.addPage(below)
outputStream = file(output_pdf, "wb")
output.write(outputStream)
outputStream.close()
os.remove(hocr_pdf)
return output_pdf
def pdf2hocr2pdf(input_file, output_file=None, ln='en', return_working_dir=False, extract_only_text=False, pdfopt=True, font='Courier', draft=False, **dummy):
"""
Return the text content in input_file.
@param ln is a two letter language code to give the OCR tool a hint.
@param return_working_dir if set to True, will return output_file path and the working_dir path, instead of deleting the working_dir. This is useful in case you need the intermediate images to build again a PDF.
"""
def _perform_rotate(working_dir, imagefile, angle):
"""Rotate imagefile of the corresponding angle. Creates a new file
with rotated.ppm."""
get_file_converter_logger().debug('Performing rotate on %s by %s degrees' % (imagefile, angle))
if not angle:
#execute_command('%s %s %s', CFG_PATH_CONVERT, os.path.join(working_dir, imagefile), os.path.join(working_dir, 'rotated-%s' % imagefile))
shutil.copy(os.path.join(working_dir, imagefile), os.path.join(working_dir, 'rotated.ppm'))
else:
execute_command(CFG_PATH_CONVERT, os.path.join(working_dir, imagefile), '-rotate', str(angle), '-depth', str(8), os.path.join(working_dir, 'rotated.ppm'))
return True
def _perform_deskew(working_dir):
"""Perform ocroscript deskew. Expect to work on rotated-imagefile.
Creates deskewed.ppm.
Return True if deskewing was fine."""
get_file_converter_logger().debug('Performing deskew')
try:
dummy, stderr = execute_command_with_stderr(CFG_PATH_OCROSCRIPT, os.path.join(CFG_ETCDIR, 'websubmit', 'file_converter_templates', 'deskew.lua'), os.path.join(working_dir, 'rotated.ppm'), os.path.join(working_dir, 'deskewed.ppm'))
if stderr.strip():
get_file_converter_logger().debug('Errors found during deskewing')
return False
else:
return True
except InvenioWebSubmitFileConverterError as err:
get_file_converter_logger().debug('Deskewing error: %s' % err)
return False
def _perform_recognize(working_dir):
"""Perform ocroscript recognize. Expect to work on deskewed.ppm.
Creates recognized.out Return True if recognizing was fine."""
get_file_converter_logger().debug('Performing recognize')
if extract_only_text:
output_mode = 'text'
else:
output_mode = 'hocr'
try:
dummy, stderr = execute_command_with_stderr(CFG_PATH_OCROSCRIPT, 'recognize', '--tesslanguage=%s' % ln, '--output-mode=%s' % output_mode, os.path.join(working_dir, 'deskewed.ppm'), filename_out=os.path.join(working_dir, 'recognize.out'))
if stderr.strip():
## There was some output on stderr
get_file_converter_logger().debug('Errors found in recognize.err')
return False
return not guess_ocropus_produced_garbage(os.path.join(working_dir, 'recognize.out'), not extract_only_text)
except InvenioWebSubmitFileConverterError as err:
get_file_converter_logger().debug('Recognizer error: %s' % err)
return False
def _perform_dummy_recognize(working_dir):
"""Return an empty text or an empty hocr referencing the image."""
get_file_converter_logger().debug('Performing dummy recognize')
if extract_only_text:
out = ''
else:
out = """<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"><head><meta content="ocr_line ocr_page" name="ocr-capabilities"/><meta content="en" name="ocr-langs"/><meta content="Latin" name="ocr-scripts"/><meta content="" name="ocr-microformats"/><title>OCR Output</title></head>
<body><div class="ocr_page" title="bbox 0 0 1 1; image deskewed.ppm">
</div></body></html>"""
open(os.path.join(working_dir, 'recognize.out'), 'w').write(out)
def _find_image_file(working_dir, imageprefix, page):
ret = '%s-%d.ppm' % (imageprefix, page)
if os.path.exists(os.path.join(working_dir, ret)):
return ret
ret = '%s-%02d.ppm' % (imageprefix, page)
if os.path.exists(os.path.join(working_dir, ret)):
return ret
ret = '%s-%03d.ppm' % (imageprefix, page)
if os.path.exists(os.path.join(working_dir, ret)):
return ret
ret = '%s-%04d.ppm' % (imageprefix, page)
if os.path.exists(os.path.join(working_dir, ret)):
return ret
ret = '%s-%05d.ppm' % (imageprefix, page)
if os.path.exists(os.path.join(working_dir, ret)):
return ret
ret = '%s-%06d.ppm' % (imageprefix, page)
if os.path.exists(os.path.join(working_dir, ret)):
return ret
## I guess we won't have documents with more than million pages
return None
def _ocr(tmp_output_file):
"""
Append to tmp_output_file the partial results of OCROpus recognize.
Return a list of rotations.
"""
page = 0
rotations = []
while True:
page += 1
get_file_converter_logger().debug('Page %d.' % page)
execute_command(CFG_PATH_PDFTOPPM, '-f', str(page), '-l', str(page), '-r', str(CFG_PPM_RESOLUTION), '-aa', 'yes', '-freetype', 'yes', input_file, os.path.join(working_dir, 'image'))
imagefile = _find_image_file(working_dir, 'image', page)
if imagefile == None:
break
for angle in (0, 180, 90, 270):
get_file_converter_logger().debug('Trying %d degrees...' % angle)
if _perform_rotate(working_dir, imagefile, angle) and _perform_deskew(working_dir) and _perform_recognize(working_dir):
rotations.append(angle)
break
else:
get_file_converter_logger().debug('Dummy recognize')
rotations.append(0)
_perform_dummy_recognize(working_dir)
open(tmp_output_file, 'a').write(open(os.path.join(working_dir, 'recognize.out')).read())
# clean
os.remove(os.path.join(working_dir, imagefile))
return rotations
if CFG_PATH_OCROSCRIPT:
if len(ln) == 2:
ln = CFG_TWO2THREE_LANG_CODES.get(ln, 'eng')
if extract_only_text:
input_file, output_file, working_dir = prepare_io(input_file, output_file, output_ext='.txt')
_ocr(output_file)
else:
input_file, tmp_output_hocr, working_dir = prepare_io(input_file, output_ext='.hocr')
rotations = _ocr(tmp_output_hocr)
if pdfopt:
input_file, tmp_output_pdf, dummy = prepare_io(input_file, output_ext='.pdf', need_working_dir=False)
tmp_output_pdf, output_file, dummy = prepare_io(tmp_output_pdf, output_file, output_ext='.pdf', need_working_dir=False)
pdf2pdfhocr(input_file, tmp_output_hocr, tmp_output_pdf, rotations=rotations, font=font, draft=draft)
pdf2pdfopt(tmp_output_pdf, output_file)
os.remove(tmp_output_pdf)
else:
input_file, output_file, dummy = prepare_io(input_file, output_file, output_ext='.pdf', need_working_dir=False)
pdf2pdfhocr(input_file, tmp_output_hocr, output_file, rotations=rotations, font=font, draft=draft)
clean_working_dir(working_dir)
return output_file
else:
raise InvenioWebSubmitFileConverterError("It's impossible to generate HOCR output from PDF. OCROpus is not available.")
def pdf2text(input_file, output_file=None, perform_ocr=True, ln='en', **dummy):
"""
Return the text content in input_file.
"""
input_file, output_file, dummy = prepare_io(input_file, output_file, '.txt', need_working_dir=False)
execute_command(CFG_PATH_PDFTOTEXT, '-enc', 'UTF-8', '-eol', 'unix', '-nopgbrk', input_file, output_file)
if perform_ocr and can_perform_ocr():
ocred_output = pdf2hocr2pdf(input_file, ln=ln, extract_only_text=True)
try:
output = open(output_file, 'a')
for row in open(ocred_output):
output.write(row)
output.close()
finally:
silent_remove(ocred_output)
return output_file
def txt2text(input_file, output_file=None, **dummy):
"""
Return the text content in input_file
"""
input_file, output_file, dummy = prepare_io(input_file, output_file, '.txt', need_working_dir=False)
shutil.copy(input_file, output_file)
return output_file
def html2text(input_file, output_file=None, **dummy):
"""
Return the text content of an HTML/XML file.
"""
class HTMLStripper(HTMLParser.HTMLParser):
def __init__(self, output_file):
HTMLParser.HTMLParser.__init__(self)
self.output_file = output_file
def handle_entityref(self, name):
if name in entitydefs:
self.output_file.write(entitydefs[name].decode('latin1').encode('utf8'))
def handle_data(self, data):
if data.strip():
self.output_file.write(_RE_CLEAN_SPACES.sub(' ', data))
def handle_charref(self, data):
try:
self.output_file.write(unichr(int(data)).encode('utf8'))
except:
pass
def close(self):
self.output_file.close()
HTMLParser.HTMLParser.close(self)
input_file, output_file, dummy = prepare_io(input_file, output_file, '.txt', need_working_dir=False)
html_stripper = HTMLStripper(open(output_file, 'w'))
for line in open(input_file):
html_stripper.feed(line)
html_stripper.close()
return output_file
def djvu2text(input_file, output_file=None, **dummy):
"""
Return the text content in input_file.
"""
input_file, output_file, dummy = prepare_io(input_file, output_file, '.txt', need_working_dir=False)
execute_command(CFG_PATH_DJVUTXT, input_file, output_file)
return output_file
def djvu2ps(input_file, output_file=None, level=2, compress=True, **dummy):
"""
Convert a djvu into a .ps[.gz]
"""
if compress:
input_file, output_file, working_dir = prepare_io(input_file, output_file, output_ext='.ps.gz')
try:
execute_command(CFG_PATH_DJVUPS, input_file, os.path.join(working_dir, 'output.ps'))
execute_command(CFG_PATH_GZIP, '-c', os.path.join(working_dir, 'output.ps'), filename_out=output_file)
finally:
clean_working_dir(working_dir)
else:
try:
input_file, output_file, working_dir = prepare_io(input_file, output_file, output_ext='.ps')
execute_command(CFG_PATH_DJVUPS, '-level=%i' % level, input_file, output_file)
finally:
clean_working_dir(working_dir)
return output_file
def tiff2pdf(input_file, output_file=None, pdfopt=True, pdfa=True, perform_ocr=True, **args):
"""
Convert a .tiff into a .pdf
"""
if pdfa or pdfopt or perform_ocr:
input_file, output_file, working_dir = prepare_io(input_file, output_file, '.pdf')
try:
partial_output = os.path.join(working_dir, 'output.pdf')
execute_command(CFG_PATH_TIFF2PDF, '-o', partial_output, input_file)
if perform_ocr:
pdf2hocr2pdf(partial_output, output_file, pdfopt=pdfopt, **args)
elif pdfa:
pdf2pdfa(partial_output, output_file, pdfopt=pdfopt, **args)
else:
pdfopt(partial_output, output_file)
finally:
clean_working_dir(working_dir)
else:
input_file, output_file, dummy = prepare_io(input_file, output_file, '.pdf', need_working_dir=False)
execute_command(CFG_PATH_TIFF2PDF, '-o', output_file, input_file)
return output_file
def pstotext(input_file, output_file=None, **dummy):
"""
Convert a .ps[.gz] into text.
"""
input_file, output_file, working_dir = prepare_io(input_file, output_file, '.txt')
try:
if input_file.endswith('.gz'):
new_input_file = os.path.join(working_dir, 'input.ps')
execute_command(CFG_PATH_GUNZIP, '-c', input_file, filename_out=new_input_file)
input_file = new_input_file
execute_command(CFG_PATH_PSTOTEXT, '-output', output_file, input_file)
finally:
clean_working_dir(working_dir)
return output_file
def gzip(input_file, output_file=None, **dummy):
"""
Compress a file.
"""
input_file, output_file, dummy = prepare_io(input_file, output_file, '.gz', need_working_dir=False)
execute_command(CFG_PATH_GZIP, '-c', input_file, filename_out=output_file)
return output_file
def gunzip(input_file, output_file=None, **dummy):
"""
Uncompress a file.
"""
from invenio.legacy.bibdocfile.api import decompose_file
input_ext = decompose_file(input_file, skip_version=True)[2]
if input_ext.endswith('.gz'):
input_ext = input_ext[:-len('.gz')]
else:
input_ext = None
input_file, output_file, dummy = prepare_io(input_file, output_file, input_ext, need_working_dir=False)
execute_command(CFG_PATH_GUNZIP, '-c', input_file, filename_out=output_file)
return output_file
def prepare_io(input_file, output_file=None, output_ext=None, need_working_dir=True):
"""Clean input_file and the output_file."""
from invenio.legacy.bibdocfile.api import decompose_file, normalize_format
output_ext = normalize_format(output_ext)
get_file_converter_logger().debug('Preparing IO for input=%s, output=%s, output_ext=%s' % (input_file, output_file, output_ext))
if output_ext is None:
if output_file is None:
output_ext = '.tmp'
else:
output_ext = decompose_file(output_file, skip_version=True)[2]
if output_file is None:
try:
(fd, output_file) = tempfile.mkstemp(suffix=output_ext, dir=CFG_TMPDIR)
os.close(fd)
except IOError as err:
raise InvenioWebSubmitFileConverterError("It's impossible to create a temporary file: %s" % err)
else:
output_file = os.path.abspath(output_file)
if os.path.exists(output_file):
os.remove(output_file)
if need_working_dir:
try:
working_dir = tempfile.mkdtemp(dir=CFG_TMPDIR, prefix='conversion')
except IOError as err:
raise InvenioWebSubmitFileConverterError("It's impossible to create a temporary directory: %s" % err)
input_ext = decompose_file(input_file, skip_version=True)[2]
new_input_file = os.path.join(working_dir, 'input' + input_ext)
shutil.copy(input_file, new_input_file)
input_file = new_input_file
else:
working_dir = None
input_file = os.path.abspath(input_file)
get_file_converter_logger().debug('IO prepared: input_file=%s, output_file=%s, working_dir=%s' % (input_file, output_file, working_dir))
return (input_file, output_file, working_dir)
def clean_working_dir(working_dir):
"""
Remove the working_dir.
"""
get_file_converter_logger().debug('Cleaning working_dir: %s' % working_dir)
shutil.rmtree(working_dir)
def execute_command(*args, **argd):
"""Wrapper to run_process_with_timeout."""
get_file_converter_logger().debug("Executing: %s" % (args, ))
args = [str(arg) for arg in args]
sudo = argd.get('sudo') or None
if sudo:
pass
# May be forbidden by sudo
# args = ['CFG_OPENOFFICE_TMPDIR="%s"' % CFG_OPENOFFICE_TMPDIR] + args
else:
os.putenv('CFG_OPENOFFICE_TMPDIR', CFG_OPENOFFICE_TMPDIR)
res, stdout, stderr = run_process_with_timeout(args,
cwd=argd.get('cwd'),
filename_out=argd.get('filename_out'),
filename_err=argd.get('filename_err'),
sudo=sudo)
get_file_converter_logger().debug('res: %s, stdout: %s, stderr: %s' % (res, stdout, stderr))
if res != 0:
message = "ERROR: Error in running %s\n stdout:\n%s\nstderr:\n%s\n" % (args, stdout, stderr)
get_file_converter_logger().error(message)
raise InvenioWebSubmitFileConverterError(message)
return stdout
def execute_command_with_stderr(*args, **argd):
"""Wrapper to run_process_with_timeout."""
get_file_converter_logger().debug("Executing: %s" % (args, ))
res, stdout, stderr = run_process_with_timeout(args, cwd=argd.get('cwd'), filename_out=argd.get('filename_out'), sudo=argd.get('sudo'))
if res != 0:
message = "ERROR: Error in running %s\n stdout:\n%s\nstderr:\n%s\n" % (args, stdout, stderr)
get_file_converter_logger().error(message)
raise InvenioWebSubmitFileConverterError(message)
return stdout, stderr
def silent_remove(path):
"""Remove without errors a path."""
if os.path.exists(path):
try:
os.remove(path)
except OSError:
pass
__CONVERSION_MAP = get_conversion_map()
def main_cli():
"""
main function when the library behaves as a normal CLI tool.
"""
from invenio.legacy.bibdocfile.api import normalize_format
parser = OptionParser()
parser.add_option("-c", "--convert", dest="input_name",
help="convert the specified FILE", metavar="FILE")
parser.add_option("-d", "--debug", dest="debug", action="store_true", help="Enable debug information")
parser.add_option("--special-pdf2hocr2pdf", dest="ocrize", help="convert the given scanned PDF into a PDF with OCRed text", metavar="FILE")
parser.add_option("-f", "--format", dest="output_format", help="the desired output format", metavar="FORMAT")
parser.add_option("-o", "--output", dest="output_name", help="the desired output FILE (if not specified a new file will be generated with the desired output format)")
parser.add_option("--without-pdfa", action="store_false", dest="pdf_a", default=True, help="don't force creation of PDF/A PDFs")
parser.add_option("--without-pdfopt", action="store_false", dest="pdfopt", default=True, help="don't force optimization of PDFs files")
parser.add_option("--without-ocr", action="store_false", dest="ocr", default=True, help="don't force OCR")
parser.add_option("--can-convert", dest="can_convert", help="display all the possible format that is possible to generate from the given format", metavar="FORMAT")
parser.add_option("--is-ocr-needed", dest="check_ocr_is_needed", help="check if OCR is needed for the FILE specified", metavar="FILE")
parser.add_option("-t", "--title", dest="title", help="specify the title (used when creating PDFs)", metavar="TITLE")
parser.add_option("-l", "--language", dest="ln", help="specify the language (used when performing OCR, e.g. en, it, fr...)", metavar="LN", default='en')
(options, dummy) = parser.parse_args()
if options.debug:
from logging import basicConfig
basicConfig()
get_file_converter_logger().setLevel(DEBUG)
if options.can_convert:
if options.can_convert:
input_format = normalize_format(options.can_convert)
if input_format == '.pdf':
if can_pdfopt(True):
print("PDF linearization supported")
else:
print("No PDF linearization support")
if can_pdfa(True):
print("PDF/A generation supported")
else:
print("No PDF/A generation support")
if can_perform_ocr(True):
print("OCR supported")
else:
print("OCR not supported")
print('Can convert from "%s" to:' % input_format[1:], end=' ')
for output_format in __CONVERSION_MAP:
if can_convert(input_format, output_format):
print('"%s"' % output_format[1:], end=' ')
print()
elif options.check_ocr_is_needed:
print("Checking if OCR is needed on %s..." % options.check_ocr_is_needed, end=' ')
sys.stdout.flush()
if guess_is_OCR_needed(options.check_ocr_is_needed):
print("needed.")
else:
print("not needed.")
elif options.ocrize:
try:
output = pdf2hocr2pdf(options.ocrize, output_file=options.output_name, title=options.title, ln=options.ln)
print("Output stored in %s" % output)
except InvenioWebSubmitFileConverterError as err:
print("ERROR: %s" % err)
sys.exit(1)
else:
try:
if not options.output_name and not options.output_format:
parser.error("Either --format, --output should be specified")
if not options.input_name:
parser.error("An input should be specified!")
output = convert_file(options.input_name, output_file=options.output_name, output_format=options.output_format, pdfopt=options.pdfopt, pdfa=options.pdf_a, title=options.title, ln=options.ln)
print("Output stored in %s" % output)
except InvenioWebSubmitFileConverterError as err:
print("ERROR: %s" % err)
sys.exit(1)
if __name__ == "__main__":
main_cli()
diff --git a/invenio/legacy/websubmit/file_stamper.py b/invenio/legacy/websubmit/file_stamper.py
index e286e5c3f..9e640e10d 100644
--- a/invenio/legacy/websubmit/file_stamper.py
+++ b/invenio/legacy/websubmit/file_stamper.py
@@ -1,1605 +1,1698 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2008, 2009, 2010, 2011 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
"""This is websubmit_file_stamper.py
This tool is used to create a stamped version of a PDF file.
+ Python API:
Please see stamp_file().
+ CLI API:
$ python ~invenio/lib/python/invenio/websubmit_file_stamper.py \\
--latex-template=demo-stamp-left.tex \\
--latex-template-var='REPORTNUMBER=TEST-THESIS-2008-019' \\
--latex-template-var='DATE=27/02/2008' \\
--stamp='first' \\
--layer='background' \\
--output-file=testfile_stamped.pdf \\
testfile.pdf
"""
__revision__ = "$Id$"
import getopt, sys, re, os, time, shutil, tempfile
from six import iteritems
from invenio.config import \
CFG_PATH_PS2PDF, \
CFG_PATH_GFILE,\
CFG_PATH_PDFLATEX
from invenio.ext.logging import register_exception
from invenio.config import CFG_TMPDIR
from invenio.config import CFG_ETCDIR
CFG_WEBSUBMIT_FILE_STAMPER_TEMPLATES_DIR = \
pkg_resources.resource_filename('invenio.legacy.websubmit', 'file_stamper_templates')
from invenio.config import CFG_PATH_PDFTK
from invenio.utils.shell import escape_shell_arg
from invenio.legacy.websubmit.config import InvenioWebSubmitFileStamperError
## ***** Functions related to the creation of the PDF Stamp file: *****
re_latex_includegraphics = re.compile('\\includegraphics\[.*?\]\{(?P<image>.*?)\}')
def copy_template_files_to_stampdir(path_workingdir, latex_template):
"""In order to stamp a PDF fulltext file, LaTeX is used to create a
"stamp" page that is then merged with the fulltext PDF.
The stamp page is created in a temporary stamp "working directory".
This means that the LaTeX file and its image files must be copied
locally into this working directory. This function handles
copying them into the working directory.
Note: Copying of the LaTeX template and its included image files is
fairly naive and assumes that it is a very basic LaTeX file
consisting of a main file and any included graphics.
No other included file items will be copied.
Also note that the order of searching for the LaTeX file and
its associated graphics is as follows:
+ If the templatename provided has a path attached to it,
look here first;
+ If there is no path, look in the current dir.
+ If there is no template in the current dir, look in
~invenio/etc/websubmit/latex
+ Images included within the LaTeX file are sought in the
same way. Full path is used if provided; if not, current
dir and failing that ~invenio/etc/websubmit/latex.
@param path_workingdir: (string) - the working directory into which the
latex templates should be copied.
@param latex_template: (string) - the name of the LaTeX template to copy
to the working dir.
"""
## Get the "base name" of the latex template:
(template_path, template_name) = os.path.split(latex_template)
if template_path != "":
## A full path to the template was provided. We look for it there.
## Test to see whether the template is a real file and is readable:
if os.access("%s/%s" % (template_path, template_name), os.R_OK):
## Template is readable. Copy it locally to the working directory:
try:
shutil.copyfile("%s/%s" % (template_path, template_name), \
"%s/%s" % (path_workingdir, template_name))
except IOError:
## Unable to copy the LaTeX template file to the
## working directory:
msg = """Error: Unable to copy LaTeX file [%s/%s] to """ \
"""working directory for stamping [%s].""" \
% (template_path, template_name, path_workingdir)
raise InvenioWebSubmitFileStamperError(msg)
else:
## Unable to read the template file:
msg = """Error: Unable to copy LaTeX file [%s/%s] to """ \
"""working directory for stamping [%s]. (File not """ \
"""readable.)""" \
% (template_path, template_name, path_workingdir)
raise InvenioWebSubmitFileStamperError(msg)
else:
## There is no path to the template file.
## Look for it first in the current working directory, then in
## ~invenio/websubmit/latex;
## If not found in either, give up.
if os.access("%s" % (template_name), os.F_OK):
## Template has been found in the current working directory.
## Copy it locally to the stamping working directory:
try:
shutil.copyfile("%s" % (template_name), \
"%s/%s" % (path_workingdir, template_name))
except IOError:
## Unable to copy the LaTeX template file to the
## working stamping directory:
msg = """Error: Unable to copy LaTeX file [%s] to """ \
"""working directory for stamping [%s].""" \
% (template_name, path_workingdir)
raise InvenioWebSubmitFileStamperError(msg)
elif os.access("%s/%s" % (CFG_WEBSUBMIT_FILE_STAMPER_TEMPLATES_DIR, \
template_name), os.F_OK):
## The template has been found in WebSubmit's latex templates
## directory. Copy it locally to the stamping working directory:
try:
shutil.copyfile("%s/%s" \
% (CFG_WEBSUBMIT_FILE_STAMPER_TEMPLATES_DIR, \
template_name), \
"%s/%s" % (path_workingdir, template_name))
except IOError:
## Unable to copy the LaTeX template file to the
## working stamping directory:
msg = """Error: Unable to copy LaTeX file [%s/%s] to """ \
"""working directory for stamping [%s].""" \
% (CFG_WEBSUBMIT_FILE_STAMPER_TEMPLATES_DIR, \
template_name, path_workingdir)
raise InvenioWebSubmitFileStamperError(msg)
else:
## Now that the template has been found, set the "template
## path" to the WebSubmit latex templates directory:
template_path = CFG_WEBSUBMIT_FILE_STAMPER_TEMPLATES_DIR
else:
## Unable to locate the latex template.
msg = """Error: Unable to locate LaTeX file [%s].""" % template_name
raise InvenioWebSubmitFileStamperError(msg)
## Now that the LaTeX template file has been copied locally, extract
## the names of graphics files to be included in the resulting
## document and attempt to copy them to the working "stamp" directory:
template_desc = file(os.path.join(path_workingdir, template_name))
template_code = template_desc.read()
template_desc.close()
graphic_names = [match_obj.group('image') for match_obj in \
re_latex_includegraphics.finditer(template_code)]
## Copy each include-graphic extracted from the template
## into the working stamp directory:
for graphic in graphic_names:
## Remove any leading/trailing whitespace:
graphic = graphic.strip()
## Get the path and "base name" of the included graphic:
(graphic_path, graphic_name) = os.path.split(graphic)
## If there is a graphic_name to work with, try copy the file:
if graphic_name != "":
if graphic_path != "":
## The graphic is included from an absolute path:
if os.access("%s/%s" % (graphic_path, graphic_name), os.F_OK):
try:
shutil.copyfile("%s/%s" % (graphic_path, \
graphic_name), \
"%s/%s" % (path_workingdir, \
graphic_name))
except IOError:
## Unable to copy the LaTeX template file to
## the current directory
msg = """Unable to stamp file. There was """ \
"""a problem when trying to copy an image """ \
"""[%s/%s] included by the LaTeX template""" \
""" [%s].""" \
% (graphic_path, graphic_name, template_name)
raise InvenioWebSubmitFileStamperError(msg)
else:
msg = """Unable to locate an image [%s/%s] included""" \
""" by the LaTeX template file [%s].""" \
% (graphic_path, graphic_name, template_name)
raise InvenioWebSubmitFileStamperError(msg)
else:
## The graphic is included from a relative path. Try to obtain
## it from the same directory that the latex template file was
## taken from:
if template_path != "":
## Since template path is not empty, try to get the images
## from that location:
if os.access("%s/%s" % (template_path, graphic_name), \
os.F_OK):
try:
shutil.copyfile("%s/%s" % (template_path, \
graphic_name), \
"%s/%s" % (path_workingdir, \
graphic_name))
except IOError:
## Unable to copy the LaTeX template file to
## the current directory
msg = """Unable to stamp file. There was """ \
"""a problem when trying to copy images """ \
"""included by the LaTeX template."""
raise InvenioWebSubmitFileStamperError(msg)
else:
msg = """Unable to locate an image [%s] included""" \
""" by the LaTeX template file [%s].""" \
% (graphic_name, template_name)
raise InvenioWebSubmitFileStamperError(msg)
else:
## There is no template path. Try to get the images from
## current dir:
if os.access("%s" % graphic_name, os.F_OK):
try:
shutil.copyfile("%s" % graphic_name, \
"%s/%s" % (path_workingdir, \
graphic_name))
except IOError:
## Unable to copy the LaTeX template file to
## the current directory
msg = """Unable to stamp file. There was """ \
"""a problem when trying to copy images """ \
"""included by the LaTeX template."""
raise InvenioWebSubmitFileStamperError(msg)
else:
msg = """Unable to locate an image [%s] included""" \
""" by the LaTeX template file [%s].""" \
% (graphic_name, template_name)
raise InvenioWebSubmitFileStamperError(msg)
## Return the basename of the template so that it can be used to create
## the PDF stamp file:
return template_name
def create_final_latex_template(working_dirname, \
latex_template, \
latex_template_var):
"""In the working directory, create a copy of the the orginal
latex template with all the possible xxx--xxx in the template
replaced with the values identified by the keywords in the
latex_template_var dictionary.
@param working_dirname: (string) the working directory used for the
creation of the PDF stamp file.
@latex_template: (string) name of the latex template before it has
been parsed for replacements.
@latex_template_var: (dict) dictionnary whose keys are the string to
replace in latex_template and values are the replacement content
@return: name of the final latex template (after replacements)
"""
## Regexp used for finding a substitution line in the original template:
re_replacement = re.compile("""XXX-(.+?)-XXX""")
## Now, read-in the local copy of the template and parse it line-by-line,
## replacing any ocurrences of "XXX-SEARCHWORD-XXX" with either:
##
## (a) The value from the "replacements" dictionary;
## (b) Nothing if there was no search-term in the dictionary;
try:
## Open the original latex template for reading:
fpread = open("%s/%s" \
% (working_dirname, latex_template), "r")
## Open a file to contain the "parsed" latex template:
fpwrite = open("%s/create%s" \
% (working_dirname, latex_template), "w")
for line in fpread.readlines():
## For each line in the template file, test for
## substitution-markers:
replacement_markers = re_replacement.finditer(line)
## For each replacement-pattern detected in this line, process it:
for replacement_marker in replacement_markers:
## Found a match:
search_term = replacement_marker.group(1)
try:
## Get the replacement-term for this match
## from the dictionary
replacement_term = latex_template_var[search_term]
except KeyError:
## This search-term was not in the list of replacements
## to be made. It should be replaced with an empty string
## in the template:
line = line[0:replacement_marker.start()] + \
line[replacement_marker.end():]
else:
## Is the replacement term of the form date(XXXX)? If yes,
## take it literally and generate a pythonic date with it:
if replacement_term.find("date(") == 0 \
and replacement_term[-1] == ")":
## Take the date format string, use it to
## generate today's date
date_format = replacement_term[5:-1].strip('\'"')
try:
replacement = time.strftime(date_format, \
time.localtime())
except TypeError:
## Bad date format
replacement = ""
elif replacement_term.find("include(") == 0 \
and replacement_term[-1] == ")":
## Replacement term is a directive to include a file
## in the LaTeX template:
replacement = replacement_term[8:-1].strip('\'"')
else:
## Take replacement_term as a literal string
## to be inserted into the template at this point.
replacement = replacement_term
## Now substitute replacement into the line of the template:
line = line[0:replacement_marker.start()] + replacement \
+ line[replacement_marker.end():]
## Write the modified line to the new template:
fpwrite.write(line)
fpwrite.flush()
## Close up the template files and unlink the original:
fpread.close()
fpwrite.close()
except IOError:
msg = "Unable to read LaTeX template [%s/%s]. Cannot Stamp File" \
% (working_dirname, latex_template)
raise InvenioWebSubmitFileStamperError(msg)
## Return the name of the LaTeX template to be used:
return "create%s" % latex_template
def escape_latex_meta_characters(text):
"""The following are LaTeX meta characters that must be escaped with a
backslash:
# $ % & _ { }
This function therefore takes a string as input and does a simple
replace of these characters with escaped versions.
@param text: (string) - the string to be escaped.
@return: (string) - the string in which the LaTeX meta characters
have been escaped.
"""
text = text.replace('#', '\#')
text = text.replace('$', '\$')
text = text.replace('%', '\%')
text = text.replace('&', '\&')
text = text.replace('_', '\_')
text = text.replace('{', '\{')
text = text.replace('}', '\}')
return text
def escape_latex_template_vars(template_vars, strict=False):
"""Take a dictionary of LaTeX template variables/values and escape LaTeX
meta characters in some of them, or all of them depending upon whether
a call is made in strict mode (if strict is set, ALL values are
escaped.)
Operating in non-strict mode, the rules for escaping are as follows:
* If the string does not contain $ { or }, it must be escaped.
* If the string contains $, then there must be an even number of
these. If the count is even, do not escape. Else, escape.
* If the string contains { or }, it must be balanced with a
counterpart. That's to say that the count of "{" must match the
count of "}". If it does, do not escape. Else, escape.
@param template_vars: (dictionary) - the LaTeX template variables and
their values.
@param strict: (boolean) - a flag indicating whether or not to
operate in strict mode. Strict mode means that all values are
escaped regardless of whether or not they are considered to be
"good" LaTeX.
@return: (dictionary) - the LaTeX template variables with their
values escaped.
"""
## Make a copy of the LaTeX template variables so as not to corrupt
## the original:
working_template_vars = template_vars.copy()
##
## For each of the variables, escape LaTeX meta characteras in the
## value according to the strict flag:
varnames = working_template_vars.keys()
for varname in varnames:
escape_value = False
varval = working_template_vars[varname]
## We don't want to escape values that are date or include directives
## so unfortunately, this if is needed here:
if (varval.find("date(") == 0 or varval.find("include(") == 0) and \
varval[-1] == ")":
## This is a date or include directive:
continue
## Count the number of "$", "{" and "}" in it. If any are present,
## they should be balanced. If so, we will assume that they are
## wanted and that the LaTeX in the string is good.
## If, however, they are not balanced, we will assume that they are
## not valid LaTeX commands and that the string should be escaped.
## If they are not present at all, we assume that the string should
## be escaped.
if "$" in varval and varval.count("$") % 2 != 0:
## $ is present, but not in an even number. This string must
## be escaped:
escape_value = True
elif "{" in varval or "}" in varval:
## "{" and/or "}" is in the value string. Count each of them.
## If they are not matched one to one, consider the string to be
## in need of escaping:
if varval.count("{") != varval.count("}"):
escape_value = True
elif "$" not in varval and "{" not in varval and "}" not in varval:
## Since none of $ { } are in the string, it should be escaped
## to be safe:
escape_value = True
##
if strict:
## If operating in strict mode, escape everything whatever the
## results of the above tests:
escape_value = True
## If the value is to be escaped, go ahead and do so:
if escape_value:
escaped_varval = escape_latex_meta_characters(varval)
working_template_vars[varname] = escaped_varval
## Return the "escaped" LaTeX template variables:
return working_template_vars
def create_pdf_stamp(path_workingdir, latex_template, latex_template_var):
"""Retrieve the LaTeX (and associated) files and use them to create a
PDF "Stamp" file that can be merged with the main file.
The PDF stamp is created in a temporary working directory.
@param path_workingdir: (string) the path to the working directory
that should be used for creating the PDF stamp file.
@param latex_template: (string) - the name of the latex template
to be used for the creation of the stamp.
@param latex_template_var: (dictionary) - key-value pairs of strings
to be sought and replaced within the latex template.
@return: (string) - the name of the PDF stamp file.
"""
## Copy the LaTeX (and helper) files should be copied into the working dir:
template_name = copy_template_files_to_stampdir(path_workingdir, \
latex_template)
##
####
## Make a first attempt at the template PDF creation, escaping the variables
## in non-strict mode:
escaped_latex_template_var = escape_latex_template_vars(latex_template_var)
## Now that the latex template and its helper files have been retrieved,
## the Stamp PDF can be created.
final_template = create_final_latex_template(path_workingdir, \
template_name, \
escaped_latex_template_var)
##
## The name that will be givem to the PDF stamp file:
pdf_stamp_name = "%s.pdf" % os.path.splitext(final_template)[0]
## Now, build the Stamp PDF from the LaTeX template:
cmd_latex = """cd %(workingdir)s; %(path_pdflatex)s """ \
"""-interaction=batchmode """ \
"""%(template-path)s > /dev/null 2>&1""" \
% { 'template-path' : escape_shell_arg("%s/%s" \
% (path_workingdir, final_template)),
'workingdir' : path_workingdir,
'path_pdflatex' : CFG_PATH_PDFLATEX,
}
## Log the latex command
os.system("""echo %s > %s""" % (escape_shell_arg(cmd_latex), \
escape_shell_arg("%s/latex_cmd_first_try" \
% path_workingdir)))
## Run the latex command
errcode_latex = os.system("%s" % cmd_latex)
## Was the PDF stamp file successfully created without error?
if errcode_latex:
## No it wasn't. Perhaps there was a problem with some of the variable
## values that we substituted into the template?
## To be certain, try to create the PDF one more time - this time
## escaping all of the variable values.
##
## Unlink the PDF file if one was created on the previous attempt:
if os.access("%s/%s" % (path_workingdir, pdf_stamp_name), os.F_OK):
try:
os.unlink("%s/%s" % (path_workingdir, pdf_stamp_name))
except OSError:
## Unable to unlink the PDF file.
err_msg = "Unable to unlink the PDF stamp file [%s]. " \
"Stamping has failed." \
% pdf_stamp_name
register_exception(prefix=err_msg)
raise InvenioWebSubmitFileStamperError(err_msg)
##
## Unlink the LaTeX template file that was created with the previously
## escaped variables:
if os.access("%s/%s" % (path_workingdir, final_template), os.F_OK):
try:
os.unlink("%s/%s" % (path_workingdir, final_template))
except OSError:
## Unable to unlink the LaTeX file.
err_msg = "Unable to unlink the LaTeX stamp template file " \
"[%s]. Stamping has failed." \
% final_template
register_exception(prefix=err_msg)
raise InvenioWebSubmitFileStamperError(err_msg)
##
####
## Make another attempt at the template PDF creation, this time escaping
## the variables in strict mode:
escaped_latex_template_var = \
escape_latex_template_vars(latex_template_var, strict=True)
## Now that the latex template and its helper files have been retrieved,
## the Stamp PDF can be created.
final_template = create_final_latex_template(path_workingdir, \
template_name, \
escaped_latex_template_var)
##
## The name that will be givem to the PDF stamp file:
pdf_stamp_name = "%s.pdf" % os.path.splitext(final_template)[0]
## Now, build the Stamp PDF from the LaTeX template:
cmd_latex = """cd %(workingdir)s; %(path_pdflatex)s """ \
"""-interaction=batchmode """ \
"""%(template-path)s > /dev/null 2>&1""" \
% { 'template-path' : escape_shell_arg("%s/%s" \
% (path_workingdir, final_template)),
'workingdir' : path_workingdir,
'path_pdflatex' : CFG_PATH_PDFLATEX,
}
## Log the latex command
os.system("""echo %s > %s""" \
% (escape_shell_arg(cmd_latex), \
escape_shell_arg("%s/latex_cmd_second_try" \
% path_workingdir)))
## Run the latex command
errcode_latex = os.system("%s" % cmd_latex)
## Was the PDF stamp file successfully created?
if errcode_latex or \
not os.access("%s/%s" % (path_workingdir, pdf_stamp_name), os.F_OK):
## It was not possible to create the PDF stamp file. Fail.
msg = """Error: Unable to create a PDF stamp file."""
raise InvenioWebSubmitFileStamperError(msg)
## Return the name of the PDF stamp file:
return pdf_stamp_name
## ***** Functions related to the actual stamping of the file: *****
def apply_stamp_cover_page(path_workingdir, \
stamp_file_name, \
subject_file, \
output_file):
"""Carry out the stamping:
This function adds a cover-page to the file.
@param path_workingdir: (string) - the path to the working directory
that contains all of the files needed for the stamping process to be
carried out.
@param stamp_file_name: (string) - the name of the PDF stamp file (i.e.
the cover-page itself).
@param subject_file: (string) - the name of the file to be stamped.
@param output_file: (string) - the name of the final "stamped" file (i.e.
that with the cover page added) that will be written in the working
directory after the function has ended.
"""
## Build the stamping command:
cmd_add_cover_page = \
"""%(pdftk)s %(cover-page-path)s """ \
"""%(file-to-stamp-path)s """ \
"""cat output %(stamped-file-path)s """ \
"""2>/dev/null"""% \
{ 'pdftk' : CFG_PATH_PDFTK,
'cover-page-path' : escape_shell_arg("%s/%s" \
% (path_workingdir, \
stamp_file_name)),
'file-to-stamp-path' : escape_shell_arg("%s/%s" \
% (path_workingdir, \
subject_file)),
'stamped-file-path' : escape_shell_arg("%s/%s" \
% (path_workingdir, \
output_file)),
}
## Execute the stamping command:
errcode_add_cover_page = os.system(cmd_add_cover_page)
## Was the PDF merged with the coverpage without error?
if errcode_add_cover_page:
## There was a problem:
msg = "Error: Unable to stamp file [%s/%s]. There was an error when " \
"trying to add the cover page [%s/%s] to the file. Stamping " \
"has failed." \
% (path_workingdir, \
subject_file, \
path_workingdir, \
stamp_file_name)
raise InvenioWebSubmitFileStamperError(msg)
def apply_stamp_first_page(path_workingdir, \
stamp_file_name, \
subject_file, \
output_file, \
stamp_layer):
"""Carry out the stamping:
This function adds a stamp to the first page of the file.
@param path_workingdir: (string) - the path to the working directory
that contains all of the files needed for the stamping process to be
carried out.
@param stamp_file_name: (string) - the name of the PDF stamp file (i.e.
the stamp itself).
@param subject_file: (string) - the name of the file to be stamped.
@param output_file: (string) - the name of the final "stamped" file that
will be written in the working directory after the function has ended.
@param stamp_layer: (string) - the layer to consider when stamping
"""
## Since only the first page of the subject file is to be stamped,
## it's safest to separate this into its own temporary file, stamp
## it, then re-merge it with the remaining pages of the original
## document. In this way, the PDF to be stamped will probably be
## simpler (pages with complex figures and tables will probably be
## avoided) and the process will hopefully have a smaller chance of
## failure.
##
## First of all, separate the first page of the subject file into a
## temporary document:
##
## Name to be given to the first page of the document:
output_file_first_page = "p1-%s" % output_file
## Name to be given to the first page of the document once it has
## been stamped:
stamped_output_file_first_page = "stamped-%s" % output_file_first_page
## Perform the separation:
cmd_get_first_page = \
"%(pdftk)s A=%(file-to-stamp-path)s " \
"cat A1 output %(first-page-path)s " \
"2>/dev/null" \
% { 'pdftk' : CFG_PATH_PDFTK,
'file-to-stamp-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, subject_file)),
'first-page-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
output_file_first_page)),
}
errcode_get_first_page = os.system(cmd_get_first_page)
## Check that the separation was successful:
if errcode_get_first_page or \
not os.access("%s/%s" % (path_workingdir, \
output_file_first_page), os.F_OK):
## Separation was unsuccessful. Fail.
msg = "Error: Unable to stamp file [%s/%s] - it wasn't possible to " \
"separate the first page from the rest of the document. " \
"Stamping has failed." \
% (path_workingdir, subject_file)
raise InvenioWebSubmitFileStamperError(msg)
## Now stamp the first page:
cmd_stamp_first_page = \
"%(pdftk)s %(first-page-path)s %(stamp_layer)s " \
"%(stamp-file-path)s output " \
"%(stamped-first-page-path)s 2>/dev/null" \
% { 'pdftk' : CFG_PATH_PDFTK,
'first-page-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
output_file_first_page)),
'stamp-file-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
stamp_file_name)),
'stamped-first-page-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
stamped_output_file_first_page)),
'stamp_layer' : stamp_layer == 'foreground' and 'stamp' or 'background'
}
errcode_stamp_first_page = os.system(cmd_stamp_first_page)
## Check that the first page was stamped successfully:
if errcode_stamp_first_page or \
not os.access("%s/%s" % (path_workingdir, \
stamped_output_file_first_page), os.F_OK):
## Unable to stamp the first page. Fail.
msg = "Error: Unable to stamp the file [%s/%s] - it was not possible " \
"to add the stamp to the first page. Stamping has failed." \
% (path_workingdir, subject_file)
raise InvenioWebSubmitFileStamperError(msg)
## Now that the first page has been stamped successfully, merge it with
## the remaining pages of the original file:
cmd_merge_stamped_and_original_files = \
"%(pdftk)s A=%(stamped-first-page-path)s " \
"B=%(original-file-path)s cat A1 B2-end output " \
"%(stamped-file-path)s 2>/dev/null" \
% { 'pdftk' : CFG_PATH_PDFTK,
'stamped-first-page-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
stamped_output_file_first_page)),
'original-file-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
subject_file)),
'stamped-file-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
output_file)),
}
errcode_merge_stamped_and_original_files = \
os.system(cmd_merge_stamped_and_original_files)
## Check to see whether the command exited with an error:
if errcode_merge_stamped_and_original_files:
## There was an error when trying to merge the stamped first-page
## with pages 2 onwards of the original file. One possible
## explanation for this could be that the original file only had
## one page (in which case trying to reference pages 2-end would
## cause an error because they don't exist.
##
## Try to get the number of pages in the original PDF. If it only
## has 1 page, the stamped first page file can become the final
## stamped PDF. If it has more than 1 page, there really was an
## error when merging the stamped first page with the rest of the
## pages and stamping can be considered to have failed.
cmd_find_number_pages = \
"""%(pdftk)s %(original-file-path)s dump_data | """ \
"""grep NumberOfPages | """ \
"""sed -n 's/^NumberOfPages: \\([0-9]\\{1,\\}\\)$/\\1/p'""" \
% { 'pdftk' : CFG_PATH_PDFTK,
'original-file-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
subject_file)),
}
fh_find_number_pages = os.popen(cmd_find_number_pages, "r")
match_number_pages = fh_find_number_pages.read()
errcode_find_number_pages = fh_find_number_pages.close()
if errcode_find_number_pages is not None:
## There was an error while checking for the number of pages.
## Fail.
msg = "Error: Unable to stamp file [%s/%s]. There was an error " \
"when attempting to merge the file containing the " \
"first page of the stamped file with the remaining " \
"pages of the original file and when an attempt was " \
"made to count the number of pages in the file, an " \
"error was also encountered. Stamping has failed." \
% (path_workingdir, subject_file)
raise InvenioWebSubmitFileStamperError(msg)
else:
try:
number_pages_in_subject_file = int(match_number_pages)
except ValueError:
## Unable to get the number of pages in the original file.
## Fail.
msg = "Error: Unable to stamp file [%s/%s]. There was an " \
"error when attempting to merge the file containing the" \
" first page of the stamped file with the remaining " \
"pages of the original file and when an attempt was " \
"made to count the number of pages in the file, an " \
"error was also encountered. Stamping has failed." \
% (path_workingdir, subject_file)
raise InvenioWebSubmitFileStamperError(msg)
else:
## Do we have just one page?
if number_pages_in_subject_file == 1:
## There was only one page in the subject file.
## copy the version that was stamped on the first page to
## the output_file filename:
try:
shutil.copyfile("%s/%s" \
% (path_workingdir, \
stamped_output_file_first_page), \
"%s/%s" \
% (path_workingdir, output_file))
except IOError:
## Unable to copy the file that was stamped on page 1
## Stamping has failed.
msg = "Error: It was not possible to copy the " \
"temporary file that was stamped on the " \
"first page [%s/%s] to the final stamped " \
"file [%s/%s]. Stamping has failed." \
% (path_workingdir, \
stamped_output_file_first_page, \
path_workingdir, \
output_file)
raise InvenioWebSubmitFileStamperError(msg)
else:
## Despite the fact that there was NOT only one page
## in the original file, there was an error when trying
## to merge it with the file that was stamped on the
## first page. Fail.
msg = "Error: Unable to stamp file [%s/%s]. There " \
"was an error when attempting to merge the " \
"file containing the first page of the " \
"stamped file with the remaining pages of the " \
"original file. Stamping has failed." \
% (path_workingdir, subject_file)
raise InvenioWebSubmitFileStamperError(msg)
elif not os.access("%s/%s" % (path_workingdir, output_file), os.F_OK):
## A final version of the stamped file was NOT created even though
## no error signal was encountered during the merging process.
## Fail.
msg = "Error: Unable to stamp file [%s/%s]. When attempting to " \
"merge the file containing the first page of the stamped " \
"file with the remaining pages of the original file, no " \
"final file was created. Stamping has failed." \
% (path_workingdir, subject_file)
raise InvenioWebSubmitFileStamperError(msg)
def apply_stamp_all_pages(path_workingdir, \
stamp_file_name, \
subject_file, \
output_file, \
stamp_layer):
"""Carry out the stamping:
This function adds a stamp to all pages of the file.
@param path_workingdir: (string) - the path to the working directory
that contains all of the files needed for the stamping process to be
carried out.
@param stamp_file_name: (string) - the name of the PDF stamp file (i.e.
the stamp itself).
@param subject_file: (string) - the name of the file to be stamped.
@param output_file: (string) - the name of the final "stamped" file that
will be written in the working directory after the function has ended.
@param stamp_layer: (string) - the layer to consider when stamping
"""
cmd_stamp_all_pages = \
"%(pdftk)s %(file-to-stamp-path)s %(stamp_layer)s " \
"%(stamp-file-path)s output " \
"%(stamped-file-all-pages-path)s 2>/dev/null" \
% { 'pdftk' : CFG_PATH_PDFTK,
'file-to-stamp-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
subject_file)),
'stamp-file-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
stamp_file_name)),
'stamped-file-all-pages-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
output_file)),
'stamp_layer' : stamp_layer == 'foreground' and 'stamp' or 'background'
}
errcode_stamp_all_pages = os.system(cmd_stamp_all_pages)
if errcode_stamp_all_pages or \
not os.access("%s/%s" % (path_workingdir, output_file), os.F_OK):
## There was a problem stamping the document. Fail.
msg = "Error: Unable to stamp file [%s/%s]. Stamping has failed." \
% (path_workingdir, subject_file)
raise InvenioWebSubmitFileStamperError(msg)
def apply_stamp_to_file(path_workingdir,
stamp_type,
stamp_file_name,
subject_file,
output_file,
- stamp_layer):
+ stamp_layer,
+ skip_metadata):
"""Given a stamp-file, the details of the type of stamp to apply, and the
details of the file to be stamped, coordinate the process of having
that stamp applied to the file.
@param path_workingdir: (string) - the path to the working directory
that contains all of the files needed for the stamping process to be
carried out.
@param stamp_type: (string) - the type of stamp to be applied to the
file.
@param stamp_file_name: (string) - the name of the PDF stamp file (i.e.
the stamp itself).
@param subject_file: (string) - the name of the file to be stamped.
@param output_file: (string) - the name of the final "stamped" file that
will be written in the working directory after the function has ended.
@param stamp_layer: (string) - the layer to consider when stamping the file.
@return: (string) - the name of the stamped file that has been created.
It will be found in the stamping working directory.
"""
## Stamping is performed on PDF files. We therefore need to test for the
## type of the subject file before attempting to stamp it:
##
## Initialize a variable to hold the "file type" of the subject file:
subject_filetype = ""
## Using the file command, test for the file-type of "subject_file":
cmd_gfile = "%(gfile)s %(file-to-stamp-path)s 2> /dev/null" \
% { 'gfile' : CFG_PATH_GFILE,
'file-to-stamp-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
subject_file)),
}
## Execute the file command:
fh_gfile = os.popen(cmd_gfile, "r")
## Read the results string output by gfile:
output_gfile = fh_gfile.read()
## Close the pipe and capture its error code:
errcode_gfile = fh_gfile.close()
## If a result was obtained from gfile, scan it for an acceptable file-type:
if errcode_gfile is None and output_gfile != "":
output_gfile = output_gfile.lower()
if "pdf document" in output_gfile:
## This is a PDF file.
subject_filetype = "pdf"
elif "postscript" in output_gfile:
## This is a PostScript file.
subject_filetype = "ps"
## Unable to determine the file type using gfile.
## Try to determine the file type by examining its extension:
if subject_filetype == "":
## split the name of the file to be stamped on "." and take the last
## part of it. This should be the "extension", once cleaned from
## the possible "version" suffix (for eg. ';2' in "foo.pdf;2")
tmp_file_extension = subject_file.split(".")[-1]
tmp_file_extension = tmp_file_extension.split(';')[0]
if tmp_file_extension.lower() == "pdf":
subject_filetype = "pdf"
elif tmp_file_extension.lower() == "ps":
subject_filetype = "ps"
if subject_filetype not in ("ps", "pdf"):
## unable to process file.
msg = """Error: Input file [%s] is not PDF or PS. - unable to """ \
"""perform stamping.""" % subject_file
raise InvenioWebSubmitFileStamperError(msg)
if subject_filetype == "ps":
## Convert the subject file from PostScript to PDF:
if subject_file[-3:].lower() == ".ps":
## The name of the file to be stamped has a PostScript extension.
## Strip it and give the name of the PDF file to be created a
## PDF extension:
created_pdfname = "%s.pdf" % subject_file[:-3]
elif len(subject_file.split(".")) > 1:
## The file name has an extension - strip it and add a PDF
## extension:
raw_name = subject_file[:subject_file.rfind(".")]
if raw_name != "":
created_pdfname = "%s.pdf" % raw_name
else:
## It would appear that the file had no extension and that its
## name started with a period. Just use the original name with
## a .pdf suffix:
created_pdfname = "%s.pdf" % subject_file
else:
## No extension - use the original name with a .pdf suffix:
created_pdfname = "%s.pdf" % subject_file
## Build the distilling command:
cmd_distill = """%(distiller)s %(ps-file-path)s """ \
"""%(pdf-file-path)s 2>/dev/null""" % \
{ 'distiller' : CFG_PATH_PS2PDF,
'ps-file-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
subject_file)),
'pdf-file-path' : escape_shell_arg("%s/%s" % \
(path_workingdir, \
created_pdfname)),
}
## Distill the PS into a PDF:
errcode_distill = os.system(cmd_distill)
## Test to see whether the PS was distilled into a PDF without error:
if errcode_distill or \
not os.access("%s/%s" % (path_workingdir, created_pdfname), os.F_OK):
## The PDF file was not correctly created in the working directory.
## Unable to continue with the stamping process.
msg = "Error: Unable to correctly convert PostScript file [%s] to" \
" PDF. Cannot stamp file." % subject_file
raise InvenioWebSubmitFileStamperError(msg)
## Now assign the name of the created PDF file to subject_file:
subject_file = created_pdfname
## Treat the name of "output_file":
if output_file in (None, ""):
## there is no value for outputfile. outfile should be given the same
## name as subject_file, but with "stamped-" appended to the front.
## E.g.: subject_file: test.pdf; outfile: stamped-test.pdf
output_file = "stamped-%s" % subject_file
else:
## If output_file has an extension, strip it and add a PDF extension:
if len(output_file.split(".")) > 1:
## The file name has an extension - strip it and add a PDF
## extension:
raw_name = output_file[:output_file.rfind(".")]
if raw_name != "":
output_file = "%s.pdf" % raw_name
else:
## It would appear that the file had no extension and that its
## name started with a period. Just use the original name with
## a .pdf suffix:
output_file = "%s.pdf" % output_file
else:
## No extension - use the original name with a .pdf suffix:
output_file = "%s.pdf" % output_file
+ if not skip_metadata:
+ ## Get the PDF file metadata
+ # Metadata file name
+ metadata_file = "metadata"
+ # Get metadata command
+ cmd_get_metadata = \
+ "%(pdftk)s %(file-to-stamp-path)s dump_data output \
+ %(metadata-file-path)s 2>/dev/null" % \
+ { 'pdftk' : CFG_PATH_PDFTK,
+ 'file-to-stamp-path' : escape_shell_arg("%s/%s" % \
+ (path_workingdir,
+ subject_file)),
+ 'metadata-file-path' : escape_shell_arg("%s/%s" % \
+ (path_workingdir,
+ metadata_file)), }
+ # Get metadata errors
+ err_get_metadata = os.system(cmd_get_metadata) or not \
+ os.access("%s/%s" % (path_workingdir, metadata_file),
+ os.F_OK)
+
if stamp_type == 'coverpage':
## The stamp to be applied to the document is in fact a "cover page".
## This means that the entire PDF "stamp" that was created from the
## LaTeX template is to be appended to the subject file as the first
## page (i.e. a cover-page).
apply_stamp_cover_page(path_workingdir, \
stamp_file_name, \
subject_file, \
output_file)
elif stamp_type == "first":
apply_stamp_first_page(path_workingdir, \
stamp_file_name, \
subject_file, \
output_file, \
stamp_layer)
elif stamp_type == 'all':
## The stamp to be applied to the document is a simple that that should
## be applied to ALL pages of the document (i.e. merged onto each page.)
apply_stamp_all_pages(path_workingdir, \
stamp_file_name, \
subject_file, \
output_file, \
stamp_layer)
else:
## Unexpcted stamping mode.
msg = """Error: Unexpected stamping mode [%s]. Stamping has failed.""" \
% stamp_type
raise InvenioWebSubmitFileStamperError(msg)
+ if not skip_metadata:
+ ## Set the PDF file metadata
+ # Were we able to get the metadata correctly in the first place?
+ if not err_get_metadata:
+ # Output file -with-metadata- name
+ with_metadata_output_file = "with-metadata-" + output_file
+ # Set metadata command
+ cmd_set_metadata = \
+ "%(pdftk)s %(stamped-file-path)s update_info \
+ %(metadata-file-path)s output \
+ %(with-metadata-stamped-file-path)s 2>/dev/null" % \
+ { 'pdftk' : CFG_PATH_PDFTK,
+ 'stamped-file-path' : escape_shell_arg("%s/%s" % \
+ (path_workingdir,
+ output_file)),
+ 'metadata-file-path' : escape_shell_arg("%s/%s" % \
+ (path_workingdir,
+ metadata_file)),
+ 'with-metadata-stamped-file-path' : \
+ escape_shell_arg("%s/%s" % \
+ (path_workingdir,
+ with_metadata_output_file)), }
+ # Set metadata errors
+ err_set_metadata = os.system(cmd_set_metadata) or not \
+ os.access("%s/%s" % (path_workingdir,
+ with_metadata_output_file),
+ os.F_OK)
+ # Were we able to set the metadata correctly in the output file?
+ if not err_set_metadata:
+ without_metadata_output_file = "without-metadata-" + output_file
+ try:
+ os.rename("%s/%s" % (path_workingdir, output_file),
+ "%s/%s" % (path_workingdir,
+ without_metadata_output_file))
+ except:
+ pass
+ else:
+ try:
+ os.rename("%s/%s" % (path_workingdir,
+ with_metadata_output_file),
+ "%s/%s" % (path_workingdir, output_file))
+ except:
+ try:
+ os.rename("%s/%s" % (path_workingdir,
+ without_metadata_output_file),
+ "%s/%s" % (path_workingdir, output_file))
+ except:
+ msg = "Error: Encoutered problems when renaming " + \
+ "the output files after copying the PDF " + \
+ "metadata"
+ raise InvenioWebSubmitFileStamperError(msg)
+
## Finally, if the original subject file was a PS, convert the stamped
## PDF back to PS:
if subject_filetype == "ps":
if output_file[-4:].lower() == ".pdf":
## The name of the file to be stamped has a PDF extension.
## Strip it and give the name of the PDF file to be created a
## PDF extension:
stamped_psname = "%s.ps" % output_file[:-4]
elif len(output_file.split(".")) > 1:
## The file name has an extension - strip it and add a PDF
## extension:
raw_name = output_file[:output_file.rfind(".")]
if raw_name != "":
stamped_psname = "%s.ps" % raw_name
else:
## It would appear that the file had no extension and that its
## name started with a period. Just use the original name with
## a .pdf suffix:
stamped_psname = "%s.ps" % output_file
else:
## No extension - use the original name with a .pdf suffix:
stamped_psname = "%s.ps" % output_file
## Build the conversion command:
cmd_pdf2ps = "%s %s %s 2>/dev/null" % (CFG_PATH_PDF2PS,
escape_shell_arg("%s/%s" % \
(path_workingdir, \
output_file)),
escape_shell_arg("%s/%s" % \
(path_workingdir, \
stamped_psname)))
errcode_pdf2ps = os.system(cmd_pdf2ps)
## Check to see that the command executed OK:
if not errcode_pdf2ps and \
os.access("%s/%s" % (path_workingdir, stamped_psname), os.F_OK):
## No problem converting the PDF to PS.
output_file = stamped_psname
## Return the name of the "stamped" file:
return output_file
def copy_subject_file_to_working_directory(path_workingdir, input_file):
"""Attempt to copy the subject file (that which is to be stamped) to the
current working directory, returning the name of the subject file if
successful.
@param path_workingdir: (string) - the path to the working directory
for the current stamping session.
@param input_file: (string) - the path to the subject file (that which
is to be stamped).
@return: (string) - the name of the subject file, which has been copied
to the current working directory.
@Exceptions raised: (InvenioWebSubmitFileStamperError) - upon failure
to successfully copy the subject file to the working directory.
"""
## Divide the input filename into path and basename:
(dummy, name_input_file) = os.path.split(input_file)
if name_input_file == "":
## The input file is just a path - not a valid filename. Fail.
msg = """Error: unable to determine the name of the file to be """ \
"""stamped."""
raise InvenioWebSubmitFileStamperError(msg)
## Test to see whether the stamping subject file is a real file and
## is readable:
if os.access("%s" % input_file, os.R_OK):
## File is readable. Copy it locally to the working directory:
try:
shutil.copyfile("%s" % input_file, \
"%s/%s" % (path_workingdir, name_input_file))
except IOError:
## Unable to copy the stamping subject file to the
## working directory. Fail.
msg = """Error: Unable to copy stamping file [%s] to """ \
"""working directory for stamping [%s].""" \
% (input_file, path_workingdir)
raise InvenioWebSubmitFileStamperError(msg)
else:
## Unable to read the subject file. Fail.
msg = """Error: Unable to copy stamping file [%s] to """ \
"""working directory [%s]. (File not readable.)""" \
% (input_file, path_workingdir)
raise InvenioWebSubmitFileStamperError(msg)
## Now that the stamping file has been successfully copied to the working
## directory, return its base name:
return name_input_file
def create_working_directory():
"""Create a "working directory" in which the files related to the stamping
process can be stored, and return the full path to it.
The working directory will be created in ~invenio/var/tmp.
If it cannot be created there, an exception
(InvenioWebSubmitFileStamperError) will be raised.
The working directory will have the prefix
"websubmit_file_stamper_", and could be given a name something like:
- websubmit_file_stamper_Tzs3St
@return: (string) - the full path to the working directory.
@Exceptions raised: InvenioWebSubmitFileStamperError.
"""
## Create the temporary directory in which to place the LaTeX template
## and its helper files in ~invenio/var/tmp:
path_workingdir = None
try:
path_workingdir = tempfile.mkdtemp(prefix="websubmit_file_stamper_", \
dir="%s" % CFG_TMPDIR)
except OSError as err:
## Unable to create the temporary directory in ~invenio/var/tmp
msg = "Error: Unable to create a temporary working directory in " \
"which to carry out the stamping process. An attempt was made " \
"to create the directory in [%s]; the error encountered was " \
"<%s>. Stamping has failed." % (CFG_TMPDIR, str(err))
raise InvenioWebSubmitFileStamperError(msg)
## return the path to the working-directory:
return path_workingdir
## ***** Functions Specific to CLI calling of the program: *****
def usage(wmsg="", err_code=0):
"""Print a "usage" message (along with an optional additional warning/error
message) to stderr and exit with a given error code.
@param wmsg: (string) - some kind of warning message for the user.
@param err_code: (integer) - an error code to be passed to sys.exit,
which is called after the usage message has been printed.
@return: None.
"""
## Wash the warning message:
if wmsg != "":
wmsg = wmsg.strip() + "\n"
## The usage message:
msg = """ Usage:
python ~invenio/lib/python/invenio/websubmit_file_stamper.py \\
[options] input-file.pdf
websubmit_file_stamper.py is used to add a "stamp" to a PDF file.
A LaTeX template is used to create the stamp and this stamp is then
concatenated with the original PDF file.
The stamp can take the form of either a separate "cover page" that is
appended to the document; or a "mark" that is applied somewhere either
on the document's first page or on all of its pages.
Options:
-h, --help Print this help.
-V, --version Print version information.
-v, --verbose=LEVEL Verbose level (0=min, 1=default, 9=max).
[NOT IMPLEMENTED]
-t, --latex-template=PATH
Path to the LaTeX template file that should be used
for the creation of the PDF stamp. (Note, if it's
just a basename, it will be sought first in the
current working directory, and then in the invenio
file-stamper templates directory; If there is a
qualifying path to the template name, it will be
sought only in that location);
-c, --latex-template-var='VARNAME=VALUE'
A variable that should be replaced in the LaTeX
template file with its corresponding value. Of the
following format:
VARNAME=VALUE
This option is repeatable - one for each template
variable;
-s, --stamp=STAMP-TYPE
The type of stamp to be applied to the subject
file. Must be one of 3 values:
+ "first" - stamp only the first page;
+ "all" - stamp all pages;
+ "coverpage" - add a cover page to the
document;
The default value is "first";
-l, --layer=LAYER
The position of the stamp. Should be one of:
+ "background" (invisible if original file has
a white -not transparent- background layer)
+ "foreground" (on top of the stamped file.
If the stamp does not have a transparent
background, will hide all of the document
layers)
The default value is "background"
-o, --output-file=XYZ
The optional name to be given to the finished
(stamped) file IN THE WORKING DIRECTORY
(Specify a file name, including
extension, not a path). If this is
omitted, the stamped file will be given
the same name as the input file, but will
be prefixed by"stamped-";
+ --skip-metadata
+ Do not copy the PDF metadata of the input file
+ to the output (stamped) file.
+ If not specified, the PDF metadata will be
+ copied by default.
Example:
python ~invenio/lib/python/invenio/websubmit_file_stamper.py \\
--latex-template=demo-stamp-left.tex \\
--latex-template-var='REPORTNUMBER=TEST-THESIS-2008-019' \\
--latex-template-var='DATE=27/02/2008' \\
--stamp='first' \\
--layer='background' \\
--output-file=testfile_stamped.pdf \\
testfile.pdf
"""
sys.stderr.write(wmsg + msg)
sys.exit(err_code)
def get_cli_options():
"""From the options and arguments supplied by the user via the CLI,
build a dictionary of options to drive websubmit-file-stamper.
For reference, the CLI options available to the user are as follows:
-h, --help -> Display help/usage message and exit;
-V, --version -> Display version information and exit;
-v, --verbose= -> Set verbosity level (0=min, 1=default,
9=max).
-t, --latex-template= -> Path to the LaTeX template file that
should be used for the creation of the
PDF stamp. (Note, if it's just a
basename, it will be sought first in the
current working directory, and then in
the invenio file-stamper templates
directory; If there is a qualifying
path to the template name, it will be
sought only in that location);
-c, --latex-template-var= -> A variable that should be
replaced in the LaTeX template file
with its corresponding value. Of the
following format:
varname=value
This option is repeatable - one for each
template variable;
-s, --stamp= The type of stamp to be applied to the
subject file. Must be one of 3 values:
+ "first" - stamp only the first page;
+ "all" - stamp all pages;
+ "coverpage" - add a cover page to the
document;
The default value is "first";
-l, --layer= -> The position of the stamp. Should be one
of:
+ "background" (invisible if original
file has a white -not transparent-
background layer)
+ "foreground" (on top of the stamped
file. If the stamp does not have a
transparent background, will hide all
of the document layers).
The default value is "background"
-o, --output-file= -> The optional name to be given to the
finished (stamped) file IN THE
WORKING DIRECTORY (Specify a
name, not a path). If this is
omitted, the stamped file will
be given the same name as the
input file, but will be
prefixed by"stamped-";
+ --skip-metadata -> Do not copy the PDF metadata of the
+ input file to the output (stamped)
+ file. If not specified, the metadata
+ will be copied by default.
@return: (dictionary) of input options and flags, set as
appropriate. The dictionary has the following structure:
+ latex-template: (string) - the path to the LaTeX template to be
used for the creation of the stamp itself;
+ latex-template-var: (dictionary) - This dictionary contains
variables that should be sought in the LaTeX template file, and
the values that should be substituted in their place. E.g.:
{ "TITLE" : "An Introduction to Invenio" }
+ input-file: (string) - the path to the input file (i.e. that
which is to be stamped;
+ output-file: (string) - the name of the stamped file that should
be created by the program. This is optional - if not provided,
a default name will be applied to a file instead;
+ stamp: (string) - the type of stamp that is to be applied to the
input file. It must take one of 3 values:
- "first": Stamp only the first page of the document;
- "all": Apply the stamp to all pages of the document;
- "coverpage": Add a "cover page" to the document;
+ layer: (string) - the position of the stamp in the layers of the
file. Will be one of the following values:
- "background": stamp applied to the background layer;
- "foreground": stamp applied to the foreground layer;
+ verbosity: (integer) - the verbosity level under which the program
is to run;
+ + skip-metadata: (boolean) - whether to skip copying the metadata
+ or not;
So, an example of the returned dictionary would be something like:
{ 'latex-template' : "demo-stamp-left.tex",
'latex-template-var' : { "REPORTNUMBER" : "TEST-2008-001",
"DATE" : "15/02/2008",
},
'input-file' : "test-doc.pdf",
'output-file' : "",
'stamp' : "first",
'layer' : "background",
'verbosity' : 0,
+ 'skip-metadata' : False,
}
"""
## dictionary of important values relating to cli call of program:
options = { 'latex-template' : "",
'latex-template-var' : {},
'input-file' : "",
'output-file' : "",
'stamp' : "first",
'layer' : "background",
'verbosity' : 0,
+ 'skip-metadata' : False,
}
## Get the options and arguments provided by the user via the CLI:
try:
myoptions, myargs = getopt.getopt(sys.argv[1:], "hVv:t:c:s:l:o:", \
["help",
"version",
"verbosity=",
"latex-template=",
"latex-template-var=",
"stamp=",
"layer=",
- "output-file="])
+ "output-file=",
+ "skip-metadata"])
except getopt.GetoptError as err:
## Invalid option provided - usage message
usage(wmsg="Error: %(msg)s." % { 'msg' : str(err) })
## Get the input file from the arguments list (it should be the
## first argument):
if len(myargs) > 0:
options["input-file"] = myargs[0]
## Extract the details of the options:
for opt in myoptions:
if opt[0] in ("-V","--version"):
## version message and exit
sys.stdout.write("%s\n" % __revision__)
sys.stdout.flush()
sys.exit(0)
elif opt[0] in ("-h","--help"):
## help message and exit
usage()
elif opt[0] in ("-v", "--verbosity"):
## Get verbosity level:
if not opt[1].isdigit():
options['verbosity'] = 0
elif int(opt[1]) not in xrange(0, 10):
options['verbosity'] = 0
else:
options['verbosity'] = int(opt[1])
elif opt[0] in ("-o", "--output-file"):
## Get the name of the "output file" that is to be created after
## stamping (i.e. the "stamped file"):
options["output-file"] = opt[1]
if '/' in options["output-file"]:
# probably user specified a file path, which is not
# supported
print("Warning: you seem to have specifed a path for option '--output-file'.")
print("Only a file name can be specified. Stamping might fail.")
elif opt[0] in ("-t", "--latex-template"):
## Get the path to the latex template to be used for the creation
## of the stamp file:
options["latex-template"] = opt[1]
elif opt[0] in ("-m", "--stamp"):
## The type of stamp that is to be applied to the document:
## Options are coverpage, first, all:
if str(opt[1].lower()) in ("coverpage", "first", "all"):
## Valid stamp type, accept it;
options["stamp"] = str(opt[1]).lower()
else:
## Invalid stamp type. Print usage message and quit.
usage(wmsg="Chosen stamp type '%s' is not valid" % opt[1])
elif opt[0] in ("-l", "--layer"):
## The layer to consider for the stamp
if str(opt[1].lower()) in ("background", "foreground"):
## Valid layer type, accept it;
options["layer"] = str(opt[1]).lower()
else:
## Invalid layer type. Print usage message and quit.
usage(wmsg="Chosen layer type '%s' is not valid" % opt[1])
elif opt[0] in ("-c", "--latex-template-var"):
## This is a variable to be replaced in the LaTeX template.
## It should take the following form:
## varname=value
## We can therefore split it on the first "=" sign - anything to
## left will be considered to be the name of the variable to search
## for; anything to the right will be considered as the value that
## should replace the variable in the LaTeX template.
## Note: If the user supplies the same variable name more than once,
## the latter occurrence will be kept and the previous value will be
## overwritten.
## Note also that if the variable string does not take the
## expected format a=b, it will be ignored.
##
## Get the complete string:
varstring = str(opt[1])
## Split into 2 string based on the first "=":
split_varstring = varstring.split("=", 1)
if len(split_varstring) == 2:
## Split based on equals sign was successful:
if split_varstring[0] != "":
## The variable name was not empty - keep it:
options["latex-template-var"]["%s" % split_varstring[0]] = \
"%s" % split_varstring[1]
-
+ elif opt[0] in ("--skip-metadata"):
+ options["skip-metadata"] = True
## Return the input options:
return options
def stamp_file(options):
"""The driver for the stamping process. This is effectively the function
that is responsible for coordinating the stamping of a file.
@param options: (dictionary) - a dictionary of options that are required
by the function in order to carry out the stamping process.
The dictionary must have the following structure:
+ latex-template: (string) - the path to the LaTeX template to be
used for the creation of the stamp itself;
+ latex-template-var: (dictionary) - This dictionary contains
variables that should be sought in the LaTeX template file, and
the values that should be substituted in their place. E.g.:
{ "TITLE" : "An Introduction to Invenio" }
+ input-file: (string) - the path to the input file (i.e. that
which is to be stamped;
+ output-file: (string) - the name of the stamped file that
should be created by the program IN THE WORKING
DIRECTORY (Specify a name, not a path).
This is optional - if not provided, a default name will
be applied to a file instead;
+ stamp: (string) - the type of stamp that is to be applied to the
input file. It must take one of 3 values:
- "first": Stamp only the first page of the document;
- "all": Apply the stamp to all pages of the document;
- "coverpage": Add a "cover page" to the document;
+ layer: (string) - the layer to consider to stamp the file. Can be
one of the following values:
- "background": stamp the background layer;
- "foreground": stamp the foreground layer;
+ verbosity: (integer) - the verbosity level under which the program
is to run;
+ + skip-metadata: (boolean) - whether to skip copying the metadata
+ or not;
So, an example of the returned dictionary would be something like:
{ 'latex-template' : "demo-stamp-left.tex",
'latex-template-var' : { "REPORTNUMBER" : "TEST-2008-001",
"DATE" : "15/02/2008",
},
'input-file' : "test-doc.pdf",
'output-file' : "",
'stamp' : "first",
'layer' : "background"
'verbosity' : 0,
+ 'skip-metadata' : False,
}
@return: (tuple) - consisting of two strings:
1. the path to the working directory in which all stamping-related
files are stored;
2. The name of the "stamped" file;
@Exceptions raised: (InvenioWebSubmitFileStamperError) exceptions may
be raised or propagated by this function when the stamping process
fails for one reason or another.
"""
## SANITY CHECKS:
## Does the options dictionary contain all mandatory keys?
##
## A list of the names of the expected options:
mandatory_option_names = ["latex-template", \
"latex-template-var", \
"input-file", \
"output-file"]
- optional_option_names_and_defaults = {"layer": "background", \
- "verbosity": 0,
- "stamp": "first"}
+ optional_option_names_and_defaults = {"layer" : "background",
+ "verbosity" : 0,
+ "stamp" : "first",
+ "skip-metadata" : False}
## Are we missing some mandatory parameters?
received_option_names = options.keys()
for mandatory_option_name in mandatory_option_names:
if not mandatory_option_name in received_option_names:
msg = """Error: Mandatory parameter %s is missing""" % mandatory_option_name
raise InvenioWebSubmitFileStamperError(msg)
## Are we getting some unknown option?
for received_option_name in received_option_names:
if not received_option_name in mandatory_option_names and \
not received_option_name in optional_option_names_and_defaults.keys():
## Error: the dictionary of options had an illegal structure:
msg = """Error: Option %s is not a recognized parameter""" % received_option_name
raise InvenioWebSubmitFileStamperError(msg)
## Set default options when not specified
for opt, value in iteritems(optional_option_names_and_defaults):
if opt not in options:
options[opt] = value
## Do we have an input file to work on?
if options["input-file"] in (None, ""):
## No input file - stop the stamping:
msg = "Error: unable to determine the name of the file to be stamped."
raise InvenioWebSubmitFileStamperError(msg)
## Do we have a LaTeX file for creation of the stamp?
if options["latex-template"] in (None, ""):
## No latex stamp file - stop the stamping:
msg = "Error: unable to determine the name of the LaTeX template " \
"file to be used for stamp creation."
raise InvenioWebSubmitFileStamperError(msg)
## OK - begin the document stamping process:
##
## Get the output file:
(dummy, name_outfile) = os.path.split(options["output-file"])
if name_outfile != "":
## Take just the basename component of outfile:
options["output-file"] = name_outfile
## Create a working directory (in which to store the various files used and
## created during the stamping process) and get the full path to it:
path_workingdir = create_working_directory()
## Copy the file to be stamped into the working directory:
basename_input_file = \
copy_subject_file_to_working_directory(path_workingdir, \
options["input-file"])
## Now import the LaTeX (and associated) files into a temporary directory
## and use them to create the "stamp" PDF:
pdf_stamp_name = create_pdf_stamp(path_workingdir, \
options["latex-template"], \
options["latex-template-var"])
## Everything is now ready to merge the "stamping subject" file with the
## PDF "stamp" file that has been created:
name_stamped_file = apply_stamp_to_file(path_workingdir, \
options["stamp"], \
pdf_stamp_name, \
basename_input_file, \
options["output-file"], \
- options["layer"])
+ options["layer"], \
+ options["skip-metadata"])
## Return a tuple containing the working directory and the name of the
## stamped file to the caller:
return (path_workingdir, name_stamped_file)
def stamp_file_cli():
"""The function responsible for triggering the stamping process when called
via the CLI.
This function will effectively get the CLI options, then pass them to
function that is responsible for coordinating the stamping process
itself.
Once stamping has been completed, an attempt will be made to copy the
stamped file to the current working directory.
"""
## Get CLI options and arguments:
input_options = get_cli_options()
## Stamp the file and obtain the working directory in which the stamped file
## is situated and the name of the stamped file:
try:
(working_dir, stamped_file) = stamp_file(input_options)
except InvenioWebSubmitFileStamperError as err:
## Something went wrong:
sys.stderr.write("Stamping failed: [%s]\n" % str(err))
sys.stderr.flush()
sys.exit(1)
if not os.access("./%s" % stamped_file, os.F_OK):
## Copy the stamped file into the current directory:
try:
shutil.copyfile("%s/%s" % (working_dir, stamped_file), \
"./%s" % stamped_file)
except IOError:
## Report that it wasn't possible to copy the stamped file locally
## and offer the user a path to it:
msg = "It was not possible to copy the stamped file to the " \
"current working directory.\nYou can find it here: " \
"[%s/%s].\n" \
% (working_dir, stamped_file)
sys.stderr.write(msg)
sys.stderr.flush()
else:
## A file exists in curdir with the same name as the final stamped file.
## just print out a message stating this fact, along with the path to
## the stamped file in the temporary working directory:
msg = "The stamped file [%s] has not been copied to the current " \
"working directory because a file with this name already " \
"existed there.\nYou can find the stamped file here: " \
"[%s/%s].\n" % (stamped_file, working_dir, stamped_file)
sys.stderr.write(msg)
sys.stderr.flush()
## Start proceedings for CLI calls:
if __name__ == "__main__":
stamp_file_cli()
diff --git a/invenio/legacy/websubmit/functions/Create_Modify_Interface.py b/invenio/legacy/websubmit/functions/Create_Modify_Interface.py
index d527cc3bb..f19598078 100644
--- a/invenio/legacy/websubmit/functions/Create_Modify_Interface.py
+++ b/invenio/legacy/websubmit/functions/Create_Modify_Interface.py
@@ -1,271 +1,335 @@
## This file is part of Invenio.
-## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 CERN.
+## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
This is the Create_Modify_Interface function (along with its helpers).
It is used by WebSubmit for the "Modify Bibliographic Information" action.
"""
__revision__ = "$Id$"
import os
import re
import time
import pprint
+import cgi
from invenio.legacy.dbquery import run_sql
from invenio.legacy.websubmit.config import InvenioWebSubmitFunctionError
from invenio.legacy.websubmit.functions.Retrieve_Data import Get_Field
from invenio.ext.logging import register_exception
+from invenio.utils.html import escape_javascript_string
+from invenio.base.i18n import gettext_set_language, wash_language
def Create_Modify_Interface_getfieldval_fromfile(cur_dir, fld=""):
"""Read a field's value from its corresponding text file in 'cur_dir' (if it exists) into memory.
Delete the text file after having read-in its value.
This function is called on the reload of the modify-record page. This way, the field in question
can be populated with the value last entered by the user (before reload), instead of always being
populated with the value still found in the DB.
"""
fld_val = ""
if len(fld) > 0 and os.access("%s/%s" % (cur_dir, fld), os.R_OK|os.W_OK):
fp = open( "%s/%s" % (cur_dir, fld), "r" )
fld_val = fp.read()
fp.close()
try:
os.unlink("%s/%s"%(cur_dir, fld))
except OSError:
# Cannot unlink file - ignore, let WebSubmit main handle this
pass
fld_val = fld_val.strip()
return fld_val
def Create_Modify_Interface_getfieldval_fromDBrec(fieldcode, recid):
"""Read a field's value from the record stored in the DB.
This function is called when the Create_Modify_Interface function is called for the first time
when modifying a given record, and field values must be retrieved from the database.
"""
fld_val = ""
if fieldcode != "":
for next_field_code in [x.strip() for x in fieldcode.split(",")]:
fld_val += "%s\n" % Get_Field(next_field_code, recid)
fld_val = fld_val.rstrip('\n')
return fld_val
def Create_Modify_Interface_transform_date(fld_val):
"""Accept a field's value as a string. If the value is a date in one of the following formats:
DD Mon YYYY (e.g. 23 Apr 2005)
YYYY-MM-DD (e.g. 2005-04-23)
...transform this date value into "DD/MM/YYYY" (e.g. 23/04/2005).
"""
if re.search("^[0-9]{2} [a-z]{3} [0-9]{4}$", fld_val, re.IGNORECASE) is not None:
try:
fld_val = time.strftime("%d/%m/%Y", time.strptime(fld_val, "%d %b %Y"))
except (ValueError, TypeError):
# bad date format:
pass
elif re.search("^[0-9]{4}-[0-9]{2}-[0-9]{2}$", fld_val, re.IGNORECASE) is not None:
try:
fld_val = time.strftime("%d/%m/%Y", time.strptime(fld_val, "%Y-%m-%d"))
except (ValueError,TypeError):
# bad date format:
pass
return fld_val
def Create_Modify_Interface(parameters, curdir, form, user_info=None):
"""
Create an interface for the modification of a document, based on
the fields that the user has chosen to modify. This avoids having
to redefine a submission page for the modifications, but rely on
the elements already defined for the initial submission i.e. SBI
action (The only page that needs to be built for the modification
is the page letting the user specify a document to modify).
This function should be added at step 1 of your modification
workflow, after the functions that retrieves report number and
record id (Get_Report_Number, Get_Recid). Functions at step 2 are
the one executed upon successful submission of the form.
Create_Modify_Interface expects the following parameters:
* "fieldnameMBI" - the name of a text file in the submission
working directory that contains a list of the names of the
WebSubmit fields to include in the Modification interface.
These field names are separated by"\n" or "+".
+ * "prefix" - some content displayed before the main
+ modification interface. Can contain HTML (i.e. needs to be
+ pre-escaped). The prefix can make use of Python string
+ replacement for common values (such as 'rn'). Percent signs
+ (%) must consequently be escaped (with %%).
+
+ * "suffix" - some content displayed after the main modification
+ interface. Can contain HTML (i.e. needs to be
+ pre-escaped). The suffix can make use of Python string
+ replacement for common values (such as 'rn'). Percent signs
+ (%) must consequently be escaped (with %%).
+
+ * "button_label" - the label for the "END" button.
+
+ * "button_prefix" - some content displayed before the button to
+ submit the form. Can contain HTML (i.e. needs to be
+ pre-escaped). The prefix can make use of Python string
+ replacement for common values (such as 'rn'). Percent signs
+ (%) must consequently be escaped (with %%).
+
+ * "dates_conversion" - by default, values interpreted as dates
+ are converted to their 'DD/MM/YYYY' format, whenever
+ possible. Set another value for a different behaviour
+ (eg. 'none' for no conversion)
+
Given the list of WebSubmit fields to be included in the
modification interface, the values for each field are retrieved
for the given record (by way of each WebSubmit field being
configured with a MARC Code in the WebSubmit database). An HTML
FORM is then created. This form allows a user to modify certain
field values for a record.
The file referenced by 'fieldnameMBI' is usually generated from a
multiple select form field): users can then select one or several
fields to modify
Note that the function will display WebSubmit Response elements,
but will not be able to set an initial value: this must be done by
the Response element iteself.
Additionally the function creates an internal field named
'Create_Modify_Interface_DONE' on the interface, that can be
retrieved in curdir after the form has been submitted.
This flag is an indicator for the function that displayed values
should not be retrieved from the database, but from the submitted
values (in case the page is reloaded). You can also rely on this
value when building your WebSubmit Response element in order to
retrieve value either from the record, or from the submission
directory.
"""
+ ln = wash_language(form['ln'])
+ _ = gettext_set_language(ln)
+
global sysno,rn
t = ""
# variables declaration
fieldname = parameters['fieldnameMBI']
+ prefix = ''
+ suffix = ''
+ end_button_label = 'END'
+ end_button_prefix = ''
+ date_conversion_setting = ''
+ if parameters.has_key('prefix'):
+ prefix = parameters['prefix']
+ if parameters.has_key('suffix'):
+ suffix = parameters['suffix']
+ if parameters.has_key('button_label') and parameters['button_label']:
+ end_button_label = parameters['button_label']
+ if parameters.has_key('button_prefix'):
+ end_button_prefix = parameters['button_prefix']
+ if parameters.has_key('dates_conversion'):
+ date_conversion_setting = parameters['dates_conversion']
# Path of file containing fields to modify
the_globals = {
'doctype' : doctype,
'action' : action,
'act' : action, ## for backward compatibility
'step' : step,
'access' : access,
'ln' : ln,
'curdir' : curdir,
'uid' : user_info['uid'],
'uid_email' : user_info['email'],
'rn' : rn,
'last_step' : last_step,
'action_score' : action_score,
'__websubmit_in_jail__' : True,
'form': form,
'sysno': sysno,
'user_info' : user_info,
'__builtins__' : globals()['__builtins__'],
'Request_Print': Request_Print
}
if os.path.exists("%s/%s" % (curdir, fieldname)):
fp = open( "%s/%s" % (curdir, fieldname), "r" )
fieldstext = fp.read()
fp.close()
fieldstext = re.sub("\+","\n", fieldstext)
fields = fieldstext.split("\n")
else:
res = run_sql("SELECT fidesc FROM sbmFIELDDESC WHERE name=%s", (fieldname,))
if len(res) == 1:
fields = res[0][0].replace(" ", "")
fields = re.findall("<optionvalue=.*>", fields)
regexp = re.compile("""<optionvalue=(?P<quote>['|"]?)(?P<value>.*?)(?P=quote)""")
fields = [regexp.search(x) for x in fields]
fields = [x.group("value") for x in fields if x is not None]
fields = [x for x in fields if x not in ("Select", "select")]
else:
raise InvenioWebSubmitFunctionError("cannot find fields to modify")
#output some text
- t = t+"<CENTER bgcolor=\"white\">The document <B>%s</B> has been found in the database.</CENTER><br />Please modify the following fields:<br />Then press the 'END' button at the bottom of the page<br />\n" % rn
+ if not prefix:
+ t += "<center bgcolor=\"white\">The document <b>%s</b> has been found in the database.</center><br />Please modify the following fields:<br />Then press the '%s' button at the bottom of the page<br />\n" % \
+ (rn, cgi.escape(_(end_button_label)))
+ else:
+ t += prefix % the_globals
for field in fields:
subfield = ""
value = ""
marccode = ""
text = ""
# retrieve and display the modification text
t = t + "<FONT color=\"darkblue\">\n"
res = run_sql("SELECT modifytext FROM sbmFIELDDESC WHERE name=%s", (field,))
if len(res)>0:
t = t + "<small>%s</small> </FONT>\n" % res[0][0]
# retrieve the marc code associated with the field
res = run_sql("SELECT marccode FROM sbmFIELDDESC WHERE name=%s", (field,))
if len(res) > 0:
marccode = res[0][0]
# then retrieve the previous value of the field
if os.path.exists("%s/%s" % (curdir, "Create_Modify_Interface_DONE")):
# Page has been reloaded - get field value from text file on server, not from DB record
value = Create_Modify_Interface_getfieldval_fromfile(curdir, field)
else:
# First call to page - get field value from DB record
value = Create_Modify_Interface_getfieldval_fromDBrec(marccode, sysno)
- # If field is a date value, transform date into format DD/MM/YYYY:
- value = Create_Modify_Interface_transform_date(value)
+ if date_conversion_setting != 'none':
+ # If field is a date value, transform date into format DD/MM/YYYY:
+ value = Create_Modify_Interface_transform_date(value)
res = run_sql("SELECT * FROM sbmFIELDDESC WHERE name=%s", (field,))
if len(res) > 0:
element_type = res[0][3]
numcols = res[0][6]
numrows = res[0][5]
size = res[0][4]
maxlength = res[0][7]
val = res[0][8]
fidesc = res[0][9]
if element_type == "T":
- text = "<TEXTAREA name=\"%s\" rows=%s cols=%s wrap>%s</TEXTAREA>" % (field, numrows, numcols, value)
+ text = "<textarea name=\"%s\" rows=%s cols=%s wrap>%s</textarea>" % (field, numrows, numcols, cgi.escape(value))
elif element_type == "F":
- text = "<INPUT TYPE=\"file\" name=\"%s\" size=%s maxlength=\"%s\">" % (field, size, maxlength)
+ text = "<input type=\"file\" name=\"%s\" size=%s maxlength=\"%s\">" % (field, size, maxlength)
elif element_type == "I":
- value = re.sub("[\n\r\t]+", "", value)
- text = "<INPUT name=\"%s\" size=%s value=\"%s\"> " % (field, size, val)
- text = text + "<SCRIPT>document.forms[0].%s.value=\"%s\";</SCRIPT>" % (field, value)
+ text = "<input name=\"%s\" size=%s value=\"%s\"> " % (field, size, val and escape_javascript_string(val, escape_quote_for_html=True) or '')
+ text = text + '''<script type="text/javascript">/*<![CDATA[*/
+ document.forms[0].%s.value="%s";
+ /*]]>*/</script>''' % (field, escape_javascript_string(value, escape_for_html=False))
elif element_type == "H":
- text = "<INPUT type=\"hidden\" name=\"%s\" value=\"%s\">" % (field, val)
- text = text + "<SCRIPT>document.forms[0].%s.value=\"%s\";</SCRIPT>" % (field, value)
+ text = "<input type=\"hidden\" name=\"%s\" value=\"%s\">" % (field, val and escape_javascript_string(val, escape_quote_for_html=True) or '')
+ text = text + '''<script type="text/javascript">/*<![CDATA[*/
+ document.forms[0].%s.value="%s";
+ /*]]>*/</script>''' % (field, escape_javascript_string(value, escape_for_html=False))
elif element_type == "S":
values = re.split("[\n\r]+", value)
text = fidesc
if re.search("%s\[\]" % field, fidesc):
multipletext = "[]"
else:
multipletext = ""
if len(values) > 0 and not(len(values) == 1 and values[0] == ""):
- text += "<SCRIPT>\n"
+ text += '<script type="text/javascript">/*<![CDATA[*/\n'
text += "var i = 0;\n"
text += "el = document.forms[0].elements['%s%s'];\n" % (field, multipletext)
text += "max = el.length;\n"
for val in values:
text += "var found = 0;\n"
text += "var i=0;\n"
text += "while (i != max) {\n"
- text += " if (el.options[i].value == \"%s\" || el.options[i].text == \"%s\") {\n" % (val, val)
+ text += " if (el.options[i].value == \"%s\" || el.options[i].text == \"%s\") {\n" % \
+ (escape_javascript_string(val, escape_for_html=False), escape_javascript_string(val, escape_for_html=False))
text += " el.options[i].selected = true;\n"
text += " found = 1;\n"
text += " }\n"
text += " i=i+1;\n"
text += "}\n"
#text += "if (found == 0) {\n"
#text += " el[el.length] = new Option(\"%s\", \"%s\", 1,1);\n"
#text += "}\n"
- text += "</SCRIPT>\n"
+ text += "/*]]>*/</script>\n"
elif element_type == "D":
text = fidesc
elif element_type == "R":
try:
co = compile(fidesc.replace("\r\n", "\n"), "<string>", "exec")
## Note this exec is safe WRT global variable because the
## Create_Modify_Interface has already been parsed by
## execfile within a protected environment.
the_globals['text'] = ''
exec co in the_globals
text = the_globals['text']
except:
msg = "Error in evaluating response element %s with globals %s" % (pprint.pformat(field), pprint.pformat(globals()))
register_exception(req=None, alert_admin=True, prefix=msg)
raise InvenioWebSubmitFunctionError(msg)
else:
text = "%s: unknown field type" % field
t = t + "<small>%s</small>" % text
# output our flag field
t += '<input type="hidden" name="Create_Modify_Interface_DONE" value="DONE\n" />'
+
+ t += '<br />'
+
+ if end_button_prefix:
+ t += end_button_prefix % the_globals
+
# output some more text
- t = t + "<br /><br /><CENTER><small><INPUT type=\"button\" width=400 height=50 name=\"End\" value=\"END\" onClick=\"document.forms[0].step.value = 2;user_must_confirm_before_leaving_page = false;document.forms[0].submit();\"></small></CENTER></H4>"
+ t += "<br /><CENTER><small><INPUT type=\"button\" width=400 height=50 name=\"End\" value=\"%(end_button_label)s\" onClick=\"document.forms[0].step.value = 2;user_must_confirm_before_leaving_page = false;document.forms[0].submit();\"></small></CENTER></H4>" % {'end_button_label': escape_javascript_string(_(end_button_label), escape_quote_for_html=True)}
+
+ if suffix:
+ t += suffix % the_globals
return t
diff --git a/invenio/legacy/websubmit/functions/Create_Upload_Files_Interface.py b/invenio/legacy/websubmit/functions/Create_Upload_Files_Interface.py
index 1ddb80b72..e4271eb4a 100644
--- a/invenio/legacy/websubmit/functions/Create_Upload_Files_Interface.py
+++ b/invenio/legacy/websubmit/functions/Create_Upload_Files_Interface.py
@@ -1,500 +1,521 @@
## $Id: Revise_Files.py,v 1.37 2009/03/26 15:11:05 jerome Exp $
## This file is part of Invenio.
-## Copyright (C) 2009, 2010, 2011 CERN.
+## Copyright (C) 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""WebSubmit function - Displays a generic interface to upload, delete
and revise files.
To be used on par with Move_Uploaded_Files_to_Storage function:
- Create_Upload_Files_Interface records the actions performed by user.
- Move_Uploaded_Files_to_Storage execute the recorded actions.
NOTE:
=====
- Due to the way WebSubmit works, this function can only work when
positionned at step 1 in WebSubmit admin, and
Move_Uploaded_Files_to_Storage is at step 2
FIXME:
======
- One issue: if we allow deletion or renaming, we might lose track of
a bibdoc: someone adds X, renames X->Y, and adds again another file
with name X: when executing actions, we will add the second X, and
rename it to Y
-> need to go back in previous action when renaming... or check
that name has never been used..
"""
__revision__ = "$Id$"
import os
from invenio.config import \
CFG_SITE_LANG
from invenio.base.i18n import gettext_set_language, wash_language
from invenio.legacy.bibdocfile.managedocfiles import create_file_upload_interface
def Create_Upload_Files_Interface(parameters, curdir, form, user_info=None):
"""
List files for revisions.
You should use Move_Uploaded_Files_to_Storage.py function in your
submission to apply the changes performed by users with this
interface.
@param parameters:(dictionary) - must contain:
+ maxsize: the max size allowed for uploaded files
+ minsize: the max size allowed for uploaded files
+ doctypes: the list of doctypes (like 'Main' or 'Additional')
and their description that users can choose from
when adding new files.
- When no value is provided, users cannot add new
file (they can only revise/delete/add format)
- When a single value is given, it is used as
default doctype for all new documents
Eg:
main=Main document|additional=Figure, schema. etc
('=' separates doctype and description
'|' separates each doctype/description group)
+ restrictions: the list of restrictions (like 'Restricted' or
'No Restriction') and their description that
users can choose from when adding/revising
files. Restrictions can then be configured at
the level of WebAccess.
- When no value is provided, no restriction is
applied
- When a single value is given, it is used as
default resctriction for all documents.
- The first value of the list is used as default
restriction if the user if not given the
choice of the restriction. CHOOSE THE ORDER!
Eg:
=No restriction|restr=Restricted
('=' separates restriction and description
'|' separates each restriction/description group)
+ canDeleteDoctypes: the list of doctypes that users are
allowed to delete.
Eg:
Main|Additional
('|' separated values)
Use '*' for all doctypes
+ canReviseDoctypes: the list of doctypes that users are
allowed to revise
Eg:
Main|Additional
('|' separated values)
Use '*' for all doctypes
+ canDescribeDoctypes: the list of doctypes that users are
allowed to describe
Eg:
Main|Additional
('|' separated values)
Use '*' for all doctypes
+ canCommentDoctypes: the list of doctypes that users are
allowed to comment
Eg:
Main|Additional
('|' separated values)
Use '*' for all doctypes
+ canKeepDoctypes: the list of doctypes for which users can
choose to keep previous versions visible when
revising a file (i.e. 'Keep previous version'
checkbox). See also parameter 'keepDefault'.
Note that this parameter is ~ignored when
revising the attributes of a file (comment,
description) without uploading a new
file. See also parameter
Move_Uploaded_Files_to_Storage.forceFileRevision
Eg:
Main|Additional
('|' separated values)
Use '*' for all doctypes
+ canAddFormatDoctypes: the list of doctypes for which users can
add new formats. If there is no value,
then no 'add format' link nor warning
about losing old formats are displayed.
Eg:
Main|Additional
('|' separated values)
Use '*' for all doctypes
+ canRestrictDoctypes: the list of doctypes for which users can
choose the access restrictions when adding or
revising a file. If no value is given:
- no restriction is applied if none is defined
in the 'restrictions' parameter.
- else the *first* value of the 'restrictions'
parameter is used as default restriction.
Eg:
Main|Additional
('|' separated values)
Use '*' for all doctypes
+ canRenameDoctypes: the list of doctypes that users are allowed
to rename (when revising)
Eg:
Main|Additional
('|' separated values)
Use '*' for all doctypes
+ canNameNewFiles: if user can choose the name of the files they
upload (1) or not (0)
+ defaultFilenameDoctypes: Rename uploaded files to admin-chosen
values. List here the the files in
current submission directory that
contain the names to use for each doctype.
Eg:
Main=RN|Additional=additional_filename
('=' separates doctype and file in curdir
'|' separates each doctype/file group).
If the same doctype is submitted
several times, a"-%i" suffix is added
to the name defined in the file.
The default filenames are overriden
by user-chosen names if you allow
'canNameNewFiles' or
'canRenameDoctypes'.
+ maxFilesDoctypes: the maximum number of files that users can
upload for each doctype.
Eg:
Main=1|Additional=2
('|' separated values)
Do not specify the doctype here to have an
unlimited number of files for a given
doctype.
+ createRelatedFormats: if uploaded files get converted to
whatever format we can (1) or not (0)
+
+ + deferRelatedFormatsCreation: if creation of related format is
+ scheduled to be run later,
+ offline (1, default) or
+ immediately/online just after the
+ user has uploaded the file
+ (0). Setting immediate conversion
+ enables workflows to process the
+ created files in following
+ functions, but "blocks" the user.
+
+ keepDefault: the default behaviour for keeping or not previous
version of files when users cannot choose (no
value in canKeepDoctypes): keep (1) or not (0)
Note that this parameter is ignored when revising
the attributes of a file (comment, description)
without uploading a new file. See also parameter
Move_Uploaded_Files_to_Storage.forceFileRevision
+ showLinks: if we display links to files (1) when possible or
not (0)
+ fileLabel: the label for the file field
+ filenameLabel: the label for the file name field
+ descriptionLabel: the label for the description field
+ commentLabel: the label for the comments field
+ restrictionLabel: the label in front of the restrictions list
+ startDoc: the name of a file in curdir that contains some
text/markup to be printed *before* the file revision
box
+ endDoc: the name of a file in curdir that contains some
text/markup to be printed *after* the file revision
box
"""
global sysno
ln = wash_language(form['ln'])
_ = gettext_set_language(ln)
out = ''
## Fetch parameters defined for this function
(minsize, maxsize, doctypes_and_desc, doctypes,
can_delete_doctypes, can_revise_doctypes, can_describe_doctypes,
can_comment_doctypes, can_keep_doctypes, can_rename_doctypes,
can_add_format_to_doctypes, createRelatedFormats_p,
can_name_new_files, keep_default, show_links, file_label,
filename_label, description_label, comment_label, startDoc,
endDoc, restrictions_and_desc, can_restrict_doctypes,
restriction_label, doctypes_to_default_filename,
- max_files_for_doctype) = \
+ max_files_for_doctype, deferRelatedFormatsCreation_p) = \
wash_function_parameters(parameters, curdir, ln)
try:
recid = int(sysno)
except:
recid = None
out += '<center>'
out += startDoc
out += create_file_upload_interface(recid,
form=form,
print_outside_form_tag=True,
print_envelope=True,
include_headers=True,
ln=ln,
minsize=minsize, maxsize=maxsize,
doctypes_and_desc=doctypes_and_desc,
can_delete_doctypes=can_delete_doctypes,
can_revise_doctypes=can_revise_doctypes,
can_describe_doctypes=can_describe_doctypes,
can_comment_doctypes=can_comment_doctypes,
can_keep_doctypes=can_keep_doctypes,
can_rename_doctypes=can_rename_doctypes,
can_add_format_to_doctypes=can_add_format_to_doctypes,
create_related_formats=createRelatedFormats_p,
can_name_new_files=can_name_new_files,
keep_default=keep_default, show_links=show_links,
file_label=file_label, filename_label=filename_label,
description_label=description_label, comment_label=comment_label,
restrictions_and_desc=restrictions_and_desc,
can_restrict_doctypes=can_restrict_doctypes,
restriction_label=restriction_label,
doctypes_to_default_filename=doctypes_to_default_filename,
max_files_for_doctype=max_files_for_doctype,
sbm_indir=None, sbm_doctype=None, sbm_access=None,
- uid=None, sbm_curdir=curdir)[1]
+ uid=None, sbm_curdir=curdir,
+ defer_related_formats_creation=deferRelatedFormatsCreation_p)[1]
out += endDoc
out += '</center>'
return out
def wash_function_parameters(parameters, curdir, ln=CFG_SITE_LANG):
"""
Returns the functions (admin-defined) parameters washed and
initialized properly, as a tuple:
Parameters:
check Create_Upload_Files_Interface(..) docstring
Returns:
tuple (minsize, maxsize, doctypes_and_desc, doctypes,
can_delete_doctypes, can_revise_doctypes,
can_describe_doctypes can_comment_doctypes, can_keep_doctypes,
can_rename_doctypes, can_add_format_to_doctypes,
createRelatedFormats_p, can_name_new_files, keep_default,
show_links, file_label, filename_label, description_label,
comment_label, startDoc, endDoc, access_restrictions_and_desc,
can_restrict_doctypes, restriction_label,
- doctypes_to_default_filename, max_files_for_doctype)
+ doctypes_to_default_filename, max_files_for_doctype,
+ deferRelatedFormatsCreation_p)
"""
_ = gettext_set_language(ln)
# The min and max files sizes that users can upload
minsize = parameters['minsize']
maxsize = parameters['maxsize']
# The list of doctypes + description that users can select when
# adding new files. If there are no values, then user cannot add
# new files. '|' is used to separate doctypes groups, and '=' to
# separate doctype and description. Eg:
# main=Main document|additional=Figure, schema. etc
doctypes_and_desc = [doctype.strip().split("=") for doctype \
in parameters['doctypes'].split('|') \
if doctype.strip() != '']
doctypes = [doctype for (doctype, desc) in doctypes_and_desc]
doctypes_and_desc = [[doctype, _(desc)] for \
(doctype, desc) in doctypes_and_desc]
# The list of doctypes users are allowed to delete
# (list of values separated by "|")
can_delete_doctypes = [doctype.strip() for doctype \
in parameters['canDeleteDoctypes'].split('|') \
if doctype.strip() != '']
# The list of doctypes users are allowed to revise
# (list of values separated by "|")
can_revise_doctypes = [doctype.strip() for doctype \
in parameters['canReviseDoctypes'].split('|') \
if doctype.strip() != '']
# The list of doctypes users are allowed to describe
# (list of values separated by "|")
can_describe_doctypes = [doctype.strip() for doctype \
in parameters['canDescribeDoctypes'].split('|') \
if doctype.strip() != '']
# The list of doctypes users are allowed to comment
# (list of values separated by "|")
can_comment_doctypes = [doctype.strip() for doctype \
in parameters['canCommentDoctypes'].split('|') \
if doctype.strip() != '']
# The list of doctypes for which users are allowed to decide
# if they want to keep old files or not when revising
# (list of values separated by "|")
can_keep_doctypes = [doctype.strip() for doctype \
in parameters['canKeepDoctypes'].split('|') \
if doctype.strip() != '']
# The list of doctypes users are allowed to rename
# (list of values separated by "|")
can_rename_doctypes = [doctype.strip() for doctype \
in parameters['canRenameDoctypes'].split('|') \
if doctype.strip() != '']
# The mapping from doctype to default filename.
# '|' is used to separate doctypes groups, and '=' to
# separate doctype and file in curdir where the default name is. Eg:
# main=main_filename|additional=additional_filename. etc
default_doctypes_and_curdir_files = [doctype.strip().split("=") for doctype \
in parameters['defaultFilenameDoctypes'].split('|') \
if doctype.strip() != '']
doctypes_to_default_filename = {}
for doctype, curdir_file in default_doctypes_and_curdir_files:
default_filename = read_file(curdir, curdir_file)
if default_filename:
doctypes_to_default_filename[doctype] = os.path.basename(default_filename)
# The maximum number of files that can be uploaded for each doctype
# Eg:
# main=1|additional=3
doctypes_and_max_files = [doctype.strip().split("=") for doctype \
in parameters['maxFilesDoctypes'].split('|') \
if doctype.strip() != '']
max_files_for_doctype = {}
for doctype, max_files in doctypes_and_max_files:
if max_files.isdigit():
max_files_for_doctype[doctype] = int(max_files)
# The list of doctypes for which users are allowed to add new formats
# (list of values separated by "|")
can_add_format_to_doctypes = [doctype.strip() for doctype \
in parameters['canAddFormatDoctypes'].split('|') \
if doctype.strip() != '']
# The list of access restrictions + description that users can
# select when adding new files. If there are no values, no
# restriction is applied . '|' is used to separate access
# restrictions groups, and '=' to separate access restriction and
# description. Eg: main=Main document|additional=Figure,
# schema. etc
access_restrictions_and_desc = [access.strip().split("=") for access \
in parameters['restrictions'].split('|') \
if access.strip() != '']
access_restrictions_and_desc = [[access, _(desc)] for \
(access, desc) in access_restrictions_and_desc]
# The list of doctypes users are allowed to restrict
# (list of values separated by "|")
can_restrict_doctypes = [restriction.strip() for restriction \
in parameters['canRestrictDoctypes'].split('|') \
if restriction.strip() != '']
# If we should create additional formats when applicable (1) or
# not (0)
try:
- createRelatedFormats_p = int(parameters['createRelatedFormats'])
+ createRelatedFormats_p = bool(int(parameters['createRelatedFormats']))
except ValueError as e:
createRelatedFormats_p = False
+ # If we should create additional formats right now (1) or
+ # later (0)
+ try:
+ deferRelatedFormatsCreation_p = bool(int(parameters['deferRelatedFormatsCreation']))
+ except ValueError, e:
+ deferRelatedFormatsCreation_p = True
+
# If users can name the files they add
# Value should be 0 (Cannot rename) or 1 (Can rename)
try:
can_name_new_files = int(parameters['canNameNewFiles'])
except ValueError as e:
can_name_new_files = False
# The default behaviour wrt keeping previous files or not.
# 0 = do not keep, 1 = keep
try:
keep_default = int(parameters['keepDefault'])
except ValueError as e:
keep_default = False
# If we display links to files (1) or not (0)
try:
show_links = int(parameters['showLinks'])
except ValueError as e:
show_links = True
file_label = parameters['fileLabel']
if file_label == "":
file_label = _('Choose a file')
filename_label = parameters['filenameLabel']
if filename_label == "":
filename_label = _('Name')
description_label = parameters['descriptionLabel']
if description_label == "":
description_label = _('Description')
comment_label = parameters['commentLabel']
if comment_label == "":
comment_label = _('Comment')
restriction_label = parameters['restrictionLabel']
if restriction_label == "":
restriction_label = _('Access')
startDoc = parameters['startDoc']
endDoc = parameters['endDoc']
prefix = read_file(curdir, startDoc)
if prefix is None:
prefix = ""
suffix = read_file(curdir, endDoc)
if suffix is None:
suffix = ""
return (minsize, maxsize, doctypes_and_desc, doctypes,
can_delete_doctypes, can_revise_doctypes,
can_describe_doctypes, can_comment_doctypes,
can_keep_doctypes, can_rename_doctypes,
can_add_format_to_doctypes, createRelatedFormats_p,
can_name_new_files, keep_default, show_links, file_label,
filename_label, description_label, comment_label,
prefix, suffix, access_restrictions_and_desc,
can_restrict_doctypes, restriction_label,
- doctypes_to_default_filename, max_files_for_doctype)
+ doctypes_to_default_filename, max_files_for_doctype,
+ deferRelatedFormatsCreation_p)
def read_file(curdir, filename):
"""
Reads a file in curdir.
Returns None if does not exist, cannot be read, or if file is not
really in curdir
"""
try:
file_path = os.path.abspath(os.path.join(curdir, filename))
if not file_path.startswith(curdir):
return None
file_desc = file(file_path, 'r')
content = file_desc.read()
file_desc.close()
except:
content = None
return content
diff --git a/invenio/legacy/websubmit/functions/Link_Records.py b/invenio/legacy/websubmit/functions/Link_Records.py
index eef91b0e9..53eedbdd3 100644
--- a/invenio/legacy/websubmit/functions/Link_Records.py
+++ b/invenio/legacy/websubmit/functions/Link_Records.py
@@ -1,191 +1,385 @@
## This file is part of Invenio.
-## Copyright (C) 2012 CERN.
+## Copyright (C) 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
This function schedule a BibUpload append that will create a symmetric link
between two records based on the MARC field 787 OTHER RELATIONSHIP ENTRY (R)
+(or base on other MARC field, see parameters C{directRelationshipMARC} and
+C {reverseRelationshipMARC})
787 OTHER RELATIONSHIP ENTRY (R)
Indicators
First Note controller
0 - Display note (in $i)
1 - Do not display note
Subfield Code(s)
$i Relationship information (R) - [CER]
$r Report number
$w Record control number (R) - [CER]
NOTE: Used to link Conference papers and Slides records ($i Conference paper/Slides - $w CDS recid)
Example:
http://cds.cern.ch/record/1372158
7870_ $$iSlides$$rLHCb-TALK-2011-087$$w1353576
We need to include in the submission form for LHCb-PROC a field for the related repnr, from which to create the 7870 field. It would be perfect if at the same time the inverse 7870 field could be inserted in the TALK record:
7870_ $$iConference paper$$rLHCb-PROC-2011-041$$w1372158
"""
import re
import tempfile
import time
import os
from os.path import exists, join
-from invenio.legacy.bibrecord import record_xml_output, record_add_field
+from invenio.legacy.bibrecord import \
+ record_xml_output, \
+ record_add_field, \
+ record_add_fields, \
+ record_get_field_instances, \
+ create_record
+from invenio.modules.formatter import format_record
from invenio.modules.formatter.api import get_tag_from_name
from invenio.legacy.search_engine import search_pattern, get_fieldvalues
from invenio.config import CFG_TMPDIR
-from invenio.legacy.bibsched.bibtask import task_low_level_submission
+from invenio.legacy.bibsched.bibtask import task_low_level_submission, bibtask_allocate_sequenceid
from invenio.legacy.websubmit.config import InvenioWebSubmitFunctionError
CFG_OTHER_RELATIONSHIP_ENTRY = (get_tag_from_name('other relationship entry') or '787')[:3]
CFG_PRIMARY_REPORTNUMBER = get_tag_from_name('primary report number') or '037__a'
RE_FILENAME = re.compile("\\<pa\\>file\\:(.+)\\<\\/pa\\>", re.I)
def Link_Records(parameters, curdir, form, user_info=None):
"""
This function create a MARC link between two records (the 1st specified in the
edsrn file or SN, the second specified by edsrn2 file, where you can store
the reportnumber or directly the recid.
- In "directRelationship" you should specify either the name of a file (by using
- <pa>file:filename</pa>) or direclty, what is the relationship
- of the second record to be stored in the metadata of the 1st record.
- In the file "reverseRelationship" you can similarly specify the other
- direction of the harrow.
+
+ Parameters:
+
+ * edsrn: the file containing the report number or recid of the
+ first record (A) to be linked.
+
+ * edsrn2: the file containing the report number(s) or recid(s) of
+ the second record(s) (B) to be linked (one value per line).
+
+ * In "directRelationship" you should specify either the name of a file (by using
+ <pa>file:filename</pa>) or directly, what is the relationship
+ of the second record to be stored in the metadata of the 1st record (A->B).
+ Use the keyword "none" if you explicitely want to skip the recording of
+ this relation (no modification of record A).
+
+ * In the value/file "reverseRelationship" you can similarly specify the other
+ direction of the arrow (B->A)
+ Use the keyword "none" if you explicitely want to skip the recording of
+ this relation (no modification of record(s) B).
+
+ * keep_original_edsrn2: if edsrn2 is a report number, should we
+ use it as label when linking, or shall we use instead the
+ report number retrieved from the matching record?
+
+ * directRelationshipMARC: in which MARC tag + indicators shall we
+ store the relation in the first record (A). By default uses the
+ value found in tag name "other relationship entry" or 7870_.
+ The value can be directly provided or specifed in file (using
+ <pa>file:filename</pa>)
+
+ * reverseRelationshipMARC: in which MARC tag + indicators shall we
+ store the relation in the second record (B). By default uses the
+ value found in tag name "other relationship entry" or 7870_.
+ The value can be directly provided or specifed in file (using
+ <pa>file:filename</pa>)
+
+ * bibuploadMode: shall the created XML be sent in --append mode
+ (default) or using --correct. Possible values are:
+ * append (or leave empty)
+ * correct
+ This setting will depend on how you have set up your
+ submisson workflow.
+
+ * silentFailures: if set to "True", do not raise an exception
+ when the linking fails due to impossibility to retrieve the
+ corresponding "remote" record(s) B (for eg. non-existing report
+ number, report number matching several records, etc.). In these
+ cases the faulty report number is ignored.
+
+ * considerEmpty: when using bibuploadMode with 'correct', should
+ missing linking information (edsrn2 values) removes the linking
+ or simply not do anything? You might want to tweak this setting
+ depending on how the submission is presenting MBI pages (either
+ full form, or selected fields). If False (or empty), and no
+ linking information is provided, the linking is not removed
+ from the original record. If True (or any other value), and no
+ linking information is provided, the linking is removed from
+ the record. The value can be directly provided or specifed in
+ file (using <pa>file:filename</pa>)
"""
global sysno
edsrn = parameters["edsrn"]
edsrn2 = parameters["edsrn2"]
direct_relationship = parameters["directRelationship"]
reverse_relationship = parameters["reverseRelationship"]
keep_original_edsrn2 = parameters.get("keep_original_edsrn2", "True")
if keep_original_edsrn2 == "True":
keep_original_edsrn2 = True
elif keep_original_edsrn2 == "False":
keep_original_edsrn2 = False
else:
keep_original_edsrn2 = True
+ direct_relationship_MARC = parameters["directRelationshipMARC"]
+ reverse_relationship_MARC = parameters["reverseRelationshipMARC"]
+ bibupload_mode = parameters["bibuploadMode"]
+ if not bibupload_mode in ('append', 'correct'):
+ bibupload_mode = 'append'
+ silent_failures_p = parameters.get("silentFailures", "True") == 'True'
+
+ consider_empty_p = parameters.get("considerEmpty", "False")
+ g = RE_FILENAME.match(consider_empty_p)
+ if g:
+ filename = g.group(1)
+ if exists(join(curdir, filename)):
+ consider_empty_p = open(join(curdir, filename)).read().strip()
+ else:
+ consider_empty_p = ''
+ if consider_empty_p in ('False', ''):
+ consider_empty_p = False
+ else:
+ consider_empty_p = True
+
recid_a = int(sysno)
if exists(join(curdir, edsrn)):
rn_a = open(join(curdir, edsrn)).read().strip()
else:
rn_a = ""
if not rn_a:
try:
recid_a, rn_a = get_recid_and_reportnumber(recid=sysno)
except ValueError as err:
raise InvenioWebSubmitFunctionError("Error in finding the current record and its reportnumber: %s" % err)
- if exists(join(curdir, edsrn2)):
- rn_b = open(join(curdir, edsrn2)).read().strip()
- else:
- return ""
- if not rn_b:
- return ""
-
- if rn_b.isdigit():
- recid_b = int(rn_b)
- rn_b = ""
- recid_b, rn_b = get_recid_and_reportnumber(recid=recid_b)
- else:
- recid_b, rn_b = get_recid_and_reportnumber(reportnumber=rn_b,
- keep_original_reportnumber=keep_original_edsrn2)
-
g = RE_FILENAME.match(direct_relationship)
if g:
filename = g.group(1)
if exists(join(curdir, filename)):
direct_relationship = open(join(curdir, filename)).read().strip()
if not direct_relationship:
raise InvenioWebSubmitFunctionError("Can not retrieve direct relationship")
+ elif direct_relationship == 'none':
+ direct_relationship = None
g = RE_FILENAME.match(reverse_relationship)
if g:
filename = g.group(1)
if exists(join(curdir, filename)):
reverse_relationship = open(join(curdir, filename)).read().strip()
if not reverse_relationship:
raise InvenioWebSubmitFunctionError("Can not retrieve reverse relationship")
+ elif reverse_relationship == 'none':
+ reverse_relationship = None
+
+ g = RE_FILENAME.match(direct_relationship_MARC)
+ if g:
+ filename = g.group(1)
+ if exists(join(curdir, filename)):
+ direct_relationship_MARC = open(join(curdir, filename)).read().strip()
+
+ g = RE_FILENAME.match(reverse_relationship_MARC)
+ if g:
+ filename = g.group(1)
+ if exists(join(curdir, filename)):
+ reverse_relationship_MARC = open(join(curdir, filename)).read().strip()
- marcxml = _prepare_marcxml(recid_a, rn_a, recid_b, rn_b, reverse_relationship, direct_relationship)
+ recids_and_rns_b = []
+ if exists(join(curdir, edsrn2)):
+ for rn_b in open(join(curdir, edsrn2)).readlines():
+ rn_b = rn_b.strip()
+ if not rn_b:
+ continue
+
+ if rn_b.isdigit():
+ recid_b = int(rn_b)
+ rn_b = ""
+ try:
+ recid_b, rn_b = get_recid_and_reportnumber(recid=recid_b)
+ except ValueError, err:
+ if silent_failures_p:
+ continue
+ raise
+ else:
+ try:
+ recid_b, rn_b = get_recid_and_reportnumber(reportnumber=rn_b,
+ keep_original_reportnumber=keep_original_edsrn2)
+ except ValueError, err:
+ if silent_failures_p:
+ continue
+ raise
+ recids_and_rns_b.append((recid_b, rn_b))
+
+ if not recids_and_rns_b and bibupload_mode == 'append':
+ return ""
+
+ marcxml = _prepare_marcxml(recid_a, rn_a, recids_and_rns_b, reverse_relationship, direct_relationship,
+ marc_for_a=direct_relationship_MARC, marc_for_b=reverse_relationship_MARC,
+ upload_mode=bibupload_mode, consider_empty_p=consider_empty_p)
fd, name = tempfile.mkstemp(dir=CFG_TMPDIR, prefix="%s_%s" % \
(rn_a.replace('/', '_'),
time.strftime("%Y-%m-%d_%H:%M:%S")), suffix=".xml")
try:
os.write(fd, marcxml)
finally:
os.close(fd)
- bibupload_id = task_low_level_submission('bibupload', 'websubmit.Link_Records', '-a', name, '-P', '3')
+ sequence_id = bibtask_allocate_sequenceid(curdir)
+ bibupload_id = task_low_level_submission('bibupload', 'websubmit.Link_Records', '--' + bibupload_mode, name, '-P', '3', '-I', str(sequence_id))
open(join(curdir, 'bibupload_link_record_id'), 'w').write(str(bibupload_id))
return ""
def get_recid_and_reportnumber(recid=None, reportnumber=None, keep_original_reportnumber=True):
"""
Given at least a recid or a reportnumber, this function will look into
the system for the matching record and will return a normalized
recid and the primary reportnumber.
@raises ValueError: in case of no record matched.
"""
if recid:
## Recid specified receives priority.
recid = int(recid)
values = get_fieldvalues(recid, CFG_PRIMARY_REPORTNUMBER)
if values:
## Let's take whatever reportnumber is stored in the matching record
reportnumber = values[0]
return recid, reportnumber
else:
raise ValueError("The record %s does not have a primary report number" % recid)
elif reportnumber:
## Ok reportnumber specified, let's better try 1st with primary and then
## with other reportnumber
recids = search_pattern(p='%s:"%s"' % (CFG_PRIMARY_REPORTNUMBER, reportnumber))
if not recids:
## Not found as primary
recids = search_pattern(p='reportnumber:"%s"' % reportnumber)
if len(recids) > 1:
raise ValueError('More than one record matches the reportnumber "%s": %s' % (reportnumber, ', '.join(recids)))
elif len(recids) == 1:
recid = list(recids)[0]
if keep_original_reportnumber:
return recid, reportnumber
else:
reportnumbers = get_fieldvalues(recid, CFG_PRIMARY_REPORTNUMBER)
if not reportnumbers:
raise ValueError("The matched record %s does not have a primary report number" % recid)
return recid, reportnumbers[0]
else:
raise ValueError("No records are matched by the provided reportnumber: %s" % reportnumber)
raise ValueError("At least the recid or the reportnumber must be specified")
+def get_unlinked_records(recid_a, marc_for_b, display_in_b, upload_mode, recids_and_rns_b):
+ """
+ Retrieve list of recids that were already linked to recid_a using
+ this relation (marc_for_b), and that should no longer be linked
+ after this update (in 'correct' mode) as they are no longer part of
+ recids_and_rns_b.
+ """
+ unlinked_recids = []
+ if upload_mode == 'correct':
+ marc_tag_for_b, marc_ind1_for_b, marc_ind2_for_b = \
+ _prepare_marc(marc_for_b, CFG_OTHER_RELATIONSHIP_ENTRY, display_in_b and "0" or "1")
+ already_linked_recids = search_pattern(p=str(recid_a), m='e', f=marc_tag_for_b + marc_ind1_for_b + marc_ind2_for_b + 'w')
+ to_be_linked_recids = [recid for recid, rn in recids_and_rns_b]
+ unlinked_recids = [recid for recid in already_linked_recids if not recid in to_be_linked_recids]
+ return unlinked_recids
-def _prepare_marcxml(recid_a, rn_a, recid_b, rn_b, what_is_a_for_b, what_is_b_for_a, display_in_a=True, display_in_b=True):
+def _prepare_marcxml(recid_a, rn_a, recids_and_rns_b, what_is_a_for_b, what_is_b_for_a, display_in_a=True, display_in_b=True, marc_for_a=None, marc_for_b=None, upload_mode='append', consider_empty_p=False):
+ output = '<collection>'
record_a = {}
record_b = {}
- record_add_field(record_a, "001", controlfield_value=str(recid_a))
- record_add_field(record_a, CFG_OTHER_RELATIONSHIP_ENTRY, ind1=display_in_a and "0" or "1", subfields=[('i', what_is_b_for_a), ('r', rn_b), ('w', str(recid_b))])
- record_add_field(record_b, "001", controlfield_value=str(recid_b))
- record_add_field(record_b, CFG_OTHER_RELATIONSHIP_ENTRY, ind1=display_in_b and "0" or "1", subfields=[('i', what_is_a_for_b), ('r', rn_a), ('w', str(recid_a))])
- return "<collection>\n%s\n%s</collection>" % (record_xml_output(record_a), record_xml_output(record_b))
+ if what_is_b_for_a is not None:
+ marc_tag_for_a, marc_ind1_for_a, marc_ind2_for_a = \
+ _prepare_marc(marc_for_a, CFG_OTHER_RELATIONSHIP_ENTRY, display_in_a and "0" or "1")
+ record_add_field(record_a, "001", controlfield_value=str(recid_a))
+ if upload_mode == 'correct' and not recids_and_rns_b and consider_empty_p:
+ # Add empty field in order to account for cases where all
+ # linkings are removed by the submitter
+ record_add_field(record_a, marc_tag_for_a, ind1=marc_ind1_for_a, ind2=marc_ind2_for_a)
+ for recid_b, rn_b in recids_and_rns_b:
+ record_add_field(record_a, marc_tag_for_a, ind1=marc_ind1_for_a, ind2=marc_ind2_for_a,
+ subfields=[('i', what_is_b_for_a), ('r', rn_b), ('w', str(recid_b))])
+ output += record_xml_output(record_a)
+
+ if what_is_a_for_b is not None:
+ marc_tag_for_b, marc_ind1_for_b, marc_ind2_for_b = \
+ _prepare_marc(marc_for_b, CFG_OTHER_RELATIONSHIP_ENTRY, display_in_b and "0" or "1")
+ for recid_b, rn_b in recids_and_rns_b:
+ record_b = {}
+ record_add_field(record_b, "001", controlfield_value=str(recid_b))
+ if upload_mode == 'correct':
+ original_linking_fields = _get_record_linking_fields(recid_b, recid_a, marc_tag_for_b, marc_ind1_for_b, marc_ind2_for_b)
+ record_add_fields(record_b, marc_tag_for_b, original_linking_fields)
+ record_add_field(record_b, marc_tag_for_b, ind1=marc_ind1_for_b, ind2=marc_ind2_for_b,
+ subfields=[('i', what_is_a_for_b), ('r', rn_a), ('w', str(recid_a))])
+ output += record_xml_output(record_b)
+ # Remove linking in remote records where adequate
+ if consider_empty_p:
+ unlinked_recids = get_unlinked_records(recid_a, marc_for_b, display_in_b, upload_mode, recids_and_rns_b)
+ for recid_b in unlinked_recids:
+ record_b = {}
+ record_add_field(record_b, "001", controlfield_value=str(recid_b))
+ original_linking_fields = _get_record_linking_fields(recid_b, recid_a, marc_tag_for_b, marc_ind1_for_b, marc_ind2_for_b)
+ if not original_linking_fields:
+ # Add empty field in order to account for cases where all
+ # linkings are removed by the submitter
+ record_add_field(record_b, marc_tag_for_b, ind1=marc_ind1_for_b, ind2=marc_ind2_for_b)
+ record_add_fields(record_b, marc_tag_for_b, original_linking_fields)
+ output += record_xml_output(record_b)
+ output += '</collection>'
+ return output
+
+def _get_record_linking_fields(recid_b, recid_a, tag, ind1, ind2):
+ """
+ Returns the fields (defined by tag, ind1, ind2) in record (given
+ by recid_b) that do not link to another given record (recid_a).
+ """
+ fields = []
+ rec = create_record(format_record(recid_b, "xm"))[0]
+ for field_instance in record_get_field_instances(rec, tag=tag, ind1=ind1, ind2=ind2):
+ if not ('w', str(recid_a)) in field_instance[0]:
+ fields.append(field_instance)
+ return fields
+
+def _prepare_marc(marc_txt, default_tag, default_ind1=" ", default_ind2=" "):
+ """Returns (tag, ind1, ind2) tuple by parsing input marc_txt and
+ falling back to default value if needed"""
+ marc_tag = default_tag
+ marc_ind1 = default_ind1
+ marc_ind2 = default_ind2
+
+ if marc_txt:
+ if len(marc_txt) > 2:
+ marc_tag = marc_txt[:3]
+ if len(marc_txt) > 3:
+ marc_ind1 = marc_txt[3]
+ if len(marc_txt) > 4:
+ marc_ind2 = marc_txt[4]
+ return (marc_tag, marc_ind1, marc_ind2)
diff --git a/invenio/legacy/websubmit/functions/Shared_Functions.py b/invenio/legacy/websubmit/functions/Shared_Functions.py
index 5fb5a2c12..49b2c225b 100644
--- a/invenio/legacy/websubmit/functions/Shared_Functions.py
+++ b/invenio/legacy/websubmit/functions/Shared_Functions.py
@@ -1,271 +1,295 @@
## This file is part of Invenio.
-## Copyright (C) 2007, 2008, 2009, 2010, 2011 CERN.
+## Copyright (C) 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""Functions shared by websubmit_functions"""
from __future__ import print_function
__revision__ = "$Id$"
import os
import cgi
import glob
import sys
from logging import DEBUG
from six import iteritems
from invenio.config import \
CFG_PATH_CONVERT, \
- CFG_SITE_LANG
-from invenio.legacy.bibdocfile.api import decompose_file
+ CFG_SITE_LANG, \
+ CFG_BIBDOCFILE_FILEDIR
+from invenio.legacy.bibdocfile.api import decompose_file, decompose_file_with_version
from invenio.ext.logging import register_exception
from invenio.legacy.websubmit.file_converter import convert_file, InvenioWebSubmitFileConverterError, get_missing_formats, get_file_converter_logger
from invenio.legacy.websubmit.config import InvenioWebSubmitFunctionError
from invenio.legacy.dbquery import run_sql
from invenio.legacy.bibsched.cli import server_pid
from invenio.base.i18n import gettext_set_language
from invenio.legacy.search_engine import get_record
from invenio.legacy.bibrecord import record_get_field_values, record_get_field_value
-def createRelatedFormats(fullpath, overwrite=True, debug=False):
+def createRelatedFormats(fullpath, overwrite=True, debug=False, consider_version=False):
"""Given a fullpath, this function extracts the file's extension and
finds in which additional format the file can be converted and converts it.
@param fullpath: (string) complete path to file
@param overwrite: (bool) overwrite already existing formats
+ @param consider_version: (bool) if True, consider the version info
+ in C{fullpath} to find missing format
+ for that specific version, if C{fullpath}
+ contains version info
Return a list of the paths to the converted files
"""
file_converter_logger = get_file_converter_logger()
old_logging_level = file_converter_logger.getEffectiveLevel()
if debug:
file_converter_logger.setLevel(DEBUG)
try:
createdpaths = []
- basedir, filename, extension = decompose_file(fullpath)
+ if consider_version:
+ try:
+ basedir, filename, extension, version = decompose_file_with_version(fullpath)
+ except:
+ basedir, filename, extension = decompose_file(fullpath)
+ version = 0
+ else:
+ basedir, filename, extension = decompose_file(fullpath)
+ version = 0
extension = extension.lower()
if debug:
print("basedir: %s, filename: %s, extension: %s" % (basedir, filename, extension), file=sys.stderr)
- filelist = glob.glob(os.path.join(basedir, '%s*' % filename))
- if debug:
- print("filelist: %s" % filelist, file=sys.stderr)
- missing_formats = get_missing_formats(filelist)
+ if overwrite:
+ missing_formats = get_missing_formats([fullpath])
+ else:
+ if version:
+ filelist = glob.glob(os.path.join(basedir, '%s*;%s' % (filename, version)))
+ else:
+ filelist = glob.glob(os.path.join(basedir, '%s*' % filename))
+ if debug:
+ print("filelist: %s" % filelist, file=sys.stderr)
+ missing_formats = get_missing_formats(filelist)
if debug:
print("missing_formats: %s" % missing_formats, file=sys.stderr)
for path, formats in iteritems(missing_formats):
if debug:
print("... path: %s, formats: %s" % (path, formats), file=sys.stderr)
for aformat in formats:
if debug:
print("...... aformat: %s" % aformat, file=sys.stderr)
newpath = os.path.join(basedir, filename + aformat)
if debug:
print("...... newpath: %s" % newpath, file=sys.stderr)
try:
- convert_file(path, newpath)
+ if CFG_BIBDOCFILE_FILEDIR in basedir:
+ # We should create the new files in a temporary location, not
+ # directly inside the BibDoc directory.
+ newpath = convert_file(path, output_format=aformat)
+ else:
+ convert_file(path, newpath)
createdpaths.append(newpath)
except InvenioWebSubmitFileConverterError as msg:
if debug:
print("...... Exception: %s" % msg, file=sys.stderr)
register_exception(alert_admin=True)
finally:
if debug:
file_converter_logger.setLevel(old_logging_level)
return createdpaths
def createIcon(fullpath, iconsize):
"""Given a fullpath, this function extracts the file's extension and
if the format is compatible it converts it to icon.
@param fullpath: (string) complete path to file
Return the iconpath if successful otherwise None
"""
basedir = os.path.dirname(fullpath)
filename = os.path.basename(fullpath)
filename, extension = os.path.splitext(filename)
if extension == filename:
extension == ""
iconpath = "%s/icon-%s.gif" % (basedir, filename)
if os.path.exists(fullpath) and extension.lower() in ['.pdf', '.gif', '.jpg', '.jpeg', '.ps']:
os.system("%s -scale %s %s %s" % (CFG_PATH_CONVERT, iconsize, fullpath, iconpath))
if os.path.exists(iconpath):
return iconpath
else:
return None
def get_dictionary_from_string(dict_string):
"""Given a string version of a "dictionary", split the string into a
python dictionary.
For example, given the following string:
{'TITLE' : 'EX_TITLE', 'AUTHOR' : 'EX_AUTHOR', 'REPORTNUMBER' : 'EX_RN'}
A dictionary in the following format will be returned:
{
'TITLE' : 'EX_TITLE',
'AUTHOR' : 'EX_AUTHOR',
'REPORTNUMBER' : 'EX_RN',
}
@param dict_string: (string) - the string version of the dictionary.
@return: (dictionary) - the dictionary build from the string.
"""
try:
# Evaluate the dictionary string in an empty local/global
# namespaces. An empty '__builtins__' variable is still
# provided, otherwise Python will add the real one for us,
# which would access to undesirable functions, such as
# 'file()', 'open()', 'exec()', etc.
evaluated_dict = eval(dict_string, {"__builtins__": {}}, {})
except:
evaluated_dict = {}
# Check that returned value is a dict. Do not check with
# isinstance() as we do not even want to match subclasses of dict.
if type(evaluated_dict) is dict:
return evaluated_dict
else:
return {}
def ParamFromFile(afile):
""" Pipe a multi-line file into a single parameter"""
parameter = ''
afile = afile.strip()
if afile == '': return parameter
try:
fp = open(afile, "r")
lines = fp.readlines()
for line in lines:
parameter = parameter + line
fp.close()
except IOError:
pass
return parameter
def write_file(filename, filedata):
"""Open FILENAME and write FILEDATA to it."""
filename1 = filename.strip()
try:
of = open(filename1,'w')
except IOError:
raise InvenioWebSubmitFunctionError('Cannot open ' + filename1 + ' to write')
of.write(filedata)
of.close()
return ""
def get_nice_bibsched_related_message(curdir, ln=CFG_SITE_LANG):
"""
@return: a message suitable to display to the user, explaining the current
status of the system.
@rtype: string
"""
bibupload_id = ParamFromFile(os.path.join(curdir, 'bibupload_id'))
if not bibupload_id:
## No BibUpload scheduled? Then we don't care about bibsched
return ""
## Let's get an estimate about how many processes are waiting in the queue.
## Our bibupload might be somewhere in it, but it's not really so important
## WRT informing the user.
_ = gettext_set_language(ln)
res = run_sql("SELECT id,proc,runtime,status,priority FROM schTASK WHERE (status='WAITING' AND runtime<=NOW()) OR status='SLEEPING'")
pre = _("Note that your submission has been inserted into the bibliographic task queue and is waiting for execution.\n")
if server_pid():
## BibSched is up and running
msg = _("The task queue is currently running in automatic mode, and there are currently %(x_num)s tasks waiting to be executed. Your record should be available within a few minutes and searchable within an hour or thereabouts.\n", x_num=(len(res)))
else:
msg = _("Because of a human intervention or a temporary problem, the task queue is currently set to the manual mode. Your submission is well registered but may take longer than usual before it is fully integrated and searchable.\n")
return pre + msg
def txt2html(msg):
"""Transform newlines into paragraphs."""
rows = msg.split('\n')
rows = [cgi.escape(row) for row in rows]
rows = "<p>" + "</p><p>".join(rows) + "</p>"
return rows
def get_all_values_in_curdir(curdir):
"""
Return a dictionary with all the content of curdir.
@param curdir: the path to the current directory.
@type curdir: string
@return: the content
@rtype: dict
"""
ret = {}
for filename in os.listdir(curdir):
if not filename.startswith('.') and os.path.isfile(os.path.join(curdir, filename)):
ret[filename] = open(os.path.join(curdir, filename)).read().strip()
return ret
def get_current_record(curdir, system_number_file='SN'):
"""
Return the current record (in case it's being modified).
@param curdir: the path to the current directory.
@type curdir: string
@param system_number_file: is the name of the file on disk in curdir, that
is supposed to contain the record id.
@type system_number_file: string
@return: the record
@rtype: as in L{get_record}
"""
if os.path.exists(os.path.join(curdir, system_number_file)):
recid = open(os.path.join(curdir, system_number_file)).read().strip()
if recid:
recid = int(recid)
return get_record(recid)
return {}
def retrieve_field_values(curdir, field_name, separator=None, system_number_file='SN', tag=None):
"""
This is a handy function to retrieve values either from the current
submission directory, when a form has been just submitted, or from
an existing record (e.g. during MBI action).
@param curdir: is the current submission directory.
@type curdir: string
@param field_name: is the form field name that might exists on disk.
@type field_name: string
@param separator: is an optional separator. If it exists, it will be used
to retrieve multiple values contained in the field.
@type separator: string
@param system_number_file: is the name of the file on disk in curdir, that
is supposed to contain the record id.
@type system_number_file: string
@param tag: is the full MARC tag (tag+ind1+ind2+code) that should
contain values. If not specified, only values in curdir will
be retrieved.
@type tag: 6-chars
@return: the field value(s).
@rtype: list of strings.
@note: if field_name exists in curdir it will take precedence over
retrieving the values from the record.
"""
field_file = os.path.join(curdir, field_name)
if os.path.exists(field_file):
field_value = open(field_file).read()
if separator is not None:
return [value.strip() for value in field_value.split(separator) if value.strip()]
else:
return [field_value.strip()]
elif tag is not None:
system_number_file = os.path.join(curdir, system_number_file)
if os.path.exists(system_number_file):
recid = int(open(system_number_file).read().strip())
record = get_record(recid)
if separator:
return record_get_field_values(record, tag[:3], tag[3], tag[4], tag[5])
else:
return [record_get_field_value(record, tag[:3], tag[3], tag[4], tag[5])]
return []
diff --git a/invenio/legacy/websubmit/templates.py b/invenio/legacy/websubmit/templates.py
index 135b7ea33..8cf5b620a 100644
--- a/invenio/legacy/websubmit/templates.py
+++ b/invenio/legacy/websubmit/templates.py
@@ -1,2903 +1,2905 @@
## This file is part of Invenio.
## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
__revision__ = "$Id$"
import cgi
import re
import operator
from invenio.config import CFG_SITE_URL, CFG_SITE_LANG, CFG_SITE_RECORD, \
CFG_SITE_SECURE_URL, CFG_INSPIRE_SITE
from invenio.base.i18n import gettext_set_language
from invenio.utils.date import convert_datetext_to_dategui
from invenio.utils.url import create_html_link
from invenio.utils.mail import email_quoted_txt2html
from invenio.utils.html import escape_html, escape_javascript_string
from invenio.legacy.websubmit.config import CFG_WEBSUBMIT_CHECK_USER_LEAVES_SUBMISSION
class Template:
# Parameters allowed in the web interface for fetching files
files_default_urlargd = {
'version': (str, ""), # version "" means "latest"
'docname': (str, ""), # the docname (optional)
'format' : (str, ""), # the format
'verbose' : (int, 0), # the verbosity
'subformat': (str, ""), # the subformat
'download': (int, 0), # download as attachment
}
def tmpl_submit_home_page(self, ln, catalogues, user_info=None):
"""
The content of the home page of the submit engine
Parameters:
- 'ln' *string* - The language to display the interface in
- 'catalogues' *string* - The HTML code for the catalogues list
- 'user_info' *dict* - The user info object
"""
# load the right message language
_ = gettext_set_language(ln)
login_note = ""
if user_info and user_info['guest'] == '1':
login_note = '<em>(' + create_html_link(CFG_SITE_SECURE_URL + '/youraccount/login',
urlargd={'referer': CFG_SITE_SECURE_URL + user_info['uri'],
'ln': ln},
link_label=cgi.escape(_("Login to display all document types you can access"))) + \
')</em>'
return """
<script type="text/javascript" language="Javascript1.2">
var allLoaded = 1;
</script>
<table class="searchbox" width="100%%" summary="">
<tr>
<th class="portalboxheader">%(document_types)s: %(login_note)s</th>
</tr>
<tr>
<td class="portalboxbody">
<br />
%(please_select)s:
<br /><br />
<table width="100%%">
<tr>
<td width="50%%" class="narrowsearchboxbody">
%(catalogues)s
</td>
</tr>
</table>
</td>
</tr>
</table>""" % {
'document_types' : _("Document types available for submission"),
'please_select' : _("Please select the type of document you want to submit"),
'catalogues' : catalogues,
'ln' : ln,
'login_note' : login_note,
}
def tmpl_submit_home_catalog_no_content(self, ln):
"""
The content of the home page of submit in case no doctypes are available
Parameters:
- 'ln' *string* - The language to display the interface in
"""
# load the right message language
_ = gettext_set_language(ln)
out = "<h3>" + _("No document types available.") + "</h3>\n"
return out
def tmpl_submit_home_catalogs(self, ln, catalogs):
"""
Produces the catalogs' list HTML code
Parameters:
- 'ln' *string* - The language to display the interface in
- 'catalogs' *array* - The catalogs of documents, each one a hash with the properties:
- 'id' - the internal id
- 'name' - the name
- 'sons' - sub-catalogs
- 'docs' - the contained document types, in the form:
- 'id' - the internal id
- 'name' - the name
There is at least one catalog
"""
# load the right message language
_ = gettext_set_language(ln)
# import pprint
# out = "<pre>" + pprint.pformat(catalogs)
out = ""
for catalog in catalogs:
out += "\n<ul>"
out += self.tmpl_submit_home_catalogs_sub(ln, catalog)
out += "\n</ul>\n"
return out
def tmpl_submit_home_catalogs_sub(self, ln, catalog):
"""
Recursive function that produces a catalog's HTML display
Parameters:
- 'ln' *string* - The language to display the interface in
- 'catalog' *array* - A catalog of documents, with the properties:
- 'id' - the internal id
- 'name' - the name
- 'sons' - sub-catalogs
- 'docs' - the contained document types, in the form:
- 'id' - the internal id
- 'name' - the name
"""
# load the right message language
_ = gettext_set_language(ln)
if catalog['level'] == 1:
out = "<li><font size=\"+1\"><strong>%s</strong></font>\n" % catalog['name']
else:
if catalog['level'] == 2:
out = "<li>%s\n" % cgi.escape(catalog['name'])
else:
if catalog['level'] > 2:
out = "<li>%s\n" % cgi.escape(catalog['name'])
if len(catalog['docs']) or len(catalog['sons']):
out += "<ul>\n"
if len(catalog['docs']) != 0:
for row in catalog['docs']:
out += self.tmpl_submit_home_catalogs_doctype(ln, row)
if len(catalog['sons']) != 0:
for row in catalog['sons']:
out += self.tmpl_submit_home_catalogs_sub(ln, row)
if len(catalog['docs']) or len(catalog['sons']):
out += "</ul></li>"
else:
out += "</li>"
return out
def tmpl_submit_home_catalogs_doctype(self, ln, doc):
"""
Recursive function that produces a catalog's HTML display
Parameters:
- 'ln' *string* - The language to display the interface in
- 'doc' *array* - A catalog of documents, with the properties:
- 'id' - the internal id
- 'name' - the name
"""
# load the right message language
_ = gettext_set_language(ln)
return """<li>%s</li>""" % create_html_link('%s/submit' % CFG_SITE_URL, {'doctype' : doc['id'], 'ln' : ln}, doc['name'])
def tmpl_action_page(self, ln, uid, pid, now, doctype,
description, docfulldesc, snameCateg,
lnameCateg, actionShortDesc, indir,
statustext):
"""
Recursive function that produces a catalog's HTML display
Parameters:
- 'ln' *string* - The language to display the interface in
- 'pid' *string* - The current process id
- 'now' *string* - The current time (security control features)
- 'doctype' *string* - The selected doctype
- 'description' *string* - The description of the doctype
- 'docfulldesc' *string* - The title text of the page
- 'snameCateg' *array* - The short names of all the categories of documents
- 'lnameCateg' *array* - The long names of all the categories of documents
- 'actionShortDesc' *array* - The short names (codes) for the different actions
- 'indir' *array* - The directories for each of the actions
- 'statustext' *array* - The names of the different action buttons
"""
# load the right message language
_ = gettext_set_language(ln)
out = ""
out += """
<script language="JavaScript" type="text/javascript">
var checked = 0;
function tester() {
"""
out += """
if (checked == 0) {
alert ("%(select_cat)s");
return false;
} else {
return true;
}
}
function clicked() {
checked=1;
}
function selectdoctype(nb) {
document.forms[0].act.value = docname[nb];
}
</script>
<form method="get" action="/submit">
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="indir" />
<input type="hidden" name="access" value="%(now)i_%(pid)s" />
<input type="hidden" name="act" />
<input type="hidden" name="startPg" value="1" />
<input type="hidden" name="mainmenu" value="/submit?doctype=%(doctype)s&amp;ln=%(ln)s" />
<input type="hidden" name="ln" value="%(ln)s" />
<table class="searchbox" width="100%%" summary="">
<tr>
<th class="portalboxheader">%(docfulldesc)s</th>
</tr>
<tr>
<td class="portalboxbody">%(description)s
<br />
<script language="JavaScript" type="text/javascript">
var nbimg = document.images.length + 1;
</script>
<br />
<table align="center" cellpadding="0" cellspacing="0" border="0">
<tr valign="top">
""" % {
'select_cat' : _("Please select a category"),
'doctype' : doctype,
'now' : now,
'pid' : pid,
'docfulldesc' : docfulldesc,
'description' : description,
'ln' : ln,
}
if len(snameCateg):
out += """<td align="right">"""
selected = ""
if len(snameCateg) == 1:
# If there is only one category, we check it automatically
selected = "checked"
for i in range(0, len(snameCateg)):
out += """<label for="combo%(shortname)s">%(longname)s</label><input type="radio" name="combo%(doctype)s" id="combo%(shortname)s" %(selected)s value="%(shortname)s" onclick="clicked();" />&nbsp;<br />""" % {
'longname' : lnameCateg[i],
'doctype' : doctype,
'selected' : selected,
'shortname' : snameCateg[i],
}
out += "</td>"
out += "<td>"
if len(snameCateg) < 2:
out += '<script type="text/javascript">checked=1;</script>'
out += """&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>
<td>
<table><tr><td>
"""
#display list of actions
for i in range(0, len(actionShortDesc)):
out += """<input type="submit" class="adminbutton" value="%(status)s" onclick="if (tester()) { document.forms[0].indir.value='%(indir)s';document.forms[0].act.value='%(act)s';document.forms[0].submit();}; return false;" /><br />""" % {
'status' : statustext[i],
'indir' : indir[i],
'act' : actionShortDesc[i]
}
out += """ </td></tr></table>
</td>
</tr>
</table>
<br />"""
if len(snameCateg) :
out += """<strong class="headline">%(notice)s:</strong><br />
%(select_cat)s""" % {
'notice' : _("Notice"),
'select_cat' : _("Select a category and then click on an action button."),
}
out += """
<br /><br />
</td>
</tr>
</table>
</form>
<form action="/submit/continue"><hr />
<font color="black"><small>%(continue_explain)s</small></font>
<table border="0" bgcolor="#CCCCCC" width="100%%"><tr>
<td width="100%%">
<small>Access Number: <input size="15" name="access" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="ln" value="%(ln)s" />
<input class="adminbutton" type="submit" value=" %(go)s " />
</small>
</td></tr>
</table>
<hr />
</form>
""" % {
'continue_explain' : _("To continue with a previously interrupted submission, enter an access number into the box below:"),
'doctype' : doctype,
'go' : _("GO"),
'ln' : ln,
}
return out
def tmpl_page_interface(self, ln, docname, actname, curpage, nbpages, nextPg, access, nbPg, doctype, act, fields, javascript, mainmenu):
"""
Produces a page with the specified fields (in the submit chain)
Parameters:
- 'ln' *string* - The language to display the interface in
- 'doctype' *string* - The document type
- 'docname' *string* - The document type name
- 'actname' *string* - The action name
- 'act' *string* - The action
- 'curpage' *int* - The current page of submitting engine
- 'nbpages' *int* - The total number of pages
- 'nextPg' *int* - The next page
- 'access' *string* - The submission number
- 'nbPg' *string* - ??
- 'fields' *array* - the fields to display in the page, with each record having the structure:
- 'fullDesc' *string* - the description of the field
- 'text' *string* - the HTML code of the field
- 'javascript' *string* - if the field has some associated javascript code
- 'type' *string* - the type of field (T, F, I, H, D, S, R)
- 'name' *string* - the name of the field
- 'rows' *string* - the number of rows for textareas
- 'cols' *string* - the number of columns for textareas
- 'val' *string* - the default value of the field
- 'size' *string* - the size for text fields
- 'maxlength' *string* - the maximum length for text fields
- 'htmlcode' *string* - the complete HTML code for user-defined fields
- 'typename' *string* - the long name of the type
- 'javascript' *string* - the javascript code to insert in the page
- 'mainmenu' *string* - the url of the main menu
"""
# load the right message language
_ = gettext_set_language(ln)
# top menu
out = """
<form method="post" action="/submit" enctype="multipart/form-data" onsubmit="return tester();" accept-charset="UTF-8">
<center><table cellspacing="0" cellpadding="0" border="0">
<tr>
<td class="submitHeader"><b>%(docname)s&nbsp;</b></td>
<td class="submitHeader"><small>&nbsp;%(actname)s&nbsp;</small></td>
<td valign="bottom">
<table cellspacing="0" cellpadding="0" border="0" width="100%%">
<tr><td class="submitEmptyPage">&nbsp;&nbsp;</td>
""" % {
'docname' : docname,
'actname' : actname,
}
for i in range(1, nbpages+1):
if i == int(curpage):
out += """<td class="submitCurrentPage"><small>&nbsp;page: %s&nbsp;</small></td>""" % curpage
else:
out += """<td class="submitPage"><small>&nbsp;<a href='' onclick="if (tester2() == 1){document.forms[0].curpage.value=%s;user_must_confirm_before_leaving_page = false;document.forms[0].submit();return false;} else { return false; }">%s</a>&nbsp;</small></td>""" % (i, i)
out += """ <td class="submitEmptyPage">&nbsp;&nbsp;
</td></tr></table>
</td>
<td class="submitHeader" align="right">&nbsp;<a href="" onclick="window.open('/submit/summary?doctype=%(doctype)s&amp;act=%(act)s&amp;access=%(access)s&amp;ln=%(ln)s','summary','scrollbars=yes,menubar=no,width=500,height=250');return false;"><font color="white"><small>%(summary)s(2)</small></font></a>&nbsp;</td>
</tr>
<tr><td colspan="5" class="submitHeader">
<table border="0" cellspacing="0" cellpadding="15" width="100%%" class="submitBody"><tr><td>
<br />
<input type="hidden" name="nextPg" value="%(nextPg)s" />
<input type="hidden" name="access" value="%(access)s" />
<input type="hidden" name="curpage" value="%(curpage)s" />
<input type="hidden" name="nbPg" value="%(nbPg)s" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="act" value="%(act)s" />
<input type="hidden" name="mode" value="U" />
<input type="hidden" name="step" value="0" />
<input type="hidden" name="ln" value="%(ln)s" />
""" % {
'summary' : _("SUMMARY"),
'doctype' : cgi.escape(doctype),
'act' : cgi.escape(act),
'access' : cgi.escape(access),
'nextPg' : cgi.escape(nextPg),
'curpage' : cgi.escape(curpage),
'nbPg' : cgi.escape(nbPg),
'ln' : cgi.escape(ln),
}
for field in fields:
if field['javascript']:
out += """<script language="JavaScript1.1" type="text/javascript">
%s
</script>
""" % field['javascript']
# now displays the html form field(s)
out += "%s\n%s\n" % (field['fullDesc'], field['text'])
out += javascript
out += "<br />&nbsp;<br />&nbsp;</td></tr></table></td></tr>\n"
# Display the navigation cell
# Display "previous page" navigation arrows
out += """<tr><td colspan="5"><table border="0" cellpadding="0" cellspacing="0" width="100%%"><tr>"""
if int(curpage) != 1:
out += """ <td class="submitHeader" align="left">&nbsp;
<a href='' onclick="if (tester2() == 1) {document.forms[0].curpage.value=%(prpage)s;user_must_confirm_before_leaving_page = false;document.forms[0].submit();return false;} else { return false; }">
<img src="%(images)s/left-trans.gif" alt="%(prevpage)s" border="0" />
<strong><font color="white">%(prevpage)s</font></strong>
</a>
</td>
""" % {
'prpage' : int(curpage) - 1,
'images' : CFG_SITE_URL + '/img',
'prevpage' : _("Previous page"),
}
else:
out += """ <td class="submitHeader">&nbsp;</td>"""
# Display the submission number
out += """ <td class="submitHeader" align="center"><small>%(submission)s: %(access)s</small></td>\n""" % {
'submission' : _("Submission number") + '(1)',
'access' : cgi.escape(access),
}
# Display the "next page" navigation arrow
if int(curpage) != int(nbpages):
out += """ <td class="submitHeader" align="right">
<a href='' onclick="if (tester2()){document.forms[0].curpage.value=%(nxpage)s;user_must_confirm_before_leaving_page = false;document.forms[0].submit();return false;} else {return false;}; return false;">
<strong><font color="white">%(nextpage)s</font></strong>
<img src="%(images)s/right-trans.gif" alt="%(nextpage)s" border="0" />
</a>
</td>
""" % {
'nxpage' : int(curpage) + 1,
'images' : CFG_SITE_URL + '/img',
'nextpage' : _("Next page"),
}
else:
out += """ <td class="submitHeader">&nbsp;</td>"""
out += """</tr></table></td></tr></table></center></form>
<br />
<br />
<a href="%(mainmenu)s" onclick="if (%(check_not_already_enabled)s){return confirm('%(surequit)s')}">
<img src="%(images)s/mainmenu.gif" border="0" alt="%(back)s" align="right" /></a>
<br /><br />
<hr />
<small>%(take_note)s</small><br />
<small>%(explain_summary)s</small><br />
""" % {
'surequit' : _("Are you sure you want to quit this submission?"),
'check_not_already_enabled': CFG_WEBSUBMIT_CHECK_USER_LEAVES_SUBMISSION and 'false' or 'true',
'back' : _("Back to main menu"),
'mainmenu' : cgi.escape(mainmenu),
'images' : CFG_SITE_URL + '/img',
'take_note' : '(1) ' + _("This is your submission access number. It can be used to continue with an interrupted submission in case of problems."),
'explain_summary' : not CFG_INSPIRE_SITE and '(2) ' + _("Mandatory fields appear in red in the SUMMARY window.") or ''
}
return out
def tmpl_submit_field(self, ln, field):
"""
Produces the HTML code for the specified field
Parameters:
- 'ln' *string* - The language to display the interface in
- 'field' *array* - the field to display in the page, with the following structure:
- 'javascript' *string* - if the field has some associated javascript code
- 'type' *string* - the type of field (T, F, I, H, D, S, R)
- 'name' *string* - the name of the field
- 'rows' *string* - the number of rows for textareas
- 'cols' *string* - the number of columns for textareas
- 'val' *string* - the default value of the field
- 'size' *string* - the size for text fields
- 'maxlength' *string* - the maximum length for text fields
- 'htmlcode' *string* - the complete HTML code for user-defined fields
- 'typename' *string* - the long name of the type
"""
# load the right message language
_ = gettext_set_language(ln)
# If the field is a textarea
if field['type'] == 'T':
## Field is a textarea:
text = "<textarea name=\"%s\" rows=\"%s\" cols=\"%s\">%s</textarea>" \
% (field['name'], field['rows'], field['cols'], cgi.escape(str(field['val']), 1))
# If the field is a file upload
elif field['type'] == 'F':
## the field is a file input:
text = """<input type="file" name="%s" size="%s"%s />""" \
% (field['name'], field['size'], "%s" \
% ((field['maxlength'] in (0, None) and " ") or (""" maxlength="%s\"""" % field['maxlength'])) )
# If the field is a text input
elif field['type'] == 'I':
## Field is a text input:
text = """<input type="text" name="%s" size="%s" value="%s"%s />""" \
% (field['name'], field['size'], field['val'], "%s" \
% ((field['maxlength'] in (0, None) and " ") or (""" maxlength="%s\"""" % field['maxlength'])) )
# If the field is a hidden input
elif field['type'] == 'H':
text = "<input type=\"hidden\" name=\"%s\" value=\"%s\" />" % (field['name'], field['val'])
# If the field is user-defined
elif field['type'] == 'D':
text = field['htmlcode']
# If the field is a select box
elif field['type'] == 'S':
text = field['htmlcode']
# If the field type is not recognized
else:
text = "%s: unknown field type" % field['typename']
return text
def tmpl_page_interface_js(self, ln, upload, field, fieldhtml, txt, check, level, curdir, values, select, radio, curpage, nbpages, returnto):
"""
Produces the javascript for validation and value filling for a submit interface page
Parameters:
- 'ln' *string* - The language to display the interface in
- 'upload' *array* - booleans if the field is a <input type="file"> field
- 'field' *array* - the fields' names
- 'fieldhtml' *array* - the fields' HTML representation
- 'txt' *array* - the fields' long name
- 'check' *array* - if the fields should be checked (in javascript)
- 'level' *array* - strings, if the fields should be filled (M) or not (O)
- 'curdir' *array* - the current directory of the submission
- 'values' *array* - the current values of the fields
- 'select' *array* - booleans, if the controls are "select" controls
- 'radio' *array* - booleans, if the controls are "radio" controls
- 'curpage' *int* - the current page
- 'nbpages' *int* - the total number of pages
- 'returnto' *array* - a structure with 'field' and 'page', if a mandatory field on antoher page was not completed
"""
# load the right message language
_ = gettext_set_language(ln)
nbFields = len(upload)
# if there is a file upload field, we change the encoding type
out = """<script language="JavaScript1.1" type="text/javascript">
+ /*<![CDATA[*/
"""
for i in range(0, nbFields):
if upload[i] == 1:
out += "document.forms[0].encoding = \"multipart/form-data\";\n"
break
# we don't want the form to be submitted if the user enters 'Return'
# tests if mandatory fields are well filled
out += """function tester(){
return false;
}
function tester2() {
"""
for i in range(0, nbFields):
if re.search("%s\[\]" % field[i], fieldhtml[i]):
fieldname = "%s[]" % field[i]
else:
fieldname = field[i]
out += " el = document.forms[0].elements['%s'];\n" % fieldname
# If the field must be checked we call the checking function
if check[i] != "":
out += """if (%(check)s(el.value) == 0) {
el.focus();
return 0;
} """ % {
'check' : check[i]
}
# If the field is mandatory, we check a value has been selected
if level[i] == 'M':
if select[i] != 0:
# If the field is a select box
out += """if ((el.selectedIndex == -1)||(el.selectedIndex == 0)){
alert("%(field_mandatory)s");
return 0;
} """ % {
'field_mandatory' : _("The field %(field)s is mandatory.", field=txt[i]) + '\\n' + _("Please make a choice in the select box")
}
elif radio[i] != 0:
# If the field is a radio buttonset
out += """var check=0;
for (var j = 0; j < el.length; j++) {
if (el.options[j].checked){
check++;
}
}
if (check == 0) {
alert("%(press_button)s");
return 0;
}""" % {
'press_button':_("Please press a button.")
}
else:
# If the field is a text input
out += """if (el.value == '') {
alert("%(field_mandatory)s");
return 0;
}""" % {
'field_mandatory' : _("The field %(field)s is mandatory. Please fill it in.", field=txt[i])
}
out += """ return 1;
}
<!-- Fill the fields in with the previous saved values-->
"""
# # # # # # # # # # # # # # # # # # # # # # # # #
# Fill the fields with the previously saved values
# # # # # # # # # # # # # # # # # # # # # # # # #
for i in range(0, nbFields):
if re.search("%s\[\]"%field[i], fieldhtml[i]):
fieldname = "%s[]" % field[i]
else:
fieldname = field[i]
text = values[i]
if text != '':
if select[i] != 0:
# If the field is a SELECT element
vals = text.split("\n")
tmp = ""
for val in vals:
if tmp != "":
tmp += " || "
- tmp += "el.options[j].value == \"%s\" || el.options[j].text == \"%s\"" % (val, val)
+ tmp += "el.options[j].value == \"%s\" || el.options[j].text == \"%s\"" % \
+ (escape_javascript_string(val, escape_for_html=False),
+ escape_javascript_string(val, escape_for_html=False))
if tmp != "":
out += """
<!--SELECT field found-->
el = document.forms[0].elements['%(fieldname)s'];
for (var j = 0; j < el.length; j++){
if (%(tmp)s){
el.options[j].selected = true;
}
}""" % {
'fieldname' : fieldname,
'tmp' : tmp,
}
elif radio[i] != 0:
# If the field is a RADIO element
out += """<!--RADIO field found-->
el = document.forms[0].elements['%(fieldname)s'];
if (el.value == "%(text)s"){
el.checked=true;
}""" % {
'fieldname' : fieldname,
- 'text' : cgi.escape(str(text)).replace('"', '\\"'),
+ 'text' : escape_javascript_string(text, escape_for_html=False),
}
elif upload[i] == 0:
- text = text.replace('"','\"')
- text = text.replace("\n","\\n")
# If the field is not an upload element
out += """<!--input field found-->
el = document.forms[0].elements['%(fieldname)s'];
el.value="%(text)s";
""" % {
'fieldname' : fieldname,
- 'text' : cgi.escape(str(text)).replace('"', '\\"'),
+ 'text': escape_javascript_string(text, escape_for_html=False),
}
out += """<!--End Fill in section-->
"""
# JS function finish
# This function tests each mandatory field in the whole submission and checks whether
# the field has been correctly filled in or not
# This function is called when the user presses the "End
# Submission" button
if int(curpage) == int(nbpages):
out += """function finish() {
"""
if returnto:
out += """alert ("%(msg)s");
document.forms[0].curpage.value="%(page)s";
user_must_confirm_before_leaving_page = false;
document.forms[0].submit();
}
""" % {
'msg' : _("The field %(field)s is mandatory.") % returnto + '\\n' \
+ _("Going back to page") \
+ " " + str(returnto['page']),
'page' : returnto['page']
}
else:
out += """ if (tester2()) {
+ $(this).attr("disabled", true);
document.forms[0].action="/submit";
document.forms[0].step.value=1;
user_must_confirm_before_leaving_page = false;
document.forms[0].submit();
} else {
return false;
}
}"""
- out += """</script>"""
+ out += """ /*]]>*/</script>"""
return out
def tmpl_page_do_not_leave_submission_js(self, ln, enabled=CFG_WEBSUBMIT_CHECK_USER_LEAVES_SUBMISSION):
"""
Code to ask user confirmation when leaving the page, so that the
submission is not interrupted by mistake.
All submission functions should set the Javascript variable
'user_must_confirm_before_leaving_page' to 'false' before
programmatically submitting the submission form.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'enabled' *bool* - If the check applies or not
"""
# load the right message language
_ = gettext_set_language(ln)
out = '''
<script language="JavaScript">
var user_must_confirm_before_leaving_page = %s;
window.onbeforeunload = confirmExit;
function confirmExit() {
if (user_must_confirm_before_leaving_page)
return "%s";
}
</script>
''' % (enabled and 'true' or 'false',
_('Your modifications will not be saved.').replace('"', '\\"'))
return out
def tmpl_page_endaction(self, ln, nextPg, startPg, access, curpage, nbPg, nbpages, doctype, act, docname, actname, mainmenu, finished, function_content, next_action):
"""
Produces the pages after all the fields have been submitted.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'doctype' *string* - The document type
- 'act' *string* - The action
- 'docname' *string* - The document type name
- 'actname' *string* - The action name
- 'curpage' *int* - The current page of submitting engine
- 'startPg' *int* - The start page
- 'nextPg' *int* - The next page
- 'access' *string* - The submission number
- 'nbPg' *string* - total number of pages
- 'nbpages' *string* - number of pages (?)
- 'mainmenu' *string* - the url of the main menu
- 'finished' *bool* - if the submission is finished
- 'function_content' *string* - HTML code produced by some function executed
- 'next_action' *string* - if there is another action to be completed, the HTML code for linking to it
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<form ENCTYPE="multipart/form-data" action="/submit" onsubmit="user_must_confirm_before_leaving_page=false;" method="post" accept-charset="UTF-8">
<input type="hidden" name="nextPg" value="%(nextPg)s" />
<input type="hidden" name="startPg" value="%(startPg)s" />
<input type="hidden" name="access" value="%(access)s" />
<input type="hidden" name="curpage" value="%(curpage)s" />
<input type="hidden" name="nbPg" value="%(nbPg)s" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="act" value="%(act)s" />
<input type="hidden" name="fromdir" value="" />
<input type="hidden" name="mainmenu" value="%(mainmenu)s" />
<input type="hidden" name="mode" value="U" />
<input type="hidden" name="step" value="1" />
<input type="hidden" name="deleted" value="no" />
<input type="hidden" name="file_path" value="" />
<input type="hidden" name="userfile_name" value="" />
<input type="hidden" name="ln" value="%(ln)s" />
<center><table cellspacing="0" cellpadding="0" border="0"><tr>
<td class="submitHeader"><b>%(docname)s&nbsp;</b></td>
<td class="submitHeader"><small>&nbsp;%(actname)s&nbsp;</small></td>
<td valign="bottom">
<table cellspacing="0" cellpadding="0" border="0" width="100%%">
<tr><td class="submitEmptyPage">&nbsp;&nbsp;</td>
""" % {
'nextPg' : cgi.escape(nextPg),
'startPg' : cgi.escape(startPg),
'access' : cgi.escape(access),
'curpage' : cgi.escape(curpage),
'nbPg' : cgi.escape(nbPg),
'doctype' : cgi.escape(doctype),
'act' : cgi.escape(act),
'docname' : docname,
'actname' : actname,
'mainmenu' : cgi.escape(mainmenu),
'ln' : cgi.escape(ln),
}
if finished == 1:
out += """<td class="submitCurrentPage">%(finished)s</td>
<td class="submitEmptyPage">&nbsp;&nbsp;</td>
</tr></table>
</td>
<td class="submitEmptyPage" align="right">&nbsp;</td>
""" % {
'finished' : _("finished!"),
}
else:
for i in range(1, nbpages + 1):
out += """<td class="submitPage"><small>&nbsp;
<a href='' onclick="document.forms[0].curpage.value=%s;document.forms[0].action='/submit';document.forms[0].step.value=0;user_must_confirm_before_leaving_page = false;document.forms[0].submit();return false;">%s</a>&nbsp;</small></td>""" % (i, i)
out += """<td class="submitCurrentPage">%(end_action)s</td><td class="submitEmptyPage">&nbsp;&nbsp;</td></tr></table></td>
<td class="submitHeader" align="right">&nbsp;<a href='' onclick="window.open('/submit/summary?doctype=%(doctype)s&amp;act=%(act)s&amp;access=%(access)s&amp;ln=%(ln)s','summary','scrollbars=yes,menubar=no,width=500,height=250');return false;"><font color="white"><small>%(summary)s(2)</small></font></a>&nbsp;</td>""" % {
'end_action' : _("end of action"),
'summary' : _("SUMMARY"),
'doctype' : cgi.escape(doctype),
'act' : cgi.escape(act),
'access' : cgi.escape(access),
'ln' : cgi.escape(ln),
}
out += """</tr>
<tr>
<td colspan="5" class="submitBody">
<small><br /><br />
%(function_content)s
%(next_action)s
<br /><br />
</td>
</tr>
<tr class="submitHeader">
<td class="submitHeader" colspan="5" align="center">""" % {
'function_content' : function_content,
'next_action' : next_action,
}
if finished == 0:
out += """<small>%(submission)s</small>&sup2;:
<small>%(access)s</small>""" % {
'submission' : _("Submission no"),
'access' : cgi.escape(access),
}
else:
out += "&nbsp;\n"
out += """
</td>
</tr>
</table>
</center>
</form>
<br />
<br />"""
# Add the "back to main menu" button
if finished == 0:
out += """ <a href="%(mainmenu)s" onclick="if (%(check_not_already_enabled)s){return confirm('%(surequit)s')}">
<img src="%(images)s/mainmenu.gif" border="0" alt="%(back)s" align="right" /></a>
<br /><br />""" % {
'surequit' : _("Are you sure you want to quit this submission?"),
'back' : _("Back to main menu"),
'images' : CFG_SITE_URL + '/img',
'mainmenu' : cgi.escape(mainmenu),
'check_not_already_enabled': CFG_WEBSUBMIT_CHECK_USER_LEAVES_SUBMISSION and 'false' or 'true',
}
else:
out += """ <a href="%(mainmenu)s">
<img src="%(images)s/mainmenu.gif" border="0" alt="%(back)s" align="right" /></a>
<br /><br />""" % {
'back' : _("Back to main menu"),
'images' : CFG_SITE_URL + '/img',
'mainmenu' : cgi.escape(mainmenu),
}
return out
def tmpl_function_output(self, ln, display_on, action, doctype, step, functions):
"""
Produces the output of the functions.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'display_on' *bool* - If debug information should be displayed
- 'doctype' *string* - The document type
- 'action' *string* - The action
- 'step' *int* - The current step in submission
- 'functions' *aray* - HTML code produced by functions executed and informations about the functions
- 'name' *string* - the name of the function
- 'score' *string* - the score of the function
- 'error' *bool* - if the function execution produced errors
- 'text' *string* - the HTML code produced by the function
"""
# load the right message language
_ = gettext_set_language(ln)
out = ""
if display_on:
out += """<br /><br />%(function_list)s<P>
<table border="1" cellpadding="15">
<tr><th>%(function)s</th><th>%(score)s</th><th>%(running)s</th></tr>
""" % {
'function_list' : _("Here is the %(x_action)s function list for %(x_doctype)s documents at level %(x_step)s") % {
'x_action' : action,
'x_doctype' : doctype,
'x_step' : step,
},
'function' : _("Function"),
'score' : _("Score"),
'running' : _("Running function"),
}
for function in functions:
out += """<tr><td>%(name)s</td><td>%(score)s</td><td>%(result)s</td></tr>""" % {
'name' : function['name'],
'score' : function['score'],
'result' : function['error'] and (_("Function %(x_name)s does not exist.", x_name=function['name']) + "<br />") or function['text']
}
out += "</table>"
else:
for function in functions:
if not function['error']:
out += function['text']
return out
def tmpl_next_action(self, ln, actions):
"""
Produces the output of the functions.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'actions' *array* - The actions to display, in the structure
- 'page' *string* - the starting page
- 'action' *string* - the action (in terms of submission)
- 'doctype' *string* - the doctype
- 'nextdir' *string* - the path to the submission data
- 'access' *string* - the submission number
- 'indir' *string* - ??
- 'name' *string* - the name of the action
"""
# load the right message language
_ = gettext_set_language(ln)
out = "<br /><br />%(haveto)s<ul>" % {
'haveto' : _("You must now"),
}
i = 0
for action in actions:
if i > 0:
out += " <b>" + _("or") + "</b> "
i += 1
out += """<li><a href="" onclick="document.forms[0].action='/submit';document.forms[0].curpage.value='%(page)s';document.forms[0].startPg.value='%(page)s';document.forms[0].act.value='%(action)s';document.forms[0].doctype.value='%(doctype)s';document.forms[0].indir.value='%(nextdir)s';document.forms[0].access.value='%(access)s';document.forms[0].fromdir.value='%(indir)s';user_must_confirm_before_leaving_page = falsedocument.forms[0].submit();return false;"> %(name)s </a></li>""" % action
out += "</ul>"
return out
def tmpl_submit_summary (self, ln, values):
"""
Displays the summary for the submit procedure.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'values' *array* - The values of submit. Each of the records contain the following fields:
- 'name' *string* - The name of the field
- 'mandatory' *bool* - If the field is mandatory or not
- 'value' *string* - The inserted value
- 'page' *int* - The submit page on which the field is entered
"""
# load the right message language
_ = gettext_set_language(ln)
out = """<body style="background-image: url(%(images)s/header_background.gif);"><table border="0">""" % \
{ 'images' : CFG_SITE_URL + '/img' }
for value in values:
if value['mandatory']:
color = "red"
else:
color = ""
out += """<tr>
<td align="right">
<small>
<a href='' onclick="window.opener.document.forms[0].curpage.value='%(page)s';window.opener.document.forms[0].action='/submit?ln=%(ln)s';window.opener.document.forms[0].submit();return false;">
<font color="%(color)s">%(name)s</font>
</a>
</small>
</td>
<td>
<i><small><font color="black">%(value)s</font></small></i>
</td>
</tr>""" % {
'color' : color,
'name' : value['name'],
'value' : value['value'],
'page' : value['page'],
'ln' : ln
}
out += "</table>"
return out
def tmpl_yoursubmissions(self, ln, order, doctypes, submissions):
"""
Displays the list of the user's submissions.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'order' *string* - The ordering parameter
- 'doctypes' *array* - All the available doctypes, in structures:
- 'id' *string* - The doctype id
- 'name' *string* - The display name of the doctype
- 'selected' *bool* - If the doctype should be selected
- 'submissions' *array* - The available submissions, in structures:
- 'docname' *string* - The document name
- 'actname' *string* - The action name
- 'status' *string* - The status of the document
- 'cdate' *string* - Creation date
- 'mdate' *string* - Modification date
- 'id' *string* - The id of the submission
- 'reference' *string* - The display name of the doctype
- 'pending' *bool* - If the submission is pending
- 'act' *string* - The action code
- 'doctype' *string* - The doctype code
"""
# load the right message language
_ = gettext_set_language(ln)
out = ""
out += """
<br />
<form action="">
<input type="hidden" value="%(order)s" name="order" />
<input type="hidden" name="deletedId" />
<input type="hidden" name="deletedDoctype" />
<input type="hidden" name="deletedAction" />
<input type="hidden" name="ln" value="%(ln)s"/>
<table class="searchbox" width="100%%" summary="" >
<tr>
<th class="portalboxheader">%(for)s&nbsp;
<select name="doctype" onchange="document.forms[0].submit();">
<option value="">%(alltype)s</option>
""" % {
'order' : order,
'for' : _("For"),
'alltype' : _("all types of document"),
'ln' : ln,
}
for doctype in doctypes:
out += """<option value="%(id)s" %(sel)s>%(name)s</option>""" % {
'id' : doctype['id'],
'name' : doctype['name'],
'sel' : doctype['selected'] and "selected=\"selected\"" or ""
}
out += """ </select>
</th>
</tr>
<tr>
<td class="portalboxbody">
<table>
<tr>
<td></td>
</tr>
"""
num = 0
docname = ""
for submission in submissions:
if submission['docname'] != docname:
docname = submission['docname']
out += """</table>
<br/>&nbsp;<br/><h3>%(docname)s</h3>
<table border="0" class="searchbox" align="left" width="100%%">
<tr>
<th class="headerselected">%(action)s&nbsp;&nbsp;
<a href='' onclick='document.forms[0].order.value="actiondown";document.forms[0].submit();return false;'><img src="%(images)s/smalldown.gif" alt="down" border="0" /></a>&nbsp;
<a href='' onclick='document.forms[0].order.value="actionup";document.forms[0].submit();return false;'><img src="%(images)s/smallup.gif" alt="up" border="0" /></a>
</th>
<th class="headerselected">%(status)s&nbsp;&nbsp;
<a href='' onclick='document.forms[0].order.value="statusdown";document.forms[0].submit();return false;'><img src="%(images)s/smalldown.gif" alt="down" border="0" /></a>&nbsp;
<a href='' onclick='document.forms[0].order.value="statusup";document.forms[0].submit();return false;'><img src="%(images)s/smallup.gif" alt="up" border="0" /></a>
</th>
<th class="headerselected">%(id)s</th>
<th class="headerselected">%(reference)s&nbsp;&nbsp;
<a href='' onclick='document.forms[0].order.value="refdown";document.forms[0].submit();return false;'><img src="%(images)s/smalldown.gif" alt="down" border="0" /></a>&nbsp;
<a href='' onclick='document.forms[0].order.value="refup";document.forms[0].submit();return false;'><img src="%(images)s/smallup.gif" alt="up" border="0" /></a>
</th>
<th class="headerselected">%(first)s&nbsp;&nbsp;
<a href='' onclick='document.forms[0].order.value="cddown";document.forms[0].submit();return false;'><img src="%(images)s/smalldown.gif" alt="down" border="0" /></a>&nbsp;
<a href='' onclick='document.forms[0].order.value="cdup";document.forms[0].submit();return false;'><img src="%(images)s/smallup.gif" alt="up" border="0" /></a>
</th>
<th class="headerselected">%(last)s&nbsp;&nbsp;
<a href='' onclick='document.forms[0].order.value="mddown";document.forms[0].submit();return false;'><img src="%(images)s/smalldown.gif" alt="down" border="0" /></a>&nbsp;
<a href='' onclick='document.forms[0].order.value="mdup";document.forms[0].submit();return false;'><img src="%(images)s/smallup.gif" alt="up" border="0" /></a>
</th>
</tr>
""" % {
'docname' : submission['docname'],
'action' : _("Action"),
'status' : _("Status"),
'id' : _("Subm.No."),
'reference' : _("Reference"),
'images' : CFG_SITE_URL + '/img',
'first' : _("First access"),
'last' : _("Last access"),
}
if submission['pending']:
idtext = """<a href="submit/direct?access=%(id)s&sub=%(action)s%(doctype)s%(ln_link)s">%(id)s</a>
&nbsp;<a onclick='if (confirm("%(sure)s")){document.forms[0].deletedId.value="%(id)s";document.forms[0].deletedDoctype.value="%(doctype)s";document.forms[0].deletedAction.value="%(action)s";document.forms[0].submit();return true;}else{return false;}' href=''><img src="%(images)s/smallbin.gif" border="0" alt='%(delete)s' /></a>
""" % {
'images' : CFG_SITE_URL + '/img',
'id' : submission['id'],
'action' : submission['act'],
'doctype' : submission['doctype'],
'sure' : _("Are you sure you want to delete this submission?"),
'delete' : _("Delete submission %(x_id)s in %(x_docname)s") % {
'x_id' : str(submission['id']),
'x_docname' : str(submission['docname'])
},
'ln_link': (ln != CFG_SITE_LANG and '&amp;ln=' + ln) or ''
}
else:
idtext = submission['id']
if operator.mod(num, 2) == 0:
color = "#e2e2e2"
else:
color = "#f0f0f0"
if submission['reference']:
reference = submission['reference']
if not submission['pending']:
# record was integrated, so propose link:
reference = create_html_link('%s/search' % CFG_SITE_URL, {
'ln' : ln,
'p' : submission['reference'],
'f' : 'reportnumber'
}, submission['reference'])
else:
reference = """<font color="red">%s</font>""" % _("Reference not yet given")
cdate = str(submission['cdate']).replace(" ", "&nbsp;")
mdate = str(submission['mdate']).replace(" ", "&nbsp;")
out += """
<tr bgcolor="%(color)s">
<td align="center" class="mycdscell">
%(actname)s
</td>
<td align="center" class="mycdscell">
%(status)s
</td>
<td class="mycdscell">
%(idtext)s
</td>
<td class="mycdscell">
&nbsp;%(reference)s
</td>
<td class="mycdscell">
%(cdate)s
</td>
<td class="mycdscell">
%(mdate)s
</td>
</tr>
""" % {
'color' : color,
'actname' : submission['actname'],
'status' : submission['status'],
'idtext' : idtext,
'reference' : reference,
'cdate' : cdate,
'mdate' : mdate,
}
num += 1
out += "</table></td></tr></table></form>"
return out
def tmpl_yourapprovals(self, ln, referees):
"""
Displays the doctypes and categories for which the user is referee
Parameters:
- 'ln' *string* - The language to display the interface in
- 'referees' *array* - All the doctypes for which the user is referee:
- 'doctype' *string* - The doctype
- 'docname' *string* - The display name of the doctype
- 'categories' *array* - The specific categories for which the user is referee:
- 'id' *string* - The category id
- 'name' *string* - The display name of the category
"""
# load the right message language
_ = gettext_set_language(ln)
out = """ <table class="searchbox" width="100%%" summary="">
<tr>
<th class="portalboxheader">%(refdocs)s</th>
</tr>
<tr>
<td class="portalboxbody">""" % {
'refdocs' : _("Refereed Documents"),
}
for doctype in referees:
out += """<ul><li><b>%(docname)s</b><ul>""" % doctype
if doctype ['categories'] is None:
out += '''<li><a href="publiline.py?doctype=%(doctype)s%(ln_link)s">%(generalref)s</a></li>''' % {
'docname' : doctype['docname'],
'doctype' : doctype['doctype'],
'generalref' : _("You are a general referee"),
'ln_link': '&amp;ln=' + ln}
else:
for category in doctype['categories']:
out += """<li><a href="publiline.py?doctype=%(doctype)s&amp;categ=%(categ)s%(ln_link)s">%(referee)s</a></li>""" % {
'referee' : _("You are a referee for category:") + ' ' + str(category['name']) + ' (' + str(category['id']) + ')',
'doctype' : doctype['doctype'],
'categ' : category['id'],
'ln_link': '&amp;ln=' + ln}
out += "</ul><br /></li></ul>"
out += "</td></tr></table>"
out += '''<p>To see the status of documents for which approval has been requested, click <a href=\"%(url)s/publiline.py?flow=cplx\">here</a></p>''' % {'url' : CFG_SITE_URL}
return out
def tmpl_publiline_selectdoctype(self, ln, docs):
"""
Displays the doctypes that the user can select
Parameters:
- 'ln' *string* - The language to display the interface in
- 'docs' *array* - All the doctypes that the user can select:
- 'doctype' *string* - The doctype
- 'docname' *string* - The display name of the doctype
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<table class="searchbox" width="100%%" summary="">
<tr>
<th class="portalboxheader">%(list)s</th>
</tr>
<tr>
<td class="portalboxbody">
%(select)s:
</small>
<blockquote>""" % {
'list' : _("List of refereed types of documents"),
'select' : _("Select one of the following types of documents to check the documents status"),
}
for doc in docs:
params = {'ln' : ln}
params.update(doc)
out += '<li><a href="publiline.py?doctype=%(doctype)s&amp;ln=%(ln)s">%(docname)s</a></li><br />' % params
out += """</blockquote>
</td>
</tr>
</table>
<a href="publiline.py?flow=cplx&amp;ln=%s">%s</a>""" % (ln, _("Go to specific approval workflow"))
return out
def tmpl_publiline_selectcplxdoctype(self, ln, docs):
"""
Displays the doctypes that the user can select in a complex workflow
Parameters:
- 'ln' *string* - The language to display the interface in
- 'docs' *array* - All the doctypes that the user can select:
- 'doctype' *string* - The doctype
- 'docname' *string* - The display name of the doctype
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<table class="searchbox" width="100%%" summary="">
<tr>
<th class="portalboxheader">%(list)s</th>
</tr>
<tr>
<td class="portalboxbody">
%(select)s:
</small>
<blockquote>""" % {
'list' : _("List of refereed types of documents"),
'select' : _("Select one of the following types of documents to check the documents status"),
}
for doc in docs:
params = {'ln' : ln}
params.update(doc)
out += '<li><a href="publiline.py?flow=cplx&doctype=%(doctype)s&amp;ln=%(ln)s">%(docname)s</a></li><br />' % params
out += """</blockquote> </td> </tr> </table> </li><br/>"""
return out
def tmpl_publiline_selectcateg(self, ln, doctype, title, categories):
"""
Displays the categories from a doctype that the user can select
Parameters:
- 'ln' *string* - The language to display the interface in
- 'doctype' *string* - The doctype
- 'title' *string* - The doctype name
- 'categories' *array* - All the categories that the user can select:
- 'id' *string* - The id of the category
- 'waiting' *int* - The number of documents waiting
- 'approved' *int* - The number of approved documents
- 'rejected' *int* - The number of rejected documents
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<table class="searchbox" width="100%%" summary="">
<tr>
<th class="portalboxheader">%(title)s: %(list_categ)s</th>
</tr>
<tr>
<td class="portalboxbody">
%(choose_categ)s
<blockquote>
<form action="publiline.py" method="get">
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="categ" value="" />
<input type="hidden" name="ln" value="%(ln)s" />
</form>
<table>
<tr>
<td align="left">""" % {
'title' : title,
'doctype' : doctype,
'list_categ' : _("List of refereed categories"),
'choose_categ' : _("Please choose a category"),
'ln' : ln,
}
for categ in categories:
num = categ['waiting'] + categ['approved'] + categ['rejected']
if categ['waiting'] != 0:
classtext = "class=\"blocknote\""
else:
classtext = ""
out += """<a href="" onclick="document.forms[0].categ.value='%(id)s';document.forms[0].submit();return false;"><span %(classtext)s>%(id)s</span></a> (%(num)s document(s)""" % {
'id' : categ['id'],
'classtext' : classtext,
'num' : num,
}
if categ['waiting'] != 0:
out += """| %(waiting)s <img alt="%(pending)s" src="%(images)s/waiting_or.gif" border="0" />""" % {
'waiting' : categ['waiting'],
'pending' : _("Pending"),
'images' : CFG_SITE_URL + '/img',
}
if categ['approved'] != 0:
out += """| %(approved)s<img alt="%(approved_text)s" src="%(images)s/smchk_gr.gif" border="0" />""" % {
'approved' : categ['approved'],
'approved_text' : _("Approved"),
'images' : CFG_SITE_URL + '/img',
}
if categ['rejected'] != 0:
out += """| %(rejected)s<img alt="%(rejected_text)s" src="%(images)s/cross_red.gif" border="0" />""" % {
'rejected' : categ['rejected'],
'rejected_text' : _("Rejected"),
'images' : CFG_SITE_URL + '/img',
}
out += ")<br />"
out += """
</td>
<td>
<table class="searchbox" width="100%%" summary="">
<tr>
<th class="portalboxheader">%(key)s:</th>
</tr>
<tr>
<td>
<img alt="%(pending)s" src="%(images)s/waiting_or.gif" border="0" /> %(waiting)s<br />
<img alt="%(approved)s" src="%(images)s/smchk_gr.gif" border="0" /> %(already_approved)s<br />
<img alt="%(rejected)s" src="%(images)s/cross_red.gif" border="0" /> %(rejected_text)s<br /><br />
<small class="blocknote">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</small> %(somepending)s<br />
</td>
</tr>
</table>
</td>
</tr>
</table>
</blockquote>
</td>
</tr>
</table>""" % {
'key' : _("Key"),
'pending' : _("Pending"),
'images' : CFG_SITE_URL + '/img',
'waiting' : _("Waiting for approval"),
'approved' : _("Approved"),
'already_approved' : _("Already approved"),
'rejected' : _("Rejected"),
'rejected_text' : _("Rejected"),
'somepending' : _("Some documents are pending."),
}
return out
def tmpl_publiline_selectcplxcateg(self, ln, doctype, title, types):
"""
Displays the categories from a doctype that the user can select
Parameters:
- 'ln' *string* - The language to display the interface in
- 'doctype' *string* - The doctype
- 'title' *string* - The doctype name
- 'categories' *array* - All the categories that the user can select:
- 'id' *string* - The id of the category
- 'waiting' *int* - The number of documents waiting
- 'approved' *int* - The number of approved documents
- 'rejected' *int* - The number of rejected documents
"""
# load the right message language
_ = gettext_set_language(ln)
out = ""
#out = """
# <table class="searchbox" width="100%%" summary="">
# <tr>
# <th class="portalboxheader">%(title)s: %(list_type)s</th>
# </tr>
# </table><br />
# <table class="searchbox" width="100%%" summary="">
# <tr>""" % {
# 'title' : title,
# 'list_type' : _("List of specific approvals"),
# }
columns = []
columns.append ({'apptype' : 'RRP',
'list_categ' : _("List of refereing categories"),
'id_form' : 0,
})
#columns.append ({'apptype' : 'RPB',
# 'list_categ' : _("List of publication categories"),
# 'id_form' : 1,
# })
#columns.append ({'apptype' : 'RDA',
# 'list_categ' : _("List of direct approval categories"),
# 'id_form' : 2,
# })
for column in columns:
out += """
<td>
<table class="searchbox" width="100%%" summary="">
<tr>
<th class="portalboxheader">%(list_categ)s</th>
</tr>
<tr>
<td class="portalboxbody">
%(choose_categ)s
<blockquote>
<form action="publiline.py" method="get">
<input type="hidden" name="flow" value="cplx" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="categ" value="" />
<input type="hidden" name="apptype" value="%(apptype)s" />
<input type="hidden" name="ln" value="%(ln)s" />
</form>
<table>
<tr>
<td align="left">""" % {
'doctype' : doctype,
'apptype' : column['apptype'],
'list_categ' : column['list_categ'],
'choose_categ' : _("Please choose a category"),
'ln' : ln,
}
for categ in types[column['apptype']]:
num = categ['waiting'] + categ['approved'] + categ['rejected'] + categ['cancelled']
if categ['waiting'] != 0:
classtext = "class=\"blocknote\""
else:
classtext = ""
out += """<table><tr><td width="200px">*&nbsp<a href="" onclick="document.forms[%(id_form)s].categ.value='%(id)s';document.forms[%(id_form)s].submit();return false;"><small %(classtext)s>%(desc)s</small></td><td width="150px"></a><small> Total document(s) : %(num)s """ % {
'id' : categ['id'],
'id_form' : column['id_form'],
'classtext' : classtext,
'num' : num,
'desc' : categ['desc'],
}
out += """<td width="100px">"""
#if categ['waiting'] != 0:
out += """ %(waiting)s &nbsp&nbsp<img alt="%(pending)s" src="%(images)s/waiting_or.gif" border="0" /></td>""" % {
'waiting' : categ['waiting'],
'pending' : _("Pending"),
'images' : CFG_SITE_URL + '/img',
}
out += """<td width="100px">"""
#if categ['approved'] != 0:
out += """ %(approved)s &nbsp&nbsp<img alt="%(approved_text)s" src="%(images)s/smchk_gr.gif" border="0" /></td>""" % {
'approved' : categ['approved'],
'approved_text' : _("Approved"),
'images' : CFG_SITE_URL + '/img',
}
out += """<td width="100px">"""
#if categ['rejected'] != 0:
out += """ %(rejected)s&nbsp&nbsp<img alt="%(rejected_text)s" src="%(images)s/cross_red.gif" border="0" /></td>""" % {
'rejected' : categ['rejected'],
'rejected_text' : _("Rejected"),
'images' : CFG_SITE_URL + '/img',
}
out += """<td width="100px">"""
#if categ['cancelled'] != 0:
out += """ %(cancelled)s&nbsp&nbsp<img alt="%(cancelled_text)s" src="%(images)s/smchk_rd.gif" border="0" /></td>""" % {
'cancelled' : categ['cancelled'],
'cancelled_text' : _("Cancelled"),
'images' : CFG_SITE_URL + '/img',
}
out += "</small></td></tr>"
out += """
</table>
</td>
</tr>
</table>
</blockquote>
</td>
</tr>
</table>
</td>"""
# Key
out += """
<table class="searchbox" width="100%%" summary="">
<tr>
<th class="portalboxheader">%(key)s:</th>
</tr>
<tr>
<td>
<img alt="%(pending)s" src="%(images)s/waiting_or.gif" border="0" /> %(waiting)s<br />
<img alt="%(approved)s" src="%(images)s/smchk_gr.gif" border="0" /> %(already_approved)s<br />
<img alt="%(rejected)s" src="%(images)s/cross_red.gif" border="0" /> %(rejected_text)s<br />
<img alt="%(cancelled)s" src="%(images)s/smchk_rd.gif" border="0" /> %(cancelled_text)s<br /><br />
<small class="blocknote">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</small> %(somepending)s<br />
</td>
</tr>
</table>
</blockquote>
</td>
</tr>
</table>""" % {
'key' : _("Key"),
'pending' : _("Pending"),
'images' : CFG_SITE_URL + '/img',
'waiting' : _("Waiting for approval"),
'approved' : _("Approved"),
'already_approved' : _("Already approved"),
'rejected' : _("Rejected"),
'rejected_text' : _("Rejected"),
'cancelled' : _("Cancelled"),
'cancelled_text' : _("Cancelled"),
'somepending' : _("Some documents are pending."),
}
return out
def tmpl_publiline_selectdocument(self, ln, doctype, title, categ, docs):
"""
Displays the documents that the user can select in the specified category
Parameters:
- 'ln' *string* - The language to display the interface in
- 'doctype' *string* - The doctype
- 'title' *string* - The doctype name
- 'categ' *string* - the category
- 'docs' *array* - All the categories that the user can select:
- 'RN' *string* - The id of the document
- 'status' *string* - The status of the document
"""
# load the right message language
_ = gettext_set_language(ln)
out = """
<table class="searchbox" width="100%%" summary="">
<tr>
<th class="portalboxheader">%(title)s - %(categ)s: %(list)s</th>
</tr>
<tr>
<td class="portalboxbody">
%(choose_report)s
<blockquote>
<form action="publiline.py" method="get">
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="categ" value="%(categ)s" />
<input type="hidden" name="RN" value="" />
<input type="hidden" name="ln" value="%(ln)s">
</form>
<table class="searchbox">
<tr>
<th class="portalboxheader">%(report_no)s</th>
<th class="portalboxheader">%(pending)s</th>
<th class="portalboxheader">%(approved)s</th>
<th class="portalboxheader">%(rejected)s</th>
</tr>
""" % {
'doctype' : doctype,
'title' : title,
'categ' : categ,
'list' : _("List of refereed documents"),
'choose_report' : _("Click on a report number for more information."),
'report_no' : _("Report Number"),
'pending' : _("Pending"),
'approved' : _("Approved"),
'rejected' : _("Rejected"),
'ln': ln,
}
for doc in docs:
status = doc ['status']
if status == "waiting":
out += """<tr>
<td align="center">
<a href="" onclick="document.forms[0].RN.value='%(rn)s';document.forms[0].submit();return false;">%(rn)s</a>
</td>
<td align="center">
<img alt="check" src="%(images)s/waiting_or.gif" />
</td>
<td align="center">&nbsp;</td>
<td align="center">&nbsp;</td>
</tr>
""" % {
'rn' : doc['RN'],
'images' : CFG_SITE_URL + '/img',
}
elif status == "rejected":
out += """<tr>
<td align="center">
<a href="" onclick="document.forms[0].RN.value='%(rn)s';document.forms[0].submit();return false;">%(rn)s</a>
</td>
<td align="center">&nbsp;</td>
<td align="center">&nbsp;</td>
<td align="center"><img alt="check" src="%(images)s/cross_red.gif" /></td>
</tr>
""" % {
'rn' : doc['RN'],
'images' : CFG_SITE_URL + '/img',
}
elif status == "approved":
out += """<tr>
<td align="center">
<a href="" onclick="document.forms[0].RN.value='%(rn)s';document.forms[0].submit();return false;">%(rn)s</a>
</td>
<td align="center">&nbsp;</td>
<td align="center"><img alt="check" src="%(images)s/smchk_gr.gif" /></td>
<td align="center">&nbsp;</td>
</tr>
""" % {
'rn' : doc['RN'],
'images' : CFG_SITE_URL + '/img',
}
out += """ </table>
</blockquote>
</td>
</tr>
</table>"""
return out
def tmpl_publiline_selectcplxdocument(self, ln, doctype, title, categ, categname, docs, apptype):
"""
Displays the documents that the user can select in the specified category
Parameters:
- 'ln' *string* - The language to display the interface in
- 'doctype' *string* - The doctype
- 'title' *string* - The doctype name
- 'categ' *string* - the category
- 'docs' *array* - All the categories that the user can select:
- 'RN' *string* - The id of the document
- 'status' *string* - The status of the document
- 'apptype' *string* - the approval type
"""
# load the right message language
_ = gettext_set_language(ln)
listtype = ""
if apptype == "RRP":
listtype = _("List of refereed documents")
elif apptype == "RPB":
listtype = _("List of publication documents")
elif apptype == "RDA":
listtype = _("List of direct approval documents")
out = """
<table class="searchbox" width="100%%" summary="">
<tr>
<th class="portalboxheader">%(title)s - %(categname)s: %(list)s</th>
</tr>
<tr>
<td class="portalboxbody">
%(choose_report)s
<blockquote>
<form action="publiline.py" method="get">
<input type="hidden" name="flow" value="cplx" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="categ" value="%(categ)s" />
<input type="hidden" name="RN" value="" />
<input type="hidden" name="apptype" value="%(apptype)s" />
<input type="hidden" name="ln" value="%(ln)s" />
</form>
<table class="searchbox">
<tr>
<th class="portalboxheader">%(report_no)s</th>
<th class="portalboxheader">%(pending)s</th>
<th class="portalboxheader">%(approved)s</th>
<th class="portalboxheader">%(rejected)s</th>
<th class="portalboxheader">%(cancelled)s</th>
</tr>
""" % {
'doctype' : doctype,
'title' : title,
'categname' : categname,
'categ' : categ,
'list' : listtype,
'choose_report' : _("Click on a report number for more information."),
'apptype' : apptype,
'report_no' : _("Report Number"),
'pending' : _("Pending"),
'approved' : _("Approved"),
'rejected' : _("Rejected"),
'cancelled' : _("Cancelled"),
'ln': ln,
}
for doc in docs:
status = doc ['status']
if status == "waiting":
out += """<tr>
<td align="center">
<a href="" onclick="document.forms[0].RN.value='%(rn)s';document.forms[0].submit();return false;">%(rn)s</a>
</td>
<td align="center"><img alt="check" src="%(images)s/waiting_or.gif" /></td>
<td align="center">&nbsp;</td>
<td align="center">&nbsp;</td>
<td align="center">&nbsp;</td>
</tr>
""" % {
'rn' : doc['RN'],
'images' : CFG_SITE_URL + '/img',
}
elif status == "rejected":
out += """<tr>
<td align="center">
<a href="" onclick="document.forms[0].RN.value='%(rn)s';document.forms[0].submit();return false;">%(rn)s</a>
</td>
<td align="center">&nbsp;</td>
<td align="center">&nbsp;</td>
<td align="center"><img alt="check" src="%(images)s/cross_red.gif" /></td>
<td align="center">&nbsp;</td>
</tr>
""" % {
'rn' : doc['RN'],
'images' : CFG_SITE_URL + '/img',
}
elif status == "approved":
out += """<tr>
<td align="center">
<a href="" onclick="document.forms[0].RN.value='%(rn)s';document.forms[0].submit();return false;">%(rn)s</a>
</td>
<td align="center">&nbsp;</td>
<td align="center"><img alt="check" src="%(images)s/smchk_gr.gif" /></td>
<td align="center">&nbsp;</td>
<td align="center">&nbsp;</td>
</tr>
""" % {
'rn' : doc['RN'],
'images' : CFG_SITE_URL + '/img',
}
elif status == "cancelled":
out += """<tr>
<td align="center">
<a href="" onclick="document.forms[0].RN.value='%(rn)s';document.forms[0].submit();return false;">%(rn)s</a>
</td>
<td align="center">&nbsp;</td>
<td align="center">&nbsp;</td>
<td align="center">&nbsp;</td>
<td align="center"><img alt="check" src="%(images)s/smchk_rd.gif" /></td>
</tr>
""" % {
'rn' : doc['RN'],
'images' : CFG_SITE_URL + '/img',
}
out += """ </table>
</blockquote>
</td>
</tr>
</table>"""
return out
def tmpl_publiline_displaydoc(self, ln, doctype, docname, categ, rn, status, dFirstReq, dLastReq, dAction, access, confirm_send, auth_code, auth_message, authors, title, sysno, newrn, note):
"""
Displays the categories from a doctype that the user can select
Parameters:
- 'ln' *string* - The language to display the interface in
- 'doctype' *string* - The doctype
- 'docname' *string* - The doctype name
- 'categ' *string* - the category
- 'rn' *string* - The document RN (id number)
- 'status' *string* - The status of the document
- 'dFirstReq' *string* - The date of the first approval request
- 'dLastReq' *string* - The date of the last approval request
- 'dAction' *string* - The date of the last action (approval or rejection)
- 'confirm_send' *bool* - must display a confirmation message about sending approval email
- 'auth_code' *bool* - authorised to referee this document
- 'auth_message' *string* - ???
- 'authors' *string* - the authors of the submission
- 'title' *string* - the title of the submission
- 'sysno' *string* - the unique database id for the record
- 'newrn' *string* - the record number assigned to the submission
- 'note' *string* - Note about the approval request.
"""
# load the right message language
_ = gettext_set_language(ln)
if status == "waiting":
image = """<img src="%s/waiting_or.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
elif status == "approved":
image = """<img src="%s/smchk_gr.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
elif status == "rejected":
image = """<img src="%s/iconcross.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
else:
image = ""
out = """
<table class="searchbox" summary="">
<tr>
<th class="portalboxheader">%(image)s %(rn)s</th>
</tr>
<tr>
<td class="portalboxbody">""" % {
'image' : image,
'rn' : rn,
}
if confirm_send:
out += """<i><strong class="headline">%(requestsent)s</strong></i><br /><br />""" % {
'requestsent' : _("Your request has been sent to the referee."),
}
out += """<form action="publiline.py">
<input type="hidden" name="RN" value="%(rn)s" />
<input type="hidden" name="categ" value="%(categ)s" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="ln" value="%(ln)s" />
<small>""" % {
'rn' : rn,
'categ' : categ,
'doctype' : doctype,
'ln' : ln,
}
if title != "unknown":
out += """<strong class="headline">%(title_text)s</strong>%(title)s<br /><br />""" % {
'title_text' : _("Title:"),
'title' : title,
}
if authors != "":
out += """<strong class="headline">%(author_text)s</strong>%(authors)s<br /><br />""" % {
'author_text' : _("Author:"),
'authors' : authors,
}
if sysno != "":
out += """<strong class="headline">%(more)s</strong>
<a href="%(siteurl)s/%(CFG_SITE_RECORD)s/%(sysno)s?ln=%(ln)s">%(click)s</a>
<br /><br />
""" % {
'more' : _("More information:"),
'click' : _("Click here"),
'siteurl' : CFG_SITE_URL,
'CFG_SITE_RECORD': CFG_SITE_RECORD,
'sysno' : sysno,
'ln' : ln,
}
if note and auth_code == 0:
out += """<table><tr><td valign="top"><strong class="headline">%(note_text)s</strong></td><td><em>%(note)s</em></td></tr></table>""" % {
'note_text' : _("Approval note:"),
'note' : cgi.escape(note).replace('\n', '<br />'),
}
if status == "waiting":
out += _("This document is still %(x_fmt_open)swaiting for approval%(x_fmt_close)s.") % {'x_fmt_open': '<strong class="headline">',
'x_fmt_close': '</strong>'}
out += "<br /><br />"
out += _("It was first sent for approval on:") + ' <strong class="headline">' + str(dFirstReq) + '</strong><br />'
if dLastReq == "0000-00-00 00:00:00":
out += _("Last approval email was sent on:") + ' <strong class="headline">' + str(dFirstReq) + '</strong><br />'
else:
out += _("Last approval email was sent on:") + ' <strong class="headline">' + str(dLastReq) + '</strong><br />'
out += "<br />" + _("You can send an approval request email again by clicking the following button:") + " <br />" +\
"""<input class="adminbutton" type="submit" name="send" value="%(send)s" onclick="return confirm('%(warning)s')" />""" % {
'send' : _("Send Again"),
'warning' : _("WARNING! Upon confirmation, an email will be sent to the referee.")
}
if auth_code == 0:
out += "<br />" + _("As a referee for this document, you may approve or reject it from the submission interface") + ":<br />" +\
"""<input class="adminbutton" type="submit" name="approval" value="%(approve)s" onclick="window.location='%(siteurl)s/submit?doctype=%(doctype)s&amp;ln=%(ln)s';return false;" />""" % {
'approve' : _("Approve/Reject"),
'siteurl' : CFG_SITE_URL,
'doctype' : doctype,
'ln' : ln
}
if status == "approved":
out += _("This document has been %(x_fmt_open)sapproved%(x_fmt_close)s.") % {'x_fmt_open': '<strong class="headline">', 'x_fmt_close': '</strong>'}
out += '<br />' + _("Its approved reference is:") + ' <strong class="headline">' + str(newrn) + '</strong><br /><br />'
out += _("It was first sent for approval on:") + ' <strong class="headline">' + str(dFirstReq) + '</strong><br />'
if dLastReq == "0000-00-00 00:00:00":
out += _("Last approval email was sent on:") + ' <strong class="headline">' + str(dFirstReq) + '</strong><br />'
else:
out += _("Last approval email was sent on:") + ' <strong class="headline">' + str(dLastReq) + '</strong><br />' +\
_("It was approved on:") + ' <strong class="headline">' + str(dAction) + '</strong><br />'
if status == "rejected":
out += _("This document has been %(x_fmt_open)srejected%(x_fmt_close)s.") % {'x_fmt_open': '<strong class="headline">', 'x_fmt_close': '</strong>'}
out += "<br /><br />"
out += _("It was first sent for approval on:") + ' <strong class="headline">' + str(dFirstReq) +'</strong><br />'
if dLastReq == "0000-00-00 00:00:00":
out += _("Last approval email was sent on:") + ' <strong class="headline">' + str(dFirstReq) + '</strong><br />'
else:
out += _("Last approval email was sent on:") + ' <strong class="headline">' + str(dLastReq) +'</strong><br />'
out += _("It was rejected on:") + ' <strong class="headline">' + str(dAction) + '</strong><br />'
out += """ </small></form>
<br />
</td>
</tr>
</table>"""
return out
def tmpl_publiline_displaycplxdoc(self, ln, doctype, docname, categ, rn, apptype, status, dates, isPubCom, isEdBoard, isReferee, isProjectLeader, isAuthor, authors, title, sysno, newrn):
# load the right message language
_ = gettext_set_language(ln)
if status == "waiting":
image = """<img src="%s/waiting_or.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
elif status == "approved":
image = """<img src="%s/smchk_gr.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
elif status == "rejected":
image = """<img src="%s/iconcross.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
elif status == "cancelled":
image = """<img src="%s/smchk_rd.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
else:
image = ""
out = """
<table class="searchbox" summary="">
<tr>
<th class="portalboxheader">%(image)s %(rn)s</th>
</tr>
<tr>
<td class="portalboxbody">""" % {
'image' : image,
'rn' : rn,
}
out += """<form action="publiline.py">
<input type="hidden" name="flow" value="cplx" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="categ" value="%(categ)s" />
<input type="hidden" name="RN" value="%(rn)s" />
<input type="hidden" name="apptype" value="%(apptype)s" />
<input type="hidden" name="action" value="" />
<input type="hidden" name="ln" value="%(ln)s" />
""" % {
'rn' : rn,
'categ' : categ,
'doctype' : doctype,
'apptype' : apptype,
'ln': ln,
}
out += "<table><tr height='30px'><td width='120px'>"
if title != "unknown":
out += """<strong class="headline">%(title_text)s</strong></td><td>%(title)s</td></tr>""" % {
'title_text' : _("Title:"),
'title' : title,
}
out += "<tr height='30px'><td width='120px'>"
if authors != "":
out += """<strong class="headline">%(author_text)s</strong></td><td>%(authors)s</td></tr>""" % {
'author_text' : _("Author:"),
'authors' : authors,
}
out += "<tr height='30px'><td width='120px'>"
if sysno != "":
out += """<strong class="headline">%(more)s</strong>
</td><td><a href="%(siteurl)s/%(CFG_SITE_RECORD)s/%(sysno)s?ln=%(ln)s">%(click)s</a>
</td></tr>
""" % {
'more' : _("More information:"),
'click' : _("Click here"),
'siteurl' : CFG_SITE_URL,
'CFG_SITE_RECORD': CFG_SITE_RECORD,
'sysno' : sysno,
'ln' : ln,
}
out += "</table>"
out += "<br /><br />"
if apptype == "RRP":
out += "<table><tr><td width='400px'>"
out += _("It has first been asked for refereing process on the ") + "</td><td>" + ' <strong class="headline">' + str(dates['dFirstReq']) + '</strong><br /></td></tr>'
out += "<tr><td width='400px'>"
out += _("Last request e-mail was sent to the publication committee chair on the ") + "</td><td>" + ' <strong class="headline">' + str(dates['dLastReq']) + '</strong><br /></td></tr>'
if dates['dRefereeSel'] != None:
out += "<tr><td width='400px'>"
out += _("A referee has been selected by the publication committee on the ") + "</td><td>" + ' <strong class="headline">' + str(dates['dRefereeSel']) + '</strong><br /></td></tr>'
else:
out += "<tr><td width='400px'>"
out += _("No referee has been selected yet.") + "</td><td>"
if (status != "cancelled") and (isPubCom == 0):
out += displaycplxdoc_displayauthaction (action="RefereeSel", linkText=_("Select a referee"))
out += '<br /></td></tr>'
if dates['dRefereeRecom'] != None:
out += "<tr><td width='400px'>"
out += _("The referee has sent his final recommendations to the publication committee on the ") + "</td><td>" + ' <strong class="headline">' + str(dates['dRefereeRecom']) + '</strong><br /></td></tr>'
else:
out += "<tr><td width='400px'>"
out += _("No recommendation from the referee yet.") + "</td><td>"
if (status != "cancelled") and (dates['dRefereeSel'] != None) and (isReferee == 0):
out += displaycplxdoc_displayauthaction (action="RefereeRecom", linkText=_("Send a recommendation"))
out += '<br /></td></tr>'
if dates['dPubComRecom'] != None:
out += "<tr><td width='400px'>"
out += _("The publication committee has sent his final recommendations to the project leader on the ") + "</td><td>" + ' <strong class="headline">' + str(dates['dPubComRecom']) + '</strong><br /></td></tr>'
else:
out += "<tr><td width='400px'>"
out += _("No recommendation from the publication committee yet.") + "</td><td>"
if (status != "cancelled") and (dates['dRefereeRecom'] != None) and (isPubCom == 0):
out += displaycplxdoc_displayauthaction (action="PubComRecom", linkText=_("Send a recommendation"))
out += '<br /></td></tr>'
if status == "cancelled":
out += "<tr><td width='400px'>"
out += _("It has been cancelled by the author on the ") + "</td><td>" + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br /></td></tr>'
elif dates['dProjectLeaderAction'] != None:
if status == "approved":
out += "<tr><td width='400px'>"
out += _("It has been approved by the project leader on the ") + "</td><td>" + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br /></td></tr>'
elif status == "rejected":
out += "<tr><td width='400px'>"
out += _("It has been rejected by the project leader on the ") + "</td><td>" + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br /></td></tr>'
else:
out += "<tr><td width='400px'>"
out += _("No final decision taken yet.") + "</td><td>"
if (dates['dPubComRecom'] != None) and (isProjectLeader == 0):
out += displaycplxdoc_displayauthaction (action="ProjectLeaderDecision", linkText=_("Take a decision"))
if isAuthor == 0:
out += displaycplxdoc_displayauthaction (action="AuthorCancel", linkText=_("Cancel"))
out += '<br /></table>'
elif apptype == "RPB":
out += _("It has first been asked for refereing process on the ") + ' <strong class="headline">' + str(dates['dFirstReq']) + '</strong><br />'
out += _("Last request e-mail was sent to the publication committee chair on the ") + ' <strong class="headline">' + str(dates['dLastReq']) + '</strong><br />'
if dates['dEdBoardSel'] != None:
out += _("An editorial board has been selected by the publication committee on the ") + ' <strong class="headline">' + str(dates['dEdBoardSel']) + '</strong>'
if (status != "cancelled") and (isEdBoard == 0):
out += displaycplxdoc_displayauthaction (action="AddAuthorList", linkText=_("Add an author list"))
out += '<br />'
else:
out += _("No editorial board has been selected yet.")
if (status != "cancelled") and (isPubCom == 0):
out += displaycplxdoc_displayauthaction (action="EdBoardSel", linkText=_("Select an editorial board"))
out += '<br />'
if dates['dRefereeSel'] != None:
out += _("A referee has been selected by the editorial board on the ") + ' <strong class="headline">' + str(dates['dRefereeSel']) + '</strong><br />'
else:
out += _("No referee has been selected yet.")
if (status != "cancelled") and (dates['dEdBoardSel'] != None) and (isEdBoard == 0):
out += displaycplxdoc_displayauthaction (action="RefereeSel", linkText=_("Select a referee"))
out += '<br />'
if dates['dRefereeRecom'] != None:
out += _("The referee has sent his final recommendations to the editorial board on the ") + ' <strong class="headline">' + str(dates['dRefereeRecom']) + '</strong><br />'
else:
out += _("No recommendation from the referee yet.")
if (status != "cancelled") and (dates['dRefereeSel'] != None) and (isReferee == 0):
out += displaycplxdoc_displayauthaction (action="RefereeRecom", linkText=_("Send a recommendation"))
out += '<br />'
if dates['dEdBoardRecom'] != None:
out += _("The editorial board has sent his final recommendations to the publication committee on the ") + ' <strong class="headline">' + str(dates['dRefereeRecom']) + '</strong><br />'
else:
out += _("No recommendation from the editorial board yet.")
if (status != "cancelled") and (dates['dRefereeRecom'] != None) and (isEdBoard == 0):
out += displaycplxdoc_displayauthaction (action="EdBoardRecom", linkText=_("Send a recommendation"))
out += '<br />'
if dates['dPubComRecom'] != None:
out += _("The publication committee has sent his final recommendations to the project leader on the ") + ' <strong class="headline">' + str(dates['dPubComRecom']) + '</strong><br />'
else:
out += _("No recommendation from the publication committee yet.")
if (status != "cancelled") and (dates['dEdBoardRecom'] != None) and (isPubCom == 0):
out += displaycplxdoc_displayauthaction (action="PubComRecom", linkText=_("Send a recommendation"))
out += '<br />'
if status == "cancelled":
out += _("It has been cancelled by the author on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
elif dates['dProjectLeaderAction'] != None:
if status == "approved":
out += _("It has been approved by the project leader on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
elif status == "rejected":
out += _("It has been rejected by the project leader on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
else:
out += _("No final decision taken yet.")
if (dates['dPubComRecom'] != None) and (isProjectLeader == 0):
out += displaycplxdoc_displayauthaction (action="ProjectLeaderDecision", linkText=_("Take a decision"))
if isAuthor == 0:
out += displaycplxdoc_displayauthaction (action="AuthorCancel", linkText=_("Cancel"))
out += '<br />'
elif apptype == "RDA":
out += _("It has first been asked for refereing process on the ") + ' <strong class="headline">' + str(dates['dFirstReq']) + '</strong><br />'
out += _("Last request e-mail was sent to the project leader on the ") + ' <strong class="headline">' + str(dates['dLastReq']) + '</strong><br />'
if status == "cancelled":
out += _("It has been cancelled by the author on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
elif dates['dProjectLeaderAction'] != None:
if status == "approved":
out += _("It has been approved by the project leader on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
elif status == "rejected":
out += _("It has been rejected by the project leader on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
else:
out += _("No final decision taken yet.")
if isProjectLeader == 0:
out += displaycplxdoc_displayauthaction (action="ProjectLeaderDecision", linkText=_("Take a decision"))
if isAuthor == 0:
out += displaycplxdoc_displayauthaction (action="AuthorCancel", linkText=_("Cancel"))
out += '<br />'
out += """ </form>
<br />
</td>
</tr>
</table>"""
return out
def tmpl_publiline_displaycplxdocitem(self,
doctype, categ, rn, apptype, action,
comments,
(user_can_view_comments, user_can_add_comment, user_can_delete_comment),
selected_category,
selected_topic, selected_group_id, comment_subject, comment_body, ln):
_ = gettext_set_language(ln)
if comments and user_can_view_comments:
comments_text = ''
comments_overview = '<ul>'
for comment in comments:
(cmt_uid, cmt_nickname, cmt_title, cmt_body, cmt_date, cmt_priority, cmtid) = comment
comments_overview += '<li><a href="#%s">%s - %s</a> (%s)</li>' % (cmtid, cmt_nickname, cmt_title, convert_datetext_to_dategui (cmt_date))
comments_text += """
<table class="bskbasket">
<thead class="bskbasketheader">
<tr><td class="bsktitle"><a name="%s"></a>%s - %s (%s)</td><td><a href=%s/publiline.py?flow=cplx&doctype=%s&apptype=%s&categ=%s&RN=%s&reply=true&commentId=%s&ln=%s#add_comment>Reply</a></td><td><a href="#top">Top</a></td></tr>
</thead>
<tbody>
<tr><td colspan="2">%s</td></tr>
</tbody>
</table>""" % (cmtid, cmt_nickname, cmt_title, convert_datetext_to_dategui (cmt_date), CFG_SITE_URL, doctype, apptype, categ, rn, cmt_uid, ln, email_quoted_txt2html(cmt_body))
comments_overview += '</ul>'
else:
comments_text = ''
comments_overview = 'None.'
body = ''
if user_can_view_comments:
body += """<h4>%(comments_label)s</h4>"""
if user_can_view_comments:
body += """%(comments)s"""
if user_can_add_comment:
validation = """
<input type="hidden" name="validate" value="go" />
<input type="submit" class="formbutton" value="%(button_label)s" />""" % {'button_label': _("Add Comment")}
body += self.tmpl_publiline_displaywritecomment (doctype, categ, rn, apptype, action, _("Add Comment"), comment_subject, validation, comment_body, ln)
body %= {
'comments_label': _("Comments"),
'action': action,
'button_label': _("Write a comment"),
'comments': comments_text}
content = '<br />'
out = """
<table class="bskbasket">
<thead class="bskbasketheader">
<tr>
<td class="bsktitle">
<a name="top"></a>
<h4>%(comments_overview_label)s</h4>
%(comments_overview)s
</td>
<td class="bskcmtcol"></td>
</tr>
</thead>
<tbody>
<tr>
<td colspan="2" style="padding: 5px;">
%(body)s
</td>
</tr>
</tbody>
</table>""" % {
'comments_overview_label' : _('Comments overview'),
'comments_overview' : comments_overview,
'body' : body,}
return out
def tmpl_publiline_displaywritecomment(self, doctype, categ, rn, apptype, action, write_label, title, validation, reply_message, ln):
_ = gettext_set_language(ln)
return """
<div style="width:100%%%%">
<hr />
<h2>%(write_label)s</h2>
<form action="publiline.py">
<input type="hidden" name="flow" value="cplx" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="categ" value="%(categ)s" />
<input type="hidden" name="RN" value="%(rn)s" />
<input type="hidden" name="apptype" value="%(apptype)s" />
<input type="hidden" name="action" value="%(action)s" />
<input type="hidden" name="ln" value="%(ln)s" />
<p class="bsklabel">%(title_label)s:</p>
<a name="add_comment"></a>
<input type="text" name="msg_subject" size="80" value="%(title)s"/>
<p class="bsklabel">%(comment_label)s:</p>
<textarea name="msg_body" rows="20" cols="80">%(reply_message)s</textarea><br />
%(validation)s
</form>
</div>""" % {'write_label': write_label,
'title_label': _("Title"),
'title': title,
'comment_label': _("Comment"),
'rn' : rn,
'categ' : categ,
'doctype' : doctype,
'apptype' : apptype,
'action' : action,
'validation' : validation,
'reply_message' : reply_message,
'ln' : ln,
}
def tmpl_publiline_displaydocplxaction(self, ln, doctype, categ, rn, apptype, action, status, authors, title, sysno, subtitle1, email_user_pattern, stopon1, users, extrausers, stopon2, subtitle2, usersremove, stopon3, validate_btn):
# load the right message language
_ = gettext_set_language(ln)
if status == "waiting":
image = """<img src="%s/waiting_or.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
elif status == "approved":
image = """<img src="%s/smchk_gr.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
elif status == "rejected":
image = """<img src="%s/iconcross.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
else:
image = ""
out = """
<table class="searchbox" summary="">
<tr>
<th class="portalboxheader">%(image)s %(rn)s</th>
</tr>
<tr>
<td class="portalboxbody">
""" % {
'image' : image,
'rn' : rn,
}
if title != "unknown":
out += """<strong class="headline">%(title_text)s</strong>%(title)s<br /><br />""" % {
'title_text' : _("Title:"),
'title' : title,
}
if authors != "":
out += """<strong class="headline">%(author_text)s</strong>%(authors)s<br /><br />""" % {
'author_text' : _("Author:"),
'authors' : authors,
}
if sysno != "":
out += """<strong class="headline">%(more)s</strong>
<a href="%(siteurl)s/%(CFG_SITE_RECORD)s/%(sysno)s?ln=%(ln)s">%(click)s</a>
<br /><br />
""" % {
'more' : _("More information:"),
'click' : _("Click here"),
'siteurl' : CFG_SITE_URL,
'CFG_SITE_RECORD': CFG_SITE_RECORD,
'sysno' : sysno,
'ln' : ln,
}
out += """ <br />
</td>
</tr>
</table>"""
if ((apptype == "RRP") or (apptype == "RPB")) and ((action == "EdBoardSel") or (action == "RefereeSel")):
out += """
<table class="searchbox" summary="">
<tr>
<th class="portalboxheader">%(subtitle)s</th>
</tr>
<tr>
<td class="portalboxbody">""" % {
'subtitle' : subtitle1,
}
out += """<form action="publiline.py">
<input type="hidden" name="flow" value="cplx" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="categ" value="%(categ)s" />
<input type="hidden" name="RN" value="%(rn)s" />
<input type="hidden" name="apptype" value="%(apptype)s" />
<input type="hidden" name="action" value="%(action)s" />
<input type="hidden" name="ln" value="%(ln)s" />""" % {
'rn' : rn,
'categ' : categ,
'doctype' : doctype,
'apptype' : apptype,
'action' : action,
'ln': ln,
}
out += ' <span class="adminlabel">1. %s </span>\n' % _("search for user")
out += ' <input class="admin_wvar" type="text" name="email_user_pattern" value="%s" />\n' % (email_user_pattern, )
out += ' <input class="adminbutton" type="submit" value="%s"/>\n' % (_("search for users"), )
if (stopon1 == "") and (email_user_pattern != ""):
out += ' <br /><span class="adminlabel">2. %s </span>\n' % _("select user")
out += ' <select name="id_user" class="admin_w200">\n'
out += ' <option value="0">*** %s ***</option>\n' % _("select user")
for elem in users:
elem_id = elem[0]
email = elem[1]
out += ' <option value="%s">%s</option>\n' % (elem_id, email)
for elem in extrausers:
elem_id = elem[0]
email = elem[1]
out += ' <option value="%s">%s %s</option>\n' % (elem_id, email, _("connected"))
out += ' </select>\n'
out += ' <input class="adminbutton" type="submit" value="%s" />\n' % (_("add this user"), )
out += stopon2
elif stopon1 != "":
out += stopon1
out += """
</form>
<br />
</td>
</tr>
</table>"""
if action == "EdBoardSel":
out += """
<table class="searchbox" summary="">
<tr>
<th class="portalboxheader">%(subtitle)s</th>
</tr>
<tr>
<td class="portalboxbody">""" % {
'subtitle' : subtitle2,
}
out += """<form action="publiline.py">
<input type="hidden" name="flow" value="cplx" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="categ" value="%(categ)s" />
<input type="hidden" name="RN" value="%(rn)s" />
<input type="hidden" name="apptype" value="%(apptype)s" />
<input type="hidden" name="action" value="%(action)s" />
<input type="hidden" name="ln" value="%(ln)s" />""" % {
'rn' : rn,
'categ' : categ,
'doctype' : doctype,
'apptype' : apptype,
'action' : action,
'ln': ln,
}
out += ' <span class="adminlabel">1. %s </span>\n' % _("select user")
out += ' <select name="id_user_remove" class="admin_w200">\n'
out += ' <option value="0">*** %s ***</option>\n' % _("select user")
for elem in usersremove:
elem_id = elem[0]
email = elem[1]
out += ' <option value="%s">%s</option>\n' % (elem_id, email)
out += ' </select>\n'
out += ' <input class="adminbutton" type="submit" value="%s" />\n' % (_("remove this user"), )
out += stopon3
out += """
</form>
<br />
</td>
</tr>
</table>"""
if validate_btn != "":
out += """<form action="publiline.py">
<input type="hidden" name="flow" value="cplx" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="categ" value="%(categ)s" />
<input type="hidden" name="RN" value="%(rn)s" />
<input type="hidden" name="apptype" value="%(apptype)s" />
<input type="hidden" name="action" value="%(action)s" />
<input type="hidden" name="validate" value="go" />
<input type="hidden" name="ln" value="%(ln)s" />
<input class="adminbutton" type="submit" value="%(validate_btn)s" />
</form>""" % {
'rn' : rn,
'categ' : categ,
'doctype' : doctype,
'apptype' : apptype,
'action' : action,
'validate_btn' : validate_btn,
'ln': ln,
}
return out
def tmpl_publiline_displaycplxrecom(self, ln, doctype, categ, rn, apptype, action, status, authors, title, sysno, msg_to, msg_to_group, msg_subject):
# load the right message language
_ = gettext_set_language(ln)
if status == "waiting":
image = """<img src="%s/waiting_or.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
elif status == "approved":
image = """<img src="%s/smchk_gr.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
elif status == "rejected":
image = """<img src="%s/iconcross.gif" alt="" align="right" />""" % (CFG_SITE_URL + '/img')
else:
image = ""
out = """
<table class="searchbox" summary="">
<tr>
<th class="portalboxheader">%(image)s %(rn)s</th>
</tr>
<tr>
<td class="portalboxbody">
""" % {
'image' : image,
'rn' : rn,
}
if title != "unknown":
out += """<strong class="headline">%(title_text)s</strong>%(title)s<br /><br />""" % {
'title_text' : _("Title:"),
'title' : title,
}
if authors != "":
out += """<strong class="headline">%(author_text)s</strong>%(authors)s<br /><br />""" % {
'author_text' : _("Author:"),
'authors' : authors,
}
if sysno != "":
out += """<strong class="headline">%(more)s</strong>
<a href="%(siteurl)s/%(CFG_SITE_RECORD)s/%(sysno)s?ln=%(ln)s">%(click)s</a>
<br /><br />
""" % {
'more' : _("More information:"),
'click' : _("Click here"),
'siteurl' : CFG_SITE_URL,
'CFG_SITE_RECORD': CFG_SITE_RECORD,
'sysno' : sysno,
'ln' : ln,
}
out += """ <br />
</td>
</tr>
</table>"""
# escape forbidden character
msg_to = escape_html(msg_to)
msg_to_group = escape_html(msg_to_group)
msg_subject = escape_html(msg_subject)
write_box = """
<form action="publiline.py" method="post">
<input type="hidden" name="flow" value="cplx" />
<input type="hidden" name="doctype" value="%(doctype)s" />
<input type="hidden" name="categ" value="%(categ)s" />
<input type="hidden" name="RN" value="%(rn)s" />
<input type="hidden" name="apptype" value="%(apptype)s" />
<input type="hidden" name="action" value="%(action)s" />
<input type="hidden" name="ln" value="%(ln)s" />
<div style="float: left; vertical-align:text-top; margin-right: 10px;">
<table class="mailbox">
<thead class="mailboxheader">
<tr>
<td class="inboxheader" colspan="2">
<table class="messageheader">
<tr>
<td class="mailboxlabel">%(to_label)s</td>"""
if msg_to != "":
addr_box = """
<td class="mailboxlabel">%(users_label)s</td>
<td style="width:100%%%%;" class="mailboxlabel">%(to_users)s</td>""" % {'users_label': _("User"),
'to_users' : msg_to,
}
if msg_to_group != "":
addr_box += """
</tr>
<tr>
<td class="mailboxlabel">&nbsp;</td>
<td class="mailboxlabel">%(groups_label)s</td>
<td style="width:100%%%%;" class="mailboxlabel">%(to_groups)s</td>""" % {'groups_label': _("Group"),
'to_groups': msg_to_group,
}
elif msg_to_group != "":
addr_box = """
<td class="mailboxlabel">%(groups_label)s</td>
<td style="width:100%%%%;" class="mailboxlabel">%(to_groups)s</td>""" % {'groups_label': _("Group"),
'to_groups': msg_to_group,
}
else:
addr_box = """
<td class="mailboxlabel">&nbsp;</td>
<td class="mailboxlabel">&nbsp;</td>"""
write_box += addr_box
write_box += """
</tr>
<tr>
<td class="mailboxlabel">&nbsp;</td>
<td>&nbsp;</td>
<td>&nbsp;</td>
</tr>
<tr>
<td class="mailboxlabel">%(subject_label)s</td>
<td colspan="2">
<input class="mailboxinput" type="text" name="msg_subject" value="%(subject)s" />
</td>
</tr>
</table>
</td>
</tr>
</thead>
<tfoot>
<tr>
<td style="height:0px" colspan="2"></td>
</tr>
</tfoot>
<tbody class="mailboxbody">
<tr>
<td class="mailboxlabel">%(message_label)s</td>
<td>
<textarea name="msg_body" rows="10" cols="50"></textarea>
</td>
</tr>
<tr class="mailboxfooter">
<td>
<select name="validate">
<option value="%(select)s"> %(select)s</option>
<option value="approve">%(approve)s</option>
<option value="reject">%(reject)s</option>
</select>
</td>
<td colspan="2" class="mailboxfoot">
<input type="submit" name="send_button" value="%(send_label)s" class="formbutton"/>
</td>
</tr>
</tbody>
</table>
</div>
</form>
"""
write_box = write_box % {'rn' : rn,
'categ' : categ,
'doctype' : doctype,
'apptype' : apptype,
'action' : action,
'subject' : msg_subject,
'to_label': _("To:"),
'subject_label': _("Subject:"),
'message_label': _("Message:"),
'send_label': _("SEND"),
'select' : _("Select:"),
'approve' : _("approve"),
'reject' : _("reject"),
'ln': ln,
}
out += write_box
return out
def tmpl_mathpreview_header(self, ln, https=False):
"""
Metaheader to add to submit pages in order to preview equation
rendered via MathJax.
@param ln: language.
@param https: True on https pages, False otherwise.
"""
# By default, add tooltip to any suspected 'title' and
# 'abstract' field, as well as those tagged with 'mathpreview'
# class.
# load the right message language
_ = gettext_set_language(ln)
return '''
<script src="%(siteurl)s/js/jquery.mathpreview.js" type="text/javascript"></script>
<script type="text/javascript">
<!--
$(document).ready(function() {
$('textarea[name$="TITLE"], input[type="text"][name$="TITLE"],textarea[name$="ABSTRACT"], input[type="text"][name$="ABSTRACT"], textarea[name$="TTL"], input[type="text"][name$="TTL"], textarea[name$="ABS"], input[type="text"][name$="ABS"], textarea[name$="ABSTR"], input[type="text"][name$="ABSTR"], .mathpreview textarea, .mathpreview input[type="text"], input[type="text"].mathpreview, textarea.mathpreview').mathpreview(
{'help-label': '%(help-label)s',
'help-url' : '%(siteurl)s/help/submit-guide#math-markup'});
})
-->
</script>''' % {
'siteurl': https and CFG_SITE_SECURE_URL or CFG_SITE_URL,
'help-label': escape_javascript_string(_("Use '\\$' delimiters to write LaTeX markup. Eg: \\$e=mc^{2}\\$")),
}
def displaycplxdoc_displayauthaction(action, linkText):
return """ <strong class="headline">(<a href="" onclick="document.forms[0].action.value='%(action)s';document.forms[0].submit();return false;">%(linkText)s</a>)</strong>""" % {
"action" : action,
"linkText" : linkText
}
diff --git a/invenio/modules/access/engine.py b/invenio/modules/access/engine.py
index 3b697e388..1beffc66c 100644
--- a/invenio/modules/access/engine.py
+++ b/invenio/modules/access/engine.py
@@ -1,114 +1,104 @@
## This file is part of Invenio.
## Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""Invenio Access Control Engine in mod_python."""
__revision__ = "$Id$"
import cgi
-import sys
from urllib import quote
-if sys.hexversion < 0x2040000:
- # pylint: disable=W0622
- from sets import Set as set
- # pylint: enable=W0622
-
-from invenio.modules.access.control import \
- acc_find_possible_roles,\
- acc_is_user_in_role, \
- acc_is_user_in_any_role, \
- CFG_SUPERADMINROLE_ID, acc_get_role_users, \
- acc_get_roles_emails
-from invenio.modules.access.local_config import CFG_WEBACCESS_WARNING_MSGS, CFG_WEBACCESS_MSGS
+from .control import acc_find_possible_roles, acc_is_user_in_any_role, acc_get_roles_emails
+from .local_config import CFG_WEBACCESS_WARNING_MSGS, CFG_WEBACCESS_MSGS
from invenio.legacy.webuser import collect_user_info
from invenio.modules.access.firerole import load_role_definition, acc_firerole_extract_emails
from flask.ext.login import current_user
def acc_authorize_action(req, name_action, authorized_if_no_roles=False, batch_args=False, **arguments):
"""
Given the request object (or the user_info dictionary, or the uid), checks
if the user is allowed to run name_action with the given parameters.
If authorized_if_no_roles is True and no role exists (different
than superadmin) that are authorized to execute the given action, the
authorization will be granted.
Returns (0, msg) when the authorization is granted, (1, msg) when it's not.
"""
from invenio.ext.login import UserInfo
from werkzeug.local import LocalProxy
if isinstance(req, LocalProxy):
req = req._get_current_object()
if isinstance(req, UserInfo):
user_info = req
uid = user_info.get_id()
elif type(req) is dict:
uid = req.get('uid', None)
user_info = req
elif type(req) not in [int, long]:
uid = current_user.get_id()
user_info = collect_user_info(uid) # FIXME
else:
user_info = collect_user_info(req)
roles_list = acc_find_possible_roles(name_action, always_add_superadmin=True, batch_args=batch_args, **arguments)
if not batch_args:
roles_list = [roles_list]
result = []
for roles in roles_list:
if acc_is_user_in_any_role(user_info, roles):
## User belong to at least one authorized role
## or User is SUPERADMIN
ret_val = (0, CFG_WEBACCESS_WARNING_MSGS[0])
elif len(roles) <= 1:
## No role is authorized for the given action/arguments
if authorized_if_no_roles:
## User is authorized because no authorization exists for the given
## action/arguments
ret_val = (0, CFG_WEBACCESS_WARNING_MSGS[0])
else:
## User is not authorized.
ret_val = (20, CFG_WEBACCESS_WARNING_MSGS[20] % cgi.escape(name_action))
else:
## User is not authorized
in_a_web_request_p = bool(user_info.get('uri', ''))
ret_val = (1, "%s %s" % (CFG_WEBACCESS_WARNING_MSGS[1], (in_a_web_request_p and "%s %s" % (CFG_WEBACCESS_MSGS[0] % quote(user_info.get('uri', '')), CFG_WEBACCESS_MSGS[1]) or "")))
result.append(ret_val)
+ # FIXME removed CERN specific hack!
return result if batch_args else result[0]
def acc_get_authorized_emails(name_action, **arguments):
"""
Given the action and its arguments, try to retireve all the matching
email addresses of users authorized.
This is a best effort operation, because if a role is authorized and
happens to be defined using a FireRole rule based on regular expression
or on IP addresses, non every email might be returned.
@param name_action: the name of the action.
@type name_action: string
@param arguments: the arguments to the action.
@return: the list of authorized emails.
@rtype: set of string
"""
roles = acc_find_possible_roles(name_action, always_add_superadmin=False, **arguments)
authorized_emails = acc_get_roles_emails(roles)
for id_role in roles:
firerole = load_role_definition(id_role)
authorized_emails = authorized_emails.union(acc_firerole_extract_emails(firerole))
return authorized_emails
diff --git a/invenio/modules/access/local_config.py b/invenio/modules/access/local_config.py
index 1dcf051b4..337aec8f6 100644
--- a/invenio/modules/access/local_config.py
+++ b/invenio/modules/access/local_config.py
@@ -1,709 +1,711 @@
## This file is part of Invenio.
## Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""Invenio Access Control Config. """
__revision__ = \
"$Id$"
# pylint: disable=C0301
from invenio.config import CFG_SITE_NAME, CFG_SITE_URL, CFG_SITE_SECURE_URL, CFG_SITE_SUPPORT_EMAIL, CFG_SITE_RECORD, CFG_SITE_ADMIN_EMAIL
from invenio.base.i18n import _
from invenio.base.globals import cfg as config
# VALUES TO BE EXPORTED
# CURRENTLY USED BY THE FILES access_control_engine.py modules.access.control.py webaccessadmin_lib.py
# name of the role giving superadmin rights
SUPERADMINROLE = 'superadmin'
# name of the webaccess webadmin role
WEBACCESSADMINROLE = 'webaccessadmin'
# name of the action allowing roles to access the web administrator interface
WEBACCESSACTION = 'cfgwebaccess'
# name of the action allowing roles to access the web administrator interface
VIEWRESTRCOLL = 'viewrestrcoll'
# name of the action allowing roles to delegate the rights to other roles
# ex: libraryadmin to delegate libraryworker
DELEGATEADDUSERROLE = 'accdelegaterole'
# max number of users to display in the drop down selects
MAXSELECTUSERS = 25
# max number of users to display in a page (mainly for user area)
MAXPAGEUSERS = 25
# default role definition, source:
CFG_ACC_EMPTY_ROLE_DEFINITION_SRC = 'deny all'
# default role definition, compiled:
CFG_ACC_EMPTY_ROLE_DEFINITION_OBJ = (False, ())
# default role definition, compiled and serialized:
CFG_ACC_EMPTY_ROLE_DEFINITION_SER = None
# List of tags containing (multiple) emails of users who should authorize
# to access the corresponding record regardless of collection restrictions.
#if CFG_CERN_SITE:
# CFG_ACC_GRANT_AUTHOR_RIGHTS_TO_EMAILS_IN_TAGS = ['859__f', '270__m']
#else:
CFG_ACC_GRANT_AUTHOR_RIGHTS_TO_EMAILS_IN_TAGS = ['8560_f']
#if CFG_CERN_SITE:
# CFG_ACC_GRANT_VIEWER_RIGHTS_TO_EMAILS_IN_TAGS = ['506__m']
#else:
CFG_ACC_GRANT_VIEWER_RIGHTS_TO_EMAILS_IN_TAGS = []
# Use external source for access control?
# CFG_EXTERNAL_AUTHENTICATION -- this is a dictionary with the enabled login method.
# The key is the name of the login method and the value is an instance of
# of the login method (see /help/admin/webaccess-admin-guide#5). Set the value
# to None if you wish to use the local Invenio authentication method.
# CFG_EXTERNAL_AUTH_DEFAULT -- set this to the key in CFG_EXTERNAL_AUTHENTICATION
# that should be considered as default login method
# CFG_EXTERNAL_AUTH_USING_SSO -- set this to the login method name of an SSO
# login method, if any, otherwise set this to None.
# CFG_EXTERNAL_AUTH_LOGOUT_SSO -- if CFG_EXTERNAL_AUTH_USING_SSO was not None
# set this to the URL that should be contacted to perform an SSO logout
CFG_EXTERNAL_AUTH_DEFAULT = 'Local'
CFG_EXTERNAL_AUTH_USING_SSO = False
CFG_EXTERNAL_AUTH_LOGOUT_SSO = None
CFG_EXTERNAL_AUTHENTICATION = {
"Local": None,
#"Robot": ExternalAuthRobot(enforce_external_nicknames=True, use_zlib=False),
#"ZRobot": ExternalAuthRobot(enforce_external_nicknames=True, use_zlib=True)
}
# from invenio.legacy.external_authentication.robot import ExternalAuthRobot
# if CFG_CERN_SITE:
# from invenio.legacy.external_authentication import sso as ea_sso
# CFG_EXTERNAL_AUTH_USING_SSO = "CERN"
# CFG_EXTERNAL_AUTH_DEFAULT = CFG_EXTERNAL_AUTH_USING_SSO
# CFG_EXTERNAL_AUTH_LOGOUT_SSO = 'https://login.cern.ch/adfs/ls/?wa=wsignout1.0'
# CFG_EXTERNAL_AUTHENTICATION = {
# CFG_EXTERNAL_AUTH_USING_SSO : ea_sso.ExternalAuthSSO(),
# }
# elif CFG_INSPIRE_SITE:
# # INSPIRE specific robot configuration
# CFG_EXTERNAL_AUTH_DEFAULT = 'Local'
# CFG_EXTERNAL_AUTH_USING_SSO = False
# CFG_EXTERNAL_AUTH_LOGOUT_SSO = None
# CFG_EXTERNAL_AUTHENTICATION = {
# "Local": None,
# "Robot": ExternalAuthRobot(enforce_external_nicknames=True, use_zlib=False, check_user_ip=2, external_id_attribute_name='personid'),
# "ZRobot": ExternalAuthRobot(enforce_external_nicknames=True, use_zlib=True, check_user_ip=2, external_id_attribute_name='personid')
# }
# else:
# CFG_EXTERNAL_AUTH_DEFAULT = 'Local'
# CFG_EXTERNAL_AUTH_USING_SSO = False
# CFG_EXTERNAL_AUTH_LOGOUT_SSO = None
# CFG_EXTERNAL_AUTHENTICATION = {
# "Local": None,
# "Robot": ExternalAuthRobot(enforce_external_nicknames=True, use_zlib=False),
# "ZRobot": ExternalAuthRobot(enforce_external_nicknames=True, use_zlib=True)
# }
# CFG_TEMP_EMAIL_ADDRESS
# Temporary email address for logging in with an OpenID/OAuth provider which
# doesn't supply email address
CFG_TEMP_EMAIL_ADDRESS = "%s@NOEMAIL"
# CFG_OPENID_PROVIDERS
# CFG_OAUTH1_PROVIDERS
# CFG_OAUTH2_PROVIDERS
# Choose which providers you want to use. Some providers don't supply e mail
# address, if you choose them, the users will be registered with an temporary
# email address like CFG_TEMP_EMAIL_ADDRESS % randomstring
#
# Order of the login buttons can be changed by CFG_EXTERNAL_LOGIN_BUTTON_ORDER
# in invenio.websession_config
CFG_OPENID_PROVIDERS = [
'google',
'yahoo',
'aol',
'wordpress',
'myvidoop',
'openid',
'verisign',
'myopenid',
'myspace',
'livejournal',
'blogger'
]
CFG_OAUTH1_PROVIDERS = [
'twitter',
'linkedin',
'flickr'
]
CFG_OAUTH2_PROVIDERS = [
'facebook',
'yammer',
'foursquare',
'googleoauth2',
'instagram',
'orcid'
]
# CFG_OPENID_CONFIGURATIONS
# identifier: (required) identifier url. {0} will be replaced by username (an
# input).
# trust_email: (optional, default: False) Some providers let their users
# change their emails on login page. If the provider doesn't let the user,
# set it True.
CFG_OPENID_CONFIGURATIONS = {
'openid': {
'identifier': '{0}'
},
'myvidoop': {
'identifier': '{0}.myvidoop.com'
},
'google': {
'identifier': 'https://www.google.com/accounts/o8/id',
'trust_email': True
},
'wordpress': {
'identifier': '{0}.wordpress.com'
},
'aol': {
'identifier': 'openid.aol.com/{0}',
'trust_email': True
},
'myopenid': {
'identifier': '{0}.myopenid.com'
},
'yahoo': {
'identifier': 'yahoo.com',
'trust_email': True
},
'verisign': {
'identifier': '{0}.pip.verisignlabs.com'
},
'myspace': {
'identifier': 'www.myspace.com/{0}'
},
'livejournal': {
'identifier': '{0}.livejournal.com'
},
'blogger': {
'identifier': '{0}'
}
}
# CFG_OAUTH1_CONFIGURATIONS
#
# !!IMPORTANT!!
# While creating an app in the provider site, the callback uri (redirect uri)
# must be in the form of :
# CFG_SITE_SECURE_URL/youraccount/login?login_method=oauth1&provider=PROVIDERNAME
#
# consumer_key: required
# Consumer key taken from provider.
#
# consumer_secret: required
# Consumer secret taken from provider.
#
# authorize_url: required
# The url to redirect the user for authorization
#
# authorize_parameters: optional
# Additional parameters for authorize_url (ie. scope)
#
# request_token_url: required
# The url to get request token
#
# access_token_url: required
# The url to exchange the request token with the access token
#
# request_url: optional
# The url to gather the user information
#
# request_parameters: optional
# Additional parameters for request_url
#
# email, nickname: optional
# id: required
# The location where these properties in the response returned from the
# provider.
# example:
# if the response is:
# {
# 'user': {
# 'user_name': 'ABC',
# 'contact': [
# {
# 'email': 'abc@def.com'
# }
# ]
# },
# 'user_id': 'XXX',
# }
# then:
# email must be : ['user', 'contact', 0, 'email']
# id must be: ['user_id']
# nickname must be: ['user', 'user_name']
#
# debug: optional
# When debug key is set to 1, after login process, the json object
# returned from provider is displayed on the screen. It may be used
# for finding where the id, email or nickname is.
CFG_OAUTH1_CONFIGURATIONS = {
'twitter': {
'consumer_key' : '',
'consumer_secret' : '',
'request_token_url' : 'https://api.twitter.com/oauth/request_token',
'access_token_url' : 'https://api.twitter.com/oauth/access_token',
'authorize_url' : 'https://api.twitter.com/oauth/authorize',
'id': ['user_id'],
'nickname': ['screen_name']
},
'flickr': {
'consumer_key' : '',
'consumer_secret' : '',
'request_token_url' : 'http://www.flickr.com/services/oauth/request_token',
'access_token_url' : 'http://www.flickr.com/services/oauth/access_token',
'authorize_url' : 'http://www.flickr.com/services/oauth/authorize',
'authorize_parameters': {
'perms': 'read'
},
'nickname': ['username'],
'id': ['user_nsid']
},
'linkedin': {
'consumer_key' : '',
'consumer_secret' : '',
'request_token_url' : 'https://api.linkedin.com/uas/oauth/requestToken',
'access_token_url' : 'https://api.linkedin.com/uas/oauth/accessToken',
'authorize_url' : 'https://www.linkedin.com/uas/oauth/authorize',
'request_url': 'http://api.linkedin.com/v1/people/~:(id)',
'request_parameters': {
'format': 'json'
},
'id': ['id']
}
}
# CFG_OAUTH2_CONFIGURATIONS
#
# !!IMPORTANT!!
# While creating an app in the provider site, the callback uri (redirect uri)
# must be in the form of :
# CFG_SITE_SECURE_URL/youraccount/login?login_method=oauth2&provider=PROVIDERNAME
#
# consumer_key: required
# Consumer key taken from provider.
#
# consumer_secret: required
# Consumer secret taken from provider.
#
# authorize_url: required
# The url to redirect the user for authorization
#
# authorize_parameters:
# Additional parameters for authorize_url (like scope)
#
# access_token_url: required
# The url to get the access token.
#
# request_url: required
# The url to gather the user information.
# {access_token} will be replaced by access token
#
# email, nickname: optional
# id: required
# The location where these properties in the response returned from the
# provider.
# !! See the example in CFG_OAUTH1_CONFIGURATIONS !!
#
# debug: optional
# When debug key is set to 1, after login process, the json object
# returned from provider is displayed on the screen. It may be used
# for finding where the id, email or nickname is.
CFG_OAUTH2_CONFIGURATIONS = {
'facebook': {
'consumer_key': '',
'consumer_secret': '',
'access_token_url': 'https://graph.facebook.com/oauth/access_token',
'authorize_url': 'https://www.facebook.com/dialog/oauth',
'authorize_parameters': {
'scope': 'email'
},
'request_url' : 'https://graph.facebook.com/me?access_token={access_token}',
'email': ['email'],
'id': ['id'],
'nickname': ['username']
},
'foursquare': {
'consumer_key': '',
'consumer_secret': '',
'access_token_url': 'https://foursquare.com/oauth2/access_token',
'authorize_url': 'https://foursquare.com/oauth2/authorize',
'request_url': 'https://api.foursquare.com/v2/users/self?oauth_token={access_token}',
'id': ['response', 'user', 'id'],
'email': ['response', 'user', 'contact' ,'email']
},
'yammer': {
'consumer_key': '',
'consumer_secret': '',
'access_token_url': 'https://www.yammer.com/oauth2/access_token.json',
'authorize_url': 'https://www.yammer.com/dialog/oauth',
'request_url': 'https://www.yammer.com/oauth2/access_token.json?access_token={access_token}',
'email':['user', 'contact', 'email_addresses', 0, 'address'],
'id': ['user', 'id'],
'nickname': ['user', 'name']
},
'googleoauth2': {
'consumer_key': '',
'consumer_secret': '',
'access_token_url': 'https://accounts.google.com/o/oauth2/token',
'authorize_url': 'https://accounts.google.com/o/oauth2/auth',
'authorize_parameters': {
'scope': 'https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email'
},
'request_url': 'https://www.googleapis.com/oauth2/v1/userinfo?access_token={access_token}',
'email':['email'],
'id': ['id']
},
'instagram': {
'consumer_key': '',
'consumer_secret': '',
'access_token_url': 'https://api.instagram.com/oauth/access_token',
'authorize_url': 'https://api.instagram.com/oauth/authorize/',
'authorize_parameters': {
'scope': 'basic'
},
'id': ['user', 'id'],
'nickname': ['user', 'username']
},
'orcid': {
- 'consumer_key': '0000-0002-8651-6707',
- 'consumer_secret': '215e23ca-7421-4543-8306-a105fe0b5688',
+ 'consumer_key': '',
+ 'consumer_secret': '',
'authorize_url': 'http://sandbox-1.orcid.org/oauth/authorize',
'access_token_url': 'http://api.sandbox-1.orcid.org/oauth/token',
'request_url': 'http://api.sandbox-1.orcid.org/{id}/orcid-profile',
'authorize_parameters': {
'scope': '/orcid-profile/read-limited',
'response_type': 'code',
'access_type': 'offline',
},
'id': ['orcid'],
}
}
## Let's override OpenID/OAuth1/OAuth2 configuration from invenio(-local).conf
CFG_OPENID_PROVIDERS = config['CFG_OPENID_PROVIDERS']
CFG_OAUTH1_PROVIDERS = config['CFG_OAUTH1_PROVIDERS']
CFG_OAUTH2_PROVIDERS = config['CFG_OAUTH2_PROVIDERS']
if config['CFG_OPENID_CONFIGURATIONS']:
for provider, configuration in config['CFG_OPENID_CONFIGURATIONS'].items():
if provider in CFG_OPENID_CONFIGURATIONS:
CFG_OPENID_CONFIGURATIONS[provider].update(configuration)
else:
CFG_OPENID_CONFIGURATIONS[provider] = configuration
if config['CFG_OAUTH1_CONFIGURATIONS']:
for provider, configuration in config['CFG_OAUTH1_CONFIGURATIONS'].items():
if provider in CFG_OAUTH1_CONFIGURATIONS:
CFG_OAUTH1_CONFIGURATIONS[provider].update(configuration)
else:
CFG_OAUTH1_CONFIGURATIONS[provider] = configuration
if config['CFG_OAUTH2_CONFIGURATIONS']:
for provider, configuration in config['CFG_OAUTH2_CONFIGURATIONS'].items():
if provider in CFG_OAUTH2_CONFIGURATIONS:
CFG_OAUTH2_CONFIGURATIONS[provider].update(configuration)
else:
CFG_OAUTH2_CONFIGURATIONS[provider] = configuration
# If OpenID authentication is enabled, add 'openid' to login methods
CFG_OPENID_AUTHENTICATION = bool(CFG_OPENID_PROVIDERS)
if CFG_OPENID_AUTHENTICATION:
from invenio.legacy.external_authentication.openid import ExternalOpenID
CFG_EXTERNAL_AUTHENTICATION['openid'] = ExternalOpenID(enforce_external_nicknames=True)
# If OAuth1 authentication is enabled, add 'oauth1' to login methods.
CFG_OAUTH1_AUTHENTICATION = bool(CFG_OAUTH1_PROVIDERS)
-if CFG_OAUTH1_PROVIDERS:
+if CFG_OAUTH1_AUTHENTICATION:
from invenio.legacy.external_authentication.oauth1 import ExternalOAuth1
CFG_EXTERNAL_AUTHENTICATION['oauth1'] = ExternalOAuth1(enforce_external_nicknames=True)
# If OAuth2 authentication is enabled, add 'oauth2' to login methods.
CFG_OAUTH2_AUTHENTICATION = bool(CFG_OAUTH2_PROVIDERS)
if CFG_OAUTH2_AUTHENTICATION:
from invenio.legacy.external_authentication.oauth2 import ExternalOAuth2
CFG_EXTERNAL_AUTHENTICATION['oauth2'] = ExternalOAuth2(enforce_external_nicknames=True)
## If using SSO, this is the number of seconds after which the keep-alive
## SSO handler is pinged again to provide fresh SSO information.
CFG_EXTERNAL_AUTH_SSO_REFRESH = 600
# default data for the add_default_settings function
# Note: by default the definition is set to deny any. This won't be a problem
# because userid directly connected with roles will still be allowed.
# roles
# name description definition
DEF_ROLES = ((SUPERADMINROLE, 'superuser with all rights', 'deny any'),
(WEBACCESSADMINROLE, 'WebAccess administrator', 'deny any'),
('anyuser', 'Any user', 'allow any'),
('basketusers', 'Users who can use baskets', 'allow any'),
('loanusers', 'Users who can use loans', 'allow any'),
('groupusers', 'Users who can use groups', 'allow any'),
('alertusers', 'Users who can use alerts', 'allow any'),
('messageusers', 'Users who can use messages', 'allow any'),
('holdingsusers', 'Users who can view holdings', 'allow any'),
('statisticsusers', 'Users who can view statistics', 'allow any'),
('claimpaperusers', 'Users who can perform changes to their own paper attributions without the need for an operator\'s approval', 'allow any'),
('claimpaperoperators', 'Users who can perform changes to _all_ paper attributions without the need for an operator\'s approval', 'deny any'),
('paperclaimviewers', 'Users who can view "claim my paper" facilities.', 'allow all'),
('paperattributionviewers', 'Users who can view "attribute this paper" facilities', 'allow all'),
('paperattributionlinkviewers', 'Users who can see attribution links in the search', 'allow all'),
('authorlistusers', 'Users who can user Authorlist tools', 'deny all'),
)
# Demo site roles
DEF_DEMO_ROLES = (('photocurator', 'Photo collection curator', 'deny any'),
('thesesviewer', 'Theses and Drafts viewer', 'allow group "Theses and Drafts viewers"'),
('ALEPHviewer', 'ALEPH viewer', 'allow group "ALEPH viewers"'),
('ISOLDEnotesviewer', 'ISOLDE Internal Notes viewer', 'allow group "ISOLDE Internal Notes viewers"'), ('thesescurator', 'Theses collection curator', 'deny any'),
('swordcurator', 'BibSword client curator', 'deny any'),
('referee_DEMOBOO_*', 'Book collection curator', 'deny any'),
('restrictedpicturesviewer', 'Restricted pictures viewer', 'deny any'),
('curator', 'Curator', 'deny any'),
('basketusers', 'Users who can use baskets', 'deny email "hyde@cds.cern.ch"\nallow any'),
('claimpaperusers', 'Users who can perform changes to their own paper attributions without the need for an operator\'s approval', 'deny email "hyde@cds.cern.ch"\nallow any'),
('submit_DEMOJRN_*', 'Users who can submit (and modify) "Atlantis Times" articles', 'deny all'),
('atlantiseditor', 'Users who can configure "Atlantis Times" journal', 'deny all'),
('commentmoderator', 'Users who can moderate comments', 'deny all'),
('poetrycommentreader', 'Users who can view comments in Poetry collection', 'deny all'))
DEF_DEMO_USER_ROLES = (('jekyll@cds.cern.ch', 'thesesviewer'),
('balthasar.montague@cds.cern.ch', 'ALEPHviewer'),
('dorian.gray@cds.cern.ch', 'ISOLDEnotesviewer'),
('jekyll@cds.cern.ch', 'swordcurator'),
('jekyll@cds.cern.ch', 'claimpaperusers'),
('dorian.gray@cds.cern.ch', 'referee_DEMOBOO_*'),
('balthasar.montague@cds.cern.ch', 'curator'),
('romeo.montague@cds.cern.ch', 'restrictedpicturesviewer'),
('romeo.montague@cds.cern.ch', 'swordcurator'),
('romeo.montague@cds.cern.ch', 'thesescurator'),
('juliet.capulet@cds.cern.ch', 'restrictedpicturesviewer'),
('juliet.capulet@cds.cern.ch', 'photocurator'),
('romeo.montague@cds.cern.ch', 'submit_DEMOJRN_*'),
('juliet.capulet@cds.cern.ch', 'submit_DEMOJRN_*'),
('balthasar.montague@cds.cern.ch', 'atlantiseditor'),
('romeo.montague@cds.cern.ch', 'poetrycommentreader'),
('jekyll@cds.cern.ch', 'authorlistusers'),)
# users
# list of e-mail addresses
DEF_USERS = []
# actions
# name desc allowedkeywords optional
DEF_ACTIONS = (
('cfgwebsearch', 'configure WebSearch', '', 'no'),
('cfgbibformat', 'configure BibFormat', '', 'no'),
('cfgbibknowledge', 'configure BibKnowledge', '', 'no'),
('cfgwebsubmit', 'configure WebSubmit', '', 'no'),
('cfgbibrank', 'configure BibRank', '', 'no'),
('cfgwebcomment', 'configure WebComment', '', 'no'),
('cfgweblinkback', 'configure WebLinkback' , '', 'no'),
('cfgoaiharvest', 'configure OAI Harvest', '', 'no'),
('cfgoairepository', 'configure OAI Repository', '', 'no'),
('cfgbibindex', 'configure BibIndex', '', 'no'),
('cfgbibexport', 'configure BibExport', '', 'no'),
('cfgrobotkeys', 'configure Robot keys', 'login_method,robot', 'yes'),
('cfgbibsort', 'configure BibSort', '', 'no'),
('runbibindex', 'run BibIndex', '', 'no'),
('runbibupload', 'run BibUpload', '', 'no'),
('runwebcoll', 'run webcoll', 'collection', 'yes'),
('runbibformat', 'run BibFormat', 'format', 'yes'),
('runbibclassify', 'run BibClassify', 'taxonomy', 'yes'),
('runbibtaskex', 'run BibTaskEx example', '', 'no'),
('runbibrank', 'run BibRank', '', 'no'),
('runoaiharvest', 'run oaiharvest task', '', 'no'),
('runoairepository', 'run oairepositoryupdater task', '', 'no'),
('runbibedit', 'run Record Editor', 'collection', 'yes'),
('runbibeditmulti', 'run Multi-Record Editor', '', 'no'),
('runbibdocfile', 'run Document File Manager', '', 'no'),
('runbibmerge', 'run Record Merger', '', 'no'),
('runbibswordclient', 'run BibSword client', '', 'no'),
('runwebstatadmin', 'run WebStadAdmin', '', 'no'),
('runinveniogc', 'run InvenioGC', '', 'no'),
('runbibexport', 'run BibExport', '', 'no'),
('runauthorlist', 'run Authorlist tools', '', 'no'),
('referee', 'referee document type doctype/category categ', 'doctype,categ', 'yes'),
('submit', 'use webSubmit', 'doctype,act,categ', 'yes'),
('viewrestrdoc', 'view restricted document', 'status', 'no'),
('viewrestrcomment', 'view restricted comment', 'status', 'no'),
(WEBACCESSACTION, 'configure WebAccess', '', 'no'),
(DELEGATEADDUSERROLE, 'delegate subroles inside WebAccess', 'role', 'no'),
(VIEWRESTRCOLL, 'view restricted collection', 'collection', 'no'),
('cfgwebjournal', 'configure WebJournal', 'name,with_editor_rights', 'no'),
('viewcomment', 'view comments', 'collection', 'no'),
('viewlinkbacks', 'view linkbacks', 'collection', 'no'),
('sendcomment', 'send comments', 'collection', 'no'),
('attachcommentfile', 'attach files to comments', 'collection', 'no'),
('attachsubmissionfile', 'upload files to drop box during submission', '', 'no'),
('cfgbibexport', 'configure BibExport', '', 'no'),
('runbibexport', 'run BibExport', '', 'no'),
('usebaskets', 'use baskets', '', 'no'),
('useloans', 'use loans', '', 'no'),
('usegroups', 'use groups', '', 'no'),
('usealerts', 'use alerts', '', 'no'),
('usemessages', 'use messages', '', 'no'),
('viewholdings', 'view holdings', 'collection', 'yes'),
('viewstatistics', 'view statistics', 'collection', 'yes'),
('runbibcirculation', 'run BibCirculation', '', 'no'),
('moderatecomments', 'moderate comments', 'collection', 'no'),
('moderatelinkbacks', 'moderate linkbacks', 'collection', 'no'),
('runbatchuploader', 'run batchuploader', 'collection', 'yes'),
('runbibtasklet', 'run BibTaskLet', '', 'no'),
('claimpaper_view_pid_universe', 'View the Claim Paper interface', '', 'no'),
('claimpaper_claim_own_papers', 'Clam papers to his own personID', '', 'no'),
('claimpaper_claim_others_papers', 'Claim papers for others', '', 'no'),
('claimpaper_change_own_data', 'Change data associated to his own person ID', '', 'no'),
('claimpaper_change_others_data', 'Change data of any person ID', '', 'no'),
('runbibtasklet', 'run BibTaskLet', '', 'no'),
('cfgbibsched', 'configure BibSched', '', 'no'),
('runinfomanager', 'run Info Space Manager', '', 'no')
)
# Default authorizations
# role action arguments
DEF_AUTHS = (('basketusers', 'usebaskets', {}),
('loanusers', 'useloans', {}),
('groupusers', 'usegroups', {}),
('alertusers', 'usealerts', {}),
('messageusers', 'usemessages', {}),
('holdingsusers', 'viewholdings', {}),
('statisticsusers', 'viewstatistics', {}),
('authorlistusers', 'runauthorlist', {}),
('claimpaperusers', 'claimpaper_view_pid_universe', {}),
('claimpaperoperators', 'claimpaper_view_pid_universe', {}),
('claimpaperusers', 'claimpaper_claim_own_papers', {}),
('claimpaperoperators', 'claimpaper_claim_own_papers', {}),
('claimpaperoperators', 'claimpaper_claim_others_papers', {}),
('claimpaperusers', 'claimpaper_change_own_data', {}),
('claimpaperoperators', 'claimpaper_change_own_data', {}),
('claimpaperoperators', 'claimpaper_change_others_data', {}),
)
# Demo site authorizations
# role action arguments
DEF_DEMO_AUTHS = (
('photocurator', 'runwebcoll', {'collection': 'Pictures'}),
('restrictedpicturesviewer', 'viewrestrdoc', {'status': 'restricted_picture'}),
('thesesviewer', VIEWRESTRCOLL, {'collection': 'Theses'}),
('thesesviewer', VIEWRESTRCOLL, {'collection': 'Drafts'}),
('ALEPHviewer', VIEWRESTRCOLL, {'collection': 'ALEPH Theses'}),
('ALEPHviewer', VIEWRESTRCOLL, {'collection': 'ALEPH Internal Notes'}),
('ISOLDEnotesviewer', VIEWRESTRCOLL, {'collection': 'ISOLDE Internal Notes'}),
('referee_DEMOBOO_*', 'referee', {'doctype': 'DEMOBOO', 'categ': '*'}),
('curator', 'cfgbibknowledge', {}),
('curator', 'runbibedit', {}),
('curator', 'runbibeditmulti', {}),
('curator', 'runbibmerge', {}),
('swordcurator', 'runbibswordclient', {}),
('thesescurator', 'runbibedit', {'collection': 'Theses'}),
('thesescurator', VIEWRESTRCOLL, {'collection': 'Theses'}),
('photocurator', 'runbibedit', {'collection': 'Pictures'}),
('referee_DEMOBOO_*', 'runbibedit', {'collection': 'Books'}),
('submit_DEMOJRN_*', 'submit', {'doctype': 'DEMOJRN', 'act': 'SBI', 'categ': '*'}),
('submit_DEMOJRN_*', 'submit', {'doctype': 'DEMOJRN', 'act': 'MBI', 'categ': '*'}),
('submit_DEMOJRN_*', 'cfgwebjournal', {'name': 'AtlantisTimes', 'with_editor_rights': 'no'}),
('atlantiseditor', 'cfgwebjournal', {'name': 'AtlantisTimes', 'with_editor_rights': 'yes'}),
('referee_DEMOBOO_*', 'runbatchuploader', {'collection': 'Books'}),
('poetrycommentreader', 'viewcomment', {'collection': 'Poetry'}),
('atlantiseditor', VIEWRESTRCOLL, {'collection': 'Atlantis Times Drafts'}),
('anyuser', 'submit', {'doctype': 'DEMOART', 'act': 'SBI', 'categ': 'ARTICLE'}),
)
# Activities (i.e. actions) for which exists an administrative web interface.
CFG_ACC_ACTIVITIES_URLS = {
'runbibedit' : (_("Run Record Editor"), "%s/%s/edit/?ln=%%s" % (CFG_SITE_URL, CFG_SITE_RECORD)),
'runbibeditmulti' : (_("Run Multi-Record Editor"), "%s/%s/multiedit/?ln=%%s" % (CFG_SITE_URL, CFG_SITE_RECORD)),
'runbibdocfile' : (_("Run Document File Manager"), "%s/%s/managedocfiles?ln=%%s" % (CFG_SITE_URL, CFG_SITE_RECORD)),
'runbibmerge' : (_("Run Record Merger"), "%s/%s/merge/?ln=%%s" % (CFG_SITE_URL, CFG_SITE_RECORD)),
'runbibswordclient' : (_("Run BibSword client"), "%s/bibsword/?ln=%%s" % CFG_SITE_URL),
'cfgbibknowledge' : (_("Configure BibKnowledge"), "%s/kb?ln=%%s" % CFG_SITE_URL),
'cfgbibformat' : (_("Configure BibFormat"), "%s/admin/bibformat/bibformatadmin.py?ln=%%s" % CFG_SITE_URL),
'cfgoaiharvest' : (_("Configure OAI Harvest"), "%s/admin/oaiharvest/oaiharvestadmin.py?ln=%%s" % CFG_SITE_URL),
'cfgoairepository' : (_("Configure OAI Repository"), "%s/admin/oairepository/oairepositoryadmin.py?ln=%%s" % CFG_SITE_URL),
'cfgbibindex' : (_("Configure BibIndex"), "%s/admin/bibindex/bibindexadmin.py?ln=%%s" % CFG_SITE_URL),
'cfgbibrank' : (_("Configure BibRank"), "%s/admin/bibrank/bibrankadmin.py?ln=%%s" % CFG_SITE_URL),
'cfgwebaccess' : (_("Configure WebAccess"), "%s/admin/webaccess/webaccessadmin.py?ln=%%s" % CFG_SITE_URL),
'cfgwebcomment' : (_("Configure WebComment"), "%s/admin/webcomment/webcommentadmin.py?ln=%%s" % CFG_SITE_URL),
'cfgweblinkback' : (_("Configure WebLinkback"), "%s/admin/weblinkback/weblinkbackadmin.py?ln=%%s" % CFG_SITE_URL),
'cfgwebsearch' : (_("Configure WebSearch"), "%s/admin/websearch/websearchadmin.py?ln=%%s" % CFG_SITE_URL),
'cfgwebsubmit' : (_("Configure WebSubmit"), "%s/admin/websubmit/websubmitadmin.py?ln=%%s" % CFG_SITE_URL),
'cfgwebjournal' : (_("Configure WebJournal"), "%s/admin/webjournal/webjournaladmin.py?ln=%%s" % CFG_SITE_URL),
'cfgbibsort' : (_("Configure BibSort"), "%s/admin/bibsort/bibsortadmin.py?ln=%%s" % CFG_SITE_URL),
'runbibcirculation' : (_("Run BibCirculation"), "%s/admin/bibcirculation/bibcirculationadmin.py?ln=%%s" % CFG_SITE_URL),
'runbatchuploader' : (_("Run Batch Uploader"), "%s/batchuploader/metadata?ln=%%s" % CFG_SITE_URL),
'runinfomanager' : (_("Run Info Space Manager"), "%s/info/manage?ln=%%s" % CFG_SITE_URL),
'claimpaper_claim_others_papers' : (_("Run Person/Author Manager"), "%s/author/search?ln=%%s" % CFG_SITE_URL)
}
CFG_WEBACCESS_MSGS = {
0: 'Try to <a href="%s/youraccount/login?referer=%%s">login</a> with another account.' % (CFG_SITE_SECURE_URL),
1: '<br />If you think this is not correct, please contact: <a href="mailto:%s">%s</a>' % (CFG_SITE_SUPPORT_EMAIL, CFG_SITE_SUPPORT_EMAIL),
2: '<br />If you have any questions, please write to <a href="mailto:%s">%s</a>' % (CFG_SITE_SUPPORT_EMAIL, CFG_SITE_SUPPORT_EMAIL),
3: 'Guest users are not allowed, please <a href="%s/youraccount/login">login</a>.' % CFG_SITE_SECURE_URL,
4: 'The site is temporarily closed for maintenance. Please come back soon.',
5: 'Authorization failure',
6: '%s temporarily closed' % CFG_SITE_NAME,
7: 'This functionality is temporarily closed due to server maintenance. Please use only the search engine in the meantime.',
- 8: 'Functionality temporarily closed'
+ 8: 'Functionality temporarily closed',
+ 9: '<br />If you think this is not correct, please contact: <a href="mailto:%s">%s</a>',
+ 10: '<br />You might also want to check <a href="%s">%s</a>',
}
CFG_WEBACCESS_WARNING_MSGS = {
0: 'Authorization granted',
1: 'You are not authorized to perform this action.',
2: 'You are not authorized to perform any action.',
3: 'The action %s does not exist.',
4: 'Unexpected error occurred.',
5: 'Missing mandatory keyword argument(s) for this action.',
6: 'Guest accounts are not authorized to perform this action.',
7: 'Not enough arguments, user ID and action name required.',
8: 'Incorrect keyword argument(s) for this action.',
9: """Account '%s' is not yet activated.""",
10: """You were not authorized by the authentication method '%s'.""",
11: """The selected login method '%s' is not the default method for this account, please try another one.""",
12: """Selected login method '%s' does not exist.""",
13: """Could not register '%s' account.""",
14: """Could not login using '%s', because this user is unknown.""",
15: """Could not login using your '%s' account, because you have introduced a wrong password.""",
16: """External authentication troubles using '%s' (maybe temporary network problems).""",
17: """You have not yet confirmed the email address for the '%s' authentication method.""",
18: """The administrator has not yet activated your account for the '%s' authentication method.""",
19: """The site is having troubles in sending you an email for confirming your email address. The error has been logged and will be taken care of as soon as possible.""",
20: """No roles are authorized to perform action %s with the given parameters.""",
21: """Verification cancelled""",
22: """Verification failed. Please try again or use another provider to login""",
23: """Verification failed. It is probably because the configuration isn't set properly. Please contact with the <a href="mailto:%s">administator</a>""" % CFG_SITE_ADMIN_EMAIL
}
diff --git a/invenio/modules/comments/api.py b/invenio/modules/comments/api.py
index d0cf4bd45..9b81ee084 100644
--- a/invenio/modules/comments/api.py
+++ b/invenio/modules/comments/api.py
@@ -1,2181 +1,2193 @@
# -*- coding: utf-8 -*-
## This file is part of Invenio.
-## Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2012, 2013 CERN.
+## Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
""" Comments and reviews for records """
__revision__ = "$Id$"
# non Invenio imports:
import time
import math
import os
+import shutil
import cgi
import re
from datetime import datetime, timedelta
from six import iteritems
# Invenio imports:
from invenio.legacy.dbquery import run_sql
from invenio.config import CFG_PREFIX, \
CFG_SITE_LANG, \
CFG_WEBALERT_ALERT_ENGINE_EMAIL,\
CFG_SITE_SUPPORT_EMAIL,\
CFG_WEBCOMMENT_ALERT_ENGINE_EMAIL,\
CFG_SITE_URL,\
CFG_SITE_NAME,\
CFG_WEBCOMMENT_ALLOW_REVIEWS,\
CFG_WEBCOMMENT_ALLOW_SHORT_REVIEWS,\
CFG_WEBCOMMENT_ALLOW_COMMENTS,\
CFG_WEBCOMMENT_ADMIN_NOTIFICATION_LEVEL,\
CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN,\
CFG_WEBCOMMENT_TIMELIMIT_PROCESSING_COMMENTS_IN_SECONDS,\
CFG_WEBCOMMENT_DEFAULT_MODERATOR, \
CFG_SITE_RECORD, \
CFG_WEBCOMMENT_EMAIL_REPLIES_TO, \
CFG_WEBCOMMENT_ROUND_DATAFIELD, \
CFG_WEBCOMMENT_RESTRICTION_DATAFIELD, \
CFG_WEBCOMMENT_MAX_COMMENT_THREAD_DEPTH
from invenio.utils.mail import \
email_quote_txt, \
email_quoted_txt2html
from invenio.utils.html import tidy_html
from invenio.legacy.webuser import get_user_info, get_email, collect_user_info
from invenio.utils.date import convert_datetext_to_dategui, \
datetext_default, \
convert_datestruct_to_datetext
from invenio.ext.email import send_email
from invenio.ext.logging import register_exception
from invenio.base.i18n import wash_language, gettext_set_language
from invenio.utils.url import wash_url_argument
from .config import CFG_WEBCOMMENT_ACTION_CODE, \
InvenioWebCommentError, \
InvenioWebCommentWarning
from invenio.modules.access.engine import acc_authorize_action
from invenio.legacy.search_engine import \
guess_primary_collection_of_a_record, \
check_user_can_view_record, \
get_collection_reclist, \
get_colID
from invenio.legacy.bibrecord import get_fieldvalues
from invenio.utils.htmlwasher import EmailWasher
try:
import invenio.legacy.template
webcomment_templates = invenio.legacy.template.load('webcomment')
except:
pass
def perform_request_display_comments_or_remarks(req, recID, display_order='od', display_since='all', nb_per_page=100, page=1, ln=CFG_SITE_LANG, voted=-1, reported=-1, subscribed=0, reviews=0, uid=-1, can_send_comments=False, can_attach_files=False, user_is_subscribed_to_discussion=False, user_can_unsubscribe_from_discussion=False, display_comment_rounds=None):
"""
Returns all the comments (reviews) of a specific internal record or external basket record.
@param recID: record id where (internal record IDs > 0) or (external basket record IDs < -100)
@param display_order: hh = highest helpful score, review only
lh = lowest helpful score, review only
hs = highest star score, review only
ls = lowest star score, review only
od = oldest date
nd = newest date
@param display_since: all= no filtering by date
nd = n days ago
nw = n weeks ago
nm = n months ago
ny = n years ago
where n is a single digit integer between 0 and 9
@param nb_per_page: number of results per page
@param page: results page
@param voted: boolean, active if user voted for a review, see perform_request_vote function
@param reported: boolean, active if user reported a certain comment/review, perform_request_report function
@param subscribed: int, 1 if user just subscribed to discussion, -1 if unsubscribed
@param reviews: boolean, enabled if reviews, disabled for comments
@param uid: the id of the user who is reading comments
@param can_send_comments: if user can send comment or not
@param can_attach_files: if user can attach file to comment or not
@param user_is_subscribed_to_discussion: True if user already receives new comments by email
@param user_can_unsubscribe_from_discussion: True is user is allowed to unsubscribe from discussion
@return html body.
"""
_ = gettext_set_language(ln)
warnings = []
nb_reviews = 0
nb_comments = 0
# wash arguments
recID = wash_url_argument(recID, 'int')
ln = wash_language(ln)
display_order = wash_url_argument(display_order, 'str')
display_since = wash_url_argument(display_since, 'str')
nb_per_page = wash_url_argument(nb_per_page, 'int')
page = wash_url_argument(page, 'int')
voted = wash_url_argument(voted, 'int')
reported = wash_url_argument(reported, 'int')
reviews = wash_url_argument(reviews, 'int')
# vital argument check
(valid, error_body) = check_recID_is_in_range(recID, warnings, ln)
if not(valid):
return error_body
# CERN hack begins: filter out ATLAS comments
from invenio.config import CFG_CERN_SITE
if CFG_CERN_SITE:
restricted_comments_p = False
for report_number in get_fieldvalues(recID, '088__a'):
if report_number.startswith("ATL-"):
restricted_comments_p = True
break
if restricted_comments_p:
err_code, err_msg = acc_authorize_action(uid, 'viewrestrcoll',
collection='ATLAS Communications')
if err_code:
return err_msg
# CERN hack ends
# Query the database and filter results
user_info = collect_user_info(uid)
res = query_retrieve_comments_or_remarks(recID, display_order, display_since, reviews, user_info=user_info)
# res2 = query_retrieve_comments_or_remarks(recID, display_order, display_since, not reviews, user_info=user_info)
nb_res = len(res)
from invenio.legacy.webcomment.adminlib import get_nb_reviews, get_nb_comments
nb_reviews = get_nb_reviews(recID, count_deleted=False)
nb_comments = get_nb_comments(recID, count_deleted=False)
# checking non vital arguemnts - will be set to default if wrong
#if page <= 0 or page.lower() != 'all':
if page < 0:
page = 1
try:
raise InvenioWebCommentWarning(_('Bad page number --> showing first page.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_INVALID_PAGE_NB',))
if nb_per_page < 0:
nb_per_page = 100
try:
raise InvenioWebCommentWarning(_('Bad number of results per page --> showing 10 results per page.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_INVALID_NB_RESULTS_PER_PAGE',))
if CFG_WEBCOMMENT_ALLOW_REVIEWS and reviews:
if display_order not in ['od', 'nd', 'hh', 'lh', 'hs', 'ls']:
display_order = 'hh'
try:
raise InvenioWebCommentWarning(_('Bad display order --> showing most helpful first.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_INVALID_REVIEW_DISPLAY_ORDER',))
else:
if display_order not in ['od', 'nd']:
display_order = 'od'
try:
raise InvenioWebCommentWarning(_('Bad display order --> showing oldest first.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_INVALID_DISPLAY_ORDER',))
if not display_comment_rounds:
display_comment_rounds = []
# filter results according to page and number of reults per page
if nb_per_page > 0:
if nb_res > 0:
last_page = int(math.ceil(nb_res / float(nb_per_page)))
else:
last_page = 1
if page > last_page:
page = 1
try:
raise InvenioWebCommentWarning(_('Bad page number --> showing first page.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(("WRN_WEBCOMMENT_INVALID_PAGE_NB",))
if nb_res > nb_per_page: # if more than one page of results
if page < last_page:
res = res[(page-1)*(nb_per_page) : (page*nb_per_page)]
else:
res = res[(page-1)*(nb_per_page) : ]
else: # one page of results
pass
else:
last_page = 1
# Add information regarding visibility of comment for user
user_collapsed_comments = get_user_collapsed_comments_for_record(uid, recID)
if reviews:
res = [row[:] + (row[10] in user_collapsed_comments,) for row in res]
else:
res = [row[:] + (row[6] in user_collapsed_comments,) for row in res]
# Send to template
avg_score = 0.0
if not CFG_WEBCOMMENT_ALLOW_COMMENTS and not CFG_WEBCOMMENT_ALLOW_REVIEWS: # comments not allowed by admin
try:
raise InvenioWebCommentError(_('Comments on records have been disallowed by the administrator.'))
except InvenioWebCommentError as exc:
register_exception(req=req)
body = webcomment_templates.tmpl_error(exc.message, ln)
return body
# errors.append(('ERR_WEBCOMMENT_COMMENTS_NOT_ALLOWED',))
if reported > 0:
try:
raise InvenioWebCommentWarning(_('Your feedback has been recorded, many thanks.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, 'green'))
#warnings.append(('WRN_WEBCOMMENT_FEEDBACK_RECORDED',))
elif reported == 0:
try:
raise InvenioWebCommentWarning(_('You have already reported an abuse for this comment.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_ALREADY_REPORTED',))
elif reported == -2:
try:
raise InvenioWebCommentWarning(_('The comment you have reported no longer exists.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_INVALID_REPORT',))
if CFG_WEBCOMMENT_ALLOW_REVIEWS and reviews:
avg_score = calculate_avg_score(res)
if voted > 0:
try:
raise InvenioWebCommentWarning(_('Your feedback has been recorded, many thanks.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, 'green'))
#warnings.append(('WRN_WEBCOMMENT_FEEDBACK_RECORDED',))
elif voted == 0:
try:
raise InvenioWebCommentWarning(_('Sorry, you have already voted. This vote has not been recorded.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_ALREADY_VOTED',))
if subscribed == 1:
try:
raise InvenioWebCommentWarning(_('You have been subscribed to this discussion. From now on, you will receive an email whenever a new comment is posted.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, 'green'))
#warnings.append(('WRN_WEBCOMMENT_SUBSCRIBED',))
elif subscribed == -1:
try:
raise InvenioWebCommentWarning(_('You have been unsubscribed from this discussion.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, 'green'))
#warnings.append(('WRN_WEBCOMMENT_UNSUBSCRIBED',))
grouped_comments = group_comments_by_round(res, reviews)
# Clean list of comments round names
if not display_comment_rounds:
display_comment_rounds = []
elif 'all' in display_comment_rounds:
display_comment_rounds = [cmtgrp[0] for cmtgrp in grouped_comments]
elif 'latest' in display_comment_rounds:
if grouped_comments:
display_comment_rounds.append(grouped_comments[-1][0])
display_comment_rounds.remove('latest')
body = webcomment_templates.tmpl_get_comments(req,
recID,
ln,
nb_per_page, page, last_page,
display_order, display_since,
CFG_WEBCOMMENT_ALLOW_REVIEWS,
grouped_comments, nb_comments, avg_score,
warnings,
border=0,
reviews=reviews,
total_nb_reviews=nb_reviews,
uid=uid,
can_send_comments=can_send_comments,
can_attach_files=can_attach_files,
user_is_subscribed_to_discussion=\
user_is_subscribed_to_discussion,
user_can_unsubscribe_from_discussion=\
user_can_unsubscribe_from_discussion,
display_comment_rounds=display_comment_rounds)
return body
def perform_request_vote(cmt_id, client_ip_address, value, uid=-1):
"""
Vote positively or negatively for a comment/review
@param cmt_id: review id
@param value: +1 for voting positively
-1 for voting negatively
@return: integer 1 if successful, integer 0 if not
"""
cmt_id = wash_url_argument(cmt_id, 'int')
client_ip_address = wash_url_argument(client_ip_address, 'str')
value = wash_url_argument(value, 'int')
uid = wash_url_argument(uid, 'int')
if cmt_id > 0 and value in [-1, 1] and check_user_can_vote(cmt_id, client_ip_address, uid):
action_date = convert_datestruct_to_datetext(time.localtime())
action_code = CFG_WEBCOMMENT_ACTION_CODE['VOTE']
query = """INSERT INTO cmtACTIONHISTORY (id_cmtRECORDCOMMENT,
id_bibrec, id_user, client_host, action_time,
action_code)
VALUES (%s, NULL ,%s, inet_aton(%s), %s, %s)"""
params = (cmt_id, uid, client_ip_address, action_date, action_code)
run_sql(query, params)
return query_record_useful_review(cmt_id, value)
else:
return 0
def check_user_can_comment(recID, client_ip_address, uid=-1):
""" Check if a user hasn't already commented within the last seconds
time limit: CFG_WEBCOMMENT_TIMELIMIT_PROCESSING_COMMENTS_IN_SECONDS
@param recID: record id
@param client_ip_address: IP => use: str(req.remote_ip)
@param uid: user id, as given by invenio.legacy.webuser.getUid(req)
"""
recID = wash_url_argument(recID, 'int')
client_ip_address = wash_url_argument(client_ip_address, 'str')
uid = wash_url_argument(uid, 'int')
max_action_time = time.time() - CFG_WEBCOMMENT_TIMELIMIT_PROCESSING_COMMENTS_IN_SECONDS
max_action_time = convert_datestruct_to_datetext(time.localtime(max_action_time))
action_code = CFG_WEBCOMMENT_ACTION_CODE['ADD_COMMENT']
query = """SELECT id_bibrec
FROM cmtACTIONHISTORY
WHERE id_bibrec=%s AND
action_code=%s AND
action_time>%s
"""
params = (recID, action_code, max_action_time)
if uid < 0:
query += " AND client_host=inet_aton(%s)"
params += (client_ip_address,)
else:
query += " AND id_user=%s"
params += (uid,)
res = run_sql(query, params)
return len(res) == 0
def check_user_can_review(recID, client_ip_address, uid=-1):
""" Check if a user hasn't already reviewed within the last seconds
time limit: CFG_WEBCOMMENT_TIMELIMIT_PROCESSING_REVIEWS_IN_SECONDS
@param recID: record ID
@param client_ip_address: IP => use: str(req.remote_ip)
@param uid: user id, as given by invenio.legacy.webuser.getUid(req)
"""
action_code = CFG_WEBCOMMENT_ACTION_CODE['ADD_REVIEW']
query = """SELECT id_bibrec
FROM cmtACTIONHISTORY
WHERE id_bibrec=%s AND
action_code=%s
"""
params = (recID, action_code)
if uid < 0:
query += " AND client_host=inet_aton(%s)"
params += (client_ip_address,)
else:
query += " AND id_user=%s"
params += (uid,)
res = run_sql(query, params)
return len(res) == 0
def check_user_can_vote(cmt_id, client_ip_address, uid=-1):
""" Checks if a user hasn't already voted
@param cmt_id: comment id
@param client_ip_address: IP => use: str(req.remote_ip)
@param uid: user id, as given by invenio.legacy.webuser.getUid(req)
"""
cmt_id = wash_url_argument(cmt_id, 'int')
client_ip_address = wash_url_argument(client_ip_address, 'str')
uid = wash_url_argument(uid, 'int')
query = """SELECT id_cmtRECORDCOMMENT
FROM cmtACTIONHISTORY
WHERE id_cmtRECORDCOMMENT=%s"""
params = (cmt_id,)
if uid < 0:
query += " AND client_host=inet_aton(%s)"
params += (client_ip_address,)
else:
query += " AND id_user=%s"
params += (uid, )
res = run_sql(query, params)
return (len(res) == 0)
def get_comment_collection(cmt_id):
"""
Extract the collection where the comment is written
"""
query = "SELECT id_bibrec FROM cmtRECORDCOMMENT WHERE id=%s"
recid = run_sql(query, (cmt_id,))
record_primary_collection = guess_primary_collection_of_a_record(recid[0][0])
return record_primary_collection
def get_collection_moderators(collection):
"""
Return the list of comment moderators for the given collection.
"""
from invenio.modules.access.engine import acc_get_authorized_emails
res = list(acc_get_authorized_emails('moderatecomments', collection=collection))
if not res:
return [CFG_WEBCOMMENT_DEFAULT_MODERATOR,]
return res
def perform_request_report(cmt_id, client_ip_address, uid=-1):
"""
Report a comment/review for inappropriate content.
Will send an email to the administrator if number of reports is a multiple of CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN
@param cmt_id: comment id
@return: integer 1 if successful, integer 0 if not. -2 if comment does not exist
"""
cmt_id = wash_url_argument(cmt_id, 'int')
if cmt_id <= 0:
return 0
(query_res, nb_abuse_reports) = query_record_report_this(cmt_id)
if query_res == 0:
return 0
elif query_res == -2:
return -2
if not(check_user_can_report(cmt_id, client_ip_address, uid)):
return 0
action_date = convert_datestruct_to_datetext(time.localtime())
action_code = CFG_WEBCOMMENT_ACTION_CODE['REPORT_ABUSE']
query = """INSERT INTO cmtACTIONHISTORY (id_cmtRECORDCOMMENT, id_bibrec,
id_user, client_host, action_time, action_code)
VALUES (%s, NULL, %s, inet_aton(%s), %s, %s)"""
params = (cmt_id, uid, client_ip_address, action_date, action_code)
run_sql(query, params)
if nb_abuse_reports % CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN == 0:
(cmt_id2,
id_bibrec,
id_user,
cmt_body,
cmt_date,
cmt_star,
cmt_vote, cmt_nb_votes_total,
cmt_title,
cmt_reported,
round_name,
restriction) = query_get_comment(cmt_id)
(user_nb_abuse_reports,
user_votes,
user_nb_votes_total) = query_get_user_reports_and_votes(int(id_user))
(nickname, user_email, last_login) = query_get_user_contact_info(id_user)
from_addr = '%s Alert Engine <%s>' % (CFG_SITE_NAME, CFG_WEBALERT_ALERT_ENGINE_EMAIL)
comment_collection = get_comment_collection(cmt_id)
to_addrs = get_collection_moderators(comment_collection)
subject = "A comment has been reported as inappropriate by a user"
body = '''
The following comment has been reported a total of %(cmt_reported)s times.
Author: nickname = %(nickname)s
email = %(user_email)s
user_id = %(uid)s
This user has:
total number of reports = %(user_nb_abuse_reports)s
%(votes)s
Comment: comment_id = %(cmt_id)s
record_id = %(id_bibrec)s
date written = %(cmt_date)s
nb reports = %(cmt_reported)s
%(review_stuff)s
body =
---start body---
%(cmt_body)s
---end body---
Please go to the record page %(comment_admin_link)s to delete this message if necessary. A warning will be sent to the user in question.''' % \
{ 'cfg-report_max' : CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN,
'nickname' : nickname,
'user_email' : user_email,
'uid' : id_user,
'user_nb_abuse_reports' : user_nb_abuse_reports,
'user_votes' : user_votes,
'votes' : CFG_WEBCOMMENT_ALLOW_REVIEWS and \
"total number of positive votes\t= %s\n\t\ttotal number of negative votes\t= %s" % \
(user_votes, (user_nb_votes_total - user_votes)) or "\n",
'cmt_id' : cmt_id,
'id_bibrec' : id_bibrec,
'cmt_date' : cmt_date,
'cmt_reported' : cmt_reported,
'review_stuff' : CFG_WEBCOMMENT_ALLOW_REVIEWS and \
"star score\t= %s\n\treview title\t= %s" % (cmt_star, cmt_title) or "",
'cmt_body' : cmt_body,
'comment_admin_link' : CFG_SITE_URL + "/"+ CFG_SITE_RECORD +"/" + str(id_bibrec) + '/comments#' + str(cmt_id),
'user_admin_link' : "user_admin_link" #! FIXME
}
#FIXME to be added to email when websession module is over:
#If you wish to ban the user, you can do so via the User Admin Panel %(user_admin_link)s.
send_email(from_addr, to_addrs, subject, body)
return 1
def check_user_can_report(cmt_id, client_ip_address, uid=-1):
""" Checks if a user hasn't already reported a comment
@param cmt_id: comment id
@param client_ip_address: IP => use: str(req.remote_ip)
@param uid: user id, as given by invenio.legacy.webuser.getUid(req)
"""
cmt_id = wash_url_argument(cmt_id, 'int')
client_ip_address = wash_url_argument(client_ip_address, 'str')
uid = wash_url_argument(uid, 'int')
query = """SELECT id_cmtRECORDCOMMENT
FROM cmtACTIONHISTORY
WHERE id_cmtRECORDCOMMENT=%s"""
params = (uid,)
if uid < 0:
query += " AND client_host=inet_aton(%s)"
params += (client_ip_address,)
else:
query += " AND id_user=%s"
params += (uid,)
res = run_sql(query, params)
return (len(res) == 0)
def query_get_user_contact_info(uid):
"""
Get the user contact information
@return: tuple (nickname, email, last_login), if none found return ()
Note: for the moment, if no nickname, will return email address up to the '@'
"""
query1 = """SELECT nickname, email,
DATE_FORMAT(last_login, '%%Y-%%m-%%d %%H:%%i:%%s')
FROM user WHERE id=%s"""
params1 = (uid,)
res1 = run_sql(query1, params1)
if res1:
return res1[0]
else:
return ()
def query_get_user_reports_and_votes(uid):
"""
Retrieve total number of reports and votes of a particular user
@param uid: user id
@return: tuple (total_nb_reports, total_nb_votes_yes, total_nb_votes_total)
if none found return ()
"""
query1 = """SELECT nb_votes_yes,
nb_votes_total,
nb_abuse_reports
FROM cmtRECORDCOMMENT
WHERE id_user=%s"""
params1 = (uid,)
res1 = run_sql(query1, params1)
if len(res1) == 0:
return ()
nb_votes_yes = nb_votes_total = nb_abuse_reports = 0
for cmt_tuple in res1:
nb_votes_yes += int(cmt_tuple[0])
nb_votes_total += int(cmt_tuple[1])
nb_abuse_reports += int(cmt_tuple[2])
return (nb_abuse_reports, nb_votes_yes, nb_votes_total)
def query_get_comment(comID):
"""
Get all fields of a comment
@param comID: comment id
@return: tuple (comID, id_bibrec, id_user, body, date_creation, star_score, nb_votes_yes, nb_votes_total, title, nb_abuse_reports, round_name, restriction)
if none found return ()
"""
query1 = """SELECT id,
id_bibrec,
id_user,
body,
DATE_FORMAT(date_creation, '%%Y-%%m-%%d %%H:%%i:%%s'),
star_score,
nb_votes_yes,
nb_votes_total,
title,
nb_abuse_reports,
round_name,
restriction
FROM cmtRECORDCOMMENT
WHERE id=%s"""
params1 = (comID,)
res1 = run_sql(query1, params1)
if len(res1)>0:
return res1[0]
else:
return ()
def query_record_report_this(comID):
"""
Increment the number of reports for a comment
@param comID: comment id
@return: tuple (success, new_total_nb_reports_for_this_comment) where
success is integer 1 if success, integer 0 if not, -2 if comment does not exist
"""
#retrieve nb_abuse_reports
query1 = "SELECT nb_abuse_reports FROM cmtRECORDCOMMENT WHERE id=%s"
params1 = (comID,)
res1 = run_sql(query1, params1)
if len(res1) == 0:
return (-2, 0)
#increment and update
nb_abuse_reports = int(res1[0][0]) + 1
query2 = "UPDATE cmtRECORDCOMMENT SET nb_abuse_reports=%s WHERE id=%s"
params2 = (nb_abuse_reports, comID)
res2 = run_sql(query2, params2)
return (int(res2), nb_abuse_reports)
def query_record_useful_review(comID, value):
"""
private funciton
Adjust the number of useful votes and number of total votes for a comment.
@param comID: comment id
@param value: +1 or -1
@return: integer 1 if successful, integer 0 if not
"""
# retrieve nb_useful votes
query1 = "SELECT nb_votes_total, nb_votes_yes FROM cmtRECORDCOMMENT WHERE id=%s"
params1 = (comID,)
res1 = run_sql(query1, params1)
if len(res1)==0:
return 0
# modify and insert new nb_useful votes
nb_votes_yes = int(res1[0][1])
if value >= 1:
nb_votes_yes = int(res1[0][1]) + 1
nb_votes_total = int(res1[0][0]) + 1
query2 = "UPDATE cmtRECORDCOMMENT SET nb_votes_total=%s, nb_votes_yes=%s WHERE id=%s"
params2 = (nb_votes_total, nb_votes_yes, comID)
res2 = run_sql(query2, params2)
return int(res2)
def query_retrieve_comments_or_remarks(recID, display_order='od', display_since='0000-00-00 00:00:00',
ranking=0, limit='all', user_info=None):
"""
Private function
Retrieve tuple of comments or remarks from the database
@param recID: record id
@param display_order: hh = highest helpful score
lh = lowest helpful score
hs = highest star score
ls = lowest star score
od = oldest date
nd = newest date
@param display_since: datetime, e.g. 0000-00-00 00:00:00
@param ranking: boolean, enabled if reviews, disabled for comments
@param limit: number of comments/review to return
@return: tuple of comment where comment is
tuple (nickname, uid, date_creation, body, status, id) if ranking disabled or
tuple (nickname, uid, date_creation, body, status, nb_votes_yes, nb_votes_total, star_score, title, id)
Note: for the moment, if no nickname, will return email address up to '@'
"""
display_since = calculate_start_date(display_since)
order_dict = { 'hh' : "cmt.nb_votes_yes/(cmt.nb_votes_total+1) DESC, cmt.date_creation DESC ",
'lh' : "cmt.nb_votes_yes/(cmt.nb_votes_total+1) ASC, cmt.date_creation ASC ",
'ls' : "cmt.star_score ASC, cmt.date_creation DESC ",
'hs' : "cmt.star_score DESC, cmt.date_creation DESC ",
'nd' : "cmt.reply_order_cached_data DESC ",
'od' : "cmt.reply_order_cached_data ASC "
}
# Ranking only done for comments and when allowed
if ranking and recID > 0:
try:
display_order = order_dict[display_order]
except:
display_order = order_dict['od']
else:
# in case of recID > 0 => external record => no ranking!
ranking = 0
try:
if display_order[-1] == 'd':
display_order = order_dict[display_order]
else:
display_order = order_dict['od']
except:
display_order = order_dict['od']
#display_order = order_dict['nd']
query = """SELECT user.nickname,
cmt.id_user,
DATE_FORMAT(cmt.date_creation, '%%%%Y-%%%%m-%%%%d %%%%H:%%%%i:%%%%s'),
cmt.body,
cmt.status,
cmt.nb_abuse_reports,
%(ranking)s cmt.id,
cmt.round_name,
cmt.restriction,
%(reply_to_column)s
FROM cmtRECORDCOMMENT cmt LEFT JOIN user ON
user.id=cmt.id_user
WHERE cmt.id_bibrec=%%s
%(ranking_only)s
%(display_since)s
ORDER BY %(display_order)s
""" % {'ranking' : ranking and ' cmt.nb_votes_yes, cmt.nb_votes_total, cmt.star_score, cmt.title, ' or '',
'ranking_only' : ranking and ' AND cmt.star_score>0 ' or ' AND cmt.star_score=0 ',
# 'id_bibrec' : recID > 0 and 'cmt.id_bibrec' or 'cmt.id_bibrec_or_bskEXTREC',
# 'table' : recID > 0 and 'cmtRECORDCOMMENT' or 'bskRECORDCOMMENT',
'display_since' : display_since == '0000-00-00 00:00:00' and ' ' or 'AND cmt.date_creation>=\'%s\' ' % display_since,
'display_order': display_order,
'reply_to_column': recID > 0 and 'cmt.in_reply_to_id_cmtRECORDCOMMENT' or 'cmt.in_reply_to_id_bskRECORDCOMMENT'}
params = (recID,)
res = run_sql(query, params)
# return res
new_limit = limit
comments_list = []
for row in res:
if ranking:
# when dealing with reviews, row[12] holds restriction info:
restriction = row[12]
else:
# when dealing with comments, row[8] holds restriction info:
restriction = row[8]
if user_info and check_user_can_view_comment(user_info, None, restriction)[0] != 0:
# User cannot view comment. Look further
continue
comments_list.append(row)
if limit.isdigit():
new_limit -= 1
if limit < 1:
break
if comments_list:
if limit.isdigit():
return comments_list[:limit]
else:
return comments_list
return ()
## def get_comment_children(comID):
## """
## Returns the list of children (i.e. direct descendants) ordered by time of addition.
## @param comID: the ID of the comment for which we want to retrieve children
## @type comID: int
## @return the list of children
## @rtype: list
## """
## res = run_sql("SELECT id FROM cmtRECORDCOMMENT WHERE in_reply_to_id_cmtRECORDCOMMENT=%s", (comID,))
## return [row[0] for row in res]
## def get_comment_descendants(comID, depth=None):
## """
## Returns the list of descendants of the given comment, orderd from
## oldest to newest ("top-down"), down to depth specified as parameter.
## @param comID: the ID of the comment for which we want to retrieve descendant
## @type comID: int
## @param depth: the max depth down to which we want to retrieve
## descendants. Specify None for no limit, 1 for direct
## children only, etc.
## @return the list of ancestors
## @rtype: list(tuple(comment ID, descendants comments IDs))
## """
## if depth == 0:
## return (comID, [])
## res = run_sql("SELECT id FROM cmtRECORDCOMMENT WHERE in_reply_to_id_cmtRECORDCOMMENT=%s", (comID,))
## if res:
## children_comID = [row[0] for row in res]
## children_descendants = []
## if depth:
## depth -= 1
## children_descendants = [get_comment_descendants(child_comID, depth) for child_comID in children_comID]
## return (comID, children_descendants)
## else:
## return (comID, [])
def get_comment_ancestors(comID, depth=None):
"""
Returns the list of ancestors of the given comment, ordered from
oldest to newest ("top-down": direct parent of comID is at last position),
up to given depth
@param comID: the ID of the comment for which we want to retrieve ancestors
@type comID: int
@param depth: the maximum of levels up from the given comment we
want to retrieve ancestors. None for no limit, 1 for
direct parent only, etc.
@type depth: int
@return the list of ancestors
@rtype: list
"""
if depth == 0:
return []
res = run_sql("SELECT in_reply_to_id_cmtRECORDCOMMENT FROM cmtRECORDCOMMENT WHERE id=%s", (comID,))
if res:
parent_comID = res[0][0]
if parent_comID == 0:
return []
parent_ancestors = []
if depth:
depth -= 1
parent_ancestors = get_comment_ancestors(parent_comID, depth)
parent_ancestors.append(parent_comID)
return parent_ancestors
else:
return []
def get_reply_order_cache_data(comid):
"""
Prepare a representation of the comment ID given as parameter so
that it is suitable for byte ordering in MySQL.
"""
return "%s%s%s%s" % (chr((comid >> 24) % 256), chr((comid >> 16) % 256),
chr((comid >> 8) % 256), chr(comid % 256))
def query_add_comment_or_remark(reviews=0, recID=0, uid=-1, msg="",
note="", score=0, priority=0,
client_ip_address='', editor_type='textarea',
req=None, reply_to=None, attached_files=None):
"""
Private function
Insert a comment/review or remarkinto the database
@param recID: record id
@param uid: user id
@param msg: comment body
@param note: comment title
@param score: review star score
@param priority: remark priority #!FIXME
@param editor_type: the kind of editor used to submit the comment: 'textarea', 'ckeditor'
@param req: request object. If provided, email notification are sent after we reply to user request.
@param reply_to: the id of the comment we are replying to with this inserted comment.
@return: integer >0 representing id if successful, integer 0 if not
"""
current_date = calculate_start_date('0d')
#change utf-8 message into general unicode
msg = msg.decode('utf-8')
note = note.decode('utf-8')
#change general unicode back to utf-8
msg = msg.encode('utf-8')
note = note.encode('utf-8')
msg_original = msg
(restriction, round_name) = get_record_status(recID)
if attached_files is None:
attached_files = {}
if reply_to and CFG_WEBCOMMENT_MAX_COMMENT_THREAD_DEPTH >= 0:
# Check that we have not reached max depth
comment_ancestors = get_comment_ancestors(reply_to)
if len(comment_ancestors) >= CFG_WEBCOMMENT_MAX_COMMENT_THREAD_DEPTH:
if CFG_WEBCOMMENT_MAX_COMMENT_THREAD_DEPTH == 0:
reply_to = None
else:
reply_to = comment_ancestors[CFG_WEBCOMMENT_MAX_COMMENT_THREAD_DEPTH - 1]
# Inherit restriction and group/round of 'parent'
comment = query_get_comment(reply_to)
if comment:
(round_name, restriction) = comment[10:12]
if editor_type == 'ckeditor':
# Here we remove the line feeds introduced by CKEditor (they
# have no meaning for the user) and replace the HTML line
# breaks by linefeeds, so that we are close to an input that
# would be done without the CKEditor. That's much better if a
# reply to a comment is made with a browser that does not
# support CKEditor.
msg = msg.replace('\n', '').replace('\r', '')
# We clean the quotes that could have been introduced by
# CKEditor when clicking the 'quote' button, as well as those
# that we have introduced when quoting the original message.
# We can however not use directly '>>' chars to quote, as it
# will be washed/fixed when calling tidy_html(): double-escape
# all &gt; first, and use &gt;&gt;
msg = msg.replace('&gt;', '&amp;gt;')
msg = re.sub('^\s*<blockquote', '<br/> <blockquote', msg)
msg = re.sub('<blockquote.*?>\s*<(p|div).*?>', '&gt;&gt;', msg)
msg = re.sub('</(p|div)>\s*</blockquote>', '', msg)
# Then definitely remove any blockquote, whatever it is
msg = re.sub('<blockquote.*?>', '<div>', msg)
msg = re.sub('</blockquote>', '</div>', msg)
# Tidy up the HTML
msg = tidy_html(msg)
# We remove EOL that might have been introduced when tidying
msg = msg.replace('\n', '').replace('\r', '')
# Now that HTML has been cleaned, unescape &gt;
msg = msg.replace('&gt;', '>')
msg = msg.replace('&amp;gt;', '&gt;')
msg = re.sub('<br .*?(/>)', '\n', msg)
msg = msg.replace('&nbsp;', ' ')
# In case additional <p> or <div> got inserted, interpret
# these as new lines (with a sad trick to do it only once)
# (note that it has been deactivated, as it is messing up
# indentation with >>)
#msg = msg.replace('</div><', '</div>\n<')
#msg = msg.replace('</p><', '</p>\n<')
query = """INSERT INTO cmtRECORDCOMMENT (id_bibrec,
id_user,
body,
date_creation,
star_score,
nb_votes_total,
title,
round_name,
restriction,
in_reply_to_id_cmtRECORDCOMMENT)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"""
params = (recID, uid, msg, current_date, score, 0, note, round_name, restriction, reply_to or 0)
res = run_sql(query, params)
if res:
new_comid = int(res)
move_attached_files_to_storage(attached_files, recID, new_comid)
parent_reply_order = run_sql("""SELECT reply_order_cached_data from cmtRECORDCOMMENT where id=%s""", (reply_to,))
if not parent_reply_order or parent_reply_order[0][0] is None:
# This is not a reply, but a first 0-level comment
parent_reply_order = ''
else:
parent_reply_order = parent_reply_order[0][0]
run_sql("""UPDATE cmtRECORDCOMMENT SET reply_order_cached_data=%s WHERE id=%s""",
(parent_reply_order + get_reply_order_cache_data(new_comid), new_comid))
action_code = CFG_WEBCOMMENT_ACTION_CODE[reviews and 'ADD_REVIEW' or 'ADD_COMMENT']
action_time = convert_datestruct_to_datetext(time.localtime())
query2 = """INSERT INTO cmtACTIONHISTORY (id_cmtRECORDCOMMENT,
id_bibrec, id_user, client_host, action_time, action_code)
VALUES (%s, %s, %s, inet_aton(%s), %s, %s)"""
params2 = (res, recID, uid, client_ip_address, action_time, action_code)
run_sql(query2, params2)
def notify_subscribers_callback(data):
"""
Define a callback that retrieves subscribed users, and
notify them by email.
@param data: contains the necessary parameters in a tuple:
(recid, uid, comid, msg, note, score, editor_type, reviews)
"""
recid, uid, comid, msg, note, score, editor_type, reviews = data
# Email this comment to 'subscribers'
(subscribers_emails1, subscribers_emails2) = \
get_users_subscribed_to_discussion(recid)
email_subscribers_about_new_comment(recid, reviews=reviews,
emails1=subscribers_emails1,
emails2=subscribers_emails2,
comID=comid, msg=msg,
note=note, score=score,
editor_type=editor_type, uid=uid)
# Register our callback to notify subscribed people after
# having replied to our current user.
data = (recID, uid, res, msg, note, score, editor_type, reviews)
if req:
req.register_cleanup(notify_subscribers_callback, data)
else:
notify_subscribers_callback(data)
return int(res)
def move_attached_files_to_storage(attached_files, recID, comid):
"""
Move the files that were just attached to a new comment to their
final location.
@param attached_files: the mappings of desired filename to attach
and path where to find the original file
@type attached_files: dict {filename, filepath}
@param recID: the record ID to which we attach the files
@param comid: the comment ID to which we attach the files
"""
for filename, filepath in iteritems(attached_files):
- os.renames(filepath,
- os.path.join(CFG_PREFIX, 'var', 'data', 'comments',
- str(recID), str(comid), filename))
+ dest_dir = os.path.join(CFG_PREFIX, 'var', 'data', 'comments',
+ str(recID), str(comid))
+ try:
+ os.makedirs(dest_dir)
+ except:
+ # Dir most probably already existed
+ pass
+ shutil.move(filepath,
+ os.path.join(dest_dir, filename))
def get_attached_files(recid, comid):
"""
Returns a list with tuples (filename, filepath, fileurl)
@param recid: the recid to which the comment belong
@param comid: the commment id for which we want to retrieve files
"""
base_dir = os.path.join(CFG_PREFIX, 'var', 'data', 'comments',
str(recid), str(comid))
if os.path.isdir(base_dir):
filenames = os.listdir(base_dir)
return [(filename, os.path.join(CFG_PREFIX, 'var', 'data', 'comments',
str(recid), str(comid), filename),
CFG_SITE_URL + '/'+ CFG_SITE_RECORD +'/' + str(recid) + '/comments/attachments/get/' + str(comid) + '/' + filename) \
for filename in filenames]
else:
return []
def subscribe_user_to_discussion(recID, uid):
"""
Subscribe a user to a discussion, so the she receives by emails
all new new comments for this record.
@param recID: record ID corresponding to the discussion we want to
subscribe the user
@param uid: user id
"""
query = """INSERT INTO cmtSUBSCRIPTION (id_bibrec, id_user, creation_time)
VALUES (%s, %s, %s)"""
params = (recID, uid, convert_datestruct_to_datetext(time.localtime()))
try:
run_sql(query, params)
except:
return 0
return 1
def unsubscribe_user_from_discussion(recID, uid):
"""
Unsubscribe users from a discussion.
@param recID: record ID corresponding to the discussion we want to
unsubscribe the user
@param uid: user id
@return 1 if successful, 0 if not
"""
query = """DELETE FROM cmtSUBSCRIPTION
WHERE id_bibrec=%s AND id_user=%s"""
params = (recID, uid)
try:
res = run_sql(query, params)
except:
return 0
if res > 0:
return 1
return 0
def get_user_subscription_to_discussion(recID, uid):
"""
Returns the type of subscription for the given user to this
discussion. This does not check authorizations (for eg. if user
was subscribed, but is suddenly no longer authorized).
@param recID: record ID
@param uid: user id
@return:
- 0 if user is not subscribed to discussion
- 1 if user is subscribed, and is allowed to unsubscribe
- 2 if user is subscribed, but cannot unsubscribe
"""
user_email = get_email(uid)
(emails1, emails2) = get_users_subscribed_to_discussion(recID, check_authorizations=False)
if user_email in emails1:
return 1
elif user_email in emails2:
return 2
else:
return 0
def get_users_subscribed_to_discussion(recID, check_authorizations=True):
"""
Returns the lists of users subscribed to a given discussion.
Two lists are returned: the first one is the list of emails for
users who can unsubscribe from the discussion, the second list
contains the emails of users who cannot unsubscribe (for eg. author
of the document, etc).
Users appear in only one list. If a user has manually subscribed
to a discussion AND is an automatic recipients for updates, it
will only appear in the second list.
@param recID: record ID for which we want to retrieve subscribed users
@param check_authorizations: if True, check again if users are authorized to view comment
@return tuple (emails1, emails2)
"""
subscribers_emails = {}
# Get users that have subscribed to this discussion
query = """SELECT id_user FROM cmtSUBSCRIPTION WHERE id_bibrec=%s"""
params = (recID,)
res = run_sql(query, params)
for row in res:
uid = row[0]
if check_authorizations:
user_info = collect_user_info(uid)
(auth_code, auth_msg) = check_user_can_view_comments(user_info, recID)
else:
# Don't check and grant access
auth_code = False
if auth_code:
# User is no longer authorized to view comments.
# Delete subscription
unsubscribe_user_from_discussion(recID, uid)
else:
email = get_email(uid)
if '@' in email:
subscribers_emails[email] = True
# Get users automatically subscribed, based on the record metadata
collections_with_auto_replies = CFG_WEBCOMMENT_EMAIL_REPLIES_TO.keys()
for collection in collections_with_auto_replies:
if (get_colID(collection) is not None) and \
(recID in get_collection_reclist(collection)):
fields = CFG_WEBCOMMENT_EMAIL_REPLIES_TO[collection]
for field in fields:
emails = get_fieldvalues(recID, field)
for email in emails:
if not '@' in email:
# Is a group: add domain name
subscribers_emails[email + '@' + \
CFG_SITE_SUPPORT_EMAIL.split('@')[1]] = False
else:
subscribers_emails[email] = False
return ([email for email, can_unsubscribe_p \
in iteritems(subscribers_emails) if can_unsubscribe_p],
[email for email, can_unsubscribe_p \
in iteritems(subscribers_emails) if not can_unsubscribe_p] )
def email_subscribers_about_new_comment(recID, reviews, emails1,
emails2, comID, msg="",
note="", score=0,
editor_type='textarea',
ln=CFG_SITE_LANG, uid=-1):
"""
Notify subscribers that a new comment was posted.
FIXME: consider recipient preference to send email in correct language.
@param recID: record id
@param emails1: list of emails for users who can unsubscribe from discussion
@param emails2: list of emails for users who cannot unsubscribe from discussion
@param comID: the comment id
@param msg: comment body
@param note: comment title
@param score: review star score
@param editor_type: the kind of editor used to submit the comment: 'textarea', 'ckeditor'
@rtype: bool
@return: True if email was sent okay, False if it was not.
"""
_ = gettext_set_language(ln)
if not emails1 and not emails2:
return 0
# Get title
titles = get_fieldvalues(recID, "245__a")
if not titles:
# usual title not found, try conference title:
titles = get_fieldvalues(recID, "111__a")
title = ''
if titles:
title = titles[0]
else:
title = _("Record %(x_rec)i", x_rec=recID)
# Get report number
report_numbers = get_fieldvalues(recID, "037__a")
if not report_numbers:
report_numbers = get_fieldvalues(recID, "088__a")
if not report_numbers:
report_numbers = get_fieldvalues(recID, "021__a")
# Prepare email subject and body
if reviews:
email_subject = _('%(report_number)s"%(title)s" has been reviewed') % \
{'report_number': report_numbers and ('[' + report_numbers[0] + '] ') or '',
'title': title}
else:
email_subject = _('%(report_number)s"%(title)s" has been commented') % \
{'report_number': report_numbers and ('[' + report_numbers[0] + '] ') or '',
'title': title}
washer = EmailWasher()
msg = washer.wash(msg)
msg = msg.replace('&gt;&gt;', '>')
email_content = msg
if note:
email_content = note + email_content
# Send emails to people who can unsubscribe
email_header = webcomment_templates.tmpl_email_new_comment_header(recID,
title,
reviews,
comID,
report_numbers,
can_unsubscribe=True,
ln=ln,
uid=uid)
email_footer = webcomment_templates.tmpl_email_new_comment_footer(recID,
title,
reviews,
comID,
report_numbers,
can_unsubscribe=True,
ln=ln)
res1 = True
if emails1:
res1 = send_email(fromaddr=CFG_WEBCOMMENT_ALERT_ENGINE_EMAIL,
toaddr=emails1,
subject=email_subject,
content=email_content,
header=email_header,
footer=email_footer,
ln=ln)
# Then send email to people who have been automatically
# subscribed to the discussion (they cannot unsubscribe)
email_header = webcomment_templates.tmpl_email_new_comment_header(recID,
title,
reviews,
comID,
report_numbers,
can_unsubscribe=False,
ln=ln,
uid=uid)
email_footer = webcomment_templates.tmpl_email_new_comment_footer(recID,
title,
reviews,
comID,
report_numbers,
can_unsubscribe=False,
ln=ln)
res2 = True
if emails2:
res2 = send_email(fromaddr=CFG_WEBCOMMENT_ALERT_ENGINE_EMAIL,
toaddr=emails2,
subject=email_subject,
content=email_content,
header=email_header,
footer=email_footer,
ln=ln)
return res1 and res2
def get_record_status(recid):
"""
Returns the current status of the record, i.e. current restriction to apply for newly submitted
comments, and current commenting round.
The restriction to apply can be found in the record metadata, in
field(s) defined by config CFG_WEBCOMMENT_RESTRICTION_DATAFIELD. The restriction is empty string ""
in cases where the restriction has not explicitely been set, even
if the record itself is restricted.
@param recid: the record id
@type recid: int
@return tuple(restriction, round_name), where 'restriction' is empty string when no restriction applies
@rtype (string, int)
"""
collections_with_rounds = CFG_WEBCOMMENT_ROUND_DATAFIELD.keys()
commenting_round = ""
for collection in collections_with_rounds:
# Find the first collection defines rounds field for this
# record
if get_colID(collection) is not None and \
(recid in get_collection_reclist(collection)):
commenting_rounds = get_fieldvalues(recid, CFG_WEBCOMMENT_ROUND_DATAFIELD.get(collection, ""))
if commenting_rounds:
commenting_round = commenting_rounds[0]
break
collections_with_restrictions = CFG_WEBCOMMENT_RESTRICTION_DATAFIELD.keys()
restriction = ""
for collection in collections_with_restrictions:
# Find the first collection that defines restriction field for
# this record
if get_colID(collection) is not None and \
recid in get_collection_reclist(collection):
restrictions = get_fieldvalues(recid, CFG_WEBCOMMENT_RESTRICTION_DATAFIELD.get(collection, ""))
if restrictions:
restriction = restrictions[0]
break
return (restriction, commenting_round)
def calculate_start_date(display_since):
"""
Private function
Returns the datetime of display_since argument in MYSQL datetime format
calculated according to the local time.
@param display_since: = all= no filtering
nd = n days ago
nw = n weeks ago
nm = n months ago
ny = n years ago
where n is a single digit number
@return: string of wanted datetime.
If 'all' given as argument, will return datetext_default
datetext_default is defined in miscutils/lib/dateutils and
equals 0000-00-00 00:00:00 => MySQL format
If bad arguement given, will return datetext_default
If library 'dateutil' is not found return datetext_default
and register exception.
"""
time_types = {'d':0, 'w':0, 'm':0, 'y':0}
today = datetime.today()
try:
nb = int(display_since[:-1])
except:
return datetext_default
if display_since in [None, 'all']:
return datetext_default
if str(display_since[-1]) in time_types:
time_type = str(display_since[-1])
else:
return datetext_default
# year
if time_type == 'y':
if (int(display_since[:-1]) > today.year - 1) or (int(display_since[:-1]) < 1):
# 1 < nb years < 2008
return datetext_default
else:
final_nb_year = today.year - nb
yesterday = today.replace(year=final_nb_year)
# month
elif time_type == 'm':
try:
from dateutil.relativedelta import relativedelta
except ImportError:
# The dateutil library is only recommended: if not
# available, then send warning about this.
register_exception(alert_admin=True)
return datetext_default
# obtain only the date: yyyy-mm-dd
date_today = datetime.now().date()
final_date = date_today - relativedelta(months=nb)
yesterday = today.replace(year=final_date.year, month=final_date.month, day=final_date.day)
# week
elif time_type == 'w':
delta = timedelta(weeks=nb)
yesterday = today - delta
# day
elif time_type == 'd':
delta = timedelta(days=nb)
yesterday = today - delta
return yesterday.strftime("%Y-%m-%d %H:%M:%S")
def get_first_comments_or_remarks(recID=-1,
ln=CFG_SITE_LANG,
nb_comments='all',
nb_reviews='all',
voted=-1,
reported=-1,
user_info=None,
show_reviews=False):
"""
Gets nb number comments/reviews or remarks.
In the case of comments, will get both comments and reviews
Comments and remarks sorted by most recent date, reviews sorted by highest helpful score
@param recID: record id
@param ln: language
@param nb_comments: number of comment or remarks to get
@param nb_reviews: number of reviews or remarks to get
@param voted: 1 if user has voted for a remark
@param reported: 1 if user has reported a comment or review
@return: if comment, tuple (comments, reviews) both being html of first nb comments/reviews
if remark, tuple (remakrs, None)
"""
_ = gettext_set_language(ln)
warnings = []
voted = wash_url_argument(voted, 'int')
reported = wash_url_argument(reported, 'int')
## check recID argument
if type(recID) is not int:
return ()
if recID >= 1: #comment or review. NB: suppressed reference to basket (handled in webbasket)
if CFG_WEBCOMMENT_ALLOW_REVIEWS:
res_reviews = query_retrieve_comments_or_remarks(recID=recID, display_order="hh", ranking=1,
limit=nb_comments, user_info=user_info)
nb_res_reviews = len(res_reviews)
## check nb argument
if type(nb_reviews) is int and nb_reviews < len(res_reviews):
first_res_reviews = res_reviews[:nb_reviews]
else:
first_res_reviews = res_reviews
if CFG_WEBCOMMENT_ALLOW_COMMENTS:
res_comments = query_retrieve_comments_or_remarks(recID=recID, display_order="od", ranking=0,
limit=nb_reviews, user_info=user_info)
nb_res_comments = len(res_comments)
## check nb argument
if type(nb_comments) is int and nb_comments < len(res_comments):
first_res_comments = res_comments[:nb_comments]
else:
first_res_comments = res_comments
else: #error
try:
raise InvenioWebCommentError(_('%(recid)s is an invalid record ID', recid=recID))
except InvenioWebCommentError as exc:
register_exception()
body = webcomment_templates.tmpl_error(exc.message, ln)
return body
#errors.append(('ERR_WEBCOMMENT_RECID_INVALID', recID)) #!FIXME dont return error anywhere since search page
# comment
if recID >= 1:
comments = reviews = ""
if reported > 0:
try:
raise InvenioWebCommentWarning(_('Your feedback has been recorded, many thanks.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning')
warnings.append((exc.message, 'green'))
#warnings.append(('WRN_WEBCOMMENT_FEEDBACK_RECORDED_GREEN_TEXT',))
elif reported == 0:
try:
raise InvenioWebCommentWarning(_('Your feedback could not be recorded, please try again.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning')
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_FEEDBACK_NOT_RECORDED_RED_TEXT',))
if CFG_WEBCOMMENT_ALLOW_COMMENTS: # normal comments
grouped_comments = group_comments_by_round(first_res_comments, ranking=0)
comments = webcomment_templates.tmpl_get_first_comments_without_ranking(recID, ln, grouped_comments, nb_res_comments, warnings)
if show_reviews:
if CFG_WEBCOMMENT_ALLOW_REVIEWS: # ranked comments
#calculate average score
avg_score = calculate_avg_score(res_reviews)
if voted > 0:
try:
raise InvenioWebCommentWarning(_('Your feedback has been recorded, many thanks.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning')
warnings.append((exc.message, 'green'))
#warnings.append(('WRN_WEBCOMMENT_FEEDBACK_RECORDED_GREEN_TEXT',))
elif voted == 0:
try:
raise InvenioWebCommentWarning(_('Your feedback could not be recorded, please try again.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning')
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_FEEDBACK_NOT_RECORDED_RED_TEXT',))
grouped_reviews = group_comments_by_round(first_res_reviews, ranking=0)
reviews = webcomment_templates.tmpl_get_first_comments_with_ranking(recID, ln, grouped_reviews, nb_res_reviews, avg_score, warnings)
return (comments, reviews)
# remark
else:
return(webcomment_templates.tmpl_get_first_remarks(first_res_comments, ln, nb_res_comments), None)
def group_comments_by_round(comments, ranking=0):
"""
Group comments by the round to which they belong
"""
comment_rounds = {}
ordered_comment_round_names = []
for comment in comments:
comment_round_name = ranking and comment[11] or comment[7]
if comment_round_name not in comment_rounds:
comment_rounds[comment_round_name] = []
ordered_comment_round_names.append(comment_round_name)
comment_rounds[comment_round_name].append(comment)
return [(comment_round_name, comment_rounds[comment_round_name]) \
for comment_round_name in ordered_comment_round_names]
def calculate_avg_score(res):
"""
private function
Calculate the avg score of reviews present in res
@param res: tuple of tuple returned from query_retrieve_comments_or_remarks
@return: a float of the average score rounded to the closest 0.5
"""
c_star_score = 6
avg_score = 0.0
nb_reviews = 0
for comment in res:
if comment[c_star_score] > 0:
avg_score += comment[c_star_score]
nb_reviews += 1
if nb_reviews == 0:
return 0.0
avg_score = avg_score / nb_reviews
avg_score_unit = avg_score - math.floor(avg_score)
if avg_score_unit < 0.25:
avg_score = math.floor(avg_score)
elif avg_score_unit > 0.75:
avg_score = math.floor(avg_score) + 1
else:
avg_score = math.floor(avg_score) + 0.5
if avg_score > 5:
avg_score = 5.0
return avg_score
def perform_request_add_comment_or_remark(recID=0,
uid=-1,
action='DISPLAY',
ln=CFG_SITE_LANG,
msg=None,
score=None,
note=None,
priority=None,
reviews=0,
comID=0,
client_ip_address=None,
editor_type='textarea',
can_attach_files=False,
subscribe=False,
req=None,
attached_files=None,
warnings=None):
"""
Add a comment/review or remark
@param recID: record id
@param uid: user id
@param action: 'DISPLAY' to display add form
'SUBMIT' to submit comment once form is filled
'REPLY' to reply to an existing comment
@param ln: language
@param msg: the body of the comment/review or remark
@param score: star score of the review
@param note: title of the review
@param priority: priority of remark (int)
@param reviews: boolean, if enabled will add a review, if disabled will add a comment
@param comID: if replying, this is the comment id of the comment we are replying to
@param editor_type: the kind of editor/input used for the comment: 'textarea', 'ckeditor'
@param can_attach_files: if user can attach files to comments or not
@param subscribe: if True, subscribe user to receive new comments by email
@param req: request object. Used to register callback to send email notification
@param attached_files: newly attached files to this comment, mapping filename to filepath
@type attached_files: dict
@param warnings: list of warning tuples (warning_text, warning_color) that should be considered
@return:
- html add form if action is display or reply
- html successful added form if action is submit
"""
_ = gettext_set_language(ln)
if warnings is None:
warnings = []
actions = ['DISPLAY', 'REPLY', 'SUBMIT']
_ = gettext_set_language(ln)
## check arguments
check_recID_is_in_range(recID, warnings, ln)
if uid <= 0:
try:
raise InvenioWebCommentError(_('%(uid)s is an invalid user ID.', uid=uid))
except InvenioWebCommentError as exc:
register_exception()
body = webcomment_templates.tmpl_error(exc.message, ln)
return body
#errors.append(('ERR_WEBCOMMENT_UID_INVALID', uid))
return ''
if attached_files is None:
attached_files = {}
user_contact_info = query_get_user_contact_info(uid)
nickname = ''
if user_contact_info:
if user_contact_info[0]:
nickname = user_contact_info[0]
# show the form
if action == 'DISPLAY':
if reviews and CFG_WEBCOMMENT_ALLOW_REVIEWS:
return webcomment_templates.tmpl_add_comment_form_with_ranking(recID, uid, nickname, ln, msg, score, note, warnings, can_attach_files=can_attach_files)
elif not reviews and CFG_WEBCOMMENT_ALLOW_COMMENTS:
return webcomment_templates.tmpl_add_comment_form(recID, uid, nickname, ln, msg, warnings, can_attach_files=can_attach_files)
else:
try:
raise InvenioWebCommentError(_('Comments on records have been disallowed by the administrator.'))
except InvenioWebCommentError as exc:
register_exception(req=req)
body = webcomment_templates.tmpl_error(exc.message, ln)
return body
#errors.append(('ERR_WEBCOMMENT_COMMENTS_NOT_ALLOWED',))
elif action == 'REPLY':
if reviews and CFG_WEBCOMMENT_ALLOW_REVIEWS:
try:
raise InvenioWebCommentError(_('Cannot reply to a review.'))
except InvenioWebCommentError as exc:
register_exception(req=req)
body = webcomment_templates.tmpl_error(exc.message, ln)
return body
#errors.append(('ERR_WEBCOMMENT_REPLY_REVIEW',))
return webcomment_templates.tmpl_add_comment_form_with_ranking(recID, uid, nickname, ln, msg, score, note, warnings, can_attach_files=can_attach_files)
elif not reviews and CFG_WEBCOMMENT_ALLOW_COMMENTS:
textual_msg = msg
if comID > 0:
comment = query_get_comment(comID)
if comment:
user_info = get_user_info(comment[2])
if user_info:
date_creation = convert_datetext_to_dategui(str(comment[4]))
# Build two msg: one mostly textual, the other one with HTML markup, for the CkEditor.
msg = _("%(x_name)s wrote on %(x_date)s:")% {'x_name': user_info[2], 'x_date': date_creation}
textual_msg = msg
# 1 For CkEditor input
msg += '\n\n'
msg += comment[3]
msg = email_quote_txt(text=msg)
# Now that we have a text-quoted version, transform into
# something that CkEditor likes, using <blockquote> that
# do still enable users to insert comments inline
msg = email_quoted_txt2html(text=msg,
indent_html=('<blockquote><div>', '&nbsp;&nbsp;</div></blockquote>'),
linebreak_html="&nbsp;<br/>",
indent_block=False)
# Add some space for users to easily add text
# around the quoted message
msg = '<br/>' + msg + '<br/>'
# Due to how things are done, we need to
# escape the whole msg again for the editor
msg = cgi.escape(msg)
# 2 For textarea input
textual_msg += "\n\n"
textual_msg += comment[3]
textual_msg = email_quote_txt(text=textual_msg)
return webcomment_templates.tmpl_add_comment_form(recID, uid, nickname, ln, msg, warnings, textual_msg, can_attach_files=can_attach_files, reply_to=comID)
else:
try:
raise InvenioWebCommentError(_('Comments on records have been disallowed by the administrator.'))
except InvenioWebCommentError as exc:
register_exception(req=req)
body = webcomment_templates.tmpl_error(exc.message, ln)
return body
#errors.append(('ERR_WEBCOMMENT_COMMENTS_NOT_ALLOWED',))
# check before submitting form
elif action == 'SUBMIT':
if reviews and CFG_WEBCOMMENT_ALLOW_REVIEWS:
if note.strip() in ["", "None"] and not CFG_WEBCOMMENT_ALLOW_SHORT_REVIEWS:
try:
raise InvenioWebCommentWarning(_('You must enter a title.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_ADD_NO_TITLE',))
if score == 0 or score > 5:
try:
raise InvenioWebCommentWarning(_('You must choose a score.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(("WRN_WEBCOMMENT_ADD_NO_SCORE",))
if msg.strip() in ["", "None"] and not CFG_WEBCOMMENT_ALLOW_SHORT_REVIEWS:
try:
raise InvenioWebCommentWarning(_('You must enter a text.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_ADD_NO_BODY',))
# if no warnings, submit
if len(warnings) == 0:
if reviews:
if check_user_can_review(recID, client_ip_address, uid):
success = query_add_comment_or_remark(reviews, recID=recID, uid=uid, msg=msg,
note=note, score=score, priority=0,
client_ip_address=client_ip_address,
editor_type=editor_type,
req=req,
reply_to=comID)
else:
try:
raise InvenioWebCommentWarning(_('You already wrote a review for this record.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append('WRN_WEBCOMMENT_CANNOT_REVIEW_TWICE')
success = 1
else:
if check_user_can_comment(recID, client_ip_address, uid):
success = query_add_comment_or_remark(reviews, recID=recID, uid=uid, msg=msg,
note=note, score=score, priority=0,
client_ip_address=client_ip_address,
editor_type=editor_type,
req=req,
reply_to=comID, attached_files=attached_files)
if success > 0 and subscribe:
subscribe_user_to_discussion(recID, uid)
else:
try:
raise InvenioWebCommentWarning(_('You already posted a comment short ago. Please retry later.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append('WRN_WEBCOMMENT_TIMELIMIT')
success = 1
if success > 0:
if CFG_WEBCOMMENT_ADMIN_NOTIFICATION_LEVEL > 0:
notify_admin_of_new_comment(comID=success)
return webcomment_templates.tmpl_add_comment_successful(recID, ln, reviews, warnings, success)
else:
try:
raise InvenioWebCommentError(_('Failed to insert your comment to the database. Please try again.'))
except InvenioWebCommentError as exc:
register_exception(req=req)
body = webcomment_templates.tmpl_error(exc.message, ln)
return body
#errors.append(('ERR_WEBCOMMENT_DB_INSERT_ERROR'))
# if are warnings or if inserting comment failed, show user where warnings are
if reviews and CFG_WEBCOMMENT_ALLOW_REVIEWS:
return webcomment_templates.tmpl_add_comment_form_with_ranking(recID, uid, nickname, ln, msg, score, note, warnings, can_attach_files=can_attach_files)
else:
return webcomment_templates.tmpl_add_comment_form(recID, uid, nickname, ln, msg, warnings, can_attach_files=can_attach_files)
# unknown action send to display
else:
try:
raise InvenioWebCommentWarning(_('Unknown action --> showing you the default add comment form.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning', req=req)
warnings.append((exc.message, ''))
#warnings.append(('WRN_WEBCOMMENT_ADD_UNKNOWN_ACTION',))
if reviews and CFG_WEBCOMMENT_ALLOW_REVIEWS:
return webcomment_templates.tmpl_add_comment_form_with_ranking(recID, uid, ln, msg, score, note, warnings, can_attach_files=can_attach_files)
else:
return webcomment_templates.tmpl_add_comment_form(recID, uid, ln, msg, warnings, can_attach_files=can_attach_files)
return ''
def notify_admin_of_new_comment(comID):
"""
Sends an email to the admin with details regarding comment with ID = comID
"""
comment = query_get_comment(comID)
if len(comment) > 0:
(comID2,
id_bibrec,
id_user,
body,
date_creation,
star_score, nb_votes_yes, nb_votes_total,
title,
nb_abuse_reports, round_name, restriction) = comment
else:
return
user_info = query_get_user_contact_info(id_user)
if len(user_info) > 0:
(nickname, email, last_login) = user_info
if not len(nickname) > 0:
nickname = email.split('@')[0]
else:
nickname = email = last_login = "ERROR: Could not retrieve"
review_stuff = '''
Star score = %s
Title = %s''' % (star_score, title)
washer = EmailWasher()
try:
body = washer.wash(body)
except:
body = cgi.escape(body)
record_info = webcomment_templates.tmpl_email_new_comment_admin(id_bibrec)
out = '''
The following %(comment_or_review)s has just been posted (%(date)s).
AUTHOR:
Nickname = %(nickname)s
Email = %(email)s
User ID = %(uid)s
RECORD CONCERNED:
Record ID = %(recID)s
URL = <%(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/%(comments_or_reviews)s/>
%(record_details)s
%(comment_or_review_caps)s:
%(comment_or_review)s ID = %(comID)s %(review_stuff)s
Body =
<--------------->
%(body)s
<--------------->
ADMIN OPTIONS:
To moderate the %(comment_or_review)s go to %(siteurl)s/%(CFG_SITE_RECORD)s/%(recID)s/%(comments_or_reviews)s/display?%(arguments)s
''' % \
{ 'comment_or_review' : star_score > 0 and 'review' or 'comment',
'comment_or_review_caps': star_score > 0 and 'REVIEW' or 'COMMENT',
'comments_or_reviews' : star_score > 0 and 'reviews' or 'comments',
'date' : date_creation,
'nickname' : nickname,
'email' : email,
'uid' : id_user,
'recID' : id_bibrec,
'record_details' : record_info,
'comID' : comID2,
'review_stuff' : star_score > 0 and review_stuff or "",
'body' : body.replace('<br />','\n'),
'siteurl' : CFG_SITE_URL,
'CFG_SITE_RECORD' : CFG_SITE_RECORD,
'arguments' : 'ln=en&do=od#%s' % comID
}
from_addr = '%s WebComment <%s>' % (CFG_SITE_NAME, CFG_WEBALERT_ALERT_ENGINE_EMAIL)
comment_collection = get_comment_collection(comID)
to_addrs = get_collection_moderators(comment_collection)
rec_collection = guess_primary_collection_of_a_record(id_bibrec)
report_nums = get_fieldvalues(id_bibrec, "037__a")
report_nums += get_fieldvalues(id_bibrec, "088__a")
report_nums = ', '.join(report_nums)
subject = "A new comment/review has just been posted [%s|%s]" % (rec_collection, report_nums)
send_email(from_addr, to_addrs, subject, out)
def check_recID_is_in_range(recID, warnings=[], ln=CFG_SITE_LANG):
"""
Check that recID is >= 0
@param recID: record id
@param warnings: list of warning tuples (warning_text, warning_color)
@return: tuple (boolean, html) where boolean (1=true, 0=false)
and html is the body of the page to display if there was a problem
"""
_ = gettext_set_language(ln)
try:
recID = int(recID)
except:
pass
if type(recID) is int:
if recID > 0:
from invenio.legacy.search_engine import record_exists
success = record_exists(recID)
if success == 1:
return (1,"")
else:
try:
- raise InvenioWebCommentWarning(_('Record ID %(recid)s does not exist in the database.', recid=recID))
+ if success == -1:
+ status = 'deleted'
+ raise InvenioWebCommentWarning(_('The record has been deleted.'))
+ else:
+ status = 'inexistant'
+ raise InvenioWebCommentWarning(_('Record ID %s does not exist in the database.') % recID)
except InvenioWebCommentWarning as exc:
register_exception(stream='warning')
warnings.append((exc.message, ''))
#warnings.append(('ERR_WEBCOMMENT_RECID_INEXISTANT', recID))
- return (0, webcomment_templates.tmpl_record_not_found(status='inexistant', recID=recID, ln=ln))
+ return (0, webcomment_templates.tmpl_record_not_found(status=status, recID=recID, ln=ln))
elif recID == 0:
try:
raise InvenioWebCommentWarning(_('No record ID was given.'))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning')
warnings.append((exc.message, ''))
#warnings.append(('ERR_WEBCOMMENT_RECID_MISSING',))
return (0, webcomment_templates.tmpl_record_not_found(status='missing', recID=recID, ln=ln))
else:
try:
raise InvenioWebCommentWarning(_('Record ID %(recid)s is an invalid ID.', recid=recID))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning')
warnings.append((exc.message, ''))
#warnings.append(('ERR_WEBCOMMENT_RECID_INVALID', recID))
return (0, webcomment_templates.tmpl_record_not_found(status='invalid', recID=recID, ln=ln))
else:
try:
raise InvenioWebCommentWarning(_('Record ID %(recid)s is not a number.', recid=recID))
except InvenioWebCommentWarning as exc:
register_exception(stream='warning')
warnings.append((exc.message, ''))
#warnings.append(('ERR_WEBCOMMENT_RECID_NAN', recID))
return (0, webcomment_templates.tmpl_record_not_found(status='nan', recID=recID, ln=ln))
def check_int_arg_is_in_range(value, name, gte_value, lte_value=None):
"""
Check that variable with name 'name' >= gte_value and optionally <= lte_value
@param value: variable value
@param name: variable name
@param errors: list of error tuples (error_id, value)
@param gte_value: greater than or equal to value
@param lte_value: less than or equal to value
@return: boolean (1=true, 0=false)
"""
if type(value) is not int:
try:
raise InvenioWebCommentError('%s is not a number.' % value)
except InvenioWebCommentError as exc:
register_exception()
body = webcomment_templates.tmpl_error(exc.message)
return body
#errors.append(('ERR_WEBCOMMENT_ARGUMENT_NAN', value))
return 0
if value < gte_value:
try:
raise InvenioWebCommentError('%s invalid argument.' % value)
except InvenioWebCommentError as exc:
register_exception()
body = webcomment_templates.tmpl_error(exc.message)
return body
#errors.append(('ERR_WEBCOMMENT_ARGUMENT_INVALID', value))
return 0
if lte_value:
if value > lte_value:
try:
raise InvenioWebCommentError('%s invalid argument.' % value)
except InvenioWebCommentError as exc:
register_exception()
body = webcomment_templates.tmpl_error(exc.message)
return body
#errors.append(('ERR_WEBCOMMENT_ARGUMENT_INVALID', value))
return 0
return 1
def get_mini_reviews(recid, ln=CFG_SITE_LANG):
"""
Returns the web controls to add reviews to a record from the
detailed record pages mini-panel.
@param recid: the id of the displayed record
@param ln: the user's language
"""
if CFG_WEBCOMMENT_ALLOW_SHORT_REVIEWS:
action = 'SUBMIT'
else:
action = 'DISPLAY'
reviews = query_retrieve_comments_or_remarks(recid, ranking=1)
return webcomment_templates.tmpl_mini_review(recid, ln, action=action,
avg_score=calculate_avg_score(reviews),
nb_comments_total=len(reviews))
def check_user_can_view_comments(user_info, recid):
"""Check if the user is authorized to view comments for given
recid.
Returns the same type as acc_authorize_action
"""
# Check user can view the record itself first
(auth_code, auth_msg) = check_user_can_view_record(user_info, recid)
if auth_code:
return (auth_code, auth_msg)
# Check if user can view the comments
## But first can we find an authorization for this case action,
## for this collection?
record_primary_collection = guess_primary_collection_of_a_record(recid)
return acc_authorize_action(user_info, 'viewcomment', authorized_if_no_roles=True, collection=record_primary_collection)
def check_user_can_view_comment(user_info, comid, restriction=None):
"""Check if the user is authorized to view a particular comment,
given the comment restriction. Note that this function does not
check if the record itself is restricted to the user, which would
mean that the user should not see the comment.
You can omit 'comid' if you already know the 'restriction'
@param user_info: the user info object
@param comid: the comment id of that we want to check
@param restriction: the restriction applied to given comment (if known. Otherwise retrieved automatically)
@return: the same type as acc_authorize_action
"""
if restriction is None:
comment = query_get_comment(comid)
if comment:
restriction = comment[11]
else:
return (1, 'Comment %i does not exist' % comid)
if restriction == "":
return (0, '')
return acc_authorize_action(user_info, 'viewrestrcomment', status=restriction)
def check_user_can_send_comments(user_info, recid):
"""Check if the user is authorized to comment the given
recid. This function does not check that user can view the record
or view the comments
Returns the same type as acc_authorize_action
"""
## First can we find an authorization for this case, action + collection
record_primary_collection = guess_primary_collection_of_a_record(recid)
return acc_authorize_action(user_info, 'sendcomment', authorized_if_no_roles=True, collection=record_primary_collection)
def check_comment_belongs_to_record(comid, recid):
"""
Return True if the comment is indeed part of given record (even if comment or/and record have
been "deleted"). Else return False.
@param comid: the id of the comment to check membership
@param recid: the recid of the record we want to check if comment belongs to
"""
query = """SELECT id_bibrec from cmtRECORDCOMMENT WHERE id=%s"""
params = (comid,)
res = run_sql(query, params)
if res and res[0][0] == recid:
return True
return False
def check_user_can_attach_file_to_comments(user_info, recid):
"""Check if the user is authorized to attach a file to comments
for given recid. This function does not check that user can view
the comments or send comments.
Returns the same type as acc_authorize_action
"""
## First can we find an authorization for this case action, for
## this collection?
record_primary_collection = guess_primary_collection_of_a_record(recid)
return acc_authorize_action(user_info, 'attachcommentfile', authorized_if_no_roles=False, collection=record_primary_collection)
def toggle_comment_visibility(uid, comid, collapse, recid):
"""
Toggle the visibility of the given comment (collapse) for the
given user. Return the new visibility
@param uid: the user id for which the change applies
@param comid: the comment id to close/open
@param collapse: if the comment is to be closed (1) or opened (0)
@param recid: the record id to which the comment belongs
@return: if the comment is visible or not after the update
"""
# We rely on the client to tell if comment should be collapsed or
# developed, to ensure consistency between our internal state and
# client state. Even if not strictly necessary, we store the
# record ID for quicker retrieval of the collapsed comments of a
# given discussion page. To prevent unnecessary population of the
# table, only one distinct tuple (record ID, comment ID, user ID)
# can be inserted (due to table definition). For the same purpose
# we also check that comment to collapse exists, and corresponds
# to an existing record: we cannot rely on the recid found as part
# of the URL, as no former check is done. This rule is not applied
# when deleting an entry, as in the worst case no line would be
# removed. For optimized retrieval of row to delete, the id_bibrec
# column is used, though not strictly necessary.
if collapse:
query = """SELECT id_bibrec from cmtRECORDCOMMENT WHERE id=%s"""
params = (comid,)
res = run_sql(query, params)
if res:
query = """INSERT DELAYED IGNORE INTO cmtCOLLAPSED (id_bibrec, id_cmtRECORDCOMMENT, id_user)
VALUES (%s, %s, %s)"""
params = (res[0][0], comid, uid)
run_sql(query, params)
return True
else:
query = """DELETE FROM cmtCOLLAPSED WHERE
id_cmtRECORDCOMMENT=%s and
id_user=%s and
id_bibrec=%s"""
params = (comid, uid, recid)
run_sql(query, params)
return False
def get_user_collapsed_comments_for_record(uid, recid):
"""
Get the comments collapsed for given user on given recid page
"""
# Collapsed state is not an attribute of cmtRECORDCOMMENT table
# (vary per user) so it cannot be found when querying for the
# comment. We must therefore provide a efficient way to retrieve
# the collapsed state for a given discussion page and user.
query = """SELECT id_cmtRECORDCOMMENT from cmtCOLLAPSED WHERE id_user=%s and id_bibrec=%s"""
params = (uid, recid)
return [res[0] for res in run_sql(query, params)]
def is_comment_deleted(comid):
"""
Return True of the comment is deleted. Else False
@param comid: ID of comment to check
"""
query = "SELECT status from cmtRECORDCOMMENT WHERE id=%s"
params = (comid,)
res = run_sql(query, params)
if res and res[0][0] != 'ok':
return True
return False
def perform_display_your_comments(user_info,
page_number=1,
selected_order_by_option="lcf",
selected_display_number_option="all",
selected_display_format_option="rc",
ln=CFG_SITE_LANG):
"""
Display all comments submitted by the user.
@TODO: support reviews too
@param user_info: standard user info object.
@param comments: ordered list of tuples (id_bibrec, comid, date_creation, body, status, in_reply_to_id_cmtRECORDCOMMENT)
@param page_number: page on which the user is.
@type page_number: integer
@param selected_order_by_option: seleccted ordering option. Can be one of:
- ocf: Oldest comment first
- lcf: Latest comment first
- grof: Group by record, oldest commented first
- grlf: Group by record, latest commented first
@type selected_order_by_option: string
@param selected_display_number_option: number of results to show per page. Can be a string-digit or 'all'.
@type selected_display_number_option: string
@param selected_display_format_option: how to show records. Can be one of:
- rc: Records and comments
- ro: Records only
- co: Comments only
@type selected_display_format_option: string
@ln: language
@type ln: string
"""
query_params = ""
nb_total_pages = 0
if selected_display_format_option in ('rc', 'co'):
nb_total_results = run_sql("SELECT count(id) from cmtRECORDCOMMENT WHERE id_user=%s AND star_score = 0", \
(user_info['uid'], ))[0][0]
else:
if selected_order_by_option in ('grlf', 'grof'):
nb_total_results = run_sql("SELECT count(distinct(id_bibrec)) from cmtRECORDCOMMENT WHERE id_user=%s AND star_score = 0", \
(user_info['uid'], ))[0][0]
else:
nb_total_results = run_sql("SELECT count(id_bibrec) from cmtRECORDCOMMENT WHERE id_user=%s AND star_score = 0", \
(user_info['uid'], ))[0][0]
if page_number < 1:
page_number = 1
if selected_display_number_option != 'all' and \
not selected_display_number_option.isdigit():
# must be some garbage
selected_display_number_option = 'all'
query = ''
if selected_order_by_option == "lcf":
query_params += " ORDER BY date_creation DESC"
elif selected_order_by_option == "ocf":
query_params += " ORDER BY date_creation ASC"
elif selected_order_by_option == "grlf":
query = "SELECT cmt.id_bibrec, cmt.id, cmt.date_creation, cmt.body, cmt.status, cmt.in_reply_to_id_cmtRECORDCOMMENT FROM cmtRECORDCOMMENT as cmt left join (SELECT max(date_creation) as maxdatecreation, id_bibrec FROM cmtRECORDCOMMENT WHERE id_user=%s AND star_score = 0 GROUP BY id_bibrec) as grp on cmt.id_bibrec = grp.id_bibrec WHERE id_user=%s AND star_score = 0 ORDER BY grp.maxdatecreation DESC, cmt.date_creation DESC"
elif selected_order_by_option == "grof":
query = "SELECT cmt.id_bibrec, cmt.id, cmt.date_creation, cmt.body, cmt.status, cmt.in_reply_to_id_cmtRECORDCOMMENT FROM cmtRECORDCOMMENT as cmt left join (SELECT min(date_creation) as mindatecreation, id_bibrec FROM cmtRECORDCOMMENT WHERE id_user=%s AND star_score = 0 GROUP BY id_bibrec) as grp on cmt.id_bibrec = grp.id_bibrec WHERE id_user=%s AND star_score = 0 ORDER BY grp.mindatecreation ASC"
if selected_display_number_option.isdigit():
selected_display_number_option_as_int = int(selected_display_number_option)
if selected_display_number_option_as_int < 5:
selected_display_number_option_as_int = 5
selected_display_number_option = str(selected_display_number_option_as_int)
from_index = (page_number - 1) * int(selected_display_number_option)
query_params += ' LIMIT ' + \
str(from_index) + \
',' + \
str(int(selected_display_number_option))
nb_total_pages = int(math.ceil(float(nb_total_results) / selected_display_number_option_as_int))
if selected_order_by_option in ("grlf", "grof"):
res = run_sql(query + query_params, (user_info['uid'], user_info['uid']))
else:
res = run_sql("SELECT id_bibrec, id, date_creation, body, status, in_reply_to_id_cmtRECORDCOMMENT FROM cmtRECORDCOMMENT WHERE id_user=%s AND star_score = 0" + query_params, (user_info['uid'], ))
return webcomment_templates.tmpl_your_comments(user_info, res,
page_number=page_number,
selected_order_by_option=selected_order_by_option,
selected_display_number_option=selected_display_number_option,
selected_display_format_option=selected_display_format_option,
nb_total_results=nb_total_results,
nb_total_pages=nb_total_pages,
ln=ln)
diff --git a/invenio/modules/deposit/tasks.py b/invenio/modules/deposit/tasks.py
index 06997ed51..5e65fd179 100644
--- a/invenio/modules/deposit/tasks.py
+++ b/invenio/modules/deposit/tasks.py
@@ -1,523 +1,523 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
"""
from __future__ import print_function
import os
import dictdiffer
import dateutil.parser
from tempfile import mkstemp
from functools import partial
from flask import current_app, abort, request
from flask.ext.login import current_user
from invenio.modules.records.api import get_record
from invenio.modules.deposit.models import Deposition, Agent, \
DepositionDraftCacheManager
from invenio.ext.logging import register_exception
from invenio.ext.restful import error_codes
from invenio.modules.formatter import format_record
from .helpers import record_to_draft, make_record, \
deposition_record
from invenio.legacy.bibdocfile.api import BibRecDocs
-from invenio.legacy.bibsched.bibtask import task_low_level_submission, \
- bibtask_allocate_sequenceid
from invenio.modules.pidstore.models import PersistentIdentifier
#
# Helpers
#
def filter_empty_helper(keys=None):
""" Remove empty elements from a list"""
def _inner(elem):
if isinstance(elem, dict):
for k, v in elem.items():
if (keys is None or k in keys) and v:
return True
return False
else:
return bool(elem)
return _inner
#
# Workflow tasks
#
def is_api_request(obj, eng):
""" Check if request is an API request """
return getattr(request, 'is_api_request', False)
def has_submission(obj, eng):
"""
"""
d = Deposition(obj)
return d.has_sip()
def is_sip_uploaded(sip, record=None):
"""
Check if a submission information package for a record has been uploaded
"""
if not sip.is_sealed():
return False
if record is None:
record = get_record(sip.metadata.get('recid'), reset_cache=True)
sip_version_id = sip.metadata.get('modification_date')
if sip_version_id:
sip_version_id = dateutil.parser.parse(sip_version_id)
record_version_id = record.get('modification_date') if record else None
# Check of record in latest SIP has been uploaded (record version must
# be newer than SIP record version.
if record_version_id is None or (sip_version_id and
sip_version_id >= record_version_id):
return False
else:
return True
def authorize_user(action, **params):
"""
Check if current user is authorized to perform the action.
"""
def _authorize_user(obj, dummy_eng):
from invenio.modules.access.engine import acc_authorize_action
auth, message = acc_authorize_action(
current_user.get_id(),
action,
**dict((k, v() if callable(v) else v)
for (k, v) in params.items()))
if auth != 0:
current_app.logger.info(message)
abort(401)
return _authorize_user
def prefill_draft(draft_id='_default', clear=True):
"""
Fill draft values with values from pre-filled cache
"""
def _prefill_draft(obj, eng):
if not getattr(request, 'is_api_request', False):
draft_cache = DepositionDraftCacheManager.get()
if draft_cache.has_data():
d = Deposition(obj)
draft_cache.fill_draft(d, draft_id, clear=clear)
d.update()
return _prefill_draft
def render_form(draft_id='_default'):
"""
Renders a form if the draft associated with it has not yet been completed.
:param draft_id: The name of the draft to create. Must be specified if you
put more than two ``render_form'''s in your deposition workflow.
"""
def _render_form(obj, eng):
d = Deposition(obj)
draft = d.get_or_create_draft(draft_id)
if getattr(request, 'is_api_request', False):
form = draft.get_form(validate_draft=True)
if form.errors:
error_messages = []
for field, msgs in form.errors:
for m in msgs:
error_messages.append(
field=field,
message=m,
code=error_codes['validation_error'],
)
d.set_render_context(dict(
response=dict(
message="Bad request",
status=400,
errors=error_messages,
),
status=400,
))
eng.halt("API: Draft did not validate")
else:
if draft.is_completed():
eng.jumpCallForward(1)
else:
form = draft.get_form(validate_draft=draft.validate)
form.validate = True
d.set_render_context(dict(
template_name_or_list=form.get_template(),
deposition=d,
deposition_type=(
None if d.type.is_default() else
d.type.get_identifier()
),
uuid=d.id,
draft=draft,
form=form,
my_depositions=Deposition.get_depositions(
current_user, type=d.type
),
))
d.update()
eng.halt('Wait for form submission.')
return _render_form
def load_record(draft_id='_default', producer='json_for_form',
pre_process=None, post_process=None):
"""
Load a record and map to draft data.
"""
def _load_record(obj, eng):
d = Deposition(obj)
sip = d.get_latest_sip(sealed=True)
record = get_record(sip.metadata.get('recid'), reset_cache=True)
if not is_sip_uploaded(sip, record=record):
if getattr(request, 'is_api_request', False):
d.set_render_context(dict(
response=dict(
message="Conflict",
status=409,
errors="Upload not yet fully integrated. Please wait"
" a few moments.",
),
status=409,
))
else:
from flask import flash
flash(
"Editing is only possible after your upload have been"
" fully integrated. Please wait a few moments, then try"
" to reload the page.",
category='warning'
)
d.set_render_context(dict(
template_name_or_list="deposit/completed.html",
deposition=d,
deposition_type=(
None if d.type.is_default() else
d.type.get_identifier()
),
uuid=d.id,
sip=sip,
my_depositions=Deposition.get_depositions(
current_user, type=d.type
),
format_record=format_record,
))
d.update()
eng.halt("Wait for record to be uploaded")
# Check if record is already loaded, if so, skip.
if d.drafts:
eng.jumpCallForward(1)
# Load draft
draft = d.get_or_create_draft(draft_id)
# Fill draft with values from recjson
record_to_draft(
record, draft=draft, post_process=post_process, producer=producer
)
d.update()
# Stop API request
if getattr(request, 'is_api_request', False):
d.set_render_context(dict(
response=d.marshal(),
status=201,
))
eng.halt("API request")
return _load_record
def merge_changes(deposition, dest, a, b):
"""
Find changes between two dictionaries A and B, and apply the changes
to a destination dictionary.
This method is useful when A is a subset of the destination dictionary.
"""
# Generate patch
patch = dictdiffer.diff(a, b)
# Apply patch (returns a deep copy of dest with patch applied)
return dictdiffer.patch(patch, dest)
def merge_record(draft_id='_default', pre_process_load=None,
post_process_load=None, process_export=None,
merge_func=merge_changes):
"""
Merge recjson with a record
This task will load the current record, diff the changes from the
deposition against it, and apply the patch.
The merge algorithm works in the following way:
* First the current record is loaded.
* Then all fields which is not related to the deposition is removed from
the current record, to produce a simplified version of the record.
* Next the simplified version of the record is compared against the
changes the user have made.
* These changes are then applied to the full current record.
"""
if not merge_func or not callable(merge_func):
raise RuntimeError("No merge function given.")
def _merge_record(obj, eng):
d = Deposition(obj)
sip = d.get_latest_sip(sealed=False)
# Get the current record, which contains all fields.
current_record = get_record(
sip.metadata.get('recid'), reset_cache=True
)
form_class = d.get_draft(draft_id).form_class
# Create a simplified record from the current record, that only
# contains fields concerning this deposition.
current_simple_record = deposition_record(
current_record,
[form_class],
pre_process_load=pre_process_load,
post_process_load=post_process_load,
process_export=partial(process_export, d),
)
# Create a simplified record from the changes the user have made.
changed_simple_record = make_record(sip.metadata, is_dump=True)
# Make an initial patch of current record (e.g. some default values set
# by the form, might not exists in the current record)
for k in current_simple_record:
if k not in current_record:
current_record[k] = current_simple_record[k]
# Export clean dumps
current_simple_json = current_simple_record.dumps(clean=True)
changed_simple_json = changed_simple_record.dumps(clean=True)
current_full_json = current_record.dumps(clean=True)
# Merge changes from changed record into the current record.
sip.metadata = merge_func(
d,
current_full_json,
current_simple_json,
changed_simple_json,
)
# Ensure we are based on latest version_id to prevent being rejected in
# the bibupload queue.
sip.metadata['modification_date'] = \
current_full_json['modification_date']
d.update()
return _merge_record
def create_recid():
"""
Create a new record id.
"""
def _create_recid(obj, dummy_eng):
d = Deposition(obj)
sip = d.get_latest_sip(sealed=False)
if sip is None:
raise Exception("No submission information package found.")
if 'recid' not in sip.metadata:
from invenio.legacy.bibupload.engine import create_new_record
sip.metadata['recid'] = create_new_record()
d.update()
return _create_recid
def mint_pid(pid_field='doi', pid_creator=None, pid_store_type='doi',
existing_pid_checker=None):
"""
Register a persistent identifier internally.
:param pid_field: The recjson key for where to look for a pre-reserved pid.
Defaults to 'pid'.
:param pid_creator: Callable taking one argument (the recjson) that when
called will generate and return a pid string.
:param pid_store_type: The PID store type. Defaults to 'doi'.
:param existing_pid_checker: A callable taking two arguments
(pid_str, recjson) that will check if an pid found using ``pid_field''
should be registered or not.
"""
def _mint_pid(obj, dummy_eng):
d = Deposition(obj)
recjson = d.get_latest_sip(sealed=False).metadata
if 'recid' not in recjson:
raise Exception("'recid' not found in sip metadata.")
pid_text = None
pid = recjson.get(pid_field, None)
if not pid:
# No pid found in recjson, so create new pid with user supplied
# function.
pid_text = recjson[pid_field] = pid_creator(recjson)
else:
# Pid found - check if it should be minted
if existing_pid_checker and existing_pid_checker(pid, recjson):
pid_text = pid
# Create an assign pid internally - actually registration will happen
# asynchronously later.
if pid_text:
current_app.logger.info("Registering pid %s" % pid_text)
pid_obj = PersistentIdentifier.create(pid_store_type, pid_text)
if pid_obj is None:
pid_obj = PersistentIdentifier.get(pid_store_type, pid_text)
try:
pid_obj.assign("rec", recjson['recid'])
except Exception:
register_exception(alert_admin=True)
d.update()
return _mint_pid
def process_bibdocfile(process=None):
"""
Process bibdocfiles with custom processor
"""
def _bibdocfile_update(obj, eng):
if process:
d = Deposition(obj)
sip = d.get_latest_sip(sealed=False)
recid = sip.metadata.get('recid')
if recid:
brd = BibRecDocs(int(recid))
process(d, brd)
d.update()
return _bibdocfile_update
def prepare_sip():
"""
Prepare a submission information package
"""
def _prepare_sip(obj, dummy_eng):
d = Deposition(obj)
sip = d.get_latest_sip(sealed=False)
if sip is None:
sip = d.create_sip()
# FIXME: Move to somewhere more appropriate
# create_sip by default stick files into the files attribute.
if 'files' in sip.metadata:
sip.metadata['fft'] = sip.metadata['files']
del sip.metadata['files']
sip.agents = [Agent(role='creator', from_request_context=True)]
d.update()
return _prepare_sip
def process_sip_metadata(processor=None):
"""
Process metadata in submission information package using a custom
processor.
"""
def _prepare_sip(obj, dummy_eng):
d = Deposition(obj)
metadata = d.get_latest_sip(sealed=False).metadata
if processor is not None:
processor(d, metadata)
elif processor is None and hasattr(d.type, 'process_sip_metadata'):
d.type.process_sip_metadata(d, metadata)
d.update()
return _prepare_sip
def finalize_record_sip(is_dump=True):
"""
Finalizes the SIP by generating the MARC and storing it in the SIP.
"""
def _finalize_sip(obj, dummy_eng):
d = Deposition(obj)
sip = d.get_latest_sip(sealed=False)
sip.package = make_record(
sip.metadata, is_dump=is_dump
).legacy_export_as_marc()
d.update()
return _finalize_sip
def upload_record_sip():
"""
Generates the record from marc.
The function requires the marc to be generated,
so the function export_marc_from_json must have been called successfully
before
"""
def create(obj, dummy_eng):
#FIXME change share tmp directory
from invenio.config import CFG_TMPSHAREDDIR
+ from invenio.legacy.bibsched.bibtask import task_low_level_submission, \
+ bibtask_allocate_sequenceid
d = Deposition(obj)
sip = d.get_latest_sip(sealed=False)
sip.seal()
tmp_file_fd, tmp_file_path = mkstemp(
prefix="webdeposit-%s-%s" % (d.id, sip.uuid),
suffix='.xml',
dir=CFG_TMPSHAREDDIR,
)
os.write(tmp_file_fd, sip.package)
os.close(tmp_file_fd)
# Trick to have access to task_sequence_id in subsequent tasks.
d.workflow_object.task_sequence_id = bibtask_allocate_sequenceid()
task_id = task_low_level_submission(
'bibupload', 'webdeposit',
'-r' if 'recid' in sip.metadata else '-i', tmp_file_path, '-P5',
'-I', str(d.workflow_object.task_sequence_id)
)
sip.task_ids.append(task_id)
d.update()
return create
diff --git a/invenio/modules/encoder/batch_engine.py b/invenio/modules/encoder/batch_engine.py
index 198545666..8796f62e1 100644
--- a/invenio/modules/encoder/batch_engine.py
+++ b/invenio/modules/encoder/batch_engine.py
@@ -1,760 +1,762 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2011 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""Bibencode batch processing submodule"""
from string import Template
from pprint import pprint
import os
import shutil
import uuid
from pprint import pformat
from invenio.legacy.bibsched.bibtask import (
task_update_progress,
write_message,
task_low_level_submission
)
from invenio.legacy.bibdocfile.api import BibRecDocs, compose_file, compose_format, decompose_file
from invenio.legacy.search_engine import (
record_exists,
get_collection_reclist,
search_pattern,
get_fieldvalues
)
from invenio.modules.encoder.encode import encode_video, assure_quality
from invenio.modules.encoder.extract import extract_frames
from invenio.modules.encoder.profiles import (
get_encoding_profile,
get_extract_profile
)
from invenio.legacy.bibdocfile.cli import cli_fix_marc
from invenio.modules.encoder.utils import chose2
from invenio.modules.encoder.metadata import (
pbcore_metadata
)
from invenio.modules.encoder.utils import getval, chose2, generate_timestamp
from invenio.modules.encoder.config import (
CFG_BIBENCODE_DAEMON_DIR_NEWJOBS,
CFG_BIBENCODE_PBCORE_MARC_XSLT,
CFG_BIBENCODE_ASPECT_RATIO_MARC_FIELD
)
from invenio.ext.email import send_email
from invenio.base.i18n import gettext_set_language
from invenio.legacy.webuser import emailUnique, get_user_preferences
from invenio.modules.formatter.engines.xslt import format
from invenio.utils.json import json, json_decode_file
import invenio.config
## Stored messages for email notifications
global _BATCH_STEP, _BATCH_STEPS
_BATCH_STEP = 1
_BATCH_STEPS = 1
global _MSG_HISTORY, _UPD_HISTORY
_MSG_HISTORY = []
_UPD_HISTORY = []
def _notify_error_admin(batch_job,
email_admin=invenio.config.CFG_SITE_ADMIN_EMAIL):
"""Sends a notification email to the specified address, containing
admin-only information. Is called by process_batch_job() if an error
occured during the processing.
@param email_admin: email address of the admin
@type email_admin: string
"""
if not email_admin:
return
template = ("BibEncode batch processing has reported an error during the"
"execution of a job within the batch description <br/><br/>"
"This is the batch description: <br/><br/>"
"%(batch_description)s <br/><br/>"
"This is the message log: <br/><br/>"
"%(message_log)s")
html_text = template % {"batch_description": pformat(batch_job).replace("\n", "<br/>"),
"message_log": "\n".join(_MSG_HISTORY)}
text = html_text.replace("<br/>", "\n")
send_email(fromaddr=invenio.config.CFG_SITE_ADMIN_EMAIL,
toaddr=email_admin,
subject="Error during BibEncode batch processing",
content=text,
html_content=html_text)
def _notify_error_user(email_user, original_filename, recid, submission_title, ln=invenio.config.CFG_SITE_LANG):
"""Sends an error notification to the specified addres of the user.
Is called by process_batch_job() if an error occured during the processing.
@param email_user: email address of the user
@type email_user: string
@param email_admin: email address of the admin
@type email_admin: string
"""
if not email_user:
return
uid = emailUnique(email_user)
if uid != -1 and uid != 0:
language = getval(get_user_preferences(uid), "language")
if language:
ln = language
_ = gettext_set_language(ln)
rec_url = invenio.config.CFG_SITE_URL + "/record/" + str(recid)
template = ("<br/>" +
_("We are sorry, a problem has occured during the processing of"
" your video upload%(submission_title)s.") +
"<br/><br/>" +
_("The file you uploaded was %(input_filename)s.") +
"<br/><br/>" +
_("Your video might not be fully available until intervention.") +
"<br/>" +
_("You can check the status of your video here: %(record_url)s.") +
"<br/>" +
_("You might want to take a look at "
" %(guidelines_url)s"
" and modify or redo your submission."))
text = template % {"input_filename": "%s" % original_filename,
"submission_title": " %s" % submission_title,
"record_url": "%s" % rec_url,
"guidelines_url": "localhost"}
text = text.replace("<br/>", "\n")
html_text = template % {"input_filename": "<strong>%s</strong>" % original_filename,
"submission_title": " <strong>%s</strong>" % submission_title,
"record_url": "<a href=\"%s\">%s</a>" % (rec_url, rec_url),
"guidelines_url": "<a href=\"locahost\">%s</a>" % _("the video guidelines")}
send_email(fromaddr=invenio.config.CFG_SITE_ADMIN_EMAIL,
toaddr=email_user,
subject="Problem during the processing of your video",
content=text,
html_content=html_text
)
def _notify_success_user(email_user, original_filename, recid, submission_title, ln=invenio.config.CFG_SITE_LANG):
"""Sends an success notification to the specified addres of the user.
Is called by process_batch_job() if the processing was successfull.
@param email_user: email address of the user
@type email_user: string
@param email_admin: email address of the admin
@type email_admin: string
"""
uid = emailUnique(email_user)
if uid != -1 and uid != 0:
language = getval(get_user_preferences(uid), "language")
if language:
ln = language
_ = gettext_set_language(ln)
rec_url = invenio.config.CFG_SITE_URL + "/record/" + str(recid)
template = ("<br/>" +
_("Your video submission%(submission_title)s was successfully processed.") +
"<br/><br/>" +
_("The file you uploaded was %(input_filename)s.") +
"<br/><br/>" +
_("Your video is now available here: %(record_url)s.") +
"<br/>" +
_("If the videos quality is not as expected, you might want to take "
"a look at %(guidelines_url)s"
" and modify or redo your submission."))
text = template % {"input_filename": "%s" % original_filename,
"submission_title": " %s" % submission_title,
"record_url": "%s" % rec_url,
"guidelines_url": "localhost"}
text = text.replace("<br/>", "\n")
html_text = template % {"input_filename": "<strong>%s</strong>" % original_filename,
"submission_title": " <strong>%s</strong>" % submission_title,
"record_url": "<a href=\"%s\">%s</a>" % (rec_url, rec_url),
"guidelines_url": "<a href=\"locahost\">%s</a>" % _("the video guidelines")}
send_email(fromaddr=invenio.config.CFG_SITE_ADMIN_EMAIL,
toaddr=email_user,
subject="Your video submission is now complete",
content=text,
html_content=html_text
)
def _task_update_overall_status(message):
""" Generates an overall update message for the BibEncode task.
Stores the messages in a global list for notifications
@param message: the message that should be printed as task status
@type message: string
"""
message = "[%d/%d]%s" % (_BATCH_STEP, _BATCH_STEPS, message)
task_update_progress(message)
global _UPD_HISTORY
_UPD_HISTORY.append(message)
def _task_write_message(message):
""" Stores the messages in a global list for notifications
@param message: the message that should be printed as task status
@type message: string
"""
write_message(message)
global _MSG_HISTORY
_MSG_HISTORY.append(message)
def clean_job_for_quality(batch_job_dict, fallback=True):
"""
Removes jobs from the batch description that are not suitable for the master
video's quality. It applies only for encoding jobs!
@param batch_job_dict: the dict containing the batch description
@type batch_job_dict: dict
@param
@return: the cleaned dict
@rtype: dict
"""
survived_jobs = []
fallback_jobs = []
other_jobs = []
for job in batch_job_dict['jobs']:
if job['mode'] == 'encode':
if getval(job, 'fallback') and fallback:
fallback_jobs.append(job)
if getval(job, 'enforce'):
survived_jobs.append(job)
else:
profile = None
if getval(job, 'profile'):
profile = get_encoding_profile(job['profile'])
if assure_quality(input_file=batch_job_dict['input'],
aspect=chose2('aspect', job, profile),
target_width=chose2('width', job, profile),
target_height=chose2('height', job, profile),
target_bitrate=chose2('videobitrate', job, profile)):
survived_jobs.append(job)
else:
other_jobs.append(job)
if survived_jobs:
survived_jobs.extend(other_jobs)
new_jobs = survived_jobs
else:
fallback_jobs.extend(other_jobs)
new_jobs = fallback_jobs
pprint(locals())
batch_job_dict['jobs'] = new_jobs
return batch_job_dict
def create_update_jobs_by_collection(
batch_template_file,
collection,
job_directory=CFG_BIBENCODE_DAEMON_DIR_NEWJOBS):
""" Creates the job description files to update a whole collection
@param batch_template_file: fullpath to the template for the update
@type batch_tempalte_file: string
@param collection: name of the collection that should be updated
@type collection: string
@param job_directory: fullpath to the directory storing the job files
@type job_directory: string
"""
recids = get_collection_reclist(collection)
return create_update_jobs_by_recids(recids, batch_template_file,
job_directory)
def create_update_jobs_by_search(pattern,
batch_template_file,
job_directory=CFG_BIBENCODE_DAEMON_DIR_NEWJOBS
):
""" Creates the job description files to update all records that fit a
search pattern. Be aware of the search limitations!
@param search_pattern: The pattern to search for
@type search_pattern: string
@param batch_template_file: fullpath to the template for the update
@type batch_tempalte_file: string
@param job_directory: fullpath to the directory storing the job files
@type job_directory: string
"""
recids = search_pattern(p=pattern)
return create_update_jobs_by_recids(recids, batch_template_file,
job_directory)
def create_update_jobs_by_recids(recids,
batch_template_file,
job_directory=CFG_BIBENCODE_DAEMON_DIR_NEWJOBS
):
""" Creates the job description files to update all given recids
@param recids: Iterable set of recids
@type recids: iterable
@param batch_template_file: fullpath to the template for the update
@type batch_tempalte_file: string
@param job_directory: fullpath to the directory storing the job files
@type job_directory: string
"""
batch_template = json_decode_file(batch_template_file)
for recid in recids:
task_update_progress("Creating Update Job for %d" % recid)
write_message("Creating Update Job for %d" % recid)
job = dict(batch_template)
job['recid'] = recid
timestamp = generate_timestamp()
job_filename = "update_%d_%s.job" % (recid, timestamp)
create_job_from_dictionary(job, job_filename, job_directory)
return 1
def create_job_from_dictionary(
job_dict,
job_filename=None,
job_directory=CFG_BIBENCODE_DAEMON_DIR_NEWJOBS
):
""" Creates a job from a given dictionary
@param job_dict: Dictionary that contains the job description
@type job_dict: job_dict
@param job_filename: Filename for the job
@type job_filename: string
@param job_directory: fullpath to the directory storing the job files
@type job_directory: string
"""
if not job_filename:
job_filename = str(uuid.uuid4())
if not job_filename.endswith(".job"):
job_filename += ".job"
job_fullpath = os.path.join(job_directory, job_filename)
job_string = json.dumps(job_dict, sort_keys=False, indent=4)
file = open(job_fullpath, "w")
file.write(job_string)
file.close()
def sanitise_batch_job(batch_job):
""" Checks the correctness of the batch job dictionary and additionally
sanitises some values.
@param batch_job: The batch description dictionary
@type batch_job: dictionary
"""
def san_bitrate(bitrate):
""" Sanitizes bitrates
"""
if type(str()) == type(bitrate):
if bitrate.endswith('k'):
try:
bitrate = int(bitrate[:-1])
bitrate *= 1000
return bitrate
except ValueError:
raise Exception("Could not parse bitrate")
elif type(int) == type(bitrate):
return bitrate
else:
raise Exception("Could not parse bitrate")
if not getval(batch_job, 'update_from_master'):
if not getval(batch_job, 'input'):
raise Exception("No input file in batch description")
if not getval(batch_job, 'recid'):
raise Exception("No recid in batch description")
if not getval(batch_job, 'jobs'):
raise Exception("No job list in batch description")
if getval(batch_job, 'update_from_master'):
if (not getval(batch_job, 'bibdoc_master_comment') and
not getval(batch_job, 'bibdoc_master_description') and
not getval(batch_job, 'bibdoc_master_subformat')):
raise Exception("If update_from_master ist set, a comment or"
" description or subformat for matching must be given")
if getval(batch_job, 'marc_snippet'):
if not os.path.exists(getval(batch_job, 'marc_snippet')):
raise Exception("The marc snipped file %s was not found" %
getval(batch_job, 'marc_snippet'))
for job in batch_job['jobs']:
if job['mode'] == 'encode':
if getval(job, 'videobitrate'):
job['videobitrate'] = san_bitrate(getval(job, 'videobitrate'))
if getval(job, 'audiobitrate'):
job['audiobitrate'] = san_bitrate(getval(job, 'audiobitrate'))
return batch_job
def process_batch_job(batch_job_file):
""" Processes a batch job description dictionary
@param batch_job_file: a fullpath to a batch job file
@type batch_job_file: string
@return: 1 if the process was successfull, 0 if not
@rtype; int
"""
def upload_marcxml_file(marcxml):
""" Creates a temporary marcxml file and sends it to bibupload
"""
xml_filename = 'bibencode_'+ str(batch_job['recid']) + '_' + str(uuid.uuid4()) + '.xml'
xml_filename = os.path.join(invenio.config.CFG_TMPSHAREDDIR, xml_filename)
xml_file = file(xml_filename, 'w')
xml_file.write(marcxml)
xml_file.close()
targs = ['-c', xml_filename]
task_low_level_submission('bibupload', 'bibencode', *targs)
#---------#
# GENERAL #
#---------#
_task_write_message("----------- Handling Master -----------")
## Check the validity of the batch file here
batch_job = json_decode_file(batch_job_file)
## Sanitise batch description and raise errrors
batch_job = sanitise_batch_job(batch_job)
## Check if the record exists
if record_exists(batch_job['recid']) < 1:
raise Exception("Record not found")
recdoc = BibRecDocs(batch_job['recid'])
#--------------------#
# UPDATE FROM MASTER #
#--------------------#
## We want to add new stuff to the video's record, using the master as input
if getval(batch_job, 'update_from_master'):
found_master = False
bibdocs = recdoc.list_bibdocs()
for bibdoc in bibdocs:
bibdocfiles = bibdoc.list_all_files()
for bibdocfile in bibdocfiles:
comment = bibdocfile.get_comment()
description = bibdocfile.get_description()
subformat = bibdocfile.get_subformat()
m_comment = getval(batch_job, 'bibdoc_master_comment', comment)
m_description = getval(batch_job, 'bibdoc_master_description', description)
m_subformat = getval(batch_job, 'bibdoc_master_subformat', subformat)
if (comment == m_comment and
description == m_description and
subformat == m_subformat):
found_master = True
batch_job['input'] = bibdocfile.get_full_path()
## Get the aspect of the from the record
try:
## Assumes pbcore metadata mapping
batch_job['aspect'] = get_fieldvalues(124, CFG_BIBENCODE_ASPECT_RATIO_MARC_FIELD)[0]
except IndexError:
pass
break
if found_master:
break
if not found_master:
_task_write_message("Video master for record %d not found"
% batch_job['recid'])
task_update_progress("Video master for record %d not found"
% batch_job['recid'])
## Maybe send an email?
return 1
## Clean the job to do no upscaling etc
if getval(batch_job, 'assure_quality'):
batch_job = clean_job_for_quality(batch_job)
global _BATCH_STEPS
_BATCH_STEPS = len(batch_job['jobs'])
## Generate the docname from the input filename's name or given name
bibdoc_video_docname, bibdoc_video_extension = decompose_file(batch_job['input'])[1:]
if not bibdoc_video_extension or getval(batch_job, 'bibdoc_master_extension'):
bibdoc_video_extension = getval(batch_job, 'bibdoc_master_extension')
if getval(batch_job, 'bibdoc_master_docname'):
bibdoc_video_docname = getval(batch_job, 'bibdoc_master_docname')
write_message("Creating BibDoc for %s" % bibdoc_video_docname)
## If the bibdoc exists, receive it
if bibdoc_video_docname in recdoc.get_bibdoc_names():
bibdoc_video = recdoc.get_bibdoc(bibdoc_video_docname)
## Create a new bibdoc if it does not exist
else:
bibdoc_video = recdoc.add_bibdoc(docname=bibdoc_video_docname)
## Get the directory auf the newly created bibdoc to copy stuff there
bibdoc_video_directory = bibdoc_video.get_base_dir()
#--------#
# MASTER #
#--------#
if not getval(batch_job, 'update_from_master'):
if getval(batch_job, 'add_master'):
## Generate the right name for the master
## The master should be hidden first an then renamed
## when it is really available
## !!! FIX !!!
_task_write_message("Adding %s master to the BibDoc"
% bibdoc_video_docname)
master_format = compose_format(
bibdoc_video_extension,
getval(batch_job, 'bibdoc_master_subformat', 'master')
)
## If a file of the same format is there, something is wrong, remove it!
## it might be caused by a previous corrupted submission etc.
if bibdoc_video.format_already_exists_p(master_format):
bibdoc_video.delete_file(master_format, 1)
bibdoc_video.add_file_new_format(
batch_job['input'],
version=1,
description=getval(batch_job, 'bibdoc_master_description'),
comment=getval(batch_job, 'bibdoc_master_comment'),
docformat=master_format
)
#-----------#
# JOBS LOOP #
#-----------#
return_code = 1
global _BATCH_STEP
for job in batch_job['jobs']:
_task_write_message("----------- Job %s of %s -----------"
% (_BATCH_STEP, _BATCH_STEPS))
## Try to substitute docname with master docname
if getval(job, 'bibdoc_docname'):
job['bibdoc_docname'] = Template(job['bibdoc_docname']).safe_substitute({'bibdoc_master_docname': bibdoc_video_docname})
#-------------#
# TRANSCODING #
#-------------#
if job['mode'] == 'encode':
## Skip the job if assure_quality is not set and marked as fallback
if not getval(batch_job, 'assure_quality') and getval(job, 'fallback'):
continue
if getval(job, 'profile'):
profile = get_encoding_profile(job['profile'])
else:
profile = None
## We need an extension defined fot the video container
bibdoc_video_extension = getval(job, 'extension',
getval(profile, 'extension'))
if not bibdoc_video_extension:
raise Exception("No container/extension defined")
## Get the docname and subformat
bibdoc_video_subformat = getval(job, 'bibdoc_subformat')
bibdoc_slave_video_docname = getval(job, 'bibdoc_docname', bibdoc_video_docname)
## The subformat is incompatible with ffmpegs name convention
## We do the encoding without and rename it afterwards
bibdoc_video_fullpath = compose_file(
bibdoc_video_directory,
bibdoc_slave_video_docname,
bibdoc_video_extension
)
_task_write_message("Transcoding %s to %s;%s" % (bibdoc_slave_video_docname,
bibdoc_video_extension,
bibdoc_video_subformat))
## We encode now directly into the bibdocs directory
encoding_result = encode_video(
input_file=batch_job['input'],
output_file=bibdoc_video_fullpath,
acodec=getval(job, 'audiocodec'),
vcodec=getval(job, 'videocodec'),
abitrate=getval(job, 'videobitrate'),
vbitrate=getval(job, 'audiobitrate'),
resolution=getval(job, 'resolution'),
passes=getval(job, 'passes', 1),
special=getval(job, 'special'),
specialfirst=getval(job, 'specialfirst'),
specialsecond=getval(job, 'specialsecond'),
metadata=getval(job, 'metadata'),
width=getval(job, 'width'),
height=getval(job, 'height'),
aspect=getval(batch_job, 'aspect'), # Aspect for every job
profile=getval(job, 'profile'),
update_fnc=_task_update_overall_status,
message_fnc=_task_write_message
)
return_code &= encoding_result
## only on success
if encoding_result:
## Rename it, adding the subformat
os.rename(bibdoc_video_fullpath,
compose_file(bibdoc_video_directory,
bibdoc_video_extension,
bibdoc_video_subformat,
1,
bibdoc_slave_video_docname)
)
- bibdoc_video._build_file_list()
+ #bibdoc_video._build_file_list()
+ bibdoc_video.touch()
+ bibdoc_video._sync_to_db()
bibdoc_video_format = compose_format(bibdoc_video_extension,
bibdoc_video_subformat)
if getval(job, 'bibdoc_comment'):
bibdoc_video.set_comment(getval(job, 'bibdoc_comment'),
bibdoc_video_format)
if getval(job, 'bibdoc_description'):
bibdoc_video.set_description(getval(job, 'bibdoc_description'),
bibdoc_video_format)
#------------#
# EXTRACTION #
#------------#
# if there are multiple extraction jobs, all the produced files
# with the same name will be in the same bibdoc! Make sure that
# you use different subformats or docname templates to avoid
# conflicts.
if job['mode'] == 'extract':
if getval(job, 'profile'):
profile = get_extract_profile(job['profile'])
else:
profile = {}
bibdoc_frame_subformat = getval(job, 'bibdoc_subformat')
_task_write_message("Extracting frames to temporary directory")
tmpdir = invenio.config.CFG_TMPDIR + "/" + str(uuid.uuid4())
os.mkdir(tmpdir)
#Move this to the batch description
bibdoc_frame_docname = getval(job, 'bibdoc_docname', bibdoc_video_docname)
tmpfname = (tmpdir + "/" + bibdoc_frame_docname + '.'
+ getval(profile, 'extension',
getval(job, 'extension', 'jpg')))
extraction_result = extract_frames(input_file=batch_job['input'],
output_file=tmpfname,
size=getval(job, 'size'),
positions=getval(job, 'positions'),
numberof=getval(job, 'numberof'),
width=getval(job, 'width'),
height=getval(job, 'height'),
aspect=getval(batch_job, 'aspect'),
profile=getval(job, 'profile'),
update_fnc=_task_update_overall_status,
)
return_code &= extraction_result
## only on success:
if extraction_result:
## for every filename in the directorys, create a bibdoc that contains
## all sizes of the frame from the two directories
files = os.listdir(tmpdir)
for filename in files:
## The docname was altered by BibEncode extract through substitution
## Retrieve it from the filename again
bibdoc_frame_docname, bibdoc_frame_extension = os.path.splitext(filename)
_task_write_message("Creating new bibdoc for %s" % bibdoc_frame_docname)
## If the bibdoc exists, receive it
if bibdoc_frame_docname in recdoc.get_bibdoc_names():
bibdoc_frame = recdoc.get_bibdoc(bibdoc_frame_docname)
## Create a new bibdoc if it does not exist
else:
bibdoc_frame = recdoc.add_bibdoc(docname=bibdoc_frame_docname)
## The filename including path from tmpdir
fname = os.path.join(tmpdir, filename)
bibdoc_frame_format = compose_format(bibdoc_frame_extension, bibdoc_frame_subformat)
## Same as with the master, if the format allready exists,
## override it, because something went wrong before
if bibdoc_frame.format_already_exists_p(bibdoc_frame_format):
bibdoc_frame.delete_file(bibdoc_frame_format, 1)
_task_write_message("Adding %s jpg;%s to BibDoc"
% (bibdoc_frame_docname,
getval(job, 'bibdoc_subformat')))
bibdoc_frame.add_file_new_format(
fname,
version=1,
description=getval(job, 'bibdoc_description'),
comment=getval(job, 'bibdoc_comment'),
docformat=bibdoc_frame_format)
## Remove the temporary folders
_task_write_message("Removing temporary directory")
shutil.rmtree(tmpdir)
_BATCH_STEP = _BATCH_STEP + 1
#-----------------#
# FIX BIBDOC/MARC #
#-----------------#
_task_write_message("----------- Handling MARCXML -----------")
## Fix the BibDoc for all the videos previously created
_task_write_message("Updating BibDoc of %s" % bibdoc_video_docname)
bibdoc_video._build_file_list()
## Fix the MARC
_task_write_message("Fixing MARC")
cli_fix_marc({}, [batch_job['recid']], False)
if getval(batch_job, 'collection'):
## Make the record visible by moving in from the collection
marcxml = ("<record><controlfield tag=\"001\">%d</controlfield>"
"<datafield tag=\"980\" ind1=\" \" ind2=\" \">"
"<subfield code=\"a\">%s</subfield></datafield></record>"
) % (batch_job['recid'], batch_job['collection'])
upload_marcxml_file(marcxml)
#---------------------#
# ADD MASTER METADATA #
#---------------------#
if getval(batch_job, 'add_master_metadata'):
_task_write_message("Adding master metadata")
pbcore = pbcore_metadata(input_file = getval(batch_job, 'input'),
pbcoreIdentifier = batch_job['recid'],
aspect_override = getval(batch_job, 'aspect'))
marcxml = format(pbcore, CFG_BIBENCODE_PBCORE_MARC_XSLT)
upload_marcxml_file(marcxml)
#------------------#
# ADD MARC SNIPPET #
#------------------#
if getval(batch_job, 'marc_snippet'):
marc_snippet = open(getval(batch_job, 'marc_snippet'))
marcxml = marc_snippet.read()
marc_snippet.close()
upload_marcxml_file(marcxml)
#--------------#
# DELETE INPUT #
#--------------#
if getval(batch_job, 'delete_input'):
_task_write_message("Deleting input file")
# only if successfull
if not return_code:
# only if input matches pattern
if getval(batch_job, 'delete_input_pattern', '') in getval(batch_job, 'input'):
try:
os.remove(getval(batch_job, 'input'))
except OSError:
pass
#--------------#
# NOTIFICATION #
#--------------#
## Send Notification emails on errors
if not return_code:
if getval(batch_job, 'notify_user'):
_notify_error_user(getval(batch_job, 'notify_user'),
getval(batch_job, 'submission_filename', batch_job['input']),
getval(batch_job, 'recid'),
getval(batch_job, 'submission_title', ""))
_task_write_message("Notify user because of an error")
if getval(batch_job, 'notify_admin'):
_task_write_message("Notify admin because of an error")
if type(getval(batch_job, 'notify_admin') == type(str()) ):
_notify_error_admin(batch_job,
getval(batch_job, 'notify_admin'))
else:
_notify_error_admin(batch_job)
else:
if getval(batch_job, 'notify_user'):
_task_write_message("Notify user because of success")
_notify_success_user(getval(batch_job, 'notify_user'),
getval(batch_job, 'submission_filename', batch_job['input']),
getval(batch_job, 'recid'),
getval(batch_job, 'submission_title', ""))
return 1
diff --git a/invenio/modules/exporter/exporterext/configurations/googlescholar.cfg b/invenio/modules/exporter/exporterext/configurations/googlescholar.cfg
index 06aac345f..19ae609be 100644
--- a/invenio/modules/exporter/exporterext/configurations/googlescholar.cfg
+++ b/invenio/modules/exporter/exporterext/configurations/googlescholar.cfg
@@ -1,2 +1,5 @@
[export_job]
export_method = googlescholar
+collection1 = Articles
+collection2 = Theses
+collection3 = Preprints
\ No newline at end of file
diff --git a/invenio/modules/formatter/engine.py b/invenio/modules/formatter/engine.py
index 29c8750ff..449f06d83 100644
--- a/invenio/modules/formatter/engine.py
+++ b/invenio/modules/formatter/engine.py
@@ -1,2388 +1,2388 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
Formats a single XML Marc record using specified format.
There is no API for the engine. Instead use module L{bibformat}.
You can have a look at the various escaping modes available in
X{BibFormatObject} in function L{escape_field}
Still it is useful sometimes for debugging purpose to use the
L{BibFormatObject} class directly. For eg:
>>> from invenio.modules.formatter.engine import BibFormatObject
>>> bfo = BibFormatObject(102)
>>> bfo.field('245__a')
The order Rodentia in South America
>>> from invenio.modules.formatter.format_elements import bfe_title
>>> bfe_title.format_element(bfo)
The order Rodentia in South America
@see: bibformat.py, bibformat_utils.py
"""
__revision__ = "$Id$"
import re
import sys
import os
import inspect
import traceback
import cgi
import types
from flask import has_app_context, current_app
from operator import itemgetter
from six import iteritems, reraise
from werkzeug.utils import cached_property
from invenio.base.globals import cfg
from invenio.base.utils import (autodiscover_template_context_functions,
autodiscover_format_elements)
from invenio.config import \
CFG_SITE_LANG, \
CFG_BIBFORMAT_CACHED_FORMATS, \
CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS, \
CFG_BIBFORMAT_HIDDEN_TAGS
CFG_PATH_PHP, \
CFG_BINDIR, \
CFG_SITE_LANG
from invenio.ext.logging import \
register_exception
from invenio.legacy.bibrecord import \
create_record, \
record_get_field_instances, \
record_get_field_value, \
record_get_field_values, \
record_xml_output, \
record_empty
from . import registry
from .engines import xslt
from invenio.legacy.dbquery import run_sql
from invenio.base.i18n import \
language_list_long, \
wash_language, \
gettext_set_language
from . import api as bibformat_dblayer
from .config import \
CFG_BIBFORMAT_TEMPLATES_DIR, \
CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION, \
CFG_BIBFORMAT_FORMAT_JINJA_TEMPLATE_EXTENSION, \
CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION, \
CFG_BIBFORMAT_OUTPUTS_PATH, \
InvenioBibFormatError
from invenio.modules.formatter.utils import \
record_get_xml, \
parse_tag
from invenio.utils.html import \
HTMLWasher, \
CFG_HTML_BUFFER_ALLOWED_TAG_WHITELIST, \
CFG_HTML_BUFFER_ALLOWED_ATTRIBUTE_WHITELIST
from invenio.modules.knowledge.api import get_kbr_values
from invenio.ext.template import render_template_to_string
from HTMLParser import HTMLParseError
from invenio.modules.access.engine import acc_authorize_action
# Cache for data we have already read and parsed
format_templates_cache = {}
format_elements_cache = {}
format_outputs_cache = {}
html_field = '<!--HTML-->' # String indicating that field should be
# treated as HTML (and therefore no escaping of
# HTML tags should occur.
# Appears in some field values.
washer = HTMLWasher() # Used to remove dangerous tags from HTML
# sources
# Regular expression for finding <lang>...</lang> tag in format templates
pattern_lang = re.compile(r'''
<lang #<lang tag (no matter case)
\s* #any number of white spaces
> #closing <lang> start tag
(?P<langs>.*?) #anything but the next group (greedy)
(</lang\s*>) #end tag
''', re.IGNORECASE | re.DOTALL | re.VERBOSE)
# Builds regular expression for finding each known language in <lang> tags
ln_pattern_text = r"<("
for lang in language_list_long(enabled_langs_only=False):
ln_pattern_text += lang[0] +r"|"
ln_pattern_text = ln_pattern_text.rstrip(r"|")
ln_pattern_text += r")>(.*?)</\1>"
ln_pattern = re.compile(ln_pattern_text, re.IGNORECASE | re.DOTALL)
# Regular expression for finding text to be translated
TRANSLATION_PATTERN = re.compile(r'_\((?P<word>.*?)\)_',
re.IGNORECASE | re.DOTALL | re.VERBOSE)
# Regular expression for finding <name> tag in format templates
pattern_format_template_name = re.compile(r'''
<name #<name tag (no matter case)
\s* #any number of white spaces
> #closing <name> start tag
(?P<name>.*?) #name value. any char that is not end tag
(</name\s*>)(\n)? #end tag
''', re.IGNORECASE | re.DOTALL | re.VERBOSE)
# Regular expression for finding <description> tag in format templates
pattern_format_template_desc = re.compile(r'''
<description #<decription tag (no matter case)
\s* #any number of white spaces
> #closing <description> start tag
(?P<desc>.*?) #description value. any char that is not end tag
</description\s*>(\n)? #end tag
''', re.IGNORECASE | re.DOTALL | re.VERBOSE)
# Regular expression for finding <BFE_ > tags in format templates
pattern_tag = re.compile(r'''
<BFE_ #every special tag starts with <BFE_ (no matter case)
(?P<function_name>[^/\s]+) #any char but a space or slash
\s* #any number of spaces
(?P<params>(\s* #params here
(?P<param>([^=\s])*)\s* #param name: any chars that is not a white space or equality. Followed by space(s)
=\s* #equality: = followed by any number of spaces
(?P<sep>[\'"]) #one of the separators
(?P<value>.*?) #param value: any chars that is not a separator like previous one
(?P=sep) #same separator as starting one
)*) #many params
\s* #any number of spaces
(/)?> #end of the tag
''', re.IGNORECASE | re.DOTALL | re.VERBOSE)
# Regular expression for finding params inside <BFE_ > tags in format templates
pattern_function_params = re.compile(r'''
(?P<param>([^=\s])*)\s* # Param name: any chars that is not a white space or equality. Followed by space(s)
=\s* # Equality: = followed by any number of spaces
(?P<sep>[\'"]) # One of the separators
(?P<value>.*?) # Param value: any chars that is not a separator like previous one
(?P=sep) # Same separator as starting one
''', re.VERBOSE | re.DOTALL)
# Regular expression for finding format elements "params" attributes
# (defined by @param)
pattern_format_element_params = re.compile(r'''
@param\s* # Begins with AT param keyword followed by space(s)
(?P<name>[^\s=]*):\s* # A single keyword and comma, then space(s)
#(=\s*(?P<sep>[\'"]) # Equality, space(s) and then one of the separators
#(?P<default>.*?) # Default value: any chars that is not a separator like previous one
#(?P=sep) # Same separator as starting one
#)?\s* # Default value for param is optional. Followed by space(s)
(?P<desc>.*) # Any text that is not end of line (thanks to MULTILINE parameter)
''', re.VERBOSE | re.MULTILINE)
# Regular expression for finding format elements "see also" attribute
# (defined by @see)
pattern_format_element_seealso = re.compile(r'''@see:\s*(?P<see>.*)''',
re.VERBOSE | re.MULTILINE)
#Regular expression for finding 2 expressions in quotes, separated by
#comma (as in template("1st","2nd") )
#Used when parsing output formats
## pattern_parse_tuple_in_quotes = re.compile('''
## (?P<sep1>[\'"])
## (?P<val1>.*)
## (?P=sep1)
## \s*,\s*
## (?P<sep2>[\'"])
## (?P<val2>.*)
## (?P=sep2)
## ''', re.VERBOSE | re.MULTILINE)
sub_non_alnum = re.compile('[^0-9a-zA-Z]+')
fix_tag_name = lambda s: sub_non_alnum.sub('_', s.lower())
from invenio.utils.memoise import memoize
class LazyTemplateContextFunctionsCache(object):
"""Loads bibformat elements using plugin builder and caches results."""
@cached_property
def template_context_functions(self):
"""Returns template context functions"""
modules = autodiscover_template_context_functions()
elem = {}
for m in modules:
register_func = getattr(m, 'template_context_function', None)
if register_func and isinstance(register_func, types.FunctionType):
elem[m.__name__.split('.')[-1]] = register_func
return elem
@memoize
def bibformat_elements(self, modules=None):
"""Returns bibformat elements."""
if modules is None:
modules = registry.format_elements
elem = {}
for m in modules:
if m is None:
continue
name = m.__name__.split('.')[-1]
if name in elem:
continue
filename = m.__file__[:-1] if m.__file__.endswith('.pyc') \
else m.__file__
register_func = getattr(m, 'format_element',
getattr(m, 'format', None))
escape_values = getattr(m, 'escape_values', None)
if register_func and isinstance(register_func, types.FunctionType):
register_func._escape_values = escape_values
register_func.__file__ = filename
elem[name] = register_func
return elem
@cached_property
def functions(self):
def insert(name):
def _bfe_element(bfo, **kwargs):
# convert to utf-8 for legacy app
kwargs = dict((k, v.encode('utf-8') if isinstance(v, unicode) else v)
for k, v in iteritems(kwargs))
format_element = get_format_element(name)
(out, dummy) = eval_format_element(format_element,
bfo,
kwargs)
# returns unicode for jinja2
return out.decode('utf-8')
return _bfe_element
# Old bibformat templates
tfn_from_files = dict((name.lower(), insert(name.lower()))
for name in self.bibformat_elements().keys())
# Update with new template context functions
tfn_from_files.update(self.template_context_functions)
bfe_from_tags = {}
if has_app_context():
from invenio.ext.sqlalchemy import db
from invenio.modules.search.models import Tag
# get functions from tag table
bfe_from_tags = dict(('bfe_'+fix_tag_name(name),
insert(fix_tag_name(name)))
for name in map(itemgetter(0),
db.session.query(Tag.name).all()))
# overwrite functions from tag table with functions from files
bfe_from_tags.update(tfn_from_files)
return bfe_from_tags
TEMPLATE_CONTEXT_FUNCTIONS_CACHE = LazyTemplateContextFunctionsCache()
def get_format_element_path(filename):
if filename.endswith('.py'):
filename = filename[:-3]
return TEMPLATE_CONTEXT_FUNCTIONS_CACHE.bibformat_elements()[filename].__file__
def format_record(recID, of, ln=CFG_SITE_LANG, verbose=0,
search_pattern=None, xml_record=None, user_info=None, qid="", extra_context={}):
"""
Formats a record given output format. Main entry function of
bibformat engine.
Returns a formatted version of the record in the specified
language, search pattern, and with the specified output format.
The function will define which format template must be applied.
You can either specify an record ID to format, or give its xml
representation. if 'xml_record' is not None, then use it instead
of recID.
'user_info' allows to grant access to some functionalities on a
page depending on the user's priviledges. 'user_info' is the same
object as the one returned by 'webuser.collect_user_info(req)'
@param recID: the ID of record to format
@param of: an output format code (or short identifier for the output format)
@param ln: the language to use to format the record
@param verbose: the level of verbosity from 0 to 9 (O: silent,
5: errors,
7: errors and warnings, stop if error in format elements
9: errors and warnings, stop if error (debug mode ))
@param search_pattern: list of strings representing the user request in web interface
@param xml_record: an xml string representing the record to format
@param user_info: the information of the user who will view the formatted page
@return: formatted record
"""
if search_pattern is None:
search_pattern = []
out = ""
ln = wash_language(ln)
_ = gettext_set_language(ln)
# Temporary workflow (during migration of formats):
# Call new BibFormat
# But if format not found for new BibFormat, then call old BibFormat
#Create a BibFormat Object to pass that contain record and context
bfo = BibFormatObject(recID, ln, search_pattern, xml_record, user_info, of)
if of.lower() != 'xm' and (not bfo.get_record()
or record_empty(bfo.get_record())):
# Record only has recid: do not format, excepted
# for xm format
return "", False
#Find out which format template to use based on record and output format.
template = decide_format_template(bfo, of)
if verbose == 9 and template is not None:
out += """\n<br/><span class="quicknote">
Using %s template for record %i.
</span>""" % (template, recID)
path = registry.format_templates_lookup.get(template)
if template is None or not (
template.endswith("." + CFG_BIBFORMAT_FORMAT_JINJA_TEMPLATE_EXTENSION)
or path is None or os.access(path, os.R_OK)
):
# template not found in new BibFormat. Call old one
if verbose == 9:
if template is None:
out += """\n<br/><span class="quicknote">
No template found for output format %s and record %i.
(Check invenio.err log file for more details)
</span>""" % (of, recID)
else:
out += """\n<br/><span class="quicknote">
Template %s could not be read.
</span>""" % (template)
try:
raise InvenioBibFormatError(_('No template could be found for output format %(code)s.', code=of))
except InvenioBibFormatError as exc:
register_exception(req=bfo.req)
if verbose > 5:
out += """\n<br/><span class="quicknote">
%s
</span>""" % str(exc)
return out, False
# Format with template
out_, needs_2nd_pass = format_with_format_template(template, bfo, verbose)
out += out_
return out, needs_2nd_pass
def format_record_1st_pass(recID, of, ln=CFG_SITE_LANG, verbose=0,
search_pattern=None, xml_record=None,
user_info=None, on_the_fly=False,
save_missing=True):
"""
Format a record in given output format.
Return a formatted version of the record in the specified
language, search pattern, and with the specified output format.
The function will define which format template must be applied.
The record to be formatted can be specified with its ID (with
'recID' parameter) or given as XML representation (with
'xml_record' parameter). If 'xml_record' is specified 'recID' is
ignored (but should still be given for reference. A dummy recid 0
or -1 could be used).
'user_info' allows to grant access to some functionalities on a
page depending on the user's priviledges. The 'user_info' object
makes sense only in the case of on-the-fly formatting. 'user_info'
is the same object as the one returned by
'webuser.collect_user_info(req)'
@param recID: the ID of record to format.
@type recID: int
@param of: an output format code (or short identifier for the output format)
@type of: string
@param ln: the language to use to format the record
@type ln: string
@param verbose: the level of verbosity from 0 to 9 (O: silent,
5: errors,
7: errors and warnings, stop if error in format elements
9: errors and warnings, stop if error (debug mode ))
@type verbose: int
@param search_pattern: list of strings representing the user request in web interface
@type search_pattern: list(string)
@param xml_record: an xml string represention of the record to format
@type xml_record: string or None
@param user_info: the information of the user who will view the formatted page (if applicable)
@param on_the_fly: if False, try to return an already preformatted version of the record in the database
@type on_the_fly: boolean
@return: formatted record
@rtype: string
"""
from invenio.search_engine import record_exists
if search_pattern is None:
search_pattern = []
out = ""
if verbose == 9:
out += """\n<span class="quicknote">
Formatting record %i with output format %s.
</span>""" % (recID, of)
if not on_the_fly and \
(ln == CFG_SITE_LANG or
of.lower() == 'xm' or
(of.lower() in CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS)) and \
record_exists(recID) != -1:
# Try to fetch preformatted record. Only possible for records
# formatted in CFG_SITE_LANG language (other are never
# stored), or of='xm' which does not depend on language.
# Exceptions are made for output formats defined in
# CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS, which are
# always served from the same cache for any language. Also,
# do not fetch from DB when record has been deleted: we want
# to return an "empty" record in that case
res, needs_2nd_pass = bibformat_dblayer.get_preformatted_record(recID, of)
if res is not None:
# record 'recID' is formatted in 'of', so return it
if verbose == 9:
last_updated = bibformat_dblayer.get_preformatted_record_date(recID, of)
out += """\n<br/><span class="quicknote">
Found preformatted output for record %i (cache updated on %s).
</span><br/>""" % (recID, last_updated)
if of.lower() == 'xm':
res = filter_hidden_fields(res, user_info)
# try to replace language links in pre-cached res, if applicable:
if ln != CFG_SITE_LANG and of.lower() in CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS:
# The following statements try to quickly replace any
# language arguments in URL links. Not an exact
# science, but should work most of the time for most
# of the formats, with not too many false positives.
# We don't have time to parse output much here.
res = res.replace('?ln=' + CFG_SITE_LANG, '?ln=' + ln)
res = res.replace('&ln=' + CFG_SITE_LANG, '&ln=' + ln)
res = res.replace('&amp;ln=' + CFG_SITE_LANG, '&amp;ln=' + ln)
out += res
return out, needs_2nd_pass
else:
if verbose == 9:
out += """\n<br/><span class="quicknote">
No preformatted output found for record %s.
</span>"""% recID
# Live formatting of records in all other cases
if verbose == 9:
out += """\n<br/><span class="quicknote">
Formatting record %i on-the-fly.
</span>""" % recID
try:
out_, needs_2nd_pass = format_record(recID=recID,
of=of,
ln=ln,
verbose=verbose,
search_pattern=search_pattern,
xml_record=xml_record,
user_info=user_info)
out += out_
- if of.lower() == 'xm':
- out = filter_hidden_fields(out, user_info)
+ if of.lower() in ('xm', 'xoaimarc'):
+ out = filter_hidden_fields(out, user_info, force_filtering=of.lower()=='xoaimarc')
# We have spent time computing this format
# We want to save this effort if the format is cached
if save_missing and recID and ln == CFG_SITE_LANG \
and of.lower() in CFG_BIBFORMAT_CACHED_FORMATS and verbose == 0:
bibformat_dblayer.save_preformatted_record(recID,
of,
out,
needs_2nd_pass)
return out, needs_2nd_pass
except Exception, e:
register_exception(prefix="An error occured while formatting record %s in %s" %
(recID, of),
alert_admin=True)
#Failsafe execution mode
import invenio.template
websearch_templates = invenio.template.load('websearch')
if verbose == 9:
out += """\n<br/><span class="quicknote">
An error occured while formatting record %s. (%s)
</span>""" % (recID, str(e))
if of.lower() == 'hd':
if verbose == 9:
out += """\n<br/><span class="quicknote">
Formatting record %i with websearch_templates.tmpl_print_record_detailed.
</span><br/>""" % recID
return out + websearch_templates.tmpl_print_record_detailed(
ln=ln,
recID=recID,
)
if verbose == 9:
out += """\n<br/><span class="quicknote">
Formatting record %i with websearch_templates.tmpl_print_record_brief.
</span><br/>""" % recID
return out + websearch_templates.tmpl_print_record_brief(ln=ln,
recID=recID,
), False
def format_record_2nd_pass(recID, template, ln=CFG_SITE_LANG,
search_pattern=None, xml_record=None,
user_info=None, of=None, verbose=0):
# Create light bfo object
bfo = BibFormatObject(recID, ln, search_pattern, xml_record, user_info, of)
# Translations
template = translate_template(template, ln)
# Format template
r, dummy = format_with_format_template(format_template_filename=None,
format_template_code=template,
bfo=bfo,
verbose=verbose)
return r
def decide_format_template(bfo, of):
"""
Returns the format template name that should be used for formatting
given output format and L{BibFormatObject}.
Look at of rules, and take the first matching one.
If no rule matches, returns None
To match we ignore lettercase and spaces before and after value of
rule and value of record
@param bfo: a L{BibFormatObject}
@param of: the code of the output format to use
@return: name of a format template
"""
output_format = get_output_format(of)
for rule in output_format['rules']:
if rule['field'].startswith('00'):
# Rule uses controlfield
values = [bfo.control_field(rule['field']).strip()] #Remove spaces
else:
# Rule uses datafield
values = bfo.fields(rule['field'])
# loop over multiple occurences, but take the first match
if len(values) > 0:
for value in values:
value = value.strip() #Remove spaces
pattern = rule['value'].strip() #Remove spaces
match_obj = re.match(pattern, value, re.IGNORECASE)
if match_obj is not None and \
match_obj.end() == len(value):
return rule['template']
template = output_format['default']
if template != '':
return template
else:
return None
def translate_template(template, ln=CFG_SITE_LANG):
_ = gettext_set_language(ln)
def translate(match):
"""
Translate matching values
"""
word = match.group("word")
translated_word = _(word)
return translated_word
filtered_template = filter_languages(template, ln)
evaluated_format = TRANSLATION_PATTERN.sub(translate, filtered_template)
return evaluated_format
def format_with_format_template(format_template_filename, bfo,
verbose=0, format_template_code=None, qid="", extra_context={}):
""" Format a record given a
format template.
Returns a formatted version of the record represented by bfo,
in the language specified in bfo, and with the specified format template.
If format_template_code is provided, the template will not be loaded from
format_template_filename (but format_template_filename will still be used to
determine if bft or xsl transformation applies). This allows to preview format
code without having to save file on disk.
@param format_template_filename: the dilename of a format template
@param bfo: the object containing parameters for the current formatting
@param format_template_code: if not empty, use code as template instead of reading format_template_filename (used for previews)
@param verbose: the level of verbosity from 0 to 9 (O: silent,
5: errors,
7: errors and warnings,
9: errors and warnings, stop if error (debug mode ))
@return: formatted text
"""
if format_template_code is not None:
format_content = str(format_template_code)
elif not format_template_filename.endswith("." + CFG_BIBFORMAT_FORMAT_JINJA_TEMPLATE_EXTENSION):
format_content = get_format_template(format_template_filename)['code']
if format_template_filename is None or \
format_template_filename.endswith("." + CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION):
# .bft
evaluated_format, needs_2nd_pass = eval_format_template_elements(
format_content,
bfo,
verbose)
if not needs_2nd_pass:
evaluated_format = translate_template(evaluated_format, bfo.lang)
elif format_template_filename.endswith("." + CFG_BIBFORMAT_FORMAT_JINJA_TEMPLATE_EXTENSION):
evaluated_format = '<!-- empty -->'
#try:
from functools import wraps
from invenio.modules.records.api import \
create_record as new_create_record, \
get_record as new_get_record
from flask.ext.login import current_user
from invenio.base.helpers import unicodifier
def _format_record(recid, of='hb', user_info=current_user, *args, **kwargs):
from invenio.modules.formatter import format_record
return format_record(recid, of, user_info=user_info, *args, **kwargs)
# Fixes unicode problems in Jinja2 templates.
def encode_utf8(f):
@wraps(f)
def wrapper(*args, **kwds):
return unicodifier(f(*args, **kwds))
return wrapper
if bfo.xml_record is None:
record = new_get_record(bfo.recID)
else:
record = new_create_record(bfo.xml_record, master_format='marc')
bfo.recID = bfo.recID if bfo.recID else 0
record.__getitem__ = encode_utf8(record.__getitem__)
record.get = encode_utf8(record.get)
ctx = TEMPLATE_CONTEXT_FUNCTIONS_CACHE.functions
ctx.update(extra_context)
evaluated_format = render_template_to_string(
'format/record/'+format_template_filename,
recid=bfo.recID,
record=record,
format_record=_format_record,
qid=qid,
bfo=bfo, **ctx).encode('utf-8')
else:
#.xsl
if bfo.xml_record:
# bfo was initialized with a custom MARCXML
xml_record = '<?xml version="1.0" encoding="UTF-8"?>\n' + \
record_xml_output(bfo.record)
else:
# Fetch MARCXML. On-the-fly xm if we are now formatting in xm
xml_record = '<?xml version="1.0" encoding="UTF-8"?>\n' + \
record_get_xml(bfo.recID, 'xm', on_the_fly=False)
# Transform MARCXML using stylesheet
evaluated_format = xslt.format(xml_record, template_source=format_content).decode('utf-8')
needs_2nd_pass = False
return evaluated_format, needs_2nd_pass
def eval_format_template_elements(format_template, bfo, verbose=0):
"""
Evalutes the format elements of the given template and replace each element with its value.
Prepare the format template content so that we can directly replace the marc code by their value.
This implies:
1. Look for special tags
2. replace special tags by their evaluation
@param format_template: the format template code
@param bfo: the object containing parameters for the current formatting
@param verbose: the level of verbosity from 0 to 9 (O: silent,
5: errors, 7: errors and warnings,
9: errors and warnings, stop if error (debug mode ))
@return: tuple (result, errors)
"""
_ = gettext_set_language(bfo.lang)
status = {'no_cache': False}
# First define insert_element_code(match), used in re.sub() function
def insert_element_code(match):
"""
Analyses 'match', interpret the corresponding code, and return the result of the evaluation.
Called by substitution in 'eval_format_template_elements(...)'
@param match: a match object corresponding to the special tag that must be interpreted
"""
function_name = match.group("function_name")
# Ignore lang tags the processing is done outside
if function_name == 'lang':
return match.group(0)
try:
format_element = get_format_element(function_name, verbose)
except Exception as e:
register_exception(req=bfo.req)
format_element = None
if verbose >= 5:
return '<b><span style="color: rgb(255, 0, 0);">' + \
cgi.escape(str(e)).replace('\n', '<br/>') + \
'</span>'
if format_element is None:
try:
raise InvenioBibFormatError(
_('Could not find format element named %(function_name)s.',
function_name=function_name))
except InvenioBibFormatError as exc:
register_exception(req=bfo.req)
if verbose >= 5:
return '<b><span style="color: rgb(255, 0, 0);">' + \
str(exc.message)+'</span></b>'
else:
params = {}
# Look for function parameters given in format template code
all_params = match.group('params')
if all_params is not None:
function_params_iterator = pattern_function_params.finditer(all_params)
for param_match in function_params_iterator:
name = param_match.group('param')
value = param_match.group('value')
params[name] = value
if params.get('no_cache') == '1':
result = match.group("function_name")
del params['no_cache']
if params:
params_str = ' '.join('%s="%s"' % (k, v) for k, v in params.iteritems())
result = "<bfe_%s %s />" % (result, params_str)
else:
result = "<bfe_%s />" % result
status['no_cache'] = True
else:
# Evaluate element with params and return (Do not return errors)
result, dummy = eval_format_element(format_element,
bfo,
params,
verbose)
return result
# Substitute special tags in the format by our own text.
# Special tags have the form <BNE_format_element_name [param="value"]* />
fmt = pattern_tag.sub(insert_element_code, format_template)
return fmt, status['no_cache']
def eval_format_element(format_element, bfo, parameters=None, verbose=0):
"""
Returns the result of the evaluation of the given format element
name, with given L{BibFormatObject} and parameters. Also returns
the errors of the evaluation.
@param format_element: a format element structure as returned by get_format_element
@param bfo: a L{BibFormatObject} used for formatting
@param parameters: a dict of parameters to be used for formatting. Key is parameter and value is value of parameter
@param verbose: the level of verbosity from 0 to 9 (O: silent,
5: errors,
7: errors and warnings,
9: errors and warnings, stop if error (debug mode ))
@return: tuple (result, errors)
"""
if parameters is None:
parameters = {}
errors = []
#Load special values given as parameters
prefix = parameters.get('prefix', "")
suffix = parameters.get('suffix', "")
default_value = parameters.get('default', "")
escape = parameters.get('escape', "")
output_text = ''
_ = gettext_set_language(bfo.lang)
# 3 possible cases:
# a) format element file is found: we execute it
# b) format element file is not found, but exist in tag table (e.g. bfe_isbn)
# c) format element is totally unknown. Do nothing or report error
if format_element is not None and format_element['type'] == "python":
# a) We found an element with the tag name, of type "python"
# Prepare a dict 'params' to pass as parameter to 'format'
# function of element
params = {}
# Look for parameters defined in format element
# Fill them with specified default values and values
# given as parameters.
# Also remember if the element overrides the 'escape'
# parameter
format_element_overrides_escape = False
for param in format_element['attrs']['params']:
name = param['name']
default = param['default']
params[name] = parameters.get(name, default)
if name == 'escape':
format_element_overrides_escape = True
# Add BibFormatObject
params['bfo'] = bfo
# Execute function with given parameters and return result.
function = format_element['code']
_ = gettext_set_language(bfo.lang)
try:
output_text = function(**params)
except Exception as e:
register_exception(req=bfo.req)
name = format_element['attrs']['name']
try:
raise InvenioBibFormatError(_('Error when evaluating format element %s with parameters %s.') % (name, str(params)))
except InvenioBibFormatError, exc:
errors.append(exc.message)
# Log exception
message = _(
"Error when evaluating format element %(format_element)s with"
" parameters %(parameters)s.",
format_element=name,
parameters=str(params)
)
current_app.logger.exception(
message
)
errors.append(message)
# In debug mode - include traceback in output
if current_app.debug:
tb = sys.exc_info()[2]
stack = traceback.format_exception(
Exception, e, tb, limit=None
)
output_text = '<span class="well"><pre style="color:red;">' \
'%s\n\n%s</pre></span>' % (message, "".join(stack))
# None can be returned when evaluating function
if output_text is None:
output_text = ""
else:
try:
output_text = str(output_text)
except:
output_text = output_text.encode('utf-8')
# Escaping:
# (1) By default, everything is escaped in mode 1
# (2) If evaluated element has 'escape_values()' function, use
# its returned value as escape mode, and override (1)
# (3) If template has a defined parameter 'escape' (in allowed
# values), use it, and override (1) and (2). If this
# 'escape' parameter is overriden by the format element
# (defined in the 'format' function of the element), leave
# the escaping job to this element
# (1)
escape_mode = 1
# (2)
escape_function = format_element['escape_function']
if escape_function is not None:
try:
escape_mode = escape_function(bfo=bfo)
except Exception as e:
try:
raise InvenioBibFormatError(_('Escape mode for format element %(x_name)s could not be retrieved. Using default mode instead.', x_name=name))
except InvenioBibFormatError as exc:
register_exception(req=bfo.req)
errors.append(exc.message)
if verbose >= 5:
tb = sys.exc_info()[2]
output_text += '<b><span style="color: rgb(255, 0, 0);">'+ \
str(exc.message) +'</span></b> '
# (3)
if escape in ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']:
escape_mode = int(escape)
# If escape is equal to 1, then escape all
# HTML reserved chars.
if escape_mode > 0 and not format_element_overrides_escape:
output_text = escape_field(output_text, mode=escape_mode)
# Add prefix and suffix if they have been given as parameters and if
# the evaluation of element is not empty
if output_text.strip() != "":
output_text = prefix + output_text + suffix
# Add the default value if output_text is empty
if output_text == "":
output_text = default_value
return output_text, errors
elif format_element is not None and format_element['type'] == "field":
# b) We have not found an element in files that has the tag
# name. Then look for it in the table "tag"
#
# <BFE_LABEL_IN_TAG prefix = "" suffix = "" separator = ""
# nbMax="" escape="0"/>
#
# Load special values given as parameters
separator = parameters.get('separator ', "")
nbMax = parameters.get('nbMax', "")
escape = parameters.get('escape', "1") # By default, escape here
# Get the fields tags that have to be printed
tags = format_element['attrs']['tags']
output_text = []
# Get values corresponding to tags
for tag in tags:
p_tag = parse_tag(tag)
values = record_get_field_values(bfo.get_record(),
p_tag[0],
p_tag[1],
p_tag[2],
p_tag[3])
if len(values)>0 and isinstance(values[0], dict):
#flatten dict to its values only
values_list = [x.values() for x in values]
#output_text.extend(values)
for values in values_list:
output_text.extend(values)
else:
output_text.extend(values)
if nbMax != "":
try:
nbMax = int(nbMax)
except ValueError:
name = format_element['attrs']['name']
try:
raise InvenioBibFormatError(_('"nbMax" parameter for %(x_param)s must be an "int".', x_param=name))
except InvenioBibFormatError as exc:
register_exception(req=bfo.req)
errors.append(exc.message)
if verbose >= 5:
output_text = output_text.append(exc.message)
else:
output_text = output_text[:nbMax]
# Add prefix and suffix if they have been given as parameters and if
# the evaluation of element is not empty.
# If evaluation is empty string, return default value if it exists.
# Else return empty string
if ("".join(output_text)).strip() != "":
# If escape is equal to 1, then escape all
# HTML reserved chars.
if escape == '1':
output_text = cgi.escape(separator.join(output_text))
else:
output_text = separator.join(output_text)
output_text = prefix + output_text + suffix
else:
#Return default value
output_text = default_value
return (output_text, errors)
else:
# c) Element is unknown
try:
raise InvenioBibFormatError(_('Could not find format element named %(format_element)s.', format_element=format_element))
except InvenioBibFormatError as exc:
register_exception(req=bfo.req)
errors.append(exc.message)
if verbose < 5:
return ("", errors)
elif verbose >= 5:
if verbose >= 9:
sys.exit(exc.message)
return ('<b><span style="color: rgb(255, 0, 0);">' +
str(exc.message)+'</span></b>', errors)
def filter_languages(format_template, ln=CFG_SITE_LANG):
"""
Filters the language tags that do not correspond to the specified language.
@param format_template: the format template code
@param ln: the language that is NOT filtered out from the template
@return: the format template with unnecessary languages filtered out
"""
# First define search_lang_tag(match) and clean_language_tag(match), used
# in re.sub() function
def search_lang_tag(match):
"""
Searches for the <lang>...</lang> tag and remove inner localized tags
such as <en>, <fr>, that are not current_lang.
If current_lang cannot be found inside <lang> ... </lang>, try to use 'CFG_SITE_LANG'
@param match: a match object corresponding to the special tag that must be interpreted
"""
current_lang = ln
def clean_language_tag(match):
"""
Return tag text content if tag language of match is output language.
Called by substitution in 'filter_languages(...)'
@param match: a match object corresponding to the special tag that must be interpreted
"""
if match.group(1) == current_lang:
return match.group(2)
else:
return ""
# End of clean_language_tag
lang_tag_content = match.group("langs")
# Try to find tag with current lang. If it does not exists,
# then current_lang becomes CFG_SITE_LANG until the end of this
# replace
pattern_current_lang = re.compile(r"<(" + current_lang +
r")\s*>(.*)(</" + current_lang + r"\s*>)", re.IGNORECASE | re.DOTALL)
if re.search(pattern_current_lang, lang_tag_content) is None:
current_lang = CFG_SITE_LANG
cleaned_lang_tag = ln_pattern.sub(clean_language_tag, lang_tag_content)
return cleaned_lang_tag.strip()
# End of search_lang_tag
filtered_format_template = pattern_lang.sub(search_lang_tag, format_template)
return filtered_format_template
def get_format_template(filename, with_attributes=False):
"""
Returns the structured content of the given formate template.
if 'with_attributes' is true, returns the name and description. Else 'attrs' is not
returned as key in dictionary (it might, if it has already been loaded previously)::
{'code':"<b>Some template code</b>"
'attrs': {'name': "a name", 'description': "a description"}
}
@param filename: the filename of an format template
@param with_attributes: if True, fetch the attributes (names and description) for format'
@return: strucured content of format template
"""
_ = gettext_set_language(CFG_SITE_LANG)
if not filename.endswith("."+CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION) and \
not filename.endswith(".xsl"):
return None
# Get from cache whenever possible
if filename in format_templates_cache:
# If we must return with attributes and template exist in
# cache with attributes then return cache.
# Else reload with attributes
if with_attributes and \
'attrs' in format_templates_cache[filename]:
return format_templates_cache[filename]
format_template = {'code': ""}
try:
path = registry.format_templates_lookup[filename]
format_file = open(path)
format_content = format_file.read()
format_file.close()
# Load format template code
# Remove name and description
if filename.endswith("."+CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION):
code_and_description = pattern_format_template_name.sub("",
format_content, 1)
code = pattern_format_template_desc.sub("", code_and_description, 1)
else:
code = format_content
format_template['code'] = code
except:
register_exception()
# Save attributes if necessary
if with_attributes:
format_template['attrs'] = get_format_template_attrs(filename)
# Cache and return
format_templates_cache[filename] = format_template
return format_template
def get_format_templates(with_attributes=False):
"""
Returns the list of all format templates, as dictionary with filenames as keys
if 'with_attributes' is true, returns the name and description. Else 'attrs' is not
returned as key in each dictionary (it might, if it has already been loaded previously)::
[{'code':"<b>Some template code</b>"
'attrs': {'name': "a name", 'description': "a description"}
},
...
}
@param with_attributes: if True, fetch the attributes (names and description) for formats
@return: the list of format templates (with code and info)
"""
format_templates = {}
for filename in registry.format_templates:
if filename.endswith("."+CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION) or \
filename.endswith(".xsl"):
filename = os.path.basename(filename)
format_templates[filename] = get_format_template(filename,
with_attributes)
return format_templates
def get_format_template_attrs(filename):
"""
Returns the attributes of the format template with given filename
The attributes are {'name', 'description'}
Caution: the function does not check that path exists or
that the format element is valid.
@param filename: the name of a format template
@return: a structure with detailed information about given format template
"""
_ = gettext_set_language(CFG_SITE_LANG)
attrs = {}
attrs['name'] = ""
attrs['description'] = ""
try:
template_file = open(registry.format_templates_lookup[filename])
code = template_file.read()
template_file.close()
match = None
if filename.endswith(".xsl"):
# .xsl
attrs['name'] = filename[:-4]
else:
# .bft
match = pattern_format_template_name.search(code)
if match is not None:
attrs['name'] = match.group('name')
else:
attrs['name'] = filename
match = pattern_format_template_desc.search(code)
if match is not None:
attrs['description'] = match.group('desc').rstrip('.')
except Exception as e:
try:
raise InvenioBibFormatError(_('Could not read format template named %(filename)s. %(error)s.', filename=filename, error=str(e)))
except InvenioBibFormatError:
register_exception()
attrs['name'] = filename
return attrs
def get_format_element(element_name, verbose=0, with_built_in_params=False,
soft_fail=False):
"""
Returns the format element structured content.
Return None if element cannot be loaded (file not found, not readable or
invalid)
The returned structure is::
{'attrs': {some attributes in dict. See get_format_element_attrs_from_*}
'code': the_function_code,
'type':"field" or "python" depending if element is defined in file or table,
'escape_function': the function to call to know if element output must be escaped}
@param element_name: the name of the format element to load
@param verbose: the level of verbosity from 0 to 9 (O: silent,
5: errors,
7: errors and warnings,
9: errors and warnings, stop if error (debug mode ))
@param with_built_in_params: if True, load the parameters built in all elements
@return: a dictionary with format element attributes
"""
_ = gettext_set_language(CFG_SITE_LANG)
# Resolve filename and prepare 'name' as key for the cache
filename = resolve_format_element_filename(element_name)
if filename is not None:
name = filename.upper()
else:
name = element_name.upper()
if name in format_elements_cache:
element = format_elements_cache[name]
if not with_built_in_params or \
(with_built_in_params and
'builtin_params' in element['attrs']):
return element
if filename is None:
# Element is maybe in tag table
if bibformat_dblayer.tag_exists_for_name(element_name):
format_element = {'attrs': get_format_element_attrs_from_table(
element_name,
with_built_in_params),
'code': None,
'escape_function': None,
'type': "field"}
# Cache and returns
format_elements_cache[name] = format_element
return format_element
elif soft_fail:
register_exception()
return None
else:
raise InvenioBibFormatError(_('Format element %s could not be found.') % element_name)
else:
format_element = {}
module_name = filename
if module_name.endswith(".py"):
module_name = module_name[:-3]
# Load function 'format_element()' inside element
try:
function_format = TEMPLATE_CONTEXT_FUNCTIONS_CACHE.\
bibformat_elements()[module_name]
format_element['code'] = function_format
except:
if soft_fail:
register_exception()
return None
else:
raise
# Load function 'escape_values()' inside element
format_element['escape_function'] = function_format._escape_values
# Prepare, cache and return
format_element['attrs'] = get_format_element_attrs_from_function(
function_format,
element_name,
with_built_in_params)
format_element['type'] = "python"
format_elements_cache[name] = format_element
return format_element
def get_format_elements(with_built_in_params=False):
"""
Returns the list of format elements attributes as dictionary structure
Elements declared in files have priority over element declared in 'tag' table
The returned object has this format::
{element_name1: {'attrs': {'description':..., 'seealso':...
'params':[{'name':..., 'default':..., 'description':...}, ...]
'builtin_params':[{'name':..., 'default':..., 'description':...}, ...]
},
'code': code_of_the_element
},
element_name2: {...},
...}
Returns only elements that could be loaded (not error in code)
@return: a dict of format elements with name as key, and a dict as attributes
@param with_built_in_params: if True, load the parameters built in all elements
"""
format_elements = {}
mappings = bibformat_dblayer.get_all_name_tag_mappings()
for name in mappings:
format_elements[name.upper().replace(" ", "_").strip()] = get_format_element(name, with_built_in_params=with_built_in_params)
for module in registry.format_elements:
filename = os.path.basename(module.__file__)
filename_test = filename.upper().replace(" ", "_")
if filename_test.endswith(".PYC"):
filename_test = filename_test[:-1]
if filename_test.endswith(".PY") and not filename.upper().startswith("__INIT__.PY"):
if filename_test.startswith("BFE_"):
filename_test = filename_test[4:]
element_name = filename_test[:-3]
element = get_format_element(element_name,
with_built_in_params=with_built_in_params,
soft_fail=True)
if element is not None:
format_elements[element_name] = element
return format_elements
def get_format_element_attrs_from_function(function, element_name,
with_built_in_params=False):
"""
Returns the attributes of the function given as parameter.
It looks for standard parameters of the function, default
values and comments in the docstring.
The attributes are::
{'name' : "name of element" #basically the name of 'name' parameter
'description': "a string description of the element",
'seealso' : ["element_1.py", "element_2.py", ...] #a list of related elements
'params': [{'name':"param_name", #a list of parameters for this element (except 'bfo')
'default':"default value",
'description': "a description"}, ...],
'builtin_params': {name: {'name':"param_name",#the parameters builtin for all elem of this kind
'default':"default value",
'description': "a description"}, ...},
}
@param function: the formatting function of a format element
@param element_name: the name of the element
@param with_built_in_params: if True, load the parameters built in all elements
@return: a structure with detailed information of a function
"""
attrs = {}
attrs['description'] = ""
attrs['name'] = element_name.replace(" ", "_").upper()
attrs['seealso'] = []
docstring = function.__doc__
if isinstance(docstring, str):
# Look for function description in docstring
#match = pattern_format_element_desc.search(docstring)
description = docstring.split("@param")[0]
description = description.split("@see:")[0]
attrs['description'] = description.strip().rstrip('.')
# Look for @see: in docstring
match = pattern_format_element_seealso.search(docstring)
if match is not None:
elements = match.group('see').rstrip('.').split(",")
for element in elements:
attrs['seealso'].append(element.strip())
params = {}
# Look for parameters in function definition
args, dummy_varargs, dummy_varkw, defaults = inspect.getargspec(function)
# Prepare args and defaults_list such that we can have a mapping
# from args to defaults
args.reverse()
if defaults is not None:
defaults_list = list(defaults)
defaults_list.reverse()
else:
defaults_list = []
for arg, default in map(None, args, defaults_list):
if arg == "bfo":
#Don't keep this as parameter. It is hidden to users, and
#exists in all elements of this kind
continue
param = {}
param['name'] = arg
if default is None:
#In case no check is made inside element, we prefer to
#print "" (nothing) than None in output
param['default'] = ""
else:
param['default'] = default
param['description'] = "(no description provided)"
params[arg] = param
if isinstance(docstring, str):
# Look for AT param descriptions in docstring.
# Add description to existing parameters in params dict
params_iterator = pattern_format_element_params.finditer(docstring)
for match in params_iterator:
name = match.group('name')
if name in params:
params[name]['description'] = match.group('desc').rstrip('.')
attrs['params'] = params.values()
# Load built-in parameters if necessary
if with_built_in_params:
builtin_params = []
# Add 'prefix' parameter
param_prefix = {}
param_prefix['name'] = "prefix"
param_prefix['default'] = ""
param_prefix['description'] = """A prefix printed only if the
record has a value for this element"""
builtin_params.append(param_prefix)
# Add 'suffix' parameter
param_suffix = {}
param_suffix['name'] = "suffix"
param_suffix['default'] = ""
param_suffix['description'] = """A suffix printed only if the
record has a value for this element"""
builtin_params.append(param_suffix)
# Add 'default' parameter
param_default = {}
param_default['name'] = "default"
param_default['default'] = ""
param_default['description'] = """A default value printed if the
record has no value for this element"""
builtin_params.append(param_default)
# Add 'escape' parameter
param_escape = {}
param_escape['name'] = "escape"
param_escape['default'] = ""
param_escape['description'] = """0 keeps value as it is. Refer to main
documentation for escaping modes
1 to 7"""
builtin_params.append(param_escape)
attrs['builtin_params'] = builtin_params
return attrs
def get_format_element_attrs_from_table(element_name,
with_built_in_params=False):
"""
Returns the attributes of the format element with given name in 'tag' table.
Returns None if element_name does not exist in tag table.
The attributes are::
{'name' : "name of element" #basically the name of 'element_name' parameter
'description': "a string description of the element",
'seealso' : [] #a list of related elements. Always empty in this case
'params': [], #a list of parameters for this element. Always empty in this case
'builtin_params': [{'name':"param_name", #the parameters builtin for all elem of this kind
'default':"default value",
'description': "a description"}, ...],
'tags':["950.1", 203.a] #the list of tags printed by this element
}
@param element_name: an element name in database
@param element_name: the name of the element
@param with_built_in_params: if True, load the parameters built in all elements
@return: a structure with detailed information of an element found in DB
"""
attrs = {}
tags = bibformat_dblayer.get_tags_from_name(element_name)
field_label = "field"
if len(tags)>1:
field_label = "fields"
attrs['description'] = "Prints %s %s of the record" % (field_label,
", ".join(tags))
attrs['name'] = element_name.replace(" ", "_").upper()
attrs['seealso'] = []
attrs['params'] = []
attrs['tags'] = tags
# Load built-in parameters if necessary
if with_built_in_params:
builtin_params = []
# Add 'prefix' parameter
param_prefix = {}
param_prefix['name'] = "prefix"
param_prefix['default'] = ""
param_prefix['description'] = """A prefix printed only if the
record has a value for this element"""
builtin_params.append(param_prefix)
# Add 'suffix' parameter
param_suffix = {}
param_suffix['name'] = "suffix"
param_suffix['default'] = ""
param_suffix['description'] = """A suffix printed only if the
record has a value for this element"""
builtin_params.append(param_suffix)
# Add 'separator' parameter
param_separator = {}
param_separator['name'] = "separator"
param_separator['default'] = " "
param_separator['description'] = """A separator between elements of
the field"""
builtin_params.append(param_separator)
# Add 'nbMax' parameter
param_nbMax = {}
param_nbMax['name'] = "nbMax"
param_nbMax['default'] = ""
param_nbMax['description'] = """The maximum number of values to
print for this element. No limit if not
specified"""
builtin_params.append(param_nbMax)
# Add 'default' parameter
param_default = {}
param_default['name'] = "default"
param_default['default'] = ""
param_default['description'] = """A default value printed if the
record has no value for this element"""
builtin_params.append(param_default)
# Add 'escape' parameter
param_escape = {}
param_escape['name'] = "escape"
param_escape['default'] = ""
param_escape['description'] = """If set to 1, replaces special
characters '&', '<' and '>' of this
element by SGML entities"""
builtin_params.append(param_escape)
attrs['builtin_params'] = builtin_params
return attrs
def get_output_format(code, with_attributes=False, verbose=0):
"""
Returns the structured content of the given output format
If 'with_attributes' is true, also returns the names and description of the output formats,
else 'attrs' is not returned in dict (it might, if it has already been loaded previously).
if output format corresponding to 'code' is not found return an empty structure.
See get_output_format_attrs() to learn more about the attributes::
{'rules': [ {'field': "980__a",
'value': "PREPRINT",
'template': "filename_a.bft",
},
{...}
],
'attrs': {'names': {'generic':"a name", 'sn':{'en': "a name", 'fr':"un nom"}, 'ln':{'en':"a long name"}}
'description': "a description"
'code': "fnm1",
'content_type': "application/ms-excel",
'visibility': 1
}
'default':"filename_b.bft"
}
@param code: the code of an output_format
@param with_attributes: if True, fetch the attributes (names and description) for format
@param verbose: the level of verbosity from 0 to 9 (O: silent,
5: errors,
7: errors and warnings,
9: errors and warnings, stop if error (debug mode ))
@return: strucured content of output format
"""
_ = gettext_set_language(CFG_SITE_LANG)
output_format = {'rules': [], 'default': ""}
filename = resolve_output_format_filename(code, verbose)
if filename is None:
try:
raise InvenioBibFormatError(_('Output format with code %(code)s could not be found.', code=code))
except InvenioBibFormatError:
register_exception()
if with_attributes: #Create empty attrs if asked for attributes
output_format['attrs'] = get_output_format_attrs(code, verbose)
return output_format
# Get from cache whenever possible
if filename in format_outputs_cache:
# If was must return with attributes but cache has not
# attributes, then load attributes
if with_attributes and 'attrs' not in format_outputs_cache[filename]:
format_outputs_cache[filename]['attrs'] = get_output_format_attrs(code, verbose)
return format_outputs_cache[filename]
try:
if with_attributes:
output_format['attrs'] = get_output_format_attrs(code, verbose)
path = registry.output_formats_lookup[filename]
format_file = open(path)
current_tag = ''
for line in format_file:
line = line.strip()
if line == "":
# Ignore blank lines
continue
if line.endswith(":"):
# Retrieve tag
# Remove : spaces and eol at the end of line
clean_line = line.rstrip(": \n\r")
# The tag starts at second position
current_tag = "".join(clean_line.split()[1:]).strip()
elif line.find('---') != -1:
words = line.split('---')
template = words[-1].strip()
condition = ''.join(words[:-1])
output_format['rules'].append({'field': current_tag,
'value': condition,
'template': template,
})
elif line.find(':') != -1:
# Default case
default = line.split(':')[1].strip()
output_format['default'] = default
except:
register_exception()
# Cache and return
format_outputs_cache[filename] = output_format
return output_format
def get_output_format_attrs(code, verbose=0):
"""
Returns the attributes of an output format.
The attributes contain 'code', which is the short identifier of the output format
(to be given as parameter in format_record function to specify the output format),
'description', a description of the output format, 'visibility' the visibility of
the format in the output format list on public pages and 'names', the localized names
of the output format. If 'content_type' is specified then the search_engine will
send a file with this content type and with result of formatting as content to the user.
The 'names' dict always contais 'generic', 'ln' (for long name) and 'sn' (for short names)
keys. 'generic' is the default name for output format. 'ln' and 'sn' contain long and short
localized names of the output format. Only the languages for which a localization exist
are used::
{'names': {'generic':"a name", 'sn':{'en': "a name", 'fr':"un nom"}, 'ln':{'en':"a long name"}}
'description': "a description"
'code': "fnm1",
'content_type': "application/ms-excel",
'visibility': 1
}
@param code: the short identifier of the format
@param verbose: the level of verbosity from 0 to 9 (O: silent,
5: errors,
7: errors and warnings,
9: errors and warnings, stop if error (debug mode ))
@return: strucured content of output format attributes
"""
if code.endswith("."+CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION):
code = code[:-(len(CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION) + 1)]
attrs = {'names': {'generic': "",
'ln': {},
'sn': {}},
'description': '',
'code': code.upper(),
'content_type': "",
'visibility': 1}
filename = resolve_output_format_filename(code, verbose)
if filename is None:
return attrs
attrs['names'] = bibformat_dblayer.get_output_format_names(code)
attrs['description'] = bibformat_dblayer.get_output_format_description(code)
attrs['content_type'] = bibformat_dblayer.get_output_format_content_type(code)
attrs['visibility'] = bibformat_dblayer.get_output_format_visibility(code)
return attrs
def get_output_formats(with_attributes=False):
"""
Returns the list of all output format, as a dictionary with their filename as key
If 'with_attributes' is true, also returns the names and description of the output formats,
else 'attrs' is not returned in dicts (it might, if it has already been loaded previously).
See get_output_format_attrs() to learn more on the attributes::
{'filename_1.bfo': {'rules': [ {'field': "980__a",
'value': "PREPRINT",
'template': "filename_a.bft",
},
{...}
],
'attrs': {'names': {'generic':"a name", 'sn':{'en': "a name", 'fr':"un nom"}, 'ln':{'en':"a long name"}}
'description': "a description"
'code': "fnm1"
}
'default':"filename_b.bft"
},
'filename_2.bfo': {...},
...
}
@param with_attributes: if returned output formats contain detailed info, or not
@type with_attributes: boolean
@return: the list of output formats
"""
output_formats = {}
for filename in registry.output_formats_lookup.values():
filename = os.path.basename(filename)
if filename.endswith("."+CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION):
code = "".join(filename.split(".")[:-1])
if filename in output_formats:
continue
output_formats[filename] = get_output_format(code, with_attributes)
return output_formats
def resolve_format_element_filename(element_name):
"""
Returns the filename of element corresponding to x{element_name}
This is necessary since format templates code call
elements by ignoring case, for eg. <BFE_AUTHOR> is the
same as <BFE_author>.
It is also recommended that format elements filenames are
prefixed with bfe_ . We need to look for these too.
The name of the element has to start with "BFE_".
@param element_name: a name for a format element
@return: the corresponding filename, with right case
"""
if not element_name.endswith(".py"):
name = element_name.replace(" ", "_").upper() +".PY"
else:
name = element_name.replace(" ", "_").upper()
files = registry.format_elements
for element in files:
filename = element.__file__
if filename.endswith('.pyc'):
filename = filename[:-1]
basename = os.path.basename(filename)
test_filename = basename.replace(" ", "_").upper()
if test_filename == name or \
test_filename == "BFE_" + name or \
"BFE_" + test_filename == name:
return basename
# No element with that name found
# Do not log error, as it might be a normal execution case:
# element can be in database
return None
def resolve_output_format_filename(code, verbose=0):
"""
Returns the filename of output corresponding to code
This is necessary since output formats names are not case sensitive
but most file systems are.
@param code: the code for an output format
@param verbose: the level of verbosity from 0 to 9 (O: silent,
5: errors,
7: errors and warnings,
9: errors and warnings, stop if error (debug mode ))
@return: the corresponding filename, with right case, or None if not found
"""
_ = gettext_set_language(CFG_SITE_LANG)
#Remove non alphanumeric chars (except . and _)
code = re.sub(r"[^.0-9a-zA-Z_]", "", code)
if not code.endswith("."+CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION):
code = re.sub(r"\W", "", code)
code += "."+CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION
code = code.upper()
for filename in registry.output_formats_lookup.keys():
if filename.upper() == code:
return filename
# No output format with that name found
raise InvenioBibFormatError(_('Could not find output format named %(x_code)s.', x_code=code))
if verbose >= 5:
sys.stderr.write(exc.message)
if verbose >= 9:
sys.exit(exc.message)
return None
def get_fresh_format_template_filename(name):
"""
Returns a new filename and name for template with given name.
Used when writing a new template to a file, so that the name
has no space, is unique in template directory
Returns (unique_filename, modified_name)
@param name: name for a format template
@return: the corresponding filename, and modified name if necessary
"""
#name = re.sub(r"\W", "", name) #Remove non alphanumeric chars
name = name.replace(" ", "_")
filename = name
# Remove non alphanumeric chars (except .)
filename = re.sub(r"[^.0-9a-zA-Z]", "", filename)
index = 1
def _get_fullname(filename):
return filename + '.' + CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION
while _get_fullname(filename) in registry.format_templates_lookup:
index += 1
filename = name + str(index)
if index > 1:
returned_name = (name + str(index)).replace("_", " ")
else:
returned_name = name.replace("_", " ")
return (_get_fullname(filename), returned_name)
def get_fresh_output_format_filename(code):
"""
Returns a new filename for output format with given code.
Used when writing a new output format to a file, so that the code
has no space, is unique in output format directory. The filename
also need to be at most 6 chars long, as the convention is that
filename == output format code (+ .extension)
We return an uppercase code
Returns (unique_filename, modified_code)
@param code: the code of an output format
@return: the corresponding filename, and modified code if necessary
"""
_ = gettext_set_language(CFG_SITE_LANG)
#code = re.sub(r"\W", "", code) #Remove non alphanumeric chars
code = code.upper().replace(" ", "_")
# Remove non alphanumeric chars (except . and _)
code = re.sub(r"[^.0-9a-zA-Z_]", "", code)
if len(code) > 6:
code = code[:6]
def _get_fullname(filename):
return filename + '.' + CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION
filename = code
index = 2
while _get_fullname(filename) in registry.output_formats_lookup:
filename = code + str(index)
if len(filename) > 6:
filename = code[:-(len(str(index)))]+str(index)
index += 1
# We should not try more than 99999... Well I don't see how we
# could get there.. Sanity check.
if index >= 99999:
try:
raise InvenioBibFormatError(_('Could not find a fresh name for output format %(x_code)s.', x_code=code))
except InvenioBibFormatError:
register_exception()
sys.exit("Output format cannot be named as %s" % code)
return (filename + "." + CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION, filename)
def clear_caches():
"""
Clear the caches (Output Format, Format Templates and Format Elements)
@return: None
"""
global format_templates_cache, format_elements_cache, format_outputs_cache
format_templates_cache = {}
format_elements_cache = {}
format_outputs_cache = {}
class BibFormatObject(object):
"""
An object that encapsulates a record and associated methods, and that is given
as parameter to all format elements 'format' function.
The object is made specifically for a given formatting, i.e. it includes
for example the language for the formatting.
The object provides basic accessors to the record. For full access, one can get
the record with get_record() and then use BibRecord methods on the returned object.
"""
# The record
record = None
# The language in which the formatting has to be done
lang = CFG_SITE_LANG
# A list of string describing the context in which the record has
# to be formatted.
# It represents the words of the user request in web interface search
search_pattern = []
# The id of the record
recID = 0
# The information about the user, as returned by
# 'webuser.collect_user_info(req)'
user_info = None
# The format in which the record is being formatted
output_format = ''
req = None # DEPRECATED: use bfo.user_info instead. Used by WebJournal.
def __init__(self, recID, ln=CFG_SITE_LANG, search_pattern=None,
xml_record=None, user_info=None, output_format=''):
"""
Creates a new bibformat object, with given record.
You can either specify an record ID to format, or give its xml representation.
if 'xml_record' is not None, use 'xml_record' instead of recID for the record.
'user_info' allows to grant access to some functionalities on
a page depending on the user's priviledges. It is a dictionary
in the following form::
user_info = {
'remote_ip' : '',
'remote_host' : '',
'referer' : '',
'uri' : '',
'agent' : '',
'uid' : -1,
'nickname' : '',
'email' : '',
'group' : [],
'guest' : '1'
}
@param recID: the id of a record
@param ln: the language in which the record has to be formatted
@param search_pattern: list of string representing the request used by the user in web interface
@param xml_record: a xml string of the record to format
@param user_info: the information of the user who will view the formatted page
@param output_format: the output_format used for formatting this record
"""
self.xml_record = None # *Must* remain empty if recid is given
if xml_record is not None:
# If record is given as parameter
self.xml_record = xml_record
self.record = create_record(xml_record)[0]
recID = record_get_field_value(self.record, "001")
self.lang = wash_language(ln)
if search_pattern is None:
search_pattern = []
self.search_pattern = search_pattern
self.recID = recID
self.output_format = output_format
self.user_info = user_info
if self.user_info is None:
from invenio.ext.login.legacy_user import UserInfo
self.user_info = UserInfo(None)
def get_record(self):
"""
Returns the record structure of this L{BibFormatObject} instance
@return: the record structure as defined by BibRecord library
"""
from invenio.legacy.search_engine import get_record
# Create record if necessary
if self.record is None:
# on-the-fly creation if current output is xm
self.record = get_record(self.recID)
return self.record
def control_field(self, tag, escape=0):
"""
Returns the value of control field given by tag in record
@param tag: the marc code of a field
@param escape: 1 if returned value should be escaped. Else 0.
@return: value of field tag in record
"""
if self.get_record() is None:
#Case where BibRecord could not parse object
return ''
p_tag = parse_tag(tag)
field_value = record_get_field_value(self.get_record(),
p_tag[0],
p_tag[1],
p_tag[2],
p_tag[3])
if escape == 0:
return field_value
else:
return escape_field(field_value, escape)
def field(self, tag, escape=0):
"""
Returns the value of the field corresponding to tag in the
current record.
If the value does not exist, return empty string. Else
returns the same as bfo.fields(..)[0] (see docstring below).
'escape' parameter allows to escape special characters
of the field. The value of escape can be:
0. no escaping
1. escape all HTML characters
2. remove unsafe HTML tags (Eg. keep <br />)
3. Mix of mode 1 and 2. If value of field starts with
<!-- HTML -->, then use mode 2. Else use mode 1.
4. Remove all HTML tags
5. Same as 2, with more tags allowed (like <img>)
6. Same as 3, with more tags allowed (like <img>)
7. Mix of mode 0 and mode 1. If field_value
starts with <!--HTML-->, then use mode 0.
Else use mode 1.
8. Same as mode 1, but also escape double-quotes
9. Same as mode 4, but also escape double-quotes
@param tag: the marc code of a field
@param escape: 1 if returned value should be escaped. Else 0. (see above for other modes)
@return: value of field tag in record
"""
list_of_fields = self.fields(tag)
if len(list_of_fields) > 0:
# Escaping below
if escape == 0:
return list_of_fields[0]
else:
return escape_field(list_of_fields[0], escape)
else:
return ""
def fields(self, tag, escape=0, repeatable_subfields_p=False):
"""
Returns the list of values corresonding to "tag".
If tag has an undefined subcode (such as 999C5),
the function returns a list of dictionaries, whoose keys
are the subcodes and the values are the values of tag.subcode.
If the tag has a subcode, simply returns list of values
corresponding to tag.
Eg. for given MARC::
999C5 $a value_1a $b value_1b
999C5 $b value_2b
999C5 $b value_3b $b value_3b_bis
>>> bfo.fields('999C5b')
>>> ['value_1b', 'value_2b', 'value_3b', 'value_3b_bis']
>>> bfo.fields('999C5')
>>> [{'a':'value_1a', 'b':'value_1b'},
{'b':'value_2b'},
{'b':'value_3b'}]
By default the function returns only one value for each
subfield (that is it considers that repeatable subfields are
not allowed). It is why in the above example 'value3b_bis' is
not shown for bfo.fields('999C5'). (Note that it is not
defined which of value_3b or value_3b_bis is returned). This
is to simplify the use of the function, as most of the time
subfields are not repeatable (in that way we get a string
instead of a list). You can allow repeatable subfields by
setting 'repeatable_subfields_p' parameter to True. In
this mode, the above example would return:
>>> bfo.fields('999C5b', repeatable_subfields_p=True)
>>> ['value_1b', 'value_2b', 'value_3b']
>>> bfo.fields('999C5', repeatable_subfields_p=True)
>>> [{'a':['value_1a'], 'b':['value_1b']},
{'b':['value_2b']},
{'b':['value_3b', 'value3b_bis']}]
NOTICE THAT THE RETURNED STRUCTURE IS DIFFERENT. Also note
that whatever the value of 'repeatable_subfields_p' is,
bfo.fields('999C5b') always show all fields, even repeatable
ones. This is because the parameter has no impact on the
returned structure (it is always a list).
'escape' parameter allows to escape special characters
of the fields. The value of escape can be:
0. No escaping
1. Escape all HTML characters
2. Remove unsafe HTML tags (Eg. keep <br />)
3. Mix of mode 1 and 2. If value of field starts with
<!-- HTML -->, then use mode 2. Else use mode 1.
4. Remove all HTML tags
5. Same as 2, with more tags allowed (like <img>)
6. Same as 3, with more tags allowed (like <img>)
7. Mix of mode 0 and mode 1. If field_value
starts with <!--HTML-->, then use mode 0.
Else use mode 1.
8. Same as mode 1, but also escape double-quotes
9. Same as mode 4, but also escape double-quotes
@param tag: the marc code of a field
@param escape: 1 if returned values should be escaped. Else 0.
@repeatable_subfields_p if True, returns the list of subfields in the dictionary
@return: values of field tag in record
"""
if self.get_record() is None:
# Case where BibRecord could not parse object
return []
p_tag = parse_tag(tag)
if p_tag[3] != "":
# Subcode has been defined. Simply returns list of values
values = record_get_field_values(self.get_record(),
p_tag[0],
p_tag[1],
p_tag[2],
p_tag[3])
if escape == 0:
return values
else:
return [escape_field(value, escape) for value in values]
else:
# Subcode is undefined. Returns list of dicts.
# However it might be the case of a control field.
instances = record_get_field_instances(self.get_record(),
p_tag[0],
p_tag[1],
p_tag[2])
if repeatable_subfields_p:
list_of_instances = []
for instance in instances:
instance_dict = {}
for subfield in instance[0]:
if subfield[0] not in instance_dict:
instance_dict[subfield[0]] = []
if escape == 0:
instance_dict[subfield[0]].append(subfield[1])
else:
instance_dict[subfield[0]].append(escape_field(subfield[1], escape))
list_of_instances.append(instance_dict)
return list_of_instances
else:
if escape == 0:
return [dict(instance[0]) for instance in instances]
else:
return [dict([(subfield[0], escape_field(subfield[1], escape))
for subfield in instance[0]])
for instance in instances]
def kb(self, kb, string, default=""):
"""
Returns the value of the "string" in the knowledge base "kb".
If kb does not exist or string does not exist in kb,
returns 'default' string or empty string if not specified.
@param kb: a knowledge base name
@param string: the string we want to translate
@param default: a default value returned if 'string' not found in 'kb'
@return: a string value corresponding to translated input with given kb
"""
if not string:
return default
val = get_kbr_values(kb, searchkey=string, searchtype='e')
try:
return val[0][0]
except IndexError:
return default
# Utility functions
##
def escape_field(value, mode=0):
"""
Utility function used to escape the value of a field in given mode.
- mode 0: no escaping
- mode 1: escaping all HTML/XML characters (escaped chars are shown as escaped)
- mode 2: escaping unsafe HTML tags to avoid XSS, but
keep basic one (such as <br />)
Escaped tags are removed.
- mode 3: mix of mode 1 and mode 2. If field_value starts with <!--HTML-->,
then use mode 2. Else use mode 1.
- mode 4: escaping all HTML/XML tags (escaped tags are removed)
- mode 5: same as 2, but allows more tags, like <img>
- mode 6: same as 3, but allows more tags, like <img>
- mode 7: mix of mode 0 and mode 1. If field_value starts with <!--HTML-->,
then use mode 0. Else use mode 1.
- mode 8: same as mode 1, but also escape double-quotes
- mode 9: same as mode 4, but also escape double-quotes
@param value: value to escape
@param mode: escaping mode to use
@return: an escaped version of X{value} according to chosen X{mode}
"""
if mode == 1:
return cgi.escape(value)
elif mode == 8:
return cgi.escape(value, True)
elif mode in [2, 5]:
allowed_attribute_whitelist = CFG_HTML_BUFFER_ALLOWED_ATTRIBUTE_WHITELIST
allowed_tag_whitelist = CFG_HTML_BUFFER_ALLOWED_TAG_WHITELIST + \
('class',)
if mode == 5:
allowed_attribute_whitelist += ('src', 'alt',
'width', 'height',
'style', 'summary',
'border', 'cellspacing',
'cellpadding')
allowed_tag_whitelist += ('img', 'table', 'td',
'tr', 'th', 'span', 'caption')
try:
return washer.wash(value,
allowed_attribute_whitelist=
allowed_attribute_whitelist,
allowed_tag_whitelist=
allowed_tag_whitelist
)
except HTMLParseError:
# Parsing failed
return cgi.escape(value)
elif mode in [3, 6]:
if value.lstrip(' \n').startswith(html_field):
allowed_attribute_whitelist = CFG_HTML_BUFFER_ALLOWED_ATTRIBUTE_WHITELIST
allowed_tag_whitelist = CFG_HTML_BUFFER_ALLOWED_TAG_WHITELIST + \
('class',)
if mode == 6:
allowed_attribute_whitelist += ('src', 'alt',
'width', 'height',
'style', 'summary',
'border', 'cellspacing',
'cellpadding')
allowed_tag_whitelist += ('img', 'table', 'td',
'tr', 'th', 'span', 'caption')
try:
return washer.wash(value,
allowed_attribute_whitelist=
allowed_attribute_whitelist,
allowed_tag_whitelist=
allowed_tag_whitelist
)
except HTMLParseError:
# Parsing failed
return cgi.escape(value)
else:
return cgi.escape(value)
elif mode in [4, 9]:
try:
out = washer.wash(value,
allowed_attribute_whitelist=[],
allowed_tag_whitelist=[]
)
if mode == 9:
out = out.replace('"', '&quot;')
return out
except HTMLParseError:
# Parsing failed
if mode == 4:
return cgi.escape(value)
else:
return cgi.escape(value, True)
elif mode == 7:
if value.lstrip(' \n').startswith(html_field):
return value
else:
return cgi.escape(value)
else:
return value
def filter_hidden_fields(recxml, user_info=None, filter_tags=CFG_BIBFORMAT_HIDDEN_TAGS,
force_filtering=False):
"""
Filter out tags specified by filter_tags from MARCXML. If the user
is allowed to run bibedit, then filter nothing, unless
force_filtering is set to True.
@param recxml: marcxml presentation of the record
@param user_info: user information; if None, then assume invoked via CLI with all rights
@param filter_tags: list of MARC tags to be filtered
@param force_filtering: do we force filtering regardless of user rights?
@return: recxml without the hidden fields
"""
if force_filtering:
pass
else:
if user_info is None:
#by default
return recxml
else:
if (acc_authorize_action(user_info, 'runbibedit')[0] == 0):
#no need to filter
return recxml
#filter..
out = ""
omit = False
for line in recxml.splitlines(True):
#check if this block needs to be omitted
for htag in filter_tags:
- if line.count('datafield tag="'+str(htag)+'"'):
+ if 'datafield tag="'+str(htag)+'"' in line:
omit = True
if not omit:
out += line
- if omit and line.count('</datafield>'):
+ if omit and ('</datafield>' in line or '</marc:datafield>' in line):
omit = False
return out
def bf_profile():
"""
Runs a benchmark
@return: None
"""
for i in range(1, 51):
format_record(i, "HD", ln=CFG_SITE_LANG, verbose=9, search_pattern=[])
return
if __name__ == "__main__":
import profile
import pstats
#bf_profile()
profile.run('bf_profile()', "bibformat_profile")
p = pstats.Stats("bibformat_profile")
p.strip_dirs().sort_stats("cumulative").print_stats()
diff --git a/invenio/modules/formatter/engines/bfx.py b/invenio/modules/formatter/engines/bfx.py
index 3a2a95948..07a9a88d1 100644
--- a/invenio/modules/formatter/engines/bfx.py
+++ b/invenio/modules/formatter/engines/bfx.py
@@ -1,1392 +1,1392 @@
## This file is part of Invenio.
-## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN.
+## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from __future__ import print_function
"""
BFX formatting engine.
For API: see format_with_bfx() docstring below.
"""
__revision__ = "$Id$"
import re
import logging
import copy as p_copy
from xml.dom import minidom, Node
from xml.sax import saxutils
from invenio.modules.formatter.engine import BibFormatObject, get_format_element, eval_format_element
from invenio.modules.formatter.engines.bfx_config import CFG_BIBFORMAT_BFX_LABEL_DEFINITIONS, CFG_BIBFORMAT_BFX_TEMPLATES_PATH
from invenio.modules.formatter.engines.bfx_config import CFG_BIBFORMAT_BFX_FORMAT_TEMPLATE_EXTENSION, CFG_BIBFORMAT_BFX_ELEMENT_NAMESPACE
from invenio.modules.formatter.engines.bfx_config import InvenioBibFormatBfxError, InvenioBibFormatBfxWarning
from invenio.ext.logging import register_exception
from invenio.base.i18n import gettext_set_language
from invenio.config import CFG_SITE_LANG
address_pattern = r'(?P<parent>[a-z_]*):?/?(?P<tag>[0-9_?\w]*)/?(?P<code>[\w_?]?)#?(?P<reg>.*)'
def format_with_bfx(recIDs, out_file, template_name, preprocess=None):
'''
Format a set of records according to a BFX template.
This is the main entry point to the BFX engine.
@param recIDs: a list of record IDs to format
@param out_file: an object to write in; this can be every object which has a 'write' method: file, req, StringIO
@param template_name: the file name of the BFX template without the path and the .bfx extension
@param preprocess: an optional function; every record is passed through this function for initial preprocessing before formatting
'''
trans = MARCTranslator(CFG_BIBFORMAT_BFX_LABEL_DEFINITIONS)
trans.set_record_ids(recIDs, preprocess)
parser = BFXParser(trans)
template_tree = parser.load_template(template_name)
parser.walk(template_tree, out_file)
return None
class BFXParser:
'''
A general-purpose parser for generating xml/xhtml/text output based on a template system.
Must be initialised with a translator. A translator is like a blackbox that returns values, calls functions, etc...
Works with every translator supporting the following simple interface:
- is_defined(name)
- get_value(name)
- iterator(name)
- call_function(func_name, list_of_parameters)
Customized for MARC to XML conversion through the use of a MARCTranslator.
Templates are strict XML files. They are built by combining any tags with the
special BFX tags living in the http://invenio-software.org/ namespace.
Easily extensible by tags of your own.
Defined tags:
- template: defines a template
- template_ref: a reference to a template
- loop structure
- if, then, elif, else structure
- text: output text
- field: query translator for field 'name'
- element: call external functions
'''
def __init__(self, translator):
'''
Create an instance of the BFXParser class. Initialize with a translator.
The BFXparser makes queries to the translator for the values of certain names.
For the communication it uses the following translator methods:
- is_defined(name)
- iterator(name)
- get_value(name, [display_specifier])
@param translator: the translator used by the class instance
'''
self.translator = translator
self.known_operators = ['style', 'format', 'template', 'template_ref', 'text', 'field', 'element', 'loop', 'if', 'then', 'else', 'elif']
self.flags = {} # store flags here;
self.templates = {} # store templates and formats here
self.start_template_name = None #the name of the template from which the 'execution' starts;
#this is usually a format or the only template found in a doc
def load_template(self, template_name, template_source=None):
'''
Load a BFX template file.
A template file can have one of two forms:
- it is a file with a single template. Root tag is 'template'.
In an API call the single template element is 'executed'.
- it is a 'style' file which contains exactly one format and zero or more templates. Root tag is 'style' with children 'format' and 'template'(s).
In this case only the format code is 'executed'. Naturally, in it, it would have references to other templates in the document.
Template can be given by name (in that case search path is in
standard directory for bfx template) or directly using the template source.
If given, template_source overrides template_name
@param template_name: the name of the BFX template, the same as the name of the filename without the extension
@return: a DOM tree of the template
'''
_ = gettext_set_language(CFG_SITE_LANG)
if template_source is None:
template_file_name = CFG_BIBFORMAT_BFX_TEMPLATES_PATH + '/' + template_name + '.' + CFG_BIBFORMAT_BFX_FORMAT_TEMPLATE_EXTENSION
#load document
doc = minidom.parse(template_file_name)
else:
doc = minidom.parseString(template_source)
#set exec flag to false and walk document to find templates and formats
self.flags['exec'] = False
self.walk(doc)
#check found templates
if self.start_template_name:
start_template = self.templates[self.start_template_name]['node']
else:
#print CFG_BIBFORMAT_BFX_WARNING_MESSAGES['WRN_BFX_NO_FORMAT_FOUND']
if len(self.templates) == 1:
# no format found, check if there is a default template
self.start_template_name = self.templates.keys()[0]
start_template = self.templates[self.start_template_name]['node']
else:
#no formats found, templates either zero or more than one
if len(self.templates) > 1:
try:
raise InvenioBibFormatBfxError(_('More than one templates found in the document. No format found.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
self.flags['exec'] = True
return start_template
def parse_attribute(self, expression):
'''
A function to check if an expression is of the special form [!name:display].
A short form for saying <bx:field name="name" display="tag">, used in element attributes.
@param expression: a string, usually taken from an attribute value
@return: if the string is special, parse it and return the corresponding value; else return the initial expression
'''
output = expression
pattern = '\[!(?P<tmp>[\w_.:]*)\]'
expr = re.compile(pattern)
match = expr.match(expression)
if match:
tmp = match.group('tmp')
tmp = tmp.split(':')
var = tmp[0]
display = ''
if len(tmp) == 2:
display = tmp[1]
output = self.translator.get_value(var, display)
output = xml_escape(output)
return output
def walk(self, parent, out_file=None):
'''
Walk a template DOM tree.
The main function in the parser. It is recursively called until all the nodes are processed.
This function is used in two different ways:
- for initial loading of the template (and validation)
- for 'execution' of a format/template
The different behaviour is achieved through the use of flags, which can be set to True or False.
@param parent: a node to process; in an API call this is the root node
@param out_file: an object to write to; must have a 'write' method
@return: None
'''
_ = gettext_set_language(CFG_SITE_LANG)
for node in parent.childNodes:
if node.nodeType == Node.TEXT_NODE:
value = get_node_value(node)
value = value.strip()
if out_file:
out_file.write(value)
if node.nodeType == Node.ELEMENT_NODE:
#get values
name, attributes, element_namespace = get_node_name(node), get_node_attributes(node), get_node_namespace(node)
# write values
if element_namespace != CFG_BIBFORMAT_BFX_ELEMENT_NAMESPACE:
#parse all the attributes
for key in attributes.keys():
attributes[key] = self.parse_attribute(attributes[key])
if node_has_subelements(node):
if out_file:
out_file.write(create_xml_element(name=name, attrs=attributes, element_type=xmlopen))
self.walk(node, out_file) #walk subnodes
if out_file:
out_file.write(create_xml_element(name=name, element_type=xmlclose))
else:
if out_file:
out_file.write(create_xml_element(name=name, attrs=attributes, element_type=xmlempty))
#name is a special name, must fall in one of the next cases:
elif node.localName == 'style':
self.ctl_style(node, out_file)
elif node.localName == 'format':
self.ctl_format(node, out_file)
elif node.localName == 'template':
self.ctl_template(node, out_file)
elif node.localName == 'template_ref':
self.ctl_template_ref(node, out_file)
elif node.localName == 'element':
self.ctl_element(node, out_file)
elif node.localName == 'field':
self.ctl_field(node, out_file)
elif node.localName == 'text':
self.ctl_text(node, out_file)
elif node.localName == 'loop':
self.ctl_loop(node, out_file)
elif node.localName == 'if':
self.ctl_if(node, out_file)
elif node.localName == 'then':
self.ctl_then(node, out_file)
elif node.localName == 'else':
self.ctl_else(node, out_file)
elif node.localName == 'elif':
self.ctl_elif(node, out_file)
else:
if node.localName in self.known_operators:
try:
raise InvenioBibFormatBfxError(_('Note for programmer: you have not implemented operator %(x_name)s.', x_name=name))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
# print 'Note for programmer: you haven\'t implemented operator %s.' % (name)
else:
try:
raise InvenioBibFormatBfxError(_('Name %(x_name)s is not recognised as a valid operator name.',x_name=name))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
def ctl_style(self, node, out_file):
'''
Process a style root node.
'''
#exec mode
if self.flags['exec']:
return None
#test mode
self.walk(node, out_file)
return None
def ctl_format(self, node, out_file):
'''
Process a format node.
Get name, description and content attributes.
This function is called only in test mode.
'''
_ = gettext_set_language(CFG_SITE_LANG)
#exec mode
if self.flags['exec']:
return None
#test mode
attrs = get_node_attributes(node)
#get template name and give control to ctl_template
if 'name' in attrs:
name = attrs['name']
if name in self.templates:
try:
raise InvenioBibFormatBfxError(_('Duplicate name: %(x_name)s.', x_name=name))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
self.start_template_name = name
self.ctl_template(node, out_file)
else:
try:
raise InvenioBibFormatBfxError(_('No name defined for the template.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
return None
def ctl_template(self, node, out_file):
'''
Process a template node.
Get name, description and content attributes.
Register name and store for later calls from template_ref.
This function is called only in test mode.
'''
_ = gettext_set_language(CFG_SITE_LANG)
#exec mode
if self.flags['exec']:
return None
#test mode
attrs = get_node_attributes(node)
#get template name
if 'name' in attrs:
name = attrs['name']
if name in self.templates:
try:
raise InvenioBibFormatBfxError(_('Duplicate name: %(x_name)s.', x_name=name))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
self.templates[name] = {}
self.templates[name]['node'] = node
else:
try:
raise InvenioBibFormatBfxError(_('No name defined for the template.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
#get template description
if 'description' in attrs:
description = attrs['description']
else:
description = ''
try:
raise InvenioBibFormatBfxWarning(_('No description entered for the template.'))
except InvenioBibFormatBfxWarning as exc:
register_exception(stream='warning')
logging.warning(exc.message)
self.templates[name]['description'] = description
#get content-type of resulting output
if 'content' in attrs:
content_type = attrs['content']
else:
content_type = 'text/xml'
try:
raise InvenioBibFormatBfxWarning(_('No content type specified for the template. Using default: text/xml.'))
except InvenioBibFormatBfxWarning as exc:
register_exception(stream='warning')
logging.warning(exc.message)
self.templates[name]['content_type'] = content_type
#walk node
self.walk(node, out_file)
return None
def ctl_template_ref(self, node, out_file):
'''
Reference to an external template.
This function is called only in execution mode. Bad references appear as run-time errors.
'''
_ = gettext_set_language(CFG_SITE_LANG)
#test mode
if not self.flags['exec']:
return None
#exec mode
attrs = get_node_attributes(node)
if 'name' not in attrs:
try:
raise InvenioBibFormatBfxError(_('Missing attribute "name" in TEMPLATE_REF.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
name = attrs['name']
#first check for a template in the same file, that is in the already cached templates
if name in self.templates:
node_to_walk = self.templates[name]['node']
self.walk(node_to_walk, out_file)
else:
#load a file and execute it
pass
#template_file_name = CFG_BIBFORMAT_BFX_TEMPLATES_PATH + name + '/' + CFG_BIBFORMAT_BFX_FORMAT_TEMPLATE_EXTENSION
#try:
# node = minidom.parse(template_file_name)
#except:
# print CFG_BIBFORMAT_BFX_ERROR_MESSAGES['ERR_BFX_TEMPLATE_NOT_FOUND'] % (template_file_name)
return None
def ctl_element(self, node, out_file):
'''
Call an external element (written in Python).
'''
_ = gettext_set_language(CFG_SITE_LANG)
#test mode
if not self.flags['exec']:
return None
#exec mode
parameters = get_node_attributes(node)
if 'name' not in parameters:
try:
raise InvenioBibFormatBfxError(_('Missing attribute "name" in ELEMENT.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
function_name = parameters['name']
del parameters['name']
#now run external bfe_name.py, with param attrs
if function_name:
value = self.translator.call_function(function_name, parameters)
value = xml_escape(value)
out_file.write(value)
return None
def ctl_field(self, node, out_file):
'''
Get the value of a field by its name.
'''
_ = gettext_set_language(CFG_SITE_LANG)
#test mode
if not self.flags['exec']:
return None
#exec mode
attrs = get_node_attributes(node)
if 'name' not in attrs:
try:
raise InvenioBibFormatBfxError(_('Missing attribute "name" in FIELD.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
display = ''
if 'display' in attrs:
display = attrs['display']
var = attrs['name']
if not self.translator.is_defined(var):
try:
raise InvenioBibFormatBfxError(_('Field %(x_field)s is not defined.', x_field=var))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
value = self.translator.get_value(var, display)
value = xml_escape(value)
out_file.write(value)
return None
def ctl_text(self, node, out_file):
'''
Output a text
'''
_ = gettext_set_language(CFG_SITE_LANG)
#test mode
if not self.flags['exec']:
return None
#exec mode
attrs = get_node_attributes(node)
if 'value' not in attrs:
try:
raise InvenioBibFormatBfxError(_('Missing attribute "value" in TEXT.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
value = attrs['value']
value = value.replace(r'\n', '\n')
#value = xml_escape(value)
if type(value) == type(u''):
value = value.encode('utf-8')
out_file.write(value)
return None
def ctl_loop(self, node, out_file):
'''
Loop through a set of values.
'''
_ = gettext_set_language(CFG_SITE_LANG)
#test mode
if not self.flags['exec']:
self.walk(node, out_file)
return None
#exec mode
attrs = get_node_attributes(node)
if 'object' not in attrs:
try:
raise InvenioBibFormatBfxError(_('Missing attribute "object" in LOOP.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
name = attrs['object']
if not self.translator.is_defined(name):
try:
raise InvenioBibFormatBfxError(_('Field %(x_field)s is not defined.',x_field=name))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
for new_object in self.translator.iterator(name):
self.walk(node, out_file)
return None
def ctl_if(self, node, out_file):
'''
An if/then/elif/.../elif/else construct.
'If' can have several forms:
<if name="var"/> : True if var is non-empty, eval as string
<if name="var" eq="value"/> : True if var=value, eval as string
<if name="var" lt="value"/> : True if var<value, try to eval as num, else eval as string
<if name="var" gt="value"/> : True if var>value, try to eval as num, else eval as string
<if name="var" le="value"/> : True if var<=value, try to eval as num, else eval as string
<if name="var" ge="value"/> : True if var>=value, try to eval as num, else eval as string
<if name="var" in="val1 val2"/> : True if var in [val1, val2], eval as string
<if name="var" nin="val1 val2"/> : True if var not in [val1, val2], eval as string
<if name="var" neq="value"/> : True if var!=value, eval as string
<if name="var" like="regexp"/> : Match against a regular expression
Example::
<if name="author" eq="Pauli">
<then>Pauli</then>
<elif name="" eq="Einstein">
<then>Pauli</then>
<else>other</else>
</elif>
</if>
'''
_ = gettext_set_language(CFG_SITE_LANG)
#test mode
if not self.flags['exec']:
self.walk(node, out_file)
return None
#exec mode
attrs = get_node_attributes(node)
if 'name' not in attrs:
try:
raise InvenioBibFormatBfxError(_('Missing attrbute "name" in IF.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
#determine result
var = attrs['name']
if not self.translator.is_defined(var):
try:
raise InvenioBibFormatBfxError(_('Field %(x_field)s is not defined.',x_field=var))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
value = self.translator.get_value(var)
value = value.strip()
#equal
if 'eq' in attrs:
pattern = attrs['eq']
if is_number(pattern) and is_number(value):
result = (float(value)==float(pattern))
else:
result = (value==pattern)
#not equal
elif 'neq' in attrs:
pattern = attrs['neq']
if is_number(pattern) and is_number(value):
result = (float(value)!=float(pattern))
else:
result = (value!=pattern)
#lower than
elif 'lt' in attrs:
pattern = attrs['lt']
if is_number(pattern) and is_number(value):
result = (float(value)<float(pattern))
else:
result = (value<pattern)
#greater than
elif 'gt' in attrs:
pattern = attrs['gt']
if is_number(pattern) and is_number(value):
result = (float(value)>float(pattern))
else:
result = (value>pattern)
#lower or equal than
elif 'le' in attrs:
pattern = attrs['le']
if is_number(pattern) and is_number(value):
result = (float(value)<=float(pattern))
else:
result = (value<=pattern)
#greater or equal than
elif 'ge' in attrs:
pattern = attrs['ge']
if is_number(pattern) and is_number(value):
result = (float(value)>=float(pattern))
else:
result = (value>=pattern)
#in
elif 'in' in attrs:
pattern = attrs['in']
values = pattern.split()
result = (value in values)
#not in
elif 'nin' in attrs:
pattern = attrs['nin']
values = pattern.split()
result = (value not in values)
#match against a regular expression
elif 'like' in attrs:
pattern = attrs['like']
try:
expr = re.compile(pattern)
result = expr.match(value)
except:
try:
raise InvenioBibFormatBfxError(_('Invalid regular expression: %(x_patt)s.',x_patt=pattern))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
#simple form: True if non-empty, otherwise False
else:
result = value
#end of evaluation
#=================
#validate subnodes
then_node = get_node_subelement(node, 'then', CFG_BIBFORMAT_BFX_ELEMENT_NAMESPACE)
else_node = get_node_subelement(node, 'else', CFG_BIBFORMAT_BFX_ELEMENT_NAMESPACE)
elif_node = get_node_subelement(node, 'elif', CFG_BIBFORMAT_BFX_ELEMENT_NAMESPACE)
#having else and elif siblings at the same time is a syntax error
if (else_node is not None) and (elif_node is not None):
try:
raise InvenioBibFormatBfxError(_('Invalid syntax of IF statement.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
#now walk appropriate nodes, according to the result
if result: #True
if then_node:
self.walk(then_node, out_file)
#todo: add short form, without 'then', just elements within if statement to walk on 'true' and no 'elif' or 'else' elements
else: #False
if elif_node:
self.ctl_if(elif_node, out_file)
elif else_node:
self.walk(else_node, out_file)
return None
def ctl_then(self, node, out_file):
'''
Calling 'then' directly from the walk function means a syntax error.
'''
_ = gettext_set_language(CFG_SITE_LANG)
#test mode
if not self.flags['exec']:
self.walk(node, out_file)
return None
#exec mode
try:
raise InvenioBibFormatBfxError(_('Invalid syntax of IF statement.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
def ctl_else(self, node, out_file):
'''
Calling 'else' directly from the walk function means a syntax error.
'''
_ = gettext_set_language(CFG_SITE_LANG)
#test mode
if not self.flags['exec']:
self.walk(node, out_file)
return None
#exec mode
try:
raise InvenioBibFormatBfxError(_('Invalid syntax of IF statement.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
def ctl_elif(self, node, out_file):
'''
Calling 'elif' directly from the walk function means a syntax error.
'''
_ = gettext_set_language(CFG_SITE_LANG)
#test mode
if not self.flags['exec']:
self.walk(node, out_file)
return None
#exec mode
try:
raise InvenioBibFormatBfxError(_('Invalid syntax of IF statement.'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return None
class MARCTranslator:
'''
memory[name]
[name]['addresses'] - the set of rules for each of the defined names
[name]['parent'] - the name of the parent; '' if none;
[name]['children'] - a list with the name of the children of every variable
[name]['object'] - stored state of object for performance efficiency
'''
def __init__(self, labels=None):
'''
Create an instance of the translator and init with the list of the defined labels and their rules.
'''
_ = gettext_set_language(CFG_SITE_LANG)
if labels is None:
labels = {}
self.recIDs = []
self.recID = 0
self.recID_index = 0
self.record = None
self.memory = {}
pattern = address_pattern
expr = re.compile(pattern)
for name in labels.keys():
self.memory[name] = {}
self.memory[name]['object'] = None
self.memory[name]['parent'] = ''
self.memory[name]['children'] = []
self.memory[name]['addresses'] = p_copy.deepcopy(labels[name])
for name in self.memory:
for i in range(len(self.memory[name]['addresses'])):
address = self.memory[name]['addresses'][i]
match = expr.match(address)
if not match:
try:
raise InvenioBibFormatBfxError(_('Invalid address: %(x_name)s %(x_addr)s', x_name=name, x_addr=address))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
# print 'Invalid address: ', name, address
else:
parent_name = match.group('parent')
if parent_name:
if parent_name not in self.memory:
try:
raise InvenioBibFormatBfxError(_('Field %(x_field)s is not defined.',x_field=parent_name))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
else:
self.memory[name]['parent'] = parent_name
#now make parent aware of children
if not name in self.memory[parent_name]['children']:
self.memory[parent_name]['children'].append(name)
level = self.determine_level(parent_name)
self.memory[name]['addresses'][i] = self.memory[name]['addresses'][i].replace(parent_name, '/'*level)
#special case 'record'
self.memory['record'] = {}
self.memory['record']['object'] = None
self.memory['record']['parent'] = ''
self.memory['record']['children'] = []
def set_record_ids(self, recIDs, preprocess=None):
'''
Initialize the translator with the set of record IDs.
@param recIDs: a list of the record IDs
@param preprocess: an optional function which acts on every record structure after creating it
This can be used to enrich the record with fields not present in the record initially,
verify the record data or whatever plausible.
Another solution is to use external function elements.
'''
self.record = None
self.recIDs = recIDs
self.preprocess = preprocess
if self.recIDs:
self.recID_index = 0
self.recID = self.recIDs[self.recID_index]
self.record = get_bfx_record(self.recID)
if self.preprocess:
self.preprocess(self.record)
return None
def determine_level(self, name):
'''
Determine the type of the variable, whether this is an instance or a subfield.
This is done by observing the first provided address for the name.
todo: define variable types in config file, remove this function, results in a clearer concept
'''
level = 0 #default value
if name in self.memory:
expr = re.compile(address_pattern)
if self.memory[name]['addresses']:
match = expr.match(self.memory[name]['addresses'][0])
if match:
tag = match.group('tag')
code = match.group('code')
reg = match.group('reg')
if reg:
level = 2 #subfield
elif code:
level = 2 #subfield
elif tag:
level = 1 #instance
return level
#========================================
#API functions for quering the translator
#========================================
def is_defined(self, name):
'''
Check whether a variable is defined.
@param name: the name of the variable
'''
return name in self.memory
def get_num_elements(self, name):
'''
An API function to get the number of elements for a variable.
Do not use this function to build loops, Use iterator instead.
'''
if name == 'record':
return len(self.recIDs)
num = 0
for part in self.iterator(name):
num = num + 1
return num
def get_value(self, name, display_type='value'):
'''
The API function for quering the translator for values of a certain variable.
Called in a loop will result in a different value each time.
Objects are cached in memory, so subsequent calls for the same variable take less time.
@param name: the name of the variable you want the value of
@param display_type: an optional value for the type of the desired output, one of: value, tag, ind1, ind2, code, fulltag;
These can be easily added in the proper place of the code (display_value)
'''
if name == 'record':
return ''
record = self.get_object(name)
return self.display_record(record, display_type)
def iterator(self, name):
'''
An iterator over the values of a certain name.
The iterator changes state of internal variables and objects.
When calling get_value in a loop, this will result each time in a different value.
'''
if name == 'record':
for self.recID in self.recIDs:
self.record = get_bfx_record(self.recID)
if self.preprocess:
self.preprocess(self.record)
yield str(self.recID)
else:
full_object = self.build_object(name)
level = self.determine_level(name)
for new_object in record_parts(full_object, level):
self.memory[name]['object'] = new_object
#parent has changed state; also set childs state to None;
for children_name in self.memory[name]['children']:
self.memory[children_name]['object'] = None
yield new_object
#the result for a call of the same name after an iterator should be the same as if there was no iterator called before
self.memory[name]['object'] = None
def call_function(self, function_name, parameters=None):
'''
Call an external element which is a Python file, using BibFormat
@param function_name: the name of the function to call
@param parameters: a dictionary of the parameters to pass as key=value pairs
@return: a string value, which is the result of the function call
'''
if parameters is None:
parameters = {}
bfo = BibFormatObject(self.recID)
format_element = get_format_element(function_name)
- (value, dummy, dummy) = eval_format_element(format_element,
- bfo,
- parameters)
+ value, dummy = eval_format_element(format_element,
+ bfo,
+ parameters)
#to do: check errors from function call
return value
#========================================
#end of API functions
#========================================
def get_object(self, name):
'''
Responsible for creating the desired object, corresponding to provided name.
If object is not cached in memory, it is build again.
Directly called by API function get_value.
The result is then formatted by display_record according to display_type.
'''
if self.memory[name]['object'] is not None:
return self.memory[name]['object']
new_object = self.build_object(name)
#if you have reached here you are not in an iterator; return first non-empty
level = self.determine_level(name)
for tmp_object in record_parts(new_object, level):
#get the first non-empty
if tmp_object:
new_object = tmp_object
break
self.memory[name]['object'] = new_object
return new_object
def build_object(self, name):
'''
Build the object from the list of addresses
A slave function for get_object.
'''
new_object = {}
parent_name = self.memory[name]['parent'];
has_parent = parent_name
for address in self.memory[name]['addresses']:
if not has_parent:
tmp_object = copy(self.record, address)
new_object = merge(new_object, tmp_object)
else: #has parent
parent_object = self.get_object(parent_name) #already returns the parents instance
tmp_object = copy(parent_object, address)
new_object = merge(new_object, tmp_object)
return new_object
def display_record(self, record, display_type='value'):
'''
Decide what the final output value is according to the display_type.
@param record: the record structure to display; this is most probably just a single subfield
@param display_type: a string specifying the desired output; can be one of: value, tag, ind1, ind2, code, fulltag
@return: a string to output
'''
_ = gettext_set_language(CFG_SITE_LANG)
output = ''
tag, ind1, ind2, code, value = '', '', '', '', ''
if record:
tags = record.keys()
tags.sort()
if tags:
fulltag = tags[0]
tag, ind1, ind2 = fulltag[0:3], fulltag[3:4], fulltag[4:5]
field_instances = record[fulltag]
if field_instances:
field_instance = field_instances[0]
codes = field_instance.keys()
codes.sort()
if codes:
code = codes[0]
value = field_instance[code]
if not display_type:
display_type = 'value'
if display_type == 'value':
output = value
elif display_type == 'tag':
output = tag
elif display_type == 'ind1':
ind1 = ind1.replace('_', ' ')
output = ind1
elif display_type=='ind2':
ind2 = ind2.replace('_', ' ')
output = ind2
elif display_type == 'code':
output = code
elif display_type == 'fulltag':
output = tag + ind1 + ind2
else:
try:
raise InvenioBibFormatBfxError(_('Invalid display type. Must be one of: value, tag, ind1, ind2, code; received: %(x_type)s.', x_type=display_type))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
return output
'''
Functions for use with the structure representing a MARC record defined here.
This record structure differs from the one defined in bibrecord.
The reason is that we want a symmetry between controlfields and datafields.
In this format controlfields are represented internally as a subfield value with code ' ' of a datafield.
This allows for easier handling of the fields.
However, there is a restriction associated with this structure and it is that subfields cannot be repeated
in the same instance. If this is the case, the result will be incorrect.
The record structure has the form:
fields={field_tag:field_instances}
field_instances=[field_instance]
field_instance={field_code:field_value}
'''
def convert_record(old_record):
'''
Convert a record from the format defined in bibrecord to the format defined here
@param old_record: the record as returned from bibrecord.create_record()
@return: a record of the new form
'''
_ = gettext_set_language(CFG_SITE_LANG)
fields = {}
old_tags = old_record.keys()
old_tags.sort()
for old_tag in old_tags:
if int(old_tag) < 11:
#controlfields
new_tag = old_tag
fields[new_tag] = [{' ':old_record[old_tag][0][3]}]
else:
#datafields
old_field_instances = old_record[old_tag]
num_fields = len(old_field_instances)
for i in range(num_fields):
old_field_instance = old_field_instances[i]
ind1 = old_field_instance[1]
if not ind1 or ind1 == ' ':
ind1 = '_'
ind2 = old_field_instance[2]
if not ind2 or ind2 == ' ':
ind2 = '_'
new_tag = old_tag + ind1 + ind2
new_field_instance = {}
for old_subfield in old_field_instance[0]:
new_code = old_subfield[0]
new_value = old_subfield[1]
if new_code in new_field_instance:
try:
raise InvenioBibFormatBfxError(_('Repeating subfield codes in the same instance!'))
except InvenioBibFormatBfxError as exc:
register_exception()
logging.error(exc.message)
# print 'Error: Repeating subfield codes in the same instance!'
new_field_instance[new_code] = new_value
if new_tag not in fields:
fields[new_tag] = []
fields[new_tag].append(new_field_instance)
return fields
def get_bfx_record(recID):
'''
Get a record with a specific recID.
@param recID: the ID of the record
@return: a record in the structure defined here
'''
bfo = BibFormatObject(recID)
return convert_record(bfo.get_record())
def print_bfx_record(record):
'''
Print a record.
'''
tags = record.keys()
tags.sort()
for tag in tags:
field_instances = record[tag]
for field_instance in field_instances:
print(tag, field_instance)
def record_fields_value(record, tag, subfield):
'''
Return a list of all the fields with a certain tag and subfield code.
Works on subfield level.
@param record: a record
@param tag: a 3 or 5 letter tag; required
@param subfield: a subfield code; required
'''
output = []
if tag in record:
for field_instance in record[tag]:
if subfield in field_instance:
output.append(field_instance[subfield])
return output
def record_add_field_instance(record, tag, field_instance):
'''
Add a field_instance to the beginning of the instances of a corresponding tag.
@param record: a record
@param tag: a 3 or 5 letter tag; required
@param field_instance: the field instance to add
@return: None
'''
if tag not in record:
record[tag] = []
record[tag] = [field_instance] + record[tag]
return None
def record_num_parts(record, level):
'''
Count the number of instances or the number of subfields in the whole record.
@param record: record to consider for counting
@param level: either 1 or 2
level=1 - view record on instance level
level=2 - view record on subfield level
@return: the number of parts
'''
num = 0
for part in record_parts(record, level):
num = num + 1
def record_parts(record, level):
'''
An iterator over the instances or subfields of a record.
@param record: record to consider for iterating
@param level: either 1 or 2
- level=1: iterate over instances
- level=2: iterate over subfields
@return: a record structure representing the part (instance or subfield)
'''
if level == 1:
names = record.keys()
names.sort()
for name in names:
old_field_instances = record[name]
for old_field_instance in old_field_instances:
new_record = {}
new_field_instances = []
new_field_instance = {}
for old_field_code in old_field_instance.keys():
new_field_code = old_field_code
new_field_value = old_field_instance[old_field_code]
new_field_instance[new_field_code] = new_field_value
new_field_instances.append(new_field_instance)
new_record[name] = []
new_record[name].extend(new_field_instances)
yield new_record
if level == 2:
names = record.keys()
names.sort()
for name in names:
old_field_instances = record[name]
for old_field_instance in old_field_instances:
old_field_codes = old_field_instance.keys()
old_field_codes.sort()
for old_field_code in old_field_codes:
new_record = {}
new_field_instances = []
new_field_instance = {}
new_field_code = old_field_code
new_field_value = old_field_instance[old_field_code]
new_field_instance[new_field_code] = new_field_value
new_field_instances.append(new_field_instance)
new_record[name] = []
new_record[name].extend(new_field_instances)
yield new_record
def copy(old_record, address=''):
'''
Copy a record by filtering all parts of the old record specified by address
(A better name for the function is filter.)
@param old_record: the initial record
@param address: an address; for examples see modules.formatter.engines.bfx_config.
If no address is specified, return the initial record.
@return: the filtered record
'''
if not old_record:
return {}
tag_pattern, code_pattern, reg_pattern = '', '', ''
expr = re.compile(address_pattern)
match = expr.match(address)
if match:
tag_pattern = match.group('tag')
code_pattern = match.group('code')
reg_pattern = match.group('reg')
if tag_pattern:
tag_pattern = tag_pattern.replace('?','[0-9_\w]')
else:
tag_pattern = r'.*'
if code_pattern:
code_pattern = code_pattern.replace('?','[\w ]')
else:
code_pattern = r'.*'
tag_expr = re.compile(tag_pattern)
code_expr = re.compile(code_pattern)
new_record = {}
for tag in old_record.keys():
tag_match = tag_expr.match(tag)
if tag_match:
if tag_match.end() == len(tag):
old_field_instances = old_record[tag]
new_field_instances = []
for old_field_instance in old_field_instances:
new_field_instance = {}
for old_field_code in old_field_instance.keys():
new_field_code = old_field_code
code_match = code_expr.match(new_field_code)
if code_match:
new_field_value = old_field_instance[old_field_code]
new_field_instance[new_field_code] = new_field_value
if new_field_instance:
new_field_instances.append(new_field_instance)
if new_field_instances:
new_record[tag] = new_field_instances
#in new_record pass all subfields through regexp
if reg_pattern:
for tag in new_record:
field_instances = new_record[tag]
for field_instance in field_instances:
field_codes = field_instance.keys()
for field_code in field_codes:
field_instance[field_code] = pass_through_regexp(field_instance[field_code], reg_pattern)
return new_record
def merge(record1, record2):
'''
Merge two records.
Controlfields with the same tag in record2 as in record1 are ignored.
@param record1: first record to merge
@param record2: second record to merge
@return: the merged record
'''
new_record = {}
if record1:
new_record = copy(record1)
if not record2:
return new_record
for tag in record2.keys():
#append only datafield tags;
#if controlfields conflict, leave first;
old_field_instances = record2[tag]
new_field_instances = []
for old_field_instance in old_field_instances:
new_field_instance = {}
for old_field_code in old_field_instance.keys():
new_field_code = old_field_code
new_field_value = old_field_instance[old_field_code]
new_field_instance[new_field_code] = new_field_value
if new_field_instance:
new_field_instances.append(new_field_instance)
if new_field_instances:
#controlfield
if len(tag) == 3:
if tag not in new_record:
new_record[tag] = []
new_record[tag].extend(new_field_instances)
#datafield
if len(tag) == 5:
if tag not in new_record:
new_record[tag] = []
new_record[tag].extend(new_field_instances)
return new_record
#======================
#Help functions
#=====================
xmlopen = 1
xmlclose = 2
xmlfull = 3
xmlempty = 4
def create_xml_element(name, value='', attrs=None, element_type=xmlfull, level=0):
'''
Create a XML element as string.
@param name: the name of the element
@param value: the element value; default is ''
@param attrs: a dictionary with the element attributes
@param element_type: a constant which defines the type of the output
xmlopen = 1 <element attr="attr_value">
xmlclose = 2 </element>
xmlfull = 3 <element attr="attr_value">value</element>
xmlempty = 4 <element attr="attr_value"/>
@return: a formatted XML string
'''
output = ''
if attrs is None:
attrs = {}
if element_type == xmlempty:
output += '<'+name
for attrname in attrs.keys():
attrvalue = attrs[attrname]
if type(attrvalue) == type(u''):
attrvalue = attrvalue.encode('utf-8')
output += ' %s="%s"' % (attrname, attrvalue)
output += ' />'
if element_type == xmlfull:
output += '<'+name
for attrname in attrs.keys():
attrvalue = attrs[attrname]
if type(attrvalue) == type(u''):
attrvalue = attrvalue.encode('utf-8')
output += ' %s="%s"' % (attrname, attrvalue)
output += '>'
output += value
output += '</'+name+'>'
if element_type == xmlopen:
output += '<'+name
for attrname in attrs.keys():
output += ' '+attrname+'="'+attrs[attrname]+'"'
output += '>'
if element_type == xmlclose:
output += '</'+name+'>'
output = ' '*level + output
if type(output) == type(u''):
output = output.encode('utf-8')
return output
def xml_escape(value):
'''
Escape a string value for use as a xml element or attribute value.
@param value: the string value to escape
@return: escaped value
'''
return saxutils.escape(value)
def xml_unescape(value):
'''
Unescape a string value for use as a xml element.
@param value: the string value to unescape
@return: unescaped value
'''
return saxutils.unescape(value)
def node_has_subelements(node):
'''
Check if a node has any childnodes.
Check for element or text nodes.
@return: True if childnodes exist, False otherwise.
'''
result = False
for node in node.childNodes:
if node.nodeType == Node.ELEMENT_NODE or node.nodeType == Node.TEXT_NODE:
result = True
return result
def get_node_subelement(parent_node, name, namespace = None):
'''
Get the first childnode with specific name and (optional) namespace
@param parent_node: the node to check
@param name: the name to search
@param namespace: An optional namespace URI. This is usually a URL: http://invenio-software.org/
@return: the found node; None otherwise
'''
output = None
for node in parent_node.childNodes:
if node.nodeType == Node.ELEMENT_NODE and node.localName == name and node.namespaceURI == namespace:
output = node
return output
return output
def get_node_value(node):
'''
Get the node value of a node. For use with text nodes.
@param node: a text node
@return: a string of the nodevalue encoded in utf-8
'''
return node.nodeValue.encode('utf-8')
def get_node_namespace(node):
'''
Get node namespace. For use with element nodes.
@param node: an element node
@return: the namespace of the node
'''
return node.namespaceURI
def get_node_name(node):
'''
Get the node value of a node. For use with element nodes.
@param node: an element node
@return: a string of the node name
'''
return node.nodeName
def get_node_attributes(node):
'''
Get attributes of an element node. For use with element nodes
@param node: an element node
@return: a dictionary of the attributes as key:value pairs
'''
attributes = {}
attrs = node.attributes
for attrname in attrs.keys():
attrnode = attrs.get(attrname)
attrvalue = attrnode.nodeValue
attributes[attrname] = attrvalue
return attributes
def pass_through_regexp(value, regexp):
'''
Pass a value through a regular expression.
@param value: a string
@param regexp: a regexp with a group 'value' in it. No group named 'value' will result in an error.
@return: if the string matches the regexp, return named group 'value', otherwise return ''
'''
output = ''
expr = re.compile(regexp)
match = expr.match(value)
if match:
output = match.group('value')
return output
def is_number(value):
'''
Check if a value is a number.
@param value: the value to check
@return: True or False
'''
result = True
try:
float(value)
except ValueError:
result = False
return result
diff --git a/invenio/modules/formatter/format_elements/bfe_arxiv_link.py b/invenio/modules/formatter/format_elements/bfe_arxiv_link.py
new file mode 100644
index 000000000..cffcf7682
--- /dev/null
+++ b/invenio/modules/formatter/format_elements/bfe_arxiv_link.py
@@ -0,0 +1,47 @@
+## This file is part of Invenio.
+## Copyright (C) 2014 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+"""BibFormat element - Links to arXiv"""
+
+from cgi import escape
+from invenio.messages import gettext_set_language
+
+def format_element(bfo, tag="037__", target="_blank"):
+ """
+ Extracts the arXiv preprint information and
+ presents it as a direct link towards arXiv.org
+ """
+ _ = gettext_set_language(bfo.lang)
+ potential_arxiv_ids = bfo.fields(tag)
+ arxiv_id = ""
+ for potential_arxiv_id in potential_arxiv_ids:
+ if potential_arxiv_id.get('9') == 'arXiv' and potential_arxiv_id.get('a', '').startswith('arXiv:'):
+ arxiv_id = potential_arxiv_id['a'][len('arXiv:'):]
+ return '<a href="http://arxiv.org/abs/%s" target="%s" alt="%s">%s</a>' % (
+ escape(arxiv_id, True),
+ escape(target, True),
+ escape(_("This article on arXiv.org"), True),
+ escape(arxiv_id))
+ return ""
+
+def escape_values(bfo):
+ """
+ Called by BibFormat in order to check if output of this element
+ should be escaped.
+ """
+ return 0
+
+
diff --git a/invenio/modules/formatter/format_elements/bfe_authors.py b/invenio/modules/formatter/format_elements/bfe_authors.py
index f4b8872e8..a2133b35e 100644
--- a/invenio/modules/formatter/format_elements/bfe_authors.py
+++ b/invenio/modules/formatter/format_elements/bfe_authors.py
@@ -1,203 +1,215 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2013 CERN.
+## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""BibFormat element - Prints authors
"""
__revision__ = "$Id$"
import re
import six
from urllib import quote
from cgi import escape
from invenio.base.i18n import gettext_set_language
from invenio.legacy.bibauthority.engine import \
get_low_level_recIDs_from_control_no
def format_element(bfo, limit, separator=' ; ',
extension='[...]',
print_links="yes",
print_affiliations='no',
affiliation_prefix=' (',
affiliation_suffix=')',
interactive="no",
highlight="no",
link_author_pages="no",
link_mobile_pages="no",
- relator_code_pattern=None):
+ relator_code_pattern=None,
+ multiple_affiliations="no"):
"""
Prints the list of authors of a record.
@param limit: the maximum number of authors to display
@param separator: the separator between authors.
@param extension: a text printed if more authors than 'limit' exist
@param print_links: if yes, prints the authors as HTML link to their publications
@param print_affiliations: if yes, make each author name followed by its affiliation
@param affiliation_prefix: prefix printed before each affiliation
@param affiliation_suffix: suffix printed after each affiliation
@param interactive: if yes, enable user to show/hide authors when there are too many (html + javascript)
@param highlight: highlights authors corresponding to search query if set to 'yes'
@param link_author_pages: should we link to author pages if print_links in on?
@param link_mobile_pages: should we link to mobile app pages if print_links in on?
@param relator_code_pattern: a regular expression to filter authors based on subfield $4 (relator code)
+ @param multiple_affiliations: whether all affiliations should be displayed
"""
from invenio.config import CFG_BASE_URL, CFG_SITE_RECORD
CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME, \
CFG_BIBAUTHORITY_TYPE_NAMES, \
CFG_BIBAUTHORITY_PREFIX_SEP
_ = gettext_set_language(bfo.lang) # load the right message language
authors = []
authors_1 = bfo.fields('100__', repeatable_subfields_p=True)
authors_2 = bfo.fields('700__', repeatable_subfields_p=True)
authors.extend(authors_1)
authors.extend(authors_2)
# make unique string per key
for author in authors:
if 'a' in author:
author['a'] = author['a'][0]
- if 'u' in author:
+ if 'u' in author and multiple_affiliations == 'no':
author['u'] = author['u'][0]
+ if 'v' in author and multiple_affiliations == 'no':
+ author['v'] = author['v'][0]
pattern = '%s' + CFG_BIBAUTHORITY_PREFIX_SEP + "("
for control_no in author.get('0', []):
if pattern % (CFG_BIBAUTHORITY_TYPE_NAMES["INSTITUTE"]) in control_no:
author['u0'] = control_no # overwrite if multiples
elif pattern % (CFG_BIBAUTHORITY_TYPE_NAMES["AUTHOR"]) in control_no:
author['a0'] = control_no # overwrite if multiples
if relator_code_pattern:
p = re.compile(relator_code_pattern)
authors = filter(lambda x: p.match(x.get('4', '')), authors)
nb_authors = len(authors)
bibrec_id = bfo.control_field("001")
# Process authors to add link, highlight and format affiliation
for author in authors:
if 'a' in author:
if highlight == 'yes':
from invenio.modules.formatter import utils as bibformat_utils
author['a'] = bibformat_utils.highlight(author['a'],
bfo.search_pattern)
if print_links.lower() == "yes":
if link_author_pages == "yes":
author['a'] = '<a rel="author" href="' + CFG_BASE_URL + \
'/author/profile/' + quote(author['a']) + \
'?recid=' + bibrec_id + \
'&ln=' + bfo.lang + \
'">' + escape(author['a']) + '</a>'
elif link_mobile_pages == 'yes':
author['a'] = '<a rel="external" href="#page=search' + \
'&amp;f=author&amp;p=' + quote(author['a']) + \
'">' + escape(author['a']) + '</a>'
else:
auth_coll_param = ''
if 'a0' in author:
recIDs = get_low_level_recIDs_from_control_no(author['a0'])
if len(recIDs):
auth_coll_param = '&amp;c=' + \
CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME
author['a'] = '<a href="' + CFG_BASE_URL + \
'/search?f=author&amp;p=' + quote(author['a']) + \
auth_coll_param + \
'&amp;ln=' + bfo.lang + \
'">' + escape(author['a']) + '</a>'
- if 'u' in author:
+ if 'u' in author or 'v' in author:
if print_affiliations == "yes":
if 'u0' in author:
recIDs = get_low_level_recIDs_from_control_no(author['u0'])
# if there is more than 1 recID, clicking on link and
# thus displaying the authority record's page should
# contain a warning that there are multiple authority
# records with the same control number
+ if isinstance(author['u'], (list, tuple)):
+ author['u'] = author['u'][0]
if len(recIDs):
author['u'] = '<a href="' + CFG_BASE_URL + '/' + CFG_SITE_RECORD + '/' + \
str(recIDs[0]) + \
'?ln=' + bfo.lang + \
'">' + author['u'] + '</a>'
- author['u'] = affiliation_prefix + author['u'] + \
+ if not 'u' in author and 'v' in author:
+ author['u'] = author['v']
+ if isinstance(author['u'], (list, tuple)):
+ author['u'] = ' '.join([affiliation_prefix + aff + \
+ affiliation_suffix for aff in author['u']])
+ else:
+ author['u'] = affiliation_prefix + author['u'] + \
affiliation_suffix
# Flatten author instances
if print_affiliations == 'yes':
authors = [author.get('a', '') + author.get('u', '')
for author in authors]
else:
authors = [author.get('a', '')
for author in authors]
if limit.isdigit() and nb_authors > int(limit) and interactive != "yes":
return separator.join(authors[:int(limit)]) + extension
elif limit.isdigit() and nb_authors > int(limit) and interactive == "yes":
out = '<a name="show_hide" />'
out += separator.join(authors[:int(limit)])
out += '<span id="more_%s" style="">' % bibrec_id + separator + \
separator.join(authors[int(limit):]) + '</span>'
out += ' <span id="extension_%s"></span>' % bibrec_id
out += ' <small><i><a id="link_%s" href="#" style="color:rgb(204,0,0);"></a></i></small>' % bibrec_id
out += '''
<script type="text/javascript">
$('#link_%(recid)s').click(function(event) {
event.preventDefault();
var more = document.getElementById('more_%(recid)s');
var link = document.getElementById('link_%(recid)s');
var extension = document.getElementById('extension_%(recid)s');
if (more.style.display=='none'){
more.style.display = '';
extension.style.display = 'none';
link.innerHTML = "%(show_less)s"
} else {
more.style.display = 'none';
extension.style.display = '';
link.innerHTML = "%(show_more)s"
}
link.style.color = "rgb(204,0,0);"
});
function set_up_%(recid)s(){
var extension = document.getElementById('extension_%(recid)s');
extension.innerHTML = "%(extension)s";
$('#link_%(recid)s').click();
}
</script>
''' % {'show_less':_("Hide"),
'show_more':_("Show all %(x_num)i authors", x_num=nb_authors),
'extension':extension,
'recid': bibrec_id}
out += '<script type="text/javascript">set_up_%s()</script>' % bibrec_id
return out
elif nb_authors > 0:
return separator.join(authors)
def escape_values(bfo):
"""
Called by BibFormat in order to check if output of this element
should be escaped.
"""
return 0
diff --git a/invenio/modules/formatter/format_elements/bfe_qrcode.py b/invenio/modules/formatter/format_elements/bfe_qrcode.py
index e0a1c63a1..e8a8ae407 100644
--- a/invenio/modules/formatter/format_elements/bfe_qrcode.py
+++ b/invenio/modules/formatter/format_elements/bfe_qrcode.py
@@ -1,83 +1,83 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN.
+## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""BibFormat element - QR code generator """
from invenio.config import CFG_SITE_SECURE_URL, CFG_WEBDIR, CFG_SITE_RECORD
from invenio.utils.hash import md5
import os
try:
import qrcode
from PIL import Image
HAS_QR = True
except ImportError:
HAS_QR = False
if not HAS_QR:
from warnings import warn
warn("Please install qrcode <https://pypi.python.org/pypi/qrcode> and "
- "Python Imaging Library (PIL) "
+ "Pillow (Python Imaging Library) "
"<https://pypi.python.org/pypi/Pillow/>.")
def _get_record_hash(link):
"""
Generate a record hash including CFG_SITE_URL so that
if CFG_SITE_URL is updated, the QR-code image is invalidated.
"""
return md5(link).hexdigest()[:8].lower()
def format_element(bfo, width="100"):
"""
Generate a QR-code image linking to the current record.
@param width: Width of QR-code image.
"""
if not HAS_QR:
return ""
width = int(width)
bibrec_id = bfo.control_field("001")
link = "%s/%s/%s" % (CFG_SITE_SECURE_URL, CFG_SITE_RECORD, bibrec_id)
hash_val = _get_record_hash(link)
filename = "%s_%s.png" % (bibrec_id, hash_val)
filename_url = "/img/qrcodes/%s" % filename
filename_path = os.path.join(CFG_WEBDIR, "img/qrcodes/%s" % filename)
if not os.path.exists(filename_path):
if not os.path.exists(os.path.dirname(filename_path)):
os.makedirs(os.path.dirname(filename_path))
img = qrcode.make(link)
img._img = img._img.convert("RGBA")
img._img = img._img.resize((width, width), Image.ANTIALIAS)
img.save(filename_path, "PNG")
return """<img src="%s" width="%s" />""" % (filename_url, width)
def escape_values(dummy_bfo):
"""
Called by BibFormat in order to check if output of this element
should be escaped.
"""
return 0
diff --git a/invenio/modules/formatter/format_templates/Authority_HTML_brief.bft b/invenio/modules/formatter/format_templates/Authority_HTML_brief.bft
old mode 100755
new mode 100644
diff --git a/invenio/modules/formatter/format_templates/Authority_HTML_detailed.bft b/invenio/modules/formatter/format_templates/Authority_HTML_detailed.bft
old mode 100755
new mode 100644
diff --git a/invenio/modules/indexer/tokenizers/BibIndexAuthorCountTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexAuthorCountTokenizer.py
index 070e2e424..6d2d2d048 100644
--- a/invenio/modules/indexer/tokenizers/BibIndexAuthorCountTokenizer.py
+++ b/invenio/modules/indexer/tokenizers/BibIndexAuthorCountTokenizer.py
@@ -1,46 +1,50 @@
# -*- coding:utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2010, 2011, 2012 CERN.
+## Copyright (C) 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""BibIndexAuthorCountTokenizer: counts number of authors for any publication
given by recID. Will look at tags: '100_a' and '700_a' which are:
'first author name' and 'additional author name'.
"""
-
from invenio.legacy.bibindex.engine_utils import get_field_count
-from invenio.modules.indexer.tokenizers.BibIndexEmptyTokenizer import BibIndexEmptyTokenizer
-
+from invenio.modules.indexer.tokenizers.BibIndexMultiFieldTokenizer import BibIndexMultiFieldTokenizer
-class BibIndexAuthorCountTokenizer(BibIndexEmptyTokenizer):
+class BibIndexAuthorCountTokenizer(BibIndexMultiFieldTokenizer):
"""
- Returns a number of authors who created a publication with given recID in the database.
+ Returns a number of authors who created a publication
+ with given recID in the database.
+
+ Takes recID of the record as an argument to tokenizing function.
+ Calculates terms based on information from multiple tags.
+ For more information on this type of tokenizers take a look on
+ BibIndexAuthorCountTokenizer base class.
"""
def __init__(self, stemming_language = None, remove_stopwords = False, remove_html_markup = False, remove_latex_markup = False):
self.tags = ['100__a', '700__a']
def tokenize(self, recID):
"""Uses get_field_count from bibindex.engine_utils
for finding a number of authors of a publication and pass it in the list"""
return [str(get_field_count(recID, self.tags)),]
def get_tokenizing_function(self, wordtable_type):
return self.tokenize
diff --git a/modules/bibauthorid/lib/bibauthorid_dbinterface_unit_tests.py b/invenio/modules/indexer/tokenizers/BibIndexDOITokenizer.py
similarity index 50%
copy from modules/bibauthorid/lib/bibauthorid_dbinterface_unit_tests.py
copy to invenio/modules/indexer/tokenizers/BibIndexDOITokenizer.py
index 873a57410..2aea35827 100644
--- a/modules/bibauthorid/lib/bibauthorid_dbinterface_unit_tests.py
+++ b/invenio/modules/indexer/tokenizers/BibIndexDOITokenizer.py
@@ -1,41 +1,36 @@
-# -*- coding: utf-8 -*-
+# -*- coding:utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
+## Copyright (C) 2013 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+from invenio.bibindex_tokenizers.BibIndexFilteringTokenizer import BibIndexFilteringTokenizer
-"""Unit tests for the search engine."""
-__revision__ = \
- "$Id$"
+class BibIndexDOITokenizer(BibIndexFilteringTokenizer):
+ """
+ Filtering tokenizer which tokenizes DOI tag (0247_a)
+ only if "0247_2" tag is present and its value equals "DOI"
+ and 909C4a tag without any constraints.
+ """
-from itertools import chain
-from invenio.testutils import InvenioTestCase, make_test_suite, run_test_suite
-from invenio.bibauthorid_cluster_set import ClusterSet
+ def __init__(self, stemming_language=None, remove_stopwords=False, remove_html_markup=False, remove_latex_markup=False):
+ self.rules = (('0247_a', '2', 'DOI'), ('909C4a', '', ''))
-class TestDummy(InvenioTestCase):
- def setUp(self):
- pass
-
- def test_one(self):
- pass
-
-TEST_SUITE = make_test_suite(TestDummy)
-
-if __name__ == "__main__":
- run_test_suite(TEST_SUITE, warn_user=True)
+ def get_tokenizing_function(self, wordtable_type):
+ """Returns proper tokenizing function"""
+ return self.tokenize
\ No newline at end of file
diff --git a/invenio/modules/indexer/tokenizers/BibIndexDefaultTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexDefaultTokenizer.py
index f46997f98..0431b9767 100644
--- a/invenio/modules/indexer/tokenizers/BibIndexDefaultTokenizer.py
+++ b/invenio/modules/indexer/tokenizers/BibIndexDefaultTokenizer.py
@@ -1,165 +1,165 @@
# -*- coding:utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2010, 2011, 2012 CERN.
+## Copyright (C) 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""BibIndexDefaultTokenizer: useful for almost all indexes.
It performs standard tokenization. It splits phrases into words/pairs or doesnt split at all, strips accents,
removes alphanumeric characters and html and latex markup if we want to. Also can stem words for a given language.
"""
from invenio.legacy.bibindex.engine_config import \
CFG_BIBINDEX_INDEX_TABLE_TYPE
from invenio.utils.html import remove_html_markup
from invenio.utils.text import wash_for_utf8, strip_accents
from invenio.legacy.bibindex.engine_washer import \
lower_index_term, remove_latex_markup, \
apply_stemming, remove_stopwords, length_check
from invenio.legacy.bibindex.engine_utils import latex_formula_re, \
re_block_punctuation_begin, \
re_block_punctuation_end, \
re_punctuation, \
re_separators, \
re_arxiv
-from invenio.modules.indexer.tokenizers.BibIndexTokenizer import BibIndexTokenizer
+from invenio.modules.indexer.tokenizers.BibIndexStringTokenizer import BibIndexStringTokenizer
-class BibIndexDefaultTokenizer(BibIndexTokenizer):
+class BibIndexDefaultTokenizer(BibIndexStringTokenizer):
"""
It's a standard tokenizer. It is useful for most of the indexes.
Its behaviour depends on stemming, remove stopwords, remove html markup and remove latex markup parameters.
"""
def __init__(self, stemming_language = None, remove_stopwords = False, remove_html_markup = False, remove_latex_markup = False):
"""initialization"""
self.stemming_language = stemming_language
self.remove_stopwords = remove_stopwords
self.remove_html_markup = remove_html_markup
self.remove_latex_markup = remove_latex_markup
def get_tokenizing_function(self, wordtable_type):
"""Picks correct tokenize_for_xxx function depending on type of tokenization (wordtable_type)"""
if wordtable_type == CFG_BIBINDEX_INDEX_TABLE_TYPE["Words"]:
return self.tokenize_for_words
elif wordtable_type == CFG_BIBINDEX_INDEX_TABLE_TYPE["Pairs"]:
return self.tokenize_for_pairs
elif wordtable_type == CFG_BIBINDEX_INDEX_TABLE_TYPE["Phrases"]:
return self.tokenize_for_phrases
def tokenize_for_words(self, phrase):
"""Return list of words found in PHRASE. Note that the phrase is
split into groups depending on the alphanumeric characters and
punctuation characters definition present in the config file.
"""
words = {}
formulas = []
if self.remove_html_markup and phrase.find("</") > -1:
phrase = remove_html_markup(phrase)
if self.remove_latex_markup:
formulas = latex_formula_re.findall(phrase)
phrase = remove_latex_markup(phrase)
phrase = latex_formula_re.sub(' ', phrase)
phrase = wash_for_utf8(phrase)
phrase = lower_index_term(phrase)
# 1st split phrase into blocks according to whitespace
for block in strip_accents(phrase).split():
# 2nd remove leading/trailing punctuation and add block:
block = re_block_punctuation_begin.sub("", block)
block = re_block_punctuation_end.sub("", block)
if block:
stemmed_block = remove_stopwords(block, self.remove_stopwords)
stemmed_block = length_check(stemmed_block)
stemmed_block = apply_stemming(stemmed_block, self.stemming_language)
if stemmed_block:
words[stemmed_block] = 1
if re_arxiv.match(block):
# special case for blocks like `arXiv:1007.5048' where
# we would like to index the part after the colon
# regardless of dot or other punctuation characters:
words[block.split(':', 1)[1]] = 1
# 3rd break each block into subblocks according to punctuation and add subblocks:
for subblock in re_punctuation.split(block):
stemmed_subblock = remove_stopwords(subblock, self.remove_stopwords)
stemmed_subblock = length_check(stemmed_subblock)
stemmed_subblock = apply_stemming(stemmed_subblock, self.stemming_language)
if stemmed_subblock:
words[stemmed_subblock] = 1
# 4th break each subblock into alphanumeric groups and add groups:
for alphanumeric_group in re_separators.split(subblock):
stemmed_alphanumeric_group = remove_stopwords(alphanumeric_group, self.remove_stopwords)
stemmed_alphanumeric_group = length_check(stemmed_alphanumeric_group)
stemmed_alphanumeric_group = apply_stemming(stemmed_alphanumeric_group, self.stemming_language)
if stemmed_alphanumeric_group:
words[stemmed_alphanumeric_group] = 1
for block in formulas:
words[block] = 1
return words.keys()
def tokenize_for_pairs(self, phrase):
"""Return list of words found in PHRASE. Note that the phrase is
split into groups depending on the alphanumeric characters and
punctuation characters definition present in the config file.
"""
words = {}
if self.remove_html_markup and phrase.find("</") > -1:
phrase = remove_html_markup(phrase)
if self.remove_latex_markup:
phrase = remove_latex_markup(phrase)
phrase = latex_formula_re.sub(' ', phrase)
phrase = wash_for_utf8(phrase)
phrase = lower_index_term(phrase)
# 1st split phrase into blocks according to whitespace
last_word = ''
for block in strip_accents(phrase).split():
# 2nd remove leading/trailing punctuation and add block:
block = re_block_punctuation_begin.sub("", block)
block = re_block_punctuation_end.sub("", block)
if block:
block = remove_stopwords(block, self.remove_stopwords)
block = length_check(block)
block = apply_stemming(block, self.stemming_language)
# 3rd break each block into subblocks according to punctuation and add subblocks:
for subblock in re_punctuation.split(block):
subblock = remove_stopwords(subblock, self.remove_stopwords)
subblock = length_check(subblock)
subblock = apply_stemming(subblock, self.stemming_language)
if subblock:
# 4th break each subblock into alphanumeric groups and add groups:
for alphanumeric_group in re_separators.split(subblock):
alphanumeric_group = remove_stopwords(alphanumeric_group, self.remove_stopwords)
alphanumeric_group = length_check(alphanumeric_group)
alphanumeric_group = apply_stemming(alphanumeric_group, self.stemming_language)
if alphanumeric_group:
if last_word:
words['%s %s' % (last_word, alphanumeric_group)] = 1
last_word = alphanumeric_group
return words.keys()
def tokenize_for_phrases(self, phrase):
"""Return list of phrases found in PHRASE. Note that the phrase is
split into groups depending on the alphanumeric characters and
punctuation characters definition present in the config file.
"""
phrase = wash_for_utf8(phrase)
return [phrase]
diff --git a/invenio/modules/indexer/tokenizers/BibIndexEmptyTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexEmptyTokenizer.py
index 2c5df8646..27c9fc647 100644
--- a/invenio/modules/indexer/tokenizers/BibIndexEmptyTokenizer.py
+++ b/invenio/modules/indexer/tokenizers/BibIndexEmptyTokenizer.py
@@ -1,61 +1,66 @@
# -*- coding:utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2010, 2011, 2012 CERN.
+## Copyright (C) 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-"""BibIndexEmptyTokenizer: useful for situations when we want to apply tokenizer
- for consistency, but we don't want to have any results from it.
"""
+ BibIndexEmptyTokenizer.
+ It's a really lazy tokenizer and doesn't do anything.
+"""
from invenio.legacy.bibindex.engine_config import CFG_BIBINDEX_INDEX_TABLE_TYPE
-from invenio.modules.indexer.tokenizers.BibIndexTokenizer import BibIndexTokenizer
+from invenio.modules.indexer.tokenizers.BibIndexStringTokenizer import BibIndexStringTokenizer
+class BibIndexEmptyTokenizer(BibIndexStringTokenizer):
+ """
+ BibIndexEmptyTokenizer doesn't do anything.
+ Irrespective of input to tokenizing function it
+ always returns empty list.
-class BibIndexEmptyTokenizer(BibIndexTokenizer):
- """Empty tokenizer do nothing.
- It returns empty lists for tokenize_for_words, tokenize_for_pairs and tokenize_for_phrases methods.
+ Can be used in some default cases or when we want to
+ turn off specific index.
"""
def __init__(self, stemming_language = None, remove_stopwords = False, remove_html_markup = False, remove_latex_markup = False):
"""@param stemming_language: dummy
@param remove_stopwords: dummy
@param remove_html_markup: dummy
@param remove_latex_markup: dummy
"""
pass
def get_tokenizing_function(self, wordtable_type):
"""Picks correct tokenize_for_xxx function depending on type of tokenization (wordtable_type)"""
if wordtable_type == CFG_BIBINDEX_INDEX_TABLE_TYPE["Words"]:
return self.tokenize_for_words
elif wordtable_type == CFG_BIBINDEX_INDEX_TABLE_TYPE["Pairs"]:
return self.tokenize_for_pairs
elif wordtable_type == CFG_BIBINDEX_INDEX_TABLE_TYPE["Phrases"]:
return self.tokenize_for_phrases
def tokenize_for_words(self, phrase):
return []
def tokenize_for_pairs(self, phrase):
return []
def tokenize_for_phrases(self, phrase):
return []
diff --git a/invenio/modules/indexer/tokenizers/BibIndexFiletypeTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexFilenameTokenizer.py
similarity index 58%
copy from invenio/modules/indexer/tokenizers/BibIndexFiletypeTokenizer.py
copy to invenio/modules/indexer/tokenizers/BibIndexFilenameTokenizer.py
index cbf735979..a6ed8cb86 100644
--- a/invenio/modules/indexer/tokenizers/BibIndexFiletypeTokenizer.py
+++ b/invenio/modules/indexer/tokenizers/BibIndexFilenameTokenizer.py
@@ -1,63 +1,69 @@
# -*- coding:utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2010, 2011, 2012, 2013 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-"""BibIndexFiletypeTokenizer: 'tokenizes' for file extensions.
- Tokenizer is adapted to work with recjson and its get_record function.
+"""BibIndexFilenameTokenizer: 'tokenizes' finds file names.
+ Tokenizer is adapted to work with bibfield and its get_record function.
"""
-from invenio.modules.indexer.tokenizers.BibIndexEmptyTokenizer import BibIndexEmptyTokenizer
+from invenio.bibindex_tokenizers.BibIndexRecJsonTokenizer import BibIndexRecJsonTokenizer
-class BibIndexFiletypeTokenizer(BibIndexEmptyTokenizer):
+class BibIndexFilenameTokenizer(BibIndexRecJsonTokenizer):
"""
- Tokenizes for file extensions.
- Tokenizer is adapted to work with recjson and its get_record function.
+ Tokenizes for file names.
+ Tokenizer is adapted to work with bibfield and its get_record function.
It accepts as an input a record created by a get_record function:
- from invenio.modules.records.api import get_record
+ from bibfield import get_record
record16 = get_record(16)
- tokenizer = BibIndexFiletypeTokenizer()
+ tokenizer = BibIndexFilenameTokenizer()
new_words = tokenizer.tokenize(record16)
+
+ Example of new_words:
+ 'thesis.ps.gz' -> ['thesis', 'thesis.ps', 'thesis.ps.gz']
"""
- def __init__(self, stemming_language = None, remove_stopwords = False, remove_html_markup = False, remove_latex_markup = False):
+ def __init__(self, stemming_language = None,
+ remove_stopwords = False,
+ remove_html_markup = False,
+ remove_latex_markup = False):
pass
def tokenize(self, record):
- """'record' is a recjson record.
+ """'record' is a recjson record from bibfield.
- Function uses derived field 'filetypes'
+ Function uses derived field 'filenames'
from the record.
@param urls: recjson record
"""
values = []
try:
- if 'filetypes' in record:
- values = record['filetypes']
+ if 'filenames' in record:
+ values = record['filenames']
except KeyError:
pass
except TypeError:
return []
return values
def get_tokenizing_function(self, wordtable_type):
return self.tokenize
diff --git a/invenio/modules/indexer/tokenizers/BibIndexFiletypeTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexFiletypeTokenizer.py
index cbf735979..1eca54b62 100644
--- a/invenio/modules/indexer/tokenizers/BibIndexFiletypeTokenizer.py
+++ b/invenio/modules/indexer/tokenizers/BibIndexFiletypeTokenizer.py
@@ -1,63 +1,62 @@
# -*- coding:utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2010, 2011, 2012, 2013 CERN.
+## Copyright (C) 2010, 2011, 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""BibIndexFiletypeTokenizer: 'tokenizes' for file extensions.
Tokenizer is adapted to work with recjson and its get_record function.
"""
+from invenio.modules.indexer.tokenizers.BibIndexRecJsonTokenizer import BibIndexRecJsonTokenizer
-from invenio.modules.indexer.tokenizers.BibIndexEmptyTokenizer import BibIndexEmptyTokenizer
-
-class BibIndexFiletypeTokenizer(BibIndexEmptyTokenizer):
+class BibIndexFiletypeTokenizer(BibIndexRecJsonTokenizer):
"""
Tokenizes for file extensions.
Tokenizer is adapted to work with recjson and its get_record function.
It accepts as an input a record created by a get_record function:
from invenio.modules.records.api import get_record
record16 = get_record(16)
tokenizer = BibIndexFiletypeTokenizer()
new_words = tokenizer.tokenize(record16)
"""
def __init__(self, stemming_language = None, remove_stopwords = False, remove_html_markup = False, remove_latex_markup = False):
pass
def tokenize(self, record):
"""'record' is a recjson record.
Function uses derived field 'filetypes'
from the record.
@param urls: recjson record
"""
values = []
try:
if 'filetypes' in record:
values = record['filetypes']
except KeyError:
pass
except TypeError:
return []
return values
def get_tokenizing_function(self, wordtable_type):
return self.tokenize
diff --git a/invenio/modules/indexer/tokenizers/BibIndexFilteringTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexFilteringTokenizer.py
new file mode 100644
index 000000000..347d53ab8
--- /dev/null
+++ b/invenio/modules/indexer/tokenizers/BibIndexFilteringTokenizer.py
@@ -0,0 +1,78 @@
+# -*- coding:utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2013, 2014 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+from invenio.search_engine import get_record
+from invenio.bibrecord import record_get_field_instances
+from invenio.bibindex_tokenizers.BibIndexMultiFieldTokenizer import BibIndexMultiFieldTokenizer
+
+class BibIndexFilteringTokenizer(BibIndexMultiFieldTokenizer):
+ """
+ This tokenizer would tokenize phrases from tag
+ only if another tag was present in the record's metadata,
+ for example it would tokenize phrases from 100__a
+ only if 100__u was found in the record's metadata.
+
+ This tokenizer is abstract and it shouldn't be used
+ for indexes. Insted of using this tokenizer one can
+ create another tokenizer iheriting after this one.
+
+ To create new tokenizer based on BibIndexFilteringTokenizer
+ you need to specify rules of tokenizing in self.rules
+ property.
+
+ Examples:
+ 1) Let's say we want to tokenize data only from 100__a if 100__u is present:
+ set: self.rules = (('100__a', 'u', ''),)
+ 2) We want to tokenize data from '0247_a' if '0247_2' == 'DOI':
+ set: self.rules = (('0247_2', '2', 'DOI'),)
+ 3) We want to tokenize data from '0247_a' if '0247_2' == 'DOI' and all data
+ from '100__a' with no constraints:
+ set: self.rules = (('0247_2', '2', 'DOI'), ('100__a', '', ''))
+
+ Definition of 'rules' tuple:
+ (tag_to_take_phrases_from, value_of_sub_tag or '', necessary_value_of_sub_tag or '')
+
+ Note: there is no get_tokenizing_function() to make this tokenizer abstract.
+ """
+
+ def __init__(self, stemming_language=None, remove_stopwords=False, remove_html_markup=False, remove_latex_markup=False):
+ self.rules = ()
+
+
+ def tokenize(self, recID):
+ phrases = []
+ try:
+ rec = get_record(recID)
+
+ for rule in self.rules:
+ tag_to_index, necessary_tag, necessary_value = rule
+ core_tag = tag_to_index[0:3]
+ ind = tag_to_index[3:5]
+ sub_tag = tag_to_index[5]
+
+ fields = [dict(instance[0]) for instance in record_get_field_instances(rec, core_tag, ind[0], ind[1])]
+ for field in fields:
+ tag_condition = necessary_tag and field.has_key(necessary_tag) or necessary_tag == ''
+ value_condition = necessary_value and field.get(necessary_tag, '') == necessary_value or \
+ necessary_value == ''
+ if tag_condition and field.has_key(sub_tag) and value_condition:
+ phrases.append(field[sub_tag])
+ return phrases
+ except KeyError:
+ return []
+ return phrases
diff --git a/invenio/modules/indexer/tokenizers/BibIndexItemCountTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexItemCountTokenizer.py
index 5176886db..194f91453 100644
--- a/invenio/modules/indexer/tokenizers/BibIndexItemCountTokenizer.py
+++ b/invenio/modules/indexer/tokenizers/BibIndexItemCountTokenizer.py
@@ -1,49 +1,48 @@
# -*- coding:utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2010, 2011, 2012 CERN.
+## Copyright (C) 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""BibIndexItemCountTokenizer: counts the number of copies of a book which is
owned by the library in the real world.
"""
-from invenio.modules.indexer.tokenizers.BibIndexEmptyTokenizer import BibIndexEmptyTokenizer
+from invenio.modules.indexer.tokenizers.BibIndexRecJsonTokenizer import BibIndexRecJsonTokenizer
-
-class BibIndexItemCountTokenizer(BibIndexEmptyTokenizer):
+class BibIndexItemCountTokenizer(BibIndexRecJsonTokenizer):
"""
Returns a number of copies of a book which is owned by the library.
"""
def __init__(self, stemming_language = None, remove_stopwords = False, remove_html_markup = False, remove_latex_markup = False):
pass
def tokenize(self, record):
"""Tokenizes for number of copies of a book in the 'real' library"""
count = 0
try:
count = record['number_of_copies']
except KeyError:
pass
except TypeError:
return []
return [str(count)]
def get_tokenizing_function(self, wordtable_type):
return self.tokenize
diff --git a/invenio/modules/indexer/tokenizers/BibIndexJournalTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexJournalTokenizer.py
index 75890b789..2e0cc1b21 100644
--- a/invenio/modules/indexer/tokenizers/BibIndexJournalTokenizer.py
+++ b/invenio/modules/indexer/tokenizers/BibIndexJournalTokenizer.py
@@ -1,108 +1,110 @@
# -*- coding:utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2010, 2011, 2012 CERN.
+## Copyright (C) 2010, 2011, 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""BibIndexJournalTokenizer: useful for journal index.
Agregates info about journal in a specific way given by its variable
journal_pubinfo_standard_form.
Behaves in the same way for all index table types:
- Words
- Pairs
- Phrases
"""
from invenio.legacy.dbquery import run_sql
-from invenio.modules.indexer.tokenizers.BibIndexEmptyTokenizer import BibIndexEmptyTokenizer
+from invenio.modules.indexer.tokenizers.BibIndexMultiFieldTokenizer import BibIndexMultiFieldTokenizer
from invenio.config import \
CFG_CERN_SITE, \
CFG_INSPIRE_SITE
-
if CFG_CERN_SITE:
CFG_JOURNAL_TAG = '773__%'
CFG_JOURNAL_PUBINFO_STANDARD_FORM = "773__p 773__v (773__y) 773__c"
CFG_JOURNAL_PUBINFO_STANDARD_FORM_REGEXP_CHECK = r'^\w.*\s\w.*\s\(\d+\)\s\w.*$'
elif CFG_INSPIRE_SITE:
CFG_JOURNAL_TAG = '773__%'
CFG_JOURNAL_PUBINFO_STANDARD_FORM = "773__p,773__v,773__c"
CFG_JOURNAL_PUBINFO_STANDARD_FORM_REGEXP_CHECK = r'^\w.*,\w.*,\w.*$'
else:
CFG_JOURNAL_TAG = '909C4%'
CFG_JOURNAL_PUBINFO_STANDARD_FORM = "909C4p 909C4v (909C4y) 909C4c"
CFG_JOURNAL_PUBINFO_STANDARD_FORM_REGEXP_CHECK = r'^\w.*\s\w.*\s\(\d+\)\s\w.*$'
-
-class BibIndexJournalTokenizer(BibIndexEmptyTokenizer):
+class BibIndexJournalTokenizer(BibIndexMultiFieldTokenizer):
"""
- Tokenizer for journal index. It returns joined title/volume/year/page as a word from journal tag.
- (In fact it's an aggregator.)
+ Tokenizer for journal index.
+ Returns joined title/volume/year/page as a word from journal tag.
+
+ Tokenizer works on multiple tags.
+ For more information on tokenizers working on per-record basis
+ take a look on BibIndexJournalTokenizer base class.
"""
def __init__(self, stemming_language = None, remove_stopwords = False, remove_html_markup = False, remove_latex_markup = False):
self.tag = CFG_JOURNAL_TAG
self.journal_pubinfo_standard_form = CFG_JOURNAL_PUBINFO_STANDARD_FORM
self.journal_pubinfo_standard_form_regexp_check = CFG_JOURNAL_PUBINFO_STANDARD_FORM_REGEXP_CHECK
def tokenize(self, recID):
"""
Special procedure to extract words from journal tags. Joins
title/volume/year/page into a standard form that is also used for
citations.
"""
# get all journal tags/subfields:
bibXXx = "bib" + self.tag[0] + self.tag[1] + "x"
bibrec_bibXXx = "bibrec_" + bibXXx
query = """SELECT bb.field_number,b.tag,b.value FROM %s AS b, %s AS bb
WHERE bb.id_bibrec=%%s
AND bb.id_bibxxx=b.id AND tag LIKE %%s""" % (bibXXx, bibrec_bibXXx)
res = run_sql(query, (recID, self.tag))
# construct journal pubinfo:
dpubinfos = {}
for row in res:
nb_instance, subfield, value = row
if subfield.endswith("c"):
# delete pageend if value is pagestart-pageend
# FIXME: pages may not be in 'c' subfield
value = value.split('-', 1)[0]
if nb_instance in dpubinfos:
dpubinfos[nb_instance][subfield] = value
else:
dpubinfos[nb_instance] = {subfield: value}
# construct standard format:
lwords = []
for dpubinfo in dpubinfos.values():
# index all journal subfields separately
for tag, val in dpubinfo.items():
lwords.append(val)
# index journal standard format:
pubinfo = self.journal_pubinfo_standard_form
for tag, val in dpubinfo.items():
pubinfo = pubinfo.replace(tag, val)
if self.tag[:-1] in pubinfo:
# some subfield was missing, do nothing
pass
else:
lwords.append(pubinfo)
# return list of words and pubinfos:
return lwords
def get_tokenizing_function(self, wordtable_type):
return self.tokenize
diff --git a/invenio/modules/indexer/tokenizers/BibIndexMultiFieldTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexMultiFieldTokenizer.py
new file mode 100644
index 000000000..0633d0696
--- /dev/null
+++ b/invenio/modules/indexer/tokenizers/BibIndexMultiFieldTokenizer.py
@@ -0,0 +1,81 @@
+# -*- coding:utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2010, 2011, 2012 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+"""
+ BibIndexMultiFieldTokenizer.
+ Base class for tokenizers that work on more than one field
+ and possibly on more than one phrase at a time.
+"""
+
+
+from invenio.bibindex_engine_config import CFG_BIBINDEX_INDEX_TABLE_TYPE
+from invenio.bibindex_tokenizers.BibIndexTokenizer import BibIndexTokenizer
+
+
+
+class BibIndexMultiFieldTokenizer(BibIndexTokenizer):
+ """
+ BibIndexMultiFieldTokenizer is an abstract tokenizer.
+ It should be used only for inheritance.
+
+ This tokenizer should be a base class for more complicated
+ tokenizers which tokenizing functions perform calculations
+ on per record basis and NOT per string basis (look for
+ BibIndexDefaultTokenizer if you want to know more about the
+ latter type of tokenization).
+
+ Tokenizing functions take as an argument recID of the record
+ we want to perform calculations on.
+ Example:
+
+ class BibIndexComplicatedTokenizer(BibIndexMultiFieldTokenizer):
+ (...)
+ recID = 10
+ a = BibIndexComplicatedTokenizer()
+ res = a.tokenize_for_words(recID)
+
+ Good examples of MultiFieldTokenizer are JournalTokenizer and
+ AuthorCountTokenizer.
+ Both return results after processing more than one field/tag
+ of the record (for more information check these tokenizers).
+
+ """
+
+ def __init__(self, stemming_language = None, remove_stopwords = False, remove_html_markup = False, remove_latex_markup = False):
+ """@param stemming_language: dummy
+ @param remove_stopwords: dummy
+ @param remove_html_markup: dummy
+ @param remove_latex_markup: dummy
+ """
+ pass
+
+
+ def get_tokenizing_function(self, wordtable_type):
+ """Picks correct tokenize_for_xxx function depending on type of tokenization (wordtable_type)"""
+ raise NotImplementedError
+
+
+ def tokenize_for_words(self, recid):
+ raise NotImplementedError
+
+ def tokenize_for_pairs(self, recid):
+ raise NotImplementedError
+
+ def tokenize_for_phrases(self, recid):
+ raise NotImplementedError
+
diff --git a/invenio/modules/indexer/tokenizers/BibIndexRecJsonTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexRecJsonTokenizer.py
new file mode 100644
index 000000000..5b9353503
--- /dev/null
+++ b/invenio/modules/indexer/tokenizers/BibIndexRecJsonTokenizer.py
@@ -0,0 +1,70 @@
+# -*- coding:utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2014 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
+"""
+ BibIndexRecJsonTokenizer.
+ It's an abstract class created only for inheritance purposes.
+
+ Tokenizers which are based on RecJsonTokenizer use bibfield/JsonAlchemy records.
+ Logic of the tokenization process is in the functions of bibfield module.
+ Tokenizer itself should perform only necessary post-processing.
+"""
+
+
+from invenio.bibindex_tokenizers.BibIndexTokenizer import BibIndexTokenizer
+
+
+class BibIndexRecJsonTokenizer(BibIndexTokenizer):
+ """
+ BibIndexRecJsonTokenizer is an abstract tokenizer.
+ It should be used only for inheritance.
+
+ It should be a base class for all tokenizers which need to use
+ bibfield/JsonAlchemy records.
+
+ Tokenizing function of RecJsonTokenizer takes a bibfield record
+ as an argument.
+ Main logic of tokenization process stays in bibfield record's
+ functions. Tokenizing functions of all tokenizers inheriting after
+ RecJsonTokenizer should only do post-processing tasks.
+
+ For example of use please check: BibIndexFiletypeTokenizer
+ """
+
+ def __init__(self, stemming_language = None, remove_stopwords = False, remove_html_markup = False, remove_latex_markup = False):
+ pass
+
+
+ def tokenize(self, record):
+ """'record' is a recjson record from bibfield module
+ @param urls: recjson record
+ """
+ raise NotImplementedError
+
+ def get_tokenizing_function(self, wordtable_type):
+ raise NotImplementedError
+
+ def tokenize_for_words(self, recid):
+ raise NotImplementedError
+
+ def tokenize_for_pairs(self, recid):
+ raise NotImplementedError
+
+ def tokenize_for_phrases(self, recid):
+ raise NotImplementedError
diff --git a/invenio/modules/indexer/tokenizers/BibIndexEmptyTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexStringTokenizer.py
similarity index 61%
copy from invenio/modules/indexer/tokenizers/BibIndexEmptyTokenizer.py
copy to invenio/modules/indexer/tokenizers/BibIndexStringTokenizer.py
index 2c5df8646..70c80f12a 100644
--- a/invenio/modules/indexer/tokenizers/BibIndexEmptyTokenizer.py
+++ b/invenio/modules/indexer/tokenizers/BibIndexStringTokenizer.py
@@ -1,61 +1,65 @@
# -*- coding:utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2010, 2011, 2012 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-"""BibIndexEmptyTokenizer: useful for situations when we want to apply tokenizer
- for consistency, but we don't want to have any results from it.
"""
+ Abstract BibIndexStringTokenizer.
+ It is a tokenizer created only for inheritance.
+ All string based tokenizers should inherit after this tokenizer.
+"""
+
+from invenio.bibindex_tokenizers.BibIndexTokenizer import BibIndexTokenizer
-from invenio.legacy.bibindex.engine_config import CFG_BIBINDEX_INDEX_TABLE_TYPE
-from invenio.modules.indexer.tokenizers.BibIndexTokenizer import BibIndexTokenizer
+class BibIndexStringTokenizer(BibIndexTokenizer):
+ """
+ BibIndexStringTokenizer is an abstract tokenizer.
+ It should be used only for inheritance.
+
+ This tokenizer should be a base class for tokenizers
+ which operates on strings/phrases and splits them
+ into multiple terms/tokens.
+ Tokenizing functions take phrase as an argument.
-class BibIndexEmptyTokenizer(BibIndexTokenizer):
- """Empty tokenizer do nothing.
- It returns empty lists for tokenize_for_words, tokenize_for_pairs and tokenize_for_phrases methods.
+ Good examples of StringTokenizer is DeafultTokenizer.
"""
def __init__(self, stemming_language = None, remove_stopwords = False, remove_html_markup = False, remove_latex_markup = False):
"""@param stemming_language: dummy
@param remove_stopwords: dummy
@param remove_html_markup: dummy
@param remove_latex_markup: dummy
"""
pass
def get_tokenizing_function(self, wordtable_type):
"""Picks correct tokenize_for_xxx function depending on type of tokenization (wordtable_type)"""
- if wordtable_type == CFG_BIBINDEX_INDEX_TABLE_TYPE["Words"]:
- return self.tokenize_for_words
- elif wordtable_type == CFG_BIBINDEX_INDEX_TABLE_TYPE["Pairs"]:
- return self.tokenize_for_pairs
- elif wordtable_type == CFG_BIBINDEX_INDEX_TABLE_TYPE["Phrases"]:
- return self.tokenize_for_phrases
+ raise NotImplementedError
def tokenize_for_words(self, phrase):
- return []
+ raise NotImplementedError
def tokenize_for_pairs(self, phrase):
- return []
+ raise NotImplementedError
def tokenize_for_phrases(self, phrase):
- return []
+ raise NotImplementedError
diff --git a/invenio/modules/indexer/tokenizers/BibIndexTokenizer.py b/invenio/modules/indexer/tokenizers/BibIndexTokenizer.py
index 80ecc9558..45c464c8a 100644
--- a/invenio/modules/indexer/tokenizers/BibIndexTokenizer.py
+++ b/invenio/modules/indexer/tokenizers/BibIndexTokenizer.py
@@ -1,116 +1,160 @@
# -*- coding:utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2010, 2011, 2012 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-"""BibIndexTokenizer: generic, not implemented tokenizer for inheritance
+"""
+ BibIndexTokenizer: generic, not implemented tokenizer for inheritance
+
+ Inheritance tree for tokenizers in Invenio:
+
+ BibIndexTokenizer
+ ^
+ |
+ |----BibIndexStringTokenizer<---|
+ | |
+ | BibIndexDefaultTokenizer<---|
+ | |
+ | BibIndexAuthorTokenizer
+ | BibIndexExactAuthorTokenizer
+ | (...)
+ |
+ |----BibIndexRecJsonTokenizer<---|
+ | |
+ | BibIndexFiletypeTokenizer
+ | (...)
+ |
+ |----BibIndexMultiFieldTokenizer<---|
+ |
+ BibIndexJournalTokenizer
+ BibIndexAuthorCountTokenizer
+ (...)
"""
class BibIndexTokenizer(object):
- """Base class for the tokenizers
-
- Tokenizers act as filters which turn input strings into lists of strings
- which represent the idexable components of that string.
+ """
+ Base class for the tokenizers.
+
+ Tokenizers are components that find terms which need to be
+ indexed and stored in DB.
+ Different types of tokenizers work in different ways.
+ Tokenizers are divided into three groups:
+ - tokenizers that take string as an input and split it into
+ tokens/terms which later are indexed
+ - tokenizers that take recID of the record and find terms
+ by processing many fields/tags from the record
+ - tokenizers that use bibfield module and their functions
+ which precomputes terms to index
"""
#words part
def scan_string_for_words(self, s):
"""Return an intermediate representation of the tokens in s.
Every tokenizer should have a scan_string function, which scans the
input string and lexically tags its components. These units are
grouped together sequentially. The output of scan_string is usually
something like:
{
'TOKEN_TAG_LIST' : a list of valid keys in this output set,
'key1' : [val1, val2, val3] - where key describes the in some
meaningful way
}
@param s: the input to be lexically tagged
@type s: string
@return: dict of lexically tagged input items
In a sample Tokenizer where scan_string simply splits s on
space, scan_string might output the following for
"Assam and Darjeeling":
{
'TOKEN_TAG_LIST' : 'word_list',
'word_list' : ['Assam', 'and', 'Darjeeling']
}
@rtype: dict
"""
raise NotImplementedError
def parse_scanned_for_words(self, o):
"""Calculate the token list from the intermediate representation o.
While this should be an interesting computation over the intermediate
representation generated by scan_string, obviously in the split-on-
space example we need only return o['word_list'].
@param t: a dictionary with a 'word_list' key
@type t: dict
@return: the token items from 'word_list'
@rtype: list of string
"""
raise NotImplementedError
def tokenize_for_words(self, s):
"""Main entry point. Return token list from input string s.
Simply composes the functionality above.
@param s: the input to be lexically tagged
@type s: string
@return: the token items derived from s
@rtype: list of string
"""
raise NotImplementedError
#pairs part
def scan_string_for_pairs(self, s):
""" See: scan_string_for_words """
raise NotImplementedError
def parse_scanned_for_pairs(self, o):
""" See: parse_scanned_for_words """
raise NotImplementedError
def tokenize_for_pairs(self, s):
""" See: tokenize_for_words """
raise NotImplementedError
#phrases part
def scan_string_for_phrases(self, s):
""" See: scan_string_for_words """
raise NotImplementedError
def parse_scanned_for_phrases(self, o):
""" See: parse_scanned_for_words """
raise NotImplementedError
def tokenize_for_phrases(self, s):
""" See: tokenize_for_words """
raise NotImplementedError
def get_tokenizing_function(self, wordtable_type):
"""Chooses tokenize_for_words, tokenize_for_phrases or tokenize_for_pairs
depending on type of tokenization we want to perform."""
raise NotImplementedError
+
+ @property
+ def implemented(self):
+ try:
+ self.get_tokenizing_function("")
+ except NotImplementedError:
+ return False
+ except AttributeError:
+ return False
+ return True
\ No newline at end of file
diff --git a/invenio/modules/records/testsuite/fields/atlantis.cfg b/invenio/modules/records/testsuite/fields/atlantis.cfg
index 6cdf31e3a..675637409 100644
--- a/invenio/modules/records/testsuite/fields/atlantis.cfg
+++ b/invenio/modules/records/testsuite/fields/atlantis.cfg
@@ -1,1083 +1,1110 @@
+# -*- coding: utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2013, 2014 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
+
###############################################################################
########## ##########
########## Invenio Atlantis Site Bibfield Configuration File ##########
########## ##########
###############################################################################
@persistent_identifier(0)
recid:
""" """
schema:
{'recid': {'type':'integer', 'min': 1, 'required': True}}
creator:
@legacy(('001', ''), )
@connect('_id')
marc, '001', int(value)
producer:
json_for_marc(), {'001': ''}
@extend
modification_date:
derived:
@legacy('marc', ('005', ''))
@depends_on('recid')
get_modification_date(self.get('recid', -1))
producer:
json_for_marc(), {"005": "self.get('modification_date').strftime('%Y%m%d%H%M%S.0')"}
@extend
creation_date:
creator:
@parse_first('recid')
@only_if('recid' not in self)
marc, '005', datetime.datetime(*(time.strptime(value, "%Y%m%d%H%M%S.0")[0:6]))
derived:
@depends_on('recid')
get_creation_date(self.get('recid', -1))
abstract:
creator:
@legacy((("520", "520__", "520__%"), "abstract", ""),
("520__a", "abstract", "summary"),
("520__b", "expansion"),
("520__9", "number"))
marc, "520__", {'summary':value['a'], 'expansion':value['b'], 'number':value['9']}
producer:
json_for_marc(), {"520__a": "summary", "520__b": "expansion", "520__9": "number"}
+ json_for_dc(), {"dc:description": "summary"}
abstract_french:
creator:
@legacy((("590", "590__", "590__%"), ""),
("590__a", "summary"),
("590__b", "expansion"))
marc, "590__", {'summary':value['a'], 'expansion':value['b']}
producer:
json_for_marc(), {"590__a": "sumary", "590__b": "expansion"}
accelerator_experiment:
creator:
@legacy((("693", "693__", "693__%"), ""),
("693__a", "accelerator"),
("693__e", "experiment"),
("693__f", "facility"))
marc, "693__", {'accelerator':value['a'], 'experiment':value['e'], 'facility':value['f']}
producer:
json_for_marc(), {"693__a": "accelerator", "693__b": "experiment", "693__f": "facility"}
action_note:
creator:
@legacy((("583", "583__", "583__%"), ""),
("583__a", "action"),
("583__c", "time"),
("583__i", "email"),
("583__z", "note"))
marc, "583__", {'action':value['a'], 'time':value['c'], 'email':value['i'], 'note':value['z']}
producer:
json_for_marc(), {"583__a": "action", "583__c": "time", "583__i": "email", "583__z": "note"}
address:
creator:
@legacy((("270", "270__", "270__%"), ""),
("270__a", "address"),
("270__b", "city"),
("270__d", "country"),
("270__e", "pc"),
("270__k", "telephone"),
("270__l", "fax"),
("270__m", "email"),
("270__p", "contact"),
("270__s", "suffix"),
("270__9", "telex"))
marc, "270__", {'address':value['a'], 'city':value['b'], 'country':value['d'], 'pc':value['e'], 'telephone':value['k'], 'fax':value['l'], 'email':value['m'], 'contact':value['p'], 'suffix':value['s'], 'telex':value['9']}
producer:
json_for_marc(), {"270__a":"address", "270__b":"city", "270__d":"country", "270__e":"pc", "270__k":"telephone", "270__l":"fax", "270__m":"email", "270__p":"contact", "270__s":"suffix", "270__9":"telex"}
affiliation:
creator:
@legacy((("901", "901__", "901__%"), ""),
("901__u", ""))
marc, "901__", value['u']
producer:
json_for_marc(), {"901__u": ""}
agency_code:
+ """It contains the code for the agency whose system control number is present in field recid"""
creator:
@legacy(("003", ""), )
marc, "003", value
producer:
json_for_marc(), {"003": ""}
- description:
- """It contains the code for the agency whose system control number is present in field recid"""
aleph_linking_page:
creator:
@legacy((("962", "962__", "962__%"), ""),
("962__a", "type"),
("962__b", "sysno"),
("962__l", "library"),
("962__n", "down_link"),
("962__m", "up_link"),
("962__y", "volume_link"),
("962__p", "part_link"),
("962__i", "issue_link"),
("962__k", "pages"),
("962__t", "base"))
marc, "962__", {'type':value['a'], 'sysno':value['b'], 'library':value['l'], 'down_link':value['n'], 'up_link':value['n'], 'volume_link':value['y'], 'part_link':value['p'], 'issue_link':value['i'], 'pages':value['k'], 'base':value['t']}
producer:
json_for_marc(), {"962__a":"type", "962__b":"sysno", "962__l":"library", "962__n":"down_link", "962__m":"up_link", "962__y":"volume_link", "962__p":"part_link", "962__i":"issue_link", "962__k":"pages", "962__t":"base"}
_first_author, first_author, creator:
creator:
@legacy((("100", "100__", "100__%"), ""),
("100__a", "first author name", "full_name"),
("100__e", "relator_name"),
("100__h", "CCID"),
("100__i", "INSPIRE_number"),
("100__u", "first author affiliation", "affiliation"))
marc, "100__", { 'full_name':value['a'], 'first_name':util_split(value['a'],',',1), 'last_name':util_split(value['a'],',',0), 'relator_name':value['e'], 'CCID':value['h'], 'INSPIRE_number':value['i'], 'affiliation':value['u'] }
+ checker:
+ check_field_existence(0,1)
+ check_field_type('str')
producer:
json_for_marc(), {"100__a": "full_name", "100__e": "relator_name", "100__h": "CCID", "100__i": "INSPIRE_number", "100__u": "affiliation"}
+ json_for_dc(), {"dc:creator": "full_name"}
_additional_authors, additional_authors, contributor:
schema:
{'_additional_authors': {'type': 'list', 'force': True}}
creator:
@legacy((("700", "700__", "700__%"), ""),
("700__a", "additional author name", "full_name"),
("700__u", "additional author affiliation", "affiliation"))
@parse_first('_first_author')
marc, "700__", {'full_name': value['a'], 'first_name':util_split(value['a'],',',1), 'last_name':util_split(value['a'],',',0), 'relator_name':value['e'], 'CCID':value['h'], 'INSPIRE_number':value['i'], 'affiliation':value['u'] }
+ checker:
+ check_field_existence(0,'n')
+ check_field_type('str')
producer:
json_for_marc(), {"700__a": "full_name", "700__e": "relator_name", "700__h": "CCID", "700__i": "INSPIRE_number", "700__u": "affiliation"}
+ json_for_dc(), {"dc:contributor": "full_name"}
authors:
"""List with all the authors, connected with main_author and rest_authors"""
derived:
@parse_first('_first_author', '_additional_authors')
@connect('_first_author', sync_authors)
@connect('_additional_authors', sync_authors)
@only_if('_firs_author' in self or '_additional_authors' in self)
util_merge_fields_info_list(self, ['_first_author', '_additional_authors'])
author_archive:
creator:
@legacy((("720", "720__", "720__%"), ""),
("720__a", ""))
marc, "720__", value['a']
producer:
json_for_marc(), {"720__a": ""}
base:
creator:
@legacy((("960", "960__", "960__%"), ""),
("960__a", ""))
marc, "960__", value['a']
producer:
json_for_marc(), {"960__a": ""}
cataloguer_info:
creator:
@legacy((("961", "961__", "961__%"), ""),
("961__a", "cataloguer"),
("961__b", "level"),
("961__c", "modification_date"),
("961__l", "library"),
("961__h", "hour"),
("961__x", "creation_date"))
marc, "961__", {'cataloguer':value['a'], 'level':value['b'], 'modification_date':value['c'], 'library':value['l'], 'hour':value['h'], 'creation_date':value['x']}
producer:
json_for_marc(), {"961__a": "cataloguer", "961__b": "level", "961__c": "modification_date", "961__l": "library", "961__h": "hour", "961__x": "creation_date"}
classification_terms:
schema:
{'classification_terms': {'type': 'list', 'force': True}}
creator:
@legacy((("694", "694__", "694__%"), ""),
("694__a", "term"),
("694__9", "institute"))
marc, "694__", {'term':value['a'], 'institute':value['9']}
producer:
json_for_marc(), {"694__a": "term", "694__9": "institute"}
cern_bookshop_statistics:
creator:
@legacy((("599", "599__", "599__%"), ""),
("599__a", "number_of_books_bought"),
("599__b", "number_of_books_sold"),
("599__c", "relation"))
marc, "599__", {'number_of_books_bought':value['a'], 'number_of_books_sold':value['b'], 'relation':value['c']}
producer:
json_for_marc(), {"599__a":"number_of_books_bought", "599__b":"number_of_books_sold", "599__c":"relation"}
code_designation:
creator:
@legacy((("033", "033__", "033__%"), ""),
("030__a", "coden", "coden"),
("030__9", "source"))
marc, "030__", {'coden':value['a'], 'source':value['9']}
producer:
json_for_marc(), {"030__a":"coden", "030__9":"source"}
collections:
schema:
{'collections': {'type': 'list', 'force': True}}
creator:
@legacy((("980", "980__", "980__%"), ""),
("980__%", "collection identifier", ""),
("980__a", "primary"),
("980__b", "secondary"),
("980__c", "deleted"))
marc, "980__", { 'primary':value['a'], 'secondary':value['b'], 'deleted':value['c'] }
producer:
json_for_marc(), {"980__a":"primary", "980__b":"secondary", "980__c":"deleted"}
comment:
creator:
@legacy((("500", "500__", "500__%"), ""),
("500__a", "comment", ""))
marc, "500__", value['a']
producer:
json_for_marc(), {"500__a": ""}
content_type:
+ """Note: use for SLIDES"""
creator:
@legacy((("336", "336__", "336__%"), ""),
("336__a", ""))
marc, "336__", value['a']
producer:
json_for_marc(), {"336__a": ""}
- description:
- """Note: use for SLIDES"""
copyright:
creator:
@legacy((("598", "598__", "598__%"), ""),
("598__a", ""))
marc, "598__", value['a']
producer:
json_for_marc(), {"580__a": ""}
_first_corporate_name, first_corporate_name:
creator:
@legacy((("110", "110__", "110__%"), ""),
("110__a", "name"),
("110__b", "subordinate_unit"),
("110__g", "collaboration"))
marc, "110__", {'name':value['a'], 'subordinate_unit':value['b'], 'collaboration':value['g']}
producer:
json_for_marc(), {"110__a":"name", "110__b":"subordinate_unit", "110__":"collaboration"}
_additional_corporate_names, additional_corporate_names:
schema:
{'_additional_corporate_names': {'type': 'list', 'force': True}}
creator:
@legacy((("710", "710__", "710__%"), ""),
("710__a", "name"),
("710__b", "subordinate_unit"),
("710__g", "collaboration", "collaboration"))
marc, "710__", {'name':value['a'], 'subordinate_unit':value['b'], 'collaboration':value['g']}
producer:
json_for_marc(), {"710__a":"name", "710__b":"subordinate_unit", "710__":"collaboration"}
corporate_names:
derived:
@parse_first('_first_corporate_name', '_additional_corporate_names')
@connect('_first_corporate_name', sync_corparate_names)
@connect('_additional_corporate_names', sync_corparate_names)
@only_if('_first_corporate_name' in self or '_additional_corporate_names' in self)
util_merge_fields_info_list(self, ['_first_corporate_name', '_additional_corporate_names'])
-
cumulative_index:
creator:
@legacy((("555", "555__", "555__%"), ""),
("555__a", ""))
marc, "555__", value['a']
producer:
json_for_marc(), {"555__a": ""}
current_publication_prequency:
creator:
@legacy((("310", "310__", "310__%"), ""),
("310__a", ""))
marc, "310__", value['a']
producer:
json_for_marc(), {"310__a": ""}
publishing_country:
creator:
@legacy((("044", "044__", "044__%"), ""),
("044__a", ""))
marc, "044__", value['a']
producer:
json_for_marc(), {"044__a": ""}
coyright:
creator:
@legacy((("542", "542__", "542__%"), ""),
("542__d", "holder"),
("542__g", "date"),
("542__u", "url"),
("542__e", "holder_contact"),
("542__f", "statement"),
("542__3", "materials"),)
marc, "542__", {'holder':value['d'], 'date':value['g'], 'url':value['u'], 'holder_contact':value['e'], 'statement':value['f'], 'materials':value['3']}
producer:
json_for_marc(), {"542__d": "holder", "542__g": "date", "542__u": "url", "542__e": "holder_contact", "542__f": "statement", "542__3": "materials"}
dewey_decimal_classification_number:
creator:
@legacy((("082", "082__", "082__%"), ""),
("082__a", ""))
marc, "082__", value['a']
producer:
json_for_marc(), {"082__a": ""}
dissertation_note:
creator:
@legacy((("502", "502__", "502__%"), ""),
("502__a","diploma"),
("502__b","university"),
("502__c","defense_date"))
marc, "502__", {'diploma':value['a'], 'university':value['b'], 'defense_date':value['c']}
producer:
json_for_marc(), {"502__a": "diploma", "502__b": "university", "502__b": "defense_date"}
@persistent_identifier(3)
doi:
creator:
@legacy((("024", "0247_", "0247_%"), ""),
("0247_a", ""))
marc, "0247_", get_doi(value)
producer:
json_for_marc(), {'0247_2': 'str("DOI")', '0247_a': ''}
edition_statement:
+ """Information relating to the edition of a work as determined by applicable cataloging rules."""
creator:
@legacy((("250", "250__", "250__%"), ""),
("250__a", ""))
marc, "250__", value['a']
producer:
json_for_marc(), {"250__a": ""}
- description:
- """Information relating to the edition of a work as determined by applicable cataloging rules."""
email:
creator:
@legacy((("856", "8560_", "8560_%"), ""),
("8560_f", "email"))
marc, "8560_", value['f']
producer:
json_for_marc(), {"8560_f": ""}
email_message:
creator:
@legacy((("859", "859__", "859__%"), ""),
("859__a","contact"),
("859__f","address"),
("859__x","date"))
marc, "859__", {'contact':value['a'], 'address':value['f'], 'date':value['x']}
producer:
json_for_marc(), {"859__a": 'contact',"859__f": 'address', "859__x": 'date'}
fft:
creator:
@legacy((('FFT', 'FFT__', 'FFT__%'), ''),
("FFT__a", "path"),
("FFT__d", "description"),
("FFT__f", "eformat"),
("FFT__i", "temporary_id"),
("FFT__m", "new_name"),
("FFT__o", "flag"),
("FFT__r", "restriction"),
("FFT__s", "timestamp"),
("FFT__t", "docfile_type"),
("FFT__v", "version"),
("FFT__x", "icon_path"),
("FFT__z", "comment"),
("FFT__w", "document_moreinfo"),
("FFT__p", "version_moreinfo"),
("FFT__b", "version_format_moreinfo"),
("FFT__f", "format_moreinfo"))
marc, "FFT__", {'path': value['a'],
'description': value['d'],
'eformat': value['f'],
'temporary_id': value['i'],
'new_name': value['m'],
'flag': value['o'],
'restriction': value['r'],
'timestamp': value['s'],
'docfile_type': value['t'],
'version': value['v'],
'icon_path': value['x'],
'comment': value['z'],
'document_moreinfo': value['w'],
'version_moreinfo': value['p'],
'version_format_moreinfo': value['b'],
'format_moreinfo': value['u']
}
@only_if_master_value(is_local_url(value['u']))
marc, "8564_", {'hots_name': value['a'],
'access_number': value['b'],
'compression_information': value['c'],
'path':value['d'],
'electronic_name': value['f'],
'request_processor': value['h'],
'institution': value['i'],
'formart': value['q'],
'settings': value['r'],
'file_size': value['s'],
'url': value['u'],
'subformat':value['x'],
'description':value['y'],
'comment':value['z']}
producer:
json_for_marc(), {"FFT__a": "path", "FFT__d": "description", "FFT__f": "eformat", "FFT__i": "temporary_id", "FFT__m": "new_name", "FFT__o": "flag", "FFT__r": "restriction", "FFT__s": "timestamp", "FFT__t": "docfile_type", "FFT__v": "version", "FFT__x": "icon_path", "FFT__z": "comment", "FFT__w": "document_moreinfo", "FFT__p": "version_moreinfo", "FFT__b": "version_format_moreinfo", "FFT__f": "format_moreinfo"}
funding_info:
creator:
@legacy((("536", "536__", "536__%"), ""),
("536__a", "agency"),
("536__c", "grant_number"),
("536__f", "project_number"),
("536__r", "access_info"))
marc, "536__", {'agency':value['a'], 'grant_number':value['c'], 'project_number':value['f'], 'access_info':value['r']}
producer:
json_for_marc(), {"536__a": "agency", "536__c": "grant_number", "536__f": "project_number", "536__r": "access_info"}
imprint:
creator:
@legacy((("260", "260__", "260__%"), ""),
("260__a", "place"),
("260__b", "publisher_name"),
("260__c", "date"),
("260__g", "reprinted_editions"))
marc, "260__", {'place':value['a'], 'publisher_name':value['b'], 'date':value['c'], 'reprinted_editions':value['g']}
producer:
json_for_marc(), {"260__a": "place", "260__b": "publisher_name", "260__c": "date", "260__g": "reprinted_editions"}
internal_notes:
creator:
@legacy((("595", "595__", "595__%"), ""),
("595__a", "internal notes", "internal_note"),
("595__d", "control_field"),
("595__i", "inspec_number"),
("595__s", "subject"))
marc, "595__", {'internal_note':value['a'], 'control_field':value['d'], 'inspec_number':value['i'], 'subject':value['s']}
producer:
json_for_marc(), {"595__a": "internal_note", "595__d": "control_field","595__i": "inspec_number", "595__s": "subject"}
isbn:
creator:
@legacy((("020", "020__", "020__%"), ""),
("020__a", "isbn", "isbn"),
("020__u", "medium"))
marc, "020__", {'isbn':value['a'], 'medium':value['u']}
producer:
json_for_marc(), {"020__a": "isbn", "020__u": "medium"}
isn:
creator:
@legacy((("021", "021__", "021__%"), ""),
("021__a", ""))
marc, "021__", value['a']
producer:
json_for_marc(), {"021__a": ""}
issn:
creator:
@legacy((("022", "022__", "022__%"), ""),
("022__a", "issn", ""))
marc, "022__", value['a']
producer:
json_for_marc(), {"022__a": ""}
item:
creator:
@legacy((("964", "964__", "964__%"), ""),
("964__a", ""))
marc, "964__", value['a']
producer:
json_for_marc(), {"964__a": ""}
journal_info:
creator:
@legacy((("909", "909C4", "909C4%"), "journal", ""),
("909C4a", "doi", "doi"),
("909C4c", "journal page", "pagination"),
("909C4d", "date"),
("909C4e", "recid"),
("909C4f", "note"),
("909C4p", "journal title", "title"),
("909C4u", "url"),
("909C4v", "journal volume", "volume"),
("909C4y", "journal year", "year"),
("909C4t", "talk"),
("909C4w", "cnum"),
("909C4x", "reference"))
marc, "909C4", {'doi':value['a'], 'pagination':value['c'], 'date':value['d'], 'recid':value['e'], 'note':value['f'], 'title':value['p'], 'url':value['u'], 'volume':value['v'], 'year':value['y'], 'talk':value['t'], 'cnum':value['w'], 'reference':value['x']}
producer:
json_for_marc(), {"909C4a": "doi","909C4c": "pagination", "909C4d": "date", "909C4e": "recid", "909C4f": "note", "909C4p": "title", "909C4u": "url","909C4v": "volume", "909C4y": "year", "909C4t": "talk", "909C4w": "cnum", "909C4x": "reference"}
keywords:
schema:
{'keywords': {'type': 'list', 'force': True}}
creator:
@legacy((("653", "6531_", "6531_%"), ""),
("6531_a", "keyword", "term"),
("6531_9", "institute"))
marc, "6531_", { 'term': value['a'], 'institute': value['9'] }
producer:
json_for_marc(), {"6531_a": "term", "6531_9": "institute"}
language:
creator:
@legacy((("041", "041__", "041__%"), ""),
("041__a", ""))
marc, "041__", value['a']
producer:
json_for_marc(), {"041__a": ""}
+ json_for_dc(), {"dc:language": ""}
language_note:
creator:
@legacy((("546", "546__", "546__%"), ""),
("546__a", "language_note"),
("546__g", "target_language"))
marc, "546__", {'language_note':value['a'], 'target_language':value['g']}
producer:
json_for_marc(), {"546__a": "language_note", "546__g": "target_language"}
library_of_congress_call_number:
creator:
@legacy((("050", "050__", "050__%"), ""),
("050__a", "classification_number"),
("050__b", "item_number"))
marc, "050__", {'classification_number':value['a'], 'item_number':value['b']}
producer:
json_for_marc(), {"050__a": "classification_number", "050__b": "item_number"}
license:
creator:
@legacy((("540", "540__", "540__%"), ""),
("540__a", "license"),
("540__b", "imposing"),
("540__u", "url"),
("540__3", "material"))
marc, "540__", {'license':value['a'], 'imposing':value['b'], 'url':value['u'], 'material':value['3'],}
producer:
json_for_marc(), {"540__a": "license", "540__b": "imposing", "540__u": "url", "540__3": "material"}
location:
creator:
@legacy((("852", "852__", "852__%"), ""),
("852__a", ""))
marc, "852__", value['a']
producer:
json_for_marc(), {"852__a": ""}
medium:
creator:
@legacy((("340", "340__", "340__%"), ""),
("340__a", "material"),
("340__c", "suface"),
("340__d", "recording_technique"),
("340__d", "cd-rom"))
marc, "340__", {'material':value['a'], 'surface':value['c'], 'recording_technique':value['d'], 'cd-rom':value['9']}
producer:
json_for_marc(), {"340__a": "material", "340__c": "suface", "340__d": "recording_technique", "340__d": "cd-rom"}
_first_meeting_name:
creator:
@legacy((("111", "111__", "111__%"), ""),
("111__a", "meeting"),
("111__c", "location"),
("111__d", "date"),
("111__f", "year"),
("111__g", "coference_code"),
("111__n", "number_of_parts"),
("111__w", "country"),
("111__z", "closing_date"),
("111__9", "opening_date"))
marc, "111__", {'meeting':value['a'], 'location':value['c'], 'date':value['d'], 'year':value['f'], 'coference_code':value['g'], 'number_of_parts':value['n'], 'country':value['w'], 'closing_date':value['z'], 'opening_date':value['9']}
producer:
json_for_marc(), {"111__a": "meeting", "111__c": "location", "111__d": "date","111__f": "year", "111__g": "coference_code", "111__n": "number_of_parts", "111__w": "country", "111__z": "closing_date", "111__9": "opening_date"}
-_additionla_meeting_names:
+_additional_meeting_names:
creator:
@legacy((("711", "711__", "711__%"), ""),
("711__a", "meeting"),
("711__c", "location"),
("711__d", "date"),
("711__f", "work_date"),
("711__g", "coference_code"),
("711__n", "number_of_parts"),
("711__9", "opening_date"))
marc, "711__", {'meeting':value['a'], 'location':value['c'], 'date':value['d'], 'work_date':value['f'], 'coference_code':value['g'], 'number_of_parts':value['n'], 'opening_date':value['9']}
producer:
- json_ffirst_authoror_marc(), {"711__a": "meeting", "711__c": "location", "711__d": "date", "711__f": "work_date", "711__g": "coference_code", "711__n": "number_of_parts", "711__9": "opening_date"}
+ json_for_marc(), {"711__a": "meeting", "711__c": "location", "711__d": "date", "711__f": "work_date", "711__g": "coference_code", "711__n": "number_of_parts", "711__9": "opening_date"}
meeting_names:
derived:
@parse_first('_first_meeting_name', '_additionla_meeting_names')
@connect('_first_meeting_name', sync_meeting_names)
@connect('_additionla_meeting_names', sync_meeting_names)
@only_if('_first_meeting_name' in self or '_additionla_meeting_names' in self)
util_merge_fields_info_list(self, ['_first_meeting_name', '_additionla_meeting_names'])
@persistent_identifier(4)
oai:
creator:
@legacy((("024", "0248_", "0248_%"), ""),
("0248_a", "oai"),
("0248_p", "indicator"))
marc, "0248_", {'value': value['a'], 'indicator': value['p']}
producer:
json_for_marc(), {"0248_a": "oai", "0248_p": "indicator"}
observation:
creator:
@legacy((("691", "691__", "691__%"), ""),
("691__a", ""))
marc, "691__", value['a']
producer:
json_for_marc(), {"691__a": ""}
observation_french:
creator:
@legacy((("597", "597__", "597__%"), ""),
("597__a", ""))
marc, "597__", value['a']
producer:
json_for_marc(), {"597__a": ""}
other_report_number:
creator:
@legacy((("084", "084__", "084__%"), ""),
("084__a", "clasification_number"),
("084__b", "collection_short"),
("084__2", "source_number"))
marc, "084__", {'clasification_number':value['a'], 'collection_short':value['b'], 'source_number':value['2'],}
producer:
json_for_marc(), {"084__a": "clasification_number", "084__b": "collection_short", "084__2": "source_number"}
owner:
creator:
@legacy((("963", "963__", "963__%"), ""),
("963__a",""))
marc, "963__", value['a']
producer:
json_for_marc(), {"963__a": ""}
prepublication:
+ """
+ note: don't use the following lines for cer base=14,2n,41-45 !!
+ note: don't use for theses
+ """
creator:
@legacy((("269", "269__", "269__%"), ""),
("269__a", "place"),
("269__b", "publisher_name"),
("269__c", "date"))
marc, "269__", {'place':value['a'], 'publisher_name': value['b'], 'date':value['c']}
producer:
json_for_marc(), {"269__a": "place", "269__b": "publisher_name", "269__c": "date"}
- description:
- """
- note: don't use the following lines for cer base=14,2n,41-45 !!
- note: don't use for theses
- """
primary_report_number:
creator:
@legacy((("037", "037__", "037__%"), ""),
("037__a", "primary report number", ""), )
marc, "037__", value['a']
producer:
json_for_marc(), {"037__a": ""}
publication_info:
+ """note: publication_info.doi not to be used, used instead doi"""
creator:
@legacy((("773", "773__", "773__%"), ""),
("773__a", "doi"),
("773__c", "pagination"),
("773__d", "date"),
("773__e", "recid"),
("773__f", "note"),
("773__p", "title"),
("773__u", "url"),
("773__v", "volume"),
("773__y", "year"),
("773__t", "talk"),
("773__w", "cnum"),
("773__x", "reference"))
marc, "773__", {'doi':value['a'], 'pagination':value['c'], 'date':value['d'], 'recid':value['e'], 'note':value['f'], 'title':value['p'], 'url':value['u'], 'volume':value['v'], 'year':value['y'], 'talk':value['t'], 'cnum':value['w'], 'reference':value['x']}
producer:
json_for_marc(), {"773__a": "doi", "773__c": "pagination", "773__d": "date", "773__e": "recid", "773__f": "note", "773__p": "title", "773__u": "url", "773__v": "volume", "773__y": "year", "773__t": "talk", "773__w": "cnum", "773__x": "reference"}
- description:
- """note: publication_info.doi not to be used, used instead doi"""
physical_description:
creator:
@legacy((("300", "300__", "300__%", "")),
("300__a", "pagination"),
("300__b", "details"))
marc, "300__", {'pagination':value['a'], 'details':value['b']}
producer:
json_for_marc(), {"300__a": "pagination", "300__b": "details"}
reference:
creator:
@legacy((("999", "999C5", "999C5%"), ""),
("999C5", "reference", ""),
("999C5a", "doi"),
("999C5h", "authors"),
("999C5m", "misc"),
("999C5n", "issue_number"),
("999C5o", "order_number"),
("999C5p", "page"),
("999C5r", "report_number"),
("999C5s", "title"),
("999C5u", "url"),
("999C5v", "volume"),
("999C5y", "year"),)
marc, "999C5", {'doi':value['a'], 'authors':value['h'], 'misc':value['m'], 'issue_number':value['n'], 'order_number':value['o'], 'page':value['p'], 'report_number':value['r'], 'title':value['s'], 'url':value['u'], 'volume':value['v'], 'year':value['y'],}
producer:
json_for_marc(), {"999C5a": "doi", "999C5h": "authors", "999C5m": "misc", "999C5n": "issue_number", "999C5o":"order_number", "999C5p":"page", "999C5r":"report_number", "999C5s":"title", "999C5u":"url", "999C5v":"volume", "999C5y": "year"}
restriction_access:
creator:
@legacy((("506", "506__", "506__%"), ""),
("506__a", "terms"),
("506__9", "local_info"))
marc, "506__", {'terms':value['a'], 'local_info':value['9']}
producer:
json_for_marc(), {"506__a": "terms", "506__9": "local_info"}
report_number:
creator:
@legacy((("088", "088__", "088__%"), ""),
("088__a", "additional report number", "report_number"),
("088__9", "internal"))
marc, "088__", {'report_number':value['a'], 'internal':value['9']}
producer:
json_for_marc(), {"088__a": "report_number", "088__9": "internal"}
series:
creator:
@legacy((("490", "490__", "490__%"), ""),
("490__a", "statement"),
("490__v", "volume"))
marc, "490__", {'statement':value['a'], 'volume':value['v']}
producer:
json_for_marc(), {"490__a": "statement", "490__v": "volume"}
slac_note:
creator:
@legacy((("596", "596__", "596__%"), ""),
("596__a", "slac_note", ""), )
marc, "596__", value['a']
producer:
json_for_marc(), {"596__a": ""}
source_of_acquisition:
creator:
@legacy((("541", "541__", "541__%"), ""),
("541__a","source_of_acquisition"),
("541__d","date"),
("541__e","accession_number"),
("541_f_","owner"),
("541__h","price_paid"),
("541__9","price_user"))
marc, "541__", {'source_of_acquisition':value['a'], 'date':value['d'], 'accession_number':value['e'], 'owner':value['f'], 'price_paid':value['h'], 'price_user':value['9']}
producer:
json_for_marc(), {"541__a": "source_of_acquisition", "541__d": "date", "541__e": "accession_number", "541_f_": "owner", "541__h": "price_paid", "541__9":"price_user"}
status_week:
creator:
@legacy((("916", "916__", "916__%"), ""),
("916__a","acquistion_proceedings"),
("916__d","display_period"),
("916__e","copies_bought"),
("916__s","status"),
("916__w","status_week"),
("916__y","year"))
marc, "916__", {'acquistion_proceedings':value['a'], 'display_period':value['d'], 'copies_bought':value['e'], 'status':value['s'], 'status_week':value['w'], 'year':value['y']}
producer:
json_for_marc(), {"916__a": "acquistion_proceedings", "916__d": "display_period", "916__e": "copies_bought", "916__s": "status", "916__w": "status_week", "916__y":"year"}
subject:
creator:
@legacy((("650", "65017", "65017%"), ""),
("65017a", "main subject", "term"),
("650172", "source"),
("65017e", "relator"))
marc, "65017", {'term':value['a'], 'source':value['2'], 'relator':value['e']}
producer:
json_for_marc(), {"65017a": "term", "650172": "source", "65017e": "relator"}
+ json_for_dc(), {"dc:subject": "term"}
subject_additional:
creator:
@legacy((("650", "65027", "65027%"), ""),
("65027a", "additional subject", "term"),
("650272", "source"),
("65027e", "relator"),
("65027p", "percentage"))
marc, "65027", {'term':value['a'], 'source':value['2'], 'relator':value['e'], 'percentage':value['p']}
producer:
json_for_marc(), {"65027a": "term", "650272": "source", "65027e": "relator", "65027p": "percentage"}
subject_indicator:
creator:
@legacy((("690", "690__", "690__%"), ""),
("690c_a", ""))
marc, "690c_", value['a']
producer:
json_for_marc(), {"690c_a": ""}
@persistent_identifier(2)
system_control_number:
creator:
@legacy((("035", "035__", "035__%"), ""),
("035__a", "system_control_number"),
("035__9", "institute"))
marc, "035__", {'value': value['a'], 'canceled':value['z'], 'linkpage':value['6'], 'institute':value['9']}
producer:
json_for_marc(), {"035__a": "system_control_number", "035__9": "institute"}
@persistent_identifier(1)
system_number:
creator:
@legacy((("970", "970__", "970__%"), ""),
("970__a", "sysno"),
("970__d", "recid"))
marc, "970__", {'value':value['a'], 'recid':value['d']}
producer:
json_for_marc(), {"970__a": "sysno", "970__d": "recid"}
thesaurus_terms:
creator:
@legacy((("695", "695__", "695__%"), ""),
("695__a", "term"),
("695__9", "institute"))
marc, "695__", {'term':value['a'], 'institute':value['9']}
producer:
json_for_marc(), {"695__a": "term", "695__9": "institute"}
time_and_place_of_event_note:
creator:
@legacy((("518", "518__", "518__%"), ""),
("518__d", "date"),
("518__g", "conference_identification"),
("518__h", "starting_time"),
("518__l", "speech_length"),
("518__r", "meeting"))
marc, "519__", {'date':value['d'], 'conference_identification':value['g'], 'starting_time':value['h'], 'speech_length':value['l'], 'meeting':value['r']}
producer:
json_for_marc(), {"518__d": "date", "518__g": "conference_identification", "518__h": "starting_time", "518__l": "speech_length", "518__r": "meeting"}
abbreviated_title:
creator:
@legacy((("210", "210__", "210__%"), ""),
("210__a", ""))
marc, "210__", value['a']
producer:
json_for_marc(), {"210__a": ""}
main_title_statement:
creator:
@legacy((("145", "145__", "145__%"), ""),
("145__a", "title"),
("145__b", "subtitle"),)
marc, "145__", {'title':value['a'], 'subtitle':value['b']}
producer:
json_for_marc(), {"145__a": "title", "145__b": "subtitle"}
title_additional:
creator:
@legacy((("246", "246__", "246__%"), ""),
("246__%", "additional title", ""),
("246__a", "title"),
("246__b", "subtitle"),
("246__g", "misc"),
("246__i", "text"),
("246__n", "part_number"),
("246__p", "part_name"))
marc, "246__", { 'title':value['a'], 'subtitle':value['b'], 'misc':value['g'], 'text':value['i'], 'part_number':value['n'], 'part_name':value['p']}
producer:
json_for_marc(), {"246__a": "title", "246__b": "subtitle", "246__g": "misc", "246__i": "text", "246__n": "part_number", "246__p": "part_name"}
title:
creator:
@legacy((("245", "245__", "245__%"), ""),
("245__%", "main title", ""),
("245__a", "title", "title"),
("245__b", "subtitle"),
("245__n", "volume"),
("245__k", "form"))
marc, "245__", { 'title':value['a'], 'subtitle':value['b'], 'volume': value['n'], 'form':value['k'] }
+ checker:
+ check_field_existence(0, 1, continuable=False)
+ check_field_type('str')
producer:
json_for_marc(), {"245__a": "title", "245__b": "subtitle", "245__k": "form"}
+ json_for_dc(), {"dc:title": "title"}
title_key:
creator:
@legacy((("222", "222__", "222__%"), ""),
("222__a", ""))
marc, "222__", value['a']
producer:
json_for_marc(), {"222__a": ""}
title_other:
creator:
@legacy((("246", "246_3", "246_3%"), ""),
("246_3a", "title"),
("246_3i", "text"),
("246_39", "sigle"))
marc, "246_3", { 'title':value['a'], 'text':value['i'], 'sigle':value['9']}
producer:
json_for_marc(), {"246_3a": "title", "246_3i": "text", "246_39": "sigle"}
title_parallel:
creator:
@legacy((("246", "246_1", "246_1%"), ""),
("246_1a", "title"),
("246_1i", "text"))
marc, "246_1", { 'title':value['a'], 'text':value['i']}
producer:
json_for_marc(), {"246_1a": "title", "246_1i": "text"}
title_translation:
creator:
@legacy((("242", "242__", "242__%"), ""),
("242__a", "title"),
("242__b", "subtitle"),
("242__y", "language"))
marc, "242__", {'title':value['a'], 'subtitle':value['b'], 'language':value['y']}
producer:
json_for_marc(), {"242__a": "title", "242__b": "subtitle", "242__y": "language"}
type:
creator:
@legacy((("594", "594__", "594__%"), ""),
("594__a", ""))
marc, "594__", value['a']
producer:
json_for_marc(), {"594__a": ""}
udc:
creator:
@legacy((("080", "080__", "080__%"), ""),
("080__a", ""))
marc, "080__", value['a']
producer:
json_for_marc(), {"080__a": ""}
- description:
- """"universal decimal classification number"""
url:
creator:
@legacy((("856", "8564_", "8564_%"), ""),
("8564_a", "host_name"),
("8564_b", "access_number"),
("8564_c", "compression_information"),
("8564_d", "path"),
("8564_f", "electronic_name"),
("8564_h", "request_processor"),
("8564_i", "institution"),
("8564_q", "eformat"),
("8564_r", "settings"),
("8564_s", "file_size"),
("8564_u", "url", "url"),
("8564_x", "subformat"),
("8564_y", "caption", "description"),
("8564_z", "comment"))
@only_if_master_value((not is_local_url(value['u']), ))
marc, "8564_", {'host_name': value['a'],
'access_number': value['b'],
'compression_information': value['c'],
'path':value['d'],
'electronic_name': value['f'],
'request_processor': value['h'],
'institution': value['i'],
'eformart': value['q'],
'settings': value['r'],
'size': value['s'],
'url': value['u'],
'subformat':value['x'],
'description':value['y'],
'comment':value['z']}
producer:
json_for_marc(), {"8564_a": "host_name", "8564_b": "access_number", "8564_c": "compression_information", "8564_d": "path", "8564_f": "electronic_name", "8564_h": "request_processor", "8564_i": "institution", "8564_q": "eformat", "8564_r": "settings", "8564_s": "file_size", "8564_u": "url", "8564_x": "subformat", "8564_y": "description", "8564_z": "comment"}
-
+ json_for_dc(), {"dc:identifier": "url"}
###############################################################################
########## ##########
########## Derived and Calculated Fields Definitions ##########
########## ##########
###############################################################################
files:
+ """
+ Retrieves all the files related with the recid that were passed to the system
+ using the FFT field described above
+
+ Note: this is a mandatory field and it shouldn't be remove from this configuration
+ file. On the other hand the function that retrieve the metadata from BibDoc could
+ be enrich.
+ """
calculated:
@legacy('marc', ("8564_z", "comment"),
("8564_y", "caption", "description"),
("8564_q", "eformat"),
("8564_f", "name"),
("8564_s", "size"),
("8564_u", "url", "url")
)
@parse_first('recid')
@memoize()
get_files_from_bibdoc(self.get('recid', -1))
producer:
json_for_marc(), {"8564_z": "comment", "8564_y": "description", "8564_q": "eformat", "8564_f": "name", "8564_s": "size", "8564_u": "url"}
- description:
- """
- Retrieves all the files related with the recid that were passed to the system
- using the FFT field described above
-
- Note: this is a mandatory field and it shouldn't be remove from this configuration
- file. On the other hand the function that retrieve the metadata from BibDoc could
- be enrich.
- """
+ json_for_dc(), {"dc:identifier": "url"}
number_of_authors:
"""Number of authors"""
derived:
@depends_on('authors')
len(self['authors'])
number_of_copies:
calculated:
@depends_on('recid', 'collections')
@only_if('BOOK' in self.get('collections.primary', []))
@memoize()
get_number_of_copies(self['recid'])
description:
"""Number of copies"""
number_of_reviews:
calculated:
@parse_first('recid')
@memoize()
get_number_of_reviews(self.get('recid'))
description:
"""Number of reviews"""
number_of_comments:
calculated:
@parse_first('recid')
@memoize(30)
get_number_of_comments(self.get('recid'))
description:
"""Number of comments"""
cited_by_count:
calculated:
@parse_first('recid')
@memoize()
get_cited_by_count(self.get('recid'))
description:
"""How many records cite given record"""
diff --git a/invenio/modules/records/testsuite/test_legacy_record.py b/invenio/modules/records/testsuite/test_legacy_record.py
index 20951caf9..e4cd8cb4c 100644
--- a/invenio/modules/records/testsuite/test_legacy_record.py
+++ b/invenio/modules/records/testsuite/test_legacy_record.py
@@ -1,1676 +1,1803 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
The BibRecord test suite.
"""
import os
import pkg_resources
from invenio.base.wrappers import lazy_import
from invenio.testsuite import make_test_suite, run_test_suite, InvenioTestCase
bibrecord = lazy_import('invenio.legacy.bibrecord')
bibrecord_config = lazy_import('invenio.legacy.bibrecord.bibrecord_config')
try:
import pyRXP
parser_pyrxp_available = True
except ImportError:
parser_pyrxp_available = False
try:
from lxml import etree
parser_lxml_available = True
except ImportError:
parser_lxml_available = False
try:
import Ft.Xml.Domlette
parser_4suite_available = True
except ImportError:
parser_4suite_available = False
try:
import xml.dom.minidom
import xml.parsers.expat
parser_minidom_available = True
except ImportError:
parser_minidom_available = False
class BibRecordSuccessTest(InvenioTestCase):
""" bibrecord - demo file parsing test """
def setUp(self):
"""Initialize stuff"""
xmltext = pkg_resources.resource_string('invenio.testsuite',
os.path.join('data', 'demo_record_marc_data.xml'))
self.recs = [rec[0] for rec in bibrecord.create_records(xmltext)]
def test_records_created(self):
""" bibrecord - demo file how many records are created """
self.assertEqual(141, len(self.recs))
def test_tags_created(self):
""" bibrecord - demo file which tags are created """
## check if the tags are correct
tags = ['003', '005', '020', '024', '035', '037', '041', '080', '084', '088',
'100', '110', '148', '150', '242', '245', '246', '250', '260', '269',
'270', '300', '340', '371', '372', '400', '410', '430', '440', '450',
'490', '500', '502', '506', '510', '520', '542', '550', '588', '590',
'595', '643', '650', '653', '670', '678', '680', '690', '691', '693',
'694', '695', '697', '700', '710', '711', '720', '773', '852', '856',
'859', '901', '909', '913', '914', '916', '920', '960', '961', '962',
'963', '964', '970', '980', '999', 'FFT']
t = []
for rec in self.recs:
t.extend(rec.keys())
t.sort()
#eliminate the elements repeated
tt = []
for x in t:
if not x in tt:
tt.append(x)
self.assertEqual(tags, tt)
def test_fields_created(self):
"""bibrecord - demo file how many fields are created"""
## check if the number of fields for each record is correct
fields = [14, 14, 8, 11, 11, 13, 11, 15, 10, 18, 15, 16,
10, 9, 15, 10, 11, 11, 11, 9, 11, 11, 10, 9, 9, 9,
10, 9, 10, 10, 8, 9, 8, 9, 14, 13, 14, 14, 15, 12,
13, 12, 15, 15, 13, 16, 16, 15, 15, 14, 16, 15, 15,
15, 16, 15, 16, 15, 15, 16, 15, 15, 14, 15, 12, 13,
11, 15, 8, 11, 14, 13, 12, 13, 6, 6, 25, 24, 27, 26,
26, 24, 26, 26, 25, 28, 24, 23, 27, 25, 25, 26, 26,
25, 20, 26, 25, 22, 9, 8, 9, 9, 8, 7, 19, 21, 27, 23,
23, 22, 9, 8, 16, 7, 7, 9, 5, 5, 3, 9, 12, 6,
8, 8, 8, 13, 20, 20, 5, 8, 7, 7, 7, 7, 7, 8, 7, 8, 7, 7, 8]
cr = []
ret = []
for rec in self.recs:
cr.append(len(rec.values()))
ret.append(rec)
self.assertEqual(fields, cr, "\n%s\n!=\n%s" % (fields, cr))
def test_create_record_with_collection_tag(self):
""" bibrecord - create_record() for single record in collection"""
xmltext = """
<collection>
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
</record>
</collection>
"""
record = bibrecord.create_record(xmltext)
record1 = bibrecord.create_records(xmltext)[0]
self.assertEqual(record1, record)
class BibRecordParsersTest(InvenioTestCase):
""" bibrecord - testing the creation of records with different parsers"""
def setUp(self):
"""Initialize stuff"""
self.xmltext = """
<!-- A first comment -->
<collection>
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<!-- A second comment -->
<subfield code="a">eng</subfield>
</datafield>
</record>
</collection>
"""
self.expected_record = {
'001': [([], ' ', ' ', '33', 1)],
'041': [([('a', 'eng')], ' ', ' ', '', 2)]
}
if parser_pyrxp_available:
def test_pyRXP(self):
""" bibrecord - create_record() with pyRXP """
record = bibrecord._create_record_rxp(self.xmltext)
self.assertEqual(record, self.expected_record)
if parser_lxml_available:
def test_lxml(self):
""" bibrecord - create_record() with lxml"""
record = bibrecord._create_record_lxml(self.xmltext)
self.assertEqual(record, self.expected_record)
if parser_4suite_available:
def test_4suite(self):
""" bibrecord - create_record() with 4suite """
record = bibrecord._create_record_4suite(self.xmltext)
self.assertEqual(record, self.expected_record)
if parser_minidom_available:
def test_minidom(self):
""" bibrecord - create_record() with minidom """
record = bibrecord._create_record_minidom(self.xmltext)
self.assertEqual(record, self.expected_record)
+class BibRecordDropDuplicateFieldsTest(InvenioTestCase):
+ def test_drop_duplicate_fields(self):
+ """bibrecord - testing record_drop_duplicate_fields()"""
+ record = """
+ <record>
+ <controlfield tag="001">123</controlfield>
+ <datafield tag="100" ind1=" " ind2=" ">
+ <subfield code="a">Doe, John</subfield>
+ <subfield code="u">Foo University</subfield>
+ </datafield>
+ <datafield tag="100" ind1=" " ind2=" ">
+ <subfield code="a">Doe, John</subfield>
+ <subfield code="u">Foo University</subfield>
+ </datafield>
+ <datafield tag="100" ind1=" " ind2=" ">
+ <subfield code="u">Foo University</subfield>
+ <subfield code="a">Doe, John</subfield>
+ </datafield>
+ <datafield tag="100" ind1=" " ind2=" ">
+ <subfield code="a">Doe, John</subfield>
+ <subfield code="u">Foo University</subfield>
+ <subfield code="a">Doe, John</subfield>
+ </datafield>
+ <datafield tag="245" ind1=" " ind2=" ">
+ <subfield cde="a">On the foo and bar</subfield>
+ </datafield>
+ <datafield tag="100" ind1=" " ind2=" ">
+ <subfield code="a">Doe, John</subfield>
+ <subfield code="u">Foo University</subfield>
+ </datafield>
+ </record>
+ """
+ record_result = """
+ <record>
+ <controlfield tag="001">123</controlfield>
+ <datafield tag="100" ind1=" " ind2=" ">
+ <subfield code="a">Doe, John</subfield>
+ <subfield code="u">Foo University</subfield>
+ </datafield>
+ <datafield tag="100" ind1=" " ind2=" ">
+ <subfield code="u">Foo University</subfield>
+ <subfield code="a">Doe, John</subfield>
+ </datafield>
+ <datafield tag="100" ind1=" " ind2=" ">
+ <subfield code="a">Doe, John</subfield>
+ <subfield code="u">Foo University</subfield>
+ <subfield code="a">Doe, John</subfield>
+ </datafield>
+ <datafield tag="245" ind1=" " ind2=" ">
+ <subfield cde="a">On the foo and bar</subfield>
+ </datafield>
+ </record>
+ """
+ rec = bibrecord.create_record(record)[0]
+ rec = bibrecord.record_drop_duplicate_fields(rec)
+ rec2 = bibrecord.create_record(record_result)[0]
+ self.maxDiff = None
+ self.assertEqual(rec, rec2)
+
+
class BibRecordBadInputTreatmentTest(InvenioTestCase):
""" bibrecord - testing for bad input treatment """
def test_empty_collection(self):
"""bibrecord - empty collection"""
xml_error0 = """<collection></collection>"""
rec = bibrecord.create_record(xml_error0)[0]
self.assertEqual(rec, {})
records = bibrecord.create_records(xml_error0)
self.assertEqual(len(records), 0)
def test_wrong_attribute(self):
"""bibrecord - bad input subfield \'cde\' instead of \'code\'"""
ws = bibrecord.CFG_BIBRECORD_WARNING_MSGS
xml_error1 = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe, John</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2=" ">
<subfield cde="a">On the foo and bar</subfield>
</datafield>
</record>
"""
e = bibrecord.create_record(xml_error1, 1, 1)[2]
ee =''
for i in e:
if type(i).__name__ == 'str':
if i.count(ws[3])>0:
ee = i
self.assertEqual(bibrecord._warning((3, '(field number: 4)')), ee)
def test_missing_attribute(self):
""" bibrecord - bad input missing \"tag\" """
ws = bibrecord.CFG_BIBRECORD_WARNING_MSGS
xml_error2 = """
<record>
<controlfield tag="001">33</controlfield>
<datafield ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe, John</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2=" ">
<subfield code="a">On the foo and bar</subfield>
</datafield>
</record>
"""
e = bibrecord.create_record(xml_error2, 1, 1)[2]
ee = ''
for i in e:
if type(i).__name__ == 'str':
if i.count(ws[1])>0:
ee = i
self.assertEqual(bibrecord._warning((1, '(field number(s): [2])')), ee)
def test_empty_datafield(self):
""" bibrecord - bad input no subfield """
ws = bibrecord.CFG_BIBRECORD_WARNING_MSGS
xml_error3 = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe, John</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2=" ">
<subfield code="a">On the foo and bar</subfield>
</datafield>
</record>
"""
e = bibrecord.create_record(xml_error3, 1, 1)[2]
ee = ''
for i in e:
if type(i).__name__ == 'str':
if i.count(ws[8])>0:
ee = i
self.assertEqual(bibrecord._warning((8, '(field number: 2)')), ee)
def test_missing_tag(self):
"""bibrecord - bad input missing end \"tag\" """
ws = bibrecord.CFG_BIBRECORD_WARNING_MSGS
xml_error4 = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe, John</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2=" ">
<subfield code="a">On the foo and bar</subfield>
</record>
"""
e = bibrecord.create_record(xml_error4, 1, 1)[2]
ee = ''
for i in e:
if type(i).__name__ == 'str':
if i.count(ws[99])>0:
ee = i
self.assertEqual(bibrecord._warning((99, '(Tagname : datafield)')), ee)
class BibRecordAccentedUnicodeLettersTest(InvenioTestCase):
""" bibrecord - testing accented UTF-8 letters """
def setUp(self):
"""Initialize stuff"""
self.xml_example_record = """<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Döè1, John</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe2, J>ohn</subfield>
<subfield code="b">editor</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="1">
<subfield code="a">Пушкин</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="2">
<subfield code="a">On the foo and bar2</subfield>
</datafield>
</record>"""
self.rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0]
def test_accented_unicode_characters(self):
"""bibrecord - accented Unicode letters"""
self.assertEqual(self.xml_example_record,
bibrecord.record_xml_output(self.rec))
self.assertEqual(bibrecord.record_get_field_instances(self.rec, "100", " ", " "),
[([('a', 'Döè1, John')], " ", " ", "", 3), ([('a', 'Doe2, J>ohn'), ('b', 'editor')], " ", " ", "", 4)])
self.assertEqual(bibrecord.record_get_field_instances(self.rec, "245", " ", "1"),
[([('a', 'Пушкин')], " ", '1', "", 5)])
class BibRecordGettingFieldValuesTest(InvenioTestCase):
""" bibrecord - testing for getting field/subfield values """
def setUp(self):
"""Initialize stuff"""
xml_example_record = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe1, John</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe2, John</subfield>
<subfield code="b">editor</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="1">
<subfield code="a">On the foo and bar1</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="2">
<subfield code="a">On the foo and bar2</subfield>
</datafield>
<datafield tag="700" ind1=" " ind2="2">
<subfield code="a">Penrose, Roger</subfield>
<subfield code="u">University College London</subfield>
</datafield>
<datafield tag="700" ind1=" " ind2="2">
<subfield code="a">Messi, Lionel</subfield>
<subfield code="u">FC Barcelona</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
def test_get_field_instances(self):
"""bibrecord - getting field instances"""
self.assertEqual(bibrecord.record_get_field_instances(self.rec, "100", " ", " "),
[([('a', 'Doe1, John')], " ", " ", "", 3), ([('a', 'Doe2, John'), ('b', 'editor')], " ", " ", "", 4)])
self.assertEqual(bibrecord.record_get_field_instances(self.rec, "", " ", " "),
[('245', [([('a', 'On the foo and bar1')], " ", '1', "", 5), ([('a', 'On the foo and bar2')], " ", '2', "", 6)]), ('001', [([], " ", " ", '33', 1)]), ('700', [([('a', 'Penrose, Roger'), ('u', "University College London")], ' ', '2', '', 7), ([('a', 'Messi, Lionel'), ('u', 'FC Barcelona')], ' ', '2', '', 8)]), ('100', [([('a', 'Doe1, John')], " ", " ", "", 3), ([('a', 'Doe2, John'), ('b', 'editor')], " ", " ", "", 4)]), ('041', [([('a', 'eng')], " ", " ", "", 2)]),])
def test_get_field_values(self):
"""bibrecord - getting field values"""
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "a"),
['Doe1, John', 'Doe2, John'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "b"),
['editor'])
def test_get_field_value(self):
"""bibrecord - getting first field value"""
self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", " ", "a"),
'Doe1, John')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", " ", "b"),
'editor')
def test_get_subfield_values(self):
"""bibrecord - getting subfield values"""
fi1, fi2 = bibrecord.record_get_field_instances(self.rec, "100", " ", " ")
self.assertEqual(bibrecord.field_get_subfield_values(fi1, "b"), [])
self.assertEqual(bibrecord.field_get_subfield_values(fi2, "b"), ["editor"])
def test_filter_field(self):
"""bibrecord - filter field instances"""
field_instances = bibrecord.record_get_field_instances(self.rec, "700", "%", "%")
out = bibrecord.filter_field_instances(field_instances, "u", "University College London", 'e')
self.assertEqual(out, [([('a', 'Penrose, Roger'), ('u', "University College London")], ' ', '2', '', 7)])
out = bibrecord.filter_field_instances(field_instances, "u", "Bar", "s")
self.assertEqual(out, [([('a', 'Messi, Lionel'), ('u', 'FC Barcelona')], ' ', '2', '', 8)])
out = bibrecord.filter_field_instances(field_instances, "u", "on", "s")
self.assertEqual(out, [([('a', 'Penrose, Roger'), ('u', "University College London")], ' ', '2', '', 7),
([('a', 'Messi, Lionel'), ('u', 'FC Barcelona')], ' ', '2', '', 8)])
out = bibrecord.filter_field_instances(field_instances, "u", r".*\scoll", "r")
self.assertEqual(out,[])
out = bibrecord.filter_field_instances(field_instances, "u", r"[FC]{2}\s.*", "r")
self.assertEqual(out, [([('a', 'Messi, Lionel'), ('u', 'FC Barcelona')], ' ', '2', '', 8)])
class BibRecordGettingFieldValuesViaWildcardsTest(InvenioTestCase):
""" bibrecord - testing for getting field/subfield values via wildcards """
def setUp(self):
"""Initialize stuff"""
xml_example_record = """
<record>
<controlfield tag="001">1</controlfield>
<datafield tag="100" ind1="C" ind2="5">
<subfield code="a">val1</subfield>
</datafield>
<datafield tag="555" ind1="A" ind2="B">
<subfield code="a">val2</subfield>
</datafield>
<datafield tag="555" ind1="A" ind2=" ">
<subfield code="a">val3</subfield>
</datafield>
<datafield tag="555" ind1=" " ind2=" ">
<subfield code="a">val4a</subfield>
<subfield code="b">val4b</subfield>
</datafield>
<datafield tag="555" ind1=" " ind2="B">
<subfield code="a">val5</subfield>
</datafield>
<datafield tag="556" ind1="A" ind2="C">
<subfield code="a">val6</subfield>
</datafield>
<datafield tag="556" ind1="A" ind2=" ">
<subfield code="a">val7a</subfield>
<subfield code="b">val7b</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
def test_get_field_instances_via_wildcard(self):
"""bibrecord - getting field instances via wildcards"""
self.assertEqual(bibrecord.record_get_field_instances(self.rec, "100", " ", " "),
[])
self.assertEqual(bibrecord.record_get_field_instances(self.rec, "100", "%", " "),
[])
self.assertEqual(bibrecord.record_get_field_instances(self.rec, "100", "%", "%"),
[([('a', 'val1')], 'C', '5', "", 2)])
self.assertEqual(bibrecord.record_get_field_instances(self.rec, "55%", "A", "%"),
[([('a', 'val2')], 'A', 'B', "", 3),
([('a', 'val3')], 'A', " ", "", 4),
([('a', 'val6')], 'A', 'C', "", 7),
([('a', 'val7a'), ('b', 'val7b')], 'A', " ", "", 8)])
self.assertEqual(bibrecord.record_get_field_instances(self.rec, "55%", "A", " "),
[([('a', 'val3')], 'A', " ", "", 4),
([('a', 'val7a'), ('b', 'val7b')], 'A', " ", "", 8)])
self.assertEqual(bibrecord.record_get_field_instances(self.rec, "556", "A", " "),
[([('a', 'val7a'), ('b', 'val7b')], 'A', " ", "", 8)])
def test_get_field_values_via_wildcard(self):
"""bibrecord - getting field values via wildcards"""
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", " "),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", " ", " "),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", "%", " "),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", "%", " "),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", "%", "z"),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "%"),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "a"),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", " ", "a"),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", "%", "a"),
['val1'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", "%", "%"),
['val1'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", "%", "a"),
['val2', 'val3', 'val6', 'val7a'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", " ", "a"),
['val3', 'val7a'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "556", "A", " ", "a"),
['val7a'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "555", " ", " ", " "),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "555", " ", " ", "z"),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "555", " ", " ", "%"),
['val4a', 'val4b'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", " ", " ", "b"),
['val4b'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "%", "%", "b"),
['val4b', 'val7b'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", " ", "b"),
['val7b'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", "%", "b"),
['val7b'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", " ", "a"),
['val3', 'val7a'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", "%", "a"),
['val2', 'val3', 'val6', 'val7a'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "%", "%", "a"),
['val2', 'val3', 'val4a', 'val5', 'val6', 'val7a'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", " ", " ", "a"),
['val4a'])
def test_get_field_values_filtering_exact(self):
"""bibrecord - getting field values and exact filtering"""
self.assertEqual(bibrecord.record_get_field_values(self.rec, "556", "%", "%", "%", 'a', 'val7a'),
['val7a', 'val7b'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "556", "%", "%", "a", 'a', 'val7a'),
['val7a'])
def test_get_field_values_filtering_substring(self):
"""bibrecord - getting field values and substring filtering"""
self.assertEqual(bibrecord.record_get_field_values(self.rec, "556", "%", "%", "%", 'a', '7a', 's'),
['val7a', 'val7b'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "556", "%", "%", "b", 'a', '7a', 's'),
['val7b'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "%", "%", "%", 'b', 'val', 's'),
['val4a', 'val4b', 'val7a', 'val7b'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "%", "%", " ", 'b', 'val', 's'),
[])
def test_get_field_values_filtering_regexp(self):
"""bibrecord - getting field values and regexp filtering"""
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "%", "%", "%", 'b', r'al', 'r'),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "%", "%", "%", 'a', r'.*al[6,7]', 'r'),
['val6', 'val7a', 'val7b'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "%", "%", "a", 'a', r'.*al[6,7]', 'r'),
['val6', 'val7a'])
def test_get_field_value_via_wildcard(self):
"""bibrecord - getting first field value via wildcards"""
self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", " ", " "),
'')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", "%", " ", " "),
'')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", "%", " "),
'')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", "%", "%", " "),
'')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", " ", "%"),
'')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", " ", "a"),
'')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", "%", " ", "a"),
'')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", "%", "%", "a"),
'val1')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", "%", "%", "%"),
'val1')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", "%", "a"),
'val2')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", " ", "a"),
'val3')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "556", "A", " ", "a"),
'val7a')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "555", " ", " ", " "),
'')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "555", " ", " ", "%"),
'val4a')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", " ", " ", "b"),
'val4b')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "%", "%", "b"),
'val4b')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", " ", "b"),
'val7b')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", "%", "b"),
'val7b')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", " ", "a"),
'val3')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", "%", "a"),
'val2')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "%", "%", "a"),
'val2')
self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", " ", " ", "a"),
'val4a')
class BibRecordAddFieldTest(InvenioTestCase):
""" bibrecord - testing adding field """
def setUp(self):
"""Initialize stuff"""
xml_example_record = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe1, John</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe2, John</subfield>
<subfield code="b">editor</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="1">
<subfield code="a">On the foo and bar1</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="2">
<subfield code="a">On the foo and bar2</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
def test_add_controlfield(self):
"""bibrecord - adding controlfield"""
field_position_global_1 = bibrecord.record_add_field(self.rec, "003",
controlfield_value="SzGeCERN")
field_position_global_2 = bibrecord.record_add_field(self.rec, "004",
controlfield_value="Test")
self.assertEqual(field_position_global_1, 2)
self.assertEqual(field_position_global_2, 3)
self.assertEqual(bibrecord.record_get_field_values(self.rec, "003", " ", " ", ""),
['SzGeCERN'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "004", " ", " ", ""),
['Test'])
def test_add_datafield(self):
"""bibrecord - adding datafield"""
field_position_global_1 = bibrecord.record_add_field(self.rec, "100",
subfields=[('a', 'Doe3, John')])
field_position_global_2 = bibrecord.record_add_field(self.rec, "100",
subfields= [('a', 'Doe4, John'), ('b', 'editor')])
self.assertEqual(field_position_global_1, 5)
self.assertEqual(field_position_global_2, 6)
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "a"),
['Doe1, John', 'Doe2, John', 'Doe3, John', 'Doe4, John'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "b"),
['editor', 'editor'])
def test_add_controlfield_on_desired_position(self):
"""bibrecord - adding controlfield on desired position"""
field_position_global_1 = bibrecord.record_add_field(self.rec, "005",
controlfield_value="Foo",
field_position_global=0)
field_position_global_2 = bibrecord.record_add_field(self.rec, "006",
controlfield_value="Bar",
field_position_global=0)
self.assertEqual(field_position_global_1, 7)
self.assertEqual(field_position_global_2, 8)
def test_add_datafield_on_desired_position_field_position_global(self):
"""bibrecord - adding datafield on desired global field position"""
field_position_global_1 = bibrecord.record_add_field(self.rec, "100",
subfields=[('a', 'Doe3, John')], field_position_global=0)
field_position_global_2 = bibrecord.record_add_field(self.rec, "100",
subfields=[('a', 'Doe4, John'), ('b', 'editor')], field_position_global=0)
self.assertEqual(field_position_global_1, 3)
self.assertEqual(field_position_global_2, 3)
def test_add_datafield_on_desired_position_field_position_local(self):
"""bibrecord - adding datafield on desired local field position"""
field_position_global_1 = bibrecord.record_add_field(self.rec, "100",
subfields=[('a', 'Doe3, John')], field_position_local=0)
field_position_global_2 = bibrecord.record_add_field(self.rec, "100",
subfields=[('a', 'Doe4, John'), ('b', 'editor')],
field_position_local=2)
self.assertEqual(field_position_global_1, 3)
self.assertEqual(field_position_global_2, 5)
class BibRecordManageMultipleFieldsTest(InvenioTestCase):
""" bibrecord - testing the management of multiple fields """
def setUp(self):
"""Initialize stuff"""
xml_example_record = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="245" ind1=" " ind2=" ">
<subfield code="a">subfield1</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2=" ">
<subfield code="a">subfield2</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2=" ">
<subfield code="a">subfield3</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2=" ">
<subfield code="a">subfield4</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
def test_delete_multiple_datafields(self):
"""bibrecord - deleting multiple datafields"""
self.fields = bibrecord.record_delete_fields(self.rec, '245', [1, 2])
self.assertEqual(self.fields[0],
([('a', 'subfield2')], ' ', ' ', '', 3))
self.assertEqual(self.fields[1],
([('a', 'subfield3')], ' ', ' ', '', 4))
def test_add_multiple_datafields_default_index(self):
"""bibrecord - adding multiple fields with the default index"""
fields = [([('a', 'subfield5')], ' ', ' ', '', 4),
([('a', 'subfield6')], ' ', ' ', '', 19)]
index = bibrecord.record_add_fields(self.rec, '245', fields)
self.assertEqual(index, None)
self.assertEqual(self.rec['245'][-2],
([('a', 'subfield5')], ' ', ' ', '', 6))
self.assertEqual(self.rec['245'][-1],
([('a', 'subfield6')], ' ', ' ', '', 7))
def test_add_multiple_datafields_with_index(self):
"""bibrecord - adding multiple fields with an index"""
fields = [([('a', 'subfield5')], ' ', ' ', '', 4),
([('a', 'subfield6')], ' ', ' ', '', 19)]
index = bibrecord.record_add_fields(self.rec, '245', fields,
field_position_local=0)
self.assertEqual(index, 0)
self.assertEqual(self.rec['245'][0],
([('a', 'subfield5')], ' ', ' ', '', 2))
self.assertEqual(self.rec['245'][1],
([('a', 'subfield6')], ' ', ' ', '', 3))
self.assertEqual(self.rec['245'][2],
([('a', 'subfield1')], ' ', ' ', '', 4))
def test_move_multiple_fields(self):
"""bibrecord - move multiple fields"""
bibrecord.record_move_fields(self.rec, '245', [1, 3])
self.assertEqual(self.rec['245'][0],
([('a', 'subfield1')], ' ', ' ', '', 2))
self.assertEqual(self.rec['245'][1],
([('a', 'subfield3')], ' ', ' ', '', 4))
self.assertEqual(self.rec['245'][2],
([('a', 'subfield2')], ' ', ' ', '', 5))
self.assertEqual(self.rec['245'][3],
([('a', 'subfield4')], ' ', ' ', '', 6))
class BibRecordDeleteFieldTest(InvenioTestCase):
""" bibrecord - testing field deletion """
def setUp(self):
"""Initialize stuff"""
xml_example_record = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe1, John</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe2, John</subfield>
<subfield code="b">editor</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="1">
<subfield code="a">On the foo and bar1</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="2">
<subfield code="a">On the foo and bar2</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
xml_example_record_empty = """
<record>
</record>
"""
self.rec_empty = bibrecord.create_record(xml_example_record_empty, 1, 1)[0]
def test_delete_controlfield(self):
"""bibrecord - deleting controlfield"""
bibrecord.record_delete_field(self.rec, "001", " ", " ")
self.assertEqual(bibrecord.record_get_field_values(self.rec, "001", " ", " ", " "),
[])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "b"),
['editor'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "2", "a"),
['On the foo and bar2'])
def test_delete_datafield(self):
"""bibrecord - deleting datafield"""
bibrecord.record_delete_field(self.rec, "100", " ", " ")
self.assertEqual(bibrecord.record_get_field_values(self.rec, "001", " ", " ", ""),
['33'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "b"),
[])
bibrecord.record_delete_field(self.rec, "245", " ", " ")
self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "1", "a"),
['On the foo and bar1'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "2", "a"),
['On the foo and bar2'])
bibrecord.record_delete_field(self.rec, "245", " ", "2")
self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "1", "a"),
['On the foo and bar1'])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "2", "a"),
[])
def test_add_delete_add_field_to_empty_record(self):
"""bibrecord - adding, deleting, and adding back a field to an empty record"""
field_position_global_1 = bibrecord.record_add_field(self.rec_empty, "003",
controlfield_value="SzGeCERN")
self.assertEqual(field_position_global_1, 1)
self.assertEqual(bibrecord.record_get_field_values(self.rec_empty, "003", " ", " ", ""),
['SzGeCERN'])
bibrecord.record_delete_field(self.rec_empty, "003", " ", " ")
self.assertEqual(bibrecord.record_get_field_values(self.rec_empty, "003", " ", " ", ""),
[])
field_position_global_1 = bibrecord.record_add_field(self.rec_empty, "003",
controlfield_value="SzGeCERN2")
self.assertEqual(field_position_global_1, 1)
self.assertEqual(bibrecord.record_get_field_values(self.rec_empty, "003", " ", " ", ""),
['SzGeCERN2'])
class BibRecordDeleteFieldFromTest(InvenioTestCase):
""" bibrecord - testing field deletion from position"""
def setUp(self):
"""Initialize stuff"""
xml_example_record = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe1, John</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe2, John</subfield>
<subfield code="b">editor</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="1">
<subfield code="a">On the foo and bar1</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="2">
<subfield code="a">On the foo and bar2</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
def test_delete_field_from(self):
"""bibrecord - deleting field from position"""
bibrecord.record_delete_field(self.rec, "100", field_position_global=4)
self.assertEqual(self.rec['100'], [([('a', 'Doe1, John')], ' ', ' ', '', 3)])
bibrecord.record_delete_field(self.rec, "100", field_position_global=3)
self.failIf('100' in self.rec)
bibrecord.record_delete_field(self.rec, "001", field_position_global=1)
bibrecord.record_delete_field(self.rec, "245", field_position_global=6)
self.failIf('001' in self.rec)
self.assertEqual(self.rec['245'], [([('a', 'On the foo and bar1')], ' ', '1', '', 5)])
# Some crash tests
bibrecord.record_delete_field(self.rec, '999', field_position_global=1)
bibrecord.record_delete_field(self.rec, '245', field_position_global=999)
class BibRecordAddSubfieldIntoTest(InvenioTestCase):
""" bibrecord - testing subfield addition """
def setUp(self):
"""Initialize stuff"""
xml_example_record = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe2, John</subfield>
<subfield code="b">editor</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="1">
<subfield code="a">On the foo and bar1</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="2">
<subfield code="a">On the foo and bar2</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
def test_add_subfield_into(self):
"""bibrecord - adding subfield into position"""
bibrecord.record_add_subfield_into(self.rec, "100", "b", "Samekniv",
field_position_global=3)
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "b"),
['editor', 'Samekniv'])
bibrecord.record_add_subfield_into(self.rec, "245", "x", "Elgokse",
field_position_global=4)
bibrecord.record_add_subfield_into(self.rec, "245", "x", "Fiskeflue",
subfield_position=0, field_position_global=4)
bibrecord.record_add_subfield_into(self.rec, "245", "z", "Ulriken",
subfield_position=2, field_position_global=4)
bibrecord.record_add_subfield_into(self.rec, "245", "z",
"Stortinget", subfield_position=999, field_position_global=4)
self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "1", "%"),
['Fiskeflue', 'On the foo and bar1', 'Ulriken', 'Elgokse', 'Stortinget'])
# Some crash tests
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_add_subfield_into, self.rec, "187", "x", "Crash",
field_position_global=1)
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_add_subfield_into, self.rec, "245", "x", "Crash",
field_position_global=999)
class BibRecordModifyControlfieldTest(InvenioTestCase):
""" bibrecord - testing controlfield modification """
def setUp(self):
"""Initialize stuff"""
xml_example_record = """
<record>
<controlfield tag="001">33</controlfield>
<controlfield tag="005">A Foo's Tale</controlfield>
<controlfield tag="008">Skeech Skeech</controlfield>
<controlfield tag="008">Whoop Whoop</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="2">
<subfield code="a">On the foo and bar2</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
def test_modify_controlfield(self):
"""bibrecord - modify controlfield"""
bibrecord.record_modify_controlfield(self.rec, "001", "34",
field_position_global=1)
bibrecord.record_modify_controlfield(self.rec, "008", "Foo Foo",
field_position_global=3)
self.assertEqual(bibrecord.record_get_field_values(self.rec, "001"), ["34"])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "005"), ["A Foo's Tale"])
self.assertEqual(bibrecord.record_get_field_values(self.rec, "008"), ["Foo Foo", "Whoop Whoop"])
# Some crash tests
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_modify_controlfield, self.rec, "187", "Crash",
field_position_global=1)
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_modify_controlfield, self.rec, "008", "Test",
field_position_global=10)
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_modify_controlfield, self.rec, "245", "Burn",
field_position_global=5)
self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "2", "%"),
["On the foo and bar2"])
class BibRecordModifySubfieldTest(InvenioTestCase):
""" bibrecord - testing subfield modification """
def setUp(self):
"""Initialize stuff"""
xml_example_record = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe2, John</subfield>
<subfield code="b">editor</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="1">
<subfield code="a">On the foo and bar1</subfield>
<subfield code="b">On writing unit tests</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="2">
<subfield code="a">On the foo and bar2</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
def test_modify_subfield(self):
"""bibrecord - modify subfield"""
bibrecord.record_modify_subfield(self.rec, "245", "a", "Holmenkollen",
0, field_position_global=4)
bibrecord.record_modify_subfield(self.rec, "245", "x", "Brann", 1,
field_position_global=4)
self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "1", "%"),
['Holmenkollen', 'Brann'])
# Some crash tests
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_modify_subfield, self.rec, "187", "x", "Crash", 0,
field_position_global=1)
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_modify_subfield, self.rec, "245", "x", "Burn", 1,
field_position_global=999)
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_modify_subfield, self.rec, "245", "a", "Burn",
999, field_position_global=4)
class BibRecordDeleteSubfieldFromTest(InvenioTestCase):
""" bibrecord - testing subfield deletion """
def setUp(self):
"""Initialize stuff"""
xml_example_record = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe2, John</subfield>
<subfield code="b">editor</subfield>
<subfield code="z">Skal vi danse?</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="1">
<subfield code="a">On the foo and bar1</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="2">
<subfield code="a">On the foo and bar2</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
def test_delete_subfield_from(self):
"""bibrecord - delete subfield from position"""
bibrecord.record_delete_subfield_from(self.rec, "100", 2,
field_position_global=3)
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "z"),
[])
bibrecord.record_delete_subfield_from(self.rec, "100", 0,
field_position_global=3)
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "%"),
['editor'])
bibrecord.record_delete_subfield_from(self.rec, "100", 0,
field_position_global=3)
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "%"),
[])
# Some crash tests
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_delete_subfield_from, self.rec, "187", 0,
field_position_global=1)
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_delete_subfield_from, self.rec, "245", 0,
field_position_global=999)
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_delete_subfield_from, self.rec, "245", 999,
field_position_global=4)
class BibRecordDeleteSubfieldTest(InvenioTestCase):
""" bibrecord - testing subfield deletion """
def setUp(self):
"""Initialize stuff"""
self.xml_example_record = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe2, John</subfield>
<subfield code="b">editor</subfield>
<subfield code="z">Skal vi danse?</subfield>
<subfield code="a">Doe3, Zbigniew</subfield>
<subfield code="d">Doe4, Joachim</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="1">
<subfield code="a">On the foo and bar1</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="2">
<subfield code="a">On the foo and bar2</subfield>
</datafield>
<datafield tag="246" ind1="1" ind2="2">
<subfield code="c">On the foo and bar1</subfield>
</datafield>
<datafield tag="246" ind1="1" ind2="2">
<subfield code="c">On the foo and bar2</subfield>
</datafield>
</record>
"""
def test_simple_removals(self):
""" bibrecord - delete subfield by its code"""
# testing a simple removals where all the fields are removed
rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0]
bibrecord.record_delete_subfield(rec, "041", "b") # nothing should change
self.assertEqual(rec["041"][0][0], [("a", "eng")])
bibrecord.record_delete_subfield(rec, "041", "a")
self.assertEqual(rec["041"][0][0], [])
def test_indices_important(self):
""" bibrecord - delete subfield where indices are important"""
rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0]
bibrecord.record_delete_subfield(rec, "245", "a", " ", "1")
self.assertEqual(rec["245"][0][0], [])
self.assertEqual(rec["245"][1][0], [("a", "On the foo and bar2")])
bibrecord.record_delete_subfield(rec, "245", "a", " ", "2")
self.assertEqual(rec["245"][1][0], [])
def test_remove_some(self):
""" bibrecord - delete subfield when some should be preserved and some removed"""
rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0]
bibrecord.record_delete_subfield(rec, "100", "a", " ", " ")
self.assertEqual(rec["100"][0][0], [("b", "editor"), ("z", "Skal vi danse?"), ("d", "Doe4, Joachim")])
def test_more_fields(self):
""" bibrecord - delete subfield where more fits criteria"""
rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0]
bibrecord.record_delete_subfield(rec, "246", "c", "1", "2")
self.assertEqual(rec["246"][1][0], [])
self.assertEqual(rec["246"][0][0], [])
def test_nonexisting_removals(self):
""" bibrecord - delete subfield that does not exist """
rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0]
# further preparation
bibrecord.record_delete_subfield(rec, "100", "a", " ", " ")
self.assertEqual(rec["100"][0][0], [("b", "editor"), ("z", "Skal vi danse?"), ("d", "Doe4, Joachim")])
#the real tests begin
# 1) removing the subfield from an empty list of subfields
bibrecord.record_delete_subfield(rec, "246", "c", "1", "2")
self.assertEqual(rec["246"][1][0], [])
self.assertEqual(rec["246"][0][0], [])
bibrecord.record_delete_subfield(rec, "246", "8", "1", "2")
self.assertEqual(rec["246"][1][0], [])
self.assertEqual(rec["246"][0][0], [])
# 2) removing a subfield from a field that has some subfields but none has an appropriate code
bibrecord.record_delete_subfield(rec, "100", "a", " ", " ")
self.assertEqual(rec["100"][0][0], [("b", "editor"), ("z", "Skal vi danse?"), ("d", "Doe4, Joachim")])
bibrecord.record_delete_subfield(rec, "100", "e", " ", " ")
self.assertEqual(rec["100"][0][0], [("b", "editor"), ("z", "Skal vi danse?"), ("d", "Doe4, Joachim")])
class BibRecordMoveSubfieldTest(InvenioTestCase):
""" bibrecord - testing subfield moving """
def setUp(self):
"""Initialize stuff"""
xml_example_record = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Doe2, John</subfield>
<subfield code="b">editor</subfield>
<subfield code="c">fisk</subfield>
<subfield code="d">eple</subfield>
<subfield code="e">hammer</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2="1">
<subfield code="a">On the foo and bar1</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
def test_move_subfield(self):
"""bibrecord - move subfields"""
bibrecord.record_move_subfield(self.rec, "100", 2, 4,
field_position_global=3)
bibrecord.record_move_subfield(self.rec, "100", 1, 0,
field_position_global=3)
bibrecord.record_move_subfield(self.rec, "100", 2, 999,
field_position_global=3)
self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "%"),
['editor', 'Doe2, John', 'hammer', 'fisk', 'eple'])
# Some crash tests
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_move_subfield, self.rec, "187", 0, 1,
field_position_global=3)
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_move_subfield, self.rec, "100", 1, 0,
field_position_global=999)
self.assertRaises(bibrecord.InvenioBibRecordFieldError,
bibrecord.record_move_subfield, self.rec, "100", 999, 0,
field_position_global=3)
+
+class BibRecordCompareSubfieldTest(InvenioTestCase):
+ """ bibrecord - """
+
+ def setUp(self):
+ """Initialize stuff"""
+ xml_example_record = """
+ <record>
+ <controlfield tag="001">33</controlfield>
+ <datafield tag="041" ind1=" " ind2=" ">
+ <subfield code="a">eng</subfield>
+ </datafield>
+ <datafield tag="100" ind1=" " ind2=" ">
+ <subfield code="a">Doe2, John</subfield>
+ <subfield code="b">editor</subfield>
+ <subfield code="c">fisk</subfield>
+ <subfield code="d">eple</subfield>
+ <subfield code="e">hammer</subfield>
+ </datafield>
+ <datafield tag="245" ind1=" " ind2="1">
+ <subfield code="a">On the foo and bar1</subfield>
+ </datafield>
+ </record>
+ """
+ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
+ # For simplicity, create an alias of the function
+ self._func = bibrecord.record_match_subfields
+
+ def test_check_subfield_exists(self):
+ self.assertEqual(self._func(self.rec, '100', sub_key='a'), 3)
+ self.assertEqual(self._func(self.rec, '100', sub_key='e'), 3)
+ self.assertFalse(self._func(self.rec, '245', sub_key='a'))
+ self.assertEqual(self._func(self.rec, '245', ind2='1', sub_key='a'), 4)
+ self.assertFalse(self._func(self.rec, '999', sub_key='x'))
+ self.assertFalse(self._func(self.rec, '100', sub_key='x'))
+
+ def test_check_subfield_values(self):
+ self.assertEqual(self._func(self.rec, '100', sub_key='b',
+ sub_value='editor'), 3)
+ self.assertEqual(self._func(self.rec, '245', ind2='1', sub_key='a',
+ sub_value='On the foo and bar1'), 4)
+ self.assertEqual(self._func(self.rec, '100', sub_key='e',
+ sub_value='ponies suck'), False)
+ self.assertEqual(self._func(self.rec, '100', sub_key='c',
+ sub_value='FISK'), False)
+ self.assertEqual(self._func(self.rec, '100', sub_key='c',
+ sub_value='FISK', case_sensitive=False), 3)
+
+ def test_compare_subfields(self):
+ self.assertEqual(self._func(self.rec, '100', sub_key='c',
+ sub_value='fisk', sub_key2='d', sub_value2='eple'), 3)
+ self.assertFalse(self._func(self.rec, '100', sub_key='c',
+ sub_value='fisk', sub_key2='d', sub_value2='tom'))
+ self.assertEqual(self._func(self.rec, '100', sub_key='c',
+ sub_value='fiSk', sub_key2='d', sub_value2='Eple',
+ case_sensitive=False), 3)
+
+ def test_error_conditions(self):
+ self.assertRaises(TypeError,
+ self._func, self.rec, '100')
+ self.assertRaises(TypeError,
+ self._func, self.rec, '100', sub_key='a',
+ sub_value='fiSk', sub_key2='d')
+
+
class BibRecordSpecialTagParsingTest(InvenioTestCase):
""" bibrecord - parsing special tags (FMT, FFT)"""
def setUp(self):
"""setting up example records"""
self.xml_example_record_with_fmt = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="FMT" ind1=" " ind2=" ">
<subfield code="f">HB</subfield>
<subfield code="g">Let us see if this gets inserted well.</subfield>
</datafield>
</record>
"""
self.xml_example_record_with_fft = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="FFT" ind1=" " ind2=" ">
<subfield code="a">file:///foo.pdf</subfield>
<subfield code="a">http://bar.com/baz.ps.gz</subfield>
</datafield>
</record>
"""
self.xml_example_record_with_xyz = """
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="041" ind1=" " ind2=" ">
<subfield code="a">eng</subfield>
</datafield>
<datafield tag="XYZ" ind1=" " ind2=" ">
<subfield code="f">HB</subfield>
<subfield code="g">Let us see if this gets inserted well.</subfield>
</datafield>
</record>
"""
def test_parsing_file_containing_fmt_special_tag_with_correcting(self):
"""bibrecord - parsing special FMT tag, correcting on"""
rec = bibrecord.create_record(self.xml_example_record_with_fmt, 1, 1)[0]
self.assertEqual(rec,
{u'001': [([], " ", " ", '33', 1)],
'FMT': [([('f', 'HB'), ('g', 'Let us see if this gets inserted well.')], " ", " ", "", 3)],
'041': [([('a', 'eng')], " ", " ", "", 2)]})
self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"),
['eng'])
self.assertEqual(bibrecord.record_get_field_values(rec, "FMT", " ", " ", "f"),
['HB'])
self.assertEqual(bibrecord.record_get_field_values(rec, "FMT", " ", " ", "g"),
['Let us see if this gets inserted well.'])
def test_parsing_file_containing_fmt_special_tag_without_correcting(self):
"""bibrecord - parsing special FMT tag, correcting off"""
rec = bibrecord.create_record(self.xml_example_record_with_fmt, 1, 0)[0]
self.assertEqual(rec,
{u'001': [([], " ", " ", '33', 1)],
'FMT': [([('f', 'HB'), ('g', 'Let us see if this gets inserted well.')], " ", " ", "", 3)],
'041': [([('a', 'eng')], " ", " ", "", 2)]})
self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"),
['eng'])
self.assertEqual(bibrecord.record_get_field_values(rec, "FMT", " ", " ", "f"),
['HB'])
self.assertEqual(bibrecord.record_get_field_values(rec, "FMT", " ", " ", "g"),
['Let us see if this gets inserted well.'])
def test_parsing_file_containing_fft_special_tag_with_correcting(self):
"""bibrecord - parsing special FFT tag, correcting on"""
rec = bibrecord.create_record(self.xml_example_record_with_fft, 1, 1)[0]
self.assertEqual(rec,
{u'001': [([], " ", " ", '33', 1)],
'FFT': [([('a', 'file:///foo.pdf'), ('a', 'http://bar.com/baz.ps.gz')], " ", " ", "", 3)],
'041': [([('a', 'eng')], " ", " ", "", 2)]})
self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"),
['eng'])
self.assertEqual(bibrecord.record_get_field_values(rec, "FFT", " ", " ", "a"),
['file:///foo.pdf', 'http://bar.com/baz.ps.gz'])
def test_parsing_file_containing_fft_special_tag_without_correcting(self):
"""bibrecord - parsing special FFT tag, correcting off"""
rec = bibrecord.create_record(self.xml_example_record_with_fft, 1, 0)[0]
self.assertEqual(rec,
{u'001': [([], " ", " ", '33', 1)],
'FFT': [([('a', 'file:///foo.pdf'), ('a', 'http://bar.com/baz.ps.gz')], " ", " ", "", 3)],
'041': [([('a', 'eng')], " ", " ", "", 2)]})
self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"),
['eng'])
self.assertEqual(bibrecord.record_get_field_values(rec, "FFT", " ", " ", "a"),
['file:///foo.pdf', 'http://bar.com/baz.ps.gz'])
def test_parsing_file_containing_xyz_special_tag_with_correcting(self):
"""bibrecord - parsing unrecognized special XYZ tag, correcting on"""
# XYZ should not get accepted when correcting is on; should get changed to 000
rec = bibrecord.create_record(self.xml_example_record_with_xyz, 1, 1)[0]
self.assertEqual(rec,
{u'001': [([], " ", " ", '33', 1)],
'000': [([('f', 'HB'), ('g', 'Let us see if this gets inserted well.')], " ", " ", "", 3)],
'041': [([('a', 'eng')], " ", " ", "", 2)]})
self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"),
['eng'])
self.assertEqual(bibrecord.record_get_field_values(rec, "XYZ", " ", " ", "f"),
[])
self.assertEqual(bibrecord.record_get_field_values(rec, "XYZ", " ", " ", "g"),
[])
self.assertEqual(bibrecord.record_get_field_values(rec, "000", " ", " ", "f"),
['HB'])
self.assertEqual(bibrecord.record_get_field_values(rec, "000", " ", " ", "g"),
['Let us see if this gets inserted well.'])
def test_parsing_file_containing_xyz_special_tag_without_correcting(self):
"""bibrecord - parsing unrecognized special XYZ tag, correcting off"""
# XYZ should get accepted without correcting
rec = bibrecord.create_record(self.xml_example_record_with_xyz, 1, 0)[0]
self.assertEqual(rec,
{u'001': [([], " ", " ", '33', 1)],
'XYZ': [([('f', 'HB'), ('g', 'Let us see if this gets inserted well.')], " ", " ", "", 3)],
'041': [([('a', 'eng')], " ", " ", "", 2)]})
self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"),
['eng'])
self.assertEqual(bibrecord.record_get_field_values(rec, "XYZ", " ", " ", "f"),
['HB'])
self.assertEqual(bibrecord.record_get_field_values(rec, "XYZ", " ", " ", "g"),
['Let us see if this gets inserted well.'])
class BibRecordPrintingTest(InvenioTestCase):
""" bibrecord - testing for printing record """
def setUp(self):
"""Initialize stuff"""
self.xml_example_record = """
<record>
<controlfield tag="001">81</controlfield>
<datafield tag="037" ind1=" " ind2=" ">
<subfield code="a">TEST-ARTICLE-2006-001</subfield>
</datafield>
<datafield tag="037" ind1=" " ind2=" ">
<subfield code="a">ARTICLE-2006-001</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2=" ">
<subfield code="a">Test ti</subfield>
</datafield>
</record>"""
self.xml_example_record_short = """
<record>
<controlfield tag="001">81</controlfield>
<datafield tag="037" ind1=" " ind2=" ">
<subfield code="a">TEST-ARTICLE-2006-001</subfield>
</datafield>
<datafield tag="037" ind1=" " ind2=" ">
<subfield code="a">ARTICLE-2006-001</subfield>
</datafield>
</record>"""
self.xml_example_multi_records = """
<record>
<controlfield tag="001">81</controlfield>
<datafield tag="037" ind1=" " ind2=" ">
<subfield code="a">TEST-ARTICLE-2006-001</subfield>
</datafield>
<datafield tag="037" ind1=" " ind2=" ">
<subfield code="a">ARTICLE-2006-001</subfield>
</datafield>
<datafield tag="245" ind1=" " ind2=" ">
<subfield code="a">Test ti</subfield>
</datafield>
</record>
<record>
<controlfield tag="001">82</controlfield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Author, t</subfield>
</datafield>
</record>"""
self.xml_example_multi_records_short = """
<record>
<controlfield tag="001">81</controlfield>
<datafield tag="037" ind1=" " ind2=" ">
<subfield code="a">TEST-ARTICLE-2006-001</subfield>
</datafield>
<datafield tag="037" ind1=" " ind2=" ">
<subfield code="a">ARTICLE-2006-001</subfield>
</datafield>
</record>
<record>
<controlfield tag="001">82</controlfield>
</record>"""
def test_record_xml_output(self):
"""bibrecord - xml output"""
rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0]
rec_short = bibrecord.create_record(self.xml_example_record_short, 1, 1)[0]
self.assertEqual(bibrecord.create_record(bibrecord.record_xml_output(rec, tags=[]), 1, 1)[0], rec)
self.assertEqual(bibrecord.create_record(bibrecord.record_xml_output(rec, tags=["001", "037"]), 1, 1)[0], rec_short)
self.assertEqual(bibrecord.create_record(bibrecord.record_xml_output(rec, tags=["037"]), 1, 1)[0], rec_short)
class BibRecordCreateFieldTest(InvenioTestCase):
""" bibrecord - testing for creating field """
def test_create_valid_field(self):
"""bibrecord - create and check a valid field"""
bibrecord.create_field()
bibrecord.create_field([('a', 'testa'), ('b', 'testb')], '2', 'n',
'controlfield', 15)
def test_invalid_field_raises_exception(self):
"""bibrecord - exception raised when creating an invalid field"""
# Invalid subfields.
self.assertRaises(bibrecord_config.InvenioBibRecordFieldError,
bibrecord.create_field, 'subfields', '1', '2', 'controlfield', 10)
self.assertRaises(bibrecord_config.InvenioBibRecordFieldError,
bibrecord.create_field, ('1', 'value'), '1', '2', 'controlfield', 10)
self.assertRaises(bibrecord_config.InvenioBibRecordFieldError,
bibrecord.create_field, [('value')], '1', '2', 'controlfield', 10)
self.assertRaises(bibrecord_config.InvenioBibRecordFieldError,
bibrecord.create_field, [('1', 'value', '2')], '1', '2', 'controlfield', 10)
# Invalid indicators.
self.assertRaises(bibrecord_config.InvenioBibRecordFieldError,
bibrecord.create_field, [], 1, '2', 'controlfield', 10)
self.assertRaises(bibrecord_config.InvenioBibRecordFieldError,
bibrecord.create_field, [], '1', 2, 'controlfield', 10)
# Invalid controlfield value
self.assertRaises(bibrecord_config.InvenioBibRecordFieldError,
bibrecord.create_field, [], '1', '2', 13, 10)
# Invalid global position
self.assertRaises(bibrecord_config.InvenioBibRecordFieldError,
bibrecord.create_field, [], '1', '2', 'controlfield', 'position')
def test_compare_fields(self):
"""bibrecord - compare fields"""
# Identical
field0 = ([('a', 'test')], '1', '2', '', 0)
field1 = ([('a', 'test')], '1', '2', '', 3)
self.assertEqual(True,
bibrecord._compare_fields(field0, field1, strict=True))
self.assertEqual(True,
bibrecord._compare_fields(field0, field1, strict=False))
# Order of the subfields changed.
field0 = ([('a', 'testa'), ('b', 'testb')], '1', '2', '', 0)
field1 = ([('b', 'testb'), ('a', 'testa')], '1', '2', '', 3)
self.assertEqual(False,
bibrecord._compare_fields(field0, field1, strict=True))
self.assertEqual(True,
bibrecord._compare_fields(field0, field1, strict=False))
# Different
field0 = ([], '3', '2', '', 0)
field1 = ([], '1', '2', '', 3)
self.assertEqual(False,
bibrecord._compare_fields(field0, field1, strict=True))
self.assertEqual(False,
bibrecord._compare_fields(field0, field1, strict=False))
class BibRecordFindFieldTest(InvenioTestCase):
""" bibrecord - testing for finding field """
def setUp(self):
"""Initialize stuff"""
xml = """
<record>
<controlfield tag="001">81</controlfield>
<datafield tag="037" ind1=" " ind2=" ">
<subfield code="a">TEST-ARTICLE-2006-001</subfield>
<subfield code="b">ARTICLE-2007-001</subfield>
</datafield>
</record>
"""
self.rec = bibrecord.create_record(xml)[0]
self.field0 = self.rec['001'][0]
self.field1 = self.rec['037'][0]
self.field2 = (
[self.field1[0][1], self.field1[0][0]],
self.field1[1],
self.field1[2],
self.field1[3],
self.field1[4],
)
def test_finding_field_strict(self):
"""bibrecord - test finding field strict"""
self.assertEqual((1, 0),
bibrecord.record_find_field(self.rec, '001', self.field0,
strict=True))
self.assertEqual((2, 0),
bibrecord.record_find_field(self.rec, '037', self.field1,
strict=True))
self.assertEqual((None, None),
bibrecord.record_find_field(self.rec, '037', self.field2,
strict=True))
def test_finding_field_loose(self):
"""bibrecord - test finding field loose"""
self.assertEqual((1, 0),
bibrecord.record_find_field(self.rec, '001', self.field0,
strict=False))
self.assertEqual((2, 0),
bibrecord.record_find_field(self.rec, '037', self.field1,
strict=False))
self.assertEqual((2, 0),
bibrecord.record_find_field(self.rec, '037', self.field2,
strict=False))
class BibRecordSingletonTest(InvenioTestCase):
""" bibrecord - testing singleton removal """
def setUp(self):
"""Initialize stuff"""
self.xml = """<collection>
<record>
<controlfield tag="001">33</controlfield>
<controlfield tag="002" />
<datafield tag="99" ind1=" " ind2=" "/>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a" />
</datafield>
<datafield tag="100" ind1=" " ind2=" ">
<subfield code="a">Some value</subfield>
</datafield>
<tagname />
</record>
<record />
<collection>"""
self.rec_expected = {
'001': [([], ' ', ' ', '33', 1)],
'100': [([('a', 'Some value')], ' ', ' ', '', 2)],
}
if parser_minidom_available:
def test_singleton_removal_minidom(self):
"""bibrecord - enforcing singleton removal with minidom"""
rec = bibrecord.create_records(self.xml, verbose=1,
correct=1, parser='minidom',
keep_singletons=False)[0][0]
self.assertEqual(rec, self.rec_expected)
if parser_4suite_available:
def test_singleton_removal_4suite(self):
"""bibrecord - enforcing singleton removal with 4suite"""
rec = bibrecord.create_records(self.xml, verbose=1,
correct=1, parser='4suite',
keep_singletons=False)[0][0]
self.assertEqual(rec, self.rec_expected)
if parser_pyrxp_available:
def test_singleton_removal_pyrxp(self):
"""bibrecord - enforcing singleton removal with pyrxp"""
rec = bibrecord.create_records(self.xml, verbose=1,
correct=1, parser='pyrxp',
keep_singletons=False)[0][0]
self.assertEqual(rec, self.rec_expected)
if parser_lxml_available:
def test_singleton_removal_lxml(self):
"""bibrecord - enforcing singleton removal with lxml"""
rec = bibrecord.create_records(self.xml, verbose=1,
correct=1, parser='lxml',
keep_singletons=False)[0][0]
self.assertEqual(rec, self.rec_expected)
class BibRecordNumCharRefTest(InvenioTestCase):
""" bibrecord - testing numerical character reference expansion"""
def setUp(self):
"""Initialize stuff"""
self.xml = """<?xml version="1.0" encoding="UTF-8"?>
<record>
<controlfield tag="001">33</controlfield>
<datafield tag="123" ind1=" " ind2=" ">
<subfield code="a">Σ &amp; &#931;</subfield>
<subfield code="a">use &amp;amp; in XML</subfield>
</datafield>
</record>"""
self.rec_expected = {
'001': [([], ' ', ' ', '33', 1)],
'123': [([('a', '\xce\xa3 & \xce\xa3'), ('a', 'use &amp; in XML'),], ' ', ' ', '', 2)],
}
if parser_minidom_available:
def test_numcharref_expansion_minidom(self):
"""bibrecord - numcharref expansion with minidom"""
rec = bibrecord.create_records(self.xml, verbose=1,
correct=1, parser='minidom')[0][0]
self.assertEqual(rec, self.rec_expected)
if parser_4suite_available:
def test_numcharref_expansion_4suite(self):
"""bibrecord - numcharref expansion with 4suite"""
rec = bibrecord.create_records(self.xml, verbose=1,
correct=1, parser='4suite')[0][0]
self.assertEqual(rec, self.rec_expected)
if parser_pyrxp_available:
def test_numcharref_expansion_pyrxp(self):
"""bibrecord - but *no* numcharref expansion with pyrxp (see notes)
FIXME: pyRXP does not seem to like num char ref entities,
so this test is mostly left here in a TDD style in order
to remind us of this fact. If we want to fix this
situation, then we should probably use pyRXPU that uses
Unicode strings internally, hence it is num char ref
friendly. Maybe we should use pyRXPU by default, if
performance is acceptable, or maybe we should introduce a
flag to govern this behaviour.
"""
rec = bibrecord.create_records(self.xml, verbose=1,
correct=1, parser='pyrxp')[0][0]
#self.assertEqual(rec, self.rec_expected)
self.assertEqual(rec, None)
if parser_lxml_available:
def test_numcharref_expansion_lxml(self):
"""bibrecord - numcharref expansion with lxml"""
rec = bibrecord.create_records(self.xml, verbose=1,
correct=1, parser='lxml')[0][0]
self.assertEqual(rec, self.rec_expected)
class BibRecordExtractIdentifiersTest(InvenioTestCase):
""" bibrecord - testing for getting identifiers from record """
def setUp(self):
"""Initialize stuff"""
from invenio.config import CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG
xml_example_record = """
<record>
<controlfield tag="001">1</controlfield>
<datafield tag="100" ind1="C" ind2="5">
<subfield code="a">val1</subfield>
</datafield>
<datafield tag="024" ind1="7" ind2=" ">
<subfield code="2">doi</subfield>
<subfield code="a">5555/TEST1</subfield>
</datafield>
<datafield tag="024" ind1="7" ind2=" ">
<subfield code="2">DOI</subfield>
<subfield code="a">5555/TEST2</subfield>
</datafield>
<datafield tag="024" ind1="7" ind2=" ">
<subfield code="2">nondoi</subfield>
<subfield code="a">5555/TEST3</subfield>
</datafield>
<datafield tag="024" ind1="8" ind2=" ">
<subfield code="2">doi</subfield>
<subfield code="a">5555/TEST4</subfield>
</datafield>
<datafield tag="%(oai_tag)s" ind1="%(oai_ind1)s" ind2="%(oai_ind2)s">
<subfield code="%(oai_subcode)s">oai:atlantis:1</subfield>
</datafield>
</record>
""" % {'oai_tag': CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[0:3],
'oai_ind1': CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[3],
'oai_ind2': CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[4],
'oai_subcode': CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[5],
}
self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0]
def test_extract_doi(self):
"""bibrecord - getting DOI identifier(s) from record"""
self.assertEqual(bibrecord.record_extract_dois(self.rec),
['5555/TEST1', '5555/TEST2'])
def test_extract_oai_id(self):
"""bibrecord - getting OAI identifier(s) from record"""
self.assertEqual(bibrecord.record_extract_oai_id(self.rec),
'oai:atlantis:1')
TEST_SUITE = make_test_suite(
BibRecordSuccessTest,
BibRecordParsersTest,
BibRecordBadInputTreatmentTest,
BibRecordGettingFieldValuesTest,
BibRecordGettingFieldValuesViaWildcardsTest,
BibRecordAddFieldTest,
BibRecordDeleteFieldTest,
BibRecordManageMultipleFieldsTest,
BibRecordDeleteFieldFromTest,
BibRecordAddSubfieldIntoTest,
BibRecordModifyControlfieldTest,
BibRecordModifySubfieldTest,
BibRecordDeleteSubfieldFromTest,
BibRecordMoveSubfieldTest,
+ BibRecordCompareSubfieldTest,
BibRecordAccentedUnicodeLettersTest,
BibRecordSpecialTagParsingTest,
BibRecordPrintingTest,
BibRecordCreateFieldTest,
BibRecordFindFieldTest,
BibRecordDeleteSubfieldTest,
BibRecordSingletonTest,
BibRecordNumCharRefTest,
BibRecordExtractIdentifiersTest,
+ BibRecordDropDuplicateFieldsTest
)
if __name__ == '__main__':
run_test_suite(TEST_SUITE)
diff --git a/invenio/modules/redirector/redirect_methods/goto_plugin_cern_hr_documents.py b/invenio/modules/redirector/redirect_methods/goto_plugin_cern_hr_documents.py
index 251f58daa..04cbcb312 100644
--- a/invenio/modules/redirector/redirect_methods/goto_plugin_cern_hr_documents.py
+++ b/invenio/modules/redirector/redirect_methods/goto_plugin_cern_hr_documents.py
@@ -1,155 +1,161 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2012 CERN.
+## Copyright (C) 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
This implements a redirection for CERN HR Documents in the CERN Document
Server. It's useful as a reference on how goto plugins could be implemented.
"""
import time
import re
from invenio.legacy.search_engine import perform_request_search
from invenio.legacy.bibrecord import get_fieldvalues
-from invenio.legacy.bibdocfile.api import BibRecDocs
+from invenio.legacy.bibdocfile.api import BibRecDocs, InvenioBibDocFileError
def make_cern_ssr_docname(lang, edition, modif=0):
if modif:
- return "CERN_SSR_%(lang)s_ed%(edition)02d_modif%(modif)02d" % {
+ return "CERN_SRR_%(lang)s_ed%(edition)02d_modif%(modif)02d" % {
'lang': lang,
'edition': edition,
'modif': modif
}
else:
- return "CERN_SSR_%(lang)s_ed%(edition)02d" % {
+ return "CERN_SRR_%(lang)s_ed%(edition)02d" % {
'lang': lang,
'edition': edition,
}
_RE_REVISION = re.compile(r"rev(\d\d)")
def _get_revision(docname):
"""
Return the revision in a docname. E.g.:
CERN_Circ_Op_en_02_rev01_Implementation measures.pdf -> 1
CERN_Circ_Op_en_02_rev02_Implementation measures.PDF -> 2
"""
g = _RE_REVISION.search(docname)
if g:
return int(g.group(1))
return 0
def _register_document(documents, docname, key):
"""
Register in the documents mapping the docname to key, but only if the
docname has a revision higher of the docname already associated with a key
"""
if key in documents:
if _get_revision(docname) > _get_revision(documents[key]):
documents[key] = docname
else:
documents[key] = docname
def goto(type, document='', number=0, lang='en', modif=0):
today = time.strftime('%Y-%m-%d')
- if type == 'SSR':
+ if type == 'SRR':
## We would like a CERN Staff Rules and Regulations
recids = perform_request_search(cc='Staff Rules and Regulations', f="925__a:1996-01-01->%s 925__b:%s->9999-99-99" % (today, today))
recid = recids[-1]
reportnumber = get_fieldvalues(recid, '037__a')[0]
edition = int(reportnumber[-2:]) ## e.g. CERN-STAFF-RULES-ED08
return BibRecDocs(recid).get_bibdoc(make_cern_ssr_docname(lang, edition, modif)).get_file('.pdf').get_url()
elif type == "OPER-CIRC":
- recids = perform_request_search(cc="Operational Circulars", p="reportnumber=\"CERN-OPER-CIRC-%s-*\"" % number, sf="925__a")
+ recids = perform_request_search(cc="Operational Circulars", p="reportnumber:\"CERN-OPER-CIRC-%s-*\"" % number, sf="925__a")
recid = recids[-1]
documents = {}
bibrecdocs = BibRecDocs(recid)
for docname in bibrecdocs.get_bibdoc_names():
ldocname = docname.lower()
if 'implementation' in ldocname:
- _register_document(documents, docname, 'implementation_en')
+ _register_document(documents, docname, 'implementation-en')
elif 'application' in ldocname:
- _register_document(documents, docname, 'implementation_fr')
+ _register_document(documents, docname, 'implementation-fr')
elif 'archiving' in ldocname:
- _register_document(documents, docname, 'archiving_en')
+ _register_document(documents, docname, 'archiving-en')
elif 'archivage' in ldocname:
- _register_document(documents, docname, 'archiving_fr')
+ _register_document(documents, docname, 'archiving-fr')
elif 'annexe' in ldocname or 'annexes_fr' in ldocname:
- _register_document(documents, docname, 'annex_fr')
+ _register_document(documents, docname, 'annex-fr')
elif 'annexes_en' in ldocname or 'annex' in ldocname:
- _register_document(documents, docname, 'annex_en')
+ _register_document(documents, docname, 'annex-en')
elif '_en_' in ldocname or '_eng_' in ldocname or '_angl_' in ldocname:
_register_document(documents, docname, 'en')
elif '_fr_' in ldocname:
_register_document(documents, docname, 'fr')
- return bibrecdocs.get_bibdoc(documents[document]).get_file('.pdf').get_url()
+ try:
+ return bibrecdocs.get_bibdoc(documents[document]).get_file('.pdf').get_url()
+ except InvenioBibDocFileError:
+ return bibrecdocs.get_bibdoc(documents[document]).get_file('.PDF').get_url()
elif type == 'ADMIN-CIRC':
- recids = perform_request_search(cc="Administrative Circulars", p="reportnumber=\"CERN-ADMIN-CIRC-%s-*\"" % number, sf="925__a")
+ recids = perform_request_search(cc="Administrative Circulars", p='reportnumber:"CERN-ADMIN-CIRC-%s-*"' % number, sf="925__a")
recid = recids[-1]
documents = {}
bibrecdocs = BibRecDocs(recid)
for docname in bibrecdocs.get_bibdoc_names():
ldocname = docname.lower()
if 'implementation' in ldocname:
_register_document(documents, docname, 'implementation-en')
elif 'application' in ldocname:
_register_document(documents, docname, 'implementation-fr')
elif 'archiving' in ldocname:
_register_document(documents, docname, 'archiving-en')
elif 'archivage' in ldocname:
_register_document(documents, docname, 'archiving-fr')
elif 'annexe' in ldocname or 'annexes_fr' in ldocname:
_register_document(documents, docname, 'annex-fr')
elif 'annexes_en' in ldocname or 'annex' in ldocname:
_register_document(documents, docname, 'annex-en')
elif '_en_' in ldocname or '_eng_' in ldocname or '_angl_' in ldocname:
_register_document(documents, docname, 'en')
elif '_fr_' in ldocname:
_register_document(documents, docname, 'fr')
- return bibrecdocs.get_bibdoc(documents[document]).get_file('.pdf').get_url()
+ try:
+ return bibrecdocs.get_bibdoc(documents[document]).get_file('.pdf').get_url()
+ except InvenioBibDocFileError:
+ return bibrecdocs.get_bibdoc(documents[document]).get_file('.PDF').get_url()
def register_hr_redirections():
"""
Run this only once
"""
from invenio.modules.redirector.api import register_redirection
plugin = 'goto_plugin_cern_hr_documents'
## Staff rules and regulations
for modif in range(1, 20):
for lang in ('en', 'fr'):
- register_redirection('hr-srr-modif%02d-%s' % (modif, lang), plugin, parameters={'type': 'SSR', 'lang': lang, 'modif': modif})
+ register_redirection('hr-srr-modif%02d-%s' % (modif, lang), plugin, parameters={'type': 'SRR', 'lang': lang, 'modif': modif})
for lang in ('en', 'fr'):
- register_redirection('hr-srr-%s' % lang, plugin, parameters={'type': 'SSR', 'lang': lang, 'modif': 0})
+ register_redirection('hr-srr-%s' % lang, plugin, parameters={'type': 'SRR', 'lang': lang, 'modif': 0})
## Operational Circulars
for number in range(1, 10):
for lang in ('en', 'fr'):
register_redirection('hr-oper-circ-%s-%s' % (number, lang), plugin, parameters={'type': 'OPER-CIRC', 'document': lang, 'number': number})
for number, special_document in ((2, 'implementation'), (2, 'annex'), (3, 'archiving'), (3, 'annex')):
for lang in ('en', 'fr'):
register_redirection('hr-circ-%s-%s-%s' % (number, special_document, lang), plugin, parameters={'type': 'OPER-CIRC', 'document': '%s-%s' % (special_document, lang), 'number': number})
## Administrative Circulars:
for number in range(1, 32):
for lang in ('en', 'fr'):
register_redirection('hr-admin-circ-%s-%s' % (number, lang), plugin, parameters={'type': 'ADMIN-CIRC', 'document': lang, 'number': number})
if __name__ == "__main__":
register_hr_redirections()
diff --git a/invenio/modules/scheduler/tasklets/bst_create_icons.py b/invenio/modules/scheduler/tasklets/bst_create_icons.py
new file mode 100644
index 000000000..3b42eb78e
--- /dev/null
+++ b/invenio/modules/scheduler/tasklets/bst_create_icons.py
@@ -0,0 +1,307 @@
+# -*- coding: utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2013, 2014 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
+"""Invenio Bibliographic Tasklet for generating subformats.
+ Usange:
+ $bibtasklet -N createicons -T bst_create_icons -a recid=123 -a icon_sizes=180,640,1440
+ $bibtasklet -N createicons -T bst_create_icons -a "collection=ABC Photos" -a icon_sizes=180,640,1440"""
+
+try:
+ from invenio.config import CFG_ICON_CREATION_FORMAT_MAPPINGS
+except ImportError, e:
+ CFG_ICON_CREATION_FORMAT_MAPPINGS = {'*': ['jpg']}
+from invenio.bibdocfile import BibRecDocs
+from invenio.bibdocfile_config import CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT
+from invenio.websubmit_icon_creator import create_icon, CFG_ALLOWED_FILE_EXTENSIONS
+from invenio.bibdocfilecli import cli_fix_marc
+from invenio.bibtask import write_message, \
+ task_update_progress, \
+ task_sleep_now_if_required
+
+import os
+import sys
+
+CFG_DEFAULT_ICON_EXTENSION = "gif"
+CFG_DEFAULT_ICON_SIZE = '180'
+
+def create_icons_for_record(recid, icon_sizes, icon_format_mappings=None,
+ docnames=None, add_default_icon=False, inherit_moreinfo=False):
+ """Generate icons, if missing, for a record
+ @param recid: the record id for which icons are being created
+ @type recid: int
+ @param icon_sizes: the list of icon sizes that need to be
+ generated. Note that upscaled is not allowed
+ @type icon_sizes: list
+ @param icon_format_mappings: defines for each "master" format in
+ which format the icons should be
+ created. If the master format is
+ not specified here, then its icons
+ will be created in the same format,
+ if possible (for eg. the icons of a
+ TIFF file would be created as TIFF,
+ while icons of a PDF or DOC file
+ would be created in JPG) and unless
+ a default mapping is not provided in
+ C{CFG_ICON_CREATION_FORMAT_MAPPINGS}.
+ @type icon_format_mappings: dict
+ @param docnames: the list of docnames for which we want to create an icon.
+ If not provided, consider all docnames.
+ @type docnames: list
+ @param add_default_icon: if a default icon (i.e. without icon
+ size suffix, matching
+ CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT)
+ should be added or not.
+ @type add_default_icon: bool
+ @param inherit_moreinfo: if the added icons should also have
+ their description and comment set to
+ the same value as the "main" bibdoc
+ or not.
+ @type inherit_moreinfo: bool
+ """
+ exceptions = [] # keep track of all exceptions
+ done = 0
+ brd = BibRecDocs(recid)
+ bibdocs = brd.list_bibdocs()
+ # Override default formats from CFG_ICON_CREATION_FORMAT_MAPPINGS
+ # with values specified in icon_format_mappings
+ if icon_format_mappings is None:
+ icon_format_mappings = {}
+ icon_format_mappings = dict(CFG_ICON_CREATION_FORMAT_MAPPINGS, **icon_format_mappings)
+ if icon_format_mappings.has_key('*') and \
+ not icon_format_mappings['*']:
+ # we must override the default in order to keep the
+ # "superformat"
+ del icon_format_mappings['*']
+ for bibdoc in bibdocs:
+ docname = brd.get_docname(bibdoc.id)
+ if docnames and not docname in docnames:
+ # Skip this docname
+ continue
+ comment, description = get_comment_and_description(bibdoc, inherit_moreinfo)
+ default_icon_added_p = False
+ bibfiles = bibdoc.list_latest_files()
+ bibdoc_formats = [bibfile.get_format() for bibfile in bibfiles]
+ for bibfile in bibfiles:
+ if bibfile.get_subformat():
+ # this is a subformat, do nothing
+ continue
+ filepath = bibfile.get_full_path()
+ #do not consider the dot in front of the format
+ superformat = bibfile.get_format()[1:].lower()
+ bibfile_icon_formats = icon_format_mappings.get(superformat, icon_format_mappings.get('*', [superformat]))
+ if isinstance(bibfile_icon_formats, str):
+ bibfile_icon_formats = [bibfile_icon_formats]
+ bibfile_icon_formats = [bibfile_icon_format for bibfile_icon_format in bibfile_icon_formats \
+ if bibfile_icon_format in CFG_ALLOWED_FILE_EXTENSIONS]
+
+ if add_default_icon and not default_icon_added_p:
+ # add default icon
+ try:
+ iconpath, iconname = _create_icon(filepath, CFG_DEFAULT_ICON_SIZE, docname,
+ icon_format=CFG_DEFAULT_ICON_EXTENSION, verbosity=9)
+ bibdoc.add_file_new_format(os.path.join(iconpath, iconname),
+ docformat=".%s;%s" % (CFG_DEFAULT_ICON_EXTENSION,
+ CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT),
+ comment=comment, description=description)
+ default_icon_added_p = True
+ write_message("Added default icon to recid: %s, format: %s" % (recid, CFG_DEFAULT_ICON_EXTENSION))
+ except Exception, ex:
+ exceptions.append("Could not add new icon to recid: %s, format: %s; exc: %s" \
+ % (recid, CFG_DEFAULT_ICON_EXTENSION, ex))
+
+ # check if the subformat that we want to create already exists
+ for icon_size in icon_sizes:
+ washed_icon_size = icon_size.replace('>', '').replace('<', '').replace('^', '').replace('!', '')
+ for icon_format in bibfile_icon_formats:
+ new_format = '.%s;%s-%s' % (icon_format, CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT, washed_icon_size)
+ if new_format in bibdoc_formats:
+ # the subformat already exists, do nothing
+ continue
+ # add icon
+ try:
+ iconpath, iconname = _create_icon(filepath, icon_size, docname,
+ icon_format=icon_format, verbosity=9)
+ bibdoc.add_file_new_format(os.path.join(iconpath, iconname), docformat=new_format,
+ comment=comment, description=description)
+ write_message("Added icon to recid: %s, format: %s %s %s %s" % (recid, new_format, iconpath, iconname, icon_size))
+ done += 1
+ except Exception, ex:
+ exceptions.append("Could not add new format to recid: %s, format: %s; exc: %s" \
+ %(recid, new_format, ex))
+ return done, exceptions
+
+
+def bst_create_icons(recid, icon_sizes, icon_format_mappings=None,
+ collection=None, docnames=None, add_default_icon=0, inherit_moreinfo=0):
+ """BibTasklet for generating missing icons.
+ @param recid: the record on which the action is being performed
+ @type recid: int
+ @param icon_sizes: a comma-separated list of icon sizes, ex 180,640
+ @type icon_sizes: string
+ @param collection: the collection name on which to run the task;
+ if recid is defined, collection will be ignored
+ @type collection: string
+ @param icon_format_mappings: defines for each "master" format in
+ which format the icons should be
+ created. If the master format is
+ not specified here, then its icons
+ will be created in the same format,
+ if possible (for eg. the icons of a
+ TIFF file would be created as TIFF,
+ while icons of a PDF or DOC file
+ would be created in JPG) and unless
+ a default mapping is not provided in
+ C{CFG_ICON_CREATION_FORMAT_MAPPINGS}.
+ Use syntax masterextension-targetextension1,targetextension2
+ (eg. "doc->png,jpg" or "png-jpg")
+ Use '*' to target extensions not
+ matched by other rules (if
+ necessary set its value to empty ''
+ in order to override/remove the
+ default star rule set in
+ C{CFG_ICON_CREATION_FORMAT_MAPPINGS}.
+ @type icon_format_mappings: list
+ @param docnames: the list of docnames for which we want to create an icon.
+ If not provided, consider all docnames.
+ Separate docnames using "/"
+ @type docnames: list
+ @param add_default_icon: if a default icon (i.e. without icon
+ size suffix, matching
+ CFG_BIBDOCFILE_DEFAULT_ICON_SUBFORMAT)
+ should be added (1) or not (0)
+ @type add_default_icon: int
+ @param inherit_moreinfo: if the added icons should also have
+ their description and comment set to
+ the same value as the "main" bibdoc
+ (1) or not (0)
+ @type inherit_moreinfo: int
+ """
+ if recid:
+ recids = [int(recid)]
+ elif collection:
+ from invenio.search_engine import get_collection_reclist
+ recids = get_collection_reclist(collection)
+ else:
+ write_message("Error: no recid found.", sys.stderr)
+ return 1
+ try:
+ add_default_icon = int(add_default_icon) and True or False
+ except:
+ add_default_icon = False
+ try:
+ inherit_moreinfo = int(inherit_moreinfo) and True or False
+ except:
+ inherit_moreinfo = False
+ if icon_format_mappings is None:
+ icon_format_mappings = []
+ if isinstance(icon_format_mappings, str):
+ icon_format_mappings = [icon_format_mappings]
+ try:
+ icon_format_mappings = dict([map(lambda x: ',' in x and x.split(',') or x, mapping.split("-", 1)) \
+ for mapping in icon_format_mappings])
+ except Exception, e:
+ write_message("Error: parameter 'icon_format_mappings' not well-formed:\n%s" % e, sys.stderr)
+ return 0
+
+ write_message("Generating formats for %s record%s." \
+ % (len(recids), len(recids) > 1 and 's' or ''))
+
+ icon_sizes = icon_sizes.split(',')
+ if isinstance(docnames, str):
+ docnames = docnames.split('/')
+
+ updated = 0
+ for i, recid in enumerate(recids):
+ done, exceptions = create_icons_for_record(recid, icon_sizes,
+ icon_format_mappings,
+ docnames, add_default_icon,
+ inherit_moreinfo)
+ updated += done
+ if exceptions:
+ for ex in exceptions:
+ write_message(ex)
+ else:
+ write_message("Recid %s DONE." % recid)
+
+ task_update_progress("Done %d out of %d." % (i, len(recids)))
+ task_sleep_now_if_required(can_stop_too=True)
+
+ if updated:
+ cli_fix_marc(None, explicit_recid_set=recids, interactive=False)
+
+ return 1
+
+def _create_icon(file_path, icon_size, docname, icon_format='gif', verbosity=9):
+ """
+ Creates icon of given file.
+
+ Returns path to the icon. If creation fails, return None, and
+ register exception (send email to admin).
+
+
+ @param file_path: full path to icon
+ @type file_path: string
+
+ @param icon_size: the scaling information to be used for the
+ creation of the new icon.
+ @type icon_size: int
+
+ @param icon_format: format to use to create the icon
+ @type icon_format: string
+
+ @param verbosity: the verbosity level under which the program
+ is to run;
+ @type verbosity: int
+ """
+ if icon_size.isdigit():
+ # It makes sense to explicitely not upscale images
+ icon_size += '>'
+ icon_properties = {
+ 'input-file' : file_path,
+ 'icon-name' : docname,
+ 'multipage-icon': False,
+ 'multipage-icon-delay': 0,
+ 'icon-file-format': "%s" % icon_format,
+ 'icon-scale' : "%s" % icon_size,
+ 'verbosity' : verbosity,
+ }
+ return create_icon(icon_properties)
+
+def get_comment_and_description(bibdoc, inherit_moreinfo):
+ """
+ Return the bibdoc comment and description, according to the
+ model/assumption that the sames are applied to all bibdocfiles of
+ the bibdoc, if inherit_moreinfo is set to True.
+ """
+ comment = None
+ description = None
+ if inherit_moreinfo:
+ all_descriptions = [bibdocfile.get_description() for bibdocfile \
+ in bibdoc.list_latest_files()
+ if bibdocfile.get_description() not in ['', None]]
+ if len(all_descriptions) > 0:
+ description = all_descriptions[0]
+
+ all_comments = [bibdocfile.get_comment() for bibdocfile \
+ in bibdoc.list_latest_files()
+ if bibdocfile.get_comment() not in ['', None]]
+ if len(all_comments) > 0:
+ comment = all_comments[0]
+
+ return (comment, description)
diff --git a/invenio/modules/scheduler/tasklets/bst_create_related_formats.py b/invenio/modules/scheduler/tasklets/bst_create_related_formats.py
new file mode 100644
index 000000000..9519dc606
--- /dev/null
+++ b/invenio/modules/scheduler/tasklets/bst_create_related_formats.py
@@ -0,0 +1,104 @@
+# -*- coding: utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2012 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
+"""
+Invenio Tasklet.
+
+Create related formats
+"""
+
+from invenio.bibdocfile import BibRecDocs
+from invenio.bibtask import write_message, \
+ task_sleep_now_if_required, \
+ task_update_progress
+from invenio.bibdocfile_managedocfiles import get_description_and_comment
+from invenio.websubmit_functions.Shared_Functions import \
+ createRelatedFormats
+from invenio.bibdocfilecli import cli_fix_marc
+
+def bst_create_related_formats(recid, docnames=None, force=0):
+ """
+ Create the related formats for the given record and docnames.
+
+ @param recid: the record ID to consider
+ @type recid: int
+ @param docnames: the list of docnames for which we want to create
+ related formats. Separate docnames using '/' character.
+ @type docnames: list
+ @param force: do we force the creation even if the format already exists (1) or not (0)?
+ @type force: int
+ """
+ force = int(force) and True or False
+ recid = int(recid)
+ if isinstance(docnames, str):
+ docnames = docnames.split('/')
+ elif docnames is None:
+ docnames = []
+
+ try:
+ bibarchive = BibRecDocs(recid)
+ except Exception, e:
+ write_message("Could not instantiate record #%s: %s" % (recid, e))
+ return 0
+
+ write_message("Going to create related file formats for record #%s" % recid)
+
+ i = 0
+ for docname in docnames:
+ i += 1
+ task_sleep_now_if_required()
+ msg = "Processing %s (%i/%i)" % (docname, i, len(docnames))
+ write_message(msg)
+ task_update_progress(msg)
+ try:
+ bibdoc = bibarchive.get_bibdoc(docname)
+ except Exception, e:
+ write_message("Could not process docname %s: %s" % (docname, e))
+ continue
+
+ (prev_desc, prev_comment) = \
+ get_description_and_comment(bibarchive.get_bibdoc(docname).list_latest_files())
+
+ # List all files that are not icons or subformats
+ current_files = [bibdocfile.get_path() for bibdocfile in bibdoc.list_latest_files() if \
+ not bibdocfile.get_subformat() and not bibdocfile.is_icon()]
+
+ ## current_files = []
+ ## if not force:
+ ## current_files = [bibdocfile.get_path() for bibdocfile bibdoc.list_latest_files()]
+ for current_filepath in current_files:
+ # Convert
+ new_files = createRelatedFormats(fullpath=current_filepath,
+ overwrite=force,
+ consider_version=True)
+ # Append
+ for new_file in new_files:
+ try:
+ bibdoc = bibarchive.add_new_format(new_file,
+ docname,
+ prev_desc,
+ prev_comment)
+ except Exception, e:
+ write_message("Could not add format to BibDoc with docname %s: %s" % (docname, e))
+ continue
+
+ write_message("All files successfully processed and attached")
+ cli_fix_marc(None, [recid], interactive=False)
+
+ return 1
diff --git a/invenio/modules/search/fixtures.py b/invenio/modules/search/fixtures.py
index 37fc786f1..2fdbc3e4d 100644
--- a/invenio/modules/search/fixtures.py
+++ b/invenio/modules/search/fixtures.py
@@ -1,2662 +1,2667 @@
# -*- coding: utf-8 -*-
#
## This file is part of Invenio.
## Copyright (C) 2012, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
from invenio.config import CFG_SITE_NAME
from fixture import DataSet
class CollectionData(DataSet):
class siteCollection:
id = 1
name = CFG_SITE_NAME
dbquery = None
class FieldData(DataSet):
class Field_1:
code = u'anyfield'
id = 1
name = u'any field'
class Field_2:
code = u'title'
id = 2
name = u'title'
class Field_3:
code = u'author'
id = 3
name = u'author'
class Field_4:
code = u'abstract'
id = 4
name = u'abstract'
class Field_5:
code = u'keyword'
id = 5
name = u'keyword'
class Field_6:
code = u'reportnumber'
id = 6
name = u'report number'
class Field_7:
code = u'subject'
id = 7
name = u'subject'
class Field_8:
code = u'reference'
id = 8
name = u'reference'
class Field_9:
code = u'fulltext'
id = 9
name = u'fulltext'
class Field_10:
code = u'collection'
id = 10
name = u'collection'
class Field_11:
code = u'division'
id = 11
name = u'division'
class Field_12:
code = u'year'
id = 12
name = u'year'
class Field_13:
code = u'experiment'
id = 13
name = u'experiment'
class Field_14:
code = u'recid'
id = 14
name = u'record ID'
class Field_15:
code = u'isbn'
id = 15
name = u'isbn'
class Field_16:
code = u'issn'
id = 16
name = u'issn'
class Field_17:
code = u'coden'
id = 17
name = u'coden'
#class Field_18:
# code = u'doi'
# id = 18
# name = u'doi'
class Field_19:
code = u'journal'
id = 19
name = u'journal'
class Field_20:
code = u'collaboration'
id = 20
name = u'collaboration'
class Field_21:
code = u'affiliation'
id = 21
name = u'affiliation'
class Field_22:
code = u'exactauthor'
id = 22
name = u'exact author'
class Field_23:
code = u'datecreated'
id = 23
name = u'date created'
class Field_24:
code = u'datemodified'
id = 24
name = u'date modified'
class Field_25:
code = u'refersto'
id = 25
name = u'refers to'
class Field_26:
code = u'citedby'
id = 26
name = u'cited by'
class Field_27:
code = u'caption'
id = 27
name = u'caption'
class Field_28:
code = u'firstauthor'
id = 28
name = u'first author'
class Field_29:
code = u'exactfirstauthor'
id = 29
name = u'exact first author'
class Field_30:
code = u'authorcount'
id = 30
name = u'author count'
class Field_31:
code = u'rawref'
id = 31
name = u'reference to'
class Field_32:
code = u'exacttitle'
id = 32
name = u'exact title'
class Field_33:
code = u'authorityauthor'
id = 33
name = u'authority author'
class Field_34:
code = u'authorityinstitute'
id = 34
name = u'authority institution'
class Field_35:
code = u'authorityjournal'
id = 35
name = u'authority journal'
class Field_36:
code = u'authoritysubject'
id = 36
name = u'authority subject'
class Field_37:
code = u'itemcount'
id = 37
name = u'item count'
class Field_38:
code = u'filetype'
id = 38
name = u'file type'
class Field_39:
code = u'miscellaneous'
id = 39
name = u'miscellaneous'
class Field_40:
code = u'referstoexcludingselfcites'
id = 40
name = u'refers to excluding self cites'
class Field_41:
code = u'citedbyexcludingselfcites'
id = 41
name = u'cited by excluding self cites'
class Field_42:
code = u'cataloguer'
id = 42
name = u'cataloguer nickname'
class Field_43:
- code = u'tag'
+ code = u'filename'
id = 43
+ name = u'file name'
+
+ class Field_44:
+ code = u'tag'
+ id = 44
name = u'tag'
class TagData(DataSet):
class Tag_1:
id = 1
value = u'100__a'
name = u'first author name'
class Tag_2:
id = 2
value = u'700__a'
name = u'additional author name'
class Tag_3:
id = 3
value = u'245__%'
name = u'main title'
class Tag_4:
id = 4
value = u'246__%'
name = u'additional title'
class Tag_5:
id = 5
value = u'520__%'
name = u'abstract'
class Tag_6:
id = 6
value = u'6531_a'
name = u'keyword'
class Tag_7:
id = 7
value = u'037__a'
name = u'primary report number'
class Tag_8:
id = 8
value = u'088__a'
name = u'additional report number'
class Tag_9:
id = 9
value = u'909C0r'
name = u'added report number'
class Tag_10:
id = 10
value = u'999C5%'
name = u'reference'
class Tag_11:
id = 11
value = u'980__%'
name = u'collection identifier'
class Tag_12:
id = 12
value = u'65017a'
name = u'main subject'
class Tag_13:
id = 13
value = u'65027a'
name = u'additional subject'
class Tag_14:
id = 14
value = u'909C0p'
name = u'division'
class Tag_15:
id = 15
value = u'909C0y'
name = u'year'
class Tag_16:
id = 16
value = u'00%'
name = u'00x'
class Tag_17:
id = 17
value = u'01%'
name = u'01x'
class Tag_18:
id = 18
value = u'02%'
name = u'02x'
class Tag_19:
id = 19
value = u'03%'
name = u'03x'
class Tag_20:
id = 20
value = u'04%'
name = u'lang'
class Tag_21:
id = 21
value = u'05%'
name = u'05x'
class Tag_22:
id = 22
value = u'06%'
name = u'06x'
class Tag_23:
id = 23
value = u'07%'
name = u'07x'
class Tag_24:
id = 24
value = u'08%'
name = u'08x'
class Tag_25:
id = 25
value = u'09%'
name = u'09x'
class Tag_26:
id = 26
value = u'10%'
name = u'10x'
class Tag_27:
id = 27
value = u'11%'
name = u'11x'
class Tag_28:
id = 28
value = u'12%'
name = u'12x'
class Tag_29:
id = 29
value = u'13%'
name = u'13x'
class Tag_30:
id = 30
value = u'14%'
name = u'14x'
class Tag_31:
id = 31
value = u'15%'
name = u'15x'
class Tag_32:
id = 32
value = u'16%'
name = u'16x'
class Tag_33:
id = 33
value = u'17%'
name = u'17x'
class Tag_34:
id = 34
value = u'18%'
name = u'18x'
class Tag_35:
id = 35
value = u'19%'
name = u'19x'
class Tag_36:
id = 36
value = u'20%'
name = u'20x'
class Tag_37:
id = 37
value = u'21%'
name = u'21x'
class Tag_38:
id = 38
value = u'22%'
name = u'22x'
class Tag_39:
id = 39
value = u'23%'
name = u'23x'
class Tag_40:
id = 40
value = u'24%'
name = u'24x'
class Tag_41:
id = 41
value = u'25%'
name = u'25x'
class Tag_42:
id = 42
value = u'26%'
name = u'internal'
class Tag_43:
id = 43
value = u'27%'
name = u'27x'
class Tag_44:
id = 44
value = u'28%'
name = u'28x'
class Tag_45:
id = 45
value = u'29%'
name = u'29x'
class Tag_46:
id = 46
value = u'30%'
name = u'pages'
class Tag_47:
id = 47
value = u'31%'
name = u'31x'
class Tag_48:
id = 48
value = u'32%'
name = u'32x'
class Tag_49:
id = 49
value = u'33%'
name = u'33x'
class Tag_50:
id = 50
value = u'34%'
name = u'34x'
class Tag_51:
id = 51
value = u'35%'
name = u'35x'
class Tag_52:
id = 52
value = u'36%'
name = u'36x'
class Tag_53:
id = 53
value = u'37%'
name = u'37x'
class Tag_54:
id = 54
value = u'38%'
name = u'38x'
class Tag_55:
id = 55
value = u'39%'
name = u'39x'
class Tag_56:
id = 56
value = u'40%'
name = u'40x'
class Tag_57:
id = 57
value = u'41%'
name = u'41x'
class Tag_58:
id = 58
value = u'42%'
name = u'42x'
class Tag_59:
id = 59
value = u'43%'
name = u'43x'
class Tag_60:
id = 60
value = u'44%'
name = u'44x'
class Tag_61:
id = 61
value = u'45%'
name = u'45x'
class Tag_62:
id = 62
value = u'46%'
name = u'46x'
class Tag_63:
id = 63
value = u'47%'
name = u'47x'
class Tag_64:
id = 64
value = u'48%'
name = u'48x'
class Tag_65:
id = 65
value = u'49%'
name = u'series'
class Tag_66:
id = 66
value = u'50%'
name = u'50x'
class Tag_67:
id = 67
value = u'51%'
name = u'51x'
class Tag_68:
id = 68
value = u'52%'
name = u'52x'
class Tag_69:
id = 69
value = u'53%'
name = u'53x'
class Tag_70:
id = 70
value = u'54%'
name = u'54x'
class Tag_71:
id = 71
value = u'55%'
name = u'55x'
class Tag_72:
id = 72
value = u'56%'
name = u'56x'
class Tag_73:
id = 73
value = u'57%'
name = u'57x'
class Tag_74:
id = 74
value = u'58%'
name = u'58x'
class Tag_75:
id = 75
value = u'59%'
name = u'summary'
class Tag_76:
id = 76
value = u'60%'
name = u'60x'
class Tag_77:
id = 77
value = u'61%'
name = u'61x'
class Tag_78:
id = 78
value = u'62%'
name = u'62x'
class Tag_79:
id = 79
value = u'63%'
name = u'63x'
class Tag_80:
id = 80
value = u'64%'
name = u'64x'
class Tag_81:
id = 81
value = u'65%'
name = u'65x'
class Tag_82:
id = 82
value = u'66%'
name = u'66x'
class Tag_83:
id = 83
value = u'67%'
name = u'67x'
class Tag_84:
id = 84
value = u'68%'
name = u'68x'
class Tag_85:
id = 85
value = u'69%'
name = u'subject'
class Tag_86:
id = 86
value = u'70%'
name = u'70x'
class Tag_87:
id = 87
value = u'71%'
name = u'71x'
class Tag_88:
id = 88
value = u'72%'
name = u'author-ad'
class Tag_89:
id = 89
value = u'73%'
name = u'73x'
class Tag_90:
id = 90
value = u'74%'
name = u'74x'
class Tag_91:
id = 91
value = u'75%'
name = u'75x'
class Tag_92:
id = 92
value = u'76%'
name = u'76x'
class Tag_93:
id = 93
value = u'77%'
name = u'77x'
class Tag_94:
id = 94
value = u'78%'
name = u'78x'
class Tag_95:
id = 95
value = u'79%'
name = u'79x'
class Tag_96:
id = 96
value = u'80%'
name = u'80x'
class Tag_97:
id = 97
value = u'81%'
name = u'81x'
class Tag_98:
id = 98
value = u'82%'
name = u'82x'
class Tag_99:
id = 99
value = u'83%'
name = u'83x'
class Tag_100:
id = 100
value = u'84%'
name = u'84x'
class Tag_101:
id = 101
value = u'85%'
name = u'electr'
class Tag_102:
id = 102
value = u'86%'
name = u'86x'
class Tag_103:
id = 103
value = u'87%'
name = u'87x'
class Tag_104:
id = 104
value = u'88%'
name = u'88x'
class Tag_105:
id = 105
value = u'89%'
name = u'89x'
class Tag_106:
id = 106
value = u'90%'
name = u'publication'
class Tag_107:
id = 107
value = u'91%'
name = u'pub-conf-cit'
class Tag_108:
id = 108
value = u'92%'
name = u'92x'
class Tag_109:
id = 109
value = u'93%'
name = u'93x'
class Tag_110:
id = 110
value = u'94%'
name = u'94x'
class Tag_111:
id = 111
value = u'95%'
name = u'95x'
class Tag_112:
id = 112
value = u'96%'
name = u'catinfo'
class Tag_113:
id = 113
value = u'97%'
name = u'97x'
class Tag_114:
id = 114
value = u'98%'
name = u'98x'
class Tag_115:
id = 115
value = u'8564_u'
name = u'url'
class Tag_116:
id = 116
value = u'909C0e'
name = u'experiment'
class Tag_117:
id = 117
value = u'001'
name = u'record ID'
class Tag_118:
id = 118
value = u'020__a'
name = u'isbn'
class Tag_119:
id = 119
value = u'022__a'
name = u'issn'
class Tag_120:
id = 120
value = u'030__a'
name = u'coden'
class Tag_121:
id = 121
value = u'909C4a'
name = u'doi'
class Tag_122:
id = 122
value = u'850%'
name = u'850x'
class Tag_123:
id = 123
value = u'851%'
name = u'851x'
class Tag_124:
id = 124
value = u'852%'
name = u'852x'
class Tag_125:
id = 125
value = u'853%'
name = u'853x'
class Tag_126:
id = 126
value = u'854%'
name = u'854x'
class Tag_127:
id = 127
value = u'855%'
name = u'855x'
class Tag_128:
id = 128
value = u'857%'
name = u'857x'
class Tag_129:
id = 129
value = u'858%'
name = u'858x'
class Tag_130:
id = 130
value = u'859%'
name = u'859x'
class Tag_131:
id = 131
value = u'909C4%'
name = u'journal'
class Tag_132:
id = 132
value = u'710__g'
name = u'collaboration'
class Tag_133:
id = 133
value = u'100__u'
name = u'first author affiliation'
class Tag_134:
id = 134
value = u'700__u'
name = u'additional author affiliation'
class Tag_135:
id = 135
value = u'8564_y'
name = u'caption'
class Tag_136:
id = 136
value = u'909C4c'
name = u'journal page'
class Tag_137:
id = 137
value = u'909C4p'
name = u'journal title'
class Tag_138:
id = 138
value = u'909C4v'
name = u'journal volume'
class Tag_139:
id = 139
value = u'909C4y'
name = u'journal year'
class Tag_140:
id = 140
value = u'500__a'
name = u'comment'
class Tag_141:
id = 141
value = u'245__a'
name = u'title'
class Tag_142:
id = 142
value = u'245__a'
name = u'main abstract'
class Tag_143:
id = 143
value = u'595__a'
name = u'internal notes'
class Tag_144:
id = 144
value = u'787%'
name = u'other relationship entry'
class Tag_146:
id = 146
value = u'400__a'
name = u'authority: alternative personal name'
class Tag_148:
id = 148
value = u'110__a'
name = u'authority: organization main name'
class Tag_149:
id = 149
value = u'410__a'
name = u'organization alternative name'
class Tag_150:
id = 150
value = u'510__a'
name = u'organization main from other record'
class Tag_151:
id = 151
value = u'130__a'
name = u'authority: uniform title'
class Tag_152:
id = 152
value = u'430__a'
name = u'authority: uniform title alternatives'
class Tag_153:
id = 153
value = u'530__a'
name = u'authority: uniform title from other record'
class Tag_154:
id = 154
value = u'150__a'
name = u'authority: subject from other record'
class Tag_155:
id = 155
value = u'450__a'
name = u'authority: subject alternative name'
class Tag_156:
id = 156
value = u'450__a'
name = u'authority: subject main name'
class Tag_157:
id = 157
value = u'031%'
name = u'031x'
class Tag_158:
id = 158
value = u'032%'
name = u'032x'
class Tag_159:
id = 159
value = u'033%'
name = u'033x'
class Tag_160:
id = 160
value = u'034%'
name = u'034x'
class Tag_161:
id = 161
value = u'035%'
name = u'035x'
class Tag_162:
id = 162
value = u'036%'
name = u'036x'
class Tag_163:
id = 163
value = u'037%'
name = u'037x'
class Tag_164:
id = 164
value = u'038%'
name = u'038x'
class Tag_165:
id = 165
value = u'080%'
name = u'080x'
class Tag_166:
id = 166
value = u'082%'
name = u'082x'
class Tag_167:
id = 167
value = u'083%'
name = u'083x'
class Tag_168:
id = 168
value = u'084%'
name = u'084x'
class Tag_169:
id = 169
value = u'085%'
name = u'085x'
class Tag_170:
id = 170
value = u'086%'
name = u'086x'
class Tag_171:
id = 171
value = u'240%'
name = u'240x'
class Tag_172:
id = 172
value = u'242%'
name = u'242x'
class Tag_173:
id = 173
value = u'243%'
name = u'243x'
class Tag_174:
id = 174
value = u'244%'
name = u'244x'
class Tag_175:
id = 175
value = u'247%'
name = u'247x'
class Tag_176:
id = 176
value = u'521%'
name = u'521x'
class Tag_177:
id = 177
value = u'522%'
name = u'522x'
class Tag_178:
id = 178
value = u'524%'
name = u'524x'
class Tag_179:
id = 179
value = u'525%'
name = u'525x'
class Tag_180:
id = 180
value = u'526%'
name = u'526x'
class Tag_181:
id = 181
value = u'650%'
name = u'650x'
class Tag_182:
id = 182
value = u'651%'
name = u'651x'
class Tag_183:
id = 183
value = u'6531_v'
name = u'6531_v'
class Tag_184:
id = 184
value = u'6531_y'
name = u'6531_y'
class Tag_185:
id = 185
value = u'6531_9'
name = u'6531_9'
class Tag_186:
id = 186
value = u'654%'
name = u'654x'
class Tag_187:
id = 187
value = u'655%'
name = u'655x'
class Tag_188:
id = 188
value = u'656%'
name = u'656x'
class Tag_189:
id = 189
value = u'657%'
name = u'657x'
class Tag_190:
id = 190
value = u'658%'
name = u'658x'
class Tag_191:
id = 191
value = u'711%'
name = u'711x'
class Tag_192:
id = 192
value = u'900%'
name = u'900x'
class Tag_193:
id = 193
value = u'901%'
name = u'901x'
class Tag_194:
id = 194
value = u'902%'
name = u'902x'
class Tag_195:
id = 195
value = u'903%'
name = u'903x'
class Tag_196:
id = 196
value = u'904%'
name = u'904x'
class Tag_197:
id = 197
value = u'905%'
name = u'905x'
class Tag_198:
id = 198
value = u'906%'
name = u'906x'
class Tag_199:
id = 199
value = u'907%'
name = u'907x'
class Tag_200:
id = 200
value = u'908%'
name = u'908x'
class Tag_201:
id = 201
value = u'909C1%'
name = u'909C1x'
class Tag_202:
id = 202
value = u'909C5%'
name = u'909C5x'
class Tag_203:
id = 203
value = u'909CS%'
name = u'909CSx'
class Tag_204:
id = 204
value = u'909CO%'
name = u'909COx'
class Tag_205:
id = 205
value = u'909CK%'
name = u'909CKx'
class Tag_206:
id = 206
value = u'909CP%'
name = u'909CPx'
class Tag_207:
id = 207
value = u'981%'
name = u'981x'
class Tag_208:
id = 208
value = u'982%'
name = u'982x'
class Tag_209:
id = 209
value = u'983%'
name = u'983x'
class Tag_210:
id = 210
value = u'984%'
name = u'984x'
class Tag_211:
id = 211
value = u'985%'
name = u'985x'
class Tag_212:
id = 212
value = u'986%'
name = u'986x'
class Tag_213:
id = 213
value = u'987%'
name = u'987x'
class Tag_214:
id = 214
value = u'988%'
name = u'988x'
class Tag_215:
id = 215
value = u'989%'
name = u'989x'
class Tag_216:
id = 216
value = u'100__0'
name = u'author control'
class Tag_217:
id = 217
value = u'110__0'
name = u'institute control'
class Tag_218:
id = 218
value = u'130__0'
name = u'journal control'
class Tag_219:
id = 219
value = u'150__0'
name = u'subject control'
class Tag_220:
id = 220
value = u'260__0'
name = u'additional institute control'
class Tag_221:
id = 221
value = u'700__0'
name = u'additional author control'
class FormatData(DataSet):
class Format_1:
code = u'hb'
last_updated = None
description = u'HTML brief output format, used for search results pages.'
content_type = u'text/html'
id = 1
visibility = 1
name = u'HTML brief'
class Format_2:
code = u'hd'
last_updated = None
description = u'HTML detailed output format, used for Detailed record pages.'
content_type = u'text/html'
id = 2
visibility = 1
name = u'HTML detailed'
class Format_3:
code = u'hm'
last_updated = None
description = u'HTML MARC.'
content_type = u'text/html'
id = 3
visibility = 1
name = u'MARC'
class Format_4:
code = u'xd'
last_updated = None
description = u'XML Dublin Core.'
content_type = u'text/xml'
id = 4
visibility = 1
name = u'Dublin Core'
class Format_5:
code = u'xm'
last_updated = None
description = u'XML MARC.'
content_type = u'text/xml'
id = 5
visibility = 1
name = u'MARCXML'
class Format_6:
code = u'hp'
last_updated = None
description = u'HTML portfolio-style output format for photos.'
content_type = u'text/html'
id = 6
visibility = 1
name = u'portfolio'
class Format_7:
code = u'hc'
last_updated = None
description = u'HTML caption-only output format for photos.'
content_type = u'text/html'
id = 7
visibility = 1
name = u'photo captions only'
class Format_8:
code = u'hx'
last_updated = None
description = u'BibTeX.'
content_type = u'text/html'
id = 8
visibility = 1
name = u'BibTeX'
class Format_9:
code = u'xe'
last_updated = None
description = u'XML EndNote.'
content_type = u'text/xml'
id = 9
visibility = 1
name = u'EndNote'
class Format_10:
code = u'xn'
last_updated = None
description = u'XML NLM.'
content_type = u'text/xml'
id = 10
visibility = 1
name = u'NLM'
class Format_11:
code = u'excel'
last_updated = None
description = u'Excel csv output'
content_type = u'application/ms-excel'
id = 11
visibility = 0
name = u'Excel'
class Format_12:
code = u'hs'
last_updated = None
description = u'Very short HTML output for similarity box (<i>people also viewed..</i>).'
content_type = u'text/html'
id = 12
visibility = 0
name = u'HTML similarity'
class Format_13:
code = u'xr'
last_updated = None
description = u'RSS.'
content_type = u'text/xml'
id = 13
visibility = 0
name = u'RSS'
class Format_14:
code = u'xoaidc'
last_updated = None
description = u'OAI DC.'
content_type = u'text/xml'
id = 14
visibility = 0
name = u'OAI DC'
class Format_15:
code = u'hdfile'
last_updated = None
description = u'Used to show fulltext files in mini-panel of detailed record pages.'
content_type = u'text/html'
id = 15
visibility = 0
name = u'File mini-panel'
class Format_16:
code = u'hdact'
last_updated = None
description = u'Used to display actions in mini-panel of detailed record pages.'
content_type = u'text/html'
id = 16
visibility = 0
name = u'Actions mini-panel'
class Format_17:
code = u'hdref'
last_updated = None
description = u'Display record references in References tab.'
content_type = u'text/html'
id = 17
visibility = 0
name = u'References tab'
class Format_18:
code = u'hcs'
last_updated = None
description = u'HTML cite summary format, used for search results pages.'
content_type = u'text/html'
id = 18
visibility = 1
name = u'HTML citesummary'
class Format_19:
code = u'xw'
last_updated = None
description = u'RefWorks.'
content_type = u'text/xml'
id = 19
visibility = 1
name = u'RefWorks'
class Format_20:
code = u'xo'
last_updated = None
description = u'Metadata Object Description Schema'
content_type = u'application/xml'
id = 20
visibility = 1
name = u'MODS'
class Format_21:
code = u'ha'
last_updated = None
description = u'Very brief HTML output format for author/paper claiming facility.'
content_type = u'text/html'
id = 21
visibility = 0
name = u'HTML author claiming'
class Format_22:
code = u'xp'
last_updated = None
description = u'Sample format suitable for multimedia feeds, such as podcasts'
content_type = u'application/rss+xml'
id = 22
visibility = 0
name = u'Podcast'
class Format_23:
code = u'wapaff'
last_updated = None
description = u'cPickled dicts'
content_type = u'text'
id = 23
visibility = 0
name = u'WebAuthorProfile affiliations helper'
class Format_24:
code = u'xe8x'
last_updated = None
description = u'XML EndNote (8-X).'
content_type = u'text/xml'
id = 24
visibility = 1
name = u'EndNote (8-X)'
class Format_25:
code = u'hcs2'
last_updated = None
description = u'HTML cite summary format, including self-citations counts.'
content_type = u'text/html'
id = 25
visibility = 0
name = u'HTML citesummary extended'
class Format_26:
code = u'dcite'
last_updated = None
description = u'DataCite XML format.'
content_type = u'text/xml'
id = 26
visibility = 0
name = u'DataCite'
class Format_27:
code = u'mobb'
last_updated = None
description = u'Mobile brief format.'
content_type = u'text/html'
id = 27
visibility = 0
name = u'Mobile brief'
class Format_28:
code = u'mobd'
last_updated = None
description = u'Mobile detailed format.'
content_type = u'text/html'
id = 28
visibility = 0
name = u'Mobile detailed'
class Format_29:
code = u'tm'
last_updated = None
description = u'Text MARC.'
content_type = u'text/plain'
id = 29
visibility = 0
name = u'Text MARC'
class FieldTagData(DataSet):
class FieldTag_10_11:
score = 100
id_tag = TagData.Tag_11.ref('id')
id_field = FieldData.Field_10.ref('id')
class FieldTag_11_14:
score = 100
id_tag = TagData.Tag_14.ref('id')
id_field = FieldData.Field_11.ref('id')
class FieldTag_12_15:
score = 10
id_tag = TagData.Tag_15.ref('id')
id_field = FieldData.Field_12.ref('id')
class FieldTag_13_116:
score = 10
id_tag = TagData.Tag_116.ref('id')
id_field = FieldData.Field_13.ref('id')
class FieldTag_14_117:
score = 100
id_tag = TagData.Tag_117.ref('id')
id_field = FieldData.Field_14.ref('id')
class FieldTag_15_118:
score = 100
id_tag = TagData.Tag_118.ref('id')
id_field = FieldData.Field_15.ref('id')
class FieldTag_16_119:
score = 100
id_tag = TagData.Tag_119.ref('id')
id_field = FieldData.Field_16.ref('id')
class FieldTag_17_120:
score = 100
id_tag = TagData.Tag_120.ref('id')
id_field = FieldData.Field_17.ref('id')
#class FieldTag_18_120:
# score = 100
# id_tag = TagData.Tag_121.ref('id')
# id_field = FieldData.Field_18.ref('id')
class FieldTag_19_131:
score = 100
id_tag = TagData.Tag_131.ref('id')
id_field = FieldData.Field_19.ref('id')
class FieldTag_20_132:
score = 100
id_tag = TagData.Tag_132.ref('id')
id_field = FieldData.Field_20.ref('id')
class FieldTag_21_133:
score = 100
id_tag = TagData.Tag_133.ref('id')
id_field = FieldData.Field_21.ref('id')
class FieldTag_21_134:
score = 90
id_tag = TagData.Tag_134.ref('id')
id_field = FieldData.Field_21.ref('id')
class FieldTag_22_1:
score = 100
id_tag = TagData.Tag_1.ref('id')
id_field = FieldData.Field_22.ref('id')
class FieldTag_22_2:
score = 90
id_tag = TagData.Tag_2.ref('id')
id_field = FieldData.Field_22.ref('id')
class FieldTag_27_135:
score = 100
id_tag = TagData.Tag_135.ref('id')
id_field = FieldData.Field_27.ref('id')
class FieldTag_28_1:
score = 100
id_tag = TagData.Tag_1.ref('id')
id_field = FieldData.Field_28.ref('id')
class FieldTag_29_1:
score = 100
id_tag = TagData.Tag_1.ref('id')
id_field = FieldData.Field_29.ref('id')
class FieldTag_2_3:
score = 100
id_tag = TagData.Tag_3.ref('id')
id_field = FieldData.Field_2.ref('id')
class FieldTag_2_4:
score = 90
id_tag = TagData.Tag_4.ref('id')
id_field = FieldData.Field_2.ref('id')
class FieldTag_30_1:
score = 100
id_tag = TagData.Tag_1.ref('id')
id_field = FieldData.Field_30.ref('id')
class FieldTag_30_2:
score = 90
id_tag = TagData.Tag_2.ref('id')
id_field = FieldData.Field_30.ref('id')
class FieldTag_32_3:
score = 100
id_tag = TagData.Tag_3.ref('id')
id_field = FieldData.Field_32.ref('id')
class FieldTag_32_4:
score = 90
id_tag = TagData.Tag_4.ref('id')
id_field = FieldData.Field_32.ref('id')
class FieldTag_3_1:
score = 100
id_tag = TagData.Tag_1.ref('id')
id_field = FieldData.Field_3.ref('id')
class FieldTag_3_2:
score = 90
id_tag = TagData.Tag_2.ref('id')
id_field = FieldData.Field_3.ref('id')
class FieldTag_4_5:
score = 100
id_tag = TagData.Tag_5.ref('id')
id_field = FieldData.Field_4.ref('id')
class FieldTag_5_6:
score = 100
id_tag = TagData.Tag_6.ref('id')
id_field = FieldData.Field_5.ref('id')
class FieldTag_6_7:
score = 30
id_tag = TagData.Tag_7.ref('id')
id_field = FieldData.Field_6.ref('id')
class FieldTag_6_8:
score = 10
id_tag = TagData.Tag_8.ref('id')
id_field = FieldData.Field_6.ref('id')
class FieldTag_6_9:
score = 20
id_tag = TagData.Tag_9.ref('id')
id_field = FieldData.Field_6.ref('id')
class FieldTag_7_12:
score = 100
id_tag = TagData.Tag_12.ref('id')
id_field = FieldData.Field_7.ref('id')
class FieldTag_7_13:
score = 90
id_tag = TagData.Tag_13.ref('id')
id_field = FieldData.Field_7.ref('id')
class FieldTag_8_10:
score = 100
id_tag = TagData.Tag_10.ref('id')
id_field = FieldData.Field_8.ref('id')
class FieldTag_9_115:
score = 100
id_tag = TagData.Tag_115.ref('id')
id_field = FieldData.Field_9.ref('id')
class FieldTag_33_1:
score = 100
id_tag = TagData.Tag_1.ref('id')
id_field = FieldData.Field_33.ref('id')
class FieldTag_33_146:
score = 100
id_tag = TagData.Tag_146.ref('id')
id_field = FieldData.Field_33.ref('id')
class FieldTag_33_140:
score = 100
id_tag = TagData.Tag_140.ref('id')
id_field = FieldData.Field_33.ref('id')
class FieldTag_34_148:
score = 100
id_tag = TagData.Tag_148.ref('id')
id_field = FieldData.Field_34.ref('id')
class FieldTag_34_149:
score = 100
id_tag = TagData.Tag_149.ref('id')
id_field = FieldData.Field_34.ref('id')
class FieldTag_34_150:
score = 100
id_tag = TagData.Tag_150.ref('id')
id_field = FieldData.Field_34.ref('id')
class FieldTag_35_151:
score = 100
id_tag = TagData.Tag_151.ref('id')
id_field = FieldData.Field_35.ref('id')
class FieldTag_35_152:
score = 100
id_tag = TagData.Tag_152.ref('id')
id_field = FieldData.Field_35.ref('id')
class FieldTag_35_153:
score = 100
id_tag = TagData.Tag_153.ref('id')
id_field = FieldData.Field_35.ref('id')
class FieldTag_36_154:
score = 100
id_tag = TagData.Tag_154.ref('id')
id_field = FieldData.Field_36.ref('id')
class FieldTag_36_155:
score = 100
id_tag = TagData.Tag_155.ref('id')
id_field = FieldData.Field_36.ref('id')
class FieldTag_36_156:
score = 100
id_tag = TagData.Tag_156.ref('id')
id_field = FieldData.Field_36.ref('id')
class FieldTag_39_17:
score = 10
id_tag = TagData.Tag_17.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_18:
score = 10
id_tag = TagData.Tag_18.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_157:
score = 10
id_tag = TagData.Tag_157.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_158:
score = 10
id_tag = TagData.Tag_158.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_159:
score = 10
id_tag = TagData.Tag_159.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_160:
score = 10
id_tag = TagData.Tag_160.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_161:
score = 10
id_tag = TagData.Tag_161.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_162:
score = 10
id_tag = TagData.Tag_162.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_163:
score = 10
id_tag = TagData.Tag_163.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_164:
score = 10
id_tag = TagData.Tag_164.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_20:
score = 10
id_tag = TagData.Tag_20.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_21:
score = 10
id_tag = TagData.Tag_21.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_22:
score = 10
id_tag = TagData.Tag_22.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_23:
score = 10
id_tag = TagData.Tag_23.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_165:
score = 10
id_tag = TagData.Tag_165.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_166:
score = 10
id_tag = TagData.Tag_166.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_167:
score = 10
id_tag = TagData.Tag_167.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_168:
score = 10
id_tag = TagData.Tag_168.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_169:
score = 10
id_tag = TagData.Tag_169.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_170:
score = 10
id_tag = TagData.Tag_170.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_25:
score = 10
id_tag = TagData.Tag_25.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_27:
score = 10
id_tag = TagData.Tag_27.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_28:
score = 10
id_tag = TagData.Tag_28.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_29:
score = 10
id_tag = TagData.Tag_29.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_30:
score = 10
id_tag = TagData.Tag_30.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_31:
score = 10
id_tag = TagData.Tag_31.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_32:
score = 10
id_tag = TagData.Tag_32.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_33:
score = 10
id_tag = TagData.Tag_33.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_34:
score = 10
id_tag = TagData.Tag_34.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_35:
score = 10
id_tag = TagData.Tag_35.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_36:
score = 10
id_tag = TagData.Tag_36.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_37:
score = 10
id_tag = TagData.Tag_37.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_38:
score = 10
id_tag = TagData.Tag_38.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_39:
score = 10
id_tag = TagData.Tag_39.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_171:
score = 10
id_tag = TagData.Tag_171.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_172:
score = 10
id_tag = TagData.Tag_172.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_173:
score = 10
id_tag = TagData.Tag_173.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_174:
score = 10
id_tag = TagData.Tag_174.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_175:
score = 10
id_tag = TagData.Tag_175.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_41:
score = 10
id_tag = TagData.Tag_41.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_42:
score = 10
id_tag = TagData.Tag_42.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_43:
score = 10
id_tag = TagData.Tag_43.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_44:
score = 10
id_tag = TagData.Tag_44.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_45:
score = 10
id_tag = TagData.Tag_45.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_46:
score = 10
id_tag = TagData.Tag_46.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_47:
score = 10
id_tag = TagData.Tag_47.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_48:
score = 10
id_tag = TagData.Tag_48.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_49:
score = 10
id_tag = TagData.Tag_49.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_50:
score = 10
id_tag = TagData.Tag_50.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_51:
score = 10
id_tag = TagData.Tag_51.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_52:
score = 10
id_tag = TagData.Tag_52.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_53:
score = 10
id_tag = TagData.Tag_53.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_54:
score = 10
id_tag = TagData.Tag_54.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_55:
score = 10
id_tag = TagData.Tag_55.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_56:
score = 10
id_tag = TagData.Tag_56.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_57:
score = 10
id_tag = TagData.Tag_57.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_58:
score = 10
id_tag = TagData.Tag_58.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_59:
score = 10
id_tag = TagData.Tag_59.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_60:
score = 10
id_tag = TagData.Tag_60.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_61:
score = 10
id_tag = TagData.Tag_61.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_62:
score = 10
id_tag = TagData.Tag_62.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_63:
score = 10
id_tag = TagData.Tag_63.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_64:
score = 10
id_tag = TagData.Tag_64.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_65:
score = 10
id_tag = TagData.Tag_65.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_66:
score = 10
id_tag = TagData.Tag_66.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_67:
score = 10
id_tag = TagData.Tag_67.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_176:
score = 10
id_tag = TagData.Tag_176.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_177:
score = 10
id_tag = TagData.Tag_177.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_178:
score = 10
id_tag = TagData.Tag_178.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_179:
score = 10
id_tag = TagData.Tag_179.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_180:
score = 10
id_tag = TagData.Tag_180.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_69:
score = 10
id_tag = TagData.Tag_69.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_70:
score = 10
id_tag = TagData.Tag_70.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_71:
score = 10
id_tag = TagData.Tag_71.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_72:
score = 10
id_tag = TagData.Tag_72.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_73:
score = 10
id_tag = TagData.Tag_73.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_74:
score = 10
id_tag = TagData.Tag_74.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_75:
score = 10
id_tag = TagData.Tag_75.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_76:
score = 10
id_tag = TagData.Tag_76.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_77:
score = 10
id_tag = TagData.Tag_77.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_78:
score = 10
id_tag = TagData.Tag_78.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_79:
score = 10
id_tag = TagData.Tag_79.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_80:
score = 10
id_tag = TagData.Tag_80.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_181:
score = 10
id_tag = TagData.Tag_181.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_182:
score = 10
id_tag = TagData.Tag_182.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_183:
score = 10
id_tag = TagData.Tag_183.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_184:
score = 10
id_tag = TagData.Tag_184.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_185:
score = 10
id_tag = TagData.Tag_185.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_186:
score = 10
id_tag = TagData.Tag_186.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_82:
score = 10
id_tag = TagData.Tag_82.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_83:
score = 10
id_tag = TagData.Tag_83.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_84:
score = 10
id_tag = TagData.Tag_84.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_85:
score = 10
id_tag = TagData.Tag_85.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_187:
score = 10
id_tag = TagData.Tag_187.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_88:
score = 10
id_tag = TagData.Tag_88.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_89:
score = 10
id_tag = TagData.Tag_89.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_90:
score = 10
id_tag = TagData.Tag_90.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_91:
score = 10
id_tag = TagData.Tag_91.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_92:
score = 10
id_tag = TagData.Tag_92.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_93:
score = 10
id_tag = TagData.Tag_93.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_94:
score = 10
id_tag = TagData.Tag_94.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_95:
score = 10
id_tag = TagData.Tag_95.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_96:
score = 10
id_tag = TagData.Tag_96.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_97:
score = 10
id_tag = TagData.Tag_97.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_98:
score = 10
id_tag = TagData.Tag_98.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_99:
score = 10
id_tag = TagData.Tag_99.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_100:
score = 10
id_tag = TagData.Tag_100.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_102:
score = 10
id_tag = TagData.Tag_102.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_103:
score = 10
id_tag = TagData.Tag_103.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_104:
score = 10
id_tag = TagData.Tag_104.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_105:
score = 10
id_tag = TagData.Tag_105.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_188:
score = 10
id_tag = TagData.Tag_188.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_189:
score = 10
id_tag = TagData.Tag_189.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_190:
score = 10
id_tag = TagData.Tag_190.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_191:
score = 10
id_tag = TagData.Tag_191.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_192:
score = 10
id_tag = TagData.Tag_192.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_193:
score = 10
id_tag = TagData.Tag_193.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_194:
score = 10
id_tag = TagData.Tag_194.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_195:
score = 10
id_tag = TagData.Tag_195.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_196:
score = 10
id_tag = TagData.Tag_196.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_107:
score = 10
id_tag = TagData.Tag_107.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_108:
score = 10
id_tag = TagData.Tag_108.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_109:
score = 10
id_tag = TagData.Tag_109.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_110:
score = 10
id_tag = TagData.Tag_110.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_111:
score = 10
id_tag = TagData.Tag_111.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_112:
score = 10
id_tag = TagData.Tag_112.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_113:
score = 10
id_tag = TagData.Tag_113.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_197:
score = 10
id_tag = TagData.Tag_197.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_198:
score = 10
id_tag = TagData.Tag_198.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_199:
score = 10
id_tag = TagData.Tag_199.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_200:
score = 10
id_tag = TagData.Tag_200.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_201:
score = 10
id_tag = TagData.Tag_201.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_202:
score = 10
id_tag = TagData.Tag_202.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_203:
score = 10
id_tag = TagData.Tag_203.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_204:
score = 10
id_tag = TagData.Tag_204.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_205:
score = 10
id_tag = TagData.Tag_205.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_206:
score = 10
id_tag = TagData.Tag_206.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_207:
score = 10
id_tag = TagData.Tag_207.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_208:
score = 10
id_tag = TagData.Tag_208.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_209:
score = 10
id_tag = TagData.Tag_209.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_210:
score = 10
id_tag = TagData.Tag_210.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_211:
score = 10
id_tag = TagData.Tag_211.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_212:
score = 10
id_tag = TagData.Tag_212.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_213:
score = 10
id_tag = TagData.Tag_213.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_214:
score = 10
id_tag = TagData.Tag_214.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_215:
score = 10
id_tag = TagData.Tag_215.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_122:
score = 10
id_tag = TagData.Tag_122.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_123:
score = 10
id_tag = TagData.Tag_123.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_124:
score = 10
id_tag = TagData.Tag_124.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_125:
score = 10
id_tag = TagData.Tag_125.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_126:
score = 10
id_tag = TagData.Tag_126.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_127:
score = 10
id_tag = TagData.Tag_127.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_128:
score = 10
id_tag = TagData.Tag_128.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_129:
score = 10
id_tag = TagData.Tag_129.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_130:
score = 10
id_tag = TagData.Tag_130.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_1:
score = 10
id_tag = TagData.Tag_1.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_2:
score = 10
id_tag = TagData.Tag_2.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_216:
score = 10
id_tag = TagData.Tag_216.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_217:
score = 10
id_tag = TagData.Tag_217.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_218:
score = 10
id_tag = TagData.Tag_218.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_219:
score = 10
id_tag = TagData.Tag_219.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_220:
score = 10
id_tag = TagData.Tag_220.ref('id')
id_field = FieldData.Field_39.ref('id')
class FieldTag_39_221:
score = 10
id_tag = TagData.Tag_221.ref('id')
id_field = FieldData.Field_39.ref('id')
diff --git a/invenio/modules/upgrader/upgrades/invenio_2013_09_10_new_param_websubmit_function.py b/invenio/modules/upgrader/upgrades/invenio_2013_09_10_new_param_websubmit_function.py
new file mode 100644
index 000000000..cbacc812d
--- /dev/null
+++ b/invenio/modules/upgrader/upgrades/invenio_2013_09_10_new_param_websubmit_function.py
@@ -0,0 +1,36 @@
+# -*- coding: utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2013 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
+from invenio.legacy.dbquery import run_sql
+
+depends_on = ['invenio_release_1_1_0']
+
+def info():
+ return "New 'prefix', 'suffix', 'button_label', 'button_prefix' and 'dates_conversion' parameters for Create_Modify_Interface WebSubmit function"
+
+def do_upgrade():
+ run_sql("""INSERT INTO sbmFUNDESC VALUES ('Create_Modify_Interface','prefix')""")
+ run_sql("""INSERT INTO sbmFUNDESC VALUES ('Create_Modify_Interface','suffix')""")
+ run_sql("""INSERT INTO sbmFUNDESC VALUES ('Create_Modify_Interface','button_label')""")
+ run_sql("""INSERT INTO sbmFUNDESC VALUES ('Create_Modify_Interface','button_prefix')""")
+ run_sql("""INSERT INTO sbmFUNDESC VALUES ('Create_Modify_Interface','dates_conversion')""")
+
+def estimate():
+ """ Estimate running time of upgrade in seconds (optional). """
+ return 1
diff --git a/invenio/modules/upgrader/upgrades/invenio_2013_10_25_new_param_websubmit_function.py b/invenio/modules/upgrader/upgrades/invenio_2013_10_25_new_param_websubmit_function.py
new file mode 100644
index 000000000..f58427e10
--- /dev/null
+++ b/invenio/modules/upgrader/upgrades/invenio_2013_10_25_new_param_websubmit_function.py
@@ -0,0 +1,36 @@
+# -*- coding: utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2013 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
+from invenio.legacy.dbquery import run_sql
+
+depends_on = ['invenio_release_1_1_0']
+
+def info():
+ return "New 'directRelationshipMARC', 'reverseRelationshipMARC' and 'bibuploadMode', parameters for Link_Records WebSubmit function"
+
+def do_upgrade():
+ run_sql("""INSERT INTO sbmFUNDESC VALUES ('Link_Records','directRelationshipMARC')""")
+ run_sql("""INSERT INTO sbmFUNDESC VALUES ('Link_Records','reverseRelationshipMARC')""")
+ run_sql("""INSERT INTO sbmFUNDESC VALUES ('Link_Records','bibuploadMode')""")
+ run_sql("""INSERT INTO sbmFUNDESC VALUES ('Link_Records','silentFailures')""")
+ run_sql("""INSERT INTO sbmFUNDESC VALUES ('Link_Records','considerEmpty')""")
+
+def estimate():
+ """ Estimate running time of upgrade in seconds (optional). """
+ return 1
diff --git a/modules/bibauthorid/lib/bibauthorid_dbinterface_unit_tests.py b/invenio/modules/upgrader/upgrades/invenio_2013_11_12_new_param_websubmit_function.py
similarity index 59%
copy from modules/bibauthorid/lib/bibauthorid_dbinterface_unit_tests.py
copy to invenio/modules/upgrader/upgrades/invenio_2013_11_12_new_param_websubmit_function.py
index 873a57410..926a5c342 100644
--- a/modules/bibauthorid/lib/bibauthorid_dbinterface_unit_tests.py
+++ b/invenio/modules/upgrader/upgrades/invenio_2013_11_12_new_param_websubmit_function.py
@@ -1,41 +1,32 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
+## Copyright (C) 2013 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-"""Unit tests for the search engine."""
+from invenio.legacy.dbquery import run_sql
-__revision__ = \
- "$Id$"
+depends_on = ['invenio_release_1_1_0']
-from itertools import chain
+def info():
+ return "New 'deferRelatedFormatsCreation' parameter for Create_Upload_Files_Interface WebSubmit function"
-from invenio.testutils import InvenioTestCase, make_test_suite, run_test_suite
-from invenio.bibauthorid_cluster_set import ClusterSet
+def do_upgrade():
+ run_sql("""INSERT INTO sbmFUNDESC VALUES ('Create_Upload_Files_Interface','deferRelatedFormatsCreation')""")
-class TestDummy(InvenioTestCase):
-
- def setUp(self):
- pass
-
- def test_one(self):
- pass
-
-TEST_SUITE = make_test_suite(TestDummy)
-
-if __name__ == "__main__":
- run_test_suite(TEST_SUITE, warn_user=True)
+def estimate():
+ """ Estimate running time of upgrade in seconds (optional). """
+ return 1
diff --git a/invenio/modules/upgrader/upgrades/invenio_2013_12_05_new_index_doi.py b/invenio/modules/upgrader/upgrades/invenio_2013_12_05_new_index_doi.py
new file mode 100644
index 000000000..6f14a5b14
--- /dev/null
+++ b/invenio/modules/upgrader/upgrades/invenio_2013_12_05_new_index_doi.py
@@ -0,0 +1,99 @@
+# -*- coding: utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2012, 2013 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
+
+from invenio.legacy.dbquery import run_sql
+
+
+depends_on = ['invenio_2013_09_25_virtual_indexes']
+
+def info():
+ return "New DOI index"
+
+
+def do_upgrade():
+ pass
+
+
+def do_upgrade_atlantis():
+ doi_index = 27
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxWORD%02dF (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term varchar(50) default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ UNIQUE KEY term (term)
+ ) ENGINE=MyISAM;
+ """ % doi_index)
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxWORD%02dR (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+ ) ENGINE=MyISAM;
+ """ % doi_index)
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxPAIR%02dF (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term varchar(100) default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ UNIQUE KEY term (term)
+ ) ENGINE=MyISAM;
+ """ % doi_index)
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxPAIR%02dR (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+ ) ENGINE=MyISAM;
+ """ % doi_index)
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxPHRASE%02dF (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term text default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ KEY term (term(50))
+ ) ENGINE=MyISAM;
+ """ % doi_index)
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxPHRASE%02dR (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+ ) ENGINE=MyISAM;
+ """ % doi_index)
+ run_sql("""INSERT INTO idxINDEX VALUES (%02d,'doi','This index contains words/phrases from DOI fields','0000-00-00 00:00:00', '', 'native','','No','No','No', 'BibIndexDOITokenizer')""" % doi_index)
+ run_sql("""INSERT INTO idxINDEX_idxINDEX (id_virtual, id_normal) VALUES (1, %02d)""" % doi_index)
+ run_sql("""INSERT INTO field VALUES (18,'doi','doi')""")
+ run_sql("""INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (%02d, 18)""" % doi_index)
+
+def estimate():
+ return 1
+
+def pre_upgrade():
+ pass
+
+def post_upgrade():
+ pass
diff --git a/invenio/modules/upgrader/upgrades/invenio_2014_01_22_queue_table_virtual_index.py b/invenio/modules/upgrader/upgrades/invenio_2014_01_22_queue_table_virtual_index.py
new file mode 100644
index 000000000..8075d9341
--- /dev/null
+++ b/invenio/modules/upgrader/upgrades/invenio_2014_01_22_queue_table_virtual_index.py
@@ -0,0 +1,76 @@
+# -*- coding: utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2012, 2013, 2014 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
+from invenio.legacy.dbquery import run_sql
+
+depends_on = ['invenio_2013_09_30_indexer_interface']
+
+def info():
+ return "New queue tables for virtual indexes: idxWORD/PAIR/PHRASExxQ"
+
+def do_upgrade():
+ global_id = 1
+
+ run_sql("""CREATE TABLE IF NOT EXISTS idxWORD%02dQ (
+ id mediumint(10) unsigned NOT NULL auto_increment,
+ runtime datetime NOT NULL default '0000-00-00 00:00:00',
+ id_bibrec_low mediumint(9) unsigned NOT NULL,
+ id_bibrec_high mediumint(9) unsigned NOT NULL,
+ index_name varchar(50) NOT NULL default '',
+ mode varchar(50) NOT NULL default 'update',
+ PRIMARY KEY (id),
+ INDEX (index_name),
+ INDEX (runtime)
+ ) ENGINE=MyISAM;""" % global_id)
+
+ run_sql("""CREATE TABLE IF NOT EXISTS idxPAIR%02dQ (
+ id mediumint(10) unsigned NOT NULL auto_increment,
+ runtime datetime NOT NULL default '0000-00-00 00:00:00',
+ id_bibrec_low mediumint(9) unsigned NOT NULL,
+ id_bibrec_high mediumint(9) unsigned NOT NULL,
+ index_name varchar(50) NOT NULL default '',
+ mode varchar(50) NOT NULL default 'update',
+ PRIMARY KEY (id),
+ INDEX (index_name),
+ INDEX (runtime)
+ ) ENGINE=MyISAM;""" % global_id)
+
+ run_sql("""CREATE TABLE IF NOT EXISTS idxPHRASE%02dQ (
+ id mediumint(10) unsigned NOT NULL auto_increment,
+ runtime datetime NOT NULL default '0000-00-00 00:00:00',
+ id_bibrec_low mediumint(9) unsigned NOT NULL,
+ id_bibrec_high mediumint(9) unsigned NOT NULL,
+ index_name varchar(50) NOT NULL default '',
+ mode varchar(50) NOT NULL default 'update',
+ PRIMARY KEY (id),
+ INDEX (index_name),
+ INDEX (runtime)
+ ) ENGINE=MyISAM;""" % global_id)
+
+
+def estimate():
+ return 1
+
+def pre_upgrade():
+ pass
+
+def post_upgrade():
+ print "NOTE: If you plan to change some of your indexes " \
+ "to virtual type, please note that you need to run " \
+ "new separate bibindex process for them"
diff --git a/modules/bibauthorid/lib/bibauthorid_prob_matrix_unit_tests.py b/invenio/modules/upgrader/upgrades/invenio_2014_01_22_redis_sessions.py
similarity index 60%
copy from modules/bibauthorid/lib/bibauthorid_prob_matrix_unit_tests.py
copy to invenio/modules/upgrader/upgrades/invenio_2014_01_22_redis_sessions.py
index 036ede59c..72252c898 100644
--- a/modules/bibauthorid/lib/bibauthorid_prob_matrix_unit_tests.py
+++ b/invenio/modules/upgrader/upgrades/invenio_2014_01_22_redis_sessions.py
@@ -1,38 +1,39 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
+## Copyright (C) 2013 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-"""Unit tests for the search engine."""
+depends_on = ['invenio_release_1_1_0']
-__revision__ = \
- "$Id$"
+def info():
+ return "Notice user about new redis sessions backend"
-from itertools import chain
+def estimate():
+ return 10
-from invenio.testutils import InvenioTestCase, make_test_suite, run_test_suite
-from invenio.bibauthorid_cluster_set import ClusterSet
+def do_upgrade():
+ pass
-class TestProbabilityMatrix(InvenioTestCase):
+def pre_upgrade():
+ pass
- def setUp(self):
- pass
-
-TEST_SUITE = make_test_suite()
-
-if __name__ == "__main__":
- run_test_suite(TEST_SUITE, warn_user=True)
+def post_upgrade():
+ print 'NOTE: Please install redis server'
+ print 'NOTE: e.g. apt-get install redis'
+ print 'NOTE: Please install nydus and redis'
+ print 'NOTE: e.g. pip install nydus redis'
+ print 'NOTE: Redis is now required by websession'
diff --git a/modules/bibauthorid/lib/bibauthorid_prob_matrix_unit_tests.py b/invenio/modules/upgrader/upgrades/invenio_2014_01_24_seqSTORE_larger_value.py
similarity index 60%
rename from modules/bibauthorid/lib/bibauthorid_prob_matrix_unit_tests.py
rename to invenio/modules/upgrader/upgrades/invenio_2014_01_24_seqSTORE_larger_value.py
index 036ede59c..83e130500 100644
--- a/modules/bibauthorid/lib/bibauthorid_prob_matrix_unit_tests.py
+++ b/invenio/modules/upgrader/upgrades/invenio_2014_01_24_seqSTORE_larger_value.py
@@ -1,38 +1,33 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
+## Copyright (C) 2013 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-"""Unit tests for the search engine."""
+from invenio.legacy.dbquery import run_sql
-__revision__ = \
- "$Id$"
+depends_on = ['invenio_2013_12_04_seqSTORE_larger_value']
-from itertools import chain
+def info():
+ return "Larger values allowed for seqSTORE.seq_value"
-from invenio.testutils import InvenioTestCase, make_test_suite, run_test_suite
-from invenio.bibauthorid_cluster_set import ClusterSet
+def do_upgrade():
+ """ Implement your upgrades here """
+ run_sql("ALTER TABLE seqSTORE MODIFY COLUMN seq_value varchar(255);")
-class TestProbabilityMatrix(InvenioTestCase):
-
- def setUp(self):
- pass
-
-TEST_SUITE = make_test_suite()
-
-if __name__ == "__main__":
- run_test_suite(TEST_SUITE, warn_user=True)
+def estimate():
+ """ Estimate running time of upgrade in seconds (optional). """
+ return 1
diff --git a/invenio/modules/upgrader/upgrades/invenio_2014_03_13_new_index_filename.py b/invenio/modules/upgrader/upgrades/invenio_2014_03_13_new_index_filename.py
new file mode 100644
index 000000000..cc0633b35
--- /dev/null
+++ b/invenio/modules/upgrader/upgrades/invenio_2014_03_13_new_index_filename.py
@@ -0,0 +1,98 @@
+# -*- coding: utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2012, 2013 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
+
+from invenio.legacy.dbquery import run_sql
+
+
+depends_on = ['invenio_2013_10_18_new_index_filetype']
+
+def info():
+ return "New index filename."
+
+
+def do_upgrade():
+ pass
+
+
+def do_upgrade_atlantis():
+ index_id = 28
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxWORD%02dF (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term varchar(50) default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ UNIQUE KEY term (term)
+ ) ENGINE=MyISAM;
+ """ % index_id)
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxWORD%02dR (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+ ) ENGINE=MyISAM;
+ """ % index_id)
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxPAIR%02dF (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term varchar(100) default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ UNIQUE KEY term (term)
+ ) ENGINE=MyISAM;
+ """ % index_id)
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxPAIR%02dR (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+ ) ENGINE=MyISAM;
+ """ % index_id)
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxPHRASE%02dF (
+ id mediumint(9) unsigned NOT NULL auto_increment,
+ term text default NULL,
+ hitlist longblob,
+ PRIMARY KEY (id),
+ KEY term (term(50))
+ ) ENGINE=MyISAM;
+ """ % index_id)
+ run_sql("""
+ CREATE TABLE IF NOT EXISTS idxPHRASE%02dR (
+ id_bibrec mediumint(9) unsigned NOT NULL,
+ termlist longblob,
+ type enum('CURRENT','FUTURE','TEMPORARY') NOT NULL default 'CURRENT',
+ PRIMARY KEY (id_bibrec,type)
+ ) ENGINE=MyISAM;
+ """ % index_id)
+ run_sql("""INSERT INTO idxINDEX VALUES (28,'filename','This index contains names of all files attached to the record.', '0000-00-00 00:00:00', '', 'native', '', 'No', 'No', 'No', 'BibIndexFilenameTokenizer')""")
+ run_sql("""INSERT INTO field VALUES (43,'file name', 'filename')""")
+ run_sql("""INSERT INTO idxINDEX_field (id_idxINDEX, id_field) VALUES (28,43)""")
+
+def estimate():
+ return 1
+
+def pre_upgrade():
+ pass
+
+def post_upgrade():
+ pass
diff --git a/invenio/testsuite/data/test_authors.cfg b/invenio/testsuite/data/test_authors.cfg
index 188454c9d..dedfc2198 100644
--- a/invenio/testsuite/data/test_authors.cfg
+++ b/invenio/testsuite/data/test_authors.cfg
@@ -1,33 +1,58 @@
+# -*- coding: utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2014 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
#Test for loops
include "test_authors.cfg"
+
authors[0], creator:
creator:
- @legacy(("100__a", "first author name", "authors[0].full_name"),
- ("100__e", "authors[0].relator_name"),
- ("100__h", "authors[0].CCID"),
- ("100__i", "authors[0].INSPIRE_number"),
- ("100__u", "first author affiliation", "authors[0].affiliation"))
+ @legacy((("100", "100__", "100__%"), ""),
+ ("100__a", "first author name", "full_name"),
+ ("100__e", "relator_name"),
+ ("100__h", "CCID"),
+ ("100__i", "INSPIRE_number"),
+ ("100__u", "first author affiliation", "affiliation"))
marc, "100__", { 'full_name':value['a'], 'first_name':util_split(value['a'],',',1), 'last_name':util_split(value['a'],',',0), 'relator_name':value['e'], 'CCID':value['h'], 'INSPIRE_number':value['i'], 'affiliation':value['u'] }
checker:
check_field_existence(0,1)
check_field_type('str')
- documentation:
- "Main Author"
- @subfield fn: "First name"
- @subfield ln: "Last name"
+ producer:
+ json_for_marc(), {"100__a": "full_name", "100__e": "relator_name", "100__h": "CCID", "100__i": "INSPIRE_number", "100__u": "affiliation"}
+ json_for_dc(), {"dc:creator": "full_name"}
+ description:
+ """Main Author"""
authors[n], contributor:
creator:
- @legacy(("700__a", "additional author name", "authors[1:].full_name"),
- ("700__u", "additional author affiliation", "authors[1:].affiliation"))
+ @legacy((("700", "700__", "700__%"), ""),
+ ("700__a", "additional author name", "full_name"),
+ ("700__u", "additional author affiliation", "affiliation"))
marc, "700__", {'full_name': value['a'], 'first_name':util_split(value['a'],',',1), 'last_name':util_split(value['a'],',',0), 'relator_name':value['e'], 'CCID':value['h'], 'INSPIRE_number':value['i'], 'affiliation':value['u'] }
checker:
check_field_existence(0,'n')
- checl_field_type('str')
- documentation:
- "Authors"
+ check_field_type('str')
+ producer:
+ json_for_marc(), {"700__a": "full_name", "700__e": "relator_name", "700__h": "CCID", "700__i": "INSPIRE_number", "700__u": "affiliation"}
+ json_for_dc(), {"dc:contributor": "full_name"}
+ description:
+ """Authors"""
+@inherit_from(("authors[0]",))
main_author:
- @inherit_from(("authors[0]",))
- documentation:
- "Just main author"
+ """Just main author"""
diff --git a/invenio/testsuite/data/test_bibfield.cfg b/invenio/testsuite/data/test_bibfield.cfg
index 988ef5a81..e7ce3bf3e 100644
--- a/invenio/testsuite/data/test_bibfield.cfg
+++ b/invenio/testsuite/data/test_bibfield.cfg
@@ -1,167 +1,274 @@
+# -*- coding: utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2014 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
include "test_authors.cfg"
include "test_title.cfg"
abstract:
creator:
- @legacy(("520__a", "abstract", "abstract.summary"),
- ("520__b", "abstract.expansion"),
- ("520__9", "abstract.number"))
+ @legacy((("520", "520__", "520__%"), "abstract", ""),
+ ("520__a", "abstract", "summary"),
+ ("520__b", "expansion"),
+ ("520__9", "number"))
marc, "520__", {'summary':value['a'], 'expansion':value['b'], 'number':value['9']}
+ producer:
+ json_for_marc(), {"520__a": "summary", "520__b": "expansion", "520__9": "number"}
+ json_for_dc(), {"dc:description":"summary"}
collection:
creator:
- @legacy(("980__%", "collection identifier", "collection"),
- ("980__a", "collection.primary"),
- ("980__b", "collection.secondary"),
- ("980__c", "collection.deleted"))
+ @legacy((("980", "980__", "980__%"), ""),
+ ("980__%", "collection identifier", ""),
+ ("980__a", "primary"),
+ ("980__b", "secondary"),
+ ("980__c", "deleted"))
marc, "980__", { 'primary':value['a'], 'secondary':value['b'], 'deleted':value['c'] }
+ producer:
+ json_for_marc(), {"980__a":"primary", "980__b":"secondary", "980__c":"deleted"}
-@persistent_identifier(1)
+@persistent_identifier(3)
doi:
- creator:
- @legacy (("0247_2", "doi"),)
- marc, "0247_", get_doi(value)
- checker:
- check_field_existence(0,1)
+ creator:
+ @legacy((("024", "0247_", "0247_%"), ""),
+ ("0247_a", ""))
+ marc, "0247_", get_doi(value)
+ checker:
+ check_field_existence(0,1)
+ producer:
+ json_for_marc(), {'0247_2': 'str("DOI")', '0247_a': ''}
-email:
+fft[n]:
creator:
- @legacy(("8560_f", "email"),)
- marc, "8560_", value['f']
+ @legacy(("FFT__a", "path"),
+ ("FFT__d", "description"),
+ ("FFT__f", "eformat"),
+ ("FFT__i", "temporary_id"),
+ ("FFT__m", "new_name"),
+ ("FFT__o", "flag"),
+ ("FFT__r", "restriction"),
+ ("FFT__s", "timestamp"),
+ ("FFT__t", "docfile_type"),
+ ("FFT__v", "version"),
+ ("FFT__x", "icon_path"),
+ ("FFT__z", "comment"),
+ ("FFT__w", "document_moreinfo"),
+ ("FFT__p", "version_moreinfo"),
+ ("FFT__b", "version_format_moreinfo"),
+ ("FFT__f", "format_moreinfo"))
+ marc, "FFT__", {'path': value['a'],
+ 'description': value['d'],
+ 'eformat': value['f'],
+ 'temporary_id': value['i'],
+ 'new_name': value['m'],
+ 'flag': value['o'],
+ 'restriction': value['r'],
+ 'timestamp': value['s'],
+ 'docfile_type': value['t'],
+ 'version': value['v'],
+ 'icon_path': value['x'],
+ 'comment': value['z'],
+ 'document_moreinfo': value['w'],
+ 'version_moreinfo': value['p'],
+ 'version_format_moreinfo': value['b'],
+ 'format_moreinfo': value['u']
+ }
+ @only_if_value((is_local_url(value['u']), ))
+ marc, "8564_", {'hots_name': value['a'],
+ 'access_number': value['b'],
+ 'compression_information': value['c'],
+ 'path':value['d'],
+ 'electronic_name': value['f'],
+ 'request_processor': value['h'],
+ 'institution': value['i'],
+ 'formart': value['q'],
+ 'settings': value['r'],
+ 'file_size': value['s'],
+ 'url': value['u'],
+ 'subformat':value['x'],
+ 'description':value['y'],
+ 'comment':value['z']}
+ producer:
+ json_for_marc(), {"FFT__a": "path", "FFT__d": "description", "FFT__f": "eformat", "FFT__i": "temporary_id", "FFT__m": "new_name", "FFT__o": "flag", "FFT__r": "restriction", "FFT__s": "timestamp", "FFT__t": "docfile_type", "FFT__v": "version", "FFT__x": "icon_path", "FFT__z": "comment", "FFT__w": "document_moreinfo", "FFT__p": "version_moreinfo", "FFT__b": "version_format_moreinfo", "FFT__f": "format_moreinfo"}
isbn:
creator:
- @legacy(("020__a", "isbn", "isbn.isbn"),
- ("020__u", "isbn.medium"))
+ @legacy((("020", "020__", "020__%"), ""),
+ ("020__a", "isbn", "isbn"),
+ ("020__u", "medium"))
marc, "020__", {'isbn':value['a'], 'medium':value['u']}
+ checker:
+ check_field_type('isbn', 'isbn')
+ producer:
+ json_for_marc(), {"020__a": "isbn", "020__u": "medium"}
-keywords:
+keywords[n]:
creator:
- @legacy(("6531_a", "keyword", "keyword.term"),
- ("6531_9", "keyword.institute"))
+ @legacy((("653", "6531_", "6531_%"), ""),
+ ("6531_a", "keyword", "term"),
+ ("6531_9", "institute"))
marc, "6531_", { 'term': value['a'], 'institute': value['9'] }
checker:
check_field_existence(0,'n')
check_field_type('str')
+ producer:
+ json_for_marc(), {"6531_a": "term", "6531_9": "institute"}
language:
creator:
- @legacy(("041__a", "language"),)
+ @legacy((("041", "041__", "041__%"), ""),
+ ("041__a", ""))
marc, "041__", value['a']
+ producer:
+ json_for_marc(), {"041__a": ""}
+ json_for_dc(), {"dc:language": ""}
+
+modification_date, version_id:
+ schema:
+ {'modification_date': {'type': 'datetime', 'required': True, 'default': lambda: __import__('datetime').datetime.now()}}
+ creator:
+ @legacy(('005', ''),)
+ marc, '005', datetime.datetime(*(time.strptime(value, "%Y%m%d%H%M%S.0")[0:6]))
+ json:
+ dumps, lambda d: d.isoformat()
+ loads, lambda d: __import__('datetime').datetime.strptime(d, "%Y-%m-%dT%H:%M:%S")
primary_report_number:
creator:
- @legacy(("037__a", "primary report number", "primary_report_number"),)
+ @legacy((("037", "037__", "037__%"), ""),
+ ("037__a", "primary report number", ""), )
marc, "037__", value['a']
+ producer:
+ json_for_marc(), {"037__a": ""}
-@persistent_identifier(0)
-recid:
+reference:
creator:
- @legacy(("001", "record ID", "recid"),)
- marc, "001", value
- checker:
- check_field_existence(1)
- check_field_type('num')
+ @legacy((("999", "999C5", "999C5%"), ""),
+ ("999C5", "reference", ""),
+ ("999C5a", "doi"),
+ ("999C5h", "authors"),
+ ("999C5m", "misc"),
+ ("999C5n", "issue_number"),
+ ("999C5o", "order_number"),
+ ("999C5p", "page"),
+ ("999C5r", "report_number"),
+ ("999C5s", "title"),
+ ("999C5u", "url"),
+ ("999C5v", "volume"),
+ ("999C5y", "year"),)
+ marc, "999C5", {'doi':value['a'], 'authors':value['h'], 'misc':value['m'], 'issue_number':value['n'], 'order_number':value['o'], 'page':value['p'], 'report_number':value['r'], 'title':value['s'], 'url':value['u'], 'volume':value['v'], 'year':value['y'],}
+ producer:
+ json_for_marc(), {"999C5a": "doi", "999C5h": "authors", "999C5m": "misc", "999C5n": "issue_number", "999C5o":"order_number", "999C5p":"page", "999C5r":"report_number", "999C5s":"title", "999C5u":"url", "999C5v":"volume", "999C5y": "year"}
-report_number:
+@persistent_identifier(2)
+system_control_number:
creator:
- @legacy(("088_a", "additional report number", "report_number.report_number"),
- ("088_9", "internal"))
- marc, "088__", {'report_number':value['a'], 'internal':value['9']}
+ @legacy((("035", "035__", "035__%"), ""),
+ ("035__a", "system_control_number"),
+ ("035__9", "institute"))
+ marc, "035__", {'value': value['a'], 'canceled':value['z'], 'linkpage':value['6'], 'institute':value['9']}
+ producer:
+ json_for_marc(), {"035__a": "system_control_number", "035__9": "institute"}
-#To be overwritten by test_tiltle.cfg
-title:
+@persistent_identifier(1)
+system_number:
creator:
- marc, "245__", value
+ @legacy((("970", "970__", "970__%"), ""),
+ ("970__a", "sysno"),
+ ("970__d", "recid"))
+ marc, "970__", {'value':value['a'], 'recid':value['d']}
checker:
check_field_existence(0,1)
- documentation:
- "Some useless documentation"
+ producer:
+ json_for_marc(), {"970__a": "sysno", "970__d": "recid"}
-subject:
+#To be overwritten by test_tiltle.cfg
+title:
+ """Some useless documentation"""
creator:
- @legacy(("65017a", "main subject", "subject.term"),
- ("650172", "subject.source"),
- ("65017e", "subject.relator"))
- marc, "65017", {'term':value['a'], 'source':value['2'], 'relator':value['e']}
- documentation:
- @subfield term: "Topical term or geographic name"
- @subfield source: "Source of heading or term"
- @subfield relator: "Specifies the relationship between the topical heading and the described materials"
+ marc, "245__", value['foo']
url:
creator:
- @legacy(("8564_p", "url.path"),
- ("8564_q", "url.eformat"),
- ("8564_s", "url.file_size"),
- ("8564_u", "url", "url.url"),
- ("8564_x", "url.nonpublic_note"),
- ("8564_y", "caption", "url.link"),
- ("8564_z", "url.public_note"))
- marc, "8564_", {'path':value['d'], 'eformart':value['q'], 'file_size':value['s'], 'url':value['u'], 'nonpublic_note':value['x'], 'link':value['y'], 'public_note':value['z']}
- documentation:
- @subfield url: "used for URL and URN, repeatable for URN. repeat 856 for several url's"
- @subfield public_note: "Stamped by WebSubmit: DATE"
-
-###############################################################################
-########## ##########
-########## DERIVED AND CALCULATED FIELDS DEFINITION ##########
-########## ##########
-###############################################################################
+ @legacy((("856", "8564_", "8564_%"), ""),
+ ("8564_a", "host_name"),
+ ("8564_b", "access_number"),
+ ("8564_c", "compression_information"),
+ ("8564_d", "path"),
+ ("8564_f", "electronic_name"),
+ ("8564_h", "request_processor"),
+ ("8564_i", "institution"),
+ ("8564_q", "eformat"),
+ ("8564_r", "settings"),
+ ("8564_s", "file_size"),
+ ("8564_u", "url", "url"),
+ ("8564_x", "subformat"),
+ ("8564_y", "caption", "description"),
+ ("8564_z", "comment"))
+ @only_if_value((not is_local_url(value['u']), ))
+ marc, "8564_", {'host_name': value['a'],
+ 'access_number': value['b'],
+ 'compression_information': value['c'],
+ 'path':value['d'],
+ 'electronic_name': value['f'],
+ 'request_processor': value['h'],
+ 'institution': value['i'],
+ 'eformart': value['q'],
+ 'settings': value['r'],
+ 'size': value['s'],
+ 'url': value['u'],
+ 'subformat':value['x'],
+ 'description':value['y'],
+ 'comment':value['z']}
+ producer:
+ json_for_marc(), {"8564_a": "host_name", "8564_b": "access_number", "8564_c": "compression_information", "8564_d": "path", "8564_f": "electronic_name", "8564_h": "request_processor", "8564_i": "institution", "8564_q": "eformat", "8564_r": "settings", "8564_s": "file_size", "8564_u": "url", "8564_x": "subformat", "8564_y": "description", "8564_z": "comment"}
+ json_for_dc(), {"dc:identifier": "url"}
_persistent_identifiers_keys:
calculated:
- @parse_first(('system_control_number', 'recid', 'doi', 'oai'))
+ @parse_first(('system_control_number', 'recid', 'doi', 'oai', 'system_number', '_id'))
+ @memoize()
get_persistent_identifiers_keys(self.keys())
- documentation:
+ description:
"""
This field will tell you which fields among all are considered as
persistent identifiers (decorated with @persistent_identifier)
If a new persistent identifier field is added the cached version of this
field must be rebuild.
Note: If a new persistent idenfier is added the list of fields to parse
before this one should be updated
"""
_random:
+ """Checks on the fly fields"""
derived:
- @do_not_cache
random.randint(0,100)
- documentation:
- "Checks @do_not_cache"
-
-number_of_authors:
- derived:
- @parse_first(('authors',))
- @depends_on(('authors',))
- len(self['authors'])
- checker:
- check_field_existence(0, 1)
- check_field_type('num')
- documentation:
- "Number of authors"
@persistent_identifier(2)
dummy:
- derived:
- @parse_first(('number_of_authors',))
- @depends_on(('authors',))
- self['nunmber_of_authors']
- checker:
- check_field_existence(0, 1)
- check_field_type('num')
- documentation:
- "Dummy number of authors"
+ """Dummy number of authors"""
-_number_of_copies:
- calculated:
- @parse_first(('recid', 'collection'))
- @depends_on(('recid', 'collection.primary'))
- @only_if(('BOOK' in self['collection.primary'],))
- number_of_copies(self['recid'])
+@extend
+dummy:
+ derived:
+ @depends_on(('number_of_authors',))
+ self.get('nunmber_of_authors', 0)
checker:
- check_field_existence(0, 1)
- check_field_type('num')
- documentation:
- "Number of copies"
+ check_field_existence(0, 1)
+ check_field_type('num')
diff --git a/invenio/testsuite/data/test_title.cfg b/invenio/testsuite/data/test_title.cfg
index 1e84a795a..030ad4dc9 100644
--- a/invenio/testsuite/data/test_title.cfg
+++ b/invenio/testsuite/data/test_title.cfg
@@ -1,19 +1,37 @@
+# -*- coding: utf-8 -*-
+##
+## This file is part of Invenio.
+## Copyright (C) 2014 CERN.
+##
+## Invenio is free software; you can redistribute it and/or
+## modify it under the terms of the GNU General Public License as
+## published by the Free Software Foundation; either version 2 of the
+## License, or (at your option) any later version.
+##
+## Invenio is distributed in the hope that it will be useful, but
+## WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+## General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with Invenio; if not, write to the Free Software Foundation, Inc.,
+## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
+@override
title:
+ """Overrided Title"""
creator:
- @legacy(("245__%", "main title", "title"),
- ("245__a", "title", "title.title"),
- ("245__b", "title.subtitle"),
- ("245__k", "title.form"))
+ @legacy((("245", "245__","245__%"), ""),
+ ("245__a", "title", "title"),
+ ("245__b", "subtitle"),
+ ("245__k", "form"))
marc, "245__", { 'title':value['a'], 'subtitle':value['b'], 'form':value['k'] }
checker:
- @master_format("marc",)
check_field_existence(0,1)
check_field_type('str')
- documentation:
- "Title"
title_parallel:
creator:
- @legacy(("246_1a", "title_parallel.title"),
- ("246_1i", "title_parallel.text"))
+ @legacy(("246_1a", "title"),
+ ("246_1i", "text"))
marc, "246_1", { 'title':value['a'], 'text':value['i']}
diff --git a/modules/bibauthorid/lib/bibauthorid_dbinterface_unit_tests.py b/invenio/testsuite/test_utils_redis.py
similarity index 51%
rename from modules/bibauthorid/lib/bibauthorid_dbinterface_unit_tests.py
rename to invenio/testsuite/test_utils_redis.py
index 873a57410..29a170d06 100644
--- a/modules/bibauthorid/lib/bibauthorid_dbinterface_unit_tests.py
+++ b/invenio/testsuite/test_utils_redis.py
@@ -1,41 +1,51 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
+## Copyright (C) 2006, 2007, 2008, 2010, 2011, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+"""Redis utils Regression Test Suite."""
-"""Unit tests for the search engine."""
+import time
-__revision__ = \
- "$Id$"
+from invenio.testsuite import make_test_suite, run_test_suite, InvenioTestCase
-from itertools import chain
-from invenio.testutils import InvenioTestCase, make_test_suite, run_test_suite
-from invenio.bibauthorid_cluster_set import ClusterSet
-
-class TestDummy(InvenioTestCase):
+class RedisUtilsTest(InvenioTestCase):
def setUp(self):
- pass
+ from invenio.utils.redis import get_redis
+ self.db = get_redis()
+
+ def test_set(self):
+ from invenio.config import CFG_REDIS_HOSTS
+ self.db.set('hello_test', 'a')
+ if CFG_REDIS_HOSTS:
+ self.assertEqual(self.db.get('hello_test'), 'a')
+ else:
+ self.assertEqual(self.db.get('hello_test'), None)
+ self.db.delete('hello_test')
+
+ def test_expire(self):
+ self.db.set('hello', 'a', 1)
+ time.sleep(2)
+ self.assertEqual(self.db.get('hello'), None)
+
- def test_one(self):
- pass
+TEST_SUITE = make_test_suite(RedisUtilsTest)
-TEST_SUITE = make_test_suite(TestDummy)
if __name__ == "__main__":
run_test_suite(TEST_SUITE, warn_user=True)
diff --git a/invenio/testsuite/test_utils_text.py b/invenio/testsuite/test_utils_text.py
index 75f529e92..56a16d02d 100644
--- a/invenio/testsuite/test_utils_text.py
+++ b/invenio/testsuite/test_utils_text.py
@@ -1,500 +1,587 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2008, 2009, 2010, 2011, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""Unit tests for the textutils library."""
__revision__ = "$Id$"
try:
import chardet
CHARDET_AVAILABLE = True
except ImportError:
CHARDET_AVAILABLE = False
try:
from unidecode import unidecode
UNIDECODE_AVAILABLE = True
except ImportError:
UNIDECODE_AVAILABLE = False
from invenio.base.wrappers import lazy_import
from invenio.testsuite import make_test_suite, run_test_suite, InvenioTestCase
decode_to_unicode = lazy_import('invenio.utils.text:decode_to_unicode')
escape_latex = lazy_import('invenio.utils.text:escape_latex')
guess_minimum_encoding = lazy_import('invenio.utils.text:guess_minimum_encoding')
+show_diff = lazy_import('invenio.utils.text:show_diff')
strip_accents = lazy_import('invenio.utils.text:strip_accents')
translate_latex2unicode = lazy_import('invenio.utils.text:translate_latex2unicode')
translate_to_ascii = lazy_import('invenio.utils.text:translate_to_ascii')
transliterate_ala_lc = lazy_import('invenio.utils.text:transliterate_ala_lc')
wash_for_utf8 = lazy_import('invenio.utils.text:wash_for_utf8')
wash_for_xml = lazy_import('invenio.utils.text:wash_for_xml')
wrap_text_in_a_box = lazy_import('invenio.utils.text:wrap_text_in_a_box')
class GuessMinimumEncodingTest(InvenioTestCase):
"""Test functions related to guess_minimum_encoding function."""
def test_guess_minimum_encoding(self):
"""textutils - guess_minimum_encoding."""
self.assertEqual(guess_minimum_encoding('patata'), ('patata', 'ascii'))
self.assertEqual(guess_minimum_encoding('àèéìòù'), ('\xe0\xe8\xe9\xec\xf2\xf9', 'latin1'))
self.assertEqual(guess_minimum_encoding('Ιθάκη'), ('Ιθάκη', 'utf8'))
class WashForXMLTest(InvenioTestCase):
"""Test functions related to wash_for_xml function."""
def test_latin_characters_washing_1_0(self):
"""textutils - washing latin characters for XML 1.0."""
self.assertEqual(wash_for_xml('àèéìòùÀ'), 'àèéìòùÀ')
def test_latin_characters_washing_1_1(self):
"""textutils - washing latin characters for XML 1.1."""
self.assertEqual(wash_for_xml('àèéìòùÀ', xml_version='1.1'), 'àèéìòùÀ')
def test_chinese_characters_washing_1_0(self):
"""textutils - washing chinese characters for XML 1.0."""
self.assertEqual(wash_for_xml('''
春眠暁を覚えず
処処に啼鳥と聞く
夜来風雨の声
花落つること
知んぬ多少ぞ'''), '''
春眠暁を覚えず
処処に啼鳥と聞く
夜来風雨の声
花落つること
知んぬ多少ぞ''')
def test_chinese_characters_washing_1_1(self):
"""textutils - washing chinese characters for XML 1.1."""
self.assertEqual(wash_for_xml('''
春眠暁を覚えず
処処に啼鳥と聞く
夜来風雨の声
花落つること
知んぬ多少ぞ''', xml_version='1.1'), '''
春眠暁を覚えず
処処に啼鳥と聞く
夜来風雨の声
花落つること
知んぬ多少ぞ''')
def test_greek_characters_washing_1_0(self):
"""textutils - washing greek characters for XML 1.0."""
self.assertEqual(wash_for_xml('''
ἄνδρα μοι ἔννεπε, μου̂σα, πολύτροπον, ὃς μάλα πολλὰ
πλάγχθη, ἐπεὶ Τροίης ἱερὸν πτολίεθρον ἔπερσεν:
πολλω̂ν δ' ἀνθρώπων ἴδεν ἄστεα καὶ νόον ἔγνω,
πολλὰ δ' ὅ γ' ἐν πόντῳ πάθεν ἄλγεα ὃν κατὰ θυμόν,
ἀρνύμενος ἥν τε ψυχὴν καὶ νόστον ἑταίρων.
ἀλλ' οὐδ' ὣς ἑτάρους ἐρρύσατο, ἱέμενός περ:
αὐτω̂ν γὰρ σφετέρῃσιν ἀτασθαλίῃσιν ὄλοντο,
νήπιοι, οἳ κατὰ βου̂ς ̔Υπερίονος ̓Ηελίοιο
ἤσθιον: αὐτὰρ ὁ τοι̂σιν ἀφείλετο νόστιμον ἠ̂μαρ.
τω̂ν ἁμόθεν γε, θεά, θύγατερ Διός, εἰπὲ καὶ ἡμι̂ν.'''), '''
ἄνδρα μοι ἔννεπε, μου̂σα, πολύτροπον, ὃς μάλα πολλὰ
πλάγχθη, ἐπεὶ Τροίης ἱερὸν πτολίεθρον ἔπερσεν:
πολλω̂ν δ' ἀνθρώπων ἴδεν ἄστεα καὶ νόον ἔγνω,
πολλὰ δ' ὅ γ' ἐν πόντῳ πάθεν ἄλγεα ὃν κατὰ θυμόν,
ἀρνύμενος ἥν τε ψυχὴν καὶ νόστον ἑταίρων.
ἀλλ' οὐδ' ὣς ἑτάρους ἐρρύσατο, ἱέμενός περ:
αὐτω̂ν γὰρ σφετέρῃσιν ἀτασθαλίῃσιν ὄλοντο,
νήπιοι, οἳ κατὰ βου̂ς ̔Υπερίονος ̓Ηελίοιο
ἤσθιον: αὐτὰρ ὁ τοι̂σιν ἀφείλετο νόστιμον ἠ̂μαρ.
τω̂ν ἁμόθεν γε, θεά, θύγατερ Διός, εἰπὲ καὶ ἡμι̂ν.''')
def test_greek_characters_washing_1_1(self):
"""textutils - washing greek characters for XML 1.1."""
self.assertEqual(wash_for_xml('''
ἄνδρα μοι ἔννεπε, μου̂σα, πολύτροπον, ὃς μάλα πολλὰ
πλάγχθη, ἐπεὶ Τροίης ἱερὸν πτολίεθρον ἔπερσεν:
πολλω̂ν δ' ἀνθρώπων ἴδεν ἄστεα καὶ νόον ἔγνω,
πολλὰ δ' ὅ γ' ἐν πόντῳ πάθεν ἄλγεα ὃν κατὰ θυμόν,
ἀρνύμενος ἥν τε ψυχὴν καὶ νόστον ἑταίρων.
ἀλλ' οὐδ' ὣς ἑτάρους ἐρρύσατο, ἱέμενός περ:
αὐτω̂ν γὰρ σφετέρῃσιν ἀτασθαλίῃσιν ὄλοντο,
νήπιοι, οἳ κατὰ βου̂ς ̔Υπερίονος ̓Ηελίοιο
ἤσθιον: αὐτὰρ ὁ τοι̂σιν ἀφείλετο νόστιμον ἠ̂μαρ.
τω̂ν ἁμόθεν γε, θεά, θύγατερ Διός, εἰπὲ καὶ ἡμι̂ν.''',
xml_version='1.1'), '''
ἄνδρα μοι ἔννεπε, μου̂σα, πολύτροπον, ὃς μάλα πολλὰ
πλάγχθη, ἐπεὶ Τροίης ἱερὸν πτολίεθρον ἔπερσεν:
πολλω̂ν δ' ἀνθρώπων ἴδεν ἄστεα καὶ νόον ἔγνω,
πολλὰ δ' ὅ γ' ἐν πόντῳ πάθεν ἄλγεα ὃν κατὰ θυμόν,
ἀρνύμενος ἥν τε ψυχὴν καὶ νόστον ἑταίρων.
ἀλλ' οὐδ' ὣς ἑτάρους ἐρρύσατο, ἱέμενός περ:
αὐτω̂ν γὰρ σφετέρῃσιν ἀτασθαλίῃσιν ὄλοντο,
νήπιοι, οἳ κατὰ βου̂ς ̔Υπερίονος ̓Ηελίοιο
ἤσθιον: αὐτὰρ ὁ τοι̂σιν ἀφείλετο νόστιμον ἠ̂μαρ.
τω̂ν ἁμόθεν γε, θεά, θύγατερ Διός, εἰπὲ καὶ ἡμι̂ν.''')
def test_russian_characters_washing_1_0(self):
"""textutils - washing greek characters for XML 1.0."""
self.assertEqual(wash_for_xml('''
В тени дерев, над чистыми водами
Дерновый холм вы видите ль, друзья?
Чуть слышно там плескает в брег струя;
Чуть ветерок там дышит меж листами;
На ветвях лира и венец...
Увы! друзья, сей холм - могила;
Здесь прах певца земля сокрыла;
Бедный певец!''', xml_version='1.1'), '''
В тени дерев, над чистыми водами
Дерновый холм вы видите ль, друзья?
Чуть слышно там плескает в брег струя;
Чуть ветерок там дышит меж листами;
На ветвях лира и венец...
Увы! друзья, сей холм - могила;
Здесь прах певца земля сокрыла;
Бедный певец!''')
def test_russian_characters_washing_1_1(self):
"""textutils - washing greek characters for XML 1.1."""
self.assertEqual(wash_for_xml('''
В тени дерев, над чистыми водами
Дерновый холм вы видите ль, друзья?
Чуть слышно там плескает в брег струя;
Чуть ветерок там дышит меж листами;
На ветвях лира и венец...
Увы! друзья, сей холм - могила;
Здесь прах певца земля сокрыла;
Бедный певец!''', xml_version='1.1'), '''
В тени дерев, над чистыми водами
Дерновый холм вы видите ль, друзья?
Чуть слышно там плескает в брег струя;
Чуть ветерок там дышит меж листами;
На ветвях лира и венец...
Увы! друзья, сей холм - могила;
Здесь прах певца земля сокрыла;
Бедный певец!''')
def test_illegal_characters_washing_1_0(self):
"""textutils - washing illegal characters for XML 1.0."""
self.assertEqual(wash_for_xml(chr(8) + chr(9) + 'some chars'), '\tsome chars')
self.assertEqual(wash_for_xml('$b\bar{b}$'), '$bar{b}$')
def test_illegal_characters_washing_1_1(self):
"""textutils - washing illegal characters for XML 1.1."""
self.assertEqual(wash_for_xml(chr(8) + chr(9) + 'some chars',
xml_version='1.1'), '\x08\tsome chars')
self.assertEqual(wash_for_xml('$b\bar{b}$', xml_version='1.1'), '$b\x08ar{b}$')
class WashForUTF8Test(InvenioTestCase):
def test_normal_legal_string_washing(self):
"""textutils - testing UTF-8 washing on a perfectly normal string"""
some_str = "This is an example string"
self.assertEqual(some_str, wash_for_utf8(some_str))
def test_chinese_string_washing(self):
"""textutils - testing washing functions on chinese script"""
some_str = """春眠暁を覚えず
処処に啼鳥と聞く
夜来風雨の声
花落つること
知んぬ多少ぞ"""
self.assertEqual(some_str, wash_for_utf8(some_str))
def test_russian_characters_washing(self):
"""textutils - washing Russian characters for UTF-8"""
self.assertEqual(wash_for_utf8('''
В тени дерев, над чистыми водами
Дерновый холм вы видите ль, друзья?
Чуть слышно там плескает в брег струя;
Чуть ветерок там дышит меж листами;
На ветвях лира и венец...
Увы! друзья, сей холм - могила;
Здесь прах певца земля сокрыла;
Бедный певец!'''), '''
В тени дерев, над чистыми водами
Дерновый холм вы видите ль, друзья?
Чуть слышно там плескает в брег струя;
Чуть ветерок там дышит меж листами;
На ветвях лира и венец...
Увы! друзья, сей холм - могила;
Здесь прах певца земля сокрыла;
Бедный певец!''')
def test_remove_incorrect_unicode_characters(self):
"""textutils - washing out the incorrect characters"""
self.assertEqual(wash_for_utf8("Ź\206dź\204bło żół\203wia \202"), "Źdźbło żółwia ")
def test_empty_string_wash(self):
"""textutils - washing an empty string"""
self.assertEqual(wash_for_utf8(""), "")
def test_only_incorrect_unicode_wash(self):
"""textutils - washing an empty string"""
self.assertEqual(wash_for_utf8("\202\203\204\205"), "")
def test_raising_exception_on_incorrect(self):
"""textutils - assuring an exception on incorrect input"""
self.assertRaises(UnicodeDecodeError, wash_for_utf8, "\202\203\204\205", correct=False)
def test_already_utf8_input(self):
"""textutils - washing a Unicode string into UTF-8 binary string"""
self.assertEqual('Göppert', wash_for_utf8(u'G\xf6ppert', True))
class WrapTextInABoxTest(InvenioTestCase):
"""Test functions related to wrap_text_in_a_box function."""
def test_plain_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box plain."""
result = """
**********************************************
** foo bar **
**********************************************
"""
self.assertEqual(wrap_text_in_a_box('foo bar'), result)
def test_empty_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box empty."""
result = """
**********************************************
**********************************************
"""
self.assertEqual(wrap_text_in_a_box(), result)
def test_with_title_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box with title."""
result = """
**********************************************
** a Title! **
** **************************************** **
** foo bar **
**********************************************
"""
self.assertEqual(wrap_text_in_a_box('foo bar', title='a Title!'), result)
def test_multiline_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box multiline."""
result = """
**********************************************
** foo bar **
**********************************************
"""
self.assertEqual(wrap_text_in_a_box('foo\n bar'), result)
def test_real_multiline_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box real multiline."""
result = """
**********************************************
** foo **
** bar **
**********************************************
"""
self.assertEqual(wrap_text_in_a_box('foo\n\nbar'), result)
def test_real_no_width_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box no width."""
result = """
************
** foobar **
************
"""
self.assertEqual(wrap_text_in_a_box('foobar', min_col=0), result)
def test_real_nothing_at_all_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box nothing at all."""
result = """
******
******
"""
self.assertEqual(wrap_text_in_a_box(min_col=0), result)
def test_real_squared_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box squared style."""
result = """
+--------+
| foobar |
+--------+
"""
self.assertEqual(wrap_text_in_a_box('foobar', style='squared', min_col=0), result)
def test_indented_text_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box indented text."""
text = """
def test_real_squared_wrap_text_in_a_box(self):\n
\"""wrap_text_in_a_box - squared style.\"""\n
result = \"""\n
+--------+\n
| foobar |\n
+--------+
\"""
"""
result = """
******************************
** def test_real_square **
** d_wrap_text_in_a_box **
** (self): **
** \"""wrap_text_in_ **
** a_box - squared **
** style.\""" **
** result = \""" **
** +--------+ **
** | foobar | **
** +--------+\""" **
******************************
"""
self.assertEqual(wrap_text_in_a_box(text, min_col=0, max_col=30, break_long=True), result)
def test_single_new_line_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box single new line."""
result = """
**********************************************
** ciao come và? **
**********************************************
"""
self.assertEqual(wrap_text_in_a_box("ciao\ncome và?"), result)
def test_indented_box_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box indented box."""
result = """
**********************************************
** foobar **
**********************************************
"""
self.assertEqual(wrap_text_in_a_box('foobar', tab_num=1), result)
def test_real_conclusion_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box conclusion."""
result = """----------------------------------------
foobar \n"""
self.assertEqual(wrap_text_in_a_box('foobar', style='conclusion'), result)
def test_real_longtext_wrap_text_in_a_box(self):
"""textutils - wrap_text_in_a_box long text."""
text = """Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat."""
result = """
************************************************************************
** Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do **
** eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut **
** enim ad minim veniam, quis nostrud exercitation ullamco laboris **
** nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in **
** reprehenderit in voluptate velit esse cillum dolore eu fugiat **
** nulla pariatur. Excepteur sint occaecat cupidatat non proident, **
** sunt in culpa qui officia deserunt mollit anim id est laborum. **
** At vero eos et accusamus et iusto odio dignissimos ducimus qui **
** blanditiis praesentium voluptatum deleniti atque corrupti quos **
** dolores et quas molestias excepturi sint occaecati cupiditate non **
** provident, similique sunt in culpa qui officia deserunt mollitia **
** animi, id est laborum et dolorum fuga. Et harum quidem rerum **
** facilis est et expedita distinctio. Nam libero tempore, cum soluta **
** nobis est eligendi optio cumque nihil impedit quo minus id quod **
** maxime placeat facere possimus, omnis voluptas assumenda est, **
** omnis dolor repellendus. Temporibus autem quibusdam et aut **
** officiis debitis aut rerum necessitatibus saepe eveniet ut et **
** voluptates repudiandae sint et molestiae non recusandae. Itaque **
** earum rerum hic tenetur a sapiente delectus, ut aut reiciendis **
** voluptatibus maiores alias consequatur aut perferendis doloribus **
** asperiores repellat. **
************************************************************************
"""
self.assertEqual(wrap_text_in_a_box(text), result)
class DecodeToUnicodeTest(InvenioTestCase):
"""Test functions related to decode_to_unicode function."""
if CHARDET_AVAILABLE:
def test_decode_to_unicode(self):
"""textutils - decode_to_unicode."""
self.assertEqual(decode_to_unicode('\202\203\204\205', default_encoding='latin1'), u'\x82\x83\x84\x85')
self.assertEqual(decode_to_unicode('àèéìòù'), u'\xe0\xe8\xe9\xec\xf2\xf9')
self.assertEqual(decode_to_unicode('Ιθάκη'), u'\u0399\u03b8\u03ac\u03ba\u03b7')
else:
pass
class Latex2UnicodeTest(InvenioTestCase):
"""Test functions related to translating LaTeX symbols to Unicode."""
def test_latex_to_unicode(self):
"""textutils - latex_to_unicode"""
self.assertEqual(translate_latex2unicode("\\'a \\'i \\'U").encode('utf-8'), "á í Ú")
self.assertEqual(translate_latex2unicode("\\'N \\k{i}"), u'\u0143 \u012f')
self.assertEqual(translate_latex2unicode("\\AAkeson"), u'\u212bkeson')
self.assertEqual(translate_latex2unicode("$\\mathsl{\\Zeta}$"), u'\U0001d6e7')
class TestStripping(InvenioTestCase):
"""Test for stripping functions like accents and control characters."""
if UNIDECODE_AVAILABLE:
def test_text_to_ascii(self):
"""textutils - transliterate to ascii using unidecode"""
self.assert_(translate_to_ascii(
["á í Ú", "H\xc3\xb6hne", "Åge Øst Vær", "normal"]) in
(["a i U", "Hohne", "Age Ost Vaer", "normal"], ## unidecode < 0.04.13
['a i U', 'Hoehne', 'Age Ost Vaer', 'normal']) ## unidecode >= 0.04.13
)
self.assertEqual(translate_to_ascii("àèéìòù"), ["aeeiou"])
self.assertEqual(translate_to_ascii("ß"), ["ss"])
self.assertEqual(translate_to_ascii(None), None)
self.assertEqual(translate_to_ascii([]), [])
self.assertEqual(translate_to_ascii([None]), [None])
else:
pass
def test_strip_accents(self):
"""textutils - transliterate to ascii (basic)"""
self.assertEqual("memememe",
strip_accents('mémêmëmè'))
self.assertEqual("MEMEMEME",
strip_accents('MÉMÊMËMÈ'))
self.assertEqual("oe",
strip_accents('œ'))
self.assertEqual("OE",
strip_accents('Œ'))
+class TestDiffering(InvenioTestCase):
+ """Test for differing two strings."""
+
+ string1 = """Lorem ipsum dolor sit amet, consectetur adipiscing
+elit. Donec fringilla tellus eget fringilla sagittis. Pellentesque
+posuere lacus id erat tristique pulvinar. Morbi volutpat, diam
+eget interdum lobortis, lacus mi cursus leo, sit amet porttitor
+neque est vitae lectus. Donec tempor metus vel tincidunt fringilla.
+Nam iaculis lacinia nisl, enim sollicitudin
+convallis. Morbi ut mauris velit. Proin suscipit dolor id risus
+placerat sodales nec id elit. Morbi vel lacinia lectus, eget laoreet
+dui. Nunc commodo neque porttitor eros placerat, sed ultricies purus
+accumsan. In velit nisi, accumsan molestie gravida a, rutrum in augue.
+Nulla pharetra purus nec dolor ornare, ut aliquam odio placerat.
+Aenean ultrices condimentum quam vitae pharetra."""
+
+ string2 = """Lorem ipsum dolor sit amet, consectetur adipiscing
+elit. Donec fringilla tellus eget fringilla sagittis. Pellentesque
+posuere lacus id erat.
+eget interdum lobortis, lacus mi cursus leo, sit amet porttitor
+neque est vitae lectus. Donec tempor metus vel tincidunt fringilla.
+Nam iaculis lacinia nisl, consectetur viverra enim sollicitudin
+convallis. Morbi ut mauris velit. Proin suscipit dolor id risus
+placerat sodales nec id elit. Morbi vel lacinia lectus, eget laoreet
+placerat sodales nec id elit. Morbi vel lacinia lectus, eget laoreet
+dui. Nunc commodo neque porttitor eros placerat, sed ultricies purus
+accumsan. In velit nisi, lorem ipsum lorem gravida a, rutrum in augue.
+Nulla pharetra purus nec dolor ornare, ut aliquam odio placerat.
+Aenean ultrices condimentum quam vitae pharetra."""
+
+ def test_show_diff_plain_text(self):
+ """textutils - show_diff() with plain text"""
+
+ expected_result = """
+ Lorem ipsum dolor sit amet, consectetur adipiscing
+ elit. Donec fringilla tellus eget fringilla sagittis. Pellentesque
+-posuere lacus id erat.
++posuere lacus id erat tristique pulvinar. Morbi volutpat, diam
+ eget interdum lobortis, lacus mi cursus leo, sit amet porttitor
+ neque est vitae lectus. Donec tempor metus vel tincidunt fringilla.
+-Nam iaculis lacinia nisl, consectetur viverra enim sollicitudin
++Nam iaculis lacinia nisl, enim sollicitudin
+ convallis. Morbi ut mauris velit. Proin suscipit dolor id risus
+ placerat sodales nec id elit. Morbi vel lacinia lectus, eget laoreet
+-placerat sodales nec id elit. Morbi vel lacinia lectus, eget laoreet
+ dui. Nunc commodo neque porttitor eros placerat, sed ultricies purus
+-accumsan. In velit nisi, lorem ipsum lorem gravida a, rutrum in augue.
++accumsan. In velit nisi, accumsan molestie gravida a, rutrum in augue.
+ Nulla pharetra purus nec dolor ornare, ut aliquam odio placerat.
+ Aenean ultrices condimentum quam vitae pharetra.
+"""
+
+ self.assertEqual(show_diff(self.string1, self.string2), expected_result)
+
+ def test_show_diff_html(self):
+ """textutils - show_diff() with plain text"""
+
+ expected_result = """<pre>
+Lorem ipsum dolor sit amet, consectetur adipiscing
+elit. Donec fringilla tellus eget fringilla sagittis. Pellentesque
+<strong class="diff_field_deleted">posuere lacus id erat.</strong>
+<strong class="diff_field_added">posuere lacus id erat tristique pulvinar. Morbi volutpat, diam</strong>
+eget interdum lobortis, lacus mi cursus leo, sit amet porttitor
+neque est vitae lectus. Donec tempor metus vel tincidunt fringilla.
+<strong class="diff_field_deleted">Nam iaculis lacinia nisl, consectetur viverra enim sollicitudin</strong>
+<strong class="diff_field_added">Nam iaculis lacinia nisl, enim sollicitudin</strong>
+convallis. Morbi ut mauris velit. Proin suscipit dolor id risus
+placerat sodales nec id elit. Morbi vel lacinia lectus, eget laoreet
+<strong class="diff_field_deleted">placerat sodales nec id elit. Morbi vel lacinia lectus, eget laoreet</strong>
+dui. Nunc commodo neque porttitor eros placerat, sed ultricies purus
+<strong class="diff_field_deleted">accumsan. In velit nisi, lorem ipsum lorem gravida a, rutrum in augue.</strong>
+<strong class="diff_field_added">accumsan. In velit nisi, accumsan molestie gravida a, rutrum in augue.</strong>
+Nulla pharetra purus nec dolor ornare, ut aliquam odio placerat.
+Aenean ultrices condimentum quam vitae pharetra.
+</pre>"""
+
+ self.assertEqual(show_diff(self.string1,
+ self.string2,
+ prefix="<pre>", suffix="</pre>",
+ prefix_unchanged='',
+ suffix_unchanged='',
+ prefix_removed='<strong class="diff_field_deleted">',
+ suffix_removed='</strong>',
+ prefix_added='<strong class="diff_field_added">',
+ suffix_added='</strong>'), expected_result)
+
class TestALALC(InvenioTestCase):
"""Test for handling ALA-LC transliteration."""
if UNIDECODE_AVAILABLE:
def test_alalc(self):
msg = "眾鳥高飛盡"
encoded_text, encoding = guess_minimum_encoding(msg)
unicode_text = unicode(encoded_text.decode(encoding))
self.assertEqual("Zhong Niao Gao Fei Jin ",
transliterate_ala_lc(unicode_text))
class LatexEscape(InvenioTestCase):
"""Test for escape latex function"""
def test_escape_latex(self):
unescaped = "this is unescaped latex & % $ # _ { } ~ \ ^ and some multi-byte chars: żółw mémêmëmè"
escaped = escape_latex(unescaped)
self.assertEqual(escaped,
"this is unescaped latex \\& \\% \\$ \\# \\_ \\{ \\} \\~{} \\textbackslash{} \\^{} and some multi-byte chars: \xc5\xbc\xc3\xb3\xc5\x82w m\xc3\xa9m\xc3\xaam\xc3\xabm\xc3\xa8")
TEST_SUITE = make_test_suite(WrapTextInABoxTest, GuessMinimumEncodingTest,
WashForXMLTest, WashForUTF8Test, DecodeToUnicodeTest,
Latex2UnicodeTest, TestStripping,
- TestALALC)
+ TestALALC, TestDiffering)
if __name__ == "__main__":
run_test_suite(TEST_SUITE)
diff --git a/invenio/utils/autodiscovery/__init__.py b/invenio/utils/autodiscovery/__init__.py
index 190bfaff5..9cd5fe668 100644
--- a/invenio/utils/autodiscovery/__init__.py
+++ b/invenio/utils/autodiscovery/__init__.py
@@ -1,125 +1,123 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2013 CERN.
+## Copyright (C) 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
Invenio import helper functions.
Usage example:
autodiscover_modules(['invenio'], '.+_tasks')
An import difference from pluginutils is that modules are imported in their
package hierarchy, contrary to pluginutils where modules are imported as
standalone Python modules.
"""
import imp
import re
from werkzeug import find_modules, import_string
from .checkers import create_enhanced_plugin_builder
_RACE_PROTECTION = False
def autodiscover_modules(packages, related_name_re='.+', ignore_exceptions=False):
"""
Autodiscover function follows the pattern used by Celery.
@param packages: List of package names to auto discover modules in.
@type packages: list of str
@param related_name_re: Regular expression used to match modules names.
@type related_name_re: str
@param ignore_exceptions: Ignore exception when importing modules.
@type ignore_exceptions: bool
"""
global _RACE_PROTECTION
if _RACE_PROTECTION:
return []
_RACE_PROTECTION = True
modules = []
try:
tmp = [find_related_modules(pkg, related_name_re, ignore_exceptions)
for pkg in packages]
for l in tmp:
for m in l:
if m is not None:
modules.append(m)
# Workaround for finally-statement
except:
_RACE_PROTECTION = False
raise
_RACE_PROTECTION = False
return modules
def find_related_modules(package, related_name_re='.+', ignore_exceptions=False):
"""Given a package name and a module name pattern, tries to find matching
modules."""
package_elements = package.rsplit(".", 1)
try:
if len(package_elements) == 2:
pkg = __import__(package_elements[0], globals(), locals(), [package_elements[1]])
pkg = getattr(pkg, package_elements[1])
else:
pkg = __import__(package_elements[0], globals(), locals(), [])
pkg_path = pkg.__path__
except AttributeError:
return []
# Find all modules named according to related_name
p = re.compile(related_name_re)
modules = []
for name in find_modules(package, include_packages=True):
if p.match(name.split('.')[-1]):
try:
modules.append(import_string(name, silent=ignore_exceptions))
except Exception as e:
if not ignore_exceptions:
raise e
return modules
def import_related_module(package, pkg_path, related_name, ignore_exceptions=False):
"""
Import module from given path
"""
try:
imp.find_module(related_name, pkg_path)
except ImportError:
return
try:
return getattr(
__import__('%s' % (package), globals(), locals(), [related_name]),
related_name
)
except Exception as e:
if ignore_exceptions:
#FIXME remove invenio dependency
from invenio.ext.logging import register_exception
register_exception()
else:
raise e
-
-
diff --git a/invenio/utils/container.py b/invenio/utils/container.py
index 7a205abcb..15d28d37f 100644
--- a/invenio/utils/container.py
+++ b/invenio/utils/container.py
@@ -1,98 +1,276 @@
## This file is part of Invenio.
## Copyright (C) 2012 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
This module contains functions for basic containers (dict, list, str)
"""
+import re
+
def get_substructure(data, path):
"""
Tries to retrieve a sub-structure within some data. If the path does not
match any sub-structure, returns None.
>>> data = {'a': 5, 'b': {'c': [1, 2, [{'f': [57]}], 4], 'd': 'test'}}
>>> get_substructure(island, "bc")
[1, 2, [{'f': [57]}], 4]
>>> get_substructure(island, ['b', 'c'])
[1, 2, [{'f': [57]}], 4]
>>> get_substructure(island, ['b', 'c', 2, 0, 'f', 0])
57
>>> get_substructure(island, ['b', 'c', 2, 0, 'f', 'd'])
None
@param data: a container
@type data: str|dict|list|(an indexable container)
@param path: location of the data
@type path: list|str
@rtype: *
"""
if not len(path):
return data
try:
return get_substructure(data[path[0]], path[1:])
except (TypeError, IndexError, KeyError):
return None
try:
from collections import defaultdict
except:
class defaultdict(dict):
"""
Python 2.4 backport of defaultdict
GPL licensed code taken from NLTK - http://code.google.com/p/nltk/
collections.defaultdict
originally contributed by Yoav Goldberg <yoav.goldberg@gmail.com>
new version by Jason Kirtland from Python cookbook.
<http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/523034>
"""
def __init__(self, default_factory=None, *a, **kw):
if (default_factory is not None and
not hasattr(default_factory, '__call__')):
raise TypeError('first argument must be callable')
dict.__init__(self, *a, **kw)
self.default_factory = default_factory
def __getitem__(self, key):
try:
return dict.__getitem__(self, key)
except KeyError:
return self.__missing__(key)
def __missing__(self, key):
if self.default_factory is None:
raise KeyError(key)
self[key] = value = self.default_factory()
return value
def __reduce__(self):
if self.default_factory is None:
args = tuple()
else:
args = self.default_factory,
return type(self), args, None, None, self.items()
def copy(self):
return self.__copy__()
def __copy__(self):
return type(self)(self.default_factory, self)
def __deepcopy__(self, memo):
import copy
return type(self)(self.default_factory,
copy.deepcopy(self.items()))
def __repr__(self):
return 'defaultdict(%s, %s)' % (self.default_factory,
dict.__repr__(self))
+
+
+class SmartDict(object):
+ """
+ This dictionary allows to do some 'smart queries' to its content::
+
+ >>> d = SmartDict()
+
+ >>> d['foo'] = {'a': 'world', 'b':'hello'}
+ >>> d['a'] = [ {'b':1}, {'b':2}, {'b':3} ]
+
+ >>> d['a']
+ [ {'b':1}, {'b':2}, {'b':3} ]
+ >>> d['a[0]']
+ {'b':1}
+ >>> d['a.b']
+ [1,2,3]
+ >>> d['a[1:]']
+ [{'b':2}, {'b':3}]
+ """
+
+ split_key_pattern = re.compile('\.|\[')
+ main_key_pattern = re.compile('\..*|\[.*')
+
+ def __init__(self, d=None):
+ self._dict = d if not d is None else dict()
+
+ def __getitem__(self, key):
+ """
+ As in C{dict.__getitem__} but using 'smart queries'
+ """
+ def getitem(k, v):
+ if isinstance(v, dict):
+ return v[k]
+ elif ']' in k:
+ k = k[:-1].replace('n', '-1')
+ #Work around for list indexes and slices
+ try:
+ return v[int(k)]
+ except ValueError:
+ return v[slice(*map(lambda x: int(x.strip()) if x.strip() else None, k.split(':')))]
+ else:
+ tmp = []
+ for inner_v in v:
+ tmp.append(getitem(k, inner_v))
+ return tmp
+
+
+ #Check if we are using python regular keys
+ try:
+ return self._dict[key]
+ except KeyError:
+ pass
+
+ keys = SmartDict.split_key_pattern.split(key)
+ value = self._dict
+ for k in keys:
+ value = getitem(k, value)
+ return value
+
+ def __setitem__(self, key, value, extend=False):
+ #TODO: Check repeatable fields
+ if '.' not in key and ']' not in key and not extend:
+ self._dict[key] = value
+ else:
+ keys = SmartDict.split_key_pattern.split(key)
+ self.__setitem(self._dict, keys[0], keys[1:], value, extend)
+
+ def __delitem__(self, key):
+ """Note: It only works with first keys"""
+ del self._dict[key]
+
+ def __contains__(self, key):
+
+ if '.' not in key and '[' not in key:
+ return key in self._dict
+ try:
+ self[key]
+ except:
+ return False
+ return True
+
+ def __eq__(self, other):
+ return (isinstance(other, self.__class__)
+ and self._dict == other._dict)
+
+ def __iter__(self):
+ return iter(self._dict)
+
+ def __len__(self):
+ return len(self._dict)
+
+ def keys(self):
+ return self._dict.keys()
+
+ def items(self):
+ return self._dict.items()
+
+ def iteritems(self):
+ return self._dict.iteritems()
+
+ def iterkeys(self):
+ return self._dict.iterkeys()
+
+ def itervalues(self):
+ return self._dict.itervalues()
+
+ def has_key(self, key):
+ return key in self
+
+ def __repr__(self):
+ return repr(self._dict)
+
+ def __setitem(self, chunk, key, keys, value, extend=False):
+ """ Helper function to fill up the dictionary"""
+
+ def setitem(chunk):
+ if keys:
+ return self.__setitem(chunk, keys[0], keys[1:], value, extend)
+ else:
+ return value
+
+ if ']' in key: # list
+ key = int(key[:-1].replace('n', '-1'))
+ if extend:
+ if chunk is None:
+ chunk = [None, ]
+ else:
+ if not isinstance(chunk, list):
+ chunk = [chunk, ]
+ if key != -1:
+ chunk.insert(key, None)
+ else:
+ chunk.append(None)
+ else:
+ if chunk is None:
+ chunk = [None, ]
+ chunk[key] = setitem(chunk[key])
+ else: # dict
+ if extend:
+ if chunk is None:
+ chunk = {}
+ chunk[key] = None
+ chunk[key] = setitem(chunk[key])
+ elif not key in chunk:
+ chunk[key] = None
+ chunk[key] = setitem(chunk[key])
+ else:
+ if keys:
+ chunk[key] = setitem(chunk[key])
+ else:
+ if not isinstance(chunk[key], list):
+ chunk[key] = [chunk[key],]
+ chunk[key].append(None)
+ chunk[key][-1] = setitem(chunk[key][-1])
+ else:
+ if chunk is None:
+ chunk = {}
+ if key not in chunk:
+ chunk[key] = None
+ chunk[key] = setitem(chunk[key])
+
+ return chunk
+
+
+ def get(self, key, default=None):
+ try:
+ return self[key]
+ except KeyError:
+ return default
+
+ def set(self, key, value, extend=False):
+ self.__setitem__(key, value, extend)
+
+ def update(self, E, **F):
+ self._dict.update(E, **F)
diff --git a/invenio/utils/crossref.py b/invenio/utils/crossref.py
index 04855815b..a69e7e899 100644
--- a/invenio/utils/crossref.py
+++ b/invenio/utils/crossref.py
@@ -1,200 +1,241 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2012, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
""" API to fetch metadata in MARCXML format from crossref site using DOI """
-import urllib, urllib2
+import sys
+import urllib
+import urllib2
+import datetime
from xml.dom.minidom import parse
from time import sleep
from invenio.config import CFG_ETCDIR, CFG_CROSSREF_USERNAME, \
CFG_CROSSREF_PASSWORD, CFG_CROSSREF_EMAIL
from invenio.legacy.bibconvert.xslt_engine import convert
from invenio.legacy.bibrecord import record_get_field_value
+from invenio.utils.url import make_invenio_opener
+from invenio.utils.json import json, json_unicode_to_utf8
+
+CROSSREF_OPENER = make_invenio_opener('crossrefutils')
FIELDS_JOURNAL = 'issn,title,author,volume,issue,page,year,type,doi'.split(',')
FIELDS_BOOK = ('isbn,ser_title,vol_title,author,volume,edition_number,'
+ 'page,year,component_number,type,doi').split(',')
# Exceptions classes
class CrossrefError(Exception):
"""Crossref errors"""
def __init__(self, code):
"""Initialisation"""
self.code = code
def __str__(self):
"""Returns error code"""
return repr(self.code)
def get_marcxml_for_doi(doi):
"""
Send doi to the http://www.crossref.org/openurl page.
Attaches parameters: username, password, doi and noredirect.
Returns the MARCXML code or throws an exception, when
1. DOI is malformed
2. Record not found
"""
if not CFG_CROSSREF_USERNAME and not CFG_CROSSREF_PASSWORD:
raise CrossrefError("error_crossref_no_account")
# Clean the DOI
doi = doi.strip()
# Getting the data from external source
url = "http://www.crossref.org/openurl/?pid=" + CFG_CROSSREF_USERNAME \
+ ":" + CFG_CROSSREF_PASSWORD + "&noredirect=tru&id=doi:" + doi
request = urllib2.Request(url)
- response = urllib2.urlopen(request)
+ response = CROSSREF_OPENER.open(request)
header = response.info().getheader('Content-Type')
content = response.read()
# Check if the returned page is html - this means the DOI is malformed
if "text/html" in header:
raise CrossrefError("error_crossref_malformed_doi")
if 'status="unresolved"' in content:
raise CrossrefError("error_crossref_record_not_found")
# Convert xml to marc using convert function
# from bibconvert_xslt_engine file
# Seting the path to xsl template
xsl_crossref2marc_config = templates.get('crossref2marcxml.xsl', '')
output = convert(xmltext=content, \
template_filename=xsl_crossref2marc_config)
return output
def get_doi_for_records(records):
"""
Query crossref to obtain the DOI of a set of records
@params records: List of records
@returns dict {record_id : doi}
"""
from itertools import islice, chain
def batch(iterable, size):
sourceiter = iter(iterable)
while True:
batchiter = islice(sourceiter, size)
yield chain([batchiter.next()], batchiter)
pipes = []
for record in records:
data = [
"", # ISSN
"", # JOURNAL TITLE (773__p)
"", # AUTHOR (Family name of 100__a)
"", # VOLUME (773__v)
"", # ISSUE (773__n)
"", # PAGE (773__c)
"", # YEAR (773__y)
"", # RESOURCE TYPE
"", # KEY
"" # DOI
]
full_author = record_get_field_value(record, "100", "", "", "a").split(",")
if len(full_author) > 0:
data[2] = full_author[0]
data[8] = str(record["001"][0][3])
for subfield, position in ("p", 1), ("v", 3), ("n", 4), ("c", 5), ("y", 6):
for tag, ind1, ind2 in [("773", "", "")]:
val = record_get_field_value(record, tag, ind1, ind2, subfield)
if val:
data[position] = val
break
if not data[1] or not data[3] or not data[5]:
continue # We need journal title, volume and page
pipes.append("|".join(data))
dois = {}
if len(pipes) > 0:
for batchpipes in batch(pipes, 10):
params = {
"usr": CFG_CROSSREF_USERNAME,
"pwd": CFG_CROSSREF_PASSWORD,
"format": "unixref",
"qdata": "\n".join(batchpipes)
}
url = "http://doi.crossref.org/servlet/query"
data = urllib.urlencode(params)
retry_attempt = 0
while retry_attempt < 10:
try:
- document = parse(urllib2.urlopen(url, data))
+ document = parse(CROSSREF_OPENER.open(url, data))
break
except (urllib2.URLError, urllib2.HTTPError):
sleep(5)
retry_attempt += 1
results = document.getElementsByTagName("doi_record")
for result in results:
record_id = result.getAttribute("key")
doi_tags = result.getElementsByTagName("doi")
if len(doi_tags) == 1:
dois[record_id] = doi_tags[0].firstChild.nodeValue
# Avoid sending too many requests
sleep(0.5)
return dois
def get_metadata_for_dois(dois):
"""
Get the metadata associated with
"""
pipes = []
for doi in dois:
pipes.append("|".join([doi, doi]))
metadata = {}
if len(pipes) > 0:
params = {
"usr": CFG_CROSSREF_EMAIL,
"format": "piped",
"qdata": "\n".join(pipes)
}
url = "http://doi.crossref.org/servlet/query"
data = urllib.urlencode(params)
- for line in urllib2.urlopen(url, data):
+ for line in CROSSREF_OPENER.open(url, data):
line = line.split("|")
if len(line) == 1:
pass
elif len(line) in (10, 12):
is_book = len(line) == 12
if is_book:
record_data = dict(zip(FIELDS_BOOK, line))
else:
record_data = dict(zip(FIELDS_JOURNAL, line))
record_data["is_book"] = is_book
metadata[record_data["doi"]] = record_data
else:
raise CrossrefError("Crossref response not understood")
return metadata
+
+def get_all_modified_dois(publisher, from_date=None, re_match=None, debug=False):
+ """
+ Get all the DOIs from a given publisher (e.g. Elsevier has 10.1016
+ that were registered or updated in Crossref since from_date.
+ By default from_date is today - 3 days.
+ re_match is an optional argument where a compiled reguar expression
+ can be passed in order to match only given DOI structures.
+ """
+ if from_date is None:
+ from_date = (datetime.datetime.today() - datetime.timedelta(days=3)).strftime("%Y-%m-%d")
+ res = query_fundref_api("/publishers/%s/works" % publisher, rows=0, filter="from-update-date:%s" % from_date)
+ total_results = res['total-results']
+ if debug:
+ print >> sys.stderr, "total modified DOIs for publisher %s since %s: %s" % (publisher, from_date, total_results)
+ ret = {}
+ for offset in range(0, total_results, 1000):
+ if debug:
+ print >> sys.stderr, "Fetching %s/%s..." % (offset, total_results)
+ res = query_fundref_api("/publishers/%s/works" % publisher, rows=1000, offset=offset, filter="from-update-date:%s" % from_date)
+ if re_match:
+ ret.update([(item['DOI'], item) for item in res['items'] if re_match.match(item['DOI'])])
+ else:
+ ret.update([(item['DOI'], item) for item in res['items']])
+ return ret
+
+def query_fundref_api(query, **kwargs):
+ """
+ Perform a request to the Fundref API.
+ """
+ sleep(1)
+ req = urllib2.Request("http://api.crossref.org%s?%s" % (query, urllib.urlencode(kwargs)), headers={'content-type': 'application/vnd.crossref-api-message+json; version=1.0'})
+ res = json_unicode_to_utf8(json.load(CROSSREF_OPENER.open(req)))
+ return res['message']
diff --git a/invenio/utils/html.py b/invenio/utils/html.py
index 9fbcef764..e78ed37c9 100644
--- a/invenio/utils/html.py
+++ b/invenio/utils/html.py
@@ -1,926 +1,933 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
-## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2013 CERN.
+## Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2013, 2014 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""HTML utilities."""
from __future__ import absolute_import
from HTMLParser import HTMLParser
from six import iteritems
from werkzeug.local import LocalProxy
from invenio.base.globals import cfg
from invenio.utils.text import indent_text, encode_for_xml
default_ln = lambda ln: cfg['CFG_SITE_LANG'] if ln is None else ln
import re
import cgi
import os
import json
try:
from BeautifulSoup import BeautifulSoup
CFG_BEAUTIFULSOUP_INSTALLED = True
except ImportError:
CFG_BEAUTIFULSOUP_INSTALLED = False
try:
import tidy
CFG_TIDY_INSTALLED = True
except ImportError:
CFG_TIDY_INSTALLED = False
# List of allowed tags (tags that won't create any XSS risk)
CFG_HTML_BUFFER_ALLOWED_TAG_WHITELIST = ('a',
'p', 'br', 'blockquote',
'strong', 'b', 'u', 'i', 'em',
'ul', 'ol', 'li', 'sub', 'sup', 'div', 'strike')
# List of allowed attributes. Be cautious, some attributes may be risky:
# <p style="background: url(myxss_suite.js)">
CFG_HTML_BUFFER_ALLOWED_ATTRIBUTE_WHITELIST = ('href', 'name', 'class')
## precompile some often-used regexp for speed reasons:
RE_HTML = re.compile("(?s)<[^>]*>|&#?\w+;")
RE_HTML_WITHOUT_ESCAPED_CHARS = re.compile("(?s)<[^>]*>")
# url validation regex
regex_url = re.compile(r'^(?:http|ftp)s?://' # http:// or https://
r'(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}\.?)|' #domain...
r'localhost|' #localhost...
r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})' # ...or ip
r'(?::\d+)?' # optional port
r'(?:/?|[/?]\S+)$', re.IGNORECASE)
def nmtoken_from_string(text):
"""
Returns a Nmtoken from a string.
It is useful to produce XHTML valid values for the 'name'
attribute of an anchor.
CAUTION: the function is surjective: 2 different texts might lead to
the same result. This is improbable on a single page.
Nmtoken is the type that is a mixture of characters supported in
attributes such as 'name' in HTML 'a' tag. For example,
<a name="Articles%20%26%20Preprints"> should be tranformed to
<a name="Articles372037263720Preprints"> using this function.
http://www.w3.org/TR/2000/REC-xml-20001006#NT-Nmtoken
Also note that this function filters more characters than
specified by the definition of Nmtoken ('CombiningChar' and
'Extender' charsets are filtered out).
"""
text = text.replace('-', '--')
return ''.join( [( ((not char.isalnum() and not char in ['.', '-', '_', ':']) and str(ord(char))) or char)
for char in text] )
def escape_html(text, escape_quotes=False):
"""Escape all HTML tags, avoiding XSS attacks.
< => &lt;
> => &gt;
& => &amp:
@param text: text to be escaped from HTML tags
@param escape_quotes: if True, escape any quote mark to its HTML entity:
" => &quot;
' => &#39;
"""
text = text.replace('&', '&amp;')
text = text.replace('<', '&lt;')
text = text.replace('>', '&gt;')
if escape_quotes:
text = text.replace('"', '&quot;')
text = text.replace("'", '&#39;')
return text
CFG_JS_CHARS_MAPPINGS = {
'\\': '\\\\',
"'": "\\'",
'"': '\\"',
'\b': '\\b',
'\f': '\\f',
'\n': '\\n',
'\r': '\\r',
'\t': '\\t',
'\v': '\\v',
}
for i in range(0x20):
CFG_JS_CHARS_MAPPINGS.setdefault(chr(i), '\\u%04x' % (i,))
for i in (0x2028, 0x2029):
CFG_JS_CHARS_MAPPINGS.setdefault(unichr(i), '\\u%04x' % (i,))
RE_ESCAPE_JS_CHARS = re.compile(u'''[\\x00-\\x1f\\\\"\\\\'\\b\\f\\n\\r\\t\\v\u2028\u2029]''')
RE_CLOSING_SCRIPT_TAG = re.compile('</script>', re.IGNORECASE)
def escape_javascript_string(text, escape_for_html=True, escape_quote_for_html=False, escape_CDATA=True, escape_script_tag_with_quote='"'):
"""
Escape text in order to be used as Javascript string in various
context.
Examples::
>>> text = '''"Are you a Munchkin?" asked Dorothy.
"No, but I am their friend"'''
>>> escape_javascript_string(text)
>>> \\"&quot;Are you a Munchkin?\\" asked Dorothy.\\n\\"No, but I am their friend\\"'
The returned string can be enclosed either in single or double
quotes delimiters.
THE FUNCTION ASSUME THAT YOU HAVE ALREDADY WASHED THE STRING FROM
UNSAFE CONTENT, according to the context you plan to use the
string. The function will just make sure that the string will not
break you Javascript/HTML code/markup.
If you plan to include the string inside the body of an HTML page,
you will probably want to set C{escape_for_html} to True, in order
to produce XHTML-valid pages when the input string contain
characters such as < , > and &.
Furthermore if you plan to include the string as part of a tag
attribute (for eg. <a href="#" onclick="foo&quot;bar"), you might
want to set C{escape_quote_for_html} to True.
If you plan to include the string inside the body of an HTML page,
enclosed by CDATA delimiters, then you would *not* need to escape
HTML tags. Using CDATA delimeters enables to include Javascript
strings meant to refer to HTML tags (eg. in case you would like to
manipulate the DOM tree to add new nodes to the page), which would
not be possible when escaping the HTML. For eg.:
/*<![CDATA[*/
document.getElementById('foo').innerHTML = '<p>bar</p>'
/*]]>*/
In this case you will probably want to set C{escape_CDATA} to True
in order to produce an XHTML-valid document, in case a closing
CDATA delimeter is in your input string. Parameter C{escape_CDATA}
is not considered when C{escape_for_html} is set to True.
Note that CDATA delimiters might be automatically added by the
browser, based on the content-type used to serve the page.
When C{escape_for_html} is set to False, whatever option is chosen
for C{escape_CDATA}, the string must not contain a '</script>' tag
(apparently...). The only option to keep this '</script>' tag (if
you need it) is to split it, which requires to know which quote
delimiter your plan to use. For eg:
Examples::
>>> text = '''foo</script>bar'''
>>> val = escape_javascript_string(text, escape_for_html=False, escape_script_tag_with_quote='"')
>>> 'foo</scr"+"ipt>bar'
>>> mycode = '''alert("%s")''' % val
C{escape_script_tag_with_quote} is not considered when
C{escape_for_html} is set to True.
If you are planning to return the string as part of a pure
Javascript document, then you should in principle set both
C{escape_for_html} and C{escape_CDATA} to False, and
C{escape_script_tag_with_quote} to None.
@param text: string to be escaped
@param escape_for_html: if True, also escape input for HTML
@param escape_CDATA: if True, escape closing CDATA tags (when C{escape_for_html} is False)
@escape_script_tag_with_quote: which quote will be used to delimit your string, in case you must wash, but keep, C{</script>} tag (when C{escape_for_html} is False)
"""
if escape_quote_for_html:
text = text.replace('"', '&quot;')
if escape_for_html:
text = cgi.escape(text)
elif escape_CDATA:
text = text.replace(']]>', ']]]]><![CDATA[>')
text = json.dumps(text)[1:-1].replace("'", "\\'")
if not escape_for_html and escape_script_tag_with_quote:
text = RE_CLOSING_SCRIPT_TAG.sub('''</scr%(q)s+%(q)sipt>''' % {'q': escape_script_tag_with_quote}, text)
return text
class HTMLWasher(HTMLParser):
"""
Creates a washer for HTML, avoiding XSS attacks. See wash function for
details on parameters.
Usage::
from invenio.utils.html import HTMLWasher
washer = HTMLWasher()
escaped_text = washer.wash(unescaped_text)
Examples::
a.wash('Spam and <b><blink>eggs</blink></b>')
=> 'Spam and <b>eggs</b>'
a.wash('Spam and <b><blink>eggs</blink></b>', True)
=> 'Spam and <b>&lt;blink&gt;eggs&lt;/blink&gt;</b>'
a.wash('Spam and <b><a href="python.org">eggs</u></b>')
=> 'Spam and <b><a href="python.org">eggs</a></b>'
a.wash('Spam and <b><a href="javascript:xss();">eggs</a></b>')
=>'Spam and <b><a href="">eggs</a></b>'
a.wash('Spam and <b><a href="jaVas cRipt:xss();">poilu</a></b>')
=>'Spam and <b><a href="">eggs</a></b>'
"""
silent = False
def __init__(self):
""" Constructor; initializes washer """
HTMLParser.__init__(self)
self.result = ''
self.nb = 0
self.previous_nbs = []
self.previous_type_lists = []
self.url = ''
self.render_unallowed_tags = False
self.allowed_tag_whitelist = \
CFG_HTML_BUFFER_ALLOWED_TAG_WHITELIST
self.allowed_attribute_whitelist = \
CFG_HTML_BUFFER_ALLOWED_ATTRIBUTE_WHITELIST
# javascript:
self.re_js = re.compile( ".*(j|&#106;|&#74;)"\
"\s*(a|&#97;|&#65;)"\
"\s*(v|&#118;|&#86;)"\
"\s*(a|&#97;|&#65;)"\
"\s*(s|&#115;|&#83;)"\
"\s*(c|&#99;|&#67;)"\
"\s*(r|&#114;|&#82;)"\
"\s*(i|&#195;|&#73;)"\
"\s*(p|&#112;|&#80;)"\
"\s*(t|&#112;|&#84)"\
"\s*(:|&#58;).*", re.IGNORECASE | re.DOTALL)
# vbscript:
self.re_vb = re.compile( ".*(v|&#118;|&#86;)"\
"\s*(b|&#98;|&#66;)"\
"\s*(s|&#115;|&#83;)"\
"\s*(c|&#99;|&#67;)"\
"\s*(r|&#114;|&#82;)"\
"\s*(i|&#195;|&#73;)"\
"\s*(p|&#112;|&#80;)"\
"\s*(t|&#112;|&#84;)"\
"\s*(:|&#58;).*", re.IGNORECASE | re.DOTALL)
def wash(self, html_buffer,
render_unallowed_tags=False,
allowed_tag_whitelist=CFG_HTML_BUFFER_ALLOWED_TAG_WHITELIST,
automatic_link_transformation=False,
allowed_attribute_whitelist=\
CFG_HTML_BUFFER_ALLOWED_ATTRIBUTE_WHITELIST):
"""
Wash HTML buffer, escaping XSS attacks.
@param html_buffer: text to escape
@param render_unallowed_tags: if True, print unallowed tags escaping
< and >. Else, only print content of unallowed tags.
@param allowed_tag_whitelist: list of allowed tags
@param allowed_attribute_whitelist: list of allowed attributes
"""
self.reset()
self.result = ''
self.nb = 0
self.previous_nbs = []
self.previous_type_lists = []
self.url = ''
self.render_unallowed_tags = render_unallowed_tags
self.automatic_link_transformation = automatic_link_transformation
self.allowed_tag_whitelist = allowed_tag_whitelist
self.allowed_attribute_whitelist = allowed_attribute_whitelist
self.feed(html_buffer)
self.close()
return self.result
def handle_starttag(self, tag, attrs):
"""Function called for new opening tags"""
if tag.lower() in self.allowed_tag_whitelist:
self.result += '<' + tag
for (attr, value) in attrs:
if attr.lower() in self.allowed_attribute_whitelist:
self.result += ' %s="%s"' % \
(attr, self.handle_attribute_value(value))
self.result += '>'
else:
if self.render_unallowed_tags:
self.result += '&lt;' + cgi.escape(tag)
for (attr, value) in attrs:
self.result += ' %s="%s"' % \
(attr, cgi.escape(value, True))
self.result += '&gt;'
elif tag == 'style' or tag == 'script':
# In that case we want to remove content too
self.silent = True
def handle_data(self, data):
"""Function called for text nodes"""
if not self.silent:
possible_urls = re.findall(r'(https?://[\w\d:#%/;$()~_?\-=\\\.&]*)', data)
# validate possible urls
# we'll transform them just in case
# they are valid.
if possible_urls and self.automatic_link_transformation:
for url in possible_urls:
if regex_url.search(url):
transformed_url = '<a href="%s">%s</a>' % (url, url)
data = data.replace(url, transformed_url)
self.result += data
else:
self.result += cgi.escape(data, True)
def handle_endtag(self, tag):
"""Function called for ending of tags"""
if tag.lower() in self.allowed_tag_whitelist:
self.result += '</' + tag + '>'
else:
if self.render_unallowed_tags:
self.result += '&lt;/' + cgi.escape(tag) + '&gt;'
if tag == 'style' or tag == 'script':
self.silent = False
def handle_startendtag(self, tag, attrs):
"""Function called for empty tags (e.g. <br />)"""
if tag.lower() in self.allowed_tag_whitelist:
self.result += '<' + tag
for (attr, value) in attrs:
if attr.lower() in self.allowed_attribute_whitelist:
self.result += ' %s="%s"' % \
(attr, self.handle_attribute_value(value))
self.result += ' />'
else:
if self.render_unallowed_tags:
self.result += '&lt;' + cgi.escape(tag)
for (attr, value) in attrs:
self.result += ' %s="%s"' % \
(attr, cgi.escape(value, True))
self.result += ' /&gt;'
def handle_attribute_value(self, value):
"""Check attribute. Especially designed for avoiding URLs in the form:
javascript:myXSSFunction();"""
if self.re_js.match(value) or self.re_vb.match(value):
return ''
return value
def handle_charref(self, name):
"""Process character references of the form "&#ref;". Return it as it is."""
self.result += '&#' + name + ';'
def handle_entityref(self, name):
"""Process a general entity reference of the form "&name;".
Return it as it is."""
self.result += '&' + name + ';'
def tidy_html(html_buffer, cleaning_lib='utidylib'):
"""
Tidy up the input HTML using one of the installed cleaning
libraries.
@param html_buffer: the input HTML to clean up
@type html_buffer: string
@param cleaning_lib: chose the preferred library to clean the HTML. One of:
- utidylib
- beautifulsoup
@return: a cleaned version of the input HTML
@note: requires uTidylib or BeautifulSoup to be installed. If the chosen library is missing, the input X{html_buffer} is returned I{as is}.
"""
if CFG_TIDY_INSTALLED and cleaning_lib == 'utidylib':
options = dict(output_xhtml=1,
show_body_only=1,
merge_divs=0,
wrap=0)
try:
output = str(tidy.parseString(html_buffer, **options))
except:
output = html_buffer
elif CFG_BEAUTIFULSOUP_INSTALLED and cleaning_lib == 'beautifulsoup':
try:
output = str(BeautifulSoup(html_buffer).prettify())
except:
output = html_buffer
else:
output = html_buffer
return output
def get_mathjax_header(https=False):
"""
Return the snippet of HTML code to put in HTML HEAD tag, in order to
enable MathJax support.
@param https: when using the CDN, whether to use the HTTPS URL rather
than the HTTP one.
@type https: bool
@note: with new releases of MathJax, update this function toghether with
$MJV variable in the root Makefile.am
"""
if cfg['CFG_MATHJAX_HOSTING'].lower() == 'cdn':
if https:
mathjax_path = "https://d3eoax9i5htok0.cloudfront.net/mathjax/2.1-latest"
else:
mathjax_path = "http://cdn.mathjax.org/mathjax/2.1-latest"
else:
mathjax_path = "/MathJax"
+
+ if cfg['CFG_MATHJAX_RENDERS_MATHML']:
+ mathjax_config = "TeX-AMS-MML_HTMLorMML"
+ else:
+ mathjax_config = "TeX-AMS_HTML"
+
return """<script type="text/x-mathjax-config">
MathJax.Hub.Config({
tex2jax: {inlineMath: [['$','$']],
processEscapes: true},
showProcessingMessages: false,
messageStyle: "none"
});
</script>
-<script src="%(mathjax_path)s/MathJax.js?config=TeX-AMS_HTML" type="text/javascript">
+<script src="%(mathjax_path)s/MathJax.js?config=%(mathjax_config)s" type="text/javascript">
</script>""" % {
- 'mathjax_path': mathjax_path
+ 'mathjax_path': mathjax_path,
+ 'mathjax_config': mathjax_config,
}
def is_html_text_editor_installed():
"""
Returns True if the wysiwyg editor (CKeditor) is installed
"""
return os.path.exists(os.path.join(cfg['CFG_WEBDIR'], 'ckeditor', 'ckeditor.js'))
ckeditor_available = LocalProxy(is_html_text_editor_installed)
def get_html_text_editor(name, id=None, content='', textual_content=None, width='300px', height='200px',
enabled=True, file_upload_url=None, toolbar_set="Basic",
custom_configurations_path='/js/ckeditor/invenio-ckeditor-config.js',
ln=None):
"""
Returns a wysiwyg editor (CKEditor) to embed in html pages.
Fall back to a simple textarea when the library is not installed,
or when the user's browser is not compatible with the editor, or
when 'enable' is False, or when javascript is not enabled.
NOTE that the output also contains a hidden field named
'editor_type' that contains the kind of editor used, 'textarea' or
'ckeditor'.
Based on 'editor_type' you might want to take different actions,
like replace CRLF with <br/> when editor_type equals to
'textarea', but not when editor_type equals to 'ckeditor'.
@param name: *str* the name attribute of the returned editor
@param id: *str* the id attribute of the returned editor (when
applicable)
@param content: *str* the default content of the editor.
@param textual_content: *str* a content formatted for the case where the
wysiwyg editor is not available for user. When not
specified, use value of 'content'
@param width: *str* width of the editor in an html compatible unit:
Eg: '400px', '50%'.
@param height: *str* height of the editor in an html compatible unit:
Eg: '400px', '50%'.
@param enabled: *bool* if the wysiwyg editor is return (True) or if a
simple texteara is returned (False)
@param file_upload_url: *str* the URL used to upload new files via the
editor upload panel. You have to implement the
handler for your own use. The URL handler will get
form variables 'File' as POST for the uploaded file,
and 'Type' as GET for the type of file ('file',
'image', 'flash', 'media')
When value is not given, the file upload is disabled.
@param toolbar_set: *str* the name of the toolbar layout to
use. CKeditor comes by default with 'Basic' and
'Default'. To define other sets, customize the
config file in
/opt/cds-invenio/var/www/ckeditor/invenio-ckconfig.js
@param custom_configurations_path: *str* value for the CKeditor config
variable 'CustomConfigurationsPath',
which allows to specify the path of a
file that contains a custom configuration
for the editor. The path is relative to
/opt/invenio/var/www/
@return: the HTML markup of the editor
"""
ln = default_ln(ln)
if textual_content is None:
textual_content = content
editor = ''
if enabled and ckeditor_available:
# Prepare upload path settings
file_upload_script = ''
if file_upload_url is not None:
file_upload_script = ''',
filebrowserLinkUploadUrl: '%(file_upload_url)s',
filebrowserImageUploadUrl: '%(file_upload_url)s?type=Image',
filebrowserFlashUploadUrl: '%(file_upload_url)s?type=Flash'
''' % {'file_upload_url': file_upload_url}
# Prepare code to instantiate an editor
editor += '''
<script type="text/javascript" language="javascript">//<![CDATA[
/* Load the script only once, or else multiple instance of the editor on the same page will not work */
var INVENIO_CKEDITOR_ALREADY_LOADED
if (INVENIO_CKEDITOR_ALREADY_LOADED != 1) {
document.write('<script type="text/javascript" src="%(CFG_SITE_URL)s/ckeditor/ckeditor.js"><\/script>');
INVENIO_CKEDITOR_ALREADY_LOADED = 1;
}
//]]></script>
<input type="hidden" name="editor_type" id="%(id)seditortype" value="textarea" />
<textarea rows="100" cols="80" id="%(id)s" name="%(name)s" style="width:%(width)s;height:%(height)s">%(textual_content)s</textarea>
<textarea rows="100" cols="80" id="%(id)shtmlvalue" name="%(name)shtmlvalue" style="display:none;width:%(width)s;height:%(height)s">%(html_content)s</textarea>
<script type="text/javascript">//<![CDATA[
var CKEDITOR_BASEPATH = '/ckeditor/';
CKEDITOR.replace( '%(name)s',
{customConfig: '%(custom_configurations_path)s',
toolbar: '%(toolbar)s',
width: '%(width)s',
height:'%(height)s',
language: '%(ln)s'
%(file_upload_script)s
});
CKEDITOR.on('instanceReady',
function( evt )
{
/* If CKeditor was correctly loaded, display the nice HTML representation */
var oEditor = evt.editor;
editor_id = oEditor.id
editor_name = oEditor.name
var html_editor = document.getElementById(editor_name + 'htmlvalue');
oEditor.setData(html_editor.value);
var editor_type_field = document.getElementById(editor_name + 'editortype');
editor_type_field.value = 'ckeditor';
var writer = oEditor.dataProcessor.writer;
writer.indentationChars = ''; /*Do not indent source code with tabs*/
oEditor.resetDirty();
/* Workaround: http://dev.ckeditor.com/ticket/3674 */
evt.editor.on( 'contentDom', function( ev )
{
ev.removeListener();
evt.editor.resetDirty();
} );
/* End workaround */
})
//]]></script>
''' % \
{'textual_content': cgi.escape(textual_content),
'html_content': content,
'width': width,
'height': height,
'name': name,
'id': id or name,
'custom_configurations_path': custom_configurations_path,
'toolbar': toolbar_set,
'file_upload_script': file_upload_script,
'CFG_SITE_URL': cfg['CFG_SITE_URL'],
'ln': ln}
else:
# CKedior is not installed
textarea = '<textarea rows="100" cols="80" %(id)s name="%(name)s" style="width:%(width)s;height:%(height)s">%(content)s</textarea>' \
% {'content': cgi.escape(textual_content),
'width': width,
'height': height,
'name': name,
'id': id and ('id="%s"' % id) or ''}
editor += textarea
editor += '<input type="hidden" name="editor_type" value="textarea" />'
return editor
def remove_html_markup(text, replacechar=' ', remove_escaped_chars_p=True):
"""
Remove HTML markup from text.
@param text: Input text.
@type text: string.
@param replacechar: By which character should we replace HTML markup.
Usually, a single space or an empty string are nice values.
@type replacechar: string
@param remove_escaped_chars_p: If True, also remove escaped characters
like '&amp;', '&lt;', '&gt;' and '&quot;'.
@type remove_escaped_chars_p: boolean
@return: Input text with HTML markup removed.
@rtype: string
"""
if not remove_escaped_chars_p:
return RE_HTML_WITHOUT_ESCAPED_CHARS.sub(replacechar, text)
return RE_HTML.sub(replacechar, text)
def unescape(s, quote=False):
"""
The opposite of the cgi.escape function.
Replace escaped characters '&amp;', '&lt;' and '&gt;' with the corresponding
regular characters. If the optional flag quote is true, the escaped quotation
mark character ('&quot;') is also translated.
"""
s = s.replace('&lt;', '<')
s = s.replace('&gt;', '>')
if quote:
s = s.replace('&quot;', '"')
s = s.replace('&amp;', '&')
return s
class EscapedString(str):
"""
This class is a stub used by the MLClass machinery in order
to distinguish native string, from string that don't need to be
escaped.
"""
pass
class EscapedHTMLString(EscapedString):
"""
This class automatically escape a non-escaped string used to initialize
it, using the HTML escaping method (i.e. cgi.escape).
"""
def __new__(cls, original_string='', escape_quotes=False):
if isinstance(original_string, EscapedString):
escaped_string = str(original_string)
else:
if original_string and not str(original_string).strip():
escaped_string = '&nbsp;'
else:
escaped_string = cgi.escape(str(original_string), escape_quotes)
obj = str.__new__(cls, escaped_string)
obj.original_string = original_string
obj.escape_quotes = escape_quotes
return obj
def __repr__(self):
return 'EscapedHTMLString(%s, %s)' % (repr(self.original_string), repr(self.escape_quotes))
def __add__(self, rhs):
return EscapedHTMLString(EscapedString(str(self) + str(rhs)))
class EscapedXMLString(EscapedString):
"""
This class automatically escape a non-escaped string used to initialize
it, using the XML escaping method (i.e. encode_for_xml).
"""
def __new__(cls, original_string='', escape_quotes=False):
if isinstance(original_string, EscapedString):
escaped_string = str(original_string)
else:
if original_string and not str(original_string).strip():
escaped_string = '&nbsp;'
else:
escaped_string = encode_for_xml(str(original_string), wash=True, quote=escape_quotes)
obj = str.__new__(cls, escaped_string)
obj.original_string = original_string
obj.escape_quotes = escape_quotes
return obj
def __repr__(self):
return 'EscapedXMLString(%s, %s)' % (repr(self.original_string), repr(self.escape_quotes))
def __add__(self, rhs):
return EscapedXMLString(EscapedString(str(self) + str(rhs)))
def create_tag(tag, escaper=EscapedHTMLString, opening_only=False, body=None, escape_body=False, escape_attr=True, indent=0, attrs=None, **other_attrs):
"""
Create an XML/HTML tag.
This function create a full XML/HTML tag, putting toghether an
optional inner body and a dictionary of attributes.
>>> print create_html_tag ("select", create_html_tag("h1",
... "hello", other_attrs={'class': "foo"}))
<select>
<h1 class="foo">
hello
</h1>
</select>
@param tag: the tag (e.g. "select", "body", "h1"...).
@type tag: string
@param body: some text/HTML to put in the body of the tag (this
body will be indented WRT the tag).
@type body: string
@param escape_body: wether the body (if any) must be escaped.
@type escape_body: boolean
@param escape_attr: wether the attribute values (if any) must be
escaped.
@type escape_attr: boolean
@param indent: number of level of indentation for the tag.
@type indent: integer
@param attrs: map of attributes to add to the tag.
@type attrs: dict
@return: the HTML tag.
@rtype: string
"""
if attrs is None:
attrs = {}
for key, value in iteritems(other_attrs):
if value is not None:
if key.endswith('_'):
attrs[key[:-1]] = value
else:
attrs[key] = value
out = "<%s" % tag
for key, value in iteritems(attrs):
if escape_attr:
value = escaper(value, escape_quotes=True)
out += ' %s="%s"' % (key, value)
if body is not None:
if callable(body) and body.__name__ == 'handle_body':
body = body()
out += ">"
if escape_body and not isinstance(body, EscapedString):
body = escaper(body)
out += body
if not opening_only:
out += "</%s>" % tag
elif not opening_only:
out += " />"
if indent:
out = indent_text(out, indent)[:-1]
from invenio.utils.text import wash_for_utf8
return EscapedString(wash_for_utf8(out))
class MLClass(object):
"""
Swiss army knife to generate XML or HTML strings a la carte.
>>> from invenio.utils.html import X, H
>>> X.foo()()
... '<foo />'
>>> X.foo(bar='baz')()
... '<foo bar="baz" />'
>>> X.foo(bar='baz&pi')()
... '<foo bar="baz&amp;pi" />'
>>> X.foo("<body />", bar='baz')
... '<foo bar="baz"><body /></foo>'
>>> X.foo(bar='baz')(X.body())
... '<foo bar="baz"><body /></foo>'
>>> X.foo(bar='baz')("<body />") ## automatic escaping
... '<foo bar="baz">&lt;body /></foo>'
>>> X.foo()(X.p(), X.p()) ## magic concatenation
... '<foo><p /><p /></foo>'
>>> X.foo(class_='bar')() ## protected keywords...
... '<foo class="bar" />'
>>> X["xml-bar"]()()
... '<xml-bar />'
"""
def __init__(self, escaper):
self.escaper = escaper
def __getattr__(self, tag):
def tag_creator(body=None, opening_only=False, escape_body=False, escape_attr=True, indent=0, attrs=None, **other_attrs):
if body:
return create_tag(tag, body=body, opening_only=opening_only, escape_body=escape_body, escape_attr=escape_attr, indent=indent, attrs=attrs, **other_attrs)
else:
def handle_body(*other_bodies):
full_body = None
if other_bodies:
full_body = ""
for body in other_bodies:
if callable(body) and body.__name__ == 'handle_body':
full_body += body()
elif isinstance(body, EscapedString):
full_body += body
else:
full_body += self.escaper(str(body))
return create_tag(tag, body=full_body, opening_only=opening_only, escape_body=escape_body, escape_attr=escape_attr, indent=indent, attrs=attrs, **other_attrs)
return handle_body
return tag_creator
__getitem__ = __getattr__
H = MLClass(EscapedHTMLString)
X = MLClass(EscapedXMLString)
def create_html_select(options, name=None, selected=None, disabled=None, multiple=False, attrs=None, **other_attrs):
"""
Create an HTML select box.
>>> print create_html_select(["foo", "bar"], selected="bar", name="baz")
<select name="baz">
<option selected="selected" value="bar">
bar
</option>
<option value="foo">
foo
</option>
</select>
>>> print create_html_select([("foo", "oof"), ("bar", "rab")], selected="bar", name="baz")
<select name="baz">
<option value="foo">
oof
</option>
<option selected="selected" value="bar">
rab
</option>
</select>
@param options: this can either be a sequence of strings, or a sequence
of couples or a map of C{key->value}. In the former case, the C{select}
tag will contain a list of C{option} tags (in alphabetical order),
where the C{value} attribute is not specified. In the latter case,
the C{value} attribute will be set to the C{key}, while the body
of the C{option} will be set to C{value}.
@type options: sequence or map
@param name: the name of the form element.
@type name: string
@param selected: optional key(s)/value(s) to select by default. In case
a map has been used for options.
@type selected: string (or list of string)
@param disabled: optional key(s)/value(s) to disable.
@type disabled: string (or list of string)
@param multiple: whether a multiple select box must be created.
@type mutable: bool
@param attrs: optional attributes to create the select tag.
@type attrs: dict
@param other_attrs: other optional attributes.
@return: the HTML output.
@rtype: string
@note: the values and keys will be escaped for HTML.
@note: it is important that parameter C{value} is always
specified, in case some browser plugin play with the
markup, for eg. when translating the page.
"""
body = []
if selected is None:
selected = []
elif isinstance(selected, (str, unicode)):
selected = [selected]
if disabled is None:
disabled = []
elif isinstance(disabled, (str, unicode)):
disabled = [disabled]
if name is not None and multiple and not name.endswith('[]'):
name += "[]"
if isinstance(options, dict):
items = options.items()
items.sort(lambda item1, item2: cmp(item1[1], item2[1]))
elif isinstance(options, (list, tuple)):
options = list(options)
items = []
for item in options:
if isinstance(item, (str, unicode)):
items.append((item, item))
elif isinstance(item, (tuple, list)) and len(item) == 2:
items.append(tuple(item))
else:
raise ValueError('Item "%s" of incompatible type: %s' % (item, type(item)))
else:
raise ValueError('Options of incompatible type: %s' % type(options))
for key, value in items:
option_attrs = {}
if key in selected:
option_attrs['selected'] = 'selected'
if key in disabled:
option_attrs['disabled'] = 'disabled'
body.append(create_tag("option", body=value, escape_body=True, value=key, attrs=option_attrs))
if attrs is None:
attrs = {}
if name is not None:
attrs['name'] = name
if multiple:
attrs['multiple'] = 'multiple'
return create_tag("select", body='\n'.join(body), attrs=attrs, **other_attrs)
class _LinkGetter(HTMLParser):
"""
Hidden class that, by deriving from HTMLParser, will intercept all
<a> tags and retrieve the corresponding href attribute.
All URLs are available in the urls attribute of the class.
"""
def __init__(self):
HTMLParser.__init__(self)
self.urls = set()
def handle_starttag(self, tag, attrs):
if tag == 'a':
for (name, value) in attrs:
if name == 'href':
self.urls.add(value)
def get_links_in_html_page(html):
"""
@param html: the HTML text to parse
@type html: str
@return: the list of URLs that were referenced via <a> tags.
@rtype: set of str
"""
parser = _LinkGetter()
parser.feed(html)
return parser.urls
diff --git a/invenio/utils/redis.py b/invenio/utils/redis.py
new file mode 100644
index 000000000..98a41e4a8
--- /dev/null
+++ b/invenio/utils/redis.py
@@ -0,0 +1,48 @@
+from invenio.config import CFG_REDIS_HOSTS
+if CFG_REDIS_HOSTS:
+ from nydus.db import create_cluster
+
+
+from invenio.config import CFG_REDIS_HOSTS
+
+_REDIS_CONN = {}
+
+
+class DummyRedisClient(object):
+ def get(self, key):
+ pass
+
+ def set(self, key, value, timeout=None):
+ pass
+
+ def delete(self, key):
+ pass
+
+
+def get_redis(redis_namespace='default'):
+ """Connects to a redis using nydus
+
+ We simlulate a redis cluster by connecting to several redis servers
+ in the background and using a consistent hashing ring to choose which
+ server stores the data.
+ Returns a redis object that can be used like a regular redis object
+ see http://redis.io/
+ """
+ if not CFG_REDIS_HOSTS or not CFG_REDIS_HOSTS[redis_namespace]:
+ return DummyRedisClient()
+
+ redis = _REDIS_CONN.get(redis_namespace, None)
+ if redis:
+ return redis
+
+ hosts_dict = {}
+ for server_num, server_info in enumerate(CFG_REDIS_HOSTS[redis_namespace]):
+ hosts_dict[server_num] = server_info
+
+ redis = create_cluster({
+ 'backend': 'nydus.db.backends.redis.Redis',
+ 'router': 'nydus.db.routers.keyvalue.ConsistentHashingRouter',
+ 'hosts': hosts_dict
+ })
+ _REDIS_CONN[redis_namespace] = redis
+ return redis
diff --git a/invenio/utils/text.py b/invenio/utils/text.py
index ae9aa516a..4b10a8a6f 100644
--- a/invenio/utils/text.py
+++ b/invenio/utils/text.py
@@ -1,835 +1,850 @@
# -*- coding: utf-8 -*-
from __future__ import print_function
## This file is part of Invenio.
## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2013 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""
Functions useful for text wrapping (in a box) and indenting.
"""
__revision__ = "$Id$"
import six
import sys
import re
import textwrap
import pkg_resources
from six.moves import html_entities
try:
import chardet
CHARDET_AVAILABLE = True
except ImportError:
CHARDET_AVAILABLE = False
try:
from unidecode import unidecode
UNIDECODE_AVAILABLE = True
except ImportError:
UNIDECODE_AVAILABLE = False
CFG_LATEX_UNICODE_TRANSLATION_CONST = {}
CFG_WRAP_TEXT_IN_A_BOX_STYLES = {
'__DEFAULT' : {
'horiz_sep' : '*',
'max_col' : 72,
'min_col' : 40,
'tab_str' : ' ',
'tab_num' : 0,
'border' : ('**', '*', '**', '** ', ' **', '**', '*', '**'),
'prefix' : '\n',
'suffix' : '\n',
'break_long' : False,
'force_horiz' : False,
},
'squared' : {
'horiz_sep' : '-',
'border' : ('+', '-', '+', '| ', ' |', '+', '-', '+')
},
'double_sharp' : {
'horiz_sep' : '#',
'border' : ('##', '#', '##', '## ', ' ##', '##', '#', '##')
},
'single_sharp' : {
'horiz_sep' : '#',
'border' : ('#', '#', '#', '# ', ' #', '#', '#', '#')
},
'single_star' : {
'border' : ('*', '*', '*', '* ', ' *', '*', '*', '*',)
},
'double_star' : {
},
'no_border' : {
'horiz_sep' : '',
'border' : ('', '', '', '', '', '', '', ''),
'prefix' : '',
'suffix' : ''
},
'conclusion' : {
'border' : ('', '', '', '', '', '', '', ''),
'prefix' : '',
'horiz_sep' : '-',
'force_horiz' : True,
},
'important' : {
'tab_num' : 1,
},
'ascii' : {
'horiz_sep' : (u'├', u'─', u'┤'),
'border' : (u'┌', u'─', u'┐', u'│ ', u' │', u'└', u'─', u'┘'),
},
'ascii_double' : {
'horiz_sep' : (u'╠', u'═', u'╣'),
'border' : (u'╔', u'═', u'╗', u'║ ', u' ║', u'╚', u'═', u'╝'),
}
}
re_unicode_lowercase_a = re.compile(unicode(r"(?u)[áàäâãå]", "utf-8"))
re_unicode_lowercase_ae = re.compile(unicode(r"(?u)[æ]", "utf-8"))
re_unicode_lowercase_oe = re.compile(unicode(r"(?u)[œ]", "utf-8"))
re_unicode_lowercase_e = re.compile(unicode(r"(?u)[éèëê]", "utf-8"))
re_unicode_lowercase_i = re.compile(unicode(r"(?u)[íìïî]", "utf-8"))
re_unicode_lowercase_o = re.compile(unicode(r"(?u)[óòöôõø]", "utf-8"))
re_unicode_lowercase_u = re.compile(unicode(r"(?u)[úùüû]", "utf-8"))
re_unicode_lowercase_y = re.compile(unicode(r"(?u)[ýÿ]", "utf-8"))
re_unicode_lowercase_c = re.compile(unicode(r"(?u)[çć]", "utf-8"))
re_unicode_lowercase_n = re.compile(unicode(r"(?u)[ñ]", "utf-8"))
re_unicode_lowercase_ss = re.compile(unicode(r"(?u)[ß]", "utf-8"))
re_unicode_uppercase_a = re.compile(unicode(r"(?u)[ÁÀÄÂÃÅ]", "utf-8"))
re_unicode_uppercase_ae = re.compile(unicode(r"(?u)[Æ]", "utf-8"))
re_unicode_uppercase_oe = re.compile(unicode(r"(?u)[Œ]", "utf-8"))
re_unicode_uppercase_e = re.compile(unicode(r"(?u)[ÉÈËÊ]", "utf-8"))
re_unicode_uppercase_i = re.compile(unicode(r"(?u)[ÍÌÏÎ]", "utf-8"))
re_unicode_uppercase_o = re.compile(unicode(r"(?u)[ÓÒÖÔÕØ]", "utf-8"))
re_unicode_uppercase_u = re.compile(unicode(r"(?u)[ÚÙÜÛ]", "utf-8"))
re_unicode_uppercase_y = re.compile(unicode(r"(?u)[Ý]", "utf-8"))
re_unicode_uppercase_c = re.compile(unicode(r"(?u)[ÇĆ]", "utf-8"))
re_unicode_uppercase_n = re.compile(unicode(r"(?u)[Ñ]", "utf-8"))
re_latex_lowercase_a = re.compile("\\\\[\"H'`~^vu=k]\{?a\}?")
re_latex_lowercase_ae = re.compile("\\\\ae\\{\\}?")
re_latex_lowercase_oe = re.compile("\\\\oe\\{\\}?")
re_latex_lowercase_e = re.compile("\\\\[\"H'`~^vu=k]\\{?e\\}?")
re_latex_lowercase_i = re.compile("\\\\[\"H'`~^vu=k]\\{?i\\}?")
re_latex_lowercase_o = re.compile("\\\\[\"H'`~^vu=k]\\{?o\\}?")
re_latex_lowercase_u = re.compile("\\\\[\"H'`~^vu=k]\\{?u\\}?")
re_latex_lowercase_y = re.compile("\\\\[\"']\\{?y\\}?")
re_latex_lowercase_c = re.compile("\\\\['uc]\\{?c\\}?")
re_latex_lowercase_n = re.compile("\\\\[c'~^vu]\\{?n\\}?")
re_latex_uppercase_a = re.compile("\\\\[\"H'`~^vu=k]\\{?A\\}?")
re_latex_uppercase_ae = re.compile("\\\\AE\\{?\\}?")
re_latex_uppercase_oe = re.compile("\\\\OE\\{?\\}?")
re_latex_uppercase_e = re.compile("\\\\[\"H'`~^vu=k]\\{?E\\}?")
re_latex_uppercase_i = re.compile("\\\\[\"H'`~^vu=k]\\{?I\\}?")
re_latex_uppercase_o = re.compile("\\\\[\"H'`~^vu=k]\\{?O\\}?")
re_latex_uppercase_u = re.compile("\\\\[\"H'`~^vu=k]\\{?U\\}?")
re_latex_uppercase_y = re.compile("\\\\[\"']\\{?Y\\}?")
re_latex_uppercase_c = re.compile("\\\\['uc]\\{?C\\}?")
re_latex_uppercase_n = re.compile("\\\\[c'~^vu]\\{?N\\}?")
def get_kb_filename(filename='latex-to-unicode.kb'):
return pkg_resources.resource_filename('invenio.utils.data', filename)
def indent_text(text,
nb_tabs=0,
tab_str=" ",
linebreak_input="\n",
linebreak_output="\n",
wrap=False):
"""
add tabs to each line of text
@param text: the text to indent
@param nb_tabs: number of tabs to add
@param tab_str: type of tab (could be, for example "\t", default: 2 spaces
@param linebreak_input: linebreak on input
@param linebreak_output: linebreak on output
@param wrap: wethever to apply smart text wrapping.
(by means of wrap_text_in_a_box)
@return: indented text as string
"""
if not wrap:
lines = text.split(linebreak_input)
tabs = nb_tabs*tab_str
output = ""
for line in lines:
output += tabs + line + linebreak_output
return output
else:
return wrap_text_in_a_box(body=text, style='no_border',
tab_str=tab_str, tab_num=nb_tabs)
_RE_BEGINNING_SPACES = re.compile(r'^\s*')
_RE_NEWLINES_CLEANER = re.compile(r'\n+')
_RE_LONELY_NEWLINES = re.compile(r'\b\n\b')
def wrap_text_in_a_box(body='', title='', style='double_star', **args):
"""Return a nicely formatted text box:
e.g.
******************
** title **
**--------------**
** body **
******************
Indentation and newline are respected.
@param body: the main text
@param title: an optional title
@param style: the name of one of the style in CFG_WRAP_STYLES. By default
the double_star style is used.
You can further tune the desired style by setting various optional
parameters:
@param horiz_sep: a string that is repeated in order to produce a
separator row between the title and the body (if needed)
or a tuple of three characters in the form (l, c, r)
@param max_col: the maximum number of coulmns used by the box
(including indentation)
@param min_col: the symmetrical minimum number of columns
@param tab_str: a string to represent indentation
@param tab_num: the number of leveles of indentations
@param border: a tuple of 8 element in the form
(tl, t, tr, l, r, bl, b, br) of strings that represent the
different corners and sides of the box
@param prefix: a prefix string added before the box
@param suffix: a suffix string added after the box
@param break_long: wethever to break long words in order to respect
max_col
@param force_horiz: True in order to print the horizontal line even when
there is no title
e.g.:
print wrap_text_in_a_box(title='prova',
body=' 123 prova.\n Vediamo come si indenta',
horiz_sep='-', style='no_border', max_col=20, tab_num=1)
prova
----------------
123 prova.
Vediamo come
si indenta
"""
def _wrap_row(row, max_col, break_long):
"""Wrap a single row"""
spaces = _RE_BEGINNING_SPACES.match(row).group()
row = row[len(spaces):]
spaces = spaces.expandtabs()
return textwrap.wrap(row, initial_indent=spaces,
subsequent_indent=spaces, width=max_col,
break_long_words=break_long)
def _clean_newlines(text):
text = _RE_LONELY_NEWLINES.sub(' \n', text)
return _RE_NEWLINES_CLEANER.sub(lambda x: x.group()[:-1], text)
body = unicode(body, 'utf-8')
title = unicode(title, 'utf-8')
astyle = dict(CFG_WRAP_TEXT_IN_A_BOX_STYLES['__DEFAULT'])
if style in CFG_WRAP_TEXT_IN_A_BOX_STYLES:
astyle.update(CFG_WRAP_TEXT_IN_A_BOX_STYLES[style])
astyle.update(args)
horiz_sep = astyle['horiz_sep']
border = astyle['border']
tab_str = astyle['tab_str'] * astyle['tab_num']
max_col = max(astyle['max_col'] \
- len(border[3]) - len(border[4]) - len(tab_str), 1)
min_col = astyle['min_col']
prefix = astyle['prefix']
suffix = astyle['suffix']
force_horiz = astyle['force_horiz']
break_long = astyle['break_long']
body = _clean_newlines(body)
tmp_rows = [_wrap_row(row, max_col, break_long)
for row in body.split('\n')]
body_rows = []
for rows in tmp_rows:
if rows:
body_rows += rows
else:
body_rows.append('')
if not ''.join(body_rows).strip():
# Concrete empty body
body_rows = []
title = _clean_newlines(title)
tmp_rows = [_wrap_row(row, max_col, break_long)
for row in title.split('\n')]
title_rows = []
for rows in tmp_rows:
if rows:
title_rows += rows
else:
title_rows.append('')
if not ''.join(title_rows).strip():
# Concrete empty title
title_rows = []
max_col = max([len(row) for row in body_rows + title_rows] + [min_col])
mid_top_border_len = max_col \
+ len(border[3]) + len(border[4]) - len(border[0]) - len(border[2])
mid_bottom_border_len = max_col \
+ len(border[3]) + len(border[4]) - len(border[5]) - len(border[7])
top_border = border[0] \
+ (border[1] * mid_top_border_len)[:mid_top_border_len] + border[2]
bottom_border = border[5] \
+ (border[6] * mid_bottom_border_len)[:mid_bottom_border_len] \
+ border[7]
if type(horiz_sep) is tuple and len(horiz_sep) == 3:
horiz_line = horiz_sep[0] + (horiz_sep[1] * (max_col + 2))[:(max_col + 2)] + horiz_sep[2]
else:
horiz_line = border[3] + (horiz_sep * max_col)[:max_col] + border[4]
title_rows = [tab_str + border[3] + row
+ ' ' * (max_col - len(row)) + border[4] for row in title_rows]
body_rows = [tab_str + border[3] + row
+ ' ' * (max_col - len(row)) + border[4] for row in body_rows]
ret = []
if top_border:
ret += [tab_str + top_border]
ret += title_rows
if title_rows or force_horiz:
ret += [tab_str + horiz_line]
ret += body_rows
if bottom_border:
ret += [tab_str + bottom_border]
return (prefix + '\n'.join(ret) + suffix).encode('utf-8')
def wait_for_user(msg=""):
"""
Print MSG and a confirmation prompt, waiting for user's
confirmation, unless silent '--yes-i-know' command line option was
used, in which case the function returns immediately without
printing anything.
"""
if '--yes-i-know' in sys.argv:
return
print(msg)
try:
answer = raw_input("Please confirm by typing 'Yes, I know!': ")
except KeyboardInterrupt:
print()
answer = ''
if answer != 'Yes, I know!':
sys.stderr.write("ERROR: Aborted.\n")
sys.exit(1)
return
def guess_minimum_encoding(text, charsets=('ascii', 'latin1', 'utf8')):
"""Try to guess the minimum charset that is able to represent the given
text using the provided charsets. text is supposed to be encoded in utf8.
Returns (encoded_text, charset) where charset is the first charset
in the sequence being able to encode text.
Returns (text_in_utf8, 'utf8') in case no charset is able to encode text.
@note: If the input text is not in strict UTF-8, then replace any
non-UTF-8 chars inside it.
"""
text_in_unicode = text.decode('utf8', 'replace')
for charset in charsets:
try:
return (text_in_unicode.encode(charset), charset)
except (UnicodeEncodeError, UnicodeDecodeError):
pass
return (text_in_unicode.encode('utf8'), 'utf8')
def encode_for_xml(text, wash=False, xml_version='1.0', quote=False):
"""Encodes special characters in a text so that it would be
XML-compliant.
@param text: text to encode
@return: an encoded text"""
text = text.replace('&', '&amp;')
text = text.replace('<', '&lt;')
if quote:
text = text.replace('"', '&quot;')
if wash:
text = wash_for_xml(text, xml_version=xml_version)
return text
try:
unichr(0x100000)
RE_ALLOWED_XML_1_0_CHARS = re.compile(u'[^\U00000009\U0000000A\U0000000D\U00000020-\U0000D7FF\U0000E000-\U0000FFFD\U00010000-\U0010FFFF]')
RE_ALLOWED_XML_1_1_CHARS = re.compile(u'[^\U00000001-\U0000D7FF\U0000E000-\U0000FFFD\U00010000-\U0010FFFF]')
except ValueError:
# oops, we are running on a narrow UTF/UCS Python build,
# so we have to limit the UTF/UCS char range:
RE_ALLOWED_XML_1_0_CHARS = re.compile(u'[^\U00000009\U0000000A\U0000000D\U00000020-\U0000D7FF\U0000E000-\U0000FFFD]')
RE_ALLOWED_XML_1_1_CHARS = re.compile(u'[^\U00000001-\U0000D7FF\U0000E000-\U0000FFFD]')
def wash_for_xml(text, xml_version='1.0'):
"""
Removes any character which is not in the range of allowed
characters for XML. The allowed characters depends on the version
of XML.
- XML 1.0:
<http://www.w3.org/TR/REC-xml/#charsets>
- XML 1.1:
<http://www.w3.org/TR/xml11/#charsets>
@param text: input string to wash.
@param xml_version: version of the XML for which we wash the
input. Value for this parameter can be '1.0' or '1.1'
"""
if xml_version == '1.0':
return RE_ALLOWED_XML_1_0_CHARS.sub('', unicode(text, 'utf-8')).encode('utf-8')
else:
return RE_ALLOWED_XML_1_1_CHARS.sub('', unicode(text, 'utf-8')).encode('utf-8')
def wash_for_utf8(text, correct=True):
"""Return UTF-8 encoded binary string with incorrect characters washed away.
@param text: input string to wash (can be either a binary string or a Unicode string)
@param correct: whether to correct bad characters or throw exception
"""
if isinstance(text, unicode):
return text.encode('utf-8')
ret = []
while True:
try:
text.decode("utf-8")
except UnicodeDecodeError as e:
if correct:
ret.append(text[:e.start])
text = text[e.end:]
else:
raise e
else:
break
ret.append(text)
return ''.join(ret)
def nice_number(number, thousands_separator=',', max_ndigits_after_dot=None):
"""
Return nicely printed number NUMBER in language LN using
given THOUSANDS_SEPARATOR character.
If max_ndigits_after_dot is specified and the number is float, the
number is rounded by taking in consideration up to max_ndigits_after_dot
digit after the dot.
This version does not pay attention to locale. See
tmpl_nice_number_via_locale().
"""
if type(number) is float:
if max_ndigits_after_dot is not None:
number = round(number, max_ndigits_after_dot)
int_part, frac_part = str(number).split('.')
return '%s.%s' % (nice_number(int(int_part), thousands_separator),
frac_part)
else:
chars_in = list(str(number))
number = len(chars_in)
chars_out = []
for i in range(0, number):
if i % 3 == 0 and i != 0:
chars_out.append(thousands_separator)
chars_out.append(chars_in[number - i - 1])
chars_out.reverse()
return ''.join(chars_out)
def nice_size(size):
"""
@param size: the size.
@type size: int
@return: a nicely printed size.
@rtype: string
"""
unit = 'B'
if size > 1024:
size /= 1024.0
unit = 'KB'
if size > 1024:
size /= 1024.0
unit = 'MB'
if size > 1024:
size /= 1024.0
unit = 'GB'
return '%s %s' % (nice_number(size, max_ndigits_after_dot=2), unit)
def remove_line_breaks(text):
"""
Remove line breaks from input, including unicode 'line
separator', 'paragraph separator', and 'next line' characters.
"""
return unicode(text, 'utf-8').replace('\f', '').replace('\n', '').replace('\r', '').replace(u'\xe2\x80\xa8', '').replace(u'\xe2\x80\xa9', '').replace(u'\xc2\x85', '').encode('utf-8')
def decode_to_unicode(text, default_encoding='utf-8'):
"""
Decode input text into Unicode representation by first using the default
encoding utf-8.
If the operation fails, it detects the type of encoding used in the given text.
For optimal result, it is recommended that the 'chardet' module is installed.
NOTE: Beware that this might be slow for *very* large strings.
If chardet detection fails, it will try to decode the string using the basic
detection function guess_minimum_encoding().
Also, bear in mind that it is impossible to detect the correct encoding at all
times, other then taking educated guesses. With that said, this function will
always return some decoded Unicode string, however the data returned may not
be the same as original data in some cases.
@param text: the text to decode
@type text: string
@param default_encoding: the character encoding to use. Optional.
@type default_encoding: string
@return: input text as Unicode
@rtype: string
"""
if not text:
return ""
try:
return text.decode(default_encoding)
except (UnicodeError, LookupError):
pass
detected_encoding = None
if CHARDET_AVAILABLE:
# We can use chardet to perform detection
res = chardet.detect(text)
if res['confidence'] >= 0.8:
detected_encoding = res['encoding']
if detected_encoding == None:
# No chardet detection, try to make a basic guess
dummy, detected_encoding = guess_minimum_encoding(text)
return text.decode(detected_encoding)
def to_unicode(text):
if isinstance(text, unicode):
return text
if isinstance(text, six.string_types):
return decode_to_unicode(text)
return unicode(text)
def translate_latex2unicode(text, kb_file=None):
"""
This function will take given text, presumably containing LaTeX symbols,
and attempts to translate it to Unicode using the given or default KB
translation table located under CFG_ETCDIR/bibconvert/KB/latex-to-unicode.kb.
The translated Unicode string will then be returned.
If the translation table and compiled regular expression object is not
previously generated in the current session, they will be.
@param text: a text presumably containing LaTeX symbols.
@type text: string
@param kb_file: full path to file containing latex2unicode translations.
Defaults to CFG_ETCDIR/bibconvert/KB/latex-to-unicode.kb
@type kb_file: string
@return: Unicode representation of translated text
@rtype: unicode
"""
if kb_file is None:
kb_file = get_kb_filename()
# First decode input text to Unicode
try:
text = decode_to_unicode(text)
except UnicodeDecodeError:
text = unicode(wash_for_utf8(text))
# Load translation table, if required
if CFG_LATEX_UNICODE_TRANSLATION_CONST == {}:
_load_latex2unicode_constants(kb_file)
# Find all matches and replace text
for match in CFG_LATEX_UNICODE_TRANSLATION_CONST['regexp_obj'].finditer(text):
# If LaTeX style markers {, } and $ are before or after the matching text, it
# will replace those as well
text = re.sub("[\{\$]?%s[\}\$]?" % (re.escape(match.group()),), \
CFG_LATEX_UNICODE_TRANSLATION_CONST['table'][match.group()], \
text)
# Return Unicode representation of translated text
return text
def _load_latex2unicode_constants(kb_file=None):
"""
Load LaTeX2Unicode translation table dictionary and regular expression object
from KB to a global dictionary.
@param kb_file: full path to file containing latex2unicode translations.
Defaults to CFG_ETCDIR/bibconvert/KB/latex-to-unicode.kb
@type kb_file: string
@return: dict of type: {'regexp_obj': regexp match object,
'table': dict of LaTeX -> Unicode mappings}
@rtype: dict
"""
if kb_file is None:
kb_file = get_kb_filename()
try:
data = open(kb_file)
except IOError:
# File not found or similar
sys.stderr.write("\nCould not open LaTeX to Unicode KB file. Aborting translation.\n")
return CFG_LATEX_UNICODE_TRANSLATION_CONST
latex_symbols = []
translation_table = {}
for line in data:
# The file has form of latex|--|utf-8. First decode to Unicode.
line = line.decode('utf-8')
mapping = line.split('|--|')
translation_table[mapping[0].rstrip('\n')] = mapping[1].rstrip('\n')
latex_symbols.append(re.escape(mapping[0].rstrip('\n')))
data.close()
CFG_LATEX_UNICODE_TRANSLATION_CONST['regexp_obj'] = re.compile("|".join(latex_symbols))
CFG_LATEX_UNICODE_TRANSLATION_CONST['table'] = translation_table
def translate_to_ascii(values):
"""
Transliterate the string contents of the given sequence into ascii representation.
Returns a sequence with the modified values if the module 'unidecode' is
available. Otherwise it will fall back to the inferior strip_accents function.
For example: H\xc3\xb6hne becomes Hohne.
Note: Passed strings are returned as a list.
@param values: sequence of strings to transform
@type values: sequence
@return: sequence with values transformed to ascii
@rtype: sequence
"""
if not values and not type(values) == str:
return values
if type(values) == str:
values = [values]
for index, value in enumerate(values):
if not value:
continue
if not UNIDECODE_AVAILABLE:
ascii_text = strip_accents(value)
else:
encoded_text, encoding = guess_minimum_encoding(value)
unicode_text = unicode(encoded_text.decode(encoding))
ascii_text = unidecode(unicode_text).encode('ascii')
values[index] = ascii_text
return values
def xml_entities_to_utf8(text, skip=('lt', 'gt', 'amp')):
"""
Removes HTML or XML character references and entities from a text string
and replaces them with their UTF-8 representation, if possible.
@param text: The HTML (or XML) source text.
@type text: string
@param skip: list of entity names to skip when transforming.
@type skip: iterable
@return: The plain text, as a Unicode string, if necessary.
@author: Based on http://effbot.org/zone/re-sub.htm#unescape-html
"""
def fixup(m):
text = m.group(0)
if text[:2] == "&#":
# character reference
try:
if text[:3] == "&#x":
return unichr(int(text[3:-1], 16)).encode("utf-8")
else:
return unichr(int(text[2:-1])).encode("utf-8")
except ValueError:
pass
else:
# named entity
if text[1:-1] not in skip:
try:
text = unichr(html_entities.name2codepoint[text[1:-1]]).encode("utf-8")
except KeyError:
pass
return text # leave as is
return re.sub("&#?\w+;", fixup, text)
def strip_accents(x):
"""
Strip accents in the input phrase X (assumed in UTF-8) by replacing
accented characters with their unaccented cousins (e.g. é by e).
@param x: the input phrase to strip.
@type x: string
@return: Return such a stripped X.
"""
x = re_latex_lowercase_a.sub("a", x)
x = re_latex_lowercase_ae.sub("ae", x)
x = re_latex_lowercase_oe.sub("oe", x)
x = re_latex_lowercase_e.sub("e", x)
x = re_latex_lowercase_i.sub("i", x)
x = re_latex_lowercase_o.sub("o", x)
x = re_latex_lowercase_u.sub("u", x)
x = re_latex_lowercase_y.sub("x", x)
x = re_latex_lowercase_c.sub("c", x)
x = re_latex_lowercase_n.sub("n", x)
x = re_latex_uppercase_a.sub("A", x)
x = re_latex_uppercase_ae.sub("AE", x)
x = re_latex_uppercase_oe.sub("OE", x)
x = re_latex_uppercase_e.sub("E", x)
x = re_latex_uppercase_i.sub("I", x)
x = re_latex_uppercase_o.sub("O", x)
x = re_latex_uppercase_u.sub("U", x)
x = re_latex_uppercase_y.sub("Y", x)
x = re_latex_uppercase_c.sub("C", x)
x = re_latex_uppercase_n.sub("N", x)
# convert input into Unicode string:
try:
y = unicode(x, "utf-8")
except:
return x # something went wrong, probably the input wasn't UTF-8
# asciify Latin-1 lowercase characters:
y = re_unicode_lowercase_a.sub("a", y)
y = re_unicode_lowercase_ae.sub("ae", y)
y = re_unicode_lowercase_oe.sub("oe", y)
y = re_unicode_lowercase_e.sub("e", y)
y = re_unicode_lowercase_i.sub("i", y)
y = re_unicode_lowercase_o.sub("o", y)
y = re_unicode_lowercase_u.sub("u", y)
y = re_unicode_lowercase_y.sub("y", y)
y = re_unicode_lowercase_c.sub("c", y)
y = re_unicode_lowercase_n.sub("n", y)
y = re_unicode_lowercase_ss.sub("ss", y)
# asciify Latin-1 uppercase characters:
y = re_unicode_uppercase_a.sub("A", y)
y = re_unicode_uppercase_ae.sub("AE", y)
y = re_unicode_uppercase_oe.sub("OE", y)
y = re_unicode_uppercase_e.sub("E", y)
y = re_unicode_uppercase_i.sub("I", y)
y = re_unicode_uppercase_o.sub("O", y)
y = re_unicode_uppercase_u.sub("U", y)
y = re_unicode_uppercase_y.sub("Y", y)
y = re_unicode_uppercase_c.sub("C", y)
y = re_unicode_uppercase_n.sub("N", y)
# return UTF-8 representation of the Unicode string:
return y.encode("utf-8")
_punct_re = re.compile(r'[\t !"#$%&\'()*\-/<=>?@\[\\\]^_`{|},.]+')
def slugify(text, delim=u'-'):
"""Generates an ASCII-only slug."""
result = []
for word in _punct_re.split(text.lower()):
result.extend(unidecode(word).split())
return unicode(delim.join(result))
-def show_diff(original, modified, prefix="<pre>", sufix="</pre>"):
- """
- Returns the diff view between source and changed strings.
+def show_diff(original, modified, prefix='', suffix='',
+ prefix_unchanged=' ',
+ suffix_unchanged='',
+ prefix_removed='-',
+ suffix_removed='',
+ prefix_added='+',
+ suffix_added=''):
+ """
+ Returns the diff view between original and modified strings.
Function checks both arguments line by line and returns a string
- with additional css classes for difference view
+ with a:
+ - prefix_unchanged when line is common to both sequences
+ - prefix_removed when line is unique to sequence 1
+ - prefix_added when line is unique to sequence 2
+ and a corresponding suffix in each line
@param original: base string
@param modified: changed string
@param prefix: prefix of the output string
- @param sufix: sufix of the output string
+ @param suffix: suffix of the output string
+ @param prefix_unchanged: prefix of the unchanged line
+ @param suffix_unchanged: suffix of the unchanged line
+ @param prefix_removed: prefix of the removed line
+ @param suffix_removed: suffix of the removed line
+ @param prefix_added: prefix of the added line
+ @param suffix_added: suffix of the added line
@return: string with the comparison of the records
@rtype: string
"""
import difflib
differ = difflib.Differ()
result = [prefix]
for line in differ.compare(modified.splitlines(), original.splitlines()):
if line[0] == ' ':
- result.append(line.strip())
+ # Mark as unchanged
+ result.append(prefix_unchanged + line[2:].strip() + suffix_unchanged)
elif line[0] == '-':
- # Mark as deleted
- result.append('<strong class="diff_field_deleted">' + line[2:].strip() + "</strong>")
+ # Mark as removed
+ result.append(prefix_removed + line[2:].strip() + suffix_removed)
elif line[0] == '+':
# Mark as added/modified
- result.append('<strong class="diff_field_added">' + line[2:].strip() + "</strong>")
- else:
- continue
+ result.append(prefix_added + line[2:].strip() + suffix_added)
- result.append(sufix)
+ result.append(suffix)
return '\n'.join(result)
def transliterate_ala_lc(value):
"""
Transliterate a string.
Compatibility with the ALA-LC romanization standard:
http://www.loc.gov/catdir/cpso/roman.html
Maps from one system of writing into another, letter by letter.
Uses 'unidecode' if available.
@param values: string to transform
@type values: string
@return: string transliterated
@rtype: string
"""
if not value:
return value
if UNIDECODE_AVAILABLE:
text = unidecode(value)
else:
text = translate_to_ascii(value)
text = text.pop()
return text
def escape_latex(text):
"""
This function takes the given text and escapes characters
that have a special meaning in LaTeX: # $ % ^ & _ { } ~ \
"""
text = unicode(text.decode('utf-8'))
CHARS = {
'&': r'\&',
'%': r'\%',
'$': r'\$',
'#': r'\#',
'_': r'\_',
'{': r'\{',
'}': r'\}',
'~': r'\~{}',
'^': r'\^{}',
'\\': r'\textbackslash{}',
}
escaped = "".join([CHARS.get(char, char) for char in text])
return escaped.encode('utf-8')
diff --git a/modules/bibauthorid/lib/backbone-min.js b/modules/bibauthorid/lib/backbone-min.js
deleted file mode 100644
index 3541019c5..000000000
--- a/modules/bibauthorid/lib/backbone-min.js
+++ /dev/null
@@ -1,4 +0,0 @@
-(function(){var t=this;var e=t.Backbone;var i=[];var r=i.push;var s=i.slice;var n=i.splice;var a;if(typeof exports!=="undefined"){a=exports}else{a=t.Backbone={}}a.VERSION="1.0.0";var h=t._;if(!h&&typeof require!=="undefined")h=require("underscore");a.$=t.jQuery||t.Zepto||t.ender||t.$;a.noConflict=function(){t.Backbone=e;return this};a.emulateHTTP=false;a.emulateJSON=false;var o=a.Events={on:function(t,e,i){if(!l(this,"on",t,[e,i])||!e)return this;this._events||(this._events={});var r=this._events[t]||(this._events[t]=[]);r.push({callback:e,context:i,ctx:i||this});return this},once:function(t,e,i){if(!l(this,"once",t,[e,i])||!e)return this;var r=this;var s=h.once(function(){r.off(t,s);e.apply(this,arguments)});s._callback=e;return this.on(t,s,i)},off:function(t,e,i){var r,s,n,a,o,u,c,f;if(!this._events||!l(this,"off",t,[e,i]))return this;if(!t&&!e&&!i){this._events={};return this}a=t?[t]:h.keys(this._events);for(o=0,u=a.length;o<u;o++){t=a[o];if(n=this._events[t]){this._events[t]=r=[];if(e||i){for(c=0,f=n.length;c<f;c++){s=n[c];if(e&&e!==s.callback&&e!==s.callback._callback||i&&i!==s.context){r.push(s)}}}if(!r.length)delete this._events[t]}}return this},trigger:function(t){if(!this._events)return this;var e=s.call(arguments,1);if(!l(this,"trigger",t,e))return this;var i=this._events[t];var r=this._events.all;if(i)c(i,e);if(r)c(r,arguments);return this},stopListening:function(t,e,i){var r=this._listeners;if(!r)return this;var s=!e&&!i;if(typeof e==="object")i=this;if(t)(r={})[t._listenerId]=t;for(var n in r){r[n].off(e,i,this);if(s)delete this._listeners[n]}return this}};var u=/\s+/;var l=function(t,e,i,r){if(!i)return true;if(typeof i==="object"){for(var s in i){t[e].apply(t,[s,i[s]].concat(r))}return false}if(u.test(i)){var n=i.split(u);for(var a=0,h=n.length;a<h;a++){t[e].apply(t,[n[a]].concat(r))}return false}return true};var c=function(t,e){var i,r=-1,s=t.length,n=e[0],a=e[1],h=e[2];switch(e.length){case 0:while(++r<s)(i=t[r]).callback.call(i.ctx);return;case 1:while(++r<s)(i=t[r]).callback.call(i.ctx,n);return;case 2:while(++r<s)(i=t[r]).callback.call(i.ctx,n,a);return;case 3:while(++r<s)(i=t[r]).callback.call(i.ctx,n,a,h);return;default:while(++r<s)(i=t[r]).callback.apply(i.ctx,e)}};var f={listenTo:"on",listenToOnce:"once"};h.each(f,function(t,e){o[e]=function(e,i,r){var s=this._listeners||(this._listeners={});var n=e._listenerId||(e._listenerId=h.uniqueId("l"));s[n]=e;if(typeof i==="object")r=this;e[t](i,r,this);return this}});o.bind=o.on;o.unbind=o.off;h.extend(a,o);var d=a.Model=function(t,e){var i;var r=t||{};e||(e={});this.cid=h.uniqueId("c");this.attributes={};h.extend(this,h.pick(e,p));if(e.parse)r=this.parse(r,e)||{};if(i=h.result(this,"defaults")){r=h.defaults({},r,i)}this.set(r,e);this.changed={};this.initialize.apply(this,arguments)};var p=["url","urlRoot","collection"];h.extend(d.prototype,o,{changed:null,validationError:null,idAttribute:"id",initialize:function(){},toJSON:function(t){return h.clone(this.attributes)},sync:function(){return a.sync.apply(this,arguments)},get:function(t){return this.attributes[t]},escape:function(t){return h.escape(this.get(t))},has:function(t){return this.get(t)!=null},set:function(t,e,i){var r,s,n,a,o,u,l,c;if(t==null)return this;if(typeof t==="object"){s=t;i=e}else{(s={})[t]=e}i||(i={});if(!this._validate(s,i))return false;n=i.unset;o=i.silent;a=[];u=this._changing;this._changing=true;if(!u){this._previousAttributes=h.clone(this.attributes);this.changed={}}c=this.attributes,l=this._previousAttributes;if(this.idAttribute in s)this.id=s[this.idAttribute];for(r in s){e=s[r];if(!h.isEqual(c[r],e))a.push(r);if(!h.isEqual(l[r],e)){this.changed[r]=e}else{delete this.changed[r]}n?delete c[r]:c[r]=e}if(!o){if(a.length)this._pending=true;for(var f=0,d=a.length;f<d;f++){this.trigger("change:"+a[f],this,c[a[f]],i)}}if(u)return this;if(!o){while(this._pending){this._pending=false;this.trigger("change",this,i)}}this._pending=false;this._changing=false;return this},unset:function(t,e){return this.set(t,void 0,h.extend({},e,{unset:true}))},clear:function(t){var e={};for(var i in this.attributes)e[i]=void 0;return this.set(e,h.extend({},t,{unset:true}))},hasChanged:function(t){if(t==null)return!h.isEmpty(this.changed);return h.has(this.changed,t)},changedAttributes:function(t){if(!t)return this.hasChanged()?h.clone(this.changed):false;var e,i=false;var r=this._changing?this._previousAttributes:this.attributes;for(var s in t){if(h.isEqual(r[s],e=t[s]))continue;(i||(i={}))[s]=e}return i},previous:function(t){if(t==null||!this._previousAttributes)return null;return this._previousAttributes[t]},previousAttributes:function(){return h.clone(this._previousAttributes)},fetch:function(t){t=t?h.clone(t):{};if(t.parse===void 0)t.parse=true;var e=this;var i=t.success;t.success=function(r){if(!e.set(e.parse(r,t),t))return false;if(i)i(e,r,t);e.trigger("sync",e,r,t)};R(this,t);return this.sync("read",this,t)},save:function(t,e,i){var r,s,n,a=this.attributes;if(t==null||typeof t==="object"){r=t;i=e}else{(r={})[t]=e}if(r&&(!i||!i.wait)&&!this.set(r,i))return false;i=h.extend({validate:true},i);if(!this._validate(r,i))return false;if(r&&i.wait){this.attributes=h.extend({},a,r)}if(i.parse===void 0)i.parse=true;var o=this;var u=i.success;i.success=function(t){o.attributes=a;var e=o.parse(t,i);if(i.wait)e=h.extend(r||{},e);if(h.isObject(e)&&!o.set(e,i)){return false}if(u)u(o,t,i);o.trigger("sync",o,t,i)};R(this,i);s=this.isNew()?"create":i.patch?"patch":"update";if(s==="patch")i.attrs=r;n=this.sync(s,this,i);if(r&&i.wait)this.attributes=a;return n},destroy:function(t){t=t?h.clone(t):{};var e=this;var i=t.success;var r=function(){e.trigger("destroy",e,e.collection,t)};t.success=function(s){if(t.wait||e.isNew())r();if(i)i(e,s,t);if(!e.isNew())e.trigger("sync",e,s,t)};if(this.isNew()){t.success();return false}R(this,t);var s=this.sync("delete",this,t);if(!t.wait)r();return s},url:function(){var t=h.result(this,"urlRoot")||h.result(this.collection,"url")||U();if(this.isNew())return t;return t+(t.charAt(t.length-1)==="/"?"":"/")+encodeURIComponent(this.id)},parse:function(t,e){return t},clone:function(){return new this.constructor(this.attributes)},isNew:function(){return this.id==null},isValid:function(t){return this._validate({},h.extend(t||{},{validate:true}))},_validate:function(t,e){if(!e.validate||!this.validate)return true;t=h.extend({},this.attributes,t);var i=this.validationError=this.validate(t,e)||null;if(!i)return true;this.trigger("invalid",this,i,h.extend(e||{},{validationError:i}));return false}});var v=["keys","values","pairs","invert","pick","omit"];h.each(v,function(t){d.prototype[t]=function(){var e=s.call(arguments);e.unshift(this.attributes);return h[t].apply(h,e)}});var g=a.Collection=function(t,e){e||(e={});if(e.url)this.url=e.url;if(e.model)this.model=e.model;if(e.comparator!==void 0)this.comparator=e.comparator;this._reset();this.initialize.apply(this,arguments);if(t)this.reset(t,h.extend({silent:true},e))};var m={add:true,remove:true,merge:true};var y={add:true,merge:false,remove:false};h.extend(g.prototype,o,{model:d,initialize:function(){},toJSON:function(t){return this.map(function(e){return e.toJSON(t)})},sync:function(){return a.sync.apply(this,arguments)},add:function(t,e){return this.set(t,h.defaults(e||{},y))},remove:function(t,e){t=h.isArray(t)?t.slice():[t];e||(e={});var i,r,s,n;for(i=0,r=t.length;i<r;i++){n=this.get(t[i]);if(!n)continue;delete this._byId[n.id];delete this._byId[n.cid];s=this.indexOf(n);this.models.splice(s,1);this.length--;if(!e.silent){e.index=s;n.trigger("remove",n,this,e)}this._removeReference(n)}return this},set:function(t,e){e=h.defaults(e||{},m);if(e.parse)t=this.parse(t,e);if(!h.isArray(t))t=t?[t]:[];var i,s,a,o,u,l;var c=e.at;var f=this.comparator&&c==null&&e.sort!==false;var d=h.isString(this.comparator)?this.comparator:null;var p=[],v=[],g={};for(i=0,s=t.length;i<s;i++){if(!(a=this._prepareModel(t[i],e)))continue;if(u=this.get(a)){if(e.remove)g[u.cid]=true;if(e.merge){u.set(a.attributes,e);if(f&&!l&&u.hasChanged(d))l=true}}else if(e.add){p.push(a);a.on("all",this._onModelEvent,this);this._byId[a.cid]=a;if(a.id!=null)this._byId[a.id]=a}}if(e.remove){for(i=0,s=this.length;i<s;++i){if(!g[(a=this.models[i]).cid])v.push(a)}if(v.length)this.remove(v,e)}if(p.length){if(f)l=true;this.length+=p.length;if(c!=null){n.apply(this.models,[c,0].concat(p))}else{r.apply(this.models,p)}}if(l)this.sort({silent:true});if(e.silent)return this;for(i=0,s=p.length;i<s;i++){(a=p[i]).trigger("add",a,this,e)}if(l)this.trigger("sort",this,e);return this},reset:function(t,e){e||(e={});for(var i=0,r=this.models.length;i<r;i++){this._removeReference(this.models[i])}e.previousModels=this.models;this._reset();this.add(t,h.extend({silent:true},e));if(!e.silent)this.trigger("reset",this,e);return this},push:function(t,e){t=this._prepareModel(t,e);this.add(t,h.extend({at:this.length},e));return t},pop:function(t){var e=this.at(this.length-1);this.remove(e,t);return e},unshift:function(t,e){t=this._prepareModel(t,e);this.add(t,h.extend({at:0},e));return t},shift:function(t){var e=this.at(0);this.remove(e,t);return e},slice:function(t,e){return this.models.slice(t,e)},get:function(t){if(t==null)return void 0;return this._byId[t.id!=null?t.id:t.cid||t]},at:function(t){return this.models[t]},where:function(t,e){if(h.isEmpty(t))return e?void 0:[];return this[e?"find":"filter"](function(e){for(var i in t){if(t[i]!==e.get(i))return false}return true})},findWhere:function(t){return this.where(t,true)},sort:function(t){if(!this.comparator)throw new Error("Cannot sort a set without a comparator");t||(t={});if(h.isString(this.comparator)||this.comparator.length===1){this.models=this.sortBy(this.comparator,this)}else{this.models.sort(h.bind(this.comparator,this))}if(!t.silent)this.trigger("sort",this,t);return this},sortedIndex:function(t,e,i){e||(e=this.comparator);var r=h.isFunction(e)?e:function(t){return t.get(e)};return h.sortedIndex(this.models,t,r,i)},pluck:function(t){return h.invoke(this.models,"get",t)},fetch:function(t){t=t?h.clone(t):{};if(t.parse===void 0)t.parse=true;var e=t.success;var i=this;t.success=function(r){var s=t.reset?"reset":"set";i[s](r,t);if(e)e(i,r,t);i.trigger("sync",i,r,t)};R(this,t);return this.sync("read",this,t)},create:function(t,e){e=e?h.clone(e):{};if(!(t=this._prepareModel(t,e)))return false;if(!e.wait)this.add(t,e);var i=this;var r=e.success;e.success=function(s){if(e.wait)i.add(t,e);if(r)r(t,s,e)};t.save(null,e);return t},parse:function(t,e){return t},clone:function(){return new this.constructor(this.models)},_reset:function(){this.length=0;this.models=[];this._byId={}},_prepareModel:function(t,e){if(t instanceof d){if(!t.collection)t.collection=this;return t}e||(e={});e.collection=this;var i=new this.model(t,e);if(!i._validate(t,e)){this.trigger("invalid",this,t,e);return false}return i},_removeReference:function(t){if(this===t.collection)delete t.collection;t.off("all",this._onModelEvent,this)},_onModelEvent:function(t,e,i,r){if((t==="add"||t==="remove")&&i!==this)return;if(t==="destroy")this.remove(e,r);if(e&&t==="change:"+e.idAttribute){delete this._byId[e.previous(e.idAttribute)];if(e.id!=null)this._byId[e.id]=e}this.trigger.apply(this,arguments)}});var _=["forEach","each","map","collect","reduce","foldl","inject","reduceRight","foldr","find","detect","filter","select","reject","every","all","some","any","include","contains","invoke","max","min","toArray","size","first","head","take","initial","rest","tail","drop","last","without","indexOf","shuffle","lastIndexOf","isEmpty","chain"];h.each(_,function(t){g.prototype[t]=function(){var e=s.call(arguments);e.unshift(this.models);return h[t].apply(h,e)}});var w=["groupBy","countBy","sortBy"];h.each(w,function(t){g.prototype[t]=function(e,i){var r=h.isFunction(e)?e:function(t){return t.get(e)};return h[t](this.models,r,i)}});var b=a.View=function(t){this.cid=h.uniqueId("view");this._configure(t||{});this._ensureElement();this.initialize.apply(this,arguments);this.delegateEvents()};var x=/^(\S+)\s*(.*)$/;var E=["model","collection","el","id","attributes","className","tagName","events"];h.extend(b.prototype,o,{tagName:"div",$:function(t){return this.$el.find(t)},initialize:function(){},render:function(){return this},remove:function(){this.$el.remove();this.stopListening();return this},setElement:function(t,e){if(this.$el)this.undelegateEvents();this.$el=t instanceof a.$?t:a.$(t);this.el=this.$el[0];if(e!==false)this.delegateEvents();return this},delegateEvents:function(t){if(!(t||(t=h.result(this,"events"))))return this;this.undelegateEvents();for(var e in t){var i=t[e];if(!h.isFunction(i))i=this[t[e]];if(!i)continue;var r=e.match(x);var s=r[1],n=r[2];i=h.bind(i,this);s+=".delegateEvents"+this.cid;if(n===""){this.$el.on(s,i)}else{this.$el.on(s,n,i)}}return this},undelegateEvents:function(){this.$el.off(".delegateEvents"+this.cid);return this},_configure:function(t){if(this.options)t=h.extend({},h.result(this,"options"),t);h.extend(this,h.pick(t,E));this.options=t},_ensureElement:function(){if(!this.el){var t=h.extend({},h.result(this,"attributes"));if(this.id)t.id=h.result(this,"id");if(this.className)t["class"]=h.result(this,"className");var e=a.$("<"+h.result(this,"tagName")+">").attr(t);this.setElement(e,false)}else{this.setElement(h.result(this,"el"),false)}}});a.sync=function(t,e,i){var r=k[t];h.defaults(i||(i={}),{emulateHTTP:a.emulateHTTP,emulateJSON:a.emulateJSON});var s={type:r,dataType:"json"};if(!i.url){s.url=h.result(e,"url")||U()}if(i.data==null&&e&&(t==="create"||t==="update"||t==="patch")){s.contentType="application/json";s.data=JSON.stringify(i.attrs||e.toJSON(i))}if(i.emulateJSON){s.contentType="application/x-www-form-urlencoded";s.data=s.data?{model:s.data}:{}}if(i.emulateHTTP&&(r==="PUT"||r==="DELETE"||r==="PATCH")){s.type="POST";if(i.emulateJSON)s.data._method=r;var n=i.beforeSend;i.beforeSend=function(t){t.setRequestHeader("X-HTTP-Method-Override",r);if(n)return n.apply(this,arguments)}}if(s.type!=="GET"&&!i.emulateJSON){s.processData=false}if(s.type==="PATCH"&&window.ActiveXObject&&!(window.external&&window.external.msActiveXFilteringEnabled)){s.xhr=function(){return new ActiveXObject("Microsoft.XMLHTTP")}}var o=i.xhr=a.ajax(h.extend(s,i));e.trigger("request",e,o,i);return o};var k={create:"POST",update:"PUT",patch:"PATCH","delete":"DELETE",read:"GET"};a.ajax=function(){return a.$.ajax.apply(a.$,arguments)};var S=a.Router=function(t){t||(t={});if(t.routes)this.routes=t.routes;this._bindRoutes();this.initialize.apply(this,arguments)};var $=/\((.*?)\)/g;var T=/(\(\?)?:\w+/g;var H=/\*\w+/g;var A=/[\-{}\[\]+?.,\\\^$|#\s]/g;h.extend(S.prototype,o,{initialize:function(){},route:function(t,e,i){if(!h.isRegExp(t))t=this._routeToRegExp(t);if(h.isFunction(e)){i=e;e=""}if(!i)i=this[e];var r=this;a.history.route(t,function(s){var n=r._extractParameters(t,s);i&&i.apply(r,n);r.trigger.apply(r,["route:"+e].concat(n));r.trigger("route",e,n);a.history.trigger("route",r,e,n)});return this},navigate:function(t,e){a.history.navigate(t,e);return this},_bindRoutes:function(){if(!this.routes)return;this.routes=h.result(this,"routes");var t,e=h.keys(this.routes);while((t=e.pop())!=null){this.route(t,this.routes[t])}},_routeToRegExp:function(t){t=t.replace(A,"\\$&").replace($,"(?:$1)?").replace(T,function(t,e){return e?t:"([^/]+)"}).replace(H,"(.*?)");return new RegExp("^"+t+"$")},_extractParameters:function(t,e){var i=t.exec(e).slice(1);return h.map(i,function(t){return t?decodeURIComponent(t):null})}});var I=a.History=function(){this.handlers=[];h.bindAll(this,"checkUrl");if(typeof window!=="undefined"){this.location=window.location;this.history=window.history}};var N=/^[#\/]|\s+$/g;var P=/^\/+|\/+$/g;var O=/msie [\w.]+/;var C=/\/$/;I.started=false;h.extend(I.prototype,o,{interval:50,getHash:function(t){var e=(t||this).location.href.match(/#(.*)$/);return e?e[1]:""},getFragment:function(t,e){if(t==null){if(this._hasPushState||!this._wantsHashChange||e){t=this.location.pathname;var i=this.root.replace(C,"");if(!t.indexOf(i))t=t.substr(i.length)}else{t=this.getHash()}}return t.replace(N,"")},start:function(t){if(I.started)throw new Error("Backbone.history has already been started");I.started=true;this.options=h.extend({},{root:"/"},this.options,t);this.root=this.options.root;this._wantsHashChange=this.options.hashChange!==false;this._wantsPushState=!!this.options.pushState;this._hasPushState=!!(this.options.pushState&&this.history&&this.history.pushState);var e=this.getFragment();var i=document.documentMode;var r=O.exec(navigator.userAgent.toLowerCase())&&(!i||i<=7);this.root=("/"+this.root+"/").replace(P,"/");if(r&&this._wantsHashChange){this.iframe=a.$('<iframe src="javascript:0" tabindex="-1" />').hide().appendTo("body")[0].contentWindow;this.navigate(e)}if(this._hasPushState){a.$(window).on("popstate",this.checkUrl)}else if(this._wantsHashChange&&"onhashchange"in window&&!r){a.$(window).on("hashchange",this.checkUrl)}else if(this._wantsHashChange){this._checkUrlInterval=setInterval(this.checkUrl,this.interval)}this.fragment=e;var s=this.location;var n=s.pathname.replace(/[^\/]$/,"$&/")===this.root;if(this._wantsHashChange&&this._wantsPushState&&!this._hasPushState&&!n){this.fragment=this.getFragment(null,true);this.location.replace(this.root+this.location.search+"#"+this.fragment);return true}else if(this._wantsPushState&&this._hasPushState&&n&&s.hash){this.fragment=this.getHash().replace(N,"");this.history.replaceState({},document.title,this.root+this.fragment+s.search)}if(!this.options.silent)return this.loadUrl()},stop:function(){a.$(window).off("popstate",this.checkUrl).off("hashchange",this.checkUrl);clearInterval(this._checkUrlInterval);I.started=false},route:function(t,e){this.handlers.unshift({route:t,callback:e})},checkUrl:function(t){var e=this.getFragment();if(e===this.fragment&&this.iframe){e=this.getFragment(this.getHash(this.iframe))}if(e===this.fragment)return false;if(this.iframe)this.navigate(e);this.loadUrl()||this.loadUrl(this.getHash())},loadUrl:function(t){var e=this.fragment=this.getFragment(t);var i=h.any(this.handlers,function(t){if(t.route.test(e)){t.callback(e);return true}});return i},navigate:function(t,e){if(!I.started)return false;if(!e||e===true)e={trigger:e};t=this.getFragment(t||"");if(this.fragment===t)return;this.fragment=t;var i=this.root+t;if(this._hasPushState){this.history[e.replace?"replaceState":"pushState"]({},document.title,i)}else if(this._wantsHashChange){this._updateHash(this.location,t,e.replace);if(this.iframe&&t!==this.getFragment(this.getHash(this.iframe))){if(!e.replace)this.iframe.document.open().close();this._updateHash(this.iframe.location,t,e.replace)}}else{return this.location.assign(i)}if(e.trigger)this.loadUrl(t)},_updateHash:function(t,e,i){if(i){var r=t.href.replace(/(javascript:|#).*$/,"");t.replace(r+"#"+e)}else{t.hash="#"+e}}});a.history=new I;var j=function(t,e){var i=this;var r;if(t&&h.has(t,"constructor")){r=t.constructor}else{r=function(){return i.apply(this,arguments)}}h.extend(r,i,e);var s=function(){this.constructor=r};s.prototype=i.prototype;r.prototype=new s;if(t)h.extend(r.prototype,t);r.__super__=i.prototype;return r};d.extend=g.extend=S.extend=b.extend=I.extend=j;var U=function(){throw new Error('A "url" property or function must be specified')};var R=function(t,e){var i=e.error;e.error=function(r){if(i)i(t,r,e);t.trigger("error",t,r,e)}}}).call(this);
-/*
-//@ sourceMappingURL=backbone-min.map
-*/
\ No newline at end of file
diff --git a/modules/bibauthorid/lib/bibauthorid_bib_matrix_unit_tests.py b/modules/bibauthorid/lib/bibauthorid_bib_matrix_unit_tests.py
deleted file mode 100644
index eb591da83..000000000
--- a/modules/bibauthorid/lib/bibauthorid_bib_matrix_unit_tests.py
+++ /dev/null
@@ -1,138 +0,0 @@
-# -*- coding: utf-8 -*-
-##
-## This file is part of Invenio.
-## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2014 CERN.
-##
-## Invenio is free software; you can redistribute it and/or
-## modify it under the terms of the GNU General Public License as
-## published by the Free Software Foundation; either version 2 of the
-## License, or (at your option) any later version.
-##
-## Invenio is distributed in the hope that it will be useful, but
-## WITHOUT ANY WARRANTY; without even the implied warranty of
-## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-## General Public License for more details.
-##
-## You should have received a copy of the GNU General Public License
-## along with Invenio; if not, write to the Free Software Foundation, Inc.,
-## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-
-"""Unit tests for the search engine."""
-
-__revision__ = \
- "$Id$"
-
-from invenio.testutils import InvenioTestCase, make_test_suite, \
- run_test_suite, nottest
-
-from invenio.bibauthorid_cluster_set import ClusterSet
-from invenio.bibauthorid_bib_matrix import Bib_matrix
-
-class TestBibMatrix(InvenioTestCase):
-
- def setUp(self):
- """
- Set up an empty bibmatrix and one filled with ten clusters of 10 elements each.
- """
- self.bm = Bib_matrix('testname', storage_dir_override='/tmp/')
- self.css = ClusterSet()
- self.css.clusters = [ClusterSet.Cluster(range(i*10,i*10+10)) for i in range(10)]
- self.css.update_bibs()
- self.bmcs0 = Bib_matrix('testname2', self.css, storage_dir_override='/tmp/')
-
- def tearDown(self):
- self.bm.destroy()
- self.bmcs0.destroy()
-
- def test_resolve_entry_simmetry(self):
- '''
- Bib matrix stores a triangular matrix. Entries should be symmetric.
- '''
- for j in range(100):
- for k in range(100):
- self.assertTrue( self.bmcs0._resolve_entry((j,k))==self.bmcs0._resolve_entry((k,j)) )
-
- def test_resolve_entry_unicity(self):
- '''
- resolve_entry should produce unuque indexes for any couple of values
- '''
- ntests = 30
- testvalues = set((i,j) for i in range(ntests) for j in range(ntests))
- for k in range(ntests):
- for z in range(ntests):
- tvalues = testvalues - set([(k,z)]) - set([(z,k)])
- val = self.bmcs0._resolve_entry((k,z))
- allvalues = set(self.bmcs0._resolve_entry(v) for v in tvalues)
- self.assertFalse( val in allvalues , str(val)+' is in, from '+str((k,z)))
-
- def test_matrix_content(self):
- '''
- The matrix should be simmetric, and values should be preserved
- '''
- for i in range(100):
- for j in range(i+1):
- self.bmcs0[i,j] = (i,j)
-
- for i in range(100):
- for j in range(i+1,100):
- val = self.bmcs0[i,j]
- if i < j:
- k,z = j,i
- else:
- k,z = i,j
- self.assertTrue(val[0] == k)
- self.assertTrue(val[1] == z)
-
- def test_create_empty_matrix(self):
- """
- All elements should be None
- """
- for i in range(9,10):
- for j in range(i*10,i*10+10):
- for k in range(i*10,i*10+10):
- self.assertTrue(self.bmcs0[(j,k)] == None)
-
- @nottest
- def FIXME_1678_test_save_matrix(self):
- '''
- Matrix should save, be loadable, and stay equal to a newly loaded one on the same files
- '''
- self.bmcs0.store()
- loaded = Bib_matrix('testname2', storage_dir_override='/tmp/')
- self.assertTrue(loaded.load())
- bmcs0 = self.bmcs0
- for i in range(100):
- for j in range(100):
- self.assertTrue(bmcs0[i,j] == loaded[i,j])
-
- def test_duplicate_existing(self):
- self.bmcs0.store()
- self.bm.duplicate_existing('testname2','testnameduplicate')
- self.assertTrue(self.bmcs0.load())
- self.assertTrue(self.bm.load())
- bmcs0 = self.bmcs0
- bm = self.bm
- for i in range(100):
- for j in range(100):
- self.assertTrue(bmcs0[i,j] == bm[i,j])
-
- def test_special_items(self):
- self.bmcs0[0,0] = '+'
- self.bmcs0[0,1] = '-'
- self.bmcs0[0,2] = None
- self.assertTrue(self.bmcs0[0,0] == '+')
- self.assertTrue(self.bmcs0[0,1] == '-')
- self.assertTrue(self.bmcs0[0,2] is None)
-
- def test_getitem_numeric(self):
- self.bmcs0[0,0] = '+'
- self.bmcs0[0,1] = '-'
- self.bmcs0[0,2] = None
- self.assertTrue(self.bmcs0.getitem_numeric([0,0])[0] == -2)
- self.assertTrue(self.bmcs0.getitem_numeric([0,1])[0] == -1)
- self.assertTrue(self.bmcs0.getitem_numeric([0,2])[0] == -3)
-
-TEST_SUITE = make_test_suite(TestBibMatrix)
-
-if __name__ == "__main__":
- run_test_suite(TEST_SUITE, warn_user=True)
diff --git a/modules/bibauthorid/lib/bibauthorid_cluster_set_unit_tests.py b/modules/bibauthorid/lib/bibauthorid_cluster_set_unit_tests.py
deleted file mode 100644
index 188632156..000000000
--- a/modules/bibauthorid/lib/bibauthorid_cluster_set_unit_tests.py
+++ /dev/null
@@ -1,63 +0,0 @@
-# -*- coding: utf-8 -*-
-##
-## This file is part of Invenio.
-## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
-##
-## Invenio is free software; you can redistribute it and/or
-## modify it under the terms of the GNU General Public License as
-## published by the Free Software Foundation; either version 2 of the
-## License, or (at your option) any later version.
-##
-## Invenio is distributed in the hope that it will be useful, but
-## WITHOUT ANY WARRANTY; without even the implied warranty of
-## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-## General Public License for more details.
-##
-## You should have received a copy of the GNU General Public License
-## along with Invenio; if not, write to the Free Software Foundation, Inc.,
-## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-
-"""Unit tests for the search engine."""
-
-__revision__ = \
- "$Id$"
-
-from itertools import chain
-
-from invenio.testutils import InvenioTestCase, make_test_suite, run_test_suite
-from invenio.bibauthorid_cluster_set import ClusterSet
-
-class TestCluster(InvenioTestCase):
-
- def setUp(self):
- self.clusters = [ClusterSet.Cluster(range(i*10,i*10+10)) for i in range(3)]
-
- def test_quarrel_hate(self):
- c1 = self.clusters[0]
- c2 = self.clusters[1]
-
- self.assertFalse(c1.hates(c2))
-
- c1.quarrel(c2)
-
- self.assertTrue(c1.hates(c2))
- self.assertTrue(c2.hates(c1))
-
-class TestClusterSet(InvenioTestCase):
-
- def setUp(self):
- self.clusters = [ClusterSet.Cluster(range(i*10,i*10+5)) for i in range(10)]
-
- def test_udate_all_bibs(self):
- c = ClusterSet()
- c.clusters = self.clusters
- c.update_bibs()
-
- self.assertTrue(c.num_all_bibs == 50)
- self.assertTrue( sorted(list((c.all_bibs()))) ==
- list(chain.from_iterable(range(i*10,i*10+5) for i in range(10))))
-
-TEST_SUITE = make_test_suite(TestCluster, TestClusterSet)
-
-if __name__ == "__main__":
- run_test_suite(TEST_SUITE, warn_user=True)
diff --git a/modules/bibauthorid/lib/bibauthorid_dbinterface_old.py b/modules/bibauthorid/lib/bibauthorid_dbinterface_old.py
deleted file mode 100644
index a5609f673..000000000
--- a/modules/bibauthorid/lib/bibauthorid_dbinterface_old.py
+++ /dev/null
@@ -1,3548 +0,0 @@
-# -*- coding: utf-8 -*-
-##
-## This file is part of Invenio.
-## Copyright (C) 2011, 2012, 2013 CERN.
-##
-## Invenio is free software; you can redistribute it and/or
-## modify it under the terms of the GNU General Public License as
-## published by the Free Software Foundation; either version 2 of the
-## License, or (at your option) any later version.
-##
-## Invenio is distributed in the hope that it will be useful, but
-## WITHOUT ANY WARRANTY; without even the implied warranty of
-## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-## General Public License for more details.
-##
-## You should have received a copy of the GNU General Public License
-## along with Invenio; if not, write to the Free Software Foundation, Inc.,
-## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-
-'''
- bibauthorid_bdinterface
- This is the only file in bibauthorid which should
- use the data base. It should have an interface for
- all other files in the module.
-'''
-import invenio.bibauthorid_config as bconfig
-import numpy
-import cPickle
-from cPickle import UnpicklingError
-from invenio.htmlutils import X
-
-import os
-import gc
-
-# python2.4 compatibility
-from invenio.bibauthorid_general_utils import bai_all as all
-
-from itertools import groupby, count, ifilter, chain, imap, repeat
-from operator import itemgetter
-
-from invenio.search_engine import perform_request_search
-from invenio.access_control_engine import acc_authorize_action
-from invenio.config import CFG_SITE_URL
-from invenio.bibauthorid_config import QGRAM_LEN
-
-from invenio.bibauthorid_name_utils import split_name_parts
-from invenio.bibauthorid_name_utils import create_canonical_name
-from invenio.bibauthorid_name_utils import create_normalized_name
-from invenio.bibauthorid_general_utils import bibauthor_print
-from invenio.bibauthorid_general_utils import update_status \
- , update_status_final
-from dbquery import run_sql
-
-try:
- from collections import defaultdict
-except ImportError:
- class defaultdict(dict):
- '''
- Implementation of defaultdict to supply missing collections library in python <= 2.4
- '''
- def __init__(self, default_factory, *args, **kwargs):
- super(defaultdict, self).__init__(*args, **kwargs)
- self.default_factory = default_factory
-
- def __missing__(self, key):
- try:
- self[key] = self.default_factory()
- except TypeError:
- raise KeyError("Missing key %s" % (key,))
- else:
- return self[key]
-
- def __getitem__(self, key):
- try:
- return super(defaultdict, self).__getitem__(key)
- except KeyError:
- return self.__missing__(key)
-
-MARC_100_700_CACHE = None
-
-COLLECT_INSPIRE_ID = bconfig.COLLECT_EXTERNAL_ID_INSPIREID
-
-
-def get_sql_time():
- '''
- Returns the time according to the database. The type is datetime.datetime.
- '''
- return run_sql("select now()")[0][0]
-
-
-def set_personid_row(person_id, tag, value, opt1=None, opt2=None, opt3=None):
- '''
- Inserts data and additional info into aidPERSONIDDATA
- @param person_id:
- @type person_id: int
- @param tag:
- @type tag: string
- @param value:
- @type value: string
- @param opt1:
- @type opt1: int
- @param opt2:
- @type opt2: int
- @param opt3:
- @type opt3: string
- '''
- run_sql("INSERT INTO aidPERSONIDDATA "
- "(`personid`, `tag`, `data`, `opt1`, `opt2`, `opt3`) "
- "VALUES (%s, %s, %s, %s, %s, %s)",
- (person_id, tag, value, opt1, opt2, opt3))
-
-
-def get_personid_row(person_id, tag):
- '''
- Returns all the records associated to a person and a tag.
-
- @param person_id: id of the person to read the attribute from
- @type person_id: int
- @param tag: the tag to read.
- @type tag: string
-
- @return: the data associated with a virtual author
- @rtype: tuple of tuples
- '''
- return run_sql("SELECT data, opt1, opt2, opt3 "
- "data FROM aidPERSONIDDATA "
- "WHERE personid = %s AND tag = %s",
- (person_id, tag))
-
-
-def del_personid_row(tag, person_id=None, value=None):
- '''
- Delete the value associated to the given tag for a certain person.
- Can delete all tags regardless of person_id or value, or restrict the deletion using either of
- both of them.
- @param person_id: ID of the person
- @type person_id: int
- @param tag: tag to be updated
- @type tag: string
- @param value: value to be written for the tag
- @type value: string
- '''
- if person_id:
- if value:
- run_sql("delete from aidPERSONIDDATA where personid=%s and tag=%s and data=%s", (person_id, tag, value,))
- else:
- run_sql("delete from aidPERSONIDDATA where personid=%s and tag=%s", (person_id, tag,))
- else:
- if value:
- run_sql("delete from aidPERSONIDDATA where tag=%s and data=%s", (tag, value,))
- else:
- run_sql("delete from aidPERSONIDDATA where tag=%s", (tag,))
-
-
-def get_all_papers_of_pids(personid_list):
- '''
- Get all papers of authors in a given list and sorts the results
- by bibrefrec.
- @param personid_list: list with the authors.
- @type personid_list: iteratable of integers.
- '''
- if personid_list:
- plist = list_2_SQL_str(personid_list)
- paps = run_sql("select personid, bibref_table, bibref_value, bibrec, flag "
- "from aidPERSONIDPAPERS "
- "where personid in %s "
- % plist)
-
- inner = set(row[1:4] for row in paps if row[4] > -2)
-
- return (x for x in paps if x[1:4] in inner)
-
- return ()
-
-
-def del_person_not_manually_claimed_papers(pid):
- '''
- Deletes papers from a person which have not been manually claimed.
- @param pid:
- @type pid: int
- '''
- run_sql("delete from aidPERSONIDPAPERS "
- "where and (flag <> '-2' and flag <> '2') and personid=%s", (pid,))
-
-def get_personid_from_uid(uid):
- '''
- Returns the personID associated with the provided ui.
- If the personID is already associated with the person the secon parameter is True, false otherwise.
- @param uid: userID
- @type uid: ((int,),)
- '''
- pid = run_sql("select personid from aidPERSONIDDATA where tag=%s and data=%s", ('uid', str(uid[0][0])))
- if len(pid) == 1:
- return (pid[0], True)
- else:
- return ([-1], False)
-
-def get_uid_from_personid(pid):
- '''
- Get the invenio user id associated to a pid if exists.
- @param pid: person_id
- @type pid: int
- '''
- uid = run_sql("select data from aidPERSONIDDATA where tag='uid' and personid = %s", (pid,))
- if uid:
- return uid[0][0]
- else:
- return None
-
-
-def get_new_personid():
- '''
- Get a free personid number
- '''
- pids = (run_sql("select max(personid) from aidPERSONIDDATA")[0][0],
- run_sql("select max(personid) from aidPERSONIDPAPERS")[0][0])
-
- pids = tuple(int(p) for p in pids if p != None)
-
- if len(pids) == 2:
- return max(*pids) + 1
- elif len(pids) == 1:
- return pids[0] + 1
- else:
- return 0
-
-def get_existing_personids(with_papers_only=False):
- '''
- Get a set of existing person_ids.
- @param with_papers_only: if True, returns only ids holding papers discarding ids holding only information in aidPERSONIDDATA
- @type with_papers_only: Bool
- '''
- if not with_papers_only:
- try:
- pids_data = set(map(int, zip(*run_sql("select distinct personid from aidPERSONIDDATA"))[0]))
- except IndexError:
- pids_data = set()
- else:
- pids_data = set()
- try:
- pids_pap = set(map(int, zip(*run_sql("select distinct personid from aidPERSONIDPAPERS"))[0]))
- except IndexError:
- pids_pap = set()
- return pids_data | pids_pap
-
-
-def get_existing_result_clusters():
- '''
- Get existing relult clusters, for private use of Tortoise and merger
- '''
- return run_sql("select distinct personid from aidRESULTS")
-
-
-def create_new_person(uid=-1, uid_is_owner=False):
- '''
- Create a new person. Set the uid as owner if requested.
- @param uid: User id to associate to the newly created person
- @type uid: int
- @param uid_is_owner: If true, the person will hold the uid as owned, otherwise the id is only remembered as the creator
- @type uid_is_owner: bool
- '''
- personid_with_uid = run_sql("select personid from aidPERSONIDDATA where tag=%s and data=%s", ('uid', uid))
-
- if personid_with_uid:
- return personid_with_uid[0][0]
-
- pid = get_new_personid()
- if uid_is_owner:
- set_personid_row(pid, 'uid', str(uid))
- else:
- set_personid_row(pid, 'user-created', str(uid))
- return pid
-
-def create_new_person_from_uid(uid):
- '''
- Commodity stub for create_new_person(...)
- @param uid: user id
- @type uid: int
- '''
- return create_new_person(uid, uid_is_owner=True)
-
-def new_person_from_signature(sig, name=None):
- '''
- Creates a new person from a signature.
- @param sig: signature tuple ([100|700],bibref,bibrec)
- @type sig: tuple
- @param name:
- @type name: string
- '''
- pid = get_new_personid()
- add_signature(sig, name, pid)
- return pid
-
-
-def add_signature(sig, name, pid):
- '''
- Inserts a signature in personid.
- @param sig: signature tuple
- @type sig: tuple
- @param name: name string
- @type name: string
- @param pid: personid to which assign the signature
- @type pid: int
- '''
- if not name:
- name = get_name_by_bibrecref(sig)
- name = create_normalized_name(split_name_parts(name))
-
- run_sql("INSERT INTO aidPERSONIDPAPERS "
- "(personid, bibref_table, bibref_value, bibrec, name) "
- "VALUES (%s, %s, %s, %s, %s)"
- , (pid, str(sig[0]), sig[1], sig[2], name))
-
-def move_signature(sig, pid, force_claimed=False, unclaim=False):
- '''
- Moves a signature to a different person id
- @param sig: signature tuple
- @type sig: tuple
- @param pid: personid
- @type pid: int
- '''
- upd = "update aidPERSONIDPAPERS set personid=%s" % pid
- if unclaim:
- upd += ',flag=0 '
- sel = " where bibref_table like '%s' and bibref_value=%s and bibrec=%s " % sig
-
- sql = upd + sel
- if not force_claimed:
- sql += ' and flag <> 2 and flag <> -2'
-
- run_sql(sql)
-
-def find_conflicts(sig, pid):
- '''
- Helper for merger algorithm, find signature given personid
- @param sig: signature tuple
- @type sig: tuple
- @param pid: personid id
- @type pid: integer
- '''
- return run_sql("select bibref_table, bibref_value, bibrec, flag "
- "from aidPERSONIDPAPERS where "
- "personid = %s and "
- "bibrec = %s and "
- "flag <> -2"
- , (pid, sig[2]))
-
-def update_request_ticket(person_id, tag_data_tuple, ticket_id=None):
- '''
- Creates / updates a request ticket for a personID
- @param: personid int
- @param: tag_data_tuples 'image' of the ticket: (('paper', '700:316,10'), ('owner', 'admin'), ('external_id', 'ticket_18'))
- @return: ticketid
- '''
- # tags: rt_owner (the owner of the ticket, associating the rt_number to the transaction)
- # rt_external_id
- # rt_paper_cornfirm, rt_paper_reject, rt_paper_forget, rt_name, rt_email, rt_whatever
- # flag: rt_number
- if not ticket_id:
- last_id = run_sql("select max(opt1) from aidPERSONIDDATA where personid=%s and tag like %s", (str(person_id), 'rt_%'))[0][0]
-
- if last_id:
- ticket_id = last_id + 1
- else:
- ticket_id = 1
- else:
- delete_request_ticket(person_id, ticket_id)
-
- for d in tag_data_tuple:
- run_sql("insert into aidPERSONIDDATA (personid, tag, data, opt1) "
- "values (%s,%s,%s,%s)",
- (str(person_id), 'rt_' + str(d[0]), str(d[1]), str(ticket_id)))
-
- return ticket_id
-
-
-def delete_request_ticket(person_id, ticket_id=None):
- '''
- Removes a ticket from a person_id.
- If ticket_id is not provider removes all the tickets pending on a person.
- '''
- if ticket_id:
- run_sql("delete from aidPERSONIDDATA where personid=%s and tag like %s and opt1 =%s", (str(person_id), 'rt_%', str(ticket_id)))
- else:
- run_sql("delete from aidPERSONIDDATA where personid=%s and tag like %s", (str(person_id), 'rt_%'))
-
-
-def get_all_personids_by_name(regexpr):
- '''
- Search personids matching SQL expression in the name field
- @param regexpr: string SQL regexp
- @type regexpr: string
- '''
- return run_sql("select personid, name "
- "from aidPERSONIDPAPERS "
- "where name like %s "
- "and flag > -2",
- (regexpr,))
-
-def get_personids_by_canonical_name(target):
- '''
- Find personids by canonical name
- @param target:
- @type target:
- '''
- return run_sql("select personid, data from aidPERSONIDDATA where "
- "tag='canonical_name' and data like %s", (target,))
-
-def get_bibref_modification_status(bibref):
- '''
- Determines if a record attached to a person has been touched by a human
- by checking the flag.
-
- @param pid: The Person ID of the person to check the assignment from
- @type pid: int
- @param bibref: The paper identifier to be checked (e.g. "100:12,144")
- @type bibref: string
-
- returns [bool:human_modified, int:lcul]
- '''
- if not bibref:
- raise ValueError("A bibref is expected!")
-
- head, rec = bibref.split(',')
- table, ref = head.split(':')
- flags = run_sql("SELECT flag, lcul FROM aidPERSONIDPAPERS WHERE "
- "bibref_table = %s and bibref_value = %s and bibrec = %s"
- , (table, ref, rec))
- if flags:
- return flags[0]
- else:
- return (False, 0)
-
-
-def get_canonical_id_from_personid(pid):
- '''
- Finds the person id canonical name (e.g. Ellis_J_R_1)
-
- @param pid
- @type int
-
- @return: sql result of the request
- @rtype: tuple of tuple
- '''
- return run_sql("SELECT data FROM aidPERSONIDDATA WHERE "
- "tag = %s AND personid = %s", ('canonical_name', str(pid)))
-
-
-def get_papers_status(paper):
- '''
- Gets the personID and flag assiciated to papers
- @param papers: list of papers
- @type papers: '100:7531,9024'
- @return: (('data','personID','flag',),)
- @rtype: tuple of tuples
- '''
- head, bibrec = paper.split(',')
- _table, bibref = head.split(':')
-
- rets = run_sql("select PersonID, flag "
- "from aidPERSONIDPAPERS "
- "where bibref_table = %s "
- "and bibref_value = %s "
- "and bibrec = %s"
- % (head, bibrec, bibref))
- return [[paper] + list(x) for x in rets]
-
-
-def get_persons_from_recids(recids, return_alt_names=False,
- return_all_person_papers=False):
- '''
- Helper for search engine indexing. Gives back a dictionary with important info about a person, for example:
- get_persons_from_recids([1], True, True) returns
- ({1: [16591L]},
- {16591L: {'alternatative_names': ['Wong, Yung Chow'],
- 'canonical_id': 'Y.C.Wong.1',
- 'person_records': [275304, 1, 51394, 128250, 311629]}})
-
- @param recids:
- @type recids:
- @param return_alt_names:
- @type return_alt_names:
- @param return_all_person_papers:
- @type return_all_person_papers:
- '''
- rec_2_pid = dict()
- pid_2_data = dict()
- all_pids = set()
-
- def get_canonical_name(pid):
- return run_sql("SELECT data "
- "FROM aidPERSONIDDATA "
- "WHERE tag = %s "
- "AND personid = %s",
- ('canonical_name', pid))
-
- for rec in recids:
- pids = run_sql("SELECT personid "
- "FROM aidPERSONIDPAPERS "
- "WHERE bibrec = %s "
- " and flag > -2 ",
- (rec,))
-
- # for some reason python's set is faster than a mysql distinct
- pids = set(p[0] for p in pids)
- all_pids |= pids
- rec_2_pid[rec] = list(pids)
-
- for pid in all_pids:
- pid_data = {}
-
- canonical = get_canonical_name(pid)
- # We can supposed that this person didn't have a chance to get a canonical name yet
- # because it was not fully processed by it's creator. Anyway it's safe to try to create one
- # before failing miserably
- if not canonical:
- update_personID_canonical_names([pid])
- canonical = get_canonical_name(pid)
-
- # assert len(canonical) == 1
- # This condition cannot hold in case claims or update daemons are run in parallel
- # with this, as it can happen that a person with papers exists for wich a canonical name
- # has not been computed yet. Hence, it will be indexed next time, so it learns.
- # Each person should have at most one canonical name, so:
- assert len(canonical) <= 1, "A person cannot have more than one canonical name"
-
- if len(canonical) == 1:
- pid_data = {'canonical_id' : canonical[0][0]}
-
- if return_alt_names:
- names = run_sql("SELECT name "
- "FROM aidPERSONIDPAPERS "
- "WHERE personid = %s "
- " and flag > -2 ",
- (pid,))
- names = set(n[0] for n in names)
-
- pid_data['alternatative_names'] = list(names)
-
- if return_all_person_papers:
- recs = run_sql("SELECT bibrec "
- "FROM aidPERSONIDPAPERS "
- "WHERE personid = %s "
- " and flag > -2 ",
- (pid,))
- recs = set(r[0] for r in recs)
-
- pid_data['person_records'] = list(recs)
-
- pid_2_data[pid] = pid_data
-
- return (rec_2_pid, pid_2_data)
-
-
-def get_person_db_names_count(pid, sort_by_count=True):
- '''
- Returns the set of name strings and count associated to a person id.
- The name strings are as found in the database.
- @param pid: ID of the person
- @type pid: ('2',)
- '''
-
- id_2_count = run_sql("select bibref_table, bibref_value "
- "from aidPERSONIDPAPERS "
- "where personid = %s "
- "and flag > -2", (pid,))
-
- ref100 = [refid[1] for refid in id_2_count if refid[0] == '100']
- ref700 = [refid[1] for refid in id_2_count if refid[0] == '700']
-
- ref100_count = dict((key, len(list(data))) for key, data in groupby(sorted(ref100)))
- ref700_count = dict((key, len(list(data))) for key, data in groupby(sorted(ref700)))
-
- if ref100:
- ref100_s = list_2_SQL_str(ref100, str)
- id100_2_str = run_sql("select id, value "
- "from bib10x "
- "where id in %s"
- % ref100_s)
- else:
- id100_2_str = tuple()
-
- if ref700:
- ref700_s = list_2_SQL_str(ref700, str)
- id700_2_str = run_sql("select id, value "
- "from bib70x "
- "where id in %s"
- % ref700_s)
- else:
- id700_2_str = tuple()
-
- ret100 = [(name, ref100_count[refid]) for refid, name in id100_2_str]
- ret700 = [(name, ref700_count[refid]) for refid, name in id700_2_str]
-
- ret = ret100 + ret700
- if sort_by_count:
- ret = sorted(ret, key=itemgetter(1), reverse=True)
- return ret
-
-
-def get_person_id_from_canonical_id(canonical_id):
- '''
- Finds the person id from a canonical name (e.g. Ellis_J_R_1)
-
- @param canonical_id: the canonical ID
- @type canonical_id: string
-
- @return: sql result of the request
- @rtype: tuple of tuple
- '''
- return run_sql("SELECT personid FROM aidPERSONIDDATA WHERE "
- "tag='canonical_name' AND data = %s", (canonical_id,))
-
-
-#def get_person_names_count(pid):
-# '''
-# Returns the set of name strings and count associated to a person id
-# @param pid: ID of the person
-# @type pid: ('2',)
-# '''
-# return run_sql("select name, count(name) from aidPERSONIDPAPERS where "
-# "personid=%s and flag > -2 group by name", (pid,))
-
-### After testing it seems that on average the above query is slower than the function below ###
-
-def get_person_names_count(pid):
- '''
- Returns the set of name strings and count associated to a person id
- @param pid: ID of the person
- @type pid: ('2',)
- '''
- res = run_sql("select name from aidPERSONIDPAPERS where "
- "personid=%s and flag > -2", (pid,))
- reslist = [x[0] for x in res]
- names_count = defaultdict(int)
- for name in reslist:
- names_count[name]+=1
-
- return names_count.items()
-
-
-def get_person_db_names_set(pid):
- '''
- Returns the set of db_name strings associated to a person id
- @param pid: ID of the person
- @type pid: 2
- '''
-
- names = get_person_db_names_count(pid)
- if names:
- return zip(set(zip(*names)[0]))
- else:
- return []
-
-def get_personids_from_bibrec(bibrec):
- '''
- Returns all the personids associated to a bibrec.
- @param bibrec: record id
- @type bibrec: int
- '''
-
- pids = run_sql("select personid from aidPERSONIDPAPERS where bibrec=%s and flag > -2", (bibrec,))
-
- if pids:
- return set((x[0] for x in pids))
- else:
- return set()
-
-def get_personids_and_papers_from_bibrecs(bibrecs, limit_by_name=None):
- '''
- Gives back a list of tuples (personid, set_of_papers_owned_by) limited to the given list of bibrecs.
- @param bibrecs:
- @type bibrecs:
- @param limit_by_name:
- @type limit_by_name:
- '''
- if not bibrecs:
- return []
- else:
- bibrecs = list_2_SQL_str(bibrecs)
- if limit_by_name:
- try:
- surname = split_name_parts(limit_by_name)[0]
- except IndexError:
- surname = None
- else:
- surname = None
- if not surname:
- data = run_sql("select personid,bibrec from aidPERSONIDPAPERS where bibrec in %s" % (bibrecs,))
- else:
- surname = split_name_parts(limit_by_name)[0]
- data = run_sql(("select personid,bibrec from aidPERSONIDPAPERS where bibrec in %s "
- "and name like " % bibrecs) + ' %s ', (surname + '%',))
- pidlist = [(k, set([s[1] for s in d]))
- for k, d in groupby(sorted(data, key=lambda x:x[0]), key=lambda x:x[0])]
- pidlist = sorted(pidlist, key=lambda x:len(x[1]), reverse=True)
- return pidlist
-
-def get_person_bibrecs(pid):
- '''
- Returns bibrecs associated with a personid
- @param pid: integer personid
- @return [bibrec1,...,bibrecN]
- '''
- papers = run_sql("select bibrec from aidPERSONIDPAPERS where personid=%s and flag > -2", (str(pid),))
- if papers:
- return list(set(zip(*papers)[0]))
- else:
- return []
-
-def get_person_papers(pid, flag,
- show_author_name=False,
- show_title=False,
- show_rt_status=False,
- show_affiliations=False,
- show_date=False,
- show_experiment=False):
- '''
- Get all papers of person with flag greater than flag. Gives back a dictionary like:
- get_person_papers(16591,-2,True,True,True,True,True,True) returns
- [{'affiliation': ['Hong Kong U.'],
- 'authorname': 'Wong, Yung Chow',
- 'data': '100:1,1',
- 'date': ('1961',),
- 'experiment': [],
- 'flag': 0,
- 'rt_status': False,
- 'title': ('Isoclinic N planes in Euclidean 2N space, Clifford parallels in elliptic (2N-1) space, and the Hurwitz matrix equations',)},
- ...]
- @param pid:
- @type pid:
- @param flag:
- @type flag:
- @param show_author_name:
- @type show_author_name:
- @param show_title:
- @type show_title:
- @param show_rt_status:
- @type show_rt_status:
- @param show_affiliations:
- @type show_affiliations:
- @param show_date:
- @type show_date:
- @param show_experiment:
- @type show_experiment:
- '''
- query = "bibref_table, bibref_value, bibrec, flag"
- if show_author_name:
- query += ", name"
-
- all_papers = run_sql("SELECT " + query + " "
- "FROM aidPERSONIDPAPERS "
- "WHERE personid = %s "
- "AND flag >= %s",
- (pid, flag))
-
- def format_paper(paper):
- bibrefrec = "%s:%d,%d" % paper[:3]
- ret = {'data' : bibrefrec,
- 'flag' : paper[3]
- }
-
- if show_author_name:
- ret['authorname'] = paper[4]
-
- if show_title:
- ret['title'] = ""
- title = get_title_from_rec(paper[2])
- if title:
- ret['title'] = (title,)
-
- if show_rt_status:
- rt_count = run_sql("SELECT count(personid) "
- "FROM aidPERSONIDDATA WHERE "
- "tag like 'rt_%%' and data = %s"
- , (bibrefrec,))
-
- ret['rt_status'] = (rt_count[0][0] > 0)
-
- if show_affiliations:
- tag = '%s__u' % paper[0]
- ret['affiliation'] = get_grouped_records(paper[:3], tag)[tag]
-
- if show_date:
- ret['date'] = []
- date_id = run_sql("SELECT id_bibxxx "
- "FROM bibrec_bib26x "
- "WHERE id_bibrec = %s "
- , (paper[2],))
-
- if date_id:
- date_id_s = list_2_SQL_str(date_id, lambda x: x[0])
- date = run_sql("SELECT value "
- "FROM bib26x "
- "WHERE id in %s "
- "AND tag = %s"
- % (date_id_s, "'269__c'"))
- if date:
- ret['date'] = zip(*date)[0]
-
-
- if show_experiment:
- ret['experiment'] = []
- experiment_id = run_sql("SELECT id_bibxxx "
- "FROM bibrec_bib69x "
- "WHERE id_bibrec = %s "
- , (paper[2],))
-
- if experiment_id:
- experiment_id_s = list_2_SQL_str(experiment_id, lambda x: x[0])
- experiment = run_sql("SELECT value "
- "FROM bib69x "
- "WHERE id in %s "
- "AND tag = %s"
- % (experiment_id_s, "'693__e'"))
- if experiment:
- ret['experiment'] = zip(*experiment)[0]
-
- return ret
-
- return [format_paper(paper) for paper in all_papers]
-
-
-def get_persons_with_open_tickets_list():
- '''
- Finds all the persons with open tickets and returns pids and count of tickets
- @return: [[pid, ticket_count]]
- '''
- return run_sql("select personid, count(distinct opt1) from "
- "aidPERSONIDDATA where tag like 'rt_%' group by personid")
-
-
-def get_request_ticket(person_id, ticket_id=None):
- '''
- Retrieves one or many requests tickets from a person
- @param: person_id: person id integer
- @param: matching: couple of values to match ('tag', 'value')
- @param: ticket_id: ticket id (flag) value
- @returns: [[[('tag', 'value')], ticket_id]]
- [[[('a', 'va'), ('b', 'vb')], 1L], [[('b', 'daOEIaoe'), ('a', 'caaoOUIe')], 2L]]
- '''
- if ticket_id:
- tstr = " and opt1='%s' " % ticket_id
- else:
- tstr = " "
- tickets = run_sql("select tag,data,opt1 from aidPERSONIDDATA where personid=%s and "
- " tag like 'rt_%%' " + tstr , (person_id,))
- return [[[(s[0][3:], s[1]) for s in d], k] for k, d in groupby(sorted(tickets, key=lambda k: k[2]), key=lambda k: k[2])]
-
-
-def get_validated_request_ticket(person_id, ticket_id=None):
- '''
- Validates request tickets before returning them.
- '''
- tickets = get_request_ticket(person_id, ticket_id)
- for ticket in list(tickets):
- for entry in list(ticket[0]):
- # those should be the only possible actions in a ticket!
- if entry[0] == 'reject' or entry[0] == 'assign':
- try:
- bibref, bibrec = entry[1].split(',')
- tab, val = bibref.split(':')
- sig = (int(tab), int(val), int(bibrec))
- present = bool(run_sql("select * from aidPERSONIDPAPERS where bibref_table like %s and bibref_value = %s and bibrec = %s ", sig))
- if not present:
- ticket[0].remove(entry)
- # No matter what goes wrong, that's an invalid entry in the ticket. let's discard it.
- except:
- ticket[0].remove(entry)
-
- for ticket in list(tickets):
- tags = [x[0] for x in ticket[0]]
- if 'reject' not in tags and 'assign' not in tags:
- tickets.remove(ticket)
-
- return tickets
-
-
-def insert_user_log(userinfo, personid, action, tag, value, comment='', transactionid=0, timestamp=None, userid=0):
- '''
- Instert log entries in the user log table.
- For example of entres look at the table generation script.
- @param userinfo: username or user identifier
- @type: string
- @param personid: personid involved in the transaction
- @type: longint
- @param action: action type
- @type: string
- @param tag: tag
- @type: string
- @param value: value for the transaction
- @type: string
- @param comment: optional comment for the transaction
- @type: string
- @param transactionid: optional id for the transaction
- @type: longint
-
- @return: the transactionid
- @rtype: longint
- '''
- if not timestamp:
- timestamp = run_sql('select now()')[0][0]
-
- run_sql('insert into aidUSERINPUTLOG '
- '(transactionid,timestamp,userinfo,userid,personid,action,tag,value,comment) values '
- '(%s,%s,%s,%s,%s,%s,%s,%s,%s)',
- (transactionid, timestamp, userinfo, userid, personid,
- action, tag, value, comment))
-
- return transactionid
-
-
-def person_bibref_is_touched_old(pid, bibref):
- '''
- Determines if a record attached to a person has been touched by a human
- by checking the flag.
-
- @param pid: The Person ID of the person to check the assignment from
- @type pid: int
- @param bibref: The paper identifier to be checked (e.g. "100:12,144")
- @type bibref: string
- '''
- bibref, rec = bibref.split(",")
- table, ref = bibref.split(":")
-
- flag = run_sql("SELECT flag "
- "FROM aidPERSONIDPAPERS "
- "WHERE personid = %s "
- "AND bibref_table = %s "
- "AND bibref_value = %s "
- "AND bibrec = %s"
- , (pid, table, ref, rec))
-
- try:
- flag = flag[0][0]
- except (IndexError):
- return False
-
- if not flag:
- return False
- elif -2 < flag < 2:
- return False
- else:
- return True
-
-
-def confirm_papers_to_person(pid, papers, user_level=0):
- '''
- Confirms the relationship between pid and paper, as from user input.
- @param pid: id of the person
- @type pid: integer
- @param papers: list of papers to confirm
- @type papers: ((str,),) e.g. (('100:7531,9024',),)
- @return: list of tuples: (status, message_key)
- @rtype: [(bool, str), ]
- '''
- pids_to_update = set([pid])
- res = []
-
- for p in papers:
- bibref, rec = p.split(",")
- rec = int(rec)
- table, ref = bibref.split(":")
- ref = int(ref)
- sig = (table, ref, rec)
-
- # Check the status of pid: the paper should be present, either assigned or rejected
- gen_papers = run_sql("select bibref_table, bibref_value, bibrec, personid, flag, name "
- "from aidPERSONIDPAPERS "
- "where bibrec=%s "
- "and flag >= -2"
- , (rec,))
-
- paps = [el[0:3] for el in gen_papers if el[3] == pid and el[4] > -2]
- # run_sql("select bibref_table, bibref_value, bibrec "
- # "from aidPERSONIDPAPERS "
- # "where personid=%s "
- # "and bibrec=%s "
- # "and flag > -2"
- # , (pid, rec))
-
- other_paps = [el[0:3] for el in gen_papers if el[3] != pid and el[4] > -2]
- # other_paps = run_sql("select bibref_table, bibref_value, bibrec "
- # "from aidPERSONIDPAPERS "
- # "where personid <> %s "
- # "and bibrec=%s "
- # "and flag > -2"
- # , (pid, rec))
-
- rej_paps = [el[0:3] for el in gen_papers if el[3] == pid and el[4] == -2]
- # rej_paps = run_sql("select bibref_table, bibref_value, bibrec "
- # "from aidPERSONIDPAPERS "
- # "where personid=%s "
- # "and bibrec=%s "
- # "and flag = -2"
- # , (pid, rec))
-
- bibref_exists = [el[0:3] for el in gen_papers if el[0] == table and el[1] == ref and el[4] > -2]
- # bibref_exists = run_sql("select * "
- # "from aidPERSONIDPAPERS "
- # "and bibref_table=%s "
- # "and bibref_value=%s "
- # "and bibrec=%s "
- # "and flag > -2"
- # , (table, ref, rec))
-
-
- # All papers that are being claimed should be present in aidPERSONIDPAPERS, thus:
- # assert paps or rej_paps or other_paps, 'There should be at least something regarding this bibrec!'
- # should always be valid.
- # BUT, it usually happens that claims get done out of the browser/session cache which is hours/days old,
- # hence it happens that papers are claimed which no longer exists in the system.
- # For the sake of mental sanity, instead of crashing from now on we just ignore such cases.
- if not (paps or other_paps or rej_paps) or not bibref_exists:
- res.append((False, 'confirm_failure'))
- continue
- res.append((True, 'confirm_success'))
-
- # It should not happen that a paper is assigned more then once to the same person.
- # But sometimes it happens in rare unfortunate cases of bad concurrency circumstances,
- # so we try to fix it directly instead of crashing here.
- # Once a better solution for dealing with concurrency will be found, the following asserts
- # shall be reenabled, to allow better control on what happens.
-
- # assert len(paps) < 2, "This paper should not be assigned to this person more then once! %s" % paps
- # assert len(other_paps) < 2, "There should not be more then one copy of this paper! %s" % other_paps
-
- # if the bibrec is present with a different bibref, the present one must be moved somwhere
- # else before we can claim the incoming one
- if paps:
- for pap in paps:
- # kick out all unwanted signatures
- if sig != pap:
- new_pid = get_new_personid()
- pids_to_update.add(new_pid)
- move_signature(pap, new_pid)
-
- # Make sure that the incoming claim is unique and get rid of all rejections, they are useless
- # from now on
- run_sql("delete from aidPERSONIDPAPERS where bibref_table like %s and "
- " bibref_value = %s and bibrec=%s"
- , sig)
-
- add_signature(sig, None, pid)
- run_sql("update aidPERSONIDPAPERS "
- "set personid = %s "
- ", flag = %s "
- ", lcul = %s "
- "where bibref_table = %s "
- "and bibref_value = %s "
- "and bibrec = %s"
- , (pid, '2', user_level,
- table, ref, rec))
-
- update_personID_canonical_names(pids_to_update)
- return res
-
-
-def reject_papers_from_person(pid, papers, user_level=0):
- '''
- Confirms the negative relationship between pid and paper, as from user input.
- @param pid: id of the person
- @type pid: integer
- @param papers: list of papers to confirm
- @type papers: ((str,),) e.g. (('100:7531,9024',),)
- @return: list of tuples: (status, message_key)
- @rtype: [(bool, str), ]
- '''
-
- new_pid = get_new_personid()
- pids_to_update = set([pid])
- res = []
-
- for p in papers:
- brr, rec = p.split(",")
- table, ref = brr.split(':')
-
- sig = (table, ref, rec)
- # To be rejected, a record should be present!
- records = personid_name_from_signature(sig)
- # For the sake of mental sanity (see commentis in confirm_papers_to_personid, just ignore in case this paper is no longer existent
- # assert(records)
- if not records:
- res.append((False, 'reject_failure'))
- continue
- res.append((True, 'reject_success'))
-
- fpid, name = records[0]
- # If the record is assigned to a different person already, the rejection is meaningless
- # Otherwise, we assign the paper to someone else (not important who it will eventually
- # get moved by tortoise) and add the rejection to the current person
-
- if fpid == pid:
- move_signature(sig, new_pid, force_claimed=True, unclaim=True)
- pids_to_update.add(new_pid)
- run_sql("INSERT INTO aidPERSONIDPAPERS "
- "(personid, bibref_table, bibref_value, bibrec, name, flag, lcul) "
- "VALUES (%s, %s, %s, %s, %s, %s, %s)"
- , (pid, table, ref, rec, name, -2, user_level))
-
- update_personID_canonical_names(pids_to_update)
-
- return res
-
-
-def reset_papers_flag(pid, papers):
- '''
- Resets the flag associated to the papers to '0'
- @param pid: id of the person
- @type pid: integer
- @param papers: list of papers to confirm
- @type papers: ((str,),) e.g. (('100:7531,9024',),)
- @return: list of tuples: (status, message_key)
- @rtype: [(bool, str), ]
- '''
- res = []
- for p in papers:
- bibref, rec = p.split(",")
- table, ref = bibref.split(":")
- ref = int(ref)
- sig = (table, ref, rec)
- gen_papers = run_sql("select bibref_table, bibref_value, bibrec, flag "
- "from aidPERSONIDPAPERS "
- "where bibrec=%s "
- "and personid=%s"
- , (rec, pid))
-
- paps = [el[0:3] for el in gen_papers]
- # run_sql("select bibref_table, bibref_value, bibrec "
- # "from aidPERSONIDPAPERS "
- # "where personid=%s "
- # "and bibrec=%s "
- # , (pid, rec))
-
- rej_paps = [el[0:3] for el in gen_papers if el[3] == -2]
- # rej_paps = run_sql("select bibref_table, bibref_value, bibrec "
- # "from aidPERSONIDPAPERS "
- # "where personid=%s "
- # "and bibrec=%s "
- # "and flag = -2"
- # , (pid, rec))
-
- pid_bibref_exists = [el[0:3] for el in gen_papers if el[0] == table and el[1] == ref and el[3] > -2]
- # bibref_exists = run_sql("select * "
- # "from aidPERSONIDPAPERS "
- # "and bibref_table=%s "
- # "and bibref_value=%s "
- # "and personid=%s "
- # "and bibrec=%s "
- # "and flag > -2"
- # , (table, ref, pid, rec))
-
- # again, see confirm_papers_to_person for the sake of mental sanity
- # assert paps or rej_paps
- if rej_paps or not pid_bibref_exists:
- res.append((False, 'reset_failure'))
- continue
- res.append((True, 'reset_success'))
- assert len(paps) < 2
-
- run_sql("delete from aidPERSONIDPAPERS where bibref_table like %s and "
- "bibref_value = %s and bibrec = %s",
- (sig))
- add_signature(sig, None, pid)
-
- return res
-
-
-def user_can_modify_data(uid, pid):
- '''
- Return True if the uid can modify data of this personID, false otherwise.
- @param uid: the user id
- @type: int
- @param pid: the person id
- @type: int
-
- @return: can user mofidfy data?
- @rtype: boolean
- '''
-
- pid_uid = run_sql("select data from aidPERSONIDDATA where tag = %s"
- " and personid = %s", ('uid', str(pid)))
-
- if len(pid_uid) >= 1 and str(uid) == str(pid_uid[0][0]):
- rights = bconfig.CLAIMPAPER_CHANGE_OWN_DATA
- else:
- rights = bconfig.CLAIMPAPER_CHANGE_OTHERS_DATA
-
- return acc_authorize_action(uid, rights)[0] == 0
-
-
-def get_possible_bibrecref(names, bibrec, always_match=False):
- '''
- Returns a list of bibrefs for which the surname is matching
- @param names: list of names strings
- @param bibrec: bibrec number
- @param always_match: match with all the names (full bibrefs list)
- '''
- splitted_names = [split_name_parts(n) for n in names]
-
- bibrec_names_100 = run_sql("select o.id, o.value from bib10x o, "
- "(select i.id_bibxxx as iid from bibrec_bib10x i "
- "where id_bibrec=%s) as dummy "
- "where o.tag='100__a' AND o.id = dummy.iid",
- (str(bibrec),))
- bibrec_names_700 = run_sql("select o.id, o.value from bib70x o, "
- "(select i.id_bibxxx as iid from bibrec_bib70x i "
- "where id_bibrec=%s) as dummy "
- "where o.tag='700__a' AND o.id = dummy.iid",
- (str(bibrec),))
-# bibrec_names_100 = run_sql("select id,value from bib10x where tag='100__a' and id in "
-# "(select id_bibxxx from bibrec_bib10x where id_bibrec=%s)",
-# (str(bibrec),))
-# bibrec_names_700 = run_sql("select id,value from bib70x where tag='700__a' and id in "
-# "(select id_bibxxx from bibrec_bib70x where id_bibrec=%s)",
-# (str(bibrec),))
- bibreflist = []
-
- for b in bibrec_names_100:
- spb = split_name_parts(b[1])
- for n in splitted_names:
- if (n[0].lower() == spb[0].lower()) or always_match:
- if ['100:' + str(b[0]), b[1]] not in bibreflist:
- bibreflist.append(['100:' + str(b[0]), b[1]])
-
- for b in bibrec_names_700:
- spb = split_name_parts(b[1])
- for n in splitted_names:
- if (n[0].lower() == spb[0].lower()) or always_match:
- if ['700:' + str(b[0]), b[1]] not in bibreflist:
- bibreflist.append(['700:' + str(b[0]), b[1]])
-
- return bibreflist
-
-
-def user_can_modify_paper(uid, paper):
- '''
- Return True if the uid can modify this paper, false otherwise.
- If the paper is assigned more then one time (from algorithms) consider the most privileged
- assignment.
- @param uid: the user id
- @type: int
- @param paper: the paper bibref,bibrec pair x00:1234,4321
- @type: str
-
- @return: can user mofidfy paper attribution?
- @rtype: boolean
- '''
- bibref, rec = paper.split(",")
- table, ref = bibref.split(":")
- prow = run_sql("select personid, lcul from aidPERSONIDPAPERS "
- "where bibref_table = %s and bibref_value = %s and bibrec = %s "
- "order by lcul desc limit 0,1",
- (table, ref, rec))
-
- if len(prow) == 0:
- return ((acc_authorize_action(uid, bconfig.CLAIMPAPER_CLAIM_OWN_PAPERS)[0] == 0) or
- (acc_authorize_action(uid, bconfig.CLAIMPAPER_CLAIM_OTHERS_PAPERS)[0] == 0))
-
- min_req_acc_n = int(prow[0][1])
- req_acc = resolve_paper_access_right(bconfig.CLAIMPAPER_CLAIM_OWN_PAPERS)
- pid_uid = run_sql("select data from aidPERSONIDDATA where tag = %s and personid = %s", ('uid', str(prow[0][0])))
- if len(pid_uid) > 0:
- if (str(pid_uid[0][0]) != str(uid)) and min_req_acc_n > 0:
- req_acc = resolve_paper_access_right(bconfig.CLAIMPAPER_CLAIM_OTHERS_PAPERS)
-
- if min_req_acc_n < req_acc:
- min_req_acc_n = req_acc
-
- min_req_acc = resolve_paper_access_right(min_req_acc_n)
-
- return (acc_authorize_action(uid, min_req_acc)[0] == 0) and (resolve_paper_access_right(min_req_acc) >= min_req_acc_n)
-
-
-def resolve_paper_access_right(acc):
- '''
- Given a string or an integer, resolves to the corresponding integer or string
- If asked for a wrong/not present parameter falls back to the minimum privilege.
- '''
- access_dict = {bconfig.CLAIMPAPER_VIEW_PID_UNIVERSE: 0,
- bconfig.CLAIMPAPER_CLAIM_OWN_PAPERS: 25,
- bconfig.CLAIMPAPER_CLAIM_OTHERS_PAPERS: 50}
-
- if isinstance(acc, str):
- try:
- return access_dict[acc]
- except:
- return 0
-
- inverse_dict = dict([[v, k] for k, v in access_dict.items()])
- lower_accs = [a for a in inverse_dict.keys() if a <= acc]
- try:
- return inverse_dict[max(lower_accs)]
- except:
- return bconfig.CLAIMPAPER_VIEW_PID_UNIVERSE
-
-
-def get_recently_modified_record_ids(date):
- '''
- Returns the bibrecs with modification date more recent then date.
- @param date: date
- '''
- touched_papers = frozenset(p[0] for p in run_sql(
- "select id from bibrec "
- "where modification_date >= %s"
- , (date,)))
- return touched_papers & frozenset(get_all_valid_bibrecs())
-
-
-def filter_modified_record_ids(bibrecs, date):
- '''
- Returns the bibrecs with modification date before the date.
- @param date: date
- '''
- return ifilter(
- lambda x: run_sql("select count(*) from bibrec "
- "where id = %s and "
- "modification_date < %s"
- , (x[2], date))[0][0]
- , bibrecs)
-
-
-def get_user_log(transactionid='', userinfo='', userid='', personID='', action='', tag='', value='', comment='', only_most_recent=False):
- '''
- Get user log table entry matching all the given parameters; all of them are optional.
- IF no parameters are given retuns the complete log table
- @param transactionid: id of the transaction
- @param userinfo: user name or identifier
- @param personid: id of the person involved
- @param action: action
- @param tag: tag
- @param value: value
- @param comment: comment
- '''
- sql_query = ('select id,transactionid,timestamp,userinfo,personid,action,tag,value,comment ' +
- 'from aidUSERINPUTLOG where 1 ')
- if transactionid:
- sql_query += ' and transactionid=\'' + str(transactionid) + '\''
- if userinfo:
- sql_query += ' and userinfo=\'' + str(userinfo) + '\''
- if userid:
- sql_query += ' and userid=\'' + str(userid) + '\''
- if personID:
- sql_query += ' and personid=\'' + str(personID) + '\''
- if action:
- sql_query += ' and action=\'' + str(action) + '\''
- if tag:
- sql_query += ' and tag=\'' + str(tag) + '\''
- if value:
- sql_query += ' and value=\'' + str(value) + '\''
- if comment:
- sql_query += ' and comment=\'' + str(comment) + '\''
- if only_most_recent:
- sql_query += ' order by timestamp desc limit 0,1'
- return run_sql(sql_query)
-
-def list_2_SQL_str(items, f=lambda x: x):
- """
- Concatenates all items in items to a sql string using f.
- @param items: a set of items
- @param type items: X
- @param f: a function which transforms each item from items to string
- @param type f: X:->str
- @return: "(x1, x2, x3, ... xn)" for xi in items
- @return type: string
- """
- strs = (str(f(x)) for x in items)
- return "(%s)" % ", ".join(strs)
-
-
-def _get_authors_from_paper_from_db(paper):
- '''
- selects all author bibrefs by a given papers
- '''
- fullbibrefs100 = run_sql("select id_bibxxx from bibrec_bib10x where id_bibrec=%s", (paper,))
- if len(fullbibrefs100) > 0:
- fullbibrefs100str = list_2_SQL_str(fullbibrefs100, lambda x: str(x[0]))
- return run_sql("select id from bib10x where tag='100__a' and id in %s" % (fullbibrefs100str,))
- return tuple()
-
-def _get_authors_from_paper_from_cache(paper):
- '''
- selects all author bibrefs by a given papers
- '''
- try:
- ids = MARC_100_700_CACHE['brb100'][paper]['id'].keys()
- refs = [i for i in ids if '100__a' in MARC_100_700_CACHE['b100'][i][0]]
- except KeyError:
- return tuple()
- return zip(refs)
-
-def get_authors_from_paper(paper):
- if MARC_100_700_CACHE:
- if bconfig.DEBUG_CHECKS:
- assert _get_authors_from_paper_from_cache(paper) == _get_authors_from_paper_from_cache(paper)
- return _get_authors_from_paper_from_cache(paper)
- else:
- return _get_authors_from_paper_from_db(paper)
-
-def _get_coauthors_from_paper_from_db(paper):
- '''
- selects all coauthor bibrefs by a given papers
- '''
- fullbibrefs700 = run_sql("select id_bibxxx from bibrec_bib70x where id_bibrec=%s", (paper,))
- if len(fullbibrefs700) > 0:
- fullbibrefs700str = list_2_SQL_str(fullbibrefs700, lambda x: str(x[0]))
- return run_sql("select id from bib70x where tag='700__a' and id in %s" % (fullbibrefs700str,))
- return tuple()
-
-def _get_coauthors_from_paper_from_cache(paper):
- '''
- selects all author bibrefs by a given papers
- '''
- try:
- ids = MARC_100_700_CACHE['brb700'][paper]['id'].keys()
- refs = [i for i in ids if '700__a' in MARC_100_700_CACHE['b700'][i][0]]
- except KeyError:
- return tuple()
- return zip(refs)
-
-def get_coauthors_from_paper(paper):
- if MARC_100_700_CACHE:
- if bconfig.DEBUG_CHECKS:
- assert _get_coauthors_from_paper_from_cache(paper) == _get_coauthors_from_paper_from_db(paper)
- return _get_coauthors_from_paper_from_cache(paper)
- else:
- return _get_coauthors_from_paper_from_db(paper)
-
-def get_bibrefrec_subset(table, papers, refs):
- table = "bibrec_bib%sx" % str(table)[:-1]
- contents = run_sql("select id_bibrec, id_bibxxx from %s" % table)
- papers = set(papers)
- refs = set(refs)
-
- # yes, there are duplicates and we must set them
- return set(ifilter(lambda x: x[0] in papers and x[1] in refs, contents))
-
-def get_deleted_papers():
- return run_sql("select o.id_bibrec from bibrec_bib98x o, "
- "(select i.id as iid from bib98x i "
- "where value = 'DELETED' "
- "and tag like '980__a') as dummy "
- "where o.id_bibxxx = dummy.iid")
-
-def add_personID_external_id(personid, external_id_str, value):
- run_sql("insert into aidPERSONIDDATA (personid,tag,data) values (%s,%s,%s)",
- (personid, 'extid:%s' % external_id_str, value))
-
-def remove_personID_external_id(personid, external_id_str, value=False):
- if not value:
- run_sql("delete from aidPERSONIDDATA where personid=%s and tag=%s",
- (personid, 'extid:%s' % external_id_str))
- else:
- run_sql("delete from aidPERSONIDDATA where personid=%s and tag=%s and data=%s",
- (personid, 'extid:%s' % external_id_str, value))
-
-def get_personiID_external_ids(personid):
- ids = run_sql("select tag,data from aidPERSONIDDATA where personid=%s and tag like 'extid:%%'",
- (personid,))
-
- extids = {}
- for i in ids:
- id_str = i[0].split(':')[1]
- idd = i[1]
- try:
- extids[id_str].append(idd)
- except KeyError:
- extids[id_str] = [idd]
- return extids
-
-def change_personID_canonical_names(personid_cname_list=None):
- '''
- Changes the existing canonical name of a person with the given one.
- @param: personid_cname_list: list of tuples [(person_id, new_canonical_name),]
- '''
- for idx, pid_cname in enumerate(personid_cname_list):
- person_id, new_canonical_name = pid_cname
- update_status(float(idx) / float(len(personid_cname_list)), "Changing canonical_names...")
- # delete the current canonical name of person_id and the current holder of new_canonical_name
- run_sql("delete from aidPERSONIDDATA where tag=%s and (personid=%s or data=%s)", ('canonical_name', person_id, new_canonical_name))
- run_sql("insert into aidPERSONIDDATA (personid, tag, data) values (%s,%s,%s) ", (person_id, 'canonical_name', new_canonical_name))
- update_status_final("Changing canonical_names finished.")
-
-
-# bibauthorid_maintenance personid update private methods
-def update_personID_canonical_names(persons_list=None, overwrite=False, suggested='', overwrite_not_claimed_only=False):
- '''
- Updates the personID table creating or updating canonical names for persons
- @param: persons_list: persons to consider for the update (('1'),)
- @param: overwrite: if to touch already existing canonical names
- @param: suggested: string to suggest a canonical name for the person
- '''
- if not persons_list and overwrite:
- persons_list = set([x[0] for x in run_sql('select personid from aidPERSONIDPAPERS')])
- elif not persons_list:
- persons_list = set([x[0] for x in run_sql('select personid from aidPERSONIDPAPERS')])
- existing_cnamed_pids = set(
- [x[0] for x in run_sql('select personid from aidPERSONIDDATA where tag=%s',
- ('canonical_name',))])
- persons_list = persons_list - existing_cnamed_pids
-
- for idx, pid in enumerate(persons_list):
- update_status(float(idx) / float(len(persons_list)), "Updating canonical_names...")
- if overwrite_not_claimed_only:
- has_claims = run_sql("select personid from aidPERSONIDPAPERS where personid = %s and flag = 2", (pid,))
- if has_claims:
- continue
- current_canonical = run_sql("select data from aidPERSONIDDATA where "
- "personid=%s and tag=%s", (pid, 'canonical_name'))
-
- if overwrite or len(current_canonical) == 0:
- run_sql("delete from aidPERSONIDDATA where personid=%s and tag=%s",
- (pid, 'canonical_name'))
-
- names = get_person_names_count(pid)
- names = sorted(names, key=lambda k: k[1], reverse=True)
- if len(names) < 1 and not suggested:
- continue
- else:
- if suggested:
- canonical_name = suggested
- else:
- canonical_name = create_canonical_name(names[0][0])
-
- existing_cnames = run_sql("select data from aidPERSONIDDATA "
- "where tag=%s and data like %s",
- ('canonical_name', str(canonical_name) + '%'))
-
- existing_cnames = set(name[0].lower() for name in existing_cnames)
- for i in count(1):
- cur_try = canonical_name + '.' + str(i)
- if cur_try.lower() not in existing_cnames:
- canonical_name = cur_try
- break
-
- run_sql("insert into aidPERSONIDDATA (personid, tag, data) values (%s,%s,%s) ",
- (pid, 'canonical_name', canonical_name))
-
- update_status_final("Updating canonical_names finished.")
-
-
-def personid_get_recids_affected_since(last_timestamp):
- '''
- Returns a list of recids which have been manually changed since timestamp)
- @param: last_timestamp: last update, datetime.datetime
- '''
- vset = set(int(v[0].split(',')[1]) for v in run_sql(
- "select distinct value from aidUSERINPUTLOG "
- "where timestamp > %s", (last_timestamp,))
- if ',' in v[0] and ':' in v[0])
-
- pids = set(int(p[0]) for p in run_sql(
- "select distinct personid from aidUSERINPUTLOG "
- "where timestamp > %s", (last_timestamp,))
- if p[0] > 0)
-
- if pids:
- pids_s = list_2_SQL_str(pids)
- vset |= set(int(b[0]) for b in run_sql(
- "select bibrec from aidPERSONIDPAPERS "
- "where personid in %s" % pids_s))
- # I'm not sure about this cast. It might work without it.
- return list(vset)
-
-
-def get_all_paper_records(pid, claimed_only=False):
- if not claimed_only:
- result = run_sql("SELECT bibrec FROM aidPERSONIDPAPERS WHERE personid = %s", (str(pid),))
- else:
- result = run_sql("SELECT bibrec FROM aidPERSONIDPAPERS WHERE "
- "personid = %s and flag=2 or flag=-2", (str(pid),))
-
- return tuple(set(result))
-
-
-def get_all_modified_names_from_personid(since=None):
- if since:
- all_pids = run_sql("SELECT DISTINCT personid "
- "FROM aidPERSONIDPAPERS "
- "WHERE flag > -2 "
- "AND last_updated > %s"
- % since)
- else:
- all_pids = run_sql("SELECT DISTINCT personid "
- "FROM aidPERSONIDPAPERS "
- "WHERE flag > -2 ")
-
- return ((name[0][0], set(n[1] for n in name), len(name))
- for name in (run_sql(
- "SELECT personid, name "
- "FROM aidPERSONIDPAPERS "
- "WHERE personid = %s "
- "AND flag > -2", p)
- for p in all_pids))
-
-
-def destroy_partial_marc_caches():
- global MARC_100_700_CACHE
- MARC_100_700_CACHE = None
- gc.collect()
-
-def populate_partial_marc_caches():
- global MARC_100_700_CACHE
-
- if MARC_100_700_CACHE:
- return
-
- def br_dictionarize(maptable):
- gc.disable()
- md = defaultdict(dict)
- maxiters = len(set(map(itemgetter(0), maptable)))
- for i, v in enumerate(groupby(maptable, itemgetter(0))):
- if i % 1000 == 0:
- update_status(float(i) / maxiters, 'br_dictionarizing...')
-# if i % 1000000 == 0:
-# update_status(float(i) / maxiters, 'br_dictionarizing...GC')
-# gc.collect()
- idx = defaultdict(list)
- fn = defaultdict(list)
- for _, k, z in v[1]:
- idx[k].append(z)
- fn[z].append(k)
- md[v[0]]['id'] = idx
- md[v[0]]['fn'] = fn
- update_status_final('br_dictionarizing done')
- gc.enable()
- return md
-
- def bib_dictionarize(bibtable):
- return dict((i[0], (i[1], i[2])) for i in bibtable)
-
- update_status(.0, 'Populating get_grouped_records_table_cache')
- bibrec_bib10x = sorted(run_sql("select id_bibrec,id_bibxxx,field_number from bibrec_bib10x"))
- update_status(.125, 'Populating get_grouped_records_table_cache')
- brd_b10x = br_dictionarize(bibrec_bib10x)
- del bibrec_bib10x
-
- update_status(.25, 'Populating get_grouped_records_table_cache')
- bibrec_bib70x = sorted(run_sql("select id_bibrec,id_bibxxx,field_number from bibrec_bib70x"))
- update_status(.375, 'Populating get_grouped_records_table_cache')
- brd_b70x = br_dictionarize(bibrec_bib70x)
- del bibrec_bib70x
-
- update_status(.5, 'Populating get_grouped_records_table_cache')
- bib10x = (run_sql("select id,tag,value from bib10x"))
- update_status(.625, 'Populating get_grouped_records_table_cache')
- bibd_10x = bib_dictionarize(bib10x)
- del bib10x
-
- update_status(.75, 'Populating get_grouped_records_table_cache')
- bib70x = (run_sql("select id,tag,value from bib70x"))
- update_status(.875, 'Populating get_grouped_records_table_cache')
- bibd_70x = bib_dictionarize(bib70x)
- del bib70x
-
- update_status_final('Finished populating get_grouped_records_table_cache')
- MARC_100_700_CACHE = {'brb100':brd_b10x, 'brb700':brd_b70x, 'b100':bibd_10x, 'b700':bibd_70x}
-
-def _get_grouped_records_using_caches(brr, *args):
- try:
- c = MARC_100_700_CACHE['brb%s' % str(brr[0])][brr[2]]
- fn = c['id'][brr[1]]
- except KeyError:
- return dict((arg, []) for arg in args)
- if not fn or len(fn) > 1:
- # if len fn > 1 it's BAD: the same signature is at least twice on the same paper.
- # Let's default to nothing, to be on the safe side.
- return dict((arg, []) for arg in args)
- ids = set(chain(*(c['fn'][i] for i in fn)))
- tuples = [MARC_100_700_CACHE['b%s' % str(brr[0])][i] for i in ids]
- results = {}
- for t in tuples:
- present = [x for x in args if x in t[0]]
- assert len(present) <= 1
- if present:
- arg = present[0]
- try:
- results[arg].append(t[1])
- except KeyError:
- results[arg] = [t[1]]
- for arg in args:
- if arg not in results.keys():
- results[arg] = []
- return results
-
-def _get_grouped_records_from_db(bibrefrec, *args):
- '''
- By a given bibrefrec: mark:ref,rec this function will scan
- bibmarkx table and extract all records with tag in argc, which
- are grouped togerther with this bibrec.
-
- Returns a dictionary with { tag : [extracted_values] }
- if the values is not found.
- @type bibrefrec: (mark(int), ref(int), rec(int))
- '''
- table, ref, rec = bibrefrec
-
- target_table = "bib%sx" % (str(table)[:-1])
- mapping_table = "bibrec_%s" % target_table
-
- group_id = run_sql("SELECT field_number "
- "FROM %s "
- "WHERE id_bibrec = %d "
- "AND id_bibxxx = %d" %
- (mapping_table, rec, ref))
-
- if len(group_id) == 0:
- # unfortunately the mapping is not found, so
- # we cannot find anything
- return dict((arg, []) for arg in args)
- elif len(group_id) == 1:
- # All is fine
- field_number = group_id[0][0]
- else:
- # sounds bad, but ignore the error
- field_number = min(x[0] for x in group_id)
-
- grouped = run_sql("SELECT id_bibxxx "
- "FROM %s "
- "WHERE id_bibrec = %d "
- "AND field_number = %d" %
- (mapping_table, rec, int(field_number)))
- assert len(grouped) > 0, "There should be a most one grouped value per tag."
- grouped_s = list_2_SQL_str(grouped, lambda x: str(x[0]))
-
- ret = {}
- for arg in args:
- qry = run_sql("SELECT value "
- "FROM %s "
- "WHERE tag LIKE '%%%s%%' "
- "AND id IN %s" %
- (target_table, arg, grouped_s))
- ret[arg] = [q[0] for q in qry]
-
- return ret
-
-def get_grouped_records(bibrefrec, *args):
- if MARC_100_700_CACHE:
- if bconfig.DEBUG_CHECKS:
- assert _get_grouped_records_using_caches(bibrefrec, *args) == _get_grouped_records_from_db(bibrefrec, *args)
- return _get_grouped_records_using_caches(bibrefrec, *args)
- else:
- return _get_grouped_records_from_db(bibrefrec, *args)
-
-def get_person_with_extid(idd, match_tag=False):
- if match_tag:
- mtag = " and tag = '%s'" % 'extid:' + match_tag
- else:
- mtag = ''
- pids = run_sql("select personid from aidPERSONIDDATA where data=%s" % '%s' + mtag, (idd,))
- return set(pids)
-
-def get_inspire_id(p):
- '''
- Gets inspire id for a signature (bibref_table,bibref_value.bibrec)
- '''
- return get_grouped_records((str(p[0]), p[1], p[2]), str(p[0]) + '__i').values()[0]
-
-def get_claimed_papers_from_papers(papers):
- '''
- Given a set of papers it returns the subset of claimed papers
- @param papers: set of papers
- @type papers: frozenset
- @return: tuple
- '''
- papers_s = list_2_SQL_str(papers)
- claimed_papers = run_sql("select bibrec from aidPERSONIDPAPERS "
- "where bibrec in %s and flag = 1" % papers_s)
- return claimed_papers
-
-def collect_personID_external_ids_from_papers(personid, limit_to_claimed_papers=False):
- gathered_ids = {}
-
- if limit_to_claimed_papers:
- flag = 1
- else:
- flag = -2
-
- person_papers = run_sql("select bibref_table,bibref_value,bibrec from aidPERSONIDPAPERS where "
- "personid=%s and flag > %s", (personid, flag))
-
- if COLLECT_INSPIRE_ID:
- inspireids = []
- for p in person_papers:
- extid = get_inspire_id(p)
- if extid:
- inspireids.append(extid)
- inspireids = set((i[0] for i in inspireids))
-
- gathered_ids['INSPIREID'] = inspireids
-
-# if COLLECT_ORCID:
-# orcids = []
-# for p in person_papers:
-# extid = get_orcid(p)
-# if extid:
-# orcids.append(extid)
-# orcids = set((i[0] for i in orcids))
-# gathered_ids['ORCID'] = orcids
-
-# if COLLECT_ARXIV_ID:
-# arxivids = []
-# for p in person_papers:
-# extid = get_arxiv_id(p)
-# if extid:
-# arxivids.append(extid)
-# arxivids = set((i[0] for i in arxivids))
-# gathered_ids['ARXIVID'] = arxivids
-
- return gathered_ids
-
-def update_personID_external_ids(persons_list=None, overwrite=False,
- limit_to_claimed_papers=False, force_cache_tables=False):
- if force_cache_tables:
- populate_partial_marc_caches()
-
- if not persons_list:
- persons_list = set([x[0] for x in run_sql('select personid from aidPERSONIDPAPERS')])
-
- for idx, pid in enumerate(persons_list):
- update_status(float(idx) / float(len(persons_list)), "Updating external ids...")
-
- collected = collect_personID_external_ids_from_papers(pid, limit_to_claimed_papers=limit_to_claimed_papers)
- present = get_personiID_external_ids(pid)
-
- if overwrite:
- for idd in present.keys():
- for k in present[idd]:
- remove_personID_external_id(pid, idd, value=k)
- present = {}
-
- for idd in collected.keys():
- for k in collected[idd]:
- if idd not in present or k not in present[idd]:
- add_personID_external_id(pid, idd, k)
-
- if force_cache_tables:
- destroy_partial_marc_caches()
-
- update_status_final("Updating external ids finished.")
-
-def _get_name_by_bibrecref_from_db(bib):
- '''
- @param bib: bibrefrec or bibref
- @type bib: (mark, bibref, bibrec) OR (mark, bibref)
- '''
- table = "bib%sx" % str(bib[0])[:-1]
- refid = bib[1]
- tag = "%s__a" % str(bib[0])
- ret = run_sql("select value from %s where id = '%s' and tag = '%s'" % (table, refid, tag))
-
- assert len(ret) == 1, "A bibrefrec must have exactly one name(%s)" % str(bib)
- return ret[0][0]
-
-def _get_name_by_bibrecref_from_cache(bib):
- '''
- @param bib: bibrefrec or bibref
- @type bib: (mark, bibref, bibrec) OR (mark, bibref)
- '''
- table = "b%s" % bib[0]
- refid = bib[1]
- tag = "%s__a" % str(bib[0])
- ret = None
- try:
- if tag in MARC_100_700_CACHE[table][refid][0]:
- ret = MARC_100_700_CACHE[table][refid][1]
- except (KeyError, IndexError), e:
- # The GC did run and the table is not clean?
- # We might want to allow empty response here
- raise Exception(str(bib) + str(e))
- if bconfig.DEBUG_CHECKS:
- assert ret == _get_name_by_bibrecref_from_db(bib)
- return ret
-
-def get_name_by_bibrecref(bib):
- if MARC_100_700_CACHE:
- if bconfig.DEBUG_CHECKS:
- assert _get_name_by_bibrecref_from_cache(bib) == _get_name_by_bibrecref_from_db(bib)
- return _get_name_by_bibrecref_from_cache(bib)
- else:
- return _get_name_by_bibrecref_from_db(bib)
-
-def get_collaboration(bibrec):
- bibxxx = run_sql("select id_bibxxx from bibrec_bib71x where id_bibrec = %s", (str(bibrec),))
-
- if len(bibxxx) == 0:
- return ()
-
- bibxxx = list_2_SQL_str(bibxxx, lambda x: str(x[0]))
-
- ret = run_sql("select value from bib71x where id in %s and tag like '%s'" % (bibxxx, "710__g"))
- return [r[0] for r in ret]
-
-
-def get_key_words(bibrec):
- if bconfig.CFG_ADS_SITE:
- bibxxx = run_sql("select id_bibxxx from bibrec_bib65x where id_bibrec = %s", (str(bibrec),))
- else:
- bibxxx = run_sql("select id_bibxxx from bibrec_bib69x where id_bibrec = %s", (str(bibrec),))
-
- if len(bibxxx) == 0:
- return ()
-
- bibxxx = list_2_SQL_str(bibxxx, lambda x: str(x[0]))
-
- if bconfig.CFG_ADS_SITE:
- ret = run_sql("select value from bib69x where id in %s and tag like '%s'" % (bibxxx, "6531_a"))
- else:
- ret = run_sql("select value from bib69x where id in %s and tag like '%s'" % (bibxxx, "695__a"))
-
- return [r[0] for r in ret]
-
-
-def get_all_authors(bibrec):
- bibxxx_1 = run_sql("select id_bibxxx from bibrec_bib10x where id_bibrec = %s", (str(bibrec),))
- bibxxx_7 = run_sql("select id_bibxxx from bibrec_bib70x where id_bibrec = %s", (str(bibrec),))
-
- if bibxxx_1:
- bibxxxs_1 = list_2_SQL_str(bibxxx_1, lambda x: str(x[0]))
- authors_1 = run_sql("select value from bib10x where tag = '%s' and id in %s" % ('100__a', bibxxxs_1,))
- else:
- authors_1 = []
-
- if bibxxx_7:
- bibxxxs_7 = list_2_SQL_str(bibxxx_7, lambda x: str(x[0]))
- authors_7 = run_sql("select value from bib70x where tag = '%s' and id in %s" % ('700__a', bibxxxs_7,))
- else:
- authors_7 = []
-
- return [a[0] for a in authors_1] + [a[0] for a in authors_7]
-
-def get_title_from_rec(rec):
- """
- Returns the name of the paper like str if found.
- Otherwise returns None.
- """
- title_id = run_sql("SELECT id_bibxxx "
- "FROM bibrec_bib24x "
- "WHERE id_bibrec = %s",
- (rec,))
-
- if title_id:
- title_id_s = list_2_SQL_str(title_id, lambda x: x[0])
- title = run_sql("SELECT value "
- "FROM bib24x "
- "WHERE id in %s "
- "AND tag = '245__a'"
- % title_id_s)
-
- if title:
- return title[0][0]
-
-def get_bib10x():
- return run_sql("select id, value from bib10x where tag like %s", ("100__a",))
-
-
-def get_bib70x():
- return run_sql("select id, value from bib70x where tag like %s", ("700__a",))
-
-
-class Bib_matrix(object):
- '''
- This small class contains the sparse matrix
- and encapsulates it.
- '''
- # please increment this value every time you
- # change the output of the comparison functions
- current_comparison_version = 10
-
- __special_items = ((None, -3.), ('+', -2.), ('-', -1.))
- special_symbols = dict((x[0], x[1]) for x in __special_items)
- special_numbers = dict((x[1], x[0]) for x in __special_items)
-
- def __init__(self, cluster_set=None):
- if cluster_set:
- self._bibmap = dict((b[1], b[0]) for b in enumerate(cluster_set.all_bibs()))
- width = len(self._bibmap)
- size = ((width - 1) * width) / 2
- self._matrix = Bib_matrix.create_empty_matrix(size)
- else:
- self._bibmap = dict()
-
- self.creation_time = get_sql_time()
-
- @staticmethod
- def create_empty_matrix(lenght):
- ret = numpy.ndarray(shape=(lenght, 2), dtype=float, order='C')
- ret.fill(Bib_matrix.special_symbols[None])
- return ret
-
- def _resolve_entry(self, bibs):
- assert len(bibs) == 2
- first = self._bibmap[bibs[0]]
- second = self._bibmap[bibs[1]]
- if first > second:
- first, second = second, first
- return first + ((second - 1) * second) / 2
-
- def __setitem__(self, bibs, val):
- self._matrix[self._resolve_entry(bibs)] = Bib_matrix.special_symbols.get(val, val)
-
- def __getitem__(self, bibs):
- #without temporary variable it's faster!
- #entry = self._resolve_entry(bibs)
- #ret = tuple(self._matrix[entry])
- #return Bib_matrix.special_numbers.get(ret[0], ret)
- ret = tuple(self._matrix[self._resolve_entry(bibs)])
- return Bib_matrix.special_numbers.get(ret[0], ret)
-
- def __contains__(self, bib):
- return bib in self._bibmap
-
- def get_keys(self):
- return self._bibmap.keys()
-
- @staticmethod
- def get_file_dir(name):
- sub_dir = name[:2]
- if not sub_dir:
- sub_dir = "empty_last_name"
- return "%s%s/" % (bconfig.TORTOISE_FILES_PATH, sub_dir)
-
- @staticmethod
- def get_map_path(dir_path, name):
- return "%s%s.bibmap" % (dir_path, name)
-
- @staticmethod
- def get_matrix_path(dir_path, name):
- return "%s%s.npy" % (dir_path, name)
-
- def load(self, name, load_map=True, load_matrix=True):
- files_dir = Bib_matrix.get_file_dir(name)
-
- if not os.path.isdir(files_dir):
- self._bibmap = dict()
- return False
-
- try:
- if load_map:
- bibmap_v = cPickle.load(open(Bib_matrix.get_map_path(files_dir, name), 'r'))
- rec_v, self.creation_time, self._bibmap = bibmap_v
- if (rec_v != Bib_matrix.current_comparison_version or
- # you can use negative version to recalculate
- Bib_matrix.current_comparison_version < 0):
-
- self._bibmap = dict()
- return False
-
- if load_matrix:
- self._matrix = numpy.load(Bib_matrix.get_matrix_path(files_dir, name))
- except (IOError, UnpicklingError):
- if load_map:
- self._bibmap = dict()
- self.creation_time = get_sql_time()
- return False
- return True
-
- def store(self, name):
- files_dir = Bib_matrix.get_file_dir(name)
-
- if not os.path.isdir(files_dir):
- try:
- os.mkdir(files_dir)
- except OSError, e:
- if e.errno == 17 or 'file exists' in str(e.strerror).lower():
- pass
- else:
- raise e
-
- bibmap_v = (Bib_matrix.current_comparison_version, self.creation_time, self._bibmap)
- cPickle.dump(bibmap_v, open(Bib_matrix.get_map_path(files_dir, name), 'w'))
-
- numpy.save(open(Bib_matrix.get_matrix_path(files_dir, name), "w"), self._matrix)
-
-def delete_paper_from_personid(rec):
- '''
- Deletes all information in PERSONID about a given paper
- '''
- run_sql("delete from aidPERSONIDPAPERS where bibrec = %s", (rec,))
-
-
-def get_signatures_from_rec(bibrec):
- '''
- Retrieves all information in PERSONID
- about a given bibrec.
- '''
- return run_sql("select personid, bibref_table, bibref_value, bibrec, name "
- "from aidPERSONIDPAPERS where bibrec = %s"
- , (bibrec,))
-
-
-def modify_signature(oldref, oldrec, newref, newname):
- '''
- Modifies a signature in aidPERSONIDpapers.
- '''
- return run_sql("UPDATE aidPERSONIDPAPERS "
- "SET bibref_table = %s, bibref_value = %s, name = %s "
- "WHERE bibref_table = %s AND bibref_value = %s AND bibrec = %s"
- , (str(newref[0]), newref[1], newname,
- str(oldref[0]), oldref[1], oldrec))
-
-
-def find_pids_by_name(name):
- '''
- Finds names and personids by a prefix name.
- '''
- return set(run_sql("SELECT personid, name "
- "FROM aidPERSONIDPAPERS "
- "WHERE name like %s"
- , (name + ',%',)))
-
-def find_pids_by_exact_name(name):
- """
- Finds names and personids by a name.
- """
- return set(run_sql("SELECT personid "
- "FROM aidPERSONIDPAPERS "
- "WHERE name = %s"
- , (name,)))
-
-def remove_sigs(signatures):
- '''
- Removes records from aidPERSONIDPAPERS
- '''
- for sig in signatures:
- run_sql("DELETE FROM aidPERSONIDPAPERS "
- "WHERE bibref_table like %s AND bibref_value = %s AND bibrec = %s"
- , (str(sig[0]), sig[1], sig[2]))
-
-
-def remove_personid_papers(pids):
- '''
- Removes all signatures from aidPERSONIDPAPERS with pid in pids
- '''
- if pids:
- run_sql("delete from aidPERSONIDPAPERS where personid in %s"
- % list_2_SQL_str(pids))
-
-
-def get_full_personid_papers(table_name="`aidPERSONIDPAPERS`"):
- '''
- Get all columns and rows from aidPERSONIDPAPERS
- or any other table with the same structure.
- '''
- return run_sql("select personid, bibref_table, "
- "bibref_value, bibrec, name, flag, "
- "lcul from %s" % table_name)
-
-
-def get_full_results():
- '''
- Depricated. Should be removed soon.
- '''
- return run_sql("select personid, bibref_table, bibref_value, bibrec "
- "from aidRESULTS")
-
-
-def get_lastname_results(last_name):
- '''
- Returns rows from aidRESULTS which share a common last name.
- '''
- return run_sql("select personid, bibref_table, bibref_value, bibrec "
- "from aidRESULTS "
- "where personid like '" + last_name + ".%'")
-
-
-def get_full_personid_data(table_name="`aidPERSONIDDATA`"):
- '''
- Get all columns and rows from aidPERSONIDDATA
- or any other table with the same structure.
- '''
- return run_sql("select personid, tag, data, "
- "opt1, opt2, opt3 from %s" % table_name)
-
-
-def get_specific_personid_full_data(pid):
- '''
- Get all columns and rows from aidPERSONIDDATA
- '''
- return run_sql("select personid, tag, data, "
- "opt1, opt2, opt3 from aidPERSONIDDATA where personid=%s "
- , (pid,))
-
-
-def get_canonical_names_by_pid(pid):
- '''
- Get all data that has as a tag canonical_name from aidPERSONIDDATA
- '''
- return run_sql("select data "
- "from aidPERSONIDDATA where personid=%s and tag=%s"
- , (pid, "canonical_name"))
-
-
-def get_orcids_by_pids(pid):
- '''
- Get all data that has as a tag extid:ORCID from aidPERSONIDDATA
- '''
- return run_sql("select data "
- "from aidPERSONIDDATA where personid=%s and tag=%s"
- , (pid, "extid:ORCID"))
-
-
-def get_inspire_ids_by_pids(pid):
- '''
- Get all data that has as a tag extid:INSPIREID from aidPERSONIDDATA
- '''
- return run_sql("select data "
- "from aidPERSONIDDATA where personid=%s and tag=%s"
- , (pid, "extid:INSPIREID"))
-
-def get_uids_by_pids(pid):
- '''
- Get all data that has as a tag uid from aidPERSONIDDATA
- '''
- return run_sql("select data "
- "from aidPERSONIDDATA where personid=%s and tag=%s"
- , (pid, "uid"))
-
-def get_name_string_to_pid_dictionary():
- '''
- Get a dictionary which maps name strigs to person ids
- '''
- namesdict = {}
- all_names = run_sql("select personid,name from aidPERSONIDPAPERS")
- for x in all_names:
- namesdict.setdefault(x[1], set()).add(x[0])
- return namesdict
-
-# could be useful to optimize rabbit, still unused and untested, watch out.
-def get_bibrecref_to_pid_dictuonary():
- brr2pid = {}
- all_brr = run_sql("select personid,bibref_table,bibref_value.bibrec from aidPERSONIDPAPERS")
- for x in all_brr:
- brr2pid.setdefault(tuple(x[1:]), set()).add(x[0])
- return brr2pid
-
-
-def check_personid_papers(output_file=None): ### check_author_paper_associations
- '''
- It examines if there are records in aidPERSONIDPAPERS table which are in an
- impaired state. If 'output_file' is specified it writes the output in that
- file, otherwise in stdout.
-
- @param output_file: file to write output
- @type output_file: str
- @return: damaged records are found
- @rtype: bool
- '''
- if output_file:
- fp = open(output_file, "w")
- printer = lambda x: fp.write(x + '\n')
- else:
- printer = bibauthor_print
-
- checkers = (check_duplicated_lines,
- check_wrong_names,
- check_duplicated_papers,
- check_duplicated_signatures,
- check_wrong_rejection,
- check_canonical_names,
- check_empty_personids
- # check_claim_inspireid_contradiction
- )
- # avoid writing f(a) or g(a), because one of the calls might be optimized
- return all([check(printer) for check in checkers])
-
-
-def repair_personid(output_file=None):
- '''
- This should make check_personid_papers() to return true.
- '''
- if output_file:
- fp = open(output_file, "w")
- printer = lambda x: fp.write(x + '\n')
- else:
- printer = bibauthor_print
-
- checkers = (
- check_duplicated_lines,
- check_wrong_names,
- check_duplicated_papers,
- check_duplicated_signatures,
- check_wrong_rejection,
- check_canonical_names,
- check_empty_personids
- # check_claim_inspireid_contradiction
- )
-
- first_check = [check(printer) for check in checkers]
- repair_pass = [check(printer, repair=True) for check in checkers]
- last_check = [check(printer) for check in checkers]
-
- if not all(first_check):
- assert not(all(repair_pass))
- assert all(last_check)
-
- return all(last_check)
-
-def check_duplicated_lines(printer, repair=False):
- '''
- For some reasons it happens that some lines in the database are duplicated. This is of course to
- be corrected, removing the duplicate line and leaving there only one copy.
- '''
-
- all_ok = True
-
- aid_table = run_sql('select personid, bibref_table, bibref_value, bibrec, name, flag, lcul, last_updated from aidPERSONIDPAPERS')
- aid_table = sorted(aid_table)
- duplicates = set([x for i, x in enumerate(aid_table[0:len(aid_table)-1]) if x == aid_table[i+1]])
-
- if duplicates:
- all_ok = False
- printer("There are duplicated lines in aidPERSONIDPAPERS: ")
- printer(str(duplicates))
-
- if repair:
- fields = ['personid=%s','bibref_table=%s','bibref_value=%s', 'bibrec=%s','name=%s','flag=%s','lcul=%s','last_updated=%s']
- for l in duplicates:
- used_fields = list()
- data = list()
- for i,f in enumerate(l):
- if f:
- used_fields.append(fields[i])
- data.append(f)
- run_sql("delete from aidPERSONIDPAPERS where "+' and '.join(used_fields), data)
- run_sql("insert into aidPERSONIDPAPERS (personid, bibref_table, bibref_value, bibrec, name, flag, lcul, last_updated) "
- "value (%s,%s,%s,%s,%s,%s,%s,%s)", l)
-
- aid_table = run_sql('select personid, tag, data, opt1, opt2, opt3, last_updated from aidPERSONIDDATA')
- aid_table = sorted(aid_table)
- duplicates = set([x for i, x in enumerate(aid_table[0:len(aid_table)-1]) if x == aid_table[i+1]])
-
- if duplicates:
- all_ok = False
- printer("There are duplicated lines in aidPERSONIDDATA: ")
- printer(str(duplicates))
-
- if repair:
- fields = ['personid=%s','tag=%s','data=%s','opt1=%s','opt2=%s','opt3=%s','last_updated=%s']
- for l in duplicates:
- used_fields = list()
- data = list()
- for i,f in enumerate(l):
- if f:
- used_fields.append(fields[i])
- data.append(f)
- run_sql("delete from aidPERSONIDDATA where "+' and '.join(used_fields), data)
- run_sql("insert into aidPERSONIDDATA (personid, tag, data, opt1, opt2, opt3, last_updated) "
- "value (%s,%s,%s,%s,%s,%s,%s)", l)
- return all_ok
-
-
-def check_duplicated_papers(printer, repair=False): ### duplicated_conirmed_papers_exist
- '''
- It examines if there are records of confirmed papers in aidPERSONIDPAPERS
- table which are in an impaired state (duplicated) and repairs them if
- specified.
-
- @param printer: for log keeping
- @type printer: func
- @param repair: fix the duplicated records
- @type repair: bool
- @return: duplicated records are found
- @rtype: bool
- '''
- all_ok = True
- author_confirmed_papers = defaultdict(list)
- to_reassign = list()
-
- confirmed_papers = run_sql("""select personid, bibrec
- from aidPERSONIDPAPERS
- where flag <> %s""",
- (-2,) )
-
- for pid, rec in confirmed_papers:
- author_confirmed_papers[pid].append(rec)
-
- for pid, recs in author_confirmed_papers.iteritems():
-
- if not len(recs) == len(set(recs)):
- all_ok = False
-
- duplicates = sorted(recs)
- duplicates = set([rec for i, rec in enumerate(duplicates[:-1]) if rec == duplicates[i+1]])
- printer("Person %d has duplicated papers: %s" % (pid, duplicates))
-
- if repair:
- for duprec in duplicates:
- printer("Repairing duplicated bibrec %s" % str(duprec))
- claimed_from_involved = run_sql("""select personid, bibref_table, bibref_value, bibrec, flag
- from aidPERSONIDPAPERS
- where personid=%s
- and bibrec=%s
- and flag >= 2""",
- (pid, duprec) )
- if len(claimed_from_involved) != 1:
- to_reassign.append(duprec)
- _delete_from_aidpersonidpapers_where(rec=duprec, pid=pid)
- else:
- run_sql("""delete from aidPERSONIDPAPERS
- where personid=%s
- and bibrec=%s
- and flag < 2""",
- (pid, duprec) )
-
- if repair and to_reassign:
- printer("Reassigning deleted bibrecs %s" % str(to_reassign))
- from bibauthorid_rabbit import rabbit
- rabbit(to_reassign)
-
- return all_ok
-
-
-def check_duplicated_signatures(printer, repair=False): # duplicated_confirmed_signatures_exist
- '''
- It examines if there are records of confirmed signatures in
- aidPERSONIDPAPERS table which are in an impaired state (duplicated) and
- repairs them if specified.
-
- @param printer: for log keeping
- @type printer: func
- @param repair: fix the duplicated signatures
- @type repair: bool
- @return: duplicated signatures are found
- @rtype: bool
- '''
- all_ok = True
- paper_confirmed_bibrefs = dict()
- to_reassign = list()
-
- confirmed_sigs = run_sql("""select bibref_table, bibref_value, bibrec
- from aidPERSONIDPAPERS
- where flag > %s""",
- (-2,) )
-
- for table, ref, rec in confirmed_sigs:
- paper_confirmed_bibrefs.setdefault(rec, []).append((table, ref))
-
- for rec, bibrefs in paper_confirmed_bibrefs.iteritems():
-
- if not len(bibrefs) == len(set(bibrefs)):
- all_ok = False
-
- duplicates = sorted(bibrefs)
- duplicates = set([bibref for i, bibref in enumerate(duplicates[:-1]) if bibref == duplicates[i+1]])
- printer("Paper %d has duplicated signatures: %s" % (rec, duplicates))
-
- if repair:
- for table, ref in duplicates:
- printer("Repairing duplicated signature %s" % str((table, ref)))
- claimed = _select_from_aidpersonidpapers_where(select=['personid', 'bibref_table', 'bibref_value', 'bibrec'], table=table, ref=ref, rec=rec, flag=2)
-
- if len(claimed) != 1:
- to_reassign.append(rec)
- _delete_from_aidpersonidpapers_where(table=table, ref=ref, rec=rec)
- else:
- run_sql("""delete from aidPERSONIDPAPERS
- where bibref_table=%s
- and bibref_value=%s
- and bibrec=%s
- and flag < 2""",
- (table, ref, rec) )
-
- if repair and to_reassign:
- printer("Reassigning deleted bibrecs %s" % str(to_reassign))
- from bibauthorid_rabbit import rabbit
- rabbit(to_reassign)
-
- return all_ok
-
-
-def get_wrong_names():
- '''
- Returns a generator with all wrong names in aidPERSONIDPAPERS.
- Every element is (table, ref, correct_name).
- '''
-
- bib100 = dict(((x[0], create_normalized_name(split_name_parts(x[1]))) for x in get_bib10x()))
- bib700 = dict(((x[0], create_normalized_name(split_name_parts(x[1]))) for x in get_bib70x()))
-
- pidnames100 = set(run_sql("select bibref_value, name from aidPERSONIDPAPERS "
- " where bibref_table='100'"))
- pidnames700 = set(run_sql("select bibref_value, name from aidPERSONIDPAPERS "
- " where bibref_table='700'"))
-
- wrong100 = set(('100', x[0], bib100.get(x[0], None)) for x in pidnames100 if x[1] != bib100.get(x[0], None))
- wrong700 = set(('700', x[0], bib700.get(x[0], None)) for x in pidnames700 if x[1] != bib700.get(x[0], None))
-
- total = len(wrong100) + len(wrong700)
-
- return chain(wrong100, wrong700), total
-
-
-def check_wrong_names(printer, repair=False): ### wrong_names_exist
- '''
- It examines if there are records in aidPERSONIDPAPERS table which carry a
- wrong name and repairs them if specified.
-
- @param printer: for log keeping
- @type printer: func
- @param repair: fix the found wrong names
- @type repair: bool
- @return: wrong names are found
- @rtype: bool
- '''
- wrong_names_exist = False
- wrong_names, wrong_names_count = get_wrong_names()
-
- if wrong_names_count > 0:
- wrong_names_exist = True
- printer("%d corrupted names in aidPERSONIDPAPERS." % wrong_names_count)
- for wrong_name in wrong_names:
- if wrong_name[2]:
- printer("Outdated name, '%s'(%s:%d)." % (wrong_name[2], wrong_name[0], wrong_name[1]))
- else:
- printer("Invalid id(%s:%d)." % (wrong_name[0], wrong_name[1]))
-
- if repair:
- printer("Fixing wrong name: %s" % str(wrong_name))
- if wrong_name[2]:
- run_sql("""update aidPERSONIDPAPERS
- set name=%s
- where bibref_table=%s
- and bibref_value=%s""",
- (wrong_name[2], wrong_name[0], wrong_name[1]) )
- else:
- _delete_from_aidpersonidpapers_where(table=wrong_name[0], ref=wrong_name[1])
-
- return not wrong_names_exist
-
-
-def check_canonical_names(printer, repair=False): ### impaired_canonical_names_exist
- '''
- It examines if there are authors who carry less or more than one canonical
- name and repairs them if specified.
-
- @param printer: for log keeping
- @type printer: func
- @param repair: fix authors with less/more than one canonical name
- @type repair: bool
- @return: authors with less/more than one canonical name exist
- @rtype: bool
- '''
- all_ok = True
-
- authors_cnames = _select_from_aidpersoniddata_where(select=['personid', 'data'], tag='canonical_name')
- authors_cnames = sorted(authors_cnames, key=itemgetter(0))
- author_cnames_count = dict((pid, len(list(cnames))) for pid, cnames in groupby(authors_cnames, key=itemgetter(0)))
-
- to_update = list()
-
- for pid in get_existing_personids():
- cnames_count = author_cnames_count.get(pid, 0)
-
- if cnames_count != 1:
- if cnames_count == 0:
- papers_count = _select_from_aidpersonidpapers_where(select=['count(*)'], pid=pid)[0][0]
- if papers_count != 0:
- all_ok = False
- printer("Personid %d does not have a canonical name, but has %d papers." % (pid, papers_count))
- to_update.append(pid)
- else:
- all_ok = False
- printer("Personid %d has %d canonical names.", (pid, cnames_count))
- to_update.append(pid)
-
- if repair and not all_ok:
- printer("Repairing canonical names for pids: %s" % str(to_update))
- update_personID_canonical_names(to_update, overwrite=True)
-
- return all_ok
-
-
-def check_empty_personids(printer, repair=False): ### empty_authors_exist
- '''
- It examines if there are empty authors (that is authors with no papers or
- other defined data) and deletes them if specified.
-
- @param printer: for log keeping
- @type printer: func
- @param repair: delete empty authors
- @type repair: bool
- @return: empty authors are found
- @rtype: bool
- '''
- empty_pids = delete_empty_persons(remove=repair)
-
- empty_authors_exist = False
- if empty_pids:
- empty_authors_exist = True
-
- for pid in empty_pids:
- printer("Personid %d has no papers and nothing else than canonical_name." % pid)
-
- if repair:
- printer("Deleting empty person %s." % pid)
-
- return not empty_authors_exist
-
-
-def check_wrong_rejection(printer, repair=False): ### impaired_rejections_exist
- '''
- It examines if there are records of rejected papers in aidPERSONIDPAPERS
- table which are in an impaired state (not assigned or both confirmed and
- rejected for the same author) and repairs them if specified.
-
- @param printer: for log keeping
- @type printer: func
- @param repair: fix the damaged records
- @type repair: bool
- @return: damaged records are found
- @rtype: bool
- '''
- all_ok = True
- to_reassign = list()
- to_deal_with = list()
-
- rejected_papers = set(_select_from_aidpersonidpapers_where(select=['bibref_table', 'bibref_value', 'bibrec'], flag=-2))
-
- confirmed_papers = set(run_sql("""select bibref_table, bibref_value, bibrec
- from aidPERSONIDPAPERS
- where flag > %s""",
- (-2,) ))
- not_assigned_papers = rejected_papers - confirmed_papers
-
- for paper in not_assigned_papers:
- printer("Paper (%s:%s,%s) was rejected but never reassigned" % paper)
- to_reassign.append(paper)
-
- rejected_papers = set(_select_from_aidpersonidpapers_where(select=['personid', 'bibref_table', 'bibref_value', 'bibrec'], flag=-2))
-
- confirmed_papers = set(run_sql("""select personid, bibref_table, bibref_value, bibrec
- from aidPERSONIDPAPERS
- where flag > %s""",
- (-2,) ))
- # papers which are both confirmed and rejected for/from the same author
- both_confirmed_and_rejected_papers = rejected_papers & confirmed_papers
-
- for paper in both_confirmed_and_rejected_papers:
- printer("Conflicting assignment/rejection: %s" % str(paper))
- to_deal_with.append(paper)
-
- if not_assigned_papers or both_confirmed_and_rejected_papers:
- all_ok = False
-
- if repair and (to_reassign or to_deal_with):
- from bibauthorid_rabbit import rabbit
-
- if to_reassign:
- # Rabbit is not designed to reassign signatures which are rejected but not assigned:
- # All signatures should be assigned. If a rejection occurs, the signature should get
- # moved to a new place and the rejection entry added, but never exist as a rejection only.
- # Hence, to force rabbit to reassign it we have to delete the rejection.
- printer("Reassigning bibrecs with missing entries: %s" % str(to_reassign))
- for sig in to_reassign:
- table, ref, rec = sig
- _delete_from_aidpersonidpapers_where(table=table, ref=ref, rec=rec, flag=-2)
-
- recs = [paper[2] for paper in to_reassign]
- rabbit(recs)
-
- if to_deal_with:
- # We got claims and rejections on the same paper for the same person. Let's forget about
- # it and reassign it automatically, they'll make up their minds sooner or later.
- printer("Deleting and reassigning bibrefrecs with conflicts %s" % str(to_deal_with))
- for sig in to_deal_with:
- pid, table, ref, rec = sig
- _delete_from_aidpersonidpapers_where(table=table, ref=ref, rec=rec, pid=pid)
-
- recs = map(itemgetter(3), to_deal_with)
- rabbit(recs)
-
- return all_ok
-
-
-def _delete_from_aidpersonidpapers_where(pid=None, table=None, ref=None, rec=None, name=None, flag=None, lcul=None):
- '''
- Deletes the records from aidPERSONIDPAPERS table with the given attributes.
- If no parameters are given it deletes all records.
-
- @param pid: author identifier
- @type pid: int
- @param table: bibref_table
- @type table: int
- @param ref: bibref_value
- @type ref: int
- @param rec: paper identifier
- @type rec: int
- @param name: author name
- @type name: str
- @param flag: flag
- @type flag: int
- @param lcul: lcul
- @type lcul: int
- '''
- conditions = list()
- add_condition = conditions.append
- args = list()
- add_arg = args.append
-
- if pid is not None:
- add_condition('personid=%s')
- add_arg(pid)
- if table is not None:
- add_condition('bibref_table=%s')
- add_arg(str(table))
- if ref is not None:
- add_condition('bibref_value=%s')
- add_arg(ref)
- if rec is not None:
- add_condition('bibrec=%s')
- add_arg(rec)
- if name is not None:
- add_condition('name=%s')
- add_arg(name)
- if flag is not None:
- add_condition('flag=%s')
- add_arg(flag)
- if lcul is not None:
- add_condition('lcul=%s')
- add_arg(lcul)
-
- if not conditions:
- return
-
- conditions_str = " and ".join(conditions)
- query = "delete from aidPERSONIDPAPERS where %s" % conditions_str
-
- run_sql(query, tuple(args) )
-
-
-def _select_from_aidpersonidpapers_where(select=None, pid=None, table=None, ref=None, rec=None, name=None, flag=None, lcul=None):
- '''
- Selects the given fields from the records of aidPERSONIDPAPERS table
- with the specified attributes. If no parameters are given it returns all
- records.
-
- @param select: fields to select
- @type select: list [str,]
- @param pid: author identifier
- @type pid: int
- @param table: bibref_table
- @type table: int
- @param ref: bibref_value
- @type ref: int
- @param rec: paper identifier
- @type rec: int
- @param name: author name
- @type name: str
- @param flag: author-paper association status
- @type flag: int
- @param lcul: lcul
- @type lcul: int
- @return: given fields of the records with the specified attributes
- @rtype: tuple
- '''
- if not select:
- return None
-
- conditions = list()
- add_condition = conditions.append
- args = list()
- add_arg = args.append
-
- add_condition('True')
- if pid is not None:
- add_condition('personid=%s')
- add_arg(pid)
- if table is not None:
- add_condition('bibref_table=%s')
- add_arg(str(table))
- if ref is not None:
- add_condition('bibref_value=%s')
- add_arg(ref)
- if rec is not None:
- add_condition('bibrec=%s')
- add_arg(rec)
- if name is not None:
- add_condition('name=%s')
- add_arg(name)
- if flag is not None:
- add_condition('flag=%s')
- add_arg(flag)
- if lcul is not None:
- add_condition('lcul=%s')
- add_arg(lcul)
-
- select_fields_str = "%s" % ", ".join(select)
- conditions_str = "%s" % " and ".join(conditions)
- query = """select %s
- from aidPERSONIDPAPERS
- where %s""" % (select_fields_str, conditions_str)
-
- return run_sql(query, tuple(args) )
-
-
-def _select_from_aidpersoniddata_where(select=None, pid=None, tag=None, data=None, opt1=None, opt2=None, opt3=None):
- '''
- Selects the given fields from the records of aidPERSONIDDATA table
- with the specified attributes. If no parameters are given it returns all
- records.
-
- @param select: fields to select
- @type select: list [str,]
- @param pid: author identifier
- @type pid: int
- @param tag: data tag
- @type tag: str
- @param data: data tag value
- @type data: str
- @param opt1: opt1
- @type opt1: int
- @param opt2: opt2
- @type opt2: int
- @param opt3: opt3
- @type opt3: str
- @return: given fields of the records with the specified attributes
- @rtype: tuple
- '''
- if not select:
- return None
-
- conditions = list()
- add_condition = conditions.append
- args = list()
- add_arg = args.append
-
- add_condition('True')
- if pid is not None:
- add_condition('personid=%s')
- add_arg(pid)
- if tag is not None:
- add_condition('tag=%s')
- add_arg(tag)
- if data is not None:
- add_condition('data=%s')
- add_arg(data)
- if opt1 is not None:
- add_condition('opt1=%s')
- add_arg(opt1)
- if opt2 is not None:
- add_condition('opt2=%s')
- add_arg(opt2)
- if opt3 is not None:
- add_condition('opt3=%s')
- add_arg(opt3)
-
- select_fields_str = "%s" % ", ".join(select)
- conditions_str = "%s" % " and ".join(conditions)
- query = """select %s
- from aidPERSONIDDATA
- where %s""" % (select_fields_str, conditions_str)
-
- return run_sql(query, tuple(args) )
-
-
-def check_merger():
- '''
- This function presumes that copy_personid was
- called before the merger.
- '''
- is_ok = True
-
- old_claims = set(run_sql("select personid, bibref_table, bibref_value, bibrec, flag "
- "from aidPERSONIDPAPERS_copy "
- "where flag = -2 or flag = 2"))
-
- cur_claims = set(run_sql("select personid, bibref_table, bibref_value, bibrec, flag "
- "from aidPERSONIDPAPERS "
- "where flag = -2 or flag = 2"))
-
- errors = ((old_claims - cur_claims, "Some claims were lost during the merge."),
- (cur_claims - old_claims, "Some new claims appeared after the merge."))
- act = { -2 : 'Rejection', 2 : 'Claim' }
-
- for err_set, err_msg in errors:
- if err_set:
- is_ok = False
- bibauthor_print(err_msg)
- bibauthor_print("".join(" %s: personid %d %d:%d,%d\n" %
- (act[cl[4]], cl[0], int(cl[1]), cl[2], cl[3]) for cl in err_set))
-
- old_assigned = set(run_sql("select bibref_table, bibref_value, bibrec "
- "from aidPERSONIDPAPERS_copy"))
- # "where flag <> -2 and flag <> 2"))
-
- cur_assigned = set(run_sql("select bibref_table, bibref_value, bibrec "
- "from aidPERSONIDPAPERS"))
- # "where flag <> -2 and flag <> 2"))
-
- errors = ((old_assigned - cur_assigned, "Some signatures were lost during the merge."),
- (cur_assigned - old_assigned, "Some new signatures appeared after the merge."))
-
- for err_sig, err_msg in errors:
- if err_sig:
- is_ok = False
- bibauthor_print(err_msg)
- bibauthor_print("".join(" %s:%d,%d\n" % sig for sig in err_sig))
-
- return is_ok
-
-def check_results():
- is_ok = True
-
- all_result_rows = run_sql("select personid,bibref_table,bibref_value,bibrec from aidRESULTS")
-
- keyfunc = lambda x: x[1:]
- duplicated = (d for d in (list(d) for k, d in groupby(sorted(all_result_rows, key=keyfunc), key=keyfunc)) if len(d) > 1)
-
- for dd in duplicated:
- is_ok = False
- for d in dd:
- print "Duplicated row in aidRESULTS"
- print "%s %s %s %s" % d
- print
-
-
- clusters = {}
- for rr in all_result_rows:
- clusters[rr[0]] = clusters.get(rr[0], []) + [rr[3]]
-
- faulty_clusters = dict((cid, len(recs) - len(set(recs)))
- for cid, recs in clusters.items()
- if not len(recs) == len(set(recs)))
-
- if faulty_clusters:
- is_ok = False
- print "Recids NOT unique in clusters!"
- print ("A total of %s clusters hold an average of %.2f duplicates" %
- (len(faulty_clusters), (sum(faulty_clusters.values()) / float(len(faulty_clusters)))))
-
- for c in faulty_clusters:
- print "Name: %-20s Size: %4d Faulty: %2d" % (c, len(clusters[c]), faulty_clusters[c])
-
- return is_ok
-
-def check_claim_inspireid_contradiction():
- iids10x = run_sql("select id from bib10x where tag = '100__i'")
- iids70x = run_sql("select id from bib70x where tag = '700__i'")
-
- refs10x = set(x[0] for x in run_sql("select id from bib10x where tag = '100__a'"))
- refs70x = set(x[0] for x in run_sql("select id from bib70x where tag = '700__a'"))
-
- if iids10x:
- iids10x = list_2_SQL_str(iids10x, lambda x: str(x[0]))
-
- iids10x = run_sql("select id_bibxxx, id_bibrec, field_number "
- "from bibrec_bib10x "
- "where id_bibxxx in %s"
- % iids10x)
-
- iids10x = ((row[0], [(ref, rec) for ref, rec in run_sql(
- "select id_bibxxx, id_bibrec "
- "from bibrec_bib10x "
- "where id_bibrec = '%s' "
- "and field_number = '%s'"
- % row[1:])
- if ref in refs10x])
- for row in iids10x)
- else:
- iids10x = ()
-
- if iids70x:
- iids70x = list_2_SQL_str(iids70x, lambda x: str(x[0]))
-
- iids70x = run_sql("select id_bibxxx, id_bibrec, field_number "
- "from bibrec_bib70x "
- "where id_bibxxx in %s"
- % iids70x)
-
- iids70x = ((row[0], [(ref, rec) for ref, rec in run_sql(
- "select id_bibxxx, id_bibrec "
- "from bibrec_bib70x "
- "where id_bibrec = '%s' "
- "and field_number = '%s'"
- % (row[1:]))
- if ref in refs70x])
- for row in iids70x)
- else:
- iids70x = ()
-
- # [(iids, [bibs])]
- inspired = list(chain(((iid, list(set(('100',) + bib for bib in bibs))) for iid, bibs in iids10x),
- ((iid, list(set(('700',) + bib for bib in bibs))) for iid, bibs in iids70x)))
-
- assert all(len(x[1]) == 1 for x in inspired)
-
- inspired = ((k, map(itemgetter(0), map(itemgetter(1), d)))
- for k, d in groupby(sorted(inspired, key=itemgetter(0)), key=itemgetter(0)))
-
- # [(inspireid, [bibs])]
- inspired = [([(run_sql("select personid "
- "from aidPERSONIDPAPERS "
- "where bibref_table = %s "
- "and bibref_value = %s "
- "and bibrec = %s "
- "and flag = '2'"
- , bib), bib)
- for bib in cluster[1]], cluster[0])
- for cluster in inspired]
-
- # [([([pid], bibs)], inspireid)]
- for cluster, iid in inspired:
- pids = set(chain.from_iterable(imap(itemgetter(0), cluster)))
-
- if len(pids) > 1:
- print "InspireID: %s links the following papers:" % iid
- print map(itemgetter(1), cluster)
- print "More than one personid claimed them:"
- print list(pids)
- print
- continue
-
- if len(pids) == 0:
- # not even one paper with this inspireid has been
- # claimed, screw it
- continue
-
- pid = list(pids)[0][0]
-
- # The last step is to check all non-claimed papers for being
- # claimed by the person on some different signature.
- problem = (run_sql("select bibref_table, bibref_value, bibrec "
- "from aidPERSONIDPAPERS "
- "where bibrec = %s "
- "and personid = %s "
- "and flag = %s"
- , (bib[2], pid, 2))
- for bib in (bib for lpid, bib in cluster if not lpid))
- problem = list(chain.from_iterable(problem))
-
- if problem:
- print "A personid has claimed a paper from an inspireid cluster and a contradictory paper."
- print "Personid %d" % pid
- print "Inspireid cluster %s" % str(map(itemgetter(1), cluster))
- print "Contradicting claims: %s" % str(problem)
- print
-
-def get_all_bibrecs():
- '''
- Get all record ids present in aidPERSONIDPAPERS
- '''
- return set([x[0] for x in run_sql("select bibrec from aidPERSONIDPAPERS")])
-
-def get_bibrefrec_to_pid_flag_mapping():
- '''
- create a map between signatures and personid/flag
- '''
- whole_table = run_sql("select bibref_table,bibref_value,bibrec,personid,flag from aidPERSONIDPAPERS")
- gc.disable()
- ret = {}
- for x in whole_table:
- sig = (x[0], x[1], x[2])
- pid_flag = (x[3], x[4])
- ret[sig] = ret.get(sig , []) + [pid_flag]
- gc.collect()
- gc.enable()
- return ret
-
-def remove_all_bibrecs(bibrecs):
- '''
- Remove give record ids from aidPERSONIDPAPERS table
- @param bibrecs:
- @type bibrecs:
- '''
- bibrecs_s = list_2_SQL_str(bibrecs)
- run_sql("delete from aidPERSONIDPAPERS where bibrec in %s" % bibrecs_s)
-
-
-def empty_results_table():
- '''
- Get rid of all tortoise results
- '''
- run_sql("TRUNCATE aidRESULTS")
-
-
-def save_cluster(named_cluster):
- '''
- Save a cluster in aidRESULTS
- @param named_cluster:
- @type named_cluster:
- '''
- name, cluster = named_cluster
- for bib in cluster.bibs:
- run_sql("INSERT INTO aidRESULTS "
- "(personid, bibref_table, bibref_value, bibrec) "
- "VALUES (%s, %s, %s, %s) "
- , (name, str(bib[0]), bib[1], bib[2]))
-
-
-def remove_result_cluster(name):
- '''
- Remove result cluster using name string
- @param name:
- @type name:
- '''
- run_sql("DELETE FROM aidRESULTS "
- "WHERE personid like '%s.%%'"
- % name)
-
-
-def personid_name_from_signature(sig):
- '''
- Find personid and name string of a signature
- @param sig:
- @type sig:
- '''
- ret = run_sql("select personid, name "
- "from aidPERSONIDPAPERS "
- "where bibref_table = %s and bibref_value = %s and bibrec = %s "
- "and flag > '-2'"
- , sig)
- assert len(ret) < 2, ret
- return ret
-
-def personid_from_signature(sig):
- '''
- Find personid owner of a signature
- @param sig:
- @type sig:
- '''
- ret = run_sql("select personid, flag "
- "from aidPERSONIDPAPERS "
- "where bibref_table = %s and bibref_value = %s and bibrec = %s "
- "and flag > '-2'"
- , sig)
- assert len(ret) < 2, ret
- return ret
-
-def get_signature_info(sig):
- '''
- Get personid and flag relative to a signature
- @param sig:
- @type sig:
- '''
- ret = run_sql("select personid, flag "
- "from aidPERSONIDPAPERS "
- "where bibref_table = %s and bibref_value = %s and bibrec = %s "
- "order by flag"
- , sig)
- return ret
-
-def get_claimed_papers(pid):
- '''
- Find all papers which have been manually claimed
- @param pid:
- @type pid:
- '''
- return run_sql("select bibref_table, bibref_value, bibrec "
- "from aidPERSONIDPAPERS "
- "where personid = %s "
- "and flag > %s",
- (pid, 1))
-
-
-def copy_personids():
- '''
- Make a copy of aidPERSONID tables to aidPERSONID*_copy tables for later comparison/restore
- '''
- run_sql("DROP TABLE IF EXISTS `aidPERSONIDDATA_copy`")
- run_sql("CREATE TABLE `aidPERSONIDDATA_copy` ( "
- "`personid` BIGINT( 8 ) UNSIGNED NOT NULL , "
- "`tag` VARCHAR( 64 ) NOT NULL , "
- "`data` VARCHAR( 256 ) NOT NULL , "
- "`opt1` MEDIUMINT( 8 ) DEFAULT NULL , "
- "`opt2` MEDIUMINT( 8 ) DEFAULT NULL , "
- "`opt3` VARCHAR( 256 ) DEFAULT NULL , "
- "`last_updated` TIMESTAMP ON UPDATE CURRENT_TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP , "
- "INDEX `personid-b` ( `personid` ) , "
- "INDEX `tag-b` ( `tag` ) , "
- "INDEX `data-b` ( `data` ) , "
- "INDEX `opt1` ( `opt1` ) , "
- "INDEX `timestamp-b` ( `last_updated` ) "
- ") ENGINE = MYISAM DEFAULT CHARSET = utf8")
- run_sql("INSERT INTO `aidPERSONIDDATA_copy` "
- "SELECT * "
- "FROM `aidPERSONIDDATA`")
-
- run_sql("DROP TABLE IF EXISTS `aidPERSONIDPAPERS_copy`")
- run_sql("CREATE TABLE `aidPERSONIDPAPERS_copy` ( "
- "`personid` bigint( 16 ) unsigned NOT NULL , "
- "`bibref_table` enum( '100', '700' ) NOT NULL , "
- "`bibref_value` mediumint( 8 ) unsigned NOT NULL , "
- "`bibrec` mediumint( 8 ) unsigned NOT NULL , "
- "`name` varchar( 256 ) NOT NULL , "
- "`flag` smallint( 2 ) NOT NULL DEFAULT '0', "
- "`lcul` smallint( 2 ) NOT NULL DEFAULT '0', "
- "`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP , "
- "INDEX `personid-b` ( `personid` ) , "
- "INDEX `reftable-b` ( `bibref_table` ) , "
- "INDEX `refvalue-b` ( `bibref_value` ) , "
- "INDEX `rec-b` ( `bibrec` ) , "
- "INDEX `name-b` ( `name` ) , "
- "INDEX `pn-b` (`personid`, `name`) , "
- "INDEX `timestamp-b` (`last_updated`) , "
- "INDEX `flag-b` (`flag`) , "
- "INDEX `personid-flag-b` (`personid`,`flag`) , "
- "INDEX `ptvrf-b` (`personid`, `bibref_table`, `bibref_value`, `bibrec`, `flag`) "
- ") ENGINE = MyISAM DEFAULT CHARSET = utf8")
- run_sql("INSERT INTO `aidPERSONIDPAPERS_copy` "
- "SELECT * "
- "FROM `aidPERSONIDPAPERS")
-
-
-def delete_empty_persons(remove=True): ### remove_empty_authors
- '''
- Gets all empty authors (that is authors with no papers or other defined
- data) and by default deletes all data associated with them, except if
- specified differently.
-
- @param remove: delete empty authors
- @type remove: bool
- @return: empty author identifiers set(pid,)
- @rtype: set set(int,)
- '''
- pids_with_papers = set(pid[0] for pid in _select_from_aidpersonidpapers_where(select=['personid']))
- pids_tags = _select_from_aidpersoniddata_where(select=['personid', 'tag'])
- pids_with_data = set(pid for pid, tag in pids_tags)
- not_empty_pids = set(pid for pid, tag in pids_tags if tag in bconfig.NON_EMPTY_PERSON_TAGS)
-
- empty_pids = pids_with_data - (pids_with_papers | not_empty_pids)
-
- if empty_pids and remove:
- run_sql("""delete from aidPERSONIDDATA
- where personid in %s"""
- % list_2_SQL_str(empty_pids) )
-
- return empty_pids
-
-
-def restore_personids():
- '''
- Restore personid tables from last copy saved with copy_personids
- '''
- run_sql("TRUNCATE `aidPERSONIDDATA`")
- run_sql("INSERT INTO `aidPERSONIDDATA` "
- "SELECT * "
- "FROM `aidPERSONIDDATA_copy`")
-
- run_sql("TRUNCATE `aidPERSONIDPAPERS`")
- run_sql("INSERT INTO `aidPERSONIDPAPERS` "
- "SELECT * "
- "FROM `aidPERSONIDPAPERS_copy")
-
-def resolve_affiliation(ambiguous_aff_string):
- """
- This is a method available in the context of author disambiguation in ADS
- only. No other platform provides the db table used by this function.
- @warning: to be used in an ADS context only.
- @param ambiguous_aff_string: Ambiguous affiliation string
- @type ambiguous_aff_string: str
- @return: The normalized version of the name string as presented in the database
- @rtype: str
- """
- if not ambiguous_aff_string or not bconfig.CFG_ADS_SITE:
- return "None"
-
- aff_id = run_sql("select aff_id from ads_affiliations where affstring=%s", (ambiguous_aff_string,))
-
- if aff_id:
- return aff_id[0][0]
- else:
- return "None"
-
-def get_free_pids():
- '''
- Returns an iterator with all free personids.
- It's cool, because it fills holes.
- '''
- all_pids = frozenset(x[0] for x in chain(
- run_sql("select personid from aidPERSONIDPAPERS") ,
- run_sql("select personid from aidPERSONIDDATA")))
- return ifilter(lambda x: x not in all_pids, count())
-
-
-def remove_results_outside(many_names):
- '''
- Delete results from aidRESULTS not including many_names
- @param many_names:
- @type many_names:
- '''
- many_names = frozenset(many_names)
- res_names = frozenset(x[0].split(".")[0] for x in run_sql("select personid from aidRESULTS"))
-
- for name in res_names - many_names:
- run_sql("delete from aidRESULTS where personid like '%s.%%'" % name)
-
-
-def get_signatures_from_bibrefs(bibrefs):
- '''
- @param bibrefs:
- @type bibrefs:
- '''
- bib10x = ifilter(lambda x: x[0] == 100, bibrefs)
- bib10x_s = list_2_SQL_str(bib10x, lambda x: x[1])
- bib70x = ifilter(lambda x: x[0] == 700, bibrefs)
- bib70x_s = list_2_SQL_str(bib70x, lambda x: x[1])
- valid_recs = set(get_all_valid_bibrecs())
-
- if bib10x_s != '()':
- sig10x = run_sql("select 100, id_bibxxx, id_bibrec "
- "from bibrec_bib10x "
- "where id_bibxxx in %s"
- % (bib10x_s,))
- else:
- sig10x = ()
-
- if bib70x_s != '()':
- sig70x = run_sql("select 700, id_bibxxx, id_bibrec "
- "from bibrec_bib70x "
- "where id_bibxxx in %s"
- % (bib70x_s,))
- else:
- sig70x = ()
-
- return ifilter(lambda x: x[2] in valid_recs, chain(set(sig10x), set(sig70x)))
-
-
-def get_all_valid_bibrecs():
- '''
- Returns a list of valid record ids
- '''
- collection_restriction_pattern = " or ".join(["980__a:\"%s\"" % x for x in bconfig.LIMIT_TO_COLLECTIONS])
- return perform_request_search(p="%s" % collection_restriction_pattern, rg=0)
-
-
-def get_coauthor_pids(pid, exclude_bibrecs=None):
- '''
- Find personids sharing bibrecs with given pid, eventually excluding a given set of common bibrecs.
- @param pid:
- @type pid:
- @param exclude_bibrecs:
- @type exclude_bibrecs:
- '''
- papers = get_person_bibrecs(pid)
- if exclude_bibrecs:
- papers = set(papers) - set(exclude_bibrecs)
-
- if not papers:
- return []
-
- papers_s = list_2_SQL_str(papers)
-
- pids = run_sql("select personid,bibrec from aidPERSONIDPAPERS "
- "where bibrec in %s and flag > -2" % papers_s)
-
-
- pids = set((int(p[0]), int(p[1])) for p in pids if p[0] != pid)
- pids = sorted([p[0] for p in pids])
- pids = groupby(pids)
- pids = [(key, len(list(val))) for key, val in pids if key != pid]
- pids = sorted(pids, key=lambda x: x[1], reverse=True)
-
- return pids
-
-def get_doi_from_rec(recid):
- """
- Returns the doi of the paper like str if found.
- Otherwise returns None.
- 0247 $2 DOI $a id
- """
- idx = run_sql("SELECT id_bibxxx, field_number FROM bibrec_bib02x WHERE id_bibrec = %s", (recid,))
- if idx:
- doi_id_s = list_2_SQL_str(idx, lambda x: x[0])
- doi = run_sql("SELECT id, tag, value FROM bib02x WHERE id in %s " % doi_id_s)
- if doi:
- grouped = groupby(idx, lambda x: x[1])
- doi_dict = dict((x[0], x[1:]) for x in doi)
- for group in grouped:
- elms = [x[0] for x in list(group[1])]
- found = False
- code = None
- for el in elms:
- if doi_dict[el][0] == '0247_2' and doi_dict[el][1] == 'DOI':
- found = True
- elif doi_dict[el][0] == '0247_a':
- code = doi_dict[el][1]
- if found and code:
- return code
- return None
-
-def export_person(person_id):
- '''list of records table: personidpapers and personiddate check existing function for getting the records!!!
- exports a structure of dictunaries of tuples of [...] if strings, like:
-
- {'name':('namestring',),
- 'repeatable_field':({'field1':('val1',)},{'field1':'val2'})}
- '''
-
- person_info = defaultdict(defaultdict)
-
- full_names = get_person_db_names_set(person_id)
- if full_names:
- splitted_names = [split_name_parts(n[0]) for n in full_names]
- splitted_names = [x + [len(x[2])] for x in splitted_names]
- max_first_names = max([x[4] for x in splitted_names])
- full_name_candidates = filter(lambda x: x[4] == max_first_names, splitted_names)
- full_name = create_normalized_name(full_name_candidates[0])
-
-
- person_info['names']['full_name'] = (full_name,)
- person_info['names']['surname'] = (full_name_candidates[0][0],)
- if full_name_candidates[0][2]:
- person_info['names']['first_names'] = (' '.join(full_name_candidates[0][2]),)
- person_info['names']['name_variants'] = ('; '.join([create_normalized_name(x) for x in splitted_names]),)
-
- bibrecs = get_person_bibrecs(person_id)
-
- recids_data = []
- for recid in bibrecs:
- recid_dict = defaultdict(defaultdict)
- recid_dict['INSPIRE-record-id'] = (str(recid),)
- recid_dict['INSPIRE-record-url'] = ('%s/record/%s' % (CFG_SITE_URL, str(recid)),)
- rec_doi = get_doi_from_rec(recid)
- if rec_doi:
- recid_dict['DOI'] = (str(rec_doi),)
- recids_data.append(recid_dict)
-
- person_info['records']['record'] = tuple(recids_data)
-
-
- person_info['identifiers']['INSPIRE_person_ID'] = (str(person_id),)
-
-
- canonical_names = get_canonical_names_by_pid(person_id)
- if canonical_names:
- person_info['identifiers']['INSPIRE_canonical_name'] = (str(canonical_names[0][0]),)
- person_info['profile_page']['INSPIRE_profile_page'] = ('%s/author/%s' % (CFG_SITE_URL, canonical_names[0][0]),)
- else:
- person_info['profile_page']['INSPIRE_profile_page'] = ('%s/author/%s' % (CFG_SITE_URL, str(person_id)),)
-
- orcids = get_orcids_by_pids(person_id)
- if orcids:
- person_info['identifiers']['ORCID'] = tuple(str(x[0]) for x in orcids)
-
-
- inspire_ids = get_inspire_ids_by_pids(person_id)
- if inspire_ids:
- person_info['identifiers']['INSPIREID'] = tuple(str(x[0]) for x in inspire_ids)
-
-
- return person_info
-
-
-def export_person_to_foaf(person_id):
- '''
- Exports to foaf xml a dictionary of dictionaries or tuples of strings as retured by export_person
- '''
-
- infodict = export_person(person_id)
-
- def export(val, indent=0):
- if isinstance(val, dict):
- contents = list()
- for k, v in val.iteritems():
- if isinstance(v, tuple):
- contents.append(''.join([ X[str(k)](indent=indent, body=export(c)) for c in v]))
- else:
- contents.append(X[str(k)](indent=indent, body=export(v, indent=indent + 1)))
- return ''.join(contents)
- elif isinstance(val, str):
- return str(X.escaper(val))
- else:
- raise Exception('WHAT THE HELL DID WE GET HERE? %s' % str(val))
-
- return X['person'](body=export(infodict, indent=1))
-
-def flush_data(table_name, column_names, args):
- '''
- docstring
-
- @param table_name:
- @type table_name:
- @param column_names:
- @type column_names:
- @param args:
- @type args:
- '''
- column_num = len(column_names)
-
- assert len(args) % column_num == 0, 'Trying to flush data in table %s. Wrong number of arguments passed.' % table_name
-
- values_tuple = "(%s)" % ", ".join(repeat("%s", column_num))
- multiple_values_tuple = ", ".join(repeat(values_tuple, len(args)/column_num))
- insert_query = 'insert into %s (%s) values %s' % (table_name, ", ".join(column_names), multiple_values_tuple)
-
- run_sql(insert_query, args)
-
-def trancate_table(table_name):
- '''
- docstring
-
- @param table_name:
- @type table_name:
- '''
- run_sql("truncate table %s" % (table_name,))
-
-def set_dense_index_ready():
- '''
- docstring
- '''
- run_sql("insert into aidDENSEINDEX (name_id,person_name,personids) values (%s,%s,%s)", (-1,'',''))
-
-def set_inverted_lists_ready():
- '''
- docstring
- '''
- run_sql("insert into aidINVERTEDLISTS (qgram,inverted_list,list_cardinality) values (%s,%s,%s)", ('!'*QGRAM_LEN,'',0))
-
-def get_inverted_lists(qgrams):
- '''
- docstring
-
- @param table_name:
- @type table_name:
- @return:
- @rtype:
- '''
- inverted_lists = run_sql("select inverted_list, list_cardinality from aidINVERTEDLISTS where qgram in %s"
- % (list_2_SQL_str(qgrams, f=lambda x: "'%s'" % x), ))
- return inverted_lists
-
-def get_indexable_name_personids(nameids):
- '''
- docstring
-
- @param table_name:
- @type table_name:
- @return:
- @rtype:
- '''
- name_personids = run_sql("select person_name, personids from aidDENSEINDEX where name_id in %s" % (list_2_SQL_str(nameids),) )
- return name_personids
-
-def check_search_engine_status():
- '''
- docstring
-
- @return:
- @rtype:
- '''
- dense_index_exists = bool(run_sql("select * from aidDENSEINDEX where name_id=%s", (-1,)))
- inverted_lists_exists = bool(run_sql("select * from aidINVERTEDLISTS where qgram=%s", ('!'*QGRAM_LEN,) ))
-
- if dense_index_exists and inverted_lists_exists:
- return True
-
- return False
diff --git a/modules/bibauthorid/lib/bibauthorid_name_utils_unit_tests.py b/modules/bibauthorid/lib/bibauthorid_name_utils_unit_tests.py
deleted file mode 100644
index 81bdf868d..000000000
--- a/modules/bibauthorid/lib/bibauthorid_name_utils_unit_tests.py
+++ /dev/null
@@ -1,281 +0,0 @@
-# -*- coding: utf-8 -*-
-##
-## This file is part of Invenio.
-## Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN.
-##
-## Invenio is free software; you can redistribute it and/or
-## modify it under the terms of the GNU General Public License as
-## published by the Free Software Foundation; either version 2 of the
-## License, or (at your option) any later version.
-##
-## Invenio is distributed in the hope that it will be useful, but
-## WITHOUT ANY WARRANTY; without even the implied warranty of
-## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-## General Public License for more details.
-##
-## You should have received a copy of the GNU General Public License
-## along with Invenio; if not, write to the Free Software Foundation, Inc.,
-## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-
-"""Unit tests for the search engine."""
-
-__revision__ = \
- "$Id$"
-
-from operator import ge,le,eq
-from invenio.testutils import InvenioTestCase, make_test_suite, run_test_suite
-
-#name_tests: name string, split_name_parts,
-
-names = [
- 'Test, m. e.',
- 'Test, me',
- 'Test, me with m. u. ltiple names a. n. d. initials',
- 'Test, me with multiple names',
- 'me test',
- 's. s. b. test',
- 'should s. b. test',
- 'should still be test',
- 'test with -aoe}[/])+ junk, but with comma',
- 'test,',
- 'test, m initials morder',
- 'test, mix initials m.',
- 'test, mix initials morder',
- 'test, s. s. b.',
- 'test, s. still be',
- 'with .[+)* ] just but without comma test']
-
-from invenio.bibauthorid_name_utils import split_name_parts, distance, create_canonical_name, create_normalized_name, create_unified_name, soft_compare_names
-from invenio.bibauthorid_name_utils import full_names_are_equal_composites, full_names_are_substrings, surname_compatibility, initials_compatibility, compare_names
-from invenio.bibauthorid_name_utils import create_name_tuples
-
-class TestDistanceFunctions(InvenioTestCase):
- """ Test string distance functions """
-
- def test_simple_distance(self):
- self.assertEqual(distance('aaaaa','aaaaa'), 0)
- self.assertEqual(distance('aaaaa','aa'), 3)
- self.assertEqual(distance('aaaaa','aaaab'), 1)
- self.assertEqual(distance('aaaaa','bbbbb'), 5)
- self.assertEqual(distance('eecbr','bbbbb'), 4)
-
-
-class TestSplitNameParts(InvenioTestCase):
- 'Test splitting of names'
- names_split_name_parts = {
- 'Test, m. e.': ['Test', ['M', 'E'], [], []],
- 'Test, me': ['Test', ['M'], ['Me'], [0]],
- 'Test, me with m. u. ltiple names a. n. d. initials': ['Test', ['M', 'W', 'M', 'U', 'L', 'N', 'A', 'N', 'D', 'I'],['Me', 'With', 'Ltiple', 'Names', 'Initials'],[0, 1, 4, 5, 9]],
- 'Test, me with multiple names': ['Test',['M', 'W', 'M', 'N'],['Me', 'With', 'Multiple', 'Names'],[0, 1, 2, 3]],
- 'me test': ['Test', ['M'], ['Me'], [0]],
- 's. s. b. test': ['Test', ['S', 'S', 'B'], [], []],
- 'should s. b. test': ['Test', ['S', 'S', 'B'], ['Should'], [0]],
- 'should still be test': ['Test',['S', 'S', 'B'],['Should', 'Still', 'Be'],[0, 1, 2]],
- 'test with -aoe}[/])+ junk, but with comma': ['Test with -Aoe}[/])+ junk',['B', 'W', 'C'],['But', 'With', 'Comma'],[0, 1, 2]],
- 'test,': ['Test', [], [], []],
- 'test, m initials morder': ['Test',['M', 'I', 'M'],['Initials', 'Morder'],[1, 2]],
- 'test, mix initials m.': ['Test',['M', 'I', 'M'],['Mix', 'Initials'],[0, 1]],
- 'test, mix initials morder': ['Test',['M', 'I', 'M'],['Mix', 'Initials', 'Morder'],[0, 1, 2]],
- 'test, s. s. b.': ['Test', ['S', 'S', 'B'], [], []],
- 'test, s. still be': ['Test', ['S', 'S', 'B'], ['Still', 'Be'], [1, 2]],
- 'with .[+)* ] just but without comma test': ['Test',['W', '[', '*', ']', 'J', 'B', 'W', 'C'],['With', '[+', 'Just', 'But', 'Without', 'Comma'],[0, 1, 4, 5, 6, 7]],
- 'test-dash': ['Test-Dash', [], [], []],
- 'test-dash,': ['Test-Dash', [], [], []]
- }
-
- def test_split_name_parss(self):
- for tn in self.names_split_name_parts.keys():
- self.assertEqual(split_name_parts(tn), self.names_split_name_parts[tn])
-
-
-class TestCreateCanonicalNames(InvenioTestCase):
- 'Test creation of canonical names'
- names_create_canonical_names = {
- 'Test': 'Test',
- 'Test, m. e.': 'M.E.Test',
- 'Test, me': 'M.Test',
- 'Test, me with m. u. ltiple names a. n. d. initials': 'M.W.M.U.L.N.A.N.D.I.Test',
- 'Test, me with multiple names': 'M.W.M.N.Test',
- 'me test': 'M.Test',
- 's. s. b. test': 'S.S.B.Test',
- 'should s. b. test': 'S.S.B.Test',
- 'should still be test': 'S.S.B.Test',
- 'test with -aoe}[/])+ junk, but with comma': 'B.W.C.Test.with.Aoe.junk',
- 'test,': 'Test',
- 'test, m initials morder': 'M.I.M.Test',
- 'test, mix initials m.': 'M.I.M.Test',
- 'test, mix initials morder': 'M.I.M.Test',
- 'test, s. s. b.': 'S.S.B.Test',
- 'test, s. still be': 'S.S.B.Test',
- 'with .[+)* ] just but without comma test': 'W..J.B.W.C.Test'
- }
- def test_create_canonical_name(self):
- for tn in self.names_create_canonical_names.keys():
- self.assertEqual(create_canonical_name(tn), self.names_create_canonical_names[tn])
-
-
-class TestCreateNormalizedName(InvenioTestCase):
- 'Test creation of normalized names'
- tc ={
- 'Hyphened-surname, hyphened-name and normal': 'Hyphened-Surname, Hyphened Name And Normal',
- 'Test, m. e.': 'Test, M. E.',
- 'Test, me': 'Test, Me',
- 'Test, me with m. u. ltiple names a. n. d. initials': 'Test, Me With M. U. Ltiple Names A. N. D. Initials',
- 'Test, me with multiple names': 'Test, Me With Multiple Names',
- 'me test': 'Test, Me',
- 's. s. b. test': 'Test, S. S. B.',
- 'should s. b. test': 'Test, Should S. B.',
- 'should still be test': 'Test, Should Still Be',
- 'test with -aoe}[/])+ junk, but with comma': 'Test with -Aoe}[/])+ junk, But With Comma',
- 'test,': 'Test',
- 'test, m initials morder': 'Test, M. Initials Morder',
- 'test, mix initials m.': 'Test, Mix Initials M.',
- 'test, mix initials morder': 'Test, Mix Initials Morder',
- 'test, s. s. b.': 'Test, S. S. B.',
- 'test, s. still be': 'Test, S. Still Be',
- 'with .[+)* ] just but without comma test': 'Test, With [+ *. ]. Just But Without Comma'}
-
-
- def test_create_normalized_name(self):
- for tn in self.tc.keys():
- self.assertEqual(create_normalized_name(split_name_parts(tn)), self.tc[tn])
-
-
-
-class TestCreateUinifiedName(InvenioTestCase):
- 'Test creation of unified names'
- tc = {
- 'Hyphened-surname, hyphened-name and normal': 'Hyphened-Surname, H. N. A. N. ',
- 'Test, m. e.': 'Test, M. E. ',
- 'Test, me': 'Test, M. ',
- 'Test, me with m. u. ltiple names a. n. d. initials': 'Test, M. W. M. U. L. N. A. N. D. I. ',
- 'Test, me with multiple names': 'Test, M. W. M. N. ',
- 'me test': 'Test, M. ',
- 's. s. b. test': 'Test, S. S. B. ',
- 'should s. b. test': 'Test, S. S. B. ',
- 'should still be test': 'Test, S. S. B. ',
- 'test with -aoe}[/])+ junk, but with comma': 'Test with -Aoe}[/])+ junk, B. W. C. ',
- 'test,': 'Test, ',
- 'test, m initials morder': 'Test, M. I. M. ',
- 'test, mix initials m.': 'Test, M. I. M. ',
- 'test, mix initials morder': 'Test, M. I. M. ',
- 'test, s. s. b.': 'Test, S. S. B. ',
- 'test, s. still be': 'Test, S. S. B. ',
- 'with .[+)* ] just but without comma test': 'Test, W. [. *. ]. J. B. W. C. '}
-
- def test_create_unified_name(self):
- for tn in self.tc.keys():
- self.assertEqual(create_unified_name(tn), self.tc[tn])
-
-class TestSoftNameComparison(InvenioTestCase):
- 'Test soft name comparison'
-
- tc = {
- 'Test, Name': ['Test, Name', [(ge, 0.7), (le, 1.)]],
- 'Test, N': ['Test, Name', [(ge, 0.7), (le, 1.)]],
- 'Test, Cane': ['Test, Name', [(ge, 0.5), (le, 1.)]],
- 'Test, C': ['Test, Name', [(ge, 0.5), (le, 1.)]],
- 'Test, Cane Name': ['Test, Name Cane', [(ge, 0.5), (le, 1.)]],
- 'Test, C N': ['Test, N C', [(ge, 0.5), (le, 1.)]],
- 'Tast, C N': ['Test, N C', [(ge, 0.0), (le, 0.5)]],
- 'Diff, C N': ['Erent, C N', [(ge, 0.0), (le, 0.4)]],
- 'Diff, Con Nome': ['Erent, Con Nome', [(ge, 0.0), (le, 0.4)]],
- 'Diff, Con Nomee': ['Erent, Che Noun', [(ge, 0.0), (le, 0.2)]],
- 'Diff, Name': ['Erent, Completely', [(ge, 0.0), (le, 0.1)]],
- 'Test-dash': ['Test-dash', [(ge, 0.5), (le, 1.)]],
- 'Test, noname': ['Test,', [(ge, 0.5), (le, 1.)]],
- 'Test, noname': ['Test', [(ge, 0.5), (le, 1.)]],
- }
-
- def test_value_ranges(self):
- cn = soft_compare_names
- for n1 in self.tc.keys():
- n2 = self.tc[n1][0]
- tests = self.tc[n1][1]
- for test in tests:
- self.assertTrue(cn(n1,n2) == cn(n2,n1))
- self.assertTrue(test[0](cn(n1,n2), test[1]))
-
-
-class TestNameComparison(InvenioTestCase):
- 'Test name comparison'
- tc = {
- 'Test, Name': ['Test, Name', [(ge, 0.7), (le, 1.)]],
- 'Test, N': ['Test, Name', [(ge, 0.6), (le, 0.8)]],
- 'Test, Cane': ['Test, Name', [(ge, 0.0), (le, .2)]],
- 'Test, C': ['Test, Name', [(ge, 0.0), (le, .2)]],
- 'Test, Cane Name': ['Test, Name Cane', [(ge, 0.0), (le, 1.)]],
- 'Test, C N': ['Test, N C', [(ge, 0.4), (le, 0.7)]],
- 'Tast, C N': ['Test, C N', [(ge, 0.4), (le, 0.7)]],
- 'Diff, C N': ['Erent, C N', [(ge, 0.0), (le, 0.2)]],
- 'Diff, Con Nome': ['Erent, Con Nome', [(ge, 0.0), (le, 0.2)]],
- 'Diff, Con Nomee': ['Erent, Che Noun', [(ge, 0.0), (le, 0.2)]],
- 'Diff, Name': ['Erent, Completely', [(ge, 0.0), (le, 0.1)]],
- 'Chen, Xiaoli': ['Chen, Xiao Li', [(ge, 0.6), (le, 0.8)]],
- 'Chen, Xiaolo': ['Chen, Xiao', [(ge, 0.7), (le, 0.8)]],
- 'Chen, Xiaola': ['Chen, Xia', [(ge, 0.6), (le, 0.7)]],
- 'Chen, Xiaolu': ['Chen, Xi', [(ge, 0.4), (le, 0.6)]],
- }
-
- def test_create_name_tuples(self):
- cn = create_name_tuples
- self.assertEqual(cn(['n1','n2','n3']), ['n1 n2 n3', 'N1n2 n3', 'n1 N2n3', 'N1n2n3'])
- self.assertEqual(cn(['n1','n2']), ['n1 n2', 'N1n2'])
- self.assertEqual(cn(['n1']), ['n1'])
-
- def test_full_names_are_equal_composites(self):
- cn = full_names_are_equal_composites
- self.assertTrue(cn('surname, comp-osite', 'surname, comp osite'))
- self.assertTrue(cn('surname, comp [{} ) osite', 'surname, comp osite'))
- self.assertTrue(cn('surname, composite', 'surname, comp osite'))
- self.assertFalse(cn('surname, comp-osite', 'surname, notcomp osite'))
- self.assertFalse(cn('surname, comp-osite', 'surname, comp nosite'))
- self.assertFalse(cn('surname, comp-osite', 'surname, compo osite'))
- self.assertFalse(cn('surname, comp-osite', 'surname, comp osited'))
-
- def test_full_names_are_substrings(self):
- cn = full_names_are_substrings
- self.assertTrue(cn('Sur, longname', 'Sur, long'))
- self.assertFalse(cn('Sur, longname', 'Sur, name'))
- self.assertTrue(cn('Sur, long', 'Sur, long'))
- self.assertFalse(cn('Sur, l', 'Sur, long'))
- self.assertFalse(cn('Sur, name', 'Sur, word'))
-
- def test_surname_compatibility(self):
- cn = surname_compatibility
- self.assertEqual(cn('Surname','Surname'), 1.0)
- self.assertEqual(cn('Surname','Verynotthesame'), 0.0)
- self.assertTrue(cn('Surname', 'Surnam') > 0.5)
- self.assertTrue(cn('Surname','Surnam' ) < 1.0)
- self.assertTrue(cn('Surname', 'Curname') > 0.5)
- self.assertTrue(cn('test','tast') > 0.5)
-
- def test_initials_compatibility(self):
- cn = initials_compatibility
- self.assertEqual(cn(['a','b','c'],['a','b','c']), 1.0)
- self.assertEqual(cn(['a','b','c'],['d','e','f']), 0.0)
- self.assertTrue(cn(['a','b'], ['a','c']) > 0.25)
- self.assertTrue(cn(['a','b'], ['a','c']) < 0.5)
- self.assertTrue(cn(['a','b'], ['c','a']) > 0.25)
- self.assertTrue(cn(['a','b'], ['c','a']) < 0.5)
-
- def test_value_ranges(self):
- cn = compare_names
- for n1 in self.tc.keys():
- n2 = self.tc[n1][0]
- tests = self.tc[n1][1]
- for test in tests:
- #print 'TESTING: ', n1, n2, ' -- ', cn(n1,n2)
- self.assertTrue(cn(n1,n2) == cn(n2,n1))
- self.assertTrue(test[0](cn(n1,n2), test[1]))
-
-TEST_SUITE = make_test_suite(TestDistanceFunctions,
- TestSoftNameComparison,
- TestSplitNameParts,
- TestCreateCanonicalNames,
- TestCreateNormalizedName,
- TestSoftNameComparison,
- TestNameComparison)
-
-if __name__ == "__main__":
- run_test_suite(TEST_SUITE, warn_user=True)
diff --git a/modules/bibcatalog/lib/bibcatalog_unit_tests.py b/modules/bibcatalog/lib/bibcatalog_unit_tests.py
deleted file mode 100644
index 4b442dbad..000000000
--- a/modules/bibcatalog/lib/bibcatalog_unit_tests.py
+++ /dev/null
@@ -1,94 +0,0 @@
-# -*- coding: utf-8 -*-
-##
-## This file is part of Invenio.
-## Copyright (C) 2013 CERN.
-##
-## Invenio is free software; you can redistribute it and/or
-## modify it under the terms of the GNU General Public License as
-## published by the Free Software Foundation; either version 2 of the
-## License, or (at your option) any later version.
-##
-## Invenio is distributed in the hope that it will be useful, but
-## WITHOUT ANY WARRANTY; without even the implied warranty of
-## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-## General Public License for more details.
-##
-## You should have received a copy of the GNU General Public License
-## along with Invenio; if not, write to the Free Software Foundation, Inc.,
-## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-
-"""
-BibCatalog unit tests
-"""
-
-from invenio.testutils import make_test_suite, run_test_suite, InvenioTestCase
-
-from invenio.bibcatalog_utils import \
- load_tag_code_from_name, \
- split_tag_code, \
- record_get_value_with_provenence, \
- record_id_from_record, \
- record_in_collection, \
- BibCatalogTagNotFound
-from invenio.bibformat_dblayer import get_all_name_tag_mappings
-
-
-class TestUtilityFunctions(InvenioTestCase):
- """Test non-data-specific utility functions for BibCatalog."""
-
- def setUp(self):
- self.record = {'001': [([], ' ', ' ', '1', 1)],
- '650': [([('2', 'SzGeCERN'), ('a', 'Experiments and Tracks')],
- '1', '7', '', 2),
- ([('9', 'arXiv'), ('a', 'hep-ph')],
- '1', '7', '', 3),
- ([('9', 'arXiv'), ('a', 'hep-th')],
- '1', '7', '', 4)],
- '980': [([('a', 'PICTURE')], ' ', ' ', '', 3)]}
-
- def test_load_tag_code_from_name(self):
- """Tests function bibcatalog_utils.load_tag_code_from_name"""
- if 'record ID' in get_all_name_tag_mappings():
- self.assertEqual(load_tag_code_from_name("record ID"), "001")
- # Name "foo" should not exist and raise an exception
- self.assertRaises(BibCatalogTagNotFound, load_tag_code_from_name, "foo")
-
- def test_split_tag_code(self):
- """Tests function bibcatalog_utils.split_tag_code"""
- self.assertEqual(split_tag_code('035__a'),
- {"tag": "035",
- "ind1": "_",
- "ind2": "_",
- "code": "a"})
-
- self.assertEqual(split_tag_code('035'),
- {"tag": "035",
- "ind1": "%",
- "ind2": "%",
- "code": "%"})
-
- def test_record_in_collection(self):
- """Tests function bibcatalog_utils.record_in_collection"""
- self.assertFalse(record_in_collection(self.record, "THESIS"))
- self.assertTrue(record_in_collection(self.record, "picture"))
-
- def test_record_id_from_record(self):
- """Tests function bibcatalog_utils.record_id_from_record"""
- self.assertEqual("1", record_id_from_record(self.record))
-
- def test_record_get_value_with_provenence(self):
- """Tests function bibcatalog_utils.record_get_value_with_provenence"""
- self.assertEqual(["hep-ph", "hep-th"],
- record_get_value_with_provenence(record=self.record,
- provenence_value="arXiv",
- provenence_code="9",
- tag="650",
- ind1="1",
- ind2="7",
- code="a"))
-
-
-TEST_SUITE = make_test_suite(TestUtilityFunctions)
-
-if __name__ == "__main__":
- run_test_suite(TEST_SUITE)
diff --git a/modules/bibcheck/lib/bibcheck_plugins_unit_tests.py b/modules/bibcheck/lib/bibcheck_plugins_unit_tests.py
deleted file mode 100644
index 89de0dadf..000000000
--- a/modules/bibcheck/lib/bibcheck_plugins_unit_tests.py
+++ /dev/null
@@ -1,170 +0,0 @@
-# -*- coding: utf-8 -*-
-##
-## This file is part of Invenio.
-## Copyright (C) 2013 CERN.
-##
-## Invenio is free software; you can redistribute it and/or
-## modify it under the terms of the GNU General Public License as
-## published by the Free Software Foundation; either version 2 of the
-## License, or (at your option) any later version.
-##
-## Invenio is distributed in the hope that it will be useful, but
-## WITHOUT ANY WARRANTY; without even the implied warranty of
-## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-## General Public License for more details.
-##
-## You should have received a copy of the GNU General Public License
-## along with Invenio; if not, write to the Free Software Foundation, Inc.,
-## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-
-
-"""BibTask default plugins Test Suite."""
-
-__revision__ = "$Id$"
-
-from invenio.testutils import make_test_suite, run_test_suite, InvenioTestCase
-from invenio.bibcheck_plugins import mandatory, \
- trailing_space, \
- regexp, \
- utf8, \
- enum, \
- dates, \
- texkey, \
- url
-from invenio.bibcheck_task import AmendableRecord
-
-MOCK_RECORD = {
- '001': [([], ' ', ' ', '1', 7)],
- '005': [([], ' ', ' ', '20130621172205.0', 7)],
- '100': [([('a', 'Photolab ')], ' ', ' ', '', 7)], # Trailing spaces
- '260': [([('c', '2000-06-14')], ' ', ' ', '', 7)],
- '261': [([('c', '14 Jun 2000')], ' ', ' ', '', 7)],
- '262': [([('c', '14 06 00')], ' ', ' ', '', 7)],
- '263': [([('c', '2000 06 14')], ' ', ' ', '', 7)],
- '264': [([('c', '1750 06 14')], ' ', ' ', '', 7)],
- '265': [([('c', '2100 06 14')], ' ', ' ', '', 7)],
- '340': [([('a', 'FI\xc3\x28LM')], ' ', ' ', '', 7)], # Invalid utf-8
- '595': [([('a', ' Press')], ' ', ' ', '', 7)], # Leading spaces
- '653': [([('a', 'LEP')], '1', ' ', '', 7)],
- '856': [([('f', 'neil.calder@cern.ch')], '0', ' ', '', 7)],
- '994': [([('u', 'http://httpstat.us/200')], '4', ' ', '', 7)], # Url that works
- '995': [([('u', 'www.google.com/favicon.ico')], '4', ' ', '', 7)], # url without protocol
- '996': [([('u', 'httpstat.us/301')], '4', ' ', '', 7)], # redirection without protocol
- '997': [([('u', 'http://httpstat.us/404')], '4', ' ', '', 7)], # Error 404
- '998': [([('u', 'http://httpstat.us/500')], '4', ' ', '', 7)], # Error 500
- '999': [([('u', 'http://httpstat.us/301')], '4', ' ', '', 7)], # Permanent redirect
-}
-
-RULE_MOCK = {
- "name": "test_rule",
- "holdingpen": True
-}
-
-class BibCheckPluginsTest(InvenioTestCase):
- """ Bibcheck default plugins test """
-
- def assertAmends(self, test, changes, **kwargs):
- """
- Assert that the plugin "test" amends the mock record when called with
- params kwargs.
- """
- record = AmendableRecord(MOCK_RECORD)
- record.set_rule(RULE_MOCK)
- test.check_record(record, **kwargs)
- self.assertTrue(record.amended)
- self.assertEqual(len(record.amendments), len(changes))
- for field, val in changes.iteritems():
- if val is not None:
- self.assertEqual(
- [((field, 0, 0), val)],
- list(record.iterfield(field))
- )
- else:
- self.assertEqual(len(list(record.iterfield(field))), 1)
-
- def assertFails(self, test, **kwargs):
- """
- Assert that the plugin test marks the record as invalid when called with
- params kwargs.
- """
- record = AmendableRecord(MOCK_RECORD)
- record.set_rule(RULE_MOCK)
- test.check_record(record, **kwargs)
- self.assertFalse(record.valid)
- self.assertTrue(len(record.errors) > 0)
-
- def assertOk(self, test, **kwargs):
- """
- Assert that the plugin test doesn't make any modification to the record
- when called with params kwargs.
- """
- record = AmendableRecord(MOCK_RECORD)
- record.set_rule(RULE_MOCK)
- test.check_record(record, **kwargs)
- self.assertTrue(record.valid)
- self.assertFalse(record.amended)
- self.assertEqual(len(record.amendments), 0)
- self.assertEqual(len(record.errors), 0)
-
- def test_mandatory(self):
- """ Mandatory fields plugin test """
- self.assertOk(mandatory, fields=["100%%a", "260%%c"])
- self.assertFails(mandatory, fields=["100%%b"])
- self.assertFails(mandatory, fields=["111%%%"])
-
- def test_trailing_space(self):
- """ Trailing space plugin test """
- self.assertAmends(trailing_space, {"100__a": "Photolab"}, fields=["100%%a"])
- self.assertAmends(trailing_space, {"100__a": "Photolab"}, fields=["595%%a", "100%%a"])
- self.assertOk(trailing_space, fields=["653%%a"])
-
- def test_regexp(self):
- """ Regexp plugin test """
- self.assertOk(regexp, regexps={
- "856%%f": "[^@]+@[^@]+$",
- "260%%c": r"\d{4}-\d\d-\d\d$"
- })
- self.assertFails(regexp, regexps={
- "340%%a": "foobar"
- })
-
- def test_utf8(self):
- """ Valid utf-8 plugin test """
- self.assertFails(utf8, fields=["%%%%%%"])
- self.assertOk(utf8, fields=["856%%%"])
-
- def test_enum(self):
- """ Enum plugin test """
- self.assertFails(enum, allowed_values={"100__a": ["Pepe", "Juan"]})
- self.assertOk(enum, allowed_values={"6531_a": ["LEP", "Other"]})
-
- def test_date(self):
- """ Date plugin test """
- self.assertOk(dates, fields=["260__c"])
- self.assertAmends(dates, {"261__c": "2000-06-14"}, fields=["261__c"])
- self.assertAmends(dates, {"262__c": "2000-06-14"}, fields=["262__c"])
- self.assertAmends(dates, {"263__c": "2000-06-14"}, fields=["263__c"])
- self.assertFails(dates, fields=["264__c"]) # Date in the far past
- self.assertFails(dates, fields=["265__c"], allow_future=False) # Date in the future
- self.assertFails(dates, fields=["100__a"]) # Invalid date
-
- def test_texkey(self):
- """ TexKey plugin test """
- self.assertAmends(texkey, {"035__a": None})
-
- # Test skipped by default because it involved making slow http requests
- #def test_url(self):
- # """ Url checker plugin test. This plugin is disabled by default """
- # self.assertOk(url, fields=["994%%u"])
- # self.assertAmends(url, {"9954_u": "http://www.google.com/favicon.ico"}, fields=["995%%u"])
- # self.assertAmends(url, {"9964_u": "http://httpstat.us"}, fields=["996%%u"])
- # self.assertFails(url, fields=["997%%u"])
- # self.assertFails(url, fields=["998%%u"])
- # self.assertAmends(url, {"9994_u": "http://httpstat.us"}, fields=["999%%u"])
-
-
-TEST_SUITE = make_test_suite(BibCheckPluginsTest)
-
-if __name__ == "__main__":
- run_test_suite(TEST_SUITE)
-
diff --git a/modules/bibcheck/lib/bibcheck_unit_tests.py b/modules/bibcheck/lib/bibcheck_unit_tests.py
deleted file mode 100644
index 6957fd28b..000000000
--- a/modules/bibcheck/lib/bibcheck_unit_tests.py
+++ /dev/null
@@ -1,201 +0,0 @@
-# -*- coding: utf-8 -*-
-##
-## This file is part of Invenio.
-## Copyright (C) 2013 CERN.
-##
-## Invenio is free software; you can redistribute it and/or
-## modify it under the terms of the GNU General Public License as
-## published by the Free Software Foundation; either version 2 of the
-## License, or (at your option) any later version.
-##
-## Invenio is distributed in the hope that it will be useful, but
-## WITHOUT ANY WARRANTY; without even the implied warranty of
-## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-## General Public License for more details.
-##
-## You should have received a copy of the GNU General Public License
-## along with Invenio; if not, write to the Free Software Foundation, Inc.,
-## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-
-
-"""BibTask Regression Test Suite."""
-
-__revision__ = "$Id$"
-
-from ConfigParser import RawConfigParser
-from invenio.testutils import make_test_suite, run_test_suite, InvenioTestCase
-from invenio.bibcheck_task import RulesParseError, \
- load_rule, \
- AmendableRecord
-from invenio.bibcheck_plugins_unit_tests import MOCK_RECORD
-
-PLUGINS_MOCK = {
- "valid_checker": {
- "check_record": lambda x: None,
- "all_args": set(
- "param_true param_number param_str param_obj param_arr".split()
- ),
- "mandatory_args": set()
- },
- "mandatory_args": {
- "check_record": lambda x: None,
- "all_args": set(["mandatory_arg"]),
- "mandatory_args": set(["mandatory_arg"])
- },
-}
-PLUGINS_MOCK["other_checker"] = PLUGINS_MOCK["valid_checker"]
-
-RULE_MOCK = {
- "name": "test_rule",
- "holdingpen": True
-}
-
-INVALID_RULES = (
- ("empty_rule", {}, ""),
- ("invalid_checker", {
- "check": "non_existent_checker"
- }, "Invalid checker"),
- ("invalid_parameter", {
- "check": "valid_checker",
- "check.param": "Invalid JSON"
- }, "Invalid value"),
- ("invalid_option", {
- "check": "valid_checker",
- "non_existent_option": "true"
- }, "Invalid rule option"),
- ("no_checker", {
- "filter_collection": "CERN"
- }, "Doesn't have a checker"),
- ("unknown_parameter", {
- "check": "mandatory_args",
- "check.non_existent_arg": "true",
- "check.mandatory_arg": "true"
- }, "Unknown plugin argument"),
- ("mandatory_arg", {
- "check": "mandatory_args"
- }, "mandatory argument"),
-)
-
-
-class BibCheckRulesParseTest(InvenioTestCase):
- """ Tests the rule parse functionality """
- def test_invalid_rule(self):
- """ Makes sure the parser raises an error with invalid rules """
- config = RawConfigParser()
-
- # Create sections in the config file
- for rule_name, options, _ in INVALID_RULES:
- config.add_section(rule_name)
- for option_name, val in options.items():
- config.set(rule_name, option_name, val)
-
- # Test invalid sections that should fail to parse
- for rule_name, _, exception in INVALID_RULES:
- try:
- load_rule(config, PLUGINS_MOCK, rule_name)
- self.fail()
- except RulesParseError, ex:
- if str(ex).find(exception) < 0:
- self.fail()
-
- def test_valid_rule(self):
- """ Checks that rules are parsed correctly """
- config = RawConfigParser()
- config.add_section("rule1")
- config.set("rule1", "check", "valid_checker")
- config.set("rule1", "check.param_true", "true")
- config.set("rule1", "check.param_number", "1337")
- config.set("rule1", "check.param_str", '"foobar"')
- config.set("rule1", "check.param_obj", '{"foo":"bar"}')
- config.set("rule1", "check.param_arr", '[true, 1337, ["foobar"]]')
- config.set("rule1", "filter_pattern", "foo")
- config.set("rule1", "filter_field", "bar")
- config.set("rule1", "filter_collection", "baz")
- config.set("rule1", "holdingpen", "true")
-
- config.add_section("rule2")
- config.set("rule2", "check", "other_checker")
-
- rule1 = load_rule(config, PLUGINS_MOCK, "rule1")
- rule2 = load_rule(config, PLUGINS_MOCK, "rule2")
-
- self.assertEqual(rule1["check"], "valid_checker")
- self.assertTrue(rule1["checker_params"]["param_true"])
- self.assertEqual(rule1["checker_params"]["param_number"], 1337)
- self.assertEqual(rule1["checker_params"]["param_str"], "foobar")
- self.assertEqual(rule1["checker_params"]["param_obj"], {"foo": "bar"})
- self.assertEqual(rule1["checker_params"]["param_arr"], [True, 1337, ["foobar"]])
- self.assertEqual(rule1["filter_pattern"], "foo")
- self.assertEqual(rule1["filter_field"], "bar")
- self.assertEqual(rule1["filter_collection"], "baz")
- self.assertEqual(rule1["holdingpen"], True)
-
- self.assertEqual(rule2["check"], "other_checker")
-
-
-class BibCheckAmendableRecordTest(InvenioTestCase):
- """ Check the AmendableRecord class """
-
- def setUp(self):
- """ Create a mock amenda record to test with """
- self.record = AmendableRecord(MOCK_RECORD)
- self.record.set_rule(RULE_MOCK)
-
- def test_valid(self):
- """ Test the set_invalid method """
- self.assertTrue(self.record.valid)
- self.record.set_invalid("test message")
- self.assertFalse(self.record.valid)
- self.assertEqual(self.record.errors, ["Rule test_rule: test message"])
-
- def test_amend(self):
- """ Test the amend method """
- self.assertFalse(self.record.amendments)
- self.record.amend_field(("100__a", 0, 0), "Pepe", "Changed author")
- self.assertEqual(self.record["100"][0][0][0][1], "Pepe")
- self.assertTrue(self.record.amended)
- self.assertEqual(self.record.amendments, ["Rule test_rule: Changed author"])
-
- def test_itertags(self):
- """ Test the itertags method """
- self.assertEqual(
- set(self.record.keys()),
- set(self.record.itertags("%%%"))
- )
- self.assertEqual(set(['100']), set(self.record.itertags("100")))
- self.assertEqual(set(['001', '005']), set(self.record.itertags("00%")))
- self.assertEqual(set(), set(self.record.itertags("111")))
-
- def test_iterfields(self):
- """ Test the iterfields method """
- self.assertEqual(set(), set(self.record.iterfields(["111%%%"])))
- self.assertEqual(
- set([(("100__a", 0, 0), "Pepe")]),
- set(self.record.iterfields(["1%%%%%"]))
- )
- self.assertEqual(
- set([(("9944_u", 0, 0), self.record["994"][0][0][0][1]),
- (("9954_u", 0, 0), self.record["995"][0][0][0][1]),
- (("9964_u", 0, 0), self.record["996"][0][0][0][1]),
- (("9974_u", 0, 0), self.record["997"][0][0][0][1]),
- (("9984_u", 0, 0), self.record["998"][0][0][0][1]),
- (("9994_u", 0, 0), self.record["999"][0][0][0][1])]),
- set(self.record.iterfields(["9%%%%u"]))
- )
-
- def test_is_dummy(self):
- """ Test the is_dummy method """
- dummy_record = {
- '001': [([], ' ', ' ', '1', 1)]
- }
- record = AmendableRecord(dummy_record)
- self.assertTrue(record.is_dummy())
-
-TEST_SUITE = make_test_suite(
- BibCheckRulesParseTest,
- BibCheckAmendableRecordTest
- )
-
-if __name__ == "__main__":
- run_test_suite(TEST_SUITE)
-
diff --git a/modules/bibrank/lib/bibrank_publication_grapher.py b/modules/bibrank/lib/bibrank_publication_grapher.py
deleted file mode 100644
index 1d7c84bf1..000000000
--- a/modules/bibrank/lib/bibrank_publication_grapher.py
+++ /dev/null
@@ -1,75 +0,0 @@
-# -*- coding: utf-8 -*-
-##
-## This file is part of Invenio.
-## Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011 CERN.
-##
-## Invenio is free software; you can redistribute it and/or
-## modify it under the terms of the GNU General Public License as
-## published by the Free Software Foundation; either version 2 of the
-## License, or (at your option) any later version.
-##
-## Invenio is distributed in the hope that it will be useful, but
-## WITHOUT ANY WARRANTY; without even the implied warranty of
-## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-## General Public License for more details.
-##
-## You should have received a copy of the GNU General Public License
-## along with Invenio; if not, write to the Free Software Foundation, Inc.,
-## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
-
-import os
-
-from invenio.config import CFG_SITE_URL, CFG_WEBDIR
-from invenio.bibrank_grapher import create_temporary_image, write_coordinates_in_tmp_file
-
-def html_image(filename, width, height):
- """
- Returns html code for showing publication graph image (with lightbox js effect)
- @param filename: str (graph image name)
- @param width: int (image width)
- @param height: int (image height)
- @return: str (html code)
- """
- html = """<a href='%s/img/%s' rel="lightbox"> <img src='%s/img/%s' width="%s" height="%s"
- alt=""> </a>""" % (CFG_SITE_URL, filename, CFG_SITE_URL, filename, str(width), str(height))
- return html
-
-def create_graph_image(graph_file_name, graph_data):
- """
- Creates a new graph image with the given data.
- @param graph_file_name: str (graph image name)
- @param graph_data: list (data for the graph plot)
- @return: str (full name of the graph image)
- """
- res = ''
- if not os.path.exists("%s/img/tmp" % (CFG_WEBDIR)):
- os.mkdir("%s/img/tmp" % (CFG_WEBDIR))
- if not os.path.exists("%s/img/tmp/%s" % (CFG_WEBDIR, graph_file_name[0])):
- os.mkdir("%s/img/tmp/%s" % (CFG_WEBDIR, graph_file_name[0]))
- datas_info = write_coordinates_in_tmp_file([graph_data])
- years = [tup[0] for tup in graph_data]
- graph = create_temporary_image(graph_file_name, 'pubs_per_year', datas_info[0], 'Year',
- 'Times published', [0, 0], datas_info[1], [], ' ', years)
- graph_image = graph[0]
- graph_source_file = graph[1]
- if graph_image and graph_source_file and os.path.exists(graph_source_file):
- os.unlink(datas_info[0])
- res = graph_image
- return res
-
-def get_graph_code(graph_file_name, graph_data):
- """
- Creates HTML code refering to the 'publications per year' graph image.
- If the graph image does not exist in the filesystem, it creates a new one.
- @param graph_file_name: str (graph image name)
- @param graph_data: list (data for the graph plot)
- @return: str (html code)
- """
- html_graph_code = ''
- if os.path.exists('%s/img/tmp/%s/%s.png' % (CFG_WEBDIR, graph_file_name[0], graph_file_name)):
- graph_image = 'tmp/%s/%s.png' % (graph_file_name[0], graph_file_name)
- html_graph_code = """<p>%s""" % html_image(graph_image, 350, 200)
- else:
- graph_image = create_graph_image(graph_file_name, graph_data)
- html_graph_code = """<p>%s""" % html_image(graph_image, 350, 200)
- return html_graph_code
diff --git a/modules/webauthorprofile/lib/webauthorprofile_unit_tests.py b/modules/webauthorprofile/lib/webauthorprofile_unit_tests.py
index 2bc983283..981086193 100644
--- a/modules/webauthorprofile/lib/webauthorprofile_unit_tests.py
+++ b/modules/webauthorprofile/lib/webauthorprofile_unit_tests.py
@@ -1,71 +1,71 @@
# -*- coding: utf-8 -*-
##
## This file is part of Invenio.
## Copyright (C) 2010, 2011 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
# pylint: disable=E1102
"""
WebAuthorProfile unit tests
"""
from threading import Thread
from time import sleep
from invenio.testutils import make_test_suite, run_test_suite, InvenioTestCase
from invenio.webauthorprofile_dbapi import expire_cache_element
from invenio.webauthorprofile_corefunctions import foo, _foo
class WebAuthorProfileTest(InvenioTestCase):
""" Test functions to check the validator of WebAuthorProfile. """
def test_caching(self):
""" Test if the main corefuntions work correctly. """
- res1 = _foo(1,2,3,0)
- res2, status2, = foo(1,2,3,0)
- res3, status3, = foo(1,2,3,0)
+ res1 = _foo(1, 2, 3, 0)
+ res2, status2 = foo(1, 2, 3, 0)
+ res3, status3 = foo(1, 2, 3, 0)
self.assertEqual(res1, res2)
self.assertEqual(True, status2)
self.assertEqual(res2, res3)
self.assertEqual(status2, status3)
def test_caching2(self):
""" Test if precaching works """
def handler(reslist, secs):
- reslist.append(foo(1,2,3,secs))
+ reslist.append(foo(1, 2, 3, secs))
def make_thread(secs):
result = []
thread = Thread(target=handler, args=(result, secs))
return (thread, result)
expire_cache_element('foo', 1)
thread1, res1 = make_thread(1)
thread2, res2 = make_thread(0)
thread1.start()
sleep(0.5)
thread2.start()
thread1.join()
thread2.join()
self.assertNotEqual(res1[0][0], res2[0][0])
self.assertNotEqual(res1[0][1], res2[0][1])
expire_cache_element('foo', 1)
TEST_SUITE = make_test_suite(WebAuthorProfileTest)
if __name__ == "__main__":
run_test_suite(TEST_SUITE, warn_user=False)
diff --git a/po/fa.po b/po/fa.po
index 72b2ba982..8ff7e10c1 100644
--- a/po/fa.po
+++ b/po/fa.po
@@ -1,12925 +1,12936 @@
# # This file is part of Invenio.
-# # Copyright (C) 2013 CERN.
+# # Copyright (C) 2013, 2014 CERN.
# #
# # Invenio is free software; you can redistribute it and/or
# # modify it under the terms of the GNU General Public License as
# # published by the Free Software Foundation; either version 2 of the
# # License, or (at your option) any later version.
# #
# # Invenio is distributed in the hope that it will be useful, but
# # WITHOUT ANY WARRANTY; without even the implied warranty of
# # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# # General Public License for more details.
# #
# # You should have received a copy of the GNU General Public License
# # along with Invenio; if not, write to the Free Software Foundation, Inc.,
# # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
msgid ""
msgstr ""
"Project-Id-Version: Invenio 1.0.0-rc0\n"
"Report-Msgid-Bugs-To: info@invenio-software.org\n"
"POT-Creation-Date: 2011-12-19 22:12+0100\n"
-"PO-Revision-Date: 2013-10-30 22:17+0100\n"
-"Last-Translator: Tibor Simko <tibor.simko@cern.ch>\n"
+"PO-Revision-Date: 2014-04-12 06:30+0330\n"
+"Last-Translator: Mehdi Zahedi <mehdizahedin@gmail.com>\n"
"Language-Team: FA <info@invenio-software.org>\n"
-"Language: \n"
+"Language: fa_IR\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: pygettext.py 1.5\n"
+"X-Generator: Poedit 1.6.4\n"
+"Plural-Forms: nplurals=1; plural=0;\n"
#: modules/websearch/doc/search-guide.webdoc:361
#: modules/websearch/doc/search-guide.webdoc:396
#: modules/websearch/doc/search-guide.webdoc:493
#: modules/websearch/doc/search-guide.webdoc:528
#: modules/websearch/doc/search-guide.webdoc:630
#: modules/websearch/doc/search-guide.webdoc:665
#: modules/websearch/doc/search-guide.webdoc:768
#: modules/websearch/doc/search-guide.webdoc:803
#: modules/websearch/lib/search_engine.py:1200
#: modules/websearch/lib/websearch_templates.py:1152
msgid "AND NOT"
-msgstr ""
+msgstr "و نه"
#: modules/webhelp/web/admin/admin.webdoc:18
#: modules/websearch/doc/admin/websearch-admin-guide.webdoc:21
#: modules/websubmit/doc/admin/websubmit-admin-guide.webdoc:21
#: modules/bibedit/doc/admin/bibedit-admin-guide.webdoc:21
#: modules/bibupload/doc/admin/bibupload-admin-guide.webdoc:21
#: modules/bibformat/doc/admin/bibformat-admin-guide.webdoc:21
#: modules/bibharvest/doc/admin/bibharvest-admin-guide.webdoc:21
#: modules/webmessage/doc/admin/webmessage-admin-guide.webdoc:21
#: modules/webalert/doc/admin/webalert-admin-guide.webdoc:21
#: modules/bibclassify/doc/admin/bibclassify-admin-guide.webdoc:22
#: modules/bibmatch/doc/admin/bibmatch-admin-guide.webdoc:21
#: modules/bibconvert/doc/admin/bibconvert-admin-guide.webdoc:21
#: modules/bibsched/doc/admin/bibsched-admin-guide.webdoc:21
#: modules/bibrank/doc/admin/bibrank-admin-guide.webdoc:21
#: modules/webstat/doc/admin/webstat-admin-guide.webdoc:21
#: modules/bibindex/doc/admin/bibindex-admin-guide.webdoc:21
#: modules/webbasket/doc/admin/webbasket-admin-guide.webdoc:21
#: modules/webcomment/doc/admin/webcomment-admin-guide.webdoc:21
#: modules/websession/doc/admin/websession-admin-guide.webdoc:21
#: modules/webstyle/doc/admin/webstyle-admin-guide.webdoc:21
#: modules/elmsubmit/doc/admin/elmsubmit-admin-guide.webdoc:21
#: modules/bibformat/lib/bibformatadminlib.py:61
#: modules/bibformat/web/admin/bibformatadmin.py:70
#: modules/webcomment/lib/webcommentadminlib.py:45
#: modules/webstyle/lib/webdoc_webinterface.py:153
#: modules/bibcheck/web/admin/bibcheckadmin.py:57
#: modules/bibcheck/web/admin/bibcheckadmin.py:159
#: modules/bibcheck/web/admin/bibcheckadmin.py:203
#: modules/bibcheck/web/admin/bibcheckadmin.py:263
#: modules/bibcheck/web/admin/bibcheckadmin.py:302
#: modules/bibknowledge/lib/bibknowledgeadmin.py:70
msgid "Admin Area"
-msgstr ""
+msgstr "ناحیه مدیر"
#: modules/websearch/doc/search-guide.webdoc:427
#: modules/websearch/doc/search-guide.webdoc:559
#: modules/websearch/doc/search-guide.webdoc:696
#: modules/websearch/doc/search-guide.webdoc:834
#: modules/websearch/lib/search_engine.py:4692
#: modules/websearch/lib/websearch_templates.py:793
#: modules/websearch/lib/websearch_templates.py:871
#: modules/websearch/lib/websearch_templates.py:994
#: modules/websearch/lib/websearch_templates.py:1965
#: modules/websearch/lib/websearch_templates.py:2056
#: modules/websearch/lib/websearch_templates.py:2113
#: modules/websearch/lib/websearch_templates.py:2170
#: modules/websearch/lib/websearch_templates.py:2209
#: modules/websearch/lib/websearch_templates.py:2232
#: modules/websearch/lib/websearch_templates.py:2263
msgid "Browse"
-msgstr ""
+msgstr "مرور"
#: modules/webhelp/web/help-central.webdoc:50
#: modules/websearch/doc/search-tips.webdoc:20
#: modules/websearch/lib/websearch_templates.py:794
#: modules/websearch/lib/websearch_templates.py:872
#: modules/websearch/lib/websearch_templates.py:995
#: modules/websearch/lib/websearch_templates.py:2060
#: modules/websearch/lib/websearch_templates.py:2117
#: modules/websearch/lib/websearch_templates.py:2174
msgid "Search Tips"
-msgstr ""
+msgstr "نکات جستجو"
#: modules/websearch/doc/search-guide.webdoc:343
#: modules/websearch/doc/search-guide.webdoc:378
#: modules/websearch/doc/search-guide.webdoc:413
#: modules/websearch/doc/search-guide.webdoc:475
#: modules/websearch/doc/search-guide.webdoc:510
#: modules/websearch/doc/search-guide.webdoc:545
#: modules/websearch/doc/search-guide.webdoc:612
#: modules/websearch/doc/search-guide.webdoc:647
#: modules/websearch/doc/search-guide.webdoc:682
#: modules/websearch/doc/search-guide.webdoc:750
#: modules/websearch/doc/search-guide.webdoc:785
#: modules/websearch/doc/search-guide.webdoc:820
#: modules/miscutil/lib/inveniocfg.py:482
msgid "abstract"
-msgstr ""
+msgstr "چکیده"
#: modules/websearch/doc/search-guide.webdoc:348
#: modules/websearch/doc/search-guide.webdoc:383
#: modules/websearch/doc/search-guide.webdoc:418
#: modules/websearch/doc/search-guide.webdoc:480
#: modules/websearch/doc/search-guide.webdoc:515
#: modules/websearch/doc/search-guide.webdoc:550
#: modules/websearch/doc/search-guide.webdoc:617
#: modules/websearch/doc/search-guide.webdoc:652
#: modules/websearch/doc/search-guide.webdoc:687
#: modules/websearch/doc/search-guide.webdoc:755
#: modules/websearch/doc/search-guide.webdoc:790
#: modules/websearch/doc/search-guide.webdoc:825
#: modules/miscutil/lib/inveniocfg.py:487
msgid "fulltext"
-msgstr ""
+msgstr "تمام متن"
#: modules/websearch/doc/search-guide.webdoc:337
#: modules/websearch/doc/search-guide.webdoc:373
#: modules/websearch/doc/search-guide.webdoc:408
#: modules/websearch/doc/search-guide.webdoc:469
#: modules/websearch/doc/search-guide.webdoc:505
#: modules/websearch/doc/search-guide.webdoc:540
#: modules/websearch/doc/search-guide.webdoc:606
#: modules/websearch/doc/search-guide.webdoc:642
#: modules/websearch/doc/search-guide.webdoc:677
#: modules/websearch/doc/search-guide.webdoc:744
#: modules/websearch/doc/search-guide.webdoc:780
#: modules/websearch/doc/search-guide.webdoc:815
#: modules/websearch/lib/search_engine.py:1222
#: modules/websearch/lib/websearch_templates.py:1108
msgid "Regular expression:"
msgstr ""
#: modules/websearch/doc/search-guide.webdoc:333
#: modules/websearch/doc/search-guide.webdoc:369
#: modules/websearch/doc/search-guide.webdoc:404
#: modules/websearch/doc/search-guide.webdoc:465
#: modules/websearch/doc/search-guide.webdoc:501
#: modules/websearch/doc/search-guide.webdoc:536
#: modules/websearch/doc/search-guide.webdoc:602
#: modules/websearch/doc/search-guide.webdoc:638
#: modules/websearch/doc/search-guide.webdoc:673
#: modules/websearch/doc/search-guide.webdoc:740
#: modules/websearch/doc/search-guide.webdoc:776
#: modules/websearch/doc/search-guide.webdoc:811
#: modules/websearch/lib/search_engine.py:1218
#: modules/websearch/lib/websearch_templates.py:1100
msgid "All of the words:"
-msgstr ""
+msgstr "همه کلمات"
#: modules/websearch/doc/search-guide.webdoc:351
#: modules/websearch/doc/search-guide.webdoc:386
#: modules/websearch/doc/search-guide.webdoc:421
#: modules/websearch/doc/search-guide.webdoc:483
#: modules/websearch/doc/search-guide.webdoc:518
#: modules/websearch/doc/search-guide.webdoc:553
#: modules/websearch/doc/search-guide.webdoc:620
#: modules/websearch/doc/search-guide.webdoc:655
#: modules/websearch/doc/search-guide.webdoc:690
#: modules/websearch/doc/search-guide.webdoc:758
#: modules/websearch/doc/search-guide.webdoc:793
#: modules/websearch/doc/search-guide.webdoc:828
#: modules/miscutil/lib/inveniocfg.py:484
msgid "report number"
-msgstr ""
+msgstr "شماره گزارش"
#: modules/websearch/doc/search-tips.webdoc:406
#: modules/websearch/doc/search-tips.webdoc:413
#: modules/websearch/doc/search-tips.webdoc:414
#: modules/websearch/doc/search-tips.webdoc:415
#: modules/websearch/doc/search-tips.webdoc:433
#: modules/websearch/doc/search-tips.webdoc:434
#: modules/websearch/doc/search-tips.webdoc:435
#: modules/websearch/doc/search-guide.webdoc:354
#: modules/websearch/doc/search-guide.webdoc:389
#: modules/websearch/doc/search-guide.webdoc:424
#: modules/websearch/doc/search-guide.webdoc:486
#: modules/websearch/doc/search-guide.webdoc:521
#: modules/websearch/doc/search-guide.webdoc:556
#: modules/websearch/doc/search-guide.webdoc:623
#: modules/websearch/doc/search-guide.webdoc:658
#: modules/websearch/doc/search-guide.webdoc:693
#: modules/websearch/doc/search-guide.webdoc:761
#: modules/websearch/doc/search-guide.webdoc:796
#: modules/websearch/doc/search-guide.webdoc:831
#: modules/miscutil/lib/inveniocfg.py:490
#: modules/bibcirculation/lib/bibcirculation_templates.py:7218
#: modules/bibcirculation/lib/bibcirculation_templates.py:7917
msgid "year"
-msgstr ""
+msgstr "سال"
#: modules/websearch/doc/search-guide.webdoc:352
#: modules/websearch/doc/search-guide.webdoc:387
#: modules/websearch/doc/search-guide.webdoc:422
#: modules/websearch/doc/search-guide.webdoc:484
#: modules/websearch/doc/search-guide.webdoc:519
#: modules/websearch/doc/search-guide.webdoc:554
#: modules/websearch/doc/search-guide.webdoc:621
#: modules/websearch/doc/search-guide.webdoc:656
#: modules/websearch/doc/search-guide.webdoc:691
#: modules/websearch/doc/search-guide.webdoc:759
#: modules/websearch/doc/search-guide.webdoc:794
#: modules/websearch/doc/search-guide.webdoc:829
#: modules/miscutil/lib/inveniocfg.py:485
msgid "subject"
-msgstr ""
+msgstr "موضوع"
#: modules/websearch/doc/search-guide.webdoc:336
#: modules/websearch/doc/search-guide.webdoc:372
#: modules/websearch/doc/search-guide.webdoc:407
#: modules/websearch/doc/search-guide.webdoc:468
#: modules/websearch/doc/search-guide.webdoc:504
#: modules/websearch/doc/search-guide.webdoc:539
#: modules/websearch/doc/search-guide.webdoc:605
#: modules/websearch/doc/search-guide.webdoc:641
#: modules/websearch/doc/search-guide.webdoc:676
#: modules/websearch/doc/search-guide.webdoc:743
#: modules/websearch/doc/search-guide.webdoc:779
#: modules/websearch/doc/search-guide.webdoc:814
#: modules/websearch/lib/search_engine.py:1221
#: modules/websearch/lib/websearch_templates.py:1106
msgid "Partial phrase:"
-msgstr ""
+msgstr "عبارت نسبی"
#: modules/websearch/doc/search-guide.webdoc:350
#: modules/websearch/doc/search-guide.webdoc:385
#: modules/websearch/doc/search-guide.webdoc:420
#: modules/websearch/doc/search-guide.webdoc:482
#: modules/websearch/doc/search-guide.webdoc:517
#: modules/websearch/doc/search-guide.webdoc:552
#: modules/websearch/doc/search-guide.webdoc:619
#: modules/websearch/doc/search-guide.webdoc:654
#: modules/websearch/doc/search-guide.webdoc:689
#: modules/websearch/doc/search-guide.webdoc:757
#: modules/websearch/doc/search-guide.webdoc:792
#: modules/websearch/doc/search-guide.webdoc:827
#: modules/miscutil/lib/inveniocfg.py:486
msgid "reference"
-msgstr ""
+msgstr "ارجاع"
#: modules/websearch/doc/search-tips.webdoc:37
#: modules/websearch/doc/search-tips.webdoc:68
#: modules/websearch/doc/search-tips.webdoc:106
#: modules/websearch/doc/search-tips.webdoc:154
#: modules/websearch/doc/search-tips.webdoc:177
#: modules/websearch/doc/search-tips.webdoc:201
#: modules/websearch/doc/search-tips.webdoc:246
#: modules/websearch/doc/search-tips.webdoc:286
#: modules/websearch/doc/search-tips.webdoc:297
#: modules/websearch/doc/search-tips.webdoc:317
#: modules/websearch/doc/search-tips.webdoc:337
#: modules/websearch/doc/search-tips.webdoc:371
#: modules/websearch/doc/search-tips.webdoc:405
#: modules/websearch/doc/search-tips.webdoc:426
#: modules/websearch/doc/search-tips.webdoc:446
#: modules/websearch/doc/search-tips.webdoc:480
#: modules/websearch/doc/search-tips.webdoc:498
#: modules/websearch/doc/search-tips.webdoc:517
#: modules/websearch/doc/search-tips.webdoc:550
#: modules/websearch/doc/search-tips.webdoc:575
#: modules/websearch/doc/search-tips.webdoc:583
#: modules/websearch/doc/search-tips.webdoc:586
#: modules/websearch/doc/search-tips.webdoc:588
#: modules/websearch/doc/search-tips.webdoc:599
#: modules/websearch/doc/search-tips.webdoc:620
#: modules/websearch/doc/search-guide.webdoc:226
#: modules/websearch/doc/search-guide.webdoc:250
#: modules/websearch/doc/search-guide.webdoc:276
#: modules/websearch/doc/search-guide.webdoc:301
#: modules/websearch/doc/search-guide.webdoc:344
#: modules/websearch/doc/search-guide.webdoc:379
#: modules/websearch/doc/search-guide.webdoc:414
#: modules/websearch/doc/search-guide.webdoc:476
#: modules/websearch/doc/search-guide.webdoc:511
#: modules/websearch/doc/search-guide.webdoc:546
#: modules/websearch/doc/search-guide.webdoc:613
#: modules/websearch/doc/search-guide.webdoc:648
#: modules/websearch/doc/search-guide.webdoc:683
#: modules/websearch/doc/search-guide.webdoc:751
#: modules/websearch/doc/search-guide.webdoc:786
#: modules/websearch/doc/search-guide.webdoc:821
#: modules/websearch/doc/search-guide.webdoc:881
#: modules/websearch/doc/search-guide.webdoc:912
#: modules/websearch/doc/search-guide.webdoc:952
#: modules/websearch/doc/search-guide.webdoc:986
#: modules/websearch/doc/search-guide.webdoc:1026
#: modules/websearch/doc/search-guide.webdoc:1048
#: modules/websearch/doc/search-guide.webdoc:1068
#: modules/websearch/doc/search-guide.webdoc:1084
#: modules/websearch/doc/search-guide.webdoc:1124
#: modules/websearch/doc/search-guide.webdoc:1147
#: modules/websearch/doc/search-guide.webdoc:1168
#: modules/websearch/doc/search-guide.webdoc:1183
#: modules/websearch/doc/search-guide.webdoc:1227
#: modules/websearch/doc/search-guide.webdoc:1252
#: modules/websearch/doc/search-guide.webdoc:1273
#: modules/websearch/doc/search-guide.webdoc:1289
#: modules/websearch/doc/search-guide.webdoc:1334
#: modules/websearch/doc/search-guide.webdoc:1357
#: modules/websearch/doc/search-guide.webdoc:1379
#: modules/websearch/doc/search-guide.webdoc:1395
#: modules/websearch/doc/search-guide.webdoc:1765
#: modules/websearch/doc/search-guide.webdoc:1779
#: modules/websearch/doc/search-guide.webdoc:1797
#: modules/websearch/doc/search-guide.webdoc:1816
#: modules/websearch/doc/search-guide.webdoc:1829
#: modules/websearch/doc/search-guide.webdoc:1847
#: modules/websearch/doc/search-guide.webdoc:1867
#: modules/websearch/doc/search-guide.webdoc:1882
#: modules/websearch/doc/search-guide.webdoc:1901
#: modules/websearch/doc/search-guide.webdoc:1924
#: modules/websearch/doc/search-guide.webdoc:1939
#: modules/websearch/doc/search-guide.webdoc:1958
#: modules/websearch/doc/search-guide.webdoc:1986
#: modules/websearch/doc/search-guide.webdoc:2024
#: modules/websearch/doc/search-guide.webdoc:2035
#: modules/websearch/doc/search-guide.webdoc:2049
#: modules/websearch/doc/search-guide.webdoc:2063
#: modules/websearch/doc/search-guide.webdoc:2076
#: modules/websearch/doc/search-guide.webdoc:2092
#: modules/websearch/doc/search-guide.webdoc:2103
#: modules/websearch/doc/search-guide.webdoc:2117
#: modules/websearch/doc/search-guide.webdoc:2131
#: modules/websearch/doc/search-guide.webdoc:2144
#: modules/websearch/doc/search-guide.webdoc:2160
#: modules/websearch/doc/search-guide.webdoc:2171
#: modules/websearch/doc/search-guide.webdoc:2185
#: modules/websearch/doc/search-guide.webdoc:2199
#: modules/websearch/doc/search-guide.webdoc:2212
#: modules/websearch/doc/search-guide.webdoc:2230
#: modules/websearch/doc/search-guide.webdoc:2241
#: modules/websearch/doc/search-guide.webdoc:2255
#: modules/websearch/doc/search-guide.webdoc:2269
#: modules/websearch/doc/search-guide.webdoc:2282
#: modules/websearch/doc/search-guide.webdoc:2311
#: modules/websearch/doc/search-guide.webdoc:2325
#: modules/websearch/doc/search-guide.webdoc:2342
#: modules/websearch/doc/search-guide.webdoc:2355
#: modules/websearch/doc/search-guide.webdoc:2372
#: modules/websearch/doc/search-guide.webdoc:2386
#: modules/websearch/doc/search-guide.webdoc:2404
#: modules/websearch/doc/search-guide.webdoc:2418
#: modules/websearch/doc/search-guide.webdoc:2449
#: modules/websearch/doc/search-guide.webdoc:2464
#: modules/websearch/doc/search-guide.webdoc:2478
#: modules/websearch/doc/search-guide.webdoc:2493
#: modules/websearch/doc/search-guide.webdoc:2521
#: modules/websearch/doc/search-guide.webdoc:2536
#: modules/websearch/doc/search-guide.webdoc:2550
#: modules/websearch/doc/search-guide.webdoc:2566
#: modules/websearch/doc/search-guide.webdoc:2598
#: modules/websearch/doc/search-guide.webdoc:2614
#: modules/websearch/doc/search-guide.webdoc:2628
#: modules/websearch/doc/search-guide.webdoc:2643
#: modules/websearch/doc/search-guide.webdoc:2674
#: modules/websearch/doc/search-guide.webdoc:2690
#: modules/websearch/doc/search-guide.webdoc:2704
#: modules/websearch/doc/search-guide.webdoc:2719
#: modules/websearch/doc/search-guide.webdoc:2761
#: modules/websearch/doc/search-guide.webdoc:2776
#: modules/websearch/doc/search-guide.webdoc:2790
#: modules/websearch/doc/search-guide.webdoc:2815
#: modules/websearch/doc/search-guide.webdoc:2830
#: modules/websearch/doc/search-guide.webdoc:2844
#: modules/websearch/doc/search-guide.webdoc:2873
#: modules/websearch/doc/search-guide.webdoc:2888
#: modules/websearch/doc/search-guide.webdoc:2902
#: modules/websearch/doc/search-guide.webdoc:2930
#: modules/websearch/doc/search-guide.webdoc:2945
#: modules/websearch/doc/search-guide.webdoc:2958
#: modules/websearch/doc/search-guide.webdoc:2993
#: modules/websearch/doc/search-guide.webdoc:3015
#: modules/websearch/doc/search-guide.webdoc:3039
#: modules/websearch/doc/search-guide.webdoc:3063
#: modules/websearch/doc/search-guide.webdoc:3087
#: modules/websearch/doc/search-guide.webdoc:3102
#: modules/websearch/doc/search-guide.webdoc:3118
#: modules/websearch/doc/search-guide.webdoc:3135
#: modules/websearch/doc/search-guide.webdoc:3155
#: modules/websearch/doc/search-guide.webdoc:3173
#: modules/websearch/doc/search-guide.webdoc:3191
#: modules/websearch/doc/search-guide.webdoc:3210
#: modules/websearch/doc/search-guide.webdoc:3231
#: modules/websearch/doc/search-guide.webdoc:3245
#: modules/websearch/doc/search-guide.webdoc:3265
#: modules/websearch/doc/search-guide.webdoc:3280
#: modules/websearch/doc/search-guide.webdoc:3299
#: modules/websearch/doc/search-guide.webdoc:3314
#: modules/websearch/doc/search-guide.webdoc:3334
#: modules/websearch/doc/search-guide.webdoc:3349
#: modules/websearch/doc/search-guide.webdoc:3411
#: modules/websearch/doc/search-guide.webdoc:3425
#: modules/websearch/doc/search-guide.webdoc:3442
#: modules/websearch/doc/search-guide.webdoc:3455
#: modules/websearch/doc/search-guide.webdoc:3473
#: modules/websearch/doc/search-guide.webdoc:3488
#: modules/websearch/doc/search-guide.webdoc:3506
#: modules/websearch/doc/search-guide.webdoc:3521
#: modules/websearch/doc/search-guide.webdoc:3546
#: modules/websearch/doc/search-guide.webdoc:3559
#: modules/websearch/doc/search-guide.webdoc:3572
#: modules/websearch/doc/search-guide.webdoc:3588
#: modules/websearch/doc/search-guide.webdoc:3604
#: modules/websearch/doc/search-guide.webdoc:3621
#: modules/websearch/doc/search-guide.webdoc:3654
#: modules/websearch/doc/search-guide.webdoc:3670
#: modules/websearch/doc/search-guide.webdoc:3687
#: modules/websearch/doc/search-guide.webdoc:3707
#: modules/websearch/doc/search-guide.webdoc:3721
#: modules/websearch/doc/search-guide.webdoc:3739
#: modules/websearch/doc/search-guide.webdoc:3760
#: modules/websearch/doc/search-guide.webdoc:3779
#: modules/websearch/doc/search-guide.webdoc:3797
#: modules/websearch/doc/search-guide.webdoc:3819
#: modules/websearch/doc/search-guide.webdoc:3838
#: modules/websearch/doc/search-guide.webdoc:3855
#: modules/websearch/doc/search-guide.webdoc:3976
#: modules/websearch/doc/search-guide.webdoc:4001
#: modules/websearch/doc/search-guide.webdoc:4024
#: modules/websearch/doc/search-guide.webdoc:4050
#: modules/websearch/doc/search-guide.webdoc:4074
#: modules/websearch/doc/search-guide.webdoc:4101
#: modules/websearch/doc/search-guide.webdoc:4126
#: modules/websearch/doc/search-guide.webdoc:4152
#: modules/websearch/doc/search-guide.webdoc:4181
#: modules/websearch/doc/search-guide.webdoc:4201
#: modules/websearch/doc/search-guide.webdoc:4225
#: modules/websearch/doc/search-guide.webdoc:4252
#: modules/websearch/doc/search-guide.webdoc:4292
#: modules/websearch/doc/search-guide.webdoc:4313
#: modules/websearch/doc/search-guide.webdoc:4337
#: modules/websearch/doc/search-guide.webdoc:4367
#: modules/websearch/doc/search-guide.webdoc:4411
#: modules/websearch/doc/search-guide.webdoc:4433
#: modules/websearch/doc/search-guide.webdoc:4458
#: modules/websearch/doc/search-guide.webdoc:4488
#: modules/websearch/doc/search-guide.webdoc:4533
#: modules/websearch/doc/search-guide.webdoc:4554
#: modules/websearch/doc/search-guide.webdoc:4579
#: modules/websearch/doc/search-guide.webdoc:4609
#: modules/websearch/doc/search-guide.webdoc:4901
#: modules/websearch/doc/search-guide.webdoc:4917
#: modules/websearch/doc/search-guide.webdoc:4937
#: modules/websearch/doc/search-guide.webdoc:4956
#: modules/websearch/doc/search-guide.webdoc:4977
#: modules/websearch/doc/search-guide.webdoc:4995
#: modules/websearch/doc/search-guide.webdoc:5016
#: modules/websearch/doc/search-guide.webdoc:5034
#: modules/websearch/doc/search-guide.webdoc:5067
#: modules/websearch/doc/search-guide.webdoc:5081
#: modules/websearch/doc/search-guide.webdoc:5096
#: modules/websearch/doc/search-guide.webdoc:5112
#: modules/websearch/doc/search-guide.webdoc:5131
#: modules/websearch/doc/search-guide.webdoc:5145
#: modules/websearch/doc/search-guide.webdoc:5161
#: modules/websearch/doc/search-guide.webdoc:5179
#: modules/websearch/doc/search-guide.webdoc:5198
#: modules/websearch/doc/search-guide.webdoc:5213
#: modules/websearch/doc/search-guide.webdoc:5228
#: modules/websearch/doc/search-guide.webdoc:5246
#: modules/websearch/doc/search-guide.webdoc:5266
#: modules/websearch/doc/search-guide.webdoc:5281
#: modules/websearch/doc/search-guide.webdoc:5296
#: modules/websearch/doc/search-guide.webdoc:5316
#: modules/webstyle/doc/hacking/webstyle-webdoc-syntax.webdoc:131
#: modules/miscutil/lib/inveniocfg.py:481
#: modules/bibcirculation/lib/bibcirculation_templates.py:2019
#: modules/bibcirculation/lib/bibcirculation_templates.py:7219
#: modules/bibcirculation/lib/bibcirculation_templates.py:7918
#: modules/bibcirculation/lib/bibcirculation_templates.py:9140
#: modules/bibcirculation/lib/bibcirculation_templates.py:9148
#: modules/bibcirculation/lib/bibcirculation_templates.py:9156
#: modules/bibcirculation/lib/bibcirculation_templates.py:9164
#: modules/bibcirculation/lib/bibcirculation_templates.py:16025
msgid "author"
-msgstr ""
+msgstr "نویسنده"
#: modules/webhelp/web/help-central.webdoc:98
#: modules/websearch/doc/search-guide.webdoc:20
msgid "Search Guide"
-msgstr ""
+msgstr "راهنمای جستجو"
#: modules/websearch/doc/search-guide.webdoc:347
#: modules/websearch/doc/search-guide.webdoc:382
#: modules/websearch/doc/search-guide.webdoc:417
#: modules/websearch/doc/search-guide.webdoc:479
#: modules/websearch/doc/search-guide.webdoc:514
#: modules/websearch/doc/search-guide.webdoc:549
#: modules/websearch/doc/search-guide.webdoc:616
#: modules/websearch/doc/search-guide.webdoc:651
#: modules/websearch/doc/search-guide.webdoc:686
#: modules/websearch/doc/search-guide.webdoc:754
#: modules/websearch/doc/search-guide.webdoc:789
#: modules/websearch/doc/search-guide.webdoc:824
#: modules/miscutil/lib/inveniocfg.py:492
msgid "experiment"
-msgstr ""
+msgstr "آزمایش"
#: modules/websearch/doc/search-guide.webdoc:334
#: modules/websearch/doc/search-guide.webdoc:370
#: modules/websearch/doc/search-guide.webdoc:405
#: modules/websearch/doc/search-guide.webdoc:466
#: modules/websearch/doc/search-guide.webdoc:502
#: modules/websearch/doc/search-guide.webdoc:537
#: modules/websearch/doc/search-guide.webdoc:603
#: modules/websearch/doc/search-guide.webdoc:639
#: modules/websearch/doc/search-guide.webdoc:674
#: modules/websearch/doc/search-guide.webdoc:741
#: modules/websearch/doc/search-guide.webdoc:777
#: modules/websearch/doc/search-guide.webdoc:812
#: modules/websearch/lib/search_engine.py:1219
#: modules/websearch/lib/websearch_templates.py:1102
msgid "Any of the words:"
-msgstr ""
+msgstr "هر کدام از کلمات"
#: modules/websearch/doc/search-guide.webdoc:346
#: modules/websearch/doc/search-guide.webdoc:381
#: modules/websearch/doc/search-guide.webdoc:416
#: modules/websearch/doc/search-guide.webdoc:478
#: modules/websearch/doc/search-guide.webdoc:513
#: modules/websearch/doc/search-guide.webdoc:548
#: modules/websearch/doc/search-guide.webdoc:615
#: modules/websearch/doc/search-guide.webdoc:650
#: modules/websearch/doc/search-guide.webdoc:685
#: modules/websearch/doc/search-guide.webdoc:753
#: modules/websearch/doc/search-guide.webdoc:788
#: modules/websearch/doc/search-guide.webdoc:823
#: modules/miscutil/lib/inveniocfg.py:489
msgid "division"
-msgstr ""
+msgstr "بخش ها"
#: modules/websearch/doc/search-tips.webdoc:39
#: modules/websearch/doc/search-tips.webdoc:70
#: modules/websearch/doc/search-tips.webdoc:108
#: modules/websearch/doc/search-tips.webdoc:156
#: modules/websearch/doc/search-tips.webdoc:179
#: modules/websearch/doc/search-tips.webdoc:203
#: modules/websearch/doc/search-tips.webdoc:248
#: modules/websearch/doc/search-tips.webdoc:288
#: modules/websearch/doc/search-tips.webdoc:299
#: modules/websearch/doc/search-tips.webdoc:319
#: modules/websearch/doc/search-tips.webdoc:339
#: modules/websearch/doc/search-tips.webdoc:373
#: modules/websearch/doc/search-tips.webdoc:408
#: modules/websearch/doc/search-tips.webdoc:428
#: modules/websearch/doc/search-tips.webdoc:448
#: modules/websearch/doc/search-tips.webdoc:482
#: modules/websearch/doc/search-tips.webdoc:500
#: modules/websearch/doc/search-tips.webdoc:519
#: modules/websearch/doc/search-tips.webdoc:552
#: modules/websearch/doc/search-tips.webdoc:577
#: modules/websearch/doc/search-tips.webdoc:601
#: modules/websearch/doc/search-tips.webdoc:622
#: modules/websearch/doc/search-guide.webdoc:227
#: modules/websearch/doc/search-guide.webdoc:251
#: modules/websearch/doc/search-guide.webdoc:277
#: modules/websearch/doc/search-guide.webdoc:302
#: modules/websearch/doc/search-guide.webdoc:427
#: modules/websearch/doc/search-guide.webdoc:559
#: modules/websearch/doc/search-guide.webdoc:696
#: modules/websearch/doc/search-guide.webdoc:834
#: modules/websearch/doc/search-guide.webdoc:882
#: modules/websearch/doc/search-guide.webdoc:913
#: modules/websearch/doc/search-guide.webdoc:953
#: modules/websearch/doc/search-guide.webdoc:987
#: modules/websearch/doc/search-guide.webdoc:1027
#: modules/websearch/doc/search-guide.webdoc:1049
#: modules/websearch/doc/search-guide.webdoc:1069
#: modules/websearch/doc/search-guide.webdoc:1085
#: modules/websearch/doc/search-guide.webdoc:1125
#: modules/websearch/doc/search-guide.webdoc:1148
#: modules/websearch/doc/search-guide.webdoc:1169
#: modules/websearch/doc/search-guide.webdoc:1184
#: modules/websearch/doc/search-guide.webdoc:1228
#: modules/websearch/doc/search-guide.webdoc:1253
#: modules/websearch/doc/search-guide.webdoc:1274
#: modules/websearch/doc/search-guide.webdoc:1290
#: modules/websearch/doc/search-guide.webdoc:1335
#: modules/websearch/doc/search-guide.webdoc:1358
#: modules/websearch/doc/search-guide.webdoc:1380
#: modules/websearch/doc/search-guide.webdoc:1396
#: modules/websearch/doc/search-guide.webdoc:1766
#: modules/websearch/doc/search-guide.webdoc:1780
#: modules/websearch/doc/search-guide.webdoc:1798
#: modules/websearch/doc/search-guide.webdoc:1817
#: modules/websearch/doc/search-guide.webdoc:1830
#: modules/websearch/doc/search-guide.webdoc:1848
#: modules/websearch/doc/search-guide.webdoc:1868
#: modules/websearch/doc/search-guide.webdoc:1883
#: modules/websearch/doc/search-guide.webdoc:1902
#: modules/websearch/doc/search-guide.webdoc:1925
#: modules/websearch/doc/search-guide.webdoc:1940
#: modules/websearch/doc/search-guide.webdoc:1959
#: modules/websearch/doc/search-guide.webdoc:1988
#: modules/websearch/doc/search-guide.webdoc:2025
#: modules/websearch/doc/search-guide.webdoc:2036
#: modules/websearch/doc/search-guide.webdoc:2050
#: modules/websearch/doc/search-guide.webdoc:2064
#: modules/websearch/doc/search-guide.webdoc:2077
#: modules/websearch/doc/search-guide.webdoc:2093
#: modules/websearch/doc/search-guide.webdoc:2104
#: modules/websearch/doc/search-guide.webdoc:2118
#: modules/websearch/doc/search-guide.webdoc:2132
#: modules/websearch/doc/search-guide.webdoc:2145
#: modules/websearch/doc/search-guide.webdoc:2161
#: modules/websearch/doc/search-guide.webdoc:2172
#: modules/websearch/doc/search-guide.webdoc:2186
#: modules/websearch/doc/search-guide.webdoc:2200
#: modules/websearch/doc/search-guide.webdoc:2213
#: modules/websearch/doc/search-guide.webdoc:2231
#: modules/websearch/doc/search-guide.webdoc:2242
#: modules/websearch/doc/search-guide.webdoc:2256
#: modules/websearch/doc/search-guide.webdoc:2270
#: modules/websearch/doc/search-guide.webdoc:2283
#: modules/websearch/doc/search-guide.webdoc:2312
#: modules/websearch/doc/search-guide.webdoc:2326
#: modules/websearch/doc/search-guide.webdoc:2343
#: modules/websearch/doc/search-guide.webdoc:2356
#: modules/websearch/doc/search-guide.webdoc:2373
#: modules/websearch/doc/search-guide.webdoc:2387
#: modules/websearch/doc/search-guide.webdoc:2405
#: modules/websearch/doc/search-guide.webdoc:2419
#: modules/websearch/doc/search-guide.webdoc:2450
#: modules/websearch/doc/search-guide.webdoc:2465
#: modules/websearch/doc/search-guide.webdoc:2479
#: modules/websearch/doc/search-guide.webdoc:2494
#: modules/websearch/doc/search-guide.webdoc:2522
#: modules/websearch/doc/search-guide.webdoc:2537
#: modules/websearch/doc/search-guide.webdoc:2551
#: modules/websearch/doc/search-guide.webdoc:2567
#: modules/websearch/doc/search-guide.webdoc:2599
#: modules/websearch/doc/search-guide.webdoc:2615
#: modules/websearch/doc/search-guide.webdoc:2629
#: modules/websearch/doc/search-guide.webdoc:2644
#: modules/websearch/doc/search-guide.webdoc:2675
#: modules/websearch/doc/search-guide.webdoc:2691
#: modules/websearch/doc/search-guide.webdoc:2705
#: modules/websearch/doc/search-guide.webdoc:2720
#: modules/websearch/doc/search-guide.webdoc:2762
#: modules/websearch/doc/search-guide.webdoc:2777
#: modules/websearch/doc/search-guide.webdoc:2791
#: modules/websearch/doc/search-guide.webdoc:2816
#: modules/websearch/doc/search-guide.webdoc:2831
#: modules/websearch/doc/search-guide.webdoc:2845
#: modules/websearch/doc/search-guide.webdoc:2874
#: modules/websearch/doc/search-guide.webdoc:2889
#: modules/websearch/doc/search-guide.webdoc:2903
#: modules/websearch/doc/search-guide.webdoc:2931
#: modules/websearch/doc/search-guide.webdoc:2946
#: modules/websearch/doc/search-guide.webdoc:2959
#: modules/websearch/doc/search-guide.webdoc:2994
#: modules/websearch/doc/search-guide.webdoc:3016
#: modules/websearch/doc/search-guide.webdoc:3040
#: modules/websearch/doc/search-guide.webdoc:3064
#: modules/websearch/doc/search-guide.webdoc:3088
#: modules/websearch/doc/search-guide.webdoc:3103
#: modules/websearch/doc/search-guide.webdoc:3119
#: modules/websearch/doc/search-guide.webdoc:3136
#: modules/websearch/doc/search-guide.webdoc:3156
#: modules/websearch/doc/search-guide.webdoc:3174
#: modules/websearch/doc/search-guide.webdoc:3192
#: modules/websearch/doc/search-guide.webdoc:3211
#: modules/websearch/doc/search-guide.webdoc:3232
#: modules/websearch/doc/search-guide.webdoc:3246
#: modules/websearch/doc/search-guide.webdoc:3266
#: modules/websearch/doc/search-guide.webdoc:3281
#: modules/websearch/doc/search-guide.webdoc:3300
#: modules/websearch/doc/search-guide.webdoc:3315
#: modules/websearch/doc/search-guide.webdoc:3335
#: modules/websearch/doc/search-guide.webdoc:3350
#: modules/websearch/doc/search-guide.webdoc:3412
#: modules/websearch/doc/search-guide.webdoc:3426
#: modules/websearch/doc/search-guide.webdoc:3443
#: modules/websearch/doc/search-guide.webdoc:3456
#: modules/websearch/doc/search-guide.webdoc:3474
#: modules/websearch/doc/search-guide.webdoc:3489
#: modules/websearch/doc/search-guide.webdoc:3507
#: modules/websearch/doc/search-guide.webdoc:3522
#: modules/websearch/doc/search-guide.webdoc:3547
#: modules/websearch/doc/search-guide.webdoc:3560
#: modules/websearch/doc/search-guide.webdoc:3573
#: modules/websearch/doc/search-guide.webdoc:3589
#: modules/websearch/doc/search-guide.webdoc:3605
#: modules/websearch/doc/search-guide.webdoc:3622
#: modules/websearch/doc/search-guide.webdoc:3655
#: modules/websearch/doc/search-guide.webdoc:3671
#: modules/websearch/doc/search-guide.webdoc:3688
#: modules/websearch/doc/search-guide.webdoc:3708
#: modules/websearch/doc/search-guide.webdoc:3722
#: modules/websearch/doc/search-guide.webdoc:3740
#: modules/websearch/doc/search-guide.webdoc:3761
#: modules/websearch/doc/search-guide.webdoc:3780
#: modules/websearch/doc/search-guide.webdoc:3798
#: modules/websearch/doc/search-guide.webdoc:3820
#: modules/websearch/doc/search-guide.webdoc:3839
#: modules/websearch/doc/search-guide.webdoc:3856
#: modules/websearch/doc/search-guide.webdoc:3977
#: modules/websearch/doc/search-guide.webdoc:4002
#: modules/websearch/doc/search-guide.webdoc:4025
#: modules/websearch/doc/search-guide.webdoc:4051
#: modules/websearch/doc/search-guide.webdoc:4075
#: modules/websearch/doc/search-guide.webdoc:4102
#: modules/websearch/doc/search-guide.webdoc:4127
#: modules/websearch/doc/search-guide.webdoc:4153
#: modules/websearch/doc/search-guide.webdoc:4182
#: modules/websearch/doc/search-guide.webdoc:4202
#: modules/websearch/doc/search-guide.webdoc:4226
#: modules/websearch/doc/search-guide.webdoc:4253
#: modules/websearch/doc/search-guide.webdoc:4293
#: modules/websearch/doc/search-guide.webdoc:4314
#: modules/websearch/doc/search-guide.webdoc:4338
#: modules/websearch/doc/search-guide.webdoc:4368
#: modules/websearch/doc/search-guide.webdoc:4412
#: modules/websearch/doc/search-guide.webdoc:4434
#: modules/websearch/doc/search-guide.webdoc:4459
#: modules/websearch/doc/search-guide.webdoc:4489
#: modules/websearch/doc/search-guide.webdoc:4534
#: modules/websearch/doc/search-guide.webdoc:4555
#: modules/websearch/doc/search-guide.webdoc:4580
#: modules/websearch/doc/search-guide.webdoc:4610
#: modules/websearch/doc/search-guide.webdoc:4902
#: modules/websearch/doc/search-guide.webdoc:4918
#: modules/websearch/doc/search-guide.webdoc:4938
#: modules/websearch/doc/search-guide.webdoc:4957
#: modules/websearch/doc/search-guide.webdoc:4978
#: modules/websearch/doc/search-guide.webdoc:4996
#: modules/websearch/doc/search-guide.webdoc:5017
#: modules/websearch/doc/search-guide.webdoc:5035
#: modules/websearch/doc/search-guide.webdoc:5068
#: modules/websearch/doc/search-guide.webdoc:5082
#: modules/websearch/doc/search-guide.webdoc:5097
#: modules/websearch/doc/search-guide.webdoc:5113
#: modules/websearch/doc/search-guide.webdoc:5132
#: modules/websearch/doc/search-guide.webdoc:5146
#: modules/websearch/doc/search-guide.webdoc:5162
#: modules/websearch/doc/search-guide.webdoc:5180
#: modules/websearch/doc/search-guide.webdoc:5199
#: modules/websearch/doc/search-guide.webdoc:5214
#: modules/websearch/doc/search-guide.webdoc:5229
#: modules/websearch/doc/search-guide.webdoc:5247
#: modules/websearch/doc/search-guide.webdoc:5267
#: modules/websearch/doc/search-guide.webdoc:5282
#: modules/websearch/doc/search-guide.webdoc:5297
#: modules/websearch/doc/search-guide.webdoc:5317
#: modules/webstyle/doc/hacking/webstyle-webdoc-syntax.webdoc:133
#: modules/websearch/lib/websearch_templates.py:792
#: modules/websearch/lib/websearch_templates.py:870
#: modules/websearch/lib/websearch_templates.py:993
#: modules/websearch/lib/websearch_templates.py:1962
#: modules/websearch/lib/websearch_templates.py:2055
#: modules/websearch/lib/websearch_templates.py:2112
#: modules/websearch/lib/websearch_templates.py:2169
#: modules/webstyle/lib/webstyle_templates.py:433
#: modules/webstyle/lib/webstyle_templates.py:502
#: modules/webstyle/lib/webdoc_tests.py:86
#: modules/bibedit/lib/bibeditmulti_templates.py:312
#: modules/bibcirculation/lib/bibcirculation_templates.py:161
#: modules/bibcirculation/lib/bibcirculation_templates.py:207
#: modules/bibcirculation/lib/bibcirculation_templates.py:1977
#: modules/bibcirculation/lib/bibcirculation_templates.py:2052
#: modules/bibcirculation/lib/bibcirculation_templates.py:2245
#: modules/bibcirculation/lib/bibcirculation_templates.py:4088
#: modules/bibcirculation/lib/bibcirculation_templates.py:6806
#: modules/bibcirculation/lib/bibcirculation_templates.py:7251
#: modules/bibcirculation/lib/bibcirculation_templates.py:7948
#: modules/bibcirculation/lib/bibcirculation_templates.py:8600
#: modules/bibcirculation/lib/bibcirculation_templates.py:9190
#: modules/bibcirculation/lib/bibcirculation_templates.py:9687
#: modules/bibcirculation/lib/bibcirculation_templates.py:10162
#: modules/bibcirculation/lib/bibcirculation_templates.py:14248
#: modules/bibcirculation/lib/bibcirculation_templates.py:14571
#: modules/bibcirculation/lib/bibcirculation_templates.py:15243
#: modules/bibcirculation/lib/bibcirculation_templates.py:16055
#: modules/bibcirculation/lib/bibcirculation_templates.py:17122
#: modules/bibcirculation/lib/bibcirculation_templates.py:17604
#: modules/bibcirculation/lib/bibcirculation_templates.py:17788
#: modules/bibknowledge/lib/bibknowledge_templates.py:82
#: modules/bibknowledge/lib/bibknowledge_templates.py:423
msgid "Search"
-msgstr ""
+msgstr "جستجو"
#: modules/webhelp/web/help-central.webdoc:133
msgid "Citation Metrics"
-msgstr ""
+msgstr "سنجه های استنادی"
#: modules/websearch/doc/search-tips.webdoc:35
#: modules/websearch/doc/search-tips.webdoc:66
#: modules/websearch/doc/search-tips.webdoc:104
#: modules/websearch/doc/search-tips.webdoc:152
#: modules/websearch/doc/search-tips.webdoc:175
#: modules/websearch/doc/search-tips.webdoc:199
#: modules/websearch/doc/search-tips.webdoc:244
#: modules/websearch/doc/search-tips.webdoc:284
#: modules/websearch/doc/search-tips.webdoc:295
#: modules/websearch/doc/search-tips.webdoc:315
#: modules/websearch/doc/search-tips.webdoc:335
#: modules/websearch/doc/search-tips.webdoc:369
#: modules/websearch/doc/search-tips.webdoc:403
#: modules/websearch/doc/search-tips.webdoc:424
#: modules/websearch/doc/search-tips.webdoc:444
#: modules/websearch/doc/search-tips.webdoc:478
#: modules/websearch/doc/search-tips.webdoc:496
#: modules/websearch/doc/search-tips.webdoc:515
#: modules/websearch/doc/search-tips.webdoc:548
#: modules/websearch/doc/search-tips.webdoc:559
#: modules/websearch/doc/search-tips.webdoc:561
#: modules/websearch/doc/search-tips.webdoc:564
#: modules/websearch/doc/search-tips.webdoc:573
#: modules/websearch/doc/search-tips.webdoc:597
#: modules/websearch/doc/search-tips.webdoc:618
#: modules/websearch/doc/search-guide.webdoc:224
#: modules/websearch/doc/search-guide.webdoc:248
#: modules/websearch/doc/search-guide.webdoc:274
#: modules/websearch/doc/search-guide.webdoc:299
#: modules/websearch/doc/search-guide.webdoc:342
#: modules/websearch/doc/search-guide.webdoc:377
#: modules/websearch/doc/search-guide.webdoc:412
#: modules/websearch/doc/search-guide.webdoc:474
#: modules/websearch/doc/search-guide.webdoc:509
#: modules/websearch/doc/search-guide.webdoc:544
#: modules/websearch/doc/search-guide.webdoc:611
#: modules/websearch/doc/search-guide.webdoc:646
#: modules/websearch/doc/search-guide.webdoc:681
#: modules/websearch/doc/search-guide.webdoc:749
#: modules/websearch/doc/search-guide.webdoc:784
#: modules/websearch/doc/search-guide.webdoc:819
#: modules/websearch/doc/search-guide.webdoc:879
#: modules/websearch/doc/search-guide.webdoc:910
#: modules/websearch/doc/search-guide.webdoc:950
#: modules/websearch/doc/search-guide.webdoc:984
#: modules/websearch/doc/search-guide.webdoc:1024
#: modules/websearch/doc/search-guide.webdoc:1046
#: modules/websearch/doc/search-guide.webdoc:1066
#: modules/websearch/doc/search-guide.webdoc:1082
#: modules/websearch/doc/search-guide.webdoc:1122
#: modules/websearch/doc/search-guide.webdoc:1145
#: modules/websearch/doc/search-guide.webdoc:1166
#: modules/websearch/doc/search-guide.webdoc:1181
#: modules/websearch/doc/search-guide.webdoc:1225
#: modules/websearch/doc/search-guide.webdoc:1250
#: modules/websearch/doc/search-guide.webdoc:1271
#: modules/websearch/doc/search-guide.webdoc:1287
#: modules/websearch/doc/search-guide.webdoc:1332
#: modules/websearch/doc/search-guide.webdoc:1355
#: modules/websearch/doc/search-guide.webdoc:1377
#: modules/websearch/doc/search-guide.webdoc:1393
#: modules/websearch/doc/search-guide.webdoc:1763
#: modules/websearch/doc/search-guide.webdoc:1777
#: modules/websearch/doc/search-guide.webdoc:1795
#: modules/websearch/doc/search-guide.webdoc:1814
#: modules/websearch/doc/search-guide.webdoc:1827
#: modules/websearch/doc/search-guide.webdoc:1845
#: modules/websearch/doc/search-guide.webdoc:1865
#: modules/websearch/doc/search-guide.webdoc:1880
#: modules/websearch/doc/search-guide.webdoc:1899
#: modules/websearch/doc/search-guide.webdoc:1922
#: modules/websearch/doc/search-guide.webdoc:1937
#: modules/websearch/doc/search-guide.webdoc:1956
#: modules/websearch/doc/search-guide.webdoc:1984
#: modules/websearch/doc/search-guide.webdoc:2022
#: modules/websearch/doc/search-guide.webdoc:2033
#: modules/websearch/doc/search-guide.webdoc:2047
#: modules/websearch/doc/search-guide.webdoc:2061
#: modules/websearch/doc/search-guide.webdoc:2074
#: modules/websearch/doc/search-guide.webdoc:2090
#: modules/websearch/doc/search-guide.webdoc:2101
#: modules/websearch/doc/search-guide.webdoc:2115
#: modules/websearch/doc/search-guide.webdoc:2129
#: modules/websearch/doc/search-guide.webdoc:2142
#: modules/websearch/doc/search-guide.webdoc:2158
#: modules/websearch/doc/search-guide.webdoc:2169
#: modules/websearch/doc/search-guide.webdoc:2183
#: modules/websearch/doc/search-guide.webdoc:2197
#: modules/websearch/doc/search-guide.webdoc:2210
#: modules/websearch/doc/search-guide.webdoc:2228
#: modules/websearch/doc/search-guide.webdoc:2239
#: modules/websearch/doc/search-guide.webdoc:2253
#: modules/websearch/doc/search-guide.webdoc:2267
#: modules/websearch/doc/search-guide.webdoc:2280
#: modules/websearch/doc/search-guide.webdoc:2309
#: modules/websearch/doc/search-guide.webdoc:2323
#: modules/websearch/doc/search-guide.webdoc:2340
#: modules/websearch/doc/search-guide.webdoc:2353
#: modules/websearch/doc/search-guide.webdoc:2370
#: modules/websearch/doc/search-guide.webdoc:2384
#: modules/websearch/doc/search-guide.webdoc:2402
#: modules/websearch/doc/search-guide.webdoc:2416
#: modules/websearch/doc/search-guide.webdoc:2447
#: modules/websearch/doc/search-guide.webdoc:2462
#: modules/websearch/doc/search-guide.webdoc:2476
#: modules/websearch/doc/search-guide.webdoc:2491
#: modules/websearch/doc/search-guide.webdoc:2519
#: modules/websearch/doc/search-guide.webdoc:2534
#: modules/websearch/doc/search-guide.webdoc:2548
#: modules/websearch/doc/search-guide.webdoc:2564
#: modules/websearch/doc/search-guide.webdoc:2596
#: modules/websearch/doc/search-guide.webdoc:2612
#: modules/websearch/doc/search-guide.webdoc:2626
#: modules/websearch/doc/search-guide.webdoc:2641
#: modules/websearch/doc/search-guide.webdoc:2672
#: modules/websearch/doc/search-guide.webdoc:2688
#: modules/websearch/doc/search-guide.webdoc:2702
#: modules/websearch/doc/search-guide.webdoc:2717
#: modules/websearch/doc/search-guide.webdoc:2759
#: modules/websearch/doc/search-guide.webdoc:2774
#: modules/websearch/doc/search-guide.webdoc:2788
#: modules/websearch/doc/search-guide.webdoc:2813
#: modules/websearch/doc/search-guide.webdoc:2828
#: modules/websearch/doc/search-guide.webdoc:2842
#: modules/websearch/doc/search-guide.webdoc:2871
#: modules/websearch/doc/search-guide.webdoc:2886
#: modules/websearch/doc/search-guide.webdoc:2900
#: modules/websearch/doc/search-guide.webdoc:2928
#: modules/websearch/doc/search-guide.webdoc:2943
#: modules/websearch/doc/search-guide.webdoc:2956
#: modules/websearch/doc/search-guide.webdoc:2991
#: modules/websearch/doc/search-guide.webdoc:3013
#: modules/websearch/doc/search-guide.webdoc:3037
#: modules/websearch/doc/search-guide.webdoc:3061
#: modules/websearch/doc/search-guide.webdoc:3085
#: modules/websearch/doc/search-guide.webdoc:3100
#: modules/websearch/doc/search-guide.webdoc:3116
#: modules/websearch/doc/search-guide.webdoc:3133
#: modules/websearch/doc/search-guide.webdoc:3153
#: modules/websearch/doc/search-guide.webdoc:3171
#: modules/websearch/doc/search-guide.webdoc:3189
#: modules/websearch/doc/search-guide.webdoc:3208
#: modules/websearch/doc/search-guide.webdoc:3229
#: modules/websearch/doc/search-guide.webdoc:3243
#: modules/websearch/doc/search-guide.webdoc:3263
#: modules/websearch/doc/search-guide.webdoc:3278
#: modules/websearch/doc/search-guide.webdoc:3297
#: modules/websearch/doc/search-guide.webdoc:3312
#: modules/websearch/doc/search-guide.webdoc:3332
#: modules/websearch/doc/search-guide.webdoc:3347
#: modules/websearch/doc/search-guide.webdoc:3409
#: modules/websearch/doc/search-guide.webdoc:3423
#: modules/websearch/doc/search-guide.webdoc:3440
#: modules/websearch/doc/search-guide.webdoc:3453
#: modules/websearch/doc/search-guide.webdoc:3471
#: modules/websearch/doc/search-guide.webdoc:3486
#: modules/websearch/doc/search-guide.webdoc:3504
#: modules/websearch/doc/search-guide.webdoc:3519
#: modules/websearch/doc/search-guide.webdoc:3544
#: modules/websearch/doc/search-guide.webdoc:3557
#: modules/websearch/doc/search-guide.webdoc:3570
#: modules/websearch/doc/search-guide.webdoc:3586
#: modules/websearch/doc/search-guide.webdoc:3602
#: modules/websearch/doc/search-guide.webdoc:3619
#: modules/websearch/doc/search-guide.webdoc:3652
#: modules/websearch/doc/search-guide.webdoc:3668
#: modules/websearch/doc/search-guide.webdoc:3685
#: modules/websearch/doc/search-guide.webdoc:3705
#: modules/websearch/doc/search-guide.webdoc:3719
#: modules/websearch/doc/search-guide.webdoc:3737
#: modules/websearch/doc/search-guide.webdoc:3758
#: modules/websearch/doc/search-guide.webdoc:3777
#: modules/websearch/doc/search-guide.webdoc:3795
#: modules/websearch/doc/search-guide.webdoc:3817
#: modules/websearch/doc/search-guide.webdoc:3836
#: modules/websearch/doc/search-guide.webdoc:3853
#: modules/websearch/doc/search-guide.webdoc:3974
#: modules/websearch/doc/search-guide.webdoc:3999
#: modules/websearch/doc/search-guide.webdoc:4022
#: modules/websearch/doc/search-guide.webdoc:4048
#: modules/websearch/doc/search-guide.webdoc:4072
#: modules/websearch/doc/search-guide.webdoc:4099
#: modules/websearch/doc/search-guide.webdoc:4124
#: modules/websearch/doc/search-guide.webdoc:4150
#: modules/websearch/doc/search-guide.webdoc:4179
#: modules/websearch/doc/search-guide.webdoc:4199
#: modules/websearch/doc/search-guide.webdoc:4223
#: modules/websearch/doc/search-guide.webdoc:4250
#: modules/websearch/doc/search-guide.webdoc:4290
#: modules/websearch/doc/search-guide.webdoc:4311
#: modules/websearch/doc/search-guide.webdoc:4335
#: modules/websearch/doc/search-guide.webdoc:4365
#: modules/websearch/doc/search-guide.webdoc:4409
#: modules/websearch/doc/search-guide.webdoc:4431
#: modules/websearch/doc/search-guide.webdoc:4456
#: modules/websearch/doc/search-guide.webdoc:4486
#: modules/websearch/doc/search-guide.webdoc:4531
#: modules/websearch/doc/search-guide.webdoc:4552
#: modules/websearch/doc/search-guide.webdoc:4577
#: modules/websearch/doc/search-guide.webdoc:4607
#: modules/websearch/doc/search-guide.webdoc:4899
#: modules/websearch/doc/search-guide.webdoc:4915
#: modules/websearch/doc/search-guide.webdoc:4935
#: modules/websearch/doc/search-guide.webdoc:4954
#: modules/websearch/doc/search-guide.webdoc:4975
#: modules/websearch/doc/search-guide.webdoc:4993
#: modules/websearch/doc/search-guide.webdoc:5014
#: modules/websearch/doc/search-guide.webdoc:5032
#: modules/websearch/doc/search-guide.webdoc:5065
#: modules/websearch/doc/search-guide.webdoc:5079
#: modules/websearch/doc/search-guide.webdoc:5094
#: modules/websearch/doc/search-guide.webdoc:5110
#: modules/websearch/doc/search-guide.webdoc:5129
#: modules/websearch/doc/search-guide.webdoc:5143
#: modules/websearch/doc/search-guide.webdoc:5159
#: modules/websearch/doc/search-guide.webdoc:5177
#: modules/websearch/doc/search-guide.webdoc:5196
#: modules/websearch/doc/search-guide.webdoc:5211
#: modules/websearch/doc/search-guide.webdoc:5226
#: modules/websearch/doc/search-guide.webdoc:5244
#: modules/websearch/doc/search-guide.webdoc:5264
#: modules/websearch/doc/search-guide.webdoc:5279
#: modules/websearch/doc/search-guide.webdoc:5294
#: modules/websearch/doc/search-guide.webdoc:5314
#: modules/webstyle/doc/hacking/webstyle-webdoc-syntax.webdoc:129
#: modules/miscutil/lib/inveniocfg.py:479
#: modules/bibcirculation/lib/bibcirculation_templates.py:2019
#: modules/bibcirculation/lib/bibcirculation_templates.py:7218
#: modules/bibcirculation/lib/bibcirculation_templates.py:7917
#: modules/bibcirculation/lib/bibcirculation_templates.py:9140
#: modules/bibcirculation/lib/bibcirculation_templates.py:9148
#: modules/bibcirculation/lib/bibcirculation_templates.py:9156
#: modules/bibcirculation/lib/bibcirculation_templates.py:9164
#: modules/bibcirculation/lib/bibcirculation_templates.py:16024
msgid "any field"
-msgstr ""
+msgstr "هر فیلد"
#: modules/webhelp/web/help-central.webdoc:20
#: modules/webhelp/web/help-central.webdoc:25
#: modules/webhelp/web/help-central.webdoc:26
#: modules/webhelp/web/help-central.webdoc:27
#: modules/webhelp/web/help-central.webdoc:28
#: modules/webhelp/web/help-central.webdoc:29
#: modules/webhelp/web/help-central.webdoc:30
#: modules/webhelp/web/help-central.webdoc:31
#: modules/webhelp/web/help-central.webdoc:32
#: modules/webhelp/web/help-central.webdoc:33
#: modules/webhelp/web/help-central.webdoc:34
#: modules/webhelp/web/help-central.webdoc:35
#: modules/webhelp/web/help-central.webdoc:36
#: modules/webhelp/web/help-central.webdoc:37
#: modules/webhelp/web/help-central.webdoc:38
#: modules/webhelp/web/help-central.webdoc:39
#: modules/webhelp/web/help-central.webdoc:40
#: modules/webhelp/web/help-central.webdoc:41
#: modules/webhelp/web/help-central.webdoc:42
#: modules/webhelp/web/help-central.webdoc:43
#: modules/webhelp/web/help-central.webdoc:44
#: modules/webhelp/web/help-central.webdoc:45
#: modules/websearch/doc/search-tips.webdoc:21
#: modules/websearch/doc/search-guide.webdoc:21
#: modules/websubmit/doc/submit-guide.webdoc:21
#: modules/webstyle/lib/webdoc_tests.py:105
#: modules/webstyle/lib/webdoc_webinterface.py:155
msgid "Help Central"
msgstr ""
#: modules/bibformat/etc/format_templates/Default_HTML_actions.bft:6
msgid "Export as"
-msgstr ""
+msgstr "خروجی به صورت "
#: modules/websearch/doc/search-guide.webdoc:345
#: modules/websearch/doc/search-guide.webdoc:380
#: modules/websearch/doc/search-guide.webdoc:415
#: modules/websearch/doc/search-guide.webdoc:477
#: modules/websearch/doc/search-guide.webdoc:512
#: modules/websearch/doc/search-guide.webdoc:547
#: modules/websearch/doc/search-guide.webdoc:614
#: modules/websearch/doc/search-guide.webdoc:649
#: modules/websearch/doc/search-guide.webdoc:684
#: modules/websearch/doc/search-guide.webdoc:752
#: modules/websearch/doc/search-guide.webdoc:787
#: modules/websearch/doc/search-guide.webdoc:822
#: modules/miscutil/lib/inveniocfg.py:488
msgid "collection"
-msgstr ""
+msgstr "مجموعه"
#: modules/websearch/doc/admin/websearch-admin-guide.webdoc:20
msgid "WebSearch Admin Guide"
msgstr ""
#: modules/websearch/doc/search-guide.webdoc:335
#: modules/websearch/doc/search-guide.webdoc:371
#: modules/websearch/doc/search-guide.webdoc:406
#: modules/websearch/doc/search-guide.webdoc:467
#: modules/websearch/doc/search-guide.webdoc:503
#: modules/websearch/doc/search-guide.webdoc:538
#: modules/websearch/doc/search-guide.webdoc:604
#: modules/websearch/doc/search-guide.webdoc:640
#: modules/websearch/doc/search-guide.webdoc:675
#: modules/websearch/doc/search-guide.webdoc:742
#: modules/websearch/doc/search-guide.webdoc:778
#: modules/websearch/doc/search-guide.webdoc:813
#: modules/websearch/lib/search_engine.py:1220
#: modules/websearch/lib/websearch_templates.py:1104
msgid "Exact phrase:"
-msgstr ""
+msgstr "عبارت دقیق"
#: modules/webhelp/web/help-central.webdoc:108
#: modules/websubmit/doc/submit-guide.webdoc:20
msgid "Submit Guide"
-msgstr ""
+msgstr "راهنمای واگذاری"
#: modules/websearch/doc/search-guide.webdoc:360
#: modules/websearch/doc/search-guide.webdoc:395
#: modules/websearch/doc/search-guide.webdoc:492
#: modules/websearch/doc/search-guide.webdoc:527
#: modules/websearch/doc/search-guide.webdoc:629
#: modules/websearch/doc/search-guide.webdoc:664
#: modules/websearch/doc/search-guide.webdoc:767
#: modules/websearch/doc/search-guide.webdoc:802
#: modules/websearch/lib/search_engine.py:1053
#: modules/websearch/lib/search_engine.py:1199
#: modules/websearch/lib/websearch_templates.py:1151
#: modules/websearch/lib/websearch_webcoll.py:592
msgid "OR"
-msgstr ""
+msgstr "یا"
#: modules/websearch/doc/search-guide.webdoc:359
#: modules/websearch/doc/search-guide.webdoc:394
#: modules/websearch/doc/search-guide.webdoc:491
#: modules/websearch/doc/search-guide.webdoc:526
#: modules/websearch/doc/search-guide.webdoc:628
#: modules/websearch/doc/search-guide.webdoc:663
#: modules/websearch/doc/search-guide.webdoc:766
#: modules/websearch/doc/search-guide.webdoc:801
#: modules/websearch/lib/search_engine.py:1198
#: modules/websearch/lib/websearch_templates.py:1150
msgid "AND"
-msgstr ""
+msgstr "و"
#: modules/websearch/doc/search-guide.webdoc:349
#: modules/websearch/doc/search-guide.webdoc:384
#: modules/websearch/doc/search-guide.webdoc:419
#: modules/websearch/doc/search-guide.webdoc:481
#: modules/websearch/doc/search-guide.webdoc:516
#: modules/websearch/doc/search-guide.webdoc:551
#: modules/websearch/doc/search-guide.webdoc:618
#: modules/websearch/doc/search-guide.webdoc:653
#: modules/websearch/doc/search-guide.webdoc:688
#: modules/websearch/doc/search-guide.webdoc:756
#: modules/websearch/doc/search-guide.webdoc:791
#: modules/websearch/doc/search-guide.webdoc:826
#: modules/miscutil/lib/inveniocfg.py:483
msgid "keyword"
-msgstr ""
+msgstr "کلیدواژه"
#: modules/websearch/doc/search-tips.webdoc:36
#: modules/websearch/doc/search-tips.webdoc:67
#: modules/websearch/doc/search-tips.webdoc:105
#: modules/websearch/doc/search-tips.webdoc:153
#: modules/websearch/doc/search-tips.webdoc:176
#: modules/websearch/doc/search-tips.webdoc:200
#: modules/websearch/doc/search-tips.webdoc:245
#: modules/websearch/doc/search-tips.webdoc:285
#: modules/websearch/doc/search-tips.webdoc:296
#: modules/websearch/doc/search-tips.webdoc:316
#: modules/websearch/doc/search-tips.webdoc:336
#: modules/websearch/doc/search-tips.webdoc:370
#: modules/websearch/doc/search-tips.webdoc:404
#: modules/websearch/doc/search-tips.webdoc:425
#: modules/websearch/doc/search-tips.webdoc:445
#: modules/websearch/doc/search-tips.webdoc:479
#: modules/websearch/doc/search-tips.webdoc:497
#: modules/websearch/doc/search-tips.webdoc:516
#: modules/websearch/doc/search-tips.webdoc:549
#: modules/websearch/doc/search-tips.webdoc:574
#: modules/websearch/doc/search-tips.webdoc:598
#: modules/websearch/doc/search-tips.webdoc:619
#: modules/websearch/doc/search-guide.webdoc:225
#: modules/websearch/doc/search-guide.webdoc:249
#: modules/websearch/doc/search-guide.webdoc:275
#: modules/websearch/doc/search-guide.webdoc:300
#: modules/websearch/doc/search-guide.webdoc:353
#: modules/websearch/doc/search-guide.webdoc:388
#: modules/websearch/doc/search-guide.webdoc:423
#: modules/websearch/doc/search-guide.webdoc:485
#: modules/websearch/doc/search-guide.webdoc:520
#: modules/websearch/doc/search-guide.webdoc:555
#: modules/websearch/doc/search-guide.webdoc:622
#: modules/websearch/doc/search-guide.webdoc:657
#: modules/websearch/doc/search-guide.webdoc:692
#: modules/websearch/doc/search-guide.webdoc:760
#: modules/websearch/doc/search-guide.webdoc:795
#: modules/websearch/doc/search-guide.webdoc:830
#: modules/websearch/doc/search-guide.webdoc:880
#: modules/websearch/doc/search-guide.webdoc:911
#: modules/websearch/doc/search-guide.webdoc:951
#: modules/websearch/doc/search-guide.webdoc:985
#: modules/websearch/doc/search-guide.webdoc:1025
#: modules/websearch/doc/search-guide.webdoc:1047
#: modules/websearch/doc/search-guide.webdoc:1067
#: modules/websearch/doc/search-guide.webdoc:1083
#: modules/websearch/doc/search-guide.webdoc:1123
#: modules/websearch/doc/search-guide.webdoc:1146
#: modules/websearch/doc/search-guide.webdoc:1167
#: modules/websearch/doc/search-guide.webdoc:1182
#: modules/websearch/doc/search-guide.webdoc:1226
#: modules/websearch/doc/search-guide.webdoc:1251
#: modules/websearch/doc/search-guide.webdoc:1272
#: modules/websearch/doc/search-guide.webdoc:1288
#: modules/websearch/doc/search-guide.webdoc:1333
#: modules/websearch/doc/search-guide.webdoc:1356
#: modules/websearch/doc/search-guide.webdoc:1378
#: modules/websearch/doc/search-guide.webdoc:1394
#: modules/websearch/doc/search-guide.webdoc:1764
#: modules/websearch/doc/search-guide.webdoc:1778
#: modules/websearch/doc/search-guide.webdoc:1796
#: modules/websearch/doc/search-guide.webdoc:1815
#: modules/websearch/doc/search-guide.webdoc:1828
#: modules/websearch/doc/search-guide.webdoc:1846
#: modules/websearch/doc/search-guide.webdoc:1866
#: modules/websearch/doc/search-guide.webdoc:1881
#: modules/websearch/doc/search-guide.webdoc:1900
#: modules/websearch/doc/search-guide.webdoc:1923
#: modules/websearch/doc/search-guide.webdoc:1938
#: modules/websearch/doc/search-guide.webdoc:1957
#: modules/websearch/doc/search-guide.webdoc:1985
#: modules/websearch/doc/search-guide.webdoc:2023
#: modules/websearch/doc/search-guide.webdoc:2034
#: modules/websearch/doc/search-guide.webdoc:2048
#: modules/websearch/doc/search-guide.webdoc:2062
#: modules/websearch/doc/search-guide.webdoc:2075
#: modules/websearch/doc/search-guide.webdoc:2091
#: modules/websearch/doc/search-guide.webdoc:2102
#: modules/websearch/doc/search-guide.webdoc:2116
#: modules/websearch/doc/search-guide.webdoc:2130
#: modules/websearch/doc/search-guide.webdoc:2143
#: modules/websearch/doc/search-guide.webdoc:2159
#: modules/websearch/doc/search-guide.webdoc:2170
#: modules/websearch/doc/search-guide.webdoc:2184
#: modules/websearch/doc/search-guide.webdoc:2198
#: modules/websearch/doc/search-guide.webdoc:2211
#: modules/websearch/doc/search-guide.webdoc:2229
#: modules/websearch/doc/search-guide.webdoc:2240
#: modules/websearch/doc/search-guide.webdoc:2254
#: modules/websearch/doc/search-guide.webdoc:2268
#: modules/websearch/doc/search-guide.webdoc:2281
#: modules/websearch/doc/search-guide.webdoc:2310
#: modules/websearch/doc/search-guide.webdoc:2324
#: modules/websearch/doc/search-guide.webdoc:2341
#: modules/websearch/doc/search-guide.webdoc:2354
#: modules/websearch/doc/search-guide.webdoc:2371
#: modules/websearch/doc/search-guide.webdoc:2385
#: modules/websearch/doc/search-guide.webdoc:2403
#: modules/websearch/doc/search-guide.webdoc:2417
#: modules/websearch/doc/search-guide.webdoc:2448
#: modules/websearch/doc/search-guide.webdoc:2463
#: modules/websearch/doc/search-guide.webdoc:2477
#: modules/websearch/doc/search-guide.webdoc:2492
#: modules/websearch/doc/search-guide.webdoc:2520
#: modules/websearch/doc/search-guide.webdoc:2535
#: modules/websearch/doc/search-guide.webdoc:2549
#: modules/websearch/doc/search-guide.webdoc:2565
#: modules/websearch/doc/search-guide.webdoc:2597
#: modules/websearch/doc/search-guide.webdoc:2613
#: modules/websearch/doc/search-guide.webdoc:2627
#: modules/websearch/doc/search-guide.webdoc:2642
#: modules/websearch/doc/search-guide.webdoc:2673
#: modules/websearch/doc/search-guide.webdoc:2689
#: modules/websearch/doc/search-guide.webdoc:2703
#: modules/websearch/doc/search-guide.webdoc:2718
#: modules/websearch/doc/search-guide.webdoc:2760
#: modules/websearch/doc/search-guide.webdoc:2775
#: modules/websearch/doc/search-guide.webdoc:2789
#: modules/websearch/doc/search-guide.webdoc:2814
#: modules/websearch/doc/search-guide.webdoc:2829
#: modules/websearch/doc/search-guide.webdoc:2843
#: modules/websearch/doc/search-guide.webdoc:2872
#: modules/websearch/doc/search-guide.webdoc:2887
#: modules/websearch/doc/search-guide.webdoc:2901
#: modules/websearch/doc/search-guide.webdoc:2929
#: modules/websearch/doc/search-guide.webdoc:2944
#: modules/websearch/doc/search-guide.webdoc:2957
#: modules/websearch/doc/search-guide.webdoc:2992
#: modules/websearch/doc/search-guide.webdoc:3014
#: modules/websearch/doc/search-guide.webdoc:3038
#: modules/websearch/doc/search-guide.webdoc:3062
#: modules/websearch/doc/search-guide.webdoc:3086
#: modules/websearch/doc/search-guide.webdoc:3101
#: modules/websearch/doc/search-guide.webdoc:3117
#: modules/websearch/doc/search-guide.webdoc:3134
#: modules/websearch/doc/search-guide.webdoc:3154
#: modules/websearch/doc/search-guide.webdoc:3172
#: modules/websearch/doc/search-guide.webdoc:3190
#: modules/websearch/doc/search-guide.webdoc:3209
#: modules/websearch/doc/search-guide.webdoc:3230
#: modules/websearch/doc/search-guide.webdoc:3244
#: modules/websearch/doc/search-guide.webdoc:3264
#: modules/websearch/doc/search-guide.webdoc:3279
#: modules/websearch/doc/search-guide.webdoc:3298
#: modules/websearch/doc/search-guide.webdoc:3313
#: modules/websearch/doc/search-guide.webdoc:3333
#: modules/websearch/doc/search-guide.webdoc:3348
#: modules/websearch/doc/search-guide.webdoc:3410
#: modules/websearch/doc/search-guide.webdoc:3424
#: modules/websearch/doc/search-guide.webdoc:3441
#: modules/websearch/doc/search-guide.webdoc:3454
#: modules/websearch/doc/search-guide.webdoc:3472
#: modules/websearch/doc/search-guide.webdoc:3487
#: modules/websearch/doc/search-guide.webdoc:3505
#: modules/websearch/doc/search-guide.webdoc:3520
#: modules/websearch/doc/search-guide.webdoc:3545
#: modules/websearch/doc/search-guide.webdoc:3558
#: modules/websearch/doc/search-guide.webdoc:3571
#: modules/websearch/doc/search-guide.webdoc:3587
#: modules/websearch/doc/search-guide.webdoc:3603
#: modules/websearch/doc/search-guide.webdoc:3620
#: modules/websearch/doc/search-guide.webdoc:3653
#: modules/websearch/doc/search-guide.webdoc:3669
#: modules/websearch/doc/search-guide.webdoc:3686
#: modules/websearch/doc/search-guide.webdoc:3706
#: modules/websearch/doc/search-guide.webdoc:3720
#: modules/websearch/doc/search-guide.webdoc:3738
#: modules/websearch/doc/search-guide.webdoc:3759
#: modules/websearch/doc/search-guide.webdoc:3778
#: modules/websearch/doc/search-guide.webdoc:3796
#: modules/websearch/doc/search-guide.webdoc:3818
#: modules/websearch/doc/search-guide.webdoc:3837
#: modules/websearch/doc/search-guide.webdoc:3854
#: modules/websearch/doc/search-guide.webdoc:3975
#: modules/websearch/doc/search-guide.webdoc:4000
#: modules/websearch/doc/search-guide.webdoc:4023
#: modules/websearch/doc/search-guide.webdoc:4049
#: modules/websearch/doc/search-guide.webdoc:4073
#: modules/websearch/doc/search-guide.webdoc:4100
#: modules/websearch/doc/search-guide.webdoc:4125
#: modules/websearch/doc/search-guide.webdoc:4151
#: modules/websearch/doc/search-guide.webdoc:4180
#: modules/websearch/doc/search-guide.webdoc:4200
#: modules/websearch/doc/search-guide.webdoc:4224
#: modules/websearch/doc/search-guide.webdoc:4251
#: modules/websearch/doc/search-guide.webdoc:4291
#: modules/websearch/doc/search-guide.webdoc:4312
#: modules/websearch/doc/search-guide.webdoc:4336
#: modules/websearch/doc/search-guide.webdoc:4366
#: modules/websearch/doc/search-guide.webdoc:4410
#: modules/websearch/doc/search-guide.webdoc:4432
#: modules/websearch/doc/search-guide.webdoc:4457
#: modules/websearch/doc/search-guide.webdoc:4487
#: modules/websearch/doc/search-guide.webdoc:4532
#: modules/websearch/doc/search-guide.webdoc:4553
#: modules/websearch/doc/search-guide.webdoc:4578
#: modules/websearch/doc/search-guide.webdoc:4608
#: modules/websearch/doc/search-guide.webdoc:4900
#: modules/websearch/doc/search-guide.webdoc:4916
#: modules/websearch/doc/search-guide.webdoc:4936
#: modules/websearch/doc/search-guide.webdoc:4955
#: modules/websearch/doc/search-guide.webdoc:4976
#: modules/websearch/doc/search-guide.webdoc:4994
#: modules/websearch/doc/search-guide.webdoc:5015
#: modules/websearch/doc/search-guide.webdoc:5033
#: modules/websearch/doc/search-guide.webdoc:5066
#: modules/websearch/doc/search-guide.webdoc:5080
#: modules/websearch/doc/search-guide.webdoc:5095
#: modules/websearch/doc/search-guide.webdoc:5111
#: modules/websearch/doc/search-guide.webdoc:5130
#: modules/websearch/doc/search-guide.webdoc:5144
#: modules/websearch/doc/search-guide.webdoc:5160
#: modules/websearch/doc/search-guide.webdoc:5178
#: modules/websearch/doc/search-guide.webdoc:5197
#: modules/websearch/doc/search-guide.webdoc:5212
#: modules/websearch/doc/search-guide.webdoc:5227
#: modules/websearch/doc/search-guide.webdoc:5245
#: modules/websearch/doc/search-guide.webdoc:5265
#: modules/websearch/doc/search-guide.webdoc:5280
#: modules/websearch/doc/search-guide.webdoc:5295
#: modules/websearch/doc/search-guide.webdoc:5315
#: modules/webstyle/doc/hacking/webstyle-webdoc-syntax.webdoc:130
#: modules/miscutil/lib/inveniocfg.py:480
#: modules/bibcirculation/lib/bibcirculation_templates.py:2020
#: modules/bibcirculation/lib/bibcirculation_templates.py:7219
#: modules/bibcirculation/lib/bibcirculation_templates.py:7918
#: modules/bibcirculation/lib/bibcirculation_templates.py:9140
#: modules/bibcirculation/lib/bibcirculation_templates.py:9148
#: modules/bibcirculation/lib/bibcirculation_templates.py:9156
#: modules/bibcirculation/lib/bibcirculation_templates.py:9164
#: modules/bibcirculation/lib/bibcirculation_templates.py:16025
#: modules/bibcirculation/lib/bibcirculation_templates.py:17724
msgid "title"
-msgstr ""
+msgstr "عنوان"
#: modules/websearch/doc/search-tips.webdoc:72
#: modules/websearch/doc/search-tips.webdoc:110
#: modules/websearch/lib/websearch_templates.py:1264
msgid "Narrow by collection:"
-msgstr ""
+msgstr "خاص براساس مجموعه"
#: modules/bibformat/etc/format_templates/Default_HTML_actions.bft:5
msgid "Add to personal basket"
-msgstr ""
+msgstr "اضافه به سبد شخصی"
#: modules/websubmit/doc/admin/websubmit-admin-guide.webdoc:20
msgid "WebSubmit Admin Guide"
msgstr ""
#: modules/websearch/doc/search-tips.webdoc:290
#: modules/websubmit/lib/websubmit_templates.py:1119
#: modules/bibharvest/lib/oai_harvest_admin.py:450
#: modules/bibharvest/lib/oai_harvest_admin.py:458
#: modules/bibharvest/lib/oai_harvest_admin.py:472
#: modules/bibharvest/lib/oai_harvest_admin.py:487
msgid "or"
-msgstr ""
+msgstr "یا"
#: modules/bibedit/lib/bibedit_templates.py:295
msgid "Comparison of:"
msgstr ""
#: modules/bibedit/lib/bibedit_templates.py:296
msgid "Revision"
-msgstr ""
+msgstr "بازنگری"
#: modules/bibformat/lib/bibformat_templates.py:315
#: modules/bibformat/lib/bibformat_templates.py:429
#: modules/bibformat/lib/bibformat_templates.py:579
#: modules/bibformat/lib/bibformat_templates.py:595
#: modules/bibformat/lib/bibformat_templates.py:626
#: modules/bibformat/lib/bibformat_templates.py:937
#: modules/bibformat/lib/bibformat_templates.py:1069
#: modules/bibformat/lib/bibformat_templates.py:1385
#: modules/bibformat/lib/bibformat_templates.py:1491
#: modules/bibformat/lib/bibformat_templates.py:1550
#: modules/webcomment/lib/webcomment_templates.py:1475
#: modules/webjournal/lib/webjournal_templates.py:165
#: modules/webjournal/lib/webjournal_templates.py:537
#: modules/webjournal/lib/webjournal_templates.py:678
#: modules/bibknowledge/lib/bibknowledge_templates.py:327
#: modules/bibknowledge/lib/bibknowledge_templates.py:587
#: modules/bibknowledge/lib/bibknowledge_templates.py:656
msgid "Menu"
-msgstr ""
+msgstr "فهرست"
#: modules/bibformat/lib/bibformat_templates.py:317
#: modules/bibformat/lib/bibformat_templates.py:430
#: modules/bibformat/lib/bibformat_templates.py:582
#: modules/bibformat/lib/bibformat_templates.py:598
#: modules/bibformat/lib/bibformat_templates.py:629
#: modules/bibknowledge/lib/bibknowledge_templates.py:323
#: modules/bibknowledge/lib/bibknowledge_templates.py:586
#: modules/bibknowledge/lib/bibknowledge_templates.py:655
msgid "Close Editor"
-msgstr ""
+msgstr "بستن ویرایشگر"
#: modules/bibformat/lib/bibformat_templates.py:318
#: modules/bibformat/lib/bibformat_templates.py:431
#: modules/bibformat/lib/bibformat_templates.py:583
#: modules/bibformat/lib/bibformat_templates.py:599
#: modules/bibformat/lib/bibformat_templates.py:630
msgid "Modify Template Attributes"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:319
#: modules/bibformat/lib/bibformat_templates.py:432
#: modules/bibformat/lib/bibformat_templates.py:584
#: modules/bibformat/lib/bibformat_templates.py:600
#: modules/bibformat/lib/bibformat_templates.py:631
msgid "Template Editor"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:320
#: modules/bibformat/lib/bibformat_templates.py:433
#: modules/bibformat/lib/bibformat_templates.py:585
#: modules/bibformat/lib/bibformat_templates.py:601
#: modules/bibformat/lib/bibformat_templates.py:632
#: modules/bibformat/lib/bibformat_templates.py:1184
#: modules/bibformat/lib/bibformat_templates.py:1384
#: modules/bibformat/lib/bibformat_templates.py:1490
msgid "Check Dependencies"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:370
#: modules/bibformat/lib/bibformat_templates.py:935
#: modules/bibformat/lib/bibformat_templates.py:1060
#: modules/bibupload/lib/batchuploader_templates.py:464
#: modules/webalert/lib/webalert_templates.py:320
#: modules/websubmit/lib/websubmit_managedocfiles.py:385
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:194
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:255
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:345
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:398
#: modules/bibcirculation/lib/bibcirculation_utils.py:454
#: modules/bibcirculation/lib/bibcirculation_templates.py:1147
#: modules/bibcirculation/lib/bibcirculation_templates.py:1347
#: modules/bibcirculation/lib/bibcirculation_templates.py:1522
#: modules/bibcirculation/lib/bibcirculation_templates.py:1797
#: modules/bibcirculation/lib/bibcirculation_templates.py:2387
#: modules/bibcirculation/lib/bibcirculation_templates.py:2504
#: modules/bibcirculation/lib/bibcirculation_templates.py:2736
#: modules/bibcirculation/lib/bibcirculation_templates.py:3094
#: modules/bibcirculation/lib/bibcirculation_templates.py:3940
#: modules/bibcirculation/lib/bibcirculation_templates.py:4043
#: modules/bibcirculation/lib/bibcirculation_templates.py:4266
#: modules/bibcirculation/lib/bibcirculation_templates.py:4327
#: modules/bibcirculation/lib/bibcirculation_templates.py:4454
#: modules/bibcirculation/lib/bibcirculation_templates.py:5602
#: modules/bibcirculation/lib/bibcirculation_templates.py:6185
#: modules/bibcirculation/lib/bibcirculation_templates.py:6234
#: modules/bibcirculation/lib/bibcirculation_templates.py:6533
#: modules/bibcirculation/lib/bibcirculation_templates.py:6596
#: modules/bibcirculation/lib/bibcirculation_templates.py:6699
#: modules/bibcirculation/lib/bibcirculation_templates.py:6928
#: modules/bibcirculation/lib/bibcirculation_templates.py:7027
#: modules/bibcirculation/lib/bibcirculation_templates.py:7427
#: modules/bibcirculation/lib/bibcirculation_templates.py:8079
#: modules/bibcirculation/lib/bibcirculation_templates.py:8236
#: modules/bibcirculation/lib/bibcirculation_templates.py:9025
#: modules/bibcirculation/lib/bibcirculation_templates.py:9270
#: modules/bibcirculation/lib/bibcirculation_templates.py:9595
#: modules/bibcirculation/lib/bibcirculation_templates.py:9834
#: modules/bibcirculation/lib/bibcirculation_templates.py:9879
#: modules/bibcirculation/lib/bibcirculation_templates.py:10070
#: modules/bibcirculation/lib/bibcirculation_templates.py:10313
#: modules/bibcirculation/lib/bibcirculation_templates.py:10357
#: modules/bibcirculation/lib/bibcirculation_templates.py:10521
#: modules/bibcirculation/lib/bibcirculation_templates.py:10748
#: modules/bibcirculation/lib/bibcirculation_templates.py:11221
#: modules/bibcirculation/lib/bibcirculation_templates.py:11349
#: modules/bibcirculation/lib/bibcirculation_templates.py:11854
#: modules/bibcirculation/lib/bibcirculation_templates.py:12210
#: modules/bibcirculation/lib/bibcirculation_templates.py:12831
#: modules/bibcirculation/lib/bibcirculation_templates.py:12994
#: modules/bibcirculation/lib/bibcirculation_templates.py:13604
#: modules/bibcirculation/lib/bibcirculation_templates.py:13867
#: modules/bibcirculation/lib/bibcirculation_templates.py:14070
#: modules/bibcirculation/lib/bibcirculation_templates.py:14140
#: modules/bibcirculation/lib/bibcirculation_templates.py:14389
#: modules/bibcirculation/lib/bibcirculation_templates.py:14460
#: modules/bibcirculation/lib/bibcirculation_templates.py:14713
#: modules/bibcirculation/lib/bibcirculation_templates.py:15143
#: modules/bibcirculation/lib/bibcirculation_templates.py:15515
#: modules/bibcirculation/lib/bibcirculation_templates.py:15862
#: modules/bibcirculation/lib/bibcirculation_templates.py:17909
#: modules/websubmit/lib/functions/Create_Upload_Files_Interface.py:447
#: modules/bibknowledge/lib/bibknowledge_templates.py:79
msgid "Name"
-msgstr ""
+msgstr "نام"
#: modules/bibformat/lib/bibformat_templates.py:389
#: modules/bibformat/lib/bibformat_templates.py:936
#: modules/bibformat/lib/bibformat_templates.py:1061
#: modules/webbasket/lib/webbasket_templates.py:1273
#: modules/websession/lib/websession_templates.py:1504
#: modules/websession/lib/websession_templates.py:1578
#: modules/websession/lib/websession_templates.py:1641
#: modules/websubmit/lib/websubmit_managedocfiles.py:387
#: modules/bibcirculation/lib/bibcirculation_templates.py:357
#: modules/bibcirculation/lib/bibcirculation_templates.py:3187
#: modules/bibcirculation/lib/bibcirculation_templates.py:6050
#: modules/bibcirculation/lib/bibcirculation_templates.py:7476
#: modules/bibcirculation/lib/bibcirculation_templates.py:7654
#: modules/bibcirculation/lib/bibcirculation_templates.py:7813
#: modules/bibcirculation/lib/bibcirculation_templates.py:8111
#: modules/bibcirculation/lib/bibcirculation_templates.py:8321
#: modules/bibcirculation/lib/bibcirculation_templates.py:8481
#: modules/bibcirculation/lib/bibcirculation_templates.py:9330
#: modules/bibcirculation/lib/bibcirculation_templates.py:17958
#: modules/bibcirculation/lib/bibcirculation_templates.py:18044
#: modules/websubmit/lib/functions/Create_Upload_Files_Interface.py:451
#: modules/bibknowledge/lib/bibknowledge_templates.py:80
msgid "Description"
-msgstr ""
+msgstr "توصیف"
#: modules/bibformat/lib/bibformat_templates.py:390
msgid "Update Format Attributes"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:580
#: modules/bibformat/lib/bibformat_templates.py:596
#: modules/bibformat/lib/bibformat_templates.py:627
msgid "Show Documentation"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:581
#: modules/bibformat/lib/bibformat_templates.py:597
#: modules/bibformat/lib/bibformat_templates.py:628
#: modules/bibformat/lib/bibformat_templates.py:679
msgid "Hide Documentation"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:588
#: modules/websubmit/lib/websubmit_templates.py:868
msgid "Your modifications will not be saved."
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:938
#: modules/bibformat/lib/bibformat_templates.py:1062
#: modules/bibupload/lib/batchuploader_templates.py:253
#: modules/bibupload/lib/batchuploader_templates.py:295
#: modules/bibupload/lib/batchuploader_templates.py:464
#: modules/websubmit/lib/websubmit_templates.py:1491
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:536
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:633
#: modules/bibcirculation/lib/bibcirculation_templates.py:358
#: modules/bibcirculation/lib/bibcirculation_templates.py:771
#: modules/bibcirculation/lib/bibcirculation_templates.py:2849
#: modules/bibcirculation/lib/bibcirculation_templates.py:2957
#: modules/bibcirculation/lib/bibcirculation_templates.py:3160
#: modules/bibcirculation/lib/bibcirculation_templates.py:7451
#: modules/bibcirculation/lib/bibcirculation_templates.py:7686
#: modules/bibcirculation/lib/bibcirculation_templates.py:7815
#: modules/bibcirculation/lib/bibcirculation_templates.py:8105
#: modules/bibcirculation/lib/bibcirculation_templates.py:8352
#: modules/bibcirculation/lib/bibcirculation_templates.py:8483
#: modules/bibcirculation/lib/bibcirculation_templates.py:9331
#: modules/bibcirculation/lib/bibcirculation_templates.py:10597
#: modules/bibcirculation/lib/bibcirculation_templates.py:10817
#: modules/bibcirculation/lib/bibcirculation_templates.py:10914
#: modules/bibcirculation/lib/bibcirculation_templates.py:11516
#: modules/bibcirculation/lib/bibcirculation_templates.py:11637
#: modules/bibcirculation/lib/bibcirculation_templates.py:12228
#: modules/bibcirculation/lib/bibcirculation_templates.py:13012
#: modules/bibcirculation/lib/bibcirculation_templates.py:13678
#: modules/bibcirculation/lib/bibcirculation_templates.py:13909
#: modules/bibcirculation/lib/bibcirculation_templates.py:15589
#: modules/bibcirculation/lib/bibcirculation_templates.py:17933
#: modules/bibcirculation/lib/bibcirculation_templates.py:18037
msgid "Status"
-msgstr ""
+msgstr "وضعیت"
#: modules/bibformat/lib/bibformat_templates.py:939
#: modules/bibformat/lib/bibformat_templates.py:1063
msgid "Last Modification Date"
-msgstr ""
+msgstr "تاریخ آخرین اصلاح"
#: modules/bibformat/lib/bibformat_templates.py:940
#: modules/bibformat/lib/bibformat_templates.py:1064
#: modules/webalert/lib/webalert_templates.py:327
#: modules/webalert/lib/webalert_templates.py:464
#: modules/webmessage/lib/webmessage_templates.py:89
#: modules/websubmit/lib/websubmit_templates.py:1490
#: modules/bibknowledge/lib/bibknowledge_templates.py:81
msgid "Action"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:942
#: modules/bibformat/lib/bibformat_templates.py:1066
#: modules/bibformat/lib/bibformat_templates.py:1551
#: modules/bibformat/web/admin/bibformatadmin.py:104
#: modules/bibformat/web/admin/bibformatadmin.py:167
#: modules/bibformat/web/admin/bibformatadmin.py:240
#: modules/bibformat/web/admin/bibformatadmin.py:287
#: modules/bibformat/web/admin/bibformatadmin.py:384
#: modules/bibformat/web/admin/bibformatadmin.py:999
msgid "Manage Output Formats"
-msgstr ""
+msgstr "مدیریت قالب های خروجی"
#: modules/bibformat/lib/bibformat_templates.py:943
#: modules/bibformat/lib/bibformat_templates.py:1067
#: modules/bibformat/lib/bibformat_templates.py:1552
#: modules/bibformat/web/admin/bibformatadmin.py:465
#: modules/bibformat/web/admin/bibformatadmin.py:500
#: modules/bibformat/web/admin/bibformatadmin.py:573
#: modules/bibformat/web/admin/bibformatadmin.py:620
#: modules/bibformat/web/admin/bibformatadmin.py:694
#: modules/bibformat/web/admin/bibformatadmin.py:1020
msgid "Manage Format Templates"
-msgstr ""
+msgstr "مدیریت اگلوهای قالب بندی"
#: modules/bibformat/lib/bibformat_templates.py:944
#: modules/bibformat/lib/bibformat_templates.py:1068
#: modules/bibformat/lib/bibformat_templates.py:1553
#: modules/bibformat/web/admin/bibformatadmin.py:887
#: modules/bibformat/web/admin/bibformatadmin.py:911
#: modules/bibformat/web/admin/bibformatadmin.py:946
#: modules/bibformat/web/admin/bibformatadmin.py:1038
msgid "Format Elements Documentation"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:996
#: modules/bibformat/web/admin/bibformatadmin.py:405
#: modules/bibformat/web/admin/bibformatadmin.py:407
#: modules/bibformat/web/admin/bibformatadmin.py:714
#: modules/bibformat/web/admin/bibformatadmin.py:716
#: modules/webbasket/lib/webbasket_templates.py:2894
#: modules/webmessage/lib/webmessage_templates.py:115
#: modules/webjournal/lib/webjournaladminlib.py:116
#: modules/webjournal/lib/webjournaladminlib.py:119
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:175
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:324
#: modules/bibcirculation/lib/bibcirculation_templates.py:1220
#: modules/bibcirculation/lib/bibcirculation_templates.py:15932
#: modules/bibcirculation/lib/bibcirculation_templates.py:18078
#: modules/bibcheck/web/admin/bibcheckadmin.py:137
#: modules/bibknowledge/lib/bibknowledge_templates.py:102
#: modules/bibknowledge/lib/bibknowledge_templates.py:529
#: modules/bibknowledge/lib/bibknowledgeadmin.py:747
#: modules/bibknowledge/lib/bibknowledgeadmin.py:749
msgid "Delete"
-msgstr ""
+msgstr "حذف"
#: modules/bibformat/lib/bibformat_templates.py:1019
msgid "Add New Format Template"
-msgstr ""
+msgstr "فزودن الگوی قالب بندی جدید"
#: modules/bibformat/lib/bibformat_templates.py:1020
msgid "Check Format Templates Extensively"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:1059
msgid "Code"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:1142
msgid "Add New Output Format"
-msgstr ""
+msgstr "افزودن قالب خروجی جدید"
#: modules/bibformat/lib/bibformat_templates.py:1180
msgid "menu"
-msgstr ""
+msgstr "فهرست"
#: modules/bibformat/lib/bibformat_templates.py:1181
#: modules/bibformat/lib/bibformat_templates.py:1381
#: modules/bibformat/lib/bibformat_templates.py:1487
msgid "Close Output Format"
-msgstr ""
+msgstr "بستن قالب خروجی جدید"
#: modules/bibformat/lib/bibformat_templates.py:1182
#: modules/bibformat/lib/bibformat_templates.py:1382
#: modules/bibformat/lib/bibformat_templates.py:1488
msgid "Rules"
-msgstr ""
+msgstr "قوانین"
#: modules/bibformat/lib/bibformat_templates.py:1183
#: modules/bibformat/lib/bibformat_templates.py:1383
#: modules/bibformat/lib/bibformat_templates.py:1489
msgid "Modify Output Format Attributes"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:1282
#: modules/bibformat/lib/bibformatadminlib.py:565
msgid "Remove Rule"
-msgstr ""
+msgstr "حذف قوانین"
#: modules/bibformat/lib/bibformat_templates.py:1335
#: modules/bibformat/lib/bibformatadminlib.py:572
msgid "Add New Rule"
-msgstr ""
+msgstr "افزودن قانون جدید"
#: modules/bibformat/lib/bibformat_templates.py:1336
#: modules/bibformat/lib/bibformatadminlib.py:569
#: modules/bibcheck/web/admin/bibcheckadmin.py:239
msgid "Save Changes"
-msgstr ""
+msgstr "ذخیره تغییرات"
#: modules/bibformat/lib/bibformat_templates.py:1910
msgid "No problem found with format"
msgstr ""
#: modules/bibformat/lib/bibformat_templates.py:1912
msgid "An error has been found"
-msgstr ""
+msgstr "یک خطا پیدا شده است"
#: modules/bibformat/lib/bibformat_templates.py:1914
msgid "The following errors have been found"
msgstr ""
#: modules/bibformat/lib/bibformatadminlib.py:61
#: modules/bibformat/web/admin/bibformatadmin.py:72
msgid "BibFormat Admin"
msgstr ""
#: modules/bibformat/lib/bibformatadminlib.py:357
#: modules/bibformat/lib/bibformatadminlib.py:396
#: modules/bibformat/lib/bibformatadminlib.py:398
msgid "Test with record:"
msgstr ""
#: modules/bibformat/lib/bibformatadminlib.py:358
msgid "Enter a search query here."
msgstr ""
#: modules/bibformat/lib/elements/bfe_aid_authors.py:287
#: modules/bibformat/lib/elements/bfe_authors.py:127
msgid "Hide"
-msgstr ""
+msgstr "پنهان"
#: modules/bibformat/lib/elements/bfe_aid_authors.py:288
#: modules/bibformat/lib/elements/bfe_authors.py:128
#, python-format
msgid "Show all %i authors"
msgstr ""
#: modules/bibformat/lib/elements/bfe_fulltext.py:78
#: modules/bibformat/lib/elements/bfe_fulltext_mini.py:72
#: modules/bibformat/lib/elements/bfe_fulltext_mini.py:75
#: modules/bibformat/lib/elements/bfe_fulltext_mini.py:113
#: modules/bibformat/lib/elements/bfe_fulltext_mini.py:116
#: modules/bibformat/lib/elements/bfe_fulltext_mini.py:133
#: modules/bibformat/lib/elements/bfe_fulltext_mini.py:135
msgid "Download fulltext"
-msgstr ""
+msgstr "دانلود تمام متن"
#: modules/bibformat/lib/elements/bfe_fulltext.py:87
#: modules/bibformat/lib/elements/bfe_fulltext_mini.py:61
msgid "additional files"
-msgstr ""
+msgstr "فایل های اضافی"
#: modules/bibformat/lib/elements/bfe_fulltext.py:130
#: modules/bibformat/lib/elements/bfe_fulltext_mini.py:120
#, python-format
msgid "%(x_sitename)s link"
msgstr ""
#: modules/bibformat/lib/elements/bfe_fulltext.py:130
#: modules/bibformat/lib/elements/bfe_fulltext_mini.py:120
#, python-format
msgid "%(x_sitename)s links"
msgstr ""
#: modules/bibformat/lib/elements/bfe_fulltext.py:139
#: modules/bibformat/lib/elements/bfe_fulltext_mini.py:138
msgid "external link"
-msgstr ""
+msgstr "پیوند خارجی"
#: modules/bibformat/lib/elements/bfe_fulltext.py:139
#: modules/bibformat/lib/elements/bfe_fulltext_mini.py:138
msgid "external links"
-msgstr ""
+msgstr "پیوندهای خارجی"
#: modules/bibformat/lib/elements/bfe_fulltext.py:234
#: modules/bibformat/lib/elements/bfe_fulltext.py:238
#: modules/bibformat/lib/elements/bfe_fulltext.py:289
msgid "Fulltext"
-msgstr ""
+msgstr "متن کامل"
#: modules/bibformat/lib/elements/bfe_edit_files.py:50
msgid "Manage Files of This Record"
-msgstr ""
+msgstr "مدیریت فایل های این رکورد"
#: modules/bibformat/lib/elements/bfe_edit_record.py:46
msgid "Edit This Record"
-msgstr ""
+msgstr "ویرایش این رکورد"
#: modules/bibformat/web/admin/bibformatadmin.py:182
#: modules/bibformat/web/admin/bibformatadmin.py:252
#: modules/bibformat/web/admin/bibformatadmin.py:299
#: modules/bibformat/web/admin/bibformatadmin.py:1002
msgid "Restricted Output Format"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:208
#: modules/bibformat/web/admin/bibformatadmin.py:534
#: modules/bibknowledge/lib/bibknowledgeadmin.py:563
msgid "Ok"
-msgstr ""
+msgstr "خوب"
#: modules/bibformat/web/admin/bibformatadmin.py:210
#, python-format
msgid "Output Format %s Rules"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:265
#, python-format
msgid "Output Format %s Attributes"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:312
#, python-format
msgid "Output Format %s Dependencies"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:384
msgid "Delete Output Format"
-msgstr ""
+msgstr "حذف قالب خروجی"
#: modules/bibformat/web/admin/bibformatadmin.py:405
#: modules/bibformat/web/admin/bibformatadmin.py:714
#: modules/webbasket/lib/webbasket_templates.py:1451
#: modules/webbasket/lib/webbasket_templates.py:1519
#: modules/webbasket/lib/webbasket_templates.py:1626
#: modules/webbasket/lib/webbasket_templates.py:1681
#: modules/webbasket/lib/webbasket_templates.py:1771
#: modules/webbasket/lib/webbasket_templates.py:2839
#: modules/webbasket/lib/webbasket_templates.py:3642
#: modules/websession/lib/websession_templates.py:1781
#: modules/websession/lib/websession_templates.py:1889
#: modules/websession/lib/websession_templates.py:2091
#: modules/websession/lib/websession_templates.py:2174
#: modules/websubmit/lib/websubmit_managedocfiles.py:877
#: modules/websubmit/lib/websubmit_templates.py:2535
#: modules/websubmit/lib/websubmit_templates.py:2598
#: modules/websubmit/lib/websubmit_templates.py:2618
#: modules/websubmit/web/publiline.py:1228
#: modules/webjournal/lib/webjournaladminlib.py:117
#: modules/webjournal/lib/webjournaladminlib.py:230
#: modules/bibedit/lib/bibeditmulti_templates.py:564
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:261
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:403
#: modules/bibcirculation/lib/bibcirculation_templates.py:784
#: modules/bibcirculation/lib/bibcirculation_templates.py:1411
#: modules/bibcirculation/lib/bibcirculation_templates.py:1556
#: modules/bibcirculation/lib/bibcirculation_templates.py:4705
#: modules/bibknowledge/lib/bibknowledgeadmin.py:747
msgid "Cancel"
-msgstr ""
+msgstr "لغو"
#: modules/bibformat/web/admin/bibformatadmin.py:434
msgid "Cannot create output format"
-msgstr ""
+msgstr "عدم توانایی ایجاد قالب خروجی"
#: modules/bibformat/web/admin/bibformatadmin.py:513
#: modules/bibformat/web/admin/bibformatadmin.py:587
#: modules/bibformat/web/admin/bibformatadmin.py:1023
msgid "Restricted Format Template"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:539
#, python-format
msgid "Format Template %s"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:598
#, python-format
msgid "Format Template %s Attributes"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:632
#, python-format
msgid "Format Template %s Dependencies"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:694
msgid "Delete Format Template"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:920
#, python-format
msgid "Format Element %s Dependencies"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:953
#, python-format
msgid "Test Format Element %s"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:1016
#, python-format
msgid "Validation of Output Format %s"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:1034
#, python-format
msgid "Validation of Format Template %s"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:1042
msgid "Restricted Format Element"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:1050
#, python-format
msgid "Validation of Format Element %s"
msgstr ""
#: modules/bibformat/web/admin/bibformatadmin.py:1053
msgid "Format Validation"
msgstr ""
#: modules/bibharvest/lib/bibharvest_templates.py:53
#: modules/bibharvest/lib/bibharvest_templates.py:70
msgid "See Guide"
-msgstr ""
+msgstr "راهنما را ببینید"
#: modules/bibharvest/lib/bibharvest_templates.py:81
msgid "OAI sources currently present in the database"
msgstr ""
#: modules/bibharvest/lib/bibharvest_templates.py:82
msgid "No OAI sources currently present in the database"
msgstr ""
#: modules/bibharvest/lib/bibharvest_templates.py:92
msgid "Next oaiharvest task"
msgstr ""
#: modules/bibharvest/lib/bibharvest_templates.py:93
msgid "scheduled time:"
msgstr ""
#: modules/bibharvest/lib/bibharvest_templates.py:94
msgid "current status:"
-msgstr ""
+msgstr "وضعیت جاری:"
#: modules/bibharvest/lib/bibharvest_templates.py:95
msgid "No oaiharvest task currently scheduled."
msgstr ""
#: modules/bibharvest/lib/bibharvest_templates.py:202
msgid "successfully validated"
msgstr ""
#: modules/bibharvest/lib/bibharvest_templates.py:203
msgid "does not seem to be a OAI-compliant baseURL"
msgstr ""
#: modules/bibharvest/lib/bibharvest_templates.py:284
msgid "View next entries..."
-msgstr ""
+msgstr "مشاهده مدخل های بعدی ..."
#: modules/bibharvest/lib/bibharvest_templates.py:341
msgid "previous month"
-msgstr ""
+msgstr "ماه پیشین"
#: modules/bibharvest/lib/bibharvest_templates.py:348
msgid "next month"
-msgstr ""
+msgstr "ماه آینده"
#: modules/bibharvest/lib/bibharvest_templates.py:443
msgid "main Page"
-msgstr ""
+msgstr "صفحه اصلی"
#: modules/bibharvest/lib/bibharvest_templates.py:450
#: modules/bibharvest/lib/oai_harvest_admin.py:94
msgid "edit"
-msgstr ""
+msgstr "ویرایش"
#: modules/bibharvest/lib/bibharvest_templates.py:454
#: modules/websubmit/lib/websubmit_managedocfiles.py:1037
#: modules/bibharvest/lib/oai_harvest_admin.py:98
msgid "delete"
-msgstr ""
+msgstr "حذف"
#: modules/bibharvest/lib/bibharvest_templates.py:458
#: modules/bibharvest/lib/oai_harvest_admin.py:102
msgid "test"
msgstr ""
#: modules/bibharvest/lib/bibharvest_templates.py:462
#: modules/bibharvest/lib/oai_harvest_admin.py:106
msgid "history"
-msgstr ""
+msgstr "تاریخچه"
#: modules/bibharvest/lib/bibharvest_templates.py:466
#: modules/bibharvest/lib/oai_harvest_admin.py:110
msgid "harvest"
msgstr ""
#: modules/bibrank/lib/bibrank_citation_grapher.py:137
msgid "Citation history:"
-msgstr ""
+msgstr "تاریخچه استنادی"
#: modules/bibrank/lib/bibrank_downloads_grapher.py:85
msgid "Download history:"
-msgstr ""
+msgstr "تاریخچه دانلود"
#: modules/bibrank/lib/bibrank_downloads_grapher.py:107
msgid "Download user distribution:"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:138
msgid "Warning: Please, select a valid time"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:142
msgid "Warning: Please, select a valid file"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:146
msgid "Warning: The date format is not correct"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:150
msgid "Warning: Please, select a valid date"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:185
msgid "Select file to upload"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:186
msgid "File type"
-msgstr ""
+msgstr "نوع فایل"
#: modules/bibupload/lib/batchuploader_templates.py:187
#: modules/bibupload/lib/batchuploader_templates.py:395
msgid "Upload mode"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:188
#: modules/bibupload/lib/batchuploader_templates.py:396
msgid "Upload later? then select:"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:189
#: modules/bibupload/lib/batchuploader_templates.py:397
#: modules/webmessage/lib/webmessage_templates.py:88
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:535
msgid "Date"
-msgstr ""
+msgstr "تاریخ"
#: modules/bibupload/lib/batchuploader_templates.py:190
#: modules/bibupload/lib/batchuploader_templates.py:398
#: modules/bibupload/lib/batchuploader_templates.py:464
#: modules/webstyle/lib/webstyle_templates.py:670
msgid "Time"
-msgstr ""
+msgstr "زمان"
#: modules/bibupload/lib/batchuploader_templates.py:191
#: modules/bibupload/lib/batchuploader_templates.py:393
#: modules/bibupload/lib/batchuploader_templates.py:399
#: modules/websession/lib/websession_templates.py:161
#: modules/websession/lib/websession_templates.py:164
#: modules/websession/lib/websession_templates.py:1038
msgid "Example"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:192
#: modules/bibupload/lib/batchuploader_templates.py:400
#, python-format
msgid "All fields with %(x_fmt_open)s*%(x_fmt_close)s are mandatory"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:194
#: modules/bibupload/lib/batchuploader_templates.py:391
msgid "Upload priority"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:214
#, python-format
msgid ""
"Your file has been successfully queued. You can check your "
"%(x_url1_open)supload history%(x_url1_close)s or %(x_url2_open)ssubmit "
"another file%(x_url2_close)s"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:225
#, python-format
msgid ""
"The MARCXML submitted is not valid. Please, review the file and "
"%(x_url2_open)sresubmit it%(x_url2_close)s"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:237
msgid "No metadata files have been uploaded yet."
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:250
#: modules/bibupload/lib/batchuploader_templates.py:292
msgid "Submit time"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:251
#: modules/bibupload/lib/batchuploader_templates.py:293
msgid "File name"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:252
#: modules/bibupload/lib/batchuploader_templates.py:294
msgid "Execution time"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:279
msgid "No document files have been uploaded yet."
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:334
#: modules/bibupload/lib/batchuploader_webinterface.py:73
#: modules/bibupload/lib/batchuploader_webinterface.py:243
msgid "Metadata batch upload"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:337
#: modules/bibupload/lib/batchuploader_webinterface.py:96
#: modules/bibupload/lib/batchuploader_webinterface.py:151
msgid "Document batch upload"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:340
#: modules/bibupload/lib/batchuploader_webinterface.py:267
msgid "Upload history"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:343
msgid "Daemon monitor"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:392
msgid "Input directory"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:394
msgid "Filename matching"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:409
#, python-format
msgid "<b>%s documents</b> have been found."
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:411
msgid "The following files have been successfully queued:"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:416
msgid "The following errors have occurred:"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:423
msgid ""
"Some files could not be moved to DONE folder. Please remove them manually."
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:425
msgid "All uploaded files were moved to DONE folder."
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:435
#, python-format
msgid ""
"Using %(x_fmt_open)sweb interface upload%(x_fmt_close)s, actions are "
"executed a single time."
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:437
#, python-format
msgid ""
"Check the %(x_url_open)sBatch Uploader daemon help page%(x_url_close)s for "
"executing these actions periodically."
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:442
msgid "Metadata folders"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:464
#: modules/bibcirculation/lib/bibcirculation_templates.py:2301
#: modules/bibcirculation/lib/bibcirculation_templates.py:2414
#: modules/bibcirculation/lib/bibcirculation_templates.py:2658
#: modules/bibcirculation/lib/bibcirculation_templates.py:4390
#: modules/bibcirculation/lib/bibcirculation_templates.py:5551
#: modules/bibcirculation/lib/bibcirculation_templates.py:6453
#: modules/bibcirculation/lib/bibcirculation_templates.py:8976
#: modules/bibcirculation/lib/bibcirculation_templates.py:9109
#: modules/bibcirculation/lib/bibcirculation_templates.py:9878
#: modules/bibcirculation/lib/bibcirculation_templates.py:10356
#: modules/bibcirculation/lib/bibcirculation_templates.py:11101
#: modules/bibcirculation/lib/bibcirculation_templates.py:11517
#: modules/bibcirculation/lib/bibcirculation_templates.py:11638
#: modules/bibcirculation/lib/bibcirculation_templates.py:15342
msgid "ID"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:464
msgid "Progress"
-msgstr ""
+msgstr "پیشرفت"
#: modules/bibupload/lib/batchuploader_templates.py:466
msgid "Last BibSched tasks:"
msgstr ""
#: modules/bibupload/lib/batchuploader_templates.py:475
msgid "Next scheduled BibSched run:"
msgstr ""
#: modules/bibupload/lib/batchuploader_webinterface.py:154
msgid "Document batch upload result"
msgstr ""
#: modules/bibupload/lib/batchuploader_webinterface.py:238
msgid "Invalid MARCXML"
msgstr ""
#: modules/bibupload/lib/batchuploader_webinterface.py:241
msgid "Upload successful"
-msgstr ""
+msgstr "فراگذاری موفقیت آمیز"
#: modules/bibupload/lib/batchuploader_webinterface.py:291
msgid "Batch Uploader: Daemon monitor"
msgstr ""
#: modules/miscutil/lib/dateutils.py:82 modules/miscutil/lib/dateutils.py:109
#: modules/webbasket/lib/webbasket.py:208
#: modules/webbasket/lib/webbasket.py:870
#: modules/webbasket/lib/webbasket.py:965
#: modules/websession/lib/webuser.py:301
#: modules/webstyle/lib/webstyle_templates.py:579
msgid "N/A"
msgstr ""
#: modules/miscutil/lib/dateutils.py:172
msgid "Sun"
-msgstr ""
+msgstr "یکشنبه"
#: modules/miscutil/lib/dateutils.py:173
msgid "Mon"
-msgstr ""
+msgstr "دوشنبه"
#: modules/miscutil/lib/dateutils.py:174
msgid "Tue"
-msgstr ""
+msgstr "سه شنبه"
#: modules/miscutil/lib/dateutils.py:175
msgid "Wed"
-msgstr ""
+msgstr "چهارشنبه"
#: modules/miscutil/lib/dateutils.py:176
msgid "Thu"
-msgstr ""
+msgstr "پنج شنبه"
#: modules/miscutil/lib/dateutils.py:177
msgid "Fri"
-msgstr ""
+msgstr "جمعه"
#: modules/miscutil/lib/dateutils.py:178
msgid "Sat"
-msgstr ""
+msgstr "شنبه"
#: modules/miscutil/lib/dateutils.py:180
msgid "Sunday"
-msgstr ""
+msgstr "یکشنبه"
#: modules/miscutil/lib/dateutils.py:181
msgid "Monday"
-msgstr ""
+msgstr "دوشنبه"
#: modules/miscutil/lib/dateutils.py:182
msgid "Tuesday"
-msgstr ""
+msgstr "سه شنبه"
#: modules/miscutil/lib/dateutils.py:183
msgid "Wednesday"
-msgstr ""
+msgstr "چهارشنبه"
#: modules/miscutil/lib/dateutils.py:184
msgid "Thursday"
-msgstr ""
+msgstr "پنج شنبه"
#: modules/miscutil/lib/dateutils.py:185
msgid "Friday"
-msgstr ""
+msgstr "جمعه"
#: modules/miscutil/lib/dateutils.py:186
msgid "Saturday"
-msgstr ""
+msgstr "شنبه"
#: modules/miscutil/lib/dateutils.py:200 modules/miscutil/lib/dateutils.py:214
msgid "Month"
-msgstr ""
+msgstr "ماه"
#: modules/miscutil/lib/dateutils.py:201
msgid "Jan"
-msgstr ""
+msgstr "ژانویه"
#: modules/miscutil/lib/dateutils.py:202
msgid "Feb"
-msgstr ""
+msgstr "فوریه"
#: modules/miscutil/lib/dateutils.py:203
msgid "Mar"
-msgstr ""
+msgstr "مارچ"
#: modules/miscutil/lib/dateutils.py:204
msgid "Apr"
-msgstr ""
+msgstr "آپریل"
#: modules/miscutil/lib/dateutils.py:205 modules/miscutil/lib/dateutils.py:219
#: modules/websearch/lib/search_engine.py:981
#: modules/websearch/lib/websearch_templates.py:1190
msgid "May"
-msgstr ""
+msgstr "مِی"
#: modules/miscutil/lib/dateutils.py:206
msgid "Jun"
-msgstr ""
+msgstr "ژوئن"
#: modules/miscutil/lib/dateutils.py:207
msgid "Jul"
-msgstr ""
+msgstr "ژولای"
#: modules/miscutil/lib/dateutils.py:208
msgid "Aug"
-msgstr ""
+msgstr "آوگوست"
#: modules/miscutil/lib/dateutils.py:209
msgid "Sep"
-msgstr ""
+msgstr "سپتامبر"
#: modules/miscutil/lib/dateutils.py:210
msgid "Oct"
-msgstr ""
+msgstr "اکتبر"
#: modules/miscutil/lib/dateutils.py:211
msgid "Nov"
-msgstr ""
+msgstr "نوامبر"
#: modules/miscutil/lib/dateutils.py:212
msgid "Dec"
-msgstr ""
+msgstr "دسامبر"
#: modules/miscutil/lib/dateutils.py:215
#: modules/websearch/lib/search_engine.py:980
#: modules/websearch/lib/websearch_templates.py:1189
msgid "January"
-msgstr ""
+msgstr "ژانویه"
#: modules/miscutil/lib/dateutils.py:216
#: modules/websearch/lib/search_engine.py:980
#: modules/websearch/lib/websearch_templates.py:1189
msgid "February"
-msgstr ""
+msgstr "فوریه"
#: modules/miscutil/lib/dateutils.py:217
#: modules/websearch/lib/search_engine.py:980
#: modules/websearch/lib/websearch_templates.py:1189
msgid "March"
-msgstr ""
+msgstr "مارچ"
#: modules/miscutil/lib/dateutils.py:218
#: modules/websearch/lib/search_engine.py:980
#: modules/websearch/lib/websearch_templates.py:1189
msgid "April"
-msgstr ""
+msgstr "آپریل"
#: modules/miscutil/lib/dateutils.py:220
#: modules/websearch/lib/search_engine.py:981
#: modules/websearch/lib/websearch_templates.py:1190
msgid "June"
-msgstr ""
+msgstr "ژوئن"
#: modules/miscutil/lib/dateutils.py:221
#: modules/websearch/lib/search_engine.py:981
#: modules/websearch/lib/websearch_templates.py:1190
msgid "July"
-msgstr ""
+msgstr "زولای"
#: modules/miscutil/lib/dateutils.py:222
#: modules/websearch/lib/search_engine.py:981
#: modules/websearch/lib/websearch_templates.py:1190
msgid "August"
-msgstr ""
+msgstr "آوگوست"
#: modules/miscutil/lib/dateutils.py:223
#: modules/websearch/lib/search_engine.py:982
#: modules/websearch/lib/websearch_templates.py:1191
msgid "September"
-msgstr ""
+msgstr "سپتامبر"
#: modules/miscutil/lib/dateutils.py:224
#: modules/websearch/lib/search_engine.py:982
#: modules/websearch/lib/websearch_templates.py:1191
msgid "October"
-msgstr ""
+msgstr "اکتبر"
#: modules/miscutil/lib/dateutils.py:225
#: modules/websearch/lib/search_engine.py:982
#: modules/websearch/lib/websearch_templates.py:1191
msgid "November"
-msgstr ""
+msgstr "نوامبر"
#: modules/miscutil/lib/dateutils.py:226
#: modules/websearch/lib/search_engine.py:982
#: modules/websearch/lib/websearch_templates.py:1191
msgid "December"
-msgstr ""
+msgstr "دسامبر"
#: modules/miscutil/lib/dateutils.py:244
msgid "Day"
-msgstr ""
+msgstr "روز"
#: modules/miscutil/lib/dateutils.py:295
#: modules/bibcirculation/lib/bibcirculation_utils.py:432
#: modules/bibcirculation/lib/bibcirculation_templates.py:1757
#: modules/bibcirculation/lib/bibcirculation_templates.py:2743
#: modules/bibcirculation/lib/bibcirculation_templates.py:3096
#: modules/bibcirculation/lib/bibcirculation_templates.py:7154
#: modules/bibcirculation/lib/bibcirculation_templates.py:7431
#: modules/bibcirculation/lib/bibcirculation_templates.py:8083
#: modules/bibcirculation/lib/bibcirculation_templates.py:8240
#: modules/bibcirculation/lib/bibcirculation_templates.py:9597
#: modules/bibcirculation/lib/bibcirculation_templates.py:9836
#: modules/bibcirculation/lib/bibcirculation_templates.py:10072
#: modules/bibcirculation/lib/bibcirculation_templates.py:10315
#: modules/bibcirculation/lib/bibcirculation_templates.py:10525
#: modules/bibcirculation/lib/bibcirculation_templates.py:10750
#: modules/bibcirculation/lib/bibcirculation_templates.py:11209
#: modules/bibcirculation/lib/bibcirculation_templates.py:11353
#: modules/bibcirculation/lib/bibcirculation_templates.py:11858
#: modules/bibcirculation/lib/bibcirculation_templates.py:11954
#: modules/bibcirculation/lib/bibcirculation_templates.py:12071
#: modules/bibcirculation/lib/bibcirculation_templates.py:12155
#: modules/bibcirculation/lib/bibcirculation_templates.py:12835
#: modules/bibcirculation/lib/bibcirculation_templates.py:12938
#: modules/bibcirculation/lib/bibcirculation_templates.py:13608
#: modules/bibcirculation/lib/bibcirculation_templates.py:13869
#: modules/bibcirculation/lib/bibcirculation_templates.py:14924
#: modules/bibcirculation/lib/bibcirculation_templates.py:15146
#: modules/bibcirculation/lib/bibcirculation_templates.py:15422
#: modules/bibcirculation/lib/bibcirculation_templates.py:16843
#: modules/bibcirculation/lib/bibcirculation_templates.py:17031
#: modules/bibcirculation/lib/bibcirculation_templates.py:17316
#: modules/bibcirculation/lib/bibcirculation_templates.py:17514
#: modules/bibcirculation/lib/bibcirculation_templates.py:17913
msgid "Year"
-msgstr ""
+msgstr "سال"
#: modules/miscutil/lib/errorlib_webinterface.py:64
#: modules/miscutil/lib/errorlib_webinterface.py:69
#: modules/miscutil/lib/errorlib_webinterface.py:74
#: modules/miscutil/lib/errorlib_webinterface.py:79
msgid "Sorry"
msgstr ""
#: modules/miscutil/lib/errorlib_webinterface.py:65
#: modules/miscutil/lib/errorlib_webinterface.py:70
#: modules/miscutil/lib/errorlib_webinterface.py:75
#: modules/miscutil/lib/errorlib_webinterface.py:80
#, python-format
msgid "Cannot send error request, %s parameter missing."
msgstr ""
#: modules/miscutil/lib/errorlib_webinterface.py:98
msgid "The error report has been sent."
msgstr ""
#: modules/miscutil/lib/errorlib_webinterface.py:99
msgid "Many thanks for helping us to improve the service."
msgstr ""
#: modules/miscutil/lib/errorlib_webinterface.py:101
msgid "Use the back button of your browser to return to the previous page."
msgstr ""
#: modules/miscutil/lib/errorlib_webinterface.py:103
msgid "Thank you!"
-msgstr ""
+msgstr "متشکرم!"
#: modules/miscutil/lib/inveniocfg.py:491
msgid "journal"
-msgstr ""
+msgstr "مجله"
#: modules/miscutil/lib/inveniocfg.py:493
msgid "record ID"
msgstr ""
#: modules/miscutil/lib/inveniocfg.py:506
msgid "word similarity"
msgstr ""
#: modules/miscutil/lib/inveniocfg.py:507
msgid "journal impact factor"
-msgstr ""
+msgstr "ضریب تأثیر مجله"
#: modules/miscutil/lib/inveniocfg.py:508
msgid "times cited"
-msgstr ""
+msgstr "دفعات استناد شده"
#: modules/miscutil/lib/inveniocfg.py:509
msgid "time-decay cite count"
msgstr ""
#: modules/miscutil/lib/inveniocfg.py:510
msgid "all-time-best cite rank"
msgstr ""
#: modules/miscutil/lib/inveniocfg.py:511
msgid "time-decay cite rank"
msgstr ""
#: modules/miscutil/lib/mailutils.py:210 modules/miscutil/lib/mailutils.py:223
#: modules/webcomment/lib/webcomment_templates.py:2107
msgid "Hello:"
msgstr ""
#: modules/miscutil/lib/mailutils.py:241 modules/miscutil/lib/mailutils.py:261
msgid "Best regards"
msgstr ""
#: modules/miscutil/lib/mailutils.py:243 modules/miscutil/lib/mailutils.py:263
msgid "Need human intervention? Contact"
-msgstr ""
+msgstr "نیازمند به مداخله انسانی دارید؟ تماس بگیرید"
#: modules/webaccess/lib/access_control_config.py:300
#: modules/websession/lib/websession_templates.py:1090
#: modules/websession/lib/webuser.py:896 modules/websession/lib/webuser.py:905
#: modules/websession/lib/webuser.py:906
msgid "Run Record Editor"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:301
#: modules/websession/lib/websession_templates.py:1092
msgid "Run Multi-Record Editor"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:302
#: modules/websession/lib/websession_templates.py:1123
#: modules/websession/lib/webuser.py:897 modules/websession/lib/webuser.py:907
#: modules/websession/lib/webuser.py:908
msgid "Run Document File Manager"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:303
#: modules/websession/lib/websession_templates.py:1096
msgid "Run Record Merger"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:304
msgid "Run BibSword client"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:305
#: modules/websession/lib/websession_templates.py:1103
msgid "Configure BibKnowledge"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:306
#: modules/websession/lib/websession_templates.py:1102
msgid "Configure BibFormat"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:307
#: modules/websession/lib/websession_templates.py:1105
msgid "Configure OAI Harvest"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:308
#: modules/websession/lib/websession_templates.py:1107
msgid "Configure OAI Repository"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:309
#: modules/websession/lib/websession_templates.py:1109
msgid "Configure BibIndex"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:310
#: modules/websession/lib/websession_templates.py:1111
msgid "Configure BibRank"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:311
#: modules/websession/lib/websession_templates.py:1113
msgid "Configure WebAccess"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:312
#: modules/websession/lib/websession_templates.py:1115
msgid "Configure WebComment"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:313
#: modules/websession/lib/websession_templates.py:1119
msgid "Configure WebSearch"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:314
#: modules/websession/lib/websession_templates.py:1121
msgid "Configure WebSubmit"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:315
#: modules/websession/lib/websession_templates.py:1117
msgid "Configure WebJournal"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:316
#: modules/websession/lib/websession_templates.py:1094
msgid "Run BibCirculation"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:317
#: modules/websession/lib/websession_templates.py:1100
msgid "Run Batch Uploader"
msgstr ""
#: modules/webaccess/lib/access_control_config.py:318
msgid "Run Person/Author Manager"
msgstr ""
#: modules/webaccess/lib/webaccessadmin_lib.py:3703
#, python-format
msgid "Your account on '%s' has been activated"
msgstr ""
#: modules/webaccess/lib/webaccessadmin_lib.py:3704
#, python-format
msgid "Your account earlier created on '%s' has been activated:"
msgstr ""
#: modules/webaccess/lib/webaccessadmin_lib.py:3706
#: modules/webaccess/lib/webaccessadmin_lib.py:3719
#: modules/webaccess/lib/webaccessadmin_lib.py:3745
msgid "Username/Email:"
-msgstr ""
+msgstr "نام کاربری/ پست الکترونیکی:"
#: modules/webaccess/lib/webaccessadmin_lib.py:3707
#: modules/webaccess/lib/webaccessadmin_lib.py:3720
msgid "Password:"
-msgstr ""
+msgstr "کلمه عبور:"
#: modules/webaccess/lib/webaccessadmin_lib.py:3717
#, python-format
msgid "Account created on '%s'"
msgstr ""
#: modules/webaccess/lib/webaccessadmin_lib.py:3718
#, python-format
msgid "An account has been created for you on '%s':"
msgstr ""
#: modules/webaccess/lib/webaccessadmin_lib.py:3730
#, python-format
msgid "Account rejected on '%s'"
msgstr ""
#: modules/webaccess/lib/webaccessadmin_lib.py:3731
#, python-format
msgid "Your request for an account has been rejected on '%s':"
msgstr ""
#: modules/webaccess/lib/webaccessadmin_lib.py:3733
#, python-format
msgid "Username/Email: %s"
msgstr ""
#: modules/webaccess/lib/webaccessadmin_lib.py:3743
#, python-format
msgid "Account deleted on '%s'"
msgstr ""
#: modules/webaccess/lib/webaccessadmin_lib.py:3744
#, python-format
msgid "Your account on '%s' has been deleted:"
msgstr ""
#: modules/webalert/lib/htmlparser.py:186
#: modules/webbasket/lib/webbasket_templates.py:2373
#: modules/webbasket/lib/webbasket_templates.py:3249
#: modules/websearch/lib/websearch_templates.py:1610
#: modules/websearch/lib/websearch_templates.py:3330
#: modules/websearch/lib/websearch_templates.py:3336
#: modules/websearch/lib/websearch_templates.py:3341
msgid "Detailed record"
msgstr ""
#: modules/webalert/lib/htmlparser.py:187
#: modules/websearch/lib/websearch_templates.py:1613
#: modules/websearch/lib/websearch_templates.py:3348
#: modules/webstyle/lib/webstyle_templates.py:845
msgid "Similar records"
-msgstr ""
+msgstr "رکوردهای مشابه"
#: modules/webalert/lib/htmlparser.py:188
msgid "Cited by"
-msgstr ""
+msgstr "استناد داده شده به وسیله"
#: modules/webalert/lib/webalert.py:54
#, python-format
msgid "You already have an alert named %s."
msgstr ""
#: modules/webalert/lib/webalert.py:111
msgid "unknown"
-msgstr ""
+msgstr "نامشخص"
#: modules/webalert/lib/webalert.py:163 modules/webalert/lib/webalert.py:217
#: modules/webalert/lib/webalert.py:303 modules/webalert/lib/webalert.py:341
msgid "You do not have rights for this operation."
msgstr ""
#: modules/webalert/lib/webalert.py:198
msgid "You already have an alert defined for the specified query and basket."
msgstr ""
#: modules/webalert/lib/webalert.py:221 modules/webalert/lib/webalert.py:345
msgid "The alert name cannot be empty."
-msgstr ""
+msgstr "نام هشدار نمی تواند خالی باشد."
#: modules/webalert/lib/webalert.py:226
msgid "You are not the owner of this basket."
-msgstr ""
+msgstr "شما مالک این سبد نیستید."
#: modules/webalert/lib/webalert.py:237
#, python-format
msgid "The alert %s has been added to your profile."
msgstr ""
#: modules/webalert/lib/webalert.py:376
#, python-format
msgid "The alert %s has been successfully updated."
msgstr ""
#: modules/webalert/lib/webalert.py:428
#, python-format
msgid ""
"You have made %(x_nb)s queries. A %(x_url_open)sdetailed list%(x_url_close)s "
"is available with a possibility to (a) view search results and (b) subscribe "
"to an automatic email alerting service for these queries."
msgstr ""
#: modules/webalert/lib/webalert_templates.py:75
msgid "Pattern"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:77
#: modules/bibedit/lib/bibeditmulti_templates.py:556
msgid "Field"
-msgstr ""
+msgstr "فیلد"
#: modules/webalert/lib/webalert_templates.py:79
msgid "Pattern 1"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:81
msgid "Field 1"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:83
msgid "Pattern 2"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:85
msgid "Field 2"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:87
msgid "Pattern 3"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:89
msgid "Field 3"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:91
msgid "Collections"
-msgstr ""
+msgstr "مجموعه ها"
#: modules/webalert/lib/webalert_templates.py:93
#: modules/bibcirculation/lib/bibcirculation_templates.py:356
#: modules/bibcirculation/lib/bibcirculation_templates.py:3179
#: modules/bibcirculation/lib/bibcirculation_templates.py:6049
#: modules/bibcirculation/lib/bibcirculation_templates.py:7469
#: modules/bibcirculation/lib/bibcirculation_templates.py:7620
#: modules/bibcirculation/lib/bibcirculation_templates.py:7812
#: modules/bibcirculation/lib/bibcirculation_templates.py:8110
#: modules/bibcirculation/lib/bibcirculation_templates.py:8298
#: modules/bibcirculation/lib/bibcirculation_templates.py:8480
#: modules/bibcirculation/lib/bibcirculation_templates.py:9329
#: modules/bibcirculation/lib/bibcirculation_templates.py:17951
#: modules/bibcirculation/lib/bibcirculation_templates.py:18043
msgid "Collection"
-msgstr ""
+msgstr "مجموعه"
#: modules/webalert/lib/webalert_templates.py:114
msgid "You own the following alerts:"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:115
msgid "alert name"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:124
msgid "SHOW"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:173
msgid ""
"This alert will notify you each time/only if a new item satisfies the "
"following query:"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:174
msgid "QUERY"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:212
msgid "Alert identification name:"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:214
msgid "Search-checking frequency:"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:218
#: modules/webalert/lib/webalert_templates.py:338
#: modules/bibharvest/lib/oai_harvest_admin.py:142
msgid "monthly"
-msgstr ""
+msgstr "ماهانه"
#: modules/webalert/lib/webalert_templates.py:219
#: modules/webalert/lib/webalert_templates.py:336
#: modules/bibharvest/lib/oai_harvest_admin.py:141
msgid "weekly"
-msgstr ""
+msgstr "هفتگی"
#: modules/webalert/lib/webalert_templates.py:220
#: modules/webalert/lib/webalert_templates.py:333
#: modules/bibharvest/lib/oai_harvest_admin.py:140
msgid "daily"
-msgstr ""
+msgstr "روزانه"
#: modules/webalert/lib/webalert_templates.py:221
msgid "Send notification email?"
-msgstr ""
+msgstr "ارسال ایمیل اطلاع دهی؟"
#: modules/webalert/lib/webalert_templates.py:224
#: modules/webalert/lib/webalert_templates.py:341
msgid "yes"
-msgstr ""
+msgstr "بله"
#: modules/webalert/lib/webalert_templates.py:225
#: modules/webalert/lib/webalert_templates.py:343
msgid "no"
-msgstr ""
+msgstr "نه"
#: modules/webalert/lib/webalert_templates.py:226
#, python-format
msgid "if %(x_fmt_open)sno%(x_fmt_close)s you must specify a basket"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:228
msgid "Store results in basket?"
-msgstr ""
+msgstr "ذخیره نتایج در سبد؟"
#: modules/webalert/lib/webalert_templates.py:249
msgid "SET ALERT"
-msgstr ""
+msgstr "تنظیم هشدار"
#: modules/webalert/lib/webalert_templates.py:250
msgid "CLEAR DATA"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:301
#, python-format
msgid ""
"Set a new alert from %(x_url1_open)syour searches%(x_url1_close)s, the "
"%(x_url2_open)spopular searches%(x_url2_close)s, or the input form."
msgstr ""
#: modules/webalert/lib/webalert_templates.py:319
#: modules/webcomment/lib/webcomment_templates.py:233
#: modules/webcomment/lib/webcomment_templates.py:664
#: modules/webcomment/lib/webcomment_templates.py:1965
#: modules/webcomment/lib/webcomment_templates.py:1989
#: modules/webcomment/lib/webcomment_templates.py:2015
#: modules/webmessage/lib/webmessage_templates.py:509
#: modules/websession/lib/websession_templates.py:2215
#: modules/websession/lib/websession_templates.py:2255
msgid "No"
-msgstr ""
+msgstr "نه"
#: modules/webalert/lib/webalert_templates.py:321
msgid "Search checking frequency"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:322
msgid "Notification by email"
-msgstr ""
+msgstr "اطلاع دهی به وسیله پست الکترونیکی"
#: modules/webalert/lib/webalert_templates.py:323
msgid "Result in basket"
-msgstr ""
+msgstr "نتیجه در سبد"
#: modules/webalert/lib/webalert_templates.py:324
msgid "Date last run"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:325
msgid "Creation date"
-msgstr ""
+msgstr "تاریخ ایجاد"
#: modules/webalert/lib/webalert_templates.py:326
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:346
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:399
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:459
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:632
msgid "Query"
-msgstr ""
+msgstr "عبارت جستجو"
#: modules/webalert/lib/webalert_templates.py:369
#: modules/webbasket/lib/webbasket_templates.py:1786
msgid "no basket"
-msgstr ""
+msgstr "بدون سبد"
#: modules/webalert/lib/webalert_templates.py:386
msgid "Modify"
-msgstr ""
+msgstr "اصلاح کردن"
#: modules/webalert/lib/webalert_templates.py:392
#: modules/webjournal/lib/webjournaladminlib.py:231
#: modules/webjournal/lib/webjournaladminlib.py:237
msgid "Remove"
-msgstr ""
+msgstr "حذف کردن"
#: modules/webalert/lib/webalert_templates.py:394
#: modules/webalert/lib/webalert_templates.py:484
msgid "Execute search"
-msgstr ""
+msgstr "اجرای جستجو"
#: modules/webalert/lib/webalert_templates.py:400
#, python-format
msgid "You have defined %s alerts."
msgstr ""
#: modules/webalert/lib/webalert_templates.py:438
#, python-format
msgid ""
"You have not executed any search yet. Please go to the %(x_url_open)ssearch "
"interface%(x_url_close)s first."
msgstr ""
#: modules/webalert/lib/webalert_templates.py:447
#, python-format
msgid ""
"You have performed %(x_nb1)s searches (%(x_nb2)s different questions) during "
"the last 30 days or so."
msgstr ""
#: modules/webalert/lib/webalert_templates.py:452
#, python-format
msgid "Here are the %s most popular searches."
msgstr ""
#: modules/webalert/lib/webalert_templates.py:463
msgid "Question"
-msgstr ""
+msgstr "سؤال"
#: modules/webalert/lib/webalert_templates.py:467
msgid "Last Run"
msgstr ""
#: modules/webalert/lib/webalert_templates.py:485
msgid "Set new alert"
-msgstr ""
+msgstr "تنظیم هشدار جدید"
#: modules/webalert/lib/webalert_webinterface.py:76
#: modules/webalert/lib/webalert_webinterface.py:139
#: modules/webalert/lib/webalert_webinterface.py:224
#: modules/webalert/lib/webalert_webinterface.py:302
#: modules/webalert/lib/webalert_webinterface.py:358
#: modules/webalert/lib/webalert_webinterface.py:435
#: modules/webalert/lib/webalert_webinterface.py:509
msgid "You are not authorized to use alerts."
msgstr ""
#: modules/webalert/lib/webalert_webinterface.py:79
msgid "Popular Searches"
-msgstr ""
+msgstr "جستجوی های رایج"
#: modules/webalert/lib/webalert_webinterface.py:81
#: modules/websession/lib/websession_templates.py:457
#: modules/websession/lib/websession_templates.py:619
msgid "Your Searches"
-msgstr ""
+msgstr "جستجوهای شما"
#: modules/webalert/lib/webalert_webinterface.py:98
#: modules/webalert/lib/webalert_webinterface.py:150
#: modules/webalert/lib/webalert_webinterface.py:183
#: modules/webalert/lib/webalert_webinterface.py:235
#: modules/webalert/lib/webalert_webinterface.py:268
#: modules/webalert/lib/webalert_webinterface.py:319
#: modules/webalert/lib/webalert_webinterface.py:369
#: modules/webalert/lib/webalert_webinterface.py:395
#: modules/webalert/lib/webalert_webinterface.py:446
#: modules/webalert/lib/webalert_webinterface.py:472
#: modules/webalert/lib/webalert_webinterface.py:520
#: modules/webalert/lib/webalert_webinterface.py:548
#: modules/webbasket/lib/webbasket.py:2349
#: modules/webbasket/lib/webbasket_webinterface.py:800
#: modules/webbasket/lib/webbasket_webinterface.py:894
#: modules/webbasket/lib/webbasket_webinterface.py:1015
#: modules/webbasket/lib/webbasket_webinterface.py:1110
#: modules/webbasket/lib/webbasket_webinterface.py:1240
#: modules/webmessage/lib/webmessage_templates.py:466
#: modules/websession/lib/websession_templates.py:605
#: modules/websession/lib/websession_templates.py:2328
#: modules/websession/lib/websession_webinterface.py:216
#: modules/websession/lib/websession_webinterface.py:238
#: modules/websession/lib/websession_webinterface.py:280
#: modules/websession/lib/websession_webinterface.py:508
#: modules/websession/lib/websession_webinterface.py:531
#: modules/websession/lib/websession_webinterface.py:559
#: modules/websession/lib/websession_webinterface.py:575
#: modules/websession/lib/websession_webinterface.py:627
#: modules/websession/lib/websession_webinterface.py:650
#: modules/websession/lib/websession_webinterface.py:676
#: modules/websession/lib/websession_webinterface.py:749
#: modules/websession/lib/websession_webinterface.py:805
#: modules/websession/lib/websession_webinterface.py:840
#: modules/websession/lib/websession_webinterface.py:871
#: modules/websession/lib/websession_webinterface.py:943
#: modules/websession/lib/websession_webinterface.py:981
#: modules/websubmit/web/publiline.py:136
#: modules/websubmit/web/publiline.py:157
#: modules/websubmit/web/yourapprovals.py:91
#: modules/websubmit/web/yoursubmissions.py:163
msgid "Your Account"
-msgstr ""
+msgstr "حساب شما"
#: modules/webalert/lib/webalert_webinterface.py:100
#, python-format
msgid "%s Personalize, Display searches"
msgstr ""
#: modules/webalert/lib/webalert_webinterface.py:101
#: modules/webalert/lib/webalert_webinterface.py:153
#: modules/webalert/lib/webalert_webinterface.py:186
#: modules/webalert/lib/webalert_webinterface.py:238
#: modules/webalert/lib/webalert_webinterface.py:271
#: modules/webalert/lib/webalert_webinterface.py:322
#: modules/webalert/lib/webalert_webinterface.py:372
#: modules/webalert/lib/webalert_webinterface.py:398
#: modules/webalert/lib/webalert_webinterface.py:449
#: modules/webalert/lib/webalert_webinterface.py:475
#: modules/webalert/lib/webalert_webinterface.py:523
#: modules/webalert/lib/webalert_webinterface.py:551
#: modules/websession/lib/websession_webinterface.py:219
#: modules/websession/lib/websession_webinterface.py:241
#: modules/websession/lib/websession_webinterface.py:282
#: modules/websession/lib/websession_webinterface.py:510
#: modules/websession/lib/websession_webinterface.py:533
#: modules/websession/lib/websession_webinterface.py:562
#: modules/websession/lib/websession_webinterface.py:578
#: modules/websession/lib/websession_webinterface.py:596
#: modules/websession/lib/websession_webinterface.py:606
#: modules/websession/lib/websession_webinterface.py:629
#: modules/websession/lib/websession_webinterface.py:652
#: modules/websession/lib/websession_webinterface.py:678
#, python-format
msgid "%s, personalize"
msgstr ""
#: modules/webalert/lib/webalert_webinterface.py:145
#: modules/webalert/lib/webalert_webinterface.py:230
#: modules/webalert/lib/webalert_webinterface.py:364
#: modules/webalert/lib/webalert_webinterface.py:441
#: modules/webalert/lib/webalert_webinterface.py:515
#: modules/webstyle/lib/webstyle_templates.py:583
#: modules/webstyle/lib/webstyle_templates.py:620
#: modules/webstyle/lib/webstyle_templates.py:622
#: modules/websubmit/lib/websubmit_engine.py:1734
#: modules/websubmit/lib/websubmit_webinterface.py:1361
#: modules/bibcatalog/lib/bibcatalog_templates.py:37
#: modules/bibedit/lib/bibedit_webinterface.py:193
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:496
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:559
#: modules/bibknowledge/lib/bibknowledgeadmin.py:279
msgid "Error"
-msgstr ""
+msgstr "خطا"
#: modules/webalert/lib/webalert_webinterface.py:152
#: modules/webalert/lib/webalert_webinterface.py:185
#: modules/webalert/lib/webalert_webinterface.py:237
#: modules/webalert/lib/webalert_webinterface.py:371
#: modules/webalert/lib/webalert_webinterface.py:448
#: modules/webalert/lib/webalert_webinterface.py:522
#, python-format
msgid "%s Personalize, Set a new alert"
msgstr ""
#: modules/webalert/lib/webalert_webinterface.py:178
msgid "Set a new alert"
-msgstr ""
+msgstr "تنظیم یک هشدار جدید"
#: modules/webalert/lib/webalert_webinterface.py:263
msgid "Modify alert settings"
-msgstr ""
+msgstr "اصلاح تنظیمات هشدار"
#: modules/webalert/lib/webalert_webinterface.py:270
#, python-format
msgid "%s Personalize, Modify alert settings"
msgstr ""
#: modules/webalert/lib/webalert_webinterface.py:314
#: modules/websession/lib/websession_templates.py:474
msgid "Your Alerts"
-msgstr ""
+msgstr "هشدارهای شما"
#: modules/webalert/lib/webalert_webinterface.py:321
#: modules/webalert/lib/webalert_webinterface.py:397
#: modules/webalert/lib/webalert_webinterface.py:474
#: modules/webalert/lib/webalert_webinterface.py:550
#, python-format
msgid "%s Personalize, Display alerts"
msgstr ""
#: modules/webalert/lib/webalert_webinterface.py:390
#: modules/webalert/lib/webalert_webinterface.py:467
#: modules/webalert/lib/webalert_webinterface.py:543
msgid "Display alerts"
-msgstr ""
+msgstr "هشدارهای نمایش"
#: modules/webbasket/lib/webbasket.py:104
#: modules/webbasket/lib/webbasket.py:2151
#: modules/webbasket/lib/webbasket.py:2181
msgid ""
"The selected public basket does not exist or you do not have access to it."
-msgstr ""
+msgstr "سبد عمومی انتخاب شده وجود ندارد یا اینکه شما دسترسی به آن را ندارید."
#: modules/webbasket/lib/webbasket.py:112
msgid "Please select a valid public basket from the list of public baskets."
-msgstr ""
+msgstr "لطفا یک سبد عمومی درست از سیاهه سبدهای عمومی انتخاب کنید. "
#: modules/webbasket/lib/webbasket.py:135
#: modules/webbasket/lib/webbasket.py:298
#: modules/webbasket/lib/webbasket.py:1000
msgid "The selected item does not exist or you do not have access to it."
-msgstr ""
+msgstr "آیتم انتخاب شده وجود ندارد یا اینکه شما دسترسی به آن را ندارید."
#: modules/webbasket/lib/webbasket.py:141
msgid "Returning to the public basket view."
-msgstr ""
+msgstr "بازگشت به نمای سبد عمومی."
#: modules/webbasket/lib/webbasket.py:416
#: modules/webbasket/lib/webbasket.py:474
#: modules/webbasket/lib/webbasket.py:1419
#: modules/webbasket/lib/webbasket.py:1483
msgid "You do not have permission to write notes to this item."
-msgstr ""
+msgstr "شما اجازه نوشتن یادداشت برای این آیتم را ندارید."
#: modules/webbasket/lib/webbasket.py:429
#: modules/webbasket/lib/webbasket.py:1431
msgid ""
"The note you are quoting does not exist or you do not have access to it."
msgstr ""
#: modules/webbasket/lib/webbasket.py:484
#: modules/webbasket/lib/webbasket.py:1495
msgid "You must fill in both the subject and the body of the note."
msgstr ""
#: modules/webbasket/lib/webbasket.py:581
#: modules/webbasket/lib/webbasket.py:657
#: modules/webbasket/lib/webbasket.py:713
#: modules/webbasket/lib/webbasket.py:2680
msgid "The selected topic does not exist or you do not have access to it."
msgstr ""
#: modules/webbasket/lib/webbasket.py:623
#: modules/webbasket/lib/webbasket.py:743
#: modules/webbasket/lib/webbasket.py:2707
#: modules/webbasket/lib/webbasket.py:2715
#: modules/webbasket/lib/webbasket.py:2722
msgid "The selected basket does not exist or you do not have access to it."
msgstr ""
#: modules/webbasket/lib/webbasket.py:734
msgid "The selected basket is no longer public."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1548
msgid "You do not have permission to delete this note."
-msgstr ""
+msgstr "شما مجوز حذف این یدداشت را ندارید."
#: modules/webbasket/lib/webbasket.py:1559
msgid ""
"The note you are deleting does not exist or you do not have access to it."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1656
#, python-format
msgid "Sorry, you do not have sufficient rights to add record #%i."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1662
msgid "Some of the items were not added due to lack of sufficient rights."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1679
msgid "Please provide a title for the external source."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1685
msgid "Please provide a description for the external source."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1691
msgid "Please provide a url for the external source."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1700
msgid "The url you have provided is not valid."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1707
msgid ""
"The url you have provided is not valid: The request contains bad syntax or "
"cannot be fulfilled."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1714
msgid ""
"The url you have provided is not valid: The server failed to fulfil an "
"apparently valid request."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1763
#: modules/webbasket/lib/webbasket.py:1884
#: modules/webbasket/lib/webbasket.py:1953
msgid "Sorry, you do not have sufficient rights on this basket."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1772
msgid "No records to add."
-msgstr ""
+msgstr "بدون رکوردی برای افزودن"
#: modules/webbasket/lib/webbasket.py:1812
#: modules/webbasket/lib/webbasket.py:2652
msgid "Cannot add items to the selected basket. Invalid parameters."
msgstr ""
#: modules/webbasket/lib/webbasket.py:1824
msgid ""
"A default topic and basket have been automatically created. Edit them to "
"rename them as you see fit."
msgstr ""
#: modules/webbasket/lib/webbasket.py:2101
msgid "Please provide a name for the new basket."
-msgstr ""
+msgstr "لطفا یک نام برای سبد جدید قرار دهید."
#: modules/webbasket/lib/webbasket.py:2107
msgid "Please select an existing topic or create a new one."
msgstr ""
#: modules/webbasket/lib/webbasket.py:2143
msgid ""
"You cannot subscribe to this basket, you are the either owner or you have "
"already subscribed."
msgstr ""
#: modules/webbasket/lib/webbasket.py:2173
msgid ""
"You cannot unsubscribe from this basket, you are the either owner or you "
"have already unsubscribed."
msgstr ""
#: modules/webbasket/lib/webbasket.py:2266
#: modules/webbasket/lib/webbasket.py:2379
#: modules/webbasket/lib/webbasket_templates.py:101
#: modules/webbasket/lib/webbasket_templates.py:151
#: modules/webbasket/lib/webbasket_templates.py:157
#: modules/webbasket/lib/webbasket_templates.py:605
#: modules/webbasket/lib/webbasket_templates.py:657
msgid "Personal baskets"
-msgstr ""
+msgstr "سبدهای شخصی"
#: modules/webbasket/lib/webbasket.py:2290
#: modules/webbasket/lib/webbasket.py:2396
#: modules/webbasket/lib/webbasket_templates.py:167
#: modules/webbasket/lib/webbasket_templates.py:199
#: modules/webbasket/lib/webbasket_templates.py:205
#: modules/webbasket/lib/webbasket_templates.py:614
#: modules/webbasket/lib/webbasket_templates.py:691
msgid "Group baskets"
-msgstr ""
+msgstr "سبدهای گروهی"
#: modules/webbasket/lib/webbasket.py:2316
msgid "Others' baskets"
-msgstr ""
+msgstr "سبدهای دیگر"
#: modules/webbasket/lib/webbasket.py:2352
#: modules/websession/lib/websession_templates.py:465
#: modules/websession/lib/websession_templates.py:613
msgid "Your Baskets"
-msgstr ""
+msgstr "سبدهای شما"
#: modules/webbasket/lib/webbasket.py:2357
#: modules/webbasket/lib/webbasket_webinterface.py:1273
#: modules/webbasket/lib/webbasket_webinterface.py:1358
#: modules/webbasket/lib/webbasket_webinterface.py:1401
#: modules/webbasket/lib/webbasket_webinterface.py:1461
msgid "List of public baskets"
-msgstr ""
+msgstr "سیاهه سبدهای عمومی"
#: modules/webbasket/lib/webbasket.py:2368
#: modules/webbasket/lib/webbasket_webinterface.py:428
msgid "Search baskets"
-msgstr ""
+msgstr "جستجوی سبدها"
#: modules/webbasket/lib/webbasket.py:2373
#: modules/webbasket/lib/webbasket_webinterface.py:738
#: modules/websearch/lib/websearch_templates.py:2852
#: modules/websearch/lib/websearch_templates.py:3038
msgid "Add to basket"
-msgstr ""
+msgstr "افزودن به سبد"
#: modules/webbasket/lib/webbasket.py:2413
#: modules/webbasket/lib/webbasket_templates.py:218
#: modules/webbasket/lib/webbasket_templates.py:224
#: modules/webbasket/lib/webbasket_templates.py:623
#: modules/webbasket/lib/webbasket_templates.py:725
msgid "Public baskets"
-msgstr ""
+msgstr "سبدهای عمومی"
#: modules/webbasket/lib/webbasket.py:2443
#, python-format
msgid ""
"You have %(x_nb_perso)s personal baskets and are subscribed to "
"%(x_nb_group)s group baskets and %(x_nb_public)s other users public baskets."
msgstr ""
#: modules/webbasket/lib/webbasket.py:2629
#: modules/webbasket/lib/webbasket.py:2667
msgid ""
"The category you have selected does not exist. Please select a valid "
"category."
msgstr ""
#: modules/webbasket/lib/webbasket.py:2693
msgid "The selected group does not exist or you do not have access to it."
msgstr ""
#: modules/webbasket/lib/webbasket.py:2738
msgid "The selected output format is not available or is invalid."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:87
msgid ""
"You have no personal or group baskets or are subscribed to any public "
"baskets."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:88
#, python-format
msgid ""
"You may want to start by %(x_url_open)screating a new basket%(x_url_close)s."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:112
#: modules/webbasket/lib/webbasket_templates.py:178
msgid "Back to Your Baskets"
-msgstr ""
+msgstr "بازگشت به سبدهای شما"
#: modules/webbasket/lib/webbasket_templates.py:118
#: modules/webbasket/lib/webbasket_webinterface.py:1243
msgid "Create basket"
-msgstr ""
+msgstr "ایجاد سبد"
#: modules/webbasket/lib/webbasket_templates.py:124
#: modules/webbasket/lib/webbasket_webinterface.py:1132
msgid "Edit topic"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:559
msgid "Search baskets for"
-msgstr ""
+msgstr "جستجوی سبدها برای"
#: modules/webbasket/lib/webbasket_templates.py:560
msgid "Search also in notes (where allowed)"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:597
msgid "Results overview"
-msgstr ""
+msgstr "نمای کلی نتایج"
#: modules/webbasket/lib/webbasket_templates.py:598
#: modules/webbasket/lib/webbasket_templates.py:607
#: modules/webbasket/lib/webbasket_templates.py:616
#: modules/webbasket/lib/webbasket_templates.py:625
#: modules/webbasket/lib/webbasket_templates.py:634
#: modules/webbasket/lib/webbasket_templates.py:659
#: modules/webbasket/lib/webbasket_templates.py:677
#: modules/webbasket/lib/webbasket_templates.py:693
#: modules/webbasket/lib/webbasket_templates.py:711
#: modules/webbasket/lib/webbasket_templates.py:727
#: modules/webbasket/lib/webbasket_templates.py:744
#: modules/webbasket/lib/webbasket_templates.py:760
#: modules/webbasket/lib/webbasket_templates.py:776
#, python-format
msgid "%i items found"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:632
#: modules/webbasket/lib/webbasket_templates.py:758
msgid "All public baskets"
-msgstr ""
+msgstr "همه سبدهای عمومی"
#: modules/webbasket/lib/webbasket_templates.py:648
msgid "No items found."
-msgstr ""
+msgstr "هیچ آیتمی یافت نشد."
#: modules/webbasket/lib/webbasket_templates.py:675
#: modules/webbasket/lib/webbasket_templates.py:709
#: modules/webbasket/lib/webbasket_templates.py:742
#: modules/webbasket/lib/webbasket_templates.py:774
#, python-format
msgid "In %(x_linked_basket_name)s"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:869
#: modules/webbasket/lib/webbasket_webinterface.py:1291
#: modules/webbasket/lib/webbasket_webinterface.py:1416
#: modules/webbasket/lib/webbasket_webinterface.py:1476
msgid "Public basket"
-msgstr ""
+msgstr "سبد عمومی"
#: modules/webbasket/lib/webbasket_templates.py:870
msgid "Owner"
-msgstr ""
+msgstr "مالک"
#: modules/webbasket/lib/webbasket_templates.py:871
msgid "Last update"
-msgstr ""
+msgstr "آخرین به روز رسانی"
#: modules/webbasket/lib/webbasket_templates.py:872
#: modules/bibcirculation/lib/bibcirculation_templates.py:113
msgid "Items"
-msgstr ""
+msgstr "آیتم ها"
#: modules/webbasket/lib/webbasket_templates.py:873
msgid "Views"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:955
msgid "There is currently no publicly accessible basket"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:977
#, python-format
msgid ""
"Displaying public baskets %(x_from)i - %(x_to)i out of "
"%(x_total_public_basket)i public baskets in total."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1044
#: modules/webbasket/lib/webbasket_templates.py:1068
#, python-format
msgid "%(x_title)s, by %(x_name)s on %(x_date)s:"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1047
#: modules/webbasket/lib/webbasket_templates.py:1071
#: modules/webcomment/lib/webcomment.py:1605
#, python-format
msgid "%(x_name)s wrote on %(x_date)s:"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1127
msgid "Select topic"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1143
#: modules/webbasket/lib/webbasket_templates.py:1541
#: modules/webbasket/lib/webbasket_templates.py:1550
msgid "Choose topic"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1144
#: modules/webbasket/lib/webbasket_templates.py:1552
msgid "or create a new one"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1144
msgid "Create new topic"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1145
#: modules/webbasket/lib/webbasket_templates.py:1538
msgid "Basket name"
-msgstr ""
+msgstr "نام سبد"
#: modules/webbasket/lib/webbasket_templates.py:1147
msgid "Create a new basket"
-msgstr ""
+msgstr "ایجاد یک سبد جدید"
#: modules/webbasket/lib/webbasket_templates.py:1199
msgid "Create new basket"
-msgstr ""
+msgstr "ایجاد سبد جدید"
#: modules/webbasket/lib/webbasket_templates.py:1269
#: modules/webbasket/lib/webbasket_templates.py:2297
#: modules/webbasket/lib/webbasket_templates.py:2673
#: modules/webbasket/lib/webbasket_templates.py:3182
#: modules/webbasket/lib/webbasket_templates.py:3498
msgid "External item"
-msgstr ""
+msgstr "آیتم خروجی"
#: modules/webbasket/lib/webbasket_templates.py:1270
msgid ""
"Provide a url for the external item you wish to add and fill in a title and "
"description"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1271
#: modules/websubmit/lib/websubmit_templates.py:2726
#: modules/bibcirculation/lib/bibcirculation_utils.py:428
#: modules/bibcirculation/lib/bibcirculation_templates.py:2101
#: modules/bibcirculation/lib/bibcirculation_templates.py:2741
#: modules/bibcirculation/lib/bibcirculation_templates.py:5991
#: modules/bibcirculation/lib/bibcirculation_templates.py:8850
#: modules/bibcirculation/lib/bibcirculation_templates.py:11207
#: modules/bibcirculation/lib/bibcirculation_templates.py:11950
#: modules/bibcirculation/lib/bibcirculation_templates.py:12151
#: modules/bibcirculation/lib/bibcirculation_templates.py:12934
#: modules/bibcirculation/lib/bibcirculation_templates.py:15419
#: modules/bibcirculation/lib/bibcirculation_templates.py:16118
#: modules/bibcirculation/lib/bibcirculation_templates.py:16839
#: modules/bibcirculation/lib/bibcirculation_templates.py:17027
msgid "Title"
-msgstr ""
+msgstr "عنوان"
#: modules/webbasket/lib/webbasket_templates.py:1275
msgid "URL"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1305
#, python-format
msgid "%i items have been successfully added to your basket"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1306
#, python-format
msgid "Proceed to the %(x_url_open)sbasket%(x_url_close)s"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1311
#, python-format
msgid " or return to your %(x_url_open)sprevious basket%(x_url_close)s"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1315
#, python-format
msgid " or return to your %(x_url_open)ssearch%(x_url_close)s"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1428
#, python-format
msgid "Adding %i items to your baskets"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1429
#, python-format
msgid ""
"Please choose a basket: %(x_basket_selection_box)s %(x_fmt_open)s(or "
"%(x_url_open)screate a new one%(x_url_close)s first)%(x_fmt_close)s"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1443
msgid "Optionally, add a note to each one of these items"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1444
msgid "Optionally, add a note to this item"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1450
msgid "Add items"
-msgstr ""
+msgstr "افزودن آیتم ها"
#: modules/webbasket/lib/webbasket_templates.py:1474
msgid "Are you sure you want to delete this basket?"
-msgstr ""
+msgstr "آیا برای حذف این سبد مطمئن هستید؟"
#: modules/webbasket/lib/webbasket_templates.py:1476
#, python-format
msgid "%i users are subscribed to this basket."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1478
#, python-format
msgid "%i user groups are subscribed to this basket."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1480
#, python-format
msgid "You have set %i alerts on this basket."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1518
#: modules/webcomment/lib/webcomment_templates.py:232
#: modules/webcomment/lib/webcomment_templates.py:662
#: modules/webcomment/lib/webcomment_templates.py:1965
#: modules/webcomment/lib/webcomment_templates.py:1989
#: modules/webcomment/lib/webcomment_templates.py:2015
#: modules/webmessage/lib/webmessage_templates.py:508
#: modules/websession/lib/websession_templates.py:2214
#: modules/websession/lib/websession_templates.py:2254
msgid "Yes"
-msgstr ""
+msgstr "بله"
#: modules/webbasket/lib/webbasket_templates.py:1555
#: modules/webbasket/lib/webbasket_templates.py:1651
msgid "General settings"
-msgstr ""
+msgstr "تنظیمات کلی"
#: modules/webbasket/lib/webbasket_templates.py:1570
#: modules/webbasket/lib/webbasket_templates.py:1745
#: modules/webbasket/lib/webbasket_templates.py:1772
msgid "Add group"
-msgstr ""
+msgstr "افزودن گروه"
#: modules/webbasket/lib/webbasket_templates.py:1575
msgid "Manage group rights"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1587
msgid "Manage global sharing rights"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1592
#: modules/webbasket/lib/webbasket_templates.py:1658
#: modules/webbasket/lib/webbasket_templates.py:2006
#: modules/webbasket/lib/webbasket_templates.py:2085
msgid "Delete basket"
-msgstr ""
+msgstr "حذف سبد"
#: modules/webbasket/lib/webbasket_templates.py:1616
#, python-format
msgid "Editing basket %(x_basket_name)s"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1625
#: modules/webbasket/lib/webbasket_templates.py:1680
msgid "Save changes"
-msgstr ""
+msgstr "ذخیره تغییرات"
#: modules/webbasket/lib/webbasket_templates.py:1646
msgid "Topic name"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1675
#, python-format
msgid "Editing topic: %(x_topic_name)s"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1692
#: modules/webbasket/lib/webbasket_templates.py:1707
msgid "No rights"
-msgstr ""
+msgstr "بدون حقوق"
#: modules/webbasket/lib/webbasket_templates.py:1694
#: modules/webbasket/lib/webbasket_templates.py:1709
msgid "View records"
-msgstr ""
+msgstr "دیدن رکوردها"
#: modules/webbasket/lib/webbasket_templates.py:1696
#: modules/webbasket/lib/webbasket_templates.py:1698
#: modules/webbasket/lib/webbasket_templates.py:1711
#: modules/webbasket/lib/webbasket_templates.py:1713
#: modules/webbasket/lib/webbasket_templates.py:1715
#: modules/webbasket/lib/webbasket_templates.py:1717
#: modules/webbasket/lib/webbasket_templates.py:1719
#: modules/webbasket/lib/webbasket_templates.py:1721
msgid "and"
-msgstr ""
+msgstr "و"
#: modules/webbasket/lib/webbasket_templates.py:1696
msgid "view comments"
-msgstr ""
+msgstr "دیدن نظرها"
#: modules/webbasket/lib/webbasket_templates.py:1698
msgid "add comments"
-msgstr ""
+msgstr "افزودن نظرها"
#: modules/webbasket/lib/webbasket_templates.py:1711
msgid "view notes"
-msgstr ""
+msgstr "دیدن یادداشت ها"
#: modules/webbasket/lib/webbasket_templates.py:1713
msgid "add notes"
-msgstr ""
+msgstr "افزودن یادداشت ها"
#: modules/webbasket/lib/webbasket_templates.py:1715
msgid "add records"
-msgstr ""
+msgstr "افزودن رکوردها"
#: modules/webbasket/lib/webbasket_templates.py:1717
msgid "delete notes"
-msgstr ""
+msgstr "پاک کردن یادداشت ها"
#: modules/webbasket/lib/webbasket_templates.py:1719
msgid "remove records"
-msgstr ""
+msgstr "حذف رکوردها"
#: modules/webbasket/lib/webbasket_templates.py:1721
msgid "manage sharing rights"
-msgstr ""
+msgstr "مدیریت حقوق اشتراک"
#: modules/webbasket/lib/webbasket_templates.py:1743
msgid "You are not a member of a group."
-msgstr ""
+msgstr "شما عضوی از یک گروه نیستید"
#: modules/webbasket/lib/webbasket_templates.py:1765
msgid "Sharing basket to a new group"
-msgstr ""
+msgstr "اشترات سبد برای یک گروه جدید"
#: modules/webbasket/lib/webbasket_templates.py:1794
#: modules/websession/lib/websession_templates.py:510
msgid ""
"You are logged in as a guest user, so your baskets will disappear at the end "
"of the current session."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1795
#: modules/webbasket/lib/webbasket_templates.py:1810
#: modules/websession/lib/websession_templates.py:513
#, python-format
msgid ""
"If you wish you can %(x_url_open)slogin or register here%(x_url_close)s."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1809
#: modules/websession/lib/websession_webinterface.py:263
msgid "This functionality is forbidden to guest users."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1862
#: modules/webcomment/lib/webcomment_templates.py:864
msgid "Back to search results"
-msgstr ""
+msgstr "بازگشت به نتایج جستجو"
#: modules/webbasket/lib/webbasket_templates.py:1990
#: modules/webbasket/lib/webbasket_templates.py:3022
#, python-format
msgid "%i items"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1991
#: modules/webbasket/lib/webbasket_templates.py:3024
#, python-format
msgid "%i notes"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1991
#: modules/webbasket/lib/webbasket_templates.py:3024
msgid "no notes yet"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1994
#, python-format
msgid "%i subscribers"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:1996
#: modules/webbasket/lib/webbasket_templates.py:3026
msgid "last update"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2000
#: modules/webbasket/lib/webbasket_templates.py:2079
msgid "Add item"
-msgstr ""
+msgstr "افزودن آیتم"
#: modules/webbasket/lib/webbasket_templates.py:2003
#: modules/webbasket/lib/webbasket_templates.py:2082
#: modules/webbasket/lib/webbasket_webinterface.py:1037
msgid "Edit basket"
-msgstr ""
+msgstr "ویرایش سبد"
#: modules/webbasket/lib/webbasket_templates.py:2016
#: modules/webbasket/lib/webbasket_templates.py:2093
#: modules/webbasket/lib/webbasket_templates.py:3034
#: modules/webbasket/lib/webbasket_templates.py:3087
msgid "Unsubscribe from basket"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2098
msgid "This basket is publicly accessible at the following address:"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2162
#: modules/webbasket/lib/webbasket_templates.py:3137
msgid "Basket is empty"
-msgstr ""
+msgstr "سبد خالی است"
#: modules/webbasket/lib/webbasket_templates.py:2196
msgid "You do not have sufficient rights to view this basket's content."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2239
msgid "Move item up"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2243
msgid "You cannot move this item up"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2257
msgid "Move item down"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2261
msgid "You cannot move this item down"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2275
#: modules/webbasket/lib/webbasket_templates.py:3178
msgid "Copy item"
-msgstr ""
+msgstr "کپی آیتم"
#: modules/webbasket/lib/webbasket_templates.py:2291
msgid "Remove item"
-msgstr ""
+msgstr "حذف آیتم"
#: modules/webbasket/lib/webbasket_templates.py:2363
#: modules/webbasket/lib/webbasket_templates.py:2718
#: modules/webbasket/lib/webbasket_templates.py:3239
#: modules/webbasket/lib/webbasket_templates.py:3539
msgid "This record does not seem to exist any more"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2366
#: modules/webbasket/lib/webbasket_templates.py:2868
#: modules/webbasket/lib/webbasket_templates.py:3242
#: modules/webbasket/lib/webbasket_templates.py:3671
#: modules/bibcirculation/lib/bibcirculation_templates.py:3945
#: modules/bibcirculation/lib/bibcirculation_templates.py:4048
#: modules/bibcirculation/lib/bibcirculation_templates.py:4271
#: modules/bibcirculation/lib/bibcirculation_templates.py:4332
#: modules/bibcirculation/lib/bibcirculation_templates.py:4459
#: modules/bibcirculation/lib/bibcirculation_templates.py:6190
#: modules/bibcirculation/lib/bibcirculation_templates.py:6239
#: modules/bibcirculation/lib/bibcirculation_templates.py:6629
#: modules/bibcirculation/lib/bibcirculation_templates.py:6704
#: modules/bibcirculation/lib/bibcirculation_templates.py:10669
#: modules/bibcirculation/lib/bibcirculation_templates.py:10821
#: modules/bibcirculation/lib/bibcirculation_templates.py:10916
#: modules/bibcirculation/lib/bibcirculation_templates.py:13775
#: modules/bibcirculation/lib/bibcirculation_templates.py:13956
#: modules/bibcirculation/lib/bibcirculation_templates.py:14071
#: modules/bibcirculation/lib/bibcirculation_templates.py:14144
#: modules/bibcirculation/lib/bibcirculation_templates.py:14717
msgid "Notes"
-msgstr ""
+msgstr "یادداشت ها"
#: modules/webbasket/lib/webbasket_templates.py:2366
#: modules/webbasket/lib/webbasket_templates.py:3242
msgid "Add a note..."
-msgstr ""
+msgstr "افزودن یک یادداشت..."
#: modules/webbasket/lib/webbasket_templates.py:2478
#: modules/webbasket/lib/webbasket_templates.py:3330
#, python-format
msgid "Item %(x_item_index)i of %(x_item_total)i"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2491
#: modules/webbasket/lib/webbasket_templates.py:2494
#: modules/webbasket/lib/webbasket_templates.py:2578
#: modules/webbasket/lib/webbasket_templates.py:2581
#: modules/webbasket/lib/webbasket_templates.py:3340
#: modules/webbasket/lib/webbasket_templates.py:3343
#: modules/webbasket/lib/webbasket_templates.py:3415
#: modules/webbasket/lib/webbasket_templates.py:3418
msgid "Previous item"
-msgstr ""
+msgstr "آیتم پیشین"
#: modules/webbasket/lib/webbasket_templates.py:2506
#: modules/webbasket/lib/webbasket_templates.py:2509
#: modules/webbasket/lib/webbasket_templates.py:2593
#: modules/webbasket/lib/webbasket_templates.py:2596
#: modules/webbasket/lib/webbasket_templates.py:3352
#: modules/webbasket/lib/webbasket_templates.py:3355
#: modules/webbasket/lib/webbasket_templates.py:3427
#: modules/webbasket/lib/webbasket_templates.py:3430
msgid "Next item"
-msgstr ""
+msgstr "آیتم پسین"
#: modules/webbasket/lib/webbasket_templates.py:2519
#: modules/webbasket/lib/webbasket_templates.py:2606
#: modules/webbasket/lib/webbasket_templates.py:3362
#: modules/webbasket/lib/webbasket_templates.py:3437
msgid "Return to basket"
-msgstr ""
+msgstr "بازگشت به سبد"
#: modules/webbasket/lib/webbasket_templates.py:2666
#: modules/webbasket/lib/webbasket_templates.py:3491
msgid "The item you have selected does not exist."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2694
#: modules/webbasket/lib/webbasket_templates.py:3515
msgid "You do not have sufficient rights to view this item's notes."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2735
msgid "You do not have sufficient rights to view this item."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2842
#: modules/webbasket/lib/webbasket_templates.py:2852
#: modules/webbasket/lib/webbasket_templates.py:3645
#: modules/webbasket/lib/webbasket_templates.py:3655
#: modules/webbasket/lib/webbasket_webinterface.py:493
#: modules/webbasket/lib/webbasket_webinterface.py:1538
msgid "Add a note"
-msgstr ""
+msgstr "افزودن یک یادداشت"
#: modules/webbasket/lib/webbasket_templates.py:2843
#: modules/webbasket/lib/webbasket_templates.py:3646
msgid "Add note"
-msgstr ""
+msgstr "افزودن یادداشت"
#: modules/webbasket/lib/webbasket_templates.py:2889
#: modules/webbasket/lib/webbasket_templates.py:3692
#: modules/webcomment/lib/webcomment_templates.py:373
#: modules/webmessage/lib/webmessage_templates.py:111
msgid "Reply"
-msgstr ""
+msgstr "پاسخ دادن"
#: modules/webbasket/lib/webbasket_templates.py:2919
#: modules/webbasket/lib/webbasket_templates.py:3717
#, python-format
msgid "%(x_title)s, by %(x_name)s on %(x_date)s"
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:2921
#: modules/webbasket/lib/webbasket_templates.py:3719
#: modules/websession/lib/websession_templates.py:165
#: modules/websession/lib/websession_templates.py:214
#: modules/websession/lib/websession_templates.py:915
#: modules/websession/lib/websession_templates.py:1039
msgid "Note"
-msgstr ""
+msgstr "یادداشت"
#: modules/webbasket/lib/webbasket_templates.py:3031
#: modules/webbasket/lib/webbasket_templates.py:3084
msgid "Subscribe to basket"
-msgstr ""
+msgstr "آبونه شدن سبد"
#: modules/webbasket/lib/webbasket_templates.py:3090
msgid "This public basket belongs to the user "
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:3114
msgid "This public basket belongs to you."
msgstr ""
#: modules/webbasket/lib/webbasket_templates.py:3889
msgid "All your baskets"
-msgstr ""
+msgstr "همه سبدهای شما"
#: modules/webbasket/lib/webbasket_templates.py:3891
#: modules/webbasket/lib/webbasket_templates.py:3966
msgid "Your personal baskets"
-msgstr ""
+msgstr "سبدهای شخصی شما"
#: modules/webbasket/lib/webbasket_templates.py:3897
#: modules/webbasket/lib/webbasket_templates.py:3977
msgid "Your group baskets"
-msgstr ""
+msgstr "سبدهای گروهی شما"
#: modules/webbasket/lib/webbasket_templates.py:3903
msgid "Your public baskets"
-msgstr ""
+msgstr "سبدهای عمومی شما"
#: modules/webbasket/lib/webbasket_templates.py:3904
msgid "All the public baskets"
-msgstr ""
+msgstr "همه سبدهای عمومی"
#: modules/webbasket/lib/webbasket_templates.py:3961
msgid "*** basket name ***"
msgstr ""
#: modules/webbasket/lib/webbasket_webinterface.py:158
#: modules/webbasket/lib/webbasket_webinterface.py:319
#: modules/webbasket/lib/webbasket_webinterface.py:406
#: modules/webbasket/lib/webbasket_webinterface.py:471
#: modules/webbasket/lib/webbasket_webinterface.py:538
#: modules/webbasket/lib/webbasket_webinterface.py:615
#: modules/webbasket/lib/webbasket_webinterface.py:702
#: modules/webbasket/lib/webbasket_webinterface.py:780
#: modules/webbasket/lib/webbasket_webinterface.py:864
#: modules/webbasket/lib/webbasket_webinterface.py:961
#: modules/webbasket/lib/webbasket_webinterface.py:1077
#: modules/webbasket/lib/webbasket_webinterface.py:1177
#: modules/webbasket/lib/webbasket_webinterface.py:1397
#: modules/webbasket/lib/webbasket_webinterface.py:1457
#: modules/webbasket/lib/webbasket_webinterface.py:1519
#: modules/webbasket/lib/webbasket_webinterface.py:1581
msgid "You are not authorized to use baskets."
-msgstr ""
+msgstr "شما برای استفاده این سبدها اجازه ندارید"
#: modules/webbasket/lib/webbasket_webinterface.py:169
msgid "You are not authorized to view this attachment"
-msgstr ""
+msgstr "شما برای دیدن این ضمیمه اجازه ندارید"
#: modules/webbasket/lib/webbasket_webinterface.py:361
msgid "Display baskets"
-msgstr ""
+msgstr "نمایش سبدها"
#: modules/webbasket/lib/webbasket_webinterface.py:564
#: modules/webbasket/lib/webbasket_webinterface.py:639
#: modules/webbasket/lib/webbasket_webinterface.py:1604
msgid "Display item and notes"
-msgstr ""
+msgstr "نمایش آیتم و یادداشت ها"
#: modules/webbasket/lib/webbasket_webinterface.py:821
msgid "Delete a basket"
-msgstr ""
+msgstr "حذف یک سبد"
#: modules/webbasket/lib/webbasket_webinterface.py:879
msgid "Copy record to basket"
-msgstr ""
+msgstr "کپی رکورد به سبد"
#: modules/webcomment/lib/webcommentadminlib.py:122
msgid "Invalid comment ID."
msgstr ""
#: modules/webcomment/lib/webcommentadminlib.py:142
#, python-format
msgid "Comment ID %s does not exist."
msgstr ""
#: modules/webcomment/lib/webcommentadminlib.py:156
#, python-format
msgid "Record ID %s does not exist."
msgstr ""
#: modules/webcomment/lib/webcomment.py:166
#: modules/webcomment/lib/webcomment.py:210
msgid "Bad page number --> showing first page."
msgstr ""
#: modules/webcomment/lib/webcomment.py:174
+#, fuzzy
msgid "Bad number of results per page --> showing 10 results per page."
-msgstr ""
+msgstr "تعداد نتایج جستجو در هر صفحه"
#: modules/webcomment/lib/webcomment.py:183
msgid "Bad display order --> showing most helpful first."
msgstr ""
#: modules/webcomment/lib/webcomment.py:192
msgid "Bad display order --> showing oldest first."
msgstr ""
#: modules/webcomment/lib/webcomment.py:229
#: modules/webcomment/lib/webcomment.py:1579
#: modules/webcomment/lib/webcomment.py:1632
msgid "Comments on records have been disallowed by the administrator."
msgstr ""
#: modules/webcomment/lib/webcomment.py:237
#: modules/webcomment/lib/webcomment.py:260
#: modules/webcomment/lib/webcomment.py:1419
#: modules/webcomment/lib/webcomment.py:1440
msgid "Your feedback has been recorded, many thanks."
msgstr ""
#: modules/webcomment/lib/webcomment.py:244
msgid "You have already reported an abuse for this comment."
msgstr ""
#: modules/webcomment/lib/webcomment.py:251
msgid "The comment you have reported no longer exists."
msgstr ""
#: modules/webcomment/lib/webcomment.py:267
msgid "Sorry, you have already voted. This vote has not been recorded."
msgstr ""
#: modules/webcomment/lib/webcomment.py:274
msgid ""
"You have been subscribed to this discussion. From now on, you will receive "
"an email whenever a new comment is posted."
msgstr ""
#: modules/webcomment/lib/webcomment.py:281
msgid "You have been unsubscribed from this discussion."
msgstr ""
#: modules/webcomment/lib/webcomment.py:1171
#, python-format
msgid "Record %i"
msgstr ""
#: modules/webcomment/lib/webcomment.py:1182
#, python-format
msgid "%(report_number)s\"%(title)s\" has been reviewed"
msgstr ""
#: modules/webcomment/lib/webcomment.py:1186
#, python-format
msgid "%(report_number)s\"%(title)s\" has been commented"
msgstr ""
#: modules/webcomment/lib/webcomment.py:1407
#, python-format
msgid "%s is an invalid record ID"
msgstr ""
#: modules/webcomment/lib/webcomment.py:1426
#: modules/webcomment/lib/webcomment.py:1447
msgid "Your feedback could not be recorded, please try again."
msgstr ""
#: modules/webcomment/lib/webcomment.py:1555
#, python-format
msgid "%s is an invalid user ID."
msgstr ""
#: modules/webcomment/lib/webcomment.py:1589
msgid "Cannot reply to a review."
msgstr ""
#: modules/webcomment/lib/webcomment.py:1644
msgid "You must enter a title."
-msgstr ""
+msgstr "شما باید یک عنوان را وارد کنید."
#: modules/webcomment/lib/webcomment.py:1651
msgid "You must choose a score."
-msgstr ""
+msgstr "شما باید یک نمره را انتخاب کنید."
#: modules/webcomment/lib/webcomment.py:1658
msgid "You must enter a text."
-msgstr ""
+msgstr "شما باید یک متن را وارد کنید."
#: modules/webcomment/lib/webcomment.py:1675
msgid "You already wrote a review for this record."
-msgstr ""
+msgstr "شما قبلا یک بررسی برای این رکورد نوشته اید."
#: modules/webcomment/lib/webcomment.py:1693
msgid "You already posted a comment short ago. Please retry later."
msgstr ""
#: modules/webcomment/lib/webcomment.py:1705
msgid "Failed to insert your comment to the database. Please try again."
msgstr ""
#: modules/webcomment/lib/webcomment.py:1719
msgid "Unknown action --> showing you the default add comment form."
msgstr ""
#: modules/webcomment/lib/webcomment.py:1841
#, python-format
msgid "Record ID %s does not exist in the database."
msgstr ""
#: modules/webcomment/lib/webcomment.py:1849
msgid "No record ID was given."
msgstr ""
#: modules/webcomment/lib/webcomment.py:1857
#, python-format
msgid "Record ID %s is an invalid ID."
msgstr ""
#: modules/webcomment/lib/webcomment.py:1865
#, python-format
msgid "Record ID %s is not a number."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:79
#: modules/webcomment/lib/webcomment_templates.py:839
#: modules/websubmit/lib/websubmit_templates.py:2674
msgid "Write a comment"
-msgstr ""
+msgstr "یک نظر بنویسید"
#: modules/webcomment/lib/webcomment_templates.py:94
#, python-format
msgid ""
"<div class=\"webcomment_comment_round_header\">%(x_nb)i Comments for round "
"\"%(x_name)s\""
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:96
#, python-format
msgid "<div class=\"webcomment_comment_round_header\">%(x_nb)i Comments"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:124
#, python-format
msgid "Showing the latest %i comments:"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:137
#: modules/webcomment/lib/webcomment_templates.py:163
msgid "Discuss this document"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:164
#: modules/webcomment/lib/webcomment_templates.py:849
msgid "Start a discussion about any aspect of this document."
-msgstr ""
+msgstr "شروع یک بحث درباره هر جنبه ای از این مدرک."
#: modules/webcomment/lib/webcomment_templates.py:180
#, python-format
msgid "Sorry, the record %s does not seem to exist."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:182
#, python-format
msgid "Sorry, %s is not a valid ID value."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:184
msgid "Sorry, no record ID was provided."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:188
#, python-format
msgid "You may want to start browsing from %s"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:244
#: modules/webcomment/lib/webcomment_templates.py:704
#: modules/webcomment/lib/webcomment_templates.py:714
#, python-format
msgid "%(x_nb)i comments for round \"%(x_name)s\""
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:267
#: modules/webcomment/lib/webcomment_templates.py:763
msgid "Was this review helpful?"
-msgstr ""
+msgstr "آیا این بررسی سودمند بود؟"
#: modules/webcomment/lib/webcomment_templates.py:278
#: modules/webcomment/lib/webcomment_templates.py:315
#: modules/webcomment/lib/webcomment_templates.py:839
msgid "Write a review"
-msgstr ""
+msgstr "یک بررسی بنویسید"
#: modules/webcomment/lib/webcomment_templates.py:285
#: modules/webcomment/lib/webcomment_templates.py:827
#: modules/webcomment/lib/webcomment_templates.py:2036
#, python-format
msgid "Average review score: %(x_nb_score)s based on %(x_nb_reviews)s reviews"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:288
#, python-format
msgid "Readers found the following %s reviews to be most helpful."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:291
#: modules/webcomment/lib/webcomment_templates.py:314
#, python-format
msgid "View all %s reviews"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:310
#: modules/webcomment/lib/webcomment_templates.py:332
#: modules/webcomment/lib/webcomment_templates.py:2077
msgid "Rate this document"
-msgstr ""
+msgstr "این مدرک را درجه بندی کنید"
#: modules/webcomment/lib/webcomment_templates.py:333
msgid ""
"<div class=\"webcomment_review_first_introduction\">Be the first to review "
"this document.</div>"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:368
#, python-format
msgid "%(x_name)s"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:375
#: modules/webcomment/lib/webcomment_templates.py:764
msgid "Report abuse"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:390
msgid "Undelete comment"
-msgstr ""
+msgstr "بازگرداندن نظر"
#: modules/webcomment/lib/webcomment_templates.py:399
#: modules/webcomment/lib/webcomment_templates.py:401
msgid "Delete comment"
-msgstr ""
+msgstr "حذف نظر"
#: modules/webcomment/lib/webcomment_templates.py:407
msgid "Unreport comment"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:418
msgid "Attached file"
-msgstr ""
+msgstr "فایل پیوست شده"
#: modules/webcomment/lib/webcomment_templates.py:418
msgid "Attached files"
-msgstr ""
+msgstr "فایل های پیوست شده"
#: modules/webcomment/lib/webcomment_templates.py:484
#, python-format
msgid "Reviewed by %(x_nickname)s on %(x_date)s"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:488
#, python-format
msgid "%(x_nb_people)s out of %(x_nb_total)s people found this review useful"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:510
msgid "Undelete review"
-msgstr ""
+msgstr "بازگرداندن بررسی"
#: modules/webcomment/lib/webcomment_templates.py:519
msgid "Delete review"
-msgstr ""
+msgstr "حذف بررسی"
#: modules/webcomment/lib/webcomment_templates.py:525
msgid "Unreport review"
-msgstr ""
+msgstr "بررسی گزارش نشده"
#: modules/webcomment/lib/webcomment_templates.py:631
#: modules/webcomment/lib/webcomment_templates.py:646
#: modules/webcomment/lib/webcomment_webinterface.py:237
#: modules/webcomment/lib/webcomment_webinterface.py:429
#: modules/websubmit/lib/websubmit_templates.py:2672
msgid "Comments"
-msgstr ""
+msgstr "نظرها"
#: modules/webcomment/lib/webcomment_templates.py:632
#: modules/webcomment/lib/webcomment_templates.py:647
#: modules/webcomment/lib/webcomment_webinterface.py:237
#: modules/webcomment/lib/webcomment_webinterface.py:429
msgid "Reviews"
-msgstr ""
+msgstr "بررسی ها"
#: modules/webcomment/lib/webcomment_templates.py:802
#: modules/websearch/lib/websearch_templates.py:1864
#: modules/bibcatalog/lib/bibcatalog_templates.py:50
#: modules/bibknowledge/lib/bibknowledge_templates.py:167
msgid "Previous"
-msgstr ""
+msgstr "قبلی"
#: modules/webcomment/lib/webcomment_templates.py:818
#: modules/bibcatalog/lib/bibcatalog_templates.py:72
#: modules/bibknowledge/lib/bibknowledge_templates.py:165
msgid "Next"
-msgstr ""
+msgstr "بعدی"
#: modules/webcomment/lib/webcomment_templates.py:842
#, python-format
msgid "There is a total of %s reviews"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:844
#, python-format
msgid "There is a total of %s comments"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:851
msgid "Be the first to review this document."
-msgstr ""
+msgstr "نخستین فرد بررسی کننده این مدرک باشید."
#: modules/webcomment/lib/webcomment_templates.py:863
#: modules/webcomment/lib/webcomment_templates.py:1643
#: modules/websearch/lib/websearch_templates.py:558
msgid "Record"
-msgstr ""
+msgstr "رکورد"
#: modules/webcomment/lib/webcomment_templates.py:870
#: modules/webcomment/lib/webcomment_templates.py:929
msgid "review"
-msgstr ""
+msgstr "بررسی"
#: modules/webcomment/lib/webcomment_templates.py:870
#: modules/webcomment/lib/webcomment_templates.py:929
msgid "comment"
-msgstr ""
+msgstr "نظر"
#: modules/webcomment/lib/webcomment_templates.py:871
#: modules/webcomment/lib/webcomment_templates.py:1879
msgid "Review"
-msgstr ""
+msgstr "بررسی"
#: modules/webcomment/lib/webcomment_templates.py:871
#: modules/webcomment/lib/webcomment_templates.py:1202
#: modules/webcomment/lib/webcomment_templates.py:1647
#: modules/webcomment/lib/webcomment_templates.py:1879
#: modules/websubmit/lib/websubmit_managedocfiles.py:389
#: modules/websubmit/lib/websubmit_templates.py:2728
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:347
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:401
#: modules/websubmit/lib/functions/Create_Upload_Files_Interface.py:455
msgid "Comment"
-msgstr ""
+msgstr "نظر"
#: modules/webcomment/lib/webcomment_templates.py:927
msgid "Viewing"
-msgstr ""
+msgstr "دیدن"
#: modules/webcomment/lib/webcomment_templates.py:928
msgid "Page:"
-msgstr ""
+msgstr "صفحه:"
#: modules/webcomment/lib/webcomment_templates.py:946
msgid "Subscribe"
-msgstr ""
+msgstr "مشترک شدن"
#: modules/webcomment/lib/webcomment_templates.py:955
msgid "Unsubscribe"
-msgstr ""
+msgstr "مشترک نشدن"
#: modules/webcomment/lib/webcomment_templates.py:962
msgid "You are not authorized to comment or review."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1132
#, python-format
msgid "Note: Your nickname, %s, will be displayed as author of this comment."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1136
#: modules/webcomment/lib/webcomment_templates.py:1253
#, python-format
msgid ""
"Note: you have not %(x_url_open)sdefined your nickname%(x_url_close)s. "
"%(x_nickname)s will be displayed as the author of this comment."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1153
msgid "Once logged in, authorized users can also attach files."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1168
msgid "Optionally, attach a file to this comment"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1169
msgid "Optionally, attach files to this comment"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1170
msgid "Max one file"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1171
#, python-format
msgid "Max %i files"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1172
#, python-format
msgid "Max %(x_nb_bytes)s per file"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1187
msgid "Send me an email when a new comment is posted"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1201
#: modules/webcomment/lib/webcomment_templates.py:1324
msgid "Article"
-msgstr ""
+msgstr "مقاله"
#: modules/webcomment/lib/webcomment_templates.py:1203
msgid "Add comment"
-msgstr ""
+msgstr "افزودن نظر"
#: modules/webcomment/lib/webcomment_templates.py:1248
#, python-format
msgid ""
"Note: Your nickname, %s, will be displayed as the author of this review."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1325
msgid "Rate this article"
-msgstr ""
+msgstr "این مقاله را درجه بندی کنید"
#: modules/webcomment/lib/webcomment_templates.py:1326
msgid "Select a score"
-msgstr ""
+msgstr "یک نمره انتخاب کنید"
#: modules/webcomment/lib/webcomment_templates.py:1327
msgid "Give a title to your review"
-msgstr ""
+msgstr "یک عنوان به بررسی تان بدهید"
#: modules/webcomment/lib/webcomment_templates.py:1328
msgid "Write your review"
-msgstr ""
+msgstr "بررسی تان را بنویسید"
#: modules/webcomment/lib/webcomment_templates.py:1333
msgid "Add review"
-msgstr ""
+msgstr "افزودن بررسی"
#: modules/webcomment/lib/webcomment_templates.py:1343
#: modules/webcomment/lib/webcomment_webinterface.py:474
msgid "Add Review"
-msgstr ""
+msgstr "افزودن بررسی"
#: modules/webcomment/lib/webcomment_templates.py:1365
msgid "Your review was successfully added."
-msgstr ""
+msgstr "بررسی شما به صورت موفقیت آمیزی اضافه شد."
#: modules/webcomment/lib/webcomment_templates.py:1367
msgid "Your comment was successfully added."
-msgstr ""
+msgstr "نظر شما به صورت موفقیت آمیزی اضافه شد."
#: modules/webcomment/lib/webcomment_templates.py:1370
msgid "Back to record"
-msgstr ""
+msgstr "برگشت به رکورد"
#: modules/webcomment/lib/webcomment_templates.py:1448
#: modules/webcomment/web/admin/webcommentadmin.py:171
msgid "View most commented records"
-msgstr ""
+msgstr "دیدن رکوردهای دارای بیشترین نظر"
#: modules/webcomment/lib/webcomment_templates.py:1450
#: modules/webcomment/web/admin/webcommentadmin.py:207
msgid "View latest commented records"
-msgstr ""
+msgstr "دیدن آخرین رکوردهای نظرداده شده"
#: modules/webcomment/lib/webcomment_templates.py:1452
#: modules/webcomment/web/admin/webcommentadmin.py:140
msgid "View all comments reported as abuse"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1456
#: modules/webcomment/web/admin/webcommentadmin.py:170
msgid "View most reviewed records"
-msgstr ""
+msgstr "دیدن رکوردهای دارای بیشترین بررسی"
#: modules/webcomment/lib/webcomment_templates.py:1458
#: modules/webcomment/web/admin/webcommentadmin.py:206
msgid "View latest reviewed records"
-msgstr ""
+msgstr "دیدن آخرین رکوردهای بررسی شده"
#: modules/webcomment/lib/webcomment_templates.py:1460
#: modules/webcomment/web/admin/webcommentadmin.py:140
msgid "View all reviews reported as abuse"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1468
msgid "View all users who have been reported"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1470
msgid "Guide"
-msgstr ""
+msgstr "راهنما"
#: modules/webcomment/lib/webcomment_templates.py:1472
msgid "Comments and reviews are disabled"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1492
msgid ""
"Please enter the ID of the comment/review so that you can view it before "
"deciding whether to delete it or not"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1516
msgid "Comment ID:"
-msgstr ""
+msgstr "هویت نظر:"
#: modules/webcomment/lib/webcomment_templates.py:1517
msgid "Or enter a record ID to list all the associated comments/reviews:"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1518
msgid "Record ID:"
-msgstr ""
+msgstr "هویت رکورد:"
#: modules/webcomment/lib/webcomment_templates.py:1520
msgid "View Comment"
-msgstr ""
+msgstr "مشاهده نظر"
#: modules/webcomment/lib/webcomment_templates.py:1541
msgid "There have been no reports so far."
-msgstr ""
+msgstr "تا کنون هیچ گزارشی وجود نداشته است."
#: modules/webcomment/lib/webcomment_templates.py:1545
#, python-format
msgid "View all %s reported comments"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1548
#, python-format
msgid "View all %s reported reviews"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1585
msgid ""
"Here is a list, sorted by total number of reports, of all users who have had "
"a comment reported at least once."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1593
#: modules/webcomment/lib/webcomment_templates.py:1622
#: modules/websession/lib/websession_templates.py:158
#: modules/websession/lib/websession_templates.py:1034
msgid "Nickname"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1594
#: modules/webcomment/lib/webcomment_templates.py:1626
#: modules/bibcirculation/lib/bibcirculation_utils.py:457
#: modules/bibcirculation/lib/bibcirculation_templates.py:2390
#: modules/bibcirculation/lib/bibcirculation_templates.py:2507
#: modules/bibcirculation/lib/bibcirculation_templates.py:2739
#: modules/bibcirculation/lib/bibcirculation_templates.py:3942
#: modules/bibcirculation/lib/bibcirculation_templates.py:4045
#: modules/bibcirculation/lib/bibcirculation_templates.py:4268
#: modules/bibcirculation/lib/bibcirculation_templates.py:4329
#: modules/bibcirculation/lib/bibcirculation_templates.py:4457
#: modules/bibcirculation/lib/bibcirculation_templates.py:5605
#: modules/bibcirculation/lib/bibcirculation_templates.py:6186
#: modules/bibcirculation/lib/bibcirculation_templates.py:6235
#: modules/bibcirculation/lib/bibcirculation_templates.py:6536
#: modules/bibcirculation/lib/bibcirculation_templates.py:6597
#: modules/bibcirculation/lib/bibcirculation_templates.py:6700
#: modules/bibcirculation/lib/bibcirculation_templates.py:6929
#: modules/bibcirculation/lib/bibcirculation_templates.py:7028
#: modules/bibcirculation/lib/bibcirculation_templates.py:9028
#: modules/bibcirculation/lib/bibcirculation_templates.py:9273
#: modules/bibcirculation/lib/bibcirculation_templates.py:9882
#: modules/bibcirculation/lib/bibcirculation_templates.py:10360
#: modules/bibcirculation/lib/bibcirculation_templates.py:11224
#: modules/bibcirculation/lib/bibcirculation_templates.py:12211
#: modules/bibcirculation/lib/bibcirculation_templates.py:12995
#: modules/bibcirculation/lib/bibcirculation_templates.py:14071
#: modules/bibcirculation/lib/bibcirculation_templates.py:14141
#: modules/bibcirculation/lib/bibcirculation_templates.py:14390
#: modules/bibcirculation/lib/bibcirculation_templates.py:14461
#: modules/bibcirculation/lib/bibcirculation_templates.py:14715
#: modules/bibcirculation/lib/bibcirculation_templates.py:15518
msgid "Email"
-msgstr ""
+msgstr "ایمیل"
#: modules/webcomment/lib/webcomment_templates.py:1595
#: modules/webcomment/lib/webcomment_templates.py:1624
msgid "User ID"
-msgstr ""
+msgstr "هویت کاربر"
#: modules/webcomment/lib/webcomment_templates.py:1597
msgid "Number positive votes"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1598
msgid "Number negative votes"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1599
msgid "Total number votes"
-msgstr ""
+msgstr "کل تعداد رأی ها"
#: modules/webcomment/lib/webcomment_templates.py:1600
msgid "Total number of reports"
-msgstr ""
+msgstr "کل تعداد گزارش ها"
#: modules/webcomment/lib/webcomment_templates.py:1601
msgid "View all user's reported comments/reviews"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1634
#, python-format
msgid "This review has been reported %i times"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1636
#, python-format
msgid "This comment has been reported %i times"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1880
msgid "Written by"
-msgstr ""
+msgstr "نوشته شده به وسیله"
#: modules/webcomment/lib/webcomment_templates.py:1881
msgid "General informations"
-msgstr ""
+msgstr "اطلاعات کلی"
#: modules/webcomment/lib/webcomment_templates.py:1882
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:652
msgid "Select"
-msgstr ""
+msgstr "انتخاب"
#: modules/webcomment/lib/webcomment_templates.py:1896
msgid "Delete selected reviews"
-msgstr ""
+msgstr "حذف بررسی های انتخاب شده"
#: modules/webcomment/lib/webcomment_templates.py:1897
#: modules/webcomment/lib/webcomment_templates.py:1904
msgid "Suppress selected abuse report"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1898
msgid "Undelete selected reviews"
-msgstr ""
+msgstr "عدم حذف بررسی های انتخاب شده"
#: modules/webcomment/lib/webcomment_templates.py:1902
msgid "Undelete selected comments"
-msgstr ""
+msgstr "عدم حذف نظرات انتخاب شده"
#: modules/webcomment/lib/webcomment_templates.py:1903
msgid "Delete selected comments"
-msgstr ""
+msgstr "حذف نظرات انتخاب شده"
#: modules/webcomment/lib/webcomment_templates.py:1912
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:494
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:557
#: modules/bibcirculation/lib/bibcirculation_templates.py:1635
msgid "OK"
-msgstr ""
+msgstr "بسیارخوب"
#: modules/webcomment/lib/webcomment_templates.py:1918
#, python-format
msgid "Here are the reported reviews of user %s"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1920
#, python-format
msgid "Here are the reported comments of user %s"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1924
#, python-format
msgid "Here is review %s"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1926
#, python-format
msgid "Here is comment %s"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1929
#, python-format
msgid "Here is review %(x_cmtID)s written by user %(x_user)s"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1931
#, python-format
msgid "Here is comment %(x_cmtID)s written by user %(x_user)s"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1937
msgid "Here are all reported reviews sorted by the most reported"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1939
msgid "Here are all reported comments sorted by the most reported"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1944
#, python-format
msgid "Here are all reviews for record %i, sorted by the most reported"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1945
msgid "Show comments"
-msgstr ""
+msgstr "نمایش نظرها"
#: modules/webcomment/lib/webcomment_templates.py:1947
#, python-format
msgid "Here are all comments for record %i, sorted by the most reported"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:1948
msgid "Show reviews"
-msgstr ""
+msgstr "نمایش بررسی ها"
#: modules/webcomment/lib/webcomment_templates.py:1973
#: modules/webcomment/lib/webcomment_templates.py:1997
#: modules/webcomment/lib/webcomment_templates.py:2023
msgid "comment ID"
-msgstr ""
+msgstr "هویت نظر"
#: modules/webcomment/lib/webcomment_templates.py:1973
msgid "successfully deleted"
-msgstr ""
+msgstr "به طور موفقیت آمیزی حذف شد"
#: modules/webcomment/lib/webcomment_templates.py:1997
msgid "successfully undeleted"
-msgstr ""
+msgstr "به طور موفقیت آمیزی حذف نشد"
#: modules/webcomment/lib/webcomment_templates.py:2023
msgid "successfully suppressed abuse report"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:2040
msgid "Not yet reviewed"
-msgstr ""
+msgstr "هنوز بررسی نشده است"
#: modules/webcomment/lib/webcomment_templates.py:2108
#, python-format
msgid ""
"The following review was sent to %(CFG_SITE_NAME)s by %(user_nickname)s:"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:2109
#, python-format
msgid ""
"The following comment was sent to %(CFG_SITE_NAME)s by %(user_nickname)s:"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:2136
msgid "This is an automatic message, please don't reply to it."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:2138
#, python-format
msgid "To post another comment, go to <%(x_url)s> instead."
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:2143
#, python-format
msgid "To specifically reply to this comment, go to <%(x_url)s>"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:2148
#, python-format
msgid "To unsubscribe from this discussion, go to <%(x_url)s>"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:2152
#, python-format
msgid "For any question, please use <%(CFG_SITE_SUPPORT_EMAIL)s>"
msgstr ""
#: modules/webcomment/lib/webcomment_templates.py:2219
msgid "Your comment will be lost."
-msgstr ""
+msgstr "نظر شما گم خواهد شد."
#: modules/webcomment/lib/webcomment_webinterface.py:261
#: modules/webcomment/lib/webcomment_webinterface.py:493
msgid "Record Not Found"
-msgstr ""
+msgstr "رکورد یافت نشد"
#: modules/webcomment/lib/webcomment_webinterface.py:394
#, python-format
msgid ""
"The size of file \\\"%s\\\" (%s) is larger than maximum allowed file size "
"(%s). Select files again."
msgstr ""
#: modules/webcomment/lib/webcomment_webinterface.py:476
#: modules/websubmit/lib/websubmit_templates.py:2668
#: modules/websubmit/lib/websubmit_templates.py:2669
msgid "Add Comment"
-msgstr ""
+msgstr "افزودن نظر"
#: modules/webcomment/lib/webcomment_webinterface.py:734
#: modules/webcomment/lib/webcomment_webinterface.py:768
msgid "Page Not Found"
-msgstr ""
+msgstr "صفحه پیدا نشد"
#: modules/webcomment/lib/webcomment_webinterface.py:735
msgid "The requested comment could not be found"
msgstr ""
#: modules/webcomment/lib/webcomment_webinterface.py:769
msgid "The requested file could not be found"
-msgstr ""
+msgstr "فایل درخواستی پیدا نشد"
#: modules/webcomment/web/admin/webcommentadmin.py:45
#: modules/webcomment/web/admin/webcommentadmin.py:59
#: modules/webcomment/web/admin/webcommentadmin.py:83
#: modules/webcomment/web/admin/webcommentadmin.py:126
#: modules/webcomment/web/admin/webcommentadmin.py:164
#: modules/webcomment/web/admin/webcommentadmin.py:192
#: modules/webcomment/web/admin/webcommentadmin.py:228
#: modules/webcomment/web/admin/webcommentadmin.py:266
msgid "WebComment Admin"
msgstr ""
#: modules/webcomment/web/admin/webcommentadmin.py:50
#: modules/webcomment/web/admin/webcommentadmin.py:88
#: modules/webcomment/web/admin/webcommentadmin.py:131
#: modules/webcomment/web/admin/webcommentadmin.py:197
#: modules/webcomment/web/admin/webcommentadmin.py:233
#: modules/webcomment/web/admin/webcommentadmin.py:271
#: modules/websearch/lib/websearch_webinterface.py:1563
#: modules/websearch/web/admin/websearchadmin.py:1040
#: modules/websession/lib/websession_webinterface.py:937
#: modules/webstyle/lib/webstyle_templates.py:585
#: modules/webjournal/web/admin/webjournaladmin.py:390
#: modules/bibcheck/web/admin/bibcheckadmin.py:331
msgid "Internal Error"
-msgstr ""
+msgstr "خطای درونی"
#: modules/webcomment/web/admin/webcommentadmin.py:102
msgid "Delete/Undelete Reviews"
-msgstr ""
+msgstr "حذف/ عدم حذف بررسی ها"
#: modules/webcomment/web/admin/webcommentadmin.py:102
msgid "Delete/Undelete Comments"
-msgstr ""
+msgstr "حذف/ عدم حذف نظرات"
#: modules/webcomment/web/admin/webcommentadmin.py:102
msgid " or Suppress abuse reports"
msgstr ""
#: modules/webcomment/web/admin/webcommentadmin.py:242
msgid "View all reported users"
-msgstr ""
+msgstr "نمایش تمام کاربران گزارش شده"
#: modules/webcomment/web/admin/webcommentadmin.py:289
msgid "Delete comments"
-msgstr ""
+msgstr "حذف نظرها"
#: modules/webcomment/web/admin/webcommentadmin.py:292
msgid "Suppress abuse reports"
msgstr ""
#: modules/webcomment/web/admin/webcommentadmin.py:295
msgid "Undelete comments"
-msgstr ""
+msgstr "بازگرداندن نظرها"
#: modules/webmessage/lib/webmessage.py:58
#: modules/webmessage/lib/webmessage.py:137
#: modules/webmessage/lib/webmessage.py:203
msgid "Sorry, this message in not in your mailbox."
msgstr ""
#: modules/webmessage/lib/webmessage.py:75
#: modules/webmessage/lib/webmessage.py:219
msgid "This message does not exist."
-msgstr ""
+msgstr "این پیام وجود ندارد."
#: modules/webmessage/lib/webmessage.py:144
msgid "The message could not be deleted."
-msgstr ""
+msgstr "این پیام نمی تواند حذف شود."
#: modules/webmessage/lib/webmessage.py:146
msgid "The message was successfully deleted."
-msgstr ""
+msgstr "پیام به طور موفقیت آمیزی حذف شد."
#: modules/webmessage/lib/webmessage.py:162
msgid "Your mailbox has been emptied."
-msgstr ""
+msgstr "صندوق پستی شما خالی دشه است."
#: modules/webmessage/lib/webmessage.py:368
#, python-format
msgid "The chosen date (%(x_year)i/%(x_month)i/%(x_day)i) is invalid."
msgstr ""
#: modules/webmessage/lib/webmessage.py:377
msgid "Please enter a user name or a group name."
-msgstr ""
+msgstr "لطفا نام کاربری یا نام یک گروه را وارد کنید"
#: modules/webmessage/lib/webmessage.py:381
#, python-format
msgid ""
"Your message is too long, please edit it. Maximum size allowed is %i "
"characters."
msgstr ""
#: modules/webmessage/lib/webmessage.py:396
#, python-format
msgid "Group %s does not exist."
msgstr ""
#: modules/webmessage/lib/webmessage.py:421
#, python-format
msgid "User %s does not exist."
msgstr ""
#: modules/webmessage/lib/webmessage.py:434
#: modules/webmessage/lib/webmessage_webinterface.py:145
#: modules/webmessage/lib/webmessage_webinterface.py:242
msgid "Write a message"
-msgstr ""
+msgstr "نوشتن یک پیام"
#: modules/webmessage/lib/webmessage.py:449
msgid ""
"Your message could not be sent to the following recipients due to their "
"quota:"
msgstr ""
#: modules/webmessage/lib/webmessage.py:453
msgid "Your message has been sent."
-msgstr ""
+msgstr "پیام شما ارسال شده است."
#: modules/webmessage/lib/webmessage.py:458
#: modules/webmessage/lib/webmessage_templates.py:472
#: modules/webmessage/lib/webmessage_webinterface.py:87
#: modules/webmessage/lib/webmessage_webinterface.py:311
#: modules/webmessage/lib/webmessage_webinterface.py:357
#: modules/websession/lib/websession_templates.py:607
msgid "Your Messages"
-msgstr ""
+msgstr "پیام های شما"
#: modules/webmessage/lib/webmessage_templates.py:86
#: modules/bibcirculation/lib/bibcirculation_templates.py:5322
msgid "Subject"
-msgstr ""
+msgstr "موضوع"
#: modules/webmessage/lib/webmessage_templates.py:87
msgid "Sender"
-msgstr ""
+msgstr "ارسال کننده"
#: modules/webmessage/lib/webmessage_templates.py:96
msgid "No messages"
-msgstr ""
+msgstr "بدون پیام"
#: modules/webmessage/lib/webmessage_templates.py:100
msgid "No subject"
-msgstr ""
+msgstr "بدون موضوع"
#: modules/webmessage/lib/webmessage_templates.py:146
msgid "Write new message"
-msgstr ""
+msgstr "نوشتن پیام جدید"
#: modules/webmessage/lib/webmessage_templates.py:147
msgid "Delete All"
-msgstr ""
+msgstr "حذف همه"
#: modules/webmessage/lib/webmessage_templates.py:189
msgid "Re:"
msgstr ""
#: modules/webmessage/lib/webmessage_templates.py:281
msgid "Send later?"
-msgstr ""
+msgstr "بعد ارسال شود؟"
#: modules/webmessage/lib/webmessage_templates.py:282
#: modules/websubmit/lib/websubmit_templates.py:3080
msgid "To:"
-msgstr ""
+msgstr "به:"
#: modules/webmessage/lib/webmessage_templates.py:283
msgid "Users"
-msgstr ""
+msgstr "کاربران"
#: modules/webmessage/lib/webmessage_templates.py:284
msgid "Groups"
-msgstr ""
+msgstr "گروه ها"
#: modules/webmessage/lib/webmessage_templates.py:285
#: modules/webmessage/lib/webmessage_templates.py:447
#: modules/websubmit/lib/websubmit_templates.py:3081
msgid "Subject:"
-msgstr ""
+msgstr "موضوع:"
#: modules/webmessage/lib/webmessage_templates.py:286
#: modules/websubmit/lib/websubmit_templates.py:3082
msgid "Message:"
-msgstr ""
+msgstr "پیام:"
#: modules/webmessage/lib/webmessage_templates.py:287
#: modules/websubmit/lib/websubmit_templates.py:3083
msgid "SEND"
-msgstr ""
+msgstr "ارسال"
#: modules/webmessage/lib/webmessage_templates.py:446
msgid "From:"
-msgstr ""
+msgstr "از:"
#: modules/webmessage/lib/webmessage_templates.py:448
msgid "Sent on:"
msgstr ""
#: modules/webmessage/lib/webmessage_templates.py:449
msgid "Received on:"
msgstr ""
#: modules/webmessage/lib/webmessage_templates.py:450
msgid "Sent to:"
-msgstr ""
+msgstr "ارسال به:"
#: modules/webmessage/lib/webmessage_templates.py:451
msgid "Sent to groups:"
-msgstr ""
+msgstr "ارسال به گروه ها:"
#: modules/webmessage/lib/webmessage_templates.py:452
msgid "REPLY"
-msgstr ""
+msgstr "پاسخ دادن"
#: modules/webmessage/lib/webmessage_templates.py:453
msgid "DELETE"
-msgstr ""
+msgstr "حذف کردن"
#: modules/webmessage/lib/webmessage_templates.py:506
msgid "Are you sure you want to empty your whole mailbox?"
-msgstr ""
+msgstr "آیا مطمئن هستید که می خواهید کل صندوق پستی خود را خالی کنید؟"
#: modules/webmessage/lib/webmessage_templates.py:582
#, python-format
msgid "Quota used: %(x_nb_used)i messages out of max. %(x_nb_total)i"
msgstr ""
#: modules/webmessage/lib/webmessage_templates.py:600
msgid "Please select one or more:"
-msgstr ""
+msgstr "لطفا یکی یا بیشتر را انتخاب کنید:"
#: modules/webmessage/lib/webmessage_templates.py:631
msgid "Add to users"
-msgstr ""
+msgstr "افزودن به کاربران"
#: modules/webmessage/lib/webmessage_templates.py:633
msgid "Add to groups"
-msgstr ""
+msgstr "افزودن به گروه ها "
#: modules/webmessage/lib/webmessage_templates.py:636
msgid "No matching user"
msgstr ""
#: modules/webmessage/lib/webmessage_templates.py:638
#: modules/websession/lib/websession_templates.py:1819
msgid "No matching group"
msgstr ""
#: modules/webmessage/lib/webmessage_templates.py:675
msgid "Find users or groups:"
-msgstr ""
+msgstr "یافتن کاربران یا گروه ها:"
#: modules/webmessage/lib/webmessage_templates.py:676
msgid "Find a user"
-msgstr ""
+msgstr "یافتن یک کاربر"
#: modules/webmessage/lib/webmessage_templates.py:677
msgid "Find a group"
-msgstr ""
+msgstr "یافتن یک گروه"
#: modules/webmessage/lib/webmessage_templates.py:692
#, python-format
msgid "You have %(x_nb_new)s new messages out of %(x_nb_total)s messages"
msgstr ""
#: modules/webmessage/lib/webmessage_webinterface.py:82
#: modules/webmessage/lib/webmessage_webinterface.py:134
#: modules/webmessage/lib/webmessage_webinterface.py:228
#: modules/webmessage/lib/webmessage_webinterface.py:305
#: modules/webmessage/lib/webmessage_webinterface.py:351
#: modules/webmessage/lib/webmessage_webinterface.py:397
msgid "You are not authorized to use messages."
msgstr ""
#: modules/webmessage/lib/webmessage_webinterface.py:403
msgid "Read a message"
-msgstr ""
+msgstr "خواندن یک پیام"
#: modules/websearch/lib/search_engine.py:833
#: modules/websearch/lib/search_engine.py:860
#: modules/websearch/lib/search_engine.py:4715
#: modules/websearch/lib/search_engine.py:4768
msgid "Search Results"
-msgstr ""
+msgstr "نتایج جستجو"
#: modules/websearch/lib/search_engine.py:973
#: modules/websearch/lib/websearch_templates.py:1174
msgid "any day"
-msgstr ""
+msgstr "هر روزی"
#: modules/websearch/lib/search_engine.py:979
#: modules/websearch/lib/websearch_templates.py:1186
msgid "any month"
-msgstr ""
+msgstr "هر ماهی"
#: modules/websearch/lib/search_engine.py:987
#: modules/websearch/lib/websearch_templates.py:1200
msgid "any year"
-msgstr ""
+msgstr "هر سالی"
#: modules/websearch/lib/search_engine.py:1028
#: modules/websearch/lib/search_engine.py:1047
msgid "any public collection"
-msgstr ""
+msgstr "هر مجموعه عمومی"
#: modules/websearch/lib/search_engine.py:1032
msgid "remove this collection"
-msgstr ""
+msgstr "پاک کردن این مجموعه"
#: modules/websearch/lib/search_engine.py:1043
msgid "add another collection"
-msgstr ""
+msgstr "افزودن مجوعه دیگر"
#: modules/websearch/lib/search_engine.py:1053
#: modules/websearch/lib/websearch_webcoll.py:592
msgid "rank by"
-msgstr ""
+msgstr "رتبه بندی به وسیله"
#: modules/websearch/lib/search_engine.py:1177
#: modules/websearch/lib/websearch_webcoll.py:562
msgid "latest first"
msgstr ""
#: modules/websearch/lib/search_engine.py:1827
msgid "No values found."
-msgstr ""
+msgstr "هیچ مقداری یافت نشد."
#: modules/websearch/lib/search_engine.py:1945
msgid ""
"Warning: full-text search is only available for a subset of papers mostly "
"from 2006-2011."
msgstr ""
#: modules/websearch/lib/search_engine.py:1947
msgid ""
"Warning: figure caption search is only available for a subset of papers "
"mostly from 2008-2011."
msgstr ""
#: modules/websearch/lib/search_engine.py:1953
#, python-format
msgid "There is no index %s. Searching for %s in all fields."
msgstr ""
#: modules/websearch/lib/search_engine.py:1957
#, python-format
msgid "Instead searching %s."
msgstr ""
#: modules/websearch/lib/search_engine.py:1963
msgid "Search term too generic, displaying only partial results..."
msgstr ""
#: modules/websearch/lib/search_engine.py:1966
msgid ""
"No phrase index available for fulltext yet, looking for word combination..."
msgstr ""
#: modules/websearch/lib/search_engine.py:2006
#, python-format
msgid "No exact match found for %(x_query1)s, using %(x_query2)s instead..."
msgstr ""
#: modules/websearch/lib/search_engine.py:2016
#: modules/websearch/lib/search_engine.py:2025
#: modules/websearch/lib/search_engine.py:4687
#: modules/websearch/lib/search_engine.py:4725
#: modules/websearch/lib/search_engine.py:4776
#: modules/websubmit/lib/websubmit_webinterface.py:112
#: modules/websubmit/lib/websubmit_webinterface.py:155
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:839
msgid "Requested record does not seem to exist."
-msgstr ""
+msgstr "رکورد درخواست شده بنظر نمی رسد وجود داشته باشد."
#: modules/websearch/lib/search_engine.py:2148
msgid ""
"Search syntax misunderstood. Ignoring all parentheses in the query. If this "
"doesn't help, please check your search and try again."
msgstr ""
#: modules/websearch/lib/search_engine.py:2573
#, python-format
msgid ""
"No match found in collection %(x_collection)s. Other public collections gave "
"%(x_url_open)s%(x_nb_hits)d hits%(x_url_close)s."
msgstr ""
#: modules/websearch/lib/search_engine.py:2582
msgid ""
"No public collection matched your query. If you were looking for a non-"
"public document, please choose the desired restricted collection first."
msgstr ""
#: modules/websearch/lib/search_engine.py:2696
msgid "No match found, please enter different search terms."
msgstr ""
#: modules/websearch/lib/search_engine.py:2702
#, python-format
msgid "There are no records referring to %s."
msgstr ""
#: modules/websearch/lib/search_engine.py:2704
#, python-format
msgid "There are no records cited by %s."
msgstr ""
#: modules/websearch/lib/search_engine.py:2709
#, python-format
msgid "No word index is available for %s."
msgstr ""
#: modules/websearch/lib/search_engine.py:2720
#, python-format
msgid "No phrase index is available for %s."
msgstr ""
#: modules/websearch/lib/search_engine.py:2767
#, python-format
msgid ""
"Search term %(x_term)s inside index %(x_index)s did not match any record. "
"Nearest terms in any collection are:"
msgstr ""
#: modules/websearch/lib/search_engine.py:2771
#, python-format
msgid ""
"Search term %s did not match any record. Nearest terms in any collection are:"
msgstr ""
#: modules/websearch/lib/search_engine.py:3486
#, python-format
msgid ""
"Sorry, sorting is allowed on sets of up to %d records only. Using default "
"sort order."
msgstr ""
#: modules/websearch/lib/search_engine.py:3510
#, python-format
msgid ""
"Sorry, %s does not seem to be a valid sort option. Choosing title sort "
"instead."
msgstr ""
#: modules/websearch/lib/search_engine.py:3703
#: modules/websearch/lib/search_engine.py:4012
#: modules/websearch/lib/search_engine.py:4191
#: modules/websearch/lib/search_engine.py:4214
#: modules/websearch/lib/search_engine.py:4222
#: modules/websearch/lib/search_engine.py:4230
#: modules/websearch/lib/search_engine.py:4276
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:837
msgid "The record has been deleted."
-msgstr ""
+msgstr "رکورد حذف شده است."
#: modules/websearch/lib/search_engine.py:3901
msgid "Use different search terms."
-msgstr ""
+msgstr "اصطلاحات جستجوی مختلفی را استفاده کنید."
#: modules/websearch/lib/search_engine.py:4997
msgid "No match within your time limits, discarding this condition..."
msgstr ""
#: modules/websearch/lib/search_engine.py:5024
msgid "No match within your search limits, discarding this condition..."
msgstr ""
#: modules/websearch/lib/websearchadminlib.py:3393
msgid "Information"
-msgstr ""
+msgstr "اطلاعات"
#: modules/websearch/lib/websearchadminlib.py:3394
msgid "References"
-msgstr ""
+msgstr "ارجاعات"
#: modules/websearch/lib/websearchadminlib.py:3395
msgid "Citations"
-msgstr ""
+msgstr "استنادها"
#: modules/websearch/lib/websearchadminlib.py:3396
msgid "Keywords"
-msgstr ""
+msgstr "کلیدواژه ها"
#: modules/websearch/lib/websearchadminlib.py:3397
msgid "Discussion"
-msgstr ""
+msgstr "بحث"
#: modules/websearch/lib/websearchadminlib.py:3398
msgid "Usage statistics"
-msgstr ""
+msgstr "آمارهای استفاده"
#: modules/websearch/lib/websearchadminlib.py:3399
msgid "Files"
msgstr ""
#: modules/websearch/lib/websearchadminlib.py:3400
msgid "Plots"
msgstr ""
#: modules/websearch/lib/websearchadminlib.py:3401
msgid "Holdings"
-msgstr ""
+msgstr "موجودی"
#: modules/websearch/lib/websearch_templates.py:458
#, python-format
msgid "Search on %(x_CFG_SITE_NAME_INTL)s"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:682
#: modules/websearch/lib/websearch_templates.py:831
#, python-format
msgid "Search %s records for:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:734
msgid "less"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:735
#: modules/websearch/lib/websearch_templates.py:1508
#: modules/websearch/lib/websearch_templates.py:3893
#: modules/websearch/lib/websearch_templates.py:3970
#: modules/websearch/lib/websearch_templates.py:4030
msgid "more"
-msgstr ""
+msgstr "بیشتر"
#: modules/websearch/lib/websearch_templates.py:740
#, python-format
msgid "Example: %(x_sample_search_query)s"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:752
#: modules/websearch/lib/websearch_templates.py:2129
#, python-format
msgid "Search in %(x_collection_name)s"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:756
#: modules/websearch/lib/websearch_templates.py:2133
msgid "Search everywhere"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:790
#: modules/websearch/lib/websearch_templates.py:867
#: modules/websearch/lib/websearch_templates.py:2101
#: modules/websearch/lib/websearch_templates.py:2159
msgid "Advanced Search"
-msgstr ""
+msgstr "جستجوی پیشرفته"
#: modules/websearch/lib/websearch_templates.py:928
#, python-format
msgid "Search %s records for"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:979
#: modules/websearch/lib/websearch_templates.py:2017
msgid "Simple Search"
-msgstr ""
+msgstr "جستجوی ساده"
#: modules/websearch/lib/websearch_templates.py:1012
msgid "Search options:"
-msgstr ""
+msgstr "گزینه های جستجو"
#: modules/websearch/lib/websearch_templates.py:1059
#: modules/websearch/lib/websearch_templates.py:2255
msgid "Added/modified since:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1060
#: modules/websearch/lib/websearch_templates.py:2256
msgid "until:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1065
#: modules/websearch/lib/websearch_templates.py:2298
msgid "Sort by:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1066
#: modules/websearch/lib/websearch_templates.py:2299
msgid "Display results:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1067
#: modules/websearch/lib/websearch_templates.py:2300
msgid "Output format:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1227
msgid "Added since:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1228
msgid "Modified since:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1265
msgid "Focus on:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1328
msgid "restricted"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1355
msgid "Search also:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1426
msgid ""
"This collection is restricted. If you are authorized to access it, please "
"click on the Search button."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1441
msgid ""
"This is a hosted external collection. Please click on the Search button to "
"see its content."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1456
msgid "This collection does not contain any document yet."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1523
msgid "Latest additions:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1626
#: modules/websearch/lib/websearch_templates.py:3361
#, python-format
msgid "Cited by %i records"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1692
#, python-format
msgid "Words nearest to %(x_word)s inside %(x_field)s in any collection are:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1695
#, python-format
msgid "Words nearest to %(x_word)s in any collection are:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1787
msgid "Hits"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:1866
#: modules/websearch/lib/websearch_templates.py:2518
#: modules/websearch/lib/websearch_templates.py:2708
#: modules/bibedit/lib/bibeditmulti_templates.py:657
msgid "next"
-msgstr ""
+msgstr "بعدی"
#: modules/websearch/lib/websearch_templates.py:2204
msgid "collections"
-msgstr ""
+msgstr "مجموعه ها"
#: modules/websearch/lib/websearch_templates.py:2226
msgid "Limit to:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:2268
#: modules/websearch/lib/websearch_webcoll.py:610
msgid "results"
-msgstr ""
+msgstr "نتایج"
#: modules/websearch/lib/websearch_templates.py:2304
#: modules/websearch/lib/websearch_webcoll.py:580
msgid "asc."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:2307
#: modules/websearch/lib/websearch_webcoll.py:581
msgid "desc."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:2313
#: modules/websearch/lib/websearch_webcoll.py:624
msgid "single list"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:2316
#: modules/websearch/lib/websearch_webcoll.py:623
msgid "split by collection"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:2354
msgid "MARC tag"
-msgstr ""
+msgstr "برچسب مارک"
#: modules/websearch/lib/websearch_templates.py:2469
#: modules/websearch/lib/websearch_templates.py:2474
#: modules/websearch/lib/websearch_templates.py:2652
#: modules/websearch/lib/websearch_templates.py:2664
#: modules/websearch/lib/websearch_templates.py:2985
#: modules/websearch/lib/websearch_templates.py:2994
#, python-format
msgid "%s records found"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:2501
#: modules/websearch/lib/websearch_templates.py:2691
#: modules/bibedit/lib/bibeditmulti_templates.py:655
msgid "begin"
-msgstr ""
+msgstr "شروع"
#: modules/websearch/lib/websearch_templates.py:2506
#: modules/websearch/lib/websearch_templates.py:2696
#: modules/websubmit/lib/websubmit_templates.py:1241
#: modules/bibedit/lib/bibeditmulti_templates.py:656
msgid "previous"
-msgstr ""
+msgstr "قبلی"
#: modules/websearch/lib/websearch_templates.py:2525
#: modules/websearch/lib/websearch_templates.py:2715
msgid "end"
-msgstr ""
+msgstr "پایان"
#: modules/websearch/lib/websearch_templates.py:2545
#: modules/websearch/lib/websearch_templates.py:2735
msgid "jump to record:"
-msgstr ""
+msgstr "پرش به رکورد"
#: modules/websearch/lib/websearch_templates.py:2558
#: modules/websearch/lib/websearch_templates.py:2748
#, python-format
msgid "Search took %s seconds."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:2952
#, python-format
msgid ""
"%(x_fmt_open)sResults overview:%(x_fmt_close)s Found %(x_nb_records)s "
"records in %(x_nb_seconds)s seconds."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:2964
#, python-format
msgid "%(x_fmt_open)sResults overview%(x_fmt_close)s"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:2972
#, python-format
msgid ""
"%(x_fmt_open)sResults overview:%(x_fmt_close)s Found at least "
"%(x_nb_records)s records in %(x_nb_seconds)s seconds."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3049
msgid "No results found..."
-msgstr ""
+msgstr "هیچ نتیجه ای یافت نشد ..."
#: modules/websearch/lib/websearch_templates.py:3082
msgid ""
"Boolean query returned no hits. Please combine your search terms differently."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3114
msgid "See also: similar author names"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3362
msgid "Cited by 1 record"
-msgstr ""
+msgstr "به وسیله 1 رکورد استناد شده"
#: modules/websearch/lib/websearch_templates.py:3377
#, python-format
msgid "%i comments"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3378
msgid "1 comment"
-msgstr ""
+msgstr "1 نظر"
#: modules/websearch/lib/websearch_templates.py:3388
#, python-format
msgid "%i reviews"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3389
msgid "1 review"
-msgstr ""
+msgstr "1 بررسی"
#: modules/websearch/lib/websearch_templates.py:3602
#: modules/websearch/lib/websearch_webinterface.py:1580
#, python-format
msgid "Collection %s Not Found"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3614
#: modules/websearch/lib/websearch_webinterface.py:1576
#, python-format
msgid "Sorry, collection %s does not seem to exist."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3616
#: modules/websearch/lib/websearch_webinterface.py:1577
#, python-format
msgid "You may want to start browsing from %s."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3643
#, python-format
msgid ""
"Set up a personal %(x_url1_open)semail alert%(x_url1_close)s\n"
" or subscribe to the %(x_url2_open)sRSS feed"
"%(x_url2_close)s."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3650
#, python-format
msgid "Subscribe to the %(x_url2_open)sRSS feed%(x_url2_close)s."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3659
msgid "Interested in being notified about new results for this query?"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3746
#: modules/websearch/lib/websearch_templates.py:3796
msgid "Back to search"
-msgstr ""
+msgstr "بازگشت به جستجو"
#: modules/websearch/lib/websearch_templates.py:3756
#: modules/websearch/lib/websearch_templates.py:3772
#: modules/websearch/lib/websearch_templates.py:3787
#, python-format
msgid "%s of"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:3886
msgid "People who downloaded this document also downloaded:"
msgstr ""
+"افرادی که این مدرک را دانلود کرده اند همچنین موارد زیر را دانلود کرده اند:"
#: modules/websearch/lib/websearch_templates.py:3902
msgid "People who viewed this page also viewed:"
-msgstr ""
+msgstr "افرادی که این صفحه را دیده اند همچنین صفحه های زیر را دیده اند:"
#: modules/websearch/lib/websearch_templates.py:3956
#, python-format
msgid "Cited by: %s records"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4023
#, python-format
msgid "Co-cited with: %s records"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4065
#, python-format
msgid ".. of which self-citations: %s records"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4157
msgid "Name variants"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4168
msgid "No Name Variants"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4176
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:763
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:920
msgid "Papers"
-msgstr ""
+msgstr "مقاله ها"
#: modules/websearch/lib/websearch_templates.py:4180
msgid "downloaded"
-msgstr ""
+msgstr "دانلود شده"
#: modules/websearch/lib/websearch_templates.py:4181
msgid "times"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4210
msgid "No Papers"
-msgstr ""
+msgstr "بدون مقاله ها"
#: modules/websearch/lib/websearch_templates.py:4223
msgid "unknown affiliation"
-msgstr ""
+msgstr "وابستگی نامشخص"
#: modules/websearch/lib/websearch_templates.py:4230
msgid "No Affiliations"
-msgstr ""
+msgstr "بدون وابستگی ها"
#: modules/websearch/lib/websearch_templates.py:4232
msgid "Affiliations"
-msgstr ""
+msgstr "وابستگی ها"
#: modules/websearch/lib/websearch_templates.py:4248
msgid "No Keywords"
-msgstr ""
+msgstr "بدون کلیدواژه ها"
#: modules/websearch/lib/websearch_templates.py:4251
msgid "Frequent keywords"
-msgstr ""
+msgstr "کلیدواژه های تکراری"
#: modules/websearch/lib/websearch_templates.py:4256
msgid "Frequent co-authors"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4267
msgid "No Frequent Co-authors"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4291
msgid "This is me. Verify my publication list."
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4330
msgid "Citations:"
-msgstr ""
+msgstr "استنادها"
#: modules/websearch/lib/websearch_templates.py:4334
msgid "No Citation Information available"
-msgstr ""
+msgstr "هیچ اطلاعات استنادی قابل استفاده نیست"
#: modules/websearch/lib/websearch_templates.py:4401
msgid "Citation summary results"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4406
msgid "Total number of citable papers analyzed:"
-msgstr ""
+msgstr "تعداد کل مقالات قابل استناد تحلیل شده:"
#: modules/websearch/lib/websearch_templates.py:4429
msgid "Total number of citations:"
-msgstr ""
+msgstr "تعداد کل استنادها:"
#: modules/websearch/lib/websearch_templates.py:4434
msgid "Average citations per paper:"
-msgstr ""
+msgstr "میانگین استنادهای هر مقاله:"
#: modules/websearch/lib/websearch_templates.py:4444
msgid "Total number of citations excluding self-citations:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4449
msgid "Average citations per paper excluding self-citations:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4458
msgid "Breakdown of papers by citations:"
msgstr ""
#: modules/websearch/lib/websearch_templates.py:4492
msgid "Additional Citation Metrics"
-msgstr ""
+msgstr "سنجه های استنادی اضافی"
#: modules/websearch/lib/websearch_webinterface.py:735
#: modules/websearch/lib/websearch_webinterface.py:747
#, python-format
msgid ""
"We're sorry. The requested author \"%s\" seems not to be listed on the "
"specified paper."
msgstr ""
#: modules/websearch/lib/websearch_webinterface.py:738
#: modules/websearch/lib/websearch_webinterface.py:750
msgid "Please try the following link to start a broader search on the author: "
msgstr ""
#: modules/websearch/lib/websearch_webinterface.py:1163
msgid "You are not authorized to view this area."
msgstr ""
#: modules/websearch/lib/websearch_webinterface.py:1582
msgid "Not found"
-msgstr ""
+msgstr "پیدا نشد"
#: modules/websearch/lib/websearch_external_collections.py:145
msgid "in"
-msgstr ""
+msgstr "در"
#: modules/websearch/lib/websearch_external_collections_templates.py:51
msgid ""
"Haven't found what you were looking for? Try your search on other servers:"
msgstr ""
#: modules/websearch/lib/websearch_external_collections_templates.py:79
msgid "External collections results overview:"
msgstr ""
#: modules/websearch/lib/websearch_external_collections_templates.py:119
msgid "Search timed out."
msgstr ""
#: modules/websearch/lib/websearch_external_collections_templates.py:120
msgid ""
"The external search engine has not responded in time. You can check its "
"results here:"
msgstr ""
#: modules/websearch/lib/websearch_external_collections_templates.py:146
#: modules/websearch/lib/websearch_external_collections_templates.py:154
#: modules/websearch/lib/websearch_external_collections_templates.py:167
msgid "No results found."
-msgstr ""
+msgstr "نتایج یافت نشد."
#: modules/websearch/lib/websearch_external_collections_templates.py:150
#, python-format
msgid "%s results found"
msgstr ""
#: modules/websearch/lib/websearch_external_collections_templates.py:152
#, python-format
msgid "%s seconds"
msgstr ""
#: modules/websearch/lib/websearch_webcoll.py:645
msgid "brief"
-msgstr ""
+msgstr "خلاصه"
#: modules/websession/lib/webaccount.py:116
#, python-format
msgid ""
"You are logged in as guest. You may want to %(x_url_open)slogin"
"%(x_url_close)s as a regular user."
msgstr ""
#: modules/websession/lib/webaccount.py:120
#, python-format
msgid ""
"The %(x_fmt_open)sguest%(x_fmt_close)s users need to %(x_url_open)sregister"
"%(x_url_close)s first"
msgstr ""
#: modules/websession/lib/webaccount.py:125
msgid "No queries found"
msgstr ""
#: modules/websession/lib/webaccount.py:367
msgid ""
"This collection is restricted. If you think you have right to access it, "
"please authenticate yourself."
msgstr ""
#: modules/websession/lib/webaccount.py:368
msgid ""
"This file is restricted. If you think you have right to access it, please "
"authenticate yourself."
msgstr ""
#: modules/websession/lib/websession_templates.py:93
msgid "External account settings"
msgstr ""
#: modules/websession/lib/websession_templates.py:95
#, python-format
msgid ""
"You can consult the list of your external groups directly in the "
"%(x_url_open)sgroups page%(x_url_close)s."
msgstr ""
#: modules/websession/lib/websession_templates.py:99
msgid "External user groups"
-msgstr ""
+msgstr "گروه های کاربری بیرونی"
#: modules/websession/lib/websession_templates.py:156
msgid ""
"If you want to change your email or set for the first time your nickname, "
"please set new values in the form below."
msgstr ""
#: modules/websession/lib/websession_templates.py:157
msgid "Edit login credentials"
msgstr ""
#: modules/websession/lib/websession_templates.py:162
msgid "New email address"
-msgstr ""
+msgstr "نشانی پست الکترونیکی جدید"
#: modules/websession/lib/websession_templates.py:163
#: modules/websession/lib/websession_templates.py:210
#: modules/websession/lib/websession_templates.py:1036
msgid "mandatory"
-msgstr ""
+msgstr "الزامی"
#: modules/websession/lib/websession_templates.py:166
msgid "Set new values"
-msgstr ""
+msgstr "مقادیر جدید را تنظیم کنید"
#: modules/websession/lib/websession_templates.py:170
msgid ""
"Since this is considered as a signature for comments and reviews, once set "
"it can not be changed."
msgstr ""
#: modules/websession/lib/websession_templates.py:209
msgid ""
"If you want to change your password, please enter the old one and set the "
"new value in the form below."
msgstr ""
#: modules/websession/lib/websession_templates.py:211
msgid "Old password"
-msgstr ""
+msgstr "کلمه عبور قدیم"
#: modules/websession/lib/websession_templates.py:212
msgid "New password"
-msgstr ""
+msgstr "کلمه عبور جدید"
#: modules/websession/lib/websession_templates.py:213
#: modules/websession/lib/websession_templates.py:1037
msgid "optional"
-msgstr ""
+msgstr "انتخابی"
#: modules/websession/lib/websession_templates.py:215
#: modules/websession/lib/websession_templates.py:1040
msgid "The password phrase may contain punctuation, spaces, etc."
msgstr ""
#: modules/websession/lib/websession_templates.py:216
msgid "You must fill the old password in order to set a new one."
msgstr ""
#: modules/websession/lib/websession_templates.py:217
msgid "Retype password"
-msgstr ""
+msgstr "کلمه عبور را دوباره تایپ کنید"
#: modules/websession/lib/websession_templates.py:218
msgid "Set new password"
-msgstr ""
+msgstr "کلمه عبور جدید را بگذارید"
#: modules/websession/lib/websession_templates.py:223
#, python-format
msgid ""
"If you are using a lightweight CERN account you can\n"
" %(x_url_open)sreset the password%(x_url_close)s."
msgstr ""
#: modules/websession/lib/websession_templates.py:229
#, python-format
msgid ""
"You can change or reset your CERN account password by means of the "
"%(x_url_open)sCERN account system%(x_url_close)s."
msgstr ""
#: modules/websession/lib/websession_templates.py:253
msgid "Edit cataloging interface settings"
msgstr ""
#: modules/websession/lib/websession_templates.py:254
#: modules/websession/lib/websession_templates.py:900
msgid "Username"
-msgstr ""
+msgstr "نام کاربری"
#: modules/websession/lib/websession_templates.py:255
#: modules/websession/lib/websession_templates.py:901
#: modules/websession/lib/websession_templates.py:1035
msgid "Password"
-msgstr ""
+msgstr "کلمه عبور"
#: modules/websession/lib/websession_templates.py:256
#: modules/websession/lib/websession_templates.py:282
#: modules/websession/lib/websession_templates.py:316
msgid "Update settings"
-msgstr ""
+msgstr "بروزرسانی تنظیمات"
#: modules/websession/lib/websession_templates.py:270
msgid "Edit language-related settings"
msgstr ""
#: modules/websession/lib/websession_templates.py:281
msgid "Select desired language of the web interface."
msgstr ""
#: modules/websession/lib/websession_templates.py:299
msgid "Edit search-related settings"
msgstr ""
#: modules/websession/lib/websession_templates.py:300
msgid "Show the latest additions box"
msgstr ""
#: modules/websession/lib/websession_templates.py:302
msgid "Show collection help boxes"
msgstr ""
#: modules/websession/lib/websession_templates.py:317
msgid "Number of search results per page"
-msgstr ""
+msgstr "تعداد نتایج جستجو در هر صفحه"
#: modules/websession/lib/websession_templates.py:347
msgid "Edit login method"
msgstr ""
#: modules/websession/lib/websession_templates.py:348
msgid ""
"Please select which login method you would like to use to authenticate "
"yourself"
msgstr ""
#: modules/websession/lib/websession_templates.py:349
#: modules/websession/lib/websession_templates.py:363
msgid "Select method"
msgstr ""
#: modules/websession/lib/websession_templates.py:381
#, python-format
msgid ""
"If you have lost the password for your %(sitename)s %(x_fmt_open)sinternal "
"account%(x_fmt_close)s, then please enter your email address in the "
"following form in order to have a password reset link emailed to you."
msgstr ""
#: modules/websession/lib/websession_templates.py:403
#: modules/websession/lib/websession_templates.py:1033
msgid "Email address"
-msgstr ""
+msgstr "نشانی پست الکترونیکی"
#: modules/websession/lib/websession_templates.py:404
msgid "Send password reset link"
msgstr ""
#: modules/websession/lib/websession_templates.py:408
#, python-format
msgid ""
"If you have been using the %(x_fmt_open)sCERN login system%(x_fmt_close)s, "
"then you can recover your password through the %(x_url_open)sCERN "
"authentication system%(x_url_close)s."
msgstr ""
#: modules/websession/lib/websession_templates.py:411
msgid ""
"Note that if you have been using an external login system, then we cannot do "
"anything and you have to ask there."
msgstr ""
#: modules/websession/lib/websession_templates.py:412
#, python-format
msgid ""
"Alternatively, you can ask %s to change your login system from external to "
"internal."
msgstr ""
#: modules/websession/lib/websession_templates.py:439
#, python-format
msgid ""
"%s offers you the possibility to personalize the interface, to set up your "
"own personal library of documents, or to set up an automatic alert query "
"that would run periodically and would notify you of search results by email."
msgstr ""
#: modules/websession/lib/websession_templates.py:449
#: modules/websession/lib/websession_webinterface.py:276
msgid "Your Settings"
-msgstr ""
+msgstr "تنظیمات شما"
#: modules/websession/lib/websession_templates.py:450
msgid ""
"Set or change your account email address or password. Specify your "
"preferences about the look and feel of the interface."
msgstr ""
#: modules/websession/lib/websession_templates.py:458
msgid "View all the searches you performed during the last 30 days."
msgstr ""
#: modules/websession/lib/websession_templates.py:466
msgid ""
"With baskets you can define specific collections of items, store interesting "
"records you want to access later or share with others."
msgstr ""
#: modules/websession/lib/websession_templates.py:475
msgid ""
"Subscribe to a search which will be run periodically by our service. The "
"result can be sent to you via Email or stored in one of your baskets."
msgstr ""
#: modules/websession/lib/websession_templates.py:484
#: modules/websession/lib/websession_templates.py:610
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:126
msgid "Your Loans"
-msgstr ""
+msgstr "امانت های شما"
#: modules/websession/lib/websession_templates.py:485
msgid ""
"Check out book you have on loan, submit borrowing requests, etc. Requires "
"CERN ID."
msgstr ""
#: modules/websession/lib/websession_templates.py:512
msgid ""
"You are logged in as a guest user, so your alerts will disappear at the end "
"of the current session."
msgstr ""
#: modules/websession/lib/websession_templates.py:535
#, python-format
msgid ""
"You are logged in as %(x_user)s. You may want to a) %(x_url1_open)slogout"
"%(x_url1_close)s; b) edit your %(x_url2_open)saccount settings"
"%(x_url2_close)s."
msgstr ""
#: modules/websession/lib/websession_templates.py:616
msgid "Your Alert Searches"
msgstr ""
#: modules/websession/lib/websession_templates.py:622
#, python-format
msgid ""
"You can consult the list of %(x_url_open)syour groups%(x_url_close)s you are "
"administering or are a member of."
msgstr ""
#: modules/websession/lib/websession_templates.py:625
#: modules/websession/lib/websession_templates.py:2326
#: modules/websession/lib/websession_webinterface.py:1020
msgid "Your Groups"
-msgstr ""
+msgstr "گروه های شما"
#: modules/websession/lib/websession_templates.py:628
#, python-format
msgid ""
"You can consult the list of %(x_url_open)syour submissions%(x_url_close)s "
"and inquire about their status."
msgstr ""
#: modules/websession/lib/websession_templates.py:631
#: modules/websubmit/web/yoursubmissions.py:160
msgid "Your Submissions"
-msgstr ""
+msgstr "واگذاری های شما"
#: modules/websession/lib/websession_templates.py:634
#, python-format
msgid ""
"You can consult the list of %(x_url_open)syour approvals%(x_url_close)s with "
"the documents you approved or refereed."
msgstr ""
#: modules/websession/lib/websession_templates.py:637
#: modules/websubmit/web/yourapprovals.py:88
msgid "Your Approvals"
msgstr ""
#: modules/websession/lib/websession_templates.py:641
#, python-format
msgid "You can consult the list of %(x_url_open)syour tickets%(x_url_close)s."
msgstr ""
#: modules/websession/lib/websession_templates.py:644
msgid "Your Tickets"
msgstr ""
#: modules/websession/lib/websession_templates.py:646
#: modules/websession/lib/websession_webinterface.py:625
msgid "Your Administrative Activities"
msgstr ""
#: modules/websession/lib/websession_templates.py:673
#: modules/bibharvest/lib/oai_harvest_admin.py:470
#: modules/bibharvest/lib/oai_harvest_admin.py:485
msgid "Try again"
-msgstr ""
+msgstr "مجدد تلاش کنید"
#: modules/websession/lib/websession_templates.py:695
#, python-format
msgid ""
"Somebody (possibly you) coming from %(x_ip_address)s has asked\n"
"for a password reset at %(x_sitename)s\n"
"for the account \"%(x_email)s\"."
msgstr ""
#: modules/websession/lib/websession_templates.py:703
msgid "If you want to reset the password for this account, please go to:"
msgstr ""
#: modules/websession/lib/websession_templates.py:709
#: modules/websession/lib/websession_templates.py:746
msgid "in order to confirm the validity of this request."
msgstr ""
#: modules/websession/lib/websession_templates.py:710
#: modules/websession/lib/websession_templates.py:747
#, python-format
msgid ""
"Please note that this URL will remain valid for about %(days)s days only."
msgstr ""
#: modules/websession/lib/websession_templates.py:732
#, python-format
msgid ""
"Somebody (possibly you) coming from %(x_ip_address)s has asked\n"
"to register a new account at %(x_sitename)s\n"
"for the email address \"%(x_email)s\"."
msgstr ""
#: modules/websession/lib/websession_templates.py:740
msgid "If you want to complete this account registration, please go to:"
msgstr ""
#: modules/websession/lib/websession_templates.py:766
#, python-format
msgid "Okay, a password reset link has been emailed to %s."
msgstr ""
#: modules/websession/lib/websession_templates.py:781
msgid "Deleting your account"
-msgstr ""
+msgstr "حذف حساب شما"
#: modules/websession/lib/websession_templates.py:795
msgid "You are no longer recognized by our system."
msgstr ""
#: modules/websession/lib/websession_templates.py:797
#, python-format
msgid ""
"You are still recognized by the centralized\n"
" %(x_fmt_open)sSSO%(x_fmt_close)s system. You can\n"
" %(x_url_open)slogout from SSO%(x_url_close)s, too."
msgstr ""
#: modules/websession/lib/websession_templates.py:804
#, python-format
msgid "If you wish you can %(x_url_open)slogin here%(x_url_close)s."
msgstr ""
#: modules/websession/lib/websession_templates.py:835
msgid "If you already have an account, please login using the form below."
msgstr ""
#: modules/websession/lib/websession_templates.py:839
#, python-format
msgid ""
"If you don't own a CERN account yet, you can register a %(x_url_open)snew "
"CERN lightweight account%(x_url_close)s."
msgstr ""
#: modules/websession/lib/websession_templates.py:842
#, python-format
msgid ""
"If you don't own an account yet, please %(x_url_open)sregister"
"%(x_url_close)s an internal account."
msgstr ""
#: modules/websession/lib/websession_templates.py:850
#, python-format
msgid "If you don't own an account yet, please contact %s."
msgstr ""
#: modules/websession/lib/websession_templates.py:873
msgid "Login method:"
-msgstr ""
+msgstr "روش ورود:"
#: modules/websession/lib/websession_templates.py:902
msgid "Remember login on this computer."
msgstr ""
#: modules/websession/lib/websession_templates.py:903
#: modules/websession/lib/websession_templates.py:1182
#: modules/websession/lib/websession_webinterface.py:103
#: modules/websession/lib/websession_webinterface.py:198
#: modules/websession/lib/websession_webinterface.py:748
#: modules/websession/lib/websession_webinterface.py:839
msgid "login"
-msgstr ""
+msgstr "ورود"
#: modules/websession/lib/websession_templates.py:908
#: modules/websession/lib/websession_webinterface.py:529
msgid "Lost your password?"
-msgstr ""
+msgstr "کلمه عبورتان را از دست داده اید؟"
#: modules/websession/lib/websession_templates.py:916
msgid "You can use your nickname or your email address to login."
msgstr ""
#: modules/websession/lib/websession_templates.py:940
msgid ""
"Your request is valid. Please set the new desired password in the following "
"form."
msgstr ""
#: modules/websession/lib/websession_templates.py:963
msgid "Set a new password for"
-msgstr ""
+msgstr "تنظیم یک کلمه عبور جدید برای"
#: modules/websession/lib/websession_templates.py:964
msgid "Type the new password"
-msgstr ""
+msgstr "کلمه عبور جدید را تایپ کنید"
#: modules/websession/lib/websession_templates.py:965
msgid "Type again the new password"
-msgstr ""
+msgstr "کلمه عبور جدید را دوباره تایپ کنید"
#: modules/websession/lib/websession_templates.py:966
msgid "Set the new password"
-msgstr ""
+msgstr "تنظیم کلمه عبور جدید"
#: modules/websession/lib/websession_templates.py:988
msgid "Please enter your email address and desired nickname and password:"
msgstr ""
#: modules/websession/lib/websession_templates.py:990
msgid ""
"It will not be possible to use the account before it has been verified and "
"activated."
msgstr ""
#: modules/websession/lib/websession_templates.py:1041
msgid "Retype Password"
-msgstr ""
+msgstr "تایپ دوباره کلمه عبور"
#: modules/websession/lib/websession_templates.py:1042
#: modules/websession/lib/websession_webinterface.py:942
msgid "register"
-msgstr ""
+msgstr "ثبت"
#: modules/websession/lib/websession_templates.py:1043
#, python-format
msgid ""
"Please do not use valuable passwords such as your Unix, AFS or NICE "
"passwords with this service. Your email address will stay strictly "
"confidential and will not be disclosed to any third party. It will be used "
"to identify you for personal services of %s. For example, you may set up an "
"automatic alert search that will look for new preprints and will notify you "
"daily of new arrivals by email."
msgstr ""
#: modules/websession/lib/websession_templates.py:1047
#, python-format
msgid ""
"It is not possible to create an account yourself. Contact %s if you want an "
"account."
msgstr ""
#: modules/websession/lib/websession_templates.py:1073
#, python-format
msgid ""
"You seem to be a guest user. You have to %(x_url_open)slogin%(x_url_close)s "
"first."
msgstr ""
#: modules/websession/lib/websession_templates.py:1079
msgid "You are not authorized to access administrative functions."
msgstr ""
#: modules/websession/lib/websession_templates.py:1082
#, python-format
msgid "You are enabled to the following roles: %(x_role)s."
msgstr ""
#: modules/websession/lib/websession_templates.py:1098
msgid "Run BibSword Client"
msgstr ""
#: modules/websession/lib/websession_templates.py:1125
msgid "Here are some interesting web admin links for you:"
msgstr ""
#: modules/websession/lib/websession_templates.py:1127
#, python-format
msgid ""
"For more admin-level activities, see the complete %(x_url_open)sAdmin Area"
"%(x_url_close)s."
msgstr ""
#: modules/websession/lib/websession_templates.py:1180
msgid "guest"
-msgstr ""
+msgstr "مهمان"
#: modules/websession/lib/websession_templates.py:1194
msgid "logout"
-msgstr ""
+msgstr "خروج"
#: modules/websession/lib/websession_templates.py:1242
#: modules/webstyle/lib/webstyle_templates.py:435
#: modules/webstyle/lib/webstyle_templates.py:504
msgid "Personalize"
-msgstr ""
+msgstr "شخص کردن"
#: modules/websession/lib/websession_templates.py:1250
msgid "Your account"
-msgstr ""
+msgstr "حساب شما"
#: modules/websession/lib/websession_templates.py:1256
msgid "Your alerts"
-msgstr ""
+msgstr "هشدارهای شما"
#: modules/websession/lib/websession_templates.py:1262
msgid "Your approvals"
-msgstr ""
+msgstr "تأییدهای شما"
#: modules/websession/lib/websession_templates.py:1268
msgid "Your baskets"
-msgstr ""
+msgstr "سبدهای شما"
#: modules/websession/lib/websession_templates.py:1274
msgid "Your groups"
-msgstr ""
+msgstr "گروه های شما"
#: modules/websession/lib/websession_templates.py:1280
msgid "Your loans"
-msgstr ""
+msgstr "امانت های شما"
#: modules/websession/lib/websession_templates.py:1286
msgid "Your messages"
-msgstr ""
+msgstr "پیام های شما"
#: modules/websession/lib/websession_templates.py:1292
msgid "Your submissions"
-msgstr ""
+msgstr "واگذاری های شما"
#: modules/websession/lib/websession_templates.py:1298
msgid "Your searches"
-msgstr ""
+msgstr "جستجوهای شما"
#: modules/websession/lib/websession_templates.py:1351
msgid "Administration"
-msgstr ""
+msgstr "مدیریت"
#: modules/websession/lib/websession_templates.py:1367
msgid "Statistics"
-msgstr ""
+msgstr "آمارها"
#: modules/websession/lib/websession_templates.py:1484
msgid "You are an administrator of the following groups:"
-msgstr ""
+msgstr "شما اداره کننده گروه های ذیل هستید:"
#: modules/websession/lib/websession_templates.py:1504
#: modules/websession/lib/websession_templates.py:1578
#: modules/websession/lib/websession_templates.py:1641
#: modules/websubmit/lib/websubmit_templates.py:3012
#: modules/websubmit/lib/websubmit_templates.py:3018
msgid "Group"
-msgstr ""
+msgstr "گروه"
#: modules/websession/lib/websession_templates.py:1511
msgid "You are not an administrator of any groups."
-msgstr ""
+msgstr "شما اداره کننده هیچ گروهی نیستید"
#: modules/websession/lib/websession_templates.py:1518
msgid "Edit group"
-msgstr ""
+msgstr "ویرایش گروه"
#: modules/websession/lib/websession_templates.py:1525
#, python-format
msgid "Edit %s members"
msgstr ""
#: modules/websession/lib/websession_templates.py:1548
#: modules/websession/lib/websession_templates.py:1688
#: modules/websession/lib/websession_templates.py:1690
#: modules/websession/lib/websession_webinterface.py:1076
msgid "Create new group"
-msgstr ""
+msgstr "ایجاد گروه جدید"
#: modules/websession/lib/websession_templates.py:1562
msgid "You are a member of the following groups:"
-msgstr ""
+msgstr "شما عضوی از گروه های ذیل هستید:"
#: modules/websession/lib/websession_templates.py:1585
msgid "You are not a member of any groups."
-msgstr ""
+msgstr "شما عضو هیچ گروهی نیستید."
#: modules/websession/lib/websession_templates.py:1609
msgid "Join new group"
-msgstr ""
+msgstr "به گروه جدیدی بپیوندید"
#: modules/websession/lib/websession_templates.py:1610
#: modules/websession/lib/websession_templates.py:2162
#: modules/websession/lib/websession_templates.py:2173
msgid "Leave group"
-msgstr ""
+msgstr "گروه را ترک کنید"
#: modules/websession/lib/websession_templates.py:1625
msgid "You are a member of the following external groups:"
-msgstr ""
+msgstr "شما عضوی از گروهای بیرونی ذیل هستید:"
#: modules/websession/lib/websession_templates.py:1648
msgid "You are not a member of any external groups."
-msgstr ""
+msgstr "شما عضو هیچ گروه بیرونی نیستید."
#: modules/websession/lib/websession_templates.py:1696
msgid "Update group"
-msgstr ""
+msgstr "بروز رسانی گروه"
#: modules/websession/lib/websession_templates.py:1698
#, python-format
msgid "Edit group %s"
msgstr ""
#: modules/websession/lib/websession_templates.py:1700
msgid "Delete group"
-msgstr ""
+msgstr "حذف گروه"
#: modules/websession/lib/websession_templates.py:1773
msgid "Group name:"
-msgstr ""
+msgstr "نام گروه:"
#: modules/websession/lib/websession_templates.py:1775
msgid "Group description:"
-msgstr ""
+msgstr "توصیف گروه:"
#: modules/websession/lib/websession_templates.py:1776
msgid "Group join policy:"
-msgstr ""
+msgstr "خط مشی پیوستن به گروه:"
#: modules/websession/lib/websession_templates.py:1817
#: modules/websession/lib/websession_templates.py:1890
#: modules/websession/lib/websession_templates.py:2031
#: modules/websession/lib/websession_templates.py:2040
#: modules/websession/lib/websession_templates.py:2160
#: modules/websession/lib/websession_templates.py:2272
msgid "Please select:"
-msgstr ""
+msgstr "لطفا انتخاب کنید:"
#: modules/websession/lib/websession_templates.py:1883
msgid "Join group"
-msgstr ""
+msgstr "به گروه بپیوندید"
#: modules/websession/lib/websession_templates.py:1885
msgid "or find it"
-msgstr ""
+msgstr "یا آن را بیابید"
#: modules/websession/lib/websession_templates.py:1886
msgid "Choose group:"
-msgstr ""
+msgstr "انتخاب گروه:"
#: modules/websession/lib/websession_templates.py:1888
msgid "Find group"
-msgstr ""
+msgstr "یافتن گروه"
#: modules/websession/lib/websession_templates.py:2036
msgid "Remove member"
-msgstr ""
+msgstr "پاک کردن عضو"
#: modules/websession/lib/websession_templates.py:2038
msgid "No members."
-msgstr ""
+msgstr "بدون اعضاء."
#: modules/websession/lib/websession_templates.py:2048
msgid "Accept member"
-msgstr ""
+msgstr "پذیرش عضو"
#: modules/websession/lib/websession_templates.py:2048
msgid "Reject member"
-msgstr ""
+msgstr "رد عضو"
#: modules/websession/lib/websession_templates.py:2050
msgid "No members awaiting approval."
-msgstr ""
+msgstr "بدون اعضاء منتظر تأیید."
#: modules/websession/lib/websession_templates.py:2052
#: modules/websession/lib/websession_templates.py:2086
msgid "Current members"
-msgstr ""
+msgstr "اعضاء جاری"
#: modules/websession/lib/websession_templates.py:2053
#: modules/websession/lib/websession_templates.py:2087
msgid "Members awaiting approval"
-msgstr ""
+msgstr "اعضاء منتظر تأیید"
#: modules/websession/lib/websession_templates.py:2054
#: modules/websession/lib/websession_templates.py:2088
msgid "Invite new members"
-msgstr ""
+msgstr "دعوت اعضاء جدید"
#: modules/websession/lib/websession_templates.py:2059
#, python-format
msgid "Invitation to join \"%s\" group"
msgstr ""
#: modules/websession/lib/websession_templates.py:2060
#, python-format
msgid ""
"Hello:\n"
"\n"
"I think you might be interested in joining the group \"%(x_name)s\".\n"
"You can join by clicking here: %(x_url)s.\n"
"\n"
"Best regards.\n"
msgstr ""
#: modules/websession/lib/websession_templates.py:2074
#, python-format
msgid ""
"If you want to invite new members to join your group, please use the "
"%(x_url_open)sweb message%(x_url_close)s system."
msgstr ""
#: modules/websession/lib/websession_templates.py:2078
#, python-format
msgid "Group: %s"
msgstr ""
#: modules/websession/lib/websession_templates.py:2161
msgid "Group list"
-msgstr ""
+msgstr "سیاهه گروه"
#: modules/websession/lib/websession_templates.py:2164
msgid "You are not member of any group."
-msgstr ""
+msgstr "شما عضو هیچ گروهی نیستید."
#: modules/websession/lib/websession_templates.py:2212
msgid "Are you sure you want to delete this group?"
-msgstr ""
+msgstr "آیا مطمئن هستند که می خواهید این گروه را حذف کنید؟"
#: modules/websession/lib/websession_templates.py:2252
msgid "Are you sure you want to leave this group?"
-msgstr ""
+msgstr "آیا مطمئن هستید که می خواهید این گروه را ترک کنید؟"
#: modules/websession/lib/websession_templates.py:2268
msgid "Visible and open for new members"
-msgstr ""
+msgstr "آشکار و آزاد برای اعضاء جدید"
#: modules/websession/lib/websession_templates.py:2270
msgid "Visible but new members need approval"
-msgstr ""
+msgstr "آشکار اما اعضاء جدید نیاز به تأیید دارند"
#: modules/websession/lib/websession_templates.py:2355
#, python-format
msgid "Group %s: New membership request"
msgstr ""
#: modules/websession/lib/websession_templates.py:2359
#, python-format
msgid "A user wants to join the group %s."
msgstr ""
#: modules/websession/lib/websession_templates.py:2360
#, python-format
msgid ""
"Please %(x_url_open)saccept or reject%(x_url_close)s this user's request."
msgstr ""
#: modules/websession/lib/websession_templates.py:2377
#, python-format
msgid "Group %s: Join request has been accepted"
msgstr ""
#: modules/websession/lib/websession_templates.py:2378
#, python-format
msgid "Your request for joining group %s has been accepted."
msgstr ""
#: modules/websession/lib/websession_templates.py:2380
#, python-format
msgid "Group %s: Join request has been rejected"
msgstr ""
#: modules/websession/lib/websession_templates.py:2381
#, python-format
msgid "Your request for joining group %s has been rejected."
msgstr ""
#: modules/websession/lib/websession_templates.py:2384
#: modules/websession/lib/websession_templates.py:2402
#, python-format
msgid "You can consult the list of %(x_url_open)syour groups%(x_url_close)s."
msgstr ""
#: modules/websession/lib/websession_templates.py:2398
#, python-format
msgid "Group %s has been deleted"
msgstr ""
#: modules/websession/lib/websession_templates.py:2400
#, python-format
msgid "Group %s has been deleted by its administrator."
msgstr ""
#: modules/websession/lib/websession_templates.py:2417
#, python-format
msgid ""
"You can consult the list of %(x_url_open)s%(x_nb_total)i groups"
"%(x_url_close)s you are subscribed to (%(x_nb_member)i) or administering "
"(%(x_nb_admin)i)."
msgstr ""
#: modules/websession/lib/websession_templates.py:2436
msgid ""
"Warning: The password set for MySQL root user is the same as the default "
"Invenio password. For security purposes, you may want to change the password."
msgstr ""
#: modules/websession/lib/websession_templates.py:2442
msgid ""
"Warning: The password set for the Invenio MySQL user is the same as the "
"shipped default. For security purposes, you may want to change the password."
msgstr ""
#: modules/websession/lib/websession_templates.py:2448
msgid ""
"Warning: The password set for the Invenio admin user is currently empty. For "
"security purposes, it is strongly recommended that you add a password."
msgstr ""
#: modules/websession/lib/websession_templates.py:2454
msgid ""
"Warning: The email address set for support email is currently set to "
"info@invenio-software.org. It is recommended that you change this to your "
"own address."
msgstr ""
#: modules/websession/lib/websession_templates.py:2460
msgid ""
"A newer version of Invenio is available for download. You may want to visit "
msgstr ""
+"نسخه جدیدتری از اینونیو برای دانلود در دسترس است. شما ممکن است خواهان دیدن "
+"آن باشید."
#: modules/websession/lib/websession_templates.py:2467
msgid ""
"Cannot download or parse release notes from http://invenio-software.org/repo/"
"invenio/tree/RELEASE-NOTES"
msgstr ""
#: modules/websession/lib/webuser.py:149
msgid "Database problem"
-msgstr ""
+msgstr "مشکل پایگاه داده"
#: modules/websession/lib/webuser.py:299
#: modules/websession/lib/webgroup_dblayer.py:314
msgid "user"
-msgstr ""
+msgstr "کاربر"
#: modules/websession/lib/webuser.py:470
#, python-format
msgid "Account registration at %s"
msgstr ""
#: modules/websession/lib/webuser.py:714
msgid "New account on"
msgstr ""
#: modules/websession/lib/webuser.py:716
msgid "PLEASE ACTIVATE"
-msgstr ""
+msgstr "لطفا فعال کنید"
#: modules/websession/lib/webuser.py:717
msgid "A new account has been created on"
msgstr ""
#: modules/websession/lib/webuser.py:719
msgid " and is awaiting activation"
-msgstr ""
+msgstr "و منتظر فعال سازی است"
#: modules/websession/lib/webuser.py:721
msgid " Username/Email"
-msgstr ""
+msgstr "نام کاربری/ پست الکترونیکی"
#: modules/websession/lib/webuser.py:722
msgid "You can approve or reject this account request at"
msgstr ""
#: modules/websession/lib/websession_webinterface.py:85
msgid "Mail Cookie Service"
msgstr ""
#: modules/websession/lib/websession_webinterface.py:95
msgid "Role authorization request"
msgstr ""
#: modules/websession/lib/websession_webinterface.py:95
msgid "This request for an authorization has already been authorized."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:98
#, python-format
msgid ""
"You have successfully obtained an authorization as %(x_role)s! This "
"authorization will last until %(x_expiration)s and until you close your "
"browser if you are a guest user."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:116
msgid "You have confirmed the validity of your email address!"
msgstr ""
#: modules/websession/lib/websession_webinterface.py:119
#: modules/websession/lib/websession_webinterface.py:129
msgid "Please, wait for the administrator to enable your account."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:123
#: modules/websession/lib/websession_webinterface.py:132
#, python-format
msgid "You can now go to %(x_url_open)syour account page%(x_url_close)s."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:124
#: modules/websession/lib/websession_webinterface.py:133
msgid "Email address successfully activated"
-msgstr ""
+msgstr "نشانی پست الکترونیکی به طور موفقیت آمیز فعال شد."
#: modules/websession/lib/websession_webinterface.py:127
msgid "You have already confirmed the validity of your email address!"
msgstr ""
#: modules/websession/lib/websession_webinterface.py:136
msgid ""
"This request for confirmation of an email address is not valid or is expired."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:141
msgid "This request for an authorization is not valid or is expired."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:154
msgid "Reset password"
-msgstr ""
+msgstr "بازتنظیم کلمه عبور"
#: modules/websession/lib/websession_webinterface.py:160
msgid "This request for resetting a password has already been used."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:163
msgid "This request for resetting a password is not valid or is expired."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:168
msgid "This request for resetting the password is not valid or is expired."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:181
msgid "The two provided passwords aren't equal."
-msgstr ""
+msgstr "دو کلمه عبور قرار داده شده، یکسان نیستند."
#: modules/websession/lib/websession_webinterface.py:196
msgid "The password was successfully set! You can now proceed with the login."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:281
#, python-format
msgid "%s Personalize, Your Settings"
msgstr ""
#: modules/websession/lib/websession_webinterface.py:335
#: modules/websession/lib/websession_webinterface.py:402
#: modules/websession/lib/websession_webinterface.py:464
#: modules/websession/lib/websession_webinterface.py:477
#: modules/websession/lib/websession_webinterface.py:490
msgid "Settings edited"
-msgstr ""
+msgstr "تنظیمات ویرایش شده"
#: modules/websession/lib/websession_webinterface.py:337
#: modules/websession/lib/websession_webinterface.py:401
#: modules/websession/lib/websession_webinterface.py:442
#: modules/websession/lib/websession_webinterface.py:466
#: modules/websession/lib/websession_webinterface.py:479
#: modules/websession/lib/websession_webinterface.py:485
msgid "Show account"
-msgstr ""
+msgstr "نمایش حساب"
#: modules/websession/lib/websession_webinterface.py:341
msgid "Unable to change login method."
-msgstr ""
+msgstr "نمی تواند روش ورود را تغییر دهد."
#: modules/websession/lib/websession_webinterface.py:349
msgid "Switched to internal login method."
-msgstr ""
+msgstr "به روش ورود درونی تعویض شد."
#: modules/websession/lib/websession_webinterface.py:350
msgid ""
"Please note that if this is the first time that you are using this account "
"with the internal login method then the system has set for you a randomly "
"generated password. Please click the following button to obtain a password "
"reset request link sent to you via email:"
msgstr ""
#: modules/websession/lib/websession_webinterface.py:358
msgid "Send Password"
-msgstr ""
+msgstr "ارسال کلمه عبور"
#: modules/websession/lib/websession_webinterface.py:366
#, python-format
msgid ""
"Unable to switch to external login method %s, because your email address is "
"unknown."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:370
#, python-format
msgid ""
"Unable to switch to external login method %s, because your email address is "
"unknown to the external login system."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:374
msgid "Login method successfully selected."
-msgstr ""
+msgstr "روش ورود به طور موفقیت آمیز انتخاب شد."
#: modules/websession/lib/websession_webinterface.py:376
#, python-format
msgid ""
"The external login method %s does not support email address based logins. "
"Please contact the site administrators."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:395
msgid "Settings successfully edited."
-msgstr ""
+msgstr "تنظیمات به طور موفقیت آمیز ویرایش شد."
#: modules/websession/lib/websession_webinterface.py:396
#, python-format
msgid ""
"Note that if you have changed your email address, you will have to "
"%(x_url_open)sreset your password%(x_url_close)s anew."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:404
#: modules/websession/lib/websession_webinterface.py:912
#, python-format
msgid "Desired nickname %s is invalid."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:405
#: modules/websession/lib/websession_webinterface.py:411
#: modules/websession/lib/websession_webinterface.py:424
#: modules/websession/lib/websession_webinterface.py:446
#: modules/websession/lib/websession_webinterface.py:452
#: modules/websession/lib/websession_webinterface.py:903
#: modules/websession/lib/websession_webinterface.py:908
#: modules/websession/lib/websession_webinterface.py:913
#: modules/websession/lib/websession_webinterface.py:924
msgid "Please try again."
-msgstr ""
+msgstr "لطفا دوباره تلاش کنید."
#: modules/websession/lib/websession_webinterface.py:407
#: modules/websession/lib/websession_webinterface.py:413
#: modules/websession/lib/websession_webinterface.py:420
#: modules/websession/lib/websession_webinterface.py:426
#: modules/websession/lib/websession_webinterface.py:448
#: modules/websession/lib/websession_webinterface.py:454
#: modules/websession/lib/websession_webinterface.py:501
msgid "Edit settings"
-msgstr ""
+msgstr "ویرایش تنظیمات"
#: modules/websession/lib/websession_webinterface.py:408
#: modules/websession/lib/websession_webinterface.py:414
#: modules/websession/lib/websession_webinterface.py:421
#: modules/websession/lib/websession_webinterface.py:427
#: modules/websession/lib/websession_webinterface.py:503
msgid "Editing settings failed"
-msgstr ""
+msgstr "ویرایش تنظیمات عمل نکرد"
#: modules/websession/lib/websession_webinterface.py:410
#: modules/websession/lib/websession_webinterface.py:907
#, python-format
msgid "Supplied email address %s is invalid."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:416
#: modules/websession/lib/websession_webinterface.py:917
#, python-format
msgid "Supplied email address %s already exists in the database."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:418
#: modules/websession/lib/websession_webinterface.py:919
msgid "Or please try again."
-msgstr ""
+msgstr "یا لطفا دوباره تلاش کنید."
#: modules/websession/lib/websession_webinterface.py:423
#, python-format
msgid "Desired nickname %s is already in use."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:432
msgid "Users cannot edit passwords on this site."
-msgstr ""
+msgstr "کاربران نمی توانند کلمه های عبور این سایت را ویرایش کنند."
#: modules/websession/lib/websession_webinterface.py:440
msgid "Password successfully edited."
-msgstr ""
+msgstr "کلمه عبور به طور موفقیت آمیز ویرایش شد."
#: modules/websession/lib/websession_webinterface.py:443
msgid "Password edited"
-msgstr ""
+msgstr "کلمه عبور ویرایش شد"
#: modules/websession/lib/websession_webinterface.py:445
#: modules/websession/lib/websession_webinterface.py:902
msgid "Both passwords must match."
-msgstr ""
+msgstr "هر دو کلمه عبور باید منطبق باشند."
#: modules/websession/lib/websession_webinterface.py:449
#: modules/websession/lib/websession_webinterface.py:455
msgid "Editing password failed"
-msgstr ""
+msgstr "ویرایش کلمه عبور عمل نکرد"
#: modules/websession/lib/websession_webinterface.py:451
msgid "Wrong old password inserted."
-msgstr ""
+msgstr "کلمه عبور قدیمی اشتباه وارد شده است."
#: modules/websession/lib/websession_webinterface.py:467
#: modules/websession/lib/websession_webinterface.py:480
#: modules/websession/lib/websession_webinterface.py:494
msgid "User settings saved correctly."
-msgstr ""
+msgstr "تنظیمات کاربری به طور صحیح ذخیره شد."
#: modules/websession/lib/websession_webinterface.py:487
msgid "Editing bibcatalog authorization failed"
msgstr ""
#: modules/websession/lib/websession_webinterface.py:488
msgid "Empty username or password"
-msgstr ""
+msgstr "خالی کردن نام کاربری یا کلمه عبور"
#: modules/websession/lib/websession_webinterface.py:497
msgid "Unable to update settings."
-msgstr ""
+msgstr "نمی توان تنظیمات را بروزرسانی کرد."
#: modules/websession/lib/websession_webinterface.py:558
msgid ""
"Cannot send password reset request since you are using external "
"authentication system."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:574
msgid "The entered email address does not exist in the database."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:588
msgid "Password reset request for"
-msgstr ""
+msgstr "درخواست بازتنظیم کلمه عبور برای"
#: modules/websession/lib/websession_webinterface.py:592
msgid ""
"The entered email address is incorrect, please check that it is written "
"correctly (e.g. johndoe@example.com)."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:593
msgid "Incorrect email address"
-msgstr ""
+msgstr "نشانی پست الکترونیکی نادرست"
#: modules/websession/lib/websession_webinterface.py:603
msgid "Reset password link sent"
-msgstr ""
+msgstr "لینک بازتنظیم کلمه عبور ارسال شد"
#: modules/websession/lib/websession_webinterface.py:648
msgid "Delete Account"
-msgstr ""
+msgstr "حذف حساب"
#: modules/websession/lib/websession_webinterface.py:674
msgid "Logout"
-msgstr ""
+msgstr "خروج"
#: modules/websession/lib/websession_webinterface.py:747
#: modules/websession/lib/websession_webinterface.py:803
#: modules/websession/lib/websession_webinterface.py:838
msgid "Login"
-msgstr ""
+msgstr "ورود"
#: modules/websession/lib/websession_webinterface.py:869
msgid "Register"
-msgstr ""
+msgstr "ثبت"
#: modules/websession/lib/websession_webinterface.py:872
#: modules/websession/lib/websession_webinterface.py:944
#, python-format
msgid "%s Personalize, Main page"
msgstr ""
#: modules/websession/lib/websession_webinterface.py:889
msgid "Your account has been successfully created."
-msgstr ""
+msgstr "حساب شما به طور موفقیت آمیزی ایجاد شد."
#: modules/websession/lib/websession_webinterface.py:890
msgid "Account created"
-msgstr ""
+msgstr "حساب ایجاد شد"
#: modules/websession/lib/websession_webinterface.py:892
msgid ""
"In order to confirm its validity, an email message containing an account "
"activation key has been sent to the given email address."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:893
msgid ""
"Please follow instructions presented there in order to complete the account "
"registration process."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:895
msgid ""
"A second email will be sent when the account has been activated and can be "
"used."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:898
#, python-format
msgid "You can now access your %(x_url_open)saccount%(x_url_close)s."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:905
#: modules/websession/lib/websession_webinterface.py:910
#: modules/websession/lib/websession_webinterface.py:915
#: modules/websession/lib/websession_webinterface.py:921
#: modules/websession/lib/websession_webinterface.py:926
#: modules/websession/lib/websession_webinterface.py:930
#: modules/websession/lib/websession_webinterface.py:934
#: modules/websession/lib/websession_webinterface.py:939
msgid "Registration failure"
-msgstr ""
+msgstr "ثبت نام شکست خورد"
#: modules/websession/lib/websession_webinterface.py:923
#, python-format
msgid "Desired nickname %s already exists in the database."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:928
msgid "Users cannot register themselves, only admin can register them."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:932
msgid ""
"The site is having troubles in sending you an email for confirming your "
"email address."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:932
#: modules/websubmit/lib/websubmit_webinterface.py:151
msgid ""
"The error has been logged and will be taken in consideration as soon as "
"possible."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:979
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:729
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:733
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:765
msgid "Your tickets"
msgstr ""
#: modules/websession/lib/websession_webinterface.py:1015
#: modules/websession/lib/websession_webinterface.py:1056
#: modules/websession/lib/websession_webinterface.py:1115
#: modules/websession/lib/websession_webinterface.py:1177
#: modules/websession/lib/websession_webinterface.py:1234
#: modules/websession/lib/websession_webinterface.py:1305
msgid "You are not authorized to use groups."
msgstr ""
#: modules/websession/lib/websession_webinterface.py:1140
msgid "Join New Group"
-msgstr ""
+msgstr "پیوستن گروه جدید"
#: modules/websession/lib/websession_webinterface.py:1192
msgid "Leave Group"
-msgstr ""
+msgstr "ترک گروه"
#: modules/websession/lib/websession_webinterface.py:1262
msgid "Edit Group"
-msgstr ""
+msgstr "ویرایش گروه"
#: modules/websession/lib/websession_webinterface.py:1334
msgid "Edit group members"
-msgstr ""
+msgstr "ویرایش اعضاء گروه"
#: modules/websession/lib/webgroup.py:158
#: modules/websession/lib/webgroup.py:432
msgid "Please enter a group name."
-msgstr ""
+msgstr "لطفا نام یک گروه را وارد کنید."
#: modules/websession/lib/webgroup.py:168
#: modules/websession/lib/webgroup.py:442
msgid "Please enter a valid group name."
msgstr ""
#: modules/websession/lib/webgroup.py:178
#: modules/websession/lib/webgroup.py:452
msgid "Please choose a group join policy."
msgstr ""
#: modules/websession/lib/webgroup.py:188
#: modules/websession/lib/webgroup.py:462
msgid "Group name already exists. Please choose another group name."
msgstr ""
#: modules/websession/lib/webgroup.py:260
msgid "You are already member of the group."
-msgstr ""
+msgstr "شما قبلا عضو گروه بوده اید."
#: modules/websession/lib/webgroup.py:302
msgid "Please select only one group."
-msgstr ""
+msgstr "لطفا تنها یک گروه را انتخاب کنید."
#: modules/websession/lib/webgroup.py:359
msgid "Please select one group."
-msgstr ""
+msgstr "لطفا یک گروه را انتخاب کنید."
#: modules/websession/lib/webgroup.py:384
#: modules/websession/lib/webgroup.py:399
#: modules/websession/lib/webgroup.py:510
#: modules/websession/lib/webgroup.py:555
#: modules/websession/lib/webgroup.py:570
#: modules/websession/lib/webgroup.py:604
#: modules/websession/lib/webgroup.py:644
#: modules/websession/lib/webgroup.py:711
msgid "Sorry, there was an error with the database."
msgstr ""
#: modules/websession/lib/webgroup.py:391
#: modules/websession/lib/webgroup.py:562
msgid "Sorry, you do not have sufficient rights on this group."
msgstr ""
#: modules/websession/lib/webgroup.py:499
msgid "The group has already been deleted."
-msgstr ""
+msgstr "گروه قبلا حذف شده است."
#: modules/websession/lib/webgroup.py:611
msgid "Please choose a member if you want to remove him from the group."
-msgstr ""
+msgstr "اگر می خواهید عضوی از گروه را حذف کنید لطفا وی را انتخاب کنید."
#: modules/websession/lib/webgroup.py:651
msgid ""
"Please choose a user from the list if you want him to be added to the group."
msgstr ""
+"اگر می خواهید کاربری را به گروه اضافه کنید لطفا وی را از سیاهه انتخاب کنید."
#: modules/websession/lib/webgroup.py:664
msgid "The user is already member of the group."
-msgstr ""
+msgstr "کاربر قبلا عضو گروه بوده است."
#: modules/websession/lib/webgroup.py:718
msgid ""
"Please choose a user from the list if you want him to be removed from "
"waiting list."
msgstr ""
+"اگر می خواهید کاربری را از سیاهه انتظار حذف کنید لطفا وی را از سیاهه انتخاب "
+"کنید."
#: modules/websession/lib/webgroup.py:731
msgid "The user request for joining group has already been rejected."
-msgstr ""
+msgstr "درخواست کاربر برای پیوستن گروه قبلا رد شده است."
#: modules/webstyle/lib/webstyle_templates.py:86
#: modules/webstyle/lib/webstyle_templates.py:95
#: modules/bibcirculation/lib/bibcirculation_templates.py:111
msgid "Home"
-msgstr ""
+msgstr "خانه"
#: modules/webstyle/lib/webstyle_templates.py:434
#: modules/webstyle/lib/webstyle_templates.py:503
#: modules/websubmit/lib/websubmit_engine.py:697
#: modules/websubmit/lib/websubmit_engine.py:1142
#: modules/websubmit/lib/websubmit_engine.py:1188
#: modules/websubmit/lib/websubmit_engine.py:1421
msgid "Submit"
-msgstr ""
+msgstr "واگذاری"
#: modules/webstyle/lib/webstyle_templates.py:436
#: modules/webstyle/lib/webstyle_templates.py:505
#: modules/bibcirculation/lib/bibcirculation_templates.py:215
msgid "Help"
-msgstr ""
+msgstr "راهنما"
#: modules/webstyle/lib/webstyle_templates.py:469
msgid "Last updated"
-msgstr ""
+msgstr "آخرین بروزرسانی"
#: modules/webstyle/lib/webstyle_templates.py:507
msgid "Powered by"
msgstr ""
#: modules/webstyle/lib/webstyle_templates.py:508
msgid "Maintained by"
msgstr ""
#: modules/webstyle/lib/webstyle_templates.py:554
msgid "This site is also available in the following languages:"
msgstr "این صفحه در زبان های زیر نیز قابل دسترسی است:"
#: modules/webstyle/lib/webstyle_templates.py:587
msgid "Browser"
msgstr ""
#: modules/webstyle/lib/webstyle_templates.py:609
msgid "System Error"
-msgstr ""
+msgstr "خطای سیستم"
#: modules/webstyle/lib/webstyle_templates.py:624
msgid "Traceback"
msgstr ""
#: modules/webstyle/lib/webstyle_templates.py:671
msgid "Client"
msgstr ""
#: modules/webstyle/lib/webstyle_templates.py:673
msgid "Please send an error report to the administrator."
-msgstr ""
+msgstr "لطفا گزارش خطا را به مدیر ارسال کنید."
#: modules/webstyle/lib/webstyle_templates.py:674
msgid "Send error report"
-msgstr ""
+msgstr "ارسال گزارش خطا"
#: modules/webstyle/lib/webstyle_templates.py:678
#, python-format
msgid "Please contact %s quoting the following information:"
msgstr ""
#: modules/webstyle/lib/webstyle_templates.py:728
#: modules/websubmit/lib/websubmit_templates.py:1231
msgid "Restricted"
msgstr ""
#: modules/webstyle/lib/webstyle_templates.py:848
#, python-format
msgid ""
"Record created %(x_date_creation)s, last modified %(x_date_modification)s"
msgstr ""
#: modules/webstyle/lib/webstyle_templates.py:919
msgid "The server encountered an error while dealing with your request."
msgstr ""
#: modules/webstyle/lib/webstyle_templates.py:920
msgid "The system administrators have been alerted."
msgstr ""
#: modules/webstyle/lib/webstyle_templates.py:921
#, python-format
msgid "In case of doubt, please contact %(x_admin_email)s."
msgstr ""
#: modules/webstyle/lib/webdoc.py:550
#, python-format
msgid "%(category)s Pages"
msgstr ""
#: modules/webstyle/lib/webdoc_webinterface.py:144
#: modules/webstyle/lib/webdoc_webinterface.py:149
msgid "Admin Pages"
msgstr ""
#: modules/webstyle/lib/webdoc_webinterface.py:146
#: modules/webstyle/lib/webdoc_webinterface.py:150
msgid "Help Pages"
msgstr ""
#: modules/webstyle/lib/webdoc_webinterface.py:148
#: modules/webstyle/lib/webdoc_webinterface.py:151
msgid "Hacking Pages"
msgstr ""
#: modules/webstyle/lib/webdoc_webinterface.py:157
msgid "Hacking Invenio"
msgstr ""
#: modules/webstyle/lib/webdoc_webinterface.py:159
msgid "Latest modifications:"
msgstr ""
#: modules/webstyle/lib/webdoc_webinterface.py:162
#, python-format
msgid "This is the table of contents of the %(x_category)s pages."
msgstr ""
#: modules/webstyle/lib/webdoc_webinterface.py:164
msgid "See also"
msgstr ""
#: modules/webstyle/lib/webdoc_webinterface.py:179
#, python-format
msgid "Page %s Not Found"
msgstr ""
#: modules/webstyle/lib/webdoc_webinterface.py:187
#, python-format
msgid "Sorry, page %s does not seem to exist."
msgstr ""
#: modules/webstyle/lib/webdoc_webinterface.py:190
#, python-format
msgid ""
"You may want to look at the %(x_url_open)s%(x_category)s pages"
"%(x_url_close)s."
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:383
#: modules/websubmit/lib/functions/Create_Upload_Files_Interface.py:443
msgid "Choose a file"
-msgstr ""
+msgstr "انتخاب یک فایل"
#: modules/websubmit/lib/websubmit_managedocfiles.py:391
#: modules/websubmit/lib/functions/Create_Upload_Files_Interface.py:459
msgid "Access"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:543
msgid ""
"The file you want to edit is protected against modifications. Your action "
"has not been applied"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:562
#, python-format
msgid ""
"The uploaded file is too small (<%i o) and has therefore not been considered"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:567
#, python-format
msgid ""
"The uploaded file is too big (>%i o) and has therefore not been considered"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:574
msgid ""
"The uploaded file name is too long and has therefore not been considered"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:586
msgid ""
"You have already reached the maximum number of files for this type of "
"document"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:609
#: modules/websubmit/lib/websubmit_managedocfiles.py:620
#: modules/websubmit/lib/websubmit_managedocfiles.py:730
#, python-format
msgid "A file named %s already exists. Please choose another name."
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:631
#, python-format
msgid "A file with format '%s' already exists. Please upload another format."
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:639
msgid ""
"You are not allowed to use dot '.', slash '/', or backslash '\\\\' in file "
"names. Choose a different name and upload your file again. In particular, "
"note that you should not include the extension in the renaming field."
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:811
msgid "Choose how you want to restrict access to this file."
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:848
msgid "Add new file"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:873
msgid "You can decide to hide or not previous version(s) of this file."
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:874
msgid ""
"When you revise a file, the additional formats that you might have "
"previously uploaded are removed, since they no longer up-to-date with the "
"new file."
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:875
msgid ""
"Alternative formats uploaded for current version of this file will be removed"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:876
msgid "Keep previous versions"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:878
#: modules/bibknowledge/lib/bibknowledge_templates.py:207
msgid "Upload"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:890
#: modules/websubmit/lib/websubmit_webinterface.py:481
#: modules/bibedit/lib/bibeditmulti_templates.py:321
msgid "Apply changes"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:895
#, python-format
msgid "Need help revising or adding files to record %(recid)s"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:897
#, python-format
msgid ""
"Dear Support,\n"
"I would need help to revise or add a file to record %(recid)s.\n"
"I have attached the new version to this email.\n"
"Best regards"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:902
#, python-format
msgid ""
"Having a problem revising a file? Send the revised version to "
"%(mailto_link)s."
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:905
#, python-format
msgid ""
"Having a problem adding or revising a file? Send the new/revised version to "
"%(mailto_link)s."
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:1029
msgid "revise"
msgstr ""
#: modules/websubmit/lib/websubmit_managedocfiles.py:1073
msgid "add format"
msgstr ""
#: modules/websubmit/lib/functions/Shared_Functions.py:176
msgid ""
"Note that your submission as been inserted into the bibliographic task queue "
"and is waiting for execution.\n"
msgstr ""
#: modules/websubmit/lib/functions/Shared_Functions.py:179
#, python-format
msgid ""
"The task queue is currently running in automatic mode, and there are "
"currently %s tasks waiting to be executed. Your record should be available "
"within a few minutes and searchable within an hour or thereabouts.\n"
msgstr ""
#: modules/websubmit/lib/functions/Shared_Functions.py:181
msgid ""
"Because of a human intervention or a temporary problem, the task queue is "
"currently set to the manual mode. Your submission is well registered but may "
"take longer than usual before it is fully integrated and searchable.\n"
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:179
#: modules/websubmit/lib/websubmit_engine.py:797
#: modules/websubmit/web/yoursubmissions.py:61
msgid "Sorry, you must log in to perform this action."
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:186
#: modules/websubmit/lib/websubmit_engine.py:804
msgid "Not enough information to go ahead with the submission."
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:192
#: modules/websubmit/lib/websubmit_engine.py:273
#: modules/websubmit/lib/websubmit_engine.py:283
#: modules/websubmit/lib/websubmit_engine.py:360
#: modules/websubmit/lib/websubmit_engine.py:401
#: modules/websubmit/lib/websubmit_engine.py:818
#: modules/websubmit/lib/websubmit_engine.py:856
#: modules/websubmit/lib/websubmit_engine.py:919
#: modules/websubmit/lib/websubmit_engine.py:960
msgid "Invalid parameters"
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:198
#: modules/websubmit/lib/websubmit_engine.py:810
msgid "Invalid doctype and act parameters"
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:229
#: modules/websubmit/lib/websubmit_engine.py:845
#, python-format
msgid "Unable to find the submission directory for the action: %s"
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:238
#: modules/websubmit/lib/websubmit_engine.py:1003
msgid "Unknown document type"
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:244
#: modules/websubmit/lib/websubmit_engine.py:1009
msgid "Unknown action"
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:252
#: modules/websubmit/lib/websubmit_engine.py:1016
msgid "Unable to determine the number of submission pages."
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:293
#: modules/websubmit/lib/websubmit_engine.py:864
msgid ""
"Unable to create a directory for this submission. The administrator has been "
"alerted."
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:407
#: modules/websubmit/lib/websubmit_engine.py:967
msgid "Cannot create submission directory. The administrator has been alerted."
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:429
#: modules/websubmit/lib/websubmit_engine.py:989
msgid "No file uploaded?"
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:470
#: modules/websubmit/lib/websubmit_engine.py:473
#: modules/websubmit/lib/websubmit_engine.py:606
#: modules/websubmit/lib/websubmit_engine.py:609
msgid "Unknown form field found on submission page."
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:1055
msgid ""
"A serious function-error has been encountered. Adminstrators have been "
"alerted. <br /><em>Please not that this might be due to wrong characters "
"inserted into the form</em> (e.g. by copy and pasting some text from a PDF "
"file)."
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:1384
#, python-format
msgid "Unable to find document type: %s"
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:1670
msgid "The chosen action is not supported by the document type."
msgstr ""
#: modules/websubmit/lib/websubmit_engine.py:1747
#: modules/websubmit/lib/websubmit_webinterface.py:1377
#: modules/websubmit/web/approve.py:81
msgid "Warning"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:85
msgid "Document types available for submission"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:86
msgid "Please select the type of document you want to submit"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:103
msgid "No document types available."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:269
msgid "Please log in first."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:269
msgid "Use the top-right menu to log in."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:313
msgid "Please select a category"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:352
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:2114
msgid "Notice"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:353
msgid "Select a category and then click on an action button."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:376
msgid ""
"To continue with a previously interrupted submission, enter an access number "
"into the box below:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:378
msgid "GO"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:499
#: modules/websubmit/lib/websubmit_templates.py:968
msgid "SUMMARY"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:535
#: modules/bibharvest/lib/oai_harvest_admin.py:858
#: modules/bibharvest/lib/oai_harvest_admin.py:911
msgid "Previous page"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:541
msgid "Submission number"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:555
#: modules/bibharvest/lib/oai_harvest_admin.py:843
#: modules/bibharvest/lib/oai_harvest_admin.py:899
msgid "Next page"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:570
#: modules/websubmit/lib/websubmit_templates.py:1009
msgid "Are you sure you want to quit this submission?"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:572
#: modules/websubmit/lib/websubmit_templates.py:1010
#: modules/websubmit/lib/websubmit_templates.py:1019
msgid "Back to main menu"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:575
msgid ""
"This is your submission access number. It can be used to continue with an "
"interrupted submission in case of problems."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:576
msgid "Mandatory fields appear in red in the SUMMARY window."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:722
#, python-format
msgid "The field %s is mandatory."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:722
msgid "Please make a choice in the select box"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:736
msgid "Please press a button."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:744
#, python-format
msgid "The field %s is mandatory. Please fill it in."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:821
#, python-format
msgid "The field %(field)s is mandatory."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:822
msgid "Going back to page"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:959
msgid "finished!"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:967
msgid "end of action"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:991
msgid "Submission no"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1062
#, python-format
msgid ""
"Here is the %(x_action)s function list for %(x_doctype)s documents at level "
"%(x_step)s"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1067
msgid "Function"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1068
msgid "Score"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1069
msgid "Running function"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1075
#, python-format
msgid "Function %s does not exist."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1114
msgid "You must now"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1146
msgid "record"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1148
msgid "document"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1150
#: modules/websubmit/lib/websubmit_templates.py:1253
msgid "version"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1185
msgid "file(s)"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1236
msgid "see"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1436
#: modules/bibauthorid/lib/bibauthorid_templates.py:1011
msgid "For"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1437
msgid "all types of document"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1492
msgid "Subm.No."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1493
msgid "Reference"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1495
msgid "First access"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1496
msgid "Last access"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1506
msgid "Are you sure you want to delete this submission?"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1507
#, python-format
msgid "Delete submission %(x_id)s in %(x_docname)s"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1531
msgid "Reference not yet given"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1602
msgid "Refereed Documents"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1612
msgid "You are a general referee"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1618
msgid "You are a referee for category:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1657
#: modules/websubmit/lib/websubmit_templates.py:1702
msgid "List of refereed types of documents"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1658
#: modules/websubmit/lib/websubmit_templates.py:1703
msgid ""
"Select one of the following types of documents to check the documents status"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1671
msgid "Go to specific approval workflow"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1759
msgid "List of refereed categories"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1760
#: modules/websubmit/lib/websubmit_templates.py:1909
msgid "Please choose a category"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1780
#: modules/websubmit/lib/websubmit_templates.py:1821
#: modules/websubmit/lib/websubmit_templates.py:1932
#: modules/websubmit/lib/websubmit_templates.py:1990
#: modules/websubmit/lib/websubmit_templates.py:2056
#: modules/websubmit/lib/websubmit_templates.py:2181
msgid "Pending"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1786
#: modules/websubmit/lib/websubmit_templates.py:1824
#: modules/websubmit/lib/websubmit_templates.py:1939
#: modules/websubmit/lib/websubmit_templates.py:1993
#: modules/websubmit/lib/websubmit_templates.py:2057
#: modules/websubmit/lib/websubmit_templates.py:2182
msgid "Approved"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1792
#: modules/websubmit/lib/websubmit_templates.py:1826
#: modules/websubmit/lib/websubmit_templates.py:1827
#: modules/websubmit/lib/websubmit_templates.py:1946
#: modules/websubmit/lib/websubmit_templates.py:1995
#: modules/websubmit/lib/websubmit_templates.py:1996
#: modules/websubmit/lib/websubmit_templates.py:2058
#: modules/websubmit/lib/websubmit_templates.py:2183
msgid "Rejected"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1820
#: modules/websubmit/lib/websubmit_templates.py:1989
msgid "Key"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1823
#: modules/websubmit/lib/websubmit_templates.py:1992
msgid "Waiting for approval"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1825
#: modules/websubmit/lib/websubmit_templates.py:1994
msgid "Already approved"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1828
#: modules/websubmit/lib/websubmit_templates.py:1999
msgid "Some documents are pending."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1873
msgid "List of refereing categories"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:1953
#: modules/websubmit/lib/websubmit_templates.py:1997
#: modules/websubmit/lib/websubmit_templates.py:1998
#: modules/websubmit/lib/websubmit_templates.py:2184
msgid "Cancelled"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2053
#: modules/websubmit/lib/websubmit_templates.py:2141
msgid "List of refereed documents"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2054
#: modules/websubmit/lib/websubmit_templates.py:2178
msgid "Click on a report number for more information."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2055
#: modules/websubmit/lib/websubmit_templates.py:2180
msgid "Report Number"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2143
msgid "List of publication documents"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2145
msgid "List of direct approval documents"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2318
msgid "Your request has been sent to the referee."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2334
#: modules/websubmit/lib/websubmit_templates.py:2455
#: modules/websubmit/lib/websubmit_templates.py:2766
#: modules/websubmit/lib/websubmit_templates.py:2950
msgid "Title:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2340
#: modules/websubmit/lib/websubmit_templates.py:2462
#: modules/websubmit/lib/websubmit_templates.py:2772
#: modules/websubmit/lib/websubmit_templates.py:2956
msgid "Author:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2348
#: modules/websubmit/lib/websubmit_templates.py:2471
#: modules/websubmit/lib/websubmit_templates.py:2780
#: modules/websubmit/lib/websubmit_templates.py:2964
msgid "More information:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2349
#: modules/websubmit/lib/websubmit_templates.py:2472
#: modules/websubmit/lib/websubmit_templates.py:2781
#: modules/websubmit/lib/websubmit_templates.py:2965
msgid "Click here"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2358
msgid "Approval note:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2363
#, python-format
msgid ""
"This document is still %(x_fmt_open)swaiting for approval%(x_fmt_close)s."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2366
#: modules/websubmit/lib/websubmit_templates.py:2386
#: modules/websubmit/lib/websubmit_templates.py:2395
msgid "It was first sent for approval on:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2368
#: modules/websubmit/lib/websubmit_templates.py:2370
#: modules/websubmit/lib/websubmit_templates.py:2388
#: modules/websubmit/lib/websubmit_templates.py:2390
#: modules/websubmit/lib/websubmit_templates.py:2397
#: modules/websubmit/lib/websubmit_templates.py:2399
msgid "Last approval email was sent on:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2371
msgid ""
"You can send an approval request email again by clicking the following "
"button:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2373
#: modules/websubmit/web/publiline.py:366
msgid "Send Again"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2374
msgid "WARNING! Upon confirmation, an email will be sent to the referee."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2377
msgid ""
"As a referee for this document, you may click this button to approve or "
"reject it"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2379
msgid "Approve/Reject"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2384
#, python-format
msgid "This document has been %(x_fmt_open)sapproved%(x_fmt_close)s."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2385
msgid "Its approved reference is:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2391
msgid "It was approved on:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2393
#, python-format
msgid "This document has been %(x_fmt_open)srejected%(x_fmt_close)s."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2400
msgid "It was rejected on:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2484
#: modules/websubmit/lib/websubmit_templates.py:2539
#: modules/websubmit/lib/websubmit_templates.py:2602
msgid "It has first been asked for refereing process on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2487
#: modules/websubmit/lib/websubmit_templates.py:2541
msgid "Last request e-mail was sent to the publication committee chair on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2491
msgid "A referee has been selected by the publication committee on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2494
#: modules/websubmit/lib/websubmit_templates.py:2557
msgid "No referee has been selected yet."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2496
#: modules/websubmit/lib/websubmit_templates.py:2559
msgid "Select a referee"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2501
msgid ""
"The referee has sent his final recommendations to the publication committee "
"on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2504
#: modules/websubmit/lib/websubmit_templates.py:2565
msgid "No recommendation from the referee yet."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2506
#: modules/websubmit/lib/websubmit_templates.py:2516
#: modules/websubmit/lib/websubmit_templates.py:2567
#: modules/websubmit/lib/websubmit_templates.py:2575
#: modules/websubmit/lib/websubmit_templates.py:2583
msgid "Send a recommendation"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2511
#: modules/websubmit/lib/websubmit_templates.py:2579
msgid ""
"The publication committee has sent his final recommendations to the project "
"leader on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2514
#: modules/websubmit/lib/websubmit_templates.py:2581
msgid "No recommendation from the publication committee yet."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2521
#: modules/websubmit/lib/websubmit_templates.py:2587
#: modules/websubmit/lib/websubmit_templates.py:2607
msgid "It has been cancelled by the author on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2525
#: modules/websubmit/lib/websubmit_templates.py:2590
#: modules/websubmit/lib/websubmit_templates.py:2610
msgid "It has been approved by the project leader on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2528
#: modules/websubmit/lib/websubmit_templates.py:2592
#: modules/websubmit/lib/websubmit_templates.py:2612
msgid "It has been rejected by the project leader on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2531
#: modules/websubmit/lib/websubmit_templates.py:2594
#: modules/websubmit/lib/websubmit_templates.py:2614
msgid "No final decision taken yet."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2533
#: modules/websubmit/lib/websubmit_templates.py:2596
#: modules/websubmit/lib/websubmit_templates.py:2616
#: modules/websubmit/web/publiline.py:1136
#: modules/websubmit/web/publiline.py:1146
msgid "Take a decision"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2544
msgid ""
"An editorial board has been selected by the publication committee on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2546
msgid "Add an author list"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2549
msgid "No editorial board has been selected yet."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2551
msgid "Select an editorial board"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2555
msgid "A referee has been selected by the editorial board on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2563
msgid ""
"The referee has sent his final recommendations to the editorial board on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2571
msgid ""
"The editorial board has sent his final recommendations to the publication "
"committee on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2573
msgid "No recommendation from the editorial board yet."
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2604
msgid "Last request e-mail was sent to the project leader on the "
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2698
msgid "Comments overview"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2820
msgid "search for user"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2822
msgid "search for users"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2825
#: modules/websubmit/lib/websubmit_templates.py:2827
#: modules/websubmit/lib/websubmit_templates.py:2880
#: modules/websubmit/lib/websubmit_templates.py:2882
msgid "select user"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2836
msgid "connected"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2839
msgid "add this user"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:2889
msgid "remove this user"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:3003
msgid "User"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:3084
#: modules/websubmit/web/publiline.py:1133
msgid "Select:"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:3085
msgid "approve"
msgstr ""
#: modules/websubmit/lib/websubmit_templates.py:3086
msgid "reject"
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:116
msgid "Requested record does not seem to have been integrated."
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:150
msgid ""
"The system has encountered an error in retrieving the list of files for this "
"document."
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:216
msgid "This file is restricted: "
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:228
msgid "An error has happened in trying to stream the request file."
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:231
msgid "The requested file is hidden and can not be accessed."
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:238
msgid "Requested file does not seem to exist."
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:272
msgid "Access to Fulltext"
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:321
msgid "An error has happened in trying to retrieve the requested file."
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:323
msgid "Not enough information to retrieve the document"
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:331
msgid "An error has happened in trying to retrieving the requested file."
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:383
msgid "Manage Document Files"
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:401
#, python-format
msgid "Your modifications to record #%i have been submitted"
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:409
#, python-format
msgid "Your modifications to record #%i have been cancelled"
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:418
msgid "Edit"
-msgstr ""
+msgstr "ویرایش"
#: modules/websubmit/lib/websubmit_webinterface.py:419
msgid "Edit record"
-msgstr ""
+msgstr "ویرایش رکورد"
#: modules/websubmit/lib/websubmit_webinterface.py:434
#: modules/websubmit/lib/websubmit_webinterface.py:490
msgid "Document File Manager"
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:435
#: modules/websubmit/lib/websubmit_webinterface.py:490
#, python-format
msgid "Record #%i"
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:482
msgid "Cancel all changes"
-msgstr ""
+msgstr "لغو همه تغییرات"
#: modules/websubmit/lib/websubmit_webinterface.py:1171
msgid "Sorry, 'sub' parameter missing..."
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:1174
msgid "Sorry. Cannot analyse parameter"
msgstr ""
#: modules/websubmit/lib/websubmit_webinterface.py:1235
msgid "Sorry, invalid URL..."
msgstr ""
#: modules/websubmit/lib/websubmitadmin_engine.py:3902
#, python-format
msgid ""
"Unable to move field at position %s to position %s on page %s of submission "
"'%s%s' - Invalid Field Position Numbers"
msgstr ""
#: modules/websubmit/lib/websubmitadmin_engine.py:3913
#, python-format
msgid ""
"Unable to swap field at position %s with field at position %s on page %s of "
"submission %s - could not move field at position %s to temporary field "
"location"
msgstr ""
#: modules/websubmit/lib/websubmitadmin_engine.py:3924
#, python-format
msgid ""
"Unable to swap field at position %s with field at position %s on page %s of "
"submission %s - could not move field at position %s to position %s. Please "
"ask Admin to check that a field was not stranded in a temporary position"
msgstr ""
#: modules/websubmit/lib/websubmitadmin_engine.py:3935
#, python-format
msgid ""
"Unable to swap field at position %s with field at position %s on page %s of "
"submission %s - could not move field that was located at position %s to "
"position %s from temporary position. Field is now stranded in temporary "
"position and must be corrected manually by an Admin"
msgstr ""
#: modules/websubmit/lib/websubmitadmin_engine.py:3946
#, python-format
msgid ""
"Unable to move field at position %s to position %s on page %s of submission "
"%s - could not decrement the position of the fields below position %s. Tried "
"to recover - please check that field ordering is not broken"
msgstr ""
#: modules/websubmit/lib/websubmitadmin_engine.py:3957
#, python-format
msgid ""
"Unable to move field at position %s to position %s on page %s of submission "
"%s%s - could not increment the position of the fields at and below position "
"%s. The field that was at position %s is now stranded in a temporary "
"position."
msgstr ""
#: modules/websubmit/lib/websubmitadmin_engine.py:3968
#, python-format
msgid ""
"Moved field from position %s to position %s on page %s of submission '%s%s'."
msgstr ""
#: modules/websubmit/lib/websubmitadmin_engine.py:3989
#, python-format
msgid "Unable to delete field at position %s from page %s of submission '%s'"
msgstr ""
#: modules/websubmit/lib/websubmitadmin_engine.py:3999
#, python-format
msgid "Unable to delete field at position %s from page %s of submission '%s%s'"
msgstr ""
#: modules/websubmit/web/approve.py:55
msgid "approve.py: cannot determine document reference"
msgstr ""
#: modules/websubmit/web/approve.py:58
msgid "approve.py: cannot find document in database"
msgstr ""
#: modules/websubmit/web/approve.py:72
msgid "Sorry parameter missing..."
msgstr ""
#: modules/websubmit/web/publiline.py:133
msgid "Document Approval Workflow"
msgstr ""
#: modules/websubmit/web/publiline.py:154
msgid "Approval and Refereeing Workflow"
msgstr ""
#: modules/websubmit/web/publiline.py:333
#: modules/websubmit/web/publiline.py:434
#: modules/websubmit/web/publiline.py:660
msgid "Approval has never been requested for this document."
msgstr ""
#: modules/websubmit/web/publiline.py:356
#: modules/websubmit/web/publiline.py:358
#: modules/websubmit/web/publiline.py:460
#: modules/websubmit/web/publiline.py:685
msgid "Unable to display document."
msgstr ""
#: modules/websubmit/web/publiline.py:689
#: modules/websubmit/web/publiline.py:813
#: modules/websubmit/web/publiline.py:928
#: modules/websubmit/web/publiline.py:992
#: modules/websubmit/web/publiline.py:1033
#: modules/websubmit/web/publiline.py:1089
#: modules/websubmit/web/publiline.py:1152
#: modules/websubmit/web/publiline.py:1202
msgid "Action unauthorized for this document."
msgstr ""
#: modules/websubmit/web/publiline.py:692
#: modules/websubmit/web/publiline.py:816
#: modules/websubmit/web/publiline.py:931
#: modules/websubmit/web/publiline.py:995
#: modules/websubmit/web/publiline.py:1036
#: modules/websubmit/web/publiline.py:1092
#: modules/websubmit/web/publiline.py:1155
#: modules/websubmit/web/publiline.py:1205
msgid "Action unavailable for this document."
msgstr ""
#: modules/websubmit/web/publiline.py:702
msgid "Adding users to the editorial board"
msgstr ""
#: modules/websubmit/web/publiline.py:730
#: modules/websubmit/web/publiline.py:853
msgid "no qualified users, try new search."
msgstr ""
#: modules/websubmit/web/publiline.py:732
#: modules/websubmit/web/publiline.py:855
msgid "hits"
msgstr ""
#: modules/websubmit/web/publiline.py:732
#: modules/websubmit/web/publiline.py:855
msgid "too many qualified users, specify more narrow search."
msgstr ""
#: modules/websubmit/web/publiline.py:732
#: modules/websubmit/web/publiline.py:855
msgid "limit"
msgstr ""
#: modules/websubmit/web/publiline.py:748
msgid "users in brackets are already attached to the role, try another one..."
msgstr ""
#: modules/websubmit/web/publiline.py:754
msgid "Removing users from the editorial board"
msgstr ""
#: modules/websubmit/web/publiline.py:790
msgid "Validate the editorial board selection"
msgstr ""
#: modules/websubmit/web/publiline.py:835
msgid "Referee selection"
msgstr ""
#: modules/websubmit/web/publiline.py:921
msgid "Come back to the document"
msgstr ""
#: modules/websubmit/web/publiline.py:1106
msgid "Back to the document"
msgstr ""
#: modules/websubmit/web/publiline.py:1134
#: modules/websubmit/web/publiline.py:1194
msgid "Approve"
-msgstr ""
+msgstr "تأیید کردن"
#: modules/websubmit/web/publiline.py:1135
#: modules/websubmit/web/publiline.py:1195
msgid "Reject"
-msgstr ""
+msgstr "رد کردن"
#: modules/websubmit/web/publiline.py:1233
msgid "Wrong action for this document."
msgstr ""
#: modules/websubmit/web/yourapprovals.py:57
msgid "You are not authorized to use approval system."
msgstr ""
#: modules/webjournal/lib/webjournal_templates.py:50
msgid "Available Journals"
msgstr ""
#: modules/webjournal/lib/webjournal_templates.py:59
#: modules/webjournal/lib/webjournal_templates.py:98
#, python-format
msgid "Contact %(x_url_open)sthe administrator%(x_url_close)s"
msgstr ""
#: modules/webjournal/lib/webjournal_templates.py:143
msgid "Regeneration Error"
msgstr ""
#: modules/webjournal/lib/webjournal_templates.py:144
msgid ""
"The issue could not be correctly regenerated. Please contact your "
"administrator."
msgstr ""
#: modules/webjournal/lib/webjournal_templates.py:270
#, python-format
msgid "If you cannot read this email please go to %(x_journal_link)s"
msgstr ""
#: modules/webjournal/lib/webjournal_templates.py:417
#: modules/webjournal/lib/webjournal_templates.py:666
#: modules/webjournal/lib/webjournaladminlib.py:299
#: modules/webjournal/lib/webjournaladminlib.py:319
msgid "Add"
-msgstr ""
+msgstr "افزودن"
#: modules/webjournal/lib/webjournal_templates.py:418
#: modules/webjournal/lib/webjournaladminlib.py:338
msgid "Publish"
-msgstr ""
+msgstr "منتشر کردن"
#: modules/webjournal/lib/webjournal_templates.py:419
#: modules/webjournal/lib/webjournaladminlib.py:299
#: modules/webjournal/lib/webjournaladminlib.py:316
msgid "Refresh"
-msgstr ""
+msgstr "تازه کردن"
#: modules/webjournal/lib/webjournal_templates.py:482
#: modules/webjournal/lib/webjournaladminlib.py:364
#: modules/bibcirculation/lib/bibcirculation_templates.py:3254
#: modules/bibcirculation/lib/bibcirculation_templates.py:3275
#: modules/bibcirculation/lib/bibcirculation_templates.py:3296
#: modules/bibcirculation/lib/bibcirculation_templates.py:3317
#: modules/bibcirculation/lib/bibcirculation_templates.py:3947
#: modules/bibcirculation/lib/bibcirculation_templates.py:4499
#: modules/bibcirculation/lib/bibcirculation_templates.py:4507
#: modules/bibcirculation/lib/bibcirculation_templates.py:8140
#: modules/bibcirculation/lib/bibcirculation_templates.py:14718
msgid "Update"
-msgstr ""
+msgstr "بروز رسانی"
#: modules/webjournal/lib/webjournal_templates.py:661
msgid "Apply"
-msgstr ""
+msgstr "اجرا کردن"
#: modules/webjournal/lib/webjournal_config.py:64
msgid "Page not found"
-msgstr ""
+msgstr "صفحه پیدا نشد"
#: modules/webjournal/lib/webjournal_config.py:65
msgid "The requested page does not exist"
-msgstr ""
+msgstr "صفحه درخواستی وجود ندارد"
#: modules/webjournal/lib/webjournal_config.py:96
msgid "No journal articles"
-msgstr ""
+msgstr "نبود مقاله های مجله"
#: modules/webjournal/lib/webjournal_config.py:97
#: modules/webjournal/lib/webjournal_config.py:138
msgid "Problem with the configuration of this journal"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:137
msgid "No journal issues"
-msgstr ""
+msgstr "نبود شماره های مجله"
#: modules/webjournal/lib/webjournal_config.py:176
msgid "Journal article error"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:177
msgid "We could not know which article you were looking for"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:211
msgid "No journals available"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:212
msgid "We could not provide you any journals"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:213
msgid ""
"It seems that there are no journals defined on this server. Please contact "
"support if this is not right."
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:239
msgid "Select a journal on this server"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:240
msgid "We couldn't guess which journal you are looking for"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:241
msgid ""
"You did not provide an argument for a journal name. Please select the "
"journal you want to read in the list below."
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:268
msgid "No current issue"
-msgstr ""
+msgstr "نبود شماره جاری"
#: modules/webjournal/lib/webjournal_config.py:269
msgid "We could not find any informtion on the current issue"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:270
msgid ""
"The configuration for the current issue seems to be empty. Try providing an "
"issue number or check with support."
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:298
msgid "Issue number badly formed"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:299
msgid "We could not read the issue number you provided"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:329
msgid "Archive date badly formed"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:330
msgid "We could not read the archive date you provided"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:365
msgid "No popup record"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:366
msgid "We could not deduce the popup article you requested"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:399
msgid "Update error"
-msgstr ""
+msgstr "خطای بروزرسانی"
#: modules/webjournal/lib/webjournal_config.py:400
#: modules/webjournal/lib/webjournal_config.py:431
msgid "There was an internal error"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:430
msgid "Journal publishing DB error"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:463
msgid "Journal issue error"
-msgstr ""
+msgstr "خطای شماره مجله"
#: modules/webjournal/lib/webjournal_config.py:464
msgid "Issue not found"
-msgstr ""
+msgstr "شماره پیدا نشد"
#: modules/webjournal/lib/webjournal_config.py:494
msgid "Journal ID error"
-msgstr ""
+msgstr "خطای تعیین هویت مجله"
#: modules/webjournal/lib/webjournal_config.py:495
msgid "We could not find the id for this journal in the Database"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:527
#: modules/webjournal/lib/webjournal_config.py:529
#, python-format
msgid "Category \"%(category_name)s\" not found"
msgstr ""
#: modules/webjournal/lib/webjournal_config.py:531
msgid "Sorry, this category does not exist for this journal and issue."
msgstr ""
#: modules/webjournal/lib/webjournaladminlib.py:350
msgid "Please select an issue"
-msgstr ""
+msgstr "لطفا یک شماره انتخاب کنید"
#: modules/webjournal/web/admin/webjournaladmin.py:77
msgid "WebJournal Admin"
msgstr ""
#: modules/webjournal/web/admin/webjournaladmin.py:119
#, python-format
msgid "Administrate %(journal_name)s"
msgstr ""
#: modules/webjournal/web/admin/webjournaladmin.py:158
msgid "Feature a record"
msgstr ""
#: modules/webjournal/web/admin/webjournaladmin.py:220
msgid "Email Alert System"
msgstr ""
#: modules/webjournal/web/admin/webjournaladmin.py:273
msgid "Issue regenerated"
msgstr ""
#: modules/webjournal/web/admin/webjournaladmin.py:324
msgid "Publishing Interface"
msgstr ""
#: modules/webjournal/web/admin/webjournaladmin.py:350
msgid "Add Journal"
-msgstr ""
+msgstr "افزودن مجله"
#: modules/webjournal/web/admin/webjournaladmin.py:352
msgid "Edit Settings"
-msgstr ""
+msgstr "ویرایش تنظیمات"
#: modules/bibcatalog/lib/bibcatalog_templates.py:43
#, python-format
msgid "You have %i tickets."
msgstr ""
#: modules/bibcatalog/lib/bibcatalog_templates.py:62
msgid "show"
-msgstr ""
+msgstr "نمایش"
#: modules/bibcatalog/lib/bibcatalog_templates.py:63
msgid "close"
-msgstr ""
+msgstr "بستن"
#: modules/webjournal/lib/widgets/bfe_webjournal_widget_seminars.py:68
#: modules/webjournal/lib/widgets/bfe_webjournal_widget_seminars.py:82
#: modules/webjournal/lib/widgets/bfe_webjournal_widget_weather.py:125
#: modules/webjournal/lib/widgets/bfe_webjournal_widget_weather.py:140
msgid "No information available"
-msgstr ""
+msgstr "هیچ اطلاعاتی قابل استفاده نیست"
#: modules/webjournal/lib/widgets/bfe_webjournal_widget_seminars.py:86
msgid "No seminars today"
msgstr ""
#: modules/webjournal/lib/widgets/bfe_webjournal_widget_seminars.py:184
msgid "What's on today"
msgstr ""
#: modules/webjournal/lib/widgets/bfe_webjournal_widget_seminars.py:185
msgid "Seminars of the week"
-msgstr ""
+msgstr "سمینارهای هفته"
#: modules/webjournal/lib/widgets/bfe_webjournal_widget_whatsNew.py:163
msgid "There are no new articles for the moment"
msgstr ""
#: modules/webjournal/lib/widgets/bfe_webjournal_widget_whatsNew.py:285
msgid "What's new"
msgstr ""
#: modules/webjournal/lib/widgets/bfe_webjournal_widget_weather.py:224
msgid "Under the CERN sky"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_article_author.py:48
#, python-format
msgid "About your article at %(url)s"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_imprint.py:122
msgid "Issue No."
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_article_body.py:221
msgid "Did you know?"
-msgstr ""
+msgstr "آیا شما می دانید؟"
#: modules/webjournal/lib/elements/bfe_webjournal_article_body.py:253
#, python-format
msgid ""
"Hi,\n"
"\n"
"Have a look at the following article:\n"
"<%(url)s>"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_article_body.py:260
msgid "Send this article"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_rss.py:136
msgid "Subscribe by RSS"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_main_navigation.py:90
msgid "News Articles"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_main_navigation.py:91
msgid "Official News"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_main_navigation.py:92
msgid "Training and Development"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_main_navigation.py:93
msgid "General Information"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_main_navigation.py:94
msgid "Announcements"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_main_navigation.py:95
msgid "Training"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_main_navigation.py:96
msgid "Events"
-msgstr ""
+msgstr "رخدادها"
#: modules/webjournal/lib/elements/bfe_webjournal_main_navigation.py:97
msgid "Staff Association"
msgstr ""
#: modules/webjournal/lib/elements/bfe_webjournal_archive.py:106
msgid "Archive"
-msgstr ""
+msgstr "آرشیو"
#: modules/webjournal/lib/elements/bfe_webjournal_archive.py:133
msgid "Select Year:"
-msgstr ""
+msgstr "انتخاب سال:"
#: modules/webjournal/lib/elements/bfe_webjournal_archive.py:139
msgid "Select Issue:"
-msgstr ""
+msgstr "انتخاب شماره:"
#: modules/webjournal/lib/elements/bfe_webjournal_archive.py:143
msgid "Select Date:"
-msgstr ""
+msgstr "انتخاب تاریخ"
#: modules/bibedit/lib/bibedit_webinterface.py:169
msgid "Comparing two record revisions"
msgstr ""
#: modules/bibedit/lib/bibedit_webinterface.py:192
msgid "Failed to create a ticket"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:313
msgid "Next Step"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:314
msgid "Search criteria"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:315
msgid "Output tags"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:316
msgid "Filter collection"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:317
msgid "1. Choose search criteria"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:318
msgid ""
"Specify the criteria you'd like to use for filtering records that will be "
"changed. Use \"Search\" to see which records would have been filtered using "
"these criteria."
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:320
msgid "Preview results"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:552
msgid "2. Define changes"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:553
msgid ""
"Specify fields and their subfields that should be changed in every record "
"matching the search criteria."
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:554
msgid "Define new field action"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:555
msgid "Define new subfield action"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:557
msgid "Select action"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:558
msgid "Add field"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:559
msgid "Delete field"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:560
msgid "Update field"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:561
msgid "Add subfield"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:562
msgid "Delete subfield"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:563
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:260
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:402
#: modules/bibknowledge/lib/bibknowledge_templates.py:280
#: modules/bibknowledge/lib/bibknowledge_templates.py:528
msgid "Save"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:565
msgid "Replace substring"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:566
msgid "Replace full content"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:567
msgid "with"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:568
msgid "when subfield $$"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:569
msgid "new value"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:570
msgid "is equal to"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:571
msgid "contains"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:572
msgid "condition"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:573
msgid "when other subfield"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:574
msgid "when subfield"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:575
msgid "Apply only to specific field instances"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:576
msgid "value"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:605
msgid "Back to Results"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_templates.py:703
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:515
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:571
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:590
msgid "records found"
msgstr ""
#: modules/bibedit/lib/bibeditmulti_webinterface.py:102
msgid "Multi-Record Editor"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:116
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:132
msgid "Export Job Overview"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:117
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:189
msgid "New Export Job"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:118
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:443
msgid "Export Job History"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:174
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:195
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:323
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:534
msgid "Run"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:176
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:325
msgid "New"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:196
msgid "Last run"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:256
msgid "Frequency"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:257
msgid "Output Format"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:258
msgid "Start"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:259
msgid "Output Directory"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:262
msgid "Edit Queries"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:348
msgid "Output Fields"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:400
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:460
msgid "Output fields"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:438
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:605
msgid "Download"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:439
msgid "View as: "
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:533
msgid "Job"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:588
msgid "Total"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:653
msgid "All"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:654
msgid "None"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:672
msgid "Manually"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:674
msgid "Daily"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:676
msgid "Weekly"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_templates.py:678
msgid "Monthly"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:192
msgid "Edit Export Job"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:239
msgid "Query Results"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:246
msgid "Export Job Queries"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:320
msgid "New Query"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:325
msgid "Edit Query"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:356
msgid "Export Job Results"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:389
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:423
msgid "Export Job Result"
msgstr ""
#: modules/bibexport/lib/bibexport_method_fieldexporter_webinterface.py:465
#: modules/bibexport/lib/bibexport_method_fieldexporter.py:500
#: modules/bibexport/lib/bibexport_method_fieldexporter.py:515
#: modules/bibexport/lib/bibexport_method_fieldexporter.py:530
msgid "You are not authorised to access this resource."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:399
#: modules/bibcirculation/lib/bibcirculation_templates.py:8849
#: modules/bibcirculation/lib/bibcirculation_templates.py:10425
msgid "Loan information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:403
msgid "This book has been sent to you:"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:429
#: modules/bibcirculation/lib/bibcirculation_templates.py:1756
#: modules/bibcirculation/lib/bibcirculation_templates.py:2102
#: modules/bibcirculation/lib/bibcirculation_templates.py:5993
#: modules/bibcirculation/lib/bibcirculation_templates.py:16119
msgid "Author"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:430
msgid "Editor"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:431
#: modules/bibcirculation/lib/bibcirculation_templates.py:1759
#: modules/bibcirculation/lib/bibcirculation_templates.py:2745
#: modules/bibcirculation/lib/bibcirculation_templates.py:3098
#: modules/bibcirculation/lib/bibcirculation_templates.py:5995
#: modules/bibcirculation/lib/bibcirculation_templates.py:6099
#: modules/bibcirculation/lib/bibcirculation_templates.py:7156
#: modules/bibcirculation/lib/bibcirculation_templates.py:7435
#: modules/bibcirculation/lib/bibcirculation_templates.py:8087
#: modules/bibcirculation/lib/bibcirculation_templates.py:8244
#: modules/bibcirculation/lib/bibcirculation_templates.py:9599
#: modules/bibcirculation/lib/bibcirculation_templates.py:9838
#: modules/bibcirculation/lib/bibcirculation_templates.py:10074
#: modules/bibcirculation/lib/bibcirculation_templates.py:10317
#: modules/bibcirculation/lib/bibcirculation_templates.py:10529
#: modules/bibcirculation/lib/bibcirculation_templates.py:10752
#: modules/bibcirculation/lib/bibcirculation_templates.py:11211
#: modules/bibcirculation/lib/bibcirculation_templates.py:11357
#: modules/bibcirculation/lib/bibcirculation_templates.py:11862
#: modules/bibcirculation/lib/bibcirculation_templates.py:11956
#: modules/bibcirculation/lib/bibcirculation_templates.py:12157
#: modules/bibcirculation/lib/bibcirculation_templates.py:12839
#: modules/bibcirculation/lib/bibcirculation_templates.py:12940
#: modules/bibcirculation/lib/bibcirculation_templates.py:13612
#: modules/bibcirculation/lib/bibcirculation_templates.py:13871
#: modules/bibcirculation/lib/bibcirculation_templates.py:14926
#: modules/bibcirculation/lib/bibcirculation_templates.py:15149
#: modules/bibcirculation/lib/bibcirculation_templates.py:15425
#: modules/bibcirculation/lib/bibcirculation_templates.py:16846
#: modules/bibcirculation/lib/bibcirculation_templates.py:17033
#: modules/bibcirculation/lib/bibcirculation_templates.py:17917
msgid "ISBN"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:455
#: modules/bibcirculation/lib/bibcirculation_templates.py:2389
#: modules/bibcirculation/lib/bibcirculation_templates.py:2506
#: modules/bibcirculation/lib/bibcirculation_templates.py:2738
#: modules/bibcirculation/lib/bibcirculation_templates.py:4456
#: modules/bibcirculation/lib/bibcirculation_templates.py:5604
#: modules/bibcirculation/lib/bibcirculation_templates.py:6189
#: modules/bibcirculation/lib/bibcirculation_templates.py:6238
#: modules/bibcirculation/lib/bibcirculation_templates.py:6535
#: modules/bibcirculation/lib/bibcirculation_templates.py:9027
#: modules/bibcirculation/lib/bibcirculation_templates.py:9272
#: modules/bibcirculation/lib/bibcirculation_templates.py:9881
#: modules/bibcirculation/lib/bibcirculation_templates.py:10359
#: modules/bibcirculation/lib/bibcirculation_templates.py:11223
#: modules/bibcirculation/lib/bibcirculation_templates.py:12212
#: modules/bibcirculation/lib/bibcirculation_templates.py:12996
#: modules/bibcirculation/lib/bibcirculation_templates.py:15517
msgid "Mailbox"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:456
#: modules/bibcirculation/lib/bibcirculation_templates.py:2388
#: modules/bibcirculation/lib/bibcirculation_templates.py:2505
#: modules/bibcirculation/lib/bibcirculation_templates.py:2737
#: modules/bibcirculation/lib/bibcirculation_templates.py:3941
#: modules/bibcirculation/lib/bibcirculation_templates.py:4044
#: modules/bibcirculation/lib/bibcirculation_templates.py:4267
#: modules/bibcirculation/lib/bibcirculation_templates.py:4328
#: modules/bibcirculation/lib/bibcirculation_templates.py:4455
#: modules/bibcirculation/lib/bibcirculation_templates.py:5603
#: modules/bibcirculation/lib/bibcirculation_templates.py:6188
#: modules/bibcirculation/lib/bibcirculation_templates.py:6237
#: modules/bibcirculation/lib/bibcirculation_templates.py:6534
#: modules/bibcirculation/lib/bibcirculation_templates.py:6597
#: modules/bibcirculation/lib/bibcirculation_templates.py:6702
#: modules/bibcirculation/lib/bibcirculation_templates.py:6931
#: modules/bibcirculation/lib/bibcirculation_templates.py:7030
#: modules/bibcirculation/lib/bibcirculation_templates.py:9026
#: modules/bibcirculation/lib/bibcirculation_templates.py:9271
#: modules/bibcirculation/lib/bibcirculation_templates.py:9880
#: modules/bibcirculation/lib/bibcirculation_templates.py:10358
#: modules/bibcirculation/lib/bibcirculation_templates.py:11222
#: modules/bibcirculation/lib/bibcirculation_templates.py:14071
#: modules/bibcirculation/lib/bibcirculation_templates.py:14143
#: modules/bibcirculation/lib/bibcirculation_templates.py:14392
#: modules/bibcirculation/lib/bibcirculation_templates.py:14463
#: modules/bibcirculation/lib/bibcirculation_templates.py:14714
#: modules/bibcirculation/lib/bibcirculation_templates.py:15516
msgid "Address"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:467
#: modules/bibcirculation/lib/bibcirculation_templates.py:358
#: modules/bibcirculation/lib/bibcirculation_templates.py:673
#: modules/bibcirculation/lib/bibcirculation_templates.py:2513
#: modules/bibcirculation/lib/bibcirculation_templates.py:3162
#: modules/bibcirculation/lib/bibcirculation_templates.py:3603
#: modules/bibcirculation/lib/bibcirculation_templates.py:3808
#: modules/bibcirculation/lib/bibcirculation_templates.py:4787
#: modules/bibcirculation/lib/bibcirculation_templates.py:4980
#: modules/bibcirculation/lib/bibcirculation_templates.py:5158
#: modules/bibcirculation/lib/bibcirculation_templates.py:5430
#: modules/bibcirculation/lib/bibcirculation_templates.py:7452
#: modules/bibcirculation/lib/bibcirculation_templates.py:8853
#: modules/bibcirculation/lib/bibcirculation_templates.py:9331
#: modules/bibcirculation/lib/bibcirculation_templates.py:10427
#: modules/bibcirculation/lib/bibcirculation_templates.py:11519
#: modules/bibcirculation/lib/bibcirculation_templates.py:12463
#: modules/bibcirculation/lib/bibcirculation_templates.py:12564
#: modules/bibcirculation/lib/bibcirculation_templates.py:13261
#: modules/bibcirculation/lib/bibcirculation_templates.py:13373
#: modules/bibcirculation/lib/bibcirculation_templates.py:15588
#: modules/bibcirculation/lib/bibcirculation_templates.py:17934
#: modules/bibcirculation/lib/bibcirculation_templates.py:18038
msgid "Due date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:513
msgid "List of pending hold requests"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:533
#: modules/bibcirculation/lib/bibcirculation_templates.py:1754
#: modules/bibcirculation/lib/bibcirculation_templates.py:2783
#: modules/bibcirculation/lib/bibcirculation_templates.py:2847
#: modules/bibcirculation/lib/bibcirculation_templates.py:2956
#: modules/bibcirculation/lib/bibcirculation_templates.py:3711
#: modules/bibcirculation/lib/bibcirculation_templates.py:3803
#: modules/bibcirculation/lib/bibcirculation_templates.py:4976
#: modules/bibcirculation/lib/bibcirculation_templates.py:5154
#: modules/bibcirculation/lib/bibcirculation_templates.py:5427
#: modules/bibcirculation/lib/bibcirculation_templates.py:11513
#: modules/bibcirculation/lib/bibcirculation_templates.py:11633
msgid "Borrower"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:534
#: modules/bibcirculation/lib/bibcirculation_templates.py:671
#: modules/bibcirculation/lib/bibcirculation_templates.py:769
#: modules/bibcirculation/lib/bibcirculation_templates.py:860
#: modules/bibcirculation/lib/bibcirculation_templates.py:1148
#: modules/bibcirculation/lib/bibcirculation_templates.py:1348
#: modules/bibcirculation/lib/bibcirculation_templates.py:1523
#: modules/bibcirculation/lib/bibcirculation_templates.py:1755
#: modules/bibcirculation/lib/bibcirculation_templates.py:1798
#: modules/bibcirculation/lib/bibcirculation_templates.py:2511
#: modules/bibcirculation/lib/bibcirculation_templates.py:2795
#: modules/bibcirculation/lib/bibcirculation_templates.py:2848
#: modules/bibcirculation/lib/bibcirculation_templates.py:3506
#: modules/bibcirculation/lib/bibcirculation_templates.py:3598
#: modules/bibcirculation/lib/bibcirculation_templates.py:4668
#: modules/bibcirculation/lib/bibcirculation_templates.py:4784
#: modules/bibcirculation/lib/bibcirculation_templates.py:4977
#: modules/bibcirculation/lib/bibcirculation_templates.py:5155
#: modules/bibcirculation/lib/bibcirculation_templates.py:5633
#: modules/bibcirculation/lib/bibcirculation_templates.py:10910
#: modules/bibcirculation/lib/bibcirculation_templates.py:11514
#: modules/bibcirculation/lib/bibcirculation_templates.py:11634
#: modules/bibcirculation/lib/bibcirculation_templates.py:15583
#: modules/bibcirculation/lib/bibcirculation_templates.py:15863
msgid "Item"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:535
#: modules/bibcirculation/lib/bibcirculation_templates.py:356
#: modules/bibcirculation/lib/bibcirculation_templates.py:1149
#: modules/bibcirculation/lib/bibcirculation_templates.py:1349
#: modules/bibcirculation/lib/bibcirculation_templates.py:2512
#: modules/bibcirculation/lib/bibcirculation_templates.py:2958
#: modules/bibcirculation/lib/bibcirculation_templates.py:3163
#: modules/bibcirculation/lib/bibcirculation_templates.py:3506
#: modules/bibcirculation/lib/bibcirculation_templates.py:3600
#: modules/bibcirculation/lib/bibcirculation_templates.py:3713
#: modules/bibcirculation/lib/bibcirculation_templates.py:3805
#: modules/bibcirculation/lib/bibcirculation_templates.py:4670
#: modules/bibcirculation/lib/bibcirculation_templates.py:7453
#: modules/bibcirculation/lib/bibcirculation_templates.py:7571
#: modules/bibcirculation/lib/bibcirculation_templates.py:7810
#: modules/bibcirculation/lib/bibcirculation_templates.py:8106
#: modules/bibcirculation/lib/bibcirculation_templates.py:8270
#: modules/bibcirculation/lib/bibcirculation_templates.py:8478
#: modules/bibcirculation/lib/bibcirculation_templates.py:9329
#: modules/bibcirculation/lib/bibcirculation_templates.py:10638
#: modules/bibcirculation/lib/bibcirculation_templates.py:10820
#: modules/bibcirculation/lib/bibcirculation_templates.py:12419
#: modules/bibcirculation/lib/bibcirculation_templates.py:12562
#: modules/bibcirculation/lib/bibcirculation_templates.py:12647
#: modules/bibcirculation/lib/bibcirculation_templates.py:13217
#: modules/bibcirculation/lib/bibcirculation_templates.py:13371
#: modules/bibcirculation/lib/bibcirculation_templates.py:13458
#: modules/bibcirculation/lib/bibcirculation_templates.py:15864
#: modules/bibcirculation/lib/bibcirculation_templates.py:17935
#: modules/bibcirculation/lib/bibcirculation_templates.py:18039
msgid "Library"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:536
#: modules/bibcirculation/lib/bibcirculation_templates.py:357
#: modules/bibcirculation/lib/bibcirculation_templates.py:1150
#: modules/bibcirculation/lib/bibcirculation_templates.py:1350
#: modules/bibcirculation/lib/bibcirculation_templates.py:2512
#: modules/bibcirculation/lib/bibcirculation_templates.py:2959
#: modules/bibcirculation/lib/bibcirculation_templates.py:3168
#: modules/bibcirculation/lib/bibcirculation_templates.py:3507
#: modules/bibcirculation/lib/bibcirculation_templates.py:3601
#: modules/bibcirculation/lib/bibcirculation_templates.py:3714
#: modules/bibcirculation/lib/bibcirculation_templates.py:3806
#: modules/bibcirculation/lib/bibcirculation_templates.py:4671
#: modules/bibcirculation/lib/bibcirculation_templates.py:6003
#: modules/bibcirculation/lib/bibcirculation_templates.py:7458
#: modules/bibcirculation/lib/bibcirculation_templates.py:7613
#: modules/bibcirculation/lib/bibcirculation_templates.py:7811
#: modules/bibcirculation/lib/bibcirculation_templates.py:8107
#: modules/bibcirculation/lib/bibcirculation_templates.py:8297
#: modules/bibcirculation/lib/bibcirculation_templates.py:8479
#: modules/bibcirculation/lib/bibcirculation_templates.py:9330
#: modules/bibcirculation/lib/bibcirculation_templates.py:15865
#: modules/bibcirculation/lib/bibcirculation_templates.py:17940
#: modules/bibcirculation/lib/bibcirculation_templates.py:18040
msgid "Location"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:537
#: modules/bibcirculation/lib/bibcirculation_templates.py:977
#: modules/bibcirculation/lib/bibcirculation_templates.py:1153
#: modules/bibcirculation/lib/bibcirculation_templates.py:1351
#: modules/bibcirculation/lib/bibcirculation_templates.py:1525
#: modules/bibcirculation/lib/bibcirculation_templates.py:1800
#: modules/bibcirculation/lib/bibcirculation_templates.py:2850
#: modules/bibcirculation/lib/bibcirculation_templates.py:2960
#: modules/bibcirculation/lib/bibcirculation_templates.py:3507
#: modules/bibcirculation/lib/bibcirculation_templates.py:3715
#: modules/bibcirculation/lib/bibcirculation_templates.py:4672
#: modules/bibcirculation/lib/bibcirculation_templates.py:5288
#: modules/bibcirculation/lib/bibcirculation_templates.py:15866
#: modules/bibcirculation/lib/bibcirculation_templates.py:17765
msgid "From"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:538
#: modules/bibcirculation/lib/bibcirculation_templates.py:977
#: modules/bibcirculation/lib/bibcirculation_templates.py:1154
#: modules/bibcirculation/lib/bibcirculation_templates.py:1352
#: modules/bibcirculation/lib/bibcirculation_templates.py:1526
#: modules/bibcirculation/lib/bibcirculation_templates.py:1801
#: modules/bibcirculation/lib/bibcirculation_templates.py:2851
#: modules/bibcirculation/lib/bibcirculation_templates.py:2961
#: modules/bibcirculation/lib/bibcirculation_templates.py:3508
#: modules/bibcirculation/lib/bibcirculation_templates.py:3716
#: modules/bibcirculation/lib/bibcirculation_templates.py:4673
#: modules/bibcirculation/lib/bibcirculation_templates.py:5290
#: modules/bibcirculation/lib/bibcirculation_templates.py:15867
#: modules/bibcirculation/lib/bibcirculation_templates.py:17766
#: modules/bibknowledge/lib/bibknowledge_templates.py:363
msgid "To"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_utils.py:539
#: modules/bibcirculation/lib/bibcirculation_templates.py:770
#: modules/bibcirculation/lib/bibcirculation_templates.py:1155
#: modules/bibcirculation/lib/bibcirculation_templates.py:1353
#: modules/bibcirculation/lib/bibcirculation_templates.py:1527
#: modules/bibcirculation/lib/bibcirculation_templates.py:1802
#: modules/bibcirculation/lib/bibcirculation_templates.py:2852
#: modules/bibcirculation/lib/bibcirculation_templates.py:2962
#: modules/bibcirculation/lib/bibcirculation_templates.py:3508
#: modules/bibcirculation/lib/bibcirculation_templates.py:3717
#: modules/bibcirculation/lib/bibcirculation_templates.py:4674
#: modules/bibcirculation/lib/bibcirculation_templates.py:12357
#: modules/bibcirculation/lib/bibcirculation_templates.py:12420
#: modules/bibcirculation/lib/bibcirculation_templates.py:12563
#: modules/bibcirculation/lib/bibcirculation_templates.py:12648
#: modules/bibcirculation/lib/bibcirculation_templates.py:13152
#: modules/bibcirculation/lib/bibcirculation_templates.py:13218
#: modules/bibcirculation/lib/bibcirculation_templates.py:13371
#: modules/bibcirculation/lib/bibcirculation_templates.py:13458
#: modules/bibcirculation/lib/bibcirculation_templates.py:15585
#: modules/bibcirculation/lib/bibcirculation_templates.py:15868
msgid "Request date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:117
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:162
msgid "You are not authorized to use loans."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:167
#: modules/bibcirculation/lib/bibcirculation_templates.py:633
msgid "Loans - historical overview"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:222
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:317
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:404
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:493
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:565
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:640
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:692
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:774
msgid "You are not authorized to use ill."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:232
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:340
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:426
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:723
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:792
#: modules/bibcirculation/lib/bibcirculation_templates.py:11346
msgid "Interlibrary loan request for books"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:503
msgid "Purchase request"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:570
msgid ""
"Payment method information is mandatory. Please, type your budget code or "
"tick the 'cash' checkbox."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:601
#: modules/bibcirculation/lib/bibcirculation_templates.py:206
msgid "Register purchase request"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:649
msgid "Interlibrary loan request for articles"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_webinterface.py:721
msgid "Wrong user id"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:110
msgid "Main navigation links"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:111
msgid "Loan"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:111
#: modules/bibcirculation/lib/bibcirculation_templates.py:4845
#: modules/bibcirculation/lib/bibcirculation_templates.py:5483
msgid "Return"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:112
#: modules/bibcirculation/lib/bibcirculation_templates.py:371
#: modules/bibcirculation/lib/bibcirculation_templates.py:9355
msgid "Request"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:112
msgid "Borrowers"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:154
#: modules/bibcirculation/lib/bibcirculation_templates.py:208
msgid "Lists"
-msgstr ""
+msgstr "سیاهه ها"
#: modules/bibcirculation/lib/bibcirculation_templates.py:155
msgid "Last loans"
-msgstr ""
+msgstr "آخرین امانت ها"
#: modules/bibcirculation/lib/bibcirculation_templates.py:156
msgid "Overdue loans"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:157
msgid "Items on shelf with holds"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:158
msgid "Items on loan with holds"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:159
msgid "Overdue loans with holds"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:160
msgid "Ordered books"
-msgstr ""
+msgstr "کتاب های سفارش شده"
#: modules/bibcirculation/lib/bibcirculation_templates.py:160
msgid "Others"
-msgstr ""
+msgstr "دیگران"
#: modules/bibcirculation/lib/bibcirculation_templates.py:161
#: modules/bibcirculation/lib/bibcirculation_templates.py:6836
#: modules/bibcirculation/lib/bibcirculation_templates.py:8643
msgid "Libraries"
-msgstr ""
+msgstr "کتابخانه ها"
#: modules/bibcirculation/lib/bibcirculation_templates.py:162
msgid "Add new library"
-msgstr ""
+msgstr "افزودن کتابخانه جدید"
#: modules/bibcirculation/lib/bibcirculation_templates.py:163
msgid "Update info"
-msgstr ""
+msgstr "بروز رسانی اطلاعات"
#: modules/bibcirculation/lib/bibcirculation_templates.py:164
msgid "Acquisitions"
-msgstr ""
+msgstr "فراهم آوری"
#: modules/bibcirculation/lib/bibcirculation_templates.py:165
msgid "List of ordered books"
-msgstr ""
+msgstr "سیاهه کتاب های سفارش شده"
#: modules/bibcirculation/lib/bibcirculation_templates.py:166
msgid "Order new book"
-msgstr ""
+msgstr "سفارش کتاب جدید"
#: modules/bibcirculation/lib/bibcirculation_templates.py:167
msgid "Vendors"
-msgstr ""
+msgstr "فروشندگان"
#: modules/bibcirculation/lib/bibcirculation_templates.py:168
msgid "Add new vendor"
-msgstr ""
+msgstr "افزودن فروشنده جدید"
#: modules/bibcirculation/lib/bibcirculation_templates.py:203
#: modules/bibcirculation/lib/bibcirculation_templates.py:4606
#: modules/bibcirculation/lib/bibcirculation_templates.py:4613
msgid "ILL"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:204
msgid "Register Book request"
-msgstr ""
+msgstr "ثبت درخواست کتاب"
#: modules/bibcirculation/lib/bibcirculation_templates.py:205
msgid "Register Article"
-msgstr ""
+msgstr "ثبت مقاله"
#: modules/bibcirculation/lib/bibcirculation_templates.py:209
msgid "Purchase"
-msgstr ""
+msgstr "خرید"
#: modules/bibcirculation/lib/bibcirculation_templates.py:216
msgid "Admin guide"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:217
msgid "Contact Support"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:267
msgid "This record does not exist."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:271
msgid "This record has no copies."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:278
msgid "Add a new copy"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:291
msgid "ILL services"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:295
#, python-format
msgid ""
"All the copies of %(strong_tag_open)s%(title)s%(strong_tag_close)s are "
"missing. You can request a copy using %(strong_tag_open)s%(ill_link)s"
"%(strong_tag_close)s"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:302
#: modules/bibcirculation/lib/bibcirculation_templates.py:9308
msgid "This item has no holdings."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:356
#: modules/bibcirculation/lib/bibcirculation_templates.py:1354
#: modules/bibcirculation/lib/bibcirculation_templates.py:11641
msgid "Options"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:357
#: modules/bibcirculation/lib/bibcirculation_templates.py:3173
#: modules/bibcirculation/lib/bibcirculation_templates.py:6010
#: modules/bibcirculation/lib/bibcirculation_templates.py:7463
#: modules/bibcirculation/lib/bibcirculation_templates.py:7661
#: modules/bibcirculation/lib/bibcirculation_templates.py:7814
#: modules/bibcirculation/lib/bibcirculation_templates.py:8108
#: modules/bibcirculation/lib/bibcirculation_templates.py:8328
#: modules/bibcirculation/lib/bibcirculation_templates.py:8482
#: modules/bibcirculation/lib/bibcirculation_templates.py:8855
#: modules/bibcirculation/lib/bibcirculation_templates.py:9330
#: modules/bibcirculation/lib/bibcirculation_templates.py:17945
#: modules/bibcirculation/lib/bibcirculation_templates.py:18041
msgid "Loan period"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:358
#: modules/bibcirculation/lib/bibcirculation_templates.py:1617
#: modules/bibcirculation/lib/bibcirculation_templates.py:2511
#: modules/bibcirculation/lib/bibcirculation_templates.py:3159
#: modules/bibcirculation/lib/bibcirculation_templates.py:3506
#: modules/bibcirculation/lib/bibcirculation_templates.py:3599
#: modules/bibcirculation/lib/bibcirculation_templates.py:3712
#: modules/bibcirculation/lib/bibcirculation_templates.py:3804
#: modules/bibcirculation/lib/bibcirculation_templates.py:4785
#: modules/bibcirculation/lib/bibcirculation_templates.py:4978
#: modules/bibcirculation/lib/bibcirculation_templates.py:5156
#: modules/bibcirculation/lib/bibcirculation_templates.py:5428
#: modules/bibcirculation/lib/bibcirculation_templates.py:5636
#: modules/bibcirculation/lib/bibcirculation_templates.py:6048
#: modules/bibcirculation/lib/bibcirculation_templates.py:7450
#: modules/bibcirculation/lib/bibcirculation_templates.py:7570
#: modules/bibcirculation/lib/bibcirculation_templates.py:7809
#: modules/bibcirculation/lib/bibcirculation_templates.py:8104
#: modules/bibcirculation/lib/bibcirculation_templates.py:8269
#: modules/bibcirculation/lib/bibcirculation_templates.py:8477
#: modules/bibcirculation/lib/bibcirculation_templates.py:8851
#: modules/bibcirculation/lib/bibcirculation_templates.py:9042
#: modules/bibcirculation/lib/bibcirculation_templates.py:9329
#: modules/bibcirculation/lib/bibcirculation_templates.py:9600
#: modules/bibcirculation/lib/bibcirculation_templates.py:9839
#: modules/bibcirculation/lib/bibcirculation_templates.py:10075
#: modules/bibcirculation/lib/bibcirculation_templates.py:10318
#: modules/bibcirculation/lib/bibcirculation_templates.py:10556
#: modules/bibcirculation/lib/bibcirculation_templates.py:10814
#: modules/bibcirculation/lib/bibcirculation_templates.py:12376
#: modules/bibcirculation/lib/bibcirculation_templates.py:12482
#: modules/bibcirculation/lib/bibcirculation_templates.py:12580
#: modules/bibcirculation/lib/bibcirculation_templates.py:12663
#: modules/bibcirculation/lib/bibcirculation_templates.py:17932
#: modules/bibcirculation/lib/bibcirculation_templates.py:18036
msgid "Barcode"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:402
msgid "See this book on BibCirculation"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:453
msgid "This item is not for loan."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:458
msgid "Server busy. Please, try again in a few seconds."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:463
msgid ""
"Your request has been registered and the document will be sent to you via "
"internal mail."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:468
msgid "Your request has been registered."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:473
#: modules/bibcirculation/lib/bibcirculation_templates.py:481
msgid "It is not possible to validate your request. "
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:474
#: modules/bibcirculation/lib/bibcirculation_templates.py:482
msgid "Your office address is not available. "
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:475
#: modules/bibcirculation/lib/bibcirculation_templates.py:483
#, python-format
msgid "Please contact %(librarian_email)s"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:489
msgid "Your purchase request has been registered."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:509
msgid "No messages to be displayed"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:539
#: modules/bibcirculation/lib/bibcirculation_templates.py:544
msgid "0 borrowers found."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:539
msgid "Search by CCID."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:543
msgid "Register new borrower."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:569
msgid "Borrower(s)"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:607
#: modules/bibcirculation/lib/bibcirculation_templates.py:894
#: modules/bibcirculation/lib/bibcirculation_templates.py:979
#: modules/bibcirculation/lib/bibcirculation_templates.py:1123
#: modules/bibcirculation/lib/bibcirculation_templates.py:1268
#: modules/bibcirculation/lib/bibcirculation_templates.py:1325
#: modules/bibcirculation/lib/bibcirculation_templates.py:1456
#: modules/bibcirculation/lib/bibcirculation_templates.py:1577
#: modules/bibcirculation/lib/bibcirculation_templates.py:1977
#: modules/bibcirculation/lib/bibcirculation_templates.py:2052
#: modules/bibcirculation/lib/bibcirculation_templates.py:2143
#: modules/bibcirculation/lib/bibcirculation_templates.py:2392
#: modules/bibcirculation/lib/bibcirculation_templates.py:2573
#: modules/bibcirculation/lib/bibcirculation_templates.py:2814
#: modules/bibcirculation/lib/bibcirculation_templates.py:2905
#: modules/bibcirculation/lib/bibcirculation_templates.py:3012
#: modules/bibcirculation/lib/bibcirculation_templates.py:3457
#: modules/bibcirculation/lib/bibcirculation_templates.py:3547
#: modules/bibcirculation/lib/bibcirculation_templates.py:3660
#: modules/bibcirculation/lib/bibcirculation_templates.py:3758
#: modules/bibcirculation/lib/bibcirculation_templates.py:3852
#: modules/bibcirculation/lib/bibcirculation_templates.py:3964
#: modules/bibcirculation/lib/bibcirculation_templates.py:4146
#: modules/bibcirculation/lib/bibcirculation_templates.py:4354
#: modules/bibcirculation/lib/bibcirculation_templates.py:4615
#: modules/bibcirculation/lib/bibcirculation_templates.py:4725
#: modules/bibcirculation/lib/bibcirculation_templates.py:4896
#: modules/bibcirculation/lib/bibcirculation_templates.py:4953
#: modules/bibcirculation/lib/bibcirculation_templates.py:5072
#: modules/bibcirculation/lib/bibcirculation_templates.py:5129
#: modules/bibcirculation/lib/bibcirculation_templates.py:5253
#: modules/bibcirculation/lib/bibcirculation_templates.py:5365
#: modules/bibcirculation/lib/bibcirculation_templates.py:5524
#: modules/bibcirculation/lib/bibcirculation_templates.py:5672
#: modules/bibcirculation/lib/bibcirculation_templates.py:5771
#: modules/bibcirculation/lib/bibcirculation_templates.py:5871
#: modules/bibcirculation/lib/bibcirculation_templates.py:6191
#: modules/bibcirculation/lib/bibcirculation_templates.py:6258
#: modules/bibcirculation/lib/bibcirculation_templates.py:6278
#: modules/bibcirculation/lib/bibcirculation_templates.py:6539
#: modules/bibcirculation/lib/bibcirculation_templates.py:6629
#: modules/bibcirculation/lib/bibcirculation_templates.py:6705
#: modules/bibcirculation/lib/bibcirculation_templates.py:6806
#: modules/bibcirculation/lib/bibcirculation_templates.py:6866
#: modules/bibcirculation/lib/bibcirculation_templates.py:6962
#: modules/bibcirculation/lib/bibcirculation_templates.py:7032
#: modules/bibcirculation/lib/bibcirculation_templates.py:7179
#: modules/bibcirculation/lib/bibcirculation_templates.py:7251
#: modules/bibcirculation/lib/bibcirculation_templates.py:7309
#: modules/bibcirculation/lib/bibcirculation_templates.py:7725
#: modules/bibcirculation/lib/bibcirculation_templates.py:7817
#: modules/bibcirculation/lib/bibcirculation_templates.py:7948
#: modules/bibcirculation/lib/bibcirculation_templates.py:8005
#: modules/bibcirculation/lib/bibcirculation_templates.py:8156
#: modules/bibcirculation/lib/bibcirculation_templates.py:8395
#: modules/bibcirculation/lib/bibcirculation_templates.py:8485
#: modules/bibcirculation/lib/bibcirculation_templates.py:8599
#: modules/bibcirculation/lib/bibcirculation_templates.py:8675
#: modules/bibcirculation/lib/bibcirculation_templates.py:8779
#: modules/bibcirculation/lib/bibcirculation_templates.py:8901
#: modules/bibcirculation/lib/bibcirculation_templates.py:9071
#: modules/bibcirculation/lib/bibcirculation_templates.py:9190
#: modules/bibcirculation/lib/bibcirculation_templates.py:9461
#: modules/bibcirculation/lib/bibcirculation_templates.py:9946
#: modules/bibcirculation/lib/bibcirculation_templates.py:10430
#: modules/bibcirculation/lib/bibcirculation_templates.py:10669
#: modules/bibcirculation/lib/bibcirculation_templates.py:10822
#: modules/bibcirculation/lib/bibcirculation_templates.py:11073
#: modules/bibcirculation/lib/bibcirculation_templates.py:11238
#: modules/bibcirculation/lib/bibcirculation_templates.py:11436
#: modules/bibcirculation/lib/bibcirculation_templates.py:12703
#: modules/bibcirculation/lib/bibcirculation_templates.py:13518
#: modules/bibcirculation/lib/bibcirculation_templates.py:13775
#: modules/bibcirculation/lib/bibcirculation_templates.py:13957
#: modules/bibcirculation/lib/bibcirculation_templates.py:14072
#: modules/bibcirculation/lib/bibcirculation_templates.py:14145
#: modules/bibcirculation/lib/bibcirculation_templates.py:14248
#: modules/bibcirculation/lib/bibcirculation_templates.py:14315
#: modules/bibcirculation/lib/bibcirculation_templates.py:14393
#: modules/bibcirculation/lib/bibcirculation_templates.py:14464
#: modules/bibcirculation/lib/bibcirculation_templates.py:14571
#: modules/bibcirculation/lib/bibcirculation_templates.py:14637
#: modules/bibcirculation/lib/bibcirculation_templates.py:14735
#: modules/bibcirculation/lib/bibcirculation_templates.py:14818
#: modules/bibcirculation/lib/bibcirculation_templates.py:15015
#: modules/bibcirculation/lib/bibcirculation_templates.py:15532
#: modules/bibcirculation/lib/bibcirculation_templates.py:15685
#: modules/bibcirculation/lib/bibcirculation_templates.py:15788
#: modules/bibcirculation/lib/bibcirculation_templates.py:16055
#: modules/bibcirculation/lib/bibcirculation_templates.py:16161
#: modules/bibcirculation/lib/bibcirculation_templates.py:16925
#: modules/bibcirculation/lib/bibcirculation_templates.py:17387
#: modules/bibcirculation/lib/bibcirculation_templates.py:17788
#: modules/bibcirculation/lib/bibcirculation_templates.py:18076
#: modules/bibknowledge/lib/bibknowledgeadmin.py:142
msgid "Back"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:628
#: modules/bibcirculation/lib/bibcirculation_templates.py:4879
msgid "Renew all loans"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:647
msgid "You don't have any book on loan."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:672
#: modules/bibcirculation/lib/bibcirculation_templates.py:3602
#: modules/bibcirculation/lib/bibcirculation_templates.py:3807
#: modules/bibcirculation/lib/bibcirculation_templates.py:4979
#: modules/bibcirculation/lib/bibcirculation_templates.py:5157
#: modules/bibcirculation/lib/bibcirculation_templates.py:5429
msgid "Loaned on"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:674
#: modules/bibcirculation/lib/bibcirculation_templates.py:772
msgid "Action(s)"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:686
#: modules/bibcirculation/lib/bibcirculation_templates.py:4844
#: modules/bibcirculation/lib/bibcirculation_templates.py:5482
msgid "Renew"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:739
#: modules/bibcirculation/lib/bibcirculation_templates.py:768
msgid "Your Requests"
-msgstr ""
+msgstr "درخواست های شما"
#: modules/bibcirculation/lib/bibcirculation_templates.py:740
msgid "You don't have any request (waiting or pending)."
msgstr ""
+"شما هیچ درخواستی (در حال انتظار یا در حال بررسی) ندارید.؟؟؟؟؟؟؟؟؟؟؟؟؟؟؟؟؟؟"
+"pending"
#: modules/bibcirculation/lib/bibcirculation_templates.py:742
#: modules/bibcirculation/lib/bibcirculation_templates.py:823
#: modules/bibcirculation/lib/bibcirculation_templates.py:1017
#: modules/bibcirculation/lib/bibcirculation_templates.py:1054
#: modules/bibcirculation/lib/bibcirculation_templates.py:1497
#: modules/bibcirculation/lib/bibcirculation_templates.py:2605
#: modules/bibcirculation/lib/bibcirculation_templates.py:2640
#: modules/bibcirculation/lib/bibcirculation_templates.py:2746
#: modules/bibcirculation/lib/bibcirculation_templates.py:6313
#: modules/bibcirculation/lib/bibcirculation_templates.py:6740
#: modules/bibcirculation/lib/bibcirculation_templates.py:7072
#: modules/bibcirculation/lib/bibcirculation_templates.py:7860
#: modules/bibcirculation/lib/bibcirculation_templates.py:8531
#: modules/bibcirculation/lib/bibcirculation_templates.py:9502
#: modules/bibcirculation/lib/bibcirculation_templates.py:9979
#: modules/bibcirculation/lib/bibcirculation_templates.py:10864
#: modules/bibcirculation/lib/bibcirculation_templates.py:11280
#: modules/bibcirculation/lib/bibcirculation_templates.py:11470
#: modules/bibcirculation/lib/bibcirculation_templates.py:13998
#: modules/bibcirculation/lib/bibcirculation_templates.py:14184
#: modules/bibcirculation/lib/bibcirculation_templates.py:14503
#: modules/bibcirculation/lib/bibcirculation_templates.py:15825
msgid "Back to home"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:861
msgid "Loaned"
-msgstr ""
+msgstr "امانت داده شده"
#: modules/bibcirculation/lib/bibcirculation_templates.py:862
msgid "Returned"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:863
msgid "Renewalls"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:976
msgid "Enter your period of interest"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:979
#: modules/bibcirculation/lib/bibcirculation_templates.py:2815
#: modules/bibcirculation/lib/bibcirculation_templates.py:4361
#: modules/bibcirculation/lib/bibcirculation_templates.py:5673
#: modules/bibcirculation/lib/bibcirculation_templates.py:5772
#: modules/bibcirculation/lib/bibcirculation_templates.py:5873
#: modules/bibcirculation/lib/bibcirculation_templates.py:6705
#: modules/bibcirculation/lib/bibcirculation_templates.py:8485
#: modules/bibcirculation/lib/bibcirculation_templates.py:8780
#: modules/bibcirculation/lib/bibcirculation_templates.py:9072
#: modules/bibcirculation/lib/bibcirculation_templates.py:9461
#: modules/bibcirculation/lib/bibcirculation_templates.py:11074
#: modules/bibcirculation/lib/bibcirculation_templates.py:14145
#: modules/bibcirculation/lib/bibcirculation_templates.py:14782
#: modules/bibcirculation/lib/bibcirculation_templates.py:15789
msgid "Confirm"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1013
#, python-format
msgid "You can see your loans %(x_url_open)shere%(x_url_close)s."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1052
msgid "A new loan has been registered."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1100
msgid "Delete this request?"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1101
#: modules/bibcirculation/lib/bibcirculation_templates.py:1368
msgid "Request not deleted."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1122
#: modules/bibcirculation/lib/bibcirculation_templates.py:1324
msgid "No more requests are pending."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1151
msgid "Vol."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1152
msgid "Ed."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1156
#: modules/bibcirculation/lib/bibcirculation_templates.py:3188
#: modules/bibcirculation/lib/bibcirculation_templates.py:15869
msgid "Actions"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1222
#: modules/bibcirculation/lib/bibcirculation_templates.py:1417
#: modules/bibcirculation/lib/bibcirculation_templates.py:15939
msgid "Associate barcode"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1496
msgid "No hold requests waiting."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1524
#: modules/bibcirculation/lib/bibcirculation_templates.py:1799
#: modules/bibcirculation/lib/bibcirculation_templates.py:4669
msgid "Request status"
-msgstr ""
+msgstr "وضعیت درخواست"
#: modules/bibcirculation/lib/bibcirculation_templates.py:1528
#: modules/bibcirculation/lib/bibcirculation_templates.py:1803
msgid "Request options"
-msgstr ""
+msgstr "گزینه های درخواست"
#: modules/bibcirculation/lib/bibcirculation_templates.py:1558
msgid "Select hold request"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1634
#: modules/bibcirculation/lib/bibcirculation_templates.py:5366
msgid "Reset"
-msgstr ""
+msgstr "بازتنطیم"
#: modules/bibcirculation/lib/bibcirculation_templates.py:1680
#, python-format
msgid ""
"The item %(x_strong_tag_open)s%(x_title)s%(x_strong_tag_close)s, with "
"barcode %(x_strong_tag_open)s%(x_barcode)s%(x_strong_tag_close)s, has been "
"returned with success."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1694
#, python-format
msgid ""
"There are %(x_strong_tag_open)s%(x_number_of_requests)s requests"
"%(x_strong_tag_close)s on the book that has been returned."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1753
msgid "Loan informations"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1758
#: modules/bibcirculation/lib/bibcirculation_templates.py:2102
#: modules/bibcirculation/lib/bibcirculation_templates.py:2744
#: modules/bibcirculation/lib/bibcirculation_templates.py:3097
#: modules/bibcirculation/lib/bibcirculation_templates.py:5996
#: modules/bibcirculation/lib/bibcirculation_templates.py:7153
#: modules/bibcirculation/lib/bibcirculation_templates.py:7433
#: modules/bibcirculation/lib/bibcirculation_templates.py:8085
#: modules/bibcirculation/lib/bibcirculation_templates.py:8242
#: modules/bibcirculation/lib/bibcirculation_templates.py:9598
#: modules/bibcirculation/lib/bibcirculation_templates.py:9837
#: modules/bibcirculation/lib/bibcirculation_templates.py:10073
#: modules/bibcirculation/lib/bibcirculation_templates.py:10316
#: modules/bibcirculation/lib/bibcirculation_templates.py:10527
#: modules/bibcirculation/lib/bibcirculation_templates.py:10751
#: modules/bibcirculation/lib/bibcirculation_templates.py:11210
#: modules/bibcirculation/lib/bibcirculation_templates.py:11355
#: modules/bibcirculation/lib/bibcirculation_templates.py:11860
#: modules/bibcirculation/lib/bibcirculation_templates.py:11953
#: modules/bibcirculation/lib/bibcirculation_templates.py:12070
#: modules/bibcirculation/lib/bibcirculation_templates.py:12154
#: modules/bibcirculation/lib/bibcirculation_templates.py:12837
#: modules/bibcirculation/lib/bibcirculation_templates.py:12937
#: modules/bibcirculation/lib/bibcirculation_templates.py:13610
#: modules/bibcirculation/lib/bibcirculation_templates.py:13870
#: modules/bibcirculation/lib/bibcirculation_templates.py:14923
#: modules/bibcirculation/lib/bibcirculation_templates.py:15147
#: modules/bibcirculation/lib/bibcirculation_templates.py:15423
#: modules/bibcirculation/lib/bibcirculation_templates.py:16119
#: modules/bibcirculation/lib/bibcirculation_templates.py:16842
#: modules/bibcirculation/lib/bibcirculation_templates.py:17030
#: modules/bibcirculation/lib/bibcirculation_templates.py:17915
msgid "Publisher"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1760
#: modules/bibcirculation/lib/bibcirculation_templates.py:12565
#: modules/bibcirculation/lib/bibcirculation_templates.py:13373
msgid "Return date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1796
msgid "Waiting requests"
-msgstr ""
+msgstr "درخواست های در حال انتظار"
#: modules/bibcirculation/lib/bibcirculation_templates.py:1838
msgid "Select request"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1878
msgid "Welcome to Invenio BibCirculation Admin"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1904
#: modules/bibcirculation/lib/bibcirculation_templates.py:2207
#: modules/bibcirculation/lib/bibcirculation_templates.py:2214
#: modules/bibcirculation/lib/bibcirculation_templates.py:2221
#: modules/bibcirculation/lib/bibcirculation_templates.py:10128
#: modules/bibcirculation/lib/bibcirculation_templates.py:10135
#: modules/bibcirculation/lib/bibcirculation_templates.py:10142
#: modules/bibcirculation/lib/bibcirculation_templates.py:15206
#: modules/bibcirculation/lib/bibcirculation_templates.py:15213
#: modules/bibcirculation/lib/bibcirculation_templates.py:15220
#: modules/bibcirculation/lib/bibcirculation_templates.py:17087
#: modules/bibcirculation/lib/bibcirculation_templates.py:17094
#: modules/bibcirculation/lib/bibcirculation_templates.py:17101
#: modules/bibcirculation/lib/bibcirculation_templates.py:17569
#: modules/bibcirculation/lib/bibcirculation_templates.py:17576
#: modules/bibcirculation/lib/bibcirculation_templates.py:17583
msgid "id"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1917
msgid "register new borrower"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1948
#: modules/bibcirculation/lib/bibcirculation_templates.py:2200
#: modules/bibcirculation/lib/bibcirculation_templates.py:9646
#: modules/bibcirculation/lib/bibcirculation_templates.py:15199
#: modules/bibcirculation/lib/bibcirculation_templates.py:17080
#: modules/bibcirculation/lib/bibcirculation_templates.py:17562
msgid "Search borrower by"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1949
#: modules/bibcirculation/lib/bibcirculation_templates.py:2180
#: modules/bibcirculation/lib/bibcirculation_templates.py:2187
#: modules/bibcirculation/lib/bibcirculation_templates.py:2194
#: modules/bibcirculation/lib/bibcirculation_templates.py:2207
#: modules/bibcirculation/lib/bibcirculation_templates.py:2214
#: modules/bibcirculation/lib/bibcirculation_templates.py:2221
#: modules/bibcirculation/lib/bibcirculation_templates.py:4088
#: modules/bibcirculation/lib/bibcirculation_templates.py:6776
#: modules/bibcirculation/lib/bibcirculation_templates.py:8570
#: modules/bibcirculation/lib/bibcirculation_templates.py:9626
#: modules/bibcirculation/lib/bibcirculation_templates.py:9633
#: modules/bibcirculation/lib/bibcirculation_templates.py:9640
#: modules/bibcirculation/lib/bibcirculation_templates.py:9653
#: modules/bibcirculation/lib/bibcirculation_templates.py:9660
#: modules/bibcirculation/lib/bibcirculation_templates.py:9667
#: modules/bibcirculation/lib/bibcirculation_templates.py:10101
#: modules/bibcirculation/lib/bibcirculation_templates.py:10108
#: modules/bibcirculation/lib/bibcirculation_templates.py:10115
#: modules/bibcirculation/lib/bibcirculation_templates.py:10128
#: modules/bibcirculation/lib/bibcirculation_templates.py:10135
#: modules/bibcirculation/lib/bibcirculation_templates.py:10142
#: modules/bibcirculation/lib/bibcirculation_templates.py:14221
#: modules/bibcirculation/lib/bibcirculation_templates.py:14540
#: modules/bibcirculation/lib/bibcirculation_templates.py:15179
#: modules/bibcirculation/lib/bibcirculation_templates.py:15186
#: modules/bibcirculation/lib/bibcirculation_templates.py:15193
#: modules/bibcirculation/lib/bibcirculation_templates.py:15206
#: modules/bibcirculation/lib/bibcirculation_templates.py:15213
#: modules/bibcirculation/lib/bibcirculation_templates.py:15220
#: modules/bibcirculation/lib/bibcirculation_templates.py:17060
#: modules/bibcirculation/lib/bibcirculation_templates.py:17067
#: modules/bibcirculation/lib/bibcirculation_templates.py:17074
#: modules/bibcirculation/lib/bibcirculation_templates.py:17087
#: modules/bibcirculation/lib/bibcirculation_templates.py:17094
#: modules/bibcirculation/lib/bibcirculation_templates.py:17101
#: modules/bibcirculation/lib/bibcirculation_templates.py:17542
#: modules/bibcirculation/lib/bibcirculation_templates.py:17549
#: modules/bibcirculation/lib/bibcirculation_templates.py:17556
#: modules/bibcirculation/lib/bibcirculation_templates.py:17569
#: modules/bibcirculation/lib/bibcirculation_templates.py:17576
#: modules/bibcirculation/lib/bibcirculation_templates.py:17583
msgid "name"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:1949
#: modules/bibcirculation/lib/bibcirculation_templates.py:2180
#: modules/bibcirculation/lib/bibcirculation_templates.py:2187
#: modules/bibcirculation/lib/bibcirculation_templates.py:2194
#: modules/bibcirculation/lib/bibcirculation_templates.py:2207
#: modules/bibcirculation/lib/bibcirculation_templates.py:2214
#: modules/bibcirculation/lib/bibcirculation_templates.py:2221
#: modules/bibcirculation/lib/bibcirculation_templates.py:4088
#: modules/bibcirculation/lib/bibcirculation_templates.py:6777
#: modules/bibcirculation/lib/bibcirculation_templates.py:8570
#: modules/bibcirculation/lib/bibcirculation_templates.py:9626
#: modules/bibcirculation/lib/bibcirculation_templates.py:9633
#: modules/bibcirculation/lib/bibcirculation_templates.py:9640
#: modules/bibcirculation/lib/bibcirculation_templates.py:9653
#: modules/bibcirculation/lib/bibcirculation_templates.py:9660
#: modules/bibcirculation/lib/bibcirculation_templates.py:9667
#: modules/bibcirculation/lib/bibcirculation_templates.py:10101
#: modules/bibcirculation/lib/bibcirculation_templates.py:10108
#: modules/bibcirculation/lib/bibcirculation_templates.py:10115
#: modules/bibcirculation/lib/bibcirculation_templates.py:10128
#: modules/bibcirculation/lib/bibcirculation_templates.py:10135
#: modules/bibcirculation/lib/bibcirculation_templates.py:10142
#: modules/bibcirculation/lib/bibcirculation_templates.py:14222
#: modules/bibcirculation/lib/bibcirculation_templates.py:14541
#: modules/bibcirculation/lib/bibcirculation_templates.py:15179
#: modules/bibcirculation/lib/bibcirculation_templates.py:15186
#: modules/bibcirculation/lib/bibcirculation_templates.py:15193
#: modules/bibcirculation/lib/bibcirculation_templates.py:15206
#: modules/bibcirculation/lib/bibcirculation_templates.py:15213
#: modules/bibcirculation/lib/bibcirculation_templates.py:15220
#: modules/bibcirculation/lib/bibcirculation_templates.py:17060
#: modules/bibcirculation/lib/bibcirculation_templates.py:17067
#: modules/bibcirculation/lib/bibcirculation_templates.py:17074
#: modules/bibcirculation/lib/bibcirculation_templates.py:17087
#: modules/bibcirculation/lib/bibcirculation_templates.py:17094
#: modules/bibcirculation/lib/bibcirculation_templates.py:17101
#: modules/bibcirculation/lib/bibcirculation_templates.py:17542
#: modules/bibcirculation/lib/bibcirculation_templates.py:17549
#: modules/bibcirculation/lib/bibcirculation_templates.py:17556
#: modules/bibcirculation/lib/bibcirculation_templates.py:17569
#: modules/bibcirculation/lib/bibcirculation_templates.py:17576
#: modules/bibcirculation/lib/bibcirculation_templates.py:17583
msgid "email"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2019
#: modules/bibcirculation/lib/bibcirculation_templates.py:7218
#: modules/bibcirculation/lib/bibcirculation_templates.py:7917
#: modules/bibcirculation/lib/bibcirculation_templates.py:9132
#: modules/bibcirculation/lib/bibcirculation_templates.py:16024
msgid "Search item by"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2019
#: modules/bibcirculation/lib/bibcirculation_templates.py:9140
#: modules/bibcirculation/lib/bibcirculation_templates.py:9148
#: modules/bibcirculation/lib/bibcirculation_templates.py:9156
#: modules/bibcirculation/lib/bibcirculation_templates.py:9164
#: modules/bibcirculation/lib/bibcirculation_templates.py:16024
msgid "barcode"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2019
msgid "recid"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2076
msgid "0 item(s) found."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2101
#, python-format
msgid "%i items found."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2103
#: modules/bibcirculation/lib/bibcirculation_templates.py:16120
msgid "# copies"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2173
#: modules/bibcirculation/lib/bibcirculation_templates.py:9619
#: modules/bibcirculation/lib/bibcirculation_templates.py:10094
#: modules/bibcirculation/lib/bibcirculation_templates.py:10121
#: modules/bibcirculation/lib/bibcirculation_templates.py:15172
#: modules/bibcirculation/lib/bibcirculation_templates.py:17053
#: modules/bibcirculation/lib/bibcirculation_templates.py:17535
msgid "Search user by"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2279
#: modules/bibcirculation/lib/bibcirculation_templates.py:9725
#: modules/bibcirculation/lib/bibcirculation_templates.py:10212
#: modules/bibcirculation/lib/bibcirculation_templates.py:15294
#: modules/bibcirculation/lib/bibcirculation_templates.py:17176
#: modules/bibcirculation/lib/bibcirculation_templates.py:17654
msgid "Select user"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2304
#: modules/bibcirculation/lib/bibcirculation_templates.py:2417
#: modules/bibcirculation/lib/bibcirculation_templates.py:2661
#: modules/bibcirculation/lib/bibcirculation_templates.py:4393
#: modules/bibcirculation/lib/bibcirculation_templates.py:5554
#: modules/bibcirculation/lib/bibcirculation_templates.py:8979
#: modules/bibcirculation/lib/bibcirculation_templates.py:9112
#: modules/bibcirculation/lib/bibcirculation_templates.py:11104
#: modules/bibcirculation/lib/bibcirculation_templates.py:15345
msgid "CCID"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2321
#: modules/bibcirculation/lib/bibcirculation_templates.py:2502
msgid "User information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2391
#: modules/bibcirculation/lib/bibcirculation_templates.py:2508
#: modules/bibcirculation/lib/bibcirculation_templates.py:2740
#: modules/bibcirculation/lib/bibcirculation_templates.py:3943
#: modules/bibcirculation/lib/bibcirculation_templates.py:4046
#: modules/bibcirculation/lib/bibcirculation_templates.py:4269
#: modules/bibcirculation/lib/bibcirculation_templates.py:4330
#: modules/bibcirculation/lib/bibcirculation_templates.py:4458
#: modules/bibcirculation/lib/bibcirculation_templates.py:5606
#: modules/bibcirculation/lib/bibcirculation_templates.py:6187
#: modules/bibcirculation/lib/bibcirculation_templates.py:6236
#: modules/bibcirculation/lib/bibcirculation_templates.py:6537
#: modules/bibcirculation/lib/bibcirculation_templates.py:6597
#: modules/bibcirculation/lib/bibcirculation_templates.py:6701
#: modules/bibcirculation/lib/bibcirculation_templates.py:6930
#: modules/bibcirculation/lib/bibcirculation_templates.py:7029
#: modules/bibcirculation/lib/bibcirculation_templates.py:9029
#: modules/bibcirculation/lib/bibcirculation_templates.py:9274
#: modules/bibcirculation/lib/bibcirculation_templates.py:9883
#: modules/bibcirculation/lib/bibcirculation_templates.py:10361
#: modules/bibcirculation/lib/bibcirculation_templates.py:11225
#: modules/bibcirculation/lib/bibcirculation_templates.py:14071
#: modules/bibcirculation/lib/bibcirculation_templates.py:14142
#: modules/bibcirculation/lib/bibcirculation_templates.py:14391
#: modules/bibcirculation/lib/bibcirculation_templates.py:14462
#: modules/bibcirculation/lib/bibcirculation_templates.py:14716
#: modules/bibcirculation/lib/bibcirculation_templates.py:15519
msgid "Phone"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2392
msgid "Barcode(s)"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2393
#: modules/bibcirculation/lib/bibcirculation_templates.py:2573
#: modules/bibcirculation/lib/bibcirculation_templates.py:6191
#: modules/bibcirculation/lib/bibcirculation_templates.py:6278
#: modules/bibcirculation/lib/bibcirculation_templates.py:6539
#: modules/bibcirculation/lib/bibcirculation_templates.py:6629
#: modules/bibcirculation/lib/bibcirculation_templates.py:6962
#: modules/bibcirculation/lib/bibcirculation_templates.py:7032
#: modules/bibcirculation/lib/bibcirculation_templates.py:7179
#: modules/bibcirculation/lib/bibcirculation_templates.py:7726
#: modules/bibcirculation/lib/bibcirculation_templates.py:7817
#: modules/bibcirculation/lib/bibcirculation_templates.py:8395
#: modules/bibcirculation/lib/bibcirculation_templates.py:9946
#: modules/bibcirculation/lib/bibcirculation_templates.py:10430
#: modules/bibcirculation/lib/bibcirculation_templates.py:10669
#: modules/bibcirculation/lib/bibcirculation_templates.py:10822
#: modules/bibcirculation/lib/bibcirculation_templates.py:11238
#: modules/bibcirculation/lib/bibcirculation_templates.py:11436
#: modules/bibcirculation/lib/bibcirculation_templates.py:12703
#: modules/bibcirculation/lib/bibcirculation_templates.py:13518
#: modules/bibcirculation/lib/bibcirculation_templates.py:13775
#: modules/bibcirculation/lib/bibcirculation_templates.py:13957
#: modules/bibcirculation/lib/bibcirculation_templates.py:14072
#: modules/bibcirculation/lib/bibcirculation_templates.py:14393
#: modules/bibcirculation/lib/bibcirculation_templates.py:14464
#: modules/bibcirculation/lib/bibcirculation_templates.py:15015
#: modules/bibcirculation/lib/bibcirculation_templates.py:15532
#: modules/bibcirculation/lib/bibcirculation_templates.py:16925
#: modules/bibcirculation/lib/bibcirculation_templates.py:17387
msgid "Continue"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2509
msgid "List of borrowed books"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2513
msgid "Write note(s)"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2639
msgid "Notification has been sent!"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2742
#: modules/bibcirculation/lib/bibcirculation_templates.py:3095
#: modules/bibcirculation/lib/bibcirculation_templates.py:7151
#: modules/bibcirculation/lib/bibcirculation_templates.py:7429
#: modules/bibcirculation/lib/bibcirculation_templates.py:8081
#: modules/bibcirculation/lib/bibcirculation_templates.py:8238
#: modules/bibcirculation/lib/bibcirculation_templates.py:9596
#: modules/bibcirculation/lib/bibcirculation_templates.py:9835
#: modules/bibcirculation/lib/bibcirculation_templates.py:10071
#: modules/bibcirculation/lib/bibcirculation_templates.py:10314
#: modules/bibcirculation/lib/bibcirculation_templates.py:10523
#: modules/bibcirculation/lib/bibcirculation_templates.py:10749
#: modules/bibcirculation/lib/bibcirculation_templates.py:11208
#: modules/bibcirculation/lib/bibcirculation_templates.py:11351
#: modules/bibcirculation/lib/bibcirculation_templates.py:11856
#: modules/bibcirculation/lib/bibcirculation_templates.py:11951
#: modules/bibcirculation/lib/bibcirculation_templates.py:12064
#: modules/bibcirculation/lib/bibcirculation_templates.py:12152
#: modules/bibcirculation/lib/bibcirculation_templates.py:12833
#: modules/bibcirculation/lib/bibcirculation_templates.py:12935
#: modules/bibcirculation/lib/bibcirculation_templates.py:13606
#: modules/bibcirculation/lib/bibcirculation_templates.py:13868
#: modules/bibcirculation/lib/bibcirculation_templates.py:14921
#: modules/bibcirculation/lib/bibcirculation_templates.py:15144
#: modules/bibcirculation/lib/bibcirculation_templates.py:15420
#: modules/bibcirculation/lib/bibcirculation_templates.py:16840
#: modules/bibcirculation/lib/bibcirculation_templates.py:17028
#: modules/bibcirculation/lib/bibcirculation_templates.py:17311
#: modules/bibcirculation/lib/bibcirculation_templates.py:17509
#: modules/bibcirculation/lib/bibcirculation_templates.py:17911
msgid "Author(s)"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2748
msgid "Print loan information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2853
#: modules/bibcirculation/lib/bibcirculation_templates.py:2963
#: modules/bibcirculation/lib/bibcirculation_templates.py:10917
#: modules/bibcirculation/lib/bibcirculation_templates.py:11521
msgid "Option(s)"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2889
#: modules/bibcirculation/lib/bibcirculation_templates.py:2990
msgid "Cancel hold request"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:2924
#: modules/bibcirculation/lib/bibcirculation_templates.py:3479
#: modules/bibcirculation/lib/bibcirculation_templates.py:3682
#: modules/bibcirculation/lib/bibcirculation_templates.py:4637
msgid "There are no requests."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3093
#: modules/bibcirculation/lib/bibcirculation_templates.py:7149
#: modules/bibcirculation/lib/bibcirculation_templates.py:7426
#: modules/bibcirculation/lib/bibcirculation_templates.py:8078
#: modules/bibcirculation/lib/bibcirculation_templates.py:8235
#: modules/bibcirculation/lib/bibcirculation_templates.py:9594
#: modules/bibcirculation/lib/bibcirculation_templates.py:9833
#: modules/bibcirculation/lib/bibcirculation_templates.py:10069
#: modules/bibcirculation/lib/bibcirculation_templates.py:10312
#: modules/bibcirculation/lib/bibcirculation_templates.py:10519
#: modules/bibcirculation/lib/bibcirculation_templates.py:10747
#: modules/bibcirculation/lib/bibcirculation_templates.py:11206
#: modules/bibcirculation/lib/bibcirculation_templates.py:11347
#: modules/bibcirculation/lib/bibcirculation_templates.py:11853
#: modules/bibcirculation/lib/bibcirculation_templates.py:11949
#: modules/bibcirculation/lib/bibcirculation_templates.py:12061
#: modules/bibcirculation/lib/bibcirculation_templates.py:12150
#: modules/bibcirculation/lib/bibcirculation_templates.py:12830
#: modules/bibcirculation/lib/bibcirculation_templates.py:12933
#: modules/bibcirculation/lib/bibcirculation_templates.py:13602
#: modules/bibcirculation/lib/bibcirculation_templates.py:13866
#: modules/bibcirculation/lib/bibcirculation_templates.py:14864
#: modules/bibcirculation/lib/bibcirculation_templates.py:15142
#: modules/bibcirculation/lib/bibcirculation_templates.py:15418
#: modules/bibcirculation/lib/bibcirculation_templates.py:17025
#: modules/bibcirculation/lib/bibcirculation_templates.py:17506
#: modules/bibcirculation/lib/bibcirculation_templates.py:17908
msgid "Item details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3100
msgid "Edit this record"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3101
msgid "Book Cover"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3102
msgid "Additional details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3174
#: modules/bibcirculation/lib/bibcirculation_templates.py:7464
#: modules/bibcirculation/lib/bibcirculation_templates.py:8109
#: modules/bibcirculation/lib/bibcirculation_templates.py:17946
#: modules/bibcirculation/lib/bibcirculation_templates.py:18042
msgid "No of loans"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3253
#: modules/bibcirculation/lib/bibcirculation_templates.py:3274
#: modules/bibcirculation/lib/bibcirculation_templates.py:3295
#: modules/bibcirculation/lib/bibcirculation_templates.py:3316
#: modules/bibcirculation/lib/bibcirculation_templates.py:4843
#: modules/bibcirculation/lib/bibcirculation_templates.py:5481
msgid "Select an action"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3255
#: modules/bibcirculation/lib/bibcirculation_templates.py:3276
#: modules/bibcirculation/lib/bibcirculation_templates.py:3297
#: modules/bibcirculation/lib/bibcirculation_templates.py:3318
msgid "Add similar copy"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3256
#: modules/bibcirculation/lib/bibcirculation_templates.py:3277
#: modules/bibcirculation/lib/bibcirculation_templates.py:3298
#: modules/bibcirculation/lib/bibcirculation_templates.py:3319
#: modules/bibcirculation/lib/bibcirculation_templates.py:4490
msgid "New request"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3257
#: modules/bibcirculation/lib/bibcirculation_templates.py:3278
#: modules/bibcirculation/lib/bibcirculation_templates.py:3299
#: modules/bibcirculation/lib/bibcirculation_templates.py:3320
#: modules/bibcirculation/lib/bibcirculation_templates.py:4489
msgid "New loan"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3258
#: modules/bibcirculation/lib/bibcirculation_templates.py:3279
#: modules/bibcirculation/lib/bibcirculation_templates.py:3300
#: modules/bibcirculation/lib/bibcirculation_templates.py:3321
msgid "Delete copy"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3432
msgid "Add new copy"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3433
msgid "Order new copy"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3434
msgid "ILL request"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3435
#, python-format
msgid "Hold requests and loans overview on %(date)s"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3437
#: modules/bibcirculation/lib/bibcirculation_templates.py:3439
msgid "Hold requests"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3437
#: modules/bibcirculation/lib/bibcirculation_templates.py:3438
#: modules/bibcirculation/lib/bibcirculation_templates.py:3440
#: modules/bibcirculation/lib/bibcirculation_templates.py:3441
#: modules/bibcirculation/lib/bibcirculation_templates.py:4603
#: modules/bibcirculation/lib/bibcirculation_templates.py:4605
#: modules/bibcirculation/lib/bibcirculation_templates.py:4607
#: modules/bibcirculation/lib/bibcirculation_templates.py:4610
#: modules/bibcirculation/lib/bibcirculation_templates.py:4612
#: modules/bibcirculation/lib/bibcirculation_templates.py:4614
msgid "More details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3438
#: modules/bibcirculation/lib/bibcirculation_templates.py:3440
#: modules/bibcirculation/lib/bibcirculation_templates.py:4604
#: modules/bibcirculation/lib/bibcirculation_templates.py:4611
msgid "Loans"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3439
#: modules/bibcirculation/lib/bibcirculation_templates.py:4608
msgid "Historical overview"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3569
#: modules/bibcirculation/lib/bibcirculation_templates.py:4747
#: modules/bibcirculation/lib/bibcirculation_templates.py:5389
msgid "There are no loans."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3604
#: modules/bibcirculation/lib/bibcirculation_templates.py:3809
msgid "Returned on"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3605
#: modules/bibcirculation/lib/bibcirculation_templates.py:3810
#: modules/bibcirculation/lib/bibcirculation_templates.py:4788
#: modules/bibcirculation/lib/bibcirculation_templates.py:4981
#: modules/bibcirculation/lib/bibcirculation_templates.py:5159
#: modules/bibcirculation/lib/bibcirculation_templates.py:5431
msgid "Renewals"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3606
#: modules/bibcirculation/lib/bibcirculation_templates.py:3811
#: modules/bibcirculation/lib/bibcirculation_templates.py:4789
#: modules/bibcirculation/lib/bibcirculation_templates.py:4982
#: modules/bibcirculation/lib/bibcirculation_templates.py:5160
msgid "Overdue letters"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3878
#: modules/bibcirculation/lib/bibcirculation_templates.py:3988
#: modules/bibcirculation/lib/bibcirculation_templates.py:4182
#: modules/bibcirculation/lib/bibcirculation_templates.py:4197
#: modules/bibcirculation/lib/bibcirculation_templates.py:4398
#: modules/bibcirculation/lib/bibcirculation_templates.py:4808
#: modules/bibcirculation/lib/bibcirculation_templates.py:5449
#: modules/bibcirculation/lib/bibcirculation_templates.py:10933
#: modules/bibcirculation/lib/bibcirculation_templates.py:14664
#: modules/bibcirculation/lib/bibcirculation_templates.py:15641
msgid "No notes"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3883
#: modules/bibcirculation/lib/bibcirculation_templates.py:3993
#: modules/bibcirculation/lib/bibcirculation_templates.py:4187
#: modules/bibcirculation/lib/bibcirculation_templates.py:4202
msgid "Notes about this library"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3939
msgid "Library details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3944
#: modules/bibcirculation/lib/bibcirculation_templates.py:4047
#: modules/bibcirculation/lib/bibcirculation_templates.py:4270
#: modules/bibcirculation/lib/bibcirculation_templates.py:4331
#: modules/bibcirculation/lib/bibcirculation_templates.py:4790
#: modules/bibcirculation/lib/bibcirculation_templates.py:6597
#: modules/bibcirculation/lib/bibcirculation_templates.py:6703
#: modules/bibcirculation/lib/bibcirculation_templates.py:6932
#: modules/bibcirculation/lib/bibcirculation_templates.py:7031
#: modules/bibcirculation/lib/bibcirculation_templates.py:11520
#: modules/bibcirculation/lib/bibcirculation_templates.py:11640
#: modules/bibcirculation/lib/bibcirculation_templates.py:13060
#: modules/bibcirculation/lib/bibcirculation_templates.py:13095
#: modules/bibcirculation/lib/bibcirculation_templates.py:13217
#: modules/bibcirculation/lib/bibcirculation_templates.py:13370
#: modules/bibcirculation/lib/bibcirculation_templates.py:13457
#: modules/bibcirculation/lib/bibcirculation_templates.py:17026
msgid "Type"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3946
#: modules/bibcirculation/lib/bibcirculation_templates.py:4049
#: modules/bibcirculation/lib/bibcirculation_templates.py:4272
#: modules/bibcirculation/lib/bibcirculation_templates.py:4333
msgid "No of items"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:3948
msgid "Duplicated library?"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4042
#: modules/bibcirculation/lib/bibcirculation_templates.py:4265
msgid "Library to be deleted"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4087
msgid "Search library"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4125
msgid "Select library"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4218
msgid "Please, note that this action is NOT reversible"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4277
#: modules/bibcirculation/lib/bibcirculation_templates.py:4338
msgid "Library not found"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4326
msgid "Merged library"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4403
msgid "Notes about this borrower"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4452
#: modules/bibcirculation/lib/bibcirculation_templates.py:5600
#: modules/bibcirculation/lib/bibcirculation_templates.py:9023
msgid "Personal details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4491
msgid "New ILL request"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4492
msgid "Notify this borrower"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4600
msgid "Requests, Loans and ILL overview on"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4602
#: modules/bibcirculation/lib/bibcirculation_templates.py:4609
msgid "Requests"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4675
msgid "Request option(s)"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4786
#: modules/bibcirculation/lib/bibcirculation_templates.py:8852
#: modules/bibcirculation/lib/bibcirculation_templates.py:10426
msgid "Loan date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4791
#: modules/bibcirculation/lib/bibcirculation_templates.py:5434
msgid "Loan notes"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4792
msgid "Loans status"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4793
#: modules/bibcirculation/lib/bibcirculation_templates.py:5435
msgid "Loan options"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4813
#: modules/bibcirculation/lib/bibcirculation_templates.py:5453
#: modules/bibcirculation/lib/bibcirculation_templates.py:10938
msgid "See notes"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4850
#: modules/bibcirculation/lib/bibcirculation_templates.py:4854
#: modules/bibcirculation/lib/bibcirculation_templates.py:5488
#: modules/bibcirculation/lib/bibcirculation_templates.py:5492
msgid "Change due date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4863
#: modules/bibcirculation/lib/bibcirculation_templates.py:5032
#: modules/bibcirculation/lib/bibcirculation_templates.py:5212
#: modules/bibcirculation/lib/bibcirculation_templates.py:5347
#: modules/bibcirculation/lib/bibcirculation_templates.py:5500
msgid "Send recall"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4952
#: modules/bibcirculation/lib/bibcirculation_templates.py:5128
msgid "No result for your search."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4983
#: modules/bibcirculation/lib/bibcirculation_templates.py:5161
msgid "Loan Notes"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:4996
#: modules/bibcirculation/lib/bibcirculation_templates.py:5175
msgid "see notes"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5000
#: modules/bibcirculation/lib/bibcirculation_templates.py:5180
msgid "no notes"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5289
msgid "CERN Library"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5324
msgid "Message"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5325
msgid "Choose a template"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5343
msgid "Templates"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5344
#: modules/bibcirculation/lib/bibcirculation_templates.py:5432
msgid "Overdue letter"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5345
msgid "Reminder"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5346
msgid "Notification"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5348
msgid "Load"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5367
msgid "Send"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5433
#: modules/bibcirculation/lib/bibcirculation_templates.py:8854
msgid "Loan status"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5652
#: modules/bibcirculation/lib/bibcirculation_templates.py:9055
#: modules/bibcirculation/lib/bibcirculation_templates.py:10428
msgid "Write notes"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5713
msgid "Notes about borrower"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5722
#: modules/bibcirculation/lib/bibcirculation_templates.py:5823
#: modules/bibcirculation/lib/bibcirculation_templates.py:8732
#: modules/bibcirculation/lib/bibcirculation_templates.py:11026
#: modules/bibcirculation/lib/bibcirculation_templates.py:11782
#: modules/bibcirculation/lib/bibcirculation_templates.py:12760
#: modules/bibcirculation/lib/bibcirculation_templates.py:13736
#: modules/bibcirculation/lib/bibcirculation_templates.py:15738
msgid "[delete]"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5768
#: modules/bibcirculation/lib/bibcirculation_templates.py:5870
#: modules/bibcirculation/lib/bibcirculation_templates.py:8776
#: modules/bibcirculation/lib/bibcirculation_templates.py:11071
#: modules/bibcirculation/lib/bibcirculation_templates.py:15785
msgid "Write new note"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5814
msgid "Notes about loan"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5990
msgid "Book Information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5994
msgid "EAN"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5997
msgid "Publication date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5998
msgid "Publication place"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:5999
#: modules/bibcirculation/lib/bibcirculation_templates.py:7155
#: modules/bibcirculation/lib/bibcirculation_templates.py:11955
#: modules/bibcirculation/lib/bibcirculation_templates.py:12156
#: modules/bibcirculation/lib/bibcirculation_templates.py:12939
#: modules/bibcirculation/lib/bibcirculation_templates.py:14925
#: modules/bibcirculation/lib/bibcirculation_templates.py:15148
#: modules/bibcirculation/lib/bibcirculation_templates.py:15424
#: modules/bibcirculation/lib/bibcirculation_templates.py:16844
#: modules/bibcirculation/lib/bibcirculation_templates.py:17032
msgid "Edition"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:6000
msgid "Number of pages"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:6001
msgid "Sub-library"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:6002
msgid "CERN Central Library"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:6099
msgid "Retrieve book information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:6312
msgid "A new borrower has been registered."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:6531
msgid "Borrower information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:6596
#: modules/bibcirculation/lib/bibcirculation_templates.py:6698
msgid "New library information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:6739
msgid "A new library has been registered."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:6775
#: modules/bibcirculation/lib/bibcirculation_templates.py:8569
msgid "Search library by"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:6927
#: modules/bibcirculation/lib/bibcirculation_templates.py:7026
msgid "Library information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:7071
#: modules/bibcirculation/lib/bibcirculation_templates.py:14502
msgid "The information has been updated."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:7150
#: modules/bibcirculation/lib/bibcirculation_templates.py:14920
msgid "Book title"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:7152
#: modules/bibcirculation/lib/bibcirculation_templates.py:11952
#: modules/bibcirculation/lib/bibcirculation_templates.py:12068
#: modules/bibcirculation/lib/bibcirculation_templates.py:12153
#: modules/bibcirculation/lib/bibcirculation_templates.py:12936
#: modules/bibcirculation/lib/bibcirculation_templates.py:14922
#: modules/bibcirculation/lib/bibcirculation_templates.py:15145
#: modules/bibcirculation/lib/bibcirculation_templates.py:15421
#: modules/bibcirculation/lib/bibcirculation_templates.py:16841
#: modules/bibcirculation/lib/bibcirculation_templates.py:17029
msgid "Place"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:7188
msgid "Coming soon..."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:7438
#: modules/bibcirculation/lib/bibcirculation_templates.py:17920
#, python-format
msgid "Copies of %s"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:7570
msgid "New copy details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:7725
#: modules/bibcirculation/lib/bibcirculation_templates.py:7816
#: modules/bibcirculation/lib/bibcirculation_templates.py:8394
#: modules/bibcirculation/lib/bibcirculation_templates.py:8484
msgid "Expected arrival date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:7859
#, python-format
msgid "A %(x_url_open)snew copy%(x_url_close)s has been added."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:7883
msgid "Back to the record"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:7975
#, python-format
msgid "%(nb_items_found)i items found"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:8268
msgid "Update copy information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:8476
msgid "New copy information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:8530
msgid "This item has been updated."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:8625
msgid "0 libraries found."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:8723
msgid "Notes about library"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:8856
msgid "Requested ?"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:8876
msgid "New due date: "
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:8901
msgid "Submit new due date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:8947
#, python-format
msgid "The due date has been updated. New due date: %s"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:8948
msgid "Back to borrower's loans"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:9225
msgid "Select item"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:9268
#: modules/bibcirculation/lib/bibcirculation_templates.py:9877
#: modules/bibcirculation/lib/bibcirculation_templates.py:10355
#: modules/bibcirculation/lib/bibcirculation_templates.py:11219
#: modules/bibcirculation/lib/bibcirculation_templates.py:15513
msgid "Borrower details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:9438
#: modules/bibcirculation/lib/bibcirculation_templates.py:9942
msgid "Enter the period of interest"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:9439
#: modules/bibcirculation/lib/bibcirculation_templates.py:9943
msgid "From: "
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:9441
#: modules/bibcirculation/lib/bibcirculation_templates.py:9944
msgid "To: "
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:9501
#: modules/bibcirculation/lib/bibcirculation_templates.py:9978
msgid "A new request has been registered with success."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:9626
#: modules/bibcirculation/lib/bibcirculation_templates.py:9633
#: modules/bibcirculation/lib/bibcirculation_templates.py:9640
#: modules/bibcirculation/lib/bibcirculation_templates.py:9653
#: modules/bibcirculation/lib/bibcirculation_templates.py:9660
#: modules/bibcirculation/lib/bibcirculation_templates.py:9667
#: modules/bibcirculation/lib/bibcirculation_templates.py:10101
#: modules/bibcirculation/lib/bibcirculation_templates.py:10108
#: modules/bibcirculation/lib/bibcirculation_templates.py:10115
msgid "ccid"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:10178
msgid "Please select one borrower to continue."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:10429
msgid "This note will be associate to this new loan, not to the borrower."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:10556
#: modules/bibcirculation/lib/bibcirculation_templates.py:10813
#: modules/bibcirculation/lib/bibcirculation_templates.py:13630
#: modules/bibcirculation/lib/bibcirculation_templates.py:13906
msgid "Order details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:10556
#: modules/bibcirculation/lib/bibcirculation_templates.py:10815
#: modules/bibcirculation/lib/bibcirculation_templates.py:10911
#: modules/bibcirculation/lib/bibcirculation_templates.py:13102
#: modules/bibcirculation/lib/bibcirculation_templates.py:13630
#: modules/bibcirculation/lib/bibcirculation_templates.py:13907
msgid "Vendor"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:10584
#: modules/bibcirculation/lib/bibcirculation_templates.py:10816
#: modules/bibcirculation/lib/bibcirculation_templates.py:10913
#: modules/bibcirculation/lib/bibcirculation_templates.py:13908
msgid "Price"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:10636
#: modules/bibcirculation/lib/bibcirculation_templates.py:10818
#: modules/bibcirculation/lib/bibcirculation_templates.py:13724
#: modules/bibcirculation/lib/bibcirculation_templates.py:13910
msgid "Order date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:10637
#: modules/bibcirculation/lib/bibcirculation_templates.py:10819
#: modules/bibcirculation/lib/bibcirculation_templates.py:10915
#: modules/bibcirculation/lib/bibcirculation_templates.py:12359
#: modules/bibcirculation/lib/bibcirculation_templates.py:12421
#: modules/bibcirculation/lib/bibcirculation_templates.py:12563
#: modules/bibcirculation/lib/bibcirculation_templates.py:12648
#: modules/bibcirculation/lib/bibcirculation_templates.py:13154
#: modules/bibcirculation/lib/bibcirculation_templates.py:13219
#: modules/bibcirculation/lib/bibcirculation_templates.py:13372
#: modules/bibcirculation/lib/bibcirculation_templates.py:13459
#: modules/bibcirculation/lib/bibcirculation_templates.py:13725
#: modules/bibcirculation/lib/bibcirculation_templates.py:13911
#: modules/bibcirculation/lib/bibcirculation_templates.py:15586
msgid "Expected date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:10863
msgid "A new purchase has been registered with success."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:10912
msgid "Ordered date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:10962
#: modules/bibcirculation/lib/bibcirculation_templates.py:11577
#: modules/bibcirculation/lib/bibcirculation_templates.py:11584
#: modules/bibcirculation/lib/bibcirculation_templates.py:11690
msgid "select"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:11017
#: modules/bibcirculation/lib/bibcirculation_templates.py:15729
msgid "Notes about acquisition"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:11212
#: modules/bibcirculation/lib/bibcirculation_templates.py:11428
#: modules/bibcirculation/lib/bibcirculation_templates.py:12217
#: modules/bibcirculation/lib/bibcirculation_templates.py:14959
#: modules/bibcirculation/lib/bibcirculation_templates.py:15150
#: modules/bibcirculation/lib/bibcirculation_templates.py:15466
#: modules/bibcirculation/lib/bibcirculation_templates.py:17381
#: modules/bibcirculation/lib/bibcirculation_templates.py:17517
msgid "ILL request details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:11213
#: modules/bibcirculation/lib/bibcirculation_templates.py:11429
#: modules/bibcirculation/lib/bibcirculation_templates.py:15152
#: modules/bibcirculation/lib/bibcirculation_templates.py:16921
#: modules/bibcirculation/lib/bibcirculation_templates.py:17037
#: modules/bibcirculation/lib/bibcirculation_templates.py:17382
#: modules/bibcirculation/lib/bibcirculation_templates.py:17518
msgid "Period of interest - From"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:11215
#: modules/bibcirculation/lib/bibcirculation_templates.py:11431
#: modules/bibcirculation/lib/bibcirculation_templates.py:15154
#: modules/bibcirculation/lib/bibcirculation_templates.py:16923
#: modules/bibcirculation/lib/bibcirculation_templates.py:17039
#: modules/bibcirculation/lib/bibcirculation_templates.py:17384
#: modules/bibcirculation/lib/bibcirculation_templates.py:17520
msgid "Period of interest - To"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:11217
#: modules/bibcirculation/lib/bibcirculation_templates.py:11433
#: modules/bibcirculation/lib/bibcirculation_templates.py:15013
#: modules/bibcirculation/lib/bibcirculation_templates.py:15156
#: modules/bibcirculation/lib/bibcirculation_templates.py:15470
#: modules/bibcirculation/lib/bibcirculation_templates.py:16925
#: modules/bibcirculation/lib/bibcirculation_templates.py:17041
#: modules/bibcirculation/lib/bibcirculation_templates.py:17386
#: modules/bibcirculation/lib/bibcirculation_templates.py:17522
msgid "Additional comments"
-msgstr ""
+msgstr "نظرهای اضافی"
#: modules/bibcirculation/lib/bibcirculation_templates.py:11218
#: modules/bibcirculation/lib/bibcirculation_templates.py:15471
msgid "Only this edition"
-msgstr ""
+msgstr "فقط این ویرایش"
#: modules/bibcirculation/lib/bibcirculation_templates.py:11279
msgid "A new ILL request has been registered with success."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:11434
#, python-format
msgid ""
"I accept the %(x_url_open)sconditions%(x_url_close)s of the service in "
"particular the return of books in due time."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:11435
msgid "I want this edition only."
-msgstr ""
+msgstr "من فقط این ویرایش را می خواهم."
#: modules/bibcirculation/lib/bibcirculation_templates.py:11466
#, python-format
msgid "You can see your loans %(here_link)s."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:11468
msgid "here"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:11515
#: modules/bibcirculation/lib/bibcirculation_templates.py:11635
#: modules/bibcirculation/lib/bibcirculation_templates.py:15584
msgid "Supplier"
-msgstr ""
+msgstr "تأمین کننده"
#: modules/bibcirculation/lib/bibcirculation_templates.py:11518
msgid "Interest from"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:11636
#: modules/bibcirculation/lib/bibcirculation_templates.py:12361
#: modules/bibcirculation/lib/bibcirculation_templates.py:12470
#: modules/bibcirculation/lib/bibcirculation_templates.py:12566
#: modules/bibcirculation/lib/bibcirculation_templates.py:12650
#: modules/bibcirculation/lib/bibcirculation_templates.py:13156
#: modules/bibcirculation/lib/bibcirculation_templates.py:13270
#: modules/bibcirculation/lib/bibcirculation_templates.py:13375
#: modules/bibcirculation/lib/bibcirculation_templates.py:13461
#: modules/bibcirculation/lib/bibcirculation_templates.py:13649
msgid "Cost"
-msgstr ""
+msgstr "هزینه"
#: modules/bibcirculation/lib/bibcirculation_templates.py:11639
msgid "Date requested"
-msgstr ""
+msgstr "تاریخ درخواست"
#: modules/bibcirculation/lib/bibcirculation_templates.py:12062
msgid "Periodical Title"
-msgstr ""
+msgstr "عنوان ادواری"
#: modules/bibcirculation/lib/bibcirculation_templates.py:12063
msgid "Article Title"
-msgstr ""
+msgstr "عنوان مقاله"
#: modules/bibcirculation/lib/bibcirculation_templates.py:12065
#: modules/bibcirculation/lib/bibcirculation_templates.py:17313
#: modules/bibcirculation/lib/bibcirculation_templates.py:17511
msgid "Volume"
-msgstr ""
+msgstr "جلد/ دوره"
#: modules/bibcirculation/lib/bibcirculation_templates.py:12066
#: modules/bibcirculation/lib/bibcirculation_templates.py:17314
#: modules/bibcirculation/lib/bibcirculation_templates.py:17512
msgid "Issue"
-msgstr ""
+msgstr "شماره"
#: modules/bibcirculation/lib/bibcirculation_templates.py:12067
#: modules/bibcirculation/lib/bibcirculation_templates.py:17315
#: modules/bibcirculation/lib/bibcirculation_templates.py:17513
msgid "Page"
-msgstr ""
+msgstr "صفحه"
#: modules/bibcirculation/lib/bibcirculation_templates.py:12069
#: modules/bibcirculation/lib/bibcirculation_templates.py:17318
#: modules/bibcirculation/lib/bibcirculation_templates.py:17516
msgid "ISSN"
-msgstr ""
+msgstr "شماره استاندارد بين المللي پيايندها (شاپا)"
#: modules/bibcirculation/lib/bibcirculation_templates.py:12210
#: modules/bibcirculation/lib/bibcirculation_templates.py:12994
msgid "Borrower request"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:12213
#: modules/bibcirculation/lib/bibcirculation_templates.py:12997
#: modules/bibcirculation/lib/bibcirculation_templates.py:14960
#: modules/bibcirculation/lib/bibcirculation_templates.py:15468
msgid "Period of interest (From)"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:12214
#: modules/bibcirculation/lib/bibcirculation_templates.py:12998
#: modules/bibcirculation/lib/bibcirculation_templates.py:15011
#: modules/bibcirculation/lib/bibcirculation_templates.py:15469
msgid "Period of interest (To)"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:12215
#: modules/bibcirculation/lib/bibcirculation_templates.py:12999
msgid "Borrower comments"
-msgstr ""
+msgstr "نظرهای امانت گیرنده"
#: modules/bibcirculation/lib/bibcirculation_templates.py:12216
#: modules/bibcirculation/lib/bibcirculation_templates.py:13000
msgid "Only this edition?"
-msgstr ""
+msgstr "تنها این ویرایش؟"
#: modules/bibcirculation/lib/bibcirculation_templates.py:12271
#: modules/bibcirculation/lib/bibcirculation_templates.py:12303
#: modules/bibcirculation/lib/bibcirculation_templates.py:12419
#: modules/bibcirculation/lib/bibcirculation_templates.py:12562
#: modules/bibcirculation/lib/bibcirculation_templates.py:12647
#: modules/bibcirculation/lib/bibcirculation_templates.py:13059
#: modules/bibcirculation/lib/bibcirculation_templates.py:13094
#: modules/bibcirculation/lib/bibcirculation_templates.py:13216
#: modules/bibcirculation/lib/bibcirculation_templates.py:13370
#: modules/bibcirculation/lib/bibcirculation_templates.py:13457
msgid "ILL request ID"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:12272
#: modules/bibcirculation/lib/bibcirculation_templates.py:12377
#: modules/bibcirculation/lib/bibcirculation_templates.py:12482
#: modules/bibcirculation/lib/bibcirculation_templates.py:12580
#: modules/bibcirculation/lib/bibcirculation_templates.py:12663
#: modules/bibcirculation/lib/bibcirculation_templates.py:13062
#: modules/bibcirculation/lib/bibcirculation_templates.py:13173
#: modules/bibcirculation/lib/bibcirculation_templates.py:13285
#: modules/bibcirculation/lib/bibcirculation_templates.py:13388
#: modules/bibcirculation/lib/bibcirculation_templates.py:13475
#: modules/bibcirculation/lib/bibcirculation_templates.py:13726
#: modules/bibcirculation/lib/bibcirculation_templates.py:13912
msgid "Previous notes"
-msgstr ""
+msgstr "یادداشت های قبلی"
#: modules/bibcirculation/lib/bibcirculation_templates.py:12293
#: modules/bibcirculation/lib/bibcirculation_templates.py:12397
#: modules/bibcirculation/lib/bibcirculation_templates.py:12500
#: modules/bibcirculation/lib/bibcirculation_templates.py:12600
#: modules/bibcirculation/lib/bibcirculation_templates.py:12683
#: modules/bibcirculation/lib/bibcirculation_templates.py:13082
#: modules/bibcirculation/lib/bibcirculation_templates.py:13192
#: modules/bibcirculation/lib/bibcirculation_templates.py:13306
#: modules/bibcirculation/lib/bibcirculation_templates.py:13408
#: modules/bibcirculation/lib/bibcirculation_templates.py:13495
#: modules/bibcirculation/lib/bibcirculation_templates.py:15590
msgid "Library notes"
-msgstr ""
+msgstr "یادداشت های کتابخانه "
#: modules/bibcirculation/lib/bibcirculation_templates.py:12310
msgid "Library/Supplier"
-msgstr ""
+msgstr "کتابخانه/ تأمین کننده"
#: modules/bibcirculation/lib/bibcirculation_templates.py:12463
#: modules/bibcirculation/lib/bibcirculation_templates.py:12564
#: modules/bibcirculation/lib/bibcirculation_templates.py:12649
#: modules/bibcirculation/lib/bibcirculation_templates.py:13261
#: modules/bibcirculation/lib/bibcirculation_templates.py:13372
#: modules/bibcirculation/lib/bibcirculation_templates.py:13459
#: modules/bibcirculation/lib/bibcirculation_templates.py:15587
msgid "Arrival date"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:12941
#: modules/bibcirculation/lib/bibcirculation_templates.py:16847
#: modules/bibcirculation/lib/bibcirculation_templates.py:17034
msgid "Standard number"
-msgstr ""
+msgstr "شماره استاندارد"
#: modules/bibcirculation/lib/bibcirculation_templates.py:13001
#: modules/bibcirculation/lib/bibcirculation_templates.py:16919
#: modules/bibcirculation/lib/bibcirculation_templates.py:17035
msgid "Request details"
-msgstr ""
+msgstr "جزئیات درخواست"
#: modules/bibcirculation/lib/bibcirculation_templates.py:13061
#: modules/bibcirculation/lib/bibcirculation_templates.py:13172
#: modules/bibcirculation/lib/bibcirculation_templates.py:13285
#: modules/bibcirculation/lib/bibcirculation_templates.py:13388
#: modules/bibcirculation/lib/bibcirculation_templates.py:13475
#: modules/bibcirculation/lib/bibcirculation_templates.py:14959
#: modules/bibcirculation/lib/bibcirculation_templates.py:15151
#: modules/bibcirculation/lib/bibcirculation_templates.py:15467
#: modules/bibcirculation/lib/bibcirculation_templates.py:16920
#: modules/bibcirculation/lib/bibcirculation_templates.py:17036
#: modules/bibcirculation/lib/bibcirculation_templates.py:17317
#: modules/bibcirculation/lib/bibcirculation_templates.py:17515
msgid "Budget code"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:13997
msgid "Purchase information updated with success."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:14070
#: modules/bibcirculation/lib/bibcirculation_templates.py:14139
msgid "New vendor information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:14183
msgid "A new vendor has been registered."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:14220
#: modules/bibcirculation/lib/bibcirculation_templates.py:14539
msgid "Search vendor by"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:14283
#: modules/bibcirculation/lib/bibcirculation_templates.py:14606
msgid "Vendor(s)"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:14388
#: modules/bibcirculation/lib/bibcirculation_templates.py:14459
msgid "Vendor information"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:14669
#: modules/bibcirculation/lib/bibcirculation_templates.py:14758
msgid "Notes about this vendor"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:14712
msgid "Vendor details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:14797
msgid "Add notes"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:14850
#, python-format
msgid "Book does not exists in %(CFG_SITE_NAME)s"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:14852
msgid "Please fill the following form."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:15014
#, python-format
msgid ""
"Borrower accepts the %(x_url_open)sconditions%(x_url_close)s of the service "
"in particular the return of books in due time."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:15015
msgid "Borrower wants this edition only."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:15158
msgid "Only this edition."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:15582
msgid "ILL ID"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:15645
msgid "Notes about this ILL"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:15823
msgid "No more requests are pending or waiting."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:15975
msgid "Printable format"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:16006
#, python-format
msgid ""
"Check if the book already exists on %(CFG_SITE_NAME)s, before sending your "
"ILL request."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:16078
msgid "0 items found."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:16161
msgid "Proceed anyway"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:16730
msgid ""
"According to a decision from the Scientific Information Policy Board, books "
"purchased with budget codes other than Team accounts will be added to the "
"Library catalogue, with the indication of the purchaser."
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:16751
msgid "Document details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:16751
msgid "Document type"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:16845
msgid "This edition only"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:16920
msgid "Cash"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:17308
msgid "Article details"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:17309
#: modules/bibcirculation/lib/bibcirculation_templates.py:17507
msgid "Periodical title"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:17310
#: modules/bibcirculation/lib/bibcirculation_templates.py:17508
msgid "Article title"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:17312
#: modules/bibcirculation/lib/bibcirculation_templates.py:17510
msgid "Report number"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:17724
msgid "Search ILL request by"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:17725
msgid "ILL request id"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:17725
msgid "cost"
-msgstr ""
+msgstr "هزینه"
#: modules/bibcirculation/lib/bibcirculation_templates.py:17725
msgid "notes"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:17764
msgid "date restriction"
msgstr ""
#: modules/bibcirculation/lib/bibcirculation_templates.py:17765
msgid "the beginning"
-msgstr ""
+msgstr "آغاز"
#: modules/bibcirculation/lib/bibcirculation_templates.py:17766
msgid "now"
-msgstr ""
+msgstr "اکنون"
#: modules/bibcheck/web/admin/bibcheckadmin.py:60
msgid "BibCheck Admin"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:70
#: modules/bibcheck/web/admin/bibcheckadmin.py:250
#: modules/bibcheck/web/admin/bibcheckadmin.py:289
#: modules/bibcheck/web/admin/bibcheckadmin.py:326
msgid "Not authorized"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:80
#, python-format
msgid "ERROR: %s does not exist"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:82
#, python-format
msgid "ERROR: %s is not a directory"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:84
#, python-format
msgid "ERROR: %s is not writable"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:117
msgid "Limit to knowledge bases containing string:"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:135
msgid "Really delete"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:141
msgid "Verify syntax"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:146
msgid "Create new"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:166
#, python-format
msgid "File %s does not exist."
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:175
msgid "Calling bibcheck -verify failed."
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:182
msgid "Verify BibCheck config file"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:183
msgid "Verify problem"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:205
msgid "File"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:241
msgid "Edit BibCheck config file"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:269
#, python-format
msgid "File %s already exists."
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:272
#, python-format
msgid "File %s: written OK."
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:278
#, python-format
msgid "File %s: write failed."
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:280
msgid "Save BibCheck config file"
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:313
#, python-format
msgid "File %s deleted."
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:315
#, python-format
msgid "File %s: delete failed."
msgstr ""
#: modules/bibcheck/web/admin/bibcheckadmin.py:317
msgid "Delete BibCheck config file"
msgstr ""
#: modules/bibharvest/lib/oai_repository_admin.py:155
#: modules/bibharvest/lib/oai_repository_admin.py:260
#: modules/bibharvest/lib/oai_repository_admin.py:339
msgid "Return to main selection"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:119
msgid "Overview of sources"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:120
msgid "Harvesting status"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:138
msgid "Not Set"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:139
msgid "never"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:150
msgid "Never harvested"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:162
msgid "View Holding Pen"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:187
#: modules/bibharvest/lib/oai_harvest_admin.py:559
msgid "No OAI source ID selected."
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:290
#: modules/bibharvest/lib/oai_harvest_admin.py:463
#: modules/bibharvest/lib/oai_harvest_admin.py:477
#: modules/bibharvest/lib/oai_harvest_admin.py:492
#: modules/bibharvest/lib/oai_harvest_admin.py:500
#: modules/bibharvest/lib/oai_harvest_admin.py:547
msgid "Go back to the OAI sources overview"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:449
msgid "Try again with another url"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:456
msgid "Continue anyway"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:830
msgid "Return to the month view"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:1104
msgid "Compare with original"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:1110
#: modules/bibharvest/lib/oai_harvest_admin.py:1155
msgid "Delete from holding pen"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:1128
msgid "Error when retrieving the Holding Pen entry"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:1136
msgid "Error when retrieving the record"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:1144
msgid ""
"Error when formatting the Holding Pen entry. Probably its content is broken"
msgstr ""
#: modules/bibharvest/lib/oai_harvest_admin.py:1149
msgid "Accept Holding Pen version"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:51
#, python-format
msgid ""
"Limit display to knowledge bases matching %(keyword_field)s in their rules "
"and descriptions"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:89
msgid "No Knowledge Base"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:148
msgid "Add New Knowledge Base"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:149
msgid "Configure a dynamic KB"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:150
msgid "Add New Taxonomy"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:191
msgid "This knowledge base already has a taxonomy file."
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:192
msgid "If you upload another file, the current version will be replaced."
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:194
#, python-format
msgid "The current taxonomy can be accessed with this URL: %s"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:197
#, python-format
msgid "Please upload the RDF file for taxonomy %s"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:234
msgid "Please configure"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:235
msgid ""
"A dynamic knowledge base is a list of values of a "
"given field. The list is generated dynamically by "
"searching the records using a search expression."
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:239
msgid ""
"Example: Your records contain field 270__a for the "
"name and address of the author's institute. If you "
"set the field to '270__a' and the expression to "
"'270__a:*Paris*', a list of institutes in Paris "
"will be created."
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:244
msgid ""
"If the expression is empty, a list of all values in "
"270__a will be created."
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:246
msgid ""
"If the expression contains '%', like '270__a:*%*', "
"it will be replaced by a search string when the "
"knowledge base is used."
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:249
msgid ""
"You can enter a collection name if the expression "
"should be evaluated in a specific collection."
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:251
msgid ""
"Example 1: Your records contain field 270__a for "
"the name and address of the author's institute. If "
"you set the field to '270__a' and the expression to "
"'270__a:*Paris*', a list of institutes in Paris "
"will be created."
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:256
msgid ""
"Example 2: Return the institute's name (100__a) when "
"the user gives its postal code "
"(270__a): Set field to 100__a, expression to 270__a:"
"*%*."
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:260
msgid "Any collection"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:282
msgid "Exporting: "
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:324
#: modules/bibknowledge/lib/bibknowledge_templates.py:588
#: modules/bibknowledge/lib/bibknowledge_templates.py:657
msgid "Knowledge Base Mappings"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:325
#: modules/bibknowledge/lib/bibknowledge_templates.py:589
#: modules/bibknowledge/lib/bibknowledge_templates.py:658
msgid "Knowledge Base Attributes"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:326
#: modules/bibknowledge/lib/bibknowledge_templates.py:590
#: modules/bibknowledge/lib/bibknowledge_templates.py:659
msgid "Knowledge Base Dependencies"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:347
msgid ""
"Here you can add new mappings to this base and "
"change the base attributes."
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:362
msgid "Map From"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:425
msgid "Search for a mapping"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:480
msgid "Knowledge base is empty"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:545
msgid "You can get a these mappings in textual format by: "
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:547
msgid "And the KBA version by:"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:627
msgid "Update Base Attributes"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:670
msgid "This knowledge base is not used in any format elements."
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:700
#, python-format
msgid "Your rule: %s"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:702
#, python-format
msgid ""
"The left side of the rule (%s) already appears in these knowledge bases:"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:705
#, python-format
msgid ""
"The right side of the rule (%s) already appears in these knowledge bases:"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:719
msgid "Please select action"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:720
msgid "Replace the selected rules with this rule"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:721
msgid "Add this rule in the current knowledge base"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:722
msgid "Cancel: do not add this rule"
msgstr ""
#: modules/bibknowledge/lib/bibknowledge_templates.py:755
msgid ""
"It is not possible to have two rules with the same left side in the same "
"knowledge base."
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:72
msgid "BibKnowledge Admin"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:92
msgid "Knowledge Bases"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:106
#: modules/bibknowledge/lib/bibknowledgeadmin.py:117
#: modules/bibknowledge/lib/bibknowledgeadmin.py:129
msgid "Cannot upload file"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:107
msgid "You have not selected a file to upload"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:141
#, python-format
msgid "File %s uploaded."
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:143
msgid "File uploaded"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:172
#: modules/bibknowledge/lib/bibknowledgeadmin.py:216
#: modules/bibknowledge/lib/bibknowledgeadmin.py:266
#: modules/bibknowledge/lib/bibknowledgeadmin.py:303
#: modules/bibknowledge/lib/bibknowledgeadmin.py:356
#: modules/bibknowledge/lib/bibknowledgeadmin.py:465
#: modules/bibknowledge/lib/bibknowledgeadmin.py:524
#: modules/bibknowledge/lib/bibknowledgeadmin.py:590
#: modules/bibknowledge/lib/bibknowledgeadmin.py:686
#: modules/bibknowledge/lib/bibknowledgeadmin.py:703
#: modules/bibknowledge/lib/bibknowledgeadmin.py:718
#: modules/bibknowledge/lib/bibknowledgeadmin.py:754
msgid "Manage Knowledge Bases"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:185
#: modules/bibknowledge/lib/bibknowledgeadmin.py:230
#: modules/bibknowledge/lib/bibknowledgeadmin.py:316
#: modules/bibknowledge/lib/bibknowledgeadmin.py:370
#: modules/bibknowledge/lib/bibknowledgeadmin.py:478
#: modules/bibknowledge/lib/bibknowledgeadmin.py:543
#: modules/bibknowledge/lib/bibknowledgeadmin.py:730
msgid "Unknown Knowledge Base"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:192
#, python-format
msgid "Knowledge Base %s"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:239
#, python-format
msgid "Knowledge Base %s Attributes"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:325
#, python-format
msgid "Knowledge Base %s Dependencies"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:407
msgid "Left side exists"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:415
msgid "Right side exists"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:592
msgid "Knowledge base name missing"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:612
msgid "Unknown knowledge base"
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:613
msgid "There is no knowledge base with that name."
msgstr ""
#: modules/bibknowledge/lib/bibknowledgeadmin.py:718
msgid "Delete Knowledge Base"
msgstr ""
#: modules/bibsword/lib/bibsword_webinterface.py:157
msgid "BibSword Admin Interface"
msgstr ""
#: modules/bibsword/lib/bibsword_webinterface.py:171
#: modules/bibsword/lib/bibsword_webinterface.py:277
#: modules/bibsword/lib/bibsword_webinterface.py:301
#: modules/bibsword/lib/bibsword_webinterface.py:330
msgid "Export with BibSword: Step 2/4"
msgstr ""
#: modules/bibsword/lib/bibsword_webinterface.py:222
#: modules/bibsword/lib/bibsword_webinterface.py:233
#: modules/bibsword/lib/bibsword_webinterface.py:291
msgid "Export with BibSword: Step 1/4"
msgstr ""
#: modules/bibsword/lib/bibsword_webinterface.py:315
#: modules/bibsword/lib/bibsword_webinterface.py:343
#: modules/bibsword/lib/bibsword_webinterface.py:374
msgid "Export with BibSword: Step 3/4"
msgstr ""
#: modules/bibsword/lib/bibsword_webinterface.py:358
#: modules/bibsword/lib/bibsword_webinterface.py:389
msgid "Export with BibSword: Step 4/4"
msgstr ""
#: modules/bibsword/lib/bibsword_webinterface.py:434
msgid "Export with BibSword: Acknowledgement"
msgstr ""
#: modules/bibupload/lib/batchuploader_engine.py:243
msgid "More than one possible recID, ambiguous behaviour"
msgstr ""
#: modules/bibupload/lib/batchuploader_engine.py:243
msgid "No records match that file name"
msgstr ""
#: modules/bibupload/lib/batchuploader_engine.py:244
msgid "File already exists"
msgstr ""
#: modules/bibupload/lib/batchuploader_engine.py:244
msgid "A file with the same name and format already exists"
msgstr ""
#: modules/bibupload/lib/batchuploader_engine.py:245
#, python-format
msgid "No rights to upload to collection '%s'"
msgstr ""
#: modules/bibupload/lib/batchuploader_engine.py:449
msgid "Guests are not authorized to run batchuploader"
msgstr ""
#: modules/bibupload/lib/batchuploader_engine.py:451
#, python-format
msgid "The user '%s' is not authorized to run batchuploader"
msgstr ""
#: modules/bibupload/lib/batchuploader_engine.py:506
#: modules/bibupload/lib/batchuploader_engine.py:519
#, python-format
msgid ""
"The user '%(x_user)s' is not authorized to modify collection '%(x_coll)s'"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:267
msgid "Fatal: Author ID capabilities are disabled on this system."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:270
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:313
msgid "Fatal: You are not allowed to access this functionality."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:662
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:763
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:920
msgid "Papers removed from this profile"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:663
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:667
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:728
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:732
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:764
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:768
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:921
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:925
msgid "Papers in need of review"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:664
msgid "Open Tickets"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:664
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:729
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:765
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:922
msgid "Data"
-msgstr ""
+msgstr "داده"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:665
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:766
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:923
msgid "Papers of this Person"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:666
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:767
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:924
msgid "Papers _not_ of this Person"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:668
msgid "Tickets for this Person"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:669
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:734
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:770
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:927
msgid "Additional Data for this Person"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:671
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:735
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:771
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:947
msgid "Sorry, there are currently no documents to be found in this category."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:672
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:772
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:948
msgid "Yes, those papers are by this person."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:673
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:773
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:949
msgid "No, those papers are not by this person"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:674
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:774
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:950
msgid "Assign to other person"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:675
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:739
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:775
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:951
msgid "Forget decision"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:676
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:690
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:776
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:790
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:952
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:966
msgid "Confirm!"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:677
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:777
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:953
msgid "Yes, this paper is by this person."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:678
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:778
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:954
msgid "Rejected!"
-msgstr ""
+msgstr "رد شده!"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:679
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:779
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:955
msgid "No, this paper is <i>not</i> by this person"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:680
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:688
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:696
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:744
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:752
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:760
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:780
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:788
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:796
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:956
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:964
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:972
msgid "Assign to another person"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:681
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:689
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:697
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:745
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:753
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:761
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:781
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:789
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:797
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:957
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:965
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:973
msgid "To other person!"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:682
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:782
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:958
msgid "Confirmed."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:683
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:783
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:959
msgid "Marked as this person's paper"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:684
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:692
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:748
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:756
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:757
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:784
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:792
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:960
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:968
msgid "Forget decision!"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:685
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:693
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:785
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:793
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:961
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:969
msgid "Forget decision."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:686
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:786
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:962
msgid "Repeal!"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:687
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:787
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:963
msgid "But it's <i>not</i> this person's paper."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:691
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:791
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:967
msgid "But it <i>is</i> this person's paper."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:694
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:794
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:970
msgid "Repealed"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:695
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:795
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:971
msgid "Marked as not this person's paper"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:727
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:730
msgid "Your papers"
-msgstr ""
+msgstr "مقالات شما"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:727
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:731
msgid "Not your papers"
-msgstr ""
+msgstr "مقالات غیر شما"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:736
msgid "These are mine!"
-msgstr ""
+msgstr "این ها مال من هستند!"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:737
msgid "These are not mine!"
-msgstr ""
+msgstr "اینها مال من نیستند!"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:738
msgid "It's not mine, but I know whose it is!"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:740
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:754
msgid "Mine!"
-msgstr ""
+msgstr "مال من!"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:741
msgid "This is my paper!"
-msgstr ""
+msgstr "این، مقاله من است!"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:742
msgid "Not mine!"
-msgstr ""
+msgstr "مال من نیست!"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:743
msgid "This is not my paper!"
-msgstr ""
+msgstr "این، مقاله من نیست!"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:746
msgid "Not Mine."
-msgstr ""
+msgstr "مال من نیست"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:747
msgid "Marked as my paper!"
-msgstr ""
+msgstr "نشانه گذاری به عنوان مقاله من"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:749
msgid "Forget assignment decision"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:750
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:758
msgid "Not Mine!"
-msgstr ""
+msgstr "مال من نیست!"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:751
msgid "But this is mine!"
-msgstr ""
+msgstr "اما این مال من است!"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:755
msgid "But this is my paper!"
-msgstr ""
+msgstr "اما این مقاله ام است!"
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:759
msgid "Marked as not your paper."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:769
msgid "Tickes you created about this person"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:922
msgid "Tickets"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:926
msgid "Request Tickets"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:1178
msgid "Submit Attribution Information"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:1323
msgid "Please review your actions"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:2008
msgid "Claim this paper"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:2109
msgid ""
"<p>We're sorry. An error occurred while handling your request. Please find "
"more information below:</p>"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:2187
msgid "Person search for assignment in progress!"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:2188
msgid "You are searching for a person to assign the following papers:"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:2349
#, python-format
msgid "You are going to claim papers for: %s"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:2377
msgid "This page in not accessible directly."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_webinterface.py:2379
msgid "Welcome!"
-msgstr ""
+msgstr "خوش آمدید!"
#: modules/bibauthorid/lib/bibauthorid_templates.py:153
msgid "Click here to review the transactions."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:196
msgid "Quit searching."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:417
msgid "You are about to attribute the following paper"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:439
msgid "Info"
-msgstr ""
+msgstr "اطلاعات"
#: modules/bibauthorid/lib/bibauthorid_templates.py:451
msgid " Search for a person to attribute the paper to"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:512
#: modules/bibauthorid/lib/bibauthorid_templates.py:607
#: modules/bibauthorid/lib/bibauthorid_templates.py:679
msgid "Select All"
-msgstr ""
+msgstr "انتخاب همگی"
#: modules/bibauthorid/lib/bibauthorid_templates.py:513
#: modules/bibauthorid/lib/bibauthorid_templates.py:608
#: modules/bibauthorid/lib/bibauthorid_templates.py:680
msgid "Select None"
-msgstr ""
+msgstr "انتخاب هیچ یک"
#: modules/bibauthorid/lib/bibauthorid_templates.py:514
#: modules/bibauthorid/lib/bibauthorid_templates.py:609
#: modules/bibauthorid/lib/bibauthorid_templates.py:681
msgid "Invert Selection"
-msgstr ""
+msgstr "انتخاب معکوس"
#: modules/bibauthorid/lib/bibauthorid_templates.py:516
#: modules/bibauthorid/lib/bibauthorid_templates.py:611
msgid "Hide successful claims"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:576
msgid "No status information found."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:598
msgid "Operator review of user actions pending"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:642
msgid "Sorry, there are currently no records to be found in this category."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:671
msgid "Review Transaction"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:678
msgid " On all pages: "
-msgstr ""
+msgstr "بر روی همه صفحات"
#: modules/bibauthorid/lib/bibauthorid_templates.py:714
msgid "Names variants:"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:836
msgid "These records have been marked as not being from this person."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:837
msgid "They will be regarded in the next run of the author "
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:838
msgid "disambiguation algorithm and might disappear from this listing."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:864
#: modules/bibauthorid/lib/bibauthorid_templates.py:865
#: modules/bibauthorid/lib/bibauthorid_templates.py:868
msgid "Not provided"
-msgstr ""
+msgstr "تأمین نشده"
#: modules/bibauthorid/lib/bibauthorid_templates.py:866
msgid "Not available"
-msgstr ""
+msgstr "غیرقابل استفاده"
#: modules/bibauthorid/lib/bibauthorid_templates.py:867
msgid "No comments"
-msgstr ""
+msgstr "بدون نظر "
#: modules/bibauthorid/lib/bibauthorid_templates.py:869
msgid "Not Available"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:889
msgid " Delete this ticket"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:893
msgid " Commit this entire ticket"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:952
msgid "... This tab is currently under construction ... "
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:973
msgid ""
"We could not reliably determine the name of the author on the records below "
"to automatically perform an assignment."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:975
msgid "Please select an author for the records in question."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:976
msgid "Boxes not selected will be ignored in the process."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:983
msgid "Select name for"
-msgstr ""
+msgstr "انتخاب نام برای"
#: modules/bibauthorid/lib/bibauthorid_templates.py:992
#: modules/bibauthorid/lib/bibauthorid_templates.py:1018
#: modules/bibauthorid/lib/bibauthorid_templates.py:1162
msgid "Error retrieving record title"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:994
msgid "Paper title: "
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1006
msgid "The following names have been automatically chosen:"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1021
msgid " -- With name: "
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1027
msgid "Ignore"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1076
#: modules/bibauthorid/lib/bibauthorid_templates.py:1092
msgid "Navigation:"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1077
msgid "Run paper attribution for another author"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1078
#: modules/bibauthorid/lib/bibauthorid_templates.py:1095
msgid "Person Interface FAQ"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1093
msgid "Person Search"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1094
msgid "Open tickets"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1123
msgid "Symbols legend: "
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1128
#: modules/bibauthorid/lib/bibauthorid_templates.py:1186
msgid "Everything is shiny, captain!"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1129
msgid "The result of this request will be visible immediately"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1134
msgid "Confirmation needed to continue"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1135
msgid ""
"The result of this request will be visible immediately but we need your "
"confirmation to do so for this paper have been manually claimed before"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1140
msgid "This will create a change request for the operators"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1141
msgid ""
"The result of this request will be visible upon confirmation through an "
"operator"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1179
msgid "Selected name on paper"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1190
msgid "Verification needed to continue"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1194
msgid "This will create a request for the operators"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1216
msgid "Please Check your entries"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1216
msgid "Sorry."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1221
msgid "Please provide at least one transaction."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1221
msgid "Error:"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1232
msgid "Please provide your information"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1239
msgid "Please provide your first name"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1243
#: modules/bibauthorid/lib/bibauthorid_templates.py:1245
msgid "Your first name:"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1251
msgid "Please provide your last name"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1256
#: modules/bibauthorid/lib/bibauthorid_templates.py:1258
msgid "Your last name:"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1266
msgid "Please provide your eMail address"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1272
msgid ""
"This eMail address is reserved by a user. Please log in or provide an "
"alternative eMail address"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1277
#: modules/bibauthorid/lib/bibauthorid_templates.py:1279
msgid "Your eMail:"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1283
msgid "You may leave a comment (optional)"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1296
msgid "Continue claiming*"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1298
msgid "Confirm these changes**"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1301
msgid "!Delete the entire request!"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1314
msgid "Mark as your documents"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1329
msgid "Mark as _not_ your documents"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1340
msgid "Nothing staged as not yours"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1344
msgid "Mark as their documents"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1355
#: modules/bibauthorid/lib/bibauthorid_templates.py:1370
msgid "Nothing staged in this category"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1359
msgid "Mark as _not_ their documents"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1376
msgid " * You can come back to this page later. Nothing will be lost. <br />"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1377
msgid ""
" ** Performs all requested changes. Changes subject to permission "
"restrictions will be submitted to an operator for manual review."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1433
#, python-format
msgid "We do not have a publication list for '%s'."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1448
#: modules/bibauthorid/lib/bibauthorid_templates.py:1560
msgid "Create a new Person for your search"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1503
msgid "Recent Papers"
-msgstr ""
+msgstr "مقاله های اخیر"
#: modules/bibauthorid/lib/bibauthorid_templates.py:1515
msgid "YES!"
-msgstr ""
+msgstr "بله!"
#: modules/bibauthorid/lib/bibauthorid_templates.py:1516
msgid " Attribute Papers To "
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1522
#: modules/bibauthorid/lib/bibauthorid_templates.py:1544
msgid "Publication List "
-msgstr ""
+msgstr "سیاهه انتشار"
#: modules/bibauthorid/lib/bibauthorid_templates.py:1529
msgid "Showing the"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1529
msgid "most recent documents:"
-msgstr ""
+msgstr "تازه ترین مدارک:"
#: modules/bibauthorid/lib/bibauthorid_templates.py:1538
msgid "Sorry, there are no documents known for this person"
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1540
msgid ""
"Information not shown to increase performances. Please refine your search."
msgstr ""
#: modules/bibauthorid/lib/bibauthorid_templates.py:1648
msgid "Correct my publication lists!"
msgstr ""
diff --git a/requirements-extras.txt b/requirements-extras.txt
index 575753468..581c0d268 100644
--- a/requirements-extras.txt
+++ b/requirements-extras.txt
@@ -1,23 +1,24 @@
# More requirements files are needed, since e.g gnuplot-py and h5py
# import numpy in their setup.py, which means they have to be
# installed in a second step.
gnuplot-py==1.8
h5py==2.0.1
# Following packages are optional (if you do development you probably want to install them):
pylint
http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download#egg=pychecker-0.8.19
pep8
selenium
winpdb
mock
cython
nose
nosexcover
flake8
dropbox
oauth2client
apiclient
urllib3
python-onedrive
python-openid
+raven==4.2.1
diff --git a/requirements.txt b/requirements.txt
index 81974c4b7..6835b049f 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,74 +1,76 @@
MySQL-python==1.2.5
rdflib==2.4.2
reportlab==2.5
python-dateutil<=1.9999
python-magic==0.4.2
https://www.reportlab.com/ftp/pyRXP-1.16-daily-unix.tar.gz#egg=pyRXP-1.16
numpy==1.7.0
lxml==3.1.2
mechanize==0.2.5
python-Levenshtein==0.10.2
http://pybrary.net/pyPdf/pyPdf-1.13.tar.gz#egg=pyPdf-1.13
PyStemmer==1.3.0
https://py-editdist.googlecode.com/files/py-editdist-0.3.tar.gz#egg=editdist-0.3
feedparser==5.1.3
BeautifulSoup==3.2.1
beautifulsoup4==4.3.2
python-twitter==0.8.7
celery==3.0.17
msgpack-python==0.3.0
pyparsing==2.0.1
git+git://github.com/lnielsen-cern/dictdiffer.git#egg=dictdiffer-0.0.3
git+git://github.com/romanchyla/workflow.git@e41299579501704b1486c72cc2509a9f82e63ea6#egg=workflow-1.1.0
-requests==1.2.3
+requests==2.2.0
PyPDF2
rauth
unidecode
https://pyfilesystem.googlecode.com/files/fs-0.4.0.tar.gz#egg=fs-0.4.0
libmagic==1.0
Babel==1.3
git+https://github.com/inveniosoftware/intbitset#egg=intbitset-2.0
Jinja2==2.7.2
SQLAlchemy==0.8.3
Flask==0.10.1
# Flask-WTF 0.9.5 doesn't support WTForms 2.0 as of yet.
WTForms>=1.0.5,<2.0
fixture==1.5
-redis==2.8.0
+redis==2.9.0
unittest2==0.5.1
git+https://github.com/david-e/flask-admin.git@bootstrap3#egg=Flask-Admin-1.0.7
Flask-Assets==0.9
Flask-Babel==0.9
Flask-Cache==0.12
Flask-DebugToolbar==0.9.0
Flask-Email==1.4.4
Flask-Gravatar==0.4.0
Flask-Login==0.2.7
Flask-Principal==0.4.0
Flask-RESTful==0.2.12
Flask-OAuthlib==0.4.3
#Flask-SQLAlchemy==1.0
git+https://github.com/mitsuhiko/flask-sqlalchemy#egg=Flask-SQLAlchemy-1.99
git+https://github.com/inveniosoftware/flask-menu.git#egg=Flask-Menu-0.1
git+https://github.com/inveniosoftware/flask-breadcrumbs.git#egg=Flask-Breadcrumbs-0.1
git+https://github.com/inveniosoftware/flask-registry.git#egg=Flask-Registry-0.1
Flask-Script>=2.0.3
#Flask-Testing==0.4
https://github.com/jarus/flask-testing/zipball/master#egg=Flask-Testing-0.4.1
Flask-WTF==0.9.5
Cerberus==0.5
#Flask-Collect>=0.2.2
git+https://github.com/greut/Flask-Collect.git@setup-py-fix#egg=Flask-Collect-0.2.3-dev
Sphinx
alembic==0.6.2
git+https://github.com/lnielsen-cern/setuptools-bower.git#egg=setuptools-bower-0.1
pytz
#PyLD=0.5.0 + python 2.6 patches
git+https://github.com/inveniosoftware/pyld.git@python2.6#egg=PyLD-26.0.5.0-dev
SQLAlchemy-Utils<0.24.0
wtforms-alchemy==0.12.4
mock==1.0.1
setuptools>=2.0
httpretty==0.8.0
raven==4.2.1
six
+unidecode==0.04.14
+nydus==0.10.6

Event Timeline