diff --git a/INSTALL b/INSTALL index 1afde67bc..51afc1799 100644 --- a/INSTALL +++ b/INSTALL @@ -1,642 +1,640 @@ CDS Invenio v0.99.1 INSTALL =========================== About ===== This document specifies how to build, customize, and install CDS Invenio v0.99.1 for the first time. See RELEASE-NOTES if you are upgrading from a previous CDS Invenio release. Contents ======== 0. Prerequisites 1. Quick instructions for the impatient CDS Invenio admin 2. Detailed instructions for the patient CDS Invenio admin 0. Prerequisites ================ Here is the software you need to have around before you start installing CDS Invenio: a) Unix-like operating system. The main development and production platforms for CDS Invenio at CERN are GNU/Linux distributions Debian, Gentoo, Scientific Linux (aka RHEL), Ubuntu, but we also develop on Mac OS X. Basically any Unix system supporting the software listed below should do. If you are using Debian GNU/Linux ``Lenny'' or later, then you can install most of the below-mentioned prerequisites and recommendations by running: $ sudo aptitude install python-dev apache2-mpm-prefork \ mysql-server mysql-client python-mysqldb \ python-4suite-xml python-simplejson python-xml \ python-libxml2 python-libxslt1 gnuplot poppler-utils \ gs-common antiword catdoc wv html2text ppthtml xlhtml \ clisp gettext libapache2-mod-wsgi unzip python-numpy \ python-rdflib python-gnuplot python-magic pdftk \ html2text giflib-tools pstotext You may also want to install some of the following packages, if you have them available on your concrete architecture: $ sudo aptitude install rxp python-psyco sbcl cmucl \ pylint pychecker pyflakes python-profiler python-epydoc Moreover, you should install some Message Transfer Agent (MTA) such as Postfix so that CDS Invenio can email notification alerts or registration information to the end users, contact moderators and reviewers of submitted documents, inform administrators about various runtime system information, etc: $ sudo aptitude install postfix After running the above-quoted aptitude command(s), you can proceed to configuring your MySQL server instance (max_allowed_packet in my.cnf, see item 0b below) and then to installing the CDS Invenio software package in the section 1 below. If you are using another operating system, then please continue reading the rest of this prerequisites section, and please consult our wiki pages for any concrete hints for your specific operating system. b) MySQL server (may be on a remote machine), and MySQL client (must be available locally too). MySQL versions 4.1 or 5.0 are supported. Please set the variable "max_allowed_packet" in your "my.cnf" init file to at least 4M. You may perhaps also want to run your MySQL server natively in UTF-8 mode by setting "default-character-set=utf8" in various parts of your "my.cnf" file, such as in the "[mysql]" part and elsewhere; but this is not really required. c) Apache 2 server, with support for loading DSO modules, and optionally with SSL support for HTTPS-secure user authentication. d) Python v2.4 or above: as well as the following Python modules: - (mandatory) MySQLdb (version >= 1.2.1_p2; see below) - (recommended) PyXML, for XML processing: - (recommended) PyRXP, for very fast XML MARC processing: - (recommended) libxml2-python, for XML/XLST processing: - (recommended) simplejson, for AJAX apps: Note that if you are using Python-2.6, you don't need to install simplejson, because the module is already included in the main Python distribution. - (recommended) Gnuplot.Py, for producing graphs: - (recommended) Snowball Stemmer, for stemming: - (recommended) py-editdist, for record merging: - (recommended) numpy, for citerank methods: - (recommended) magic, for full-text file handling: - (optional) 4suite, slower alternative to PyRXP and libxml2-python: - (optional) feedparser, for web journal creation: - (optional) Psyco, if you are running on a 32-bit OS: - (optional) RDFLib, to use RDF ontologies and thesauri: - (optional) mechanize, to run regression web test suite: Note: MySQLdb version 1.2.1_p2 or higher is recommended. If you are using an older version of MySQLdb, you may get into problems with character encoding. e) mod_wsgi Apache module. Note: for the time being, the WSGI daemon must be run with threads=1, because Invenio is not fully thread safe yet. This will come later. The Apache configuration example snippets (created below) will use threads=1. Note: if you are using Python 2.4 or earlier, then you should also install the wsgiref Python module, available from: (As of Python 2.5 this module is included in standard Python distribution.) f) If you want to be able to extract references from PDF fulltext files, then you need to install pdftotext version 3 at least. g) If you want to be able to search for words in the fulltext files (i.e. to have fulltext indexing) or to stamp submitted files, then you need as well to install some of the following tools: - for PDF file stamping: pdftk, pdf2ps - for PDF files: pdftotext or pstotext - for PostScript files: pstotext or ps2ascii - for MS Word files: antiword, catdoc, or wvText - for MS PowerPoint files: pptHtml and html2text - for MS Excel files: xlhtml and html2text h) If you have chosen to install fast XML MARC Python processors in the step d) above, then you have to install the parsers themselves: - - (optional) RXP: - - (optional) 4suite: i) (recommended) Gnuplot, the command-line driven interactive plotting program. It is used to display download and citation history graphs on the Detailed record pages on the web interface. Note that Gnuplot must be compiled with PNG output support, that is, with the GD library. Note also that Gnuplot is not required, only recommended. j) (recommended) A Common Lisp implementation, such as CLISP, SBCL or CMUCL. It is used for the web server log analysing tool and the metadata checking program. Note that any of the three implementations CLISP, SBCL, or CMUCL will do. CMUCL produces fastest machine code, but it does not support UTF-8 yet. Pick up CLISP if you don't know what to do. Note that a Common Lisp implementation is not required, only recommended. k) GNU gettext, a set of tools that makes it possible to translate the application in multiple languages. This is available by default on many systems. Note that the configure script checks whether you have all the prerequisite software installed and that it won't let you continue unless everything is in order. It also warns you if it cannot find some optional but recommended software. 1. Quick instructions for the impatient CDS Invenio admin ========================================================= 1a. Installation ---------------- $ cd $HOME/src/ $ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz $ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz.md5 $ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz.sig $ md5sum -v -c cds-invenio-0.99.1.tar.gz.md5 $ gpg --verify cds-invenio-0.99.1.tar.gz.sig cds-invenio-0.99.1.tar.gz $ tar xvfz cds-invenio-0.99.1.tar.gz $ cd cds-invenio-0.99.1 $ ./configure $ make $ make install $ make install-jsmath-plugin ## optional $ make install-jquery-plugins ## optional $ make install-fckeditor-plugin ## optional 1b. Configuration ----------------- $ sudo chown -R www-data.www-data /opt/cds-invenio $ sudo -u www-data emacs /opt/cds-invenio/etc/invenio-local.conf $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --update-all $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-tables $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --load-webstat-conf $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-apache-conf $ sudo /etc/init.d/apache2 restart $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-demo-site $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --load-demo-records $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-unit-tests $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-regression-tests $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-web-tests $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --remove-demo-records $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --drop-demo-site $ firefox http://your.site.com/help/admin/howto-run 2. Detailed instructions for the patient CDS Invenio admin ========================================================== 2a. Installation ---------------- The CDS Invenio uses standard GNU autoconf method to build and install its files. This means that you proceed as follows: $ cd $HOME/src/ Change to a directory where we will build the CDS Invenio sources. (The built files will be installed into different "target" directories later.) $ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz $ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz.md5 $ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz.sig Fetch CDS Invenio source tarball from the CDS Software Consortium distribution server, together with MD5 checksum and GnuPG cryptographic signature files useful for verifying the integrity of the tarball. $ md5sum -v -c cds-invenio-0.99.1.tar.gz.md5 Verify MD5 checksum. $ gpg --verify cds-invenio-0.99.1.tar.gz.sig cds-invenio-0.99.1.tar.gz Verify GnuPG cryptographic signature. Note that you may first have to import my public key into your keyring, if you haven't done that already: $ gpg --keyserver wwwkeys.eu.pgp.net --recv-keys 0xBA5A2B67 The output of the gpg --verify command should then read: Good signature from "Tibor Simko " You can safely ignore any trusted signature certification warning that may follow after the signature has been successfully verified. $ tar xvfz cds-invenio-0.99.1.tar.gz Untar the distribution tarball. $ cd cds-invenio-0.99.1 Go to the source directory. $ ./configure Configure CDS Invenio software for building on this specific platform. You can use the following optional parameters: --prefix=/opt/cds-invenio Optionally, specify the CDS Invenio general installation directory (default is /opt/cds-invenio). It will contain command-line binaries and program libraries containing the core CDS Invenio functionality, but also store web pages, runtime log and cache information, document data files, etc. Several subdirs like `bin', `etc', `lib', or `var' will be created inside the prefix directory to this effect. Note that the prefix directory should be chosen outside of the Apache htdocs tree, since only one its subdirectory (prefix/var/www) is to be accessible directly via the Web (see below). Note that CDS Invenio won't install to any other directory but to the prefix mentioned in this configuration line. --with-python=/opt/python/bin/python2.4 Optionally, specify a path to some specific Python binary. This is useful if you have more than one Python installation on your system. If you don't set this option, then the first Python that will be found in your PATH will be chosen for running CDS Invenio. --with-mysql=/opt/mysql/bin/mysql Optionally, specify a path to some specific MySQL client binary. This is useful if you have more than one MySQL installation on your system. If you don't set this option, then the first MySQL client executable that will be found in your PATH will be chosen for running CDS Invenio. --with-clisp=/opt/clisp/bin/clisp Optionally, specify a path to CLISP executable. This is useful if you have more than one CLISP installation on your system. If you don't set this option, then the first executable that will be found in your PATH will be chosen for running CDS Invenio. --with-cmucl=/opt/cmucl/bin/lisp Optionally, specify a path to CMUCL executable. This is useful if you have more than one CMUCL installation on your system. If you don't set this option, then the first executable that will be found in your PATH will be chosen for running CDS Invenio. --with-sbcl=/opt/sbcl/bin/sbcl Optionally, specify a path to SBCL executable. This is useful if you have more than one SBCL installation on your system. If you don't set this option, then the first executable that will be found in your PATH will be chosen for running CDS Invenio. This configuration step is mandatory. Usually, you do this step only once. (Note that if you are building CDS Invenio not from a released tarball, but from the Git sources, then you have to generate the configure file via autotools: $ sudo aptitude install automake1.9 autoconf $ aclocal-1.9 $ automake-1.9 -a $ autoconf after which you proceed with the usual configure command.) $ make Launch the CDS Invenio build. Since many messages are printed during the build process, you may want to run it in a fast-scrolling terminal such as rxvt or in a detached screen session. During this step all the pages and scripts will be pre-created and customized based on the config you have edited in the previous step. Note that on systems such as FreeBSD or Mac OS X you have to use GNU make ("gmake") instead of "make". $ make install Install the web pages, scripts, utilities and everything needed for CDS Invenio runtime into respective installation directories, as specified earlier by the configure command. Note that if you are installing CDS Invenio for the first time, you will be asked to create symbolic link(s) from Python's site-packages system-wide directory(ies) to the installation location. This is in order to instruct Python where to find CDS Invenio's Python files. You will be hinted as to the exact command to use based on the parameters you have used in the configure command. $ make install-jsmath-plugin ## optional This will automatically download and install in the proper place jsMath, a Javascript library to render LaTeX formulas in the client browser. Note that in order to enable the rendering you will have to set the variable CFG_WEBSEARCH_USE_JSMATH_FOR_FORMATS in invenio-local.conf to a suitable list of output format codes. For example: CFG_WEBSEARCH_USE_JSMATH_FOR_FORMATS = hd,hb $ make install-jquery-plugins ## optional This will automatically download and install in the proper place jQuery and related plugins. They are used for AJAX applications such as the record editor. Note that `unzip' is needed when installing jquery plugins. $ make install-fckeditor-plugin ## optional This will automatically download and install in the proper place FCKeditor, a WYSIWYG Javascript-based editor (e.g. for the WebComment module). Note that in order to enable the editor you have to set the CFG_WEBCOMMENT_USE_FCKEDITOR to True. 2b. Configuration ----------------- Once the basic software installation is done, we proceed to configuring your Invenio system. $ sudo chown -R www-data.www-data /opt/cds-invenio For the sake of simplicity, let us assume that your CDS Invenio installation will run under the `www-data' user process identity. The above command changes ownership of installed files to www-data, so that we shall run everything under this user identity from now on. For production purposes, you would typically enable Apache server to read all files from the installation place but to write only to the `var' subdirectory of your installation place. You could achieve this by configuring Unix directory group permissions, for example. $ sudo -u www-data emacs /opt/cds-invenio/etc/invenio-local.conf Customize your CDS Invenio installation. Please read the 'invenio.conf' file located in the same directory that contains the vanilla default configuration parameters of your CDS Invenio installation. If you want to customize some of these parameters, you should create a file named 'invenio-local.conf' in the same directory where 'invenio.conf' lives and write there only the customizations that you want to be different from the vanilla defaults. Here is a minimalist example what you would put there: $ cat /opt/cds-invenio/etc/invenio-local.conf [Invenio] CFG_SITE_URL = http://your.site.com CFG_SITE_SECURE_URL = https://your.site.com CFG_SITE_ADMIN_EMAIL = john.doe@your.site.com CFG_SITE_SUPPORT_EMAIL = john.doe@your.site.com You should override at least the parameters from the top of invenio.conf file in order to define some very essential runtime parameters such as the visible URL of your document server (look for CFG_SITE_URL and CFG_SITE_SECURE_URL), the database credentials (look for CFG_DATABASE_*), the name of your document server (look for CFG_SITE_NAME and CFG_SITE_NAME_INTL_*), or the email address of the local CDS Invenio administrator (look for CFG_SITE_SUPPORT_EMAIL and CFG_SITE_ADMIN_EMAIL). The CDS Invenio system will then read both the default invenio.conf file and your customized invenio-local.conf file and it will override any default options with the ones you have specifield in your local file. This cascading of configuration parameters will ease your future upgrades. $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --update-all Make the rest of the Invenio system aware of your invenio-local.conf changes. This step is mandatory each time you edit your conf files. $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-tables If you are installing CDS Invenio for the first time, you have to create database tables. Note that this step checks for potential problems such as the database connection rights and may ask you to perform some more administrative steps in case it detects a problem. Notably, it may ask you to set up database access permissions, based on your configure values. If you are installing CDS Invenio for the first time, you have to create a dedicated database on your MySQL server that the CDS Invenio can use for its purposes. Please contact your MySQL administrator and ask him to execute the commands this step proposes you. At this point you should now have successfully completed the "make install" process. We continue by setting up the Apache web server. $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --load-webstat-conf Load the configuration file of webstat module. It will create the tables in the database for register customevents, such as basket hits. $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-apache-conf Running this command will generate Apache virtual host configurations matching your installation. You will be instructed to check created files (usually they are located under /opt/cds-invenio/etc/apache/) and edit your httpd.conf to activate Invenio virtual hosts. If you are using Debian GNU/Linux ``Lenny'' or later, then you can do the following to create your SSL certificate and to activate your Invenio vhosts: ## make SSL certificate: $ sudo aptitude install ssl-cert $ sudo mkdir /etc/apache2/ssl $ sudo /usr/sbin/make-ssl-cert /usr/share/ssl-cert/ssleay.cnf \ /etc/apache2/ssl/apache.pem ## add Invenio web sites: $ sudo ln -s /opt/cds-invenio/etc/apache/invenio-apache-vhost.conf \ /etc/apache2/sites-available/invenio $ sudo ln -s /opt/cds-invenio/etc/apache/invenio-apache-vhost-ssl.conf \ /etc/apache2/sites-available/invenio-ssl ## disable Debian's default web site: $ sudo /usr/sbin/a2dissite default ## enable Invenio web sites: $ sudo /usr/sbin/a2ensite invenio $ sudo /usr/sbin/a2ensite invenio-ssl ## enable SSL module: $ sudo /usr/sbin/a2enmod ssl If you are using another operating system, you should do the equivalent, for example edit your system-wide httpd.conf and put the following include statements: Include /opt/cds-invenio/etc/apache/invenio-apache-vhost.conf Include /opt/cds-invenio/etc/apache/invenio-apache-vhost-ssl.conf Note that you may need to adapt generated vhost file snippets to match your concrete operating system specifics. Note also that you may want to tweak the generated example configurations, especially with respect to the WSGIDaemonProcess parameters. E.g. increase the `processes' parameter if you have lots of RAM and many concurrent users accessing your site in parallel. $ sudo /etc/init.d/apache2 restart Please ask your webserver administrator to restart the Apache server after the above "httpd.conf" changes. $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-demo-site This step is recommended to test your local CDS Invenio installation. It should give you our "Atlantis Institute of Science" demo installation, exactly as you see it at . $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --load-demo-records Optionally, load some demo records to be able to test indexing and searching of your local CDS Invenio demo installation. $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-unit-tests Optionally, you can run the unit test suite to verify the unit behaviour of your local CDS Invenio installation. Note that this command should be run only after you have installed the whole system via `make install'. $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-regression-tests Optionally, you can run the full regression test suite to verify the functional behaviour of your local CDS Invenio installation. Note that this command requires to have created the demo site and loaded the demo records. Note also that running the regression test suite may alter the database content with junk data, so that rebuilding the demo site is strongly recommended afterwards. $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-web-tests Optionally, you can run additional automated web tests running in a real browser. This requires to have Firefox with the Selenium IDE extension installed. $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --remove-demo-records Optionally, remove the demo records loaded in the previous step, but keeping otherwise the demo collection, submission, format, and other configurations that you may reuse and modify for your own production purposes. $ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --drop-demo-site Optionally, drop also all the demo configuration so that you'll end up with a completely blank CDS Invenio system. However, you may want to find it more practical not to drop the demo site configuration but to start customizing from there. $ firefox http://your.site.com/help/admin/howto-run In order to start using your CDS Invenio installation, you can start indexing, formatting and other daemons as indicated in the "HOWTO Run" guide on the above URL. You can also use the Admin Area web interfaces to perform further runtime configurations such as the definition of data collections, document types, document formats, word indexes, etc. Good luck, and thanks for choosing CDS Invenio. - CDS Development Group diff --git a/modules/bibedit/lib/bibrecord_tests.py b/modules/bibedit/lib/bibrecord_tests.py index e406498ae..0c48d09c9 100644 --- a/modules/bibedit/lib/bibrecord_tests.py +++ b/modules/bibedit/lib/bibrecord_tests.py @@ -1,1489 +1,1539 @@ # -*- coding: utf-8 -*- ## ## This file is part of CDS Invenio. ## Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 CERN. ## ## CDS Invenio is free software; you can redistribute it and/or ## modify it under the terms of the GNU General Public License as ## published by the Free Software Foundation; either version 2 of the ## License, or (at your option) any later version. ## ## CDS Invenio is distributed in the hope that it will be useful, but ## WITHOUT ANY WARRANTY; without even the implied warranty of ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU ## General Public License for more details. ## ## You should have received a copy of the GNU General Public License ## along with CDS Invenio; if not, write to the Free Software Foundation, Inc., ## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """ The BibRecord test suite. """ import unittest from invenio.config import CFG_TMPDIR from invenio import bibrecord, bibrecord_config from invenio.testutils import make_test_suite, run_test_suite try: import pyRXP parser_pyrxp_available = True except ImportError: parser_pyrxp_available = False try: import Ft.Xml.Domlette parser_4suite_available = True except ImportError: parser_4suite_available = False try: import xml.dom.minidom import xml.parsers.expat parser_minidom_available = True except ImportError: parser_minidom_available = False class BibRecordSuccessTest(unittest.TestCase): """ bibrecord - demo file parsing test """ def setUp(self): """Initialize stuff""" f = open(CFG_TMPDIR + '/demobibdata.xml', 'r') xmltext = f.read() f.close() self.recs = [rec[0] for rec in bibrecord.create_records(xmltext)] def test_records_created(self): """ bibrecord - demo file how many records are created """ self.assertEqual(102, len(self.recs)) def test_tags_created(self): """ bibrecord - demo file which tags are created """ ## check if the tags are correct tags = [u'003', u'005', '020', '035', '037', '041', '080', '088', '100', '242', '245', '246', '250', '260', '269', '270', '300', '340', '490', '500', '502', '520', '590', '595', '650', '653', '690', '695', '700', '710', '720', '773', '856', '859', '901', '909', '916', '960', '961', '962', '963', '970', '980', '999', 'FFT'] t = [] for rec in self.recs: t.extend(rec.keys()) t.sort() #eliminate the elements repeated tt = [] for x in t: if not x in tt: tt.append(x) self.assertEqual(tags, tt) def test_fields_created(self): """bibrecord - demo file how many fields are created""" ## check if the number of fields for each record is correct fields = [14, 14, 8, 11, 11, 12, 11, 15, 10, 18, 14, 16, 10, 9, 15, 10, 11, 11, 11, 9, 11, 11, 10, 9, 9, 9, 10, 9, 10, 10, 8, 9, 8, 9, 14, 13, 14, 14, 15, 12, 12, 12, 15, 14, 12, 16, 16, 15, 15, 14, 16, 15, 15, 15, 16, 15, 16, 15, 15, 16, 15, 14, 14, 15, 12, 13, 11, 15, 8, 11, 14, 13, 12, 13, 6, 6, 25, 24, 27, 26, 26, 24, 26, 27, 25, 28, 24, 23, 27, 25, 25, 26, 26, 24, 19, 26, 9, 8, 9, 9, 8, 7] cr = [] ret = [] for rec in self.recs: cr.append(len(rec.values())) ret.append(rec) self.assertEqual(fields, cr) def test_create_record_with_collection_tag(self): """ bibrecord - create_record() for single record in collection""" xmltext = """ 33 eng """ record = bibrecord.create_record(xmltext) record1 = bibrecord.create_records(xmltext)[0] self.assertEqual(record1, record) class BibRecordParsersTest(unittest.TestCase): """ bibrecord - testing the creation of records with different parsers""" def setUp(self): """Initialize stuff""" self.xmltext = """ 33 eng """ self.expected_record = { '001': [([], ' ', ' ', '33', 1)], '041': [([('a', 'eng')], ' ', ' ', '', 2)] } if parser_pyrxp_available: def test_pyRXP(self): """ bibrecord - create_record() with pyRXP """ record = bibrecord._create_record_rxp(self.xmltext) self.assertEqual(record, self.expected_record) if parser_4suite_available: def test_4suite(self): """ bibrecord - create_record() with 4suite """ record = bibrecord._create_record_4suite(self.xmltext) self.assertEqual(record, self.expected_record) if parser_minidom_available: def test_minidom(self): """ bibrecord - create_record() with minidom """ record = bibrecord._create_record_minidom(self.xmltext) self.assertEqual(record, self.expected_record) class BibRecordBadInputTreatmentTest(unittest.TestCase): """ bibrecord - testing for bad input treatment """ def test_empty_collection(self): """bibrecord - empty collection""" xml_error0 = """""" rec = bibrecord.create_record(xml_error0)[0] self.assertEqual(rec, {}) records = bibrecord.create_records(xml_error0) self.assertEqual(len(records), 0) def test_wrong_attribute(self): """bibrecord - bad input subfield \'cde\' instead of \'code\'""" ws = bibrecord.CFG_BIBRECORD_WARNING_MSGS xml_error1 = """ 33 eng Doe, John On the foo and bar """ e = bibrecord.create_record(xml_error1, 1, 1)[2] ee ='' for i in e: if type(i).__name__ == 'str': if i.count(ws[3])>0: ee = i self.assertEqual(bibrecord._warning((3, '(field number: 4)')), ee) def test_missing_attribute(self): """ bibrecord - bad input missing \"tag\" """ ws = bibrecord.CFG_BIBRECORD_WARNING_MSGS xml_error2 = """ 33 eng Doe, John On the foo and bar """ e = bibrecord.create_record(xml_error2, 1, 1)[2] ee = '' for i in e: if type(i).__name__ == 'str': if i.count(ws[1])>0: ee = i self.assertEqual(bibrecord._warning((1, '(field number(s): [2])')), ee) def test_empty_datafield(self): """ bibrecord - bad input no subfield """ ws = bibrecord.CFG_BIBRECORD_WARNING_MSGS xml_error3 = """ 33 Doe, John On the foo and bar """ e = bibrecord.create_record(xml_error3, 1, 1)[2] ee = '' for i in e: if type(i).__name__ == 'str': if i.count(ws[8])>0: ee = i self.assertEqual(bibrecord._warning((8, '(field number: 2)')), ee) def test_missing_tag(self): """bibrecord - bad input missing end \"tag\" """ ws = bibrecord.CFG_BIBRECORD_WARNING_MSGS xml_error4 = """ 33 eng Doe, John On the foo and bar """ e = bibrecord.create_record(xml_error4, 1, 1)[2] ee = '' for i in e: if type(i).__name__ == 'str': if i.count(ws[99])>0: ee = i self.assertEqual(bibrecord._warning((99, '(Tagname : datafield)')), ee) class BibRecordAccentedUnicodeLettersTest(unittest.TestCase): """ bibrecord - testing accented UTF-8 letters """ def setUp(self): """Initialize stuff""" self.xml_example_record = """ 33 eng Döè1, John Doe2, J>ohn editor Пушкин On the foo and bar2 """ self.rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0] def test_accented_unicode_characters(self): """bibrecord - accented Unicode letters""" self.assertEqual(self.xml_example_record, bibrecord.record_xml_output(self.rec)) self.assertEqual(bibrecord.record_get_field_instances(self.rec, "100", " ", " "), [([('a', 'Döè1, John')], " ", " ", "", 3), ([('a', 'Doe2, J>ohn'), ('b', 'editor')], " ", " ", "", 4)]) self.assertEqual(bibrecord.record_get_field_instances(self.rec, "245", " ", "1"), [([('a', 'Пушкин')], " ", '1', "", 5)]) class BibRecordGettingFieldValuesTest(unittest.TestCase): """ bibrecord - testing for getting field/subfield values """ def setUp(self): """Initialize stuff""" xml_example_record = """ 33 eng Doe1, John Doe2, John editor On the foo and bar1 On the foo and bar2 """ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0] def test_get_field_instances(self): """bibrecord - getting field instances""" self.assertEqual(bibrecord.record_get_field_instances(self.rec, "100", " ", " "), [([('a', 'Doe1, John')], " ", " ", "", 3), ([('a', 'Doe2, John'), ('b', 'editor')], " ", " ", "", 4)]) self.assertEqual(bibrecord.record_get_field_instances(self.rec, "", " ", " "), [('245', [([('a', 'On the foo and bar1')], " ", '1', "", 5), ([('a', 'On the foo and bar2')], " ", '2', "", 6)]), ('001', [([], " ", " ", '33', 1)]), ('100', [([('a', 'Doe1, John')], " ", " ", "", 3), ([('a', 'Doe2, John'), ('b', 'editor')], " ", " ", "", 4)]), ('041', [([('a', 'eng')], " ", " ", "", 2)])]) def test_get_field_values(self): """bibrecord - getting field values""" self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "a"), ['Doe1, John', 'Doe2, John']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "b"), ['editor']) def test_get_field_value(self): """bibrecord - getting first field value""" self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", " ", "a"), 'Doe1, John') self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", " ", "b"), 'editor') def test_get_subfield_values(self): """bibrecord - getting subfield values""" fi1, fi2 = bibrecord.record_get_field_instances(self.rec, "100", " ", " ") self.assertEqual(bibrecord.field_get_subfield_values(fi1, "b"), []) self.assertEqual(bibrecord.field_get_subfield_values(fi2, "b"), ["editor"]) class BibRecordGettingFieldValuesViaWildcardsTest(unittest.TestCase): """ bibrecord - testing for getting field/subfield values via wildcards """ def setUp(self): """Initialize stuff""" xml_example_record = """ 1 val1 val2 val3 val4a val4b val5 val6 val7a val7b """ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0] def test_get_field_instances_via_wildcard(self): """bibrecord - getting field instances via wildcards""" self.assertEqual(bibrecord.record_get_field_instances(self.rec, "100", " ", " "), []) self.assertEqual(bibrecord.record_get_field_instances(self.rec, "100", "%", " "), []) self.assertEqual(bibrecord.record_get_field_instances(self.rec, "100", "%", "%"), [([('a', 'val1')], 'C', '5', "", 2)]) self.assertEqual(bibrecord.record_get_field_instances(self.rec, "55%", "A", "%"), [([('a', 'val2')], 'A', 'B', "", 3), ([('a', 'val3')], 'A', " ", "", 4), ([('a', 'val6')], 'A', 'C', "", 7), ([('a', 'val7a'), ('b', 'val7b')], 'A', " ", "", 8)]) self.assertEqual(bibrecord.record_get_field_instances(self.rec, "55%", "A", " "), [([('a', 'val3')], 'A', " ", "", 4), ([('a', 'val7a'), ('b', 'val7b')], 'A', " ", "", 8)]) self.assertEqual(bibrecord.record_get_field_instances(self.rec, "556", "A", " "), [([('a', 'val7a'), ('b', 'val7b')], 'A', " ", "", 8)]) def test_get_field_values_via_wildcard(self): """bibrecord - getting field values via wildcards""" self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", " "), []) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", " ", " "), []) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", "%", " "), []) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", "%", " "), []) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", "%", "z"), []) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "%"), []) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "a"), []) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", " ", "a"), []) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", "%", "a"), ['val1']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", "%", "%", "%"), ['val1']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", "%", "a"), ['val2', 'val3', 'val6', 'val7a']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", " ", "a"), ['val3', 'val7a']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "556", "A", " ", "a"), ['val7a']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "555", " ", " ", " "), []) self.assertEqual(bibrecord.record_get_field_values(self.rec, "555", " ", " ", "z"), []) self.assertEqual(bibrecord.record_get_field_values(self.rec, "555", " ", " ", "%"), ['val4a', 'val4b']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", " ", " ", "b"), ['val4b']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "%", "%", "b"), ['val4b', 'val7b']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", " ", "b"), ['val7b']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", "%", "b"), ['val7b']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", " ", "a"), ['val3', 'val7a']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "A", "%", "a"), ['val2', 'val3', 'val6', 'val7a']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", "%", "%", "a"), ['val2', 'val3', 'val4a', 'val5', 'val6', 'val7a']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "55%", " ", " ", "a"), ['val4a']) def test_get_field_value_via_wildcard(self): """bibrecord - getting first field value via wildcards""" self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", " ", " "), '') self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", "%", " ", " "), '') self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", "%", " "), '') self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", "%", "%", " "), '') self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", " ", "%"), '') self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", " ", " ", "a"), '') self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", "%", " ", "a"), '') self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", "%", "%", "a"), 'val1') self.assertEqual(bibrecord.record_get_field_value(self.rec, "100", "%", "%", "%"), 'val1') self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", "%", "a"), 'val2') self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", " ", "a"), 'val3') self.assertEqual(bibrecord.record_get_field_value(self.rec, "556", "A", " ", "a"), 'val7a') self.assertEqual(bibrecord.record_get_field_value(self.rec, "555", " ", " ", " "), '') self.assertEqual(bibrecord.record_get_field_value(self.rec, "555", " ", " ", "%"), 'val4a') self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", " ", " ", "b"), 'val4b') self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "%", "%", "b"), 'val4b') self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", " ", "b"), 'val7b') self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", "%", "b"), 'val7b') self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", " ", "a"), 'val3') self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "A", "%", "a"), 'val2') self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", "%", "%", "a"), 'val2') self.assertEqual(bibrecord.record_get_field_value(self.rec, "55%", " ", " ", "a"), 'val4a') class BibRecordAddFieldTest(unittest.TestCase): """ bibrecord - testing adding field """ def setUp(self): """Initialize stuff""" xml_example_record = """ 33 eng Doe1, John Doe2, John editor On the foo and bar1 On the foo and bar2 """ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0] def test_add_controlfield(self): """bibrecord - adding controlfield""" field_position_global_1 = bibrecord.record_add_field(self.rec, "003", controlfield_value="SzGeCERN") field_position_global_2 = bibrecord.record_add_field(self.rec, "004", controlfield_value="Test") self.assertEqual(field_position_global_1, 2) self.assertEqual(field_position_global_2, 3) self.assertEqual(bibrecord.record_get_field_values(self.rec, "003", " ", " ", ""), ['SzGeCERN']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "004", " ", " ", ""), ['Test']) def test_add_datafield(self): """bibrecord - adding datafield""" field_position_global_1 = bibrecord.record_add_field(self.rec, "100", subfields=[('a', 'Doe3, John')]) field_position_global_2 = bibrecord.record_add_field(self.rec, "100", subfields= [('a', 'Doe4, John'), ('b', 'editor')]) self.assertEqual(field_position_global_1, 5) self.assertEqual(field_position_global_2, 6) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "a"), ['Doe1, John', 'Doe2, John', 'Doe3, John', 'Doe4, John']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "b"), ['editor', 'editor']) def test_add_controlfield_on_desired_position(self): """bibrecord - adding controlfield on desired position""" field_position_global_1 = bibrecord.record_add_field(self.rec, "005", controlfield_value="Foo", field_position_global=0) field_position_global_2 = bibrecord.record_add_field(self.rec, "006", controlfield_value="Bar", field_position_global=0) self.assertEqual(field_position_global_1, 7) self.assertEqual(field_position_global_2, 8) def test_add_datafield_on_desired_position_field_position_global(self): """bibrecord - adding datafield on desired global field position""" field_position_global_1 = bibrecord.record_add_field(self.rec, "100", subfields=[('a', 'Doe3, John')], field_position_global=0) field_position_global_2 = bibrecord.record_add_field(self.rec, "100", subfields=[('a', 'Doe4, John'), ('b', 'editor')], field_position_global=0) self.assertEqual(field_position_global_1, 3) self.assertEqual(field_position_global_2, 3) def test_add_datafield_on_desired_position_field_position_local(self): """bibrecord - adding datafield on desired local field position""" field_position_global_1 = bibrecord.record_add_field(self.rec, "100", subfields=[('a', 'Doe3, John')], field_position_local=0) field_position_global_2 = bibrecord.record_add_field(self.rec, "100", subfields=[('a', 'Doe4, John'), ('b', 'editor')], field_position_local=2) self.assertEqual(field_position_global_1, 3) self.assertEqual(field_position_global_2, 5) class BibRecordManageMultipleFieldsTest(unittest.TestCase): """ bibrecord - testing the management of multiple fields """ def setUp(self): """Initialize stuff""" xml_example_record = """ 33 subfield1 subfield2 subfield3 subfield4 """ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0] def test_delete_multiple_datafields(self): """bibrecord - deleting multiple datafields""" self.fields = bibrecord.record_delete_fields(self.rec, '245', [1, 2]) self.assertEqual(self.fields[0], ([('a', 'subfield2')], ' ', ' ', '', 3)) self.assertEqual(self.fields[1], ([('a', 'subfield3')], ' ', ' ', '', 4)) def test_add_multiple_datafields_default_index(self): """bibrecord - adding multiple fields with the default index""" fields = [([('a', 'subfield5')], ' ', ' ', '', 4), ([('a', 'subfield6')], ' ', ' ', '', 19)] index = bibrecord.record_add_fields(self.rec, '245', fields) self.assertEqual(index, None) self.assertEqual(self.rec['245'][-2], ([('a', 'subfield5')], ' ', ' ', '', 6)) self.assertEqual(self.rec['245'][-1], ([('a', 'subfield6')], ' ', ' ', '', 7)) def test_add_multiple_datafields_with_index(self): """bibrecord - adding multiple fields with an index""" fields = [([('a', 'subfield5')], ' ', ' ', '', 4), ([('a', 'subfield6')], ' ', ' ', '', 19)] index = bibrecord.record_add_fields(self.rec, '245', fields, field_position_local=0) self.assertEqual(index, 0) self.assertEqual(self.rec['245'][0], ([('a', 'subfield5')], ' ', ' ', '', 2)) self.assertEqual(self.rec['245'][1], ([('a', 'subfield6')], ' ', ' ', '', 3)) self.assertEqual(self.rec['245'][2], ([('a', 'subfield1')], ' ', ' ', '', 4)) def test_move_multiple_fields(self): """bibrecord - move multiple fields""" bibrecord.record_move_fields(self.rec, '245', [1, 3]) self.assertEqual(self.rec['245'][0], ([('a', 'subfield1')], ' ', ' ', '', 2)) self.assertEqual(self.rec['245'][1], ([('a', 'subfield3')], ' ', ' ', '', 4)) self.assertEqual(self.rec['245'][2], ([('a', 'subfield2')], ' ', ' ', '', 5)) self.assertEqual(self.rec['245'][3], ([('a', 'subfield4')], ' ', ' ', '', 6)) class BibRecordDeleteFieldTest(unittest.TestCase): """ bibrecord - testing field deletion """ def setUp(self): """Initialize stuff""" xml_example_record = """ 33 eng Doe1, John Doe2, John editor On the foo and bar1 On the foo and bar2 """ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0] xml_example_record_empty = """ """ self.rec_empty = bibrecord.create_record(xml_example_record_empty, 1, 1)[0] def test_delete_controlfield(self): """bibrecord - deleting controlfield""" bibrecord.record_delete_field(self.rec, "001", " ", " ") self.assertEqual(bibrecord.record_get_field_values(self.rec, "001", " ", " ", " "), []) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "b"), ['editor']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "2", "a"), ['On the foo and bar2']) def test_delete_datafield(self): """bibrecord - deleting datafield""" bibrecord.record_delete_field(self.rec, "100", " ", " ") self.assertEqual(bibrecord.record_get_field_values(self.rec, "001", " ", " ", ""), ['33']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "b"), []) bibrecord.record_delete_field(self.rec, "245", " ", " ") self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "1", "a"), ['On the foo and bar1']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "2", "a"), ['On the foo and bar2']) bibrecord.record_delete_field(self.rec, "245", " ", "2") self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "1", "a"), ['On the foo and bar1']) self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "2", "a"), []) def test_add_delete_add_field_to_empty_record(self): """bibrecord - adding, deleting, and adding back a field to an empty record""" field_position_global_1 = bibrecord.record_add_field(self.rec_empty, "003", controlfield_value="SzGeCERN") self.assertEqual(field_position_global_1, 1) self.assertEqual(bibrecord.record_get_field_values(self.rec_empty, "003", " ", " ", ""), ['SzGeCERN']) bibrecord.record_delete_field(self.rec_empty, "003", " ", " ") self.assertEqual(bibrecord.record_get_field_values(self.rec_empty, "003", " ", " ", ""), []) field_position_global_1 = bibrecord.record_add_field(self.rec_empty, "003", controlfield_value="SzGeCERN2") self.assertEqual(field_position_global_1, 1) self.assertEqual(bibrecord.record_get_field_values(self.rec_empty, "003", " ", " ", ""), ['SzGeCERN2']) class BibRecordDeleteFieldFromTest(unittest.TestCase): """ bibrecord - testing field deletion from position""" def setUp(self): """Initialize stuff""" xml_example_record = """ 33 eng Doe1, John Doe2, John editor On the foo and bar1 On the foo and bar2 """ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0] def test_delete_field_from(self): """bibrecord - deleting field from position""" bibrecord.record_delete_field(self.rec, "100", field_position_global=4) self.assertEqual(self.rec['100'], [([('a', 'Doe1, John')], ' ', ' ', '', 3)]) bibrecord.record_delete_field(self.rec, "100", field_position_global=3) self.failIf(self.rec.has_key('100')) bibrecord.record_delete_field(self.rec, "001", field_position_global=1) bibrecord.record_delete_field(self.rec, "245", field_position_global=6) self.failIf(self.rec.has_key('001')) self.assertEqual(self.rec['245'], [([('a', 'On the foo and bar1')], ' ', '1', '', 5)]) # Some crash tests bibrecord.record_delete_field(self.rec, '999', field_position_global=1) bibrecord.record_delete_field(self.rec, '245', field_position_global=999) class BibRecordAddSubfieldIntoTest(unittest.TestCase): """ bibrecord - testing subfield addition """ def setUp(self): """Initialize stuff""" xml_example_record = """ 33 eng Doe2, John editor On the foo and bar1 On the foo and bar2 """ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0] def test_add_subfield_into(self): """bibrecord - adding subfield into position""" bibrecord.record_add_subfield_into(self.rec, "100", "b", "Samekniv", field_position_global=3) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "b"), ['editor', 'Samekniv']) bibrecord.record_add_subfield_into(self.rec, "245", "x", "Elgokse", field_position_global=4) bibrecord.record_add_subfield_into(self.rec, "245", "x", "Fiskeflue", subfield_position=0, field_position_global=4) bibrecord.record_add_subfield_into(self.rec, "245", "z", "Ulriken", subfield_position=2, field_position_global=4) bibrecord.record_add_subfield_into(self.rec, "245", "z", "Stortinget", subfield_position=999, field_position_global=4) self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "1", "%"), ['Fiskeflue', 'On the foo and bar1', 'Ulriken', 'Elgokse', 'Stortinget']) # Some crash tests self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_add_subfield_into, self.rec, "187", "x", "Crash", field_position_global=1) self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_add_subfield_into, self.rec, "245", "x", "Crash", field_position_global=999) class BibRecordModifyControlfieldTest(unittest.TestCase): """ bibrecord - testing controlfield modification """ def setUp(self): """Initialize stuff""" xml_example_record = """ 33 A Foo's Tale Skeech Skeech Whoop Whoop eng On the foo and bar2 """ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0] def test_modify_controlfield(self): """bibrecord - modify controlfield""" bibrecord.record_modify_controlfield(self.rec, "001", "34", field_position_global=1) bibrecord.record_modify_controlfield(self.rec, "008", "Foo Foo", field_position_global=3) self.assertEqual(bibrecord.record_get_field_values(self.rec, "001"), ["34"]) self.assertEqual(bibrecord.record_get_field_values(self.rec, "005"), ["A Foo's Tale"]) self.assertEqual(bibrecord.record_get_field_values(self.rec, "008"), ["Foo Foo", "Whoop Whoop"]) # Some crash tests self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_modify_controlfield, self.rec, "187", "Crash", field_position_global=1) self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_modify_controlfield, self.rec, "008", "Test", field_position_global=10) self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_modify_controlfield, self.rec, "245", "Burn", field_position_global=5) self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "2", "%"), ["On the foo and bar2"]) class BibRecordModifySubfieldTest(unittest.TestCase): """ bibrecord - testing subfield modification """ def setUp(self): """Initialize stuff""" xml_example_record = """ 33 eng Doe2, John editor On the foo and bar1 On writing unit tests On the foo and bar2 """ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0] def test_modify_subfield(self): """bibrecord - modify subfield""" bibrecord.record_modify_subfield(self.rec, "245", "a", "Holmenkollen", 0, field_position_global=4) bibrecord.record_modify_subfield(self.rec, "245", "x", "Brann", 1, field_position_global=4) self.assertEqual(bibrecord.record_get_field_values(self.rec, "245", " ", "1", "%"), ['Holmenkollen', 'Brann']) # Some crash tests self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_modify_subfield, self.rec, "187", "x", "Crash", 0, field_position_global=1) self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_modify_subfield, self.rec, "245", "x", "Burn", 1, field_position_global=999) self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_modify_subfield, self.rec, "245", "a", "Burn", 999, field_position_global=4) class BibRecordDeleteSubfieldFromTest(unittest.TestCase): """ bibrecord - testing subfield deletion """ def setUp(self): """Initialize stuff""" xml_example_record = """ 33 eng Doe2, John editor Skal vi danse? On the foo and bar1 On the foo and bar2 """ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0] def test_delete_subfield_from(self): """bibrecord - delete subfield from position""" bibrecord.record_delete_subfield_from(self.rec, "100", 2, field_position_global=3) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "z"), []) bibrecord.record_delete_subfield_from(self.rec, "100", 0, field_position_global=3) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "%"), ['editor']) bibrecord.record_delete_subfield_from(self.rec, "100", 0, field_position_global=3) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "%"), []) # Some crash tests self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_delete_subfield_from, self.rec, "187", 0, field_position_global=1) self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_delete_subfield_from, self.rec, "245", 0, field_position_global=999) self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_delete_subfield_from, self.rec, "245", 999, field_position_global=4) class BibRecordDeleteSubfieldTest(unittest.TestCase): """ bibrecord - testing subfield deletion """ def setUp(self): """Initialize stuff""" self.xml_example_record = """ 33 eng Doe2, John editor Skal vi danse? Doe3, Zbigniew Doe4, Joachim On the foo and bar1 On the foo and bar2 On the foo and bar1 On the foo and bar2 """ def test_simple_removals(self): """ bibrecord - delete subfield by its code""" # testing a simple removals where all the fields are removed rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0] bibrecord.record_delete_subfield(rec, "041", "b") # nothing should change self.assertEqual(rec["041"][0][0], [("a", "eng")]) bibrecord.record_delete_subfield(rec, "041", "a") self.assertEqual(rec["041"][0][0], []) def test_indices_important(self): """ bibrecord - delete subfield where indices are important""" rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0] bibrecord.record_delete_subfield(rec, "245", "a", " ", "1") self.assertEqual(rec["245"][0][0], []) self.assertEqual(rec["245"][1][0], [("a", "On the foo and bar2")]) bibrecord.record_delete_subfield(rec, "245", "a", " ", "2") self.assertEqual(rec["245"][1][0], []) def test_remove_some(self): """ bibrecord - delete subfield when some should be preserved and some removed""" rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0] bibrecord.record_delete_subfield(rec, "100", "a", " ", " ") self.assertEqual(rec["100"][0][0], [("b", "editor"), ("z", "Skal vi danse?"), ("d", "Doe4, Joachim")]) def test_more_fields(self): """ bibrecord - delete subfield where more fits criteria""" rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0] bibrecord.record_delete_subfield(rec, "246", "c", "1", "2") self.assertEqual(rec["246"][1][0], []) self.assertEqual(rec["246"][0][0], []) def test_nonexisting_removals(self): """ bibrecord - delete subfield that does not exist """ rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0] # further preparation bibrecord.record_delete_subfield(rec, "100", "a", " ", " ") self.assertEqual(rec["100"][0][0], [("b", "editor"), ("z", "Skal vi danse?"), ("d", "Doe4, Joachim")]) #the real tests begin # 1) removing the subfield from an empty list of subfields bibrecord.record_delete_subfield(rec, "246", "c", "1", "2") self.assertEqual(rec["246"][1][0], []) self.assertEqual(rec["246"][0][0], []) bibrecord.record_delete_subfield(rec, "246", "8", "1", "2") self.assertEqual(rec["246"][1][0], []) self.assertEqual(rec["246"][0][0], []) # 2) removing a subfield from a field that has some subfields but none has an appropriate code bibrecord.record_delete_subfield(rec, "100", "a", " ", " ") self.assertEqual(rec["100"][0][0], [("b", "editor"), ("z", "Skal vi danse?"), ("d", "Doe4, Joachim")]) bibrecord.record_delete_subfield(rec, "100", "e", " ", " ") self.assertEqual(rec["100"][0][0], [("b", "editor"), ("z", "Skal vi danse?"), ("d", "Doe4, Joachim")]) class BibRecordMoveSubfieldTest(unittest.TestCase): """ bibrecord - testing subfield moving """ def setUp(self): """Initialize stuff""" xml_example_record = """ 33 eng Doe2, John editor fisk eple hammer On the foo and bar1 """ self.rec = bibrecord.create_record(xml_example_record, 1, 1)[0] def test_move_subfield(self): """bibrecord - move subfields""" bibrecord.record_move_subfield(self.rec, "100", 2, 4, field_position_global=3) bibrecord.record_move_subfield(self.rec, "100", 1, 0, field_position_global=3) bibrecord.record_move_subfield(self.rec, "100", 2, 999, field_position_global=3) self.assertEqual(bibrecord.record_get_field_values(self.rec, "100", " ", " ", "%"), ['editor', 'Doe2, John', 'hammer', 'fisk', 'eple']) # Some crash tests self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_move_subfield, self.rec, "187", 0, 1, field_position_global=3) self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_move_subfield, self.rec, "100", 1, 0, field_position_global=999) self.assertRaises(bibrecord.InvenioBibRecordFieldError, bibrecord.record_move_subfield, self.rec, "100", 999, 0, field_position_global=3) class BibRecordSpecialTagParsingTest(unittest.TestCase): """ bibrecord - parsing special tags (FMT, FFT)""" def setUp(self): """setting up example records""" self.xml_example_record_with_fmt = """ 33 eng HB Let us see if this gets inserted well. """ self.xml_example_record_with_fft = """ 33 eng file:///foo.pdf http://bar.com/baz.ps.gz """ self.xml_example_record_with_xyz = """ 33 eng HB Let us see if this gets inserted well. """ def test_parsing_file_containing_fmt_special_tag_with_correcting(self): """bibrecord - parsing special FMT tag, correcting on""" rec = bibrecord.create_record(self.xml_example_record_with_fmt, 1, 1)[0] self.assertEqual(rec, {u'001': [([], " ", " ", '33', 1)], 'FMT': [([('f', 'HB'), ('g', 'Let us see if this gets inserted well.')], " ", " ", "", 3)], '041': [([('a', 'eng')], " ", " ", "", 2)]}) self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"), ['eng']) self.assertEqual(bibrecord.record_get_field_values(rec, "FMT", " ", " ", "f"), ['HB']) self.assertEqual(bibrecord.record_get_field_values(rec, "FMT", " ", " ", "g"), ['Let us see if this gets inserted well.']) def test_parsing_file_containing_fmt_special_tag_without_correcting(self): """bibrecord - parsing special FMT tag, correcting off""" rec = bibrecord.create_record(self.xml_example_record_with_fmt, 1, 0)[0] self.assertEqual(rec, {u'001': [([], " ", " ", '33', 1)], 'FMT': [([('f', 'HB'), ('g', 'Let us see if this gets inserted well.')], " ", " ", "", 3)], '041': [([('a', 'eng')], " ", " ", "", 2)]}) self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"), ['eng']) self.assertEqual(bibrecord.record_get_field_values(rec, "FMT", " ", " ", "f"), ['HB']) self.assertEqual(bibrecord.record_get_field_values(rec, "FMT", " ", " ", "g"), ['Let us see if this gets inserted well.']) def test_parsing_file_containing_fft_special_tag_with_correcting(self): """bibrecord - parsing special FFT tag, correcting on""" rec = bibrecord.create_record(self.xml_example_record_with_fft, 1, 1)[0] self.assertEqual(rec, {u'001': [([], " ", " ", '33', 1)], 'FFT': [([('a', 'file:///foo.pdf'), ('a', 'http://bar.com/baz.ps.gz')], " ", " ", "", 3)], '041': [([('a', 'eng')], " ", " ", "", 2)]}) self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"), ['eng']) self.assertEqual(bibrecord.record_get_field_values(rec, "FFT", " ", " ", "a"), ['file:///foo.pdf', 'http://bar.com/baz.ps.gz']) def test_parsing_file_containing_fft_special_tag_without_correcting(self): """bibrecord - parsing special FFT tag, correcting off""" rec = bibrecord.create_record(self.xml_example_record_with_fft, 1, 0)[0] self.assertEqual(rec, {u'001': [([], " ", " ", '33', 1)], 'FFT': [([('a', 'file:///foo.pdf'), ('a', 'http://bar.com/baz.ps.gz')], " ", " ", "", 3)], '041': [([('a', 'eng')], " ", " ", "", 2)]}) self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"), ['eng']) self.assertEqual(bibrecord.record_get_field_values(rec, "FFT", " ", " ", "a"), ['file:///foo.pdf', 'http://bar.com/baz.ps.gz']) def test_parsing_file_containing_xyz_special_tag_with_correcting(self): """bibrecord - parsing unrecognized special XYZ tag, correcting on""" # XYZ should not get accepted when correcting is on; should get changed to 000 rec = bibrecord.create_record(self.xml_example_record_with_xyz, 1, 1)[0] self.assertEqual(rec, {u'001': [([], " ", " ", '33', 1)], '000': [([('f', 'HB'), ('g', 'Let us see if this gets inserted well.')], " ", " ", "", 3)], '041': [([('a', 'eng')], " ", " ", "", 2)]}) self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"), ['eng']) self.assertEqual(bibrecord.record_get_field_values(rec, "XYZ", " ", " ", "f"), []) self.assertEqual(bibrecord.record_get_field_values(rec, "XYZ", " ", " ", "g"), []) self.assertEqual(bibrecord.record_get_field_values(rec, "000", " ", " ", "f"), ['HB']) self.assertEqual(bibrecord.record_get_field_values(rec, "000", " ", " ", "g"), ['Let us see if this gets inserted well.']) def test_parsing_file_containing_xyz_special_tag_without_correcting(self): """bibrecord - parsing unrecognized special XYZ tag, correcting off""" # XYZ should get accepted without correcting rec = bibrecord.create_record(self.xml_example_record_with_xyz, 1, 0)[0] self.assertEqual(rec, {u'001': [([], " ", " ", '33', 1)], 'XYZ': [([('f', 'HB'), ('g', 'Let us see if this gets inserted well.')], " ", " ", "", 3)], '041': [([('a', 'eng')], " ", " ", "", 2)]}) self.assertEqual(bibrecord.record_get_field_values(rec, "041", " ", " ", "a"), ['eng']) self.assertEqual(bibrecord.record_get_field_values(rec, "XYZ", " ", " ", "f"), ['HB']) self.assertEqual(bibrecord.record_get_field_values(rec, "XYZ", " ", " ", "g"), ['Let us see if this gets inserted well.']) class BibRecordPrintingTest(unittest.TestCase): """ bibrecord - testing for printing record """ def setUp(self): """Initialize stuff""" self.xml_example_record = """ 81 TEST-ARTICLE-2006-001 ARTICLE-2006-001 Test ti """ self.xml_example_record_short = """ 81 TEST-ARTICLE-2006-001 ARTICLE-2006-001 """ self.xml_example_multi_records = """ 81 TEST-ARTICLE-2006-001 ARTICLE-2006-001 Test ti 82 Author, t """ self.xml_example_multi_records_short = """ 81 TEST-ARTICLE-2006-001 ARTICLE-2006-001 82 """ def test_record_xml_output(self): """bibrecord - xml output""" rec = bibrecord.create_record(self.xml_example_record, 1, 1)[0] rec_short = bibrecord.create_record(self.xml_example_record_short, 1, 1)[0] self.assertEqual(bibrecord.create_record(bibrecord.record_xml_output(rec, tags=[]), 1, 1)[0], rec) self.assertEqual(bibrecord.create_record(bibrecord.record_xml_output(rec, tags=["001", "037"]), 1, 1)[0], rec_short) self.assertEqual(bibrecord.create_record(bibrecord.record_xml_output(rec, tags=["037"]), 1, 1)[0], rec_short) class BibRecordCreateFieldTest(unittest.TestCase): """ bibrecord - testing for creating field """ def test_create_valid_field(self): """bibrecord - create and check a valid field""" bibrecord.create_field() bibrecord.create_field([('a', 'testa'), ('b', 'testb')], '2', 'n', 'controlfield', 15) def test_invalid_field_raises_exception(self): """bibrecord - exception raised when creating an invalid field""" # Invalid subfields. self.assertRaises(bibrecord_config.InvenioBibRecordFieldError, bibrecord.create_field, 'subfields', '1', '2', 'controlfield', 10) self.assertRaises(bibrecord_config.InvenioBibRecordFieldError, bibrecord.create_field, ('1', 'value'), '1', '2', 'controlfield', 10) self.assertRaises(bibrecord_config.InvenioBibRecordFieldError, bibrecord.create_field, [('value')], '1', '2', 'controlfield', 10) self.assertRaises(bibrecord_config.InvenioBibRecordFieldError, bibrecord.create_field, [('1', 'value', '2')], '1', '2', 'controlfield', 10) # Invalid indicators. self.assertRaises(bibrecord_config.InvenioBibRecordFieldError, bibrecord.create_field, [], 1, '2', 'controlfield', 10) self.assertRaises(bibrecord_config.InvenioBibRecordFieldError, bibrecord.create_field, [], '1', 2, 'controlfield', 10) # Invalid controlfield value self.assertRaises(bibrecord_config.InvenioBibRecordFieldError, bibrecord.create_field, [], '1', '2', 13, 10) # Invalid global position self.assertRaises(bibrecord_config.InvenioBibRecordFieldError, bibrecord.create_field, [], '1', '2', 'controlfield', 'position') def test_compare_fields(self): """bibrecord - compare fields""" # Identical field0 = ([('a', 'test')], '1', '2', '', 0) field1 = ([('a', 'test')], '1', '2', '', 3) self.assertEqual(True, bibrecord._compare_fields(field0, field1, strict=True)) self.assertEqual(True, bibrecord._compare_fields(field0, field1, strict=False)) # Order of the subfields changed. field0 = ([('a', 'testa'), ('b', 'testb')], '1', '2', '', 0) field1 = ([('b', 'testb'), ('a', 'testa')], '1', '2', '', 3) self.assertEqual(False, bibrecord._compare_fields(field0, field1, strict=True)) self.assertEqual(True, bibrecord._compare_fields(field0, field1, strict=False)) # Different field0 = ([], '3', '2', '', 0) field1 = ([], '1', '2', '', 3) self.assertEqual(False, bibrecord._compare_fields(field0, field1, strict=True)) self.assertEqual(False, bibrecord._compare_fields(field0, field1, strict=False)) class BibRecordFindFieldTest(unittest.TestCase): """ bibrecord - testing for finding field """ def setUp(self): """Initialize stuff""" xml = """ 81 TEST-ARTICLE-2006-001 ARTICLE-2007-001 """ self.rec = bibrecord.create_record(xml)[0] self.field0 = self.rec['001'][0] self.field1 = self.rec['037'][0] self.field2 = ( [self.field1[0][1], self.field1[0][0]], self.field1[1], self.field1[2], self.field1[3], self.field1[4], ) def test_finding_field_strict(self): """bibrecord - test finding field strict""" self.assertEqual((1, 0), bibrecord.record_find_field(self.rec, '001', self.field0, strict=True)) self.assertEqual((2, 0), bibrecord.record_find_field(self.rec, '037', self.field1, strict=True)) self.assertEqual((None, None), bibrecord.record_find_field(self.rec, '037', self.field2, strict=True)) def test_finding_field_loose(self): """bibrecord - test finding field loose""" self.assertEqual((1, 0), bibrecord.record_find_field(self.rec, '001', self.field0, strict=False)) self.assertEqual((2, 0), bibrecord.record_find_field(self.rec, '037', self.field1, strict=False)) self.assertEqual((2, 0), bibrecord.record_find_field(self.rec, '037', self.field2, strict=False)) class BibRecordSingletonTest(unittest.TestCase): """ bibrecord - testing singleton removal """ def setUp(self): """Initialize stuff""" self.xml = """ 33 Some value """ self.rec_expected = { '001': [([], ' ', ' ', '33', 1)], '100': [([('a', 'Some value')], ' ', ' ', '', 2)], } if parser_minidom_available: def test_singleton_removal_minidom(self): """bibrecord - singleton removal with minidom""" rec = bibrecord.create_records(self.xml, verbose=1, correct=1, parser='minidom')[0][0] self.assertEqual(rec, self.rec_expected) if parser_4suite_available: def test_singleton_removal_4suite(self): """bibrecord - singleton removal with 4suite""" rec = bibrecord.create_records(self.xml, verbose=1, correct=1, parser='4suite')[0][0] self.assertEqual(rec, self.rec_expected) if parser_pyrxp_available: def test_singleton_removal_pyrxp(self): """bibrecord - singleton removal with pyrxp""" rec = bibrecord.create_records(self.xml, verbose=1, correct=1, parser='pyrxp')[0][0] self.assertEqual(rec, self.rec_expected) +class BibRecordNumCharRefTest(unittest.TestCase): + """ bibrecord - testing numerical character reference expansion""" + + def setUp(self): + """Initialize stuff""" + self.xml = """ + + 33 + + Σ & Σ + use & in XML + + """ + self.rec_expected = { + '001': [([], ' ', ' ', '33', 1)], + '123': [([('a', '\xce\xa3 & \xce\xa3'), ('a', 'use & in XML'),], ' ', ' ', '', 2)], + } + + if parser_minidom_available: + def test_numcharref_expansion_minidom(self): + """bibrecord - numcharref expansion with minidom""" + rec = bibrecord.create_records(self.xml, verbose=1, + correct=1, parser='minidom')[0][0] + self.assertEqual(rec, self.rec_expected) + + if parser_4suite_available: + def test_numcharref_expansion_4suite(self): + """bibrecord - numcharref expansion with 4suite""" + rec = bibrecord.create_records(self.xml, verbose=1, + correct=1, parser='4suite')[0][0] + self.assertEqual(rec, self.rec_expected) + + if parser_pyrxp_available: + def test_numcharref_expansion_pyrxp(self): + """bibrecord - but *no* numcharref expansion with pyrxp (see notes) + + FIXME: pyRXP does not seem to like num char ref entities, + so this test is mostly left here in a TDD style in order + to remind us of this fact. If we want to fix this + situation, then we should probably use pyRXPU that uses + Unicode strings internally, hence it is num char ref + friendly. Maybe we should use pyRXPU by default, if + performance is acceptable, or maybe we should introduce a + flag to govern this behaviour. + """ + rec = bibrecord.create_records(self.xml, verbose=1, + correct=1, parser='pyrxp')[0][0] + #self.assertEqual(rec, self.rec_expected) + self.assertEqual(rec, None) TEST_SUITE = make_test_suite( BibRecordSuccessTest, BibRecordParsersTest, BibRecordBadInputTreatmentTest, BibRecordGettingFieldValuesTest, BibRecordGettingFieldValuesViaWildcardsTest, BibRecordAddFieldTest, BibRecordDeleteFieldTest, BibRecordManageMultipleFieldsTest, BibRecordDeleteFieldFromTest, BibRecordAddSubfieldIntoTest, BibRecordModifyControlfieldTest, BibRecordModifySubfieldTest, BibRecordDeleteSubfieldFromTest, BibRecordMoveSubfieldTest, BibRecordAccentedUnicodeLettersTest, BibRecordSpecialTagParsingTest, BibRecordPrintingTest, BibRecordCreateFieldTest, BibRecordFindFieldTest, BibRecordDeleteSubfieldTest, - BibRecordSingletonTest + BibRecordSingletonTest, + BibRecordNumCharRefTest ) if __name__ == '__main__': run_test_suite(TEST_SUITE)