diff --git a/ABOUT-NLS b/ABOUT-NLS index 94185b50a..cd0858982 100644 --- a/ABOUT-NLS +++ b/ABOUT-NLS @@ -1,318 +1,318 @@ Invenio NATIVE LANGUAGE SUPPORT =============================== About ----- This document describes the Native Language Support (NLS) in Invenio. Native Language Support information for administrators ------------------------------------------------------ Invenio is currently available in the following languages:: af = Afrikaans ar = Arabic bg = Bulgarian ca = Catalan cs = Czech de = German el = Greek en = English es = Spanish fa = Persian (Farsi) fr = French gl = Galician hr = Croatian hu = Hungarian it = Italian ja = Japanese ka = Georgian lt = Lithuanian no = Norwegian (Bokmål) pl = Polish pt = Portuguese ro = Romanian ru = Russian rw = Kinyarwanda sk = Slovak sv = Swedish uk = Ukrainian zh_CN = Chinese (China) zh_TW = Chinese (Taiwan) If you are installing Invenio and you want to enable/disable some languages, please just follow the standard installation procedure as described in the INSTALL file. The default language of the installation as well as the list of all user-seen languages can be selected in the general invenio.conf file, see variables CFG_SITE_LANG and CFG_SITE_LANGS. (Please note that some runtime Invenio daemons -- such as webcoll, responsible for updating the collection cache, running every hour or so -- may work twice as long when twice as many user-seen languages are selected, because it creates collection cache page elements for every user-seen language. Therefore, if you have defined thousands of collections and if you find the webcoll speed to be slow in your setup, you may want to try to limit the list of selected languages.) Native Language Support information for translators --------------------------------------------------- If you want to contibute a translation to Invenio, then please follow the procedure below: - Please check out the existence of po/LL.po file for your language, where LL stands for the ISO 639 language code (e.g. `el` for Greek). If such a file exists, then this language is already supported, in which case you may want to review the existing translation (see below). If the file does not exist yet, then you can create an empty one by copying the invenio.pot template file into LL.po that you can review as described in the next item. (Please note that you would have to translate some dynamic elements that are currently not located in the PO file, see the appendix A below.) - Please edit LL.po to review existing translation. The PO file format is a standard GNU gettext one and so you can take advantage of dedicated editing modes of programs such as GNU Emacs, KBabel, or poEdit to edit it. Pay special attention to strings marked as fuzzy and untranslated. (E.g. in the Emacs PO mode, press `f` and `u` to find them.) Do not forget to remove fuzzy marks for reviewed translations. (E.g. in the Emacs PO mode, press `TAB` to remove fuzzy status of a string.) - After you are done with translations, please validate your file to make sure it does not contain formatting errors. (E.g. in the Emacs PO mode, press `V` to validate the file.) - If you have access to a test installation of Invenio, you may want to see your modified PO file in action:: $ cd po $ emacs ja.po # edit Japanese translation $ make update-gmo $ make install $ sudo apachectl restart $ firefox http://your.site/?ln=ja # check it out in context If you do not have access to a test installation, please contribute your PO file to the developers team (see the next step) and we shall install it on a test site and contact you so that you will be able to check your translation in the global context of the application. (Note to developers: note that ``make update-gmo`` command may be necessary to run before ``make`` if the latter fails, even if you are not touching translation business at all. The reason being that the gmo files are not stored in CVS, while they are included in the distribution tarball. So, if you are building from CVS, and you do not have them in your tree, you may get build errors in directories like modules/webhelp/web/admin saying things like "No rule to make target `index.bg.html`". The solution is to run ``make update-gmo`` to produce the gmo files before running ``make``. End of note to developers.) - Please contribute your translation by emailing the file to - . You help is greatly appreciated and + . You help is greatly appreciated and will be properly credited in the THANKS file. See also the GNU gettext manual, especially the chapters 5, 6 and 11. Native Language Support information for programmers --------------------------------------------------- Invenio uses standard GNU gettext I18N and L12N philosophy. In Python programs, all output strings should be made translatable via the _() convention:: from messages import gettext_set_language [...] def square(x, ln=CFG_SITE_LANG): _ = gettext_set_language(ln) print _("Hello there!") print _("The square of %s is %s.") % (x, x*x) In webdoc source files, the convention is _()_:: _(Search Help)_ Here are some tips for writing easily translatable output messages: - Do not cut big phrases into several pieces, the meaning may be harder to grasp and to render properly in another language. Leave them in the context. Do not try to economize and reuse standalone-translated words as parts of bigger sentences. The translation could differ due to gender, for example. Rather define two sentences instead:: not: _("This %s is not available.") % x, where x is either _("basket") or _("alert") but: _("This basket is not available.") and _("This alert is not available.") - If you print some value in a translatable phrase, you can use an unnamed %i or %s string replacement placeholders:: yes: _("There are %i baskets.") % nb_baskets But, as soon as you are printing more than one value, you should use named string placeholders because in some languages the parts of the sentence may be reversed when translated:: not: _("There are %i baskets shared by %i groups.") % \ (nb_baskets, nb_groups) but: _("There are %(x_nb_baskets)s baskets shared by %(x_nb_groups)s groups.") % \ {'x_nb_baskets': nb_baskets, 'x_nb_groups': nb_groups,} Please use the `x_` prefix for the named placeholder variables to ease the localization task of the translator. - Do not mix HTML presentation inside phrases. If you want to reserve space for HTML markup, please use generic replacement placeholders as prologue and epilogue:: not: _("This is cold.") but: _("This is %(x_fmt_open)scold%(x_fmt_close)s.") Ditto for links:: not: _("This is homepage.") not: _("This is %(x_url_open)shomepage%(x_url_close)s.") - Do not leave unnecessary things in short commonly used translatable expressions, such as extraneous spaces or colons before or after them. Rather put them in the business logic:: not: _(" subject") but: " " + _("subject") not: _("Record %i:") but: _("Record") + "%i:" % recID On the other hand, in long sentences when the trailing punctuation has its meaning as an integral part of the label to be shown on the interface, you should leave them:: not: _("Nearest terms in any collection are") but: _("Nearest terms in any collection are:") - Last but not least: the best is to follow the style of existing messages as a model, so that the translators are presented with a homogeneous and consistently presented output phrase set. Introducing a new language -------------------------- If you are introducing a new language for the first time, then please firstly create and edit the PO file as described above in Section 2. This will make the largest portion of the translating work done, but it is not fully enough, because we currently have also to translate some dynamic elements that aren't located in PO files. The development team can edit the respective files ourself, if the translator sends over the following translations by email: - demo server name, from invenio.conf:: Atlantis Institute of Fictive Science - demo collection names, from democfgdata.sql:: Preprints Books Theses Reports Articles Pictures CERN Divisions CERN Experiments Theoretical Physics (TH) Experimental Physics (EP) Articles & Preprints Books & Reports Multimedia & Arts Poetry Atlantis Times News Atlantis Times Arts Atlantis Times Science Atlantis Times Atlantis Institute Books Atlantis Institute Articles Atlantis Times Drafts Notes ALEPH Papers ALEPH Internal Notes ALEPH Theses ISOLDE Papers ISOLDE Internal Notes Drafts Videos Authorities People Institutes Journals Subjects - demo right-hand-side portalbox, from democfgdata.sql:: ABOUT THIS SITE Welcome to the demo site of the Invenio, a free document server software coming from CERN. Please feel free to explore all the features of this demo site to the full. SEE ALSO The development team will than edit various files (po/LINGUAS, config files, sql files, plenty of Makefile files, etc) as needed. The last phase of the initial introduction of the new language would be to translate some short static HTML pages such as: - modules/webhelp/web/help-central.webdoc Thanks for helping us to internationalize Invenio. Integrating translation contributions ------------------------------------- This appendix contains some tips on integrating translated phrases that were prepared for different Invenio releases. It is mostly of interest to Invenio developers or the release manager. Imagine that we have a working translation file sk.po and that we have received a contribution co-CONTRIB.po that was prepared for previous Invenio release, so that the messages do not fully correspond. Moreover, another person might have had worked with the sk.po file in the meantime. The goal is to integrate the contributions. Firstly, check whether the contributed file sk-CONTRIB.po was indeed prepared for different software release version:: $ msgcmp --use-fuzzy --use-untranslated sk-CONTRIB.po invenio.pot If yes, then join its translations with the ones in the latest sk.po file:: $ msgcat sk-CONTRIB.po sk.po > sk-TMP.po and update the message references:: $ msgmerge sk-TMP.po invenio.pot > sk-NEW.po This will give the new file sk-NEW.po that should now be msgcmp'rable to invenio.pot. Lastly, we will have to go manually through sk-NEW.po in order to resolve potential translation conflicts (marked via ``#-#-#-#-#`` fuzzy translations). If the conflicts are evident and easy to resolve, for example corrected typos, we can fix them. If the conflicts are of translational nature and cannot be resolved without consulting the translators, we should warn them about the conflicts. After the evident conflicts are resolved and the file validates okay, we can rename it to sk.po and we are done. (Note that we could have used ``--use-first`` option to msgcat if we were fully sure that the first translation file (sk-CONTRIB) could have been preferred as far as the quality of translation goes.) diff --git a/AUTHORS.rst b/AUTHORS.rst index bced49c55..52a13ce38 100644 --- a/AUTHORS.rst +++ b/AUTHORS.rst @@ -1,212 +1,212 @@ .. This file is part of Invenio. Copyright (C) 2015 CERN. Invenio is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. Invenio is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with Invenio; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction. Authors ======= Invenio is being co-developed by an international collaboration comprising institutes such as CERN, DESY, EPFL, FNAL, SLAC. -You can contact us at . +You can contact us at . Active contributors: * Adrian Pawel Baran * Alexander Wagner * Annette Holtkamp * Cornelia Plott * Cristian Bacchi * Dan Michael O. Heggø * Daniel Lovasko * Dimitrios Semitsoglou-Tsiapos * Eamonn Maguire * Esteban J. G. Gabancho * Ferran Jorba * Flavio Costa * Gilles Louppe * Guillaume Lastecoueres * Harris Tzovanakis * Ivan Masár * Jacopo Notarstefano * Jan Aage Lavik * Jan Stypka * Javier Martin * Jiri Kuncar * Jocelyne Jerdelet * Joe MacMahon * Jochen Klein * Johnny Mariéthoz * Kenneth Hole * Kirsten Sachs * Lars Holm Nielsen * Laura Rueda * Leonardo Rossi * Ludmila Marian * Marco Neumann * Marios Kogias * Mateusz Susik * Miguel Martín * Nicolas Harraudeau * Nikolaos Kasioumis * Øystein Blixhavn * Pamfilos Fokianos * Pedro Gaudencio * Petr Brož * Petros Ioannidis * Roman Chyla * Sami Hiltunen * Samuele Kaplun * Sebastian Witowski * Stefan Hesselbach * Theodoropoulos Theodoros * Tibor Simko * Thorsten Schwander * Wojciech Ziolek Past contributors: * Adrian-Tudor Panescu * Alberto Pepe * Alessio Deiana * Alexandra Silva * Alper Cinar * Anna Afshar * Antonios Manaras * Artem Tsikiridis * Avraam Tsantekidis * Axel Voitier * Benoit Thiell * Björn Oltmanns * Carmen Alvarez Perez * Christopher Dickinson * Christopher Hayward * Christopher Parker * Daniel Stanculescu * David Bengoa * Diane Berkovits * Dinos Kousidis * Eduardo Margallo * Eirini Psallida * Erik Simon * Eric Stahl * Fabio Souto * Federico Poli * Franck Grenier * Frederic Gobry * Fredrik Nygård Carlsen * Gabriel Hase * Georgios Kokosioulis * Georgios Papoutsakis * Gerrit Rindermann * giannistsan * Giovanni Di Milia * Glenn Gard * Graham R. Armstrong * Gregory Favre * Grzegorz Szpura * Guotie * Eduardo Benavidez * Hector Sanchez * Henning Weiler * Jaime Garcia Llopis * Jake Cowton * Jan Brice Krause * Jan Iwaszkiewicz * Jay Luker * Jerome Caffaro * João Batista * Joaquim Rodrigues Silvestre * Joe Blaylock * Joël Vogt * Johann C. Rocholl * Jorge Aranda Sumarroca * Juan Francisco Pereira Corral * Julio Pernia Aznar * Juliusz Sompolski * Jurga Girdzijauskaite * Kamil Neczaj * Kevin Bowrin * Kevin M. Flannery * Kevin Sanders * Konstantinos Kostis * Konstantinos Kousidis * Konstantinos Ntemagkos * Krzysztof Jedrzejek * Krzysztof Lis * Kyriakos Liakopoulos * Lars Christian Raae * Lewis Barnes * Luke Andrew Smith * Maja Gracco * Markus Goetz * Marcus Johansson * Marko Niinimaki * Martin Vesely * Mathieu Barras * Miguel Martinez Pedreira * Mikael Karlsson * Mikael Vik * Mike Marino * Mike Sullivan * Minn Soe * Nicholas Robinson * Nikola Yolov * Nikolaos Kalodimas * Nikolay Dyankov * Nino Jejelava * Olivier Canévet * Olivier Serres * Øyvind Østlund * Pablo Vázquez Caderno * Patrick Glauner * Paulo Cabral * Peter Halliday * Piotr Praczyk * Radoslav Ivanov * Raja Sripada * Raquel Jimenez Encinar * Richard Owen * Roberta Faggian * Ruben Pollan * Samuele Carli * Stamen Todorov Peev * Stephane Martin * Thierry Thomas * Thomas Baron * Thomas Karampelas * Thomas McCauley * Tiberiu Dondera * Tony Ohls * Tony Osborne * Travis Brooks * Trond Aksel Myklebust * Valkyrie Savage * Vasanth Venkatraman * Vasyl Ostrovskyi * Victor Engmark * Yannick Tapparel * Yoan Blanc * Yohann Paris * Željko Kraljević See also THANKS file. diff --git a/README.rst b/README.rst index d3a6348f4..3d5123a0e 100644 --- a/README.rst +++ b/README.rst @@ -1,114 +1,114 @@ .. This file is part of Invenio. Copyright (C) 2015 CERN. Invenio is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. Invenio is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with Invenio; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction. =================================== Invenio Digital Library Framework =================================== .. image:: https://img.shields.io/travis/inveniosoftware/invenio.svg :target: https://travis-ci.org/inveniosoftware/invenio .. image:: https://img.shields.io/coveralls/inveniosoftware/invenio.svg :target: https://coveralls.io/r/inveniosoftware/invenio .. image:: https://img.shields.io/github/tag/inveniosoftware/invenio.svg :target: https://github.com/inveniosoftware/invenio/releases .. image:: https://img.shields.io/pypi/dm/invenio.svg :target: https://pypi.python.org/pypi/invenio .. image:: https://img.shields.io/github/license/inveniosoftware/invenio.svg :target: https://github.com/inveniosoftware/invenio/blob/master/LICENSE .. image:: https://badges.gitter.im/Join%20Chat.svg :target: https://gitter.im/inveniosoftware/invenio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge Invenio is a free software suite enabling you to run your own digital library or document repository on the web. The technology offered by the software covers all aspects of digital library management, from document ingestion through classification, indexing, and curation up to document dissemination. Invenio complies with standards such as the Open Archives Initiative and uses MARC 21 as its underlying bibliographic format. The flexibility and performance of Invenio make it a comprehensive solution for management of document repositories of moderate to large sizes. Invenio has been originally developed at CERN to run the CERN document server, managing over 1,000,000 bibliographic records in high-energy physics since 2002, covering articles, books, journals, photos, videos, and more. Invenio is nowadays co-developed by an international collaboration comprising institutes such as CERN, DESY, EPFL, FNAL, SLAC and is being used by many more scientific institutions worldwide. We aim at user friendliness and speed. Among the features are: - Navigable collection tree - Documents organised in collections - Regular and virtual collection trees - Customisable portal pages for each collection - At CERN, over 1,000,000 documents in 700 collections - Powerful search engine - Specially designed indexes to provide fast search speed for repositories of up to 3,000,000 records - Customisable simple and advanced search interfaces - Combined metadata, fulltext and citation search in one go - Results clustering by collection - Flexible metadata - Standard metadata format (MARC) - Handling articles, books, theses, photos, videos, museum objects and more - Customisable batch import and web submission workflows - Customisable output format display and linking rules - Collaborative features and personalisation - User-defined document baskets - User-defined email notification alerts - Personalised RSS queries - Sharing documents of interest in user groups - Peer reviewing and group commenting on documents Invenio is free software licenced under the terms of the GNU General Public Licence (GPL). It is provided on an "as is" basis, in the hope that it will be useful, but without any warranty. There is a possibility to get commercial support in case of interest. Invenio runs on Unix-like systems and requires Python/Flask web application server, MySQL, PostgreSQL or SQLite database server, Elasticsearch for information retrieval and Redis for caching. Please consult the INSTALL file for more information. Happy hacking and thanks for flying Invenio. | Invenio Development Team -| Email: info@invenio-software.org +| Email: info@inveniosoftware.org | IRC: #invenio on irc.freenode.net | Twitter: http://twitter.com/inveniosoftware | Github: http://github.com/inveniosoftware -| URL: http://invenio-software.org +| URL: http://inveniosoftware.org diff --git a/RELEASE-NOTES.rst b/RELEASE-NOTES.rst index e72c952b6..9ae031805 100644 --- a/RELEASE-NOTES.rst +++ b/RELEASE-NOTES.rst @@ -1,33 +1,33 @@ ================ Invenio v3.0.0 ================ Invenio v3.0.0 was released on November 9, 2015. About ----- Invenio is a digital library framework enabling you to build your own digital library or document repository on the web. What's new ---------- Installation ------------ $ pip install invenio==3.0.0 Documentation ------------- http://pythonhosted.org/invenio/ Happy hacking and thanks for flying Invenio. | Invenio Development Team -| Email: info@invenio-software.org +| Email: info@inveniosoftware.org | IRC: #invenio on irc.freenode.net | Twitter: http://twitter.com/inveniosoftware | GitHub: https://github.com/inveniosoftware/invenio -| URL: http://invenio-software.org +| URL: http://inveniosoftware.org diff --git a/docs/configuration/overlay.rst b/docs/configuration/overlay.rst index f1f93d60a..d5813190a 100644 --- a/docs/configuration/overlay.rst +++ b/docs/configuration/overlay.rst @@ -1,631 +1,631 @@ .. This file is part of Invenio Copyright (C) 2014, 2015 CERN. Invenio is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. Invenio is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with Invenio; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. .. _overlay: ================================== How to create an Invenio overlay ================================== .. admonition:: CAVEAT LECTOR The following overlay guide was written for Invenio v2.0 release series. It is to be updated for Invenio v3.0.x .. admonition:: TODO The following items are still missing from this document, feel free to contribute to it. - celery/redis/honcho/flower - populate the data Thanks What is an overlay ================== Invenio is a library that enable the creation of an digital library but it has to be coupled with an overlay. The overlay will contain your configuration options, your desired look and feel, the extra invenio modules you've developed and are using. Creating your first Invenio overlay =================================== If you've already setup the Invenio developer's environement itself, it will feel familiar. We are reproducing here the steps for Ubuntu LTS. Other distributions may be found in the previous link. The global setup ---------------- Some softwares and libraries are required to work on your overlay. It's mostly Python, MySQL, Redis as well as XML, XSLT and graphical libraries. Node.js is solely required for development purposes. .. code-block:: console $ python --version Python 2.7.5+ $ sudo apt-get update $ sudo apt-get install build-essential git redis-server \ libmysqlclient-dev libxml2-dev libxslt-dev \ libjpeg-dev libfreetype6-dev libtiff-dev \ libffi-dev libssl-dev \ software-properties-common python-dev \ virtualenvwrapper $ sudo pip install -U virtualenvwrapper pip $ source .bashrc # Install MySQL server, and keep the root password somewhere safe. $ sudo apt-get install mysql-server # Install node.js $ curl -sL https://deb.nodesource.com/setup_4.x | sudo bash - $ sudo apt-get install nodejs The virtual environment ----------------------- Python development usually recommends to work within a ``virtualenv``, which creates an isolated environment where the libraries required by one will not intervene with the system ones or the ones of another system. We are using ``virtualenvwrapper`` but nothing prevents your from directly using ``virtualenv`` you are familiar with it. .. code-block:: console $ mkvirtualenv myoverlay (myoverlay)$ # we are in your overlay environment now and (myoverlay)$ # can leave it using the deactivate command. (myoverlay)$ deactivate $ # Now join it back, recreating it would fail. $ workon myoverlay (myoverlay)$ # That's all there is to know about it. The base of the overlay ----------------------- Let's dive in. .. code-block:: console $ workon myoverlay (myoverlay)$ cdvirtualenv (myoverlay)$ mkdir -p src/myoverlay (myoverlay)$ cd src/myoverlay (myoverlay)$ edit setup.py The ``setup.py`` file contains the definition of a python package. Having one means you can rely on existing tools like ``pip`` to package, install, deploy it later on. Here is its minimal content. .. code-block:: python from setuptools import setup from setuptools import setup, find_packages packages = find_packages() setup( name="My Overlay", version="0.1.dev0", - url="http://invenio-software.org/", + url="http://inveniosoftware.org/", author="Invenio Software", - author_email="invenio@invenio-software.org", + author_email="invenio@inveniosoftware.org", description="My first overlay", packages=packages, install_requires=[ "Invenio>=2" ], entry_points={ "invenio.config": ["myoverlay = myoverlay.config"] } ) Now we can install it in editable mode (``-e``), meaning you don't have to reinstall it after each change .. code-block:: console (myoverlay)$ pip install -e . This will fetch the latest Invenio version published on PyPI. As a developer, you may instead want to use the development version of Invenio from GitHub. To do so, create a file called ``requirements.txt`` with the following content: .. code-block:: text git+git://github.com/inveniosoftware/invenio@pu#egg=Invenio-dev -e . and install using: .. code-block:: console (myoverlay)$ pip install -r requirements.txt Configuration ============= As you've seen above, we defined an entry_point for ``myoverlay.config``. It points to a module that will contain our configuration. So create your application. .. code-block:: text src/ │ ├ myoverlay/ │ │ │ ├ base/ │ │ │ │ │ └ __init__.py │ │ │ ├ __init__.py │ └ config.py │ ├ requirements.txt └ setup.py Put the required configuration into ``config.py``. .. code-block:: python CFG_SITE_LANGS = ["en"] CFG_SITE_NAME = "My Overlay" CFG_SITE_NAME_INTL = { "en": CFG_SITE_NAME } PACKAGES = [ "myoverlay.base", # TODO list your packages here "invenio_base", ] try: from myoverlay.instance_config import * except ImportError: pass Sensitive configuration ----------------------- Other configuration elements like database username and password or the website url should not be put here as this file is not specific to the installation and may be put under a version control system such as Git or Subversion. The configuration can be handled via the `inveniomanage` command line interface (or by editing the `invenio.cfg` file in the instance folder and reloading the application). .. code-block:: console (myoverlay)$ inveniomanage config set create secret-key # MySQL configuration (myoverlay)$ inveniomanage config set CFG_DATABASE_NAME mysql-database (myoverlay)$ inveniomanage config set CFG_DATABASE_USER mysql-user # HOST configuration (for redirects, etc.) (myoverlay)$ inveniomanage config set CFG_SITE_URL http://invenio.example.com (myoverlay)$ inveniomanage config set CFG_SITE_SECURE_URL https://invenio.example.com (myoverlay)$ inveniomanage config set DEBUG True (myoverlay)$ inveniomanage config set ASSETS_DEBUG True Database setup -------------- .. code-block:: console (invenio)$ inveniomanage database init --user=root --password=$MYSQL_ROOT --yes-i-know ... >>> Database has been installed. (invenio)$ inveniomanage database create ... >>> Tables filled successfully. Assets ------ Most of the JavaScript and CSS libraries used are not bundled with invenio itself and needs to be downloaded via `bower `_. Bower is configured using two files: - `.bowerrc`: tells where the assets are downloaded - `bower.json`: lists the dependencies to be downloaded The ``bower.json`` can be automagically generated through the `inveniomanage` command. However, we will have to tell `bower` the path we want the libraries downloaded to. In most cases, it will be in the instance directory under the `static/vendors` path. .. code-block:: console $ sudo su -c "npm install -g bower less clean-css requirejs uglify-js" (myoverlay)$ cdvirtualenv var/invenio.base-instance (myoverlay)$ echo '{"directory": "static/vendors"}' > .bowerrc (myoverlay)$ inveniomanage bower > bower.json (myoverlay)$ bower install For invenio to see the static files from the ``myoverlay.base`` module, it needs to declare a Flask blueprint. Create the following file: ``myoverlay/base/views.py``. .. code-block:: python from flask import Blueprint blueprint = Blueprint( "myoverlay", __name__, url_prefix="/", template_folder="templates", # where your custom templates will go static_folder="static" # where the assets go ) The assets will now be collected into the instance static folder from your overlay, invenio itself and every libraries it uses. .. code-block:: console (myoverlay)$ inveniomanage collect Running ======= .. code-block:: console (myoverlay)$ inveniomanage runserver Translations ============ Invenio comes with full internationalization and localization support based on `Babel `_ library and `Flask-Babel `_. All strings you want to translate in your overlay have to be marked with ``_()``. When you have all strings properly marked, it is time to prepare catalog that contains all these strings for tranlations to desired languages. Configuration ------------- First of all, you have to get into the source folder of your overlay and create a configuration file for *Babel*. .. code-block:: ini [python: **.py] encoding = utf-8 [jinja2: **/templates/**] encoding = utf-8 extensions = jinja2.ext.autoescape.jinja2.ext.with_ Save it as ``babel.cfg`` next to your ``setup.py``. Before we run the extraction tool we need to add section to configure translation directory to ``setup.cfg``. .. code-block:: ini [compile_catalog] directory = myoverlay/base/translations/ [extract_messages] output-file = myoverlay/base/translations/myoverlay.pot [init_catalog] input-file = myoverlay/base/translations/myoverlay.pot output-dir = myoverlay/base/translations/ [update_catalog] input-file = myoverlay/base/translations/myoverlay.pot output-dir = myoverlay/base/translations/ Message Extraction ------------------ Then it’s time to run the Babel string extraction with given configuration: .. code-block:: console (myoverlay)$ python setup.py extract_messages Create Catalog for New Language ------------------------------- Once all translatable strings are extracted, one need to prepare catalogs for new languages. Following example shows how to prepare new catalog for French in PO (Portable Object) format. .. code-block:: console (myoverlay)$ python setup.py init_catalog -l fr Now edit the ``myoverlay/base/translations/fr/LC_MESSAGES/messages.po`` file as needed. Compiling Catalog ----------------- Next step is to prepare MO (Machine Object) files in the format which is defined by the GNU `gettext `_ tools and the GNU `translation project `_. To compile the translations for use, pybabel integration with distutils helps again: .. code-block:: console (myoverlay)$ python setup.py compile_catalog If you install Invenio in development mode you must compile catalog also from the Invenio directory project. .. note:: You should tell git to ignore your compliled translation by running: .. code-block:: console $ echo \*.mo >> .gitignore Updating Strings ---------------- It is pretty common that your strings in the code will change over the time. Pybabel provides support for updating the translation catalog with new strings or changing existing ones. What do you have to do? Create a new ``myoverlay.pot`` like above and then let pybabel merge the changes: .. code-block:: console $ python setup.py update_catalog Deployment ========== Deploying Invenio is almost a piece of cake using `Fabric `_. The following step are inspired by the Flask documentation: `Deploying with Fabric `_ Prerequisites ------------- First, you need a server with remote access (SSH), where you've installed all the python dependencies (e.g. ``build-essentials``, ``python-dev``, ``libmysqlclient-dev``, etc.). Install `fabric` locally, .. code-block:: console $ pip install fabric and create a boilerplate ``fabfile.py``: .. code-block:: python import json from fabric.api import * from fabric.utils import error from fabric.contrib.files import exists env.user = 'invenio' # remote username env.directory = '/home/invenio/www' # remote directory env.hosts = ['yourserver'] # list of servers Preparing the tarball --------------------- Before deploying anything, we need to locally prepare the python package to be installed. Thanks to our ``setup.py`` file, it's very simple. Beforehand, we have to generate the static assets into our static folder. By doing so, it's not required to install anything related to node.js on your server (no ``bower``, ``less``, ``uglifyjs``, etc.). .. code-block:: python @task def pack(): """Create a new source distribution as tarball.""" with open(".bowerrc") as fp: bower = json.load(fp) local("inveniomanage assets build --directory {directory}/gen" .format(**bower)) return local("python setup.py sdist --formats=gztar", capture=False) \ .succeeded Try it: .. code-block:: console $ fab pack ... Done $ ls dist/ My-Overlay-0.1.dev0.tar.gz This is the package that will be installed on your server. Creating the virtual environement --------------------------------- We love virtual environments. We recommend you to install each version into its own virtual env enabling quick rollbacks. .. code-block:: python @task def create_virtualenv(): """Create the virtualenv.""" package = local("python setup.py --fullname", capture=True).strip() venv = "{0}/{1}".format(env.directory, package) with cd(env.directory): if exists(package): return error("This version {0} is already installed." .format(package)) return run("virtualenv {0}".format(package)).succeeded Installing the package ---------------------- We can now upload the local tarball into the virtualenv, and install everything there. .. code-block:: python @task def install(): """Install package.""" package = local("python setup.py --fullname", capture=True).strip() venv = "{0}/{1}".format(env.directory, package) if not exists(venv): return error("Meh? I need a virtualenv first.") # Upload the package and put it into our virtualenv. put("dist/{0}.tar.gz".format(package), "/tmp/app.tgz") run("mkdir -p {0}/src".format(venv)) with cd("{0}/src".format(venv)): run("tar xzf /tmp/app.tgz") run("rm -rf /tmp/app.tgz") # Jump into the virtualenv and install stuff with cd("{0}/src/{1}".format(venv, package)): success = run("{0}/bin/python setup.py install".format(venv) if success: # post install run("{0}/bin/inveniomanage collect".format(venv)) return success Combining all the three steps: .. code-block:: console $ fab pack virtualenv install Configuration ------------- The setup doesn't have the ``invenio.cfg`` file that is generated via ``inveniomanage config``. You should do so manually. Running the server ------------------ uWSGI is super simple and neat, all you need is two files. In the example below, we've installed two versions of our overlay and a symbolic link is pointing to the one we want to run. .. code-block:: console $ ls www/ current -> My-Overlay-0.1 My-Overlay-0.1.dev1 My-Overlay-0.1.dev2 My-Overlay-0.1 wsgi.py uwsgi.ini Let's create the ``wsgi.py`` file. .. code-block:: python from invenio_base.factory import create_wsgi_app application = create_wsgi_app() And the µWSGI configuration: .. code-block:: python [uwsgi] http = 0.0.0.0:4000 master = true processes = 4 die-on-term = true vaccum = true chdir = %d virtualenv = %d/current/ module = wsgi:application touch-reload = %d/wsgi.py Let's run it. .. code-block:: console $ pip install uwsgi $ uwsgi --ini uwsgi.ini # or in daemon mode $ uwsgi -d uwsgi.log --ini uwsgi.ini If the new version causes troubles, going back to the old one is as fast as changing the symbolic link and restarting the WSGI server. .. code-block:: console $ rm current $ ln -s My-Overlay-0.1.dev1 current $ touch wsgi.py Dealing with versions --------------------- One good idea is to use symlink to point to your current virtualenv and run your overlay from there. Doing that via Fabric is left as an exercise to the reader. When installing a new version, copying the ``invenio.cfg`` file over is the only requirements. Restarting the WSGI server is usually done by ``touch``-ing the ``wsgi.py`` file. diff --git a/docs/introduction/demo.rst b/docs/introduction/demo.rst index 66352dbf0..aa5699d20 100644 --- a/docs/introduction/demo.rst +++ b/docs/introduction/demo.rst @@ -1,73 +1,73 @@ .. This file is part of Invenio Copyright (C) 2014 CERN. Invenio is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. Invenio is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with Invenio; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. Demo ==== There are several Invenio installations that you can have a look at in order to see it in action: Demo Sites ---------- -`Atlantis Institute of Fictive Science `_ +`Atlantis Institute of Fictive Science `_ *running Invenio 1.2.1, released 2015-05-21* Atlantis Institute of Fictive Science is an official demo site of Invenio. It demonstrates a basic setup of Invenio. You could (should) obtain this site if you install Invenio for the first time, by lauching ``inveniocfg --create-demo-site`` and ``inveniocfg --load-demo-records``. This is useful to verify the functionality of your installation prior to customizing the system for production. Production Sites ---------------- .. image:: /_static/CDS20120611.png :width: 200 px :align: right `CERN Document Server `_ *CERN, Geneva, Switzerland* At CERN, Invenio manages over 800,000 bibliographic records and 350,000 fulltext documents, organized in more than 500 collections, covering preprints, articles, books, journals, photographs, videos and more, of interest to people working in particle physics. You can check out this site if you want to see the performance and scalability of Invenio. .. image:: /_static/Inspire20120611.png :width: 200 px :align: right `INSPIRE `_ *CERN, Geneva, Switzerland* INSPIRE is the high-energy physics information system that combines the successful `SPIRES `_ database content, curated at `DESY `_, `Fermilab `_ and `SLAC `_ for decades, with the Invenio digital library technology developed at `CERN `_. INSPIRE is run by a collaboration of the four labs and interacts closely with high-energy physics publishers, `arXiv.org `_, `NASA-ADS `_, `PDG `_, and other information resources. [...] diff --git a/docs/technology/git.rst b/docs/technology/git.rst index 2788b4b29..d66cacdde 100644 --- a/docs/technology/git.rst +++ b/docs/technology/git.rst @@ -1,960 +1,960 @@ .. This file is part of Invenio Copyright (C) 2014, 2015 CERN. Invenio is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. Invenio is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with Invenio; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. .. _git-workflow: Git === This page describes the Git collaboration workflow that we try to follow in our development. Introduction ------------ Our collaboration model is basically a pull-on-demand model similar to the one used for the Linux kernel. You may want to read a `chapter on this collaboration model in the Mercurial book `_. In the recipes below you will encounter several personas with different roles: * Joe Bloggs, head developer and system integrator * John Doe, developer, maintainer of the *WebFoo* module * Erika Mustermann, developer, maintainer of the *WebBar* module Setting things up ----------------- S1. Setting up your Git identity ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ How to set up your Git identity. .. code-block:: console $ vim ~/.gitconfig $ cat ~/.gitconfig [alias] k = log --graph --decorate --pretty=oneline --abbrev-commit [user] email = john.doe@cern.ch name = John Doe [color] ui = auto Please use the same full name and email address that was used in the good old CVS !ChangeLog, so that the repo history looks nicely coherent over time. S2. Setting up your private repo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is how to set up your private repo (on your laptop). .. code-block:: console $ cd ~/src - $ git clone http://invenio-software.org/repo/invenio.git + $ git clone http://inveniosoftware.org/repo/invenio.git S3. Backuping your private repo (only developers at CERN) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is how to mirror your private repo to the CERN AFS space. .. code-block:: console $ rsync -rlptDvz -e ssh --delete ~/src/invenio/ \ johndoe@lxplus.cern.ch:private/src/invenio You should mirror your private repo and your working files from your laptop to your private AFS space every night or so. This will give you a safety of having AFS backups around should something bad happen to your laptop. This is important, because you may have some features existing only in your personal repo for weeks or months (see below). Also, having a mirror of your private repo on AFS will provide you with an easy way of testing your private uncommitted code on various other machines, notably CDSDEV. S4. Setting up your public repo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is how to set up your public repo (on your AFS public space). .. code-block:: console $ ssh johndoe@lxplus.cern.ch # LXPLUS (SLC5) now has git johndoe@lxplus> mkdir /afs/cern.ch/user/j/johndoe/public/repo johndoe@lxplus> cd /afs/cern.ch/user/j/johndoe/public/repo johndoe@lxplus> mkdir invenio-johndoe.git johndoe@lxplus> cd invenio-johndoe.git johndoe@lxplus> git --bare init johndoe@lxplus> chmod a+rx hooks/post-update johndoe@lxplus> echo "John Doe's personal tree." > description You can now push your private master branch (from your laptop) to your public repo (on AFS). .. code-block:: console $ cd ~/src/invenio $ git push ssh://johndoe@lxplus.cern.ch/~/public/repo/invenio-johndoe.git master You can define a shortcut called ``johndoe-public`` for your public repo in order to ease future push commands. .. code-block:: console $ git remote add johndoe-public \ ssh://johndoe@lxplus.cern.ch/~/public/repo/invenio-johndoe.git $ git push johndoe-public master S5. Pushing to your public repo from outside CERN ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In order to be able to push to your public repo from outside of CERN to a specific machine such as cdswaredev that has ssh port hidden behind the firewall, you should configure your ssh client to connect to cdswaredev via lxplus proxy gateway, using netcat to forward traffic to cdswaredev. .. code-block:: console $ cat ~/.ssh/config Host lxplus.cern.ch ProxyCommand ssh lxplus.cern.ch exec /usr/bin/nc %h %p This will enable you to have apparently direct ssh/scp/git command connection from your laptop to cdsware, as if you were inside CERN. Note that this is not needed for regular branch pushing, since LXPLUS now has git. It is only needed to access git repos on specific machines, which is rarely the typical developer use case. S6. Making your public repo visible on the Web ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Please contact Joe Bloggs in order to make your public repo visible on -Invenio's `repo web interface `_. +Invenio's `repo web interface `_. S7. Using remote repository locally ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you prefer, you can mount the remote afs filesystem in the local drive, and work as normal. To accomplish that you need sshfs package installed. .. code-block:: console $ mkdir ~/afsrepo $ sshfs -o workaround=rename @lxplus.cern.ch:/afs/cern.ch/user/r//public/repo \ ~/afsrepo/ $ git push ~/afsrepo/invenio.git/ master To unmount the repository. .. code-block:: console fusermount -u ~/afsrepo You can create aliases or edit /etc/fstab to help you mount the public repository. .. code-block:: text sshfs#@lxplus.cern.ch:/afs/cern.ch/user///public/repo fuse user,noauto 0 0 Understanding official repo branches ------------------------------------ The official Invenio repository contains several branches for maintenance and development purposes. We roughly follow the usual git model as described in `man 7 gitworkflows `_ and elsewhere. In summary, the new patchlevel releases (X.Y.Z) happen from the ``maint`` branch, the new minor feature releases (X.Y) happen from the ``master`` branch, and new major feature releases (X) happen after they mature in the optional ``next`` branch. A more detailed description follows. ``maint`` ~~~~~~~~~ This is the maintenance branch for the latest stable release. There can be several maintenance branches for every release series (**maint-0.99**, **maint-1.0**, **maint-1.1**), but typically we use only ``maint`` for the latest stable release. The code that goes to the maintenance branch is of bugfix nature only. It should not alter DB table schema, Invenio config file schema, local configurations in the ``etc`` folder or template function parameters in a backward-incompatible way. If it contains any new features, then they are switched off in order to be fully compatible with the previous releases in this series. Therefore, for installations using any Invenio released X.Y series, it should be always safe to upgrade the system at any moment in time by (1) backing up their ``etc`` folder containing local configuration, (2) installing the corresponding ``maint-X.Y`` branch updates, and (3) rolling back the ``etc`` folder with their customizations. This upgrade process will be automatized in the future via special ``inveniomanage`` options. ``master`` ~~~~~~~~~~ The ``master`` branch is where the new features are being developed and where the new feature releases are being made from. The code in ``master`` is reviewed and verified, so that it should be possible to make a new release out of this branch almost at any given point in time. However, Invenio installations that would like to track this branch should be aware that DB table definitions are not frozen and may change, the config is not frozen and may change, etc, until the release time. So while ``master`` is relatively stable for usage, it should be treated with extreme care, because updates between day D1 and day D2 may require DB schema and ``etc`` configuration changes that are not covered by usual ``inveniomanage`` update statements, so people should be prepared to study the differences and update DB schemata and config files themselves. ``next (optional)`` ~~~~~~~~~~~~~~~~~~~ If a new feature is well implemented, tested and considered stable, it goes directly into the ``master`` branch described previously. If it is cleaned, tested and almost stable, but not fully ``master`` worthy yet, then it may go to the ``next`` branch. The ``next`` branch serves as a kind of stabilization branch for ``master``. The features may stay in ``next`` for a long enough time to get stabilized, and when they are ready, they are promoted to ``master`` (or to ``maint`` in some scenarios). The code in ``next`` may have bugs, may not pass the test suite, but anyway should be stable enough so that it is almost never revoked/rebased. Usually, ``master`` contains all of ``maint``, and ``next`` contains all of ``master``. This is assured by periodical upward merges (maint-to-master, master-to-next, etc). Working on new features - overview ---------------------------------- Here is a schema summarizing how John Doe would work on new features and fixes and how Joe Bloggs would integrate them. .. image:: /_static/invenio-git-workflow.png :width: 859 :alt: invenio git workflow with features. The most important thing to recall is that *any topic branch*, be it a bugfix or a new feature, *should be started off by the developer from the lowest maint branch it applies to*, since it will then be merged upwards to all the other branches as part of the integration process. .. image:: /_static/invenio-git-branches.png :width: 348px :alt: git tree with maint and master read only branches as well as two features or bugfix branches. Example: if there is an important bug in v0.99.1 that John is going to fix, then John should create a topic branch from the tip of ``maint-0.99``, test and everything, and send it over for integration, and it will then get merged both to ``maint-0.99`` as well as to all the necessary upwards branches (``maint-1.0``, ``maint-1.1``, ``master``, etc), as needed, e.g. via periodical ``maint->master`` merges. Backporting fixes from ``master`` to ``maint`` should remain exceptional. Working on new features - details --------------------------------- W1. Cloning the repo ~~~~~~~~~~~~~~~~~~~~ After you clone the official repo (see S2 above), you keep working on your laptop in your own private git repo, using Atlantis Institute of Fictive Science setup conditions. W2. Working with local topic branches ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You never work on the master branch, you always checkout local *topic branches* for every feature you are implementing. This will permit you to switch between topics easily, implement some urgent fixes for older releases, publish some features while retaining features not yet ready for public eyes, etc. In our workflow example above we created several branches to tackle several different tasks. .. code-block:: console $ git checkout master $ git branch new-feature-a $ git branch new-feature-b $ git branch refactor-c $ git checkout next $ git branch wild-idea-d $ git checkout maint $ git branch bugfix-e The topical branches do not necessarily have to stem from the same point in the master branch. Please name your topical branches sensibly, since their names may appear in the central repo logs in case of non-trivial merges. (Please use a dash rather than an underscore in topical branch names.) W3. Working on new-feature-b ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You now have some time to work on feature B, so. .. code-block:: console $ git checkout new-feature-b [ edit, test, edit, test, commit ] [ edit, test, edit, test, commit ] [ more of the same ] until things work as they should for the Atlantis Institute of Fictive Science demo site. This can take a minute or a few weeks, depending on the complexity of B. While working on B, you can switch to other branches to work on various more urgent problems, etc. W4. Using temporary stash ~~~~~~~~~~~~~~~~~~~~~~~~~ If you want to switch branches, you have to commit all the stuff you are currently editing, which may not be what you want. In that case you can **stash** your commits into a temporary git stash, switch to a branch, do what you want, and when you come back, replay the changes from the stash. Here is an example. .. code-block:: console $ git stash # put local edits to the stash $ git stash list # list what you have there stash@`informe 0 `_: WIP on foo.py: 2340b5a... WebFoo: new support for baz $ git checkout refactor-c # work on the refactor-c branch a bit ... <...> $ git checkout new-feature-b # come back to the new-feature-b branch $ git stash apply # replay stuff from stash $ git diff # verify W5. Testing on DEV servers ~~~~~~~~~~~~~~~~~~~~~~~~~~ When your new-feature-b code works okay on the demo site, and you synced it to your /private/ AFS space, you should now test it under **CDSDEV** or **INSPIREDEV** operating conditions. Some more editing, testing, and committing may be needed if things are not working as expected. If the code is working properly on CDSDEV in itself, but say some scalability issues were encountered, then there are two options: (i) either some more of the editing/testing/committing cycle is needed, or (ii) the code is considered working fine enough to be merged now, while the performance issues are savannized to be solved later. W6. Rebasing against latest git/master ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ At this step the new-feature-b code is working both for Atlantis and for CDS contexts. You should now check the official repo for any updates to catch any changes that may have been committed to origin/master in the meantime. .. code-block:: console $ git checkout master $ git pull You can then **rebase** your new-feature-b branch against recent master. .. code-block:: console $ git checkout new-feature-b $ git rebase master In case of conflicts during the rebase, say in file foo.py, you should resolve them. .. code-block:: console $ vim foo.py $ git add foo.py $ git rebase --continue or you can stop the rebase for good. .. code-block:: console $ git rebase --abort You may prefer rebasing of your local commits rather than merging, so that the project log looks nice. (No ugly empty merge commits, no unnecessary temporary versions.) While rebasing, you may want to squash your commits together, to keep the git repo history clean. See section R4 below for more details. You should test your code once more to verify that it was not broken by the updates. W7. Publishing your work ~~~~~~~~~~~~~~~~~~~~~~~~ W7.a Pushing into your public repo ++++++++++++++++++++++++++++++++++ The new-feature-b code is now ready to be pushed into your **public repo** for public consumption. Please make sure to check again that the test cases are working well, and please check once more the basic code kwalitee, as mentioned in the section R3 below. If the test cases work and the code kwalitee is acceptable, push your branch into your public repo this way: .. code-block:: console $ git push johndoe-public new-feature-b Then alert Joe Bloggs with a request to review and integrate the branch, indicating ``git branch johndoe/new-feature-b`` in the email Subject header so that the emails will be threaded properly and given special treatment in the haystack of Joe's usual email conversation. Please also add any special observations for merge. Example: .. code-block:: console From: john.doe@cern.ch To: joe.bloggs@cern.ch Subject: git branch johndoe/new-feature-b Hi Joe: Please merge git branch johndoe/new-feature-b. Tests added, kwalitee checked, needed quickly for Jane's forthcoming new-feature-c. Cheers, John Doe W7.b Sending patches by email +++++++++++++++++++++++++++++ If some occasional code contributors do not have a public repo, they can generate and **send patches by email** to Joe. Say like this. .. code-block:: console $ git checkout master $ git pull $ git branch foo-fix $ git checkout foo-fix $ emacs bar.py $ git commit -a -m 'WebFoo: fixed bad problem' $ git format-patch master $ ls -l 0001-WebFoo-fixed-bad-problem.patch $ git send-email --to joe.bloggs@cern.ch 0001-WebFoo-fixed-bad-problem.patch Or, instead of the last command, send Joe a normal verbose email with attached ``0001-WebFoo-fixed-bad-problem.patch`` file. W7.c Sending patch-suggestions ++++++++++++++++++++++++++++++ As we said in the introduction, John usually maintains the !WebFoo module while Erika usually maintains the !WebBar module. What happens if Erika spots a problem some !WebFoo feature? If the problem and its solution is clear, Erika can simply alert John that she's up to it, fix the problem and publish a branch or send an email to Joe asking for integration. If the problem a little bit more complicated or there are several possible solutions and it is not clear which one is the best or the solution to the problem requires some deep changes inside the structure of !WebFoo that may affect other things or the problem requires optimizations of several pre-existing functions, then it may be best if Erika contacts John as the !WebFoo module maintainer about the problem. Maybe John would like to do the changes himself or John can advise Erika how to go about the problem, etc. In the latter case Erika can implement the proposed solution and send the patch-suggestion email to John as explained in W7.b. John can than review and approve the change and eventually change what has to be changed and forward the branch to Joe for integration. Note that if such a change to !WebFoo may affect other modules and/or other APIs, then these have to be usually discussed/reviewed by Joe in advance, just like other intra-module vs inter-module issues. W8. Review process ~~~~~~~~~~~~~~~~~~ W8.a Reviewing and merging branches +++++++++++++++++++++++++++++++++++ Joe now starts to **review and integrate** the new-feature-b branch. This usually takes two rounds: 1) pure reading of the patch can generate some comments; after the round one is over, 2) testing the patch can generate other comments. If the changes to be done are rather small, then Joe usually does it himself. .. code-block:: console $ git log master..johndoe/new-feature-b # even when master is well ahead in future $ git diff master...johndoe/new-feature-b $ git merge --log johndoe/new-feature-b $ git commit --amend # change log message $ vim Makefile.am # edit to fix something $ git add Makefile.am $ git commit --amend -s # commit also this new change and sign-off $ git push origin-writable master # push to public repo If the changes to be done are rather important, and may reveal a necessity to make some more amendments to the code, this can eventually lead to longer edit/test/commit iterations done in your private repo. If this happens, then, since your code was already published into a public space (even though as personal only), you should not rebase anymore (since rebase rewrites history); you should only merge your new amendments. Or, in case of bigger rewrites, you can publish a new branch. W8.b Reviewing and committing patches +++++++++++++++++++++++++++++++++++++ For patches received by email, a similar review procedure takes place. To integrate such a patch. .. code-block:: console $ less ~/0001-Foo.patch $ emacs ~/0001-Foo.patch # for small edits $ git am -3 ~/0001-Foo.patch $ git commit --amend # to change commit message or, for bigger patches that may require more integration work: .. code-block:: console $ less ~/0001-Foo.patch $ git am -3 ~/0001-Foo.patch $ emacs foo.py # change what is needed $ emacs bar.py # change what is needed # test, install, etc $ git add foo.py # add silently Joe's changes to original patch $ git add bar.py # add silently Joe's changes to original patch $ git commit --amend # commit everything in John's name Although the last process may be evil at times, since Joe kind of usurps John's name for the changes, and commits in his name. Hence this method is usually acceptable only for tiny commits (e.g. correcting typos). W8.c Reviewing and cherry-picking commits +++++++++++++++++++++++++++++++++++++++++ Instead of integrating branches in full, Joe may want to **cherry-pick** some particular commits or squash branches to keep the project history clean. An example: .. code-block:: console $ # see log of a branch: $ git log erika/cool-stuff # pick one particular commit: (e.g. some other author in Erika's branch) $ git cherry-pick 027e1524cd1b823a620620d4b60dd570596fd641 $ # edit its log message: $ git commit --amend $ # squash other commits together while merging: (e.g. other author in Erika's branch) $ git diff 027e1524cd1b823a620620d4b60dd570596fd641 394d1a2a8488cbd0554f12b627ce478c8d1ee65c > ~/z.patch $ git apply --check z.patch $ git apply ~/z.patch --check # test whether patch applies $ emacs ~/z.patch # edit some lines away, retest until applies $ git apply --reject z.patch # alternatively, apply only good junks, study rejects later $ # commit changes as Erika: $ git commit -a --author='Erika Mustermann ' W9. Checking integrated branch ++++++++++++++++++++++++++++++ Once all the integration-related iterations are over, and your new-feature-b code was integrated into the Invenio master branch, then you fetch it to **check** if it was well integrated, and you delete your new-feature-b branch since you don't need it anymore. .. code-block:: console $ git checkout master $ git pull $ git diff master..new-feature-b $ git branch -d new-feature-b If Joe edits something during merge, then the commit SHA1s may not match, but you would notice and study the differences using diff. W10. Deleting integrated branch +++++++++++++++++++++++++++++++ Once new-feature-b is fully merged, you **delete** this branch in your public repo. .. code-block:: console $ git push johndoe-public :new-feature-b Remarks often made during code review ------------------------------------- R1. Remarks on commit history ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Before a topical branch is sent for review and integration, the commit history of the branch should be checked and eventually polished. Here is an example. Consider a topical branch with history like this. .. code-block:: text commit1 WebFoo: new xyzzy facility commit2 WebFoo: fixed typo commit3 WebFoo: speed-ups for xyzzy daemon commit4 WebFoo: Python-2.4 compatibility commit5 WebFoo: Friday weekly cleanup commit6 WebFoo: even more speed ups commit7 WebFoo: oops, cleaned documention commit8 WebFoo: amendments of zyxxy This is not very good. While preserving full commit history in the git repository would be nice, the problem here is that historical versions of the xyzzy facility in the topical branch are not always working properly. The whoops commits are not eliminated. Keeping intermediary commits does not make sense if they are not working properly, they would only be making `git bisect` harder in the future. Ideally, the individual commits should be in an always-working state and they should be presented in logical groups. For example the above branch is better to be squashed as follows: .. code-block:: text commit1 WebFoo: new xyzzy facility + commit2 WebFoo: fixed typo + commit4 WebFoo: Python-2.4 compatibility + commit7a WebFoo: oops, cleaned documention commit3 WebFoo: speed-ups for xyzzy daemon + commit5 WebFoo: Friday weekly cleanup + commit7b WebFoo: oops, cleaned documention commit6 WebFoo: even more speed ups + commit7c WebFoo: oops, cleaned documention commit8 WebFoo: amendments of zyxxy That is, the initial commit should be without typos and syntax errors, should be working on Python-2.4 environment already and should contain respective documentation already. The speed optimisation is an independent improvement, so this would logically constitute our second commit. If the commit6 contained documentation bits about optimisations, the should be presented here. The same is true for the next even-more-speedups commit. Finally, feature amendments come last. Git has powerful tools to help cleaning topical branches like this. Notably, you can run `git rebase master -i` to squash/reorder commits, `git gui` to separate various hunks inside commits, etc. Here is an illustration of a typical thinking process during branch cleanups: * Is the facility fully working now as expected? If it is, keep the commit. * Is this facility or some related one broken in one of the aspects? If it is, amend and squash. * Is this commit an improvement over an already-working facility? If it is, keep the commit. * Is this commit intermediary? Is it worth keeping? Is there a chance that somebody might want to start off a new branch at this point in some day? Does this commit helps some future developer to understand the branch history better? If not, squash. * Is the primary author of this commit different? If he is, keep the commit. Alternatively, squash it, but use `Co-authored-by` commit log directive. * Is the same commit addressing more than one logically separate problem? If it is, split. Having a clean branch history helps in providing sensibly working atomic updates, helps in understanding commits and code, eases eventual future bug-hunting via git bisecting, and makes the software generally more robust. R2. Remarks on commit log messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ R2a. Commit message format ++++++++++++++++++++++++++ Invenio git commit log messages are usually formatted in the following way: * commit message headline providing short summary (maximum 50 chars) formatted in the style of ``component: short description``. (using mostly nouns, no verbs); * empty line; * commit message body with a detailed description of what this patch does, formatted as a bulletted list, with one empty line between items (using present tense). If a given element in the list should appear in the release notes (to describe changes to Invenio admins and developers) it must be prefixed with one of the following labels: ============== =========================================================== label meaning ============== =========================================================== NEW commit adds a new feature SECURITY commit fixes a security issue FIX commit fixes a bug BETTER commit improves an existing feature AMENDS commit invalidates message from different commit identified by its SHA1 INCOMPATIBLE compatibility remarks, removing features and dependencies NOTE any general note (if needed) ============== =========================================================== Here is an example: https://github.com/inveniosoftware/invenio/commit/71df9665bf5fcdd020b67e4cbcedfaddfd6cadaa. .. code-block:: text WebSearch: field-filtered MARCXML API output * Implements field-filtered MARCXML output in Python and Web APIs. This was working for the TextMARC output, not for MARCXML output. This commit fixes the problem. Usage: `/record/123?of=xm&ot=100,700` or `/search?p=ellis&of=xm&ot=100,700`. (closes #1591) * Adds new tests for trying to access hidden fields via the filtered-field API technique. Note that if you use ``vim`` or ``emacs git-modes`` to write your commit messages, you will be alerted about the excessive headline length (more than 50 characters) via colour syntax highlighting. To use ``vim`` for example as your commit message editor, add ``export EDITOR=vim`` to your ``.bashrc``, ``bash_profile`` or variants. The short commit logs are easily readable on narrow mobile devices, are helpful to quickly localise features, and ease any possible hunting for bugs via git bisecting later, should any trouble arise. Here is an example listing the last 15 commits on the master branch. .. code-block:: console $ git log -n 15 maint-1.1..master --pretty=oneline | grep -v 'Merge b' c7cd1f184188207b55903e00e78e5b1acbff33c3 BibFormat: author links for mobile app 6f0641cbde7866adc521793e434f77e2d842f40e WebSearch: display number of hits in mobile output be86ab82f632c60aea7dfc10677f091104155a86 BibFormat: initial release of mobile app formats 81dc101b4377951f345b7a174c2f673b672c1c3a BibDocFile: improve BibDoc display in Files tab d8fd1f23aa63e6c842b3aed9c1509fc1294be719 BibDocFile: raise exception in _build_file_list() e4a1804b7bbdf61f2b7fe8698684c16aced3f58a BibField: creation date addition and keyword fix b0e6e6cacfec91393ab1cbfd04ec6dcfdff32dcd BibFormat: new Solr fulltext snippet facility b98f24bf38b95dd7366d57e8d6d90804957099e5 BibDocFile: additional mimetypes support 211065f10e1a967e1050b08560e03edca58d9c34 BibField: new fft field in `atlantis.cfg` e40be7d8af9223483fe63d97f64463d2492fa890 BibRank: increase rnkDOWNLOADS.file_format size b18ee3fd919c1a06b143761f4611c02f4ac91cab BibField: Python-2.4 compatibility fix See also commit message practices used in the git world, such as `Git for the lazy: Writing good commit messages `_ and `A Note About Git Commit Messages `_. R2b. Commit message QA/review directives ++++++++++++++++++++++++++++++++++++++++ The authors can use the following commit signature directives in order to highlight the quality of the patch at hand before requesting its review and merge. Example: https://github.com/inveniosoftware/invenio/commit/e4a1804b7bbdf61f2b7fe8698684c16aced3f58a .. code-block:: text BibField: creation date addition and keyword fix * Adds new derived field 'creation_date'. * Fixes keywords defition to always return list. Signed-off-by: Jiri Kuncar Reviewed-by: Tibor Simko Here is the list of QA directives that the author may use: ``Reported-by`` Acknowledges the user who originally reported the bug that this commit fixes. ``Signed-off-by`` The author says, in essence: "I have carefully implemented the feature without any leftover to-be-fixed places, I have run all code kwalitee checks and all relevant unit and functional tests, and everything is good. To the best of my knowledge, this commit is good to go into the fast merge track". ``Co-authored-by`` Used when more persons than the current author were involved in creating the code. This usually happens in peer programming. ``Improved-by`` Acknowledges the person who improved the current code significantly after the original committer left, say. This differs from review in that the author provides much more improvements than in a usual review. The reviewers then usually add one of the following tags: ``Acked-by`` The reviewer says, in essence: "I have seen this commit from a distance while walking in the corridor, it looks useful, but I have not had time to deal with it further". Rarely used. ``Tested-by`` The reviewer says, in essence: "In addition, I have paged through the code, tested its kwalitee, tested the desired functionality that this commit implements, and all is well." ``Reviewed-by`` The reviewer says, in essence: "In addition, I have read every line of the source code in detail." Note that a similar system is used in the git world, e.g. Linux kernel `https://www.kernel.org/doc/Documentation/SubmittingPatches `_ or Git itself `http://git.kernel.org/cgit/git/git.git/plain/Documentation/SubmittingPatches `_. While we use some tags in similar context, we use some other tags slightly differently. R3. Remarks on the coding ~~~~~~~~~~~~~~~~~~~~~~~~~ Here is a small sample of often-made code remarks: * Compliance to our `coding standards `_. Stick to PEP 8, run ``pylint`` often. * Missing `test cases `_. * Sanitisation of input variables. Default value check, XSS. * Proper escaping of HTML output. Use ``cgi.escape()``. * Proper ``run_sql()`` argument quoting. SQL injection. * Compliance to Python 2.3. Test on SLC4. Ideally you should make sure they are not present in your public branches before asking for merge into the git/master. You can do a **code kwalitee** check yourself by running. .. code-block:: console $ cd src/invenio/modules/bibedit/lib ... hack on bibrecord_engine.py and friends ... make install etc until satisfaction $ python ../../miscutil/lib/kwalitee.py --check-all ./bibrecord*.py and then follow the output recommendations. (If =--check-all= is too troublesome to implement e.g. due to bad legacy code, then please fix at least the recommendations produced by running =--check-some=.) For more information on the code kwalitee checking, on the above-listed problems and on ways to solve them, as well as some other frequently made remarks on the coding, please see the dedicated InvenioQualityAssurances wiki page. R4. Notes on the review process timeline ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Our pull-on-demand collaboration model enables us to have a *clean* development version of Invenio - there are no problems anymore with the CVS HEAD being broken because people were committing things before checking etc. The price we pay for the inherent review process in the pull-on-demand collaboration model is a certain time delay before the code becomes published and visible. It is normal for John and Erika to have many branches sitting around, waiting for Joe to integrate them. The integration delay can vary depending on the complexity of the branch. E.g. it helps to check in advance the list of frequent remarks mentioned in the section R3 above. E.g. it helps to provide test cases for every bigger commit. (Especially for deep changes that may affect a lot of the codebase, not mentioning changes affecting inter-module relationships.) E.g. it does not help if a branch combines several different features together. We should not mix feature A and feature B together in the same commit and/or branch that implements some new feature C. It is always better to separate different features into different topical branches. On the other hand, it may not be good to separate too much, if features A and B are clearly logically linked. The common sense will tell how much separation is needed. (Similarly to how the common sense says when to stop the database design normalization process.) diff --git a/setup.cfg b/setup.cfg index 75f6af18f..6bfe6787c 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,49 +1,49 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2015 CERN. # # Invenio is free software; you can redistribute it # and/or modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be # useful, but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the # Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, # MA 02111-1307, USA. # # In applying this license, CERN does not # waive the privileges and immunities granted to it by virtue of its status # as an Intergovernmental Organization or submit itself to any jurisdiction. [build_sphinx] source-dir = docs/ build-dir = docs/_build all_files = 1 [bdist_wheel] universal = 1 [compile_catalog] directory = invenio/translations/ [extract_messages] copyright_holder = CERN -msgid_bugs_address = info@invenio-software.org +msgid_bugs_address = info@inveniosoftware.org mapping-file = babel.ini output-file = invenio/translations/messages.pot [init_catalog] input-file = invenio/translations/messages.pot output-dir = invenio/translations/ [update_catalog] input-file = invenio/translations/messages.pot output-dir = invenio/translations/ diff --git a/setup.py b/setup.py index 90c95aa65..9b8c250b3 100644 --- a/setup.py +++ b/setup.py @@ -1,201 +1,201 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2013, 2014, 2015, 2016 CERN. # # Invenio is free software; you can redistribute it # and/or modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be # useful, but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the # Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, # MA 02111-1307, USA. # # In applying this license, CERN does not # waive the privileges and immunities granted to it by virtue of its status # as an Intergovernmental Organization or submit itself to any jurisdiction. """Invenio Digital Library Framework.""" import os import sys from setuptools import find_packages, setup from setuptools.command.test import test as TestCommand readme = open('README.rst').read() history = open('CHANGES.rst').read() tests_require = [ 'check-manifest>=0.25', 'coverage>=4.0', 'isort>=4.2.2', 'pydocstyle>=1.0.0', 'pytest-cache>=1.0', 'pytest-cov>=1.8.0', 'pytest-pep8>=1.0.6', 'pytest>=2.8.0', ] invenio_db_version = '>=1.0.0a7,<1.1.0' extras_require = { 'access': [ 'invenio-access>=1.0.0a2,<1.1.0', ], 'accounts': [ 'invenio-accounts>=1.0.0a2,<1.1.0', ], 'records': [ 'dojson>=0.4.0', 'invenio-pidstore>=1.0.0a1,<1.1.0', 'invenio-records>=1.0.0a8,<1.1.0', 'invenio-records-ui>=1.0.0a4,<1.1.0', 'invenio-records-rest>=1.0.0a2,<1.1.0', ], 'search': [ 'invenio-search>=1.0.0a1,<1.1.0', ], 'theme': [ 'invenio-assets>=1.0.0a1,<1.1.0', 'invenio-theme>=1.0.0a6,<1.1.0', ], 'utils': [ 'invenio-mail>=1.0.0a1,<1.1.0', 'invenio-rest>=1.0.0a2,<1.1.0', 'invenio-logging>=1.0.0a1,<1.1.0', ], 'mysql': [ 'invenio-db[mysql]' + invenio_db_version, ], 'postgresql': [ 'invenio-db[postgresql]' + invenio_db_version, ], 'docs': [ 'Sphinx>=1.4.2', ], 'tests': tests_require, } # # Aliases allow for easy installation of a specific type of Invenio instances. # pip install invenio[repository] # aliases = { 'minimal': ['accounts', 'theme', 'utils', ], 'full': ['access', 'accounts', 'records', 'search', 'theme', 'utils'], } for name, requires in aliases.items(): extras_require[name] = [] for r in requires: extras_require[name].extend(extras_require[r]) # All alias to install every possible dependency. extras_require['all'] = [] for name, reqs in extras_require.items(): if name in {'mysql', 'postgresql'}: continue extras_require['all'].extend(reqs) # # Minimal required packages for an Invenio instance (basically just the # Flask application loading). # setup_requires = [ 'Babel>=1.3', ] install_requires = [ 'invenio-base>=1.0.0a6,<1.1.0', 'invenio-celery>=1.0.0a2,<1.1.0', 'invenio-config>=1.0.0a1,<1.1.0', 'invenio-i18n>=1.0.0a1,<1.1.0', ] packages = find_packages() class PyTest(TestCommand): """PyTest Test.""" user_options = [('pytest-args=', 'a', "Arguments to pass to py.test")] def initialize_options(self): """Init pytest.""" TestCommand.initialize_options(self) self.pytest_args = [] try: from ConfigParser import ConfigParser except ImportError: from configparser import ConfigParser config = ConfigParser() config.read('pytest.ini') self.pytest_args = config.get('pytest', 'addopts').split(' ') def finalize_options(self): """Finalize pytest.""" TestCommand.finalize_options(self) if hasattr(self, '_test_args'): self.test_suite = '' else: self.test_args = [] self.test_suite = True def run_tests(self): """Run tests.""" # import here, cause outside the eggs aren't loaded import pytest errno = pytest.main(self.pytest_args) sys.exit(errno) # Get the version string. Cannot be done with import! g = {} with open(os.path.join('invenio', 'version.py'), 'rt') as fp: exec(fp.read(), g) version = g['__version__'] setup( name='invenio', version=version, description=__doc__, long_description=readme + '\n\n' + history, keywords='Invenio digital library framework', license='GPLv2', author='CERN', - author_email='info@invenio-software.org', + author_email='info@inveniosoftware.org', url='https://github.com/inveniosoftware/invenio', packages=packages, zip_safe=False, include_package_data=True, platforms='any', entry_points={}, extras_require=extras_require, install_requires=install_requires, setup_requires=setup_requires, tests_require=tests_require, classifiers=[ 'Environment :: Web Environment', 'Intended Audience :: Developers', 'License :: OSI Approved :: GNU General Public License v2 (GPLv2)', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Topic :: Internet :: WWW/HTTP :: Dynamic Content', 'Topic :: Software Development :: Libraries :: Python Modules', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Development Status :: 3 - Alpha', ], cmdclass={'test': PyTest}, )