Homec4science
Diffusion invenio-infoscience (iugw2017) Browse

R3600/docs/architecture/system-overview.rst
72817e1a1dafiugw2017

Last Change
Raw File

system-overview.rst
View Options

.. This file is part of Invenio
Copyright (C) 2015, 2016 CERN.
Invenio is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of the
License, or (at your option) any later version.
Invenio is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public License
along with Invenio; if not, write to the Free Software Foundation, Inc.,
59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
System overview
===============
Invenio v3.x
------------
.. admonition:: CAVEAT LECTOR
Invenio v3.0 alpha is a bleeding-edge developer preview version.
Invenio v3.0 build on top of `Flask`_ web development framework, using `Jinja2`_
template engine, `SQLAlchemy`_ Object Relational Mapper, `JSONSchema`_ data
model, `PostgreSQL`_ database for persistence, and `Elasticsearch`_ for
information retrieval.
.. _Flask: http://flask.pocoo.org/
.. _Jinja2: http://jinja.pocoo.org/docs/
.. _SQLAlchemy: http://www.sqlalchemy.org/
.. _JSONSchema: http://json-schema.org/
.. _PostgreSQL: http://www.postgresql.org/
.. _Elasticsearch: https://www.elastic.co/products/elasticsearch
Invenio's architecture is modular. The code base is split into more than 50
independent components that are `released independently on PyPI
<https://pypi.python.org/pypi?:action=search&term=inveniosoftware&submit=search>`_.
This ensures strict separation of components that talk among themselves over API
and permits rapid development of independent components by independent teams.
Invenio compontents, named *modules*, can be roughly split in three categories:
1. **base modules** provide interfaces to the Flask ecosystem, the Database, and
other system tools and technologies that the Invenio ecosystem uses. Example:
``Invenio-Celery`` that talks to the Celery worker system.
2. **core feature modules** provide most common functionality that each digital
library instance is likely interested in using. Example: ``Invenio-Records``
provide record object store.
3. **additional feature modules** offer additional functionality suitable for
various particular use cases, such as the Integrated Library System, the
Multimedia Store, or the Data Repository. Example: ``Invenio-Circulation``
offers circulation and holdings capabilities.
Here is a basic bird-eye overview of available Invenio components and their
dependencies: (*work in progress*)
.. graphviz::
digraph invenio3 {
size="20.0 20.0";
ratio="compress";
// helper floors
node [shape=plaintext,style=invis];
{
Floor9 -> Floor8 [style=invis];
Floor8 -> Floor7 [style=invis];
Floor7 -> Floor6 [style=invis];
Floor6 -> Floor5 [style=invis];
Floor5 -> Floor4 [style=invis];
Floor4 -> Floor3 [style=invis];
Floor3 -> Floor2 [style=invis];
Floor2 -> Floor1 [style=invis];
Floor1 -> Floor0 [style=invis];
}
// invenio tools family
node [shape=ellipse,style=dotted];
Elasticsearch;
"JSON Schema";
MySQL;
PostgreSQL;
FS;
Drive;
Dropbox;
S3;
" Celery ";
Flask;
// invenio base plate family
node [shape=box,style=filled];
Access;
Admin;
Assets;
Base;
Celery -> " Celery ";
Config;
DB -> MySQL;
DB -> PostgreSQL;
I18N;
JSONSchemas -> "JSON Schema";
Logging;
REST;
Theme;
Upgrader -> DB;
// invenio search family
node [shape=box,style=filled, color=green];
"Records-UI" -> Records;
"Records-REST" -> Records;
"Records-REST" -> PIDStore;
Records -> DB;
"Search-UI" -> Search;
Search -> Records;
"Records-REST" -> Search;
PIDStore -> Records;
PIDStore -> DB;
node [shape=ellipse,style=filled,color=grey];
"Search-UI" -> "Query-Parser" ;
"Search-UI" -> unAPI;
node [shape=ellipse,style=dotted,color=black];
Search -> Elasticsearch;
// invenio deposit family
node [shape=box,style=filled, color=red];
"Deposit-UI" -> Deposit;
"Deposit-REST" -> Deposit;
Deposit -> Workflows;
Deposit -> Knowledge;
Deposit -> Sequencegenerator;
Workflows -> Records;
Workflows -> Documents;
// invenio accounts family
node [shape=box,style=filled, color="0.5 0.5 1.0"];
"Profiles-UI" -> Profiles;
"Profiles-REST" -> Profiles;
"Groups-UI" -> Groups;
"Groups-REST" -> Groups;
Profiles -> Access;
Profiles -> Accounts;
Accounts -> Access;
Groups -> Accounts;
// invenio helpers family
node [shape=ellipse,style=filled,color=grey];
Documents;
Cloudconnector;
Testing;
Utils;
Ext;
Webhooks;
Redirector;
// invenio OAIS family
node [shape=box,style=filled,color=orange];
"OAIS-Audit-Store" -> DB;
"OAIS-SIP-Store" -> DB;
"OAIS-AIP-Store" -> Cloudconnector;
"OAIS-DIP-Store" -> DB;
Archiver;
Deposit -> "OAIS-SIP-Store";
Workflows -> "OAIS-SIP-Store";
Records -> Archiver;
Documents -> Archiver;
Archiver -> "OAIS-AIP-Store";
Records -> "OAIS-Audit-Store";
// invenio add-ons family
node [shape=box, style=filled, color=yellow];
Alerts -> Records;
Annotations -> Records;
Annotations -> Profiles;
Classifier -> Records;
Client -> "Records-REST";
Client -> "Groups-REST";
Client -> "Profiles-REST";
Client -> "Deposit-REST";
Documents -> Cloudconnector;
Documents -> FS;
Cloudconnector -> Dropbox;
Cloudconnector -> Drive;
Cloudconnector -> S3;
Collections -> Records;
Comments -> Records;
Comments -> Profiles;
Communities -> Collections;
Communities -> Groups;
Communities -> Profiles;
Deposit -> Documents;
Deposit -> Records;
Deposit -> PIDStore;
Documents -> Records;
Formatter -> Records;
Formatter -> "OAIS-DIP-Store";
Records -> JSONSchemas;
News -> Theme;
OAIHarvester -> DB;
OAIHarvester -> Workflows;
OAIHarvester -> Records;
OAuthClient -> Accounts;
OAuth2Server -> Accounts;
Pages -> Theme;
Previewer -> Records;
Previewer -> "Previewer-ISPY";
Editor -> "Records-REST";
Checker -> "Records-REST";
Merger -> "Records-REST";
Statistics;
Tags -> Records;
Tags -> Profiles;
// invenio ILS family
node [shape=box, style=filled, color=purple];
"Circulation-UI" -> Circulation;
"Circulation-REST" -> Circulation;
"Acquisition-UI" -> Acquisition;
"Acquisition-REST" -> Acquisition;
Client -> "Circulation-REST";
Client -> "Acquisition-REST";
Circulation -> Records;
Circulation -> Accounts;
Acquisition -> Records;
Acquisition -> Accounts;
// invenio end user
node [shape=plaintext, color=white];
Users;
Users -> "Deposit-UI";
Users -> "Search-UI";
Users -> "Records-UI";
Users -> "Circulation-UI";
Users -> "Acquisition-UI";
// floor 0
{
rank = same;
Floor0;
Elasticsearch;
MySQL;
PostgreSQL;
" Celery ";
"JSON Schema";
Flask;
Drive;
Dropbox;
S3;
FS;
}
// floor 1
{
rank = same;
Floor1;
Access;
Admin;
Assets;
Base;
Celery;
Config;
DB;
I18N;
JSONSchemas;
Logging;
Theme;
REST;
Upgrader;
DB;
Testing;
Utils;
Ext;
Webhooks;
Redirector;
}
// floor 8
{
rank = same;
Floor8;
"Records-UI";
"Records-REST";
"Deposit-UI";
"Deposit-REST";
"Search-UI";
"Profiles-UI";
"Profiles-REST";
"Groups-UI";
"Groups-REST";
"Circulation-UI";
"Circulation-REST";
"Acquisition-UI";
"Acquisition-REST";
}
// floor 9
{
rank = same;
Floor9;
Client;
Users;
}
}
For further information, see :ref:`module_anatomy` and :ref:`module_list`
sections.
Invenio v2.x
------------
Invenio v2.x was a transitional release series combining legacy code base
(Invenio v1.x) with new technology (Flask etc as used in Invenio v3.x).
Invenio v1.x
------------
Invenio consists of several more or less independent modules with precisely
defined functionality. The general criterion for module names is to use the
"Bib" prefix to denote modules that work more with the bibliographic data, and
the "Web" prefix to denote modules that work more with the Web interface. (The
difference is of course blurred in some cases, as in the case of search engine
that has got a web interface but searches bibliographic data.)
.. image:: /_static/modules-overview-diagram.jpeg
Follows a brief description of what each module does.
- BibCheck permits administrators and library cataloguers to automate various
kind of tests on the metadata to see whether the metadata comply with quality
standards. For example, that certain metadata fields are of a certain length,
that they have numeric content, that they must not be present when other field
exists, that their content is governed by an authority base depending on
values of other fields, etc. The module can report its findings or can even
automatically correct some kind of errors.
- BibClassify allows automatic extraction of keywords from fulltext documents,
based on the frequency of specific terms, taken from a controlled vocabulary.
Controlled vocabularies can be expressed as simple text thesauri or as
structured, RDF-compliant, taxonomies, to allow a semantic classification.
- BibConvert allows metadata conversion from any structured or semi-structured
proprietary format into any other format, typically the MARC XML that is
natively used in Invenio. Nevertheless the input and output formats are fully
configurable and have been tested on data importations from more than one
hundred data sources. The power of this utility lies in the fact that no
structural attributes of data source are presumed, but they are defined in an
extensive data source configuration. Inevitably, this leads to a high
complexity of the BibConvert configuration language. Most frequent
configurations are provided with the Invenio distribution, such as a sample
configuration from Qualified Dublin Core into the MARCXML. In general the
BibConvert configuration consists from the source data descriptions and target
data descriptions. The processor then analyzes and parses the input data and
creates the resulting data structure, similarly as the XSLT processor would
do. Typically the BibConvert is aimed at usage for input data that do not
dispose of an XML representation. The source data is required to be structured
or semi-structured, (i.e. not expressed in natural language that is a subject
of information extraction task) and its processing involves several steps
including record separation and field extraction upto transformation of source
field values and their formatting.
- BibEdit permits one to edit the metadata via a Web interface.
- BibFormat is in charge of formatting the bibliographic metadata in numerous
ways. This truly enables the separation of data content administration and
formatting layout. BibFormat can act in the background and format the records
when needed, or can preformat records for some often used outputs, such as the
brief format used when displaying search results. The BibFormat settings can
be administered either through a user-friendly web interface, or directly by
editing human-readable configuration files.
- OAIHarvest represents the OAi-PMH compatible harvester allowing the repository
to gather metadata from fellow OAi-compliant repositories and the OAi-PMH
repository management. Repository is built directly on top of the database and
disposes of an OAi repository manager that allows to perform the
administrative tasks on the repository aside from the principal generic data
administration module. The database can be partially or completely open for
harvesting in the scope of the OAi-PMH protocol. In this case, all data is
provided in raw form, where the semantics of individual tags is indicated
uniquely by the MARC21 naming convention. This is particularly interesting for
institutes that are specialized in cross-archive and cross-disciplinary
services provision, as for example the ARC service provider.
- BibIndex module takes care of the indexation of metadata, references and full
text files. Two kinds of indexes -- word and phrase index -- are being
maintained. The user can define several logical indexes (e.g. author index,
title index, etc.) and the correspondence of which physical MARC21 metadata
tag goes into which logical field index. An index consists of two parts: (i) a
forward index listing various words (or phrases) found in the given field,
with the set of record identifiers where the given word can be found; and (ii)
a reverse index listing record identifiers, with the set of words of the given
record that go to the forward index. Such a two-part indexing technique allows
one to rapidly update only those words that have changed in the input metadata
record. The indexes were designed with the aim to provide fast user-response
search times and are faster than native MySQL (full text) indexes.
- BibMatch permits to filter input XML files against the database content,
attempting to match records via certain criteria, for example to avoid
doubly-inputted records.
- BibRank permits to set up various ranking criteria that will be used later by
the search engine. For example, ranking by the word frequency, or by some
metadata tag value such as journal name by means of the journal impact factor
knowledge base, or even by the number of downloads of a particular paper. Note
that BibRank is independent of BibIndex.
- BibSched The bibliographic task scheduler is central unit of the system that
allows all other modules to access the bibliographic database in a controlled
manner, preventing sharing violation threats and assuring the coherent
execution of the database update tasks. The module comes with an
administrative interface that allows to monitor the task queue including
various possibilities of a manual intervention, for example to re-schedule
queued tasks, change the task order, etc.
- BibUpload allows to load the new bibliographic data into the database. To
effectuate this task the data must be a well-formed XML file that complies
with the current metadata tag selection schema. Usually, the properly
structured input files of BibUpload come from the BibConvert utility.
- ElmSubmit is an email submission gateway that permits for automatic document
uploads from trusted sources via email. (Usually web submission or harvesting
is preferred.)
- MiscUtil is a collection of miscellaneous utilities that other modules are
using, like the international messages, etc.
- WebAccess module is responsible for granting access to users for performing
various actions within the system. A Role-Based Access Control (RBAC)
technique is used, where users belong to several groups according to their
role in the system. Each user group can be granted to perform certain actions
depending on possible one more action arguments. WebAccess is presently used
mainly for the administrative interface. There are basically two kinds of
actions: (i) configuration of administrative modules and (ii) running
administrative tasks.
- WebAlert module allows the end user to be alerted whenever a new document
matching her personal criteria is inserted into the database. The criteria
correspond to a typical user query as if it would be done via the search
interface. For example, a user may want to get notified whenever a new
document containing certain words, or of a certain subject, is inserted. A
user may create several alerts with a daily, weekly, or a monthly frequency.
The results of alert searches are either sent back to the user by email or can
also be stored into her baskets.
- WebBasket module enables the end user of the system to store the documents she
is interested in in a personal basket or a personal shelf. The concept is
similar to popular shopping carts. One user may own several baskets. A basket
can be either private or public, allowing a simple document sharing mechanism
within a group.
- WebComment provides a community-oriented tool to rank documents by the readers
or to share comments on the documents by the readers. Integrated with the
group-aware WebBasket, WebGroup, WebMessage tools, WebComment is at the heart
of the social network features of the Invenio software.
- WebHelp presents some global user-level, admin-level, and hacker-level
documenation on Invenio. The module-specific documentation is included within
each particular module.
- WebMessage permits the communication between (possibly anonymous) end users
via web message boards, to invite readers to join the groups, etc.
- WebSearch module handles user requests to search for a certain words or
phrases in the database. Two types of searching can be performed: a word
search or a phrase search. The system allows for complex boolean queries,
regular expression searching, or a combined metadata, references and full text
file searching in one go. Users have a possibility to browse for present index
terms. If no direct match could have been found with the user-typed query
pattern, the system proposes alternative matches as a search guidance. The
search indexes were designed to provide fast response times for middle-sized
data collections of up to 106 records. The metadata corpus is organized into
metadata collections that are directly accessible through the browse function,
similarly to the popular concept of Web Directories. Orthogonal views on the
document corpus are enabled in the search interface via a concept of virtual
collections: for example, a document may be classified both according to its
type (e.g. preprint, book) and according to its Dewey decimal classification
number. Such a flexible organization views allows for the creation of easy
navigation schemata to the end users.
- WebSession is a session and user management module that permits to
differentiate between users. Useful for personalization of the interface and
services like personal baskets and alerts.
- WebStat is a configurable system that permits to gather statistics about the
health of the server, the usage of the system, as well as about some
particular system features.
- WebStyle is a library of design-related modules that defines look and feel of
Invenio pages.
- WebSubmit is a comprehensive submission system allowing authorized individuals
(authors, secretaries and repository maintenance staff) to submit individual
documents into the system. The submission system disposes of a flow-control
mechanism that assures the data approval by authorized units. In total there
are several different exploitable submission schemas at a disposal, including
an automated full text document conversion from various textual and image
formats. This module also disposes of information extraction functionality,
focusing on bibliographic entities such as references, authors, keywords or
other implicit metadata.
c4science · Help