OAIHarvest: several improvements

• Updates the OAI harvest admin interface with the possibility to add various arguments to post-harvest processes. (fixes #547)

• Arguments for each post-process are now defined inside a new config file: oai_harvest_config.py

• Adds several arguments to the existing post-harvest processes run after a scheduled OAI harvest.

• Enhances and simplifies several parts of the UI workflow and look.

• Moves DB functions in business-model into a new database-layer: oai_harvest_dblayer.py

• Comments for OAI harvest sources are now actually displayed.

• Changes column 'argument' in oaiHARVEST table from TEXT to BLOB. (invenio_upgrader compatible)

• Removes column 'bibconvertcfgfile' and 'bibfilterprogram' in oaiHARVEST table. (invenio_upgrader compatible)

• Fixes bug causing holding-pen UI to break with python > 2.4, as well as another display bug with the same interface.

• Moves Javascript relating to holding-pen into a new .js file, away from standard Javascript for the OAI Harvest admin interface.

• Adds a new CSS file for OAI harvest admin pages.

• Adds the option to harvest a certain OAI record that will be taken through the correct source configuration.

• Adds new functions to oai_harvest_dblayer.

• Re-structures code in oai_harvest_deamon to be more clear, with less duplication and easier to understand.

• Improves the web-interface for manual OAI harvest to create deamon tasks for one-shot harvest.

• Adds mail and ticket reporting features after finished harvests. Available through --notify-mail-to=MAIL and --create-ticket-in=QUEUE CLI commands.

• The task log viewer web-interface is now automatically refreshed every minute.

• Adds a new file oai_harvest_utils to contain useful functions across the module as well as reducing the amount of code in oai_harvest_daemon.py

• Contains several bug-fixes and minor improvements to both source code and web-interfaces.

• Adds a temporary hack to the harvesting step of extracting authorlists used by INSPIRE to save the current arXiv ID into a file. Should go away with SPIRES-shutoff.

• Reduces the error level of failed harvests of files during harvesting workflow, in order to not stop the queue and only log errors.

• Adds better error handling while creating tickets.

• Adds error-handling when writing to the temporary arXiv ID list for authorlist block.

Co-authored-by: Daniel Stanculescu <daniel.stanculescu@cern.ch>
Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>