Homec4science

OAIHarvest: move to workflows

Authored by Guillaume Lastecoueres <PX9e@gmx.fr> on Dec 6 2013, 23:28.

Description

OAIHarvest: move to workflows

  • Changes OAI Harvest daemon logic to use the new workflows module, with many changes overall.
  • Refactors DB-layer for oaiharvest part. Uses the new sqlalchemy models.
  • Changes OaiHARVEST table schema to add workflow options and removes frequency parameter. BibSched is the master of task frequency.
  • Allows new CLI parameters --workflow or -r/--repository, --workflow or just only -r/--repository.
  • Adds support to harvest with all combinations. It means one repository or one workflow, one repository N workflows N repositories one workflow or N repositories N workflows.
  • Optimized request in daemon to mimize the number of workflows to start to complete the work asked.
  • Improves task_run with better error reporting.
  • Cleans the code of several functions, simplifies the harvesting a bit. Changes the way the updating of the last_time update.
  • Updates OaiHarvest model in oaiharvester.

Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>

Details

Committed
kuncarJun 4 2014, 16:04
Parents
R3600:9c5f81e5c5cb: grunt: fix Prism configuration
Branches
Unknown
Tags
Unknown

Event Timeline

Jiri Kuncar <jiri.kuncar@cern.ch> committed R3600:258d117275af: OAIHarvest: move to workflows (authored by Guillaume Lastecoueres <PX9e@gmx.fr>).Jun 4 2014, 16:04