BibHarvest: configurable workflow with refextract
2a32ba8187ae
Actions

Authored by Jan Aage Lavik <jan.age.lavik@cern.ch> on Feb 14 2011, 14:40.

Description

BibHarvest: configurable workflow with refextract

Changed OAI Harvest admin interface for post-process selection from dropdown to checkbox style.

Added a check that the bibfilter-script path is valid when 'filter' (f) step is selected in OAI Harvest admin interface.

Changed 'extract' post-process to 'extract plots' (p) and added two new post-processes: 'extract references' (r) and 'attach full-text' (t).

Harvesting post-process modes are now generated and processed on-the-fly, based on users choice from the OAI Harvest admin interface.

Seperated downloading of materials (tarballs, pdf) from the various steps in order to only download required materials once per OAI record. i.e. the download of full-text pdf only happens once even if it is required several times. Modified the plotextractor_getter slightly to allow this behaviour.

Improved feedback messages from the OAI Harvest deamon and error messages will now flow all the way to Bibtask output logs for each post-process step.

Committed

Tibor Simko <tibor.simko@cern.ch>

Feb 28 2011, 11:31

Parents

Branches

Unknown

Tags

Unknown

Tibor Simko <tibor.simko@cern.ch> committed R3600:2a32ba8187ae: BibHarvest: configurable workflow with refextract (authored by Jan Aage Lavik <jan.age.lavik@cern.ch>).Feb 28 2011, 11:31

				Path
	M			modules/bibharvest/doc/admin/oai-admin-guide.webdoc
	M			modules/bibharvest/lib/bibharvest_templates.py
	M			modules/bibharvest/lib/oai_harvest_admin.py
	M			modules/bibharvest/lib/oai_harvest_config.py
	M			modules/bibharvest/lib/oai_harvest_daemon.py
	M			modules/miscutil/lib/plotextractor_getter.py