workflows: move to new structure and fixes
- Fixes handling BibWorkflowObject created by the user outside of the workflow. When BibWorkflowObject is created by user and its in RUNNING version and it doesnt have id_workflow it will be the object that worfklow is working on (data and state will be changed). There will be extra INITIAL object created to save state of the object at the begining. It allows the user to know object id befor processing it.
- Adds more things to redis sorting.
- Refactors the logging system in BibWorkflowObject and BibWorkflowEngine, adding new logging handlers to save logs to both database and stream output.
- Logs are now added using the API of the logging module, for example obj.log.info instead of obj.log_info as before.
NOTE: There is however a known issue that log.debug is not functioning properly - possibly due to Celery resetting logging levels.
- Changes the BibWorkflowObject model to store extra_data in the same way as data. Meaning, using the "hack" made for storing data as LargeBinary. This means that outside the scope of workflow tasks, obj.get_extra_data() and obj.set_extra_data() must be used to access and update this field.
- Updated the new logging structure to work better overall and fit with the new code structure.
Co-authored-by: Wojciech Ziolek <wojciech.ziolek@cern.ch>
Co-authored-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
NOTE: beware, patch amended by Tibor to use new file names