BibSched: refactoring and improvements

• Moves the bibsched monitor code to bibsched_monitor.py.

• Order node_relevant_bibupload_tasks by status first then by id so that we wake up slept bibuploads first.

• The row limit was affecting the running tasks instead of the archived ones

• Remove bibupload from monotasks for INSPIRE.

• Handles specific sleeping of bibupload tasks: If a bibupload comes with higher priority than a bibupload that is already in the queue, it sleeps the existing one and executes. (closes #1274, closes #1275)

• Related to this behavior, on INSPIRE we do not need to change the priority of bibuploads anymore so bibupload priority is never changed.

• Prevents tasks that are still in SCHEDULED status to be asked to go to sleep. The task expects to be in SCHEDULED status when starting.

• When the email-logs option is set in a task via --email-log-to we shall not send emergency messages to admins. The error is already reported to the submitter via email. (closes #1449)

• When the a motd line was longer than the window width, curses would crash and burn. We now wrap the motd to the screen width.

• Moves motd to CFG_TMPSHAREDDIR, so that it is visible on all servers that can run bibsched.

• Add timeout to javascript code, so that ping stops after some inactivity time.

• BibSched was not checking properly the number of tasks running. It was counting tasks that were running on the same host too.

• Reworks tasks list (running, sleeping, etc.) so that we handle them in a cleaner way and with less bugs.

• Adds a debug parameter to Log() in order to avoid having to write ifs protecting debug output.

• Sets the task to ERROR status after it fails to start properly instead of stopping the queue.

• Introduces a new config variable that allows us 2 tasks that should not run concurrently.

• Also introduces a new config variable that allows us to configurare incompatible tasks. The difference is the non concurrent tasks can run while the other task is sleeping. Incomptible tasks can never run even if the other task is sleeping.

• Selected line keeps to the same task even if the task list changes It matches the tasks by id

• task_sleep_now_if_required would raise a SystemExit which was interpreted as an exception. We now ignore if the code is 0.

• Optimizes update_rows() to use "select...in" instead of "select...not in"

• Prevents waking up tasks that are sleeping on other nodes. (it is not possible and the task would be marked as failed)

• If a monotask wants to run on node A, the tasks on node B can still wake up from sleep, but the monotask sleeps them again right after they have woken, so they enter an infinite cycle.

• The automatic does not kill the daemon anymore. Also to enter the automatic mode from the bibsched monitor you need to select a duration and enter a motd if this duration is more than 5 minutes.

• Handles bibindex:author, specifically the name given after : in tasks specified in CFG_BIBSCHED_NON_CONCURRENT_TASKS.

• Fixes a bug that would allow 2 non concurrent tasks to run at the same time.

• Adds a new configurable in invenio.conf, CFG_BIBSCHED_LOGDIR, where the location of bibsched task logs of type bibsched_task_%d.err|log will be stored.
  
  The default log location will still be CFG_LOGDIR.

• Amends all references to the bibsched logs to use the new variable.

• Adds --host which allows to force the execution of a task to a certain host

Co-authored-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Reviewed-by: Samuele Kaplun <samuele.kaplun@cern.ch>