Homec4science

BibFormat: several bibreformat optimisations

Authored by Alessio Deiana <alessio.deiana@cern.ch> on Jul 10 2012, 11:31.

Description

BibFormat: several bibreformat optimisations

  • Optimises queries that fetch updated records. Uses last_updated technique like the other daemons (indexer, ranker). (closes #1106)
  • Improves concurrency of bibfmt queries. All queries modifying bibfmt are set to low priority to prevent the locking problem with MyISAM:
    • a long select comes from bibreformat to check formats that need updating;
    • a small insert or update or delete comes from bibupload;
    • all subsequent selects are blocked. With LOW_PRIORITY, the update/delete/insert query will wait till all the selects are done before trying to acquire a lock.
  • Range queries for missing formats to not block bibfmt. A single JOIN was taking too long as it does have to analyze at least a million of rows. The query is now split in chunks.

Co-authored-by: Tibor Simko <tibor.simko@cern.ch>

Details

Committed
Tibor Simko <tibor.simko@cern.ch>Sep 4 2012, 18:50
Parents
R3600:b1034cd741b0: shellutils: helper to parse ids from CLI
Branches
Unknown
Tags
Unknown

Event Timeline

Tibor Simko <tibor.simko@cern.ch> committed R3600:b85ef87f1db0: BibFormat: several bibreformat optimisations (authored by Alessio Deiana <alessio.deiana@cern.ch>).Sep 4 2012, 18:50