Homec4science

Make Drydock lease infrastructure more nimble

Authored by epriestley <git@epriestley.com> on Sep 28 2015, 18:35.

Description

Make Drydock lease infrastructure more nimble

Summary:
Ref T9252. Currently, Harbormaster does this when trying to acquire a working copy:

  • Ask for a working copy.
  • Yield for 15 seconds.
  • Check if we have a working copy yet.

That's OK, but Drydock takes ~1s to acquire a working copy lease if a resource is already available, so we end up doing this:

  • T+0: Ask for a working copy.
  • T+0: Yield for 15 seconds.
  • T+1: Working copy lease activates.
  • T+15: Working copy lease is used.
  • T+16: Build finishes.

So we end up spending about 2 seconds doing work and 14 seconds sleeping.

One way to fix this would be to fiddle with the yield duration, so we yield for 1, 2, 4, ... seconds or something. This probably isn't a bad idea for longer leases (i.e., wait for 15, 30, 45 ... seconds or similar) but it implies a lot of churn for short leases.

Instead, let tasks "awaken" other tasks when they complete. The "awaken" operation means: if a task is in a yielded state (no failures, no owner, explicitly yielded, future expires time), pretend it only yielded until right now instead of whenever it really yielded to.

Basically, this rewrites history so that even though Harbormaster did a yield(15), we pretend it did a yield(4) after we activate the lease if lease activation took 4 seconds.

If this misses, it's fine: we fall back to the normal yield behavior and things move forward normally a few seconds later.

If it hits, we get a more nimble process pretty cleanly.

Test Plan:

  • Restarted a build plan (lease working copy + run ls) with this patch no-op'd, took about 16 seconds.
  • Restarted a build plan with this patch active, took about 1 second.

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14178

Details

Committed
epriestley <git@epriestley.com>Sep 28 2015, 18:35
Pushed
aubortJan 31 2017, 17:16
Parents
rPHcd2dd2a08f81: Give visual feedback when a Drydock resource or lease is releasing
Branches
Unknown
Tags
Unknown

Event Timeline

epriestley <git@epriestley.com> committed rPH9b29d46e60f3: Make Drydock lease infrastructure more nimble (authored by epriestley <git@epriestley.com>).Sep 28 2015, 18:35