Homec4science

Remove Drydock host resource limits and give working copies simple limits

Authored by epriestley <git@epriestley.com> on Oct 26 2015, 20:39.

Description

Remove Drydock host resource limits and give working copies simple limits

Summary:
Ref T9252. Right now, we have very strict limits on Drydock: one lease per host, and one working copy per working copy blueprint.

These are silly and getting in the way of using "Land Revision" more widely, since we need at least one working copy for each landable repository.

For now, just remove the host limit and put a simple limit on working copies. This might need to be fancier some day (e.g., limit working copies per-host) but it is generally reasonable for the use cases of today.

Also add a --background flag to make testing a little easier.

(Limits are also less important nowadays than they were in the past, because pools expand slowly now and we seem to have stamped out all the "runaway train" bugs where allocators go crazy and allocate a million things.)

Test Plan:

  • With a limit of 5, ran 10 concurrent builds and saw them finish after allocating 5 total resources.
  • Removed limit, raised taskmaster concurrency to 128, ran thousands of builds in blocks of 128 or 256.
    • Saw Drydock gradually expand the pool, allocating a few more working copies at first and a lot of working copies later.
    • Got ~256 builds in ~140 seconds, which isn't a breakneck pace or anything but isn't too bad.
    • This stuff seems to be mostly bottlenecked on sbuild throttling inbound SSH connections. I haven't tweaked it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14334

Details

Committed
epriestley <git@epriestley.com>Oct 26 2015, 20:39
Pushed
aubortJan 31 2017, 17:16
Parents
rPH3f193cb9e06c: Give Harbormaster build steps a "View" page
Branches
Unknown
Tags
Unknown

Event Timeline

epriestley <git@epriestley.com> committed rPHc059149eb98e: Remove Drydock host resource limits and give working copies simple limits (authored by epriestley <git@epriestley.com>).Oct 26 2015, 20:39