Homec4science

Make Drydock reclaim unused resources when it reaches a resource limit

Authored by epriestley <git@epriestley.com> on Dec 18 2015, 17:44.

Description

Make Drydock reclaim unused resources when it reaches a resource limit

Summary:
Fixes T9994. Currently, when Drydock can't allocate a new resource because some limit has been reached, it waits patiently for a resource to become available.

It is possible that no resource will ever become available. Particularly with "Working Copy" resources, the new lease may want a copy of rB, but the resource may already be maxed out on rA.

Right now, no process exists to automatically reclaim the unused rA.

When we encounter this situation, try to reclaim one of the other resources if it is just sitting there unused.

Specifically:

  • Add a "reclaim" command which means "release this resource if it is completely unused".
  • Add a bin/drydock reclaim to send this command to every active resource.
  • When we try to acquire a resource and can't, but only because of some kind of limit / utilization problem, try to release an unused resource to free up some room.

Test Plan:

  • Set "Working Copy" resource limit to 1.
  • Ran "Test Configuration" in rA, which worked.
  • Ran "Test Configuration" in rB, which hung forever.
  • Applied patch.
  • Ran "Test Configuration" in rB, saw it reclaim the rA resource, use the slot, then succeed.
  • Ran "Test Configuration" in rA again, saw it grab the slot back.
  • Ran bin/drydock reclaim and saw it reclaim a bunch of old orphaned resources.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9994

Differential Revision: https://secure.phabricator.com/D14819

Details

Committed
epriestley <git@epriestley.com>Dec 18 2015, 20:55
Pushed
aubortJan 31 2017, 17:16
Parents
rPHe9af4f897079: Fix an issue where Drydock followup tasks would not queue if the main task…
Branches
Unknown
Tags
Unknown

Event Timeline

epriestley <git@epriestley.com> committed rPH7168d8edd971: Make Drydock reclaim unused resources when it reaches a resource limit (authored by epriestley <git@epriestley.com>).Dec 18 2015, 20:55