Homec4science

Fix some issues with empty daemon autoscale pools

Authored by epriestley <git@epriestley.com> on Jun 13 2016, 22:22.

Description

Fix some issues with empty daemon autoscale pools

Summary:
Ref T10811. I'm not sure this causes any real problems, but I caught a couple of errors in the production logs in this vein:

[13-Jun-2016 14:26:55 UTC] [2016-06-13 14:26:55] ERROR 8: Undefined index:  at [/core/lib/libphutil/src/daemon/PhutilDaemonOverseer.php:302]
[13-Jun-2016 14:26:55 UTC] arcanist(head=stable, ref.master=29839e8c72c5, ref.stable=7b0aac5c6f31), libcore(), phabricator(head=stable, ref.stable=b26ecb189e48), phutil(head=stable, ref.master=992abe4a420c, ref.stable=52748950bb36), services(head=stable, ref.master=5b51b63027b7, ref.stable=66cf59af4580)
[13-Jun-2016 14:26:55 UTC]   #0 PhutilDaemonOverseer::updateAutoscale() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:224]
[13-Jun-2016 14:26:55 UTC]   #1 PhutilDaemonOverseer::run() called at [<phabricator>/scripts/daemon/launch_daemon.php:20]
[13-Jun-2016 14:26:55 UTC] [2016-06-13 14:26:55] EXCEPTION: (InvalidArgumentException) Argument 1 passed to PhutilDaemonOverseer::getAutoscaleProperty() must be an instance of PhutilDaemonHandle, null given, called in /core/lib/libphutil/src/daemon/PhutilDaemonOverseer.php on line 303 and defined at [<phutil>/src/error/PhutilErrorHandler.php:200]
[13-Jun-2016 14:26:55 UTC] arcanist(head=stable, ref.master=29839e8c72c5, ref.stable=7b0aac5c6f31), libcore(), phabricator(head=stable, ref.stable=b26ecb189e48), phutil(head=stable, ref.master=992abe4a420c, ref.stable=52748950bb36), services(head=stable, ref.master=5b51b63027b7, ref.stable=66cf59af4580)
[13-Jun-2016 14:26:55 UTC]   #0 PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:278]
[13-Jun-2016 14:26:55 UTC]   #1 PhutilDaemonOverseer::getAutoscaleProperty(NULL, string, integer) called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:303]
[13-Jun-2016 14:26:55 UTC]   #2 PhutilDaemonOverseer::updateAutoscale() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:224]
[13-Jun-2016 14:26:55 UTC]   #3 PhutilDaemonOverseer::run() called at [<phabricator>/scripts/daemon/launch_daemon.php:20]

These are caused by attempting to update empty autoscale pools. Currently, autoscale pools read configuration from the first daemon in the pool, but obviously that doesn't work if the pool is empty.

This could occur when shutting down or possibly while reloading daemons.

Also, add a diagnostic message bout pool autoscaling. This might be a little too chatty, but maybe useful for some other issues.

Test Plan:

  • A reliable way to hit this was bin/phd debug task --autoscale, then ^C it. It no longer errors.
  • I used bin/worker flood to artificially create a lot of work, and saw the pool autoscale up and down properly.
  • I launched daemons normally (no autoscale).
  • I faked daemons as busy and saw the pool autoscale to full.
  • I changed config and saw daemons automatically reload properly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10811

Differential Revision: https://secure.phabricator.com/D16110

Details

Committed
epriestley <git@epriestley.com>Jun 13 2016, 22:53
Pushed
aubortMar 17 2017, 12:03
Parents
rPHUa815d2d850b8: Don't detect long sentences which happen to have a colon in them as Git URIs
Branches
Unknown
Tags
Unknown

Event Timeline

epriestley <git@epriestley.com> committed rPHUfb1e159d3640: Fix some issues with empty daemon autoscale pools (authored by epriestley <git@epriestley.com>).Jun 13 2016, 22:53