Fix some issues with empty daemon autoscale pools
Summary:
Ref T10811. I'm not sure this causes any real problems, but I caught a couple of errors in the production logs in this vein:
[13-Jun-2016 14:26:55 UTC] [2016-06-13 14:26:55] ERROR 8: Undefined index: at [/core/lib/libphutil/src/daemon/PhutilDaemonOverseer.php:302] [13-Jun-2016 14:26:55 UTC] arcanist(head=stable, ref.master=29839e8c72c5, ref.stable=7b0aac5c6f31), libcore(), phabricator(head=stable, ref.stable=b26ecb189e48), phutil(head=stable, ref.master=992abe4a420c, ref.stable=52748950bb36), services(head=stable, ref.master=5b51b63027b7, ref.stable=66cf59af4580) [13-Jun-2016 14:26:55 UTC] #0 PhutilDaemonOverseer::updateAutoscale() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:224] [13-Jun-2016 14:26:55 UTC] #1 PhutilDaemonOverseer::run() called at [<phabricator>/scripts/daemon/launch_daemon.php:20] [13-Jun-2016 14:26:55 UTC] [2016-06-13 14:26:55] EXCEPTION: (InvalidArgumentException) Argument 1 passed to PhutilDaemonOverseer::getAutoscaleProperty() must be an instance of PhutilDaemonHandle, null given, called in /core/lib/libphutil/src/daemon/PhutilDaemonOverseer.php on line 303 and defined at [<phutil>/src/error/PhutilErrorHandler.php:200] [13-Jun-2016 14:26:55 UTC] arcanist(head=stable, ref.master=29839e8c72c5, ref.stable=7b0aac5c6f31), libcore(), phabricator(head=stable, ref.stable=b26ecb189e48), phutil(head=stable, ref.master=992abe4a420c, ref.stable=52748950bb36), services(head=stable, ref.master=5b51b63027b7, ref.stable=66cf59af4580) [13-Jun-2016 14:26:55 UTC] #0 PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:278] [13-Jun-2016 14:26:55 UTC] #1 PhutilDaemonOverseer::getAutoscaleProperty(NULL, string, integer) called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:303] [13-Jun-2016 14:26:55 UTC] #2 PhutilDaemonOverseer::updateAutoscale() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:224] [13-Jun-2016 14:26:55 UTC] #3 PhutilDaemonOverseer::run() called at [<phabricator>/scripts/daemon/launch_daemon.php:20]
These are caused by attempting to update empty autoscale pools. Currently, autoscale pools read configuration from the first daemon in the pool, but obviously that doesn't work if the pool is empty.
This could occur when shutting down or possibly while reloading daemons.
Also, add a diagnostic message bout pool autoscaling. This might be a little too chatty, but maybe useful for some other issues.
Test Plan:
- A reliable way to hit this was bin/phd debug task --autoscale, then ^C it. It no longer errors.
- I used bin/worker flood to artificially create a lot of work, and saw the pool autoscale up and down properly.
- I launched daemons normally (no autoscale).
- I faked daemons as busy and saw the pool autoscale to full.
- I changed config and saw daemons automatically reload properly.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10811
Differential Revision: https://secure.phabricator.com/D16110