Phage Agent outline
Summary:
Ref T2794. This implements a rough outline for Phage agents. This design improves on the Hypershell design in several ways.
- Lighter weight: Hypershell has a fairly heavy agent which communicates directly with MySQL and is generally "smart". However, I think this was not necessarily the right design choice -- we don't really gain much from having a heavy agent. Having a heavy agent does add a lot of complexity, though. Phage uses a very lightweight agent.
- MySQL: Phage is not dependent upon MySQL. It uses the master process itself as a locking/queueing server. (The Phabricator frontend will use MySQL, but this will be an optional display layer on top of all this.)
- PHP: Hypershell depended upon having PHP on all the target servers. Although Phage still needs PHP on the master server, the agent is now lightweight enough that porting agent drivers to other languages is practical, so a server fleet could have Python or Ruby on the web tier and PHP on only a small number of ops machines.
- Startup: Hypershell has a multistage startup phase where it copies itself to agent machines, then executes itself. I think this was one of the best design decisions in Hypershell, but it means we have somewhat slow startup as we need to make multiple connections to each agent. Phage bootstraps by streaming itself over a single connection.
In more detail, the Phage bootstrap phase goes like this:
- We run ssh ... 'php -r eval(fread(STDIN, ...)' with some frills. This starts a PHP process on the remote which runs a one-line bootstrapping program: it reads another program of known length off STDIN and runs it.
- We write a more sophisticated bootstrapping program to the SSH process. On the other end, the PHP process executes its one-line bootstrap to read and evaluate it. The bootstrapping program is just a fancier version of the one-line bootstrap.
- We write the full agent program to the SSH process. The fancy boostrapper reads and evaluates it, which causes execution to transfer to the agent.
- The agent communicates with the master using the SSH pipe as a JSON channel.
Test Plan: Added a basic unit test of the bootstrapping process.
Reviewers: btrahan, edward
Reviewed By: edward
CC: aran, AnhNhan
Maniphest Tasks: T2794
Differential Revision: https://secure.phabricator.com/D5578