Homec4science

Split Harbormaster workers apart so build steps can run in parallel

Authored by epriestley <git@epriestley.com> on Jan 6 2014, 21:32.

Description

Split Harbormaster workers apart so build steps can run in parallel

Summary:
Ref T1049. Currently, the Harbormaster worker looks like this:

foreach (step) {
  run_step(step);
}

This means steps can't ever be run in parallel. Instead, split it into two workers. The "Build" worker starts things off, and basically does:

update_build();

(We could theoretically do this in the original process because it should never take very long, but since there's a lock and it's a little bit complex it seemed cleaner to separate it.)

The "Target" worker runs an individual target (like a command, or an HTTP request, or whatever), then updates the build:

run_one_step(step);
update_build();

The new update_build() mechanism in HarbormasterBuildEngine does this, roughly:

figure_out_overall_status_of_all_steps();
if (build is done) { done(); }
if (build is fail) { fail(); }
foreach (step that is ready to run) {
  queue_target_worker_for_step(step);
}

So, overall:

  • The part of the code that updates Builds is completely separated from the part of the code that updates Targets.
  • Targets can run in parallel.

Test Plan:

  • Ran a bunch of builds via bin/harbormaster build.
  • Ran a bunch of builds via web UI.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D7890

Details

Committed
epriestley <git@epriestley.com>Jan 6 2014, 21:32
Pushed
aubortJan 31 2017, 17:16
Parents
rPH6abe65bfdce3: Add mailKey to macros
Branches
Unknown
Tags
Unknown

Event Timeline

epriestley <git@epriestley.com> committed rPH4d5e8a149a7d: Split Harbormaster workers apart so build steps can run in parallel (authored by epriestley <git@epriestley.com>).Jan 6 2014, 21:32