Homec4science

Treat "skipped" unit tests as less interesting than "passed"

Authored by epriestley <git@epriestley.com> on Mar 1 2016, 14:58.

Description

Treat "skipped" unit tests as less interesting than "passed"

Summary:
Ref T10457. Skipped tests are almost always well-behaved (e.g., testWindows(), but the test is running on Linux) and not interesting, and we do not expect well-written, solid systems to necessarily have 0 skips.

Although skips could indicate that you have missing dependencies on a build server, and thus be a bit interesting, I think they almost always indicate that a particular test is not expected to run in the current environment.

If we wanted to tackle this problem in granular detail, we could eventually add a "Missing" status or similar which would serve as "a skip you could reasonably fix in this environment", but I don't think that's too interesting.

Test Plan:
Here's an example of a build result with skips: B10875

{F1136511}

I think this is clearer as "Passed", as this is the expected production state of the build.

Locally, looked at some builds.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10457

Differential Revision: https://secure.phabricator.com/D15369

Details

Committed
epriestley <git@epriestley.com>Mar 1 2016, 15:52
Pushed
aubortJan 31 2017, 17:16
Parents
rPH58cea0714bff: Fix doubling of unbroken build steps in Build Plan view
Branches
Unknown
Tags
Unknown

Event Timeline

epriestley <git@epriestley.com> committed rPH8240e0f72706: Treat "skipped" unit tests as less interesting than "passed" (authored by epriestley <git@epriestley.com>).Mar 1 2016, 15:52