Add a robots.txt file to disallow /diffusion/

Summary:
Fixes T4610. Open to suggestions, etc., if there's anything I'm missing.

Also:

• Moves these "system" endpoints into a real application.
• Makes isUnlisted() work a little more consistently.

Test Plan: Accessed /robots.txt, /status/ and /debug/.

Reviewers: chad, btrahan

Reviewed By: btrahan

Subscribers: aran, epriestley

Maniphest Tasks: T4610

Differential Revision: https://secure.phabricator.com/D8532