Add a mu-plugin which contains the customised WordPress.org robots.txt content.
mu-plugin
The intention is that we can switch from serving a static robots.txt file to serving a customised one from WordPress.
robots.txt
See #4559.