Add an 'empty' robots.txt file to s.w.org

[jonoaldersonwp]

​https://s.w.org/robots.txt currently redirects to wordpress.org.

For SEO and performance reasons, we should implement a robots.txt file at this location.

The contents should be:

User-agent: *
Disallow:

[dd32]

In 10840:

Robots: Add a specific robots.txt value for s.w.org.

See #5668.

[dd32]

[10840] has added the robots.txt, but due to how the CDN is setup for caching, it could take a bit for the changes to take effect.

# Current cached public response
$ curl -is https://s.w.org/robots.txt
HTTP/2 301
location: https://wordpress.org/
x-nc: HIT mdw 1

# New Response, uncached.
$ curl -is https://s.w.org/robots.txt?1
HTTP/2 200
x-nc: MISS mdw 1

User-agent: *
Disallow:


# New Response, cached.
$ curl -is https://s.w.org/robots.txt?3
HTTP/2 200
x-nc: HIT mdw 1

User-agent: *
Disallow:

I'm not going to request the CDN be cleared quite yet, as it should self-clear within a few days I expect. I'll keep this open to remind myself.

[jonoaldersonwp]

Looks like this still redirects to wordpress.org.