Making WordPress.org

Opened 10 months ago

Last modified 10 months ago

#5668 accepted defect

Add an 'empty' robots.txt file to s.w.org

Reported by: jonoaldersonwp Owned by: dd32
Milestone: Priority: lowest
Component: General Keywords: seo performance


https://s.w.org/robots.txt currently redirects to wordpress.org.

For SEO and performance reasons, we should implement a robots.txt file at this location.

The contents should be:

User-agent: *

Change History (2)

#1 @dd32
10 months ago

In 10840:

Robots: Add a specific robots.txt value for s.w.org.

See #5668.

#2 @dd32
10 months ago

  • Owner set to dd32
  • Status changed from new to accepted

[10840] has added the robots.txt, but due to how the CDN is setup for caching, it could take a bit for the changes to take effect.

# Current cached public response
$ curl -is https://s.w.org/robots.txt
HTTP/2 301
location: https://wordpress.org/
x-nc: HIT mdw 1

# New Response, uncached.
$ curl -is https://s.w.org/robots.txt?1
HTTP/2 200
x-nc: MISS mdw 1

User-agent: *

# New Response, cached.
$ curl -is https://s.w.org/robots.txt?1
HTTP/2 200
x-nc: HIT mdw 1

User-agent: *

I'm not going to request the CDN be cleared quite yet, as it should self-clear within a few days I expect. I'll keep this open to remind myself.

Last edited 10 months ago by dd32 (previous) (diff)
Note: See TracTickets for help on using tickets.