Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#5938 closed defect (bug) (fixed)

Add an x-robots-tag header to md5/sha1 etc file URLs

Reported by: jonoaldersonwp's profile jonoaldersonwp Owned by:
Milestone: Priority: lowest
Component: Site Keywords: seo


We expose lots of URLs like this across

These consume considerable crawl resources and often tripS the 'soft 404' warning in Google Search Console.

We should manage this by adding an x-robots-tag header to all responses ending in .sha1, .md5 and similar, with a value of noindex, follow.

Change History (4)

#1 @dd32
2 years ago

This should not affect .zip$ or .gz$ links correct?

What about* links? Same as the above?

#2 @jonoaldersonwp
2 years ago

Ah, good question!

We can indeed safely ignore zip and gz links.

Yes please for downloads.!

#3 @dd32
2 years ago

  • Component changed from General to Site
  • Resolution set to fixed
  • Status changed from new to closed


$ curl -Is | grep -i 'x-robots-tag'
x-robots-tag: noindex, follow

$ curl -Is | grep -i 'x-robots-tag'
// No output

$ curl -Is | grep -i 'x-robots-tag'
x-robots-tag: noindex, follow

$ curl -Is | grep -i 'x-robots-tag'
x-robots-tag: noindex, follow

Plugins have a .json checksum file that I haven't added it to, but they're served with the proper content-type headers and aren't linked to.. so I think those should be fine?

#4 @jonoaldersonwp
2 years ago

Nice one!
Yeah, not seeing any problems with the JSON files. Much appreciated!

Note: See TracTickets for help on using tickets.