Google not indexing core trac anymore?

[DrewAPicture]

Up until very recently, I've always used Google search with the site operator to search core trac. And for some reason, it's no longer returning very many if any results.

So I wondered if anybody knew if the crawl/robots.txt settings for core trac had changed recently prevent Google from indexing it?

Sample search:
site:core.trac.wordpress.org bug

Results (5): ​https://www.google.com/#filter=0&hl=en&q=site:core.trac.wordpress.org+bug

[nacin]

Per ​http://core.trac.wordpress.org/robots.txt, things seem kosher, at least. Should probably not block /changeset, but eh.

[bpetty]

I do this a lot too (Google search history says about 24 times in the last couple months), and had noticed last week that it actually gave me search results that something along the lines of "this site has blocked search results, check robots.txt".

The robots.txt file looked kosher to me back when this happened to me last week (and it's exactly the same as now). So I'm really not sure what's up either. Regardless, I'm also not seeing any results for things there should be hundreds of results for, and this is rather critical in my opinion not just for us, but for WordPress users looking up issues too. Someone should hop on Google Webmaster tools (who has proper authorization), and figure out what's wrong.

[ocean90]

Someone should hop on Google Webmaster tools (who has proper authorization), and figure out what's wrong.

Yes please. It's really really sad that no trac ticket is indexed.

[nacin]

I added Trac to Google webmaster tools. It's basically empty, but everything seems to work fine. This isn't my area of expertise, but I'm definitely wanting to resolve this. I'm happy to add someone who knows more than me to investigate.

[nacin]

With the help of dd32 I tracked down where bots were getting blocked. Hopefully this will clear up soon. I may submit a sitemap of all tickets.

[DrewAPicture]

Replying to nacin:

With the help of dd32 I tracked down where bots were getting blocked. Hopefully this will clear up soon. I may submit a sitemap of all tickets.

Thanks fellas. Sounds like I'll have my Trac-fu back soon.

[joostdevalk]

Saw your ping @nacin, sorry for the late response, seems resolved as far as I'm concerned. I think changeset should indeed not be blocked, and the Crawl-delay: 3 is something Google and Bing both ignore AFAIK.

[nacin]

Seems good to go. Thanks joost also.

[SergeyBiryukov]

Related: #121