(re)instate "noindex" mode on the Patterns Directory

[jonoaldersonwp]

The Patterns Directory (​https://wordpress.org/patterns/) has, until now, used a noindex directive to prevent search indexes from indexing it. Since it was launched, this directive has been removed.

It's my strong recommendation that we reintroduce the noindex directive, until site can be made to be more performant.

Performance issues

The directory homepage loads upwards of 23mb (10mb+ 'over the wire'), in a series of profoundly expensive stages of (re)loading pages within iframes, which in turn (re)load many of the same assets.

Individual pattern pages behave similarly (many loading 8mb+ / 4mb+ 'over the wire'), as well as exhibiting many performance anti-patterns. The sheer data transfer volume is a large issue, but not the only one - the entire loading cascade of this content is problematic.

Aside from being an enormous UX (and reputation/branding; not to mention the related accessibility challenges) issue, this may have a serious impact on our Core Web Vitals metrics; potentially leading to harm side-wide harm to our SEO.

Hopefully, we can spare some resources to optimize performance in the long-term (I'll begin documenting some of the challenges and options; though frankly, our options are heavily constrained by the core approach).

In the short term, we should reinstate the meta robots noindex directive.

[bradleyt]

Ouch! 10mb+ of assets on a landing page like this really isn't good. That said:

• I just did some quick tests and I seem to be getting decent CWV metrics (especially for LCP/CLS). I guess the lazy-loading is working well, even if it is loading a ton of stuff.
• Just because a page is no-indexed doesn't mean that visits to that page don't impact the CrUX report, and therefore origin-wide SEO (source: ​https://twitter.com/JohnMu/status/1334394558835257345)

The main reason I can see for keeping the page no-indexed is to avoid traffic coming to the page, and therefore being recorded against web vitals reports. Therefore, if we go down that path it is equally important to avoid prominently linking to the patterns directory from across the site until the performance issues are resolved. I don't know what the plan for advertising this new page is in that regard?

[dd32]

IMHO this is a wontfix, I don't think we'll be deindexing it.

Performance improvement tickets are welcome though, but please report them upstream to ​https://github.com/WordPress/pattern-directory/issues

[jonoaldersonwp]

@bradleyt Where did you test?
Running Lighthouse in Chrome gives me the attached results, which are pretty terrible.

@dd32 Improving the performance is going to take more than a few tickets; it's going to require fundamental re-architecting (including all the time and resource needed to even scope and plan that). We're simply not resourced to do that, and we can't afford to wait indefinitely for theoretical improvements whilst we continue to deliver a terrible experience to/for our users.

Wearing my marketing hat briefly, this also exposes us to competitive risk/ridicule. It wouldn't take much for Wix/etc to campaign on the back of how un-performant WordPress' shiny new features are, compared to them (who're all investing in improving their performance; see ​https://wordpress.slack.com/archives/C0GKJ7TFA/p1625646182449000).

Do we have a marketing plan for Patterns which involves SEO, or which relies on page-level discovery of patterns from search? Do we have an SEO plan? If not, then I don't think that re-introducing the noindex directive is a problem. And if we do, then we should noindex until the pages are optimized and performant.

[bradleyt]

Hi @jonoaldersonwp, I've been testing with the CWV Chrome extension, with emulated "slow 3G" connection applied.

Interestingly, this morning I'm still seeing good CLS scores this morning, but worse LCP than I was seeing yesterday. I wonder if I'd not applied the throttling correctly. In any case, I do agree the overall performance is terrible even if we are partially getting away with it from a CWV point-of-view.

I have noticed since yesterday that many of the Rosetta sites do have a "patterns" link in the main nav (e.g. ​https://es.wordpress.org/patterns/). Given what I said above about no-indexed pages still affecting the Chrome UX Report, and therefore SEO, these header links could continue to do damage even if the no-index was applied - although I don't know how important the Rosetta subdomains are from a SEO perspective.

[dufresnesteven]

Noting, there is an open ticket discussing performance improvements:

​https://github.com/WordPress/pattern-directory/issues/255

[jonoaldersonwp]

This is now much-improved (though there's still more work to do!), so closing.