Site Cloner: Scale to support hundreds of sites

[iandunn]

Currently the Site Cloner is limited to 50 sites, because more than that would not be easily manageable by the user, and could have performance issues. There are almost 500 sites on WordCamp.org, though, so we need a way to allow users to find a small number of relevant sites to consider.

We can probably exclude a lot of camps off the bat, and cut down on the number we need to manage. Camps that meet the following criteria probably won't be desirable by users:

• Have the Coming Soon plugin enabled, since we know they're not done building the theme
• Older than 2 or 3 years, since the design is probably no longer modern
• Any ideas for others?

Once that's done, there should definitely be a search feature, similar to Core's Theme Switcher, for the use case when there's a particular city that the user has in mind, and they just want to quickly find it.

From there, we can add the following filters to narrow down the list:

• Year the camp occurred in
• Site theme
• Custom CSS preprocessor

It'd also be nice to have these additional filters, but it's a lower priority:

• Colors used in theme
• Number of sessions
• Number of attendees

For the colors, maybe there's a library out there that will process the screenshot we already have and tell us the most commonly used 2-3?

See ​the original discussion in the old development repo.

[iandunn]

It'd also be nice to have an filter to see all of the previous years of the current site; i.e., if I'm creating the site for WordCamp Seattle 2016, I might want to see Seattle 2015, Seattle 2014, 2013, etc.

[iandunn]

Actually, we don't need the filter described in comment:2, since we'll have the search feature.

[iandunn]

Older than 2 or 3 years, since the design is probably no longer modern

We can probably scratch that one, since people will expect to see them in search results, and they can filter by year if they only want newer ones.

[prettyboymp]

Initial rough patch to add filtering to the UI of the Site Cloner Customizer control

[iandunn]

Thanks Michael, that looks great :)

A couple minor things:

1) The whitespace changes add a lot of noise to the diff, though, making it harder to review. I usually like to do those in a separate commit, after the main changes have been merged.

Sometimes IDEs do those automatically, so you may need to adjust your settings to only modify lines that you're already changing.

The easiest way I've found to revert those specific changes in SVN, while leaving the others in tact, is to check out a second copy of the folder, then use a visual diff tool like DeltaWalker to copy the whitespace-only lines back to the working copy. If you're using Git, I think git checkout -p is the simplest way.

2) The code looks like it's heading in a good direction, but site-option.php is missing from the diff, so I can't test it. Can you upload another diff with the whitespace changes reverted, and site-option.php included?

[prettyboymp]

Cleaned up the whitespace changes in the original patch. This is still an early rough patch just to illustrate the client side filtering.

[prettyboymp]

@iandunn I removed the extra whitespace changes and added the missing site-option.php.

[iandunn]

That looks great, thanks! It responds really well for me (on a 2013 Macbook Pro), even with the full data set.

[prettyboymp]

Updated patch with new wp_cron for updating sites and added search filters

[prettyboymp]

attachment:112.2.diff includes the following updates:

• A new route integrated with the rest-api to create a JSON endpoint on each site that the customizer integration uses to retrieve the site data.
• Filters for year, theme, and css_proprocessor when searching for sites to clone within the customizer
• WP Cron handling to keep the site list transient up to date.

Some notes about the WP Cron: Since nearly 100 WordCamp sites are being generated each year, having a single cron event that retrieved all of the sites and dumped them into cache didn't seem like a long term solution. Instead, I setup a daily cron event that will prime the most recent 500 WordCamps into cache. If there are more sites to prime after that point, a second event will be scheduled 10 minutes later to update the next set of sites. This will repeat until the list is exhausted.

[iandunn]

1112.2.diff looks good :)

_site_id doesn't exist for older sites on production, we only started adding it to new sites ~2 years ago, so requiring that in the query would exclude a lot of camps from the results. It might be better to use get_wordcamp_site_id() on the results instead.

[prettyboymp]

1112.3.diff includes the following updates to 1112.2.diff:

• Removed the filter for _site_id post meta in the initial WordCamp query since some older sites do not have this set.
• Implemented router for the Site Search control that reapplies the current filters for a user after a theme change (which requires a full page reload).
• Fixed bug where CSS from the host site was still be rendered in the preview if it was already using JetPack Custom CSS.
• Replaced the theme slugs in the theme dropdown filter with theme names.
• Added a label identifying the site currently being previewed.
• Cleaned up some unused javascript variables and properties.
• Added a check that JetPack is available before setting up the plugin to avoid fatal errors
• Added capability check to verify the user can switch_themes before initializing the API endpoint.

[iandunn]

In 4280:

WordCamp Site Cloner: Convert to REST/Backbone to improve scalability

See #1112
Props prettyboymp

[iandunn]

In 4281:

WordCamp Site Cloner: Re-align whitespace

See #1112

[iandunn]

In 4285:

WordCamp Site Cloner: Lower sites query limit to 250 to avoid transient bug

See #1112

[iandunn]

Sorry it took so long to merge this. I ran into some problems while testing some of my changes, so I removed the batching in order to get the rest of the patch deployed. That limits the number of sites at 500, though, so it'd be good to come back and add the batching back in, I just didn't have time to do that myself right now.

After deploying it, I noticed a bug on production, and ended up having to lower the limit to 250. The comment in r4285 has more details. I wasn't able to reproduce it in my dev environment. That'll need to be fixed before the batching can be restored.

[dd32]

This ticket has been moved to GitHub ​https://github.com/WordPress/wordcamp.org/issues/581