WordPress.org

Making WordPress.org

Opened 4 years ago

Last modified 2 years ago

#1112 assigned enhancement

Site Cloner: Scale to support hundreds of sites

Reported by: iandunn Owned by:
Milestone: Priority: normal
Component: WordCamp Site & Plugins Keywords: needs-patch
Cc:

Description

Currently the Site Cloner is limited to 50 sites, because more than that would not be easily manageable by the user, and could have performance issues. There are almost 500 sites on WordCamp.org, though, so we need a way to allow users to find a small number of relevant sites to consider.

We can probably exclude a lot of camps off the bat, and cut down on the number we need to manage. Camps that meet the following criteria probably won't be desirable by users:

  • Have the Coming Soon plugin enabled, since we know they're not done building the theme
  • Older than 2 or 3 years, since the design is probably no longer modern
  • Any ideas for others?

Once that's done, there should definitely be a search feature, similar to Core's Theme Switcher, for the use case when there's a particular city that the user has in mind, and they just want to quickly find it.

From there, we can add the following filters to narrow down the list:

  • Year the camp occurred in
  • Site theme
  • Custom CSS preprocessor

It'd also be nice to have these additional filters, but it's a lower priority:

  • Colors used in theme
  • Number of sessions
  • Number of attendees

For the colors, maybe there's a library out there that will process the screenshot we already have and tell us the most commonly used 2-3?

See the original discussion in the old development repo.

Attachments (4)

1112-site-cloner-scaling.diff (25.7 KB) - added by prettyboymp 3 years ago.
Initial rough patch to add filtering to the UI of the Site Cloner Customizer control
1112.diff (22.9 KB) - added by prettyboymp 3 years ago.
Cleaned up the whitespace changes in the original patch. This is still an early rough patch just to illustrate the client side filtering.
1112.2.diff (35.8 KB) - added by prettyboymp 3 years ago.
Updated patch with new wp_cron for updating sites and added search filters
1112.3.diff (40.9 KB) - added by prettyboymp 3 years ago.

Download all attachments as: .zip

Change History (19)

#1 @iandunn
4 years ago

  • Owner set to iandunn
  • Status changed from new to accepted

#2 @iandunn
4 years ago

It'd also be nice to have an filter to see all of the previous years of the current site; i.e., if I'm creating the site for WordCamp Seattle 2016, I might want to see Seattle 2015, Seattle 2014, 2013, etc.

#3 @iandunn
3 years ago

Actually, we don't need the filter described in comment:2, since we'll have the search feature.

#4 @iandunn
3 years ago

Older than 2 or 3 years, since the design is probably no longer modern

We can probably scratch that one, since people will expect to see them in search results, and they can filter by year if they only want newer ones.

@prettyboymp
3 years ago

Initial rough patch to add filtering to the UI of the Site Cloner Customizer control

#5 @iandunn
3 years ago

Thanks Michael, that looks great :)

A couple minor things:

1) The whitespace changes add a lot of noise to the diff, though, making it harder to review. I usually like to do those in a separate commit, after the main changes have been merged.

Sometimes IDEs do those automatically, so you may need to adjust your settings to only modify lines that you're already changing.

The easiest way I've found to revert those specific changes in SVN, while leaving the others in tact, is to check out a second copy of the folder, then use a visual diff tool like DeltaWalker to copy the whitespace-only lines back to the working copy. If you're using Git, I think git checkout -p is the simplest way.

2) The code looks like it's heading in a good direction, but site-option.php is missing from the diff, so I can't test it. Can you upload another diff with the whitespace changes reverted, and site-option.php included?

@prettyboymp
3 years ago

Cleaned up the whitespace changes in the original patch. This is still an early rough patch just to illustrate the client side filtering.

#6 @prettyboymp
3 years ago

@iandunn I removed the extra whitespace changes and added the missing site-option.php.

#7 @iandunn
3 years ago

That looks great, thanks! It responds really well for me (on a 2013 Macbook Pro), even with the full data set.

@prettyboymp
3 years ago

Updated patch with new wp_cron for updating sites and added search filters

#8 @prettyboymp
3 years ago

attachment:112.2.diff includes the following updates:

  • A new route integrated with the rest-api to create a JSON endpoint on each site that the customizer integration uses to retrieve the site data.
  • Filters for year, theme, and css_proprocessor when searching for sites to clone within the customizer
  • WP Cron handling to keep the site list transient up to date.

Some notes about the WP Cron: Since nearly 100 WordCamp sites are being generated each year, having a single cron event that retrieved all of the sites and dumped them into cache didn't seem like a long term solution. Instead, I setup a daily cron event that will prime the most recent 500 WordCamps into cache. If there are more sites to prime after that point, a second event will be scheduled 10 minutes later to update the next set of sites. This will repeat until the list is exhausted.

#9 @iandunn
3 years ago

1112.2.diff looks good :)

_site_id doesn't exist for older sites on production, we only started adding it to new sites ~2 years ago, so requiring that in the query would exclude a lot of camps from the results. It might be better to use get_wordcamp_site_id() on the results instead.

@prettyboymp
3 years ago

#10 @prettyboymp
3 years ago

1112.3.diff includes the following updates to 1112.2.diff:

  • Removed the filter for _site_id post meta in the initial WordCamp query since some older sites do not have this set.
  • Implemented router for the Site Search control that reapplies the current filters for a user after a theme change (which requires a full page reload).
  • Fixed bug where CSS from the host site was still be rendered in the preview if it was already using JetPack Custom CSS.
  • Replaced the theme slugs in the theme dropdown filter with theme names.
  • Added a label identifying the site currently being previewed.
  • Cleaned up some unused javascript variables and properties.
  • Added a check that JetPack is available before setting up the plugin to avoid fatal errors
  • Added capability check to verify the user can switch_themes before initializing the API endpoint.

#11 @iandunn
3 years ago

In 4280:

WordCamp Site Cloner: Convert to REST/Backbone to improve scalability

See #1112
Props prettyboymp

#12 @iandunn
3 years ago

In 4281:

WordCamp Site Cloner: Re-align whitespace

See #1112

#13 @iandunn
3 years ago

In 4285:

WordCamp Site Cloner: Lower sites query limit to 250 to avoid transient bug

See #1112

#14 @iandunn
3 years ago

Sorry it took so long to merge this. I ran into some problems while testing some of my changes, so I removed the batching in order to get the rest of the patch deployed. That limits the number of sites at 500, though, so it'd be good to come back and add the batching back in, I just didn't have time to do that myself right now.

After deploying it, I noticed a bug on production, and ended up having to lower the limit to 250. The comment in r4285 has more details. I wasn't able to reproduce it in my dev environment. That'll need to be fixed before the batching can be restored.

#15 @iandunn
2 years ago

  • Owner iandunn deleted
  • Status changed from accepted to assigned
Note: See TracTickets for help on using tickets.