Add i18n Team Crawler Plugin to Make Polyglots

[netweb]

Source: ​https://github.com/markoheijnen/wp-i18n-team-crawler

Per #wordpress-polyglots IRC Chat ​here

Extend plugin to include 'dormant' translations, we're at 3.8, anything 3.6 and below is dormant.

[markoheijnen]

I just uploaded my plugin I'm using on wpcentral.io. I'm sure getting the data can be smarter. Also I was curious what to do when there are no validators. Do we have a form where people could claim responsibility?

Currently the sites are requested by using the locales file of GlotPress. This is now an SVN external.

[Otto42]

Can somebody explain what the purpose of this plugin is?

It appears to be making remote requests to the api server for something, which means it makes no sense to put this on a make site as-is, or indeed to use a transient for caching it. The website itself already has access to the same data, but directly.

A good description of what the code is trying to do and display might mean that we can create a much simpler solution, such as simply writing code to pull and display the desired data.

[nacin]

The code is pretty straightforward as to what it's trying to do. Most Rosetta code isn't open-sourced yet (in progress) so this is what was written. Obviously, when adapted for WP.org we'd drop the API calls.

yoavf is gonna implement this; he can handle the transition to the internal calls.

[iandunn]

Refactor 340.diff to access data directly instead of using API

[iandunn]

340.2.diff refactors 340.diff to run locally on WordPress.org rather than using the API.

• I started with the GitHub repo instead of 340.diff, because it was more recent, but I used ​f38bc24 instead of HEAD because the commits after that were caching bits that'd be removed anyway.
• I assumed that Nacin changing the component to International Sites implied that the shortcode will run on global.wordpress.org instead of make.wordpress.org, but I can adjust it to work on Make if that was a bad assumption.
• The stats in the header vary slightly from wpcentral.io, but it looks like they're correct. I'm guessing the difference is caused by the caching in the wpcentral.io version.
• The translators/validators bits were removed because they weren't being used by anything, but I kept the existing $locale_data structure so they can be added back in the future.

Nacin, can you confirm where you intended this to live? After that, I think it's ready to commit, unless anyone has any feedback.

[markoheijnen]

I broke caching stuff till a few days ago but I did fixed that now.

Nacin did told me that the get_locale_url() method could be replaced by a WordPress.org function. I'm wondering why the plugin is not inside a folder but the root directory.

We need to look into translators/validators and how to display that data.

[samuelsidler]

It'll run on make.wordpress.org/polyglots/, specifically, not on global.wordpress.org. I think nacin moved it there because that's where a lot of the work is being done.

As Marko said, we'll want translators/validators back, but we'll need a good way to display that. It's okay to skip it for now.

[iandunn]

Nacin did told me that the get_locale_url() method could be replaced by a WordPress.org function.

Sound good. I'll take a second look and see what would be better.

I'm wondering why the plugin is not inside a folder but the root directory.

I just goofed it up when copying it from my sandbox to the Meta repo checkout. I'll fix that when I make the other changes.

It'll run on make.wordpress.org/polyglots/, specifically, not on global.wordpress.org. I think nacin moved it there because that's where a lot of the work is being done.

Sounds good, I'll make the necessary adjustments.

[iandunn]

Refactored 340.2.diff to work on make.wordpress.org

[iandunn]

340.3.diff is updated to work on make.wordpress.org instead of global.wordpress.org. but get_locale_url() still needs work. I found some code in international_forums() that looks like it could be used as a canonical source for those URLs, but it will need to be abstracted out.

This patch also adds translator/validator data back in, but I created a [wp-locale-contributors] shortcode for displaying it instead of using custom post types, since the CPT in the GitHub repo was a caching layer for wpcentral.io.

[sadeghr]

I'm at the translations.
I got these numbers translate to
​https://translate.wordpress.org/projects/buddypress/dev/fa/default
Thank you
Username:
sadeghr

[samuelsidler]

@sadeghr: Your comment doesn't belong here. If you need help with translating, try here: ​https://make.wordpress.org/polyglots/

[iandunn]

There's still more to do here, but 340.4.diff gets us a lot closer. Here are the highlights:

• Locale subdomains and latest releases pulled from MySQL instead of being hardcoded and scraped from the filesystem, respectively.
• Front end always built from cached data, which gets updated via cron every 5 minutes.
• Merged latest commits from the GitHub repo (through ​ab3f685), with a few exceptions. See below.
• Refactored HTML output into external views instead of being echo'd by PHP, to improve readability.
• Modularized the locale status counts (e.g., minor-behind, no-site, etc) for better caching and maintainability.
• Merged inc/crawler.php into the main class, since it only had 2 methods left in it after pulling more data from MySQL.
• Removed references to "crawler" since the wporg version isn't actually crawling anything.
• Removed the secondary shortcode that was introduced in 340.3.diff, so that both views are generated by the main shortcode.
• Lots of minor tweaks.

I didn't merge the flag image on the locale details page, because it was being pulled from a remote server that we can't trust, and which doesn't support HTTPS, which would lead to mixed-content warnings. It also seems wrong, since locales don't map cleanly to countries.

I also didn't merge the Core translations column, since I couldn't find a good way to access the data.

• Calling GP_Route_Locale::single() directly in PHP would require include()'ing all of GlotPress alongside WP, which seems like a recipe for disaster.
• Requesting ​https://translate.wordpress.org/api/languages/{locale} would require a separate call for each locale.
• Pulling directly from the database would mean duplicating a lot of the logic from the GlotPress API.

Any ideas for a better approach? Maybe we can create a GlotPress plugin to add the % complete to the ​https://translate.wordpress.org/api/languages endpoint?

[iandunn]

Replying to iandunn:

I didn't merge the flag image [...] because [...] locales don't map cleanly to countries.

Er, nevermind, I was thinking of languages, not locales. So, the flag might be fine, but we'll need to find an open-source image set that we can bundle, rather than relying on a remote service.

[vanillalounge]

the flag might be fine

Very sorry to barge in this late, but I'm not sure I follow the 'flag' logic. The way I see it, it is simply not needed, nor is it even relevant. The country-to-language relationship (and vice-versa) is n-to-n, so that a flag adds nothing to the list.

[ocean90]

Looking at 340.4.diff​:
The $local_data array shouldn't be key'ed by wp_locale. We stopped with this in core and on API site to be future-proof in terms for variantions of the same locale (formal/informal). I think it should also use the same sorting as we have for for the language dropdown on install, see https://meta.trac.wordpress.org/browser/sites/trunk/api.wordpress.org/public_html/translations/lib.php#L77.

[iandunn]

Replying to ocean90:

The $local_data array shouldn't be key'ed by wp_locale. We stopped with this in core and on API site to be future-proof in terms for variantions of the same locale (formal/informal).

Nice catch, thanks for the heads up :)

[iandunn]

Minor fixes

[iandunn]

340.5.diff only has a few small fixes over 340.4.diff, but it may be awhile before I have time to circle back to this again, so I wanted to make sure the latest changes were here in case anyone else wants to work on it.

[nacin]

In 1061:

First pass at ​http://make.wordpress.org/polyglots/teams/.

props markoheijnen, iandunn.
see #340.

[nacin]

There's a lot more I want to improve/fix here, but [1061] is my first pass after going through the patches here.

[markoheijnen]

Great work! After compared some data in general it looks fine but somehow nl_NL is showing it seems everyone who translated for Dutch but the API does return only a handful.

[ocean90]

In 1068:

I18N Teams: Hide download links for locales which have never released a version.

see #340.

[ocean90]

In 1139:

i18n Teams: Improve translation counts.

• Fix division by zero warning.
• Show 0% for locales which have the projects, but without translations.

see #340.

[ocean90]

In 1140:

i18n Teams: On details page use english name of the locale.

fixes #795.
see #340.

[ocean90]

In 1157:

GlotPress templates: Add link to i18n teams in header.

see #340.

[ocean90]

Add filter for locales with 100% translated strings

[ocean90]

In 1163:

i18n Teams: Add filters for translation status.

see #340.

[ocean90]

In 1164:

i18n Teams: Bump version for assets.

see #340.

[ocean90]

In 1190:

WP i18n Teams: Avoid notices if locale has no site yet.

see #340.

[samuelsidler]

Fixed a while ago. :)