Plugins directory internationalization project

[stephdau]

Update: description below is for the original scope of the ticket ("Extract and import plugins code and readme strings to GlotPress on SVN repo actions"). We're now using the same ticket for the whole project ("Plugins directory internationalization project"), in an effort to keep things in one place.

---

The attached patches introduce new methods to achieve the task in the existing SVN tracker code.

Processing:

• Dotorg_Plugins_Tracker::process_i18n( $slug, $branch = 'dev', $type = 'all' )
• Dotorg_Plugins_Tracker::process_code_i18n( $path_rel, $branch = 'dev' )
• Dotorg_Plugins_Tracker::process_readme_i18n( $path_rel, $branch = 'dev' )

Helpers:

• Dotorg_Plugins_Tracker::handle_translator_comment( $array, $key, $val )
• Dotorg_Plugins_Tracker::import_to_glotpress_project( $project, $branch, $file )

The methods can be called from within the class, externally, or be run from the CLI as:

cd ./plugins/

# processes both code and readme in dev/trunk
php bb-load.php i18n blogware-importer

# processes code only, in stable branch
php bb-load.php i18n blogware-importer stable code

# processes readme only, in dev
php bb-load.php i18n blogware-importer dev readme

Notes:

• Using ​https://translate.wordpress.org/projects/wp-plugins/blogware-importer/dev for code parsing imports (transparent, since no changes, is what’s expected)
• Now have a sample readme sub-project at ​https://translate.wordpress.org/projects/wp-plugins/blogware-importer/dev-readme
• Using the above to work the import kinks out (html entities, etc.
• Nacin gave me ok and privs to create as many GlotPress projects/sub-projects as I need, all set on that front.

Left:

• Need list of plugins we want to start with, since we decided to start with a finite number. Note: this might be better off being decided in private, asked Nacin
• Need to define when/where to trigger the code/readme i18n reruns exactly (have options, need to pick with sys/perf in mind)
• Logic to create GP projects on the fly for new and existing plugins, thing we can leave for the next cycle since only dealing with a set list of plugins at first.

[stephdau]

Logic questions:

• Dotorg_Plugins_Tracker::process_i18n() 760.diff​→L2020: if a plugin ​lists its stable tag as trunk, the patch uses the dev and dev-readme instance of the related GP project, so translators do not need to needlessly maintain 2 "sets" (dev vs stable). Is that the right way to go?
• Dotorg_Plugins_Tracker::process_code_i18n() 760.diff​→L2061: I have logic to use a plugin's own POT file, instead of generating a temporary one ourselves, if committed in the repo as {$slug}/{$slug}.pot or {$slug}/languages/{$slug}.pot. Is that also the right way to go?
  • ​https://plugins.trac.wordpress.org/browser/buddypress/trunk/buddypress.pot
  • ​https://plugins.trac.wordpress.org/browser/blogware-importer/trunk/languages/blogware-importer.pot
• Dotorg_Plugins_Tracker::process_code_i18n(): should we also have logic to use a plugin's own committed PO/MO files if any, ​like Jetpack? What if they don't have a committed POT file, and ours somehow doesn't match their PO/MO files (doubtful but possible)?

[nacin]

Replying to stephdau:

Logic questions:

• Dotorg_Plugins_Tracker::process_i18n() 760.diff​→L2020: if a plugin ​lists its stable tag as trunk, the patch uses the dev and dev-readme instance of the related GP project, so translators do not need to needlessly maintain 2 "sets" (dev vs stable). Is that the right way to go?

Yes.

• Dotorg_Plugins_Tracker::process_code_i18n() 760.diff​→L2061: I have logic to use a plugin's own POT file, instead of generating a temporary one ourselves, if committed in the repo as {$slug}/{$slug}.pot or {$slug}/languages/{$slug}.pot. Is that also the right way to go?

Hmm. Not sure. I would say no.

• Dotorg_Plugins_Tracker::process_code_i18n(): should we also have logic to use a plugin's own committed PO/MO files if any, ​like Jetpack? What if they don't have a committed POT file, and ours somehow doesn't match their PO/MO files (doubtful but possible)?

Nope, I don't think so. These will override language packs, but the goal should be to import them into GP and remove them from the plugin.

[stephdau]

1. Excellent
2. Cool
3. Sounds fair. We can do that manually for the initial short list, and work on longer-term automation afterward, along with the does-project-exist-create-if-not parts.

[stephdau]

Implements feedback so far, and introduces some experimental display related methods

[stephdau]

760.2.diff​ Implements feedback so far, and introduces some experimental display related methods:

• Dotorg_Plugins_Tracker::translate()
• related Dotorg_Plugins_Tracker::_*_gp_*() helpers
• Dotorg_Plugins_Tracker::localize_section_url()

There's no caching or anything, but the queries are based on indices, and the ​code is "functional" to showcase how things will work.

[stephdau]

Follow up on 760.2.diff: refactors code into own file/class for cleanliness, adds full caching (see 760-translate.diff for cache handling), adds translation support in search results and listings, improves all code.

[stephdau]

Adds translation support to api.wordpress.org, once 760-wporg.diff is in place

[stephdau]

GlotPress plugin for translate.wordpress.org to clear/delete the transaltion cache keys used in the localized plugins directories and api upon interaction with GP (imports, translation edits, etc).

[stephdau]

When combined together, the latest patches bring the entire solution to life (albeit still experimental)! Consider it a complete proof-of-concept of the whole plugins directories internationalization project. :D

See ​https://cloudup.com/cmAHT9qu2Wx

• 760-wporg.diff​: handles code and readme translation processing (CLI and automated in Dotorg_Plugins_Tracker::save_data()), as well as localized directories caching and display
• 760-api.diff​: handles api.wordpress.org translation when a (temp/dev) language param is passed
• 760-translate: handles clearing the translation cache keys used in the localized plugins directories and api upon interaction with GP (imports, translation edits, etc).

[stephdau]

760-wporg.2.diff​:

• Dotorg_Plugin_I18n::import_to_glotpress_project() now also ​sets some GlotPress string priorities.
  • ​https://cloudup.com/cYTSZvl3bi0
  • plugin name & short description (both used the most): high prio (1)
  • changelog items: low prio (-1)
  • everything else: normal prio (0)
  • treating description items as standard prio so the important strings are not lost in a sea of high prio ones for readmes with long descriptions.
• No longer processes the license string from the readme, as being in the WP Plugin Directory implies GPLv2 or later. No need for everybody translating that in every plugin.
• Misc code cleanups (method renaming, etc).

[stephdau]

760-wporg.3.diff​ improves on the exisitng code, as well as adds translate/bin/set-wp-plugin-project.php, which is used by Dotorg_Plugin_I18n::process() to initialize the GlotPress environment required for the targeted plugin.

php translate/bin/set-wp-plugin-project.php livejournal-importer

[stephdau]

760-wporg.4.diff​:

• adds support for both short and long locale/language slugs (fr vs fr_FR, for eg).
• handles $_REQUESTrequest?->locale for api.wordpress.org queries.
• adds and uses Dotorg_Plugin_I18n::verify_subdomain() for localized sites (fr.wordpress.org, etc)
• misc improvements over 760-wporg.3.diff​

[stephdau]

760-translate.2.diff​ makes sure that GP_WPorg_Plugins::delete_plugin_i18n_cache_keys_for() deals with cache keys for both a short and long versions of a locale (eg: fr vs fr_FR), as keys might exist under either.

[stephdau]

Production-ready version

[stephdau]

Production-ready version

[stephdau]

Production-ready version

[stephdau]

The code was put in production at the end of last week (Jan. 23rd).

[DavidAnderson]

Replying to nacin:

• Dotorg_Plugins_Tracker::process_code_i18n(): should we also have logic to use a plugin's own committed PO/MO files if any, ​like Jetpack? What if they don't have a committed POT file, and ours somehow doesn't match their PO/MO files (doubtful but possible)?

Nope, I don't think so. These will override language packs, but the goal should be to import them into GP and remove them from the plugin.

What are the implications of this, in terms of plugins that already have their own GlotPress install (or other system) elsewhere, that people already work on? And who want to keep on using it?

[Otto42]

Replying to DavidAnderson:

What are the implications of this, in terms of plugins that already have their own GlotPress install (or other system) elsewhere, that people already work on? And who want to keep on using it?

None. If a plugin includes its own po/mo files, then those get used preferentially.

[DavidAnderson]

Replying to Otto42:

Replying to DavidAnderson:

What are the implications of this, in terms of plugins that already have their own GlotPress install (or other system) elsewhere, that people already work on? And who want to keep on using it?

None. If a plugin includes its own po/mo files, then those get used preferentially.

I think what I'm really asking is, whether there'll be a way for plugin authors to provide their own PO/MO files such that they a) don't have to be included in the plugin zip download, and b) also don't have to use WP's GlotPress install to make translations, but can use their own.

e.g. Something like, if a plugin author uploads his translations (which he gets from whatever source he prefers) to assets/translations/, then those translations can be delivered through the new "don't bloat the plugin download" mechanism, instead of plugin translators being required to use WP's GlotPress install as the place where they have to make translations. (Of course, GlotPress has an 'import' function... so perhaps my question is about whether importing can be automated through some mechanism like this, rather than having to go through the WP GlotPress website).

Apologies if this is a silly question - I did some Googling to try to learn about the state of this project, but learnt almost nothing...

[Otto42]

Replying to DavidAnderson:

I think what I'm really asking is, whether there'll be a way for plugin authors to provide their own PO/MO files such that they a) don't have to be included in the plugin zip download, and b) also don't have to use WP's GlotPress install to make translations, but can use their own.

No, that's not part of the current plan or design.

[DavidAnderson]

Replying to Otto42:

Replying to DavidAnderson:

I think what I'm really asking is, whether there'll be a way for plugin authors to provide their own PO/MO files such that they a) don't have to be included in the plugin zip download, and b) also don't have to use WP's GlotPress install to make translations, but can use their own.

No, that's not part of the current plan or design.

And, what will be the rules about including strings in your POT file that aren't in the wordpress.org plugin? The reason for asking is, I have plugins which have both free and paid versions. Strings in the premium version are a superset of those in the free version. It sounds at the moment like the choice will boil down to:
1) Bloat the free plugin with all the MO/PO files
2) Add paid-only strings to WP GlotPress (which may be forbidden, and hence not actually an option)
3) Do lots of hand import/exporting between two separate GlotPress installs, or ask translators to work in both.

If 2) is forbidden, then 1) looks much better than 3 - it means "things continue as they are".

Also... as another way of doing it... is part of the current design to allow plugins to hook in so that they can specify alternative download locations? And if so, will plugins hosted on wordpress.org be allowed to use use those alternatives? (Like WooCommerce already does, I suppose - i.e. it has an off-site location for its translations, which doesn't violate the "no third party code" rule).

[dd32]

1. All strings in GlotPress will be pulled automatically from the plugin files (so proper textdomain support is required), all pot/po/mo files in the plugin are ignored
2. Plugins in the repository should not be "bloated" with extra premium-only strings, and are already not allowed to simply have premium-only code disabled, so that should extend to extra strings for that purpose
3. If an author chooses to use their own external GlotPress install, they can continue to bundle the po/mo files, they simply don't get the benefit of Language packs
4. External language pack sources will not be supported, just the same as we don't allow for external plugin update sources, but of course plugins can do anything they want (as long as they abide by all WordPress.org repo rules if hosted there)

Since GlotPress does offer both an Export and Import functionality, it should be conceivable to build a work-flow around that, and I'm sure some automated tools will pop up to help with that as the larger plugins start using the system.

[Otto42]

All plugins and themes are currently included in language packs. Closing as this is now fixed. Any further changes probably need new tickets.