WordPress.org

Making WordPress.org

Opened 5 years ago

Closed 4 years ago

#760 closed task (fixed)

Plugins directory internationalization project

Reported by: stephdau Owned by: stephdau
Milestone: Priority: normal
Component: Plugin Directory Keywords: has-patch
Cc:

Description (last modified by stephdau)

Update: description below is for the original scope of the ticket ("Extract and import plugins code and readme strings to GlotPress on SVN repo actions"). We're now using the same ticket for the whole project ("Plugins directory internationalization project"), in an effort to keep things in one place.


The attached patches introduce new methods to achieve the task in the existing SVN tracker code.

Processing:

  • Dotorg_Plugins_Tracker::process_i18n( $slug, $branch = 'dev', $type = 'all' )
  • Dotorg_Plugins_Tracker::process_code_i18n( $path_rel, $branch = 'dev' )
  • Dotorg_Plugins_Tracker::process_readme_i18n( $path_rel, $branch = 'dev' )


Helpers:

  • Dotorg_Plugins_Tracker::handle_translator_comment( $array, $key, $val )
  • Dotorg_Plugins_Tracker::import_to_glotpress_project( $project, $branch, $file )

The methods can be called from within the class, externally, or be run from the CLI as:

cd ./plugins/

# processes both code and readme in dev/trunk
php bb-load.php i18n blogware-importer

# processes code only, in stable branch
php bb-load.php i18n blogware-importer stable code

# processes readme only, in dev
php bb-load.php i18n blogware-importer dev readme

Notes:

Left:

  • Need list of plugins we want to start with, since we decided to start with a finite number. Note: this might be better off being decided in private, asked Nacin
  • Need to define when/where to trigger the code/readme i18n reruns exactly (have options, need to pick with sys/perf in mind)
  • Logic to create GP projects on the fly for new and existing plugins, thing we can leave for the next cycle since only dealing with a set list of plugins at first.

Attachments (12)

760.diff (9.3 KB) - added by stephdau 5 years ago.
760.2.diff (15.8 KB) - added by stephdau 5 years ago.
Implements feedback so far, and introduces some experimental display related methods
760-wporg.diff (21.5 KB) - added by stephdau 5 years ago.
Follow up on 760.2.diff: refactors code into own file/class for cleanliness, adds full caching (see 760-translate.diff for cache handling), adds translation support in search results and listings, improves all code.
760-api.diff (510 bytes) - added by stephdau 5 years ago.
Adds translation support to api.wordpress.org, once 760-wporg.diff is in place
760-translate.diff (4.0 KB) - added by stephdau 5 years ago.
GlotPress plugin for translate.wordpress.org to clear/delete the transaltion cache keys used in the localized plugins directories and api upon interaction with GP (imports, translation edits, etc).
760-wporg.2.diff (22.3 KB) - added by stephdau 5 years ago.
760-wporg.3.diff (32.1 KB) - added by stephdau 5 years ago.
760-wporg.4.diff (33.3 KB) - added by stephdau 5 years ago.
760-translate.2.diff (4.3 KB) - added by stephdau 5 years ago.
760-wporg.5.diff (40.5 KB) - added by stephdau 5 years ago.
Production-ready version
760-api.2.diff (511 bytes) - added by stephdau 5 years ago.
Production-ready version
760-translate.3.diff (4.9 KB) - added by stephdau 5 years ago.
Production-ready version

Download all attachments as: .zip

Change History (41)

@stephdau
5 years ago

#1 @stephdau
5 years ago

  • Description modified (diff)
  • Owner set to stephdau
  • Status changed from new to accepted

This ticket was mentioned in Slack in #meta-i18n by stephdau. View the logs.


5 years ago

#3 @stephdau
5 years ago

  • Description modified (diff)

#4 follow-up: @stephdau
5 years ago

Logic questions:

This ticket was mentioned in Slack in #meta-i18n by stephdau. View the logs.


5 years ago

#6 in reply to: ↑ 4 ; follow-up: @nacin
5 years ago

Replying to stephdau:

Logic questions:

  • Dotorg_Plugins_Tracker::process_i18n() 760.diff→L2020: if a plugin lists its stable tag as trunk, the patch uses the dev and dev-readme instance of the related GP project, so translators do not need to needlessly maintain 2 "sets" (dev vs stable). Is that the right way to go?

Yes.

  • Dotorg_Plugins_Tracker::process_code_i18n() 760.diff→L2061: I have logic to use a plugin's own POT file, instead of generating a temporary one ourselves, if committed in the repo as {$slug}/{$slug}.pot or {$slug}/languages/{$slug}.pot. Is that also the right way to go?

Hmm. Not sure. I would say no.

  • Dotorg_Plugins_Tracker::process_code_i18n(): should we also have logic to use a plugin's own committed PO/MO files if any, like Jetpack? What if they don't have a committed POT file, and ours somehow doesn't match their PO/MO files (doubtful but possible)?

Nope, I don't think so. These will override language packs, but the goal should be to import them into GP and remove them from the plugin.

#7 @stephdau
5 years ago

  1. Excellent
  2. Cool
  3. Sounds fair. We can do that manually for the initial short list, and work on longer-term automation afterward, along with the does-project-exist-create-if-not parts.

This ticket was mentioned in Slack in #meta-i18n by stephdau. View the logs.


5 years ago

@stephdau
5 years ago

Implements feedback so far, and introduces some experimental display related methods

#9 @stephdau
5 years ago

760.2.diff Implements feedback so far, and introduces some experimental display related methods:

  • Dotorg_Plugins_Tracker::translate()
  • related Dotorg_Plugins_Tracker::_*_gp_*() helpers
  • Dotorg_Plugins_Tracker::localize_section_url()

There's no caching or anything, but the queries are based on indices, and the code is "functional" to showcase how things will work.

This ticket was mentioned in Slack in #meta-i18n by stephdau. View the logs.


5 years ago

@stephdau
5 years ago

Follow up on 760.2.diff: refactors code into own file/class for cleanliness, adds full caching (see 760-translate.diff for cache handling), adds translation support in search results and listings, improves all code.

@stephdau
5 years ago

Adds translation support to api.wordpress.org, once 760-wporg.diff is in place

@stephdau
5 years ago

GlotPress plugin for translate.wordpress.org to clear/delete the transaltion cache keys used in the localized plugins directories and api upon interaction with GP (imports, translation edits, etc).

#11 @stephdau
5 years ago

When combined together, the latest patches bring the entire solution to life (albeit still experimental)! Consider it a complete proof-of-concept of the whole plugins directories internationalization project. :D

See https://cloudup.com/cmAHT9qu2Wx

  • 760-wporg.diff: handles code and readme translation processing (CLI and automated in Dotorg_Plugins_Tracker::save_data()), as well as localized directories caching and display
  • 760-api.diff: handles api.wordpress.org translation when a (temp/dev) language param is passed
  • 760-translate.diff: handles clearing the translation cache keys used in the localized plugins directories and api upon interaction with GP (imports, translation edits, etc).
Last edited 5 years ago by stephdau (previous) (diff)

@stephdau
5 years ago

#12 @stephdau
5 years ago

760-wporg.2.diff:

  • Dotorg_Plugin_I18n::import_to_glotpress_project() now also sets some GlotPress string priorities.
    • https://cloudup.com/cYTSZvl3bi0
    • plugin name & short description (both used the most): high prio (1)
    • changelog items: low prio (-1)
    • everything else: normal prio (0)
    • treating description items as standard prio so the important strings are not lost in a sea of high prio ones for readmes with long descriptions.
  • No longer processes the license string from the readme, as being in the WP Plugin Directory implies GPLv2 or later. No need for everybody translating that in every plugin.
  • Misc code cleanups (method renaming, etc).
Last edited 5 years ago by stephdau (previous) (diff)

This ticket was mentioned in Slack in #meta-i18n by stephdau. View the logs.


5 years ago

@stephdau
5 years ago

#14 @stephdau
5 years ago

760-wporg.3.diff improves on the exisitng code, as well as adds translate/bin/set-wp-plugin-project.php, which is used by Dotorg_Plugin_I18n::process() to initialize the GlotPress environment required for the targeted plugin.

php translate/bin/set-wp-plugin-project.php livejournal-importer

@stephdau
5 years ago

#15 @stephdau
5 years ago

760-wporg.4.diff:

  • adds support for both short and long locale/language slugs (fr vs fr_FR, for eg).
  • handles $_REQUESTrequest?->locale for api.wordpress.org queries.
  • adds and uses Dotorg_Plugin_I18n::verify_subdomain() for localized sites (fr.wordpress.org, etc)
  • misc improvements over 760-wporg.3.diff

#16 @stephdau
5 years ago

760-translate.2.diff makes sure that GP_WPorg_Plugins::delete_plugin_i18n_cache_keys_for() deals with cache keys for both a short and long versions of a locale (eg: fr vs fr_FR), as keys might exist under either.

This ticket was mentioned in Slack in #meta-i18n by stephdau. View the logs.


5 years ago

#18 @stephdau
5 years ago

  • Description modified (diff)
  • Summary changed from Extract and import plugins code and readme strings to GlotPress on SVN repo actions to Plugins directory internationalization project

This ticket was mentioned in Slack in #polyglots by ocean90. View the logs.


5 years ago

@stephdau
5 years ago

Production-ready version

@stephdau
5 years ago

Production-ready version

@stephdau
5 years ago

Production-ready version

#20 @stephdau
5 years ago

The code was put in production at the end of last week (Jan. 23rd).

#21 in reply to: ↑ 6 ; follow-up: @DavidAnderson
5 years ago

Replying to nacin:

  • Dotorg_Plugins_Tracker::process_code_i18n(): should we also have logic to use a plugin's own committed PO/MO files if any, like Jetpack? What if they don't have a committed POT file, and ours somehow doesn't match their PO/MO files (doubtful but possible)?

Nope, I don't think so. These will override language packs, but the goal should be to import them into GP and remove them from the plugin.

What are the implications of this, in terms of plugins that already have their own GlotPress install (or other system) elsewhere, that people already work on? And who want to keep on using it?

#22 in reply to: ↑ 21 ; follow-up: @Otto42
5 years ago

Replying to DavidAnderson:

What are the implications of this, in terms of plugins that already have their own GlotPress install (or other system) elsewhere, that people already work on? And who want to keep on using it?

None. If a plugin includes its own po/mo files, then those get used preferentially.

#23 in reply to: ↑ 22 ; follow-up: @DavidAnderson
5 years ago

Replying to Otto42:

Replying to DavidAnderson:

What are the implications of this, in terms of plugins that already have their own GlotPress install (or other system) elsewhere, that people already work on? And who want to keep on using it?

None. If a plugin includes its own po/mo files, then those get used preferentially.

I think what I'm really asking is, whether there'll be a way for plugin authors to provide their own PO/MO files such that they a) don't have to be included in the plugin zip download, and b) also don't have to use WP's GlotPress install to make translations, but can use their own.

e.g. Something like, if a plugin author uploads his translations (which he gets from whatever source he prefers) to assets/translations/, then those translations can be delivered through the new "don't bloat the plugin download" mechanism, instead of plugin translators being required to use WP's GlotPress install as the place where they have to make translations. (Of course, GlotPress has an 'import' function... so perhaps my question is about whether importing can be automated through some mechanism like this, rather than having to go through the WP GlotPress website).

Apologies if this is a silly question - I did some Googling to try to learn about the state of this project, but learnt almost nothing...

#24 in reply to: ↑ 23 ; follow-up: @Otto42
5 years ago

Replying to DavidAnderson:

I think what I'm really asking is, whether there'll be a way for plugin authors to provide their own PO/MO files such that they a) don't have to be included in the plugin zip download, and b) also don't have to use WP's GlotPress install to make translations, but can use their own.

No, that's not part of the current plan or design.

#25 in reply to: ↑ 24 @DavidAnderson
5 years ago

Replying to Otto42:

Replying to DavidAnderson:

I think what I'm really asking is, whether there'll be a way for plugin authors to provide their own PO/MO files such that they a) don't have to be included in the plugin zip download, and b) also don't have to use WP's GlotPress install to make translations, but can use their own.

No, that's not part of the current plan or design.

And, what will be the rules about including strings in your POT file that aren't in the wordpress.org plugin? The reason for asking is, I have plugins which have both free and paid versions. Strings in the premium version are a superset of those in the free version. It sounds at the moment like the choice will boil down to:
1) Bloat the free plugin with all the MO/PO files
2) Add paid-only strings to WP GlotPress (which may be forbidden, and hence not actually an option)
3) Do lots of hand import/exporting between two separate GlotPress installs, or ask translators to work in both.

If 2) is forbidden, then 1) looks much better than 3 - it means "things continue as they are".

Also... as another way of doing it... is part of the current design to allow plugins to hook in so that they can specify alternative download locations? And if so, will plugins hosted on wordpress.org be allowed to use use those alternatives? (Like WooCommerce already does, I suppose - i.e. it has an off-site location for its translations, which doesn't violate the "no third party code" rule).

#26 @dd32
5 years ago

  1. All strings in GlotPress will be pulled automatically from the plugin files (so proper textdomain support is required), all pot/po/mo files in the plugin are ignored
  2. Plugins in the repository should not be "bloated" with extra premium-only strings, and are already not allowed to simply have premium-only code disabled, so that should extend to extra strings for that purpose
  3. If an author chooses to use their own external GlotPress install, they can continue to bundle the po/mo files, they simply don't get the benefit of Language packs
  4. External language pack sources will not be supported, just the same as we don't allow for external plugin update sources, but of course plugins can do anything they want (as long as they abide by all WordPress.org repo rules if hosted there)

Since GlotPress does offer both an Export and Import functionality, it should be conceivable to build a work-flow around that, and I'm sure some automated tools will pop up to help with that as the larger plugins start using the system.

This ticket was mentioned in Slack in #polyglots by ocean90. View the logs.


5 years ago

This ticket was mentioned in Slack in #glotpress by netweb. View the logs.


4 years ago

#29 @Otto42
4 years ago

  • Resolution set to fixed
  • Status changed from accepted to closed

All plugins and themes are currently included in language packs. Closing as this is now fixed. Any further changes probably need new tickets.

Note: See TracTickets for help on using tickets.