WordPress.org

Making WordPress.org

Opened 15 months ago

Closed 7 months ago

Last modified 6 months ago

#3748 closed enhancement (fixed)

Consider using WP-CLI for string extraction

Reported by: swissspidy Owned by: ocean90
Milestone: Priority: normal
Component: Translate Site & Plugins Keywords: has-patch
Cc:

Description

The last 8 months or so I have been working on a new WP-CLI 18n-command that makes it easier for WordPress projects to generate POT files.

Before that, I have been using makepot.php for years, but the tool is not intuitive and really flexible. Also, with JavaScript internationalization (#wp20491) and Gutenberg being a thing, it was clear that makepot.php wasn't the right tool for the job anymore.

Then there's also issues like #3601 and #wp42747 that make it harder for WordPress projects to really use makepot.php. Even for WordPress core I discovered two cases where strings were not extracted at all:

  • <strong>ERROR</strong>: "Table Prefix" can only contain numbers, letters, and underscores. (source: wp-admin/setup-config.php:262)
  • <strong>ERROR</strong>: "Table Prefix" must not be empty. (source: wp-admin/setup-config.php:258)

The WP-CLI command on the other hand does not have these issues.

That's why I propose switching to this new WP-CLI command for string extraction on WordPress.org infrastructure.

Some of the features:

  • Automatically detects plugins and themes and extracts file headers.
  • Allows extraction of only a specific text domain.
  • Supports JavaScript string extraction, even for JSX and ESNext.
  • Allows merging the resulting POT file with an existing one, e.g. one created by Babel.
  • Powerful rules to include/exclude specific directories (minified JS files, vendor, .git folder, etc.)
  • Supports extracting strings from WordPress core the same way it's done today with 4 different projects. See https://github.com/wp-cli/i18n-command/pull/69 for examples.
  • Can warn about strings with wrong placeholders, as well as misleading or missing translator comments. This could be very useful for core but also plugin/developers to improve polyglots UX.

Also, the command is extensively tested with Behat (with PHPUnit tests for some parts planned) and is already used in third-party projects.

With WP-CLI being an official WordPress project and this new command being superior to makepot.php in many ways, I think we have an opportunity to consolidate lots of things here, reduce the maintenance burden, and to make it easier for people to use WordPress I18N tools and to contribute to them.

It goes without saying that I'd love to help making this a reality, even if it means adding some new features to the command or fixing nasty bugs.

I'd suggest giving this command a go locally with some plugin, theme, or core to get a feeling for it.

Since we built it with WP-CLI 2.0 in mind, which should be released tomorrow, you might need to run the following commands to install it locally:

wp cli update --nightly
wp package install wp-cli/i18n-command

Alternatively, you can check out the GitHub repository and run it from within its directory (still requiring WP-CLI 2.0 though).

Attachments (7)

3748.diff (977 bytes) - added by herregroen 12 months ago.
3748.2.diff (1.2 KB) - added by herregroen 12 months ago.
3748.3.diff (1.3 KB) - added by herregroen 12 months ago.
3748.4.diff (3.6 KB) - added by herregroen 12 months ago.
3748.5.diff (234.4 KB) - added by herregroen 12 months ago.
3748.6.diff (237.5 KB) - added by herregroen 12 months ago.
3748.7.diff (3.6 KB) - added by herregroen 12 months ago.

Download all attachments as: .zip

Change History (36)

#1 @schlessera
15 months ago

Alternatively, you can check out the GitHub repository and run it from within its directory (still requiring WP-CLI 2.0 though).

Each command should work in a self-contained way, provided you use the WP-CLI version in vendor/bin after you did a composer install, so you don't need to install anything outside of the cloned wp-cli/i18n-command repository.

Example: vendor/bin/wp i18n make-pot --help

#2 @schlessera
15 months ago

The command is now part of the nightly builds, no need to install it manually. Just run the following command to make it available:

wp cli update --nightly

This ticket was mentioned in Slack in #meta by tellyworth. View the logs.


12 months ago

@herregroen
12 months ago

@herregroen
12 months ago

#4 @herregroen
12 months ago

Added a patch that will call this command instead of loading and using makepot.php.

It should be ensured that wp-cli has been updated on wordpress.org and this command is available.

@herregroen
12 months ago

#5 @herregroen
12 months ago

Updated the patch to also set the slug command.

I'm currently simply replacing the call to makepot.php with wp-cli. I think that's the easiest and best approach to make.

This patch only covers plugin imports, I've not yet been able to locate where exactly the calls to makepot.php happen for core.

These should also be switched to use wp i18n so we can successfully gather JS translations occurring there.

@herregroen
12 months ago

#6 @herregroen
12 months ago

Patch updated to handle POT generation for theme files as well. Using WP_CLI directly instead of exec as this is occurring during another wp-cli command.

#7 @herregroen
12 months ago

  • Keywords has-patch added

#8 @ocean90
12 months ago

This patch only covers plugin imports, I've not yet been able to locate where exactly the calls to makepot.php happen for core.

The code which calls cron-svn-pots.php is not open-sourced but it's bascially this:

$POT_GEN -m wp-frontend -c $CO/wordpress -p $CO/wp-pot -n wordpress.pot --min-version=3.7
$POT_GEN -m wp-admin -c $CO/wordpress -p $CO/wp-pot -n wordpress-admin.pot -f --min-version=3.7
$POT_GEN -m wp-network-admin -c $CO/wordpress -p $CO/wp-pot -n wordpress-admin-network.pot -f --min-version=3.7
$POT_GEN -m wp-tz -c $CO/wordpress -p $CO/wp-pot -n wordpress-continents-cities.pot -f --min-version=3.7

Thanks for the plugin/theme patches. For back-compat the --ignore-domain argument should be set.

@herregroen
12 months ago

#9 @swissspidy
12 months ago

3748.5.diff contains lots of unrelated code 🤔

As for the WordPress core string extraction, I have written down how these calls can be migrated to WP-CLI here: https://github.com/wp-cli/i18n-command/pull/69

As one can see, there are lots of exceptions for things like default themes, Hello Dolly, the list of continents and cities, etc.

In theory we don't have to use WP-CLI for all parts just yet, as we only care about JS right now. However, the WP-CLI command is a bit superior because it also finds strings that the current makepot.php script actually misses.

#10 @ocean90
12 months ago

  • Owner set to ocean90
  • Status changed from new to accepted

@swissspidy Thanks for the link, I knew you posted them somewhere. I was able to use them in a first pass and it's looking good so far. Will do some more tests to have a version ready by end of this week.

@herregroen
12 months ago

@herregroen
12 months ago

#11 @ocean90
12 months ago

In 7792:

Translate: Add wporg-translate make-core-pot command to create POT files for WordPress core.

See #3748.

#12 @ocean90
12 months ago

In 7801:

Translate: Extend wporg-translate make-core-pot command to support generating POT files for WordPress 3.7+.

See #3748.

This ticket was mentioned in Slack in #cli by ocean90. View the logs.


12 months ago

#14 @ocean90
12 months ago

In 7807:

Translate: Pass --skip-audit to i18n make-pot to disable audits.

See #3748.

#15 @ocean90
12 months ago

In 7808:

Translate: In wporg-translate make-core-pot enable JavaScript string extraction if supported.

See #3748.

#16 @ocean90
12 months ago

In 7809:

Translate: Exclude wp-includes/js/tinymce/* only for WordPress 4.3+.

See #3748.

#17 @ocean90
12 months ago

In 7810:

Translate: Remove duplicated exclude for wp-includes/js/tinymce/*.

See #3748.

This ticket was mentioned in Slack in #core-js by omarreiss. View the logs.


12 months ago

This ticket was mentioned in Slack in #core by omarreiss. View the logs.


12 months ago

#20 @ocean90
12 months ago

In 7833:

Translate: Add wporg-translate export command to export translation files (po, mo, and json) for WordPress core.

Props herregroen, ocean90.
See #3748.

This ticket was mentioned in Slack in #core-js by herregroen. View the logs.


11 months ago

#22 @ocean90
11 months ago

In 7843:

Translate: Use correct variable for locale slug.

See #3748.

This ticket was mentioned in Slack in #meta by herregroen. View the logs.


11 months ago

#24 @ocean90
11 months ago

In 7893:

Plugin Directory: Use wp i18n make-pot for string extraction.

Props herregroen.
See #3748.

#25 @ocean90
11 months ago

In 7894:

Plugin Directory: Remove i18n-tools SVN external.

See #3748.

#26 @ocean90
11 months ago

In 7895:

Translate, Theme Directory: Use wp i18n make-pot for string extraction.

Props herregroen, ocean90.
See #3748.

This ticket was mentioned in Slack in #polyglots by ocean90. View the logs.


11 months ago

#28 @ocean90
7 months ago

  • Resolution set to fixed
  • Status changed from accepted to closed

#29 @ocean90
6 months ago

In 8640:

Translate: Extract JavaScript translations from admin files for WordPress 5.2+.

See #3748.

Note: See TracTickets for help on using tickets.