Consider using WP-CLI for string extraction

[swissspidy]

The last 8 months or so I have been working on a new WP-CLI ​18n-command that makes it easier for WordPress projects to generate POT files.

Before that, I have been using makepot.php for years, but the tool is not intuitive and really flexible. Also, with JavaScript internationalization (​#wp20491) and Gutenberg being a thing, it was clear that makepot.php wasn't the right tool for the job anymore.

Then there's also issues like #3601 and ​#wp42747 that make it harder for WordPress projects to really use makepot.php. Even for WordPress core I discovered two cases where strings were not extracted at all:

• <strong>ERROR</strong>: "Table Prefix" can only contain numbers, letters, and underscores. (source: wp-admin/setup-config.php:262)
• <strong>ERROR</strong>: "Table Prefix" must not be empty. (source: wp-admin/setup-config.php:258)

The WP-CLI command on the other hand does not have these issues.

That's why I propose switching to this new WP-CLI command for string extraction on WordPress.org infrastructure.

Some of the features:

• Automatically detects plugins and themes and extracts file headers.
• Allows extraction of only a specific text domain.
• Supports JavaScript string extraction, even for JSX and ESNext.
• Allows merging the resulting POT file with an existing one, e.g. one created by Babel.
• Powerful rules to include/exclude specific directories (minified JS files, vendor, .git folder, etc.)
• Supports extracting strings from WordPress core the same way it's done today with 4 different projects. See ​https://github.com/wp-cli/i18n-command/pull/69 for examples.
• Can warn about strings with wrong placeholders, as well as misleading or missing translator comments. This could be very useful for core but also plugin/developers to improve polyglots UX.

Also, the command is extensively tested with Behat (with PHPUnit tests for some parts planned) and is already used in ​third-party projects.

With WP-CLI being an official WordPress project and this new command being superior to makepot.php in many ways, I think we have an opportunity to consolidate lots of things here, reduce the maintenance burden, and to make it easier for people to use WordPress I18N tools and to contribute to them.

It goes without saying that I'd love to help making this a reality, even if it means adding some new features to the command or fixing nasty bugs.

I'd suggest giving this command a go locally with some plugin, theme, or core to get a feeling for it.

Since we built it with WP-CLI 2.0 in mind, which should be released tomorrow, you might need to run the following commands to install it locally:

wp cli update --nightly
wp package install wp-cli/i18n-command

Alternatively, you can check out the ​GitHub repository and run it from within its directory (still requiring WP-CLI 2.0 though).

[schlessera]

Alternatively, you can check out the GitHub repository and run it from within its directory (still requiring WP-CLI 2.0 though).

Each command should work in a self-contained way, provided you use the WP-CLI version in vendor/bin after you did a composer install, so you don't need to install anything outside of the cloned wp-cli/i18n-command repository.

Example: vendor/bin/wp i18n make-pot --help

[schlessera]

The command is now part of the nightly builds, no need to install it manually. Just run the following command to make it available:

wp cli update --nightly

[herregroen]

Added a patch that will call this command instead of loading and using makepot.php.

It should be ensured that wp-cli has been updated on wordpress.org and this command is available.

[herregroen]

Updated the patch to also set the slug command.

I'm currently simply replacing the call to makepot.php with wp-cli. I think that's the easiest and best approach to make.

This patch only covers plugin imports, I've not yet been able to locate where exactly the calls to makepot.php happen for core.

These should also be switched to use wp i18n so we can successfully gather JS translations occurring there.

[herregroen]

Patch updated to handle POT generation for theme files as well. Using WP_CLI directly instead of exec as this is occurring during another wp-cli command.

[ocean90]

This patch only covers plugin imports, I've not yet been able to locate where exactly the calls to makepot.php happen for core.

• ​https://i18n.trac.wordpress.org/browser/tools/trunk/cron-svn-pots.php
• ​https://i18n.trac.wordpress.org/browser/tools/trunk/makepot.php

The code which calls cron-svn-pots.php is not open-sourced but it's bascially this:

$POT_GEN -m wp-frontend -c $CO/wordpress -p $CO/wp-pot -n wordpress.pot --min-version=3.7
$POT_GEN -m wp-admin -c $CO/wordpress -p $CO/wp-pot -n wordpress-admin.pot -f --min-version=3.7
$POT_GEN -m wp-network-admin -c $CO/wordpress -p $CO/wp-pot -n wordpress-admin-network.pot -f --min-version=3.7
$POT_GEN -m wp-tz -c $CO/wordpress -p $CO/wp-pot -n wordpress-continents-cities.pot -f --min-version=3.7

Thanks for the plugin/theme patches. For back-compat the --ignore-domain argument should be set.

[swissspidy]

3748.5.diff​ contains lots of unrelated code 🤔

As for the WordPress core string extraction, I have written down how these calls can be migrated to WP-CLI here: ​https://github.com/wp-cli/i18n-command/pull/69

As one can see, there are lots of exceptions for things like default themes, Hello Dolly, the list of continents and cities, etc.

In theory we don't have to use WP-CLI for all parts just yet, as we only care about JS right now. However, the WP-CLI command is a bit superior because it also finds strings that the current makepot.php script actually misses.

[ocean90]

@swissspidy Thanks for the link, I knew you posted them somewhere. I was able to use them in a first pass and it's looking good so far. Will do some more tests to have a version ready by end of this week.

[ocean90]

In 7792:

Translate: Add wporg-translate make-core-pot command to create POT files for WordPress core.

See #3748.

[ocean90]

In 7801:

Translate: Extend wporg-translate make-core-pot command to support generating POT files for WordPress 3.7+.

See #3748.

[ocean90]

In 7807:

Translate: Pass --skip-audit to i18n make-pot to disable audits.

See #3748.

[ocean90]

In 7808:

Translate: In wporg-translate make-core-pot enable JavaScript string extraction if supported.

See #3748.

[ocean90]

In 7809:

Translate: Exclude wp-includes/js/tinymce/* only for WordPress 4.3+.

See #3748.

[ocean90]

In 7810:

Translate: Remove duplicated exclude for wp-includes/js/tinymce/*.

See #3748.

[ocean90]

In 7833:

Translate: Add wporg-translate export command to export translation files (po, mo, and json) for WordPress core.

Props herregroen, ocean90.
See #3748.

[ocean90]

In 7843:

Translate: Use correct variable for locale slug.

See #3748.

[ocean90]

In 7893:

Plugin Directory: Use wp i18n make-pot for string extraction.

Props herregroen.
See #3748.

[ocean90]

In 7894:

Plugin Directory: Remove i18n-tools SVN external.

See #3748.

[ocean90]

In 7895:

Translate, Theme Directory: Use wp i18n make-pot for string extraction.

Props herregroen, ocean90.
See #3748.

[ocean90]

In 8640:

Translate: Extract JavaScript translations from admin files for WordPress 5.2+.

See #3748.