Reliability issues with Travis theme download tests for WP-CLI

[schlessera]

We are having constant reliability issues with our functional tests in WP-CLI, because the Travis environment seems to get a lot of random error from the .org infrastructure.

Let me detail a specific example here.

In the functional tests for the wp-cli/extension-command (which includes plugin and theme commands), we have the following failure:

​https://travis-ci.org/wp-cli/automated-tests/jobs/414224296#L2018-L2049

Couldn't find 'twentytwelve' in the WordPress.org theme directory.

This is the message we are getting from the .org servers. The command to be tested was the following:

wp theme install --force twentytwelve --version=1.0

I never managed to get this to fail when running it manually. However, in our functional tests, it often fails. And the results are the same:

​https://travis-ci.org/wp-cli/automated-tests/jobs/414224296#L2170-L2201

​https://travis-ci.org/wp-cli/automated-tests/jobs/414224296#L2322-L2353

Given that the command is used and fails 3x for each run, this makes it 9 consecutive tries to download that theme that failed.

This makes me believe that there might be problems with the way Travis CI connects to the servers, and I would be grateful for any help at getting to the bottom of this.

Build system information for that environment can be found starting from here: ​https://travis-ci.org/wp-cli/automated-tests/jobs/414224296#L6

The IP addresses that Travis can use are the following:

147.75.192.163
34.233.56.198
34.234.4.53
52.3.55.28
52.45.185.117
52.45.220.64
52.54.31.11
52.54.40.118
54.208.31.17

[dd32]

@schlessera Can you please provide some more details, such as

• What API's you're using, be it URLs or core functions
• What the actual timeout you're experiencing is, and how long the timeouts are (Is it using the default WordPress 5s, something higher, etc)

The failures look like whatever code you're using is using a much too short timeout, in the context of a CLI task I'd be increasing the timeouts globally to something much higher than what WordPress uses for web contexts.

There's a higher-than-not possibility that the networks between Travis and W.org are congested, but we'd need to know what the timeouts are (and if a higher timeout helps the issue) to be able to pinpoint the issues.

[netweb]

Maybe try wrapping the wp-cli commands with the Travis CI function travis_retry if possible:

​https://docs.travis-ci.com/user/common-build-problems/#travis_retry

"For commands which do not have a built in retry feature, use the travis_retry function to retry it up three times if the return code is non-zero:"

travis_retry wp theme install --force twentytwelve --version=1.0