Encoding issues in Trac

[SergeyBiryukov]

Looks like moving to MySQL in #253 introduced some encoding issues.

See weird characters in ​wp19156:2 (also note the owner name in that ticket), ​wp21366:3, ​wp25183:6, ​wp25669:7.

Summarized in a screenshot. They were all previously displayed correctly. There are probably more, these are just the ones I could find at a glance.

Happy to help with debugging or data conversion if needed.

[SergeyBiryukov]

I was able to reproduce the same characters by treating UTF-8 data as ISO 8859-1.

[SergeyBiryukov]

You can use ​http://2cyr.com/decode/ to get an idea of what the readable text should look like. Autodetect option works for me, but there are still some invalid sequences (displayed as question marks) due to non-printable characters not being copied correctly.

[nacin]

Yeah, ocean90 also reported this to me. So the good thing is, the DB is correct. Querying the DB directly for any of these gives me proper results. So this appears to be either a connection character set issue or some encoding issue within Trac. I'll dig into this tomorrow and will provide a DB dump of some affected tickets.

[nacin]

Barry figured out what the issue is here (double-encoding). We'll be able to fix this pretty easily; just working on a script for it.

[nacin]

Note to self (when I fix this, soon) - make sure ​https://core.trac.wordpress.org/ticket/14647 works.

[nacin]

Replying to nacin:

Note to self (when I fix this, soon) - make sure ​https://core.trac.wordpress.org/ticket/14647 works.

Ah, this was unrelated. Trac got re-compiled when I applied the core patch from #127. This upgraded Genshi (Trac's templating engine) to 0.7, which we had previously downgraded to 0.6.1 for this exact reason.

[nacin]

I fixed all of these. Hat-tip to the convert(cast(convert(field using latin1) as binary) using utf8) trick. Was careful to not run it on any post-migration fields as that would cause problems.