- Original Poster
- #1
We all love to see Windows desktop computers, and their Microsoft Word installs of varying version and model year, where text gets copied and then pasted into online entry forms, like WordPress posts/page builders.
I've never really been bothered by the semantic correctness of using left doublequotes and right double quotes, when the old ANSI doublequote would do both right and left, though not at the funky cockeyed angles. I've never personally considered using left and right single quotes when an apostrophe would fit all three occasions equally well.
Conveniently, Microsoft has taken the ISO-8859-1 standard for western european characters and 'extended' it (not my words). Where the ISO standard has reserved 20 spots for control characters, Microsoft has snagged those spots and made available several other non-standard characters, which has been dubbed CP1252.
If your computer, database server, or other bit of software is fluent in ISO-8859-1, but isn't built on Windows, or doesn't know to give special care and attention, it's liable to be fed a diet of Microsoft CP1252 characters, while being told that they're ISO-8859-1 (latin1).
.. insert years of garbage data in, mixed with good d.ata...
... it's time to modernise and migrate that system to UTF8, for compatibility sake, amongst other reasons...
Run just about any of the convenient one line commands to convert from one ISO-8859-1 to UTF8 and you will be dismayed to find posts truncated when you convert to UTF8.
Yes that lovely right single quotation mark is like a hatchet that cuts through, removing everything thereafter, ouch! It happens with many routines which don't take into account the fact that the script ignores anything it doesn't recognise (like those 20 characters), or inserts the U+0092 as if it were part of the standard.
------------------------------------------------------------------------------
The above is the backdrop story to some pain du jour. Just wondering what approach others are taking, before I go piece together something myself. LAMP/WAMP with mysql is the environment. Ultimately, we're moving to UTF8MB4, but for purposes of this discussion, it's an 8859-1 to UTF8 migration.
It looks like a sanitising by transliteration, or simply replacing the Windows extended 'bad version' of the single right quote with the ISO-8859-1 compliant 'acceptable/universal' version of it, we would have sorted 95% of the issues.
For the remaining 5% of issues, one approach that seems to work is the converting to binary first, then converting to the UTF8 flavour. Tons of pitfalls with index and field size limits, but it's one way of making the data agnostic to the character set / collation.
-----------------------------------------------------------------------------------
Any suggested alternatives would be appreciated!
Cheers
I've never really been bothered by the semantic correctness of using left doublequotes and right double quotes, when the old ANSI doublequote would do both right and left, though not at the funky cockeyed angles. I've never personally considered using left and right single quotes when an apostrophe would fit all three occasions equally well.
Conveniently, Microsoft has taken the ISO-8859-1 standard for western european characters and 'extended' it (not my words). Where the ISO standard has reserved 20 spots for control characters, Microsoft has snagged those spots and made available several other non-standard characters, which has been dubbed CP1252.
If your computer, database server, or other bit of software is fluent in ISO-8859-1, but isn't built on Windows, or doesn't know to give special care and attention, it's liable to be fed a diet of Microsoft CP1252 characters, while being told that they're ISO-8859-1 (latin1).
.. insert years of garbage data in, mixed with good d.ata...
... it's time to modernise and migrate that system to UTF8, for compatibility sake, amongst other reasons...
Run just about any of the convenient one line commands to convert from one ISO-8859-1 to UTF8 and you will be dismayed to find posts truncated when you convert to UTF8.
Yes that lovely right single quotation mark is like a hatchet that cuts through, removing everything thereafter, ouch! It happens with many routines which don't take into account the fact that the script ignores anything it doesn't recognise (like those 20 characters), or inserts the U+0092 as if it were part of the standard.
------------------------------------------------------------------------------
The above is the backdrop story to some pain du jour. Just wondering what approach others are taking, before I go piece together something myself. LAMP/WAMP with mysql is the environment. Ultimately, we're moving to UTF8MB4, but for purposes of this discussion, it's an 8859-1 to UTF8 migration.
It looks like a sanitising by transliteration, or simply replacing the Windows extended 'bad version' of the single right quote with the ISO-8859-1 compliant 'acceptable/universal' version of it, we would have sorted 95% of the issues.
For the remaining 5% of issues, one approach that seems to work is the converting to binary first, then converting to the UTF8 flavour. Tons of pitfalls with index and field size limits, but it's one way of making the data agnostic to the character set / collation.
-----------------------------------------------------------------------------------
Any suggested alternatives would be appreciated!
Cheers
