I have a program that shares files between my laptop and my phone. It works well, except for apostrophes.
When I type an apostrophe ' on my laptop, it becomes ’ on my phone. And when I type 's on my phone, it becomes 痴 on my laptop.
Apparently the phone turns the apostrophe (U+0027) into a right single quote (U+2019), then bungles bytes in the UTF-8 encoding of U+2019 as three Windows-1252 characters. The bytes E28099hex are interpreted as â (E2hex), € (80hex), and ™ (99hex).
When I type 's on my phone, it is encoded as two Windows-1252 characters 92hex and 73hex. Then by the time the text appears on my laptop, the bytes 9273hex are interpreted as a Shift-JIS encoding of the CJK character 痴 (U+75F4).
Here’s Python code to reproduce the problem.
def mojibake(s: str, n: int, bad_encoding: str = 'cp1252') -> str:
for _ in range(n):
s = s.encode('utf-8').decode(bad_encoding, errors='replace')
return s
print(mojibake("’s", 1))
print(mojibake("’s", 1, 'shift_jis'))
The code won’t corrupt ASCII text. The problem started with an ASCII character being replaced by a similar non-ASCII character. Text containing a non-ASCII character gets more corrupted with each round.