utf8 issue #691

New Issue

2014-07-26T10:59:18+02:00

DarkRedman commented

2014-07-26 10:59:18 +02:00

(Migrated from github.com)

Since the last update (few days ago) since I run Bitmessage I got this error, so the program continue to run and load message but this error don't let the GUI starts unfortunately.

sock.sendall error: [Errno 32] Relais brisé (pipe)
Traceback (most recent call last):
  File "bitmessagemain.py", line 260, in <module>
    mainprogram.start()
  File "bitmessagemain.py", line 227, in start
    bitmessageqt.run()
  File "/home/darkredman/PyBitmessage/src/bitmessageqt/__init__.py", line 3833, in run
    myapp = MyForm()
  File "/home/darkredman/PyBitmessage/src/bitmessageqt/__init__.py", line 534, in __init__
    self.loadInbox()
  File "/home/darkredman/PyBitmessage/src/bitmessageqt/__init__.py", line 971, in loadInbox
    'bitmessagesettings', 'timeformat'), localtime(int(received))), 'utf-8'))
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfb in position 11: invalid start byte

Since the last update (few days ago) since I run Bitmessage I got this error, so the program continue to run and load message but this error don't let the GUI starts unfortunately. ``` python sock.sendall error: [Errno 32] Relais brisé (pipe) Traceback (most recent call last): File "bitmessagemain.py", line 260, in <module> mainprogram.start() File "bitmessagemain.py", line 227, in start bitmessageqt.run() File "/home/darkredman/PyBitmessage/src/bitmessageqt/__init__.py", line 3833, in run myapp = MyForm() File "/home/darkredman/PyBitmessage/src/bitmessageqt/__init__.py", line 534, in __init__ self.loadInbox() File "/home/darkredman/PyBitmessage/src/bitmessageqt/__init__.py", line 971, in loadInbox 'bitmessagesettings', 'timeformat'), localtime(int(received))), 'utf-8')) UnicodeDecodeError: 'utf8' codec can't decode byte 0xfb in position 11: invalid start byte ```

bmng-dev commented

2014-07-28 08:58:41 +02:00

(Migrated from github.com)

There has been no change to lines 970 and 971 for at least 8 months. From the socket error message at the top I'm guessing it is a locale issue.

The documentation for time.strftime says this:

strftime() returns a locale depedent byte string; the result may be converted to unicode by doing strftime(<myformat>).decode(locale.getlocale()[1])

Unfortunately there are a more than a few lines that need this correction.

There has been no change to lines 970 and 971 for at least 8 months. From the socket error message at the top I'm guessing it is a locale issue. The documentation for [time.strftime](https://docs.python.org/2/library/time.html#time.strftime) says this: > `strftime()` returns a locale depedent byte string; the result may be converted to unicode by doing `strftime(<myformat>).decode(locale.getlocale()[1]`) Unfortunately there are a more than a few lines that need this correction.

DarkRedman commented

2014-07-30 19:06:04 +02:00

(Migrated from github.com)

Okay, it's a bit weird since my locale is fr-FR.UTF8 but I'll try to fix the code on my own.
Thanks for the explanation.

Okay, it's a bit weird since my locale is fr-FR.UTF8 but I'll try to fix the code on my own. Thanks for the explanation.

bmng-dev commented

2014-07-31 08:22:29 +02:00

(Migrated from github.com)

I have been working on this but it is a little more complicated to get a fix working in all circumstances. The current code seems to indicate that the locale module could cause an exception simply for importing it. locale.getlocale() may return None as the language and/or encoding. On OSX in some cases locale.getlocale() and locale.getdefaultlocale() can cause an exception, which makes me wary of locale.getpreferredencoding().
Even though your locale specifies an encoding of UTF-8, time.strftime() is not actually returning UTF-8 encoded strings (as evidenced by the error) but ISO8859-1 encoded strings. I think is because Python relies on older non-unicode aware C functions and it is difficult to determine what encoding they use (due to the locale issues mentioned above).

Now all that said, an easy fix is to avoid using the day and month names when formatting timestamps. Just change the timeformat setting in keys.dat using the directives specified in the documentation for time.strftime.

I have been working on this but it is a little more complicated to get a fix working in all circumstances. The current code seems to indicate that the locale module could cause an exception simply for importing it. `locale.getlocale()` may return None as the language and/or encoding. On OSX in some cases `locale.getlocale()` and `locale.getdefaultlocale()` can cause an exception, which makes me wary of `locale.getpreferredencoding()`. Even though your locale specifies an encoding of UTF-8, `time.strftime()` is not actually returning UTF-8 encoded strings (as evidenced by the error) but ISO8859-1 encoded strings. I think is because Python relies on older non-unicode aware C functions and it is difficult to determine what encoding they use (due to the locale issues mentioned above). Now all that said, an easy fix is to avoid using the day and month names when formatting timestamps. Just change the timeformat setting in keys.dat using the directives specified in the documentation for [time.strftime](https://docs.python.org/2/library/time.html#time.strftime).

bmng-dev commented

2014-08-03 17:29:57 +02:00

(Migrated from github.com)

I have arrived at two simple solutions:

The first solution, guaranteed to work but potentially annoy users, is to hardcode the datetime format as %Y-%m-%d %H:%M:%S.
The second solution, which requires some testing to ensure it works, is to call locale.setlocale(locale.LC_ALL, '') (see documentation for locale.setlocale and the locale module Background, details, hints, tips and caveats) in bitmessagemain.py and use locale.getlocale()[1] as recommended. The OSX issue is therefore avoided; locale.getdefaultlocale() was used due locale.getlocale() returning (None, None) because locale.setlocale() wasn't called as above.

Which solution is preferred?

I have arrived at two simple solutions: - The first solution, guaranteed to work but potentially annoy users, is to hardcode the datetime format as `%Y-%m-%d %H:%M:%S`. - The second solution, which requires some testing to ensure it works, is to call `locale.setlocale(locale.LC_ALL, '')` (see documentation for [locale.setlocale](https://docs.python.org/2/library/locale.html#locale.setlocale) and the locale module [Background, details, hints, tips and caveats](https://docs.python.org/2/library/locale.html#background-details-hints-tips-and-caveats)) in bitmessagemain.py and use `locale.getlocale()[1]` as recommended. The OSX issue is therefore avoided; `locale.getdefaultlocale()` was used due `locale.getlocale()` returning `(None, None)` because `locale.setlocale()` wasn't called as above. Which solution is preferred?

This repo is archived. You cannot comment on issues.