- fixes and feedback from @gfanti and @amiller
- addresses #1049
- minor refactoring
- two global child stems with fixed mapping between parent and
child stem
- allow child stems which don't support dandelion
- only allow outbound connections to be stems
- adjust stems if opening/closing outbound connections (should
allow partial dandelion functionality when not enough outbound
connections are available instead of breaking)
- reduce buffer size to 128kB (was 2MB)
- IP address handling use str instead of buffer (the latter, even
though it should be faster, breaks the code on Windows)
- read up to full buffer after fully established (otherwise
downloads become too slow due to the loop time). This reverts
a change made in d28a7bfb86
- more exception handling
- only use outbound connections for stems
(thanks to @amillter for info)
- don't create stems if config disabled
- addresses #1049
the desktop sound theme, with pycanberra for example. Plugin name should
start with 'theme' in that case, whereas the name of plugins playing the
sound file starts with 'file'.
- don't treat "@" in label as an email address
- ask for confirmation before autoregistering. It confused some
newbies into thinking that bitmessage requires payment
- calling "shutdown" now cleanly shuts down PyBitmessage, however the
call may not return so you need to add an error handler to the call.
With python for example, wrap the "shutdown()" in
"try:/except socket.error"
- allow loopback addresses, now you can bind different loopback IP
addresses on a single system and they will auto-cross-connect
- always listen for discovery on 0.0.0.0
- [network] - bind now also applies for the TCP socket as well as UDP
socket
- closing socket iterator fix
- get rid of per-connection writeQueue/receiveQueue, and instead use
strings and locking
- minor code cleanup
- all state handlers now should set expectBytes
- almost all data processing happens in ReceiveDataThread, and
AsyncoreThread is almost only I/O (plus TLS). AsyncoreThread simply
puts the connection object into the queue when it has some data for
processing
- allow poll, epoll and kqueue handlers. kqueue is untested and
unoptimised, poll and epoll seem to work ok (linux)
- stack depth threshold handler in decode_payload_content, this is
recursive and I think was causing occasional RuntimeErrors. Fixes#964
- longer asyncore loops, as now data is handled in ReceiveDataThread
- randomise node order when deciding what to download. Should prevent
retries being stuck to the same node
- socks cleanup (socks5 works ok, socks4a untested but should work too)
- implemented by ignoring getdata during the delay rather than sleeping
as it was in the threaded model
- it can happen that a valid getdata request is received during the
delay. A node should be implemented in a way that retries to download,
that may not be the case with older PyBitmessage versions or other
implementations
- now tracks downloads globally too, so it doesn't request the same
object from multiple peers at the same time
- retries at the earliest every minute
- stops trying to download an object after an hour
- minor fixes in retrying downloading invalid objects
- outbound peers now have a rating
- it's also shown in the network status tab
- currently it's between -1 to +1, changes by 0.1 steps and uses a
hyperbolic function 0.05/(1.0 - rating) to convert rating to
probability with which we should connect to that node when randomly
chosen
- it increases when we successfully establish a full outbound connection
to a node, and decreases when we fail to do that
- onion nodes have priority when using SOCKS
- this thread is for spreading new/updated addresses in active
connections, analogous to the InvThread
- it doesn't do anything yet, this is just a dummy queue at the moment
- not used yet, just an inactive helper function
- I received feedback that OpenSSL.rand isn't more secure than
os.urandom. I read several debates/analyses about it and concur
- should prevent the same object being re-requested indefinitely
- locking for object tracking
- move SSL-specific error handling to TLSDispatcher
- observe maximum connection limit when accepting a new connection
- stack depth test (for debugging purposes)
- separate download thread
- connection pool init moved to main thread
- update to 6044df5adf
- objects that are expired or in wrong stream are not re-requested
anymore, even if they aren't stored in the inventory
- the previous option "acceptmismatch" now only affects whether such
objects are stored in the inventory
- a new config file option, network/acceptmismatch, allows the inventory
to store objects that expired or are from a stream we're not
interested in. Having this on will prevent re-requesting objects that
other nodes incorrectly advertise. It defaults to false
- better handling of WSA* checks on non-windows systems
- handle EBADF on Windows/select
- better timeouts / loop lengths in main asyncore loop and
spawning new connections
- remove InvThread prints
- asyncore is now on by default
- inv announcements implemented
- bandwidth limit implemented / fixed
- stats on download / upload speed now work
- make prints into logger
- limit knownNodes to 20k as it was before
- green light fixed
- other minor fixes
- bm headers and commands are only read up to expected length.
On a very fast connection (e.g. local VM), reading verack
also read a part of the TLS handshake
- some debugging info moved from print to logger.debug
- tls handshake cleanup
- bugfixes
- UDP socket for local peer discovery
- new function assembleAddr to unify creating address command
- open port checker functionality (inactive)
- sendBigInv is done in a thread separate from the network IO
thread
- separate queue for processing blocking stuff on reception
- rewrote write buffer as a queue
- some addr handling
- number of half open connections correct
- Network status UI works but current speed isn't implemented yet
- Track per connection and global transferred bytes
- Add locking to write queue so that other threads can put stuff
there
- send ping on timeout (instead of closing the connection)
- implement open port checker (untested, never triggered yet)
- error handling on IO
- most of the stuff is done so it partially works
- disabled pollers other than select (debugging necessary)
- can switch in the settings, section network, option asyncore (defaults
to False)
- bmconfigpaser.py now allows to put default values for a specific
option in the file
- addresses as sections are now detected by "BM-" rather than
just ignoring bitmessagesettings. There can now be other sections
with a cleaner config file
- in some cases when IPv6 stack is available and onionbindip is an IPv4
address, socket.bind doesn't change the bound address, ending up
listening on everything
- immediately return from initCL() if numpy or pyopencl is unevailable
(no ImportError because of resetPoW() call)
- use glob to find C extension even if it named like
`bitmsghash.x86_64-linux-gnu.so`
If user chooses to show the Settings dialog:
- activate the "Network Settings" tab
- remove option 'dontconnect' if settings have been saved
- track pending hashId more accurately
- add timeout and a cleanup so that the download queues don't
get stuck and memory is freed
- randomise download order (only works for inv commands with
more than 1 entry)
- replace PendingDownload singleton dict with a Queue
- total memory and CPU requirements should be reduced
- get rid of somObjectsOfWhichThisRemoteNodeIsAlearedyAware. It has very
little practicle effect and only uses memory
- if too many nodes, only delete oldest nodes in bootstrap provider
mode, in normal mode ignore new nodes as it used to before
- in bootstrap provider mode, penalise nodes announced by others by 1
day instead of 3 hours
- version command struct for faster unpacking
- increase read buffer to 2MB to allow a full command to fit
- initial bitmessage protocol class (WIP)
- error handling
- remove duplicate method
- finished proxy design
- socks4a and socks5 implemented
- authentication not tested
- resolver for both socks4a and socks5
- http client example using the proxy
- if knownNodes grows to 20000, instead of ignoring new nodes, forget
the 1000 oldest ones
- drop connection after sendaddr if too many connections, even if it's
an outbound one
- if maximum total connections are lower than maximum outbound
connections, active bootstrap provider mode
- in this mode, check all addresses received before announcing them
- so basically it only annouces those addresses it successfully
connected to
- I can't get the dynamic loading to work on OSX in frozen mode
- I think that if someone wants to build a frozen executable with custom
messagetypes modules, he can edit the file
- so now it lists the existing types manually (for frozen mode only)
- maxtotalconnections = maximum number of total full connections
(incoming + outgoing) the node will allow. Default 200 as it was.
- maxbootstrapconnections = number of additional (to total) connection
that will act in bootstrap mode, closing after sending the list of
addresses. Default 20 as it was.
- maxaddrperstreamsend = initial address list maximum size, per
participating stream. Default 500. Child streams get half. The
response is chunked into pieces of max. 1000 addresses as that's the
protocol limit.
- on OpenBSD, you can't have a socket that supports both IPv4 and IPv6.
This allows handling for this error, and then it will try IPv4 only,
just like for other similar errors.
- there were reports of errors in FreeBSD (I could only reproduce some)
and Gentoo without IPv4 support (I don't have a VM for testing ready)
- adds an exception handler for double task_done in case sender thread
has to close prematurely (I saw this triggered on FreeBSD 11)
- listening socket opening error handler was broken (triggered if you
can't open a socket with both IPv4 and IPv6 support)
- error handler for socket.accept. Reported on FreeBSD 10.3
- fixes#854
- TTL to chans shouldn't be too low so the UI gives a feedback
- warning when sending wouldn either require a lot of refactoring or
wouldn't have good usability
- don't do subprocess in SafeHTMLParser, it doesn't work in frozen mode
and an attempt to fix it would take too much refactoring and I'm not
even sure it would work
- instead, make it handle broken unicode correctly
- I think the previous reports of freezes were caused by trying to
interpret data as unicode, causing a crash
- it does about 1MB/s on my machine, so a timeout is not a big problem
- spec file for pyinstaller detects architecture (32 or 64bit)
- spec file uses os.path.join
- spec file creates and adds the list of messagetypes
- added MinGW/MSyS support in Makefile
- separate Makefile.msvc for MCVC
- bitmsghash.cpp minor adjustments to build also on MSVC/MinGW
- if frozen mode, messagetypes loads the list of files from a text file
generated during archive building rather than from a directory
- separate Makefile for BSD make
- auto-compile will detect BSD and pass the correct parameters to make
- C PoW builds on OpenBSD and detects number of cores
- "new" folder consistently appears in chans and "All accounts"
- "Sent" message list sorting fix
- When editing a label, keys.dat is saved and the lineEdit completer
is updated
- addressbook is updated when adding/deleting a new chan
- saveKnownNodes replaced the repeated pickle.dump
- with knownNodesLock instead of acquire/release
- outgoingSynSender had an unnecessary loop during shutdown causing
excessive CPU usage / GUI freezing
- networkDefaultProofOfWorkNonceTrialsPerByte and
networkDefaultPayloadLengthExtraBytes cyclic import fix
- PyBitmessage should launch now when there's no keys.dat
- rearranged code to reduce cyclic dependencies
- doCleanShutdown is separated in shutdown.py
- shared queues are separated in queues.py
- some default values were moved to defaults.py
- knownnodes partially moved to knownnodes.py
- complete the version and SSL handshake first, and only then feed
errors into the stream and close connection
- this allows more accurate error handling on both sides
- also the timeOffset error trigger is now more accurate, but requires
more nodes to upgrade
- version command sends list of all participating streams
- biginv sends lists of hosts for all streams the peer wants (plus
immediate children)
- objects will spread to all peers that advertise the associated stream
- please note these are just network subsystem adjustments, streams
aren't actually usable yet
- queues were too short
- some error handling was missing
- remove nonblocking repeats in receive data thread
- singleCleaner shouldn't wait unnecessarily
- sendinv and sendaddress are sometimes being sent to connections that
haven't been established yet, resulting in complaints about stream
mismatch. The error should only be displayed once the connection has
been established and the remote node provides its stream number
- Missing renamed to PendingDownload
- PendingDownload now only retries 3 times rather than 6 to dowload an
object
- Added PendingUpload, replacing invQueueSize
- PendingUpload has both the "len" method (number of objects not
uploaded) as well as "progress" method, which is a float from 0
(nothing done) to 1 (all uploaded) which considers not only objects
but also how many nodes they are uploaded to
- PendingUpload tracks when the object is successfully uploaded to the
remote node instead of just adding an arbitrary time after they have
been send the corresponding "inv"
- Network status tab's "Objects to be synced" shows the sum of
PendingUpload and PendingDownload sizes
- sometimes a node would send an "inv" about an object but then not
provide it when requested. This could be that it expired in the
meantime or it was an attack or a bug. This patch will forget that the
object exists if was requested too many times and not received.
- remember what was requested from which node
- remember if it was received
- re-request object if we haven't received any new object for more than
a minute
- rely on dict quasi-random order instead of an additional shuffle
- request an object once per minute
- stop check after count objects have been found
- tries to avoid calling senddata it it would block receiveDataThread,
allowing fore more asynchronous operation
- request objects in chunks of 100 (CPU performance optimisation)
- moved logic into a Missing singleton
- shouldn't try to download duplicates anymore, only requests a hash
once every 5 minutes and not from the same host
- removed obsoleted variables
- the "Objects to be synced" in the Network tab should now be correct
- removed some checks which aren't necessary anymore in my opinion
- fix missing self in Throttle (thanks landscape.io)
- send buffer to send multiple commands in one TCP packet
- recv/send operation size now based on bandwith limit
- send queue limited to 100 entries
- buffer getdata commands to fill send queue, instead of waiting for the
data packet to arrive first (i.e. allow getdata to work asynchronously)
- SSL handshake would often fail, because verack packet was being sent
at the same time as the do_handshake was executed in a different
thread. This makes it so that do_handshake waits until verack is done
sending.
- also minor modifications in SSLContext initialisation
- fixes errors introduced in the earlier refactoring
- more variables moved to state.py
- path finding functions moved to paths.py
- remembers IPv6 network unreachable (in the future can be used to skip
IPv6 for a while)
- got rid of shared config parser and made it into a singleton
- refactored safeConfigGetBoolean as a method of the config singleton
- refactored safeConfigGet as a method of the config singleton
- moved softwareVersion from shared.py into version.py
- moved some global variables from shared.py into state.py
- moved some protocol-specific functions from shared.py into protocol.py
- minor refactoring, made it into singleton instead of a shared global
variable. This makes it a little bit cleaner and moves the class into
a separate file
- removed duplicate inventory locking
- renamed singleton.py to singleinstance.py (this is the code that
ensures only one instance of PyBitmessage runs at the same time)
- TLS handshake in python is apparently always asynchronous, so it needs
proper handling of SSLWantReadError and SSLWantWriteError
- also adds a timeout and a proper shutdown if handshake fails
- Linux users often don't know that the C PoW is available and complain
it's slow. This will try to build it, and adds availability
notification in the status bar
- also, the updateStatusBar signal now allows emphasised notifications,
which will remain visible for a longer period of time and also
reappear if a status change happened in the meantime
- sometimes SSL connections unnecessarily disconnected on non-fatal
errors. This should fix that. This is however a short term solution
because of migrating to asyncore which has its own error handling
- when you have multiple OpenCL drivers at the same time, e.g. intel and
nvidia, they won't mix leading to crashes. This patch makes it
possible to select which driver to use by listing the available
vendors
- refactored to use the .ui file
- input logic change, address is always optional
- interactive input validation
- runs asynchronously to the main window
- address generator thread can now validate chans in addition to just
adding them
- a user report indicated there is confusion about address error
messages. He/she thought it refers to the sender address, however it
refers to the recipient address. This makes it more clear
- if your time is off by more than an hour, you won't be able to
establish a connection to the network. This patch adds a UI
notification so that the user can understand why he can't connect.
- this has been tested on Windows as well, and has been cleaned up.
There is now a permanent parser thread, and it restarts when the
parsing takes more than 1 second
- Fixes#900
- while 448ceaa74c fixed slow rendering on
windows, there was still a bug where overly long messages caused
freezeing of the hyperlink regexp parser, which appears to happen on
all platforms. Maybe it's a freeze, maybe it just takes too long. This
patch aborts the regexp parser after 1 second and simply displays the
message without hyperlinks being clickable. This doesn't affect HTML
mode because there the links are kept as they are
- Fixes#900
- some messages (e.g. some long messages on Windows, or binary data)
cause an excessive amount of time in rendering the body. This
change is base on a workaround I found at
http://www.qtcentre.org/threads/8188-bug-setLineWrapMode
- most status messages are transient, so they are now only displayed for
10 seconds
- when trying to quit while disconnected or not fully synced, a
three-choice message box now appears: Yes for waiting, No for
closing anyway, and Cancel for aborting the shutdown procedure
- this copyright character has been plaguing pylupdate4 parser and
multiple unsuccessful attempts have been made and then reverted.
Replaced with a HTML entity, hopefully this will finally fix it.
- UI will now display notifications in the status bar if the connection
to the proxy itself is broken. This should give better feedback to
people who are unfamiliar with tor and misconfigured it
- The proxy error handling in the background was slightly improved as
well
- fixes "fast python" (multiprocessing) PoW
- python PoW (both slow and fast) interruptible on *NIX
- signal handler should handle multiple processes and threads correctly
(only tested on Linux)
- popul window asking whether to interrupt PoW when quitting QT GUI
- PoW status in "sent" folder fixes and now also displays broadcast
status which didn't exist before
- Fixes#894
- namecoin connection errors have now severity "info" instead of
"error", because it just confuses peopel who don't have namecoin
configured
- partially addresses #893
- namecoin lookup now also includes name of the record in the recipient
field
- namecoin lookups now support multiple semicolon-separated
recipients like the other recipient-related functions. If there are
multiple recipients, namecoin lookup will look up the last entry on
the line, for example if you have "a; b; c" in the recipient line,
it will lookup "c"
- bitmessage could end up having no known nodes and then it would
freeze. Now it shouldn't freeze, however it can still end up without
known nodes until a restart in some cases (e.g. when suspending the
computer for more then 3 days while BM is running)
- you can now use SMTP to send messages
- uses bmaddr.lan domain
- runs on 127.0.0.8425 if you set "smtpd" to True
- mandatory authentication with smtpdusername and smtpdpassword
- handles old dialog versions better if using curses
- can spawn SMTP delivery thread if configured (only when in daemon
mode)
- daemonized mode now works more like it's properly supposed to on unix
(double fork etc). You may have to adjust your init scripts, when
when using upstart for example you should now use "expect daemon"
- daemon mode now cleanly shuts down when TERM/INT signal is received
- PyBitmessage only used to quit on disk full when running in daemon
mode. When this happened with the QT-GUI, it would end up in a
half-frozen status instead. Quitting is a safer choice
Fixes#572
- helper classes for encoding/decoding messages
- includes both old as well as new extended one (msgpack+zlib)
- the classes are unused yet and are supposed to be for experimenting
- when running a hidden service, the IP of the tor relay was a part of
the verack message. In setups where it's not 127.0.0.1 it may leak
info about network topology
- thanks for an anonymous bug report
- will send the correct combination of hostname and port
- if proxyhostname is a hostname and an IP address, it will now allow
multiple parallel connections for hidden service
- PyBitmessage can now run as a hidden service on Tor
- three new variables in keys.dat: onionhostname, onionport, onionbindip
- you need to manually add a hidden service to tor
- bitmsghash should now build and run on BSD (thanks for
FreeBSD/Dragonfly maintainers for assistance)
- if it cannot detect the number of cores, will default to one thread
(previously it broke)
Two file merge conflicts, __init__.py and upnp.py, were not resolved
correctly by the automatic resolving (probably because the affected code
was written by other people and I merged them into mailchuck fork). This
changes it to the same code that is in the mailchuck fork)
On Windows, the encoding was always the default windows encoding and
didn't change when you use a language in BM that required a different
encoding. This affected mainly date & time in the received column and
the startup info on the network status tab.
The plural/paucal form support was not compatible with pylupdate4, it
didn't correctly parse the 3-argument calls to translate.
This fixes it, and updates the sources accordingly.
Some parts of strings did not use the proper locale. For example, date
and time strings was always output with the US locale. This fixes it.
There are still some cases where localisation is not implemented, and
could be changed from str(string) to locale.str(string).
- it shows that it needs to wait for PoW to finish
- it waits a bit for new objects to be distributed
- it displays a better progress indicator in the status bar
Previously, people who don't understand how PyBitmessage works sometimes
shut it down immediately after they wrote a message. This would have
caused the message to be stuck in the queue locally and not sent. Now,
it will indicate that the PoW still needs to work, and it will wait a
bit longer so that the message can spread. It's not a completely correct
approach, because it does not know whether the message was really
retrieved after the "inv" notification was sent.
Now only in status bar and no more popup window.
Previously, it only showed once until a restart, but now it shows every
time it detects a new version online. Since it does not show a popup
window it's not a big deal.
The language combo box is generated dynamically from the list of
available translations in the translations directory. This allows the
users to add their own translations without having to change the code.
Added a RetranslateMixin. Since PyQT does not support automated language
changes of UI files (like the C++ QT does), this implements something
similar. It assumes that the UI file has the same name as the class, but
lowercase.
Added RetraslateMixin to the new blacklist and networkstatus interfaces.
- UPnP handles errors better
- it tries to bind external interface (previously sometimes it searched
on 127.0.0.1 resulting in no routers being detected)
The quick navigation key disrupted keybindings with keyboard modifiers,
like Ctrl-C for copy. This restricts the quick navigation only where no
keyboard modifiers are active.
Fixes#184
- delete key now works when message body is focused as well
- N for next message (down)
- P for previous message (up)
- R for reply
- C for compose
- F for find
- Find is now dynamic if the search text is least 3 characters long
Fixes Bitmessage#655
Addresses #155
Rows are deleted from a message list in multiple places, and this is an
attempt to refactor it so that it is done in one function. It's not used
anywhere yet.
MessageView does not currently load external resources (QTextBrowser by
default interprets all external resources as local file names and tries
to load them like that. This can, in the future, be implemented. For
example, if SOCKS (Tor) is used, the resource could be loaded through
the SOCKS too.
This commit is a skeleton for it that does not actually do anything and
can be filled with an implementation that does the loading.
Email addresses and URIs are now clickable when viewing a message in
plain text mode. Clicking an email address moves to the Send tab, while
clicking an URI has the same result as clicking an URI in html mode, it
will ask for confirmation before opening it in external handler.
HTML parser wasn't correctly handling img tags.
Now it also by defaults disabled external schemas to prevent
deanonymisation (even though the renderer actually doesn't support
external schemas at the moment)
Addresses #178
Attackers injected node addresses with port 0 into the network. Port 0
is unusable on many OSes and can't be listened on. PyBitmessage won't
accept nodes that have port 0 anymore.
- postpone initial sleep until the first getdata is received
- also sleep when received a getdata request for an object that hasn't
been advertised to the other node yet
There was a report that by quickly asking a large number of nodes if
they have an ACK object (which the attacker knows but it is injected
into the network by the recipient of the message), it can estimate how
an object propagates through the network, and eventually pinpoint an
originating IP address of the injection, i.e. the IP address of the
message recipient.
This patch mitigates against it by stalling when asked for a nonexisting
object (so that the attacker can't spam requests), and also upon
connection before sending its own inventory list (so that reconnecting
won't help the attacker). It estimates how long a short message takes to
propagate through the network based on how many nodes are in a stream
and bases the stalling time on that. Currently that is about 15 seconds.
Initial connection delay takes into account the time that already passed
since the connection was established.
This basically gives the attacker one shot per a combination of his own
nodes and the nodes he can connect to, and thus makes the attack much
more difficult to succeed.
Editing of blacklist labels affected the rerendering, because it emits
the changed signal too, and it caused an exception because the address
field was missing at that time. This works around both.
In some situations, it's not necessary to send an ACK. For example, when
the sender is blacklisted, when the message has no content, or when the
address has ACK sending disabled.
Also it's not necessary to rebroadcast empty messages into a mailing
list.
When advertising nodes and when establishing connections, private IP
range checks were not done. This could cause private IPs to be
advertised across the network. Also, some of the checks weren't
IPv6-aware.
Fixes Bitmessage#768
If you change, add or remove an item in a treewidget or addressbook,
messagelists will now autoupdate labels, and sender comboboxes will also
update if applicable.
Fixes#69
Message compose sometimes misinterpreted quoted message as HTML, causing
rendering screwups. Since we don't support HTML composing, we will treat
all quoted messages as plain text.
Zooming in message body view / compose works in single steps
irrespective of wheel sensitivity, and info about zoom level is
displayed in percent rather than font pixel size.
if singleWorker crashed, the thread couldn't be joined. This both makes
it so that it doesn't crash, as well as reorders the shutdown sequence
so that it is less likely to be triggered.
Fixes Bitmessage#549
- it didn't shutdown correctly
- it didn't handle exception correctly (however, if I understand
correctly, this will never be triggered if using blocking get, so it
doesn't affect PyBitmessage)
- flushing size check changed from 1 to 0 (I don't know why it was 1)
Flood mitigation was done both in the ObjectProcessorQueue as well as
receiveData threads. This patch removes the mitigation in receiveData
threads and cleans up the one in the ObjectProcessorQueue
- subject on replies was not parsed correctly (or should I say was
parsed more than necessary)
- unicode can handle invalid characters without needing a special
function
Fixes#164
The previous commit broke places where UserData was read from the
mesagelist line, because it's now a string rather than a QVariant. This
actually simplifies things.
