- expiration wasn't handled correctly
- objects with no child stems never expired. While this is better for
anonymity, it can cause objects getting stuck
- upon expiration the nodes weren't marked as not having the object, causing it
to not be advertised
- I haven't tested it but at least I don't think can make things worse
- instead of being processed in the ReceiveQueue thread, uploads are now done
in a dedicated thread. Only the parsing is done in ReceiveQueue thread.
- the UploadThread is modelled based on the DownloadThred, but simpler.
- it checks for intersection attack, eliminates duplicates and restricts the
write buffer size to 2MB (may still grow slightly higher if too many big
objects are requested, but the absolute limit appears to be about 4.5MB in the
worst case scenario).
- the restriction of the write buffer may cause some upload throttling (to
about 2MB per second per connection), but can be optimised later
- fixes#1414
- in corner cases, download request could have contained an incorrect
request length. I haven't actually checked if this can be triggered
though
- wait for downloading until anti intersection delay expires. Doesn't
necessarily mean that it will always avoid peer's anti intersection
delay, but it's close enough
- tracks last time an object was received. If it was too long time ago,
reset the download request queue. This avoid situations like when a
request gets ignored during the anti intersection delay, but it will
keep thinking there are still pending requests as long as not all
missing objects have been requested. This caused staggered download
(request 1000 items, wait 1 minute, request 1000 more, wait another
minute, ...)
- with these fixes, you should end up downloading as fast as your
network and CPU allow
- best tested with trustedpeer
- objectsKnownToThem is supposed to track if it's necessary to send inv
commands to a peer
- it is supposed to enter garbage collection after 1 hour
(ObjectTracker.trackingExpires)
- due to peer not announcing all objects, or announcing them before we
announce, this contains excessive number of entries after connection
is fully established.
- profiling revealed that this creates unnecessary memory to be kept
allocated
- this patch will prevent tracking of entries sent during bigInv,
reducing the memory "leak"
- it is possible, in theory, that this does have some negative effect,
like increased bandwidth or neglecting to announce some invs. It
probably doesn't though as my review of objectsKnownToThem occurrences
didn't reveal any such case, and since the dict didn't track fully
accurately anyway (so it would have already been broken if it was a
problem), I consider it an acceptable risk at the moment. If it indeed
causes problems, they can be solved separately
- I tested this on one of the bootstrap servers with little memory, and
it increased the number of connections than can be handled by a factor
of about 3.5
- new options in network section: onionsocksproxytype,
onionsockshostname and onionsocksport. These allow to separate
connectivity types for onion and non-onion addresses, e.g. connect to
clear nodes over clearnet and onions over tor
- also remove some obsolete imports
- dandelion fixes
- try to wait as long as possible before expiration if there are no
outbound connections
- expire in invThread rather than singleCleaner thread
- deduplication of code in inv and dinv command methods
- turn on by default, seems to work correctly now
- turn off dandelion if outbound connections are disabled
- start tracking downloads earlier, and faster download loop
- remove some obsolete lines
- minor PEP8 updates
- will try to report "Server full" over protocol for 10 extra
connections over limit, instead of simply dropping them
- if connected to the same host inbound and outbound, handle as server
full (prevents duplicate connections)
- I thought this is done automatically through garbage collection, but I
think as the channel is still assigned in the asyncore map, it needs
to be done manually. Basically filehandle limit exceeded and it
crashed
- dandelion would always think there is a cycle and trigger fluff
- cycle fluff trigger didn't correctly re-download and re-announce the
object. Now it remembers between (d)inv and object commands that it's
in a fluff trigger phase.
- fixes and feedback from @gfanti and @amiller
- addresses #1049
- minor refactoring
- two global child stems with fixed mapping between parent and
child stem
- allow child stems which don't support dandelion
- only allow outbound connections to be stems
- adjust stems if opening/closing outbound connections (should
allow partial dandelion functionality when not enough outbound
connections are available instead of breaking)
- reduce buffer size to 128kB (was 2MB)
- IP address handling use str instead of buffer (the latter, even
though it should be faster, breaks the code on Windows)
- read up to full buffer after fully established (otherwise
downloads become too slow due to the loop time). This reverts
a change made in d28a7bfb86
- more exception handling
- only use outbound connections for stems
(thanks to @amillter for info)
- don't create stems if config disabled
- addresses #1049
- allow loopback addresses, now you can bind different loopback IP
addresses on a single system and they will auto-cross-connect
- always listen for discovery on 0.0.0.0
- [network] - bind now also applies for the TCP socket as well as UDP
socket
- closing socket iterator fix
- get rid of per-connection writeQueue/receiveQueue, and instead use
strings and locking
- minor code cleanup
- all state handlers now should set expectBytes
- almost all data processing happens in ReceiveDataThread, and
AsyncoreThread is almost only I/O (plus TLS). AsyncoreThread simply
puts the connection object into the queue when it has some data for
processing
- allow poll, epoll and kqueue handlers. kqueue is untested and
unoptimised, poll and epoll seem to work ok (linux)
- stack depth threshold handler in decode_payload_content, this is
recursive and I think was causing occasional RuntimeErrors. Fixes#964
- longer asyncore loops, as now data is handled in ReceiveDataThread
- randomise node order when deciding what to download. Should prevent
retries being stuck to the same node
- socks cleanup (socks5 works ok, socks4a untested but should work too)
- implemented by ignoring getdata during the delay rather than sleeping
as it was in the threaded model
- it can happen that a valid getdata request is received during the
delay. A node should be implemented in a way that retries to download,
that may not be the case with older PyBitmessage versions or other
implementations
- now tracks downloads globally too, so it doesn't request the same
object from multiple peers at the same time
- retries at the earliest every minute
- stops trying to download an object after an hour
- minor fixes in retrying downloading invalid objects
- outbound peers now have a rating
- it's also shown in the network status tab
- currently it's between -1 to +1, changes by 0.1 steps and uses a
hyperbolic function 0.05/(1.0 - rating) to convert rating to
probability with which we should connect to that node when randomly
chosen
- it increases when we successfully establish a full outbound connection
to a node, and decreases when we fail to do that
- onion nodes have priority when using SOCKS
- this thread is for spreading new/updated addresses in active
connections, analogous to the InvThread
- it doesn't do anything yet, this is just a dummy queue at the moment
