- attempt to fix#1598
- seems to work
- addresses won't be uploaded/announced anymore other than after connecting,
Later I need to find out how to announce them without causing problems, but
for the time disabling this seems an acceptable drawback
- addrthread is supposed to spread addresses as they appear. This was never
finished during migration to asyncore
- conservative to prevent flood and loops
- randomises order
- move protocol constants into a separate file
- move addr packet creation into a separate file
- see #1575
Logic borrowed from bitcoin, see CNetAddr::GetGroup() in src/netaddress.cpp
Simplified, so may not work fully identically but for our purposes it's good
enough. Won't connect to more than one host from a /16 subnet on IPv4 and a /32
subnet on IPv6.
and use logging without risk of circular import. Only subpackage
that imports from debug is bitmessageqt - because it also uses
debug.resetLogging().
Instead of from debug import logger is now recommended to use:
import logging
logger = logging.getLogger('default')
All subclasses of StoppableThread now have a logger attribute.
All threading related stuff except for set_thread_name()
was moved from helper_threading to network.threads.
Fixed two my mistakes from previous edit of debug in a1a8d3a:
- logger.handlers is not dict but iterable
- sys.excepthook should be set unconditionally
- expiration wasn't handled correctly
- objects with no child stems never expired. While this is better for
anonymity, it can cause objects getting stuck
- upon expiration the nodes weren't marked as not having the object, causing it
to not be advertised
- I haven't tested it but at least I don't think can make things worse
- instead of being processed in the ReceiveQueue thread, uploads are now done
in a dedicated thread. Only the parsing is done in ReceiveQueue thread.
- the UploadThread is modelled based on the DownloadThred, but simpler.
- it checks for intersection attack, eliminates duplicates and restricts the
write buffer size to 2MB (may still grow slightly higher if too many big
objects are requested, but the absolute limit appears to be about 4.5MB in the
worst case scenario).
- the restriction of the write buffer may cause some upload throttling (to
about 2MB per second per connection), but can be optimised later
- fixes#1414
- in corner cases, download request could have contained an incorrect
request length. I haven't actually checked if this can be triggered
though
- wait for downloading until anti intersection delay expires. Doesn't
necessarily mean that it will always avoid peer's anti intersection
delay, but it's close enough
- tracks last time an object was received. If it was too long time ago,
reset the download request queue. This avoid situations like when a
request gets ignored during the anti intersection delay, but it will
keep thinking there are still pending requests as long as not all
missing objects have been requested. This caused staggered download
(request 1000 items, wait 1 minute, request 1000 more, wait another
minute, ...)
- with these fixes, you should end up downloading as fast as your
network and CPU allow
- best tested with trustedpeer
- objectsKnownToThem is supposed to track if it's necessary to send inv
commands to a peer
- it is supposed to enter garbage collection after 1 hour
(ObjectTracker.trackingExpires)
- due to peer not announcing all objects, or announcing them before we
announce, this contains excessive number of entries after connection
is fully established.
- profiling revealed that this creates unnecessary memory to be kept
allocated
- this patch will prevent tracking of entries sent during bigInv,
reducing the memory "leak"
- it is possible, in theory, that this does have some negative effect,
like increased bandwidth or neglecting to announce some invs. It
probably doesn't though as my review of objectsKnownToThem occurrences
didn't reveal any such case, and since the dict didn't track fully
accurately anyway (so it would have already been broken if it was a
problem), I consider it an acceptable risk at the moment. If it indeed
causes problems, they can be solved separately
- I tested this on one of the bootstrap servers with little memory, and
it increased the number of connections than can be handled by a factor
of about 3.5
