Commit Graph

77 Commits

Author SHA1 Message Date
Don't put addresses into queue
- attempt to fix #1598
- seems to work
- addresses won't be uploaded/announced anymore other than after connecting,
  Later I need to find out how to announce them without causing problems, but
  for the time disabling this seems an acceptable drawback
2020-02-05 20:41:36 +08:00
Imported packages sequencing and formatting 2 2020-01-27 14:43:25 +05:30
Network fixes 2020-01-10 16:51:17 +05:30
Addrthread finish
- addrthread is supposed to spread addresses as they appear. This was never
  finished during migration to asyncore
- conservative to prevent flood and loops
- randomises order
- move protocol constants into a separate file
- move addr packet creation into a separate file
- see #1575
2019-11-30 13:47:24 +01:00
Restrict outbound connections on network groups
Logic borrowed from bitcoin, see CNetAddr::GetGroup() in src/netaddress.cpp
Simplified, so may not work fully identically but for our purposes it's good
enough. Won't connect to more than one host from a /16 subnet on IPv4 and a /32
subnet on IPv6.
2019-11-18 12:20:29 +01:00
Moved Peer from state to network.node
and trustedPeer to network.connectionpool.BMConnectionPool attribute
2019-11-11 17:13:12 +02:00
network.BMConnectionPool: added shortcuts connections()
and establishedConnections(), some formatting fixes
2019-11-11 12:03:04 +02:00
Used logger.isEnabledFor() to prevent unneeded calculations 2019-10-18 09:35:31 +03:00
New logging approach in order to reduce imports from submodules
and use logging without risk of circular import. Only subpackage
that imports from debug is bitmessageqt - because it also uses
Instead of from debug import logger is now recommended to use:

import logging

logger = logging.getLogger('default')

All subclasses of StoppableThread now have a logger attribute.
All threading related stuff except for set_thread_name()
was moved from helper_threading to network.threads.

Fixed two my mistakes from previous edit of debug in a1a8d3a:

 - logger.handlers is not dict but iterable
 - sys.excepthook should be set unconditionally
2019-10-18 09:35:24 +03:00
bmproto pylint fixes 2019-08-30 16:12:39 +05:30
Docstrings in network.bmproto from #1362 2019-07-26 12:08:56 +03:00
flake8 in network package 2019-07-26 12:07:57 +03:00
Moved unused timeOffsetWrongCount from shared to BMProto 2019-02-04 22:23:28 +02:00
shared.connectedHostsList is obsolete 2019-01-04 16:17:01 +02:00
Get rid of map 2018-12-20 20:34:09 +01:00
getdata performance optimisation
- reduce number of calls to time.time()
- no need to shuffle, that's done by the upload thread
- get rid of unused import
2018-12-20 20:34:06 +01:00
Put uploads into a separate thread
- instead of being processed in the ReceiveQueue thread, uploads are now done
  in a dedicated thread. Only the parsing is done in ReceiveQueue thread.
- the UploadThread is modelled based on the DownloadThred, but simpler.
- it checks for intersection attack, eliminates duplicates and restricts the
  write buffer size to 2MB (may still grow slightly higher if too many big
  objects are requested, but the absolute limit appears to be about 4.5MB in the
  worst case scenario).
- the restriction of the write buffer may cause some upload throttling (to
  about 2MB per second per connection), but can be optimised later
- fixes #1414
2018-12-20 20:34:01 +01:00
Moved missingObjects dict from state to network.objectracker 2018-12-16 17:44:26 +02:00
Special class BMStringParser for sending ACK (resolves #1374) 2018-10-27 17:55:02 +03:00
Closes #1360: validate knownnodes received from the network 2018-10-12 11:16:57 +03:00
Fixed: ERROR - Too many items in inv message! Closes 1274 2018-06-14 11:57:04 +01:00
Fixed: Type error, reported from the bitmessage chan 2018-05-16 12:01:51 +01:00
replace all with helper random function 2018-03-21 19:26:27 +05:30
Dandelion update
- dandelion fixes
- try to wait as long as possible before expiration if there are no
outbound connections
- expire in invThread rather than singleCleaner thread
- deduplication of code in inv and dinv command methods
- turn on by default, seems to work correctly now
- turn off dandelion if outbound connections are disabled
- start tracking downloads earlier, and faster download loop
- remove some obsolete lines
- minor PEP8 updates
2018-02-03 11:46:39 +01:00
Download optimisation
- new data structure to handle download tracking, uses less CPU
2018-02-01 12:19:39 +01:00
Server full and duplicate handling
- will try to report "Server full" over protocol for 10 extra
connections over limit, instead of simply dropping them
- if connected to the same host inbound and outbound, handle as server
full (prevents duplicate connections)
2018-01-02 14:29:21 +01:00
Protocol error handler fixes
- was broken if there was no error message in "raise"
- added default texts for network exceptions
2017-12-29 08:49:08 +01:00
Dandelion fixes
- dandelion would always think there is a cycle and trigger fluff
- cycle fluff trigger didn't correctly re-download and re-announce the
object. Now it remembers between (d)inv and object commands that it's
in a fluff trigger phase.
2017-11-18 09:47:17 +01:00
Socket closing changes
- explicit close only through asyncore error handler
- implicit close through garbage collector
- avoid duplicate closing
2017-11-17 13:37:51 +01:00
missingObjects fix
- didn't notice valid objects arriving (only invalid)
2017-10-20 23:21:25 +02:00
Pending download stats optimisations
- tracks separately a global list for a faster sum. Needs slightly
more memory
2017-10-20 23:11:33 +02:00
Stop downloading objects with insufficient PoW
- object with insufficient PoW weren't correctly detected and it
tried to download them over and over again
2017-10-20 13:21:39 +02:00
Dandelion updates
- fixes and feedback from @gfanti and @amiller
- addresses #1049
- minor refactoring
- two global child stems with fixed mapping between parent and
child stem
- allow child stems which don't support dandelion
- only allow outbound connections to be stems
- adjust stems if opening/closing outbound connections (should
allow partial dandelion functionality when not enough outbound
connections are available instead of breaking)
2017-10-20 01:21:49 +02:00
Asyncore updates
- reduce buffer size to 128kB (was 2MB)
- IP address handling use str instead of buffer (the latter, even
though it should be faster, breaks the code on Windows)
- read up to full buffer after fully established (otherwise
downloads become too slow due to the loop time). This reverts
a change made in d28a7bfb86
2017-10-20 01:07:30 +02:00
Socket closing changes
- closing reason moved to a variable
- actual closing now done in asyncore thread instead of
receivedata thread
2017-10-19 09:08:05 +02:00
Don't close UDP socket on bad packet magic 2017-10-19 09:00:02 +02:00
Forget known nodes with bad rating 2017-10-19 08:52:44 +02:00
UDP socket closing fix
- invalid data or an incomplete read on UDP socket caused
it to close
2017-10-19 01:46:32 +02:00
Asyncore performance optimisation
- use bytearray instead of strings for buffers
2017-10-16 08:07:32 +02:00
Dandelion fixes and updates
- also, randomise the item order in an inv/dinv command
2017-09-30 13:42:04 +02:00
Dandelion updates & fixes
- Addresses #1049
- Add dandelion routes for locally generated objects
- Minor bugfixes
- Send dinv commands on stem objects (instead of always sending inv
2017-09-25 08:49:21 +02:00
Dandelion++ implementation
- untested, some functionality may be missing, don't turn on
- also, it randomises upload of requested objects
- affects #1049
2017-09-25 01:17:04 +02:00
Code Quality 2017-08-22 13:49:27 +02:00
Parser fix for multi-level arrays 2017-07-11 10:29:29 +02:00
Make protocol decoder less recursive
- apparently, recursion has bad performance in Python, so the decoder is
now flat, except when parsing "version" command
2017-07-10 07:15:27 +02:00
Per connection node id part 2
- forgot to include this in the previous commit
2017-07-10 07:12:52 +02:00
Less data transferred in invThread and addrThread 2017-07-08 18:02:47 +02:00
Asyncore update: remove references to deleted files 2017-07-08 06:53:20 +02:00
Asyncore update
- get rid of per-connection writeQueue/receiveQueue, and instead use
strings and locking
- minor code cleanup
- all state handlers now should set expectBytes
- almost all data processing happens in ReceiveDataThread, and
AsyncoreThread is almost only I/O (plus TLS). AsyncoreThread simply
puts the connection object into the queue when it has some data for
- allow poll, epoll and kqueue handlers. kqueue is untested and
unoptimised, poll and epoll seem to work ok (linux)
- stack depth threshold handler  in decode_payload_content, this is
recursive and I think was causing occasional RuntimeErrors. Fixes #964
- longer asyncore loops, as now data is handled in ReceiveDataThread
- randomise node order when deciding what to download. Should prevent
retries being stuck to the same node
- socks cleanup (socks5 works ok, socks4a untested but should work too)
2017-07-06 19:45:36 +02:00
Migrate antiIntersectionDelay to asyncore
- implemented by ignoring getdata during the delay rather than sleeping
as it was in the threaded model
- it can happen that a valid getdata request is received during the
delay. A node should be implemented in a way that retries to download,
that may not be the case with older PyBitmessage versions or other
2017-07-05 09:27:52 +02:00