Commit Graph

77 Commits

Author SHA1 Message Date
11bec55be5
Don't put addresses into queue
- attempt to fix #1598
- seems to work
- addresses won't be uploaded/announced anymore other than after connecting,
  Later I need to find out how to announce them without causing problems, but
  for the time disabling this seems an acceptable drawback
2020-02-05 20:41:36 +08:00
lakshyacis
6139efc377
Imported packages sequencing and formatting 2 2020-01-27 14:43:25 +05:30
lakshyacis
f0bc74e658
Network fixes 2020-01-10 16:51:17 +05:30
a69732f060
Addrthread finish
- addrthread is supposed to spread addresses as they appear. This was never
  finished during migration to asyncore
- conservative to prevent flood and loops
- randomises order
- move protocol constants into a separate file
- move addr packet creation into a separate file
- see #1575
2019-11-30 13:47:24 +01:00
2a165380bb
Restrict outbound connections on network groups
Logic borrowed from bitcoin, see CNetAddr::GetGroup() in src/netaddress.cpp
Simplified, so may not work fully identically but for our purposes it's good
enough. Won't connect to more than one host from a /16 subnet on IPv4 and a /32
subnet on IPv6.
2019-11-18 12:20:29 +01:00
d6c1845b71
Moved Peer from state to network.node
and trustedPeer to network.connectionpool.BMConnectionPool attribute
2019-11-11 17:13:12 +02:00
7a1f803c92
network.BMConnectionPool: added shortcuts connections()
and establishedConnections(), some formatting fixes
2019-11-11 12:03:04 +02:00
d2a896697d
Used logger.isEnabledFor() to prevent unneeded calculations 2019-10-18 09:35:31 +03:00
7a89109fc9
New logging approach in order to reduce imports from submodules
and use logging without risk of circular import. Only subpackage
that imports from debug is bitmessageqt - because it also uses
debug.resetLogging().
Instead of from debug import logger is now recommended to use:

import logging

logger = logging.getLogger('default')

All subclasses of StoppableThread now have a logger attribute.
All threading related stuff except for set_thread_name()
was moved from helper_threading to network.threads.

Fixed two my mistakes from previous edit of debug in a1a8d3a:

 - logger.handlers is not dict but iterable
 - sys.excepthook should be set unconditionally
2019-10-18 09:35:24 +03:00
lakshyacis
5521c16478
bmproto pylint fixes 2019-08-30 16:12:39 +05:30
67d14f9e73
Docstrings in network.bmproto from #1362 2019-07-26 12:08:56 +03:00
996e71ae6f
flake8 in network package 2019-07-26 12:07:57 +03:00
b52cb9360f
Moved unused timeOffsetWrongCount from shared to BMProto 2019-02-04 22:23:28 +02:00
5b5ec2b63d
shared.connectedHostsList is obsolete 2019-01-04 16:17:01 +02:00
1cdbb90f22
Get rid of map 2018-12-20 20:34:09 +01:00
3a4eed0f8b
getdata performance optimisation
- reduce number of calls to time.time()
- no need to shuffle, that's done by the upload thread
- get rid of unused import
2018-12-20 20:34:06 +01:00
ca567acab3
Put uploads into a separate thread
- instead of being processed in the ReceiveQueue thread, uploads are now done
  in a dedicated thread. Only the parsing is done in ReceiveQueue thread.
- the UploadThread is modelled based on the DownloadThred, but simpler.
- it checks for intersection attack, eliminates duplicates and restricts the
  write buffer size to 2MB (may still grow slightly higher if too many big
  objects are requested, but the absolute limit appears to be about 4.5MB in the
  worst case scenario).
- the restriction of the write buffer may cause some upload throttling (to
  about 2MB per second per connection), but can be optimised later
- fixes #1414
2018-12-20 20:34:01 +01:00
38318150bd
Moved missingObjects dict from state to network.objectracker 2018-12-16 17:44:26 +02:00
49985a435f
Special class BMStringParser for sending ACK (resolves #1374) 2018-10-27 17:55:02 +03:00
b2c8b77938
Closes #1360: validate knownnodes received from the network 2018-10-12 11:16:57 +03:00
coffeedogs
02f40f5036
Fixed: ERROR - Too many items in inv message! Closes 1274 2018-06-14 11:57:04 +01:00
coffeedogs
6c268ba22f
Fixed: Type error, reported from the bitmessage chan 2018-05-16 12:01:51 +01:00
Mahendra
1a36361ef9
replace all random.xxx with helper random function 2018-03-21 19:26:27 +05:30
fd1a6c1fa1
Dandelion update
- dandelion fixes
- try to wait as long as possible before expiration if there are no
outbound connections
- expire in invThread rather than singleCleaner thread
- deduplication of code in inv and dinv command methods
- turn on by default, seems to work correctly now
- turn off dandelion if outbound connections are disabled
- start tracking downloads earlier, and faster download loop
- remove some obsolete lines
- minor PEP8 updates
2018-02-03 11:46:39 +01:00
68b58ce0c5
Download optimisation
- new data structure to handle download tracking, uses less CPU
2018-02-01 12:19:39 +01:00
8788f2d349
Server full and duplicate handling
- will try to report "Server full" over protocol for 10 extra
connections over limit, instead of simply dropping them
- if connected to the same host inbound and outbound, handle as server
full (prevents duplicate connections)
2018-01-02 14:29:21 +01:00
e9b1aa48a9
Protocol error handler fixes
- was broken if there was no error message in "raise"
- added default texts for network exceptions
2017-12-29 08:49:08 +01:00
3aa6f386db
Dandelion fixes
- dandelion would always think there is a cycle and trigger fluff
- cycle fluff trigger didn't correctly re-download and re-announce the
object. Now it remembers between (d)inv and object commands that it's
in a fluff trigger phase.
2017-11-18 09:47:17 +01:00
5a787f41d2
Socket closing changes
- explicit close only through asyncore error handler
- implicit close through garbage collector
- avoid duplicate closing
2017-11-17 13:37:51 +01:00
b025624f2a
missingObjects fix
- didn't notice valid objects arriving (only invalid)
2017-10-20 23:21:25 +02:00
6655e99aa3
Pending download stats optimisations
- tracks separately a global list for a faster sum. Needs slightly
more memory
2017-10-20 23:11:33 +02:00
a746ba9da7
Stop downloading objects with insufficient PoW
- object with insufficient PoW weren't correctly detected and it
tried to download them over and over again
2017-10-20 13:21:39 +02:00
2d34e73648
Dandelion updates
- fixes and feedback from @gfanti and @amiller
- addresses #1049
- minor refactoring
- two global child stems with fixed mapping between parent and
child stem
- allow child stems which don't support dandelion
- only allow outbound connections to be stems
- adjust stems if opening/closing outbound connections (should
allow partial dandelion functionality when not enough outbound
connections are available instead of breaking)
2017-10-20 01:21:49 +02:00
15857e6551
Asyncore updates
- reduce buffer size to 128kB (was 2MB)
- IP address handling use str instead of buffer (the latter, even
though it should be faster, breaks the code on Windows)
- read up to full buffer after fully established (otherwise
downloads become too slow due to the loop time). This reverts
a change made in d28a7bfb86
2017-10-20 01:07:30 +02:00
391d40d78b
Socket closing changes
- closing reason moved to a variable
- actual closing now done in asyncore thread instead of
receivedata thread
2017-10-19 09:08:05 +02:00
f785558ca5
Don't close UDP socket on bad packet magic 2017-10-19 09:00:02 +02:00
d44c6c6464
Forget known nodes with bad rating 2017-10-19 08:52:44 +02:00
7e0932815d
UDP socket closing fix
- invalid data or an incomplete read on UDP socket caused
it to close
2017-10-19 01:46:32 +02:00
4c9006a632
Asyncore performance optimisation
- use bytearray instead of strings for buffers
2017-10-16 08:07:32 +02:00
b1442ecb0a
Dandelion fixes and updates
- also, randomise the item order in an inv/dinv command
2017-09-30 13:42:04 +02:00
d574b167d8
Dandelion updates & fixes
- Addresses #1049
- Add dandelion routes for locally generated objects
- Minor bugfixes
- Send dinv commands on stem objects (instead of always sending inv
command)
2017-09-25 08:49:21 +02:00
6ce86b1d0a
Dandelion++ implementation
- untested, some functionality may be missing, don't turn on
- also, it randomises upload of requested objects
- affects #1049
2017-09-25 01:17:04 +02:00
660997b8f4
Code Quality 2017-08-22 13:49:27 +02:00
4f19c37fdc
Parser fix for multi-level arrays 2017-07-11 10:29:29 +02:00
db2d78c9b6
Make protocol decoder less recursive
- apparently, recursion has bad performance in Python, so the decoder is
now flat, except when parsing "version" command
2017-07-10 07:15:27 +02:00
853c8561ec
Per connection node id part 2
- forgot to include this in the previous commit
2017-07-10 07:12:52 +02:00
4fce01e34a
Less data transferred in invThread and addrThread 2017-07-08 18:02:47 +02:00
0f3a69adf4
Asyncore update: remove references to deleted files 2017-07-08 06:53:20 +02:00
ba4162d7fe
Asyncore update
- get rid of per-connection writeQueue/receiveQueue, and instead use
strings and locking
- minor code cleanup
- all state handlers now should set expectBytes
- almost all data processing happens in ReceiveDataThread, and
AsyncoreThread is almost only I/O (plus TLS). AsyncoreThread simply
puts the connection object into the queue when it has some data for
processing
- allow poll, epoll and kqueue handlers. kqueue is untested and
unoptimised, poll and epoll seem to work ok (linux)
- stack depth threshold handler  in decode_payload_content, this is
recursive and I think was causing occasional RuntimeErrors. Fixes #964
- longer asyncore loops, as now data is handled in ReceiveDataThread
- randomise node order when deciding what to download. Should prevent
retries being stuck to the same node
- socks cleanup (socks5 works ok, socks4a untested but should work too)
2017-07-06 19:45:36 +02:00
fe0664640e
Migrate antiIntersectionDelay to asyncore
- implemented by ignoring getdata during the delay rather than sleeping
as it was in the threaded model
- it can happen that a valid getdata request is received during the
delay. A node should be implemented in a way that retries to download,
that may not be the case with older PyBitmessage versions or other
implementations
2017-07-05 09:27:52 +02:00