Commit Graph

187 Commits

Author SHA1 Message Date
ca567acab3
Put uploads into a separate thread
- instead of being processed in the ReceiveQueue thread, uploads are now done
  in a dedicated thread. Only the parsing is done in ReceiveQueue thread.
- the UploadThread is modelled based on the DownloadThred, but simpler.
- it checks for intersection attack, eliminates duplicates and restricts the
  write buffer size to 2MB (may still grow slightly higher if too many big
  objects are requested, but the absolute limit appears to be about 4.5MB in the
  worst case scenario).
- the restriction of the write buffer may cause some upload throttling (to
  about 2MB per second per connection), but can be optimised later
- fixes #1414
2018-12-20 20:34:01 +01:00
e1d5604674
Fix #1408 (duplicate connections to some hosts), got from chan 2018-12-18 19:36:46 +02:00
38318150bd
Moved missingObjects dict from state to network.objectracker 2018-12-16 17:44:26 +02:00
coffeedogs
10128b7560
Changes based on style and lint checks. (final_code_quality_2) 2018-10-31 17:06:26 +00:00
coffeedogs
0bceeca524
Changes based on style and lint checks. (final_code_quality_5) 2018-10-31 14:04:35 +00:00
coffeedogs
c875769b15
Changes based on style and lint checks. (final_code_quality_3) 2018-10-31 12:58:46 +00:00
1c914d8cd8
flake8 and style correction for network.udp 2018-10-27 17:55:38 +03:00
49985a435f
Special class BMStringParser for sending ACK (resolves #1374) 2018-10-27 17:55:02 +03:00
daf556ca50
Added tests for knownnodes:
- import of pickled knownnodes;
  - default knownnodes if nothing imported;
  - knownnodes starvation (#1335), demanded changes in networkthread.
2018-10-22 17:18:24 +03:00
ef54bcaf82
The changes in #1325 are buggy. Implementing it properly. 2018-10-12 11:28:13 +03:00
b2c8b77938
Closes #1360: validate knownnodes received from the network 2018-10-12 11:16:57 +03:00
342e2a209c
Use DNS seed only if knownnodes are empty 2018-07-30 17:39:12 +03:00
coffeedogs
02f40f5036
Fixed: ERROR - Too many items in inv message! Closes 1274 2018-06-14 11:57:04 +01:00
coffeedogs
6c268ba22f
Fixed: Type error, reported from the bitmessage chan 2018-05-16 12:01:51 +01:00
ca7becb921
Download fixes
- in corner cases, download request could have contained an incorrect
  request length. I haven't actually checked if this can be triggered
  though
- wait for downloading until anti intersection delay expires. Doesn't
  necessarily mean that it will always avoid peer's anti intersection
  delay, but it's close enough
- tracks last time an object was received. If it was too long time ago,
  reset the download request queue. This avoid situations like when a
  request gets ignored during the anti intersection delay, but it will
  keep thinking there are still pending requests as long as not all
  missing objects have been requested. This caused staggered download
  (request 1000 items, wait 1 minute, request 1000 more, wait another
  minute, ...)
- with these fixes, you should end up downloading as fast as your
  network and CPU allow
- best tested with trustedpeer
2018-04-03 19:24:07 +02:00
0c9cb4824d
Memory leak (kind of) fix
- objectsKnownToThem is supposed to track if it's necessary to send inv
  commands to a peer
- it is supposed to enter garbage collection after 1 hour
  (ObjectTracker.trackingExpires)
- due to peer not announcing all objects, or announcing them before we
  announce, this contains excessive number of entries after connection
  is fully established.
- profiling revealed that this creates unnecessary memory to be kept
  allocated
- this patch will prevent tracking of entries sent during bigInv,
  reducing the memory "leak"
- it is possible, in theory, that this does have some negative effect,
  like increased bandwidth or neglecting to announce some invs. It
  probably doesn't though as my review of objectsKnownToThem occurrences
  didn't reveal any such case, and since the dict didn't track fully
  accurately anyway (so it would have already been broken if it was a
  problem), I consider it an acceptable risk at the moment. If it indeed
  causes problems, they can be solved separately
- I tested this on one of the bootstrap servers with little memory, and
  it increased the number of connections than can be handled by a factor
  of about 3.5
2018-04-01 11:39:02 +02:00
Mahendra
1a36361ef9
replace all random.xxx with helper random function 2018-03-21 19:26:27 +05:30
4b995e7996
Fix circular import
- previous commit broke
2018-03-13 11:42:43 +01:00
1e4400a207
Remove obsolete code
- PendingDownload and PendingUpload in inventory.py are obsolete
- the replacement functions only provide lengths and are in
  network/stats.py
2018-03-13 07:32:23 +01:00
0da0c10995
Socks5 parsing fix
- Fixes #1134
- Came over BM, thank you kind stranger
2018-02-26 20:04:57 +01:00
1184b23223
Separate protocol processing exceptions
- exceptions thrown by the state methods are separated from missing
  connections or states. This allows more accurate error reporting
2018-02-19 21:27:38 +01:00
757740369b
Add proxy authentication
- proxy module was missing initialisation of authentication
2018-02-18 20:53:16 +01:00
eb97face61
Show traceback in protocol parser error handler 2018-02-12 17:07:54 +01:00
066b419e16
Bugfixes
- typo in dandelion
- stealth ackdata fix for broadcasts and mailing lists
2018-02-09 00:49:08 +01:00
08bb85a952
Dandelion staticmethod fix 2018-02-07 17:22:26 +01:00
3d1fa473fb
Dandelion updates
- expiration now uses poisson distribution just like in the bitcoin
version
2018-02-06 22:28:56 +01:00
03f08b3cfd
Typo 2018-02-04 21:16:30 +01:00
a646ec4902
Allow separate proxy for onions
- new options in network section: onionsocksproxytype,
onionsockshostname and onionsocksport. These allow to separate
connectivity types for onion and non-onion addresses, e.g. connect to
clear nodes over clearnet and onions over tor
- also remove some obsolete imports
2018-02-04 21:03:54 +01:00
fd1a6c1fa1
Dandelion update
- dandelion fixes
- try to wait as long as possible before expiration if there are no
outbound connections
- expire in invThread rather than singleCleaner thread
- deduplication of code in inv and dinv command methods
- turn on by default, seems to work correctly now
- turn off dandelion if outbound connections are disabled
- start tracking downloads earlier, and faster download loop
- remove some obsolete lines
- minor PEP8 updates
2018-02-03 11:46:39 +01:00
053f434e04
Download fixes
- don't make empty requests
- don't make requests if all objects are pending already
2018-02-02 12:44:43 +01:00
57c8c7c07c
Download thread exception handling 2018-02-01 23:18:08 +01:00
8498143783
Download fixes
- check if already have object before requesting
2018-02-01 22:58:04 +01:00
290b87a49f
Download fixes
- don't expire too quickly
- ignore connections that haven't been fully established yet
2018-02-01 12:48:14 +01:00
c5dc7fc903
Typo 2018-02-01 12:26:54 +01:00
8d05742436
Typo 2018-02-01 12:20:41 +01:00
68b58ce0c5
Download optimisation
- new data structure to handle download tracking, uses less CPU
2018-02-01 12:19:39 +01:00
d223bfc6f2
Fix kqueue poller
- separate read and write filters
- make filters peristent for reduced syscall count
2018-01-31 22:25:23 +01:00
222e666a60
Network thread shutdown iterator fix
Thanks to @g1itch for pointing this out.
2018-01-31 21:09:36 +01:00
9e79386595
Set state to "close" and call handle_close() for all connections 2018-01-30 14:04:03 +02:00
d6df4470e1
No connection CPU hog fix
- the previous fix was incomplete, it shouldn't consume excessive
resources now when there are no connections
2018-01-23 15:59:58 +01:00
01c8f3b66d
Fix asyncore CPU usage on no connection
- if there are no connections, asyncore would hog CPU
- thanks to an anonymous contributor
2018-01-22 22:37:29 +01:00
ba91d21261
CPU hogging fix
- handle _command functions that don't return anything
- fix udp command function
2018-01-22 22:18:01 +01:00
c9851b9f41
Connection lookups invalid data handling
- shouldn't throw an exception if argument is a string rather than Peer
2018-01-02 22:23:03 +01:00
f74f82e54f
Start downloading earlier 2018-01-02 22:20:33 +01:00
4086253730
Bandwidth limit optimisation
- should be slightly more accurate and use slightly fewer resources
2018-01-02 15:24:47 +01:00
8788f2d349
Server full and duplicate handling
- will try to report "Server full" over protocol for 10 extra
connections over limit, instead of simply dropping them
- if connected to the same host inbound and outbound, handle as server
full (prevents duplicate connections)
2018-01-02 14:29:21 +01:00
36cc5b9cf5
Download optimisations
- don't make empty download requests
- use smaller chunks when they can be spread across multiple
connections
2018-01-01 12:51:35 +01:00
1864762a0a
Apply bandwidth limits without restart
- also minor style fixes
2018-01-01 12:49:08 +01:00
e9b1aa48a9
Protocol error handler fixes
- was broken if there was no error message in "raise"
- added default texts for network exceptions
2017-12-29 08:49:08 +01:00
02490e3286
Don't break if over 50k messages
- typo if there were over 50k messages in inventory caused PyBM to stall
2017-12-29 08:41:15 +01:00