Commit Graph

173 Commits

Author SHA1 Message Date
ca7becb921
Download fixes
- in corner cases, download request could have contained an incorrect
  request length. I haven't actually checked if this can be triggered
  though
- wait for downloading until anti intersection delay expires. Doesn't
  necessarily mean that it will always avoid peer's anti intersection
  delay, but it's close enough
- tracks last time an object was received. If it was too long time ago,
  reset the download request queue. This avoid situations like when a
  request gets ignored during the anti intersection delay, but it will
  keep thinking there are still pending requests as long as not all
  missing objects have been requested. This caused staggered download
  (request 1000 items, wait 1 minute, request 1000 more, wait another
  minute, ...)
- with these fixes, you should end up downloading as fast as your
  network and CPU allow
- best tested with trustedpeer
2018-04-03 19:24:07 +02:00
0c9cb4824d
Memory leak (kind of) fix
- objectsKnownToThem is supposed to track if it's necessary to send inv
  commands to a peer
- it is supposed to enter garbage collection after 1 hour
  (ObjectTracker.trackingExpires)
- due to peer not announcing all objects, or announcing them before we
  announce, this contains excessive number of entries after connection
  is fully established.
- profiling revealed that this creates unnecessary memory to be kept
  allocated
- this patch will prevent tracking of entries sent during bigInv,
  reducing the memory "leak"
- it is possible, in theory, that this does have some negative effect,
  like increased bandwidth or neglecting to announce some invs. It
  probably doesn't though as my review of objectsKnownToThem occurrences
  didn't reveal any such case, and since the dict didn't track fully
  accurately anyway (so it would have already been broken if it was a
  problem), I consider it an acceptable risk at the moment. If it indeed
  causes problems, they can be solved separately
- I tested this on one of the bootstrap servers with little memory, and
  it increased the number of connections than can be handled by a factor
  of about 3.5
2018-04-01 11:39:02 +02:00
Mahendra
1a36361ef9
replace all random.xxx with helper random function 2018-03-21 19:26:27 +05:30
4b995e7996
Fix circular import
- previous commit broke
2018-03-13 11:42:43 +01:00
1e4400a207
Remove obsolete code
- PendingDownload and PendingUpload in inventory.py are obsolete
- the replacement functions only provide lengths and are in
  network/stats.py
2018-03-13 07:32:23 +01:00
0da0c10995
Socks5 parsing fix
- Fixes #1134
- Came over BM, thank you kind stranger
2018-02-26 20:04:57 +01:00
1184b23223
Separate protocol processing exceptions
- exceptions thrown by the state methods are separated from missing
  connections or states. This allows more accurate error reporting
2018-02-19 21:27:38 +01:00
757740369b
Add proxy authentication
- proxy module was missing initialisation of authentication
2018-02-18 20:53:16 +01:00
eb97face61
Show traceback in protocol parser error handler 2018-02-12 17:07:54 +01:00
066b419e16
Bugfixes
- typo in dandelion
- stealth ackdata fix for broadcasts and mailing lists
2018-02-09 00:49:08 +01:00
08bb85a952
Dandelion staticmethod fix 2018-02-07 17:22:26 +01:00
3d1fa473fb
Dandelion updates
- expiration now uses poisson distribution just like in the bitcoin
version
2018-02-06 22:28:56 +01:00
03f08b3cfd
Typo 2018-02-04 21:16:30 +01:00
a646ec4902
Allow separate proxy for onions
- new options in network section: onionsocksproxytype,
onionsockshostname and onionsocksport. These allow to separate
connectivity types for onion and non-onion addresses, e.g. connect to
clear nodes over clearnet and onions over tor
- also remove some obsolete imports
2018-02-04 21:03:54 +01:00
fd1a6c1fa1
Dandelion update
- dandelion fixes
- try to wait as long as possible before expiration if there are no
outbound connections
- expire in invThread rather than singleCleaner thread
- deduplication of code in inv and dinv command methods
- turn on by default, seems to work correctly now
- turn off dandelion if outbound connections are disabled
- start tracking downloads earlier, and faster download loop
- remove some obsolete lines
- minor PEP8 updates
2018-02-03 11:46:39 +01:00
053f434e04
Download fixes
- don't make empty requests
- don't make requests if all objects are pending already
2018-02-02 12:44:43 +01:00
57c8c7c07c
Download thread exception handling 2018-02-01 23:18:08 +01:00
8498143783
Download fixes
- check if already have object before requesting
2018-02-01 22:58:04 +01:00
290b87a49f
Download fixes
- don't expire too quickly
- ignore connections that haven't been fully established yet
2018-02-01 12:48:14 +01:00
c5dc7fc903
Typo 2018-02-01 12:26:54 +01:00
8d05742436
Typo 2018-02-01 12:20:41 +01:00
68b58ce0c5
Download optimisation
- new data structure to handle download tracking, uses less CPU
2018-02-01 12:19:39 +01:00
d223bfc6f2
Fix kqueue poller
- separate read and write filters
- make filters peristent for reduced syscall count
2018-01-31 22:25:23 +01:00
222e666a60
Network thread shutdown iterator fix
Thanks to @g1itch for pointing this out.
2018-01-31 21:09:36 +01:00
9e79386595
Set state to "close" and call handle_close() for all connections 2018-01-30 14:04:03 +02:00
d6df4470e1
No connection CPU hog fix
- the previous fix was incomplete, it shouldn't consume excessive
resources now when there are no connections
2018-01-23 15:59:58 +01:00
01c8f3b66d
Fix asyncore CPU usage on no connection
- if there are no connections, asyncore would hog CPU
- thanks to an anonymous contributor
2018-01-22 22:37:29 +01:00
ba91d21261
CPU hogging fix
- handle _command functions that don't return anything
- fix udp command function
2018-01-22 22:18:01 +01:00
c9851b9f41
Connection lookups invalid data handling
- shouldn't throw an exception if argument is a string rather than Peer
2018-01-02 22:23:03 +01:00
f74f82e54f
Start downloading earlier 2018-01-02 22:20:33 +01:00
4086253730
Bandwidth limit optimisation
- should be slightly more accurate and use slightly fewer resources
2018-01-02 15:24:47 +01:00
8788f2d349
Server full and duplicate handling
- will try to report "Server full" over protocol for 10 extra
connections over limit, instead of simply dropping them
- if connected to the same host inbound and outbound, handle as server
full (prevents duplicate connections)
2018-01-02 14:29:21 +01:00
36cc5b9cf5
Download optimisations
- don't make empty download requests
- use smaller chunks when they can be spread across multiple
connections
2018-01-01 12:51:35 +01:00
1864762a0a
Apply bandwidth limits without restart
- also minor style fixes
2018-01-01 12:49:08 +01:00
e9b1aa48a9
Protocol error handler fixes
- was broken if there was no error message in "raise"
- added default texts for network exceptions
2017-12-29 08:49:08 +01:00
02490e3286
Don't break if over 50k messages
- typo if there were over 50k messages in inventory caused PyBM to stall
2017-12-29 08:41:15 +01:00
a3398d6a17
Don't crash download thread if no connections 2017-12-02 00:50:58 +01:00
41ead2bfb5
Distribute downloads more evenly
- also increases expireation of missing objects from 10 minutes to an
hour
2017-12-02 00:48:08 +01:00
e558b1fb72
Error handling
- proxy connections didn't init fullyEstablished
2017-11-19 13:48:43 +01:00
fdfbb77ed2
Close filehandles on connection reaping
- I thought this is done automatically through garbage collection, but I
think as the channel is still assigned in the asyncore map, it needs
to be done manually. Basically filehandle limit exceeded and it
crashed
2017-11-19 00:05:55 +01:00
5e042b76e7
Typo
- missing brackets
2017-11-18 10:05:41 +01:00
3aa6f386db
Dandelion fixes
- dandelion would always think there is a cycle and trigger fluff
- cycle fluff trigger didn't correctly re-download and re-announce the
object. Now it remembers between (d)inv and object commands that it's
in a fluff trigger phase.
2017-11-18 09:47:17 +01:00
4690dd6f00
Copy object contents from buffers on instantiation
- this may fix some memory issues
2017-11-17 23:53:46 +01:00
3c3d69e5de
Reap closed connection fix 2017-11-17 19:50:39 +01:00
1e02d2b48a
Download optimisation
- pending download tracking now per-connection instead of globally
2017-11-17 19:49:51 +01:00
5a787f41d2
Socket closing changes
- explicit close only through asyncore error handler
- implicit close through garbage collector
- avoid duplicate closing
2017-11-17 13:37:51 +01:00
1792289e00
close handling fix
- don't close a connection twice
2017-11-14 23:43:05 +01:00
4b40d4bce1
Download thread error handling 2017-10-22 15:28:30 +02:00
75a6f605c1
Download optimisation
- more accurate tracking
- randomise download order
- longer cycle
2017-10-22 11:32:37 +02:00
b025624f2a
missingObjects fix
- didn't notice valid objects arriving (only invalid)
2017-10-20 23:21:25 +02:00