Commit Graph

266 Commits

Author SHA1 Message Date
lakshyacis
2c71612a4b
http pylint fixes 2019-09-10 15:34:19 +05:30
lakshyacis
7eced454d1
http flake8 fixes 2019-09-03 19:56:59 +05:30
lakshyacis
68e09a5e30
http-old pylint fixes 2019-09-03 18:41:24 +05:30
lakshyacis
5fcb7fc05e
http-old flake8 fixes 2019-09-03 14:34:17 +05:30
lakshyacis
d8ea0afe40
downloadthread.py Pylint fixes 2019-09-02 19:15:15 +05:30
lakshyacis
8589f01d3d
downloadthread.py flake8 fixes 2019-09-02 19:02:50 +05:30
lakshyacis
b927d51eb3
dandelion.py pylint fixes 2019-09-02 13:53:06 +05:30
lakshyacis
f1b6b16940
connectionpool.py pylint fixes 2019-09-02 13:52:56 +05:30
lakshyacis
5521c16478
bmproto pylint fixes 2019-08-30 16:12:39 +05:30
lakshyacis
469d289a97
Announcethread Pylint Fixes 2019-08-30 15:23:20 +05:30
lakshyacis
548fcf0d0a
announcethread Flake8 Fixes 2019-08-29 19:24:13 +05:30
f180b1a5b0
Moved randomtrackingdict to network, it's used only there 2019-08-18 17:34:58 +03:00
ad2a2b3fb4
Inherit helper_threading.StoppableThread from threading.Thread
and do random.seed() in its __init__
2019-08-06 18:07:19 +03:00
27e74a15d7
flake8: network.connectionchooser 2019-07-29 15:54:19 +03:00
67d14f9e73
Docstrings in network.bmproto from #1362 2019-07-26 12:08:56 +03:00
bbab0010e6
Moved a test script from network.tcp to tests.core 2019-07-26 12:08:42 +03:00
391b5ded87
Docstrings in network from #1368 2019-07-26 12:08:32 +03:00
2998599442
Docstrings in network from #1401 2019-07-26 12:08:18 +03:00
19ab56a979
network.connectionpool: get any setting with .safeGet.. 2019-07-26 12:08:18 +03:00
996e71ae6f
flake8 in network package 2019-07-26 12:07:57 +03:00
0eb4f46923
Formatting based on review 2019-07-11 15:32:00 +02:00
465a276c02
Dandelion fixes
- expiration wasn't handled correctly
- objects with no child stems never expired. While this is better for
  anonymity, it can cause objects getting stuck
- upon expiration the nodes weren't marked as not having the object, causing it
  to not be advertised
- I haven't tested it but at least I don't think can make things worse
2019-07-11 11:03:48 +02:00
453e045ae5
Minimal implementation of onionpeer object 2019-05-10 12:24:41 +03:00
e25fb857cb
Slightly reduced TCPConnection.sendAddr() and changed
in order to send only nodes with non-negative rating
2019-02-07 18:52:08 +02:00
ffbc09758c
Moved sslProtocolVersion and sslProtocolCiphers to network.tls 2019-02-04 22:23:28 +02:00
b52cb9360f
Moved unused timeOffsetWrongCount from shared to BMProto 2019-02-04 22:23:28 +02:00
5b5ec2b63d
shared.connectedHostsList is obsolete 2019-01-04 16:17:01 +02:00
1cdbb90f22
Get rid of map 2018-12-20 20:34:09 +01:00
3a4eed0f8b
getdata performance optimisation
- reduce number of calls to time.time()
- no need to shuffle, that's done by the upload thread
- get rid of unused import
2018-12-20 20:34:06 +01:00
ca567acab3
Put uploads into a separate thread
- instead of being processed in the ReceiveQueue thread, uploads are now done
  in a dedicated thread. Only the parsing is done in ReceiveQueue thread.
- the UploadThread is modelled based on the DownloadThred, but simpler.
- it checks for intersection attack, eliminates duplicates and restricts the
  write buffer size to 2MB (may still grow slightly higher if too many big
  objects are requested, but the absolute limit appears to be about 4.5MB in the
  worst case scenario).
- the restriction of the write buffer may cause some upload throttling (to
  about 2MB per second per connection), but can be optimised later
- fixes #1414
2018-12-20 20:34:01 +01:00
e1d5604674
Fix #1408 (duplicate connections to some hosts), got from chan 2018-12-18 19:36:46 +02:00
38318150bd
Moved missingObjects dict from state to network.objectracker 2018-12-16 17:44:26 +02:00
coffeedogs
10128b7560
Changes based on style and lint checks. (final_code_quality_2) 2018-10-31 17:06:26 +00:00
coffeedogs
0bceeca524
Changes based on style and lint checks. (final_code_quality_5) 2018-10-31 14:04:35 +00:00
coffeedogs
c875769b15
Changes based on style and lint checks. (final_code_quality_3) 2018-10-31 12:58:46 +00:00
1c914d8cd8
flake8 and style correction for network.udp 2018-10-27 17:55:38 +03:00
49985a435f
Special class BMStringParser for sending ACK (resolves #1374) 2018-10-27 17:55:02 +03:00
daf556ca50
Added tests for knownnodes:
- import of pickled knownnodes;
  - default knownnodes if nothing imported;
  - knownnodes starvation (#1335), demanded changes in networkthread.
2018-10-22 17:18:24 +03:00
ef54bcaf82
The changes in #1325 are buggy. Implementing it properly. 2018-10-12 11:28:13 +03:00
b2c8b77938
Closes #1360: validate knownnodes received from the network 2018-10-12 11:16:57 +03:00
342e2a209c
Use DNS seed only if knownnodes are empty 2018-07-30 17:39:12 +03:00
coffeedogs
02f40f5036
Fixed: ERROR - Too many items in inv message! Closes 1274 2018-06-14 11:57:04 +01:00
coffeedogs
6c268ba22f
Fixed: Type error, reported from the bitmessage chan 2018-05-16 12:01:51 +01:00
ca7becb921
Download fixes
- in corner cases, download request could have contained an incorrect
  request length. I haven't actually checked if this can be triggered
  though
- wait for downloading until anti intersection delay expires. Doesn't
  necessarily mean that it will always avoid peer's anti intersection
  delay, but it's close enough
- tracks last time an object was received. If it was too long time ago,
  reset the download request queue. This avoid situations like when a
  request gets ignored during the anti intersection delay, but it will
  keep thinking there are still pending requests as long as not all
  missing objects have been requested. This caused staggered download
  (request 1000 items, wait 1 minute, request 1000 more, wait another
  minute, ...)
- with these fixes, you should end up downloading as fast as your
  network and CPU allow
- best tested with trustedpeer
2018-04-03 19:24:07 +02:00
0c9cb4824d
Memory leak (kind of) fix
- objectsKnownToThem is supposed to track if it's necessary to send inv
  commands to a peer
- it is supposed to enter garbage collection after 1 hour
  (ObjectTracker.trackingExpires)
- due to peer not announcing all objects, or announcing them before we
  announce, this contains excessive number of entries after connection
  is fully established.
- profiling revealed that this creates unnecessary memory to be kept
  allocated
- this patch will prevent tracking of entries sent during bigInv,
  reducing the memory "leak"
- it is possible, in theory, that this does have some negative effect,
  like increased bandwidth or neglecting to announce some invs. It
  probably doesn't though as my review of objectsKnownToThem occurrences
  didn't reveal any such case, and since the dict didn't track fully
  accurately anyway (so it would have already been broken if it was a
  problem), I consider it an acceptable risk at the moment. If it indeed
  causes problems, they can be solved separately
- I tested this on one of the bootstrap servers with little memory, and
  it increased the number of connections than can be handled by a factor
  of about 3.5
2018-04-01 11:39:02 +02:00
Mahendra
1a36361ef9
replace all random.xxx with helper random function 2018-03-21 19:26:27 +05:30
4b995e7996
Fix circular import
- previous commit broke
2018-03-13 11:42:43 +01:00
1e4400a207
Remove obsolete code
- PendingDownload and PendingUpload in inventory.py are obsolete
- the replacement functions only provide lengths and are in
  network/stats.py
2018-03-13 07:32:23 +01:00
0da0c10995
Socks5 parsing fix
- Fixes #1134
- Came over BM, thank you kind stranger
2018-02-26 20:04:57 +01:00
1184b23223
Separate protocol processing exceptions
- exceptions thrown by the state methods are separated from missing
  connections or states. This allows more accurate error reporting
2018-02-19 21:27:38 +01:00
757740369b
Add proxy authentication
- proxy module was missing initialisation of authentication
2018-02-18 20:53:16 +01:00
eb97face61
Show traceback in protocol parser error handler 2018-02-12 17:07:54 +01:00
066b419e16
Bugfixes
- typo in dandelion
- stealth ackdata fix for broadcasts and mailing lists
2018-02-09 00:49:08 +01:00
08bb85a952
Dandelion staticmethod fix 2018-02-07 17:22:26 +01:00
3d1fa473fb
Dandelion updates
- expiration now uses poisson distribution just like in the bitcoin
version
2018-02-06 22:28:56 +01:00
03f08b3cfd
Typo 2018-02-04 21:16:30 +01:00
a646ec4902
Allow separate proxy for onions
- new options in network section: onionsocksproxytype,
onionsockshostname and onionsocksport. These allow to separate
connectivity types for onion and non-onion addresses, e.g. connect to
clear nodes over clearnet and onions over tor
- also remove some obsolete imports
2018-02-04 21:03:54 +01:00
fd1a6c1fa1
Dandelion update
- dandelion fixes
- try to wait as long as possible before expiration if there are no
outbound connections
- expire in invThread rather than singleCleaner thread
- deduplication of code in inv and dinv command methods
- turn on by default, seems to work correctly now
- turn off dandelion if outbound connections are disabled
- start tracking downloads earlier, and faster download loop
- remove some obsolete lines
- minor PEP8 updates
2018-02-03 11:46:39 +01:00
053f434e04
Download fixes
- don't make empty requests
- don't make requests if all objects are pending already
2018-02-02 12:44:43 +01:00
57c8c7c07c
Download thread exception handling 2018-02-01 23:18:08 +01:00
8498143783
Download fixes
- check if already have object before requesting
2018-02-01 22:58:04 +01:00
290b87a49f
Download fixes
- don't expire too quickly
- ignore connections that haven't been fully established yet
2018-02-01 12:48:14 +01:00
c5dc7fc903
Typo 2018-02-01 12:26:54 +01:00
8d05742436
Typo 2018-02-01 12:20:41 +01:00
68b58ce0c5
Download optimisation
- new data structure to handle download tracking, uses less CPU
2018-02-01 12:19:39 +01:00
d223bfc6f2
Fix kqueue poller
- separate read and write filters
- make filters peristent for reduced syscall count
2018-01-31 22:25:23 +01:00
222e666a60
Network thread shutdown iterator fix
Thanks to @g1itch for pointing this out.
2018-01-31 21:09:36 +01:00
9e79386595
Set state to "close" and call handle_close() for all connections 2018-01-30 14:04:03 +02:00
d6df4470e1
No connection CPU hog fix
- the previous fix was incomplete, it shouldn't consume excessive
resources now when there are no connections
2018-01-23 15:59:58 +01:00
01c8f3b66d
Fix asyncore CPU usage on no connection
- if there are no connections, asyncore would hog CPU
- thanks to an anonymous contributor
2018-01-22 22:37:29 +01:00
ba91d21261
CPU hogging fix
- handle _command functions that don't return anything
- fix udp command function
2018-01-22 22:18:01 +01:00
c9851b9f41
Connection lookups invalid data handling
- shouldn't throw an exception if argument is a string rather than Peer
2018-01-02 22:23:03 +01:00
f74f82e54f
Start downloading earlier 2018-01-02 22:20:33 +01:00
4086253730
Bandwidth limit optimisation
- should be slightly more accurate and use slightly fewer resources
2018-01-02 15:24:47 +01:00
8788f2d349
Server full and duplicate handling
- will try to report "Server full" over protocol for 10 extra
connections over limit, instead of simply dropping them
- if connected to the same host inbound and outbound, handle as server
full (prevents duplicate connections)
2018-01-02 14:29:21 +01:00
36cc5b9cf5
Download optimisations
- don't make empty download requests
- use smaller chunks when they can be spread across multiple
connections
2018-01-01 12:51:35 +01:00
1864762a0a
Apply bandwidth limits without restart
- also minor style fixes
2018-01-01 12:49:08 +01:00
e9b1aa48a9
Protocol error handler fixes
- was broken if there was no error message in "raise"
- added default texts for network exceptions
2017-12-29 08:49:08 +01:00
02490e3286
Don't break if over 50k messages
- typo if there were over 50k messages in inventory caused PyBM to stall
2017-12-29 08:41:15 +01:00
a3398d6a17
Don't crash download thread if no connections 2017-12-02 00:50:58 +01:00
41ead2bfb5
Distribute downloads more evenly
- also increases expireation of missing objects from 10 minutes to an
hour
2017-12-02 00:48:08 +01:00
e558b1fb72
Error handling
- proxy connections didn't init fullyEstablished
2017-11-19 13:48:43 +01:00
fdfbb77ed2
Close filehandles on connection reaping
- I thought this is done automatically through garbage collection, but I
think as the channel is still assigned in the asyncore map, it needs
to be done manually. Basically filehandle limit exceeded and it
crashed
2017-11-19 00:05:55 +01:00
5e042b76e7
Typo
- missing brackets
2017-11-18 10:05:41 +01:00
3aa6f386db
Dandelion fixes
- dandelion would always think there is a cycle and trigger fluff
- cycle fluff trigger didn't correctly re-download and re-announce the
object. Now it remembers between (d)inv and object commands that it's
in a fluff trigger phase.
2017-11-18 09:47:17 +01:00
4690dd6f00
Copy object contents from buffers on instantiation
- this may fix some memory issues
2017-11-17 23:53:46 +01:00
3c3d69e5de
Reap closed connection fix 2017-11-17 19:50:39 +01:00
1e02d2b48a
Download optimisation
- pending download tracking now per-connection instead of globally
2017-11-17 19:49:51 +01:00
5a787f41d2
Socket closing changes
- explicit close only through asyncore error handler
- implicit close through garbage collector
- avoid duplicate closing
2017-11-17 13:37:51 +01:00
1792289e00
close handling fix
- don't close a connection twice
2017-11-14 23:43:05 +01:00
4b40d4bce1
Download thread error handling 2017-10-22 15:28:30 +02:00
75a6f605c1
Download optimisation
- more accurate tracking
- randomise download order
- longer cycle
2017-10-22 11:32:37 +02:00
b025624f2a
missingObjects fix
- didn't notice valid objects arriving (only invalid)
2017-10-20 23:21:25 +02:00
6655e99aa3
Pending download stats optimisations
- tracks separately a global list for a faster sum. Needs slightly
more memory
2017-10-20 23:11:33 +02:00
a746ba9da7
Stop downloading objects with insufficient PoW
- object with insufficient PoW weren't correctly detected and it
tried to download them over and over again
2017-10-20 13:21:39 +02:00
2d34e73648
Dandelion updates
- fixes and feedback from @gfanti and @amiller
- addresses #1049
- minor refactoring
- two global child stems with fixed mapping between parent and
child stem
- allow child stems which don't support dandelion
- only allow outbound connections to be stems
- adjust stems if opening/closing outbound connections (should
allow partial dandelion functionality when not enough outbound
connections are available instead of breaking)
2017-10-20 01:21:49 +02:00
15857e6551
Asyncore updates
- reduce buffer size to 128kB (was 2MB)
- IP address handling use str instead of buffer (the latter, even
though it should be faster, breaks the code on Windows)
- read up to full buffer after fully established (otherwise
downloads become too slow due to the loop time). This reverts
a change made in d28a7bfb86
2017-10-20 01:07:30 +02:00
ab458531e8
Changes in SOCKS and onion handling in connectionchooser
- onion addresses have a priority of 1 when SOCKS is on
- don't connect to private/loopback addresses when SOCKS is on
2017-10-19 09:16:29 +02:00
7b470d4b66
handle shutdown in receivequeuethread
- sometimes during shutdown, the receivequeuethread would get
stuck
2017-10-19 09:11:34 +02:00
391d40d78b
Socket closing changes
- closing reason moved to a variable
- actual closing now done in asyncore thread instead of
receivedata thread
2017-10-19 09:08:05 +02:00