Commit Graph

26 Commits

Author SHA1 Message Date
lakshyacis
d8ea0afe40
downloadthread.py Pylint fixes 2019-09-02 19:15:15 +05:30
lakshyacis
8589f01d3d
downloadthread.py flake8 fixes 2019-09-02 19:02:50 +05:30
ad2a2b3fb4
Inherit helper_threading.StoppableThread from threading.Thread
and do random.seed() in its __init__
2019-08-06 18:07:19 +03:00
38318150bd
Moved missingObjects dict from state to network.objectracker 2018-12-16 17:44:26 +02:00
ca7becb921
Download fixes
- in corner cases, download request could have contained an incorrect
  request length. I haven't actually checked if this can be triggered
  though
- wait for downloading until anti intersection delay expires. Doesn't
  necessarily mean that it will always avoid peer's anti intersection
  delay, but it's close enough
- tracks last time an object was received. If it was too long time ago,
  reset the download request queue. This avoid situations like when a
  request gets ignored during the anti intersection delay, but it will
  keep thinking there are still pending requests as long as not all
  missing objects have been requested. This caused staggered download
  (request 1000 items, wait 1 minute, request 1000 more, wait another
  minute, ...)
- with these fixes, you should end up downloading as fast as your
  network and CPU allow
- best tested with trustedpeer
2018-04-03 19:24:07 +02:00
Mahendra
1a36361ef9
replace all random.xxx with helper random function 2018-03-21 19:26:27 +05:30
fd1a6c1fa1
Dandelion update
- dandelion fixes
- try to wait as long as possible before expiration if there are no
outbound connections
- expire in invThread rather than singleCleaner thread
- deduplication of code in inv and dinv command methods
- turn on by default, seems to work correctly now
- turn off dandelion if outbound connections are disabled
- start tracking downloads earlier, and faster download loop
- remove some obsolete lines
- minor PEP8 updates
2018-02-03 11:46:39 +01:00
053f434e04
Download fixes
- don't make empty requests
- don't make requests if all objects are pending already
2018-02-02 12:44:43 +01:00
57c8c7c07c
Download thread exception handling 2018-02-01 23:18:08 +01:00
8498143783
Download fixes
- check if already have object before requesting
2018-02-01 22:58:04 +01:00
290b87a49f
Download fixes
- don't expire too quickly
- ignore connections that haven't been fully established yet
2018-02-01 12:48:14 +01:00
c5dc7fc903
Typo 2018-02-01 12:26:54 +01:00
68b58ce0c5
Download optimisation
- new data structure to handle download tracking, uses less CPU
2018-02-01 12:19:39 +01:00
36cc5b9cf5
Download optimisations
- don't make empty download requests
- use smaller chunks when they can be spread across multiple
connections
2018-01-01 12:51:35 +01:00
a3398d6a17
Don't crash download thread if no connections 2017-12-02 00:50:58 +01:00
41ead2bfb5
Distribute downloads more evenly
- also increases expireation of missing objects from 10 minutes to an
hour
2017-12-02 00:48:08 +01:00
1e02d2b48a
Download optimisation
- pending download tracking now per-connection instead of globally
2017-11-17 19:49:51 +01:00
4b40d4bce1
Download thread error handling 2017-10-22 15:28:30 +02:00
75a6f605c1
Download optimisation
- more accurate tracking
- randomise download order
- longer cycle
2017-10-22 11:32:37 +02:00
4c9006a632
Asyncore performance optimisation
- use bytearray instead of strings for buffers
2017-10-16 08:07:32 +02:00
f088e0ae21
Change thread names
- not needed to have "Thread" in the name of a thread
2017-07-10 07:05:50 +02:00
2d7d9c2f92
Asyncore update
- request downloads in bigger chunks
- don't put whole objects into the receiveDataQueue
2017-07-08 06:54:25 +02:00
ba4162d7fe
Asyncore update
- get rid of per-connection writeQueue/receiveQueue, and instead use
strings and locking
- minor code cleanup
- all state handlers now should set expectBytes
- almost all data processing happens in ReceiveDataThread, and
AsyncoreThread is almost only I/O (plus TLS). AsyncoreThread simply
puts the connection object into the queue when it has some data for
processing
- allow poll, epoll and kqueue handlers. kqueue is untested and
unoptimised, poll and epoll seem to work ok (linux)
- stack depth threshold handler  in decode_payload_content, this is
recursive and I think was causing occasional RuntimeErrors. Fixes #964
- longer asyncore loops, as now data is handled in ReceiveDataThread
- randomise node order when deciding what to download. Should prevent
retries being stuck to the same node
- socks cleanup (socks5 works ok, socks4a untested but should work too)
2017-07-06 19:45:36 +02:00
fc19e4119a
Download thread updates
- now tracks downloads globally too, so it doesn't request the same
object from multiple peers at the same time
- retries at the earliest every minute
- stops trying to download an object after an hour
- minor fixes in retrying downloading invalid objects
2017-07-05 09:25:49 +02:00
2685fe29b1
Code quality improvements 2017-06-24 12:13:35 +02:00
0cc8589b27
Asyncore updates
- should prevent the same object being re-requested indefinitely
- locking for object tracking
- move SSL-specific error handling to TLSDispatcher
- observe maximum connection limit when accepting a new connection
- stack depth test (for debugging purposes)
- separate download thread
- connection pool init moved to main thread
2017-06-21 12:16:33 +02:00