Commit Graph

272 Commits

Author SHA1 Message Date
0c9cb4824d
Memory leak (kind of) fix
- objectsKnownToThem is supposed to track if it's necessary to send inv
  commands to a peer
- it is supposed to enter garbage collection after 1 hour
  (ObjectTracker.trackingExpires)
- due to peer not announcing all objects, or announcing them before we
  announce, this contains excessive number of entries after connection
  is fully established.
- profiling revealed that this creates unnecessary memory to be kept
  allocated
- this patch will prevent tracking of entries sent during bigInv,
  reducing the memory "leak"
- it is possible, in theory, that this does have some negative effect,
  like increased bandwidth or neglecting to announce some invs. It
  probably doesn't though as my review of objectsKnownToThem occurrences
  didn't reveal any such case, and since the dict didn't track fully
  accurately anyway (so it would have already been broken if it was a
  problem), I consider it an acceptable risk at the moment. If it indeed
  causes problems, they can be solved separately
- I tested this on one of the bootstrap servers with little memory, and
  it increased the number of connections than can be handled by a factor
  of about 3.5
2018-04-01 11:39:02 +02:00
Mahendra
1a36361ef9
replace all random.xxx with helper random function 2018-03-21 19:26:27 +05:30
4b995e7996
Fix circular import
- previous commit broke
2018-03-13 11:42:43 +01:00
1e4400a207
Remove obsolete code
- PendingDownload and PendingUpload in inventory.py are obsolete
- the replacement functions only provide lengths and are in
  network/stats.py
2018-03-13 07:32:23 +01:00
0da0c10995
Socks5 parsing fix
- Fixes #1134
- Came over BM, thank you kind stranger
2018-02-26 20:04:57 +01:00
1184b23223
Separate protocol processing exceptions
- exceptions thrown by the state methods are separated from missing
  connections or states. This allows more accurate error reporting
2018-02-19 21:27:38 +01:00
757740369b
Add proxy authentication
- proxy module was missing initialisation of authentication
2018-02-18 20:53:16 +01:00
eb97face61
Show traceback in protocol parser error handler 2018-02-12 17:07:54 +01:00
066b419e16
Bugfixes
- typo in dandelion
- stealth ackdata fix for broadcasts and mailing lists
2018-02-09 00:49:08 +01:00
08bb85a952
Dandelion staticmethod fix 2018-02-07 17:22:26 +01:00
3d1fa473fb
Dandelion updates
- expiration now uses poisson distribution just like in the bitcoin
version
2018-02-06 22:28:56 +01:00
03f08b3cfd
Typo 2018-02-04 21:16:30 +01:00
a646ec4902
Allow separate proxy for onions
- new options in network section: onionsocksproxytype,
onionsockshostname and onionsocksport. These allow to separate
connectivity types for onion and non-onion addresses, e.g. connect to
clear nodes over clearnet and onions over tor
- also remove some obsolete imports
2018-02-04 21:03:54 +01:00
fd1a6c1fa1
Dandelion update
- dandelion fixes
- try to wait as long as possible before expiration if there are no
outbound connections
- expire in invThread rather than singleCleaner thread
- deduplication of code in inv and dinv command methods
- turn on by default, seems to work correctly now
- turn off dandelion if outbound connections are disabled
- start tracking downloads earlier, and faster download loop
- remove some obsolete lines
- minor PEP8 updates
2018-02-03 11:46:39 +01:00
053f434e04
Download fixes
- don't make empty requests
- don't make requests if all objects are pending already
2018-02-02 12:44:43 +01:00
57c8c7c07c
Download thread exception handling 2018-02-01 23:18:08 +01:00
8498143783
Download fixes
- check if already have object before requesting
2018-02-01 22:58:04 +01:00
290b87a49f
Download fixes
- don't expire too quickly
- ignore connections that haven't been fully established yet
2018-02-01 12:48:14 +01:00
c5dc7fc903
Typo 2018-02-01 12:26:54 +01:00
8d05742436
Typo 2018-02-01 12:20:41 +01:00
68b58ce0c5
Download optimisation
- new data structure to handle download tracking, uses less CPU
2018-02-01 12:19:39 +01:00
d223bfc6f2
Fix kqueue poller
- separate read and write filters
- make filters peristent for reduced syscall count
2018-01-31 22:25:23 +01:00
222e666a60
Network thread shutdown iterator fix
Thanks to @g1itch for pointing this out.
2018-01-31 21:09:36 +01:00
9e79386595
Set state to "close" and call handle_close() for all connections 2018-01-30 14:04:03 +02:00
d6df4470e1
No connection CPU hog fix
- the previous fix was incomplete, it shouldn't consume excessive
resources now when there are no connections
2018-01-23 15:59:58 +01:00
01c8f3b66d
Fix asyncore CPU usage on no connection
- if there are no connections, asyncore would hog CPU
- thanks to an anonymous contributor
2018-01-22 22:37:29 +01:00
ba91d21261
CPU hogging fix
- handle _command functions that don't return anything
- fix udp command function
2018-01-22 22:18:01 +01:00
c9851b9f41
Connection lookups invalid data handling
- shouldn't throw an exception if argument is a string rather than Peer
2018-01-02 22:23:03 +01:00
f74f82e54f
Start downloading earlier 2018-01-02 22:20:33 +01:00
4086253730
Bandwidth limit optimisation
- should be slightly more accurate and use slightly fewer resources
2018-01-02 15:24:47 +01:00
8788f2d349
Server full and duplicate handling
- will try to report "Server full" over protocol for 10 extra
connections over limit, instead of simply dropping them
- if connected to the same host inbound and outbound, handle as server
full (prevents duplicate connections)
2018-01-02 14:29:21 +01:00
36cc5b9cf5
Download optimisations
- don't make empty download requests
- use smaller chunks when they can be spread across multiple
connections
2018-01-01 12:51:35 +01:00
1864762a0a
Apply bandwidth limits without restart
- also minor style fixes
2018-01-01 12:49:08 +01:00
e9b1aa48a9
Protocol error handler fixes
- was broken if there was no error message in "raise"
- added default texts for network exceptions
2017-12-29 08:49:08 +01:00
02490e3286
Don't break if over 50k messages
- typo if there were over 50k messages in inventory caused PyBM to stall
2017-12-29 08:41:15 +01:00
a3398d6a17
Don't crash download thread if no connections 2017-12-02 00:50:58 +01:00
41ead2bfb5
Distribute downloads more evenly
- also increases expireation of missing objects from 10 minutes to an
hour
2017-12-02 00:48:08 +01:00
e558b1fb72
Error handling
- proxy connections didn't init fullyEstablished
2017-11-19 13:48:43 +01:00
fdfbb77ed2
Close filehandles on connection reaping
- I thought this is done automatically through garbage collection, but I
think as the channel is still assigned in the asyncore map, it needs
to be done manually. Basically filehandle limit exceeded and it
crashed
2017-11-19 00:05:55 +01:00
5e042b76e7
Typo
- missing brackets
2017-11-18 10:05:41 +01:00
3aa6f386db
Dandelion fixes
- dandelion would always think there is a cycle and trigger fluff
- cycle fluff trigger didn't correctly re-download and re-announce the
object. Now it remembers between (d)inv and object commands that it's
in a fluff trigger phase.
2017-11-18 09:47:17 +01:00
4690dd6f00
Copy object contents from buffers on instantiation
- this may fix some memory issues
2017-11-17 23:53:46 +01:00
3c3d69e5de
Reap closed connection fix 2017-11-17 19:50:39 +01:00
1e02d2b48a
Download optimisation
- pending download tracking now per-connection instead of globally
2017-11-17 19:49:51 +01:00
5a787f41d2
Socket closing changes
- explicit close only through asyncore error handler
- implicit close through garbage collector
- avoid duplicate closing
2017-11-17 13:37:51 +01:00
1792289e00
close handling fix
- don't close a connection twice
2017-11-14 23:43:05 +01:00
4b40d4bce1
Download thread error handling 2017-10-22 15:28:30 +02:00
75a6f605c1
Download optimisation
- more accurate tracking
- randomise download order
- longer cycle
2017-10-22 11:32:37 +02:00
b025624f2a
missingObjects fix
- didn't notice valid objects arriving (only invalid)
2017-10-20 23:21:25 +02:00
6655e99aa3
Pending download stats optimisations
- tracks separately a global list for a faster sum. Needs slightly
more memory
2017-10-20 23:11:33 +02:00
a746ba9da7
Stop downloading objects with insufficient PoW
- object with insufficient PoW weren't correctly detected and it
tried to download them over and over again
2017-10-20 13:21:39 +02:00
2d34e73648
Dandelion updates
- fixes and feedback from @gfanti and @amiller
- addresses #1049
- minor refactoring
- two global child stems with fixed mapping between parent and
child stem
- allow child stems which don't support dandelion
- only allow outbound connections to be stems
- adjust stems if opening/closing outbound connections (should
allow partial dandelion functionality when not enough outbound
connections are available instead of breaking)
2017-10-20 01:21:49 +02:00
15857e6551
Asyncore updates
- reduce buffer size to 128kB (was 2MB)
- IP address handling use str instead of buffer (the latter, even
though it should be faster, breaks the code on Windows)
- read up to full buffer after fully established (otherwise
downloads become too slow due to the loop time). This reverts
a change made in d28a7bfb86
2017-10-20 01:07:30 +02:00
ab458531e8
Changes in SOCKS and onion handling in connectionchooser
- onion addresses have a priority of 1 when SOCKS is on
- don't connect to private/loopback addresses when SOCKS is on
2017-10-19 09:16:29 +02:00
7b470d4b66
handle shutdown in receivequeuethread
- sometimes during shutdown, the receivequeuethread would get
stuck
2017-10-19 09:11:34 +02:00
391d40d78b
Socket closing changes
- closing reason moved to a variable
- actual closing now done in asyncore thread instead of
receivedata thread
2017-10-19 09:08:05 +02:00
d28a7bfb86
Asyncore performance optimisation
- don't transfer unnecessary amount of bytes from network buffers
- slice buffer more efficiently if it results in an empty buffer
2017-10-19 09:02:33 +02:00
7ec3fc7a5a
Prevent negative DownloadChunk in asyncore 2017-10-19 09:00:54 +02:00
f785558ca5
Don't close UDP socket on bad packet magic 2017-10-19 09:00:02 +02:00
a090eea9b0
Minor multiqueue updates
- add task_done to addrthread and invthread
- implement totalSize for multiqueue
- order in invThread changed
2017-10-19 08:56:48 +02:00
d44c6c6464
Forget known nodes with bad rating 2017-10-19 08:52:44 +02:00
a013814c6b
Network tab updates
- handle add/remove entry instead of recreating the whole connection
list
- update processed object counts after each object
2017-10-19 08:39:09 +02:00
7e0932815d
UDP socket closing fix
- invalid data or an incomplete read on UDP socket caused
it to close
2017-10-19 01:46:32 +02:00
4c9006a632
Asyncore performance optimisation
- use bytearray instead of strings for buffers
2017-10-16 08:07:32 +02:00
a49b3b5f84
Asyncore UDP fix
- when there was an error writing to a udp socket, it wasn't
handled correctly
2017-10-06 18:26:06 +02:00
333170b172
Dandelion fixes
- more exception handling
- only use outbound connections for stems
(thanks to @amillter for info)
- don't create stems if config disabled
- addresses #1049
2017-10-06 16:33:44 +02:00
6548999a49
Dandelion fix
- thanks to g1itch for reporting
- addresses #1049
2017-10-02 08:02:29 +02:00
b1442ecb0a
Dandelion fixes and updates
- also, randomise the item order in an inv/dinv command
2017-09-30 13:42:04 +02:00
9923d288e0
Dandelion fixes
- in route selector, some connections may not have the services
attribute (yet)
- Addresses #1049
2017-09-25 09:17:15 +02:00
d574b167d8
Dandelion updates & fixes
- Addresses #1049
- Add dandelion routes for locally generated objects
- Minor bugfixes
- Send dinv commands on stem objects (instead of always sending inv
command)
2017-09-25 08:49:21 +02:00
6ce86b1d0a
Dandelion++ implementation
- untested, some functionality may be missing, don't turn on
- also, it randomises upload of requested objects
- affects #1049
2017-09-25 01:17:04 +02:00
1881bcea68
Don't connect on first start until approved 2017-09-21 18:18:42 +02:00
660997b8f4
Code Quality 2017-08-22 13:49:27 +02:00
e7231f3aea
Fix multiple TCP bind address handling 2017-08-09 23:30:22 +02:00
6c695c8ac7
Remove non-asyncore network code (partial) 2017-08-09 17:36:52 +02:00
0b07b1c89a
Peer discovery updates
- allow loopback addresses, now you can bind different loopback IP
addresses on a single system and they will auto-cross-connect
- always listen for discovery on 0.0.0.0
- [network] - bind now also applies for the TCP socket as well as UDP
socket
- closing socket iterator fix
2017-08-09 17:34:47 +02:00
e071efac1a
Typo 2017-08-09 17:29:48 +02:00
cc955cd69d
Try new ports of binding fails
- API and BM protocol will try random ports for binding if those
configured are occupied
2017-08-09 17:29:23 +02:00
0324958e92
Peer discovery fixes
- incoming packets weren't correctly processed
2017-08-06 23:05:54 +02:00
f338c00f8e
Change peer discovery tracking from queue to a dict
- with a queue, a situation could occur when new entries are appended
but nothing is polling the queue
2017-08-06 21:29:54 +02:00
4564d37f5b
Typo in previous commits 2017-08-06 21:26:25 +02:00
38872159fb
Typo in previous commit 2017-08-06 20:40:35 +02:00
5895dc2f1f
Asyncore Windows error handling
- windows behaves somewhat differently when using select
2017-08-06 20:39:14 +02:00
5108d08ac9
Windows asyncore error handler fix
- WSAEWOULDBLOCK is now checked on connect and accept
2017-08-06 18:18:21 +02:00
8f14fb05a1
UDP socket setsockopt fix 2017-08-05 10:14:15 +02:00
a29f7534ee
Add EINTR handler for select and poll pollers 2017-07-21 09:06:02 +02:00
2530c62050
epoll throws IOError rather than select.error 2017-07-21 07:49:34 +02:00
aa059d6f2f
Handle TLS errors in receivequeuethread
- well at least EBADF, it seems to happen sometimes
2017-07-21 07:47:18 +02:00
4f19c37fdc
Parser fix for multi-level arrays 2017-07-11 10:29:29 +02:00
dcc181bf75
Asyncore processing thread synchronisation
- threre was a synchronisation problem where one thread could process
more data than another thread was expecting, leading to the thread
crashing
2017-07-10 23:18:58 +02:00
f6d5d93bf2
Multiple receive queues fix
- forgot to commit busy handler
2017-07-10 20:52:11 +02:00
db2d78c9b6
Make protocol decoder less recursive
- apparently, recursion has bad performance in Python, so the decoder is
now flat, except when parsing "version" command
2017-07-10 07:15:27 +02:00
853c8561ec
Per connection node id part 2
- forgot to include this in the previous commit
2017-07-10 07:12:52 +02:00
3941b39136
Randomise node id
- in order to detect if it's connected to to itself, PyBitmessage now
uses a per-connection id rather than a global one
2017-07-10 07:10:05 +02:00
bdf61489ae
Allow multiple ReceiveQueue threads
- defaults to 3
2017-07-10 07:08:10 +02:00
f088e0ae21
Change thread names
- not needed to have "Thread" in the name of a thread
2017-07-10 07:05:50 +02:00
4fce01e34a
Less data transferred in invThread and addrThread 2017-07-08 18:02:47 +02:00
2df9598774
Asyncore update: Fix incoming connections
- dereferencing wasn't done correctly for incoming connections
2017-07-08 07:33:29 +02:00
2d7d9c2f92
Asyncore update
- request downloads in bigger chunks
- don't put whole objects into the receiveDataQueue
2017-07-08 06:54:25 +02:00
0f3a69adf4
Asyncore update: remove references to deleted files 2017-07-08 06:53:20 +02:00
5ae1b6d865
Asyncore update: remove obsolete files 2017-07-08 06:52:17 +02:00
a98b8690d3
Asyncore fixes
- fix broken loops
- optimise I/O tests
2017-07-07 07:55:29 +02:00
ba4162d7fe
Asyncore update
- get rid of per-connection writeQueue/receiveQueue, and instead use
strings and locking
- minor code cleanup
- all state handlers now should set expectBytes
- almost all data processing happens in ReceiveDataThread, and
AsyncoreThread is almost only I/O (plus TLS). AsyncoreThread simply
puts the connection object into the queue when it has some data for
processing
- allow poll, epoll and kqueue handlers. kqueue is untested and
unoptimised, poll and epoll seem to work ok (linux)
- stack depth threshold handler  in decode_payload_content, this is
recursive and I think was causing occasional RuntimeErrors. Fixes #964
- longer asyncore loops, as now data is handled in ReceiveDataThread
- randomise node order when deciding what to download. Should prevent
retries being stuck to the same node
- socks cleanup (socks5 works ok, socks4a untested but should work too)
2017-07-06 19:45:36 +02:00
fe0664640e
Migrate antiIntersectionDelay to asyncore
- implemented by ignoring getdata during the delay rather than sleeping
as it was in the threaded model
- it can happen that a valid getdata request is received during the
delay. A node should be implemented in a way that retries to download,
that may not be the case with older PyBitmessage versions or other
implementations
2017-07-05 09:27:52 +02:00
fc19e4119a
Download thread updates
- now tracks downloads globally too, so it doesn't request the same
object from multiple peers at the same time
- retries at the earliest every minute
- stops trying to download an object after an hour
- minor fixes in retrying downloading invalid objects
2017-07-05 09:25:49 +02:00
846fced0a2
Remove obsolete inactive code 2017-07-05 09:19:56 +02:00
d086781ce8
AddrQueue fix
- forgot to commit import
2017-07-05 09:19:18 +02:00
a0bbd21efc
Add ratings to peers
- outbound peers now have a rating
- it's also shown in the network status tab
- currently it's between -1 to +1, changes by 0.1 steps and uses a
hyperbolic function 0.05/(1.0 - rating) to convert rating to
probability with which we should connect to that node when randomly
chosen
- it increases when we successfully establish a full outbound connection
to a node, and decreases when we fail to do that
- onion nodes have priority when using SOCKS
2017-07-05 09:17:01 +02:00
9d09f9f3ce
Reduce severity of socks connectivity errors
- Fixes #1024
- Fixes #1019
2017-07-05 09:07:00 +02:00
e00a02206b
AddrThread
- this thread is for spreading new/updated addresses in active
connections, analogous to the InvThread
- it doesn't do anything yet, this is just a dummy queue at the moment
2017-07-05 08:57:44 +02:00
cc3cf77759
New class multiqueue
- to be used for invthread and addthread
- updated invthread for multiqueue
2017-06-27 13:25:12 +02:00
189578cba3
Asyncore proxy fixes
- SOCKS5 now seems to work, SOCKS4a untested
2017-06-24 12:23:56 +02:00
916b85c862
Connection pool cleanup
- minor code quality improvement
2017-06-24 12:23:16 +02:00
0dc0b22974
Expired / Stream mismatch / duplicate object error handling
- cleanup of the code
2017-06-24 12:22:41 +02:00
0a79490e2c
Known nodes maximum configurable part 2 2017-06-24 12:21:42 +02:00
d57b0c55ee
Object validator trigger moved
- from bmproto to bmobject
2017-06-24 12:21:06 +02:00
dc5a91f326
Remove stack depth warnings
- I was never able to trigger them
2017-06-24 12:19:19 +02:00
e9edf70d3a
TLS updates
- save TLS version
- minor TLS error handling updates
2017-06-24 12:18:15 +02:00
26eb54a82e
Network status updates
- now lists each node with its info instead of a per-stream summary
2017-06-24 12:16:12 +02:00
2685fe29b1
Code quality improvements 2017-06-24 12:13:35 +02:00
0cc8589b27
Asyncore updates
- should prevent the same object being re-requested indefinitely
- locking for object tracking
- move SSL-specific error handling to TLSDispatcher
- observe maximum connection limit when accepting a new connection
- stack depth test (for debugging purposes)
- separate download thread
- connection pool init moved to main thread
2017-06-21 12:16:33 +02:00
a3a55e53c4
UDP Socket dict address fix
- fixes #1008
2017-06-11 14:11:39 +02:00
cba749088a
Asyncore updates
- mainly work on proxy support, but it's still not fully working
- minor bugfixes
2017-06-10 10:13:49 +02:00
7bde4e9445
Missing module name in bmproto 2017-06-04 10:25:16 +02:00
e8d9a7f183
Asyncore connect handling
- minor improvements in handling of connect events so that it's not
processed twice
2017-06-03 16:30:05 +02:00
a9c0000c17
Treat some invalid objects as received
- update to 6044df5adf
- objects that are expired or in wrong stream are not re-requested
anymore, even if they aren't stored in the inventory
- the previous option "acceptmismatch" now only affects whether such
objects are stored in the inventory
2017-06-03 16:29:21 +02:00
6044df5adf
Optional storing of expired and off-stream objects
- a new config file option, network/acceptmismatch, allows the inventory
to store objects that expired or are from a stream we're not
interested in. Having this on will prevent re-requesting objects that
other nodes incorrectly advertise. It defaults to false
2017-06-02 15:43:35 +02:00
d75d920a68
Asyncore updates
- clean object tracking dictionaries in the cleaner thread
- clean up close / handle_close
- add locking to tracking dictionaries
2017-06-02 07:09:35 +02:00
4c17a18006
Don't send invs to unestablished connections 2017-05-31 23:34:06 +02:00
18988ae2e6
Asyncore updates
- performance optimisation, reduce number of loops when waiting
for protocol headers / commands
2017-05-31 10:17:36 +02:00
d9e3349eeb
Fix own IP detection
- minor bug in the previous commit
2017-05-31 00:22:07 +02:00
f23c169eec
Don't connect to myself
- track local IP+port of incoming connections and don't connect
to them in the future
2017-05-31 00:04:21 +02:00
fa9811f426
Asyncore update
- duplicate checking implemented
- connection pool vs. socket closing cleanup
2017-05-30 23:53:43 +02:00
abaa2c72e5
typo 2017-05-29 15:04:22 +02:00
11d02b1e41
typo 2017-05-29 14:59:42 +02:00
3a543efd83
Typo 2017-05-29 14:54:13 +02:00
7f381c0c25
Asyncore update
- incoming object handling fix
2017-05-29 14:52:31 +02:00
1ccfd41c3f
Asyncore updates
- fix connected to myself check
2017-05-29 14:41:02 +02:00
0aa5dbd958
Asyncore update
- shutdown fix
2017-05-29 14:35:08 +02:00
bdaa939e2c
ENOTSOCK fix 2017-05-29 13:39:26 +02:00
97c44b97f4
Asyncore update
- handle WSAENOTSOCK
2017-05-29 13:14:25 +02:00
a5c1b0c529
Asyncore fixes
- better handling of WSA* checks on non-windows systems
- handle EBADF on Windows/select
- better timeouts / loop lengths in main asyncore loop and
spawning new connections
- remove InvThread prints
2017-05-29 12:56:59 +02:00
74f1a74a8c
Make SO_REUSEPORT optional
- apparently not available on Windows
2017-05-29 11:30:56 +02:00
02a07e5119
Asyncore update
- default to true (original attempt didn't work correctly)
2017-05-29 03:16:14 +02:00
65bb6648e7
Asyncore updates
- fix crash in inv thread
- more prints changed into logger
- minor fixes
2017-05-29 00:47:41 +02:00
c85d52b8e8
Asyncore updates
- asyncore is now on by default
- inv announcements implemented
- bandwidth limit implemented / fixed
- stats on download / upload speed now work
- make prints into logger
- limit knownNodes to 20k as it was before
- green light fixed
- other minor fixes
2017-05-29 00:24:07 +02:00
5d4e1e2007
asyncore fixes
- bm headers and commands are only read up to expected length.
On a very fast connection (e.g. local VM), reading verack
also read a part of the TLS handshake
- some debugging info moved from print to logger.debug
- tls handshake cleanup
2017-05-27 22:30:30 +02:00
21f6d38ec2
Asyncore fixes
- TCP fixes
2017-05-27 21:52:56 +02:00
99e714c432
UDP socket bugfixes 2017-05-27 20:43:27 +02:00
fa9ad537a5
Add task_done to asyncore-related queues 2017-05-27 19:39:19 +02:00