- get rid of per-connection writeQueue/receiveQueue, and instead use
strings and locking
- minor code cleanup
- all state handlers now should set expectBytes
- almost all data processing happens in ReceiveDataThread, and
AsyncoreThread is almost only I/O (plus TLS). AsyncoreThread simply
puts the connection object into the queue when it has some data for
processing
- allow poll, epoll and kqueue handlers. kqueue is untested and
unoptimised, poll and epoll seem to work ok (linux)
- stack depth threshold handler in decode_payload_content, this is
recursive and I think was causing occasional RuntimeErrors. Fixes#964
- longer asyncore loops, as now data is handled in ReceiveDataThread
- randomise node order when deciding what to download. Should prevent
retries being stuck to the same node
- socks cleanup (socks5 works ok, socks4a untested but should work too)
- implemented by ignoring getdata during the delay rather than sleeping
as it was in the threaded model
- it can happen that a valid getdata request is received during the
delay. A node should be implemented in a way that retries to download,
that may not be the case with older PyBitmessage versions or other
implementations
- now tracks downloads globally too, so it doesn't request the same
object from multiple peers at the same time
- retries at the earliest every minute
- stops trying to download an object after an hour
- minor fixes in retrying downloading invalid objects
- outbound peers now have a rating
- it's also shown in the network status tab
- currently it's between -1 to +1, changes by 0.1 steps and uses a
hyperbolic function 0.05/(1.0 - rating) to convert rating to
probability with which we should connect to that node when randomly
chosen
- it increases when we successfully establish a full outbound connection
to a node, and decreases when we fail to do that
- onion nodes have priority when using SOCKS
- this thread is for spreading new/updated addresses in active
connections, analogous to the InvThread
- it doesn't do anything yet, this is just a dummy queue at the moment
- not used yet, just an inactive helper function
- I received feedback that OpenSSL.rand isn't more secure than
os.urandom. I read several debates/analyses about it and concur
- should prevent the same object being re-requested indefinitely
- locking for object tracking
- move SSL-specific error handling to TLSDispatcher
- observe maximum connection limit when accepting a new connection
- stack depth test (for debugging purposes)
- separate download thread
- connection pool init moved to main thread
- update to 6044df5adf
- objects that are expired or in wrong stream are not re-requested
anymore, even if they aren't stored in the inventory
- the previous option "acceptmismatch" now only affects whether such
objects are stored in the inventory
- a new config file option, network/acceptmismatch, allows the inventory
to store objects that expired or are from a stream we're not
interested in. Having this on will prevent re-requesting objects that
other nodes incorrectly advertise. It defaults to false
- better handling of WSA* checks on non-windows systems
- handle EBADF on Windows/select
- better timeouts / loop lengths in main asyncore loop and
spawning new connections
- remove InvThread prints
- asyncore is now on by default
- inv announcements implemented
- bandwidth limit implemented / fixed
- stats on download / upload speed now work
- make prints into logger
- limit knownNodes to 20k as it was before
- green light fixed
- other minor fixes
- bm headers and commands are only read up to expected length.
On a very fast connection (e.g. local VM), reading verack
also read a part of the TLS handshake
- some debugging info moved from print to logger.debug
- tls handshake cleanup
- bugfixes
- UDP socket for local peer discovery
- new function assembleAddr to unify creating address command
- open port checker functionality (inactive)
- sendBigInv is done in a thread separate from the network IO
thread
- separate queue for processing blocking stuff on reception
- rewrote write buffer as a queue
- some addr handling
- number of half open connections correct
- Network status UI works but current speed isn't implemented yet
- Track per connection and global transferred bytes
- Add locking to write queue so that other threads can put stuff
there
- send ping on timeout (instead of closing the connection)
- implement open port checker (untested, never triggered yet)
- error handling on IO
- most of the stuff is done so it partially works
- disabled pollers other than select (debugging necessary)
- can switch in the settings, section network, option asyncore (defaults
to False)
- bmconfigpaser.py now allows to put default values for a specific
option in the file
- addresses as sections are now detected by "BM-" rather than
just ignoring bitmessagesettings. There can now be other sections
with a cleaner config file
- in some cases when IPv6 stack is available and onionbindip is an IPv4
address, socket.bind doesn't change the bound address, ending up
listening on everything
- immediately return from initCL() if numpy or pyopencl is unevailable
(no ImportError because of resetPoW() call)
- use glob to find C extension even if it named like
`bitmsghash.x86_64-linux-gnu.so`
If user chooses to show the Settings dialog:
- activate the "Network Settings" tab
- remove option 'dontconnect' if settings have been saved
- track pending hashId more accurately
- add timeout and a cleanup so that the download queues don't
get stuck and memory is freed
- randomise download order (only works for inv commands with
more than 1 entry)
- replace PendingDownload singleton dict with a Queue
- total memory and CPU requirements should be reduced
- get rid of somObjectsOfWhichThisRemoteNodeIsAlearedyAware. It has very
little practicle effect and only uses memory
- if too many nodes, only delete oldest nodes in bootstrap provider
mode, in normal mode ignore new nodes as it used to before
- in bootstrap provider mode, penalise nodes announced by others by 1
day instead of 3 hours
- version command struct for faster unpacking
- increase read buffer to 2MB to allow a full command to fit
- initial bitmessage protocol class (WIP)
- error handling
- remove duplicate method
- finished proxy design
- socks4a and socks5 implemented
- authentication not tested
- resolver for both socks4a and socks5
- http client example using the proxy
- if knownNodes grows to 20000, instead of ignoring new nodes, forget
the 1000 oldest ones
- drop connection after sendaddr if too many connections, even if it's
an outbound one
- if maximum total connections are lower than maximum outbound
connections, active bootstrap provider mode
- in this mode, check all addresses received before announcing them
- so basically it only annouces those addresses it successfully
connected to
- I can't get the dynamic loading to work on OSX in frozen mode
- I think that if someone wants to build a frozen executable with custom
messagetypes modules, he can edit the file
- so now it lists the existing types manually (for frozen mode only)
- maxtotalconnections = maximum number of total full connections
(incoming + outgoing) the node will allow. Default 200 as it was.
- maxbootstrapconnections = number of additional (to total) connection
that will act in bootstrap mode, closing after sending the list of
addresses. Default 20 as it was.
- maxaddrperstreamsend = initial address list maximum size, per
participating stream. Default 500. Child streams get half. The
response is chunked into pieces of max. 1000 addresses as that's the
protocol limit.
- on OpenBSD, you can't have a socket that supports both IPv4 and IPv6.
This allows handling for this error, and then it will try IPv4 only,
just like for other similar errors.
- there were reports of errors in FreeBSD (I could only reproduce some)
and Gentoo without IPv4 support (I don't have a VM for testing ready)
- adds an exception handler for double task_done in case sender thread
has to close prematurely (I saw this triggered on FreeBSD 11)
- listening socket opening error handler was broken (triggered if you
can't open a socket with both IPv4 and IPv6 support)
- error handler for socket.accept. Reported on FreeBSD 10.3
- fixes#854
- TTL to chans shouldn't be too low so the UI gives a feedback
- warning when sending wouldn either require a lot of refactoring or
wouldn't have good usability
- don't do subprocess in SafeHTMLParser, it doesn't work in frozen mode
and an attempt to fix it would take too much refactoring and I'm not
even sure it would work
- instead, make it handle broken unicode correctly
- I think the previous reports of freezes were caused by trying to
interpret data as unicode, causing a crash
- it does about 1MB/s on my machine, so a timeout is not a big problem
- spec file for pyinstaller detects architecture (32 or 64bit)
- spec file uses os.path.join
- spec file creates and adds the list of messagetypes
- added MinGW/MSyS support in Makefile
- separate Makefile.msvc for MCVC
- bitmsghash.cpp minor adjustments to build also on MSVC/MinGW
- if frozen mode, messagetypes loads the list of files from a text file
generated during archive building rather than from a directory
- separate Makefile for BSD make
- auto-compile will detect BSD and pass the correct parameters to make
- C PoW builds on OpenBSD and detects number of cores
- "new" folder consistently appears in chans and "All accounts"
- "Sent" message list sorting fix
- When editing a label, keys.dat is saved and the lineEdit completer
is updated
- addressbook is updated when adding/deleting a new chan
- saveKnownNodes replaced the repeated pickle.dump
- with knownNodesLock instead of acquire/release
- outgoingSynSender had an unnecessary loop during shutdown causing
excessive CPU usage / GUI freezing
- networkDefaultProofOfWorkNonceTrialsPerByte and
networkDefaultPayloadLengthExtraBytes cyclic import fix
- PyBitmessage should launch now when there's no keys.dat
- rearranged code to reduce cyclic dependencies
- doCleanShutdown is separated in shutdown.py
- shared queues are separated in queues.py
- some default values were moved to defaults.py
- knownnodes partially moved to knownnodes.py
- complete the version and SSL handshake first, and only then feed
errors into the stream and close connection
- this allows more accurate error handling on both sides
- also the timeOffset error trigger is now more accurate, but requires
more nodes to upgrade
- version command sends list of all participating streams
- biginv sends lists of hosts for all streams the peer wants (plus
immediate children)
- objects will spread to all peers that advertise the associated stream
- please note these are just network subsystem adjustments, streams
aren't actually usable yet
