Put uploads into a separate thread #1415

2018-12-18T22:54:32+01:00

PeterSurda commented

2018-12-18 22:54:32 +01:00

(Migrated from github.com)

instead of being processed in the ReceiveQueue thread, uploads are now done
in a dedicated thread. Only the parsing is done in ReceiveQueue thread.
the UploadThread is modelled based on the DownloadThred, but simpler.
it checks for intersection attack, eliminates duplicates and restricts the
write buffer size to 2MB (may still grow slightly higher if too many big
objects are requested, but the absolute limit appears to be about 4.5MB in the
worst case scenario).
the restriction of the write buffer may cause some upload throttling (to
about 2MB per second per connection), but can be optimised later
fixes #1414
I haven't actually tested it yet, I'll test it as the PR is being checked and reviewed

- instead of being processed in the ReceiveQueue thread, uploads are now done in a dedicated thread. Only the parsing is done in ReceiveQueue thread. - the UploadThread is modelled based on the DownloadThred, but simpler. - it checks for intersection attack, eliminates duplicates and restricts the write buffer size to 2MB (may still grow slightly higher if too many big objects are requested, but the absolute limit appears to be about 4.5MB in the worst case scenario). - the restriction of the write buffer may cause some upload throttling (to about 2MB per second per connection), but can be optimised later - fixes #1414 - I haven't actually tested it yet, I'll test it as the PR is being checked and reviewed

PeterSurda commented

2018-12-18 23:35:11 +01:00

(Migrated from github.com)

I changed the checks to ignore import position errors in bitmessagemain.py, as I don't know how else to work around this. I can revert it once this is merged.

PeterSurda commented

2018-12-18 23:40:02 +01:00

(Migrated from github.com)

Testing ...
Log says it's uploading objects
Throttling seems to work
Seems to work as expected on multiple systems, memory usage way down

Testing ... Log says it's uploading objects Throttling seems to work Seems to work as expected on multiple systems, memory usage way down

g1itch (Migrated from github.com) approved these changes 2018-12-20 13:53:59 +01:00

g1itch (Migrated from github.com) left a comment

Looks good.

src/network/bmproto.py

g1itch (Migrated from github.com) commented

2018-12-20 13:50:57 +01:00

Why not

for i in items:
    self.pendingUpload[str(i)] = now

?

Why not ```python for i in items: self.pendingUpload[str(i)] = now ``` ?

PeterSurda (Migrated from github.com) reviewed 2018-12-20 15:02:48 +01:00

src/network/bmproto.py

PeterSurda (Migrated from github.com) commented

2018-12-20 15:02:48 +01:00

Leftover from old code, you're right, it can be rewritten for maybe a bit of performance increase.

This repo is archived. You cannot comment on pull requests.

No reviewers

g1itch

No Label

No Milestone

No project

No Assignees

1 Participants

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Bitmessage/PyBitmessage-2024-08-21#1415