invbloom - New command for reduction of bandwidth #1404

New Issue

2018-12-08T16:32:59+01:00

PeterSurda commented

2018-12-08 16:32:59 +01:00

(Migrated from github.com)

Reason

After the version/verack handshake, Bitmessage sends a list of all inventory vectors to the other side. This causes a huge amount of bandwidth wastage, in particular for nodes that accept inbound connections. The proposal is to add a new command invbloom which will reduce this step in synchronisation.

This issue will be updated with more specifics as time goes on.

Data

After the command, there will be some metadata, and data. The bloom filter will use Murmurhash v3, just like Bitcoin's bloom filters. It's implemented in pybloomfiltermmap and it's very fast.

When to use

The support for this command will be indicated by a new bitfield in the version command. If both sides support this bitfield, instead of sending the list of all inventory vectors (referred to as "biginv" in PyBitmessage source code), the two nodes will send an invbloom.

Since a bloom filter allows for false positives, the invbloom command should be issued periodically to reduce the probability that an object goes missing. The delay should use a Poisson probability and pick a random outbound connection (similarly to how Dandelion works).

How to handle

Upon receiving invbloom, the node can send a list of inventory vectors that aren't in the bloom filter. This allows to reuse the rest of the protocol with no change.

Further development

Additional improvements can be added in the future with further protocol changes.

Anonymity

A different bloom filter should be used for each connection to prevent tracking of a node. I did tests with pybloomfilter and it looks like it isn't deterministic, so it probably isn't necessary to add random data to it.

## Reason After the `version`/`verack` handshake, Bitmessage sends a list of all inventory vectors to the other side. This causes a huge amount of bandwidth wastage, in particular for nodes that accept inbound connections. The proposal is to add a new command `invbloom` which will reduce this step in synchronisation. This issue will be updated with more specifics as time goes on. ## Data After the command, there will be some metadata, and data. The bloom filter will use Murmurhash v3, just like Bitcoin's bloom filters. It's implemented in pybloomfiltermmap and it's very fast. ## When to use The support for this command will be indicated by a new bitfield in the `version` command. If both sides support this bitfield, instead of sending the list of all inventory vectors (referred to as "biginv" in PyBitmessage source code), the two nodes will send an `invbloom`. Since a bloom filter allows for false positives, the `invbloom` command should be issued periodically to reduce the probability that an object goes missing. The delay should use a Poisson probability and pick a random outbound connection (similarly to how Dandelion works). ## How to handle Upon receiving `invbloom`, the node can send a list of inventory vectors that aren't in the bloom filter. This allows to reuse the rest of the protocol with no change. ## Further development Additional improvements can be added in the future with further protocol changes. ## Anonymity A different bloom filter should be used for each connection to prevent tracking of a node. I did tests with pybloomfilter and it looks like it isn't deterministic, so it probably isn't necessary to add random data to it.

👍 2

hub229ox commented

2018-12-10 12:03:42 +01:00

(Migrated from github.com)

sounds very good as long as precautions for anonymity are met like you describe :-) 🥇

sounds very good as long as precautions for anonymity are met like you describe :-) :1st_place_medal:

👍 2

g1itch commented

2019-01-16 12:07:25 +01:00

(Migrated from github.com)

The original pybloomfiltermmap code seems abandoned, last commit was in 2015. Also it probably contains serious bugs, e.g. https://github.com/axiak/pybloomfiltermmap/issues/84.

The original `pybloomfiltermmap` code seems abandoned, last commit was in 2015. Also it probably contains serious bugs, e.g. https://github.com/axiak/pybloomfiltermmap/issues/84.

g1itch commented

2019-01-17 16:19:12 +01:00

(Migrated from github.com)

Another option I found is pybloof

Another option I found is [pybloof](../../../jhgg/pybloof)

g1itch commented

2019-02-07 15:45:23 +01:00

(Migrated from github.com)

Also I started to write tests for any bloom filter implementation: 9298b61

This repo is archived. You cannot comment on issues.