Separate user data and network data into two SQLite databases #250

New Issue

2013-06-24T01:57:27+02:00

DivineOmega commented

2013-06-24 01:57:27 +02:00

(Migrated from github.com)

The messages.dat SQLite database contains a lot of information, but most of it, in terms of storage during general use, is network data (the encrypted messages and public keys).

I believe we should separate user data from network data. This would allow users to back up their inbox, sent items and address book separately from the network data. It could also potentially improve performance in the future, by allowing the user and network data to be managed by two separate threads.

My thoughts are as follows:

messages.dat contains the following tables:

inbox
sent
subscriptions
addressbook
blacklist
whitelist
settings

network.dat contains the following tables:

inventory
pubkeys
knownnodes
settings

The settings table needs to be included in both to allow independant database version numbers and last vacuum timestamps.

I'm not certain about the 'pubkeys' table, due to 'usedpersonally' field.

I quite like the idea of two separate SQL threads being used, one per database.

What are people's thoughts?

The messages.dat SQLite database contains a lot of information, but most of it, in terms of storage during general use, is network data (the encrypted messages and public keys). I believe we should separate user data from network data. This would allow users to back up their inbox, sent items and address book separately from the network data. It could also potentially improve performance in the future, by allowing the user and network data to be managed by two separate threads. My thoughts are as follows: _messages.dat_ contains the following tables: - inbox - sent - subscriptions - addressbook - blacklist - whitelist - settings _network.dat_ contains the following tables: - inventory - pubkeys - knownnodes - settings The settings table needs to be included in both to allow independant database version numbers and last vacuum timestamps. I'm not certain about the 'pubkeys' table, due to 'usedpersonally' field. I quite like the idea of two separate SQL threads being used, one per database. What are people's thoughts?

ghost commented

2013-06-24 10:59:33 +02:00

(Migrated from github.com)

Agree.

r14c commented

2013-06-26 01:32:47 +02:00

(Migrated from github.com)

This. Especially as people start sending more messages, a single SQLite database is going to turn into a major bottleneck.

linkerlin commented

2013-06-26 05:39:48 +02:00

(Migrated from github.com)

Agree!
Maybe we can add some NoSQL database support.
Maybe something like mongodb?

Agree! Maybe we can add some NoSQL database support. Maybe something like mongodb?

r14c commented

2013-06-26 08:49:22 +02:00

(Migrated from github.com)

A document-based datastore could be nice for unstructured things like conversations but there aren't a lot of options for embedded NoSQL in python. (shelve could work, I suppose)

A document-based datastore could be nice for unstructured things like conversations but there aren't a lot of options for embedded NoSQL in python. ([`shelve`](http://docs.python.org/2/library/shelve.html) could work, I suppose)

This repo is archived. You cannot comment on issues.

1 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Bitmessage/PyBitmessage-2025-03-02#250