Scaling Selene Wallet
Testing has shown a number of inefficiencies in my approach to handling sync and db operations. I did anticipate some of these issues ahead of time, but preferred to employ the "Premature Optimization" and "YAGNI" mantras instead, in the interest of shipping a decent wallet soon.
Some things to consider since the database is stored in-memory:
- raw transaction data (i.e. 'transactions' table) should probably live in its own db file, and should also be pruned (pruning strategy TBD)
- each wallet should get its own file ('wallets', 'addresses', 'address_transactions', and 'address_utxos' tables)
- if each wallet gets its own file, block data ('blockchain' table) should also get its own file so data is not duplicated. Can also be pruned
- the entire sync process can probably be moved to a webworker, hopefully with minimal complication
- it turns out the SQL wasm operations are actually "somewhat" expensive, so reconsider how redux interacts with DB
We should also consider how much data is required to download and when; importing a large (15k change addresses) wallet currently chokes hard, but I haven't implemented any optimizations at all. One test downloaded 1GB of data which is unacceptable for some users. We should respect them by limiting wallet sync in low/capped-bandwidth environments. Also, downloading transaction blobs for txes that we actually don't care about in order to rebuild ancient transaction history is silly. We can lazy-load additional historical transactions if the user actually requests them.
Investigation of the Electron Cash source code shows that they take a similar sync approach to the one I implemented. This makes sense, since it was derived logically from the methods available as part of the Electrum Cash protocol. However, that code takes advantage of multithreading in order to not lock the UI, and also has a specialized patch to retire change addresses with hd_index < current_index - 1000 by default. (With this in mind, I shouldn't feel bad about the fact that importing a wallet with 15k change addresses pretty much crashes Selene, especially with zero optimizations applied!)
It also has the luxury of being able to assume a high-connectivity environment. We must be able to gracefully degrade and recover functionality in a potentially unstable mobile connection environment.
Electrum subscription management can also be improved a bit. We realistically don't need to /actually/ subscribe to the change addresses, especially the ones that EC would consider "retired." We usually only need to check their current address state once, and fill in transaction history once. We're already setting up the address subscriptions on walletBoot
which will automatically trigger the sync process, so we should also just do a quick check on change addresses there.