Proposal: DDB storage layer

[edholland]

Hi,

It would be nice to have dynamoDB as a storage layer. I have implemented a basic version of this within storage.py. Please could someone take a look to see if this is something the project would be willing to incorporate - and if i have a sensible approach in implementation. I dont believe this is "production ready" quality code yet, but i think . early feedback is useful!

[ekzhu]

This is interesting. Why Dynamo DB?

[edholland]

Its an awful lot cheaper than the redis instances we're using at the moment - and there's practically zero management overhead to boot!

[ekzhu]

A few things to consider:

1. LSH's query method probes multiple hash tables, is this going to be efficient for DynamoDB (think network latency) comparing to an in-memory Redis instance on the same node as the LSH?
2. What is the memory footprint of your index and why not usepickle package to save the index to the disk periodically?

[edholland]

1. Yeah its not overly performant - overall requests take about 500 millisec. However this is "good enough" for our use-case
2. Concurrency mostly - We're going to be running this within AWS Lambda functions so there is no shared filesystem. Seems easiest to move the state out of application layer into the database layer

[ekzhu]

I see. This is quite interesting. Please give us some time.

[amirouche]

Very interesting. @edholland can you recommend a ddb emulator to help with the dev setup, and CI?