[Thali-talk] A trivial eventually consistent framework for acls, quotas and notifications

[Yaron Goland]

NOTE: This mail is going to the public mailing list


I think the problem of how to run user specified code to update ACLs, quotas and notifications in response to mutations in an eventually consistent way is really easy. So easy I'm genuinely embarrassed it didn't occur to me sooner.

Sequence IDs. D'uh!

What we do is create a framework that runs in the background. It will start itself when the app starts. It will hook into the changes feed of the DB it is watching and trigger whenever there are changes. It can even hook into a timer just in case anything goes funny with the changes feed. The framework will record the last sequence ID it has processed from the DB it is watching for mutations on. It will then call the changes feed to get changes from that sequence ID. It will then start feeding those changes to the user supplied functions. When the user functions return for a particular sequence ID then we will persist that ID as the latest ID and move on to the next ID.

If we crash, no problem, at worst we will start off behind an ID we processed and so can re-process IDs we've already done in the past. That should be o.k. All ACL, quota and notification commands needs to be idempotent. So sending the same changes to the user functions multiple times shouldn't leave anything in a bad state.

This is super easy to implement and guarantees that we will eventually process all changes.

    Thoughts?

            Thanks (and sorry for being so thick that this obvious solution didn't slap me in the face earlier)

                Yaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist10.pair.net/pipermail/thali-talk/attachments/20150513/b2d410e3/attachment.html>