[Thali-talk] Yaron's Weekly Update - 12/8/2014

[Yaron Goland]

Last Week

*        TL;DR - Got the perf tests working needed to repeat-ably show that my parallel improvement to PouchDB makes a big difference working.

*        Monday - A metric ton of email and a 1.5 hour team meeting. But most of the day was spent trying to figure out why the latest debug features in PouchDB weren't working for me. It turns out that PouchDB uses a different debug activation mechanism than Request.js and this confused the heck out of me.

*        Tuesday - I spent a bunch of time in the morning trying to get the skimdb mirror working with PouchDB. It took me a while to realize that the skimdb mirror server is having an especially awful day which is why connections were timing out everywhere. I then switched to Nolan Lawson's mirror of the skimdb mirror which worked better but still had time outs. Eventually I realized that there is basically no retry logic in request.js which is why I was seeing so many failures. So I edited PouchDB to use requestretry (a project add retry logic to request.js) and that worked pretty well. I also discovered the forever agent which is supposed to do a better job handling persistent connections and threw that in too.

*        Wednesday - Had a two hour long call with a partner. Then did a bunch of perf testing on my new code that amongst other things showed that the forever agent is useless. But the bulk of the afternoon was spent in an internal meeting.

*        Thursday - After talks with the PouchDB folks they made it clear they would only take my parallel fix if there was some kind of perf test to go along with it. So I spent most of the day understanding PouchDB's perf framework and getting a test going. But it didn't show any perf difference between my parallel code and the existing serial code! Confused I instead implemented a test in the perf framework against Nolan's mirror of the skimdb mirror which showed the perf difference.

*        Friday - The PouchDB folks were clear that they didn't want a perf test that depended on an external server. So instead I hacked in a proxy that would slow down requests to simulate the network. This took longer than you might think because I kept running into weird bugs. One proxy (hoxyproxy) I used opens a listener if you just 'require' it which means that the tests never completed because node.js didn't exit. Another proxy (netmorphic), which ran at the TCP level, only put in a delay on a TCP connection but not on individual messages. Finally I found a proxy (http-proxy) that 'just worked' that put in a delay at the HTTP message level. Not ideal, but it worked. I then figured out that the reason I wasn't seeing any perf difference between the serial and parallel code is because I only had generation 1 docs and so they were downloaded via all doc so each batch had effectively 1 HTTP request. And since batches are run in serial there is no perf difference. So I added in gen 2 docs and that showed the perf benefit nicely.

Next Week

*        Bunch of calls with potential customers for our project.

*        Clean up perf code for PouchDB.

*        Test out PouchDB's replication level retry logic versus requestretry to see how big a perf improvement requestretry provides.

*        Get the PouchDB perf framework checked in

*        Try to get the parallel change in PouchDB checked in

*        Finally run an analysis on the NPM database to figure out how much native Node.js add-ons matter

*        Start working on getting us an Android story
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist10.pair.net/pipermail/thali-talk/attachments/20141208/e5fefeaa/attachment.html>