[Thali-talk] Yaron's Weekly Update - 11/17/2014

[Yaron Goland]

Last Week:

*        TL;DR - Spent 3 days finding a bad perf bug in PouchDB replication that I will fix next week.

*        Monday - Looking for customers

*        Tuesday - Setting up a huge list of work items in Microsoft project and identifying basic dependencies to give us a rough estimate for how big this 'thing' is and how many people we could work on it in parallel. 100+ work items. This contains some Microsoft specific stuff so I can't release it publicly yet.

*        Wednesday - I was trying to figure out how to proxy SSL requests in PouchDB via Fiddler so I could debug some awful perf I was seeing. PouchDB is having an allergic reaction to HTTPS tunneling but I haven't figure out why.

o   I tried to convince request.js (the library that PouchDB uses for HTTP requests) to output debug info but setting require('request').debug=true didn't actually work.

o   So finally I gave up and just used what I thought was Fauxton but was actually Futon to replicate the NPM data down and that ran really fast. Which scared me because why would Fauxton be fast and PouchDB directly be slow? Well the correct answer is - because I wasn't using Fauxton, I was using Futon.

o   The reason this screw up happened is because I don't know either one's UX and it turns out there is a bug (that I filed with OpenTech) on Windows that node.js will happily claim a port that is already taken by a Windows service and won't throw an error even though it doesn't actually have the port! So what was happening is that Futon and Fauxton want to run on the same port, Futon already had it but due to the bug it looked like Fauxton was running on it (since node.js didn't throw when I ran Fauxton as it should have). Anyone who knows the UX couldn't possibly be fooled but I didn't so there you go. Sigh.

o   Once I figured this out I was even more scared. Futon runs 45 times faster than Fauxton/PouchDB. That's insane and totally unacceptable! I must investigate!

o   I decided to do some local perf experiments using Fiddler but again, couldn't get SSL to work. So I replicated data locally using HTTP. This worked just fine with CouchDB (which I was using as a reference) who supports the proxy argument in replication requests but PouchDB seemed broken.

o   The issue turned out to be that express-pouchDB doesn't support the proxy argument in replication requests. Sigh. So I fixed that and submitted a PR to the express-pouchdb folks.

o   Today's bugs were 2940<https://github.com/pouchdb/pouchdb/issues/2940> about perf and the proxy pull request<https://github.com/pouchdb/express-pouchdb/pull/125> for express-pouchdb. I also privately filed a bug with MS OpenTech on the node.js behavior on claiming ports that are already claimed.

*        Thursday - Now I could actually see the CouchDB vs PouchDB network traces there are a few things that jump out. First, PouchDB is more chatty but not outrageously so. Second, that PouchDB is completely serialized! It does NOTHING in parallel! So I spent the day climbing through the PouchDB replication code to try and understand what the heck was going on.

o   In looking through the code I didn't see what I should have seen and so mistakenly thought the problem was request.js (the library PouchDB uses for HTTP requests in node.js). This led me to get an updated version of request.js (since PouchDB uses a very old one) and ran some experiments to see how it behaves. The answer is that it's default behavior is to make a bunch of parallel requests.

o   So I updated PouchDB with a new version of request.js and tried to run various experiments all of which still ran serialized!

o   Meanwhile I wanted to really learn how the heck do you test something like request.js? So I spent a big chunk of the day trying to figure out how to use chai to test event emitters. This led to a fun feature request<https://github.com/domenic/chai-as-promised/issues/71> (which we agreed to close) with Domenic Denicola about how to use Chai to test event emitters.

*        Friday - On IRC with the PouchDB folks they pointed out that there is a known bug in Node.js 0.10 that it aggressively closes outgoing connections and maybe that's why all of PouchDB's requests are serialized? But I investigated the bug and the key is that the connections are closed if there is nothing waiting on the request stack. But in my tests I was replicating 10s of thousands of records so the stack was pretty seriously full. So this didn't ring true for me.

o   Eventually after walking through the PouchDB code really carefully I found what I believe is the actual issue. There is a call to getNextDoc() which is 100% serialized! It uses a promise so that it makes a GET request and won't make the next GET request until the first one finishes. Hence why we are completely serialized! I should have seen this yesterday when I reviewed the code but I missed it. :( I just didn't understand the code well enough to appreciate the consequences of that single line.

o   I talked with the PouchDB folks on IRC and they agreed about the bug and asked me to submit a fix. But to do a fix I had to switch my dev environment to Linux because I can't build PouchDB on Windows. And this then led to an investigation of testing and other issues on Linux that took the rest of the day.

Next Week:

*        Submit PR for the perf bug in PouchDB

*        Actually download the NPM records using the fixed PouchDB and figure out if we care about native add-ons by analyzing the data

*        Work on Android node.js support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist10.pair.net/pipermail/thali-talk/attachments/20141117/20e17fff/attachment-0001.html>