[Thali-talk] Yaron's Weekly Update - 12/29/2014

[Yaron Goland]

[Note: Was OOF for 2 days last week]

Last Week

*        TL;DR - Managed to spend about 1.5 days on the node-gyp work (mostly using node.js and Pouch for different kinds of queries) before getting side tracked by a PouchDB data corruption. It took me a day to investigate it and file bugs which thankfully got some of the issues fixed by the PouchDB folks in the person of Nolan Lawson.

*        Monday - Had an internal call. Got my node-gyp usage code to successfully talk to GitHub to pull down gyp data but their developer api is pretty seriously rate limited. But I decided to first check out just how hard it would be to download, well, everything. So I wrote code to issue HEAD requests against the latest TGZ for all the NPMs. I took a sample of around 5000 files and their total size as reported by HEAD would be 900 megs. Considering that there are over 100,000 packages I'm thinking no. So I cleaned up the GitHub code and worked through that.

*        Tuesday - I noticed that one can optional include an entry in package.json that states 'gypfile'. I wrote a quick query and out of 113,113 records it found 2000 or so that used that flag. This then brought up the question of - if package X is reported as having X1 downloads and X1 is dependent on Y then shouldn't Y's downloads Y1 > X1? I ran some tests and the answer seems to be no. Socket.io has 763,740 downloads/month but its dependency engine.io has 204,743 and it's dependency WS has 1,011,227. So how can engine.io have fewer downloads than socket.io? I ran the same test on PouchDB and its dependency leveldown and there the numbers at least made sense. But for now I'm not sure I can really rely on the download numbers. I wrote some code to count the downloads of all the files that depend on ws and leveldown and found their totals to be less than the totals for ws and leveldown. That makes sense. But what about engine.io? Engine.io has 204,743 downloads but the 52 projects that depend on it has 778,235 downloads. Huh?!?! Unfortunately then my day blew up because my leveldb file under PouchDB is corrupted! That's not supposed to be possible!! So I spent the rest of the day investigating that.

*        Wednesday - VACATION!

*        Thursday - I spent the entire day on a single function in a single file in PouchDB, the bulkDocs function in levedb.js. Wow... it's seriously complicated. Just summarizing what it's doing took me several pages of notes. Basically it's trying to do bulk updates but with multiple interlocking mechanisms to let it make progress asynchronous and I found at least four different places where if the app crashes at the wrong time data will be corrupted. I filed a bug on all this and started to write a much simpler and more robust version of the file but was beaten to the punch by Nolan Lawson who batched together the updates and stuck a write lock on everything, including reads.

*        Friday - VACATION!

Next Week

*        Need to investigate how FireChat does mesh on Android.

*        Maybe finally finish the node-gyp work without finding another show stopping bug in PouchDB?

*        Finally get started on TLS support in node.js?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist10.pair.net/pipermail/thali-talk/attachments/20141229/d5d79590/attachment.html>