Reading Summary 2019-08

Cassandra Time Series Bucketing

How to model timeseries data with Cassandra.

Simple GoRPC

The best way to understand something, is to build one yourself. This tutorial covers basic network programming in Go, struct design and the usage of reflect package.

Optimizing M3: How Uber Halved Our Metrics Ingestion Latency by Forking the Go Compiler

A great experience sharing blog on how to debug a performance issue in their services. And with profiling and analysis tools, the Uber team was able to pinpoint this issue in worker pool and goroutine stack allocation, and then they forked the Go compiler to prove it’s a regression in the Go compiler. A very nice read and analysis process.

Book: Programming Models for Distributed Computation

A programming book on topics in distributed computation, from teaching experience in distributed system course, from Northeastern University.

Spotify Engineering Culture

A very nice engineering blog from 2014. A excellent overview of Spotify culture, and an introduction on how to build the “agile” team.

How We Helped Our Reporters Learn to Love Spreadsheets

NYTimes has released its in-house course to teach journalists data science. Journalism can also benefit from a little coding/data analytics skills.

Paper Reading: Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center

Link to paper: https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf

Presentation: https://www.usenix.org/conference/nsdi11/mesos-platform-fine-grained-resource-sharing-data-center

Mesos is a cluster resource management software from UC Berkeley. Unlike many other frameworks already existed, Mesos is designed to support heterogeneous frameworks (Hadoop, MPI, etc) in the same cluster and share resources between them, by providing a thin layer that making resource offers to the framework schedulers, and delegate the scheduling decision to the frameworks themselves.

With this design, Mesos can achieve pretty good elasticity between frameworks, and letting frameworks choose their own resources results in better data locality.

Read More

Paper Reading: Large-scale cluster management at Google with Borg

Link: https://ai.google/research/pubs/pub43438

About: Borg is Google’s large cluster workload scheduling and management system, which handles Google’s most service and batch job workloads on a cluster on scale of thousands of machines. It hides users from burdens of management of cluster, and provides high-availability features that handles failures.

The now very famous and popular open-source docker orchestration tool Kubernetes, is an open source successor to Borg, and keeps borrowing ideas from Borg (see kubernetes).

Read More