Blog Reading: The log - What every software engineer should know about real-time data's unifying abstraction

Link: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

Kafka is a message queue, a pub-sub system, an event sourcing tool, and a stream processing infrastructure, is a key part of many streaming distributed systems that requires streaming data. Its underlying idea, is to aggregate data from a distributed sources, to a unifying linear log structure.

The blog is from Kafka’s creator Jay Kreps when he was at LinkedIn, contemplating the log abstraction as a key part of any distributed systems. This is not Kafka’s design paper, implementation or a tutorial, but rather the process of brewing the idea that led to its birth, and I found it equally interesting. The following are my notes.

The link to Kafka paper: https://www.semanticscholar.org/paper/Kafka-%3A-a-Distributed-Messaging-System-for-Log-Kreps/9f948448e7a5f0cc94cd53656410face8b31b18a

Read More

Reading-Summary 2019-03

10 Breakthrough Technologies in 2019, by Bill Gates

Take a look at what Mr. Gates thinks are the greatest technology breakthroughs right now. The list might surprise you.

What happens when you click Play button on Netflix

How Netflix leverages AWS technologies to build world-scale, highly-availbile, fault-tolerant distributed video streaming system. ​

Lyft Case Study - Amazon Web Services

Lyft architecture evolution on AWS. ​

Compounding Knowledge

From Farnam Street – an interesting blog site I found recently.

Also on Farnam Street and its “mental models”: The Mental Model Fallacy. TL;DR: The so-called “mental models” from Farnam Street is not of much value when it’s from non-practitioners. And to learn businees, like basketball, swimming, etc., you’ll need to actually practice to learn the intricate knowledge that are not easily translated into writings.

Parsing Gigabytes of JSON per Second

Unfortunately I didn’t have time to finish reading this paper. But it’s good to learn the concept of branchless algorithms to fill the CPU pipeline and achieve amazing performance.

Paper Reading: Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center

Link to paper: https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf

Presentation: https://www.usenix.org/conference/nsdi11/mesos-platform-fine-grained-resource-sharing-data-center

Mesos is a cluster resource management software from UC Berkeley. Unlike many other frameworks already existed, Mesos is designed to support heterogeneous frameworks (Hadoop, MPI, etc) in the same cluster and share resources between them, by providing a thin layer that making resource offers to the framework schedulers, and delegate the scheduling decision to the frameworks themselves.

With this design, Mesos can achieve pretty good elasticity between frameworks, and letting frameworks choose their own resources results in better data locality.

Read More

Paper Reading: Understanding Real-World Concurrency Bugs in Go

Link: https://golangweekly.com/link/59972/b208593eda

A team from Penn State University and Purdue published their latest study on concurrency bugs found in Golang projects, namely large projects from Github: Docker and Kubernetes, two datacenter container systems, etcd, a distributedkey-value store system, gRPC, an RPC library, and CockroachDB and BoltDB. The authors searched commit histories of each repository to understand concurrency bug fixes for categorization and study.

TL;DR:

  • Go’s message-passing concurrency mechanism, something Go is proud of, isn’t as easy to use as it’s generally perceived. It creates just as many bugs, if not more, than shared-memory concurrency model.
  • Shared memory synchronization is still used more in Go projects.
  • Go’s built-in race and deadlock bug detection library still cannot catch all the bugs. There’s room for more improvements.

Read More

Paper Reading: Large-scale cluster management at Google with Borg

Link: https://ai.google/research/pubs/pub43438

About: Borg is Google’s large cluster workload scheduling and management system, which handles Google’s most service and batch job workloads on a cluster on scale of thousands of machines. It hides users from burdens of management of cluster, and provides high-availability features that handles failures.

The now very famous and popular open-source docker orchestration tool Kubernetes, is an open source successor to Borg, and keeps borrowing ideas from Borg (see kubernetes).

Read More

Reading-Summary 2019-01

Becoming a magician

If you want to become a ‘magician’, the ones that with intricate moves and skills to amaze the audience, you’ll need to adopt a growing mindset:

you cannot become a ‘magician’ with the same progress rate, or by simply imagining a better self: sometimes the way to changes involves a fundamental shift in how you see the world. And to achieve that you’ll need to observe fellow ‘magicians’, learn the difference, and make non-linear progresses.

Read More

Paper Reading 10-14: A Reconfigurable Fabric for Accelerating Large-Scale Datacenter Services

https://www.microsoft.com/en-us/research/publication/a-reconfigurable-fabric-for-accelerating-large-scale-datacenter-services/

This is one of the series of papers from Microsoft’s Project Catapult,
which studies leveraging reconfigurable devices (FPGA, etc.) to accelerate data center, from very specific
accelerating algorithms like page ranking for Bing search engine, to more sophisticated machine
learning frameworks like DNN.

This is one of their early publications, which introduces the basic design and implementation
of the FPGA accelerated datacenter. It covers the very fundamental details of all aspects of
server design, from hardware, network topology, FPGA core design, fault-tolerant cluster management
software design, workload scheduling algorithm, and etc..

Read More