MIT Distributed Systems Coursework

I started from the course website (containing the problems and some advice) and thence solved the first two labs, via practical heuristic and Googling–based learning; I streamed the entire process on Twitch.

This is a bit of a mess to write up, as most of the interesting stuff is buried in 100h of videos. But I have my resultant work up on GitHub. I’ll try to fix this page up in future. I also need to move the video archive onto YouTube, because Twitch is more awful to navigate than expected.

Course website | My work | Stream page | Video archive

Why Stream?

• I wanted to form a stronger, more realtime connection between my thought process and my speech, which is of course useful for interviews.

• I have an archive of videos of myself solving problems step-by-step, so a visitor could pick parts at random to find out how I do things.

I: MapReduce

(26/5–31/5). This is a pretty basic project whose main challenge was understanding the intuition behind the abstraction into Map and Reduce tasks via case studies.

II: Raft

(7/6–12/6, then 18/7–10/9). This is a much harder project, implementing the complex Raft consensus algorithm largely from scratch (see here for an introductory visual demo of how it works). I was happy to pass each of MIT’s submission tests at least 199/200 times (including some heavy reliability tests, featuring churns and weird edge-cases):

Result: 200 runs of the full test suite executed in 162.4–182.7s each. All tests passed in each run except B2, B3 and C6, which each failed once in separate runs, and C7 and C8, which didn’t run once (after C6 failed).

Challenges:

• Learning to print-debug and introspect into a program comprising multiple peers running phase-dependent sequences of routines and sending/handling messages between each other.
• Developing conventions for cleanly transitioning between phases (leader/follower/candidate) and for (un)locking state variables via mutex.
• Getting a theoretical hold on so many conceptual processes happening in the same algorithm.