I’m both surprised and sad at my ability to be distracted when writing my thesis. I’ve always had an attention span within the order of microseconds, but this is an all time low.

Here’s how my typical weekday seems to go off late:

• 8.30: Wake up.
• 8.30 – 9:00: Ponder about the mysteries of the universe whilst showering.
• 9:00 – 9:30: Have breakfast, and watch a full episode of the Simpsons or Family Guy.
• 10:00: Reach the lab. Setup laptop, mouse, keyboard, and extra monitor. Open window for some fresh air. Go grab coffee.
• 11:30: Done checking my mail, zero-unread-ing my feed of web comics, browsing through HN, Slashdot, and some other news sites (and a few “Oooh! Cat picture!” moments).
• 11:30 – 12:30: Lunch.
• 12:30 – 13:30: Post-lunch-procrastination (see 11:30).
• 13:30 – 14:30: Body has begun processing lunch, so feeling drowsy — Need. More. Coffee.
• 14:30: Open up editor for writing thesis. The “let’s settle this once and for all!” feeling surges through my body.
• 14:31 – 14:45: Check Facebook.
• (The above two repeats for a while)
• 15:30: “This is boring! I think I’ll do something that matters. Like code!”
• 15:31: Implement new feature! “Byzantine-fault-tolerant-key-value-based-scalable-elastic-hadoop-LTE-fabric-on-the-cloud!”
• 16:00: Realise that new feature broke all unit and system tests.
• 16:01: git reset –hard HEAD
• 16:01 – 16:02: Check Google+. Doesn’t take that long though because there’s nothing there.
• 16:30: “That’s it! I’m going to do more experiments! Nothing like graphs to make you feel like a scientist!” * challenge-accepted-rage-face *
• 16:45: Fire shell script and watch as the whole testbed dances to your bidding, cables filling with packets, WiFi waves flowing through space. You feel empowered, like you’re about to introduce a tear in the fabric of space.
• 16:47: Realise that you misconfigured everything.
• 16:49: Repeat experiment. Pretty sure it’s correct this time, so need to do something useful for an hour.
• 16:50: Continue with writing thesis.
• 17:00: Time for more coffee.
• 17:05: Back to desk, “What was I doing again?”.
• 17:06: Facebook time.
• 17:08: Booooored.
• 17:10: Write a few more lines of related work. “Previous work by Joe et al [10] has been known to suck”.
• 17:15: Discover some feature in text editor. Optimise key bindings for maximum productivity.
• 17:49: Experiment’s over. Fire SQL queries to extract data from measurements database, and pass it through gnuplot.
• 17:50: Add graphs to thesis. Defend weird results with “Proof-of-concept”.
• 18:00: “Woah! Is it warm here in the lab or what? Screw you guys! I’m going home so that I can write comfortably!”
• 18:30: At home. Have dinner with the company of Homer or Peter.
• 19:00: Feel sleepy. Idle around.
• 23:00: Sleep.
• Repeat.

And I wonder why the carrot’s never getting closer.

Just like an operating system abstracts away the complexities of the underlying hardware for a user-space application, the controller abstracts away the complexities of the network for OpenFlow applications. For this reason, the controller is often referred to as a “network operating system”. Applications have some API to talk to the network-OS, and it translates those APIs into OpenFlow commands that control the switches.

For my thesis, the plan for my architecture was to have two applications that provide different services to the network, that are expected to run simultaneously. Both of them collect information from the OpenFlow switches and some other framework specific agents situated at the edges of the network to make some optimisation type decisions. But as soon as I implemented one of the applications, it was clear that I had no straightforward way of ensuring that both my applications wouldn’t make decisions that counteract each other. Although I really don’t like the idea of doing this, the easiest way to solve this is to wrap both applications into one. And from the looks of it, this is a problem that hasn’t been solved yet.

Controllers like NOX and Onix make the assumption that only one OpenFlow application is running on a given network at any point of time. This is a reasonable assumption from a systems perspective. But what’s gotten me confused is how OpenFlow applications fit into the “SDN for enterprises” picture. I was under the impression that a network operator using a particular controller could choose between different 3rd party OpenFlow applications to handle different complexities with the network: a load balancing application from vendor A for the edge, a routing daemon application from vendor B, and so forth. While these are relatively orthogonal applications, it looks like it’s possible for two OpenFlow applications to make decisions and choices that adversely affect each other (leading to oscillations in switch state). Floodlight allows you to run multiple applications at the same time, but leaves it to the developer (or user?) to ensure that applications can safely co-exist with each other.

So again, if my observation isn’t mistaken, how do OpenFlow applications fit cleanly into the SDN ecosystem?  How can I manage my network using building blocks of applications from different vendors? Will I need to rely on OneBigApplianceFromBigBadVendor per network? Does this necessitate something analogous to per-process resource allocation as in traditional operating systems? I can see that FlowVisor style slicing is one way to go about it, but will that suffice?

So what *should* the network operating system do here? Let the applications run wild and fight it out? Or provide some mechanism to enforce policies between applications?

If I am indeed mistaken in my assumption, please do let me know what I’m missing here!

If you’re an ns-3 user, don’t miss this chance to share ideas and learn more about what researchers from around the globe are upto with the project. Don’t forget, there’s also the developers’ meeting the next day. Just add yourself to the wiki if you’re interesting in attending in person or remotely.

Now if only I can find a damn youth hostel in Desenzano…

$: modprobe tcp_probe [port=]
$: cat /proc/net/tcpprobe > logfile.out

…and you’ll get clear statistics about whatever TCP flows go through ‘port’ (and of all TCP flows if you specify the port as 0).

Earlier today, I ran into a situation where tcp_probe would stop logging flows after a couple of seconds, and it always seemed to be around the 10th second in an experiment I was trying (which involved iperf-ing over a wireless link). Some quick searches made it clear that others were encountering it as well, but no one really had a solution for them. Odd.

And what do you do when Google can’t help you find the answer? You look through the source code of course!

Within a few seconds of going through tcp_probe.c, I had my answer before me. Have a look at lines 47-49 and you’ll know what was wrong in my case.

static int full __read_mostly;
MODULE_PARM_DESC(full, "Full log (1=every ack packet received, 0=only cwnd changes)");
module_param(full, int, 0);

In short, I was on a wireless channel without any nodes apart from my wireless interface and the access point, and my congestion window size was getting maxed out around the 10th second. Since tcpprobe by default logs a packet only if the congestion window size changes, I couldn’t see any more packets of the flow I was looking at in /proc/net/tcpprobe.

So the solution?

$: modprobe tcp_probe <args> full=1

Note that you might want to look at the log buffer size parameter (bufsize) as well, because tcp_probe happily ignores packets once your log buffer is filled.

My work flow has mostly involved taking measurements to make sure that the problem I’m trying to solve is indeed a problem, trying to reason about the resulting measurements in terms of protocol behaviour/interleaving, and exploring what I can do to improve these measurements in a manner that is practical. As part of the work however, I needed some baseline measurements for TCP behaviour so that at the end of my thesis, I can say something like, “See? After what I did, we now have MOAR TCP AWESUM!!111″

Of course, TCP and wireless are two things that don’t really get along well. TCP sees packet loss and delays as “congestion”, and depending on the congestion avoidance algorithm, will back down upon hitting such a situation. However, wireless networks suck (and will continue to suck). You can make it suck less probably, but it’ll always stay above the suck-thresh. Interference in the physical channel will always be there (like memories of The Phantom Menace, which will haunt us forever). Furthermore, if there’s another station on the same channel communicating over a lower bit rate, your station’s performance can be degraded real bad (known as the 802.11 performance anomaly). This means that Murphy in the physical layer (only your first hop in most cases!), can end up causing TCP to believe that there is congestion at any of the many links that lead to the destination, and actually back down.

Linux uses TCP CUBIC as the default congestion control algorithm since kernel version 2.6.19. Without going into details of how the different TCP thresholds work in CUBIC, let’s look at how its congestion window (CWND) curve looks like.

The above is a measurement of the CWND size on a per packet basis, for a flow generated using iperf. The test network comprised of my laptop running the iperf client with an access point one hop away, bridging all traffic to a server behind it via a switched LAN. I’m sharing a wireless channel with a whole bunch of other nodes. The red line is the CWND size, and the green line is the slow-start threshold (ssthresh) value. As for some quick TCP background, the red line indicates how many bytes TCP is willing to keep outstanding, and the green line indicates the threshold at which TCP says, “Ok, I think I should not get too greedy now, and will back-out if I feel the network is congested”.

The graph above is a typical TCP CUBIC graph. When the congestion window (red) is less than the ssthresh value (green), the CWND increases in a concave manner (really fast). This can be seen by the near vertical spikes around seconds 3, 9, 35, and 37. When the CWND value is higher than the ssthresh value, TCP CUBIC plays safe, and tries to increase very slowly. For instance, look at how slowly the stretch between 22 and 30 is growing.

As you can see, TCP is detecting congestion multiple times, thanks to a noisy channel, and adjusts itself accordingly.

Next, we look at the curve in a channel where there is no contention at all (that is, no else on the channel except my wireless interface, and the access point I’m talking to).

Clearly there is no congestion here. The CWND value crosses the slow-start threshold very early (around the 1st second), plateaus since that point, and then goes convex around the 5th second. It then keeps probing for more bandwidth, finds it, and increases the window size steadily over time.

The premise

So we’re all familiar with Hadoop, right? It’s the little yellow elephant that provides an excellent platform for distributed computing, which is seeing rapid adoption by the industry, and involvement from major players like Yahoo!, Facebook and recently, Microsoft. Well, Hadoop and friends use the Hadoop Distributed File System (HDFS) as their underlying storage layer. Given the kind of jobs that are expected to run on top of it, HDFS is designed to store large files, and is optimised for throughput as opposed to latency.

HDFS is a single-master-server based distributed file system. Architecturally speaking, HDFS comprises of three important entities:

1. Clients, who read/write files from/to the filesystem.
2. Datanodes, which actually store the data (blocks) associated with the files.
3. The Namenode, which is a central server that stores all the metadata associated with the files, and blocks.

This division between metadata storage and data storage is important, because typical use cases of HDFS are data intensive, and not metadata intensive. That’s fine, but the problem is, if the Namenode crashes, the entire file system becomes inoperable because clients and Datanodes still need the metadata to do anything useful. Furthermore, since the Namenode maintains all the metadata only in memory, the number of files you can store on the filesystem is directly proportional to the amount of RAM the Namenode has. As if that’s not enough, the Namenode will be completely saturated under write intensive workloads, and will be unable to respond to even simple client side queries like “ls”. Have a look at Shvachko’s paper which describes these problems at great length and depth, on which we’ve based our work.

Long story short, the needs of the hour are:

• High availability for the Namenode, i.e, no single point of failure.
• Horizontal scalability for the Namenode, i.e, to handle heavier loads, one would need to only add more Namenodes to the system than having to upgrade a single Namenode’s hardware.

Our solution

In order to recover from crashes, the Namenode maintains a journal of all changes that it makes to the metadata. This pretty much involves logging every operation made to disk, and there is quite a huge piece of code related to this as well. However, the database community has been doing journaling, checkpointing and replicated storage since quite a while. So if you haven’t guessed our solution yet, here it is:

“Move all of the Namenode’s metadata storage into an in-memory, replicated, share-nothing distributed database.”

In short, Namenodes themselves are reduced to a stateless frontend to the database, and fetch state into memory only when required. This comes with the added advantage of being able to have multiple stateless Namenodes for the same filesystem namespace. We chose MySQL Cluster as our database because of its wide spread use and stability. So for the filesystem to scale to a larger number of files, one needs to add more MySQL Cluster Datanodes, thus moving the bottleneck from the Namenode’s RAM to the DB’s storage capacity. For the filesystem to handle heavier workloads, one needs to add only more Namenode machines and divide the load amongst them. Another interesting aspect is that if a single Namenode machine has to reboot, it needn’t fetch any state into memory and will be ready for action within a few seconds (although it still has to sync with Datanodes). Another advantage of our design is that the modifications will not affect the clients or Datanodes in anyway, except that we might need to find a way to divide the load among the Namenodes.

How we did it

We first dissected all the internal protocols being used in HDFS, i.e, the client-Namenode, Namenode-Datanode, and client-Datanode protocols. Next, we stripped out all the Namenode code that we didn’t need. This was pretty much the code related to journaling, checkpointing, the secondary Namenode and so forth.

Next, we identified the key data structures we needed to move to the DB. We picked the two most memory intensive data-structures to migrate first: the Inodes, and the Blocks.

Since we were heavily time constrained (three months to deliver the project and the report), we decided to focus on functional correctness first, and then optimise later. So the easiest course of action seemed to be to modify the lowest levels of the call chain, replacing reads/writes from/to memory with query, insert, update and delete operations on the DB. We developed two helper classes, one each for Inodes and Blocks, and interfaced with the DB through these methods. We used the ClusterJ connector to talk to MySQL. This obviously meant that we needed a flat row representation for Inodes and Blocks in the DB, and we had some other problems to think of as well on the way. How do we index Inodes? How do we index Blocks? What about Triplets?

All in all, we tackled the problem of scaling the Namenode with a set of design decisions which we later found to be consistent with Shvacko’s update paper on the Namenode’s scalability, except that he suggests using HBase as the DB.

Current status

• Multiple stateless Namenodes run merrily, which store Inode and Block related metadata in MySQL Cluster. As a validation test, Clients can do an “ls” query to any Namenode and see a consistent view of the filesystem regardless of which Namenode updated the DB with the content.
• We’re trying to ensure functional correctness using the HDFS unit tests. We got the most important ones to pass, and decided to keep some more bug fixing until later because we needed to evaluate the system as part of the course.
• We’ve been evaluating the system using the Synthetic Load Generator. Horizontal scalability has been clearly achieved; adding more Namenodes improves the average number of operations per second for different work loads. With write intensive work loads, the scalability is linear in terms of total operations/sec that are executed.

Current limitations

Obviously, our work isn’t rainbows and sunshine; there’s a long way to go. Here’s what we don’t have yet and are currently addressing:

• Performance improvements. With a single load-generator thread throwing requests at the Namenode, we’re within a 10th of the original Namenode’s performance because read/writes from/to memory now go over a network to a database cluster (which is OK, I guess). But with more LoadGen threads, we’re experiencing a hefty bottleneck, which I’ll describe in the next point.
• The Namenode isn’t fully stateless yet. The most important data structures we’re yet to move are the DatanodeDescriptor entities and the file leases. There’ll surely be more, but these are the most crucial ones. Once full statelessness is achieved, we can eliminate the read-write locks in the code which are absolutely not needed any more in our implementation (the Namenode currently uses a multiple-reader-single-writer concurrency model). Profiling experiments indicated that the Namenode spends around 70% of its time waiting to acquire write locks. If we keep the Namenode fully stateless, we can wrap FSNamesystem operations into Database transactions which can be batched, and let MySQL cluster handle serialisability for us (which can handle write-heavy transactions really well). We can even break away from the single-writer model that the Namenode currently uses. Will this lead to a higher throughput for write operations than the original Namenode? Maybe.
• Clients and Datanodes have to be statically partitioned as of now (it sufficed for our evaluations). We need a way for them to pick a random Namenode to perform their operations with.

Talk is cheap, show me the code!

The code is publicly available here for thy scrutiny. You’ll also need to have a MySQL cluster setup in order to test this (we have a hard coded set of defaults in DBConnector.java which you can politely ignore).  Here’s our presentation on it as well. We’ve dubbed our prototype implementation KTHFS (because we’re students at KTH, but yes, no points for creativity on that one).

Future work

As an academic obligation, here’s future work (and no I’m not going to write stuff we’ll never do).

One member (not me) from the team will be continuing this work as part of his Masters thesis, and plans to address the above mentioned limitations as part of his work. I’ll try to contribute as well during my free time (what are weekends for anyway?). Let’s see how this goes.

Given that there’s hardly anything interesting about rocks and the occasional flash of light in a metro tunnel, I think there’s a lot that can be done to make the metro experience better. Here’s one idea I have to solve this international crisis which is oh-so-more-important than anything else on this planet. I call it, metro cartoons (the animated ones).

The basic idea is to insert cartoon panels between the tunnel walls and the tracks. Since we know the approximate speed at which the trains move, you can estimate the minimum spacing between the panels that would be required to provide a reasonable frame rate required for the animation. I’m sure artists understand these aspects better (thus, I don’t), and can come up with the right kind of drawings for this setting.

What say? I’m pretty sure someone must have tried this already, so let me know if you’ve heard of any such attempt.

Lagom (pronounced [ˈlɑ̀ːɡɔm]) is a Swedish word with no direct English equivalent, meaning “just the right amount”. – Wikipedia

If you’re in Sweden and there’s one word you should know, it would be ‘lagom’, a word that defines the essence of everything that is Swedish.

More than two months into Stockholm and I must say it’s been pretty lagom so far. The city itself is beautiful, with the only downside being that everything is at least twice as expensive as in Lisbon. Maybe that’s what you get for everything around here being so perfect and systematic. And unlike Lisbon, you actually have easy access to a *large* collection of beers, which is definitely a plus.

It’s autumn here right now, and the scenery outside is fabulous. Sadly though, the weather in Sweden is not-so-lagom. Temperatures are already hovering around 5 degrees celsius, and for an Indian like me, this is pretty cold in itself (heck, it never gets lower than 20 in my town back home). The funny thing is, this weather is cute for the Swedes. Let’s see how winter ends up looking like.

Meanwhile, here are some pictures I took from Stockholm over the last two months.

The workshop serves as an annual gathering of ns-3 users and developers to share ideas, and brainstorm over future directions for the project. We’re inviting conference style full paper submissions, which will be made available in the ACM and EU digital proceedings. Furthermore, we’re also hosting an interactive session comprising of demos and posters. So start working on your papers and proposals as soon as possible! If you have any queries, feel free to contact the workshop chairs.

Like previous editions of the workshop, we also plan to conduct a day long ns-3 developers’ meeting around the same time. More details will be out soon. Keep an eye on our mailing lists and website to not miss any announcements.

Important dates

Papers submission deadline : December 2, 2011
Notification of acceptance : January 20, 2012
Camera-ready deadline : February 10, 2012
Demos and posters proposal deadline : February 24, 2012
Workshop in Sirmione : March 23, 2012
Developers’ meeting : TBA