I finally took my network monitoring work up a level when Gaur sir asked me to go over to the BSNL head office and set up a MRTG server for the NIB (National Internet Backbone). What an opportunity! MRTG is Tobi Oetiker's (the man behind RRD Tools and Smokeping) multi router traffic grapher which is basically a tool to tell how much bandwidth your routers are using. This seems crucial for an ISP like BSNL who keep getting calls from their customers who b**** about not getting enough bandwidth. And it's really important that they have such a set up since all broadband connections to the state of Rajasthan originate from the Jaipur node where I was asked to work. I'd decided to deploy my ever favourite linux distro, Debian Etch on their server so that apt-get would make quick work out of the task at hand. Here's the roadmap I'd laid down for myself for the job:

1. Install a base Debian Etch-4.0.r1 with a netinstall with a fairly large and separate /var partition since all the .png files generated by MRTG are dumped into /var.

2. Since security is an important issue at such a level, a dist-upgrade is necessary which is meant to patch the OpenSSL security flaw that was found in Debian a couple of weeks back.

3. Use apt-get install apache2 to set up a webserver and configure it as needed.

4. Use apt-get install mrtg snmpd to set up MRTG and SNMP and the required dependencies.

5. Setup .htaccess for MRTG so that only privileged users can view the graphs.

6. Estimated time: 1.5 hours at the max.

Since I was told not to go alone, I took Yash along with me. We reached the BSNL head office at around 10:45 in the morning and after a scouring the whole office complex for around half an hour, we finally met Mr S.C Gupta, the head of the NIB (National Internet Backbone) in Jaipur. He guided us to his office and then led us into their server room which housed some seriously wicked routers and a couple of not-so-wicked servers. Quite expected actually. He showed us the server we were supposed to work on. I don't even think it was a server class machine. He'd already tried to follow the instruction manual that each head office had been sent so as to (or hoping so as to) set up MRTG on their own, and had installed Red Hat 7.1 (LOL) in it by himself as per the manual. At first, I exclaimed “Are these guys kidding me?”. But as the reader of this story will soon find out, the joke was on me. :(

I proceeded to tell Mr S.C Gupta how much Debian rules and how it's the ultimate combination of security and stability and is pretty much the best choice for running servers. He was kind of pleased with our idea and told him that the machine was all ours and we're free to do whatever it takes to get the job done. Roger that! So we proceeded to do the necessary pings to see if the server was connected to the internet and then noted down the IP address, netmask, the default gateway and the routes. Then we booted our Debian net install disc and went on with the usual procedure, did the partitioning, set up the time zones blah blah blah and set up a static IP configuration. We kidded about how this is going to be piece of cake but I guess we had spoken a moment too soon because the installer complained that it couldn't connect to ftp.iitm.ac.in to fetch packages. WTF? This was definitely a bad omen. I wondered if we'd set the network interface up correctly and we had. But it just refused to recognize the mirror. So we decided to fetch the packages ourselves after we properly configured the card once the installation of the base system was over. As soon as that was done, we realized we simply could NOT ping any host outside the LAN (after having a hell lot of problems being identified within the LAN itself, we couldn't ping our machine from other machines). What followed was a lot of trips to Mr Gupta's computer and back to see if he's given us the right details about the network which was followed by an exchange of calls between us and Gaur sir who suggested that IP forwarding wasn't enabled. It seemed kind of odd since I ran the netmon machines from this very installation disc. Then I tried it with my own Debian installed laptop but in vain. I just could not see any host outside the LAN. After a lot of attempts and reinstalls, we decided enough was enough and moved on to install Red Hat 7.1 itself :(. Yuck. I was kind of embarrassed actually. After all the hype I created around Debian. Sad.

What was supposed to be just a matter of two commands in Debian was indeed a herculean task in Red Hat. We had to fetch each of MRTG's dependencies from the net, compile and install them from source, not to mention having to manually include and link the necessary files and libraries in each case. After using links, the text based web browser we all know and love, we were able to find out the latest and available versions for zlib, gd and libpng and we successfully installed them from source inspite of having to do a lot of including and linking. Then when it was time to install MRTG, the connection of ours went down and we got awesome transfer rates of around 0-30 bytes per second. Since I hadn't brought another set of clothes, a sleeping bag and a shaving kit for myself, I decided to ssh into our college network, download it from my netmon server and scp it over to the BSNL machine and it worked like a charm. After the MRTG package was ready, it's configuration files made, the HTML template created and the MRTG binary scheduled to run every five minutes using crontab, we decided to test it. And it seemed the rotten luck which had been following us all day wasn't going to make itself scarce any time soon. We couldn't open the apache test page from Mr Gupta's browser. We ran a lot of checks on our httpd.conf and everything seemed normal. The permissions were perfect and the ownership required etc were all fine. We even tried setting up a virtual host for the job but even that was in vain. Finally, the idea dawned upon me that the firewall we'd set during installation time was set to block all incoming connections to the machine. After permitting www (http) requests to the machine and adding the prescribed set of ipchains rules, the server was good to go! We moved on to set up the finishing touches to the machine, making sure things worked even after a reboot. We then set up .htaccess and added a line to the html file saying that the maintainer was Mr S.C Gupta. Job's done!

Estimated time: 1.5 hours max.

Actual time taken: 6.5 hours flat!

All in all, an interesting experience, having had to work in a real live environment. After the ordeal, we realised we were pretty hungry because we were NOT given a single bite at the office. I went on to mow down 6 or 7 rotis (I lost count actually).

Cheers!