I finally took my network monitoring work up a level when Gaur sir asked me to go over to the BSNL head office and set up a MRTG server for the NIB (National Internet Backbone). What an opportunity! MRTG is Tobi Oetiker's (the man behind RRD Tools and Smokeping) multi router traffic grapher which is basically a tool to tell how much bandwidth your routers are using. This seems crucial for an ISP like BSNL who keep getting calls from their customers who b**** about not getting enough bandwidth. And it's really important that they have such a set up since all broadband connections to the state of Rajasthan originate from the Jaipur node where I was asked to work. I'd decided to deploy my ever favourite linux distro, Debian Etch on their server so that apt-get would make quick work out of the task at hand. Here's the roadmap I'd laid down for myself for the job:

1. Install a base Debian Etch-4.0.r1 with a netinstall with a fairly large and separate /var partition since all the .png files generated by MRTG are dumped into /var.

2. Since security is an important issue at such a level, a dist-upgrade is necessary which is meant to patch the OpenSSL security flaw that was found in Debian a couple of weeks back.

3. Use apt-get install apache2 to set up a webserver and configure it as needed.

4. Use apt-get install mrtg snmpd to set up MRTG and SNMP and the required dependencies.

5. Setup .htaccess for MRTG so that only privileged users can view the graphs.

6. Estimated time: 1.5 hours at the max.

Since I was told not to go alone, I took Yash along with me. We reached the BSNL head office at around 10:45 in the morning and after a scouring the whole office complex for around half an hour, we finally met Mr S.C Gupta, the head of the NIB (National Internet Backbone) in Jaipur. He guided us to his office and then led us into their server room which housed some seriously wicked routers and a couple of not-so-wicked servers. Quite expected actually. He showed us the server we were supposed to work on. I don't even think it was a server class machine. He'd already tried to follow the instruction manual that each head office had been sent so as to (or hoping so as to) set up MRTG on their own, and had installed Red Hat 7.1 (LOL) in it by himself as per the manual. At first, I exclaimed “Are these guys kidding me?”. But as the reader of this story will soon find out, the joke was on me. :(

I proceeded to tell Mr S.C Gupta how much Debian rules and how it's the ultimate combination of security and stability and is pretty much the best choice for running servers. He was kind of pleased with our idea and told him that the machine was all ours and we're free to do whatever it takes to get the job done. Roger that! So we proceeded to do the necessary pings to see if the server was connected to the internet and then noted down the IP address, netmask, the default gateway and the routes. Then we booted our Debian net install disc and went on with the usual procedure, did the partitioning, set up the time zones blah blah blah and set up a static IP configuration. We kidded about how this is going to be piece of cake but I guess we had spoken a moment too soon because the installer complained that it couldn't connect to ftp.iitm.ac.in to fetch packages. WTF? This was definitely a bad omen. I wondered if we'd set the network interface up correctly and we had. But it just refused to recognize the mirror. So we decided to fetch the packages ourselves after we properly configured the card once the installation of the base system was over. As soon as that was done, we realized we simply could NOT ping any host outside the LAN (after having a hell lot of problems being identified within the LAN itself, we couldn't ping our machine from other machines). What followed was a lot of trips to Mr Gupta's computer and back to see if he's given us the right details about the network which was followed by an exchange of calls between us and Gaur sir who suggested that IP forwarding wasn't enabled. It seemed kind of odd since I ran the netmon machines from this very installation disc. Then I tried it with my own Debian installed laptop but in vain. I just could not see any host outside the LAN. After a lot of attempts and reinstalls, we decided enough was enough and moved on to install Red Hat 7.1 itself :(. Yuck. I was kind of embarrassed actually. After all the hype I created around Debian. Sad.

What was supposed to be just a matter of two commands in Debian was indeed a herculean task in Red Hat. We had to fetch each of MRTG's dependencies from the net, compile and install them from source, not to mention having to manually include and link the necessary files and libraries in each case. After using links, the text based web browser we all know and love, we were able to find out the latest and available versions for zlib, gd and libpng and we successfully installed them from source inspite of having to do a lot of including and linking. Then when it was time to install MRTG, the connection of ours went down and we got awesome transfer rates of around 0-30 bytes per second. Since I hadn't brought another set of clothes, a sleeping bag and a shaving kit for myself, I decided to ssh into our college network, download it from my netmon server and scp it over to the BSNL machine and it worked like a charm. After the MRTG package was ready, it's configuration files made, the HTML template created and the MRTG binary scheduled to run every five minutes using crontab, we decided to test it. And it seemed the rotten luck which had been following us all day wasn't going to make itself scarce any time soon. We couldn't open the apache test page from Mr Gupta's browser. We ran a lot of checks on our httpd.conf and everything seemed normal. The permissions were perfect and the ownership required etc were all fine. We even tried setting up a virtual host for the job but even that was in vain. Finally, the idea dawned upon me that the firewall we'd set during installation time was set to block all incoming connections to the machine. After permitting www (http) requests to the machine and adding the prescribed set of ipchains rules, the server was good to go! We moved on to set up the finishing touches to the machine, making sure things worked even after a reboot. We then set up .htaccess and added a line to the html file saying that the maintainer was Mr S.C Gupta. Job's done!

Estimated time: 1.5 hours max.

Actual time taken: 6.5 hours flat!

All in all, an interesting experience, having had to work in a real live environment. After the ordeal, we realised we were pretty hungry because we were NOT given a single bite at the office. I went on to mow down 6 or 7 rotis (I lost count actually).

Cheers!

One important thing as far as network monitoring solutions for a CWN is concerned, involves setting up a system by which the administrators are informed in case one of our services goes down. We may use email or a messaging client like Jabber, but in the event that something goes wrong with the core switch or the gateway, and the internet and/or the intranet is out, this will be useless. In such a situation, the solution would be to use a media which doesn't involve the local area network or the internet for that matter, one of them being, SMSes. So this is where we choose to deploy Zabbix with a GSM modem on my netmon machine. Rohit, the technician who the Genus people (the company which provided us the modem) had sent over to demonstrate the modem, only knew how to operate it via hyper terminal which is an interface for Windows by which you can communicate with your serial port. He was supposed to show me how to use the AT commands and all that. But since we ITR folk at MNIT kick ass, we don't use Windows see? All our servers run on Debian, Gentoo, FreeBSD, Deeproot etc. No Windows! Now this dude had never used Linux before and even went to the extent of typing AT on my linux terminal. And since I was new to the whole idea of AT commands then, I didn't know how to talk with the serial port in linux, which is done using minicom, by the way. My immediate aim was to just test the modem and a quick look up on google told me that there's this thing called gsm-utils, which is a package used to send SMSes using a GSM modem (duh?) in linux. With apt-get install gsm-utils, I got the binaries installed in a jiffy and a quick browse through the man pages gave me all the info I needed to use this tool. So I insert my SIM into the device, and I try to send a message to Rohit's cell phone by typing the following into the console,

$: gsmsendsms -d /dev/ttyS0 XXXXXXXXXX “testing”

Then came the moment of truth, as me and Rohit stood there; waiting for some form of output to appear on the screen while the cursor kept blinking, when the welcome sound of the message-received tone on Rohit's handset sounded. I leaped in joy and punched the air with my signature style. Rohit shook hands with me and told me that it seems he isn't needed anymore. :P

When I met Gaur sir a few minutes later with the good news, he told me to test the modem from Zabbix. I didn't see why that could make any difference because as far as linux supported the modem (which I'd just tested and seen for myself) , Zabbix wouldn't complain as well. But hell was I wrong!

Now while I still could make Zabbix use the gsmsendsms command (a remote command) for the alerts, it came with it's own set of problems. First of all, Zabbix won't log the remote commands and second, it won't know if the command really executed or not. This wasn't really an issue because there's no way the command could NOT work see? But the main problem was the loss in flexibility of setting up the triggers themselves. If I use the gsmsendsms command, I'd have to write the same code over and over again for different phone numbers (for the different admins) and for different triggers. This seemed very sloppy and I'm the kind of guy who likes to keep things... well... beautiful :). This meant having to get the AT commands working at all costs.

So I made a very stupid trigger which would alert me in case the hostname on the netmon machine was changed. And since Zabbix sends an alert everytime the trigger changes values, I just had to change the condition a little bit and it kept equating to true and false. Hence, I kept switching the trigger between,

{zabbix_server: system.hostname.str(netmon)}

and

{zabbix_server: system.hostname.str(X)}

As soon as I saved the trigger, Zabbix tried to send an SMS. And as I'd expected, it didn't work! It gave me the following error,

“Expected [OK], received [ERROR]”

So to figure out what the problem was, I thought I'd install minicom and try sending an SMS manually using the darned AT commands myself. It was then that I figured out the problem. Zabbix seems to use the serial port with a default baud rate. And this didn't seem to match with the baud rate of the modem itself. Although I was able to send an SMS through minicom at a baud rate of 9600 baud, I couldnt get the same to work from Zabbix, even after setting the serial port speed to 9600 baud. At that point, I got an error like this,

“Expected [OK], received [*AT+CMEE=2\rERROR]”

(The * here, was some symbol my browser couldn't identify because of lack of unicode support)

So finally I decided enough is enough and I thought I'd give the source code a little tweak. And tucked away in a little corner of the source tree for Zabbix was a file named sms.c :).

And here are the relevant contents of the code,

#: vim ~zabbix-1.4.4/src/libs/zbxsms/sms.c

typedef struct {

char *message;

const char *result;

int timeout_sec;

} zbx_sms_scenario;

zbx_sms_scenario scenario[] = {

/* 0 */ {ZBX_AT_ESC , NULL , 0 }, /* Send */

/* 1 */ {"AT+CMEE=2\r" , "OK" , 5 }, /* verbose error values */

/* 1 */ {"ATE0\r" , "OK" , 5 }, /* Turn off echo */

/* 2 */ {"AT\r" , "OK" , 5 }, /* Init modem */

/* 3 */ {"AT+CMGF=1\r" , "OK" , 5 }, /* Switch to text mode */

/* 4 */ {"AT+CMGS=\"" , NULL ,0 }, /* Set phone number */

/* 5 */ {number , NULL , 0 }, /* Write phone number */

/* 6 */ {"\"\r" , "> " , 5 }, /* Set phone number */

/* 7 */ {message , NULL, 0 }, /* Write message */

/* 8 */ {ZBX_AT_CTRL_Z ,"+CMGS: ", 40 }, /* Send message */

/* 9 */ {NULL , "OK" , 1 }, /* ^Z */

/* EOS */ {NULL , NULL , 0 }

};

Hmmmm, AT+CMEE=2\r eh? Verbose error values? Doesn't look to important to me. Why not turn the OK beside it into ERROR and hence, trick Zabbix into sending an SMS even if it is an ERROR? Its worth a shot. So I change,

/* 1 */ {"AT+CMEE=2\r" , "OK" , 5 }, /* verbose error values */

into

/* 1 */ {"AT+CMEE=2\r" , "ERROR" , 5 }, /* verbose error values */

I exit vim and then type the following into the console,

#~zabbix-1.4.4: make && make install

And would you believe it? It actually worked! Ha! Talk about luck. It's kind of strange that inspite of receiving an ERROR, Zabbix proceeded to send the message. Strange. Oh well, there's one case where having the source code helped.

I've been trying to build a very customized debian for quite a while and doing so obviously involves manually compiling your own kernel. So I fetch the 2.6.25.1 kernel (vanilla sources), stable version, unpack it and create the necessary symlink. After checking the right options for my ATI Radeon X1200 graphics card which really sucks ass, I compile the kernel, copy the bzImage to /boot, configure grub and reboot. From grub, I select my new kernel and while I was thinking of what next to fetch for my system, I am greeted with the following message.

VFS: Cannot open root device “sda1” or unknown-block (0,0) Please append a correct “root=” boot option; here are the available options
0100 8192 ram0 (driver?)
0101 8192 ram1 (driver?)
0102 8192 ram2 (driver?)
0103 8192 ram3 (driver?)
0104 8192 ram4 (driver?)
0105 8192 ram5 (driver?)
0106 8192 ram6 (driver?)
0107 8192 ram7 (driver?)
0108 8192 ram8 (driver?)
0109 8192 ram9 (driver?)
010a 8192 ram10 (driver?)
010b 8192 ram11 (driver?)
010c 8192 ram12 (driver?)
010d 8192 ram13 (driver?)
010e 8192 ram14 (driver?)
010f 8192 ram15 (driver?)
Kernel panic – not syncing : VFS: Unable to mount root fs on unkown-block (0,0

Lo! The infamous kernel panic. I encountered the same problem while working on one of the college servers which was to be set up for being the new gateway. After compiling the kernel for Iptables support and rebooting into it, I got the same error, only that the available options were different. It showed the only hda device on the system that is, the CD-ROM drive.

Now after looking into a lot of forums, I decided there wasn't any clear cut solution provided on how to solve the problem. Unless you really DID append an incorrect “root=” option, the problem lies in the fact that your kernel is not able to recognize your SATA device (sda). Chances are, it's been compiled for IDE support.

The solution? Compile your kernel for SATA support.

$: cd /usr/src/linux
$: make menuconfig

Device drivers--->
SCSI device support--->

SCSI target support
[*] legacy /proc/scsi/ support
*** SCSI support type (disk, tape, CD-ROM) ***
SCSI disk support
SCSI CDROM support
[*] SCSI low-level drivers --->

ACARD SCSI support
Adaptec AIC7xxx Fast -> U160 support (New Driver)
(32) Maximum number of TCQ commands per device
(5000) Initial bus reset delay in milli-seconds
[*] Compile in Debugging Code
(0) Debug code enable mask (2047 for all debugging)
[*] Decode registers during diagnostics
Adaptec AIC7xxx support (old driver)
Adaptec AIC79xx U320 support
(32) Maximum number of TCQ commands per device
(4000) Initial bus reset delay in milli-seconds
(0) Debug code enable mask (16383 for all debugging)

The options under SCSI low level drivers depend on your device. These are the options that I'd set according to my hardware. If you're not able to figure out what the right options are, select everything! As far as the gateway was concerned, I'd selected almost everything which looked sensible enough.

Finally

$: make clean && make && make modules_install
$: cp arch/x86/boot/bzImage /boot/your-kernel-version

The last step would be to update grub.

$: vi /boot/grub/menu.lst

title Your Linux
root (hd0,0)
kernel /boot/your-kernel-version root=/dev/sda1

Hope this little write up proves helpful. Cheers. Feel free to post a comment if you have any problems.

!-- @page { size: 8.5in 11in; margin: 0.79in } P { margin-bottom: 0.08in } -->

Well... It's been ages since I've put up a post here. A whole semester to be precise. I wouldn't attribute it to a lack of things to write about. That most certainly is not the case. I could have written about my resolutions (not for the weak hearted) or the golden night/freshers we gave our juniors. I could have even written about the new meaning to the phrase 'looking for *insert name here*' which I learnt on my birthday. I could have even written about our awesome road to clinching the 3^rd place in the IMS Quotient quiz: Jaipur finals. That was an awesome experience and I thank my team mates Kanjan and Abhinav Kumar for the same. Am I done yet? No ways. Then came along a decent performance (after a long long long LONG time) by me in the 1^st mid terms. But that spree didn't last long because I couldnt have got owned worse in the second MT or the end terms itself. I guess I'm halfway through to being an engineer now unless one of my teachers decides to leave me a 'you-can-rot-here-extra-time-during-the-summers' present. Moving on, I had a lot of fun interacting with the team from NIT-Trichy who'd come to our college as part of a project called NIT-Connect. The idea for the project was simple, but it's kind of surprising when you think WHY no one every thought of it before. It's nothing more than a mere webpage which serves as a common platform to all NIT's where each institution can put up news, photos and videos and interact with students and faculty from their sister institutions. The sad part was, they only wanted users from our side and not developers. And furthermore, their source code wasn't open. Kix kind of owned them when they mentioned that. Cheers man!

I guess I never blogged through all these months because I never had the time to do so. I don't know how, but I NEVER had any free time. This inspite of the fact that I cut down on gaming, hanging out with friends and a whole lot of other things. Perhaps, I was turning into a social outcast but fortunately, a certain lucky charm came along and I kind of snapped out of the blindness I was in. Thanks for everything Kay!

And how can I not write about the IIT-Roorkee trip? Although Cognizance wasn't even half as good as last year, it certainly came with its own moments. Of all the times they could get their project evaluations to get scheduled, the final years had to get their projects evaluated during the Chaos weekend. And hence, both our Dota teams got dissolved. Since I was planning to go for NFS:MW as well, Sat joined GaP and hence, my clan GoD, which is yet to be defeated, couldn't go to Roorkee. But unluckily for Sat, GaP got eliminated in the second round itself. Maybe they over estimated their opponents, we don't know. On the other side of the Chaos arena, I was having an awesome time with NFS:MW. I felt intimidated by some of the other participants, including a bunch who were last year's winner, runner-up and semi finalist, who carried around a little book in which they'd jotted down the best performance tunes for the Lotus Elise, the pro's choice. I'd never even tuned a car till date. But throught the tourney, I learned and made my way up the ladder. And the greatest moment of satisfaction for me, came when I was told that my games are being tracked and that I'm a seeded player! Talk about beginner's luck.

*WARNING: BRAG MODE ENABLED*

Beginner's luck my ass. I simply owned the competition. I ripped apart player after player. I'll even take some time to tell ya'll how awesome my competitive rounds were (rounds 4 and above). Round four was against a guy who insisted he played with his Xbox controller and be given headphones by the organizers. Ok, I can understand he needs his controller, but the headphones? When the organizer asked him how it's going to matter, he went on to say something about getting into the mood (???). They could only salvage one pair of headphones but when the race began, I simply left him trailing in my dust. Wonder if he got into the mood then. My split times were somewhere around 20 secs! (which means, if I stopped where I was, and he continued at his current speed, he'd take 20 seconds to reach me). In a racing game, thats comparable to the time we all took to evolve from apes. The 5^th round was similar with my opponent telling me how he's got better scores in Bay Bridge than the defending champion himself. Just before the race started, he did act kind of humble though. He told me that he's a fairly decent player and he can give me a good fight. Oh well, it didn't take me too long to beat him either. I could have called it a good fight IF I'd seen him on my screen for at least 2 seconds. But alas, he wasn't even on my minimap.

*yawn*

*END OF BRAG MODE*

Then came the 6^th round where I got owned by last year's runner up. Not too much to say here. N20 was disabled for the semi finals and I've never raced without that, so my strategy was ruined and hence...

Anyways, being in the top 4 ain't that bad is it?

On my return, followed a series of events which include a little chat conversation with Kay on the 1^st of April, a little message from her on the 11^th and finally another chat conversation on the 18th and guess what? We're going out... :p (I assure you the process was as boring and simple as I made it sound back there).

And that, was the cherry on top for my awesome semester.

I guess all good things must come to an end. So here I am, back to the server room, building up my Debian from scratch. Maybe I'll be a bit more regular with the posts now that I'll be spending the summer here working on the Campus Wide Network. In these two months, I'll be learning all the technicalities about networking while I go about setting up a DMZ for the CWN, and making high availability clusters for all our services. I've already made a cluster for the Netmon service of ours which seems to be doing fairly well. Anyways, I guess I'll start posting technical updates from now on.

Anyways, the summer heat wave awaits. Do wish me luck people.

So adios for now. Guess I'll go dig into the awesome mess food that awaits me. I'm sure you're all jealous. Suckers...

18th, December:
On the 18th of this month, my mother and I arrived at the Mumbai airport after a one and a half hour long flight. The Jetlite flight we boarded was pretty much a disappointment. The seats weren’t exactly suited for my never-in-good-shape back and there just wasn’t enough leg space. The guy next to me wasn't having the time of his life either, becuase he was going through a bout of fever and the air conditioning wasn't exactly helping. The poor chap kept coughing, sneezing and I don’t know what, every few minutes, which kept startling me. And hence, neither he nor I managed to get a wink of sleep. And odd as it seemed, there was only one air hostess in on board, the rest were stewards. Heck. But then again, it was a plane journey and I was very happy that we got down in an hour and a half. My uncle was there to recieve us at the arrival and we made our way home. Once there I caught up with my cousins and karthu’s husband, Anoop. And soon enough, I did what I loved to do most in Mumbai… hit the sack!

19th, December:
After lazing around the whole of the next afternoon, we got ourselves ready for Karthu and Anoop’s wedding reception. We guys were dressed in black suits. I was in charge of the photography through the whole evening and I had a gala time, talking with people, taking snaps and keeping my eyes peeled [:P]. The food was amazingly delicious, and I couldn’t help but take three helpings of the rasmalai that was part of dessert. All in all, the function went on very well and we returned that night only to fall asleep again [:P]. Damn I love to sleep.

20th, December:
Nothing interesting here. I spent the whole day sleeping, browsing and ummm… sleeping. Mainly owing to the fact that Karthu, Anoop and my aunt had gone to some Sai Baba temple in Nasik. So there you go.

21st, December:
Well, the morning was spent lazing around as usual. And as far as the evening is concerned, mum and I made our way to my other uncle’s home (This uncle being my dad’s younger brother) . His daughter, Aloka, was to get married on the 23rd to her boyfriend, Amit. So Amit’s family was to come over and hand over the wedding sari and well... you know the tradition right? So hence, I met his folks for the 1st time and also caught up with Aloo and her little brother, Dinku.

22nd December:
The next afternoon, mom and I were driven over to the Chembur Gymkhana club by a car arranged by my uncle. This is where we would be staying through Aloka’s wedding. That evening, we had a mehendi party or whatever it is that you call it. It seems, all their friends were puzzled when they were told we keralites don’t have such a function. Finally they had to give in to their friends' pressure and they decided to have one! On the club terrace, the setup was rather slick. There was a two-member troupe playing songs of our choice. More like a living jukebox. I asked them to perform Hotel California, which they were rather hesitant to do because its not exactly a song that you play with a guitar and a violin! But inspite of that, he did sing half the song to claps and cheers from the guests. All the ladies had their arms covered with mehendi and I got myself acquainted with a lot of people.

23rd December:
Apart from the initial delay and confusion and the fact that they missed the muhurtam (auspicious time) by a few minutes, the wedding took place rather normally in the Chembur Gymkhana Club basement. I being Aloka’s only cousin to attend the marriage, put myself in charge of handling the gifts as and when the couple got them. Nothing worth mentioning here. The buffet comprised of all the traditional ‘sadhyam’ elements served on plates with banana leaves on them. All went well and Aloo broke down when it was time to leave, like most other brides I’ve come across [:P].




That very evening, was the highlight of my Mumbai trip. Aloka’s wedding reception at the Hilton Hotels, Mumbai (above, duh). Located by the Queen’s necklace at Nariman Point, the building was an amazing sight, blanketed by lights and standing proud in the night. “So this is what Paris Hilton is heiress to eh?” I asked myself. I was in awe of the sheer amount of class that the building radiated, both inside and outside. The reception was to take place on the rooftop (35th storey) and the view from there was simply breathtaking! One could actually see the whole city from up there. And the Queen’s necklace looked fantabulous! It kind of brought back memories of Kornich, Doha. At the party, I was introduced to all kinds of people including the owner of the Apollo chain of hospitals. I even met some family friends of ours who we last met when we were in Iran. I don’t remember them a even a bit though, because I was a baby when they left the country it seems. The hard drinks section was pretty much in a rush with just one guy having to cater to 6-7 people at a time! I had two glasses of wine for myself (that's as far as my drinking goes :P). And yeah, the food was awesome! There were around dishes different items including pasta, seafood, weird but delicious salads, Thai food, the works! The dessert itself is worthy of mention as it had four itemsfor itself. There were toblerone wafer sundaes, shahi (I cant quite recall the name) which looked like gulab jamuns in custard, but tasted different and excellent! Some kind of halwa and last but the least, roasted almond ice cream. I had two rounds of dessert [:P]. And after I bid farewell to my relatives and the people I'd gotten myself introduced to, me and my other cousins made our way home.And hence, I ame to the end of a memorable mini holiday in Mumbai. We were to leave the next afternoon, and thee train journey is by itself worthy of being put up as a separate post.