• Home at last...

    After two months of self inflicted torture at Jaipur, I've finally made it home, at last! On the 23rd of last month, I was informed that I'd been selected as the Sun Ambassador to my institute. Awesome! I'd almost forgot what joy felt like until then. As Sun Ambassador, I get trained in a multitude of Sun Microsystem's technologies such as Open Solaris, Java, Netbeans, Sun Studio and a lot more. My job is to conduct tech demos and events in my campus to promote all these technologies among both students and faculty alike. The coolest part is, I even get to code on specific projects of their's and hence get to contribute to open source. And as if that's not enough, I even get paid to do this! What more could I ask for? Four days later I was on a train home. For the first time in two years, I was super excited about going home because I'd really worn myself out throughout my fourth semester and on top of that, I stayed back at Jaipur so that I could continue my work with the institute network. I also thought I'd complete my RHCE examination training from Linux World for which I'd got a 100% scholarship for; although it (sadly) turned out to be a complete waste of my time (since I don't intend on spending 12k to get a certification). Two hours of classes everyday for which the very young and inexperienced teacher of ours teaches only for an hour. He keeps getting things wrong and the sad part is, his working knowledge of Linux is confined ONLY to Red Hat. He hasn't experienced the sheer awesomeness of distros like Debian or Gentoo. Plus, his knowledge was restricted only to whatever he'd learnt after attending a couple of similiar courses. And, he has no developing experience. In my honest opinion, Linux and everything else related to computers, is best learnt alone. But anyways, my decision to stay back this summer has been one of the most rewarding decisions I've ever made in my life. I came to know about the Sun Campus Ambassador post being open and my working knowledge on networking and system administration improved by a factor ten :) . I also worked on firewalls, name servers, proxy servers, SMS alerting systems, VPN and high availability clusters, all of which contributed to a steep learning curve. Apart from the one week of nonsensical politics we had in college during Gaur sir's absence, I'd say these two months have been the most productive months for me so far. Anyways, I hadn't actually reached wonder land yet as I'd just embarked on a two day journey which, as I was to realise soon enough, was without any doubt the closest thing to getting a nine inch nail hammered into your skull. And why was that? Anyone who knows me well enough would guess one out of these three things: brats, brats and brats. You guessed right? Awesome!

    Yes, my coupe had a family with a very little 5-6 month old brat and his big sister who had to be the brattiest fucking brat ever (not to mention the fact that the neighbouring coupes had a brat each). The stupid piece of filth could not sit still for no less than 2 seconds. And if you think i'm exaggerating, kindly ask her parents who told me that she couldnt sit still at all. tiI can understand that the little brat couldn't do much but cry, because that's the only thing little brats can do (right?), but that darn swine of a sister of his kept crying for attention while her unfortunate parents tried to attend to the routine diaper changes and what not. I'm sure they must have sinned like crazy to get a kid like that. Her folks tried 'asking' her to keep quiet, but in vain. If I were the parent, I wouldn't resort to words to get the job done. I'd be quick to toss her out of the train, preferrably off a bridge so that the chances of survival are grim and that there is no possibility of some unfortunate passerby having to find her and take her in. For most of the trip, I confined myself to my upper berth and thought of ways to get rid of the brat. I even thought of ways to get rid of the little brat just in case. You can be sure that he'll grow up to be just like his sister. I'd share those thoughts with you but it might end up being too graphic and is hence not appropriate for any age group at all. Luckily, there was a Malayali guy and his mom on their way to Calicut on the side berths. The guy's name was Nikhil and he'd just complete his twelth grade from KV Pattom and hence, I could open up a conversation with him. We talked about how crazy our batches at school were, about the people in Jaipur and a lot of other things. The blasted brats and their incompetent parents (after long last) got down at Madgaon the next evening. The train was running pretty late (five and a half hours to be precise) owing to an engine failure which happened close to an hour after it left Chiplun. I finally reached Shoranur junction at 7:00 AM on Sunday morning. After a wreched two hour long ride by bus, I finally reached home! Mom, Priyamma and Ammamma gave me a warm welcome with nice, long, crisp dosas and stew for breakfast. I mowed down six dosas.

    Dial-up blows by the way. For signing into blogger and putting up this post, it took me no less than forty minutes.

    Although it feels great to be visiting relatives and friends (after a long time!) I kind of wish I was back at college. Guess I got used to the hectic lifestyle I'd led back there. Hope my mom doesn't see this or I might get bludgeoned with a rolling pin or something. Cheers!

  • You know someone wants a favour from you when...

    Ever felt odd when that person who's always ignored you comes up to you and becomes nice all of a sudden? When that person who's always in the 'I'm busy' mode when you're online suddenly opens a chat conversation and says “Hi! How are you today?”. Well, chances are, Mr/Mrs X wants a favour from you. In this post, I'll give you a little tutorial on how to tell if someone wants something from you. Once you're through with this, you'll be unmasking those hypocritic maggots in no time! I shall move on to a little categorisation based on how cunning the maggot under consideration is. So with no further ado...

    NOTE: I don't know what you call such a person, so I guess I'll continue referring to them as maggots throughout this tutorial. No offence to the real maggots which are just nice little crawly creatures who want to make the world a better place.

    Level 1 maggot: Loser class

    Skill level: poor
    Cunningness: poor
    Brains: N/A
    Sinister rating: 1/10
    Signs: You've never had a conversation with him in your entire life, nor will you for many more lives to come, but this person actually walks into your room, sits on your bed and acts as if he's very interested in whatever you're doing. When asked, “Hey, what brings you here?”, he's dumb enough to answer, “Nothing, just dropped by to say hi, we are friends and all aren't we?”. Note, he said friends. Wrong answer. The verdict: maggot!
    Comments: The one finger salute works well. Try it out.

    Level 2 maggot: Grunt class

    Skill level: less than average
    Cunningness: less than average
    Brains: Rumours are they do have some, but I still go for N/A
    Sinister rating: 3/10

    Signs: You're sitting in the mess, cursing your rotten luck to be served cattle feed, when the suspected maggot comes running and sits right next to you. You observe he's already had his meal. You also remember you've just given a kick ass presentation that morning. He puts his arm around you like you grew up together only to say, “Hey man! How're you doing? How's your life going?” Note that both the questions are related to you. This is a sure give away. This person has nothing to do with you at all and you're not related. Chances are, he's going to ask you to help him out with his project but he's going to play around for a while before asking you for certain, just to make you feel comfortable.

    Comments: Shove your plate up his throat. And then proceed with the one finger salute.

    Level 3 maggot: Striker class

    Skill level: above average
    Cunningness: above average
    Brains: A teeny weeny bit. Or maybe not.
    Sinister rating: 6/10

    Signs: You know there are a lot of people in your friends list who don't give a beep about you and vice versa? That's the kind of people that we're talking about here. Now you're working away in the server room trying to figure out a solution to a problem that has long been elusive. Then, the maggot opens up a chat conversation saying, “hi”. You also note that he's got the 'I'm busy/Don't disturb/away/I'm dead' status message up. The conversation proceeds as follows:

    Maggot: hi
    You: hi there! ssup?

    ...10 seconds later...

    Maggot: how are you?

    ... Notice the delay...

    You: uh...i'm ok. Wat abt you?

    Maggot: i'm fine.

    ... It's time to slip up you evil scum...

    Maggot: wat are you workin on in the server room again?

    ... Gotcha!...

    This is where they all go wrong. The two of you might have been in the same class for maybe two years and it's been a whole two semesters probably since you've been working in the server room and he knows it and you know that he knows it. BUT, he comes out of the blue and decides to ask you what you've been doing all along? It's strange. It's not like he woke up one fine morning and while having coffee, realisation struck, he went OMGWTF and he figured out he didn't know what you've been doing in the server room all along. I'm sorry. What he really wants to do, is to get a conversation going so he can ask you a little while later to check his grades or something of that sort. Dork.

    Comments: Tell him he flunked in all his subjects and that he's got a sem back. It works great!

    Level 4 maggot: Elite class

    Skill level: High
    Cunningness: High
    Brains: Lots of it. Lots and lots of it. Or then again, I'm just kidding.
    Sinister rating: 10/10

    Signs: These guys are the mommy's and daddy's of all the above mentioned maggots. They're the pros. If this whole thing was Warcraft, they would be some boss like Nerzhul, the Lich King. They're pretty good at getting what they want from you. They'll plan way in advance as to get what they want from you and how they're going to go about to get it from you. Even people who rule as much as myself are prone to these things. They're a menace and we should all unite to send them to oblivion. Yet again, I'll take the example of a chat conversation but in this case, the hunter won't make it that obvious until at least half an hour.

    Elite maggot: Heya... long time! Have you forgotten me kya?
    You: oh hi. yeah right. long time!
    Elite maggot: So what have you been doing?
    You: oh nothing...just working away...
    Elite maggot: i c. when're you going home?

    ...blah blah blah...

    ...blah blah blah...

    ...blah blah blah...

    20-30 minutes later, the maggot is out of things to talk about...

    Elite maggot: temme...howz ur (continued below)

    Remember, all this person wants from you is the favour and he/she doesn't give a shit about you or your life, let alone your...

    (continued from above) ....girlfriend? ;)

    It's a 99% complete give away and this is a sign for you to become cautious. Put up your guard and be on the defense. Or better, do what I do...

    You: hey...the connection seems to have some probs here. I might get disconnected any sec...

    Elite maggot: wait... can you do me a favour?

    And it's at exactly that instant that you turn invisible to the maggot. You have to time this right ok? It's all about practice. But do keep in mind that this depends heavily on the IM client you're using. For example, in Gtalk, it takes a while for the other person to get the message that you're offline. So it's all about preventing the maggot from knowing that you're sure there's that question coming to you.

    Comments: Once you turn invisible, either block that person or ignore him/her for at least a century and a half.

    So there you go folks. Hope this helps. And don't forget, squish 'em maggots! And an advice to anyone who wants a favour from me; JUST ASK!
  • Setting up MRTG for BSNL, Jaipur.

    I finally took my network monitoring work up a level when Gaur sir asked me to go over to the BSNL head office and set up a MRTG server for the NIB (National Internet Backbone). What an opportunity! MRTG is Tobi Oetiker's (the man behind RRD Tools and Smokeping) multi router traffic grapher which is basically a tool to tell how much bandwidth your routers are using. This seems crucial for an ISP like BSNL who keep getting calls from their customers who b**** about not getting enough bandwidth. And it's really important that they have such a set up since all broadband connections to the state of Rajasthan originate from the Jaipur node where I was asked to work. I'd decided to deploy my ever favourite linux distro, Debian Etch on their server so that apt-get would make quick work out of the task at hand. Here's the roadmap I'd laid down for myself for the job:

    1. Install a base Debian Etch-4.0.r1 with a netinstall with a fairly large and separate /var partition since all the .png files generated by MRTG are dumped into /var.

    2. Since security is an important issue at such a level, a dist-upgrade is necessary which is meant to patch the OpenSSL security flaw that was found in Debian a couple of weeks back.

    3. Use apt-get install apache2 to set up a webserver and configure it as needed.

    4. Use apt-get install mrtg snmpd to set up MRTG and SNMP and the required dependencies.

    5. Setup .htaccess for MRTG so that only privileged users can view the graphs.

    6. Estimated time: 1.5 hours at the max.

    Since I was told not to go alone, I took Yash along with me. We reached the BSNL head office at around 10:45 in the morning and after a scouring the whole office complex for around half an hour, we finally met Mr S.C Gupta, the head of the NIB (National Internet Backbone) in Jaipur. He guided us to his office and then led us into their server room which housed some seriously wicked routers and a couple of not-so-wicked servers. Quite expected actually. He showed us the server we were supposed to work on. I don't even think it was a server class machine. He'd already tried to follow the instruction manual that each head office had been sent so as to (or hoping so as to) set up MRTG on their own, and had installed Red Hat 7.1 (LOL) in it by himself as per the manual. At first, I exclaimed “Are these guys kidding me?”. But as the reader of this story will soon find out, the joke was on me. :(

    I proceeded to tell Mr S.C Gupta how much Debian rules and how it's the ultimate combination of security and stability and is pretty much the best choice for running servers. He was kind of pleased with our idea and told him that the machine was all ours and we're free to do whatever it takes to get the job done. Roger that! So we proceeded to do the necessary pings to see if the server was connected to the internet and then noted down the IP address, netmask, the default gateway and the routes. Then we booted our Debian net install disc and went on with the usual procedure, did the partitioning, set up the time zones blah blah blah and set up a static IP configuration. We kidded about how this is going to be piece of cake but I guess we had spoken a moment too soon because the installer complained that it couldn't connect to ftp.iitm.ac.in to fetch packages. WTF? This was definitely a bad omen. I wondered if we'd set the network interface up correctly and we had. But it just refused to recognize the mirror. So we decided to fetch the packages ourselves after we properly configured the card once the installation of the base system was over. As soon as that was done, we realized we simply could NOT ping any host outside the LAN (after having a hell lot of problems being identified within the LAN itself, we couldn't ping our machine from other machines). What followed was a lot of trips to Mr Gupta's computer and back to see if he's given us the right details about the network which was followed by an exchange of calls between us and Gaur sir who suggested that IP forwarding wasn't enabled. It seemed kind of odd since I ran the netmon machines from this very installation disc. Then I tried it with my own Debian installed laptop but in vain. I just could not see any host outside the LAN. After a lot of attempts and reinstalls, we decided enough was enough and moved on to install Red Hat 7.1 itself :(. Yuck. I was kind of embarrassed actually. After all the hype I created around Debian. Sad.

    What was supposed to be just a matter of two commands in Debian was indeed a herculean task in Red Hat. We had to fetch each of MRTG's dependencies from the net, compile and install them from source, not to mention having to manually include and link the necessary files and libraries in each case. After using links, the text based web browser we all know and love, we were able to find out the latest and available versions for zlib, gd and libpng and we successfully installed them from source inspite of having to do a lot of including and linking. Then when it was time to install MRTG, the connection of ours went down and we got awesome transfer rates of around 0-30 bytes per second. Since I hadn't brought another set of clothes, a sleeping bag and a shaving kit for myself, I decided to ssh into our college network, download it from my netmon server and scp it over to the BSNL machine and it worked like a charm. After the MRTG package was ready, it's configuration files made, the HTML template created and the MRTG binary scheduled to run every five minutes using crontab, we decided to test it. And it seemed the rotten luck which had been following us all day wasn't going to make itself scarce any time soon. We couldn't open the apache test page from Mr Gupta's browser. We ran a lot of checks on our httpd.conf and everything seemed normal. The permissions were perfect and the ownership required etc were all fine. We even tried setting up a virtual host for the job but even that was in vain. Finally, the idea dawned upon me that the firewall we'd set during installation time was set to block all incoming connections to the machine. After permitting www (http) requests to the machine and adding the prescribed set of ipchains rules, the server was good to go! We moved on to set up the finishing touches to the machine, making sure things worked even after a reboot. We then set up .htaccess and added a line to the html file saying that the maintainer was Mr S.C Gupta. Job's done!

    Estimated time: 1.5 hours max.

    Actual time taken: 6.5 hours flat!

    All in all, an interesting experience, having had to work in a real live environment. After the ordeal, we realised we were pretty hungry because we were NOT given a single bite at the office. I went on to mow down 6 or 7 rotis (I lost count actually).


  • A little hack and lo! The SMS is sent.

    One important thing as far as network monitoring solutions for a CWN is concerned, involves setting up a system by which the administrators are informed in case one of our services goes down. We may use email or a messaging client like Jabber, but in the event that something goes wrong with the core switch or the gateway, and the internet and/or the intranet is out, this will be useless. In such a situation, the solution would be to use a media which doesn't involve the local area network or the internet for that matter, one of them being, SMSes. So this is where we choose to deploy Zabbix with a GSM modem on my netmon machine. Rohit, the technician who the Genus people (the company which provided us the modem) had sent over to demonstrate the modem, only knew how to operate it via hyper terminal which is an interface for Windows by which you can communicate with your serial port. He was supposed to show me how to use the AT commands and all that. But since we ITR folk at MNIT kick ass, we don't use Windows see? All our servers run on Debian, Gentoo, FreeBSD, Deeproot etc. No Windows! Now this dude had never used Linux before and even went to the extent of typing AT on my linux terminal. And since I was new to the whole idea of AT commands then, I didn't know how to talk with the serial port in linux, which is done using minicom, by the way. My immediate aim was to just test the modem and a quick look up on google told me that there's this thing called gsm-utils, which is a package used to send SMSes using a GSM modem (duh?) in linux. With apt-get install gsm-utils, I got the binaries installed in a jiffy and a quick browse through the man pages gave me all the info I needed to use this tool. So I insert my SIM into the device, and I try to send a message to Rohit's cell phone by typing the following into the console,
    $: gsmsendsms -d /dev/ttyS0 XXXXXXXXXX “testing”

    Then came the moment of truth, as me and Rohit stood there; waiting for some form of output to appear on the screen while the cursor kept blinking, when the welcome sound of the message-received tone on Rohit's handset sounded. I leaped in joy and punched the air with my signature style. Rohit shook hands with me and told me that it seems he isn't needed anymore. :P

    When I met Gaur sir a few minutes later with the good news, he told me to test the modem from Zabbix. I didn't see why that could make any difference because as far as linux supported the modem (which I'd just tested and seen for myself) , Zabbix wouldn't complain as well. But hell was I wrong!

    Now while I still could make Zabbix use the gsmsendsms command (a remote command) for the alerts, it came with it's own set of problems. First of all, Zabbix won't log the remote commands and second, it won't know if the command really executed or not. This wasn't really an issue because there's no way the command could NOT work see? But the main problem was the loss in flexibility of setting up the triggers themselves. If I use the gsmsendsms command, I'd have to write the same code over and over again for different phone numbers (for the different admins) and for different triggers. This seemed very sloppy and I'm the kind of guy who likes to keep things... well... beautiful :). This meant having to get the AT commands working at all costs.

    So I made a very stupid trigger which would alert me in case the hostname on the netmon machine was changed. And since Zabbix sends an alert everytime the trigger changes values, I just had to change the condition a little bit and it kept equating to true and false. Hence, I kept switching the trigger between,

    {zabbix_server: system.hostname.str(netmon)}


    {zabbix_server: system.hostname.str(X)}

    As soon as I saved the trigger, Zabbix tried to send an SMS. And as I'd expected, it didn't work! It gave me the following error,

    “Expected [OK], received [ERROR]”

    So to figure out what the problem was, I thought I'd install minicom and try sending an SMS manually using the darned AT commands myself. It was then that I figured out the problem. Zabbix seems to use the serial port with a default baud rate. And this didn't seem to match with the baud rate of the modem itself. Although I was able to send an SMS through minicom at a baud rate of 9600 baud, I couldnt get the same to work from Zabbix, even after setting the serial port speed to 9600 baud. At that point, I got an error like this,

    “Expected [OK], received [*AT+CMEE=2\rERROR]”

    (The * here, was some symbol my browser couldn't identify because of lack of unicode support)

    So finally I decided enough is enough and I thought I'd give the source code a little tweak. And tucked away in a little corner of the source tree for Zabbix was a file named sms.c :).

    And here are the relevant contents of the code,

    #: vim ~zabbix-1.4.4/src/libs/zbxsms/sms.c

    typedef struct {

    char *message;

    const char *result;

    int timeout_sec;

    } zbx_sms_scenario;

    zbx_sms_scenario scenario[] = {

    /* 0 */ {ZBX_AT_ESC , NULL , 0 }, /* Send */

    /* 1 */ {"AT+CMEE=2\r" , "OK" , 5 }, /* verbose error values */

    /* 1 */ {"ATE0\r" , "OK" , 5 }, /* Turn off echo */

    /* 2 */ {"AT\r" , "OK" , 5 }, /* Init modem */

    /* 3 */ {"AT+CMGF=1\r" , "OK" , 5 }, /* Switch to text mode */

    /* 4 */ {"AT+CMGS=\"" , NULL ,0 }, /* Set phone number */

    /* 5 */ {number , NULL , 0 }, /* Write phone number */

    /* 6 */ {"\"\r" , "> " , 5 }, /* Set phone number */

    /* 7 */ {message , NULL, 0 }, /* Write message */

    /* 8 */ {ZBX_AT_CTRL_Z ,"+CMGS: ", 40 }, /* Send message */

    /* 9 */ {NULL , "OK" , 1 }, /* ^Z */

    /* EOS */ {NULL , NULL , 0 }


    Hmmmm, AT+CMEE=2\r eh? Verbose error values? Doesn't look to important to me. Why not turn the OK beside it into ERROR and hence, trick Zabbix into sending an SMS even if it is an ERROR? Its worth a shot. So I change,

    /* 1 */ {"AT+CMEE=2\r" , "OK" , 5 }, /* verbose error values */


    /* 1 */ {"AT+CMEE=2\r" , "ERROR" , 5 }, /* verbose error values */

    I exit vim and then type the following into the console,

    #~zabbix-1.4.4: make && make install

    And would you believe it? It actually worked! Ha! Talk about luck. It's kind of strange that inspite of receiving an ERROR, Zabbix proceeded to send the message. Strange. Oh well, there's one case where having the source code helped.

  • Kernel panic anyone?

    I've been trying to build a very customized debian for quite a while and doing so obviously involves manually compiling your own kernel. So I fetch the kernel (vanilla sources), stable version, unpack it and create the necessary symlink. After checking the right options for my ATI Radeon X1200 graphics card which really sucks ass, I compile the kernel, copy the bzImage to /boot, configure grub and reboot. From grub, I select my new kernel and while I was thinking of what next to fetch for my system, I am greeted with the following message.

    VFS: Cannot open root device “sda1” or unknown-block (0,0) Please append a correct “root=” boot option; here are the available options
    0100 8192 ram0 (driver?)
    0101 8192 ram1 (driver?)
    0102 8192 ram2 (driver?)
    0103 8192 ram3 (driver?)
    0104 8192 ram4 (driver?)
    0105 8192 ram5 (driver?)
    0106 8192 ram6 (driver?)
    0107 8192 ram7 (driver?)
    0108 8192 ram8 (driver?)
    0109 8192 ram9 (driver?)
    010a 8192 ram10 (driver?)
    010b 8192 ram11 (driver?)
    010c 8192 ram12 (driver?)
    010d 8192 ram13 (driver?)
    010e 8192 ram14 (driver?)
    010f 8192 ram15 (driver?)
    Kernel panic – not syncing : VFS: Unable to mount root fs on unkown-block (0,0

    Lo! The infamous kernel panic. I encountered the same problem while working on one of the college servers which was to be set up for being the new gateway. After compiling the kernel for Iptables support and rebooting into it, I got the same error, only that the available options were different. It showed the only hda device on the system that is, the CD-ROM drive.

    Now after looking into a lot of forums, I decided there wasn't any clear cut solution provided on how to solve the problem. Unless you really DID append an incorrect “root=” option, the problem lies in the fact that your kernel is not able to recognize your SATA device (sda). Chances are, it's been compiled for IDE support.

    The solution? Compile your kernel for SATA support.

    $: cd /usr/src/linux
    $: make menuconfig

    Device drivers--->
    SCSI device support--->

    SCSI target support
    [*] legacy /proc/scsi/ support
    *** SCSI support type (disk, tape, CD-ROM) ***
    SCSI disk support
    SCSI CDROM support
    [*] SCSI low-level drivers --->

    ACARD SCSI support
    Adaptec AIC7xxx Fast -> U160 support (New Driver)
    (32) Maximum number of TCQ commands per device
    (5000) Initial bus reset delay in milli-seconds
    [*] Compile in Debugging Code
    (0) Debug code enable mask (2047 for all debugging)
    [*] Decode registers during diagnostics
    Adaptec AIC7xxx support (old driver)
    Adaptec AIC79xx U320 support
    (32) Maximum number of TCQ commands per device
    (4000) Initial bus reset delay in milli-seconds
    (0) Debug code enable mask (16383 for all debugging)
    The options under SCSI low level drivers depend on your device. These are the options that I'd set according to my hardware. If you're not able to figure out what the right options are, select everything! As far as the gateway was concerned, I'd selected almost everything which looked sensible enough.


    $: make clean && make && make modules_install
    $: cp arch/x86/boot/bzImage /boot/your-kernel-version

    The last step would be to update grub.
    $: vi /boot/grub/menu.lst

    title Your Linux
    root (hd0,0)
    kernel /boot/your-kernel-version root=/dev/sda1

    Hope this little write up proves helpful. Cheers. Feel free to post a comment if you have any problems.