RIP B.D. Manocha

My last living grandparent - my paternal grandfather - B.D. Manocha - pitaji - passed away on the 18'th of May, 2009. He was 95 years old.

This was coming - he had severe Alzheimers, had suffered a stroke that had paralyzed the left side of his body, was very frail and pretty much fully dependent on others. It is because of this that I feel this was a good thing. He had led a long and (for the most part) healthy life - lived long enough to see his great grand children (the oldest of which is now almost 19 years old), seen all his grand children (save me and my brother) get married and been there when his wife passed away. His passing away relieved both my uncles (his two oldest sons) from having to constantly worry about him (though, amazingly, I never heard them complain - infact they often fought over who would be the one taking care of him).

I debated over the content of this post - should this be a eulogy or a detailing of my trip to India to perform his final rights. I will give both a shot, starting with the former.

I didn't really know pitaji too well. From the time I remember things, I knew him as a deeply religious man - so much so that he almost completely ignored everything else - including his wife (for example, I don't remember a single time when both of them slept in the same room, not sure if he did it earlier in his life, but definitely not since I remember anything about our family) and his youngest grand children. The most vivid memories of him I have are the ones where he used to come to our part of the house for dinner or when he used to be the one chairing the prayers whenever a common one took place in the house. My older cousins tell me he was a strict taskmaster - he used to see if they finished their studies, were disciplined etc. Non compliance usually meant they would be hit with a cane. He didn't take non-compliance too well (this is all too evident in my father too) - as was witnessed by him scolding us or our mom when we used English instead of Hindi (he was a Gandhian). He was also obsessive-compulsive about his health (this may be why he lived to this age). He was never one to try to build a fun and playful relationship with us. I remember my younger self asking my mother how come pitaji didn't play with us like the grandfathers in the movies played with their grandkids. However, by the end of things, he had changed significantly - everytime I visited, he would enthusiastically hug me and ask me to sit by his side. I'm not sure if he knew who I was, but he liked people sitting with him, holding him or holding them in an embrace. He had also, as far as I could tell, given up on religion (this could by my bias) as well as on good health (part of this was 'cause he couldn't walk after his stroke, but he also started loving sweets - when he used to not eat much more than 3-4 spoon fulls of the stuff earlier). I guess, I had more fun being around him in his later years.

He passed away on Monday, 18'th May. My dad left for Delhi the same day. My mom followed the next day (Tuesday) and me the day after (Wednesday). He was cremated on Tuesday morning. We went to collect his ashes the morning after I arrived - almost all the males in the house. This was done with all pomp, ritual and ceremony - none of which I understood. We laid out different kids of food on all 4 corners of the rectangle as well as the 4 mid points of the sides of the rectangle. This was followed by some shlokas being read by the in-house pandit. We then did what's know as phool chun-na (pick the flowers). This involved saperating the bones from the rest of the ashes in the pit. We then headed home with the ashes as well as the bones. The afternoon saw a 3 hour rendition of the Bhagvat Gita followed by lunch. This was mostly attended by family members as well as mid to old aged housewives from the neighbourhood (given that it was a thursday afternoon, most of the men were apparently unavailable to attend). We then had an hour's respite and then headed to a temple where a more public ceremony was held - attendance was actually quiet good. The ceremony involved 45 minutes of funeralistic songs followed by a 20-25 minute eulogy delivered by the family priest. The ceremony ended by 6:30pm, after which we all returned home, where plans to go to Haridwar to put his bones in the Ganges were made. 11 of us in 3 cars were to go. We all left at different times the next morning, but ended up meeting just oustide of Haridwar. The drive there in my uncle's Wagon-R was actually pretty taxing - with the A/C not working fully well and the tempreatures soaring over 40C. Once we reached Haridwar, we checked into our hotel, rested for a few minutes and then headed down to the main river bank (called Har-ki-pauri) to put his ashes in the Ganges. Our priest met us at the banks - who then performed another 10-15 minute prayer. We then gave his ashes to the Ganges and then everybody but me, a cousin and his wife, followed it up with a dip in the river. We then headed back up to the hotel, took a shower and headed down to meet our Haridwar priest. This is the guy you go to to register a death or birth (though the latter rarely happens) in the family. He has records for the Manocha family going back 3-4 generations - apparently. This done, we went for a stroll in the local market. We rushed back to the main bank of the river by 7pm for the aarti held daily. The place was packed which meant people were rubbing up against everybody. Rather distressing but I survived what was a - fortunately - short prayer. A quick trip back to the hotel, we headed back out for dinner after freshening up. We had dinner at a Dhaba type place and then headed back to the hotel to call it a night. The next morning involved another dip in the river for most everybody and then checkout followed by our drive back home - which was plagued by the same ills that troubled us on our way in.

That was it for the performing last rites part of the trip. Many things about these 3 days pissed me off. Not least of them was the fact that after deciding to not go to India for this hoopla, I changed my mind after a call from my dad asking me to come. Now maybe I was being naive, but I thought he asked me to come 'cause he needed me there - somebody to talk to and stuff. It turns out I was window dressing - a body to show that the grandkids from our end of the family haven't all suddenly gone AWOL (since grandkids from the other two brothers were in attendance). But as I said, I was probably being naive to expect otherwise. Anyway, many of the rituals irked me too. The putting food on all corners of his pyre, the picking of his bones from the ashes, the floating of said bones into the Ganges (no wonder the water there was so nasty), the singing of the songs, the pseudo-eulogy delivered by the priest instead of his kids etc. I am sure if I had been there for the cremation, I'd have had problems with that whole process too. But I guess this is to be expected - I believe dying is the end of things - nothing lives on. You remember the person the best way you can without acting like you need to help him gain passage into heaven. Suffice it to say, I feel I would have been better off not going to India - just mourning his death from Kuala Lumpur, in a way that I feel acceptable, instead of going along with what others think I should do at such a time. Alas, that didn't happen.

My grandfather passing away is a bittersweet moment - it's sad but probably for the best. I can't say I'll miss him, but I will say may he Rest in Peace.

Using Twitter and Google Search API's using jQuery

Google and Twitter (API docs) both have API's available for web developers to create mashups using data from them. As a way to give users a broader view of what is happening around the constituency, state, party or representative they are viewing, GovCheck shows 4 searches on each of these pages - News, Web and Video search from Google and real time search from Twitter. These searches are all performed using the jQuery JavaScript framework. This has two advantages - the searches don't hold up page loading (as they happen asynchnously - AJAX :)) and they happen on the client side which means fewer resources (and hence load) used on the server side. So how exactly is this done?
The technology making this possible (besides the languages and TCP etc.) is JSONP. What this does is allow us to send AJAX requests to remote sites (due to security issues, browser do not allow cross domain AJAX requests. JSONP is a protocol that allows us to do so - although it has it's own issues too). Both Google as well as Twitter search support JSONP, as does jQuery - which makes querying those services from the client side very easy. Here's how to go about it.

function getGoogleContent(elem, searchtype, query) {
   $.getJSON('http://ajax.googleapis.com/ajax/services/search/' + searchtype + '?v=1.0&q=' + query + '&callback=?',
       function(data) {
           if(data.responseData.results.length == 0)
               elem.html($('
').html('No results').attr('style','text-align: center; font-weight: bold; font-size: 1.2em;'));
           else {
               elemliststr = elem.attr('id') + '-list';
               elem.html($('
').attr('id',elemliststr).attr('style','list-style-type: disc; padding-left: 30px;'));
               elemlistobj = $('#' + elemliststr);
               $.each(data.responseData.results, function(i, item) {
                   elemlistobj.append($('

').html($('').html(item.title).attr('href', (searchtype == 'video') ? item.url : item.unescapedUrl).attr('target','_blank')).append((item.publisher) ? ' (' + item.publisher + ')' : ' '));
               });
               elem.append($('
').html($('').html('More Results').attr('href', data.responseData.cursor.moreResultsUrl).attr('target', '_blank')).attr('style', 'padding-top: 10px; font-weight: bold; text-align: center;'));
           }
       }
   );
}

function getTwitterContent(query) {
   elem = $('#twitter-tab');
   $.getJSON('http://search.twitter.com/search.json?q=' + query + '&rpp=4&callback=?',
       function(data) {
           if(data.results.length == 0)
               elem.html($('
').html('No results').attr('style','text-align: center; font-weight: bold; font-size: 1.2em;'));
           else {
               elemliststr = elem.attr('id') + '-list';
               elem.html($('
').attr('id',elemliststr).attr('style','list-style-type: disc; padding-left: 30px;'));
               elemlistobj = $('#' + elemliststr);
               $.each(data.results, function(i, item) {
                   elemlistobj.append($('

').html($('').html(item.from_user).attr('href', 'http://twitter.com/' + item.from_user).attr('target', '_blank')).append(': ' + item.text));
               });
               elem.append($('
').html($('').html('More Results').attr('href', 'http://search.twitter.com/search?q=' + query).attr('target', '_blank')).attr('style', 'padding-top: 10px; font-weight: bold; text-align: center;'));
           }
       }
   );
}

We use jQuery's getJSON call to get the JSON results from these services. The "&callback=?" at the end of both the JSON queries tells getJSON that these are JSONP requests - jQuery automagically substitutes the name of the function the results of the query should be fed to.
The "elem" and "searchtype" arguments being sent to the "getGoogleContent" function allow me to call the same function for all News, Web and Video searches like so:

    getGoogleContent($('#news-tab'), 'news', query);
    getGoogleContent($('#youtube-tab'), 'video', query);
    getGoogleContent($('#web-tab'), 'web', query);

Another thing to note here is the usefulness of FireBug. When I was starting out making these queries, I relied on console.log to see the returned object structure (since neither Google nor Twitter specify it) which then allowed me to make the right calls for the relevant data.

So there you go - a quick, easy and scalable way of using Google and Twitter's API's to bring in information into your website.

Store pickled data in a database using SQLAlchemy

I've been using SQLAlchemy (and Elixir) to write the data collection code for GovCheck. My first choice would have been to use Django's ORM itself, but that proved harder and more time consuming that I thought worth it.

One of the problems that I've had to solve is how to store lots of static data in a SQL database without adding more than 100 columns into a DB table. A good solution to this problem is to store this data with a pickled dictionary. The problem with that is the in-built PickleType defined by SQLAlchemy does not also base64 encode the result of the pickle operation on the dictionary (I wanted to do this because the custom field type I'd defined within the Django app was something similar to this, which assumes base64 encoded data coming in). However, fortunately, it's rather easy to create custom Type's for SQLAlchemy and use them in your models. Here is the custom type I created to store the data.

import base64
try:
  import cPickle as pickle
except:
  import pickle

class EncodedPickleType(types.TypeDecorator):
  """
      This class should be used whenever pickled data needs to be stored
      (instead of using the in-built PickleType). The reason for this is
      that the in-built type does not encode the pickled string using
      base64, which is what the Django field type expects.
  """
  impl = types.Text

  def process_bind_param(self, value, dialect):
      dumps = pickle.dumps
      if value is None:
          return None
      return base64.b64encode(dumps(value))

  def process_result_value(self, value, dialect):
      loads = pickle.loads
      if value is None:
          return None
      if not isinstance(value, basestring):
          return value
      return loads(base64.b64decode(value))

You can then use this within your model definitions and feed it a dictionary. The field will take care of pickling and encoding it as well as decoding and unpickling the data when accessing the data.

Scraping PDF's in Python

So, in the course of grabbing some additional data sources for GovCheck, I needed to scrape a few pdf's and insert the information into my database. After looking high and low, I found an acceptable solution to do this using Python - pdfminer. It's not perfect, but it's much better than the rest of the pdf to html/txt converter tools - at-least as far as scraping goes. So I figured I'd note here how I wrote my scraping code. As a reference point, I was parsing election data for the past election using this pdf file.
You start off with running the code through pdfminer and getting the resulting HTML back.

import os
from BeautifulSoup import BeautifulSoup
for page in range(9, 552):
    soup = BeautifulSoup(os.popen('python ~/dev/pdfminer-dist-20090330/pdflib/pdf2txt.py -w -p %d Vol_II_LS_2004.pdf' % page).read())

Once you have said soup object, the next steps are pretty much the same as scraping any HTML page you would grab from some page on the web, except since the HTML generated by pdfminer user absolute positioning, you need to take care of those offsets. A couple ways I found to take those into account are listed below:

1. You can analyse a few pages of the generated html from the pdf, find the various offsets that are displayed and use them like so:



    for left_margin in (267,271,275):
        try:
            electors_total = map(int, map(strip, soup.find('span', style = 'position:absolute; writing-mode:lr-tb; left:%dpx; top:352px; font-size:8px;' % left_margin).string.split(' ')))
            break
        except AttributeError, e:
            pass



2. The previous method mentioned is actually a pretty bad way of doing things, not to say unreliable 'cause your code can break whenever an offset you did not know off is found. A better way to do things is to find a nearby constant text message (such as "MARGIN ") and then backtrack from there to find your relevant tag. This allows you to completely avoid hard coding offsets - unfortunately it's not usable in all the cases. Here's an example of how you can use this method:



self._data['Margin']['Number'], self._data['Margin']['Percentage'] = map(strip, soup.find(text = 'MARGIN ').parent.previousSibling.previousSibling.string.split(' '))

So there you go, a quick example of how to scrape a pdf using Python. It's not perfect but it works for me, for now.

A few things I've learnt from living with my parents...again

As stated earlier, I've been living with my parents for the last 3 months. A lot has transpired between us during that time - lot's of fights and make ups, bruised egos, defiance etc. Suffice it to say that it's not been easy - if I had to advise somebody if it'd be ok for them to go back to live with their folks after 6 years of not living with them, my answer will be a resounding NO!! Now to be fair, I'm not blaming them for this, it's a classic "it takes two to tango" situation. But all that's not the point of this post.

The subject of this post is what have I realized about myself living with my parents - more precisely, what kind of man I want to be and what kind of women I want to be with. I'll grant that basing these two personality traits solely on observing your parents might not be the best way to go about it, but they certainly do influence you - as they have me.

So, as I've been noticing the interactions between my mom and dad, one particular theme has become very clear - how submissive she is and how authoritative he is. She will bring him a glass of water/milk/coke (but never liquor for some odd reason) when he asks her to, she will arrange the kitchen (which in all honesty, should be her domain) as he likes her to, deal with the help as he wants her to, work or not work as he wants her to, cook as he wants her to - to the chagrin of everyone else, etc. etc. Reverse all that and you'll get a picture of what my dad's like. Even the most mundane stuff, like leaving an empty glass of beer on the side table next to the couch is for some reason unacceptable - as is eating in front of the tv or parking the car facing in instead of facing out or cooking the food a bit spicy or grabbing pickles with your meal - and yes, the list goes on. Now, I am almost certainly painting an exaggerated picture - it's not possible for me with my limited writing skills to paint the perfect picture - but this is the general trend.

Watching all this has made me realize things, if not influence me too. I've never been (at-least I'd like to think so) the kind of guy who likes to tell people how to do things or be told how to do things. It makes me uncomfortable when people ask me if they should do this thing or the other, and outright annoys me when they think they know what I should do. I can't stand people who think they know what's right for me, or ask me 10 times "why" I've done or decided to do something (a disease that plagues both my parents unfortunately). This line of thought, inherently leads me to like decisive people. And that's what I've realized - my ideal woman would be confident, decisive and unwavering. She won't feel the need to ask me what to or not to do. She'll also, hopefully, be the kind of person who would ask me to shut up and get back in my place if I try to interfere. Of-course, this does not mean I don't want her to ask me for suggestions - but that's exactly what I want them to be in her mind - suggestions. Conversely, I want to be the kind of man who will be trusting enough to not have to worry about the decisions my woman is making. Again, this is not to say that I don't want to be involved - I just don't want to be the guy "telling" her what to do. I would hate to become into an incarnation of my dad - thinking I know enough about everything to know what's right for someone else - screw how they feel about it.

Finally, let me say that I do not think their union doesn't work - it works for them. Maybe what I'm looking for is an aberration - it might be why I've been single for most of my life. That said, I'm not ready to settle, not just yet - I still have hope :).

So what happens after they win an election??

I'm not even going to bother with apologies - I'm a lazy retard and that's just the way it is. So anyways, I was watching Yuva again tonight, and it got me thinking - what happens after these so called educated do-gooders win an election?? Does that automatically mean things work out from there on out?

I mean, looking at the stats, one would see that we already do have a whole bunch of educationally well qualified individuals in the Lok Sabha (while GovCheck does not show educational data - yet - it will, soon) so there ain't a deficit of degrees in the Lower House (far as I've seen, a lot of them seem to have law degrees). This fact leads me to think that an education is not the panacea here. 

It's often said that the problem with India's laggard growth is not it's politicians but it's bureaucracy. If we take that statement at face value (which I do, to a large extent) then the problem is not having uneducated or criminal lawmakers (though that definitely does not help, in-fact it probably does hurt - just not as much as people might say) but having a bureaucratic process that is so void of accountability that even if we do get politicians who want to do something, there isn't much they can.

This is what disturbs me - it's not easy to get rid of this apathy people seem to have towards their government. Laws can only go so far - this needs a change of mind-set. And that is where I feel politicians and social workers can help - start asking the people to hold their government accountable. It'll be a slow process, but I definitely feel that is what's needed to fix the quagmire we're in as a nation.

Now what do I mean when I say "hold them accountable"? Well, for starters, make more positions in the government electable - posts such as mayor, district attorney etc. Also, allow people working for the government to be fired. As I understand it - presently it's so hard to fire people off the government payroll that they can pretty much go about doing anything with fearing the consequences. Finally, we need to have more investigative journalists (and not the kind of idiots who setup "sting operations") - not only at the national level - but at the local level too. The kind who will sift through city records, looking for dirt on that city council member, or the state representative. But none of these changes will happen unless the people actually ask for them (save maybe the last one) - which is why I say we need a change in the mind-set of people.

Indulge me now, if you will - this is the main reason why I started GovCheck. It's not something that was put up because there was an impending election in India. In-fact this is one of the things that piss me off about people - the ones pretentious enough to think they know why I started the website - because there's an election coming up. I couldn't care less about the elections - especially in a place like India - it's just a time for people to go around making promises and in some cases, pointing to the disgraces, they passed as law, as proof that they're better for the people. I don't care what they say during an election - a good lawmaker is one who's active off the campaign trail - and that's what I'm trying to bring to light with GovCheck.

So there we go - that is my answer to India's problems - not only better elected representatives, but a better and more aware electorate. That is the panacea for most of the diseases that plague our land - and I hope I can play a small part in curing it.