Creating Command Line Tools with Python

After spending considerable amount of time on a computer as a developer, the GUI starts to seem amazingly slow and you realize just how awesome command line tools can be. Especially if you follow the Unix philosophy of “one tool for one task”, you can quickly chain together multiple in-built tools and quickly accomplish most tasks without a single line of code.

However, what about when the tool you need doesn’t exist? Wouldn’t it be great to just create it? In this post, I’d like to show you how using Python. My reason for using python instead of the more native C/C++ is simple: easy to read code for even anybody new to programming to understand the basic structure, fast prototyping, a rich set of tools, and the general ease of use. You’re of course welcome to use other languages such as Ruby (I’ve seen a lot of great tools written in Ruby), but then you wouldn’t be reading this post, would you? Without further ado, let’s begin!

Project Setup

I faced a problem a couple of weeks ago where I needed a simple command to nuke a directory in order to rebuild some binaries and unfortunately no tool existed for this at the time. Hence I created nuke, a convenient command line tool that does exactly what it says, nuke directories.

To start, we create a project directory called nuke-tool. Inside nuke-tool, we create a directory called nuke and inside that directory, two files, nuke.py which will house all our main code logic and  __init__.py so that python is able to understand this is a package called nuke.

In the root of the project directory, we should also create a file test_nuke.py to create tests for our code. Don’t want to accidentally nuke something else now, do we? We also create a file called setup.py to aid pip in installing nuke. We’ll flesh these out one at a time.

I did mention a rich set of libraries to help us out in this process. I personally use argparse to help with argument parsing and clint from the awesome Kenneth Reitz to help with output text formatting and confirmation prompts (simple Yes/No prompts). Later on, you’ll see code snippets showing these libraries in action.

Nuke ’em

In any command line tool, the first thing we need is a way to parse command line arguments. It’s the arguments you pass to a command which are separated by spaces. For example, grep takes two arguments, a pattern and a file name grep main nuke.py.

In python, we can easily use the argparse module to help us create a simple argument parser. I wanted to provide two options to the user, an argument specifying the directory to nuke (with the default being the current directory), and a flag -y to override the confirmation prompt. The code to do this is:

import argparse
import os
.
.
.
def _argparse():
    parser = argparse.ArgumentParser("nuke")
    parser.add_argument("directory", nargs='?', default=os.getcwd(),
                        help="Directory to nuke! Default is current directory")
    parser.add_argument("-y", help="Confirm nuking", action="store_true")
                        args = parser.parse_args()
    return args

Continue reading

HTTP Libraries – Your Best Friends

For a programmer like me, the web is a playground like no other. Not just the information that can be retrieved, but the sheer amount of services and APIs available online make getting structured, useful data a matter of reading a few documentation pages. From social data like Facebook and Twitter to data services like Xbox Music, the web is a repository of rich information that can be manipulated as per your whim.

So while the data is omnipresent on the web, the bigger task is pulling that data from the stores they reside in. This is where the HTTP protocol comes in. With its basic operations like GET, POST, PUT and DELETE, you have a handy mechanism to retrieve, push or manipulate the data on the servers. Now, the HTTP protocol is not as simple a protocol as this blog post makes it seem. Things like HTTP Headers, application types, authentication tokens and other similar things make the task of HTTP based retrieval slightly more complex than it must have been intended to be.

This is where HTTP libraries come in. They handle the most of the heavy lifting and let you focus on the kind of content you are dealing with on a per API basis while abstracting away the inner complexities of packet and protocol rules. As someone who has been doing a lot of development in C# and Python, the HttpClient library in the .NET framework and Python’s Requests libraries have helped me unleash the raw power of the web countless times. To give you a glimpse of this power, you can take a look at how the HttpClient library handles various forms of HTTP content in this Stackoverflow answer.

The simplicity of Python’s requests library makes rapid prototyping with new web APIs a cinch. Simple methods like requests.get() and requests.post() (which do exactly as the names suggest) let you examine in real time the data coming in. This knowledge can then be leveraged to build more robust languages either in Python or in enterprise level languages like C#. The HttpClient library brings amazing power in terms of its ability to not only do various kinds of HTTP requests, such as upload images programmatically, but also manipulate headers and metadata effectively, thus helping you quickly get off the ground with your application. Add to that C#’s asynchronous operation support and you’ll be developing .NET software full-time.

Now while I learn Ruby, I have already exposed myself to the Net::Http library of Ruby to take advantage of this newly discovered ability. I personally will definitely make it a point to master the HTTP libraries of whatever language I use along with the defining features of the language. Accordingly, especially if you are a new programmer, I urge you to do the same and reward yourself with this power and flexibility to leverage existing information to build upon, create cool things and make the world a whole better place!

Eviva!

Understanding Python Decorators in 12 Easy Steps!

A really fantastic explanation to one of the more trickier concepts in Python. A must read if you want to either become a better Python Hacker or just learn better, more modern design principles!

simeonfranklin.com – Understanding Python Decorators in 12 Easy Steps!.

Birthday Wish NLP Hack

Well, it was my 22nd birthday 11 days back, and while the real-world was quite uneventful, I managed to create a small stir in the virtual-world.

For this birthday, I decided to do something cool and what is cooler (and a greater sign of laziness) than an AI program that replies to all the birthday wishes on my Facebook wall? This was definitely cool and quite possible given a basic understanding of HTTP and some Artificial Intelligence. After experimenting for 2 days with the Facebook Graph API and FQL, I had all the know-how to create my little bot.

Note: This is from a guy who has never taken a single course on Natural Language Processing and who has next to zero exposure programming NLP programs. Basically, I am a complete NLP noob and this hack is something I am really proud of.

But one major problem still remained: How to create a NLP classifier that would classify wall-posts as birthday wishes? I tried looking for a suitable dataset so I could build either a Support-Vector Machine or Naive Bayes Classifier, but all my search attempts were futile. Even looking for related papers and publications were in vain. That’s when I decided to come up with a little hack of my own. I had read Peter Norvig’s amazing essay on How to Build a Toy Spell Checker and seen how he had used his intuition to create a classifier when he lacked the necessary training dataset. I decided to follow my intuition as well and since my code was in Python (a language well suited for NLP tasks), I started off promptly. Here is the code I came up with:

def bday_wish(message):
""" Analyze message to ascertain if it is a birthday wish
Params:
meassage: The message received from FB friend
"""
keywords = ["birthday", "birth", "happy", "bday", "hpy", "hbd", "hb", "happi", "best", "memorable", "awesome", "fantastic", "super", "bappy", "hirthday", "returns", "many", "bless", "god", "belated"]
expansion = {"hbd":["happy", "birthday"], "hb":["happy", "birthday"]}
s = ''.join(c for c in message if c not in string.punctuation and c in string.printable)
t = s.lower().split()
l = []
for x in t:
if x in expansion.keys():
l.extend(expansion[x])
else:
l.append(x)
count = 0
for x in l:
if x in keywords:
count += 1
continue
else:
for k in keywords:
if x.count(k):
count += 1
break
if count >= 2:
return True
else:
return False

The first thing I do is create a list of keywords one would normally find in a birthday wish, things like “happy”, “birthday” and “returns”. My main intuition was that when wishing someone, people will use atleast 2 words in the simplest wish, e.g. “Happy Birthday”, so any messages just containing the word “Happy” will be safely ignored, and thus I simply have to check the message to see if atleast 2 such keywords exist in the message.

What I do first is remove all the punctuations from the message and get all the characters to lower-case to avoid string mismatching due to case sensitivity. Then I split the message into a list of words, the delimiter being the default whitespace. This is done by :

</p>
<p>s = ''.join(c for c in message if c not in string.punctuation and c in string.printable)<br />
t = s.lower().split()</p>
<p>

However, I later realized that there exist even lazier people than me who simply use wishes like “HBD”. This completely throws off my Atleast-2-Words theory, so I add a simple hack to check for these abbreviations and put in the expanded form into the message. Thus, I created a dictionary to hold these expansions and I simply check if the abbreviations are present. If they are, I add the expanded form of the abbreviation to a new list that contains all the other non-abbreviated message words added in verbatim [lines 15-20]. Since I never check for locations of keywords, where I add the expanded forms are irrelevant.

Then the next part is simple, bordering on trivial. I iterate through the list of words in my message and check if it is one of the keywords and simply maintain a counter telling me how many of the keywords are present. Python made this much, much easier than C++ or Java.
But alas, another problem: Some people have another bad habit of using extra characters, e.g. “birthdayyyy” instead of “birthday” and this again was throwing my classifier off. Yet another quick fix: I go through all the keywords and check if the current word I am examining has the keyword as a substring. This is done easily in Python strings using the count method [lines 31-34].

Finally, I simply apply my Atleast-2-Words theory. I check if my counter has a value of 2 or more and return True if yes, else False, thus completing a 2 class classifier in a mere 40 lines of code. In a true sense, this is a hack and I didn’t expect it to perform very well, but when put to work, it really managed to do a splendid job and managed to flummox a lot of my friends who tried posting messages that they thought could fool the classifier. Safe to say, I had the last laugh.

Hope you enjoyed reading this and now have enough intuition to create simple classifiers on your own. If you find any bugs or can provide me with improvements, please mention them in the comments.

Eviva!

Watermarking – Truly Transparent Text

Well, I just finished writing up a new software project. It wasn’t something really difficult, just a tool to help people watermark multiple images at once, made at the behest of some photographer friends of mine due to the lack of such a tool on the net. While the tool was pretty straightforward (and a great exercise in Software Engineering), what was really interesting was the way to create the watermark, which required me to make the text transparent to a certain degree. Ofcourse, I had to search for the right way to do it, but again nothing straightforward cropped up (this is becoming really common now) and while I did find some useful code snippets, they did not do exactly what I wanted. Thankfully, on reading the code, I was able to gather enough information about how to construct a basic watermarking algorithm works as well as how to manipulate the alpha value of images to achieve the transparency.

First some Image basics. Every image you see in the digital form is represented by pixels (picture elements in short), and each pixel has 4 values: 3 values which specify how much or Red, Green and Blue should be present in that pixel, and the 4th value is the alpha value, which determines the Opacity/Transparency of that pixel. RGBA in total. Now the alpha value is key here, and once I understood how the alpha value is manipulated, creating the Image Processing module was a cinch. For this example, I used the PIL library of Python.

What I first did was declare the colour and transparency of the text which would be used as the watermark. This was as simple as specifying the tuple (0, 0, 0, trans), where trans is my transparency value. Next, I create a completely transparent white image the same size as my input image. By specifying the RGB values as 255 each, the image was a plain white image, but by specifying alpha as 0, the image was truly transparent. Now comes the fun part: PIL has something called an ImageDraw module which allows one to draw text or other shape onto an image using an instance of a Draw object on the image. So I just use this Draw object on my transparent image,  using the .text method to draw the specified text at a particular position. This gives me a transparent image (or a canvas if you may), with just some text on it and nothing else seen. Remember, the image is transparent so you should not see any white or any other colour, but the text is as transparent as specified by the trans variable. But there is a slight problem, as the program ignores the alpha value  when displaying and manipulating the image. This is easily solved by using something called masking as described in the next chapter. However, we can still assume our image to be truly transparent.

Finally, I use the .paste method of my original image to paste my transparent image onto my input image. In the paste method, the most important thing is the 3rd argument which is the mask. The mask simply specifies which parts of the image being pasted should be actually pasted. The .paste method uses the alpha channel of an image to determine the mask, and since everything but the text has an alpha value of 0, only the text is pasted onto my input image. This results in simply my input image having some text on it, without a whitish blur that ruins your hard taken photo. Since both the images are of the same size, it means that the location you put your watermark will be preserved on pasting it.

Here’s the code:


from PIL import Image, ImageFont, ImageDraw

def watermark(img_file, text, wfont, text_pos, trans):

    """
    Watermarks the specified image with the text in the specified font and on the specified point.
    """

    # Open the image file
    img = Image.open(img_file)

    # The Text to be written will be black with trans as the alpha value
    t_color = (0, 0, 0, trans)

    # Specify alpha as 0 to get transparent image on which to write
    watermark = Image.new("RGBA", img.size, (255, 255, 255, 0))

    # Get a Draw object on my transparent image from the ImageDraw module
    waterdraw = ImageDraw.Draw(watermark, "RGBA")

    # Draw the text onto the transparent image
    waterdraw.text(text_pos, text, fill=t_color, font=wfont)

    # Paste the watermark image onto the input image img, using watermark image as the mask
    img.paste(watermark, None, watermark)

    return img

So you can see that the code is fairly straightforward. Now remember, this is just a demo to give you a basic idea of how to achieve watermarking. There will be similar libraries that allow you to do the same thing in almost every programming language, so all you have to do is apply the concepts. And then you too can get an amazing watermark like this:

Watermarked Image

An image that I watermarked using my own program.

Using the above ideas and techniques, I was able to code up my Watermarking tool in about 14 days, with a fun GUI and efficient processing. I have open sourced it on GitHub and hopefully I can expect you to be a contributor on it.

Eviva!

Face Detection with OpenCV 2

Finally done with my Senior project at college and wow, it was great! Will definitely post about it soon and hopefully will be a good apology for the tardiness of this post.

My project was on Facial Expression Recognition and for starters, I had to process images to detect a face(s) in the image. This is where OpenCV really came useful, since it has built-in capabilities of detecting faces, using something called a Haar Wavelet (which is basically a square wave that you use to apply some digital signal processing to your image) and Cascade Classifiers (a bunch of weighted machine learning classifiers used together).

Now, for its ease and convenience, as well as to maintain forward compatibility with future versions of OpenCV, I decided to ditch the OpenCV1.0 API in favour of the OpenCV2+ API, and moreover, I decided to do it in Python since the numpy library proves very powerful for linear algebra operations and rapid prototyping is easy with Python. But now here comes the big problem: OpenCV 1.0 was really popular (I guess it was because of the Kaehler-Bradski book) and almost all the links on the web show code in OpenCV 1.0 on how to detect faces (which is quite ugly compared to the 2.0 version), be it C or Python. So I jumped right into OpenCV2’s wonderful documentation, but alas, only code in C++. Not to fear, since I have good proficiency in both languages now, and it was just a matter of translating code (This is where iPython came really, really useful for its auto-complete functionality) .

So here I will show you how to detect faces in images with OpenCV2 and Python as succinctly as possible, due to the lack of more documentation, and hopefully you’ll come up with the next-big-thing!

from cv2 import *

# Locate face in image. Using cv2 functions
def detect_face(image):

    # Specify the trained cascade classifier
    face_cascade_name = "/usr/local/share/OpenCV/haarcascades/haarcascade_frontalface_alt.xml"

    # Create a cascade classifier
    face_cascade = cv2.CascadeClassifier()

    # Load the specified classifier
    face_cascade.load(face_cascade_name)

    #Preprocess the image
    grayimg = cv2.cvtColor(image, cv2.cv.CV_BGR2GRAY)
    grayimg = cv2.equalizeHist(grayimg)

    #Run the classifiers
    faces = face_cascade.detectMultiScale(grayimg, 1.1, 2, 0|cv2.cv.CV_HAAR_SCALE_IMAGE, (30, 30))

    print "Faces detected"

    if len(faces) != 0:            # If there are faces in the images
        for face in faces:         # For each face in the image

            # Get the origin co-ordinates and the length and width till where the face extends
            x, y, lx, ly = face[0], face[1], face[2], face[3]

            # Draw rectangles around all the faces
            cv2.rectangle(image, (x, y), (x + lx, y + ly), cv2.cv.RGB(155, 255, 25), 2)

    # Display the images with the faces marked
    cv2.imshow("Detected face", image)

    cv2.waitKey(0)

    return (x, y, lx, ly)


def main():

    # Specify the image to process and pass it to the function
    detect_face(imread('../sample.jpg'))


if __name__ == "__main__":
    main()

And there you go. In just a few dozen lines of code, you have a simple to understand and elegant face detector. It definitely works since I tested it out on plenty of datasets that I had obtained for my project as well as some personal images! So try it out and let me know about your thoughts in the comments.

Eviva!

TinyOS Installation Made Easy

Howdy people! Long time since a post but couldn’t help it, busy being a busybee.

Well, I have started a project on Wireless Sensor Networks and the best way to go about it has been using an amazing operating system called TinyOS. This OS is so tiny, it uses only 400 kb of memory to run! Holy Cow!!

But getting TinyOS on 64-bit machines hasn’t been well documented and I assume that lack of documentation brought you here. Well no fear, I will be showing a stepwise guide to getting TinyOS up-and-running on any 64-bit machine you have.

Note: I am using Ubuntu 11.10 64-bit as my host operating system, but the method should work on all distros of Linux.  If you’re using Windows, may God have mercy on your soul. Also, this post involves installing TinyOS 2.1.1. I cannot guarantee that all the steps will work as is for other older versions. You’ll have to do some research for that.

Step 1:

First up, we need Java, the JRE and the JDK as a lot of the TinyOS and Mote communication frameworks are built in Java. You can use the OpenJDK but I have found using Sun’s original JDK yields better results!

Open up a terminal and simply type in these commands to get the JRE and JDK installed in your machine.

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:ferramroberto/java
sudo apt-get update
sudo apt-get install sun-java6-jdk sun-java6-plugin

You can choose the appropriate JVM by running the following command in the terminal:

sudo update-alternatives --config java

Step 2:

Now that we have our environment setup, we are ready to install the TinyOS development environment.

  1. Install the Synaptic Package manager by running “sudo apt-get install Synaptic” in a terminal.
  2. In Synaptic, go to Settings ->Repositories.
  3. Click on 3rd party software and add the following (simple copy-paste):
    deb http://tinyos.stanford.edu/tinyos/dists/ubuntu * main
  4.  Click OK and reload your repositories.
  5. After reloading, simply search for TinyOS and select it. Click install and allow Synaptic to handle dependency installation for you.
  6. Once the packages have been properly installed, go back to a terminal and type “sudo gedit ~/.bashrc” and add this line at the absolute end:
                         source /opt/tinyos-2.1.1/tinyos.sh
  7.  Finally, close the current terminal and open up a new one and type in $MAKERULES. If all went well, you should get the path.

Congrats, we have TinyOS ready. But we’re not done yet!

Step 3:

Now, when writing TinyOS programs, you’ll very much want to simulate them to see what are the results. A very handy tool for TinyOS simulation is TOSSIM. But when you run the make micaz sim to generate your executable you might get a Python related error. Try these steps:

  1. In a terminal, run sudo apt-get install python-dev.
  2. Even after installing python, some configuration is required.
  3. Run python –version  and note the version of Python installed. In my case, it was 2.7.
  4. Go to /opt/tinyos-2.1.1/support/make/
  5. Open the sim.extra file in gedit with super user privileges.
  6. In the entry labelled PYTHON_VERSION, check to see if the entry matches with the Python version installed. For me, it was 2.6, so I changed it to 2.7.
Python Configuration

The Python version entry to be edited

Step 4:

As the final step, this involves some minor configuration to your tinyos.sh file found in /opt/tinyos-2.1.1. This is done so that when you run the make micaz sim command to generate TOSSIM related files, the appropriate Java classes are generated to help in the simulation. Otherwise you’ll get a long list of errors that will make no sense at all.

Editing the tinyos.sh file makes the change permanent which is exactly what you need. Simply go to the directory in which the tinyos.sh file is located (given in the start of  this step).

Open the tinyos.sh file in gedit with super user privileges. There is an entry called CLASSPATH. Simply edit the original to $TOSROOT/support/sdk/java/tinyos.jar, similar to how it is in the image.

CLASSPATH Edit

Change to CLASSPATH entry in tinyos.sh file

And voila!! You’re done and ready to start writing some incredible TinyOS code.

I hope this blog entry has been useful to you. For any doubts or problems, the TinyOS mailing list is a great place filled with really helpful people and I highly recommend posting both newbie and advanced doubts there.

Till next time, Eviva!

———————————————————————————————————————————
P.S. These are some of the links I used to get TinyOS up and working. Thought they might be useful for the future:

http://superuser.com/questions/353983/how-do-i-install-the-sun-java-sdk-in-ubuntu-11-10-oneric
http://mythicalcomputer.blogspot.in/2008/08/installation-of-tinyos-in-ubuntu.html
http://mail.millennium.berkeley.edu/pipermail/tinyos-help/2011-October/052537.html
https://www.millennium.berkeley.edu/pipermail/tinyos-help/2010-November/048947.html