Creating Command Line Tools with Python

After spending considerable amount of time on a computer as a developer, the GUI starts to seem amazingly slow and you realize just how awesome command line tools can be. Especially if you follow the Unix philosophy of “one tool for one task”, you can quickly chain together multiple in-built tools and quickly accomplish most tasks without a single line of code.

However, what about when the tool you need doesn’t exist? Wouldn’t it be great to just create it? In this post, I’d like to show you how using Python. My reason for using python instead of the more native C/C++ is simple: easy to read code for even anybody new to programming to understand the basic structure, fast prototyping, a rich set of tools, and the general ease of use. You’re of course welcome to use other languages such as Ruby (I’ve seen a lot of great tools written in Ruby), but then you wouldn’t be reading this post, would you? Without further ado, let’s begin!

Project Setup

I faced a problem a couple of weeks ago where I needed a simple command to nuke a directory in order to rebuild some binaries and unfortunately no tool existed for this at the time. Hence I created nuke, a convenient command line tool that does exactly what it says, nuke directories.

To start, we create a project directory called nuke-tool. Inside nuke-tool, we create a directory called nuke and inside that directory, two files, nuke.py which will house all our main code logic and  __init__.py so that python is able to understand this is a package called nuke.

In the root of the project directory, we should also create a file test_nuke.py to create tests for our code. Don’t want to accidentally nuke something else now, do we? We also create a file called setup.py to aid pip in installing nuke. We’ll flesh these out one at a time.

I did mention a rich set of libraries to help us out in this process. I personally use argparse to help with argument parsing and clint from the awesome Kenneth Reitz to help with output text formatting and confirmation prompts (simple Yes/No prompts). Later on, you’ll see code snippets showing these libraries in action.

Nuke ’em

In any command line tool, the first thing we need is a way to parse command line arguments. It’s the arguments you pass to a command which are separated by spaces. For example, grep takes two arguments, a pattern and a file name grep main nuke.py.

In python, we can easily use the argparse module to help us create a simple argument parser. I wanted to provide two options to the user, an argument specifying the directory to nuke (with the default being the current directory), and a flag -y to override the confirmation prompt. The code to do this is:

import argparse
import os
.
.
.
def _argparse():
    parser = argparse.ArgumentParser("nuke")
    parser.add_argument("directory", nargs='?', default=os.getcwd(),
                        help="Directory to nuke! Default is current directory")
    parser.add_argument("-y", help="Confirm nuking", action="store_true")
                        args = parser.parse_args()
    return args

Continue reading

Easy Password Reset With Django

I’ve recently been playing around quite a bit with Django and full stack development. While this has not been primarily affiliated to my research, it has helped me work with other research groups that need custom web applications in order to deploy their work.

Django as a web framework is really powerful. With easy URI schemas, a built-in ORM with supercharged capabilities, and a plug-and-play, modular architecture, there seems to be no limit to what django can accomplish. However, I recently came across a stumbling block where I needed to do user password recovery and it for some reason, I find the documentation sorely lacking, while other blog posts have conflicting (and many times, incorrect) opinions on how to accomplish this seemingly simple task.

Here I’ll show you the django-ic way of doing password recovery for your site, which is both and easy and, more importantly, secure. Note that by recovery, we always mean a password reset, since showing you your old password in plain text is akin to asking someone to hijack your account. Assuming you have your user models set up and user registration and authentication working well, let’s step through it.

Models

This is the easiest bit. django handles password recovery with the User model directly, so let’s move on.

URL Conf

We first modify the urlconf so that we can naturally move onto the views from there. This is the most involved step of adding user password recovery, but it is more of configuration rather than code, keeping in line with django’s batteries included  policy. First we import django’s auth module to make our life a lot easier

from django.contrib.auth import views as auth_views

Password Reset

We add a URL that the browser connects to when the user clicks on a password recovery button.

url(r'^account/password_reset/$', auth_views.password_reset, {'post_reset_redirect': '/account/password_reset/done/', 'template_name': 'account/password_reset.html', 'email_template_name': 'account/password_reset_email.html'}, name="password_reset")

This creates a link /account/password_reset which calls django auth’s built-in password_reset view. The password_reset view accepts custom templates for the page as well as the email that gets sent out, so we pass those in as a dictionary (note the key values).

Password Reset Done

This is the page that you see that notifies you that your recovery email has been sent out.

url(r'^account/password_reset/done/$', auth_views.password_reset_done, {'template_name': 'account/password_reset_done.html'})

This is similar to the above, but we just change the template that we will use to one that gives the user the message clearly. Don’t worry about the link in the email being insecure, django creates a URL with a unique UIDB64 hashed identifier and manages it for you.

Password Reset Confirm

This is where we do the actual password reset. The user is presented with a form that asks them to input their new password and confirm it. django takes this form and updates the password securely without you having to touch a line of code.

url(r'^account/password_reset/(?P[0-9A-Za-z]+)-(?P.+)/$', auth_views.password_reset_confirm, {'template_name': 'account/password_reset_confirm.html', 'post_reset_redirect': '/account/password_reset/complete/'}, name="password_reset_confirm")

In the above URL definition, we accept the link that django sent out in its email with the unique UIDB64 hash and the unique token. To customize the response, we pass in a dictionary indicating what is the template with the form we wish to serve as well as where to submit the POST request of the form. The form gets posted to the link we define below.

Password Reset Complete

Once django takes in the new password and updates your password, it will redirect to this page to show that the password reset was successful.

url(r'^account/password_reset/complete/$', auth_views.password_reset_complete, {'template_name': 'account/password_reset_complete.html'})

The final URL, this one accepts the password recovery form as a POST request and serves up the confirmation template that displays to the user if the password was successfully reset or no.

Templates

Finally, we look at the design and tidbits of the templates whose names we were passing into the URLs above.

  • “Password reset” only has a form to accept the email address or username of the account whose password we wish to recover.
  • “Password reset done” is a simple message indicating that a recovery email has been sent out.
  • “Password reset email” is the template where you can define how your password recovery email looks like so that your organization looks more professional.
  • “Password reset confirm” should just be a template with an if block checking for `validlink`. If `validlink` returns True, we define a form that takes two fields, `form.new_password1` and `form.new_password2`. If not `validlink`, we display an error message stating that the recovery link is no longer valid.
  • “Password reset complete” is again a simple notification template which is displayed if the password recovery was successful.

Emails

The last part is configuring the email account from which to send the recovery email. Although django comes with great helper functions for sending emails, it does not have its own SMTP server which is needed to send emails.

Thankfully, we can save ourselves the trouble of setting one up by using a service such as Gmail or Outlook. Since Gmail is wildly popular, let’s use that.

Firstly, go ahead and create an account on Gmail which will serve as the account from which the emails will be sent out. Once you have the email ID ready, go to your settings.py in the main django app and put in the following settings –

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
# Indicates that we are using gmail
EMAIL_HOST = 'smtp.gmail.com'
# The port number for the gmail SMTP server
EMAIL_PORT = 587
EMAIL_HOST_USER = ''
EMAIL_HOST_PASSWORD = ''
EMAIL_USE_TLS = True
DEFAULT_FROM_EMAIL = ''

Please note that you should not be putting the password to your email account (or any password for that matter) in plain text in a code file, and then checking that into source control for the world to see. You should be inserting that password via environment variables or some other secure mechanism.

Conclusion

And that’s it! We’re done. Without writing any sort of complex cryptography code, we have incorporated a full password recovery system into our web app. This is one of the many cool things you can do with django, and hopefully, this guide has explained this feature in a way which was easy and concise.

Watermarking – Truly Transparent Text

Well, I just finished writing up a new software project. It wasn’t something really difficult, just a tool to help people watermark multiple images at once, made at the behest of some photographer friends of mine due to the lack of such a tool on the net. While the tool was pretty straightforward (and a great exercise in Software Engineering), what was really interesting was the way to create the watermark, which required me to make the text transparent to a certain degree. Ofcourse, I had to search for the right way to do it, but again nothing straightforward cropped up (this is becoming really common now) and while I did find some useful code snippets, they did not do exactly what I wanted. Thankfully, on reading the code, I was able to gather enough information about how to construct a basic watermarking algorithm works as well as how to manipulate the alpha value of images to achieve the transparency.

First some Image basics. Every image you see in the digital form is represented by pixels (picture elements in short), and each pixel has 4 values: 3 values which specify how much or Red, Green and Blue should be present in that pixel, and the 4th value is the alpha value, which determines the Opacity/Transparency of that pixel. RGBA in total. Now the alpha value is key here, and once I understood how the alpha value is manipulated, creating the Image Processing module was a cinch. For this example, I used the PIL library of Python.

What I first did was declare the colour and transparency of the text which would be used as the watermark. This was as simple as specifying the tuple (0, 0, 0, trans), where trans is my transparency value. Next, I create a completely transparent white image the same size as my input image. By specifying the RGB values as 255 each, the image was a plain white image, but by specifying alpha as 0, the image was truly transparent. Now comes the fun part: PIL has something called an ImageDraw module which allows one to draw text or other shape onto an image using an instance of a Draw object on the image. So I just use this Draw object on my transparent image,  using the .text method to draw the specified text at a particular position. This gives me a transparent image (or a canvas if you may), with just some text on it and nothing else seen. Remember, the image is transparent so you should not see any white or any other colour, but the text is as transparent as specified by the trans variable. But there is a slight problem, as the program ignores the alpha value  when displaying and manipulating the image. This is easily solved by using something called masking as described in the next chapter. However, we can still assume our image to be truly transparent.

Finally, I use the .paste method of my original image to paste my transparent image onto my input image. In the paste method, the most important thing is the 3rd argument which is the mask. The mask simply specifies which parts of the image being pasted should be actually pasted. The .paste method uses the alpha channel of an image to determine the mask, and since everything but the text has an alpha value of 0, only the text is pasted onto my input image. This results in simply my input image having some text on it, without a whitish blur that ruins your hard taken photo. Since both the images are of the same size, it means that the location you put your watermark will be preserved on pasting it.

Here’s the code:


from PIL import Image, ImageFont, ImageDraw

def watermark(img_file, text, wfont, text_pos, trans):

    """
    Watermarks the specified image with the text in the specified font and on the specified point.
    """

    # Open the image file
    img = Image.open(img_file)

    # The Text to be written will be black with trans as the alpha value
    t_color = (0, 0, 0, trans)

    # Specify alpha as 0 to get transparent image on which to write
    watermark = Image.new("RGBA", img.size, (255, 255, 255, 0))

    # Get a Draw object on my transparent image from the ImageDraw module
    waterdraw = ImageDraw.Draw(watermark, "RGBA")

    # Draw the text onto the transparent image
    waterdraw.text(text_pos, text, fill=t_color, font=wfont)

    # Paste the watermark image onto the input image img, using watermark image as the mask
    img.paste(watermark, None, watermark)

    return img

So you can see that the code is fairly straightforward. Now remember, this is just a demo to give you a basic idea of how to achieve watermarking. There will be similar libraries that allow you to do the same thing in almost every programming language, so all you have to do is apply the concepts. And then you too can get an amazing watermark like this:

Watermarked Image

An image that I watermarked using my own program.

Using the above ideas and techniques, I was able to code up my Watermarking tool in about 14 days, with a fun GUI and efficient processing. I have open sourced it on GitHub and hopefully I can expect you to be a contributor on it.

Eviva!

Difference in WCF Services

Whoa! Back from an amazing internship at Microsoft Corporation where I was given some truly great work. I had to build an Operations Tool for something called an ESB and I was supposed to use Silverlight with Windows Communication Foundation (WCF).

Now, the main issue I faced when developing in Silverlight was that I had a multitude of options when it came to choosing a WCF Service Library and the appropriate settings. Here is a list:

  1. WCF Service
  2. WCF Data Service
  3. WCF RIA Service
  4. AJAX-enabled WCF Service
  5. Silverlight-enabled WCF Service

Now this gets very confusing for someone new to .NET and Silverlight (which was me 3 months ago). But with a little exploration and search, you can easily find out the difference in all the options.

There are basically 2 types of WCF Services in actuality. However there are 5 variations present which leads to most of the confusion.

The two basic types of WCF Services are:

  1. SOAP Services – Here, there is a service metadata file that helps the client generate proxy classes and the messages being transferred are generally SOAP messages with Binary encoding an option via the NetTcpBinding binding.
  2. REST Services – This service allows for a greater variety of message formats, from binary and XML to JSON, since it uses the HTTP protocol. The drawback is that there is no standard and both the client and server have to agree on the data format. However this type of service is more lightweight.

I hope this makes it very clear. And now onto the 5 variations:

  1. Core SOAP services: Simply used to build a standard SOAP service.
  2. Core REST services: Used to build a standard REST service.
  3. Data services: This is just a variation of the REST service, where the data being transferred uses a common format (OData), and thus clients know how to consume a new data service even if there’s no document. No arbitary data is allowed and only AtomPub (XML) and JSON is supported.
  4. RIA services: With RIA services, you can create one service and define multiple endpoints. So you can have both SOAP and Data Services in one package through different endpoints for your application to access.
  5. Workflow services: A variation of SOAP service. It allows you to expose a workflow(a series of executions based on operation of the application. For more info, see Windows Workflow Foundation on MSDN) instead of code as a SOAP service.

Finally, AJAX/Silverlight – enabled WCF Services are simply templates that allow for easier creation and programming of WCF Services for Silverlight and AJAX applications, owing to their lack of compatibility with a some libraries. Thus, be sure to use these if you have a AJAX/Silverlight application, irrespective of what others say!

As I am sure you now understand, WCF Services are quite simple in their hierarchy and nomenclature. If you have solid fundamentals of Computer Networks and Internet Technology, all the above terms should feel like everyday words. Also, the power that WCF grants you for your application, not only to retrieve data and information, but to enforce good design in your application really makes it worth your while to check out this great technology from Microsoft and create something really useful and meaningful.

Eviva!

Soothsayer

I and a bunch of guys at my college decided to take part in the Atos IT Challenge and try to win ourselves a trip to the UK for the 2012 London Olympics, the theme being Smart Mobility.

Our college was one of the lucky ones to have been selected to participate in this contest and I can see the competition from my institute itself, if not the entire world, is pretty nerve-wracking.

My team consists of Arth “Vyarth” Patel, Nimit “Lame-it” Shah, Sunny “Not Funny” Shah, Jigar “The Dafda” Dafda and Yours truly.

Having vetoed the original idea for a mobile disaster management system, I suggested a smart app that can mine and learn from the User’s history, match that with the current trends in the world and filter it with his/her geo-location to give the User some truly meaningful information that is highly relevant to their current situation, with all the data-crunching taking place on the Cloud.

This competition gives us a great opportunity to learn about a new field, Mobile Computing, a new platform, and also a chance to practice some of the AI techniques we’ve picked up in the past few months. But the biggest plan is to possibly Patent our idea. That would be TOTALLY COOL :-D.

Our idea page is here. If you liked our idea, please don’t hesitate to click on the FaceBook Like Button.

Now let’s hope the judges deem our idea worthy of selection so we can make it into the next round and begin the development stage.

Eviva!

TCP/IP Stack Work: Connection But No Internet

Has it sometimes happened to you that you try to connect to a site, get an unable-to-load message and after checking everything, still end up with the same problem?

If the ans is yes, then welcome to the club! 🙂 The above particular problem happened to me when I had gone to Noida for Formula 1 and was staying over at my cousin’s place.

Now they have amazing net! My uncle is an engineer by profession and a Techno-freak by obsession. That’s what led him to install an unlimited 4 Mbps plan in his house, just so he could enjoy the full freedom of  the World Wide Web (He’s also a Mac user, if you were wondering).

So there I am, trying in vain to connect to the net so I can get some work done, but pages are unable to load. My connection status is showing connected, net is working on my cousin’s machine, Chrome AND Firefox have the same problem, rebooting doesn’t do a darn thing! What in the world is going on??

And so I did what any self-respecting computer engineer would do: I asked Google.

Turns out, my TCP/IP stack was garbled up and resetting it would be the ideal solution. Thankfully, there is no data loss of any kind in this process as you are just resetting the protocols that deal with Computer Networks rather than the data being transmitted. 😀

These are the commands I used: netsh int ip reset C:\netsh.log.txt
                                                                   netsh winsock reset 

This uses a tool called Network Shell available in Windows (if this was Linux, I wouldn’t be having this problem in the first place :-P) and it simply resets the TCP/IP stack using the netsh.log file as a reference and then resets the WinSock API so that the Windows Programs continue working normally with the reset. Consider the second line as telling Windows that “Hey, I just reset the TCP/IP stack, so you better make sure you are compatible as well”.

And there you go! Problem solved. A similar trick in Linux is to use the following command in a terminal: service networking restart

Eviva!

Easy LAMP

Well, first things first: This is not a method to create your own torch! I can understand the confusion with all the recent black-outs and power trips in the hostel.

What LAMP is, is given by its full-form: Linux, Apache, MySQL, PHP. So basically this is a platform on linux for easy set-up of a web-server, with all the files in one place!! NICE (Especially considering all the recent interest in web-development by people in my college)!

L is for Linux, so yes, this method is for your linux distributions (specifically Ubuntu). Here we go:

  1. First open a terminal and type “sudo apt-get install lamp-server^” to install the LAMP server.
  2. When prompted, enter a password that will become your MySQL root password.
  3. When done, type at the terminal “sudo vi /var/www/info.php“. This is to open the default info file in PHP which is stored on your LAMP server.
  4. Put in the following lines:  <?php phpinfo(); ?> and save & close the file.
  5. Restart Apache with the following command: sudo etc/init.d/apache2 restart.
  6. And finally check if your server is running by opening your favourite web-browser and typing in the address bar : localhost/info.php. If a page opens with the PHP logo at the top, congratulations! Your server platform is ready to go!!

And now you can begin your much awaited web-development! 🙂

For more info, here are some links: http://goo.gl/in4Dx, http://goo.gl/JMEsj. They also explain how to install phpmyadmin to have complete control over your LAMP server.

Codifica Felice!