Arch Linux – First Impressions

So thanks to the recent Creator’s Update from Microsoft, my Ubuntu partition became inaccessible due to what I believe is a dirty bit being set. After recovering all my data and ranting about the ordeal online, I decided to reinstall Linux so I could get back to work. That’s when it hit me that I finally had the opportunity to really get Arch Linux working on my laptop and make it my primary distro moving forward. After silently thanking Microsoft’s bad update, I got to work.

Now fair warning, installing Arch is crazy hard, simply because you have to do everything yourself. All the way from configuring the BootLoader to partitioning your device to mounting your filesystem, it’s a lot of work and very overwhelming at first. Thankfully, the Arch Wiki is one of the greatest resources for all things Linux on the internet today.

After 4 painstaking hours of going through mounds of documentation and double checking various things to make sure I wasn’t making mistakes, I finally had Arch installed. I did document each step to streamline the process so hopefully it should take far less time the next time I attempt this herculean labour again. I now had a fully functioning Arch Linux desktop.

Finally, I began installing the necessary packages to set up my development environment. Let me go ahead and say this, pacman is one of the best package managers out there! Every package installation was smooth, the output messages were clear and meaningful, and I could see the benefit of the Arch philosophy since I always had the latest version of everything.

Software that did not have dedicated packages could be easily installed using the ArchLinux User Repository and the makepkg command, which is simply just git cloneing a repo and running a make like command on it to install the software. Indeed, this is how I installed a lot of packages such as VS Code.

The best part about Arch though was the speed. My previous Ubuntu installation would often lag on the UI end, but even with the GNOME desktop, Arch was smooth as butter.

Battery life also improved significantly. It helped that Arch was so trimmed down and didn’t waste time installing useless junk I wouldn’t use.

The main downside to Arch was its lack of mainstream support. ROS isn’t as easy to install, RVM was considerably broken due to its own fault of not supporting OpenSSL 1.0, and lots of the latest packages were just simply missing. While I hope Linuxbrew gains more momentum as the de facto way to install packages, my productivity did take a bit of a hit.

That said, Arch Linux was well worth the patience and burden to get installed. Given the option, I would always choose Arch and encourage others to do the same. Hopefully, a time will come when Arch does become mainstream and the Arch way continues to prevail without facing the same fate as Canonical.

Creating Command Line Tools with Python

After spending considerable amount of time on a computer as a developer, the GUI starts to seem amazingly slow and you realize just how awesome command line tools can be. Especially if you follow the Unix philosophy of “one tool for one task”, you can quickly chain together multiple in-built tools and quickly accomplish most tasks without a single line of code.

However, what about when the tool you need doesn’t exist? Wouldn’t it be great to just create it? In this post, I’d like to show you how using Python. My reason for using python instead of the more native C/C++ is simple: easy to read code for even anybody new to programming to understand the basic structure, fast prototyping, a rich set of tools, and the general ease of use. You’re of course welcome to use other languages such as Ruby (I’ve seen a lot of great tools written in Ruby), but then you wouldn’t be reading this post, would you? Without further ado, let’s begin!

Project Setup

I faced a problem a couple of weeks ago where I needed a simple command to nuke a directory in order to rebuild some binaries and unfortunately no tool existed for this at the time. Hence I created nuke, a convenient command line tool that does exactly what it says, nuke directories.

To start, we create a project directory called nuke-tool. Inside nuke-tool, we create a directory called nuke and inside that directory, two files, nuke.py which will house all our main code logic and  __init__.py so that python is able to understand this is a package called nuke.

In the root of the project directory, we should also create a file test_nuke.py to create tests for our code. Don’t want to accidentally nuke something else now, do we? We also create a file called setup.py to aid pip in installing nuke. We’ll flesh these out one at a time.

I did mention a rich set of libraries to help us out in this process. I personally use argparse to help with argument parsing and clint from the awesome Kenneth Reitz to help with output text formatting and confirmation prompts (simple Yes/No prompts). Later on, you’ll see code snippets showing these libraries in action.

Nuke ’em

In any command line tool, the first thing we need is a way to parse command line arguments. It’s the arguments you pass to a command which are separated by spaces. For example, grep takes two arguments, a pattern and a file name grep main nuke.py.

In python, we can easily use the argparse module to help us create a simple argument parser. I wanted to provide two options to the user, an argument specifying the directory to nuke (with the default being the current directory), and a flag -y to override the confirmation prompt. The code to do this is:

import argparse
import os
.
.
.
def _argparse():
    parser = argparse.ArgumentParser("nuke")
    parser.add_argument("directory", nargs='?', default=os.getcwd(),
                        help="Directory to nuke! Default is current directory")
    parser.add_argument("-y", help="Confirm nuking", action="store_true")
                        args = parser.parse_args()
    return args

Continue reading

The Right Way To Install Postgres

Ughh, my Ubuntu distro is acting up again, and this time it is for something as stupid as it being unable to connect to the PostgreSQL server on my machine. I mean, how hard is it for the OS to be able to report a conflict between versions rather than just give up with a “directory not found” error when it fails to get a response from a Unix socket? Apparently, with the mess that is apt-get and its lethargic rate of package updates, not to mention Canonical’s refusal to support forward compatibility of packages, very hard!

As Prof. Jennifer Widom has said, databases are ubiquitous, and Postgres is an industrial strength relational DBMS claiming to be the most advanced of its type, hence my preference for it (not going to open the NoSQL can of worms today, sorry).  However, as we are well aware, advanced and user-friendly are seldom synonymous, choosing to be strange bedfellows more often. Installing Postgres via binaries is a great idea if you work at a huge corporation and update your systems once every 5 years. However, us little folk wish to receive the latest and greatest as best as we can, and hence the reliance on package managers such as aptitude a.k.a. apt-get. So what’s the catch? There has to be one since I am making the effort to write this. Well, Postgres 9.6.2 released last week and apt-get still only shows me 9.5. 2 WHOLE patch updates later and apt-get is still coughing up furballs when I look for 9.6. So before I pull out all my hair, let’s get Postgres installed the right way, with Megadeth’s Dystopia aptly playing in the background.

First things first, how to get setup so that we can easily upgrade Postgres no matter what distro of Linux we use. Here’s where I introduce you to this handy little tool called linuxbrew. Linuxbrew is a package manager inspired from OSX’s extremely popular Homebrew package manager. What’s great about Linuxbrew is that it uses Github repos as its source rather than inaccessible private channels. This ensures we always have access to the latest fixes and updates. Installing linuxbrew is fairly straightforward:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install)"
PATH="$HOME/.linuxbrew/bin:$PATH"

Don’t forget to add that PATH update to your ~/.(zsh|bash)rc file so that you have access to the brew command. Once you have that done, the next step is super easy:

brew install postgres

This should take a while, since it will get the latest tarball, untar it, make it and get postgres up and running for you. On my dated machine, it took around 3.5 minutes, enough time for me to get a nice hot cup of mocha.

To verify that the Postgres server is up and running, type psql into your command line. This should open up the SQL prompt and you’re good to go and can stop reading here and go build something awesome. If not, then guess we have a few more steps to go. We have installed Postgres with the magic of linuxbrew, but linuxbrew isn’t a service manager, so it doesn’t quite know that it needs to start the server. Let’s do that. Type the command into your terminal:

pg_ctl start -D $HOME/.linuxbrew/var/postgres

Some online forums such as StackOverflow may have answers which add the -l flag to define a logfile. I am omitting that since I want Postgres to manage its logs and not have random logfiles throughout my filesystem. You should see the message server starting and a bunch of other status messages. Just hit enter and you should see your terminal again if you don’t already. Now if you type in psql, you should see that your Postgres SQL prompt functions perfectly.

As you have seen, with literally 4 lines of code (3 if you’re one of the lucky few), you’ve got a working installation of a super advanced RDBMS and that too one that is easily upgradeable. If you wish to update Postgres ever, just type in:

brew upgrade postgres

That’s it! Hope you enjoy using Postgres along with all your sweet NoSQL datastores. Looking forward to seeing what you build.

Eviva!

Terminal Kaio-Ken

Update 1: There are some issues with certain FontAwesome icons due to a mixup with the font codepoints. After speaking with folks who maintain the powerlvl9k project, a new release in the `next` branch has all the fixes.

Update 2: Added details about customizing your prompt for a richer developer environment.

In this post, I wish to elaborate on my command-line terminal setup and share what I believe is a productive, highly-utilitarian and not to mention gorgeous terminal experience. As a computer scientist/engineer, I spent more time using the terminal than a regular GUI. And if you’re Unix oriented like me, you will know that using the terminal is so much more efficient and nimble than using the mouse.

To begin with, I have eschewed the standard Bash shell in favor of the much more powerful ZSH shell. ZSH offers much more comprehensive file globbing and auto-complete features than Bash and is really nifty in how it uses its history to recall old commands you may wish to repeat. I highly recommend going through this slide-deck to gain an understanding of what ZSH can accomplish for you.

However, ZSH on its own is not enough. To really make our experience amazing, let me introduce you to the delightful Oh-My-ZSH, a community-driven project  which provides plugin and themes for the ZSH shell, and boy are they useful. To install ZSH, use the instructions here and for OMZ, all you need to do is run:

# An easy shell script to install Oh-My-ZSH. I use the wget version since it is more ubiquitous.
sh -c "$(wget https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh -O -)"

Now time for some styling! If you are a Dragon Ball Z fan like myself and some of my friends, the title of this post will bring back a lot of memories, specifically, this meme –

vegeta

The theme I most prefer is powerlevel9k. Based on the Powerline theme, it combines functionality with a good dose of fanciness, while making a nice little reference to a beloved anime character. Installing it though is not as straightforward as the above steps have been. To install the theme,  we first have to download it and add it to the right folder:

$ git clone https://github.com/bhilburn/powerlevel9k.git ~/.oh-my-zsh/custom/themes/powerlevel9k

Then, we open up the ~/.zshrc file and specify the theme

ZSH_THEME="powerlevel9k/powerlevel9k"

This uses the basic Oh-My-ZSH framework to install the theme, though you are more than welcome to use Antigen or ZPM if you are aware of them.

At this point, if you open your terminal, you may find some glaring deficiencies. This is because we haven’t yet set up the powerline fonts needed to render all the smooth jazz of this theme. Let’s do that. First we download the latest versions of the symbol font and the fontconfig files:

wget https://github.com/powerline/powerline/raw/develop/font/PowerlineSymbols.otf
wget https://github.com/powerline/powerline/raw/develop/font/10-powerline-symbols.conf

After this we need to create the font folder to store these files and create the font cache

  mkdir ~/.fonts

We can now move the font file to the above folder

mv PowerlineSymbols.otf ~/.fonts/

and generate the font-cache

fc-cache -vf ~/.fonts/

Note you may have to provide root privileges for the above step.

Finally, install the font config file:

mv 10-powerline-symbols.conf ~/.config/fontconfig/conf.d/

Now that we have the powerline fonts set up, restart your terminal. Doesn’t it look fabulous?

If you want a more fully featured font set for a developer, I recommend checking out Adobe’s Source Code Pro and the Hack font. To install, just copy the fonts to the ~/.fonts directory and run fc-cache -fv

If the theme seems a bit off, for example the arrows aren’t quite aligned, this is mostly a problem with your terminal emulator. Go to the terminal emulator’s preferences and select the Hack or Source Code Pro theme and it should be fixed.

But wait, there’s more! Since we’ve already made this much progress, why not go one more step ahead and add some amazing font icons from FontAwesome? This can be easily accomplished using the instructions here. I am not specifying them verbatim here since the repository is still undergoing active development and the install instructions are subject to change in the near future.

Your terminal should now look something like this:

screenshot-from-2017-01-17-225604

I generally dislike redundancy and useless information. Seeing my name on the prompt is exactly that, so why I not eschew it in favor of more meaningful information such as what Python virtualenv is currently active, or what Ruby version you are using? Let’s do that.

In your ~/.zshrc, you have to add some minor commands custom to powerlvl9k. These are:

POWERLEVEL9K_LEFT_PROMPT_ELEMENTS=(os_icon dir vcs)
POWERLEVEL9K_RIGHT_PROMPT_ELEMENTS=(virtualenv pyenv rvm rbenv go_version nvm status history time)

It should be pretty straightforward to understand what each item in the parentheses corresponds to. This is my setup, so feel free to omit things you don’t need. You can checkout what prompt elements are available here. While you’re there, be sure to look up how to stylize your prompt as well to get your creative juices flowing.

Finally, this is what you get:terminal

As a closing thought, one very useful OMZ plugin I use is zsh-syntax-highlighting which updates the color of the terminal text when it is correct, thus ensuring your commands are valid before you hit enter. This plugin can easily be installed using the plugins list in the ~/.zshrc and specifying the plugin exactly as above in the parentheses.

I hope you enjoy your new super-saiyan terminal experience. If you don’t particularly like it, the Oh-My-ZSH page has dozens of themes to choose from, and if you have the time, you can go ahead and create your own theme.

Eviva!

Arch-ing In

Alright, so I had an interesting day yesterday:

  • Opened my newly won laptop.
  • Booted Arch Linux from a USB Flash drive.
  • Setup wi-fi connectivity via terminal.
  • Installed the whole damn thing from a terminal (no GUI whatsoever!).
  • Did something dumb.
  • Erased the Window EFI System partition and hence the entry from the bootloader as a consequence of doing something dumb.
  • Managed to install an intermediate bootloader – Gummiboot (nice name).
  • Couldn’t boot into Windows at all.
  • Did a network boot to recover the Windows bootloader. May have to reinstall Gummiboot now.

So yeah, overall, a day of epic hacking around with some pretty cool learnings, such as how to manually set up mount points, create partition layouts, how the EFI bootloader works and why is it better than MBR and loads of other things.

Eviva!

PintOS on Ubuntu

Note: This process is currently broken and seems to throw up unexpected errors. I am trying to look for a solution but the internet is just not helping me at the time of writing this. Until I figure out a way to fix this, I recommend trying out NachOS or xv6 for your OS cravings.

PintOS is one brilliant skeletal Operating System and, given the right time and effort, is a great way to consolidate your knowledge on the design of modern operating systems. However, its installation can be quite a pain especially since the instructions on Stanford’s official site can be a tad bit confusing at times. Here I will walk you through the installation instructions. If you would rather just install than spend time reading my post, feel free to download an install script I wrote to automate the installation process from here: pintosInstall.

If you are using the script, please remember to change the file extension as WordPress does not accept .sh files. After that just run “bash pintosInstall.sh”, without the quotes, from a terminal for a completely automatic process. Also, while I can guarantee you the script runs well on Ubuntu in a folder you have root access to, the script is simple and generic enough for you to hack and customize to your distribution if required.

  1. Install some pre-requisites: GCC, Perl, QEMU, Make, GDB. Just run:
    sudo apt-get install gcc binutils perl make qemu gdb
  2. Create and installation folder. The script makes a folder ‘co302’ (the course number for OS in my college).
  3. Download the PintOS tar from here and extract it in the installation folder. I used the totally awesome wget tool in Linux (just like Mark Zuckerberg in the Social Network 😛 ).
  4. We need a folder where the PATH variable can point to, as it will have some executables that we need to run when coding the OS. I made a folder ‘bin’ in the base installation directory i.e. co302/pintos/bin.
  5. Move all the perl scripts from the src/utils folder of PintOS to the bin/ folder. The important ones are ‘backtrace’, ‘pintos’, ‘pintos-gdb’ and ‘pintos-mkdisk’.
  6. Edit your .bashrc file to add the path for the above bin/ folder. At the end of the .bashrc file, simply add the line
    export PATH=$PATH:$HOME/co302/pintos/bin/

    Again for this, I have used the awk tool as it provides a convenient, independent way to edit files programmantically.

  7. Now we have to make a change to one of the PintOS files. Open up the ‘pintos-gdb’ file (in Emacs I hope)  and edit the GDBMACROS variable to point to the ‘gdb-macros’ file in misc directory of the src directory. At this point, you have officially installed PintOS, so give yourself a pat on the back.
  8. Time to compile the utilities. Head over to the pintos/src/utils directory and run
    $ make

    . If you get a “Undefined reference to ‘floor’ ” error, simply open the Makefile and substitute LDFLAGS for LDLIBS and run make again.

  9. Copy the ‘squish-pty’ file to the PATH pointed bin directory.
  10. Head over to the pintos/src/threads/ directory and edit the ‘Make.vars’ file. Change the SIMULATOR variable from bochs to qemu, which should mostly be the last line of the file.
  11. Run make on the threads folder.
  12. Now we need to edit the ‘pintos’ util file in the bin directory with 3 edits. Many other sites will give you  the line numbers, but I will not use that as it is too variable and you get a chance to experiment with the text processing features of your favorite editor:
    1. Change $sim = bochs to $sim = qemu to enforce qemu as the simulator.
    2. Comment out the line push (@cmd, ‘-no-kqemu’); by prepending it with #.
    3. Put in the absolute path wherever required as Perl doesn’t seem to be able to interpret the ~ shorthand. Do this especially for the kernel.bin location path.
  13. Finally, edit the ‘Pintos.pm’ file in the bin directory and put in the absolute path for the line having the location of the loader.bin file.
  14. Congrats, you now have PintOS set up on your machine. Try running pintos run alarm-multiple as a test.

There you have it. A pretty easy and straightforward way to install a great experimental skeletal OS from Stanford University. I bet it took more work for me to write this up that it will take you to install PintOS. The good part about this little adventure of mine is that I got a chance to dabble in sed and awk, 2 Unix tools that no hacker can afford to not know the basics of. Add to that some wget magic and Emacs power, and you can potentially become a hacking superstar. Infact, this hack has proven really useful as the Computer Engineering department of my college has used it to install PintOS on all the machines so that the students can do meaningful OS practicals. My small way of giving back, you could say. 🙂

As a final note, at the time of writing, this post is as comprehensive a set of instructions you can get. This may not hold true forever as tomorrow someone might make some script-breaking changes. So if you find some change that I need to include, please feel free to comment and let me know about it.

Eviva!

TinyOS Installation Made Easy

Howdy people! Long time since a post but couldn’t help it, busy being a busybee.

Well, I have started a project on Wireless Sensor Networks and the best way to go about it has been using an amazing operating system called TinyOS. This OS is so tiny, it uses only 400 kb of memory to run! Holy Cow!!

But getting TinyOS on 64-bit machines hasn’t been well documented and I assume that lack of documentation brought you here. Well no fear, I will be showing a stepwise guide to getting TinyOS up-and-running on any 64-bit machine you have.

Note: I am using Ubuntu 11.10 64-bit as my host operating system, but the method should work on all distros of Linux.  If you’re using Windows, may God have mercy on your soul. Also, this post involves installing TinyOS 2.1.1. I cannot guarantee that all the steps will work as is for other older versions. You’ll have to do some research for that.

Step 1:

First up, we need Java, the JRE and the JDK as a lot of the TinyOS and Mote communication frameworks are built in Java. You can use the OpenJDK but I have found using Sun’s original JDK yields better results!

Open up a terminal and simply type in these commands to get the JRE and JDK installed in your machine.

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:ferramroberto/java
sudo apt-get update
sudo apt-get install sun-java6-jdk sun-java6-plugin

You can choose the appropriate JVM by running the following command in the terminal:

sudo update-alternatives --config java

Step 2:

Now that we have our environment setup, we are ready to install the TinyOS development environment.

  1. Install the Synaptic Package manager by running “sudo apt-get install Synaptic” in a terminal.
  2. In Synaptic, go to Settings ->Repositories.
  3. Click on 3rd party software and add the following (simple copy-paste):
    deb http://tinyos.stanford.edu/tinyos/dists/ubuntu * main
  4.  Click OK and reload your repositories.
  5. After reloading, simply search for TinyOS and select it. Click install and allow Synaptic to handle dependency installation for you.
  6. Once the packages have been properly installed, go back to a terminal and type “sudo gedit ~/.bashrc” and add this line at the absolute end:
                         source /opt/tinyos-2.1.1/tinyos.sh
  7.  Finally, close the current terminal and open up a new one and type in $MAKERULES. If all went well, you should get the path.

Congrats, we have TinyOS ready. But we’re not done yet!

Step 3:

Now, when writing TinyOS programs, you’ll very much want to simulate them to see what are the results. A very handy tool for TinyOS simulation is TOSSIM. But when you run the make micaz sim to generate your executable you might get a Python related error. Try these steps:

  1. In a terminal, run sudo apt-get install python-dev.
  2. Even after installing python, some configuration is required.
  3. Run python –version  and note the version of Python installed. In my case, it was 2.7.
  4. Go to /opt/tinyos-2.1.1/support/make/
  5. Open the sim.extra file in gedit with super user privileges.
  6. In the entry labelled PYTHON_VERSION, check to see if the entry matches with the Python version installed. For me, it was 2.6, so I changed it to 2.7.
Python Configuration

The Python version entry to be edited

Step 4:

As the final step, this involves some minor configuration to your tinyos.sh file found in /opt/tinyos-2.1.1. This is done so that when you run the make micaz sim command to generate TOSSIM related files, the appropriate Java classes are generated to help in the simulation. Otherwise you’ll get a long list of errors that will make no sense at all.

Editing the tinyos.sh file makes the change permanent which is exactly what you need. Simply go to the directory in which the tinyos.sh file is located (given in the start of  this step).

Open the tinyos.sh file in gedit with super user privileges. There is an entry called CLASSPATH. Simply edit the original to $TOSROOT/support/sdk/java/tinyos.jar, similar to how it is in the image.

CLASSPATH Edit

Change to CLASSPATH entry in tinyos.sh file

And voila!! You’re done and ready to start writing some incredible TinyOS code.

I hope this blog entry has been useful to you. For any doubts or problems, the TinyOS mailing list is a great place filled with really helpful people and I highly recommend posting both newbie and advanced doubts there.

Till next time, Eviva!

———————————————————————————————————————————
P.S. These are some of the links I used to get TinyOS up and working. Thought they might be useful for the future:

http://superuser.com/questions/353983/how-do-i-install-the-sun-java-sdk-in-ubuntu-11-10-oneric
http://mythicalcomputer.blogspot.in/2008/08/installation-of-tinyos-in-ubuntu.html
http://mail.millennium.berkeley.edu/pipermail/tinyos-help/2011-October/052537.html
https://www.millennium.berkeley.edu/pipermail/tinyos-help/2010-November/048947.html