Turn Vim into a powerful Python IDE: Essential Vim plugins for Python programmers

Essential plugins:

1. Pathogen: A must have if you want to keep your vim plugins in a clean state. Makes it super easy to install plugins and runtime files in their own private directories.

2. Powerline: Powerline is a statusline plugin for vim, and provides statuslines and prompts for several other applications, including zsh, bash, tmux, IPython, Awesome and Qtile.

3. jedi-vim: A python autocompletion library for VIM.

(You’ll need Vim python support enabled for this plugin to work. You can check if your Vim supports Python using this (you should see a line with +python in it as a result):

vim --version | grep +python

if not found, you need to install Vim, the following command will remove vim-tiny, and install Vim 7.3 with +python and –with-features=huge:

sudo apt-get remove vim-tiny
sudo apt-get install vim


4. NERDTree – NerdTree is a filebrowser that pops up in a split when you need it and features a tree like file browse.

5. NERDCommenter: Very useful tool for multiline commenting/uncommenting, just like IDLE.

For more information like this check the following :

1. Vim as a Python IDE: Talks about some of the plugins here and others.

2. Martin Brochhaus’s excellent talk on converting Vim into a Python IDE

You can donwload the talk slides and other stuff from his github page. You can also check his excellent .vimrc file here (which I use.)

How to install Vim 7.4 with Perl, Python and Ruby support on Ubuntu 12.04

1. Remove preexisting vim installations:

sudo apt-get remove vim-tiny vim vim-nox

2. Make sure you have mercurial install, otherwise install it:

sudo apt-get install mercurial

3. Download Vim 7.4 from vim.org repo:

hg clone https://vim.googlecode.com/hg/ vim74

4. change directory to vim74, and install, for example I have used installed with following options enabled:

./configure --enable-perlinterp --enable-pythoninterp --enable-rubyinterp --enable-cscope  --with-features=huge --prefix=/usr
make install

Vim: How to start using it

The following quote from Why, oh WHY, do those #?@! nutheads use vi? basically sums up the “philosophy” of Vim use:

The first time you stumble into vi or vim, you are shocked and disgusted that you have to use ‘i’ to start typing text. Can’t remember which of hjkl moves in which direction. And you need to press ‘a’ if you want to insert after the last character of the line. Since you’re already probably used to other editors, and if the arrow keys work in insert mode (they do if the system is configured properly, which is over 90% of the cases), you will stay in insert mode and not go back to normal mode except if absolutely necessary. You’ll probably spend 20 minutes with it, and never go back. And also complaining: “How on earth am I going to remember whether I’m in insert or normal mode?”

Turns out, this is just a completely wrong way to use vi or vim. Using vi/vim properly, you don’t use it modally. You are always in normal mode, and only enter insert mode for short bursts of typing text, after which you press to go to normal mode. Thus, the remembering-the-mode problem just doesn’t exist: you don’t answer the phone in insert mode to get back to vi and not remember where you were. If you are typing text and the phone rings, you exit insert mode and then answer the phone. Or you press when you come back. But you never think about insert mode as a mode where you stay.

The best way to learn Vim IMHO is to learn incrementaly. Here’s a great article on that – Everyone Who Tried to Convince Me to use Vim was Wrong.

Tips to getting started with Vim :

1. Create a list of substitue Vim commands/keyboard shortcuts for the commands/keyboard shortcuts that you use for your current editor of choice.

There are some great cheatsheets and reference tables available online to help you out with creating your list.

Vim Commands Cheat
Essential Vim
Vim Quick Reference Card

2. Practice with vimtutor

3. Make mode change easier

Another important tips is to use another shortcut for Esc key. I use Ctrl-C. Here’s a link for other shortcuts and tricks to use alter methods.

4. Install essential plugins: These very useful plugins will make you usage experience a whole lot easier if you intend to code in Vim.

5. Watch Derek Wyatt’s Vim screencasts: Derek Wyatt has some excellent Vim tutorial videos starting from novice to expert level which will make you a Vim expert in no time.

Some useful talks and tutorials:

1. Vim Introduction and Tutorial : Nice tutorial for beginners with visual guides.

2. Ben Orenstein – Write code faster: expert-level vim (Railsberry 2012)

you can download the lecture materials from his github page. and his .vimrc file is here.

3. Vim Cookbook by Steve Oualline

4. Vimcasts.org archive : Great resource for digging deep into vim. Advanced stuff.

How to fix Unity/Compiz freeze on Ubuntu 12.04 LTS

Sometimes while running Unity + compiz in precise (12.04), display manager hangs so that the mouse moves, but the keyboard allows to get to tty1. The recovery process is:

1. Ctl + Alt + F1 to get to tty1, then
2. Use “top” to check the compiz cpu use and get its pid, then sudo kill -9 , then
3. Perform “sudo service lightdm restart”, then
4. login to X-window environment

(Note that, if one doesn’t kill compiz, a new one will be loaded by lightdm, leaving the orphan compiz to use an entire cpu. And if you hit the problem and don’t kill the new compiz, you now have two cpus locked up.)

IPython Tip: Reloading modified code

IPython comes with some automatic reloading magic:

%load_ext autoreload
%autoreload 2

It will reload all changed modules every time before executing a new line.

Place commands in startup script:


Note .ipy extension.
To check what is being reloaded


How to debug in IPython using ipdb

There are two ways you can use ipdb for debugging in ipython:

1. run -d filename.py
Sample debugging process:
Situation – You believe a bug exists in a module but are not sure where.
For instance we are trying to debug wiener_filtering.py. Indeed the code runs, but the filtering does not work well.
Run the script in IPython with the debugger using %run -d wiener_filtering.p :

    In [1]: %run -d wiener_filtering.py
    *** Blank or comment
    *** Blank or comment
    *** Blank or comment
    Breakpoint 1 at /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py:4
    NOTE: Enter 'c' at the ipdb>  prompt to start your script.
    > <string>(1)<module>()

Set a break point at line 34 using b 34:

    ipdb> n
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(4)<module>()
    1---> 4 import numpy as np
          5 import scipy as sp

    ipdb> b 34
    Breakpoint 2 at /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py:34

Continue execution to next breakpoint with c(ont(inue)):

    ipdb> c
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(34)iterated_wiener()
         33     """
    2--> 34     noisy_img = noisy_img
         35     denoised_img = local_mean(noisy_img, size=size)

Step into code with n(ext) and s(tep): next jumps to the next statement in the current execution context, while step will go across execution contexts, i.e. enable exploring inside function calls:

    ipdb> s
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(35)iterated_wiener()
    2    34     noisy_img = noisy_img
    ---> 35     denoised_img = local_mean(noisy_img, size=size)
         36     l_var = local_var(noisy_img, size=size)

    ipdb> n
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(36)iterated_wiener()
         35     denoised_img = local_mean(noisy_img, size=size)
    ---> 36     l_var = local_var(noisy_img, size=size)
         37     for i in range(3):

Step a few lines and explore the local variables:

    ipdb> n
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(37)iterated_wiener()
         36     l_var = local_var(noisy_img, size=size)
    ---> 37     for i in range(3):
         38         res = noisy_img - denoised_img
    ipdb> print l_var
    [[5868 5379 5316 ..., 5071 4799 5149]
     [5013  363  437 ...,  346  262 4355]
     [5379  410  344 ...,  392  604 3377]
     [ 435  362  308 ...,  275  198 1632]
     [ 548  392  290 ...,  248  263 1653]
     [ 466  789  736 ..., 1835 1725 1940]]
    ipdb> print l_var.min()

Ref: For more detailed information on ipython debugging check out this excellent tutorial by Gaƫl Varoquaux.

2. Invoke ipdb.runcall(function_name, params…) on a specific function.

To do this you have to install the ipdb python module.

$sudo ~/workspace/python_2.7.5_venv/bin/easy_install ipdb

Then in ipython:

In [2]: import ipdb
In [3]: import utils.file_utils
In [4]: fu = utils.file_utils.FileUtils()

In [5]: ipdb.runcall(fu.memory_based_sampling, '/home/naved/datasets/test/test.csv', ',', .90)
> /home/naved/workspace/data_characterization/datachar/utils/file_utils.py(150)memory_based_sampling()
--> 150             sys_info = sysinfo.SystemInfo()
    151             free_memory_size = sys_info.get_system_free_memory_size()

ipdb> n

Then you can step though the program same as the previously shown example. Go to ipdb github page for more examples.

Installing PyTables 3.0.0 in Virtualenv on Ubuntu 12.04

1. Install hdf5:

sudo apt-get install libhdf5-serial-dev

2. Activate Virtualenv:

source ~/Coding/python_2.7.5_venv/bin/activate

3. Install Cython, Numexpr:

Numexpr: download from http://code.google.com/p/numexpr
$ python setup.py build
$ sudo python setup.py install


~/Coding/python_2.7.5_venv/bin/pip install -U numexpr
wget http://cython.org/release/Cython-0.19.1.tar.gz
tar zxvf Cython-0.19.1.tar.gz
cd Cython-0.19.1
sudo python setup.py install

4. Install PyTables

~/Coding/python_2.7.5_venv/bin/pip install git+https://github.com/PyTables/PyTables.git@v.3.0.0#egg=tables

5. Test pytables
To test the installation, run python or ipython and:

In [0]: import tables
In [1]: tables.test()

The test will fail, test with pytables functions from ipython:

In [2]: import pandas as pd
In [3]: import numpy as np
In [4]: import tables
In [5]: df = pd.DataFrame(np.random.randn(8,3))
In [6]: store = pd.HDFStore('test.h5')
In [7]: store.put('df',df)