Some frequently used python snippets

Jotting down here some python snippets that I seem to frequently use. So here goes –

1. creating dictionary of dictionaries

In [12]: x = {1:'a', 2:'b'}
In [13]: x
Out[13]: {1: 'a', 2: 'b'}
In [14]: x['c'] = {}
In [15]: x
Out[15]: {1: 'a', 2: 'b', 'c': {}}
In [16]: x['c']['name'] = 'abc'
In [17]: x
Out[17]: {1: 'a', 2: 'b', 'c': {'name': 'abc'}}

2. return multiple values from a function

def f(in_str):
    out_str = in_str.upper()
    return True, out_str # Creates tuple automatically

succeeded, b = f("a") # Automatic tuple unpacking

3. Check if a given key already exists in a dictionary

in is the intended way to test for the existence of a key in a dict.

d = dict()

for i in xrange(100):
    key = i % 10
    if key in d:
        d[key] += 1
    else:
        d[key] = 1

4. How do I check if a file exists

if you don’t want to touch the file if it doesn’t exist

import os.path
os.path.isfile(fname)

5. How to modify list entries during for loop

It’s considered poor form. Use a list comprehension instead, with slice assignment if you need to retain existing references to the list.

a = [1, 3, 5]
b = a
a[:] = [x + 2 for x in a]
print b

remember that even if you modify the list, the loop counter doesn’t change its value.

6. read json file

There’s no need to first read the file, then use loads. You can just use load directly.

output_json = json.load(open('/tmp/output.json'))

7. dump a dictionary to json file

d = {"name":"interpolator",
     "children":[{'name':key,"size":value} for key,value in sample.items()]}
json_string = json.dumps(d)

8. add new key-value pair to dictionary

dict.update({'item3': 3})

9 Add item or list of items to a list

list = []
x = 1
list.append(x)
another_list=[2,3,4]
list += another list

10. Check if a variable exists

To check the existence of a local variable:

if 'myVar' in locals():
  # myVar exists.

To check the existence of a global variable:

if 'myVar' in globals():
  # myVar exists.

To check if an object has an attribute:

if hasattr(obj, 'attr_name'):
  # obj.attr_name exists.

Another method,

try:
    myVar
except NameError:
    myVar = None
# Now you're free to use myVar without Python complaining.

11. Change the key value in dictionary

dict[new_key] = dict.pop(old_key)

Note that, this will raise KeyError if dict[old_key] is undefined.

Vim tip: How to fix Python exception IndentationError

The problem is usually with mixup in tabs and spaces –

Solution 1

1. Apply following command in Vim to highlight tabs, spaces and other whitespace differently.

:set listchars=tab:>-,trail:-,eol:$ list

2. Apply the following to set correct width

:set shiftwidth=4 tabstop=4 expandtab 

3. running

:retab

to fix the problem,Vim automatically fixes all indentation space and tab mix ups.

Solution 2

1. set ‘list’, so that you can see the whitespace and change.

Have the following mapping in .vimrc for this:

nnoremap    <F2> :<C-U>setlocal lcs=tab:>-,trail:-,eol:$ list! list? <CR>

2. Ensure ‘expandtab’ is reset, check using following command –

:verbose set ts? et? 

3. To expand all leading spaces (wider than ‘tabstop’), use retab.

retab takes a range, so specify % to mean “the whole file”.

:set tabstop=4      " To match the python file
:set noexpandtab    " Use tabs, not spaces
:%retab!            " Retabulate the whole file

Installing SimpleCV with Python 2.7.5 in Virtualenv in Ubuntu 13.10

1. sudo apt-get build-dep python-opencv
2. sudo apt-get install python-opencv
3. sudo apt-get install python-setuptools python-pip gfortran g++ liblapack-dev libsdl1.2-dev libsmpeg-dev mercurial
4. Get the paths for cv2.so and cv.py by applying dpkg -L python-opencv
/usr/lib/pyshared/python2.7/cv2.so
/usr/share/pyshared/cv.py
5. virtualenv simplecv_venv
cd simplecv_venv
ln -s /usr/lib/pyshared/python2.7/cv2.so lib/python2.7/site-packages/cv2.so
ln -s /usr/share/pyshared/cv.py lib/python2.7/site-packages/cv.py
sudo ./bin/pip install https://github.com/numpy/numpy/zipball/master
sudo ./bin/pip install cython
sudo ./bin/pip install scipy
sudo ./bin/pip install PIL
sudo ./bin/pip install ipython
mkdir src
wget -O src/pygame.tar.gz https://bitbucket.org/pygame/pygame/get/6625feb3fc7f.tar.gz
cd src
tar zxvf pygame.tar.gz
cd ..
sudo ./bin/python src/pygame-pygame-6625feb3fc7f/setup.py -setuptools install
sudo ./bin/pip install https://github.com/sightmachine/SimpleCV/zipball/develop
6. Test installation with the following webcam capture code, you’ll need to install the following module:
sudo bin/pip install svgwrite
7. Activate simplecv virtual environment:
$ source bin/activate
8. Open ipython and run the following code:
(simplecv_venv)$ipython
In [1]: run simplecv_helloworld.py
Code:

from SimpleCV import Camera, Display, Image
# Initialize the camera
cam = Camera()
# Initialize the display
display = Display()
# Snap a picture using the camera
img = cam.getImage()
# Show the picture on the screen
img.save(display)

Exit ipython to stop the code running.

Ubuntu tip: Getting information about your installed packages

1. Show the List of Installed Packages on Ubuntu :
The command we need to use is dpkg –get-selections, which will give us a list of all the currently installed packages.

    $ dpkg --get-selections
    adduser                                         install
    alsa-base                                       install
    alsa-utils                                      install
    apache2                                         install
    apache2-mpm-prefork                             install
    apache2-utils                                   install
    apache2.2-common                                install
    apt                                             install
    apt-utils                                       install

You can filter through grep to get results for the exact package you need. For instance, I wanted to see which php packages I had already installed through apt-get:

    dpkg --get-selections | grep php

    libapache2-mod-php5                             install
    php-db                                          install
    php-pear                                        install
    php-sqlite3                                     install
    php5                                            install
    php5-cli                                        install
    php5-common                                     install
    php5-gd                                         install
    php5-memcache                                   install
    php5-mysql                                      install
    php5-sqlite                                     install
    php5-sqlite3                                    install
    php5-xsl                                        install

2. How do I get a list of installed files from a package

To see all the files the package put onto your system, do this:

dpkg-query -L <package_name>

Example:-

dpkg -L php5-gd
/.
/usr
/usr/lib
/usr/lib/php5
/usr/lib/php5/20060613
/usr/lib/php5/20060613/gd.so
/usr/share
/usr/share/doc
/etc
/etc/php5
/etc/php5/conf.d
/etc/php5/conf.d/gd.ini
/usr/share/doc/php5-gd

3. Get a list of installed Python modules

Type

help('modules')

in a Python shell/prompt.

4. Find all packages installed with easy_install/pip

pip freeze

will output a list of installed packages and their versions. It also allows you to write those packages to a file that can later be used to set up a new environment.

Turn Vim into a powerful Python IDE: Essential Vim plugins for Python programmers

Essential plugins:

1. Pathogen: A must have if you want to keep your vim plugins in a clean state. Makes it super easy to install plugins and runtime files in their own private directories.

2. Powerline: Powerline is a statusline plugin for vim, and provides statuslines and prompts for several other applications, including zsh, bash, tmux, IPython, Awesome and Qtile.

3. jedi-vim: A python autocompletion library for VIM.

(You’ll need Vim python support enabled for this plugin to work. You can check if your Vim supports Python using this (you should see a line with +python in it as a result):

vim --version | grep +python

if not found, you need to install Vim, the following command will remove vim-tiny, and install Vim 7.3 with +python and –with-features=huge:

sudo apt-get remove vim-tiny
sudo apt-get install vim

)

4. NERDTree – NerdTree is a filebrowser that pops up in a split when you need it and features a tree like file browse.

5. NERDCommenter: Very useful tool for multiline commenting/uncommenting, just like IDLE.

For more information like this check the following :

1. Vim as a Python IDE: Talks about some of the plugins here and others.

2. Martin Brochhaus’s excellent talk on converting Vim into a Python IDE

You can donwload the talk slides and other stuff from his github page. You can also check his excellent .vimrc file here (which I use.)

Vim: How to start using it

The following quote from Why, oh WHY, do those #?@! nutheads use vi? basically sums up the “philosophy” of Vim use:

The first time you stumble into vi or vim, you are shocked and disgusted that you have to use ‘i’ to start typing text. Can’t remember which of hjkl moves in which direction. And you need to press ‘a’ if you want to insert after the last character of the line. Since you’re already probably used to other editors, and if the arrow keys work in insert mode (they do if the system is configured properly, which is over 90% of the cases), you will stay in insert mode and not go back to normal mode except if absolutely necessary. You’ll probably spend 20 minutes with it, and never go back. And also complaining: “How on earth am I going to remember whether I’m in insert or normal mode?”

Turns out, this is just a completely wrong way to use vi or vim. Using vi/vim properly, you don’t use it modally. You are always in normal mode, and only enter insert mode for short bursts of typing text, after which you press to go to normal mode. Thus, the remembering-the-mode problem just doesn’t exist: you don’t answer the phone in insert mode to get back to vi and not remember where you were. If you are typing text and the phone rings, you exit insert mode and then answer the phone. Or you press when you come back. But you never think about insert mode as a mode where you stay.

The best way to learn Vim IMHO is to learn incrementaly. Here’s a great article on that – Everyone Who Tried to Convince Me to use Vim was Wrong.

Tips to getting started with Vim :

1. Create a list of substitue Vim commands/keyboard shortcuts for the commands/keyboard shortcuts that you use for your current editor of choice.

There are some great cheatsheets and reference tables available online to help you out with creating your list.

Vim Commands Cheat
Essential Vim
Vim Quick Reference Card

2. Practice with vimtutor

3. Make mode change easier

Another important tips is to use another shortcut for Esc key. I use Ctrl-C. Here’s a link for other shortcuts and tricks to use alter methods.

4. Install essential plugins: These very useful plugins will make you usage experience a whole lot easier if you intend to code in Vim.

5. Watch Derek Wyatt’s Vim screencasts: Derek Wyatt has some excellent Vim tutorial videos starting from novice to expert level which will make you a Vim expert in no time.

Some useful talks and tutorials:

1. Vim Introduction and Tutorial : Nice tutorial for beginners with visual guides.

2. Ben Orenstein – Write code faster: expert-level vim (Railsberry 2012)

you can download the lecture materials from his github page. and his .vimrc file is here.

3. Vim Cookbook by Steve Oualline

4. Vimcasts.org archive : Great resource for digging deep into vim. Advanced stuff.

How to debug in IPython using ipdb

There are two ways you can use ipdb for debugging in ipython:

1. run -d filename.py
Sample debugging process:
Situation – You believe a bug exists in a module but are not sure where.
For instance we are trying to debug wiener_filtering.py. Indeed the code runs, but the filtering does not work well.
Run the script in IPython with the debugger using %run -d wiener_filtering.p :

    In [1]: %run -d wiener_filtering.py
    *** Blank or comment
    *** Blank or comment
    *** Blank or comment
    Breakpoint 1 at /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py:4
    NOTE: Enter 'c' at the ipdb>  prompt to start your script.
    > <string>(1)<module>()

Set a break point at line 34 using b 34:

    ipdb> n
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(4)<module>()
          3
    1---> 4 import numpy as np
          5 import scipy as sp

    ipdb> b 34
    Breakpoint 2 at /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py:34

Continue execution to next breakpoint with c(ont(inue)):

    ipdb> c
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(34)iterated_wiener()
         33     """
    2--> 34     noisy_img = noisy_img
         35     denoised_img = local_mean(noisy_img, size=size)

Step into code with n(ext) and s(tep): next jumps to the next statement in the current execution context, while step will go across execution contexts, i.e. enable exploring inside function calls:

    ipdb> s
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(35)iterated_wiener()
    2    34     noisy_img = noisy_img
    ---> 35     denoised_img = local_mean(noisy_img, size=size)
         36     l_var = local_var(noisy_img, size=size)

    ipdb> n
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(36)iterated_wiener()
         35     denoised_img = local_mean(noisy_img, size=size)
    ---> 36     l_var = local_var(noisy_img, size=size)
         37     for i in range(3):

Step a few lines and explore the local variables:

    ipdb> n
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(37)iterated_wiener()
         36     l_var = local_var(noisy_img, size=size)
    ---> 37     for i in range(3):
         38         res = noisy_img - denoised_img
    ipdb> print l_var
    [[5868 5379 5316 ..., 5071 4799 5149]
     [5013  363  437 ...,  346  262 4355]
     [5379  410  344 ...,  392  604 3377]
     ...,
     [ 435  362  308 ...,  275  198 1632]
     [ 548  392  290 ...,  248  263 1653]
     [ 466  789  736 ..., 1835 1725 1940]]
    ipdb> print l_var.min()
    0

Ref: For more detailed information on ipython debugging check out this excellent tutorial by Gaël Varoquaux.

2. Invoke ipdb.runcall(function_name, params…) on a specific function.

To do this you have to install the ipdb python module.

$sudo ~/workspace/python_2.7.5_venv/bin/easy_install ipdb

Then in ipython:

In [2]: import ipdb
In [3]: import utils.file_utils
In [4]: fu = utils.file_utils.FileUtils()

In [5]: ipdb.runcall(fu.memory_based_sampling, '/home/naved/datasets/test/test.csv', ',', .90)
> /home/naved/workspace/data_characterization/datachar/utils/file_utils.py(150)memory_based_sampling()
    149 
--> 150             sys_info = sysinfo.SystemInfo()
    151             free_memory_size = sys_info.get_system_free_memory_size()

ipdb> n

Then you can step though the program same as the previously shown example. Go to ipdb github page for more examples.

Configuration for Connecting to MSSQL Server 2008 on Virtualbox GuestOS from Ubuntu 12.04 HostOS using PyODBC 3.0.8

1. VirtualBox Configuration

– Adding a host-only network adapter
If you don’t have a host-only adapter setup then goto topmost menu:
File->Preferences->Network->Host-only network->Add ‘vboxnet0’ by clicking on the ‘+’ sign at
the right-hand corner. Go to edit and take note of the IP address assigned (lets assume its 192.168.56.1, so that is the virtual network adapter that has been created at host OS)
Now go to:
Settings->Network->Adapter 2 Tab (considering you’ve NAT set on Adapter 1 for internet) ->Attach to host-only adapter> Name select ‘vboxnet0’
[Note, if you already have a network adapter with NAT selected for accessing internet from
guest os, leave it and just add another one.]
-Add GuestOS to the network
Go to Stat->Settings->Network Connections->Select ‘Local Area Connection 2’ (‘Local Area Connection 1’ should be allocated to NAT Adapter 1 , which is used for Internet)
Right click to get properties ->Internet Protocol (TCP/IP)->Use the following IP address->Assign an IP in the same network as host-only adapter (In this example, that can be 192.168.56.2/255.255.255.0)

2. MSSQL Server 2008 R2 Configuration (GuestOS)

– Enable TCP/IP and Piping

Open SQL Server Configuration Manager (SSCM) go to ‘SQL Server Network Configuration’->Double Click on ‘TCP/IP’->Select Protocol Tab -> Select Enabled as ‘yes’
Goto IP Address Tab -> Change IP1 (or anyother) IP address to ‘192.168.56.1’ (In same network as the host OS ip)->Select Active to yes, Enabled to yes
Now go back to main menu, change ‘Named Pipes’ to ‘Enabled’

– Set SQL Server Authentication

If you’re using ‘Windows Authentication’ to login to your Database. You have to change it to ‘SQL Server Authentication’ and set username and password for it , so that it can be accessed using PyODBC.

To change security authentication mode

In SQL Server Management Studio Object Explorer, right-click the server, and then click Properties.
On the Security page, under Server authentication, select the new server authentication mode, and then click OK.
In the SQL Server Management Studio dialog box, click OK to acknowledge the requirement to restart SQL Server.
In Object Explorer, right-click your server, and then click Restart. If SQL Server Agent is running, it must also be restarted.

To enable the sa login, (‘sa’ will be your username)

In Object Explorer, expand Security, expand Logins, right-click sa, and then click Properties.
On the General page, you might have to create and confirm a password for the login.
On the Status page, in the Login section, click Enabled, and then click OK.

3. Disble Windows Firewall serivce
Simply turning off the firewall won’t do, you’ve to stop the service, do:
Run->Services.msc->Windows Firewall->Stop

4. Test connection
Using telnet:
From hostOS (192.168.56.1) do: telnet 192.168.56.2 1433
You can also use a DB client such as, dbeaver.

5. PyODBC configurations

– freetds.conf

# local sql server
[localsqlserver]
    host = 192.168.56.2
    port = 1433  
    tds version = 7.0

– odbcinst.ini

[sqlserverdatasource1]
Driver = /usr/lib/i386-linux-gnu/odbc/libtdsodbc.so
Description = ODBC connection via FreeTDS
Server = 192.168.56.2
Port = 1433 
Database = Test
TDS_Version = 8.0

– pyodbc-test.py code

import pyodbc

dsn = 'sqlserverdatasource1'
user = 'sa'
password = '******'
database = 'TestDB' 

con_string = 'DSN=%s;UID=%s;PWD=%s;DATABASE=%s;' % (dsn, user, password, database)
cnxn = pyodbc.connect(con_string)
cursor = cnxn.cursor()
print cursor

MSSQL Server connection using pyodbc 3.0.7 on Ubuntu 12.04

1. Install pyodbc from your Virtualenv pip

python_2.7.5_venv/bin/pip install pyodbc

2. Install mssql odbc drivers and other dependencies

sudo apt-get install unixodbc unixodbc-dev freetds-dev freetds-bin tdsodbc

3. open /etc/freetds/freetds.conf and add the following at the end:

# mssql server
[sqlserver]
    host = <host-adddress>
    port = <port-number>
    tds version = 7.0

4. Test server connection using the following command

tsql -S sqlserver -U <user-name> -P <password>

5. Check configuration files using –

odbcinst -j

6. Configure /etc/odbcinst.ini and insert the following:

[FreeTDS]
Description = TDS driver (Sybase/MS SQL)
# Some installations may differ in the paths
Driver = /usr/lib/i386-linux-gnu/odbc/libtdsodbc.so
Setup = /usr/lib/i386-linux-gnu/odbc/libtdsS.so
CPTimeout =
CPReuse =
FileUsage = 1

7. Open /etc/odbc.ini and insert the following:

[sqlserverdatasource]
Driver = FreeTDS
Description = ODBC connection via FreeTDS
Trace = No
Servername = sqlserver
Database = <database-name>
TDS_Version = 8.0

8. Check connection using the following command:

isql -v sqlserverdtasource <username> <password>

9. Use the following code to check pyodbc :

import pyodbc
 
dsn = 'sqlserverdatasource'
user = '<username>'
password = '<password>'
database = '<database-name>'
 
con_string = 'DSN=%s;UID=%s;PWD=%s;DATABASE=%s;' % (dsn, user, password, database)
cnxn = pyodbc.connect(con_string)

Some Python apps installation on Ubuntu

1. Installing pyzipcode-0.4 in Ubuntu 12.04

sudo apt-get install libsqlite3-dev
wget https://pypi.python.org/packages/source/p/pyzipcode/pyzipcode-0.4.tar.gz
tar zxvf pyzipcode-04.tar.gz
cd pyzipcode-0.4
sudo python setup.py install

2. IDLE Linux Installation

If you are using Debian / Ubuntu Linxu, type the following apt-get command to install idle IDE:

$ sudo apt-get install idle