IPython Tip: Reloading modified code

IPython comes with some automatic reloading magic:

%load_ext autoreload
%autoreload 2

It will reload all changed modules every time before executing a new line.

Place commands in startup script:

~/.config/ipython/profile_default/startup/ipython3_startup.ipy

Note .ipy extension.
To check what is being reloaded

%aimport

How to debug in IPython using ipdb

There are two ways you can use ipdb for debugging in ipython:

1. run -d filename.py
Sample debugging process:
Situation – You believe a bug exists in a module but are not sure where.
For instance we are trying to debug wiener_filtering.py. Indeed the code runs, but the filtering does not work well.
Run the script in IPython with the debugger using %run -d wiener_filtering.p :

    In [1]: %run -d wiener_filtering.py
    *** Blank or comment
    *** Blank or comment
    *** Blank or comment
    Breakpoint 1 at /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py:4
    NOTE: Enter 'c' at the ipdb>  prompt to start your script.
    > <string>(1)<module>()

Set a break point at line 34 using b 34:

    ipdb> n
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(4)<module>()
          3
    1---> 4 import numpy as np
          5 import scipy as sp

    ipdb> b 34
    Breakpoint 2 at /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py:34

Continue execution to next breakpoint with c(ont(inue)):

    ipdb> c
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(34)iterated_wiener()
         33     """
    2--> 34     noisy_img = noisy_img
         35     denoised_img = local_mean(noisy_img, size=size)

Step into code with n(ext) and s(tep): next jumps to the next statement in the current execution context, while step will go across execution contexts, i.e. enable exploring inside function calls:

    ipdb> s
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(35)iterated_wiener()
    2    34     noisy_img = noisy_img
    ---> 35     denoised_img = local_mean(noisy_img, size=size)
         36     l_var = local_var(noisy_img, size=size)

    ipdb> n
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(36)iterated_wiener()
         35     denoised_img = local_mean(noisy_img, size=size)
    ---> 36     l_var = local_var(noisy_img, size=size)
         37     for i in range(3):

Step a few lines and explore the local variables:

    ipdb> n
    > /home/varoquau/dev/scipy-lecture-notes/advanced/debugging_optimizing/wiener_filtering.py(37)iterated_wiener()
         36     l_var = local_var(noisy_img, size=size)
    ---> 37     for i in range(3):
         38         res = noisy_img - denoised_img
    ipdb> print l_var
    [[5868 5379 5316 ..., 5071 4799 5149]
     [5013  363  437 ...,  346  262 4355]
     [5379  410  344 ...,  392  604 3377]
     ...,
     [ 435  362  308 ...,  275  198 1632]
     [ 548  392  290 ...,  248  263 1653]
     [ 466  789  736 ..., 1835 1725 1940]]
    ipdb> print l_var.min()
    0

Ref: For more detailed information on ipython debugging check out this excellent tutorial by Gaël Varoquaux.

2. Invoke ipdb.runcall(function_name, params…) on a specific function.

To do this you have to install the ipdb python module.

$sudo ~/workspace/python_2.7.5_venv/bin/easy_install ipdb

Then in ipython:

In [2]: import ipdb
In [3]: import utils.file_utils
In [4]: fu = utils.file_utils.FileUtils()

In [5]: ipdb.runcall(fu.memory_based_sampling, '/home/naved/datasets/test/test.csv', ',', .90)
> /home/naved/workspace/data_characterization/datachar/utils/file_utils.py(150)memory_based_sampling()
    149 
--> 150             sys_info = sysinfo.SystemInfo()
    151             free_memory_size = sys_info.get_system_free_memory_size()

ipdb> n

Then you can step though the program same as the previously shown example. Go to ipdb github page for more examples.

Installing PyTables 3.0.0 in Virtualenv on Ubuntu 12.04

1. Install hdf5:

sudo apt-get install libhdf5-serial-dev

2. Activate Virtualenv:

source ~/Coding/python_2.7.5_venv/bin/activate

3. Install Cython, Numexpr:

Numexpr: download from http://code.google.com/p/numexpr
$ python setup.py build
$ sudo python setup.py install

or

~/Coding/python_2.7.5_venv/bin/pip install -U numexpr
wget http://cython.org/release/Cython-0.19.1.tar.gz
tar zxvf Cython-0.19.1.tar.gz
cd Cython-0.19.1
sudo python setup.py install

4. Install PyTables

~/Coding/python_2.7.5_venv/bin/pip install git+https://github.com/PyTables/PyTables.git@v.3.0.0#egg=tables

5. Test pytables
To test the installation, run python or ipython and:

In [0]: import tables
In [1]: tables.test()

The test will fail, test with pytables functions from ipython:

In [2]: import pandas as pd
In [3]: import numpy as np
In [4]: import tables
In [5]: df = pd.DataFrame(np.random.randn(8,3))
In [6]: store = pd.HDFStore('test.h5')
In [7]: store.put('df',df)

IPython Notebook : Keyboard Shortcuts

Ipython Notebook Keyboard Shortcuts
Shif­t-E­nter: run cell
Ctrl­-En­ter: run cell in-place
Ctrl-m x: cut cell
Ctrl-m c: copy cell
Ctrl-m v: paste cell
Ctrl-m d: delete cell
Ctrl-m a: insert cell above
Ctrl-m b: insert cell below
Ctrl-m o: toggle output
Ctrl-m 0: toggle output scroll
Ctrl-m l: toggle line numbers
Ctrl-m s: save notebook
Ctrl-m j: move cell down
Ctrl-m k: move cell up
Ctrl-m y: code cell
Ctrl-m m: markdown cell
Ctrl-m t: raw cell
Ctrl-m 1-6: heading 1-6 cell
Ctrl-m p: select previous
Ctrl-m n: select next
Ctrl-m i: interrupt kernel
Ctrl-m .: restart kernel
Ctrl-m h: show keyboard shortcuts

IPython Console Cheatsheet

Ref : http://object22.blogspot.com/2011/12/ipython-cheat-sheet.html
Part I – basic commands
var – tab completion of a variable that starts with prefix ‘var’
who [int, str, ..]- list available variables, or just those with the give type
whos – a table of all the variables, their type and values
psearch var* [type] – search for variables using the pattern and possibly type
reset – clear all variables
logstate – has the logging been started or not
logstart – start logging into (ipython_log.py by default)
logon – turn logging on
logoff – turn logging off
lsmagic – list all built in commands, called magic commands. These are prefixed with % to differentiate between variables if they have the same name.
magic – scrolling help about the magic commands.
cmd? – overview of the command, e.g lsmagic?
cmd?? – more information about the command including source code
var? – information about the variable
page obj – pretty print the object
func arg1, arg2 – automatically adds parentheses and calls func(arg1, arg2)
/func – same as func()
, func arg1, arg2 – same as func(“arg1”, “arg2”)
help(keyword) – manual pages on keyword (module, function, etc)
help() -> topics – lists all the help topics
pdef func – shows the function signature (list of arguments it takes)
pdoc func – prints doc string for the function
pinfo func – same as func?
psource func – shows the source code for the function
pfile func – opens the source file with the func
edit -x func – opens the source with default editor on the line with the function

Part II – navigation and the shell

Analogous to standard shell commands there are pwd, cd, pushd, popd, dirs Tab completion also works with these.
bookmark name [path] – bookmarks the current directory or the given path
bookmark -l – lists all saved bookmarks
bookmark -d name – removes bookmark name
dhist – prints the directory navigation history. cd can take numbers from that list as argument, e.g cd -2
!cmd – runs cmd as a shell command, e.g !ls
alias – lists defined aliases for shell commands we can use
alias newname cmd args – creates a new alias with newname as name for the cmd with args
unalias name – removes alias name
var = !cmd – store the output of cmd into variable
pycat name – prints the highlighted contents of the file
run script.py – runs a python file called script.py and prints the output. Also the functions, variables and modules from the script REMAIN in our namespace!
run -p script.py – run the profiling on the script

Part III – input manipulation

ctrl+p, ctrl+n, arrow keys – previous next command from history
_i, _ii, _iii – prints 1,2 or 3 previous command
_i[X] – prints command number X, e.g _i1
In[X] – prints X-th input commands
hist [N1-N2] – prints a list of previously typed commands, or N1 through N2 previous commands
macro name cmdA-cmdB cmdX – defines a mactor that can execute commands from the history
print macro – prints a macro
edit file – opens up the default editor to edit a file
edit [N1-N2] – edit the history N1 through N2
save file N1-N2 – saves the historey N1 through N2 into file.py
cpaste – paste in source code without, IPython will not format it.
Out[X] – prints the X-th output if it exists
func; – putting does not echo the return of the function and does not put it into Out dictionary

Note! The following commands do not work in IPython 0.11:
store var – save the variable in profile for future use
p var – shorthand for print
exec cmd – execute a command
runlog – executes the IPython log

Install IPython 1.0 and Notebook for Virtualenv Python 2.7.5

Use the pip inside virtual environment to install ipython

(py2.7.5)naved@neo:~/Coding/azureprecomplier$ py2.7.5/bin/pip install ipython

Install ipython dependencies:

sudo apt-get install libncurses5-dev
sudo ~/Coding/py2.7.5/bin/easy_install readline

Install notebook dependencies:

(py2.7.5)naved@neo:~/Coding$ py2.7.5/bin/pip install pyzmq
(py2.7.5)naved@neo:~/Coding$ py2.7.5/bin/pip install jinja2
(py2.7.5)naved@neo:~/Coding$ py2.7.5/bin/pip install tornado

Run ipython inside virtual environment

(py2.7.5)naved@neo:~/Coding/azureprecomplier$ ipython
/home/naved/Coding/azureprecomplier/py2.7.5/lib/python2.7/site-packages/IPython/frontend.py:30: UserWarning: The top-level `frontend` package has been deprecated. All its subpackages have been moved to the top `IPython` level.
  warn("The top-level `frontend` package has been deprecated. "
Python 2.7.5 (default, Aug 30 2013, 22:14:16) 
Type "copyright", "credits" or "license" for more information.

IPython 1.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]:

Run ipython notebook:

(py2.7.5)naved@neo:~/Coding/azureprecomplier$ ipython notebook