\chapter{The Improved Python Debugger \label{chapter-mpdb}}

\declaremodule{}{mpdb}			% not standard, in Python

\platform{Unix, Windows}


\moduleauthor{Matt Fleming}{mattjfleming@googlemail.com}

\sectionauthor{Matt Fleming}{mattjfleming@googlemail.com}


\modulesynopsis{An Improved Python Debugger}


This module is a Goolge Summer Of Code project undertaken in the summer of 
2006 by Matt Fleming.
The \module{mpdb} module defines operations for debugging programs remotely,
debugging threaded programs and debugging separate processes. It builds on the
work done by Rocky Bernstein in his Extended Python Debugger (\module{Pydb}), which 
in turn, builds upon the Python Debugger that is part of the Python 
Standard Library. Rocky also kindly agreed to mentor this project.



The \module{mpdb} module defines the following functions:


\begin{funcdesc}{pdbserver}{address\optional{, mpdb}}
Setup a pdbserver at \var{address} that will allows debuggers to
connect to this debugging session. \var{address} should be of the form
 \code{'protocol procotol-specific-address'}. For example, to use TCP at host
'myhost' and port '8765', \var{address} would be \code{'tcp myhost:8765'}.
To connect to a serial device \var{address} might be \code{'serial /dev/ttyC0'}.
The optional argument \var{mpdb} is an \class{MPdb} instance to use
for this pdbserver. If \var{mpdb} is ommitted, a new \class{MPdb} instance
is created.
\end{funcdesc}

\begin{funcdesc}{target}{address}
Connect to a pdbserver at \var{address}. \var{address} should use the same
format as \function{pdbserver}.
\end{funcdesc}

\begin{excdesc}{Exit}
Exception raised when the debugger is going to immediately exit.
\end{excdesc}

\section{Debugger commands}
\label{mpdb-commands}

The section describes the command that are available to the debugger when
run as a standalone program.

Most commands can be abbreviated to one or two letters;
e.g., \samp{h(elp)} means that either \samp{h} or \samp{help} can be
used to enter the help command (but not \samp{he}, \samp{hel},
\samp{H}, \samp{Help}, or \samp{HELP}).  Arguments to commands must
be separated by whitespace (spaces or tabs).  Optional arguments are
enclosed in square brackets (\samp{[]}) in the command syntax; the
square brackets must not be typed.  Alternatives in the command syntax
are separated by a vertical bar (\samp{|}).

Entering a blank line repeats the last command entered.  Exceptions: if
the last command was a \samp{list} command, the next 11 lines are
listed or if the debugger has just connected to a pdbserver, the debugger
exits.

Commands that the debugger doesn't recognize are assumed to be Python
statements and are executed in the context of the program being
debugged.  Python statements can also be prefixed with an exclamation
point (\samp{!}).  This may be a good way to inspect the program being
debugged; it is even possible to change a variable or call a function.
When an exception occurs in such a statement, the exception name is
printed but the debugger's state is not changed.

The debugger supports aliases.  Aliases can have parameters which
allow a certain level of adaptability to the context under
examination. See \ref{command:aliases}.

{\bf Debugger Prompt}:\label{debugger:prompt}\\
By default the debugger's prompt string is \samp{(MPdb) } with 
a trailing blank. Recursive invocations using the
\samp{debug} command strip off the trailing blanks, add a layer of
parenthesis around the string, and add a trailing blank. For example,
for the default prompt the first debug invocation will be
\samp{((MPdb)) }.

\emph{There's currently a bug in the code where specified trailing
blanks are chopped. Furthermore the prompt may change in the future to
add a history number. It is generally not advisable to change the
prompt.}

If you do need to change the prompt see \ref{command:prompt}.

Multiple commands may be entered on a single line, separated by
\samp{;;}.  (A single \samp{;} is not used because it is
the separator for multiple commands in a line that is passed to
the Python parser.)
No intelligence is applied to separating the commands;
the input is split at the first \samp{;;} pair, even if it is in
the middle of a quoted string.

\subsection{Status and Debugger Settings ({\tt info}, {\tt set}, {\tt show})\label{subsection-status}}

An {\tt info} command shows things about the program being debugged.A
{\tt set} command modifies parts of the debugger environment. You can
see these environment settings with the {\tt show} command.

In all of the set options that take ``on'' or ``off'' parameters, you
can also use 1 for ``on'' and 0 for ``off.''

Each command has a corresponding {\tt show} command to show the current
value. See \ref{subsubsection-show} for these counterparts.

If a
\ulink{\module{readline}}{http://docs.python.org/lib/module-readline.html}
module is available, \code{mpdb} can keep track of the commands you
type during your debugging sessions, so that you can be certain of
precisely what happened.  The \code{set history} commands to manage
the command history facility.

POSIX-style line tracing is available and the \code{set linetrace}
commands can be used to control that.

You may want to save the output of \code{mpdb} commands to a file.
See the \code{set logging} commands to control \code{mpdb}'s logging.

\subsubsection{Info ({\tt info})\label{subsubsection-info}}

Running this command without parameters will print the list of
available info commands. Below is a description of the individual
commands.

\begin{description}

\item[info args]

Show function/method parameters. See \ref{command:info-args}.

\item[info break]

Show the list of breakpoints. 

\item[info globals]

Show the global variables. See \ref{command:info-globals}. 

\item[info line]

Show the current line number in source file. If a function name is
given, the starting line of the function is reported.

\item[info locals]

Show the local variables. See \ref{command:info-locals}. 

\item[info program]

Show the execution status of the program. The possible status is that
the program is not running (e.g. in post-mortem dump), or the program is
``stoppped'' and if stopped at a breakpoint that is shown as well.

\item[info source]

Information about the current Python file.

\item[info target]

Show the target of the debugger, which will either be local or remote.

\end{description}

\subsubsection{Set ({\tt set})\label{subsubsection-set}}

\begin{description}

\item[set basename on\code{\Large{|}}off]\label{command:basename}

When showing filenames print only the basename. This option is useful
in regression testing where the base file names are the same
on different installations even though the directory path may
be different. You may want to use this in other
situtations as well, like showing a debugger session in a manual
such as this one. 

\item[set cmdtrace on\code{\Large{|}}off]\label{command:cmdtrace}
 
Show lines as they are read from the debugger command file (or
\samp{source} debugger command). This is useful in running
regression tests, but it may be helpful in tracking down a problem in
your \code{.mpdbrc} file.

\item[set history filename \var{filename}]\label{command:hist-filename}

Set the filename in which to record the command history.
(the list of previous commands of which a record is kept). The default
file is \verb|~/.mpdbhist|.

\item[set history save]

Set saving of the history record on exit.  Use ``on'' to enable the
saving, and ``off'' to disable it.  Without an argument, saving is
enabled.

\item[set history size]

Set the size of the command history, ie. the number of previous
commands to keep a record of. The default is 256.

\item[set linetrace on\code{\Large{|}}off]\label{command:linetrace}

If this is set on, the position (file and linenumber) is shown before
executing a statement. By default this is off. Using the command-line
option \samp{--trace} when invoking \code{mpdb} implicitly sets this
on.  For information on \samp{--trace}, see \ref{switch:trace}.

Using this option will slow down your
program. Unless single stepping through a program, normally the
debugger is called only at breakpoints or at the call and return of a
function or method. However when line tracing is turned on, the
debugger is called on execution of every statement. 

That said, execution may still be pretty fast. If you want to slow
down execution further, see the following option.

\item[set linetrace delay \var{time}]\label{command:linetrace-delay}

One of the useful things you can do with this debugger if you run it
via a front-end GUI is watch your program as it executes. To do this,
use \samp{set linetrace on} which prints the location before each
Python statement is run. Many front-end GUIs like the one in GNU Emacs
and \code{ddd} will read the location and update the display
accordingly.

There is however one catch---Python runs too fast. So by using this
option you can set a delay after each statement is run in order for
GNU and your eyes to catch up with Python. Specify a floating
point indicating the number of seconds to wait. For example: 

\begin{verbatim}
set linetrace delay 0.5 # 1/2 a second
\end{verbatim}

In my experience half a second is about right.

\item[set listsize \var{lines}]\label{command:listsize}

Sets how many lines are shown by the \code{list} command. See
\ref{command:list}.

\item[set logging]\label{command:logging}

Prints \code{set logging} usage.

\item[set logging on\code{\Large{|}}off]\label{command:logging}

Enable or disable logging.

\item[set logging file \var{filename}]\label{command:log-redirect}

By default, \code{mpdb} output will go to both the terminal and the
logfile.  Set \code{redirect} if you want output to go only to the log
file.

\item[set logging overwrite on\code{\Large{|}}off]\label{command:log-overwrite}

By default, \code{mpdb} will append to the logfile.  Set
\code{overwrite} if you want \code{set logging on} to overwrite the
logfile instead.

\item[set logging redirect on\code{\Large{|}}off]\label{command:log-redirect}

By default, \code{mpdb} output will go to both the terminal and the
logfile.  Set \code{redirect} if you want output to go only to the log
file.

\item[set prompt \var{prompt-string}]\label{command:prompt}

Set debugger's prompt string. By default it is \samp{(MPdb) } with
a trailing space. For information on how the prompt
changes, see \ref{debugger:prompt}.

\emph{There's currently a bug in the code where specified trailing
blanks specified. Furthermore the prompt may change in the future to
add a history number. It is generally not advisable to change the
prompt.}

\item[set target-address \var{target_addr}]\label{command:target}

Set the debugger's target. This sets the current address to connect to
for commands such as \code{attach} and \code{target}.

\end{description}

\subsubsection{Show ({\tt show}) \label{subsubsection-show}}

All of the ``show'' commands report some sort of status and all have a
corresponding ``set'' command to change the value. See
\ref{subsubsection-set} for the ``set'' counterparts.

\begin{description}

\item[show args]

Show the argument list that was given the program being debugged or it is
restarted

\item[show basename]

Show short or long filenames

\item[show cmdtrace]

Show the debugger commands before running

\item[show commands]

Show the history of commands you typed. You can supply a command
number to start with, or a `+' to start after the previous command
number shown.  A negative number starts from the end.

This command is available only if a
\ulink{\module{readline}}{http://docs.python.org/lib/module-readline.html}
module is available and supports the history saving.

\item[show history]

Generic command for showing command history parameters. The command
history filename, saving of history on exit and size of history file
are shown.

\item[show linetrace]

Show the line tracing status.

\item[show linetrace delay]

Show the delay after tracing each line.

\item[show listsize]

Show the number of source lines \code{mpdb} will list by default.

\item[show logging]

Show summary information of logging variables which can be set via
\code{set logging}.

\item[show logging file]

Show the current logging file.

\item[show logging overwrite]

Show whether logging overwrites or appends to the log file.

\item[show prompt]

Show the current debugger prompt.

\item[show target-address]

Show the current target address.

\item[show version]

Show the debugger version number.

\end{description}

\subsection{Breakpoints ({\tt break}, {\tt tbreak}, {\tt clear},
  {\tt commands}, {\tt delete}, {\tt disable}, {\tt
  ignore})\label{subsubsection-brkpts}}

\index{breakpoints} A breakpoint makes your program stop at that
point. You can add conditions for each breakpoint.  You can set
breakpoints with the \code{break} command and its variants. You can specify
the place where your program should stop by file and line number or by
function name.

The debugger assigns a number to each breakpoint when you create it;
these numbers are successive integers starting with 1.  In many of the
commands for controlling various features of breakpoints you use this
number.  Each breakpoint may be enabled or disabled; if disabled, it
has no effect on your program until you enable it again.

The debugger allows you to set any number of breakpoints at the same
place in your program.  There is nothing unusual about this because
different breakpoints can have different conditions associated with them.

\begin{description}
\item[b(reak) \optional{\optional{\var{filename}:}\var{lineno}\code{\Large{|}}\var{function}\optional{, \var{condition}}}]

With a \var{lineno} argument, set a break at that line number in the
current file.  With a \var{function} argument, set a break at the
first executable statement within that function.  The line number may
be prefixed with a filename and a colon to specify a breakpoint in
another file (probably one that hasn't been loaded yet).  The file is
searched on \code{sys.path}.  Note that each breakpoint is assigned a
number to which all the other breakpoint commands refer.

If a second argument is present, it is an expression which must
evaluate to true before the breakpoint is honored.

Without an argument, list all breaks, including for each breakpoint:
the number of times that breakpoint has been hit, the current
ignore count, and the associated condition if any.

\item[tbreak \optional{\optional{\var{filename}:}\var{lineno}\code{\Large{|}}\var{function}\optional{, \var{condition}}}]

Temporary breakpoint, which is removed automatically when it is
first hit.  The arguments are the same as those for \code{break}.

\item[cl(ear) \optional{\optional{\var{filename}:}\var{lineno}\code{\Large{|}}\var{function}}]

Clear breakpoint at specified line or function.  Argument may be line
number, function name, or `*' and an address.  If a line number is
specified, all breakpoints in that line are cleared.  If a function is
specified, the breakpoints at the beginning of the function are cleared.  If an
address is specified, breakpoints at that address are cleared.

With no argument, clears all breakpoints in the line where the selected
frame is executing.

See also the \code{delete} command below which clears breakpoints by
number.

\item[commands \optional{\optional{\var{bpnumber}}}]

Set commands to be executed when a breakpoint is hit.
Give breakpoint number as the argument after "commands".
With no bpnumber argument, commands refers to the last one set.
The commands themselves follow starting on the next line.
Type a line containing "end" to terminate the commands.

To remove all commands from a breakpoint, type commands and
follow it immediately with \code{end}; that is, give no commands.

Specifying any command resuming execution (currently \code{continue},
\code{step}, \code{next}, \code{return}, \code{jump}, and \code{quit})
terminates the command list as if that command was immediately
followed by \code{end}.  This is because any time you resume execution
(even with a simple next or step), you may encounter another
breakpoint---which could have its own command list, leading to
ambiguities about which list to execute.

If you use the \code{silent} command in the command list, the
usual message about stopping at a breakpoint is not printed.
This may be desirable for breakpoints that are to print a
specific message and then continue.  If none of the other
commands print anything, you see no sign that the breakpoint
was reached.

\item[delete \optional{\var{bpnumber} \optional{\var{bpnumber \ldots}}}]

With a space-separated list of breakpoint numbers, clear those
breakpoints.  Without argument, clear all breaks (but first
ask confirmation).

\item[disable \optional{\var{bpnumber} \optional{\var{bpnumber \ldots}}}]

Disable the breakpoints given as a space-separated list of
breakpoint numbers.  Disabling a breakpoint means it cannot cause
the program to stop execution, but unlike clearing a breakpoint, it
remains in the list of breakpoints and can be (re-)enabled.

\item[enable \optional{\var{bpnumber} \optional{\var{bpnumber \ldots}}}]

Enable the breakpoints specified.

\item[ignore \var{bpnumber} \optional{\var{count}}]

Set the ignore count for the given breakpoint number.  If count is
omitted, the ignore count is set to 0.  A breakpoint becomes active
when the ignore count is zero.  When non-zero, the count is
decremented each time the breakpoint is reached, the breakpoint is not
disabled, and any associated condition evaluates to true.

\item[condition \var{bpnumber} \optional{\var{condition}}]

Condition is an expression which must evaluate to true before
the breakpoint is honored.  If condition is absent, any existing
condition is removed; i.e., the breakpoint is made unconditional.

\end{description}

\subsection{Resuming Execution ({\tt step}, {\tt next}, {\tt
    finish}, {\tt return}, {\tt continue}, {\tt jump})\label{subsubsection-resume}}

``Continuing'' means resuming program execution until the program
completes normally.  In contrast, ``stepping'' means executing just
one statement of the program. When continuing or stepping, the program may stop even sooner, due to a breakpoint or an
exception.

\begin{description}

\item[s(tep) \optional{\var{count}}]\label{command:step}

Execute the current line, stop at the first possible occasion
(either in a function that is called or on the next line in the
current function).

\item[n(ext) \optional{\var{count}}]\label{command:next}

Continue execution until the next line in the current function
is reached or the function returns.  The difference between \samp{next} and
\samp{step} is that \samp{step} stops inside a called function, while
\samp{next} executes called functions at (nearly) full speed, 
stopping only at the next line in the current function.

\item[finish]\label{command:finish}

Continue execution until the current function returns.
that point the \samp{retval} command can be used to show the
return value. The short command name is rv.

See also \ref{command:retval} and \ref{command:return}.

\item[return]\label{command:return}

Make selected stack frame return to its caller. Control remains in the
debugger, but when you continue execution will resume at the return
statement found inside the subroutine or method.  At present we are
only able to perform this if we are in a subroutine that has a
\code{return} statement in it.  See also \ref{command:retval} and
\ref{command:finish}

\item[c(ontinue)]

Continue execution, stop only when a breakpoint is encountered.

\item[jump \var{lineno}]

Set the next line that will be executed. available only in the
bottom-most frame.  This lets you jump back and execute code
again, or jump forward to skip code that you don't want to run.

Not all jumps are allowed---for instance it
is not possible to jump into the middle of a \keyword{for} loop or out
of a \keyword{finally} clause.

One common use for the \code{jump} statement is to get out of a
loop. Sometimes the bounds of loops are computed in advance so you
can't leave a loop early by say setting the value of the loop variable

Here's an example demonstrating this:

\begin{verbatim}
mpdb ptest.py
(ptest.py:2): 
(MPdb) list
  1  	#!/bin/python
  2  ->	for i in range(1,10):
  3  	    print i
  4  	print "tired of this"
[EOF]
(MPdb) step
(ptest.py:3): 
(MPdb) i=1000
(MPdb) step
1000
(ptest.py:2): 
(MPdb) jump 4
(ptest.py:4): 
(MPdb) step
tired of this
--Return--
--Return--
The program finished and will be restarted
(ptest.py:2): 
(MPdb) 
\end{verbatim}

Not that the assignment of 1,000 to \code{i} took effect, although it
had no effect on termintating the \code{for} loop; \code{jump} was
needed to get out of the loop early.

\end{description}

\subsection{Examining Call Frames ({\tt info args}, {\tt info
    locals}, {\tt down}, {\tt frame}, {\tt up})\label{subsection-frames}}

Each line in the backtrace shows the frame number and the function
name, if it exists and the place in a file where the statement is
located.

Here is an example of a backtrace from a sample Towers of Hanoi
program that is used in regression testing:

\begin{verbatim}
## 0 hanoi() called from file '/tmp/pydb/test/hanoi.py' at line 5
-> 1 hanoi() called from file '/tmp/pydb/test/hanoi.py' at line 6
## 2 in file '/tmp/pydb/test/hanoi.py' at line 29
## 3 in file '<string>' at line 1
## 4 run() called from file '/usr/lib/python2.4/bdb.py' at line 366
\end{verbatim}

The {\tt->} arrow indicates the focus. In the example, I issued an
\samp{up} command which is why the focus is on 1 rather than 0 as it
would normally be after a stop.

There are two ``hanoi'' frames listed because this is a hanoi called
itself recursively. In frame 2 and 3 we don't have a function name
listed. That's because there is none. Furthermore in frame 3 there is
a funny ``in file {\tt '<string>'} at line 1.'' That's because there
isn't even a file assocated with the command. The command issued:

\begin{verbatim}
   exec cmd in globals, locals
\end{verbatim}

This statement can be seen in frame 4. This is a bug which Rocky hopes to
fix in \module{Pydb} with a more informative message.

Finally, note that frames 2 and 3 really are not part of the
program to be debugged but are part of the internal workings of the
debugger. It's possible to hide this, but in the open spirit of Python
for now it hasn't been hidden.

\begin{description}

\item[info args]\label{command:info-args}

Show the method or function parameters and their values. 

Here is an example of the output for the backtrace of the hanoi
program shown at the beginning of this section:

\begin{verbatim}
(MPdb) info args
n= 3
a= a
b= b
c= c
(MPdb) 
\end{verbatim}

\item[info locals]\label{command:info-locals}

Show all local variables for the given stack frame. This will include
the variables that would be shown by \samp{info args}.

\item[where\code{\Large{|}}T\code{\Large{|}}bt \optional{\var{count}}]

Print a backtrace, with the most recent frame at the top.  An
arrow indicates the current frame, which determines the context of
most commands.

With a positive number \var{count}, print at most many entries.

An example of a backtrace is given at the beginning of this section.

\item[retval\code{\Large{|}}rv]\label{command:retval}

Show the value that will be returned by the current function. This
command is meaningful only just before a return (such as you'd get
using the \code{finish} or \code{return} commands) or stepping after a
return statement.

To change the value, make an assignment to the variable
\code{__return__}.

See also \ref{command:finish}.

\item[down \optional{\var{count}}]

Move the current frame one level down in the stack trace
(to a newer frame). With a count, which can be  positive
or  negative, move that many positions.

\item[up \optional{\var{count}}]

Move the current frame one level up in the stack trace (to an older
frame). With a count, which can be positive or negative,
move that many positions.

\item[frame \optional{\var{position}}]
Move the current frame to the specified frame number. A negative
number indicates position from the end, so \code{frame -1} moves to 
the newest frame, and \code{frame 0} moves to the oldest frame.

\end{description}

\subsection{Examining Data ({\tt print}, {\tt pprint}, {\tt examine}, {\tt info globals})\label{subsection-data}}

\begin{description}

\item[display \optional{\var{format}} \var{expression}]

Print value of expression \var{expression} each time the program
stops.  \var{format} may be used before \var{expression} as in the
"print" command.  \var{format} "i" or "s" or including a size-letter
is allowed, and then \var{expression} is used to get the address to
examine.

With no argument, display all currently requested auto-display
expressions.  Use "undisplay" to cancel display requests previously
made.

\item[undisplay \optional{\var{format}} \var{expression}]

Evaluate the \var{expression} in the current context and print its
value.  \note{\samp{print} can also be used, but is not a debugger
command---it executes the Python \keyword{print} statement.}


\item[p \var{expression}]

Evaluate the \var{expression} in the current context and print its
value. One can also often have an expression printed by just typing
the expression. If the first token doesn't conflict with a debugger
built-in command Python will, by default, print the result same as if
you did this inside a Python interpreter shell. To make things even
more confused, a special case of running an arbitrary Python command
is the \samp{print} command. But note that the debugger command is
just \samp{p}.

So what's the difference? The debugger's print command encloses
everything in a \samp{repr()}, to ensure the resulting output is not
too long.  \emph{\note{Should add info as to how to customize what
``too long'' means}}. So if you want abbreviated output, or are not
sure if the expression may have an arbitrarily long (or infinite)
representation, then use \samp{p}. If you want the output as Python
would print it, just give the expression or possibly use python's
\samp{print} command.

\item[pp \var{expression}]

Like the \samp{p} command, except the value of the expression is
pretty-printed using the \module{pprint} module.

\item[examine \var{expression}]

Print the type of the expression and pretty-print its value.  For
functions, methods, classes, and modules print out the documentation
string if any.  For functions also show the argument list.

The examine debugger command in Perl is the model here, however much
more work is needed. Note that \samp{x} is not a short name for
``expression'' (as it is in Perl's debugger), although you could
easily make it be via an alias.

\item[info globals]\label{command:info-globals}

Show all global variables. These variables are not just the variables
that a programs sees via a \code{global} statement, but all of them
that can be accessible.

\end{description}

\subsection{Running Arbitrary Python Commands ({\tt debug}, {\tt !})\label{subsection-commands}}

\begin{description}

\item[\optional{!}\var{statement}]

Execute the (one-line) \var{statement} in the context of
the current stack frame.
The exclamation point can be omitted unless the first word
of the statement resembles a debugger command.
To set a global variable, you can prefix the assignment
command with a \samp{global} command on the same line, e.g.:

\begin{verbatim}
(MPdb) global list_options; list_options = ['-l']
(MPdb)
\end{verbatim}

\item[debug \var{statement}]

Enter a recursive debugger that steps through the code argument (which
is an arbitrary expression or statement to be executed in the current
environment). The prompt is changed to indicate nested behavior. See
\ref{debugger:prompt}

\end{description}

\subsection{Restarting a Python Script ({\tt restart}, {\tt run})\label{subsection-restart}}

\begin{description}

\item[restart \var{args...}]\label{command:restart}

Restart debugger and program via an \code{exec} call. All state
is lost, and new copy of the debugger is used. \footnote{The \code{restart}
command may not function properly on the Windows platform, this is because of
the way that Windows handles \code{execvp} calls.}

Sometimes in debugging it is necessary to modify module code when one
finds a bugs in them. Python will not notice dynamically that a module
has changed and thus not reimport it (which also means that module
initialization code is not rerun either). So in such a situation one
must use \code{restart} rather than \code{run}.\footnote{It may be
possible to unimport by removing a the module from a namespace, but if
there are shared dynamically loaded objects those don't get unloaded.}

\item[run \var{args...}]

Run or ``soft'' restart the debugged Python program. If a string is
supplied that becomes the new command arguments.  History,
breakpoints, actions and debugger options are preserved.  \code{R} is
a short command alias for \code{run}. 

You may notice that the sometimes you can \code{step} into modules
included via an \code{import} statement, but after a \code{run} this
stepping skips over the import rather than goes into it. A similar
situation is that you may have a breakpoint set inside class
\code{__init__} code, but after issiuing \code{run} this doesn't seem
to get called---and in fact it isn't run again!

That's because in Python the \code{import} occurs only once. In fact,
if the module was imported \emph{before\/} invoking the program, you
might not be able to step inside an \code{import} the first time as
well.

In such a situation or other situations where \code{run} doesn't seem
to have the effect of getting module initialization code executed,
you might try using \code{restart} rather than \code{run}.

\end{description}

\subsection{Interfacing to the OS ({\tt cd}, {\tt pwd},
  {\tt shell})\label{subsection-os}}

\begin{description}

\item[cd \var{directory}]\label{command:cd}

Set working directory to \var{directory} for debugger and program
being debugged.

\item[pwd]\label{command:pwd}

Print working directory.  

\item[shell \var{statement}]\label{command:shell}

Execute the rest of the line as a shell command.


\end{description}

\subsection{Listing Program Code ({\tt list}, {\tt disassemble})\label{subsection-listing}}

\begin{description}

\item[disassemble \optional{\var{arg}}]

With no argument, disassemble at the current frame location.  With a
numeric argument, disassemble at the frame location at that line
number. With a class, method, function, code or string argument,
disassemble that.

\item[l(ist) \optional{- \Large{|} \var{first}\optional{, \var{last}}}]\label{command:list}

List source code.  Without arguments, list
\emph{n} lines centered around the current line or continue the previous
listing, where \emph{n} is the value set by \samp{set listsize} or
shown by \samp{show listsize}.  The default value is 10.

\samp{list -} lists \emph{n} lines before a previous listing. With one
argument other than `-', list \emph{n} lines centered around the
specified position.  With two arguments, list the given range; if the
second argument is less than the first, it is a count. \var{first} and
\var{last} can be either a function name, a line number, or
\var{filename}:\var{line-number}.

\end{description}

\subsection{Interfacing to the debugger ({\tt alias}, {\tt complete}, 
{\tt help}, {\tt quit}, {\tt source}, {\tt unalias})\label{subsection-misc}}

\begin{description}

\item[alias \optional{\var{name} \optional{command}}]\label{command:aliases}

Create an alias called \var{name} that executes \var{command}.  The
command must not be enclosed in quotes.  Replaceable parameters
can be indicated by \samp{\%1}, \samp{\%2}, and so on, while \samp{\%*} is
replaced by all the parameters.  If no command is given, the current
alias for \var{name} is shown. If no arguments are given, all
aliases are listed.

Aliases may be nested and can contain anything that can be legally
typed at the \code{mpdb} prompt.  Note that internal \code{mpdb}
commands can be overridden by aliases.  Such a command is then hidden
until the alias is removed.  Aliasing is recursively applied to the
first word of the command line; all other words in the line are left
alone.

As an example, here are two useful aliases (especially when placed
in the \code{.mpdbrc} file):

\begin{verbatim}
#Print instance variables (usage "pi classInst")
alias pi for k in %1.__dict__.keys(): print "%1.",k,"=",%1.__dict__[k]
#Print instance variables in self
alias ps pi self
\end{verbatim}
                
\item[complete \var{command-prefix}]\label{command:complete}

If
\ulink{\module{readline}}{http://docs.python.org/lib/module-readline.html}
or one of readline-compatible interfaces such as
\ulink{\module{pyreadline}}{http://projects.scipy.org/ipython/ipython/wiki/PyReadline/Intro}
are available on your OS, the \code{complete} command will print a
list of command names that start with \var{command-prefix}.

\code{complete} will also work on \code{info}, \code{set}, and
\code{show} sub-command. 

In addition the command-completion key (usually the tab key) can be
used to complete command names, or \code{info}, \code{set}, and
\code{show} subcommands.

\item[h(elp) \optional{\var{command} \optional{subcommand}}]

Without argument, print the list of available commands.  With
\var{command} as argument, print help about that command.  \samp{help
mpdb} displays the full documentation file; if the environment
variable \envvar{PAGER} is defined, the file is piped through that
command.  Since the \var{command} argument must be an identifier,
\samp{help exec} must be entered to get help on the \samp{!} command.

Some commands, \code{info}, \code{set}, and \code{show} can accept an
additional subcommand to give help just about that particular
subcommand. For example \code{help info line} give help about the
\code{info line} command.

\item[q(uit)]\label{command:quit}

Quit the debugger. The program being executed is aborted. For now,
\code{kill} is a synonym for quit.

\item[source \var{filename}]\label{command:source}

Read commands from a file named \var{filename}.
Note that the file \code{.mpdbrc} is read automatically
this way when \code{mpdb} is started.

An error in any command terminates execution of the command and
control is returned to the console.

For tracking down problems with command files, see the \samp{set
cmdtrace on} debugger command, \ref{command:cmdtrace}. 

\item[unalias \var{name}]\label{command:unalias}

Delete the specified alias.

\end{description}

\section{Debugger Architecture}
\label{debugger-arch}
This section describes the organisation of the debugger and its 
architecture.

XXX Images coming soon..
\begin{center}
1. MPdb instance

2. Input/Output MQueues

3. MConnection
\end{center}


\subsection{The Message Queue}
Communication inside the debugger and also between debuggers (i.e. on a remote
connection) takes place via \emph{message queues}. The idea behind the message
queue is that it should provide a level of abstraction from exactly \emph{how}
messages are communicated. An object within \module{mpdb} need only know which
queue to place commands onto in order to communicate with another object.

\subsection{Instruction format}
The instructions that are sent on the queue are made up of the following
components,

Process/ThreadID SequenceID PacketLength Data

\section{MPdb Objects}
\label{mpdb-objects}

MPdb objects have the following methods. These methods should usually never
be called directly, they are documented here to serve as a programmer's
reference to the inner workings of mpdb. Refer to \ref{module-mpdb} for 
routines that are considered ``safe'' to be called from user code.

\begin{methoddesc}[mpdb]{remote_onecmd}{line}
This method is used by an \class{MPdb} instance that is connected
to a remote machine. It replaces the \code{onecmd} method inherited from the
\class{Cmd} class in the \module{cmd} module. \var{line} contains the
command to be executed on the remote machine. Instead of the debugger
instance interpreting commands, all commands are sent directly to the 
remote machine where they are interpreted, executed and the results are 
sent back to the client. The client then writes the results to the \class{MPdb}
objects stdout file-object.
\end{methoddesc}

\begin{methoddesc}[mpdb]{do_pdbserver}{addr}
Setup a pdbserver at \var{addr} and wait for incoming connections. \var{addr}
must be string containing a protocol to use and a protocol-specific address.
Refer to \ref{module-mpdb} for more information.
\end{methoddesc}

\begin{methoddesc}[mpdb]{do_target}{args}
This method is called to connect to a remote pdbserver. \var{args} should 
contain a protocol and an address to connect to.
\end{methoddesc}

\section{MTracer Objects}
\label{mtracer-objects}

MTracer objects are used to trace stack frames in threads. They
have the following methods:

\begin{methoddesc}[mtracer]{trace_dispatch}{frame, event, arg}
This method is the global trace function for this thread. It delegates
work to other methods according the value of \var{event} which may be
\code{'line'}, \code{'call'}, \code{'exception'}, \code{'return'}, 
\code{'c_call'}, \code{'c_return'} or \code{'c_exception'}.
\end{methoddesc}

\begin{methoddesc}[mtracer]{dispatch_line}{frame}
This method is called when the event of the current frame object is
\code{'line'}. This method checks to see whether we have been instructed
to stop at this frame or if we have a breakpoint set at this frame. If
either of these conditions are true, control is passed to the main
debugger via it's \function{user_line()} method.
\footnote{The main debugger is the \class{MPdb} instance. It is
the debugger that is tracing the \class{_MainThread} object (the Python
Interpreter).}
\end{methoddesc}

\begin{methoddesc}[mtracer]{dispatch_call}{frame, arg}
This method is called when the event of the current frame object is 
\code{'call'}. If the debugger has not been instructed to stop or break
inside this function, this method returns and the function is not traced. 
Otherwise the main debugger's \function{user_call()} method is called, 
passing \var{frame} and \var{arg} as arguments.
\end{methoddesc}

\begin{methoddesc}[mtracer]{dispatch_return}{frame, arg}
This method is called when the event of the current frame object is
\code{'return'}. If the debugger has been instructed to stop at this
frame the main debugger's \function{user_return()} method is called.
\end{methoddesc}

\begin{methoddesc}[mtracer]{dispatch_exception}{frame, arg}
This method is called when the event of the current frame object is 
\code{'exception'}. If the debugger has been instructed to stop here
the main debugger's \function{user_exception()} method is called.
\end{methoddesc}

\section{Thread Debugging}
\label{thread-debug}
This section provides information on Python's thread debugging facilities and
how \module{mpdb} makes use of them.

\module{mpdb} uses a master-slaves system for handling thread debugging. The
main debugger (master) is the instance of \class{MPdb} that is
used by the Python 
Interpreter. The thread that runs the Python Interpreter is a special class
called \class{_MainThread}. Whenever we refer to the main debugger, this is 
what we are refering to. The \class{MTracer} objects (slaves) 
are responsible for handling threads that are created in the script being
debugged. Any messages that need to be output to the user are placed
on the main debugger's \code{outputqueue} queue. This messages are 
periodically written to the main debugger's \code{stdout} variable.

% How does Python facilitate debugging threads
\subsection{Python's Thread Debugging Features}

This section aims to provides the reader with information into how Python
allows threads to be debugged.

There are two modules in the Python Standard Library that provide thread
functionality, \ulink{\module{thread}}{http://docs.python.org/lib/module-thread.html} and \ulink{\module{threading}}{http://docs.python.org/lib/module-threading.html}. 
There are three types of threads in Python,

- threads created with the \module{thread} module. These are low-level threads.

- threads created with the \module{threading} module. These are high-level
  threads.

- the \class{_MainThread}. This thread is the main thread, it is created
when the Python interpreter is started. All signals go to this thread.


The global trace function for threads created with the \module{threading}
module can be set by calling \function{threading.settrace()}. We can use
this method to inspect the frame objects on the stack, for example, to find
the filename and line number of the code object currently executing. See
\ulink{The Python Library Reference}
{http://docs.python.org/lib/debugger-hooks.html} for documentation on how
debugger's use the global trace function.


\subsection{\module{mpdb}'s Thread Debugging Facilities}

\module{mpdb} does not have thread debugging on by default. It can be turned
on whilst inside the debugger by issuing the command \code{set thread}.

\emph{All the information below can be considered the internals of how mpdb
works. It is a programmer's reference.}

When thread debugging is activated in \module{mpdb}, it imports the
\module{mthread} module and calls that modules \function{init()} function,
passing as an argument, the instance of \class{MPdb} that is currently
being used. This debugger argument is stored as a global variable in the
\module{mthread} module. Whenever a thread is created, a new \class{MTracer}
object is instantiated and the trace function for the thread is set to this
object's \function{trace_dispatch} function. Refer to \ulink{Section 9.2 - How It Works}{http://docs.python.org/lib/debugger-hooks.html} of the \module{Pdb} 
documentation in the Python Standard Library for more information on how
debuggers interact with trace functions.

An \class{MTracer} object provides methods for dealing with the different
possible events in that occur in the frame object \ref{mtracer-objects}.

The main debugger is repsonsible for debugging the \class{_MainThread} object
and the \class{MTracer} objects are responsible for debugging all other
threads. When a thread needs to stop exection, it acquires the global lock
\member{global_lock}, which is a \class{Lock} object. The \function{acquire}
call will block until the lock is obtained. When this lock is obtained 
execution transfers to the thread the \class{MTracer} is running in. This
means that this thread is now the current thread. This method of halting
the execution of the entire program is taken from how GDB handle's threads.

\emph{The GDB thread debugging facility allows you to observe all threads while your program runs--but whenever GDB takes control, one thread in particular is always the focus of debugging.} - \citetitle[http://sources.redhat.com/gdb/current/onlinedocs/gdb_5.html\#SEC27]{Debugging with GDB: Debugging programs with multiple threads}

and..

\emph{Whenever your program stops under GDB for any reason, all threads of execution stop, not just the current thread. This allows you to examine the overall state of the program, including switching between threads, without worrying that things may change underfoot.} - \citetitle [http://sources.redhat.com/gdb/current/onlinedocs/gdb_6.html\#SEC45]{Debugging with GDB: Stopping and starting multi-thread programs}

When the main thread exits from debugging, it is system defined as to what
happens to the other threads. On most systems, they are killed without
executing \code{try/finally} clauses or executing object destructors.

\section{Remote Debugging}
\label{remote-debug}
This section describes how \module{mpdb} handles debugging remotely.

Remote debugging in \module{mpdb} takes place between a \emph{pdbserver}
and a client. The pdbserver is analogous to GDB's gdbserver. A pdbserver
is run on the same machine as the program being debugged. This pdbserver
then allows clients to connect to it which begins a debugging session. In
this session all commands come from the client and are executed on the 
pdbserver.

\section{Debugging an already running process}
\label{proc-debug}
This section describes how \module{mpdb} debugs processes that are external
to the process in which \module{mpdb} is being run. A user can call the
\code{mpdb.process_debugging} function inside their code to enable a debugger
running as a separate process to debug their program. The way this works
is that \module{mpdb} sets up a signal handler for a specified signal to
\module{mpdb}'s signal handler. If there was a signal handler for this signal
previously, it is saved. When this signal handler is traps the signal it
starts a \code{pdbserver} that a debugger can connect to. Once the debugger
has 'detached' from the \code{pdbserver}, \module{mpdb}'s signal handler
is removed and the old signal handler (if any) is restored.

\begin{funcdesc}{process_debugging}{sig, protocol, addr}
This function enables debugging by another process by setting the signal
handler for signal \code{sig} to \module{mpdb}'s signal handler. When a client
wishes to connect to this process, it uses the \code{attach} command which
sends \module{mpdb}'s \code{debug_signal} to a process. It is up to the user
of \module{mpdb} to ensure that the programming they are debugging and the 
signal sent during the \code{attach} command are the same, \ref{command::set:}.
When the signal is received by \module{mpdb}'s signal handler a \code{pdbserver}
is started on \code{addr} using protoocol \code{protocol}.
\code{sig}, \code{protocol} and \code{addr} are all optional arguments that
specify the signal to use for debugging, the protocol to use for \code{pdbserver}
and the add for the \code{pdbserver}, respectively. 


This function returns a string which is the address used for the 
\code{pdbserver}. If an address is not specified it defaults to
a path consisting of the system's temporary
directory is used as returned by \code{tempfile.gettempdir()}, and a filename
made up from the process' pid and the word 'mpdb', i.e. 
\code{'/tmp/22987mpdb'}. If \code{protocol} is not specified a FIFO is used.

The default debug signal is \code{SIGUSR1}.
\end{funcdesc}

\subsection{Example}
If a program wishes to allow debugging from another process it must import
and call the \code{process_debugging} function from the \module{mpdb} module.
This function sets up a signal handler for \module{mpdb}'s debugging signal
(defaults to \code{USR1}).

\begin{verbatim}
# Import and initialise debugging from another process
import mpdb.process_debugging
mpdb.process_debugging()

# Regular program code
\end{verbatim}

From the debugger console a user must issue the 'attach' command, the
\code{target_addr} variable must be set \ref{command::set}. The user can
also select a signal to send to the process by calling the 
\code{set debug-signal} command, \ref{command::set}.

\begin{verbatim}
(MPdb) set target [protcol] [address]

(MPdb) attach PID [USR1]
\end{verbatim}

Protocol can be any of the protocols the 'pdbserver' and 'target' commands
can accept. Note however, that if the protcol is \code{tcp} you need not 
specify the hostname, as it can only be 'localhost'.


