Creating a C extension module on the Macintosh

This document gives a step-by-step example of how to create a new C extension module on the mac. For this example, we will create a module to interface to the programmers' API of InterSLIP, a package that allows you to use MacTCP (and, hence, all internet services) over a modem connection.

Prerequisites

There are a few things you need to pull this off. First and foremost, you need a C development environment. Actually, you need a specific development environment, CodeWarrior by MetroWerks. You will probably need the latest version. You may be able to get by with an older version of CodeWarrior or with another development environment (Up to about 1994 python was developed with THINK C, and in the dim past it was compiled with MPW C) assuming you have managed to get Python to compile under your development environment, but the step-by-step character of this document will be lost.

Next, you need a python source distribution. For PowerPC and cfm68k development you can actually get by without a full source distribution, using the Development distribution. You'll also need a functional python interpreter, and the Modulator program (which lives in Tools:Modulator in the standard source distribution). You may also find that Guido's Extending and embedding the Python interpreter is a very handy piece of documentation. I will skip lots of details that are handled there, like complete descriptions of Py_ParseTuple and such utility routines, or the general structure of extension modules.

InterSLIP and the C API to it

InterSLIP, the utility to which we are going to create a python interface, is a system extension that does all the work of connecting to the internet over a modem connection. InterSLIP is provided free-of-charge by InterCon. First it connects to your modem, then it goes through the whole process of dialling, logging in and possibly starting the SLIP software on the remote computer and finally it starts with the real work: packing up IP packets handed to it by MacTCP and sending them to the remote side (and, of course, the reverse action of receiving incoming packets, unpacking them and handing them to MacTCP). InterSLIP is a device driver, and you control it using a application supplied with it, InterSLIP Setup. The API that InterSLIP Setup uses to talk to the device driver is published in the documentation and, hence, also useable by other applications.

I happened to have a C interface to the API, which is all ugly low-level device-driver calls by itself. The C interface is in InterslipLib.c and InterslipLib.h, we'll concentrate here on how to build the Python wrapper module around it. Note that this is the "normal" situation when you are writing a Python extension module: you have some sort of functionality available to C programmers and want to make a Python interface to it.

Using Modulator

The method we describe in this document, using Modulator, is the best method for small interfaces. For large interfaces there is another tool, Bgen, which actually generates the complete module without you lifting a single finger. Bgen, however, has the disadvantage of having a very steep learning curve, so an example using it will have to wait until another document, when I have more time.

First, let us look at the InterslipLib.h header file, and see that the whole interface consists of six routines: is_open, is_connect, is_disconnect, is_status, is_getconfig and is_setconfig. Our first step will be to create a skeleton file @interslipmodule.c, a dummy module that will contain all the glue code that python expects of an extension module. Creating this glue code is a breeze with modulator, a tool that we only have to tell that we want to create a module with methods of the six names above and that will create the complete skeleton C code for us.

Why call this dummy module @interslipmodule.c and not interslipmodule.c? Self-preservation: if ever you happen to repeat the whole process after you have actually turned the skeleton module into a real module you would overwrite your hand-written code. By calling the dummy module a different name you have to make two mistakes in a row before you do this.

If you installed Tk support when you installed Python this is extremely simple. You start modulator and are provided with a form in which you fill out the details of the module you are creating.

You'll need to supply a module name (interslip, in our case), a module abbreviation (pyis, which is used as a prefix to all the routines and data structures modulator will create for you) and you enter the names of all the methods your module will export (the list above, with is_ stripped off). Note that we use pyis as the prefix instead of the more logical is, since the latter would cause our routine names to collide with those in the API we are interfacing to! The method names are the names as seen by the python program, and the C routine names will have the prefix and an underscore prepended. Modulator can do much more, like generating code for objects and such, but that is a topic for a later example.

Once you have told modulator all about the module you want to create you press "check", which checks that you haven't omitted any information and "Generate code". This will prompt you for a C output file and generate your module for you.

Using Modulator without Tk

Modulator actually uses a two-stage process to create your code: first the information you provided is turned into a number of python statements and then these statements are executed to generate your code. This is done so that you can even use modulator if you don't have Tk support in Python: you'll just have to write the modulator python statements by hand (about 10 lines, in our example) and modulator will generate the C code (about 150 lines, in our example). Here is the Python code you'll want to execute to generate our skeleton module: 

	import addpack
	addpack.addpack('Tools')
	addpack.addpack('modulator')
	import genmodule

	m = genmodule.module()
	m.name = 'interslip'
	m.abbrev = 'pyis'
	m.methodlist = ['open', 'connect', 'disconnect', 'status', \
		'getconfig', 'setconfig']
	m.objects = []

	fp = open('@interslipmodule.c', 'w')
	genmodule.write(fp, m)
Drop this program on the python interpreter and out will come your skeleton module.

Now, rename the file to interslipmodule.c and you're all set to start developing. The module is complete in the sense that it should compile, and that if you import it in a python program you will see all the methods. It is, of course, not yet complete in a functional way...

Adding a module to Classic 68K Python

What you do now depends on whether you're developing for PowerPC (or for CFM68K) or for "traditional" mac. For a traditional 68K Python, you will have to add your new module to the project file of the Python interpreter, and you have to edit "config.c" to add the module to the set of builtin modules. In config.c you will add the module at two places: near the start of the file there is a list of external declarations for all init() routines. Add a line of the form 
		extern void initinterslip();
here. Further down the file there is an array that is initialized with modulename/initfunction pairs. Add a line of the form 
		{"interslip",	initinterslip},
here. You may want to bracket these two lines with 
	#ifdef USE_INTERSLIP
	#endif
lines, that way you can easily control whether the module is incorporated into python at compile time. If you decide to do the latter edit your config file (you can find the name in the "C/C++ language" section of the MW preferences dialog, it will probably be "mwerks_nonshared_config.h") and add a 
	#define USE_INTERSLIP
Make the new interpreter and check that you can import the module, see the methods (with "dir(interslip)") and call them.

Creating a plugin module

For PowerPC or cfm68k development you could follow the same path, but it is actually a better idea to use a dynamically loadable module. The advantage of dynamically loadable modules is that they are not loaded until a python program actually uses them (resulting in less memory usage by the interpreter) and that development is a lot simpler (since your projects will all be smaller). Moreover, you can distribute a plugin module by itself without haveing to distribute a complete python interpreter.

Go to the "PlugIns" folder and copy the files xx.prj, and xx.prj.exp to interslipmodule.prj and interslipmodule.prj.exp, respectively. Edit interslipmodule.prj.exp and change the name of the exported routine "initxx" to "initinterslip". Open interslipmodule.prj with CodeWarrior, remove the file xxmodule.c and add interslipmodule.c and make a number of adjustments to the preferences:

Next, compile and link your module, fire up python and do the same tests as for 68K python.

Getting the module to do real work

So far, so good. In half an hour or so we have created a complete new extension module for Python. The downside, however, is that the module does not do anything useful. So, in the next half hour we will turn our beautiful skeleton module into something that is at least as beautiful but also gets some serious work done. For this once, I have spent that half hour for you, and you can see the results in interslipmodule.c.

We add 

	#include "InterslipLib.h"
	#include "macglue.h"
to the top of the file, and work our way through each of the methods to add the functionality needed. Starting with open, we fill in the template docstring, the value accessible from Python by looking at interslip.open.__doc__. There are not many tools using this information at the moment, but as soon as class browsers for python become available having this minimal documentation available is a good idea. We put "Load the interslip driver" as the comment here.

Next, we tackle the body of pyis_open(). Since it has no arguments and no return value we don't need to mess with that, we just have to add a call to is_open() and check the return for an error code, in which case we raise an error: 

	err = is_open();
	if ( err ) {
		PyErr_Mac(ErrorObject, err);
		return NULL;
	}
The routine PyErr_Mac() is a useful routine that raises the exception passed as its first argument. The data passed with the exception is based on the standard MacOS error code given, and PyErr_Mac() attempts to locate a textual description of the error code (which sure beats the "error -14021" messages that so many macintosh applications tell their poor users).

We will skip pyis_connect and pyis_disconnect here, which are pretty much identical to pyis_open: no arguments, no return value, just a call and an error check. With pyis_status() things get interesting again: this call still takes 3 arguments, and all happen to be values returned (a numeric connection status indicator, a message sequence number and a pointer to the message itself, in MacOS pascal-style string form). We declare variables to receive the returned values, do the call, check the error and format the return value.

Building the return value is done using Py_BuildValue: 

	return Py_BuildValue("iiO&", (int)status, (int)seqnum, PyMac_BuildStr255, message);
Py_BuildValue() is a very handy routine that builds tuples according to a format string, somewhat similar to the way printf() works. The format string specifies the arguments expected after the string, and turns them from C objects into python objects. The resulting objects are put in a python tuple object and returned. The "i" format specifier signifies an "int" (hence the cast: status and seqnum are declared as "long", which is what the is_status() routine wants, and even though we use a 4-byte project there is really no reason not to put the cast here). Py_BuildValue and its counterpart Py_ParseTuple have format codes for all the common C types like ints, shorts, C-strings, floats, etc. Also, there is a nifty escape mechanism to format values about which is does not know. This is invoked by the "O&" format: it expects two arguments, a routine pointer and an int-sized data object. The routine is called with the object as a parameter and it should return a python objects representing the data. Macglue.h declares a number of such formatting routines for common MacOS objects like Str255, FSSpec, OSType, Rect, etc. See the comments in the include file for details.

Pyis_getconfig() is again similar to pyis_getstatus, only two minor points are worth noting here. First, the C API return the input and output baudrate squashed together into a single 4-byte long. We separate them out before returning the result to python. Second, whereas the status call returned us a pointer to a Str255 it kept we are responsible for allocating the Str255 for getconfig. This is something that would have been easy to get wrong had we not used prototypes everywhere. Morale: always try to include the header files for interfaces to libraries and other stuff, so that the compiler can catch any mistakes you make.

Pyis_setconfig() finally shows off Py_ParseTuple, the companion function to Py_BuildValue. You pass it the argument tuple "args" that your method gets as its second argument, a format string and pointers to where you want the arguments stored. Again, standard C types such as strings and integers Py_ParseTuple knows all about and through the "O&" format you can extend the functionality. For each "O&" you pass a function pointer and a pointer to a data area. The function will be called with a PyObject pointer and your data pointer and it should convert the python object to the correct C type. It should return 1 on success and 0 on failure. Again, a number of converters for standard MacOS types are provided, and declared in macglue.h.

Next in our source file comes the method table for our module, which has been generated by modulator (and it did a good job too!), but which is worth looking at for a moment. Entries are of the form 

	{"open",	pyis_open,	1,	pyis_open__doc__},
where the entries are python method name, C routine pointer, flags and docstring pointer. The value to note is the 1 for the flags: this signifies that you want to use "new-style" Py_ParseTuple behaviour. If you are writing a new module always use this, but if you are modifying old code which calls something like getargs(args, "(ii)", ...) you will have to put zero here. See "extending and embedding" or possibly the getargs.c source file for details if you need them.

Finally, we add some code to the init module, to put some symbolic constants (codes that can by returned by the status method) in the module dictionary, so the python program can use "interslip.RUN" instead of the cryptic "4" when it wants to check that the interslip driver is in RUN state. Modulator has already generated code to get at the module dictionary using PyModule_GetDict() to store the exception object, so we simply call 

	PyDict_SetItemString(d, "IDLE", PyInt_FromLong(IS_IDLE));
for each of our items. Since the last bit of code in our init routine checks for previous errors with PyErr_Occurred() and since PyDict_SetItemString() gracefully handles the case of NULL parameters (if PyInt_FromLong() failed, for instance) we don't have to do error checking here. In some other cases you may have to do error checking yourself.

This concludes our crash-course on writing Python extensions in C on the Macintosh. If you are not done reading yet I suggest you look back at the MacPython Crashcourse index to find another topic to study.