HOWTO: Compiling Python Modules with MPW

This HOWTO is a slightly reformatted version of an original by Corran Webster, whose Python page may contain a more up-to-date version.

The Macintosh version of the Python programming language is usually compiled with Metrowerks CodeWarrior. As a result, C extension modules are also usually compiled with CodeWarrior, and the documentation and sample code reflects this. CodeWarrior is a commercial product, and may be beyond the budgets of hobbyist hackers, making them dependent on others to compile C extension modules. At the present time, many standard C extension modules compile "out of the box" on the Macintosh, but in only a few cases is the plugin for the Macintosh included in the distribution.

The Macintosh Programmer's Workshop (MPW) is Apple's development environment, and is freely available for download from Apple, as well as on their Developer CDs. Since Python was originally developed using MPW, before CodeWarrior became the dominant MacOS development environment, most of the idiosyncrasies of MPW are already supported, and compilation of C extension modules in MPW is possible.

This HOWTO only deals with compiling for PowerPC Macintoshes. The process should be similar for 68k Macintoshes using the code fragment manager, but I have not attempted this - my old Mac is running NetBSD.

This way of compiling modules is still experimental. Please read the caveats section below.

Setting Up MPW for Compiling Python Modules

This assumes that you have successfully installed both MPW and Python with the Developer's Kit on your Macintosh.

The first step is to let MPW know where you keep Python. This step is not strictly necessary, but will make development easier and improve portability. Create a new file in the Startup Items folder of MPW called Python. Type the lines:

set Python "Macintosh HD:Applications:Python 1.5.2c1:"
set PythonIncludes "{Python}Include"
set PythonMacIncludes "{Python}Mac:Include"
set PythonCore "{Python}PythonCore"

export Python PythonIncludes PythonMacIncludes PythonCore

where Macintosh HD:Applications:Python 1.5.2c1: is replaced by the path to the directory where you keep your copy of Python, and the other variables reflect where you keep your header files and Python core files. The locations here are the standard for Python 1.5.2c1, but they are different for Python 1.52b2 and earlier (most notably, the PythonCore is kept in the Extensions folder).

Next, you need to update the config.h file for the MrC compiler included with MPW. This header file is located in the :Mac:Include folder in the standard distribution. You can update it by hand, by adding the lines:

#ifdef __MRC__
#define BAD_STATIC_FORWARD
#endif

at the after the similar defines for __MWERKS__ and __SC__ in the file. This step is critical: many modules, including ones in the standard distribution, will not compile properly without this modification (see common problems below).

Copies of both the Python startup item for MPW and the config.h are included here for your convenience.

If you are porting Unix modules to the mac, you may find it useful to install GUSI for your copy of MPW. GUSI provides some amount of POSIX compatibility, and is used by Python itself for this purpose - at the very least having it's header files available may be useful. Also of note for people porting Unix modules, the most recent alpha version (4.1a8) of MrC and MrCpp at this writing permits using unix-style pathnames for includes via the -includes unix command line option. I have not experimented heavily with this, but will be doing so in the future and report my findings.

You now have MPW and Python set up to allow compilation of modules.

Compiling a Module

This assumes that you have a C extension module ready to compile. For instructions on how to write a module, see the Python documentation.

There are three approaches you can take to compiling in MPW: using the command line interface, using the MPW CreateMake command (available as the "Create build commands..." menu item, and writing a Makefile by hand.

Before you start any of these, you'll need to know:

The names and locations of the C source files. In the examples, this is the file xxmodule.c, and is in MPW's current working directory.
The name that Python expects to import your module under. In the examples, this is xx, so the shared library file will be called xx.ppc.slb.
The location of any additional header files use by the C source. The example does not use any additional header files.
The location of any additional shared libraries which the module needs to link to. The example does not link to any other shared libraries.
The name of the entry point to your module. This is usually the last function in the main C source file, and the name usually starts with init. In the examples, this is initxx.

Using the Command Line

For simple modules consisting of one or two C files, it's often convenient to simply use commands in a MPW Worksheet. Usually you will want to set MPW's working directory to the directory containing the C source code. The following commands compile and link the standard Python test module xxmodule.c:

MrC "xxmodule.c" -o "xx.c.x" -w off  -d HAVE_CONFIG_H ∂
        -i "{PythonMacIncludes}" ∂
        -i "{PythonIncludes}" 
PPCLink ∂
        -o "xx.ppc.slb"  ∂
        "xx.c.x" ∂
        -t 'shlb' ∂
        -c 'Pyth' ∂
        -xm s ∂
        -d ∂
        "{PythonCore}" ∂
        "{SharedLibraries}InterfaceLib" ∂
        "{SharedLibraries}MathLib" ∂
        "{SharedLibraries}StdCLib" ∂
        "{PPCLibraries}StdCRuntime.o" ∂
        "{PPCLibraries}PPCCRuntime.o" ∂
        "{PPCLibraries}PPCToolLibs.o" ∂
        -export initxx

(Note: The last character on each line should appear as "partial derivative" symbol, which you type as option-d and which is MPW's line continuation symbol.)

Any additional header files should be specified by adding their directories as extra -i options to the MrC command. Any additional shared libraries should be added before the PythonCore library in the PPCLink command.

If there is more than one source file, you will need to duplicate the compile command for each source file, and you will need to include all the object files in the place where "xx.c.x" appears in the PPCLink command.

Using CreateMake

For more complex modules, or modules that you are writing yourself, you will probably want to use a makefile. Unfortunately MPW's makefiles are incompatible with the standard Unix makefiles, so you will not be able to use any makefiles which come with a C module.

Usually, you will want the makefile to reside in the same directory as the C source code, so you should set MPW's working directory to that directory before proceeding.

To create a makefile for the standard Python test module xxmodule.c:

Select "Create build commands..." from the "Build" Menu.
Type xx.ppc.slb for the Program Name.
Select "Shared Library" for the Program Type.
Select "PowerPC Only" for the Target.
Click on the "Source Files..." button, and add your module's C source files to the list.
Click on the "Other Options..." button and change the creator type to "Pyth". If you are using additional header files, you can also add their directories at this stage. Click on "Continue" once you have done this.
Click on the "Exported Symbols..." button and type initxx into the entry field. Click on "Continue" once you have done this.
At this stage, your CreateMake window should look like this:
Click on the "CreateMake" button.

You will now need to edit the makefile that was just created. Open the file "xx.ppc.slb.make" in the current directory and make the following changes:

Change the line
```
Includes     =
```
to read
```
Includes     = -i "{PythonIncludes}" -i "{PythonMacIncludes}"
```
If you have any additional headers than need to be included, you can add them here as well.

Change the line

PPCCOptions  = {Includes} {Sym•PPC}

to read

PPCCOptions  = -w off  -d HAVE_CONFIG_H {Includes} {Sym•PPC}

After the line
```
                -xm s ∂
```
add
```
                -d ∂
                "{PythonCore}" ∂
```
If you have any other shared libraries you need to link to, add each on a line before PythonCore, terminating each line with a ∂.

Save the file. You are now ready to build.

Go to the "Build" or "Full Build" menu items, type in xx.ppc.slb, and MPW should take things from there. Any time you need to rebuild the shared library, you can simply do another "Build" or "Full Build".

Writing a Makefile by Hand

For modules which have complex interdependencies between files, you will likely need a more sophisticated makefile than the one created by CreateMake. You will need to be familiar with the MPW makefile format, but you can get a start by either using CreateMake to get a simple starting point, or taking another MPW makefile as a starting point.

It is beyond the scope of this HOWTO to go into the generalities of MPW makefiles. Documentation on MPW's Make command can be found with the MPW distribution, in particular the documents Building and Maintaining Programs with MPW (2nd Edition) and the MPW Command Reference.

There are a couple of important points to keep in mind when writing a makefile by hand:

When there are multiple symbols with the same name in object files or shared libraries, PPCLink used the symbol from the file which appears first in arguments of the PPCLink command. For this reason, you will usually want the PythonCore and any other shared libraries which are not part of the standard MPW runtime environment to appear before the standard runtime libraries. This is particularly the case with StdCLib. The "-d" option turns off the (often copious) warnings about multiply defined symbols.
You will want to make sure that the HAVE_CONFIG_H preprocessor symbol is defined for most C source files using the -d HAVE_CONFIG_H option to MrC.

The file xx.ppc.slb.make is included here for you to use as a starting point.

Using the Extension Module

Once you have compiled your extension module, you will need to let Python know where it is. You can either move it into a place on Python's search path - such as the :Mac:Plugins folder - or modify the path to include the location of your new module using the EditPythonPrefs applet.

Your work may not be completely done, as many extension modules have a Python wrapper around them. If the Python was not written with portability in mind, you may need to do some more work to get that up and running. Indeed, if the Python part uses OS-specific features, like pipes, you may have to completely rewrite it if you can make it work at all.

Common Problems

There are a couple of common problems which occur when porting a module from another platform. Fortunately, they are often easy to fix.

Static Forward Definitions

If you get a compiler error which looks something like:

File "xxmodule.c"; line 135 #Error: 'Xxo_Type' is already defined

then most likely either you have not set up config.h correctly to handle static forward definitions, or the module author has not adhered to the standard python conventions. If the second is the case, find where the variable is first defined, and replace the static with staticforward. Then find the second place it is defined (usually the line where the compiler complained) and replace static with statichere.

If you have set up things correctly, you should now be able to compile.

Automatic Type Conversion

MrC seems to be a little pickier about automatically converting from one type to another than some other C compilers. These can often be fixed by simply adding an explicit cast to the desired type.

XXX There may be a compiler option which relaxes this. That would be a better solution.

Caveats

As Jack Jansen pointed out on the Mac Python mailing list, there could potentially be conflicts between the MetroWerks C runtime which the Python core and standard modules was compiled with, and the MPW C runtime which your extension module is compiled with. While things seem to work fine in everyday use, it is possible that there are bugs which have not been discovered yet. Most likely these world take the form of standard C functions (most likely I/O functions due to conflicts between the SIOUX libraries and the SIOW libraries) not working as they are supposed to, or memory leaks caused by improper malloc/free.

Some such problems have been demonstrated by compiling modules with PythonCore linked after StdCLib - printf does not work properly in this setup, and I suspect that there will also be malloc/free problems in situations where the module allocates memory which is later disposed of by Python, or vice-versa. Compiling with PythonCore taking precedence over StdCLib seems to give the correct behaviour.

This method of compiling should be considered experimental for the time being. Use it at your own risk.

If you notice any quirks in modules compiled this way, or have insight into what may go wrong or right with this situation, please contact me so that I can add it to the HOWTO.

The ideal solution to this problem would be to get Python to compile using MPW (and a Python MPW Tool would be very neat indeed). However, that does seem to be a major project.