This text should be reworked. Just modified for cvs test. Introduction to the ideas of the new Stackless Python Draft, CT 20020122 Update 20020524: Software solution abandoned! Please review! Stackless Python now tries to follow some different goals: - the implementation should be as understandable as possible. (this part is slightly obscure due to my nature, but he is still eager to learn) - the users requests should be followed first. Especially, coroutines and microthreads are of larger interest. Generators are less interesting, since Python 2.2 includes a marveless, stack based implementation which is hard to improve. The abilities of continuations have been experienced by only a very small group of insiders, and they agreed that these are better not made available fo the public. Just too powerful. - my own porting efforts should be minimized. I am now trying to reduce the Stackless patches to their bare minimum. Partially, I avoid constructs which can be of value, but would require massive code changes. Partially, I'm now no longer reluctant to use techniques which are an absolute no-no, if I would again try to get into the distribution. My current strategy is to minimize the patches in a way that makes inclusion irrelevant. I.e. try to make very few changes, and keep the distro's and Stackless's intersection to the minimum. ***************************************************************************** A Bit Of History. I have spent about three years developing Stackless Python in several versions. This was very hard work, and it didn't pay off in the way I intended. Especially, Stackless Python was never understood good enough to be incorporated into the Standard Python distribution. Instead of a warm welcome and instructions how to make the sources suitable for inclusion, I got rejections and dissings, especially from Guido. At that time I was really disappointed, and it took some time to understand that his reaction was absolutely correct. The code was simply too tricky and complicated, and the implementation effort necessary toget complete stacklessness was high. Also, I was too much bound to the incredible concept of continuations, which finally killed my success. Today, I have a different point of view: Stackless is not ripe for Python, and Python is not ripe for Stackless. Both parties have their position. Stackless has a couple of users which have an absolute need for its feature meanwhile, and my work also continuously gets some sponsorship from game developers. In April, I was also hired by IronPort to work on Stackless, which is great. ***************************************************************************** The New Concept! After an unsuccessful port to Python 2.1, my first desperate attempt to break out was a complete redesign, based upon assembly callbacks only. Fortunately, my brain recovers quickly from such events. Years of thinking cannot be replaced by quick thoughts, and well-thought concepts are immutable against quick minds. Nevertheless, an eternal problem was the impossibility to make Python really completely stackless. This impossibility was produced by myself, since I still believed to be incorporated. This had the implication of never ever using any platform specific stuff like assembly or compiler dependant constructs. Having to do everything by highly complicated rewrites of standard concepts was the real show stoppers. The lack of support for third party extension modules did the rest. After all, these problems have vanished. It turned out that people are happy if they get their Stackless, and they don't care how we achieve this. On the other hand, doing platform specific stuff is hard, and I cannot support all of it. Of course there are helpers. but what is the general strategy? The answer is: ============== We do no longer support a native, compatible Stackless implementation but a hardware/platform dependant version only. This gives me the most effect with minimum maintenance work. My basic concept is to have a single place on the C stack where all Python calls end up with. This place is state-less. It does not keep state on the C stack. Given that, it is possible to switch between frames which match this condition. You may call these coroutines or microthreads, but it doesn't matter. We are dealing with frame chains which can be switched against their Python state structure. How was old Stackless stateless? Well, it tried it's best. It was tried to turn every recursive call to the nterpreter into tail recursion. That means: Try to avoid a recursive interpreter call. Prepare everything for its call, create all the parameters, but then do not recursively run the frame, but try to postpone this action, letting a toplevel interpeter act on it. This concept worked very well, despite Python methods. The latter either have post conditions which have to be met, or they follow a difficult protocol in object.c or abstract.c, which makes it nearly impossible to turn them into tail position, without a complete rewrite of the above modules, including all non-trivial object impementatiions. Conclusion: Old Stackless was good but incomplete. How is the new Stackless stateless? Well, in principle, it is the same. Just the technique used is a bit different. I do not rewrite tons of recursive code into something tail-recursive. Instead, I wrap up all the stack data that makes up a frame's C impact, put it into a structure and stick it into a memory area in the frame. After this, the C stack is cleared from the frame specific stuff. The frame contains a procedure which is able to restore this frame state on demand. The interested reader will have noticed, that this concept covers all of the former Stackless approach, but also much more: This concept always works, not only with Python functions, but also with all methods, and eve with C extensions. Whatever binary calls are triggered by a frame, regardless what kind of C callback it may be, everything can be considered as the activation part of a Python frame. When the frame is about to be left, this data has to be saved. When the frame resumes, this data has to be restored before. The key idea is to wrap every interpreter recursion's C stack frame into some structure. This structure is saved with the frame, when a new frame is executed. The f_execute field is changed to a function which re-establishes the stack environment which the function needs for executing.. to be continued...