Pyramid User Guide
##################
.. contents:: Table of Contents
Overview
========
The basic premise of pyramid is that a fragment of html can have a set
of slots; each slot having an identifier (or name). A mapping (or
dictionary) of data is associated with each fragment of html such that
each slot in the fragment has an associated entry in the dictionary. The
combination of html template and data mapping make up a 'fragment'.
Fragments may be more complicated than this, for instance having lists
of data items that fill a slot and associated patterns of html for each
item in the list. At it's most flexible, custom renderers may be
associated with tags in the html that can transform the html. However,
the typical user of the system will only use simple lists and mappings.
The system power comes from using other 'fragments' as the data
associated with a slot. This allows most general templating tasks to be
achieved without applying arbitrary restrictions on html structure. The
final aspect of the system that allows sites to be built around a common
templated structure it the ability for data items and templates to be
inherited from parent directories. In this way, a single html 'skin' can
be defined at the top of a directory structure; Each child fragment
will, if unable to find the specified template in it's own directory,
look in it's parent directory.
Introduction
============
A simple example template, which perhaps would be used on the news page
of a site for a list of news items, should demonstrate the basic
concepts. Here is the data structure ::
# file = news.yml
--- !fragment
template: news.html
local:
heading: The News
welcome-message: Welcome to the news
news: here is the news
and here is the html template::
# file = news.html
The !fragment in the data structure is a type identifier that tells the
yaml parser that a data structure is expected. In this case the fragment
data structure should have template, local and global keys.
The template key points to a filename for the html template to be used.
The local and global keys point to the two data structures to be used.
To work out the data structure to be used to the website, the local data
and global data are merged, with local data taking priority. (note for
later: if the item is in a sub-directory of a site, the parents global
data is also merged, this will be covered in a subsequent section).
The template will look for each fragments slot name in the keys of the
merged data item. The heading, welcome message and news will be taken
and inserted into the appropriate slots.
The result will be the following html::
The News
Welcome to the news
here is the news
If we wished to use a standalone template for the news item we would
repeat the initial html tempalte but change the 'news' key from a string
to a !fragment type::
# file = news.yml
--- !fragment
template: news.html
local:
heading: The News
welcome-message: Welcome to the news
newsitem: !fragment newsitem.yml
# file = news.html
The newsitem.yml fragment would then be used for the 'newsitem' key::
# fie = newsitem.yml
--- !fragment
template: newsitem.html
local:
title: First News Title
teaser: Here is my first item
# file = newsitem.html
This fragment is a single news item which itself is built in the same
way as described for the main template. The results of this are placed
into the 'newsitem' slot on the previous fragment.
Using the above technique we can create html fragments for the 'skin' of
a site and templates for the different types of content page.
In addition to the plain string data type (which does not need a !
prefix) other types of data may be used. One of the more useful types of
data for text documents is the !rest and !restfile data types. The !rest
data type will parse the data item as reStructuredText. The !restfile
will do the same but for an external file.
The Templating System
=====================
The template system used in pyramid is twisted web's nevow. The
documentation for this is available at but wil be described in brief
here.
Nevow has three primary types of directive. These are as follows
Slots
-----
A marker into which data is extracted using the name of the slot as a key
Data Items
----------
The data directive changes the current 'context' by drilling into the
currently used data. For example, if a dictionary:::
{'a':1.'b':2,'c': {'news':3} }
was supplied to a fragment. a data item with a name of 'c' would change
the current top level data to be a dictionary with just 'news' as a key.
e.g.::
would result in ::
1
2
3
Here is an example with an extra level::
{'a':1.'b':2,'c': {'news': {'title':4} } }
which with this html::
would generate::
1
2
4
Patterns
--------
A piece of html my be marked up as a pattern with an associated name.
Renderers can extract patterns for use as fragments of html. The typical
usage of the pattern is to mark a fragment of html as an item to be used
repeatedly when rendering a list. An example of this is shown below.
Renderers
---------
renderers pass the enclosed data back to a python function for
processing. This processing. The following example shows the 'sequence'
renderer which is one of nevow's built in renderers (the other common
renderer is a mapping renderer).
e.g. using the data::
{'a':1.'b':2,'c': [ {'title':11},{'title':12} ] }
which could be represented in yaml as::
# example.yml
--- !fragment
template: example.html
local:
a: 1
b: 2
c:
- title: 11
- title: 12
and the template::
# file = example.html
This html is generated::
1
2
Data Structure
==============
Although the pyramid idea is data agnostic, the only data reader being
used is yaml (coverted using the syck parser).
Yaml uses indentation to denote data structure. Two simple yaml
documents are shown below::
---
- item one
- item two
---
a: item one
b: item two
The first returns a list::
['item one', 'item two']
Type guessing is not used and, as such, all values are strings unless
explicitly cast.
The second returns a mapping::
{'a': 'item one', 'b': 'item two'}
All structures in pyramid are mappings. Mappings may contain any other
types. Examples of mappings containing lists of mappings is shown
below::
---
a: 1
b:
i: 10
ii: 20
iii: 30
c:
- 100
- 200
- 300
which would return::
{'a': '1',
'b': {'i': '10', 'ii': '20', 'iii':'30'},
'c': ['100', '200', '300']
}
In order to add a block of text (rather than inline) the following
syntaxes can be used)::
---
a: |
Anything indented by two
spaces is now considered a block
the preceding two spaces will
be removed
This line is kept
b: >
These
lines
are
collapsed
but a double new line is converted to a single newline
which will be converted to::
{
'a': 'Anything indented by two\nspaces is now considered a block\nthe preceding two spaces will\nbe removed\n\nThis line is kept\n',
'b': 'These lines are collapsed\nbut a double new line is converted to a single newline\n'
}
This is useful for blocks of text and also reStructuredText.
Apart from sequences and mappings, the following types are currently
used in the python site configuration.
fragment
--------
The fragment is the core type used. It is either a mapping with
template, local and global keys (which default to None, {} and {}
respectively); or it is a filename for a yml file which is itself a
fragment.
rest & restfile
---------------
The rest type allows inline rst to be used. Two examples follow::
---
data: !rest This is **bold**
would produce::
{'data': 'This is bold'}
and this::
---
data: | !rest
This is *emphasised* and
this is
**bold**
would produce::
{'data':'This is emphasised and\nthis is\nbold'}
The restfile type takes a filename for an argument and parses the
contents as reStructuredText.
url
---
The url type is used as a shortcut for a link and the link text::
---
utility:
- !url The Help Section /help
- !url Our Sitemap /sitemap
The algorithm behind !url does split on spaces and uses the last segment
as the url href and the remainder as the link text. This would produce::
{'utility':
[ 'The Help Section','Our Sitemap' ]
}
Using Pyramid
=============
Pyramid is currently configured to create the whole of a website in one
process. This can mean that it is quite processor intensive, taking over
a minute to build the hundred or so pages that have already been
created.
In the future, the process will be configured such that, when a page is
changed, it will only have to rebuild the sub-pages; and then only if
global data has changed.
The pyramid, if run without arguments, will show the following::
-d DATA, --data=DATA directory in which the fragment data is stored
-o OUT, --out=OUT directory in which to save output (will be emptied)
-r RESOURCES, --resources=RESOURCES
comma separated list of resource directories to copy
-v, --verbose print status messages to stdout
-R REBUILDDIRS, --rebuilddirs=REBUILDDIRS
only rebuild below these comma separated directories
-c CONSTANTS, --constants=CONSTANTS
pass in the names constants (e.g.
PDO=/root/pdo,PSF=/psf
for example, if you were in the test folder, the following would build
the simple test::
pyramid -d simple -o simple-out
You can now check the simple-out directory and can see the results.
It is more informative to run with the verbose flag set (possibly a
little too informative).
Viewing the Generated Pages
===========================
Because you will have installed twistd as part of the pre-requisites of
the software, there are a couple of simple commands that will allow you
to create a fully functional web server. Firstly you can use the mktap
command to generate a configuration file specifically for your usage::
mktap web --path=out
This will generate a file called web.tap that can be used with the
twistd command to run the server::
twistd -nof web.tap
The standard mktap command generates a configuration file for a
webserver running on port 8080. There is a man page for mktap that will
allow you to change the port used and some other variables if necessary.
Adding Pages to the Python Site - Full Description
==================================================
Adding new pages to the python site is fairly straightforward. For most
types of page, a simple rest document is all that is needed. This
section will talk through the addition of the 'new style classes' page
(currently in /doc/newstyle.html).
Adding initial files
--------------------
First thing is to add a link to the new page into the navigation. If we
open up the /doc/nav.yml file we can see the following::
--- !fragment
# Type of template to use
template: nav.html
# Contents of the template
global:
nav : !sectionnav |
Current docs folder
Beginner's Guide http://www.python.org/moin/BeginnersGuide
FAQs faq
Introductions intros
Other doc resources other
PEP Index http://python.org/peps
Non-English docs nonenglish
Book list books
Topic guides topics
New-style classes (aka descrintro) newstyle
We can see that the navigation has already been added. 'sectionnav' is a
custom data type that builds urls by adding the current directory to
each url. In this case the line 'FAQs faq' would be split on spaces and
all but the last element would be concetenated for the link text. The
last element would be appended onto the current directory (e.g. '/doc/'
+ 'faq') to create the link href'.
The newstyle section already as a link 'newstyle' so we don't need to
add this. You can also see that some links are absolute including the
domain name. The sectionnav datatype knows to leave these alone.
Now we have a link pointing to the newstyle section, we need to add a
folder to it (remembering that each folder in pyramid builds a single
file for the final website.)
Create a folder 'newstyle' under 'doc' and add the following files
index.yml
.........
the index.yml is the core filetype and is read first to work out how to
build the page. Most content pages will just include a basic index.yml
that defines the main template, a page title and where to find the
content::
--- !fragment
template: index.html
# The data to pass to the template
local:
title: Python Documentation Index
content: !fragment content.yml
We can either create a new index.yml or typically we can copy one from
another content page (typically most of these are very similar). In our
case we'll copy the index.yml from the /doc folder and change the title
attribute.
content.yml
...........
The content.yml file should be similar to the following::
--- !fragment
# Type of template to use
template: content.html
# The data to pass to the template
local:
content:
breadcrumb: !breadcrumb nav.yml nav
text: !restfile content.rst
This tells the system that we're using the content.html template and the
text content will be read from the content.rst file. Again we can
typically copy this file from another similar content page. Sometimes we
might want to add a sidebar to a file, if you look at the '/dev/doc' you
will see a content.yml similar to the following::
--- !fragment
# Type of template to use
template: content.html
# The data to pass to the template
local:
content:
breadcrumb: !breadcrumb nav.yml nav
text: !restfile content.rst
externallinks: !fragment externallinks.yml
This file includes the 'externallinks.yml' file which contains a list of
external links with an associated externallinks.html template. If you
want to create a sidebar of a similar nature, you can copy the same
structure to your new page.
content.html
............
A very simple template::
If you need a new template, please contact one of the pydotorg
webmasters or add a ticket to the trac at `http://psf.pollenation.net
`_.
Creating the ReStructuredText Content
-------------------------------------
Now we need to create a reStructuredText file (the ideal format for web
content). Our current newstyle.ht file is as follows::
Title: New-style Classes
Author: docs@python.org
New-style Classes
Unfortunately, new-style classes have not yet been integrated into
Python's standard documention. Fear not, however; many people have
worked to provide useful information on creating and using new-style
classes:
- Unifying types and classes
(aka descrintro) is Guido's essay on new-style classes and should be
your starting point.
- Raymond Hettinger's
How-To
Guide for Descriptors focuses on the single most useful aspect of
new-style classes (which includes properties).
- Python 2.3 Method Resolution Order
(MRO) covers multiple inheritance.
- Metaclasses will make your head explode. Here are several
approaches to discussing them:
- Types
and Objects is the start of a new-style class tutorial with lots
of figures and examples, but it's rough and not complete.
This is a fairly simple html document in rfc822 format. If we delete the
header, we can pass this through html2rst. The output of running this
through the modified html2rst parser is::
New-style Classes
=================
Unfortunately, new-style classes have not yet been integrated into Python's
standard documention. Fear not, however; many people have worked to provide
useful information on creating and using new-style classes:
- `Unifying types and classes`_ (aka descrintro) is Guido's essay on
new-style classes and should be your starting point.
- Raymond Hettinger's `How-To Guide for Descriptors`_ focuses on the
single most useful aspect of new-style classes (which includes
properties).
- `Python 2.3 Method Resolution Order`_ (MRO) covers multiple
inheritance.
- Metaclasses will make your head explode. Here are several approaches
to discussing them:
- David Mertz and Michele Simionato's `DeveloperWorks article`_
- Alex Martelli's `slideshow`_
- Mike Fletcher's `slideshow`_
- `Types and Objects`_ is the start of a new-style class tutorial with
lots of figures and examples, but it's rough and not complete.
.. _Unifying types and classes: ../2.2.3/descrintro.html
.. _How-To Guide for Descriptors:
http://users.rcn.com/python/download/Descriptor.htm
.. _Python 2.3 Method Resolution Order: ../2.3/mro.html
.. _DeveloperWorks article:
http://www-106.ibm.com/developerworks/linux/library/l-pymeta.html
.. _slideshow: http://www.strakt.com/docs/ep03_meta.pdf
.. _slideshow:
http://members.rogers.com/mcfletch/programming/metaclasses.pdf
.. _Types and Objects:
http://www.cafepy.com/articles/python_types_and_objects/index.html
This can now be saved as content.rst and the website rebuilt.
NB it may be worth mentioning that the previous example will cause
duplicate id problems in some builds of docutils. In the actual
conversion, the slideshow references have been changed to include the
authors name (e.g. Alex Martelli's slideshow)
Alternative Ways to Add Pages
=============================
Because the process of moving a lot of legacy content can be very
protracted and in order to make sure content can be synchronised from
the current python site to the beta site. A new directive has been
introduced which allows content of the current site to be directly
linked into the beta site. This new directive enhances the functionality
of the htfile directive already in use. I will first discuss the htfile
directive and then discuss how the expanded htfile directive can be used
on htfiles that are not in the build tree.
Using htfiles within pyramid
----------------------------
In order to include content in the htfile format that is used by
ht2html, two new directives were added 'htfile' and 'htfiledata'.
To show how these work, we will use the example above (new style
classes) but use a copy of the htfile instead of converting it to
reStructuredText. So the first thing we will have in our new page
directory will be the irc file renamed as 'content.ht'.
NB In many cases throughout the site we have tried to use the files name
'content. for the main body of content. This is not essential
however and is just a convention to make reading the directory easier.
index.yml
.........
Our new index.yml file will include the title directly from the htfile::
--- !fragment
template: index.html
# The data to pass to the template
local:
title: !htfiledata
file: content.ht
key: title
content: !fragment content.yml
content.yml
...........
This will now include the content directly from the htfile using the
!htfile directive::
--- !fragment
# Type of template to use
template: content.html
# The data to pass to the template
local:
content:
breadcrumb: !breadcrumb nav.yml nav
text: !htfile content.ht
These are the only changes needed.
Using htfiles sourced directly from the pydotorg tree
-----------------------------------------------------
Because the process of adding content can take such a long time, a new
htfile parser has been added that can use the pydotorg tree root to
reference files directory from the pydotorg current site. This will
ensure that the files remain uptodate until such time as it is decided
to move the content administration over to the new site.
NB: This may be useful at some point but only use it on the pydotorg
site if specifically arranged.
index.yml
.........
Just use the PDO string substitution and add the remaining path to the
ht file::
--- !fragment
template: index.html
# The data to pass to the template
local:
title: !htfiledata
file: %{PDO}s/community/irc.ht
key: title
content: !fragment content.yml
content.yml
...........
Adding the PDO variable to the htfile path ::
--- !fragment
# Type of template to use
template: content.html
# The data to pass to the template
local:
content:
breadcrumb: !breadcrumb nav.yml nav
text: !htfile %{PDO}s/community/irc.ht
The variable expansion needs providing with a location for the pydotorg
source tree. This is done using the pyramid argument -c ::
pyramid -d data -o out -c PDO=/trunk/pydotorg
This variable expansion currently only works for prefixes (i.e. you
can't use it in the middle of a path) but that is it's only intended use
case.
Creating htfile pages from the command line
-------------------------------------------
A command that generates the needed files for a remote htfile page is
included in the pyramid distribution. This command is called mkpydir and
the help file for it is as follows::
usage: mkpydir -t
[-d ]
-p
-o [-l]
examples
to create a page from a reStructuredText file assuming $PDO is the root of
the pydotorg source files directory (without a trailing slash)
mkpydir -t restfile -d $HOME/pydotorg/trunk -p /psf/weblog/pycon/intro.rst -o $BETAPYTHONDATA/weblog/pycon/intro
to create a page from a htfile file (copying the contents into the pyramid dir)
mkpydir -t htfile -d $HOME/pydotorg/trunk -p /Help.ht -o $HOME/beta.python.org/build/data/help
to link in some content from the pydotorg tree such that it always builds from the original
mkpydir -t htfile -l -d $PDO -p /Help.ht -o $HOME/beta.python.org/build/data/help
to include a wiki page
mkpydir -t wikiurl -d http://wiki.python.org/moin -p /Applications -o $HOME/beta.python.org/build/data/applications
to perform the above operation and create the page below the current directory
mkpydir -t wikiurl -d http://wiki.python.org/moin -p /Applications -o .
options:
-h, --help show this help message and exit
-t TYPE, --type=TYPE the type of pydir content to be created
-d DIR, --dir=DIR the root directory for the an asset (to be used when
copying pydotorg or wiki content)
-p PATH, --path=PATH the path to the asset
-o OUTPUTDIR, --outputdir=OUTPUTDIR
directory in which to create the pyramid directory
-T TITLE, --title=TITLE
the page title
-l, --link use a link to an existing ht file
-c, --copy copy the htfile into the directory
For example, if I was in the community directory of my beta.python.org
build folder and I wanted to create the a page from the
$PDO/community/irc.ht file I would use::
mkpydir -t htfile -d $PDO -p /community/irc.ht -o irc
this would create a directory called irc with an index.yml (which would
use the htfile for the page title) and a content.yml (which would use
the htfile for the contents). It would also include the basic
content.html template.
The -l option creates data files that pull in content from an original
pydotorg source tree. This should only be used on the python site if
specifically arranged.
Getting content out of wiki pages
---------------------------------
A lot of content on the site is managed through the python wiki. This
makes managing the content a lot easier for the community. Content can
be pulled into tyhe site directly over the web using the wikiurl
directive. The example below pulls in content from the the wiki
Applications page ::
--- !fragment
# Type of template to use
template: content.html
# The data to pass to the template
local:
content:
breadcrumb: !breadcrumb nav.yml nav
text: !wikiurl http://wiki.python.org/moin/Applications
Pyramid and the Python Site
===========================
FAQ's
-----
How do I include Images and other static assets?
................................................
Static directories are pulled into the final site which can be used for
non content assets. At the moment there is a /files directory which is
being used (this appears in the root of the build directory parralel
with the data directory).
A lot of links seem to be wrong?
................................
Once the url rewriting is added, links can be mapped in the same way
that the 301 redirects will be mapped. When adding content, if possible
try to use absolute urls. These will be easier to parse and rewrite if
content is moved around.