PEP: 3101 Title: Advanced String Formatting Version: $Revision$ Last-Modified: $Date$ Author: Talin Status: Draft Type: Standards Content-Type: text/plain Created: 16-Apr-2006 Python-Version: 3.0 Post-History: 28-Apr-2006, 6-May-2006, 10-Jun-2006 Abstract This PEP proposes a new system for built-in string formatting operations, intended as a replacement for the existing '%' string formatting operator. The C engine which supports this proposal may also be accessible from the string module to allow for additional formatting options, and may be compilable as a compatible separate extension module for use with older versions of Python. Rationale Python currently provides two methods of string interpolation: - The '%' operator for strings. [1] - The string.Template module. [2] The primary scope of this PEP concerns proposals for built-in string formatting operations (in other words, methods of the built-in string type). The '%' operator is primarily limited by the fact that it is a binary operator, and therefore can take at most two arguments. One of those arguments is already dedicated to the format string, leaving all other variables to be squeezed into the remaining argument. The current practice is to use either a dictionary or a tuple as the second argument, but as many people have commented [3], this lacks flexibility. The "all or nothing" approach (meaning that one must choose between only positional arguments, or only named arguments) is felt to be overly constraining. While there is some overlap between this proposal and string.Template, it is felt that each serves a distinct need, and that one does not obviate the other. This proposal is for a mechanism which, like '%', is efficient for small strings which are only used once, so, for example, compilation of a string into a template is not contemplated in this proposal, although the proposal does take care to define format strings and the API in such a way that an efficient template package could reuse the syntax and even some of the underlying formatting code. Specification The specification will consist of the following parts: - Specification of a new formatting method to be added to the built-in string class. - Specification of functions and flag values to be added to the string module, so that the underlying formatting engine can be used with additional options. - Specification of the module name to be used for backporting to earlier version of Python than 2.6. - Specification of a new syntax for format strings. - Specification of a new set of class methods to control the formatting and conversion of objects. - Specification of an API for user-defined formatting classes. - Specification of how formatting errors are handled. Note on string encodings: When discussing this PEP in the context of Python 3.0, it is assumed that all strings are unicode strings, and that the use of the word 'string' in the context of this document will generally refer to a Python 3.0 string, which is the same as Python 2.x unicode object. In the context of Python 2.x, the use of the word 'string' in the context of this document refers to an object which may either be a regular string or a unicode object. All of the function call interfaces described in this PEP can be used for both strings and unicode objects, and in all cases there is sufficient information to be able to properly deduce the output string type (in other words, there is no need for two separate APIs). In all cases, the type of the format string dominates - that is, the result of the conversion will always result in an object that contains the same representation of characters as the input format string. String Methods (Python 2.6 and above) The built-in string class (and also the unicode class in 2.6) will gain a new method, 'format', which takes an arbitrary number of positional and keyword arguments: "The story of {0}, {1}, and {c}".format(a, b, c=d) Within a format string, each positional argument is identified with a number, starting from zero, so in the above example, 'a' is argument 0 and 'b' is argument 1. Each keyword argument is identified by its keyword name, so in the above example, 'c' is used to refer to the third argument. Two keyword argument names are reserved for use by the format method: The 'namespace' argument, if present, should be bound to a dictionary or tuple of dictionaries, which will be searched whenevar a keyword argument is not found in the **kwargs passed to format(). The 'field_hook' argument, if present, defines a "hook" callback function which allows greater user control over the formatting operation. Automatic namespace generation: If format() is called with no parameters, a namespace will automatically be generated which contains the caller's locals() and globals(). string module additions (Python 2.6 and above) The string module will have two new functions: format(format_string, *args, **kwargs) The format function is identical in operation to the string/unicode format method. In other words, the output from: str.format(format_string, *args, **kwargs) and: string.format(format_string, *args, **kwargs) will be indentical. flag_format(flags, format_string, *args, **kwargs) The flag_format function adds an integer flags argument in front of the parameters which are passed to the format method. If this flags parameter is 0, the results of calling flag_format will be identical to the results of calling format with the same string and *args and **kwargs. Setting bits in the flags parameter allows finer control over the formatter operation. The string module will have some new variables which are bound to integer constants. These define options which can be passed to flag_format in the flags parameter. Most options can be combined by using the bitwise or (|) operator. FORMAT_SYNTAX_BRACES_BALANCED FORMAT_SYNTAX_BRACES_UNBALANCED FORMAT_SYNTAX_DOLLAR FORMAT_SYNTAX_BRACES_WHITESPACE FORMAT_SYNTAX_SINGLE_FIELD These define the initial syntax mode of the string. At most one of these can be ORed into the flags parameter. The default is FORMAT_SYNTAX_BRACES_BALANCED. The syntax modes are discussed later in this PEP. FORMAT_HOOK_ALWAYS By default, the hook function is only called on fields with a type specifier of 'h'. If this option is set, the hook function will be called on every single field. (It can defer processing to the internal field formatter by returning None instead of a result string.) FORMAT_ALLOW_LEADING_UNDERSCORES By default, leading underscores are not allowed in identifier lookups (getattr or getitem). Setting this flag will allow this. FORMAT_USE_ALL_ARGUMENTS By default, the formatting engine does not check that all arguments passed to it are used by the format string. If this flag is set, an exception will be raised if some arguments are unused. Backporting to earlier versions of Python An extension module will be maintained which is usable with at least Python 2.3, 2.4, and 2.5. The name of this extension module will be "string_format", and it will contain exactly the same functions and flag identifiers as are being added to the string module in 2.6 and 3.0. This will allow new code which requires this functionality to import it as follows: try: from string import format except ImportError: from string_format import format Format Strings Format strings consist of intermingled character data and markup. Character data is data which is transferred unchanged from the format string to the output string; markup is not transferred from the format string directly to the output, but instead consists of control information which describes to the format engine what should be placed in the output string in the place of the markup. The transition from character data to markup and back is controlled by the syntax mode currently in effect. The only markup currently defined in this PEP is the markup for a replacement field, and the markup for a syntax mode change. Syntax Modes The parsing of the transition from character data to markup is simple enough that the parser can easily handle a few different syntax modes with very little loss of efficiency, so this proposal provides for a few different syntax modes. The rationale for modes chosen or not is provided later in the "Alternate syntax" section of this PEP. The following syntax modes are supported: Mode 0 uses an open brace to transition from character data to markup, and a close brace to transition back to character data. Literal braces (which should appear in the output text unchanged) are created by placing two open braces or two close braces in the text. The constant for this mode in the string module is FORMAT_SYNTAX_BRACES_BALANCED. Mode 1 is identical to mode 0, but only requires literal open braces to be escaped, not literal close braces. The constant for this mode in the string module is FORMAT_SYNTAX_BRACES_UNBALANCED. Mode 2 is similar to the default escaping in the PEP 292 string.Template object. Transitions from character data to markup are normally accomplished via "${" and transitions back by "}", but if the markup consists of a field name with no conversion specifier, and the character data immediately following the field name is a space or newline, then the transition to markup may be specified by "$" (without the open brace), and the transition back to character data will happen implicitly on the whitespace. Literal dollar signs are created by doubling them ($$). The constant for this mode in the string module is FORMAT_SYNTAX_BRACES_DOLLAR. Mode 3, like modes 0 and 1, uses braces to transition from character data to markup. Literal closing braces are not escaped (this is like mode 1), and literal opening braces are escaped by placing whitespace ('\r', '\n', or ' ') immediately after the brace in the format string. If the whitespace is a literal space, the brace will be transferred to the output string, but the space will not. If the whitespace is a carriage return or linefeed, it will be transferred intact, along with the brace. The constant for this mode in the string module is FORMAT_SYNTAX_BRACES_WHITESPACE. Syntax mode transitions The default syntax mode is mode 0. The mode can be changed in the format string by using a markup string (delimited by the syntax mode currently in use) which consists of a leading "@" character, plus the letters "syntax" and the number "0" through "3". When the syntax mode is changed, a trailing newline (after the transition back to character data) will be considered to be part of the markup rather than part of the character data. For example, in the string: '''{@syntax1} Now I can have } without having to double them. ''' the first non-markup character will be the 'N'. The default mode can also be set in the flags parameter when using the flag_format() function. Syntax examples Throughout this document, unless otherwise stated, all syntax examples are for syntax mode 0. Here are some basic examples of mode 0 syntax: Brace characters ('curly braces') are used to indicate a replacement field within the string: "My name is {0}".format('Fred') The result of this is the string: "My name is Fred" Braces can be escaped by doubling: "My name is {0} :-{{}}".format('Fred') Which would produce: "My name is Fred :-{}" Replacement fields The usual markup element (e.g. inside the braces in syntax mode 0) is called a 'replacement field', or field for short. Fields consist of a 'field name', which can either be simple or compound, and an optional 'conversion specifier'. Simple and Compound Field Names Simple field names are either names or numbers. If numbers, they must be valid base-10 integers; if names, they must be valid Python identifiers. A number is used to identify a positional argument, while a name is used to identify a keyword argument. A compound field name is a combination of multiple simple field names in an expression: "My name is {0.name}".format(file('out.txt')) This example shows the use of the 'getattr' or 'dot' operator in a field expression. The dot operator allows an attribute of an input value to be specified as the field value. The types of expressions that can be used in a compound name have been deliberately limited in order to prevent potential security exploits resulting from the ability to place arbitrary Python expressions inside of strings. Only two operators are supported, the '.' (getattr) operator, and the '[]' (getitem) operator. An example of the 'getitem' syntax: "My name is {0[name]}".format(dict(name='Fred')) It should be noted that the use of 'getitem' within a string is much more limited than its normal use. In the above example, the string 'name' really is the literal string 'name', not a variable named 'name'. The rules for parsing an item key are very simple. If it starts with a digit, then its treated as a number, otherwise it is used as a string. It is not possible to specify arbitrary dictionary keys from within a format string. Implementation note: The implementation of this proposal is not required to enforce the rule about a name being a valid Python identifier. Instead, it will rely on the getattr function of the underlying object to throw an exception if the identifier is not legal. The format function will have a minimalist parser which only attempts to figure out when it is "done" with an identifier (by finding a '.' or a ']', or '}', etc.) The only exception to this laissez-faire approach is that, by default, strings are not allowed to have leading underscores. Conversion Specifiers Each field can also specify an optional set of 'conversion specifiers' which can be used to adjust the format of that field. Conversion specifiers follow the field name, with a colon (':') character separating the two: "My name is {0:8}".format('Fred') The meaning and syntax of the conversion specifiers depends on the type of object that is being formatted, however there is a standard set of conversion specifiers used for any object that does not override them. Conversion specifiers can themselves contain replacement fields. For example, a field whose field width is itself a parameter could be specified via: "{0:{1}}".format(a, b, c) Note that the doubled '}' at the end, which would normally be escaped, is not escaped in this case. The reason is because the '{{' and '}}' syntax for escapes is only applied when used *outside* of a format field. Within a format field, the brace characters always have their normal meaning. Also, within a format field, the markup syntax is always syntax 0, so, for example, the equivalent code for syntax 2 would be: "${0:{1}}".format(a, b, c) The syntax for conversion specifiers is open-ended, since a class can override the standard conversion specifiers. In such cases, the format() method merely passes all of the characters between the first colon and the matching brace to the relevant underlying formatting method. Standard Conversion Specifiers If an object does not define its own conversion specifiers, a standard set of conversion specifiers are used. These are similar in concept to the conversion specifiers used by the existing '%' operator, however there are also a number of significant differences. The standard conversion specifiers fall into three major categories: string conversions, integer conversions and floating point conversions. The general form of a standard conversion specifier is: [[fill]align][sign][width][.precision][type] The brackets ([]) indicate an optional element. Then the optional align flag can be one of the following: '<' - Forces the field to be left-aligned within the available space (This is the default.) '>' - Forces the field to be right-aligned within the available space. '=' - Forces the padding to be placed after the sign (if any) but before the digits. This is used for printing fields in the form '+000000120'. Note that unless a minimum field width is defined, the field width will always be the same size as the data to fill it, so that the alignment option has no meaning in this case. The optional 'fill' character defines the character to be used to pad the field to the minimum width. The alignment flag must be supplied if the character is a number other than 0 (otherwise the character would be interpreted as part of the field width specifier). A zero fill character without an alignment flag implies an alignment type of '='. The 'sign' element can be one of the following: '+' - indicates that a sign should be used for both positive as well as negative numbers '-' - indicates that a sign should be used only for negative numbers (this is the default behaviour) ' ' - indicates that a leading space should be used on positive numbers '()' - indicates that negative numbers should be surrounded by parentheses 'width' is a decimal integer defining the minimum field width. If not specified, then the field width will be determined by the content. The 'precision' is a decimal number indicating how many digits should be displayed after the decimal point in a floating point conversion. In a string conversion the field indicates how many characters will be used from the field content. The precision is ignored for integer conversions. Finally, the 'type' determines how the data should be presented. If the type field is absent, an appropriate type will be assigned based on the value to be formatted ('d' for integers and longs, 'g' for floats, and 's' for everything else.) The available string conversion types are: 's' - String format. Invokes str() on the object. This is the default conversion specifier type. 'r' - Repr format. Invokes repr() on the object. There are several integer conversion types. All invoke int() on the object before attempting to format it. The available integer conversion types are: 'b' - Binary. Outputs the number in base 2. 'c' - Character. Converts the integer to the corresponding unicode character before printing. 'd' - Decimal Integer. Outputs the number in base 10. 'o' - Octal format. Outputs the number in base 8. 'x' - Hex format. Outputs the number in base 16, using lower- case letters for the digits above 9. 'X' - Hex format. Outputs the number in base 16, using upper- case letters for the digits above 9. There are several floating point conversion types. All invoke float() on the object before attempting to format it. The available floating point conversion types are: 'e' - Exponent notation. Prints the number in scientific notation using the letter 'e' to indicate the exponent. 'E' - Exponent notation. Same as 'e' except it uses an upper case 'E' as the separator character. 'f' - Fixed point. Displays the number as a fixed-point number. 'F' - Fixed point. Same as 'f'. 'g' - General format. This prints the number as a fixed-point number, unless the number is too large, in which case it switches to 'e' exponent notation. 'G' - General format. Same as 'g' except switches to 'E' if the number gets to large. 'n' - Number. This is the same as 'g', except that it uses the current locale setting to insert the appropriate number separator characters. '%' - Percentage. Multiplies the number by 100 and displays in fixed ('f') format, followed by a percent sign. If the conversion type (the last character of the conversion specifier) is 'h', the remainder of the conversion specifier is not examined, and the object and specifier are passed to the 'field_hook' argument to the format function for processing. Objects are able to define their own conversion specifiers to replace the standard ones. An example is the 'datetime' class, whose conversion specifiers might look something like the arguments to the strftime() function: "Today is: {0:a b d H:M:S Y}".format(datetime.now()) Controlling Formatting on a per-class basis A class that wishes to implement a custom interpretation of its conversion specifiers can implement a __format__ method: class AST: def __format__(self, specifiers): ... The 'specifiers' argument will be either a string object or a unicode object, depending on the type of the original format string. The __format__ method should test the type of the specifiers parameter to determine whether to return a string or unicode object. It is the responsibility of the __format__ method to return an object of the proper type. string.format() will format each field using the following steps: 1) See if the value to be formatted has a __format__ method. If it does, then call it. 2) Otherwise, check the internal formatter within string.format that contains knowledge of certain builtin types. 3) Otherwise, call str() or unicode() as appropriate. Controlling Formatting on a per-field basis If the built-in formatting is not sufficient for a particular field, then a 'field_hook' can be passed to the formatter. This hook function has the same call signature as the class __format__ method described above; that is, it takes an object and a format specifier string, and returns a result string which format() will insert in the output string it is building at the appropriate spot: field_output_string = field_hook(fieldobj, formatspec) Note that, in this case, the last character of formatspec will always be 'h'. User-Defined Formatting Classes There will be times when customizing the formatting of fields on a per-type or per-field basis is not enough. An example might be an accounting application, which displays negative numbers in parentheses rather than using a negative sign. The string formatting system facilitates this kind of application- specific formatting by allowing user code to directly invoke the code that interprets format strings and fields. User-written code can intercept the normal formatting operations on a per-field basis, substituting their own formatting methods. For example, in the aforementioned accounting application, there could be an application-specific number formatter, which reuses the string.format templating code to do most of the work. The API for such an application-specific formatter is up to the application; here are several possible examples: cell_format("The total is: {0}", total) TemplateString("The total is: {0}").format(total) Creating an application-specific formatter is relatively straight- forward. The flag_format() function in the string module has a flag that can be set to indicate that the 'field_hook' function should be called on every single field, rather than just on fields which have a specifier type of 'h'. In this case, the field_hook function has the option of returning None rather than a string, to indicate that it wants the internal formatter to format the field. The hook function is called as follows: result = field_hook(fieldobj, fieldspec) The 'fieldobj' field corresponds to the value being formatted, which was retrieved from the arguments using the field name. The 'fieldspec' argument is the conversion spec part of the field, which will be either a string or unicode object, depending on the type of the original format string. The field_hook will be called once per field. The field_hook may take one of two actions: 1) Return a string or unicode object that is the result of the formatting operation. 2) Return None, indicating that the field_hook will not process this field and the default formatting should be used. This decision should be based on the type of the value object, and the contents of the conversion string. The hook function can even call back into the flag_format() function for help in formatting a field. In this case, it can use the FORMAT_SYNTAX_SINGLE_FIELD option to pass in a conversion specifier and the object to be formatted. This will bypass the machinery which transitions between character data and markup, and the machinery which parses the field name, and will call the internal field formatter on the single object. Error handling There are two classes of exceptions which can occur during formatting: exceptions generated by the formatter code itself, and exceptions generated by user code (such as a field object's getattr function, or the field_hook function). In general, exceptions generated by the formatter code itself are of the "ValueError" variety -- there is an error in the actual "value" of the format string. (This is not always true; for example, the string.format() function might be passed a non-string as its first parameter, which would result in a TypeError.) The text associated with these internally generated ValueError exceptions will indicate the location of the exception inside the format string, as well as the nature of the exception. For exceptions generated by user code, a trace record and dummy frame will be added to the traceback stack to help in determining the location in the string where the exception occurred. The inserted traceback will indicate that the error occurred at: File "", line XX, in column_YY where XX and YY represent the line and character position information in the string, respectively. Alternate Syntax Naturally, one of the most contentious issues is the syntax of the format strings, and in particular the markup conventions used to indicate fields. Rather than attempting to exhaustively list all of the various proposals, I will cover the ones that are most widely used already. - Shell variable syntax: $name and $(name) (or in some variants, ${name}). This is probably the oldest convention out there, and is used by Perl and many others. When used without the braces, the length of the variable is determined by lexically scanning until an invalid character is found. This scheme is generally used in cases where interpolation is implicit - that is, in environments where any string can contain interpolation variables, and no special subsitution function need be invoked. In such cases, it is important to prevent the interpolation behavior from occuring accidentally, so the '$' (which is otherwise a relatively uncommonly-used character) is used to signal when the behavior should occur. It is the author's opinion, however, that in cases where the formatting is explicitly invoked, that less care needs to be taken to prevent accidental interpolation, in which case a lighter and less unwieldy syntax can be used. - Printf and its cousins ('%'), including variations that add a field index, so that fields can be interpolated out of order. - Other bracket-only variations. Various MUDs (Multi-User Dungeons) such as MUSH have used brackets (e.g. [name]) to do string interpolation. The Microsoft .Net libraries uses braces ({}), and a syntax which is very similar to the one in this proposal, although the syntax for conversion specifiers is quite different. [4] - Backquoting. This method has the benefit of minimal syntactical clutter, however it lacks many of the benefits of a function call syntax (such as complex expression arguments, custom formatters, etc.). - Other variations include Ruby's #{}, PHP's {$name}, and so on. Some specific aspects of the syntax warrant additional comments: 1) Backslash character for escapes. The original version of this PEP used backslash rather than doubling to escape a bracket. This worked because backslashes in Python string literals that don't conform to a standard backslash sequence such as '\n' are left unmodified. However, this caused a certain amount of confusion, and led to potential situations of multiple recursive escapes, i.e. '\\\\{' to place a literal backslash in front of a bracket. 2) The use of the colon character (':') as a separator for conversion specifiers. This was chosen simply because that's what .Net uses. The inclusion of multiple syntaxes into the proposal: This proposal has a few different syntax methods, one of which might be expected to work reasonably well with most user's applications. It might be unwieldy (and require inefficient code) to support too many different markup syntaxes, but the currently supported ones are similar enough (and the set small enough) to be parsed by the same C loop quite easily. It may seem that the choice of syntax comes down to personal taste (and there is undoubtedly a lot of that), but different problem domains arguably lend themselves to different syntaxes. The "personal preference" argument is also quite powerful. Some individuals have a hard time distinguishing "{" and "}" characters (which are very lightweight in some fonts, almost indistinguishable from "(" and ")") and prefer the visual impact of seeing "$" in front of markup, while other individuals have a very visceral negative reaction to the "$". One thing that seems certain is that, if different syntaxes are to be supported, it is desirable that the format string itself have a mechanism by which it can announce which syntax it is using, so that automated tools can be used to analyze format strings, and so that the syntax conforms to the needs of the author of the format string, rather than the author of the Python code. The default syntax, as we have seen, uses {} to delineate markup from normal text. It was pointed out that XML, for instance, requires a literal < to be escaped, but not a literal >. Obviously, for some sorts of text, it will be useful to see {{ balance }}, but for other sorts of texts, the braces aren't matched, and it is cumbersome to escape the closing brace, so syntax 1 doesn't require this. In some text with a lot of braces, but few markup substitutions, it might be difficult to visually find the markup inside the text. Syntax 2 handles these situations handily by requiring a more traditional ${} or $identifier approach to markup. This brings us to the last syntax supported by the sandbox implementation. Syntax 3 would send any non-Python programmer running, because it depends on significant whitespace. It works great for things like C templates where most left braces are followed by a newline. Argument name collisions The 'namespace' and 'field_hook' arguments to format() share the same namespace as the user's keyword argument field names. There is nothing to prevent a format string which does not require the 'namespace' or 'field_hook' arguments from reusing these names for its own variables. For clarity, use of names with leading underscores, like _namespace and _field_hook, was considered and rejected, because the number of people who want to format variables named "field_hook" and "namespace" is probably vanishingly small compared to the number of people who actually want to use the namespace and field_hook of the format function. Automatic inclusion of locals() and globals() If no parameters are passed to the format() method, it will automatically use the caller's locals() and globals(). It is a tough call as to whether this should be done or not; on the one hand it is awfully convenient, but on the other hand it may be considered to violate the EIBTI principle. If it is decided to remove this capability, however, then the similar functionality of eval() should be reconsidered for Python 3000, because the arguments for and against the capability appear to be identical in both cases. Sample Implementation Talin generated a rough Python prototype of some of the functionality described in this PEP, and Patrick Maupin and Eric Smith generated a functional C version. There are a few discrepancies between the C and this proposal, but the learning experience of writing the C has helped to hone the proposal, so it is expected that it will not be too much work to bring the C code in line with the final version of this PEP. Backwards Compatibility Backwards compatibility can be maintained by leaving the existing mechanisms in place. The new system does not collide with any of the method names of the existing string formatting techniques, so both systems can co-exist until it comes time to deprecate the older system. References [1] Python Library Reference - String formating operations http://docs.python.org/lib/typesseq-strings.html [2] Python Library References - Template strings http://docs.python.org/lib/node109.html [3] [Python-3000] String formating operations in python 3k http://mail.python.org/pipermail/python-3000/2006-April/000285.html [4] Composite Formatting - [.Net Framework Developer's Guide] http://msdn.microsoft.com/library/en-us/cpguide/html/cpconcompositeformatting.asp?frame=true Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: