Werkzeug

The Swiss Army Knife For Python Web Developers

14 August 2008

Mono isn’t just for Apes

by Armin Ronacher at 23:08

On my trip through the Scotland I bought an edition of “Linux Format” to have something to read while waiting for the train. I was reading the letters to the editor when I came across one letter that was basically an insult to the magazine complaining about their Mono support. Apparently they reviewed Tomboy in an earlier edition and haven’t criticised for using Microsoft technology.

I came in contact with C# and .NET about two years ago when it was .NET 2.0 and instantly fell in love with it. It was Java inspired but implemented tons of really cool features like properties, delegates and much more. However at the time, Mono hasn’t yet implemented generics which was a 2.0 feature. Maybe it did, but at least my ubuntu installation wasn’t shipping gmcs yet. Because of that I pretty much forgot about Mono for a while and just played with it once in a while.

A few weeks ago I noticed that gmcs has a command line switch called -langversion:linq which enables all 2.0 features plus a few from 3.0 and 3.5 which made me play a lot with it. My main development environments are an OS X notebook and an Ubuntu linux one. I still do have a windows box but I just use it for playing. As a matter of fact I’m only interested in Mono and not the Microsoft versions of the .NET technologies. What I’ve seen so far is that Mono implemented all the cool features I actually want to use. That is C# with generics, “yield return”, lambdas, extension methods and LINQ (that is .NET 3.0 + some 3.5 features as far as I know, afair LINQ is a 3.5 feature). They also have XSP2 and System.Web which makes it possible to use it for web applications. What I’ve seen so far qualifies as “awesome” so I want to share my feelings about Mono and why I think it’s the best since the advent of ubuntu. My real experience with Mono is only a few weeks old therefore I may be missing quite a few things so please take this “review” with a grain of salt.

I think there are three major types of critics of Mono in the Open Source community. Those who think Microsoft pays Novell for developing Mono to take over the world, those who think Mono is a project that will face patent problems in the future because Microsoft will claim copyright infringement and those who have a problem with Novell because of the patent agreement with Microsoft. Of course there are more. There are people who actually had a look at it and compare it to established platforms like the JVM and come to the conclusion that the Sun compiler is better than Mono or something similar. I agree that it’s perfectly okay to not like Mono because one is happy with the JVM or to hate Novell to sign that patent agreement with Microsoft. However none of that makes Mono as such a bad thing to have.

Mono is largely based on two EMCA standards (ECMA-334 aka C#) and (ECMA-335 aka CLI or Common Language Infrastructure). Other than that Mono also implements parts of ASP.NET, ADO.NET as well as Windows.Forms which are subject to lots of concern from the Open Source community. I don’t know if parts of those are patented already or if that’s just a hypothetical problem but it seems to be one of the main reasons why people hate Mono. Fortunately you don’t have to use any of them to do proper development with Mono. Especially Windows.Forms is something you don’t even want to use ;-)

Mono itself is a fantastic piece of software and one of the best programming environments Linux users got since ages. The big advantage of it is that the Mono environment can host multiple programming languages, not just C#. Even though there are multiple languages running on top of it they can exchange code which is somewhat hard to achieve with the more traditional approaches. For example it’s nearly impossible to use a Python library with Ruby or the other way round. With Mono, IronRuby and IronPython this becomes somewhat possible. But I must agree that I haven’t played with that in detail so far. C# alone was convincing enough.

There are just two languages I dare to compare with C#. One is obviously Java where C# got mosts of it’s inspiration. The other one is D, a rather new language that unlike C# or Java is not running on an runtime environment but produces nativ code only. There is another one which is called Vala and popped up a while ago which basically tries to be C# compiling down to glib-C. D is a rather new language and there are not many bindings to popular libraries available so far. Vala is even newer. Until Mono popped up there was no statically compiled language with bindings to major open source libraries such as GTK or DBus and was beginner friendly. Most people used C/C++ or Python to develop GNOME applications at least and on my rather new ubuntu box most GUI applications are still made with those languages.

The language that is most similar to C# is Java. Having learned a lot from Java, Microsoft improved C# over Java a lot. C# feels a lot more like Python than Java. My favourite feature of C# is without a doubt the ability to create properties. And from all the languages with comparable features I know (which are Python, Ruby, D and C#) C# really has the best solution for the problem. No need to write getters and setters from the ground up in fear of breaking code later when code execution on setting/getting of values is required. A non-property approach to represent a user could look like this:

class User {
  public string Username;
  public User(string username) {
    Username = username;
  }
}

If you later decide to execute code on setting the Username attribute you can easily do so by making the attribute private and adding a property:

class User {
  private string username;
  public User(string username) {
    Username = username;
  }
  public Username {
    get { return username; }
    set {
      Utils.MoveHomeFolder(username, value);
      username = value;
    }
  }
}

My favorite feature after properties is definitively that you have to make methods virtual explicitly. This enables faster code and hides a lot of errors. In general the compiler can save you from quite a lot of problems you only spot with excessive unit-testing in Python. For me that is a huge advantage because I’m a) quite lazy and b) bad at typing. I get typos in the easiest words and thanks to ^P in Vim those appear multiple times before I notice :)

One of the things I love about Python is the possibility to subclass internal objects such as dicts, lists and more to given them a behavior more practical to the kind of data I store in them than the normal version of the objects. For example Werkzeug comes with tons of custom dicts, lists and sets for case insensitive data, multiple keys in a dict and similar stuff. C# makes it ridiculously easy to do that thanks to generics and the classes from System.Collections.Generic. First of all they check the types of the objects you put into them and furthermore you don’t even have to subclass them to get collections the standard library accepts as containers. In Python you pretty much have to subclass the builtins because many Python libraries perform instance checks against list, dict etc. In C# there are Interfaces for that and they are used all over the place which is clever.

A huge advantage over Java is also that you have delegates and lambdas which enable a lot of cool stuff not possible in Java. C# also knows “yield return” which is essentially a helper to generate Enumerator (iterator in Python) objects automatically which saves you tons of boilerplate code. Another neat thing about C# is that you have preprocessor directives which enable conditional compilation and allow you to affect the error reporting by providing different line numbers or filenames in “#line” comments. I often wished for something like that in Python for example when writing Jinja which has to do an ugly hack to rewrite the Tracebacks on the fly to get a proper debug output.

But C# goes far beyond that. Apparently the thread safety in C# is mostly achieved by per-object locking which you can control with lock(obj) { ... } which makes it a lot easier to write thread safe classes. The Python “with” statement is available as using (expr) { ... } which leads to much shorter code compared to Java. Take this Java example:

import java.io.*;

public class FileExample {
  public static void main (String[] args) {
    StringBuilder out = new StringBuilder();
    try {
      BufferedReader in = new BufferedReader(new FileReader("filename.txt"));
      try {
        String line, separator = System.getProperty("line.separator");
        while ((line = in.readLine() != null) {
          out.append(line);
          out.append(separator);
        }
      }
      finally {
        in.close();
      }
    }
    catch (IOException ex) {
      ex.printStackTrace();
    }
    doSomethingWith(out.toString());
  }
  public static void doSomethingWith(string s) {}
}

This is just ugly and I don’t even know if I works because I hacked up from memory without actually testing it. Now compare that with the following C# version of the above code:

using System.IO;

class FileExample {
  public static void Main(string[] args) {
    string result;
    using (StreamReader r = new StreamReader("filename.txt"))
      result = r.ReadToEnd();
    DoSomethingWith(result);
  }
  public static void DoSomethingWith(string s) {}
}

Not a single try/finally. using automatically does the right thing because the StreamReader is an object implementing IDeposable which means that after the using block C# will automatically call r.Dispose().

All the libraries I played with so far (that are the System.* ones, Gtk#, Dbus and many others) are using the language features like they should. Properties are used where wanted, all public namespaces, classes, methods and properties are named in a consistent way and attribute classes (a special feature somewhat comparable to decorators in Python) are used where useful (for example dbus). That’s a consistency in the core libraries you won’t find in Python! It’s really a pleasure to work with that because the code looks nice and there are few surprises when looking for names.

Of course there are problems too. The documentation for Mono is still lacking but you can help yourself by using the MSDN one. In general the Mono documentation is a lot better than some other open source projects.

The progress Mono makes is astonishing. They may not be as fast as Microsoft but the majority of the features work and even if we wouldn’t get any new it would be a great development platform. It really doesn’t matter if Mono can’t keep up with Microsoft’s .NET. The linux community isn’t very keen on Silverlight anyways and besides Silverlight not many non-mono applications will hit the average Linux PC. The goal of the Mono project is not to run arbitrary Windows .NET applications on Linux but to have a free implementation of the .NET framework. It’s saddening that so many people torpedo the development because of FUD or just because Microsoft came up with the idea.

So if you haven’t had a look at Mono yet because you’ve heard so many negative things about it: Forget about that and give it a try yourself. You can’t lose :)

17 July 2008

Jinja2 Final aka Jinjavitus Released

by Armin Ronacher at 22:07

The fiinal version of the Jinja2 Django-inspired template engine was just released. It comes with tons of improvements over the older Jinja1 engine and breaks backwards compatibility to some point over the old Jinja1 engine. It’s packaged as a separate package so that you can use both right next to each other for an easier transition.

Compared to Jinja1 it provides tons of new features:

  • Dynamic inheritance. It’s now possible to use dynamic inheritance which means that the name of the master template expanded at render time. This makes it easy to switch between different designs.
  • Improved macro and import system. Macros can be called with keyword arguments now, are much more lightweight and got their own import system which makes templates easier to understand. The import syntax follows the Python one with small adjustments.
  • Heavily improved for-loops. Loops can be filtered now and the length is calculated lazily. This makes it possible to iterate over generators with an unknown length in a much more efficient way.
  • Improved behavior of undefined values. Jinja1 had a very silent undefined behavior. If a variable was undefined you were able to call it without getting errors or access any attribute. Jinja2 ships three undefined types that make it easier to debug templates. The default undefined types allows you to print the undefined variables (which when printed outputs nothing) and loop over it (works like iterating over an empty list). However every other operatation raises an UndefinedError. Additionally there an undefined type with the same behavior but it prints the name of the variable or attribute missing if it’s printed. The third builtin undefined type is the strict type which doesn’t allow any operation except of testing if it’s undefined which is the closest you can get to the default Python behavior.
  • Improved sandbox. The sandboxed environment if enabled is now easier to use and more secure. In the default configuration everything starting with an underscore is considered insecure so it’s not needed any longer to mark those attributes explicitly. It’s also a lot faster now and easily modifyable by subclassing.
  • Line statements. Jinja2 allows to to specify a line statement prefix that marks a whole line as statement. This concept is inspired from Mako and Cheetah and allows very clean templates in many situations. If a line starts with the prefix character (for example “#”), everything up to the end of the line is handled as block.
  • Easier API. The API is much easier to use now on nearly every level. The loaders got refactored so that you only have to provide a single methods, filters and tests are normal Python functions now.
  • Automatic escaping. Jinja2 comes with optional automatic escaping compatible to Pylons and Genshi
  • Improved i18n support. Jinja2 integrates into Babel now which makes internationalizing web applications a charm.
  • Extension interface. Jinja2 provides a documented interface that can be used to extend the template engine. The interface could even be used to create Jinja inspired template engines on top of the existing compiler interface with a completely different syntax.

Get it while it’s hot from the Cheeseshop (or PyPI for those of you who prefer the new name).

Deploying Python Web Applications

by Armin Ronacher at 15:07

Every once in a while I’m really impressed by a library I stumble upon. A while back that was virtualenv, now i stumbled upon fabric. I was using capistrano for a project I was working on which was kinda okay but somehow I wasn’t sold to it.

Yesterday however apollo13 stumbled upon fabric which is capistrano just in Python, with a working put command and in less annoying.

In combination with a custom virtualenv bootstrapping script Python web application deployment is a charm. One “fab bootstrap” later the servers are creating a virtual python environment, compiling all dependencies, checking out all eggs and initializing the application environment. Updates are just one “fab production deploy” away.

And the best part is that fabric is not limited to Python. You can use it to deploy anything you can control over ssh.

Here an example fabfile (the file that controls the deployment)

set(
    fab_hosts = ['srv1.example.com', 'srv2.example.com']
)

def deploy():
    """Deploy the latest version."""
    # pull all changes from mercurial and touch the wsgi file to
    # tell the apache to reload the application.
    run("hg pull -u; touch application.wsgi")

def bootstrap():
    """Asks for a list of servers and bootstrapps the application there."""
    set(fab_hosts=[x.strip() for x in raw_input('Servers: ').split()])
    run("hg clone http://repository.example.com/application")
    local("./generate-wsgi-file.py > /tmp/application.wsgi")
    put("/tmp/application.wsgi", "application.wsgi")

Savead as fabfile.py “fab bootstrap” then asks for some servers and bootstraps the application there, after changes in the repository you can “fab deploy” the latest version. Of course that’s just a very basic made up example, but it shows how you can use fabric.

I’m using makefiles currently to execute common tasks for various Python projects (like releasing code, resting unittests and much more), I suppose fabric could also do that for me. And that would have the advantage that it works for windows users too.

05 July 2008

virtualenv for the Rescue

by Armin Ronacher at 14:07

Deploying Python applications is not that hard, but there are some pitfalls. The main problem everyone stumbles upon sooner or later is that sys.modules is a singleton. Modules are cached there which the effect that there can be exaclty one version of a library loaded at the same time per interpreter. There is no way around that and it’s fine. However there is another problem: Everyone installs everything into a system wide singleton called site-packages. Sooner or later everything ends up there, and while pkg_sources supports switching between versions this is very unconfortable and requires extra hackery.

My plan to solve these problems always was creating a folder, putting all libraries into that folder and site.addsitedir()ing that folder in the file that starts up the application. But especially for web applications that’s a dull repetitive task and easy_install and other libraries don’t play well with this approach.

Fortunately Ian Bicking wrote a really cool module called “virtualenv” that creates virtual Python environments. And it does that in a very cool manner. The following example shows how we run LodgeIt on our server:

First you need to install virtualenv. This assumes you already have easy_install:

mitsuhiko@nausicaa:~$ sudo easy_install virtualenv
Searching for virtualenv
Reading http://pypi.python.org/simple/virtualenv/
Best match: virtualenv 1.1
Downloading virtualenv-1.1-py2.5.egg
Processing virtualenv-1.1-py2.5.egg
creating /opt/local/lib/python2.5/site-packages/virtualenv-1.1-py2.5.egg
Extracting virtualenv-1.1-py2.5.egg to /opt/local/lib/python2.5/site-packages
Adding virtualenv 1.1 to easy-install.pth file
Installing virtualenv script to /opt/local/bin

Installed /opt/local/lib/python2.5/site-packages/virtualenv-1.1-py2.5.egg
Processing dependencies for virtualenv
Finished processing dependencies for virtualenv

Now we can create a folder folder the application. In our case the folder is called after the domain where the web application runs:

mitsuhiko@nausicaa:~$ virtualenv paste.pocoo.org
New python executable in paste.pocoo.org/bin/python
Installing setuptools.......................done.

The virtual python is now available in that folder and we can switch into that environment by sourcing the activate file:

mitsuhiko@nausicaa:$ cd paste.pocoo.org/
mitsuhiko@nausicaa:~/paste.pocoo.org$ . bin/activate
(paste.pocoo.org)mitsuhiko@nausicaa:~/paste.pocoo.org$

From that point onwards “python” and “easy_install” point to the executables in the ~/paste.pocoo.org/bin folder. The prompt is prefixed with the name of the virtual environment so that we are reminded that “python” is our virtual Python. We can now install all the dependencies:

$ easy_install SQLAlchemy==0.4.4 Jinja2 Werkzeug Pygments

Half a minute later the virtual python interpreter will now have the libraries available and we’re ready to add out application:

$ hg clone http://dev.pocoo.org/hg/lodgeit-main

Lodgeit itself doesn’t have a setup.py file but we can just cd into the folder and run the testing server to test if it works:

$ cd lodgeit-main/
$ python runlodgeit.py runserver
 * Running on http://localhost:5000/
 * Restarting with reloader...

That works perfectly for scripts invoking “python” directly such as standalone servers like CherryPy but won’t work for mod_wsgi where the interpreter is created by mod_wsgi and doesn’t point to the virtual environment.

There are two ways to solve the problem. For mod_wsgi 2.0 you can run the web application in daemon mode and pass it the path to the site packages in the apache config:

WSGIDaemonProcess lodgeit python-path=/path/to/site-packages

Alternatively you can go into the .wsgi file and add the site-packages by hand:

import site
site.addsitedir("/path/to/site-packages")

The site-packages of the virtual environment are located at /path/to/virtual/env/lib/python2.X/site-packages.

Even though I was not interested in virtualenv in the beginning I changed my mind because it’s straightforward to use and easy to maintain. More information about virtualenv are available in the Cheeseshop and the announcement blog post in Ian’s blog.

01 July 2008

Whitespace Sensitivity

by Armin Ronacher at 12:07

I was reading a thread on ruby-forum.com about Python that said that the whitespace-sensitivity of Python is from hell or something. There are people from every programming language that can rant about Whitespace sensitivity in Python but clearly not Ruby programmers. Why? Because Python doesn’t care about Whitespace at all. The only thing that somewhat has to do with whitespace is the indentation that the lexer convers into indent and outdent tokens. But after that, no whitespace any more, the parser doesn’t know anything about that.

That however is not true for Ruby! foo[42] does a completely different thing than foo [42]. The first calls foo without argument and subscribes the return value, the latter calls foo with [42] as Argument which happens to be an Array with one element. But there are more examples.

Take this example:

foo = 23
def bar
  42
end

puts bar/foo

That prints “1″.

However take this minor modification:

foo = 23
def bar
  42
end

puts bar /foo

Now this gives you an error that the regular Expression literal is unterminated. That’s what I call whitespace sensitivity :)

You’re wonderhing why I’m using a method for “bar” and not a locale variable? Because the parser keeps track of all assigned local variables or methods (Not sure what exactly it does) and the syntax ambiguities are resolved that way.

25 June 2008

… and they called it Werkzeug

by Armin Ronacher at 20:06

Werkzeug was featured in this week’s django podcast. I was joking in the past that after naming naming a library after an Japanese temple I want to see people pronounce a German name.

And haha, it payed of. Werkzeug was called the “w-tool” after some attempts to pronounce it :-) So now I uploaded a sound sample for Werkzeug.

24 June 2008

Werkzeug 0.3.1 released

by Armin Ronacher at 07:06

Today we have to push Werkzeug 0.3.1. 0.3 and below had a possible cryptographic weakness in the secure cookie that would allow attackers to inject additional information into the cookie. 0.3.1 fixes that and can be downloaded from the cheeseshop.

werkzeug.contrib.securecookie is still an undocumented module but used widely. There is a small API change that removed SecureCookie.new_salt. Also keep in mind that this fix automatically invalidates all existing cookies will be reinitialized with new values on the first request a user causes which usually means that the user is logged out automatically.

We’re sorry for the inconveniences caused.

19 June 2008

… and 0.10 follows 0.9

by Armin Ronacher at 12:06

Some open source projects jump from 0.9 to 0.10. So does Trac, so did Pygments. So please, don’t compare versions with string comparison operators. pkg_resources can parse versions for you so that you can compare them:

>>> from pkg_resources import parse_version
>>> parse_version("0.10") > parse_version("0.9")
True

If you don’t have pkg_resources you have problems getting running one of those anyways, but alternatively you can still split at every dot and try to compare the numbers into integers and falling back to strings:

>>> def parse_version(version):
...  def _trynumber(x):
...   try:
...    return int(x)
...   except ValueError:
...    return x
...  return tuple(map(_trynumber, version.split('.')))
...
>>> parse_version("1.0") > parse_version("0.9")
True

However, how this error happened I have no idea. TracMercurial doesn’t do checks on its own but uses setuptools for it, and that knows how to compare versions.

Cycling values in Python

by Armin Ronacher at 09:06

import sys
from threading import local

_cycles = local()

def cycle(*args):
    frm = sys._getframe(1)
    key = (frm.f_code, frm.f_lineno)
    d = _cycles.__dict__
    try:
        pos = d[key] = (d[key] + 1) % len(args)
    except LookupError:
        pos = d[key] = 0
    return args[pos]

Works like this:

>>> cycle(1, 2)
1
>>> cycle(1, 2)
2

And yes, that’s thread and stack save but ugly. Was more or less a “yes it can be done” kind of solution, I don’t encourage anyone to use that :-)

14 June 2008

Werkzeug 0.3 aka EUR325CAT6 released

by Armin Ronacher at 17:06

Version 0.3 codename EUR325CAT6 of Werkzeug, the WSGI toolkit was released today. There are many changes from 0.2 and I want to cover some of them here.

  • Updated routing system. We fixed some glitches in the routing system and added support for redirect rules. It’s now possible to redirect from one rule definition to another or to external URLs.
  • Refactored wrappers. The parsing functions from the request and response objects are moved into separate functions now so that it’s possible to use the parsing features without the request or response objects. This is especially useful if you want to parse some HTTP headers in Django for example.
  • Better HTTP support. Werkzeug supports parsing of auth&auth headers now and the middleare that exports shared data for the development server supports entity tags now.
  • Improved data structures. The dict-like data structures (Headers, MultiDict and others) raise a custom key error now that’s both a key error and a BadRequest exception. If you are not catching the exception and you have an error handler for http exceptions, that will kick in. So it’s save to do request.args['foo'] and not check if foo was actually transmitted
  • The same happens for unicode decode errors caused by incoming form data now if the error handling is set to strict.
  • The Python 2.3 support was improved but Werkzeug 0.3 will be the last release with 2.3 compatibility
  • Request objects can be created with shallow=True now. If shallow is set to True, the request object will raise exceptions if an access to the request object causes the input stream to be read. This makes it possible to use the request object in middlewares now without the fear of breaking applications.
  • Updated the debugging system. The debugger was rewritten from the ground up. It now loads a lot faster and the shell was expanded. Additionally it provides a command called dump() which either dumps the local variables or the namespace of an object.

Get it from the Cheeseshop while it’s hot.

09 June 2008

Jinja2 RC1

by Armin Ronacher at 17:06

I was talking about Jinja2 in the past already but that was in the middle of developing it and many things changed in the last weeks. And now I’m proud to announce the first release candidate :)

As I said earlier Jinja2 is Jinja1 without the design mistakes. But what exactly is new and why is it so cool?

It’s dynamic. Unlike Jinja1 template inheritance and inclusion is handled at runtime now. That means you can do something like {% extends master_template %} to extend from the template with the name stored in the “master_template” variable. Previously that was not possible which clearly was a design mistake. This also affects includes (and now imports). Besides the obvious advantage that this is much more flexible it also reduced the memory usage of Jinja and the possibilities. This dynamic inheritance / including directly lead to an import statement to load macros and variables from other templates.

It’s fast. The sandboxed evaluation mode in Jinja2 is now optional and the evaluation rules are simplified which gave it a huge performance improvement. Jinja2 renders a test template in under 0.04 seconds where the latest Django version needs nearly a second for.

Macros are more useful now. Macros in Jinja1 weren’t that great, but they are now! On the one hand because you can import them with a python like import syntax ({% from 'helpers.html' import input_field, textarea %} or {% import 'helpers.html' as helpers %}) on the other because they have their own namespace now. That means they are easier to debug and understand.

Includes and imports separated. Includes just include the template and render it at the current position, imports pull variables and macros from there. This makes it easier to understand templates as it’s clear from where a variable is coming from.

For loops are even cooler now. Jinja1 already had a pretty neat for loop which supported recursion and an optional else block that was rendered if the sequence was empty. It also featured a special loop variable that allowed the template designer to access the number of the current loop iteration, the total length of the loop and much more. Jinja2 extends that by adding support for loop filtering. The following example only lists the users that are visible or all if the current user is an administrator:

{% for user in users if user.is_visible or request.user.is_administrator %}
  <li>{{ user.username|e }}</li>
{% else %}
  <li><em>no users online right now</em></li>
{% endfor %}

The advantage of filtering directly in the loop head is that the special loop variable will count proplery (eg: excluding invisible users for counting). Additionally the else block will work as expected.

Better undefined behavior. Jinja1 had a very silent undefined behavior. If a variable was undefined you were able to call it without getting errors or access any attribute. Jinja2 ships three undefined types that make it easier to debug templates. The default undefined types allows you to print the undefined variables (which when printed outputs nothing) and loop over it (works like iterating over an empty list). However every other operatation raises an UndefinedError. Additionally there an undefined type with the same behavior but it prints the name of the variable or attribute missing if it’s printed. The third builtin undefined type is the strict type which doesn’t allow any operation except of testing if it’s undefined which is the closest you can get to the default python behavior. The following doctests shows the behavior of those types:

The normal undefined type:

>>> x = Undefined(name="x")
>>> unicode(x)
u''
>>> x.attribute
Traceback (most recent call last):
  ...
UndefinedError: 'x' is undefined

The strict undefined doesn’t allow anything except of testing if it’s defined from within a template. You can’t even compare it to an arbitrary value without getting an exception:

>>> x = StrictUndefined(name="x")
>>> unicode(x)
Traceback (most recent call last):
  ...
UndefinedError: 'x' is undefined
>>> x == 42
Traceback (most recent call last):
  ...

UndefinedError: 'x' is undefined

A better sandbox. The sandboxed environment if enabled is now easier to secure. In the default configuration everything starting with an underscore is considered insecure so it’s not needed any longer to mark those attributes as insecure. It’s also a lot faster now and easier to customize.

Better i18n support. Jinja2 now integrates neatly into Babel. No need to write custom string extraction scripts.

Extension interface. Like Django it’s now possible to write custom tags. I doubt anyone wants to use it as most functionality is already possible out of the box by using the expressions and tags provided, but it may be useful for some people, especially those switching from Django.

Line statements. Jinja2 allows to to specify a line statement prefix that marks a whole line as statement. This concept is inspired from Mako and Cheetah and allows very clean templates in many situations. The following example shows a template with the line statement prefix set to “%”:

<ul>
% for item in seq
  <li>{{ item }}</li>
% endfor
</ul>

Easier filters. Filters are regular functions now that get the value as first argument and the parameters provided as extra positional arguments or keyword arguments. In Jinja1 we had functions that return functions which turned out to be unnecessary complex and slow. Additionally you can now use “namespaces” for filters so do stuff like {{ variable|tools.something }}.

Automatic escaping. Jinja2 supports the special __html__ method as specified by pylons which makes it possible to use automatic escaping. (We still don’t recommend it though)

Easier API. The API is a lot easier to use now, custom loaders are nearly one-liners now and the caching is builtin automatically for all loaders. No need to funny mixins that add caching.

Grab it while it’s hot from the temporary Jinja2 website.

05 June 2008

Upgrading Postgres from 8.1 to 8.3 on Debian/Ubuntu

by Armin Ronacher at 09:06

Yesterday we upgraded hammet (the server powering the Pocoo projects) to Ubuntu LTS. Unfortunately Ubuntu 8.04 aka hardy doesn’t ship Postgres 8.1 any longer so we had to upgrade to 8.3. If you are ever in the same situation: DON’T UNINSTALL 8.1. I did that because my applications showed an auth error and I guessed that was because of two running postgres servers. Turns out that postgres databases are incompatible between versions and the only way to upgrade a cluster is having both versions running and using pg_migratecluster to upgrade them. Reinstalling in hardy won’t work because there is no package providing an 8.1 server.

What I did then was installing the 8.1 server again from dapper sources, delete the new cluster with “pg_dropcluster 8.3 main” and migrate with “pg_migratecluster 8.1 main”. After that getting rid of the 8.1 cluster and server and most of it works again.

I fought that auth error by setting local socket connections to trusted in the config responsible for authentication and had to upgrade trac because implicit string casts no longer work.

Those non portable, version incompatible database files and the fact that I hate psql and those admin tools are by the way the reason why I feel more comfortable with MySQL.

31 May 2008

The New %GHRML [haml for Genshi]

by Armin Ronacher at 09:05

I started GHRML some time ago as port of haml to Genshi but the initial reception was not that good, from the few blog comments and discussions on IRC I pretty much lost the interest in the project and haven’t used it myself.

However I’m pleased to announce that Richard Davies took over the GHRML maintenance and hosts the project at ghrml.org. He’s also using the project for a Django powered website and wrote a simple django integration module for it.

%GHRML still looks the same but some bugs are fixed now and template inheritance works. Appetizer:

%html
  %head
    %title Hello World
    %style{'type': 'text/css'}
      body { font-family: sans-serif; }
    %script{'type': 'text/javascript', 'src': 'foo.js'}

  %body
    #header
      %h1 Hello World
    %ul.navigation
      %li[for item in navigation]
        %a{'href': item.href} $item.caption

    #contents
      Hello World!

22 May 2008

Stupid Ruby Leak

by Armin Ronacher at 10:05

class Foo
  def self.bar
    "a string".split(/s+/)
  end
end

Foo.bar while true

Workaround: assign the return value to a temporary variable and then return it.

20 May 2008

How can I compile my Python Scripts?

by Armin Ronacher at 18:05

I see that question about once a week, maybe a bit less often, on the German Python forum. Very often it turns out that py2exe is what the poster was looking for because he doesn’t want to the user to install Python. I can understand that because for the average windows user it doesn’t really make sense why he should install that Python thing. But Python is a runtime environment for the application, like the JVM or the .NET framework windows users do install. So if the windows version of Python would automatically upgrade itself somehow that wouldn’t be a big problem.

However the motivation is very often a different one: “I don’t want others to see my code” aka “I want to obfuscate my code”. Often with the addition that they are afraid others steal their code. The first point is understandable, I just have to look at the code they post in their questions they ask in that forum to see that they really have to be ashamed of. However that’s part of the learning process and by sharing your applications with others including the source you can learn from that. The second argument is that people can steal your code. But surprise: they can do it even if they don’t have the sourcecode. Python has very high-level bytecode and you can decompile it into very nice looking code, even with the original variable names preserved. I would argue that it’s even easier for them to steal code from small closed source Python applications because with a little bit of work the resulting source code looks so fundamentally different when directly compared with the original that one wouldn’t see many differences.

But really. The solution is a proper license, not trying to lock people out. Especially for small applications. Either pick one of the really good open source licenses that range from having a heavy copyleft such as the GPL to something like the do-what-the-fuck-you-want license which allows everything. Or pick a license that doesn’t allow any modifications and put your Copyright into every file. If a bad person actually steals your (probably still crappy) code you can then write a blog post and show that you were the first person that wrote that code. I doubt you want to sue the person that stole your code because that’s pretty expensive, but with the power of the blogosphere you can at least give that copycat a very bad reputation.

By having an open sourced codebase (even if it’s bad) you will attract some other developers that will probably provide patches and help you improve the product. And if it’s good enough, you can make money with it. Just look at WordPress. Some parts of the code look like written by bozos not much better than the average person asking on the python forum how to obfuscate Python code. And still, they are the number one blogging platform and I can only admire what they achieved.

Do you still want to obfuscate your code?

15 May 2008

Mail Subjects

by Armin Ronacher at 19:05

I found two mails in my Junk folder today that where ham. I spotted them by accident as the subjects where very stupid. The first mail arrived with “Re:” as subject, the second with “Hi”. Please make my life easier and use subjects that don’t look like spam. Thanks in advance

And no, I haven’t written a mail without subject in the first place.

13 May 2008

Command of the Day

by Armin Ronacher at 15:05

# ssh-keygen -t dsa -b 1024 -f /etc/ssh/ssh_host_dsa_key -N '' &&
  ssh-keygen -t rsa -b 1024 -f /etc/ssh/ssh_host_rsa_key -N '' &&
  /etc/init.d/ssh restart

I really don’t think distributors should try to patch cryptographic stuff, especially not to silence debuggers.

12 May 2008

Mail Problems

by Armin Ronacher at 19:05

Small notice for persons trying to mail me the last ~three days: While I was off to Vienna I noticed that HE disable the E-Mail routing for the domains I moved to domainfactory. Mails send to me between Friday and today are probably lost.

07 May 2008

Jinja2 Documentation Online

by Armin Ronacher at 12:05

I now uploaded the documentation for Jinja2 to the website for those of you who are eager and want to play with it :-) On jinja.pocoo.org you have now the choice to chose between Jinja1 and Jinja2.

The new docs are powered by Sphinx and Jinja2 with a custom templating bridge.

Read the documenation.

Simple batch function for Python

by Armin Ronacher at 10:05

Often I have an iterable i want to group. For example a list of integers and i want to process two at once. That’s a pretty nice idom I found in the documentation (I just changed it to use a tuple and izip which performs a lot better):

from itertools import izip

def batch(iterable, n):
    return izip(*(iter(iterable),) * n)

Use it like that:

>>> for key, value in batch([1, 2, 3, 4], 2):
...  print key, value
... 
1 2
3 4

What's That?

Here a couple of Werkzeug related blogs are aggreagated into one big feed that is displayed here.

Blogs