Plnt Planet

People have asked me a few times about evalexception and pdb — they’d like to be able to use something like pdb through the browser, stepping through code.

The technique I used for tracebacks wouldn’t really work for pdb. For a traceback I saved all the information from the frames — mostly just the local variables — and then let the user interact with that through the browser. But with pdb you pause the application part way through waiting for user input, and the routine only completes much later.

While writing WaitForIt I played around with techniques to deal with very slow WSGI applications. Not that hard, really — you launch every request in a new thread, and you manage those requests in an application of its own. So I started thinking about pdb again, and it started seeming feasible. Whenever the app reads from stdin it goes into an interactive mode, showing you what comes out on stdout and letting you add input to stdin. It’s nothing specific to pdb really.

So, with a bit of hacking, I added it into WebError (which is an extraction of the exception handling in Paste). To give the demo a try, do:

hg clone http://knowledgetap.com/hg/weberror/
cd weberror
python setup.py develop
# You need Paste trunk:
easy_install Paste==dev
python weberror/pdbcapture.py

What you’ll see is not polished, it’s just working, but since I mostly did it to see if I could do it, that’s good enough for me.

For the most part by the time I finished writing my last article on licensing I had mostly convinced myself that the GPL isn’t a practical license for most projects. That is, outcomes when using the GPL aren’t likely to be any better than outcomes using a permissive license, except for certain kinds of projects, mostly projects involving big faceless companies, and I’d just as soon avoid such projects anyway.

My own thinking on this has changed over the years in part because of a greater sense of humility about what I produce. I’m really not that worried about people stealing my work because I don’t think that theft would be of much value. But also because I realize that the value in software is not so much in the code as in the process. The process is what is valuable, particularly for open source, and licensing doesn’t really address issues of process.

As an example, if I’m uncomfortable with how some member of an open source community is using the code, or the community, I will be much more effective by dealing with that head-on, talking with that member, or even confronting them if it’s really necessary. If you give someone an unwelcoming attitude, they’ll probably go away. The license doesn’t need to be your gatekeeper. It’s not a particularly effective gatekeeper anyway.

Another change is perhaps a more reasonable valuation of code. There was a time when people wanted to protect their intellectual property. Even some non-software company might have gotten the idea that it should own the code it contracts someone else to write, under a proprietary license, so they could sell that software later. That anyone would care to buy it was always an illusion, but the illusion is a little more obvious these days.

One value of the GPL that I do want to acknowledge is its expression of values. It makes this explicit:

When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things.

To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others.

But the GPL does more than just its text: adopting the GPL is a statement of principle on the part of the original authors, of the people who adopt the project, and of the people who later help maintain the project. It is a statement that freedom is valued and that it is valued in a universal sense, not just in a personal or isolated sense.

This is implicit, not explicit, in the choice of license, but despite that I see this pattern in projects. Projects that choose the GPL are more likely to engender a spirit of openness and sharing. Not of the core project itself — both GPL and permissively licensed projects accomplish this just fine so long as they are properly maintained, and their success is far more related to how the project is managed than the licensing. But I see the difference in the sofware that grows up around the project: extensions, complementary projects, documentation.

Maybe this is because of licensing. The license filters the community, and the people who are left in a GPL project are all at least open to sharing. But more than that, I think it puts people in the right state of mind to share. The project feels more principled, the participation is based less on pragmatism and more on optimism. And there’s always people coming into open source who haven’t really figured out why or what they want to get out of it. Presenting them with the principles of Free Software influences their decision. (This issue has caused some debate about terminology.)

With all that said, you don’t need the GPL to present the principles of a project. It’s certainly the easiest way to do so. The GPL is shorthand for a rich set of principles and ideals. But it’s shorthand for people who are already in the know. The ideas need to be reiterated and explained and reconsidered to stay relevant. I think a project might do more good with an explicit statement of principles. With that in place the licensing might not matter so much.

It occurred to me… Django is something like a dictatorship… or maybe an oligarchy. At first it seems like Pylons is the same… but no. Pylons is clearly feudal. I lord over Paste, WebOb, FormEncode. Mike Bayer lords over Mako and SQLAlchemy. Ben lords over Routes, Beaker, and Pylons.

I suppose in all cases there is a certain amount of democracy, because there are no serfs, and any individual is free to travel to any kingdom they like. Well, at least among the open source kingdoms. Without citizenship, and with no exclusiveness of ownership, with even property having largely disappeared, I suppose it’s inevitable that traditional metaphors of control and governance don’t really make sense.

I thought I’d take some time to talk about licensing.

Licensing is something that F/OSS programmers talk about a lot. There’s two major categories of licenses:

• The GPL, aka Copyleft. You must distribute source with your application, and users get full rights to the source code, including any additions you may make.
• Permissive licenses, X/MIT, BSD, etc. These let you do pretty much everything.

There’s also the LGPL, which vaguely applies GPL-like terms to the original code, but is not "viral". LGPL originally meant "Library GPL" but was renamed to "Lesser GPL" because people automatically used it for libraries without considering what the license actually enforced.

How much do these licenses matter? How should a person choose between these options?

First, the LGPL. It has specific phrasing that makes some sense for C code, to distinguish between extending the library itself and using the library. It doesn’t make much sense for other runtime environments. Some people have tried to clarify its meaning in other environments, but I’m not sure if a clarification like that means much. The ambiguity seems like the worst of all worlds. People who are afraid of the GPL won’t use the software anyway. People who act in bad faith can satisfy the terms of the license through trickery. You don’t get any practical protection over a permissive license, and you get most of the stigma of the GPL. The GPL has some success stories, places where source was actually released due to the terms of the license. I don’t know of any similar success stories for the LGPL. If you don’t want to use the GPL, just use a permissive license.

Then the question: GPL or permissive licensing?

For some time I’ve chosen a permissive license for my code. But I’m not really advocating that, and every so often I throw out a little GPL code. Underneath this is something of a disinterest in licensing. I don’t think it means much. If specifics of your license matters then something has gone wrong. People are leeching code, and/or the community isn’t providing enough benefit to encourage participation. I don’t believe that software has much intrinsic worth, and treating it like property doesn’t make that much sense to me. Licensing treats software as property, which is why I see the relevance of licensing as a kind of disfunction. But there is the outside chance that it’s really just a big project, or that the project is being participated in by rivals. But nothing I work with is like this, and it’s pretty uncommon generally, and anyway those projects all have their licensing pretty much figured out.

The GPL does seem to serve a constructive purpose when rivals have to cooperate in some fashion. It makes a kind of demilitarized zone where everyone has to work for the collective good. But even this is a sign that you can’t trust the participants. This is somewhat reasonable, because you can’t actually trust large corporations with faceless programmers working on the code. But even that’s a kind of disfunction, because you can at least kind of trust large corporations with programmer employees who have a public face. The individual programmers are going to be very uncomfortable participating in any kind of Machiavellian conspiracy. Good open source projects are a coming together of individuals. Institutions are not effective participants. There are however scaling issues with individual participation: particularly that only a minority of developers are actually inclined to participate in this way. An institution can pay someone to work on something they don’t care very deeply about, and that person can still do useful work. But they won’t be part of the community.

Outside of this mediation of rivalries I’m not really sure what the GPL provides. It protects users, because the terms of the GPL ensure that users get code. It doesn’t actually ensure that code is made public, though it does give developer employees leverage to make their work public. There’s some urban legends about the viral aspect of the GPL, but that’s really just bullshit. Code doesn’t magically get relicensed just because there’s a little GPL in the mix. The GPL police won’t show up at your door and confiscate your computers. The only people who make those claims are GPL-haters like Microsoft. Who should you trust: the FSF or Microsoft? If you answered "Microsoft" then you need to get your head screwed on straight.

Unfortunately there is a stigma about the GPL, and there are competent developers that avoid all GPL software. I find this frustrating, especially since I don’t actually care about licensing, and so this adds a point of contention that I don’t really care to pay attention to. Some people in this situation then become GPL haters, because they feel like it’s all GPL’s fault. But the GPL didn’t create the licensing situation — proprietary software started this. Direct your hate where it belongs. The GPL ain’t it.

If my libraries were GPL I doubt I could leverage that to make other people open source their code. But I know I would alienate some people, and as a result I choose a permissive license because it’s just strategically advantageous.

Applications I think are different strategically. You don’t just swap an application in and out like you might a library. If you choose to use an application, then often the licensing is a secondary choice. The licensing ultimately only really applies to extensions. Libraries also have a different pattern of acceptance. Stubborn GPL haters can in effect veto a library in many projects (this is because stubborn GPL haters are viral). But in an application their influence is much smaller. Developers concerns like architecture and choice of libraries are not what drives an application. An application is driven by its functionality. If you have a useful application then licensing tends to be a secondary concern. The licensing tends to define the community in some sense as well, and as a result there’s a kind of opt-in consensus. And I think the GPL in these cases really does create an environment of collaboration around extensions. It has a real benefit.

This seems to be true empirically as well. GPL’d libraries don’t tend to get very popular. People will do crazy-big projects just because of licensing issues. GPL’d Applications seem to do quite well, with examples like WordPress, Plone, The Gimp, and even the Linux kernel, which is closer in structure to an application than a library.

I haven’t studied the GPL v3 very closely, but it seems useful to me. Software-as-a-service has the potential to make the GPL irrelevant (especially for the kind of code I write), and could put the GPL in the same category as the LGPL where it retains its stigma without offering any practical advantages. Version 3 makes the permission/GPL choice seem useful, but so far I’ve never made any truly conscious choice between version 2 and 3.

If you are thinking about choosing a license, or thinking about choosing software based on the licensing, maybe these thoughts will be helpful in your own thinking. And please don’t GPL-hate in the comments, I’m not interested and if you feel the need then go write your own post.

Listening to Tavis Smiley’s show tonight, and the segment My America where they talked about gun violence. At one point they quote a man who lost his brother to gun violence:

You can be in a club and bump into somebody on accident, a little of your liquor, a little of your water spill on their coat, now, you go outside, he got five or six people out their because you spilled your damn drink. Which, a person should be able to say, "man, my fault dog, I apologize, you know how it is." You got people that just ain’t gonna be right, man.

Tavis Smiley: So you take that, you put guns into the equation, that changes mediation efforts dramatically.

Several times they talk about how small matters of respect lead to violence. The conclusion is that guns are the problem.

I don’t really know what to do with this. In my life (and I suspect all of your lives) issues of respect do not lead to violence. As a result I have a hard time thinking of this as a gun problem.

OK, so it’s a violence problem. The other thing that gets me is there’s this strong undertone to this conversation that "we aren’t doing enough." This attitude is of course the norm for an NPR show. But it’s not we — I, and everyone I know is not part of this we. My "we" does not resort to violence. My "we" does not project respect into minor social interactions. When I say it’s not "we", I don’t think it’s just that I tuned into the wrong radio show — am I being recruited into this "we"? Do they really think listeners are part of this "we"?

There is no reflection in these shows about why this (whatever the issue of the show) is a general problem. Of course most talk shows tend to generalize wildly, to turn every anecdote into a sign of some change in culture, some disease of our society, something more than just an anecdote. (Though some good NPR shows do not attempt to generalize anecdotes at all.)

There’s a strong attitude, in this show and others, that this is a problem for us all to solve. Why exactly is this a problem for me to solve? Why is this a problem for government to solve? (I’m not a conservative, but I feel it’s unfair that only conservatives seem to be able to ask that question: why should government solve this?)

I don’t ask these questions rhetorically (and maybe that makes me different from the conservatives, who tend to only ask questions rhetorically). There may be a good answer to these questions. But it’s far too easy to say "we must do something about this" without saying who and why. We (especially those of us who listen to NPR) are all far too fatigued with the constant admonitions that not enough is being done, and something has to change. This kind of approach is not an effective call to action.

And it’s yet another thing trying to make me feel bad for something that’s not my fault. And dammit, it really isn’t my fault!

My old do-it-yourself framework tutorial was getting a bit long in the tooth, so I rewrote it to use WebOb. Now: the new do-it-yourself framework.

So I promised some more technical discussion of App Engine than my last two posts. Here it is:

Google App Engine uses a somewhat CGI-like model. That is, a script is run, and it uses stdin/stdout/environ to handle the requests. To avoid the overhead of CGI a process can be reused by defining __main__.main(). But while a process can be reused, it might not be, and of course it might get run on an entirely separate server. So in many ways it’s like the CGI model, with a small optimization so that, particularly under load, your requests can run with less latency.

This part is all well and good. I’ve already come to terms with servers going up and down without warning. But the environment itself has a number of other restrictions. It seems that App Engine is providing security in the language itself. The interpreter has been modified so that code is sandboxed, with no ability to write to the disk, open sockets, import C extensions, and see quite a few things in its environment. It’s these things that are a bit harder to come to terms with.

While they claim it supports any Python framework, these restrictions don’t actually make it easy. So for the last few days quite a few of us have been hacking various things to get stuff working.

The first thing people noticed is that Mako and Genshi didn’t work, because they use the ast (via the parser module) to handle the templating, and that module has been restricted. Apparently arbitrary bytecode is not safe in this environment, and so anything that can product bytecode is considered dangerous. From what I understand Philip Jenvy has been working on Mako and the trunk is currently working. He’d already been doing work to get Mako working on Jython, which had similar issues. Genshi is also in progress and fairly close to working, though with some missing features. Genshi has the harder task as Mako was primarily reading the ast, while Genshi was writing it.

The first thing I noticed is that Setuptools doesn’t work. I’m flattered that one of the only 3 libraries included with App Engine is WebOb, but of course I am more enamored of a rich set of reusable libraries. Setuptools didn’t work because several modules and functions have been removed — this like os.open, os.uname, imp.acquire_lock, etc. Some of these are kind of reasonable, while others are not. The removal of many functions from imp doesn’t really make sense, for instance (I think the motivation was the difficulty of auditing the implementation of those functions, not that the functionality itself is dangerous). And while some functions can’t be used in the environment, the fact you can’t import those functions is more problematic. For instance, The Setuptools’ pkg_resources module has support to unzip eggs when they are imported. App Engine doesn’t support importing from zip files at all, and you certainly can’t unzip to a temporary location. But withoutthe necessary modules and objects pkg_resources won’t even import.

To work around this I started a new project: appengine-monkey, which adds several monkeypatches and replacement dummy modules to the environment to simulate a more typical environment. It’s just a small list so far (mostly in this module), but I expect as people experiment with other libraries the list will increase. For example, I would welcome implementations of things like httplib on top of urlfetch in this library. (Implementing httplib and stubbing out parts of socket would probably make urllib run.)

But the good news is that Pylons is pretty much working on App Engine, as is Setuptools and you can manage your libraries using virtualenv.

The instructions are all located in the appengine-monkey Pylons wiki page. Please leave comments if you have improvements or problems with that process. I also welcome contributors and developers to the project itself — this is a project for expediting App Engine development, it is not a project I care to champion or control. Or support to any large degree.

One ticket which is rather important is the apparent maximum number of files and blobs: 1000. Libraries involve lots of files, and the base Pylons install is only barely under this limit. Now I just wish I could use lxml, but that’s probably going to be a long time coming.

So, I was at a university the other day, talking with some people about a sprint project, and there was a student there. He was somewhat eager to write "algorithms". I’m not sure what that means exactly, but I was reminded of him because I was just about to write a function to make an absolute path relative and this somehow felt like an "algorithm". That’s not how I’d describe most of the coding I do.

But I digress… this student seemed to be excited about his ability. Well, okay, young programmers tend to be very eager, and often overestimate their ability (or the difficulty of the task). For students in this position I think it’s good to let them take on large projects, so they can get a better concept of their ability. This student seemed to take to heart the idea that a typical programmer produces 10 lines of working code a day. People who hear this are inclined to think "damn, I can write way more than 10 lines of code a day!"

I suspect a more fair way of thinking about this is that on average a programmer gets 50 lines of code into production a week, or 200 lines a month. Sure, lots of people are more productive than this, but the actual lines of code that go into production are usually way smaller than it feels like during the process that goes into getting that code there. That student had probably never gotten a single line of code into production. I’m pretty sure he wouldn’t have appreciated what that getting code into production is different than just hacking out some code.

The kid didn’t end up hanging around. I don’t think we had any algorithms for him. That’s too bad, maybe he could have seen what programming really looks like.

Another what-is-programming-really-like story… Emily was working on her blog and editing some templates to change the feed URL. It didn’t work, and I took a look at it and showed her that somehow a space had gotten into the URL, so she had to go back and correct them all. She found this frustrating. I pointed out to her that this is exactly what I spend my days doing.

Some people seem angered when people call writing HTML "programming". Personally I do not. Most programming is a lot more like writing HTML than it’s like writing a compiler. I bet writing a compiler is a lot more like writing HTML than what most programmers think it’s like to write a compiler.

I like this phrasing of the debate about App Engine’s role, from Doug Cutting: Cloud: commodity or proprietary? (via). (Well, I like it more than the sharecropping phrasing referenced in my last post <http://blog.ianbicking.org/2008/04/09/app-engine-and-open-source/>.) I guess I’m excited about this because like Doug I do want a "cloud" of sorts, and this is a move towards that in a way that makes sense to me. Maybe to state my motivations more clearly: I hate computers. I really hate them a lot. I dream of some world of Platonic ideals where software just exists, and existence that state it works. App Engine feels like a strong move in the direction of computers-not-mattering. What does App Engine run on? I don’t care! Where is the server located? I don’t care! What is BigTable? I am comfortable thinking of it only in its abstract sense, an API that works, and I don’t know how, nor do I need to know how. I don’t need to know these things if they just work. Always. Totally reliably. I’m not shy about digging into code. I tend to be light on my reading of documentation, because I’d much rather open up source code and poke around. But when something can work so reliably that I can treat it as completely opaque then it’s a blessing, because I can start to forget about it and think about bigger goals.

There was a time when people were concerned about Big Endian vs. Little Endian in computers. You had to think about this sort of little detail when programming. People formed actual opinions on which way was best. To think! Similarly XML has removed a large number of fairly pointless format decisions people might make. There is progress. Commodity hosting (reliable, consistent hosting, better than what we have now) feels like similar progress.

Unlike Doug I’m optimistic that App Engine is a move in the direction of a commodity cloud. The APIs seem to lack the stench of proprietary APIs. They are based on Google services, but they reasonably abstract and reasonably minimal. This does not seem like some kind of "play" (and the developers’ seem to be reassuring about their intentions). There’s a tendency to be cynical about any company’s work, that it has underlying intentions that are at odds with any competitor (present or future), that anything good is just a loss leader meant to hook you in so they can squeeze you later. Some companies deserve such cynicism. I don’t know that this company, or this team, deserves that.

Mind you, I don’t say this from a Best-Tool-For-The-Job perspective. I believe in the moral foundations of Free Software, not just the technical advantages of its development process. But I’m a Utilitarian, and it doesn’t make me uncomfortable that not everything is Free if I think it’s a step forward for overall freedom. I think App Engine has the potential to be a very powerful tool for enabling people to create and use web applications. If it was great, but still a dead end, then maybe that wouldn’t be good enough. But I don’t think this is a dead end.

This is about Google App Engine which probably everyone has read about already.

I’m quite excited about it. Hosting has been the bane of the Python web world for a long time. This provides a very compelling hosting situation for Python applications.

I’m not as interested in this from a competitive perspective as I am from a simple this-is-awesome perspective. Regardless of how this positions Python relative to other languages, this is something Python needs. But even looking beyond that, I think this is something the open source world needs.

Open source web development is in a funny place. There’s a lot of reasons why web programming is a good domain for open source. The barrier to entry for web development is extremely low. Developers have choice in their tools, as browsers don’t really care what software you use so long as you serve up HTML. This leads to experimentation and excitement and the kind of self-direction that is very motivating to developers. It leads to the kind of personal excitement that underlies most open source development.

Despite this, open source web application development doesn’t seem sustainable. There’s some applications, sure. WordPress, Trac, MediaWiki, MoinMoin. But most wiki software doesn’t have a vibrant community. Many a bug tracker has fallen by the wayside. Blog software projects have a horrible time building a viable community. Other website software hardly gets anywhere at all. A lot of the development that might appear to be application development really is more like a framework when you look closely (e.g., Plone, Drupal).

I think deployment concerns are a huge part of this. And, given its better deployment story, it’s no surprise PHP is the basis of most viable open source web applications. Being interested in a project requires that you be able to use the product (and usually use it casually, as that’s the point of entry for many developers). Right now most people can’t use open source web applications.

But people can use hosted applications, and that’s where all the effort has gone in the past few years. I am comfortable saying that Trac is a better issue tracker than Google Code’s issue tracker. But I’d probably recommend Google Code to someone starting a new project, because it’s so much less work. Similarly I’d try to dissuade most people from installing their own blog software. I still don’t know what to tell people about a CMS.

Many people are excited about how far up you might be able to scale something based on App Engine, but (like Dave) I’m excited about how far it could be scaled down. For the majority of sites the free quota will be more than enough. But that alone isn’t the point: there’s lots of free services people can use. The difference here is that the free services can be modified and controlled by the anyone who signs up and installs an application.

From the perspective of open source it’s a bit awkward that the platform itself is proprietary. Questions about sharecropping are a valid concern, but I’m optimistic about the ultimate outcome. The SDK is under a permissive open source license, and the APIs are all reasonable enough that they could be reimplemented with open source backends (maybe without the same scalability, but that’s not the aspect I care about anyway). Perhaps the BigTable APIs will serve as the basis for future storage APIs.

But even if other people make compatible implementations of these APIs, would it matter? If Google offers free hosting, is someone else really going to be able to provide a better hosting option? Or would these other implementations just be strawmen, a way to show that It Could Be Done? If the libraries are just written to prove a point, I can’t see them gaining much traction. But I think these could be viable as there are other constraints to the App Engine environment that people may want to escape at some point in their application development.

As to the details of App Engine? Can you run Pylons or Paste on it? Well, that’s a topic for another post.

I just saw this json-rpc recipe go by as a popular link on del.icio.us. It’s yet-another-*Server based recipe (BaseHTTPServer, XMLRPCServer, etc). I don’t know why people keep writing these. WSGI is in all ways easier, clearer, and more useful.

So I figured I’d give it a go myself, using WebOb. Then I figured it might make a good document, and I annotated it and turned it into a tutorial. It’s also an example of using WebOb as a client library.

I’ve added several tutorials in the past months, which I thought I should also point out. The wiki example is fairly pedestrian, but shows how to do typical application-style development with WebOb. A more interesting exampe is the comment middleware example, which shows how much easier it can be to write middleware using WebOb than traditional WSGI middleware.

I think WebOb’s particular strong points are with middleware (where it makes middleware vastly easier to write) and web services of various kinds (RESTful or not).