A talk given at PyCon Argentina and PyCon Brazil, 2009.
Web development sucks.
It’s true: web development, at its worst, is difficult, repetitive, and boring. The tools we have suck. At best, they make web development slightly less painful, but we’re a long way from making web development awesome.
The history of web development tools is a history of trying to solve this problem. It’s a history of asking, “how can we make this suck less?” It’s important to understand this history, because we can look at past trends and use them to predict the future.
That’s exactly what I plan to do. I want to answer three questions:
- What sucks, now, about web development?
- How will we fix it?
- Can we fix it with Python?
To do so, I’ll start with ancient1 history.
A brief, opinionated history of web development
In the beginning, TBL invented the Web, and it was good.
I like to think of this time as the “Stone Age” of web development, an age characterized by clumsy, difficult tools. We wrote HTML, by hand, usually in a text editor. The concept of dynamic web pages didn’t exist: if you wanted to publish a thousand stories online, you wrote a thousand .html documents.
This sucked. Obviously.
And it led to an obvious question: what if we could generate HTML programatically, like from a database or something?
And thus was CGI born, and the Bronze Age of the web began. We had tools, finally, to automate some of our repetitive tasks. We could handle user input! Connect web pages to databases! Flow content through templates!
But CGI sucks. It’s horribly slow, for one. Worse, it encourages incredibly bad development style. Either you have dozens (hundreds!) of .cgi programs, with all sorts of repetitive code… or you end up with a single, gigantic, monolithic go.cgi. CGI usually proves horrifically hard to maintain. The fact that most CGI of this era were written in Perl — or, worse, C — doesn’t help.
So, again, smart people started asking questions. At first, we asked, “how can we make CGI suck less?” Note that this isn’t a very big question — it’s not really a rethinking of how web development should work — but it still led to a leap forward: the first generation of application servers.
I’m talking here about things like mod_perl, mod_python, and especially PHP. PHP, which is essentially “CGI done right,” quickly become overwhelmingly dominant, and the Iron Age of the web began2.
Now, I mentioned before that the questions that led to PHP weren’t very good questions, and so the leap forward wasn’t all that dramatic. The Iron Age is very similar to the Bronze Age: the tools all look mostly the same, they just use slightly better tech. That is, PHP suffers from most of the same problems as CGI, but to a lesser extent.
The biggest problem, at least to my mind, with the first few ages of web development is that the mindset is essentially page-oriented. All we did in these early years was trade page.html for page.cgi for page.php. We still represented web sites as a collection of pages, written in a string of improving languages.
So the real revolution came when we started to question this basic assumption. “What if,” we asked, “we could think of these things as applications, not pages?”
This question led directly to the creation of Django, and to the Industrial Revolution: Web Frameworks.
Now, technical revolutions happen organically. Take the printing press: though Gutenberg is credited with its invention, in fact the press was simultaneously discovered by at least two other inventors3 in Europe around the same time. Indeed, it’s actually inaccurate to credit any of these men with building the first press: press-printed material dating back to the 7th century has been found in China and Korea4.
Like the printing press, then, frameworks existed long before the current crop (WebObjects is just one example). Like the Industrial Revolution, the Framework Revolution happened in many places, and in many different ways. I don’t want for a minute to pretend that Django was the first framework — or among the first — nor was Django born in a vacuum. Django is, like the other frameworks of our time, a product of the age and of these questions about web development.
The Industrial Age: Web Frameworks
Now we find ourselves in the Industrial Age, the Age of the Framework. Since I’m talking about “frameworks” quite a bit, I think it’s worth a bit of time to clarify what I mean. As I see it, the main characteristics of a modern web framework are:
They operate at a high conceptual level. Instead of thinking about HTTP, HTML, and web “pages”, frameworks allow us to think and to operate at the level of web application. This means less code, and it also allows us to be much more ambitious about what we’re building.
Frameworks provide much larger building blocks.
I like to use a construction analogy: traditionally, most houses are stick-built: you just nail together a whole bunch of lumber, one stick at a time, until the house finally appears. The raw materials are simple: wood, nails, glue, shingles, bricks. You’ve got all sorts of flexibility, but construction takes a long, long time.
Today there’s another option: factory-built homes. Here, the house is built in huge sections in a factory, mostly automatically. Each room, pre-built, is loaded on a truck and the a crane puts the house together on-site. Architecture is more constrained — you have to put the house together from the array of room options supported by the factory — but you can literally be moving into the house 30 days after signing off on the final blueprints.
Frameworks encourage rapid development. It’s no coincidence that the Age of the Framework is also the Age of Agile. Agile, XP, Scrum, etc. — frameworks are at their best when used in a rapid-iteration style.
Good frameworks are open source. I don’t think I need to justify this point to this particular crowd, so suffice to say that it’s no accident that there aren’t any proprietary frameworks with any real following to speak of5.
Finally, good frameworks make development fun. Business folk like to think this is a silly requirement. It’s not. The best thing about the web framework world is our sense of fun: fun motivates, leads to experimentation, and hence to innovation.
What’s next?
I’ve described where we are now… so what’s next?
The best way to predict the future of web development, I think, is to keep asking ourselves the question that led to all the past advances: what sucks, and how can we fix it?
So: what sucks about web development?
Inter-operability
Modern web frameworks suck at inter-op.
Frameworks are good. But frameworks inevitably lead to lock-in. Lock-in is bad.
It’s important to realize that the most important kind of inter-operability is with the user’s code, and frankly web frameworks often suck here. A basic truth of software is that as it grows and matures it becomes more and more domain-specific, and less and less generic. I’ll talk more about this below; the important part for now is to realize that general frameworks should be able to cede control to domain-specific replacements as the stack grows. For the most part, frameworks don’t.
Of course, most people think of inter-op in terms of inter-operability between multiple frameworks. Nobody’s doing very well here, but unfortunately the Python web world’s worse than average. There’s a great deal of fragmentation in the Python web community, and frankly Django’s not helping. That’s a bug in the Django community, and there are similar bugs all over the Python web world. We need to fix these.
WSGI is helping here; WSGI’s the best thing ever to happen to Python web development. We can’t rest on our laurels, however: WSGI’s got some serious problems. They’re off-topic here, so I’ll simply point you to James Bennett’s Let’s talk about WSGI and say, “ditto.”
I should also mention Rack. Rack, in a nutshell, came about when the Ruby world, facing similar problems we’d faced in Pythonland, created a WSGI-inspired web gateway tool. It’s been a resounding success: Rails 3 is being rewritten in Rack. Rack is frankly a bit better than WSGI; we Pythonistas should be embarrassed by that.
The big problem, though — the elephant in the room — is that gateways suck, too. Gateways aren’t APIs. There’s a limit — and it’s a low one — to the level of inter-op you can obtain when the only interface you have is a gateway. Even if we improve WSGI — and we should — it’ll only take us so far.
Even worse, tools like WSGI and Rack do nothing to help inter-language inter-operability. I’d really like to write parts of my application in Python, parts in Clojure, parts in Ruby, and even parts in Perl. Things like web proxies, SOA, ROA, and language VMs help, here, but since gateways aren’t APIs there’s only so far we can go.
This is going to be a hard problem to fix, even if we only focus on Python. We’ve got a bunch of disparate communities, all comprised of volunteers. Very few people have overlapping knowledge, few know how to navigate multiple community standards, and fewer still have the impetus to work on inter-op. Nearly nobody is thinking about multi-language inter-op.
But this stuff is incredibly important. If Django fizzles, I’ll be sad. But if Python fails as a web language I’ll be devastated.
Rich web applications
I’m extremely excited about HTML 5. In fact, I think it could be the best thing to ever happen to web frameworks. If web apps can truly replace desktop apps then frameworks are going to be the place to be, and Python could kick some serious ass here.
Right now, though, the current crop of tools suck at creating rich applications. The current state-of-the-art is pitiful. The two approaches I’ve seen seem to be either building parallel MVC layers on the client and the server and then mashing them together somehow, or else inventing a tightly coupled back-end-with-generated-front-end framework like GWT or SproutCore. Neither approach makes me all that happy.
For example, take a look at 280Slides. It’s an amazing piece of web tech — the browser truly disappears; it’s hard to tell that you’re not in a native desktop app. It’s amazing.
However, the developers believed that 280Slides would be literally impossible to write using any of the current web tools. They not only built their own framework, Cappuccino; they actually invented a new language, Objective-J! If this is a trend, it’s worrying.
Handling complexity (a.k.a. the deployment problem)
It’s a well-recognized fact that web applications are getting more and more complex, and the list of things you need to successfully develop, deploy, and scale a web app is getting longer and longer.
It turns out that writing the app is now the easy part; managing the rest of the stack you need for successful deployment can be nearly impossible. In other words, we’re all ops people now.
Some time ago, Leonard Lin collected this list of all of this “other stuff” you need to worry about after developing your app:
- API Metering
- Backups & Snapshots
- Counters
- Cloud/Cluster Management Tools
- Instrumentation/Monitoring (Ganglia, Nagios)
- Failover
- Node addition/removal and hashing
- Autoscaling for cloud resources
- CSRF/XSS Protection
- Data Retention/Archival
- Deployment Tools
- Multiple Devs, Staging, Prod
- Data model upgrades
- Rolling deployments
- Multiple versions (selective beta)
- Bucket Testing
- Rollbacks
- CDN Management
- Distributed File Storage
- Distributed Log storage, analysis
- Graphing
- HTTP Caching
- Input/Output Filtering
- Memory Caching
- Non-relational Key Stores
- Rate Limiting
- Relational Storage
- Queues
- Rate Limiting
- Real-time messaging (XMPP)
- Search
- Ranging
- Geo
- Sharding
- Smart Caching
- dirty-table management
Yes, a modern web developer really needs to understand this stuff. Yikes.
The good news is that there’s open source software to fill all of these needs. The bad news is that they’re all immature, disparate pieces with no connections to each other. Getting even half of this stuff up, running, and integrated is a monumental task.
There’s a huge opportunity here for Python. Python’s historically been used as a “glue” language, though recently we’ve tried to de-emphasize that aspect. It’s nothing to be ashamed of: Python’s a very good glue language.
Python could easily be the glue that keeps this huge stack from toppling over.
Scale
Internet usage is growing explosively. Worldwide it’s doubled twice since 2000… and global penetration is only about 25%6. This number’s just going to keep going up.
Meanwhile, web sites are getting a lot more complex. Think back to 2000 — could you have even imagined a site like 280Slides then?
Meanwhile, traffic is growing. The average user is spending more and more time on the web, and think about what’s going to happen as the mobile web explodes.
We’re going to have to learn to deal with more and more and more traffic. And frameworks suck at scaling.
Frameworks are very good at generic tasks. They’re meant to be: they abstract away common difficulties. But as applications grow in scale, they need to get more and more domain specific to be able to deal with scale. There’s a direct correlation between the size of the site and how specific it is.
This usually breaks down as follows:
- You develop your first little toy app using Framework X. (In the Django world these seems to be a blogging app — it seems like at least 75% of Django developers have built their own blog app.) This usually goes great.
- Happily successful, you develop a product with the framework, and launch it. This usually goes well, too — sites at the initial-launch stage are still very similar to each other, and frameworks are great here..
- As your site grows, you start to feel a bit of pain, and need to replace some bits of the framework with domain specific bits. This usually isn’t too bad: most frameworks, Django included, are modular enough that you can easily swap out the more common non-scalable bits.
- Then one day you become Twitter, and all hell breaks loose. You end up having to essentially ditch the framework and re-write everything, from scratch, in very very specific ways, just to deal with the crazy, mind-boggling amounts of traffic you’ve got.
Frameworks work incredibly well to get you off the ground quickly… and then usually fail miserably when faced with the specific needs of big sites.
This is an impossible situation for framework developers: by optimizing for a quick start, by focusing on common needs, we’re essentially guaranteeing future failure. Remember the “Rails doesn’t scale” pseudo-controversy last year? I guarantee it’s only a matter of time until there’s an angry “Django FAIL” moment.
Frameworks ought to gracefully fade away as you replace them, bit by bit, with domain-specific code. (This is what I meant, above, that inter-op is also a scaling issue.) Right now, they don’t.
Concurrency
Of course, if you’re talking about scale, then you need to talk about concurrency. That’s right, I’m gonna go there. I’m gonna talk about the GIL. Don’t worry, though, I won’t dwell or complain.
First let’s look at some processors, shall we?
Today, right now, you can buy a top-of-the-line Intel Nehalem for about $2,000. It’s got 2 hardware threads per core, and it’s available in an 8-core configuration. This means 16 hardware threads on a single slot, so you can easily build a box with 64 hardware threads (4 CPUs, 8 cores per CPU, 2 threads per core).
Of course, if you want to get really serious you could buy something with Sun’s UltraSPARC T2 (a.k.a. Niagara). This chip has 8 cores, 8 threads per core, and you get two of ‘em in a single box, so that’s a whopping 128 hardware threads per machine. Yes, the future of this machine is in doubt7, but Sun’s been on the leading edge of concurrency for quite some time. It’s only a matter of time until Intel and AMD catch up.
Obviously concurrency is going to be a Very Big Deal in the future. It already is.
Much of my thinking about this comes to me from Ted Leung. I look up to Ted, and I’m sad to tell you that Ted says we’re screwed. I’m afraid that I’m starting to agree. To some extent the “shared-nothing” architecture of most web applications mean that we can just StartServers 128 to deal with 128 threads, but as applications grow you’ll usually need to start throwing up “shared nothing.”
Most languages can really only saturate a single core, and if you can only use a single core you’re in a lot of trouble.
Now, there’s lots of exciting work going on in the concurrency world today. Cool shit like Actors, STM, persistent data structures, dataflow, tuple spaces, and more. Ted’s A survey of concurrency constructs is a great introduction to these terms if you’ve not heard ‘em yet.
Unfortunately, nearly all of this awesome work is going on in relatively obscure languages like Scala, Erlang, Clojure, or Haskell. There’s almost no forward motion in the Python community. Yes, I know all about Twisted, Kamelia, Eventlet, etc.; these are all just twists on threading or IO-based concurrency; there’s very little that’s really new going on in Python.
And though it’s sometimes considered taboo to say it, we have to be honest: this is partially the GIL’s fault. It’s not clear to me weather the GIL would preclude, say, STM, but it almost doesn’t matter: the existence of the GIL basically sends anyone interested in concurrency running for greener pastures.
I have hopes for Unladen Swallow: the prospect of removing the GIL from Python is a promising first step. However, really all we get from that are better threads, and threads suck as a concurrency mechanism. I want my Actors, dammit!
This is where us web guys really need your help. We operate at a higher level of abstraction so much of the time that we’re simply not qualified to figure out how to make concurrency better in Python. At least, I’m not. I frankly barely understand threads after a decade of using ‘em, and there’s no way I’ll be the one to implement STM in Python.
Halp!
In the year 2020
By way of conclusion, I want you think try to imagine web development circa 2020. That’s no arbitrary year: it’s also Last Call for HTML 5, so it makes sense to think about what the web’s going to be like when HTML 5 is mature. When we’re finally developing the types of apps we’re just starting to dream of today.
I’m not sure I’ll be using Django in 2020. I hope I will, of course, but it may be that Django simply can’t adapt in the next Age of web development.
However, if I’m not still using Python in 2020 I’m going to be seriously pissed off.
Joel tells us that good software takes ten years, so I think we need to start right now. How can we work to make Python the language of choice for the developers of 2020?
First, we need better inter-op. A better WSGI — WSGI 2? — will help, but we need more communication and more APIs that work between frameworks.
The Django community needs to do a better job here, and I’m taking responsibility for that. Keep complaining to me about Django’s lack of inter-op, and I’ll keep working to fix it.
But more than that we need real leaders here. Someone who can show us a way forward, and keep an eye on the bigger picture, not just focus on a single framework.
A Python web-inter-op BDFL, perhaps?
We’ve got to get out in front of HTML 5. There’s a huge opportunity for Python to be the backend language of choice for HTML 5 web applications. We need to start thinking about this now.
We’ve already made a great transition from thinking about “web pages” to thinking about “web applications.” It’s time for a new transition, for us to start thinking about a holistic “web site,” and all it’s associated related tech. Again, there’s a huge opportunity for Python: it could be what binds our stacks together and makes deployment pleasant again.
I dream about a complete stack deployment framework, all tied together with Python, probably built around WSGI 2.
We need to be thinking about scale from day one. This means being incredibly skeptical of our own work, and continually asking ourselves where it’s going to fail. We need plan for the day that our framework will be phased out.
And holy crap we need better concurrency.
Thank you.
| [1] | Well, by web standards, at least. |
| [2] | Much of the web, unfortunately, hasn’t progressed much beyond this point. PHP is still by leaps and bounds the most popular and widely-used web technology. The future may be here, but it’s certainly not evenly distributed yet. |
| [3] | At least two others, Procopius Waldfoghel of France and Laurens Janszoon Coster of the Netherlands, may have been working on their own presses around the same time as Gutenberg, and those are only the ones we know about today. |
| [4] | Wooden movable type in China dates to the 10th century, and there’s good evidance that both metal type printing presses were used in Korea as early as the 13th century. If you want to know more, Wikipedia’s History of typography in Easy Asia is as good a place to start as any. |
| [5] | Except maybe for ASP MVC.NET. It’s hard to know how popular Microsoft technologies are because there’s no real community to speak of, and because Microsoft tends to lie about penetration. |
| [6] | See http://www.internetworldstats.com/stats.htm. |
| [7] | You have to wonder if Sun would have failed if we’d been able to write software that made us actually need this many cores… |
Comments:
Amen!
I've loved python for a long time, but sadly it feels like it's falling behind to those "obscure" languages you mentioned. Which is why I've started learning clojure, and even poked just a bit at Scala. Just as Python was a scrappy new language at one point, now these are taking the technical fore.
It really boils down to technical debt slowing down/stopping forward motion, and I fear Python may be near that point, in spite of the cleanup in python 3.0
If your favorite framework were designed to work independent of Content-Type, then you'd be prepared for HTML 5 already. ;)
For actors dammit! there is Stackless Python. Already proven in a large commercial concurrent system.
I think something to look at is Seaside. It's written in Smalltalk, but it turns the web on it's head and makes it more about the functionality, and less about burdening the programmer with the request/response cycle. Taking a look at something like DabbleDB, you can see the insane flexibility that it present. The interesting thing is that Avi, the author, is a big Python and Ruby fan, and in fact tried to write it in Ruby, but found it totally unworkable.
It's not just about concurrency. You also need to offload as much as possible to be done outside the request/response cycle. None of the major dynamic web frameworks address this need yet, which means we all keep re-inventing job queues and the asynchronous messaging bus in a million ad-hoc ways.
The "enterprise" guys get it (Message Beans, anyone?), and the old-school UNIX server developers got it (fork() for everything!), but we seem to have lost something in the move to the web.
I'd be curious to know what you think we could do to help bridge this gap. It seems to me, as a new Djangonaut (recently converted from Rails) that some sort of deferred signals could be part of the answer, and Celery has some intriguing possibilities, but I think the final shape of the "async web stack" is still somewhat ill-defined.
There's been some discussion of this already over at my blog; maybe we can keep the thread going:
http://rcoder.net/content/n...
I hate to say this, because I came to both Ruby and Python out of frustration in waiting for it, but it seems to me what you want is Perl6 or at least Parrot. Oh I know it's been TEN DAMN YEARS , and I'm not that bitter. Still, Parrot is the path to the goal you specify.
The problem with Python has nothing to do with technical debt at this point. It's simply that Python was designed to solve a specific set of problems, which it does quite well with. The mass concurrency problem came later and is thus far outside the range of what Python excels at. Can Python adapt? Probably. However, if they adapt the way C++ adapts to new challenges - by effectively only bolting on new functionality - Python will end up a frankenstein monster like PHP or C++. In 2020 we will probably have moved on to a newer language for web programming, probably one that is only just now starting to be designed. At that time, we will still have Common Lisp users arguing about what they have to do to make their language more relevant and telling the world about how much better their language is than all of them.
Sadly; I don't think you're wrong, but I also don't think this is a strictly technical issue (concurrency + python). Part of the fundamental issue here is one of People - the only, ONLY reason unladen-swallow has any traction is because Mr. MoneyBags Google stepped up and said "we need this funding it ok go".
If someone sat down, and actually *worked* on the interpreter - and I'm not just talking about unladen/removing the GIL, but added green threads/processes ala erlang and showed it to work, people would love them.
But who has the time? Heck; we don't have enough bandwidth to flush out the patches posted in the tracker, much less the ability to take on something as big as reworking an interpreter.
This is what I ran into in my own efforts and rants on this subject. It's not that no one doesn't want this - no one has the time, or money. We all want actors, real threads and so on. We all want the Unicorn - but no one has the time, skill or money to do it.
Python "works well enough" right now - and most of us get paid working with Python as it sits today. I agree with everything you're saying, hell - I ranted and raved and fought for it myself. I just simply don't see it happening.
Heck, I'd do it myself if I knew who to make an interpreter (I don't) had the money (I don't) and the time (I don't). Even the one person who I knew working on something like this on his own (Adam Olsen) gave up for lack of participants.
All of these cool new languages (and the renaissance of some old ones) are being done by people who are funded, and have time. Neither of which are in high supply in our community.
Right now, Unladen swallow is our *only* hope to get this stuff off the ground. I encourage anyone who can to get involved with them.
I am inspired. I've never programmed or developed a thing in my life. It took me a while to read this article as I used Wikipedia to supplement my understanding into something I'd qualify as comprehension.
As a very young person who now wants to help carry this torch from scratch; from my completely innocent point of view, I see no reason Python should be used. Are you just rationalizing the effectiveness of something you're familiar with? What is the key feature Python provides that would make it the language of choice for the future of the web? I would like to know what I should be realistically expending my time and energy on in this endeavor.
It seems to me that the problems you are trying to solve really relate to the fundamental architecture of the web.
Nearly everything you mention has been developed - as you in fact imply - as a band aid for the previous set of problems.
I suspect that the type of solution you are looking for would require a fundamental poke at the roots. This doesn't seem to me to be at all probable...
this is a smart, enthusiastic piece....but surprising, too, because i thought you were going to the obvious place- appengine.
with all respect, i'd suggest that the solution you're looking for is already here, albeit in early form. sun and intel are not on the leading edge of concurrency. google is. the future of python web development is not some set of improvements on wsgi running on a 64-core intel box. it's appengine.
look at this way- google has all of the machines, all of the data, all of the algorithms. they have corporate/user accounts, payment services, data management, document management. they have tremendous video and audio capabilities, of course, including smart video analysis and working voice recognition, lots of natural language stuff. they have mobile, enterprise (gsa), and soon desktop reach. they're working on practical things for home automation and smart offices.
they are evolving their platform with astounding rapidity. and they've decided that python is as important to their future as first win32 was and now .net is to msft, and as java was to sun. sure, they have guido and alex. they're sponsoring unladen swallow. but all of their stuff is going to show up in appengine.
they're laying the groundwork now. work queues, background processing, xmpp support, inbound email processing. soon will come better corporate doc access, access to other google-managed datasets available to your organization- crawler data, click data, ad data, social network data.
reusable frameworks and libraries- from inside appengine, without goofy things like appengine-patch- will be important. better integration between java and python applications- here's your clojure and django- will come. things like vr broadly available to developers will be a big deal. and there's a lot to be said about mobile.
this is why they don't care about apps like twitter. twitter has become a great brand, but technically, it's one guy doing a week of work in appengine 2.0, and that guy gets voice recognition- push to tweet from your gphone- for free.
that's not to say their victory is inevitable. appengine seems weird and kind of icky now, and there have been a lot of problems. but where msft used to take 3 big revisions to get something right, google clearly will go at for 1000 small revisions.
they obviously have to continue to win trust, win developers, win business, help people make money and solve problems. they have to do better at a whole bunch of stuff. and it's certainly not clear they'll succeed everywhere- a friend said they're a lumpy organization in that some parts work better than others.
but they're the elephant in the room when it comes to the future of python and web development. they recognize all of the issues above. their response is appengine.
i'm not an employee, i don't own stock, i don't have a stake in their success. this is just a constructive suggestion that you not leave them out of the conversation.
A couple random comments - must keep them brief
CGI was born without databases in mind - that came later.
Cappuccino is a completely different animal then a web framework and the comparison is a bit improper. Cappuccino and SproutCore focus on brining front end application framework design to run in the browser for complex UI. The comparison is the evolution of page -> application, as is HTML/DOM/JS -> app.framework. These tools focus on having most of the application logic in the browser code, with the server just providing persistence.
I think the emphasis on glue is a good one. The problem is that individuals and teams can only be versed in so much code - hence the tendency to reinvent wheels in a language/framework most familiar.
The other issue is just the sheer mass of app logic that needs to be reinvented in each language or framework.
PHP may be incredibly flawed, but by the time Pinax gets to where Drupal as a out-of-the-box site CMS system, some other team will be starting a new CMS system in scala that will take time just to get to the same point where Pinax is at that time, and so progress is one step back, two forward.
On concurrency and scale. There will always be a need for rapid tools that do creative things without the need to scale. For every twitter, there are thousands of sites that need not scale like twitter. These projects will still need tools. For projects that must scale, I think something python would be well suited at would be a clone of Apple's Grand Central Dispatch. See the arstechnica pages describing it, but the essence is to make it easier to utilize threads.
-Preston
Thanks for the great write up!
-- WSGI 2 --
Looks like the WSGI 2 talk started back up a day before you posted this:
http://mail.python.org/pipe...
Perhaps you should participate there? (I'm not trying to be snarky. We need the best minds in the Python web world in the same "room" talking about this.)
-- Cross Language Interop --
RESTful applications communicating via HTTP (and using JSON for data interop) seems like a popular route for cross-language interop. You can integrate disparate systems speaking HTTP/JSON anywhere in your stack whether on the frontend using AJAX or on the backend building pages via RESTful remote APIs.
Lets not go about reinventing Corba/SOAP (for over-the-wire interop) or .Net (for in-the-binary-interop) just because its a more elegant solution on paper than a mess of RESTful services scattered up & down your stack. Remember, worse is better. :-)
http://www.jwz.org/doc/wors...
-- VMs --
PS To the "Parrot is the answer" poster: I think you meant "fashionable-vm-of-the-month" is the answer. Once any of them can outperform CPython, get rid of the GIL, and offer near 100% compatibility with CPython, we'll all switch overnight. I promise.
Until then, its not really as simple as you'd like to believe. :)
-- Bug --
PPS The lines in the comment box are off in FF 3.0 on Debian Testing. Me and one other guy are pissed. ;-)
very awesome write up. everything i have to add has already been summed up by jrb (app engine) and Michael Schurter (json is great for inter op)
Thanks!
You're right; but also, you're not right. Comparing the development of web development to ages of civilization doesn't quite hold, because there are millions of people still writing dinky little PHP pages and coding HTML by hand.
I think the problem is Postel's law (which you imply) combined with the culture of copy and paste. See:
http://www.robbyslaughter.c...
Nevertheless, excellent points overall!
On the low end, using only one processor with the GIL is probably fine. In the middle you can use shared memory threads for concurrency. But at the high end you'll need multiple machines (and perhaps multiple datacenters) and you'll need a different architecture, and the GIL, threads, STM, etc., aren't an issue anymore. (App Engine is an example of designing for that high end)
If there's a new architecture for web development, look beyond the medium sized problems where you think about multicore on a single machine. It doesn't matter whether you have 64 CPUs on 2 physical machines or on 32 of them; once you grow beyond one machine you're not going to be using threads with shared memory anymore.
My own gut feeling is that progress will come from adopting a philosophy more like Unix command line tools, where data is in a very simple format and the tools have a very limited interface (instead of stdin,stdout,stderr, we'll have request headers, response headers, response output, and a transactional state change monad). Like the Unix tools, it would be possible to write handlers in different languages and have them work together, even chained together to make more complex functions. But this is just a gut feeling and I don't have anything specific to point to.
Why do you hope you're using Python in 2020? I've been a Ruby developer for the last 3 years, but I don't hope I'm still using Ruby in 10 years. I hope something has come along by then and swept away my expectations of what programming can be like and replaced it with something that feels wholly new, like has happened for me multiple times. I'd feel stagnant and dependent if I was using one primary language for 10 years.
I think it's telling that the work going on that you describe in concurrency abstraction is happening in young, "cult" languages. One day, someone will write the Rails of the Erlang world (though perhaps it'll be a "full stack" framework and not just a dev framework) that makes perfect use of Erlang's strengths, and abstracts away or simply dismisses its weaknesses, and then a new migration of developers will begin.
That's what I hope, anyway. I don't want Ruby or Python to half-ass some solution that is still fighting against threading limitations, and never planned for syntax or structures that worked elegantly with concurrency. It's gonna have to be some kind of Ruby 3, or Python 5, and they'll have to be wicked backwards-incompatible. But that's a ways off, and there's plenty of time for them to have their lunches eaten before then.
I think concurrency is a red herring. As people have mentioned, appengine is an example of addressing concurrency, and it does it without any grand reinventions of Python. If anything it is more conservative than all the alternatives.
Appengine's model could be generalized a bit, i.e., generalizing the idea of safely reusing processes with a CGI interface. FastCGI basically does this, but doesn't do it well. We could make a frontend in Python, at least as a prototype, and go from there.
I think the more challenging problems are in how to reuse applications, how to compose a site out of different applications, how to handle auth, how to handle styling, how to make sure sites stay up without intervention, how to keep them backed up, how to deploy them and develop them, how to make them accessible to designers, how to make them hackable by new users... these more mushy issues are the big ones.
If there's a language issue that might make Python incapable, I expect it will be something like the code/data split, or maybe something like how data structures are shared. With I guess Seaside being one of the few examples of trying to break that split, or functional languages handling data differently. Little stuff like concurrency isn't at the heart of Python, I don't expect it to be a blocker.
I wonder if you have looked at the http://linkeddata.org/ initiative. It is tackling the interoperability problem by sharing raw data on the web, using RDF.
If we are trying to look at the web ten years in the future, I think we can't miss that world, which uses HTTP and the Web to create a big, open, global database.
It's not the silver bullet by itself, but it is highly interesting, IMHO.
All I can say is Amen!
And to see how far we are in 2009 from the past (pre Web) just watch: http://www.archive.org/deta...
I left some initial thoughts on my blog: http://www.sauria.com/blog/...
A forgotten element in the history. The rise of CGI/ HTTP POST to handle the content came from the inability of people to create the right framework for HTTP PUT and DELETE (both clients and servers have to implement it). In the browser world this is almost doomed or maybe through JetPack in Firefox.
I wonder if raising awareness about http://w3.org/TR/chips and http://w3.org/TR/webarch would help to get some of the things. right.
Thanks for this very good post.
In my eyes, Amit Patel and Leo Soto brought up two key aspects: The pipe-concept that the UNIX folks came up with is brilliant. Write small apps that do something extremely well. Then link them to solve more complex tasks; RDF brings us a simple yet powerful possibility to shovel data from one app to the other. Oh, and on top of that, RDFS and OWL provide mechanisms to finally free data models (classes, properties, ...) from the prison of concrete implementations.
I don't do Perl and Python - so what am I doing here? Oh well, what the heck - I do love programming and designing and what I think the good discussion is leaving out is the masses and perhaps security. The masses? The more sophisticated the proposed "wish list", the harder it is to get in place because of the mass (opposing interests, habits, dollars) of forces hindering it. The discussion needs to include the ways in which "things" can happen. And although my least favorite subject is security, unfortunately, it must be worked into everything internet.
As I told you here in Buenos Aires, there's Nagare, a framework inspired in seaside, for stackless python.
Thanks for your talk!
What about Tornado ? a framework build to scale ? tornadoweb.org
c#
Don't we have HTTP for interop? ;) Good talk, lots to mull over.
Hey Jacob! I attended this talk at PyCon Argentina; just wanted to say it was really interesting and well-done. Thank you for such a great talk!
It's conceptually misleading for the author to denounce @280north for "building their own framework, Cappuccino; and inventing a new language, Objective-j." Cappuccino is a strong subset of Cocoa and Objective-J is compatible with "straight" JavaScript. And I believe they constitute a breakthrough, not a trend.
I agree with Ian to some extent. It's not concurrency itself that is the issue with Python. It's quite possible to write highly concurrent applications with Python, just not from within a single process. But of course, any discussion of concurrency must eventually lead to distributed computing, so the single process model gets thrown out the window in any case.
The real issue with Python is that it is a toolbox with a single big hammer: the object. Concurrency will always be ugly in Python because there is no way to extend the language without resorting to objects, and any non-trivial abstraction must be mapped (often clumsily) via an object (or be accepted into the language proper). In other words, there is only one way to grow Python, and that is to grow the interpreter itself. I expect that in 2020 Python will either be an unspeakable mess of syntax and special cases or it will be irrelevant, or more likely, both. Perhaps GvR will break down and allow macros, but I'm not holding my breath. "Explicit is better than implicit" might be very practical in limited systems, but it spells the death of abstraction and elegance: explicit code will, by definition always be more verbose and complicated than fully abstracted code. Not every complex abstraction can be decomposed into simple objects. I've always found that particular mantra rather disingenuous for a dynamically-typed, garbage-collecting language in any case.
Thinking about "web applications" breaks what the web was supposed to be.
Leave a comment: