Last week, I wrote on Twitter:
Closed-source software gets worse with each release (Microsoft, Adobe, …). Open-source software gets better (OOo, Ubuntu, …). Discuss.
Much interesting discussion did ensue — mostly around the exceptions (Apple, Google).
However, I’m less interesting in the exceptions than I am in the general rule. Sure, you can find plenty of examples of commercial software that’s improving over time, and, yes, there’s plenty of open source software that suffers from bloat and creeping features. Still, it’s hard to ignore the trend: closed source software seems prone to featuritis, to bloatware, to a trend of more bullet points at the expense of elegance. Open source seems mostly immune to these pressures.
I see a simple explanation for this, best summed up in a single word:
“No.”
The Django team has recently emerged from a period of voting on features for the upcoming Django 1.2 release. What’s especially interesting in this context is the list of features we’re choosing not to focus on for the upcoming release. This list includes some biggies:
- Support for IBM’s DB2 database.
- Integration of the Django Debug Toolbar.
- Integration of South (the most popular schema migration library).
Can you imagine a commercial project, faced with pressure from within IBM itself, decided not to devote resources to DB2? That’s precisely the type of feature you need to list yourself as “Enterprise Ready,” and yet we decided not to focus on this feature right now1.
| [1] | The core team felt that the DB2 adaptor simply wasn’t sufficiently battle-tested to consider for inclusion in Django just yet. There’s a pretty high chance we’ll revisit it at a later date, though. |
Django Debug Toolbar is one of the most popular add-ons to Django — in fact, I’d go so far as to say that it’s basically a required tool in the professional Djangonaut’s toolbox. Including it out of the box in Django is a highly requested feature. But we left it out: the lead developer wants it to remain a separate project, so that’s that.
As for South, well… The lack of built-in schema migration might just be Django’s biggest weak spot right now, and South is, more and more, becoming the de facto standard Django solution. Again, though, the developers (both South’s developers and Django’s core team) felt that South would be better served by more time as a separate project where it can develop at its own pace. So it’s not a priority for Django 1.2. Simple as that.
Were Django a typical commercial project, I have no doubts this would have turned out quite differently. A manager, noticing that schema evolution was the #1 requested feature, would have listed it as a must-have for 1.2. A salesman would have promised a big potential client that 1.2 would ship with the debug toolbar. IBM would offer to help sell Django to its clients if we only included DB2 support.
Sure, developers would still have objected in the same way, but those objections would be overturned by management or marketing. Release dates slip while developers scurry to integrate not-quite-baked software… and hey presto, it’s Adobe® Django© CSX™.
Open source cleanly avoids these issues by letting technical voices have the last say. As a maintainer of Django, I spend a lot of my time saying “no,” and there’s no management above me to countermand those orders. There’s only a group of my peers, all concerned with technical quality over all else.
Note
This piece uses the words “open source” in a deliberately vague manner — of course, there’s nothing in any open source license that provides for this technically-driven model. I’m more specifically talking the (typical) open source development model, not any particular open source license.
I’d argue that the model of development, not the license, is the critical factor in open source’s success… but that’s another show.
Indeed, this helps explain companies who are putting out closed-source software with open source quality… but that, too, is another show.
Comments:
This is so true. Microsoft in particular is prone to this "syndrome". While Ubuntu, Django, Chryp, Wordpress, all are getting better over time.
Companies can learn from this. Take out the features you don't need or don't use. It'll make the world a better place.
I think one of the most profound things is the communal oversight, the lack of any obligations, besides to do something useful, and the fact that the only currency is mutual respect. The merits of Django, or Turbo Gears, or Rails, or any other open source project can be judged completely, with no missing pieces, everything is out in the open. The lack of obligations means we can do what we think is best for our software, without being beholden to management, sales people, investors, or anyone else with a different view of our software. And mutual respect being the only currency means we come to the table honestly.
This is about the "social" part of the "network". The basics of sociology has this pillar named "coercion".
In closed source software, people are coerced to deliver the software no matter what. This can (and often will) compromise the quality of the software.
In open source software, since everything is open and transparent, developers are coerced to deliver the best they can do. Otherwise their patch will be ignored.
This can be applied to the whole project as well. Bloated open source projects will be dropped by the community and users in favor of better ones.
The fact that someone on the outside might be able to perceive the technical quality of a project might also be a factor in the desire to keep things elegant :)
Over in the Drupal world, we're feeling the back-and-forth pull of this stuff. Historically, the ideal of pursuing the best quality possible, consequences be damned, ruled the roost. Today greater adoption and more pressure from the community of large and small has led to a lot of pressure towards Feature List Items.
Both the "make it a product" and "make it a framework" crowds can suffer due to this tug of war, as neither group's needs are met as well as they could be. Django and a number of other projects can serve as inspiration for what successful stewardship of the core platform can look like.
Your discipline is definitely inspiring: the *reasons* for your "No" answers are as enlightening as the fact that the team has the discipline to draw those lines.
I've couldn't agree more, honestly. It may sound like beating a dead horse, but technical voices should (at least, in my humble opinion) be the only voices making final call on a feature set in a product.
That's not to say that other people shouldn't have a say, it's just that it's easier to avoid bloat if you have veto flexibility.
Just my two cents. Glad to see Django doing what it does best.
I think this might be more due to how products are designed in open source vs. corporations. Open source has the system of Benevolent Dictatorships and the dictator actually can say "no."
Most corporations have design by committee... or design by marketing... or design by VPs etc... I'm guessing that Apple and Google are the exceptions largely due to their own "dictatorships" - Steve Jobs and Marissa Mayer.
Software dictatorships seem to create products with a cohesive feel.
I don't think this is an issue of open source vs. commercial, but more of an issue of who has the final say. It's a lot easier to say "No" when you're the owner of an open source project (or company) than it is to say "No" to your employer.
This reminds of Fred Brooks' thoughts about conceptual integrity (http://en.wikipedia.org/wik...).
What I've noticed is that when you have corporate buyers, the Punch List must be Punched. Reading over these requirement lists and "must-have" features, it's easy to see that they were compiled by someone writing down anything and everything, with little thought of actually using these features.
But to get the sale, the features have to be put in, which works fine for the one customer, but now they have to be maintained and forced upon other customers. The best is writing features to disable other features that should never have been put in the first place.
If a technical lead has full control, the good stuff gets in, quality is not compromised, conceptual integrity is maintained, which results in a high-quality piece of work.
Office 2007 isn't better than Office 2003? Windows 7 isn't better than Windows Vista? Xbox 360 isn't better than the original Xbox? Visual Studio 2008 isn't better than previous versions?
A well led project like Django is better than weakly led open source projects. Likewise, a well led commercial product is better than a poorly led one. Django is extremely well led. Furthermore, I agree that saying "No" is critical - turning down some features is as important as implementing others. If you had stuck with those points, I would have agreed with you 100%. But I disagree with your distinction between open and closed software. Just because the "No" isn't said in the open doesn't make it any less significant.
As someone who wants to have tried everything at least once (programming wise, no comment on everything else), I really missed migrations from django which RoR has. But after a discovering south I haven't looked back.
The question is, would including south in django really improve django and/or south?
I don't think adding South to Django is a good idea.
I don't like South because it isn't automated enough. Some time ago I started working on a parser for SQL that could reconstruct a database scheme from the migration files (which used real SQL syntax) and from that do a diff with the active schema. The generator could also generate UNDO commands automatically (only for DDL statements).
I like SQL, but migrations don't necessarily need SQL, you just need a DSL from which to construct a schema representation and then do a diff.
That's the only sane way migrations should work for Django. Automatic, to prevent both repeating yourself, and accidental complexity.
I like the direction where Python is headed. It has sacrificed backwards compatibilty for clean design. I like the fact Python 3 has removed some features, in contrast to Java for example.
I think I have found the same spirit in the Django project. I remember you talking about not adding very many features to Django but instead adding ability for anyone to sort of plug them in when needed. I like Django has slowed the pace in the race of adding features. Keep it that way please.
I haven't read most of the comments made so far, therefore I don't know if it was mentioned already.
You might be absolutely true for a large ammount of projects. Though, there are a lot of open source projects, that, if there's progress at all, MUST become better. Look at Source Forge, the cementary for dead open source projects. If you remove half of the projects at SF, it means that you just improved them.
In my opinion, there was no way Open Office could become worse in the beginning. On the other hand, Microsoft Office was "good enough" - so it was overloaded with features to have an argument for selling newer versions.
To come back to Django: I like the way Django was developed so far! The code changes are real improvements. It's easy to notice progress and it's fun to read scm commits. Go on! ;-)
This is all well and good that debug toolbar and south ar left as separate projects. However, I think that even if you don't include them in django itself, that django should somehow call attention to their existence.
It was not until I got a job doing django work that my co-worker informed me of the existence of these tools. When he did, I was like "where have these been hiding!" Even if they aren't part of the django package, please mention them on the django site, django book, etc.
I agree with Leah Culver, this is not a question of open-source vs. closed-source. As you pointed out, the issues with bloat and feature creep aren't limited to corporations and closed-source development. In fact I don't see this trend being noticeably more prevelant in corporations.
It's a design and project management issue; it's really about the person (or people) who makes the design decisions and who has the final say on which features ship and which features don't. This person (or people) can be great or can be crap but I think that it has little to do with whether the project is open or closed source.
I think being able to say "no" is the heart of it, but Leah really hit the proverbial nail with "design by committee". I can't count the number of times I've seen a good idea beaten into mediocrity (or even to the point of stupidity) due to committees and/or stupid internal politics.
>>"As a maintainer of Django, I spend a lot of my time saying “no,”
This is one of the reason why Django is amazing.
I think another factor here is what stage of its life the commercial software product or service is at. At an earlier stage, there are a lot of things to fix and improve, hence progress is indisputable. XP was definitely an improvement over Win98, which was a nice leap forward from Win 95.
On the other hand, I see Google Search suffering from this disease lately, when user-inputted search terms are silently excluded with a small note at the bottom of the page. E.g.
http://google.com/search?q=... - Tip: These results do not include the word "below".
Saying "no" to South is not the same as saying no to schema migrations, though. The difference between some of the open source examples and the corporate examples is that if the product manager said "no" to schema evolution in Corporate Django, then there would be no schema evolution and that would suck pretty bad. When you say no to schema evolution, you're saying that a developer has to spend a bit more time installing another tool or building their own. That's a bit different.
Django without debug toolbar and south would be worse for some people, just as MS Word without spellcheck and without wordcount would be for some people. "No" in open source means something different than "no" in closed source, unless there's a good way to extend a closed source product, which some actually have. I think this is a very compelling argument for proprietary offerings providing means for other people to extend their product, as the intense focus on what really matters produces a better core product. Basecamp doesn't have to provide time tracking, because it integrates with products that do. Amazon ec2 doesn't have to provide out of the box solutions for all sorts of problems because its design allows companies like Rightscale to attempt to solve those problems on that level.
Well I agree with you in the general point, but I don't agree with your examples you said it yourself. The Lead dev of the tool said "no I don't want to be part of the core" perhaps there is something else there hiding.
Traditionally, it has been software developed by open source communities that has been at most risk to featuritis: every contributer wanted a little bit of recognition, or their itch solved in the product so that the software was useful to them. While the patches might reviewed from a technical code quality perspective, the usability and overall product direction were typically less well managed.
This is a key reason that the Linux desktop has taken so long to gain traction: installation, the windowing desktop, a suite of wordprocessors, spreadsheets, and image editing software all could tick the boxes in terms of features. But most typical computer users found those features too difficult to use.
It is when tech leads perform not only code reviews but an ongoing review of usability and the coherence of the software experience and architecture overall, that the 'power of no' becomes useful. And the proportion of open source software projects which do this are few... but growing. I've not done a study, but I would also hazard a guess that commercial applications are also bolstering their usability and architecture focus, basically on the back of being convinced on the merits of this via Apple's software.
For the main point is how decisions are taken ? If like PHP, decisions are community based, we get as result something that goes in a different direction each release. In contrary, Python or Ruby are driven by a BDFL and the result is opinionated. But Apple has its own 1$ Dictator For Life and their products are also strongly opinionated.
I think this is also link to community size and homogeneity. MS has a lot of different consumers profiles, and PHP too. As Python and django get a larger audience, one challenge the communities will have to face will be to keep things smart and clean.
I disagree with "No". The real cause of the quality discrepancy is a different approach to competition.
Microsoft, Adobe, Symantec, Facebook, etc. are motivated by market dominance. They offer a single bloated product because they know that if any part of their product is able to be modularized or open-specification, it is able to be easily replaced, allowing their products to be undermined by competitors.
The standard FOSS community motivator is "Make a good product", which means that being able to swap a component for a better one makes the overall product better. FOSS projects tend to modularize everything to encourage drop-in replacements to be made. The Eclipse project is an extreme example of this. Competition directly causes quality.
Hey Jacob, I really enjoyed this post. I've been thinking about why this trend seems to be true off and on since you put it up. I wrote a follow up post on my site: http://mattmcman.us/article...
(I'm definitely not trying to spam! Sorry if it comes off that way. Your post definitely got me thinking and I wanted to keep the discussion going)
I know that this is not going win many friends for me here but I disagree with the assertion that "Closed-source software gets worse with each release …. Open-source software gets better". I don't know the exact stats but I am quite sure that majority of the open source projects never go anywhere. I have had my share of open source projects that did nothing but waste my time. Ubuntu, Django, Ruby on Rails are rare exceptions and not the rule. They are remarkable, no doubt but very rare in their longevity and success.
I complete love Open Source for this exact reason, it's streamlined and barebone enough to develop upon and had enough meat to get the basic job done. The Open Source route is all I use for any of my programming, add-ons, etc.
I suspect that the argument is not valid except by accident. All product developers must develop according to the whims of their user base, or they will have no product. The problem faced by closed-source (read 'commercial') development houses is the need to attract more and more users in order to generate income, and it is this pressure that encourages the bloat. Opensource has a different philosophy of success behind its distribution and design goals, allowing [most] opensource products to remain lean. However, while this does tend to reduce bloat in a single product, it does lead to a far more eclectic software usage pattern by the end users. While I might use one single 'bloated' closedsource product to achieve a single goal, in order to achieve the exact same goal using opensource I may resort to using a large number of much leaner individual opensource products. I am not convinced that in any complex real world example the total amount of code required is all that much different.
Personally I prefer to use single tools from single vendors because, even though individual tools may be weaker than alternatives, the productivity advantage of learning and using a single tool built from a single design philosophy is, for me, substantial. Although, of course, I realise that salad is a perfectly good food and not everyone likes McDonalds.
Leave a comment: