Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

The Ideological Dimension
of "Solaris vs. Linux" Debate

I have written several articles about the “pseudo-religious” aspects of linux but each time I return to this theme I find new dimensions of the topic.  Certain category of people tend to get infatuated with ideas, for example open source, and it becomes a religion for them. And in this case there is a symbol of this infatuation. Like the car for Americans, linux has become more that a cheap and reasonably stable OS, really more than a utility that provide the means to get from the point A to the point B.  Despite being a technologically backward project in comparison with such developments as scripting languages and LAMP stack it has occupied a central place in the open source psyche because it offers the distinct pseudo-religious, insulating from reality cocoon.

Most large organizations have mixed infrastructure with a large (sometimes predominant) Windows server presence. Linux is a part of Unix infrastructure and as such typically represents a fraction of total server park. But due to strong IBM PR machine there's constant pressure within many organizations to increase linux share of the servers and sometimes to do it at the expense of Solaris. Usually the driving force are university graduates who come brainwashed against anything that is not linux including Solaris, which they do not know and thus do not like. Most of them usually quite quickly understand that irrational  preference of one OS over another  has no place in business and that opening source is not panacea: software problems are too complex to be solved by "magic solutions" and issues, including issue of cost,  are far from being simple.  But some proved to be resistant to "deprogramming".

This example suggests that among programmers and system administrators there is a sizable minority of people who are by their psychological traits are suitable candidates for fervent participants of Mass Movements, no matter what this particular movement is about.  In Eastern Europe this issue is often discussed under the umbrella theme of "tragedy of Russian intelligencia" but has definitely more wide implications. 

Eric Hoffer called the category of people who are "pre-destined" to became a fervent participants of mass movements "true believers." They are joiners and followers ... people who want to give away their personality in the name of the great idea.  They look for answers, meaning, and enlightenment outside themselves. Hoffer, who wrote famous "The True Believer" observed that,

"true believers are not intent on bolstering and advancing a cherished self, but are those craving to be rid of unwanted self. They are followers, not because of a desire for self-advancement, but because it can satisfy their passion for self-renunciation!"

Hoffer also noted that true believers "are eternally incomplete and eternally insecure". He wrote:

The less justified a man is in claiming excellence for his own self, the more ready he is to claim all excellence for his nation, his religion, his race or his holy cause....

A man is likely to mind his own business when it is worth minding. When it is not, he takes his mind off his own meaningless affairs by minding other people's business....

The fanatic is perpetually incomplete and insecure. He cannot generate self-assurance out of his individual resources -- out of his rejected self -- but finds it only by clinging passionately to whatever support he happens to embrace. This passionate attachment is the essence of his blind devotion and religiosity, and he sees in it the source of all virtue and strength.... He easily sees himself as the supporter and defender of the holy cause to which he clings. And he is ready to sacrifice his life.

That's why true believers want to give up all personal responsibility for their beliefs and actions. Paradoxically, they personally want to be free of the burden of freedom even if they claim that their cause is to defend freedom.  Such people can be easily commercially exploited. In fact, the inside nickname for this type of selling among marketing professionals is, "sell it by zealot!"  Therefore it is important objectively discuss the ideological dimension of "Linux vs. Solaris" debate and the specifics of open source marketing which reminds me typical PR of anarchist organizations.  See brilliant satire Penguin Suicide Bombers in which author enumerates several interesting and controversial properties of open source PR which are identical to methods used by anarchist organizations for last hundred years and which he incorrectly but acceptably for satire exaggerates to  "primary characteristics or traits of the open source movement":

• First and foremost, the open source movement is to some degree a rejection and opposition to the direct capitalization of software but is perhaps more specifically and correctly defined as the rejection and opposition to what is perceived to be a “unipolar, capitalistic superpower”, in this case Microsoft. This appears to be a widely accepted attitude within the open source community as there are endless quotes spanning a large number of open source projects to the effect of “the enemy is Microsoft”.

• ...

• Third, the open source movement by and large uses crude propaganda and hate-filled rhetoric to defame and demonize its opposition. For example, this third point can be easily demonstrated by the coarse language used by Linux proponents when debating or characterizing peers that utilize Windows-based technologies. More often than not, Linux proponents and other open source advocates go out of their way to characterize their opponents as “stupid”, “ignorant”, “retarded”, “evil” or much, much worse. If you don’t believe me, go browse any forum frequented by Linux or open source proponents. In addition to the name calling and hate speech there is even advocacy of sending Windows users to concentration camps or purposefully spamming their email with viruses.

• Fourth, a favorite tactic of the open source movement is the use of fear as a weapon. Again, this can most readily be seen by Linux, Apache and Firefox proponents that tout the perceived security of their systems while attempting to instill fear, uncertainty and doubt in those that use Microsoft technologies by claiming that Microsoft systems are inherently insecure or inferior in terms of security.

We should never underestimate the potential danger of fanatical followers of any social moment and the True-believer syndrome . While such people will gladly work and, sometimes, even die for their holy cause it is not a true devotion, it is just a substitute for their lost faith in themselves and offers them a substitute for individual self-esteem.  All high demand cults are composed mainly of true believers but you can find some examples anywhere: in politics, churches, businesses, technology, etc.

Researchers who investigated high demand cults were surprised to discover than when such cults receive a major shock -- a prophecy fails to come true, a grave moral flaw of the founder is revealed -- they paradoxically come back stronger than before, with increased belief and fanaticism.  The Jehovah's Witnesses placed Armageddon in 1975, based on Biblical calculations; 1975 passed but they still doing strong. Similarly Unarius Academy of Science cult survived the nonappearance of an intergalactic spacefleet in 1975 and 2001 (the Wikipedia article mentions only a failed prophecy in 2001).

This phenomenon is connected with so called cognitive dissonance. When people have taken  expensive or "irrevocable" actions for a belief that later is discredited, they has difficulties admitting their mistake. The challenge to their belief presents an immense cognitive dissonance; they must find reinforcing evidence to counter the shock, and so become more rather then less fanatical.  We definitely observe similar behavior with "year of the linux desktop" and other prophecies.

True believers want to convert others to their way of living or impose some new way of doing things -- if necessary by force. They also share a common hatred of enemy, or devil which is crucial for increasing solidarity within the movement.  Any problems or setbacks of the movement are viewed as a result of the subversive activity of the enemy. In case of open source movement this role is often played by Microsoft but sometimes other firms are targeted.  It's funny that despite its questionable past IBM never got into this category and is viewed by most open source "true believers" extremely positively.   This effect is closely connected with the notion of groupthink. Here is how this phenomenon is described by Answers.com: 

Groupthink occurs when the pressure to conform within a group interferes with that group's analysis of a problem and causes poor group decision making. Individual creativity, uniqueness, and independent thinking are lost in the pursuit of group cohesiveness, as are the advantages that can sometimes be obtained by making a decision as a group—bringing different sources of ideas, knowledge, and experience together to solve a problem. Psychologist Irving Janis defines groupthink as: "a mode of thinking people engage in when they are deeply involved in a cohesive in-group, when the members' striving for unanimity override their motivation to realistically appraise alternative courses of action. Groupthink refers to a deterioration of mental efficiency, reality testing, and moral judgment that results from in-group pressures." It can also refer to the tendency of groups to agree with powerful, intimidating bosses.

The concept of groupthink provides a summary explanation of reasons groups sometimes make poor decisions. Indeed, groups are supposed to be better than individuals at making complex decisions, because, through the membership, a variety of differing perspectives are brought to bear. Group members not only serve to bring new ideas into the discussion but also act as error-correcting mechanisms. Groups also provide social support, which is especially critical for new ideas. But when new perspectives are rejected (as in the "not invented here" syndrome), it is hard to correct errors. And if the social support is geared toward supporting the group's "accepted wisdom," the elements that can make groups better decision makers than individuals become inverted, and instead make them worse. Just as groups can work to promote effective thinking/decision making, the same processes which enhance the group's operation can backfire and lead to disastrous results.

1. A feeling of invulnerability creates excessive optimism and encourages risk taking.
2. Discounting warnings that might challenge assumptions.
3. An unquestioned belief in the group’s morality, causing members to ignore the consequences of their actions.
4. Stereotyped views of enemy.
5. Pressure to conform against members of the group who disagree.
6. Shutting down of ideas that deviate from the apparent group consensus.
7. An illusion of unanimity with regards to going along with the group.
8. Mindguards — self-appointed members who defend the group-approved dogma and shield the group from dissenting opinions.

From the start, linux has been an object of quasi-religious longing. For many fanatic advocates (we will call them "linuxoids" for brevity) linux represents a readymade "Promised Land" (I always wondered wouldn't volunteering for organizations like UNICEF be more rewarding for those who want to help poor ). That's why linuxiods berate advocates of other Unixes and vise versa Solaris advocates usually grant linuxiods their good intentions but remind them that road to hell is paved with fallen penguins as well as "save mankind from Microsoft" aspirations [Bezroukov1999a]:

If we use the ESR metaphor in a different context, Linux can be looked as a cathedral as it was build to express passion about Unix by a large group that cared about it very much. The ability to attract a geographically distributed community is characteristic of political movements and religions. Although people are physically separated, they all are working toward a common and important goal. That fuels the Linux movement.

Along with a positive agenda, every powerful social movement requires an enemy, a target that can be used as a powerful unifying force. Therefore it is not accidental that a large proportion of the Linux and open source community really hate the 'Evil Empire'. The history of religious and political movements suggests that the OSS is not simply a loose collection of Internet-connected developers and users motivated only by mutual recognition. The OSS really operates on a larger stage as a political movement and as such has its own political agenda.

Probably the most important part of OSS political agenda is its negative aspect, taking the form of a revolt against Microsoft. I call it the ABM/BTM ("anything but Microsoft/Better than Microsoft") agenda. Both statements are simplifications and both presuppose that Microsoft software is not good enough for any task in hand and view Microsoft in black and white in the best of "we against them" war mentality. That's why the OSS movement and Linux have powerful allies in OS/2 and Macintosh communities. Both OS/2 and Macintosh are based on closed proprietary models (actually not much different from the Windows model), but with powerful communities that also has ABM/BTM as their political agendas.

The ABM/BTM agenda, like any political agenda, is simple yet attractive to some. More importantly, it appeals to certain elements of corporate management with support from a wide spectrum of executives at Intel, IBM and almost all major PC vendors. Given the licensing fees paid by Compaq, Gateway and other PC vendors to Microsoft, you begin to understand why there are programmers working on Linux development in these companies with full (and, if necessary, covert) support of management. Politically, some compare Microsoft to IBM in 70's; that attitude fuels reasonable attempts to develop opposition to Microsoft.

Among Internet providers the same agenda is fueled by a fear that Microsoft can usurp the Internet by using well-funded top corporate technocracy in large manufacturing companies. Large and influential Internet providers are resentful about expensive and restrictive licenses; they perceive Microsoft as a threat ("dark force") that tries to slow them down.

It is important to understand that there is and always was a very strong ideological component in all discussion of "Solaris vs. Linux". Comparison of those two OSes seldom is limited to purely technical issues, it is usually performed in a larger framework and is colored by ideology.  What is interesting is that many proponents of Solaris and Linux even view them as belonging to different "programming ideologies" despite the fact that are descendants of the same OS.  But in fact linux was cooped by corporations long ago and evolved from the concept of development by a group of enthusiasts to the idea of a "community edition" financed by few large corporations where the source may be available freely but still much of  the actual development and  "know-how" is concentrated in development teams belonging to and employed by corporations.  Solaris is now perceived by many as very similar in this respect to linux. Here is a quote from the letter by Bruce Shaw with the review of the earlier version of this article:

As far as I could see, you make no mention of the Open Solaris codebase in terms of Sun's strategy. It eliminates a great many of linux's last remaining "strong points" in that lowly users can now contribute their own patches, and Sun has provided a fairly open forum for discussing the direction of the operating system.

Still is some sense Solaris definitely belongs to a different ideology of programming then linux. Slightly simplifying there are two contrasting ideologies of programming: Software Realism and Software Idealism. The latter in their turn can be subdivided into two major schools: Stallmanism and Raymondism.  Here is how I defined them for readers of my two old papers[Bezroukov1999a, Bezroukov 1999b] on the subject (the quote below is taken not from the papers but from Softpanorama OSS page):

In the vision of Software Realists programmers like all humans have weaknesses and guided primarily by self-interest and as such requiring formal organization and incentives (direct or indirect) to act outside the limits defined by their own self-interest. Software Realists see the evils of the software world as given and derived from the limited and unhappy choices available, given the inherent moral and intellectual limitations of human beings and existing hardware.   Some of them try to create better software like programmers involved in creation of all BSD flavors but they consider commercial programmers as equals (actually many of them wear two hats) and do not object to reusing the code that they developed pro bono in proprietary software. 

In this slightly tragic worldview the software development is a hard and often underappreciated labor that requires special, pretty rare, talent  People with this talent like talented sport stars (for example tennis players or ice ballet dancers) risk their health while they are young creating short living but tremendously complex artifacts (large software systems are probably the most complex system even created by humans) and for this reason should be remunerated accordingly.  It is important to understand that like in case of artists creating paintings on the beach sand beach programmers creation are short-lived and the new wave of hardware often wipes them clean.  For example, who now remembers the creators of PL/1 optimizing and debugging compilers, the compilers that were real breakthrough in many areas of complier writing art and in comparison with which some modern compliers still look like junk despite the fact that they were written 30 years ago.

In other words they see that due to immense complexity of those artifacts all software sucks. It is just that some software sucks less. It can be proprietary software, it can be free software -- all depends of the talent of the creators not so much on a particular ideology (which, by the way, can be completely wrong: Soviets invented quite a few new technologies and managed to launch the first man into space).  Thus, in view of Software Realists school the perfection of software is unachievable and old good Unix with its classic codebase might sometimes be preferable to new Turks be it Windows or Linux. That does not mean that Linux or Windows codebase is all crap. That just means that they are just another OSes in a long historical line of such systems. In some areas they might be better then competition, in some worse, but none has the monopoly on innovation (actually Linux is a pretty conservative kernel despite a radical ideology behind it). 

Software Realists are unconvinced by claims of linux superiority and want to see hard facts.  Furthermore, history guided them that the proper tradeoffs between different subsystems of OSes can be ironed out only via long, expensive and painful process that requires strong highly paid managers, programmers and testers who are ready to sacrifices their health for the success of their creation. The real art requires sacrifices and it is better when such sacrifices are properly remunerated, although stories of talented artists who died in poverty are nothing new. 

Because real talents are rare good software is a very expensive thing.  Software realism school  presuppose that modern software is almost always a compromise between the demands of the talent and demands of the marketplace.

The Software Idealist school (both in its Stallmanism and Raymondism incarnations) holds that mankind in general and programmers in particular has not yet achieved their full moral potential, and that they are (at least in principle) perfectible if guided by wonderful new software development ideology.  Foolish and immoral choices inherent in the creation of proprietary software explains the all the evils of the existing software world  and revolution was needed and actually already  came in the form of either free software movement or its less pure form called open source software movement. Both major Software Idealism schools rely on volunteer labor of programmers connected via Internet to produce immortal gems of software wisdom that will crush proprietary software developers like cockroaches. 

In Software Idealism  worldview, whether purely moral incentive actually work in a long run or not and what will happen when Linus Torvalds will become yet another retired dot-com multimillionaire with his own yacht is irrelevant to the achievement of true software justice, justice for all.  This utopian view holds that  volunteer programmer potential is immense and can do everything including improving human nature that should get rid of those outdated  economic rewards for software development and be satisfied  working part time in McDonalds in order to be able to participate in the movement. So the Software Idealism vision promotes pursuit of the high moral ground of software freedom which somehow guarantee the best software solutions. At the end of the day new liberated men should all storm this evil Bastille of software oppression which is of course Microsoft and dance on its ruins. Sometimes in their enthusiasm  they can attack other sinful old fashioned proprietary software vendors instead of Microsoft.  Until opening Solaris (and even after that) sometimes their target was Sun.

And if the unwashed masses who corrupt their soils by using Windows are slow in catching on, then it is the role of the intellectual vanguard (the keepers of programming craft who in Eastern Europe might be called "programming intelligentsia") to lead them  - even if in the short run, the masses can be unhappy with the results because they might not be able to use full capabilities of their laptops, cameras or scanners.  In general Software Idealists think that higher moral considerations should guide us and that those consideration somehow guarantee creation of a better software, the software that is not only better but which is as perfect as it is free.

Again this is a slight simplification but I believe it catches the key points.  For more elaborate discussion you can try to read [Bezroukov2004a]. Another important problem with GPL is that if you work for a commercial company and want to share code it is dangerous to do it using GPL, especially if you take GPLed software as the source of inspiration. But also if you extract and submit parts of the product to the community as a library or stand-alone utility there is always a risk that zealots can blackmail company products. And if you do not study something in depth you are less inclined to contribute to the codebase. In case of BSD the more you use some product or library, the more you can contribute without any danger of "zealots blowback" for your generosity.  For example several lawsuits about usage of BusyBox probably did a lot to prevent its further usage by commercial developers.

This particular "cuckoo egg" property of GPL considerably depletes the space of capable developers  and only naive can claim that this represents "higher freedom" as if sharing of complex code with "strings attached" makes it more attractive for other developers to base their products on it. There are always alternatives as designers of scripting languages learned all too well (none of the popular scripting languages use pure GPL licensing; actually Perl pioneered a very elegant way to preserve sanity without alienating users who prefer GPL by inventing dual licensing scheme and Artistic license which is very close to BSD in spirit).  Also paradoxically many programmers who are too afraid that something will be stolen from them are too weak to produce anything of lasting value in the first place. 

You can also see that some less "strict", less fundamentalist modifications that fit somewhere in between the extremes (like LGPL) might actually increase the product  survivability properties.  The same is true about business model. Red Hat is an example of such a compromise view from the point of view of business model as it seems paying their developers  well, on par with commercial software giants like IBM, Microsoft,  or Oracle.  Novell probably also can serve as such example and it should be commended for building some bridges with Microsoft. Of course the latter caused a wrath of "true believers" and costs Novell some PR setbacks.

While in 1999 the ideas the open source as, essentially, a quasi-political movement that does not has any monopoly on technical innovation were pretty unorthodox and the amount of flames I endured for writing two articles sited above was probably above average, recently those views became more or less an established common sense.

Apt demonstration of this trend was the keynote address at EclipseCon, 2007 by former A&T Wireless chief technical architect Robert M. Lefcovits  ['Dilbert' Creator, Open-Source Maverick Address EclipseCon - Open Source, Software - CRN] who stated:

The ideology of the open-source moment is built atop some odd assumptions... including the idea that buyers need access to code because product designers can't be trusted to get it right.

... ... ...

Flashing a photo of a pill bottle, Lefkowitz quipped, "Sorry this drug was defective. I'm sorry you got sick but, you know, we included the chemical formula on the bottle! You could have fixed it yourself!"

His presentation was well received by the participants of this key open source conference (Eclipse became the dominant development environment for major open source projects).

Development Methodology Issues

From architectural standpoint Solaris is more of a complete OS, not just a kernel with a collection of partially written in-house, but mostly written by outsiders software components hacked into a particular distribution. The latter creates its own set of complex problems.

Sun developed Solaris as a typical commercial software developer with enigmatic founders and hired developers. During long life of the product it has it highs and lows.  Developers in Sun work for cash not so much for the holy purpose of saving mankind.  Which does not exclude that they enjoy their work and have a strong positive feeling toward the product. If opportunity arise Sun can switch to other supposedly more profitable products as it has done with Java. This is still not a betrayal of roots but a fair play as the company needs profitability to survive and prosper. And survival and prosperity of the company means prosperity for the developers.  In this sense Sun is not that different from Oracle, Microsoft, SAP and other software powerhouses. If it see value in open source it can use it were it suits its goals like IBM profitably did, but in no way that means that Sun brass suddenly became bunch of  GPL converts. 

Recently we observed that Solaris moved higher in Sun's agenda and Sun management recently changed attitude toward Solaris in internal war for resources between  Solaris and Java camps. As I mentioned before, Sun now really committed to Solaris on Opteron and Intel Duo CPUs and this change in attitudes can benefit enterprise customers.

As we all know that was not always the case.  For almost a decade, Sun seemed to relegate Solaris to the status of red-headed stepchild of Java, undeserving of attention, nurturing, and support (and shipped it with weak management applications like now discontinued Admintool, which paradoxically was replaced by also "not-so-good" Sun Management Console).  In the past Sun brass demonstrated very mixed level commitment toward Solaris on X86 platform.  But that completely changed with the adoption of Opteron which might eventually became the main development platform for Solaris.

In my opinion they can not betray this commitment again. As the x86 market space continues to grow, the gap in CPU speeds between RISC producers and Intel/AMD widen. With Microsoft growth outpacing Linux/Unix growth in enterprise space Intel and AMD are making the most money in server CPUs area and can afford to outspend competitors in research by a wide margin. 

IBM and Sun already feeling the heat from the recent introduction of Intel Duo CPUs and despite being competitor in RISC space they are sitting in one boat against onslaught of Intel and AMD; actually IBM even more so with the superiority in benchmarks for P5/P6 architecture against Intel and AMD CPUs now becoming history. Just looks at    Spec benchmarks (SpecInt for Intel X5160 3GHz is 16.8/17.4 vs IBM P6 4.77Ghz 17.8/21.7).

With the current gap created by introduction of Intel Duo, the strength of Sun's commitment to AMD and Intel CPUs should not be underestimated:  shipments of Opteron based servers became the major growth sector for the company and probably will remain such a sector in foreseeable future.  And that change in attitude is already reflected in training with the additional courses for Solaris on Intel released for Solaris 10 as well as the multiple of new virtualization offerings and zone management capabilities in Solaris 10 11/06.

Unlike typical for large commercial organization structured project development  in Sun that started with the traditions brought by graduates by two prestigious US universities (Stanford and Berkeley) and which from the beginning had Unix design architect of the highest caliber (Bill Joy of BSD fame), linux initially was developed in anarchist manner but semiautonomous individuals around the globe. Linux Torvalds started with really humble OS development baggage and had professionally grown with the system. Like most successful large open source project linux development is based in the talent and charisma on a single leader who works like crazy to implement his vision be it right or wrong. Workagolism is the norm for such leaders and their whole life revolves around this single project. Due to this chronic overload, sometimes they often look more like hostages of their ambitions. More then one hurt their health in the process.

As the level of commitment necessary for leading the project is extremely high, almost superhuman, large open source software projects which lost the initial leader, often remain "leaderless" and/or are directed by people who want just benefit from the implicit status associated with the leadership and who lack both vision and discipline (and sometimes even commitment). The "I'll donate some of my time to some project" developers prefer working on "cool" features, not on the preserving architectural integrity and if there are no real charismatic leaders who try to preserve architectural integrity despite all odds it gradually deteriorates and codebase become unmanageable. In this case the pressure to add  all of those supposedly "cool features" is unresisted and will eventually find their way into codebase even if half of them are completely redundant.  As the result many "leaderless" large open source projects suffer from feature creep and quick, irreversible deterioration of the quality of the architecture. Bloat of the codebase is a very serious problem, problem that is underestimated by many researchers on open source software development [Bezroukov1999a,Bezroukov1999b].   In a sense the only viable to open source is a "slim source" :-). And the only way to accomplish larder project is to select higher level language (that's why I think scripting language are so important for open source: there is not that much "open" in several hundred thousand likes of C or C++  code :-).  Like in case with severely overweight people as soon as you cross some weight to height ratio, quantity turns into quality and other, often a lot more serious, diseases are almost given.  That already happened with Gnome, Perl 6 and several other large open source projects which now wonder aimlessly in the complex maze of architectural alternatives and directions [McFarland2006] 

From this point of view the important difference is that Solaris is trademark of Sun, not the trademark of Linus Torvalds :-) .  Openness of the code is not the whole story and actually is just one of the important characteristics for a large software projects.  In no way it can prevent loss of architectural vision and in-fighting.   In general commercial software development with its motto: "we pay, you deliver" is less susceptible to ego-related problems like in-fighting. They can replace the leader of the kernel group with much less damage then open source projects where reputations and vanity fair is the name of the game. Money speaks and good money speaks quite loudly. Also there is  an advantage of severe, ruthless discipline of commercial software development over open source Internet based semi-anarchy.  Just compare Red Hat releases frequency with Debian.  Also while far from being easy, in case of the change of the leader  the commercial project might have advantages as it is easier to maintain and preserve architectural integrity just because the working assumption is that the product belongs to the firm and everybody just need to deliver the product as good as they can so that it can sells well and sustain the salaries of the developers.  And people are ready to put long hours to meet deadlines, which are often at least partially imposed from above.  Also while linux kernel development is now accomplished in best "cathedral" style Sun still has an advantage as Solaris was commercial project from the beginning and as in any such project ego-related issues were suppressed and this atmosphere hopefully preserved so that those issues cannot badly hurt the project even after the codebase was opened.

The level of complexity of project like Linux, Open Office, Python and so on is such that the idea of  CATB looks not as tied metaphor or fairy tail, it looks plain silly. Spending months understanding someone else's framework and code before you can even start on your cute "scratching itch" addition of modification requires tremendous self-discipline and real sacrifices both in time, energy and lifestyle. Most talented people would better off writing something from scratch.  Also the problem of "a large shadow" of Linus Torvald and other leaders of key open source projects is also very real.  There is an inherent tendency toward cult of personality in such projects as without proper remuneration for their efforts the best protection for the participants who already invested considerable time and energy into the project is a blind devotion to the fearless leader. You need to understand that simple fact or be bitterly disappointed at the end. Chance of being dismissed due to plain-vanilla envy are also very real.  Leaders are humans and have tremendous stake in maintaining their position at any cost due to the amount of contribution and hardship endured.

Even more self-discipline is necessary to test all dependencies so you don't break somebody else code.  Money and preferably very good money, not an enthusiasm (although it helps) are the only reliable cure. Enthusiasm is no substitute for money in large open source software projects. Volunteer work can help quite a lot but still it is peripheral to the contribution of key paid developers who constitute the core of the project. That's why Linux kernel development is now just of cooperative venture of several large corporations which provide all the funding. More of a convenient "corporatist" framework with several large corporations just  pooling their resources to counter  MS onslaught on their franchises.

That problem extents of testing which is really Achilles spot of linux. Volunteer approach to testing works only until certain level of complexity and cannot replace structured testing by specialized testing organization of a caliber that Microsoft have.  In December 2006 Open Source Development Labs(OSDL), an Oregon-based kernel development organization cooperatively funded by major hardware vendors which employs Linus Torvalds cut one-third of its staff[Shankland2006]. The OSDL is funded by IBM, Hewlett-Packard, Novell, Intel and several other computing companies and it's staff after layoffs is 18 people. Even before that to conserve resources Linux Torvalds decided to abandon the concept of production kernel and assumed that it is distributors who should bear the burden of testing...

While bureaucracy which is inherent in large organizations dampens individual creativity and initiative, deadlines (and strict responsibility for failing them) discipline and motivates people much better then FSF slogans. There is also a type of talented software developers that like elite special forces can deliver the performance close to the best even under intense pressure. Such people are often can be found in major software companies, especially game industry where life and death of the company often depends on timely production of the next blockbuster game.  Huge open source commercial projects like Open Office and Open Solaris has additional advantage of being less dependent of feature creep and theoretically can deliver slimmer versions then in case of completely closed software development.  It is still unclear if Sun will be able to utilize this potential advantage, as old habits die very slowly :-).  They have additional chance with opening Java, though.

Anyway for large open source projects the old rhetoric of CATB proved to be irrelevant. In reality actual "Bazaar" projects are mostly limited to the thousands of smaller and mostly single developer projects on SourceForge.  They are still very important, and some of them are very innovative, but in large open source project space Cathedral style development proved to be the only way to go. Just because something is free and open source doesn't mean that they can't be managed by someone with a business background. Just think about Ubuntu. Small doze of bazaar can add some spice, but that's about it. 

As this article partially is a counterargument to naive linux evangelism I would like to quote an old (Aug 09, 2004) entry in Chard Dickerson weblog Solaris vs. Linux that by-and-large coincides with my own views of Solaris vs. Linux problem especially his observations about using Oracle on Debian (or any enterprise class software on unsupported distribution) as well as classic "if you've already 'sold out' to Oracle, why try to be righteous by using a free Linux distribution anyway?" theme:

While we're mostly a Linux/Intel shop here at InfoWorld, I don't feel nearly as fired-up and evangelical about Linux as I did five years ago when I convinced the marketing department at my then-employer to issue a press release trumpeting our migration to Red Hat Linux (spawning a Slashdot posting, a Webmonkey article, a piece in PC World, and a mention in a News.com story ). Despite the flurry of attention that migration brought, the dirty little secret of this Linux proponent was the Sun E450 I had in the server room running our content management system (which was open-source based and ran great on Solaris) and our mission-critical Oracle database. Linux was a bit risky at that time, so I wasn't foolish enough to migrate *everything* to Linux -- we had a content business to run, so the core content-producing system stayed on Solaris/Sparc. An excellent choice, but no one issues press releases about rock-solid Sun boxes running Oracle. They were just too commonplace.

Fast forward to the present. The Red Hat Linux I run isn't free (as in beer) anymore, and though the mission-critical Oracle I used to run on Solaris is now humming away on Red Hat Advanced Server on HP hardware, it's not officially supported on Debian, our Linux distro of choice for our front-end web servers, mail servers, etc., so I'm stuck working with two Linux distributions. Though I can get plenty of advice on how to make Oracle work under Debian (see here, here, and here), I find the thought of fooling around with a Debian system (even minimally) to make a five-figure software package function. . . . well, it registers somewhere between "aggravating" and "pointless." Might as well just cough up the money to Red Hat and go through as normal an install as possible. (And speaking within the "social movement" context that Jonathan asserts in his original post -- if you've already "sold out" to Oracle, why try to be righteous by using a free Linux distribution anyway? Seems a little like wearing a shiny new leather coat to a PETA rally.)

Quandaries like the one I outline above led me to conclude this week's column this way:

If I’m going to be paying license fees and support to a commercial OS vendor, what difference does it make whether the core is a well-tested Linux or a well-tested Solaris? I don’t think I would lose sleep at night going with either.

Open Source as an Economic Welfare

Role of open source as a new type of social welfare is a very interesting phenomenon and pretty novel approach to the study of open source. In a very dialectical way software that initially was free and considered by many as add-on to expensive hardware get first into commercial stage and then into the stage when it the code again is free and opened but support is extra. also like in good old times software is considered as add-on, advertising expense for an inexpensive (mainly  Intel-based ) hardware.  In such roles it definitely can serve as a zero-cost substitute for poor families and disadvantaged groups who cannot afford themselves expensive commercial software. This is great progressive role that recently has an interesting hardware development twist in form of $100 laptop and its commercial variants (with ASUS as the most successful so far).

At the same time like any welfare system has its inherent drawbacks and is by the very fact of its existence limits the quality of contribution (may be not at the very beginning, but only after a short "irrational exuberance" period).  Also like any welfare system along with serving needy people its creates a special class of open source parasites, people who want everything from the society for free and, moreover, claim, that this is their immanent right. 

When people have an entitlement to get software without doing any work or paying any money, when the whole organizations rely on  "software welfare", the problem is not that so few people are participating in major open source software development projects but rather that anybody is. That problem extends  also into "open source software cooperatives" when several commercial firms pull their efforts in development of potentially useful package like Linux kernel, Open Office, Apache or Mozilla application suit.   There is not much trust in such ventures and each side suspect that other is "leaching" on the other and is not contributing enough or is not contributing in good faith.

Open Source Elite

Now let's discuss the notion of open source elite. Our discussion will be partially based on the paper "The tyranny of stuctureleness" by Jo Freeman who analyzed feminist groups.

The term "elite" refers to a small group of people who have power over a larger group of which they are part, usually without direct responsibility to that larger group, and often without their knowledge or consent. Because elites are informal does not mean they are invisible. At any small group meeting anyone with a sharp eye and an acute ear can tell who is influencing whom. The member of the "base" or "inner circle" will relate more to each other than to other people. They support each other's views and give in amiably. They tend to ignore the "outsiders". The 'outsiders' approval is not necessary for making a decision; however it is necessary for the 'outsiders' to stay on good terms with the members of the inner circle. Of course, the lines are not as sharp as I have drawn them. They are nuances of interaction. But they are discernible, and they do have their effect. Once one knows with whom it is important to check before a decision is made, and whose
approval is the stamp of acceptance of Linus, one knows who is running things.

Elites are not conspiracies: they are not bound together to take over the larger group for its own ends. In a way they are friends who also happen to participate in the same open source software development project. They would probably maintain their friendship whether or not they were involved in the particular project; and they would probably be involved in some other project if they did not get to this one. But after such group is formed in any egalitarian group their grip on the group is difficult to break. they also arise out of the need for the group to communicate with the outside world.   While none of them has consciously not chosen spokespeople, the movement itself promotes those who have the talent to communicate the goals of the group to the outside world. These people usually represent particular project. Because there are no official spokespeople nor any decision-making body the press can interview those people when it wants to know the movement's position on a subject. This way they create their "star" status that reinforces their elite position in the group, they started to be perceived as the spokespeople for the group. Thus, whether they want to or not, whether the movement likes it or not, elite assumes the role of spokesman for the group by default.

Open source "stars" are not selected by the people in the movement to represent the movement's views. That's why "rank-and-file" members of the movement often resented when the press presumes they speak for the movement as was many times the case with Eric Raymond... Thus the backlash of the 'star' system, in effect, encourages the very kind of behavior of the elite that the movement condemns. They feel more alienated from the group who now "does not understand their sacrifice" and they became more free to pursue individualistic goals and first of all the goal of personal enrichment.  This line of development often is called "Animal Farm  effect" with its immortal slogan "All pigs are equal but some pigs are more equal then others" . This slogan precisely captures the relationship between elite members of the group and rank and file members.

The friendship between the members of elite helps to form channels of communication outside any regular channels used by the project. Because they share the same values and orientations, because they talk to each other socially and consult with each other when common decisions have to be made, the people who can use additional channels of communication have more power in the group than those who don't. And it is a rare group that does not establish some informal channels of communication on the top.

Although the process of elite formation within small groups is well studied that does not mean that elites are destructive. They just limit and hierarchically structures the flow of information which in turn often can be a defensive reaction to "information overload" by the top members of community.  All large informal groups create informal structures as the "flat" model of interaction overload the members after the group achieves a certain critical mass.

There are two potential negative consequences of this implicit "structuring"  of the informal group and crystallization of elite:

• The informal structure of decision-making became biased by implicit power relationships: one in whom people listen is the one that belongs to inner cycle of the leader, not necessary one who have said or done the most significant things. As long as the movement does not need to achieve significant progress in particular area and remains purely hobbyist it does not matter. This is typical for abandonware, old computers and other hobbyist software development groups.  But as soon as their product start having some economic value the whole different game opens.
   
• The elite has no obligation to be responsible to the group at large and their goals can diverge from the group from which the elite was crystallized. Their power was not given to them by members; at the same time it is real and it cannot be taken away. Their status and influence is no longer solely based on what they do for the group; therefore they cannot be directly influenced by the group. This does not necessarily make elites irresponsible. Those who are concerned with maintaining their influence will usually try to be responsible. The group simply cannot compel such responsibility; it is dependent on the interests of the elite. We cannot just postulate that there are no additional goals. In open source development personal enrichment is one such side goal that is typical for elites. Slashing the biggest and best portions of the ‘open source cake’ for themselves becomes the main game in town.

 As long as open source movement stays dedicated to a form of the organization which stresses unstructured participation of volunteers the fertile ground for the creation of informal elite will persist. With no formar elected bodies the elite is exclusive and discriminatory against those people who are not or do not want to be
tied into the additional informal friendship networks. Those who do not fit will inevitably be discouraged from trying to participate by attempts to keep them at the low level and exclude them from major decision-making. Those who are "in" will develop vested interests in maintaining things as they are. As C.Wright Mills notes the immorality of elite cannot understood as primarily a matter of corrupt men in fundamentally sound institutions. The immorality of elite is a systemic feature of the "unstructured mass movements". "I've been rich and I've been poor," Sophie Tucker once said, "and believe me, rich is best."

The elite vested interests are sustained by the informal structures that exist and first of all the level of loyalty to the leader. that creates cult of personality problem that is clearly visible in some open source projects. At the same time without formal, structured elections the movement have no way of determining who shall exercise power within it. If the movement continues deliberately to avoid formalizing power structures, the members does not only abolish power, they also abdicate the right to demand that those who do exercise power and influence be responsible for their behavior.  If the movement continues to keep power as diffuse as possible and avoid electing its officers it cannot demand any responsible behavior from the elite, and at the same time can do nothing to prevent the forming of elite itself.

Jo Freeman in her article mentioned above also tried to outline the principles of democratic structuring. Most of them are well known to those who participated in various academic computer societies, such as ACM, IEEE Computer Society and USENIX. Of course, democracy as a very tricky mechanism, and principles are not a panacea as anyone experience with computer societies such as USENIX suggests. Elections are just a small step in the right direction and it does not prevent from forming elite that controls the organization pretty tightly. Of course, not all of them are applicable to open source movement. But at least they can provide a start of discussion: 

1. [Explicit] delegation of specific authority to specific individuals for specific tasks by democratic procedures. Letting people assume jobs or tasks by default only means they are not dependably done. If people are selected to do a task, preferably after expressing an interest or willingness to do it, they have made a commitment which cannot easily be ignored.
    
2. Requiring all those to whom authority has been delegated to be responsible to all those who selected them. This is how the group has control over people in positions of authority. Individuals may exercise power, but it is the group that has the ultimate say over how the power is exercised.
    
3. Distribution of authority among as many people as is reasonably possible. This prevents monopoly of power and requires those in positions of authority to consult with many others in the process of exercising it. It also gives many people an opportunity to have responsibility for specific tasks and thereby to learn specific skills.
    
4. Rotation of tasks among individuals. Responsibilities which are held too long by one person, formally or informally, come to be seen as that person's 'property' and are not easily relinquished or controlled by the group. Conversely, if tasks are rotated too frequently the individual does not have time to learn her job well and acquire a sense of satisfaction of doing a good job.
    
5. Allocation of tasks along rational criteria. Selecting someone for a position because they are liked by the group, or giving them hard work because they are disliked, serves neither the group nor the person in the long run. Ability, interest and responsibility have got to be the major concerns in such selection. People should be given an opportunity to learn skills they do not have, but this is best done through some sort of 'apprenticeship' programme rather than the 'sink or swim' method. Having a responsibility one can't handle well is demoralizing. Conversely, being blackballed from what one can do well does not encourage one to develop one's skills. Women have been punished for being competent throughout most of human history. The movement does not need to repeat this process.
    
6. Diffusion of information to everyone as frequently as possible. Information is power. Access to information enhances one's power. When an informal network spreads new ideas and information among themselves outside the group, they are already engaged in the process of forming an opinion without the group participating. The more one knows about how things work, the more politically effective one can be.
    
7. Equal access to resources needed by the group. This is not always perfectly possible, but should be striven for. A member who maintains a monopoly over a needed resource (like a printing press or a darkroom owned by a husband) can unduly influence the use of that resource. Skills and information are also resources. Members' skills and information can be equally available only when members are willing to teach what they know to others.