[HN Gopher] The Cathedral and the Bazaar (1999)
___________________________________________________________________
The Cathedral and the Bazaar (1999)
Author : wallflower
Score : 128 points
Date : 2023-05-14 17:01 UTC (5 hours ago)
(HTM) web link (www.catb.org)
(TXT) w3m dump (www.catb.org)
| chrisco255 wrote:
| I think for me, when I was first learning software development 12
| years ago, I had heard about Linux and open source, but I didn't
| really understand how it operated or organized itself. I had seen
| Wikipedia appear in the early 00s and understood that distributed
| groups could develop something better than centralized entities
| (such as Microsoft's Encarta or Britannica's encyclopedia), but
| the analogy of a centrally planned cathedral, carefully
| coordinated, vs the organized chaos of a bazaar, was useful for
| me in understanding why software development is quite unlike
| other engineering disciplines, especially once software was
| augmented with the ability to update itself over the internet.
|
| You _can_ build software like a traditional engineering project,
| with a chief architect and lead engineer drawing up plans along
| with a team of people that map out all the specs ahead of time.
| But the internet changed everything. It made distributed
| coordination possible, and long-running, complex open source
| projects that could outlive or outgrow their founding team,
| became achievable.
| tsimionescu wrote:
| But the reality today at least is that that is _not_ how big,
| successful open source projects get developed. They are very
| much handled as large engineering projects, with architects and
| lead engineers for each area or feature at least. You won 't
| get a new feature into the Linux networking stack without
| discussing and reviewing it with the network maintainers, for
| example. And ultimately you'll still need Linus' approval if it
| is a major feature.
|
| It's also important to note that some of the major open source
| projects are currently, in practice, collaborations of multiple
| large corporations. At least this is clearly how Linux works
| (the vast majority of contributions are coming from corporate
| players such as Microsoft, IBM, Intel etc), Clang, Kubernetes.
|
| Firefox and the GNU suite of tools are probably the largest
| exceptions, along with a few big projects developed by a single
| company with few outside contributions (JVM, ElasticSearch,
| Grafana, etc).
| microtherion wrote:
| Not only are many successful open source projects run as
| cathedrals, but often it turns out that hunchbacks are in
| charge of crucial parts of the cathedrals.
| coldtea wrote:
| That didn't age very well...
| joeman1000 wrote:
| It's an ESR post... what more could we expect?
| dang wrote:
| Related (surprisingly little over the years) - others?
|
| _The Cathedral and the Bazaar (1999)_ -
| https://news.ycombinator.com/item?id=35829361 - May 2023 (2
| comments)
|
| _On Management and the Maginot Line_ -
| https://news.ycombinator.com/item?id=33314682 - Oct 2022 (27
| comments)
|
| _The Cathedral and the Bazaar_ -
| https://news.ycombinator.com/item?id=16328219 - Feb 2018 (1
| comment)
|
| _The Cathedral and the Bazaar (2000)_ -
| https://news.ycombinator.com/item?id=12198625 - July 2016 (1
| comment)
|
| _Eric Raymond 's 'The Cathedral and the Bazaar' Turns 19_ -
| https://news.ycombinator.com/item?id=11754279 - May 2016 (57
| comments)
|
| _Revisiting the many eyes theory_ -
| https://news.ycombinator.com/item?id=8416597 - Oct 2014 (1
| comment)
|
| _A Second look at the Cathedral and the Bazaar (1999)_ -
| https://news.ycombinator.com/item?id=1222945 - March 2010 (1
| comment)
|
| _The Cathedral and the Bazaar_ -
| https://news.ycombinator.com/item?id=162376 - April 2008 (2
| comments)
| mulmen wrote:
| Is there some kind of effect where something is so well known
| that it doesn't get brought up? Like a reverse bathtub curve?
| dang wrote:
| I think there might be! It was sort of old hat by the time HN
| came along.
| bombcar wrote:
| dang failed to have hacker news running in 1999 I suspect
| [deleted]
| nickdothutton wrote:
| I feel compelled to mention that even today, the EU cannot grok
| the bazaar.
| smcin wrote:
| Can you cite some specific examples why you say that?
| zokier wrote:
| The ending paragraphs:
|
| > Perhaps this is not only the future of open-source software. No
| closed-source developer can match the pool of talent the Linux
| community can bring to bear on a problem. Very few could afford
| even to hire the more than 200 (1999: 600, 2000: 800) people who
| have contributed to fetchmail!
|
| > Perhaps in the end the open-source culture will triumph not
| because cooperation is morally right or software ``hoarding'' is
| morally wrong (assuming you believe the latter, which neither
| Linus nor I do), but simply because the closed-source world
| cannot win an evolutionary arms race with open-source communities
| that can put orders of magnitude more skilled time into a
| problem.
|
| Problematically I don't really see anywhere discussion where that
| large amount of skilled time originates; the economics of the
| situation. Even developers need to put food on the table.
| coldtea wrote:
| > _Very few could afford even to hire the more than 200 (1999:
| 600, 2000: 800) people who have contributed to fetchmail!_
|
| I'm willing to bet those "200" people are 1-3 core contributors
| and 197-199 people who once fixed some bug and have like 2
| commits in the whole codebase, or changed a comment's message.
|
| Also "Fetchmail" as some kind of high bar? Commercial companies
| have produced software with 10 to 100x the scope and user base
| (and quality). Including back then.
| [deleted]
| jwilk wrote:
| > 200 (1999: 600, 2000: 800) people who have contributed to
| fetchmail
|
| That doesn't sound believable. How did he come up with these
| numbers?
| jimbokun wrote:
| Looking through the commits?
| tptacek wrote:
| Notice how the argument breaks down.
|
| It is the case that the open source model has made Linux
| difficult to compete with (though that is in significant ways
| the result of commercial development; still, those corporate
| contributors were coerced into open source development by
| Linux's success, and would have done things differently in a
| VxWorks alternate history).
|
| But that same open source model produced Fetchmail. Most
| startups could produce something comparable to Fetchmail, even
| if forced to use 1999 tools. The open source model didn't turn
| Fetchmail into an insurmountably dominant tool in its space;
| Fetchmail had fallen into irrelevance even before Google Mail
| made all of these kinds of tools irrelevant.
| chubot wrote:
| Yeah the fallacy is that most open source comes from
| independent programmers
|
| What has undoubtedly happened since Raymond's time is that
| Intel, Google, etc became the biggest Linux contributors
|
| Mozilla has been almost entirely funded by search revenue for
| decades
|
| Big pieces of open source like the JVM and MySQL seem to
| require corporate stewards, for better or worse
|
| Microsoft did a 180 on open source, to some acclaim, which I'm
| still skeptical of
|
| So the framing is it a bit wrong now, and modern writing
| probably needs new terms to make these distinctions clear
|
| There is independent open source but it seems clear that most
| open source is funded by commercial entities
| tptacek wrote:
| Further, the largest open source projects --- Linux included
| --- are increasingly gatekept, and have their own
| priesthoods. And that's arguably been for the good!
| Silhouette wrote:
| But it's a different kind of gatekeeping to the
| increasingly exploitative and user-hostile world of
| commercial closed-source software. Perhaps "leaders" in a
| very literal sense would be a more apt term than
| "gatekeepers".
|
| That's one way that the FOSS model will always have a
| built-in advantage: if anyone steering the ship tries to
| throw the users overboard then someone else can always fork
| the project as a last resort. This has actually happened in
| a few significant cases and is always there as a warning
| sign to anyone thinking of steering a new course for the
| wrong reasons.
| tptacek wrote:
| I believe the same thing everyone else does about the
| superiority of open-source software development. I just
| don't think this article is a good explanation of where
| that superiority comes from. In large part, the
| importance of Linux's code being open is that it makes it
| safer for large corporations (Intel, Google) to invest
| time in it; that has nothing to do with a bazaar ethic!
| jasonwatkinspdx wrote:
| I've come around to thinking of corporate open source as
| a way companies can partition what they want to compete
| vs collaborate on. This is sort of a variation on the
| commoditize your complement strategy. Most of business is
| not zero sum, so I don't think it's particularly
| surprising competitors do agree on sharing the cost of
| maintaining infrastructure that isn't their moat or basis
| of market share.
| rektide wrote:
| Still I think incredibly useful frames for interpetting the
| world.
|
| Society so visibly can see & understand Cathedral entities. And
| we still are so weak at grokking the rhiziomatic Bazaar
| assemblages that happen.
| jonstewart wrote:
| The frame largely already existed, though, even in software,
| e.g., "Worse is Better" (https://dreamsongs.com/WIB.html).
| rektide wrote:
| Worse is better is a discussion on system design. Both
| parties represented in worse better are implicitly
| cathedrals: both are singular actors, both with the same
| value system even (Simple, Correct, Consistent, Complete).
|
| None of this embodies the Bazaar to me. The spirit of
| diversity, chance, chaos, & above all the pluralism: the
| ability to get Perl style post-modern in approaches, is
| largely absent from the WIB discussion.
|
| The Bazaar is multisystem, multi-agent, in a way that most
| people building software don't tangle with. It's
| fundamentally different if you can encompass much more than
| yourself, than any one effort.
|
| The Cathedral & The Bazaar is a book that should make these
| kinds of distinctions in thinking abundantly apparent. Few of
| us can ever really experience or witness the Bazaar. It's too
| vast. The book gives us an impressive tale, where we can
| begin to see wide exploration & reasoning occuring far beyond
| the scales that even the biggest boldest widest orgs on the
| planet can see.
| cgh wrote:
| Thematically related: "The Square and the Tower: Networks and
| Power, from the Freemasons to Facebook", by Niall Ferguson
| mastermedo wrote:
| The Cathedral and the Bazaar often comes in discussions along
| with the Song of Sisyphus[0], a story of how Sisyphus software
| became widely used within Google. The use of it spread over bar
| drinks, completely organically. And Google had a very hard time
| deprecating it.
|
| [0]: https://www.oreilly.com/library/view/a-case-
| study/9781098114...
| tptacek wrote:
| Right. If you want something that has perhaps a more profound
| and interesting description of the open source model (as
| Zawinski would put it, the "CADT model"), it's worth going back
| to Richard Gabriel:
|
| https://www.dreamsongs.com/WorseIsBetter.html
|
| Gabriel's essay is cited in Raymond's article. It's interesting
| to compare them; Raymond claims that Gabriel anticipated much
| of what his article was going to say, but I think they have
| markedly different takes about it.
| tptacek wrote:
| Just taken on its merits, I think a case can be made that this is
| one of the most overrated pieces of technical writing of the last
| 25 years. What's true in it isn't interesting ("the importance of
| having users", "release early release often") and what's
| interesting isn't true ("Linus's law" being perhaps the most
| notorious example). Much of the insight is taken directly from
| Brooks. The whole piece has as its backdrop the development of
| Fetchmail, which is not a well-regarded piece of software.
|
| What's notable about Cathedral is its timing; it did capture the
| zeitgeist of what was an important moment in the computing field,
| the moment where we transitioned from 386bsd-style hobby projects
| to an industry run on free and open source software. But Raymond
| isn't the reason why any of that happened, and much of his
| description of that moment is faulty; the rest of it is just a
| retrospective of the engineering decisions involved in the
| writing of a midlist mail processing utility (fetchmailrc syntax,
| password encryption, the now-largely-irrelevant distinctions
| between MDAs and MTAs).
|
| Even the high-level organizing notion of "cathedrals" and
| "bazaars", which should have been a lay-up, hasn't really proven
| out.
| ghaff wrote:
| The main criticism (IMO) isn't really the work itself but how
| it's handed down as wisdom. It wasn't really
| cathedral==proprietary and bazaar==open source. In fact it was
| about cathedral vs. bazaar in open source software and that
| story is more complicated and comes down to "It depends." There
| are certainly open source projects that have more of a
| cathedral aspect whether because of benevolent dictator model
| or, more recently, a strong commercially-oriented foundation.
| But there are also areas that look more like a bazaar which
| have some advantages with respect to innovation but can be...
| messy.
| antod wrote:
| Yeah I never felt proprietary was the cathedral, more like
| GNU was the target there.
| ghaff wrote:
| At least GCC. But proprietary vs. open was the common
| understanding.
| capr wrote:
| so much hindsight bias, sure, everything you take for granted
| now is not interesting, and for the parts that you think are
| not true you have just as little evidence as he had in 1997
| colinsane wrote:
| > what's interesting isn't true ("Linus's law" being perhaps
| the most notorious example)
|
| > Even the high-level organizing notion of "cathedrals" and
| "bazaars", which should have been a lay-up, hasn't really
| proven out.
|
| you and i live oceans apart. a year of working around NixOS and
| especially linux on mobile phones has me working in very
| bazaar-like systems -- where every component is
| swappable/composable -- and seeing/using Linus' law daily -- in
| that people from out of the blue will patch things i'd been
| unable to figure out and when i hit edge cases that look
| familiar enough in any software i install i will debug it &
| send fixes upstream.
|
| i'm well aware the above is niche. on the other hand i'm fairly
| confident that niche would be inaccessible to most of us who
| are otherwise participating in it were it not leveraging these
| two concepts.
| tptacek wrote:
| We do. As a vulnerability researcher, my take on "Linus's
| law" (which, like "Gell-mann Amnesia" isn't the product of
| the famous person it's named for) --- "given enough eyeballs,
| all bugs are shallow" --- is that has been effectively
| refuted by the recurrent discovery of grave vulnerabilities
| that have hidden in plain sight, sometimes for over a decade,
| and with the success of commercial vulnerability research
| work in eradicating those vulnerabilities.
| mjw1007 wrote:
| I don't think "bugs are shallow", or the "many eyeballs"
| section of this essay, are particularly talking about
| _discovering_ bugs.
|
| The author seems to have had a worldview in which bugs
| don't really matter if people aren't coming across them,
| and in which the difficult part of dealing with a bug is
| either reproducing it or getting from a symptom to a cause.
|
| If you were in a world where those two things are true then
| I think he's probably right that "many eyeballs" would help
| a great deal.
| tptacek wrote:
| It's not interesting to say that lots of interesting
| eyeballs are helpful. Anybody would have said that prior
| to this article's publication. Raymond makes a much
| stronger claim (which is why it has the force of "law"),
| and it hasn't borne out.
| mjw1007 wrote:
| Again: the purported law says "shallow", not "soon
| discovered".
| coldtea wrote:
| If they're not "soon discovered" then the use of the term
| "shallow" means absolutely nothing...
| tptacek wrote:
| No, you can't get away with this semantic dodge, because
| Raymond numbered what he believed were the most important
| lessons he was imparting, and the one corresponding to
| Linus' Law is:
|
| _8. Given a large enough beta-tester and co-developer
| base, almost every problem will be characterized quickly
| and the fix obvious to someone._
|
| He even attempted an axiomatic explanation:
|
| _Maybe it shouldn 't have been such a surprise, at that.
| Sociologists years ago discovered that the averaged
| opinion of a mass of equally expert (or equally ignorant)
| observers is quite a bit more reliable a predictor than
| the opinion of a single randomly-chosen one of the
| observers. They called this the Delphi effect. It appears
| that what Linus has shown is that this applies even to
| debugging an operating system--that the Delphi effect can
| tame development complexity even at the complexity level
| of an OS kernel._
| phkahler wrote:
| >> Given a large enough beta-tester and co-developer
| base...
|
| That's a qualifier for what follows it. Seems true enough
| to me. If there are enough developers to notice bugs,
| they will probably be found and fixed quickly.
| mjw1007 wrote:
| Well, I'm baffled then. From where I'm sitting that point
| 8 is clearly talking about what happens after a bug is
| discovered, and not about discovering bugs.
|
| The longer paragraph doesn't seem to contradict the
| notion either. My impression (based on the "How Many
| Eyeballs Tame Complexity" chapter) is that Raymond
| thought that "debugging" means "fixing bugs".
|
| If I were criticising this part of essay, I'd say the
| main weakness is that the things Raymond thought of as
| "taming complexity" weren't really addressing the hard
| problem of reducing the number of bugs.
| anamexis wrote:
| "Characterizing problems" reads to me like "discovering
| bugs," particularly given that he specifically mentions
| having lots of beta users.
| jasmer wrote:
| It's interesting to say 'a lot of eyeballs will do
| something' in a world where nobody believed it.
|
| I think this thread overstates the reality of the very,
| very crude metaphor that illustrates 'why open source can
| work' - even if it is very flawed.
|
| I think by now we all kind of know 'it depends' aka
| there's context to everything.
| lmm wrote:
| None of the significant security vulnerabilities that I can
| remember have been "deep" - subtle things that require
| extensive familiarity with the specific codebase and could
| never have been found by a drive-by contributor, which is
| the idea that ESR is refuting. Debian keygen bug? Dumb one-
| liner. Heartbleed? Dumb one-liner. Goto fail? Technically a
| logic bug, but a two-line thing that could be understood by
| reading that one file without any additional context.
|
| I've seen cases where exploitation was complex, but even
| then, that tends to be because the full exploit is chaining
| together several bugs, each of which is individually simple
| and could be fixed by a drive-by contributor.
| hn_throwaway_99 wrote:
| > What's true in it isn't interesting (... "release early
| release often")
|
| This feels like 20/20 hindsight to me given that "release early
| release often" was _definitely_ not the "axiom" that we think
| of today, and one could argue the contrary opinion was more
| widespread in 1997.
|
| I can easily remember some debates at a large consumer web
| company in the mid 00s, where at the time we released once per
| quarter. We were struggling with product quality, and a large,
| significant portion of the software dev team argued that the
| way to improve our product quality was to _reduce_ the number
| of our releases to something like only 2 per year. The argument
| was that we needed more QA time, needed more time to run
| integration tests, needed more time to ensure changes from all
| the different teams wouldn 't break each other before
| releasing.
|
| In our current world of CI/CD, a web company releasing only
| twice a year sounds absolutely insane. Fortunately the loud
| "slow down our releases" contingent lost (which is evident
| simply by the fact that this large tech company still exists -
| if they had reduced their release cycles as some desired I
| firmly believe the company would have died long ago). But I
| just bring this up because the mantra of "release early release
| often" was definitely not blatantly self-evident in 1997, and
| I'd argue it only became "axiomatic" after tooling support for
| CI/CD got much better.
| tptacek wrote:
| "Release early release often" was a mantra of "Extreme
| Programming", a closed-source commercial software development
| methodology that predates this article by about 4 years, and
| was _au courant_ at the time Raymond was was writing. One of
| my big thematic criticisms of Raymond 's article is that it
| doesn't seem especially in touch with how closed-source
| development worked at the time.
| dannyobrien wrote:
| I'm sympathetic to the idea that C&B is overrated, but it
| was published in 1997, and XP was only being fleshed out at
| the C3 program in 1996. The Agile manifesto -- in 2001 --
| was what gave it its major visibility bump.
|
| Release early and often was in the air in the late
| nineties, but C&B was legitimately the first time many
| people got to hear about it.
| tptacek wrote:
| I dispute some of this, only because I was doing a
| software startup from '98-'01 and we managed to hire not
| one but two XP devotees that dragged me into reading this
| stuff. To this day, though, I'm still mostly unfamiliar
| with the particulars of Agile.
|
| I get that Agile was bigger than XP (who could deny
| that?), and agree preemptively that Raymond's article is
| probably the first time many people heard about
| incremental release strategies. That's true of a lot of
| things in it! My big complaint about those ideas is that
| they're not Raymond's --- not even in the sense of
| waiting to be distilled from Linux by Raymond.
|
| The true-feeling parts of Raymond's article read to me
| like a document of, for lack of a better term, late 1990s
| programming thinking. Just a bunch of stuff that was
| floating in the air, a bunch of Fred Brooks, and then
| weird attempts to generalize design decisions in
| Fetchmail to the whole of software development.
|
| I appreciate getting called on this and forced to think
| more carefully about it. Thanks for the response!
| gonzo wrote:
| I posit that pg's initial business plan for Viaweb from
| 1995 anticipated CI/CD. http://paulgraham.com/vwplan.html
|
| IMO, CATB was never good.
| dannyobrien wrote:
| Things certainly happened very quickly in that period --
| it's staggering to think that 1997 was only four years
| after the web began to percolate, and C&B I think got its
| lift because it happened when Netscape decided to open
| source their browser.
|
| It's definitely hard to work out a chronology when things
| are piling one after the other, and also there wasn't the
| same consistency of knowledge propagation. It was still
| very hard to find out new things online.
|
| Thanks for the thanks!
| doctor_eval wrote:
| I completely agree with you about C&B. Even at the time,
| it felt to me mostly like a restatement of the zeitgeist.
| But Raymond was very influential at the time and was
| apparently one of the reasons Netscape released Mozilla
| as free software.
|
| I also agree that "release early and often", in
| particular, was a significant contribution, if not for
| originality than for reach. Not everyone was exposed to
| XP at the time, but ESR seemed to be everywhere. And for
| my part, the main take away from XP had been around pair
| programming which as a startup myself, I wasn't
| into/couldn't afford/didn't subscribe to anyway.
|
| But I do feel your comments about XP and agile miss the
| mark a tiny bit. XP was developed by Kent Beck - who went
| on to be one of the authors of the Agile Manifesto.
| Because of this, XP is considered an agile practice.
|
| But agile is really just a set of values and principles.
| There are many practices that people claim as "agile",
| many of which do not meet the criteria of the agile
| manifesto (I'm looking at you, SAFe).
|
| I think in some ways, C&B was a precursor to the Agile
| Manifesto, which itself it arguably just a restatement of
| principles and practices that already existed.
|
| (My limited personal experience with engineering in large
| traditional companies is that the manifesto is ignored,
| commercial practices and consultants who slap "agile"
| stickers on their traditional SDLCs still rule - and it
| is still, largely, the dark ages. I was heavily
| criticised for using a CD strategy for some enterprise
| software at the same brand name company that required me
| to enforce password rotation... in 2018).
|
| https://en.m.wikipedia.org/wiki/Extreme_programming
|
| https://agilemanifesto.org/
| bwaine wrote:
| Is Extreme Programming a closed source commercial software
| development methodology? What makes it so?
|
| I went through a period of my career where I dived head
| long into it, read Kent Beck's book, liked what I read.
| Tried pair programming, TDD etc, loved it. Found team that
| felt the same and had a great couple of years.
|
| Given the book, the many conference talks etc and comparing
| it to other flavours of agile that went full corporate
| (Scrum, SAFE). I'm surprised to hear it described as closed
| source.
| tptacek wrote:
| My impression of XP at the time was that it was a
| methodology designed for consulting firms, and that most
| of its force came from the idea that it was an insurgent
| effort at reprogramming stodgy waterfall development
| processes at big companies; all of those companies ---
| the whole client base of XP consulting firms (which I'm
| assuming was a big thing) was closed source, because
| almost everything was at the time.
| skeeter2020 wrote:
| I personally agree with the frequent release crowd, but my
| company sells a web-based SaaS product and releases
| quarterly, my last company released a web SaaS product 2x a
| year and SalesForce famously releases closer to once per
| year, so a large part my the world is "still insane" based on
| your opinion.
| andyjohnson0 wrote:
| What happened to esr? His blog has been dead for a few years.
| paulddraper wrote:
| https://news.ycombinator.com/item?id=35829361
| dang wrote:
| Users probably flagged your post because on HN a previous
| thread is only considered a dupe if it got significant
| attention, and that one didn't. (This is in the FAQ:
| https://news.ycombinator.com/newsfaq.html.)
|
| Btw we invited wallflower to repost it (which is part of how we
| get things into the second-chance pool
| (https://news.ycombinator.com/pool, explained at
| https://news.ycombinator.com/item?id=26998308), because we
| noticed that not only did that submission not get attention,
| the essay really hadn't been discussed much on HN over the
| years.
___________________________________________________________________
(page generated 2023-05-14 23:00 UTC)