[HN Gopher] Ten years against division of labor in software
___________________________________________________________________
Ten years against division of labor in software
Author : akkartik
Score : 130 points
Date : 2022-01-21 04:25 UTC (1 days ago)
(HTM) web link (akkartik.name)
(TXT) w3m dump (akkartik.name)
| phendrenad2 wrote:
| Do not use a library without reading it's source code. Software
| problems are hard, and you can't hide from the complexity by
| letting someone else do the thinking. You have to know what the
| library author knows, so you can determine what's possible, and
| decide if the library provides you with the right abstraction.
| cortesoft wrote:
| There is no way I have time to read and understand every
| library I use. I have to read all 100k+ lines of SQLite before
| I can use it?
| eternityforest wrote:
| ...
| erikerikson wrote:
| > Software libraries suck. Here's why, in a sentence: they
| promise to be abstractions, but they end up becoming services.
|
| The file abstraction seems fine. Not having to understand
| physical wiring of disks, the specific disk manufacturer's wire
| protocol, and so on is Labor saving. Many of the abstractions
| don't even force the seek and location concepts but allow "just
| put my string there" semantics.
|
| The addition abstraction seems fine though if you have a limited
| length number type there are a few things to consider. Arbitrary
| precision libraries eliminate those nuances though perhaps move
| you to others like numbers that cannot fit in available memory.
| Yet those are rather extreme corner cases.
|
| It's a provocative claim but the universal declaration and the
| confidence it seems to project seem as weak as some of those
| abstractions being railed against.
| abvdasker wrote:
| That's a wildly impractical position to take and isn't really
| true in my experience. I spent several years at a seed stage
| startup where it was me on backend and one other engineer on
| mobile. We would specify a JSON API and then could work
| independently (and very fast) to realize the functionality
| because he's an expert at mobile programming and I am an expert
| at backend. We didn't have the luxury of working at 50% output
| for a year to learn each other's specialization while trying to
| get the product off the ground. He didn't need to learn the ins
| and outs of rate limiting with redis and I didn't need to know
| how to animate a custom UI component for each of us to be
| maximally productive.
|
| Division of labor and the interfaces which enable it exist for a
| reason. I suspect it's because it reduces the complexity of
| collaboration between engineers with different specializations.
| It facilitates the separation of concerns.
| datavirtue wrote:
| I have been ruminating on this very thought for some time... "I
| started out mocking NPM for the left-pad debacle, but over time
| I've actually grown more sympathetic to left-pad. The problem
| isn't that somebody wrote a library for left-pad. It is that even
| something as trivial as left-pad has triggered our division-of-
| labor instincts. A library like left-pad is just begging to be
| saved into projects that use it so users have one less moving
| part to worry about at deployment time. Turning it into a first-
| class dependency also ensured that a lot less people looked
| inside it to understand its five lines of code or whatever."
|
| I think we should be extracting code from libraries until we need
| the whole thing. Dragging around black box dependencies as
| binaries has a lot of serious drawbacks. If people had extracted
| the parts they needed from log4j they wouldn't have even had to
| think about the exploits.
| Jtsummers wrote:
| Pulling out just the subcomponents you need is actually easier
| with a more modular design. But akkartik also opposes
| modularity, or at least has expressed a less-than-positive
| disposition towards it:
|
| > I think namespaces and modules are socially counter-
| productive even when implemented in a technically impeccable
| manner, because they encourage division of labor.
|
| Namespaces and modules are precisely what you need to be able
| to select parts of a larger system and ignore other parts.
| Without them, you're faced with numerous "monoliths",
| excessively large systems that don't decompose into reusable,
| isolatable bits. That forces you to import too much, or to
| recreate just the part that you care about (often a waste of
| time).
| ReactiveJelly wrote:
| I was very surprised by that line, too.
|
| I make namespaces and modules in my own projects just so that
| I can use simple names like "update" or "load" un-
| ambiguously. Projects that nobody else has ever touched and
| maybe nobody ever will.
|
| It's been a popular C++ style since before I did C++, and now
| it's also idiomatic Rust style.
|
| Is OP really advocating for a C style where everything must
| be fully-qualified? It's horrifying to read GTK+ C code
| because of that. Everything is `gtk_do_a_thing` and all code
| operates on a `GtkTypeOfGtkObject`. One of these days I'll
| get RSI, I only have so many keystrokes left in me! Have
| mercy!
|
| And you can't rename things, so good luck mixing incompatible
| major versions of Qt or GTK+ in a C or C++ project.
| (Actually, Qt has a compile-time flag to put it in a
| namespace. Maybe GTK+ has some similar ugly macro. But this
| requires you to compile your own dependencies, a task which
| is especially annoying, inconvenient, and error-prone in the
| C family)
| ReactiveJelly wrote:
| I don't even think micro-packages are that bad. The problems
| with NPM seem to have been that the "Always upgrade
| immediately! Upgrades are good for security and have no
| downsides!" people are winning the PR fight against us "Upgrade
| cautiously and never allow autonomous upgrades. Upgrades are
| important for security but they have many caveats" people.
|
| One is a more Windows attitude of "I know you've never gotten a
| virus, but if you don't upgrade, you _will_ get a virus and you
| _will_ die" whereas my attitude is more of a Linux "The only
| time my Linux system breaks is when I upgrade it."
| throwaway984393 wrote:
| HN trope #317: blog post preclaiming that the one thing we all
| know makes sense and has been proven to work for decades is
| actually wrong.
| [deleted]
| [deleted]
| draw_down wrote:
| [deleted]
| eternityforest wrote:
| The fact that a library discourages you from learning it's
| internals is how you know it's well done. If you needed to
| understand it, you could. The reason you don't is because you've
| never had a compelling reason to.
|
| If only one person knows how a library works, that is a problem.
| If a 100 person team maintains it and you're not on that team....
| it's probably because you have other stuff to do.
|
| Software is an engineering discipline. Computer science is a
| science. If you are in programming because you want to advance
| our understanding, great, go work in one of the many fields with
| large novel algorithms that need to be understood.
|
| For a typical programmer... Look around. Modern software is one
| of the best things about the modern world. It does SO much for
| us. Do you really think we, with a real world distribution of
| programmer abilities, could do all this without massive division
| of labor? How much would be missing if we insisted on
| understanding everything before we use it?
|
| I suspect very few alternative approaches to software would work
| as well to truly build the modern software defined world where
| everything from cars to taxes to photography is digital.
|
| Because... whenever someone tries an alternative approach, there
| usually seems to be the hidden unspoken assumptions that they
| actually don't want software to be as big as it is.
|
| The end product of software itself these days is a service(in the
| sense of this article, not the SASS sense), wheras software built
| with an understanding-first mindset seems to usually wind up
| being closer to a digital version of a passive analog tool.
| [deleted]
| akkartik wrote:
| > The fact that a library discourages you from learning it's
| internals is how you know it's well done. If you needed to
| understand it, you could. The reason you don't is because
| you've never had a compelling reason to.
|
| I don't think we need to protect people from learning
| internals, they do that just fine on their own. I know many
| situations where we failed to understand internals even in the
| presence of compelling reason.
|
| With apologies for continuing to quote myself:
|
| "I want to carve steps into the wall of the learning process.
| Most programs today yield insight only after days or weeks of
| unrewarded effort. I want an hour of reward for an hour (or
| three) of effort." -- http://akkartik.name/about
| geofft wrote:
| The difficult part about having opinions about the industry
| of software engineering is that it is, somehow, a largely
| enjoyable job.
|
| But those two things are never going to be perfectly aligned.
| Enjoyment of the process does not guarantee working results
| that people want to pay for, and the thing that people want
| to pay for may be deeply unenjoyable. You can choose to
| tackle a subset of the problem space that happens to be both
| enjoyable and worthwhile, but you ought to admit that you are
| looking at a subset of the problem space.
|
| I do a lot of other things with my time that are also near
| the confluence of enjoyable and marketable. I sing, for
| instance, and choir rehearsal and individual vocal practice
| are both a lot of fun. But any professional musician will
| tell you that if you want to be world-class at it, you
| occasionally have to wring the fun out of it. I've chosen the
| other option: I'm a volunteer singer with my church choir. I
| sing because it is fun, and only insofar as it is fun. We do
| new and not-very-difficult music each week, and an hour of
| effort is inherently an hour of reward. If my choir director
| said we're going to spend six months drilling some difficult
| measures and perfecting our German pronunciation, we'd all
| leave.
|
| (In that sense, a volunteer church choir is rather like
| hobbyist open-source development. If it produces an enjoyable
| result for the public, great, and we do find that rewarding.
| But there is an upper bound on just how hard we're going to
| try and on the complexity of what we commit to doing.)
|
| If you want an hour of reward after an hour of programming
| effort, that's fine! But the employment contract I've signed
| promises a year of payment after a year of effort.
| _Occasionally_ , I want to work on things that are
| immediately rewarding because that helps with motivation. And
| it's important that such work is available. But I have a job
| that ultimately demands specific products, some of which just
| aren't going to be immediately rewarding, some of which do
| actually take a lengthy learning process, and - most
| importantly - many of which require building on top of other
| people's work that I _could_ understand if I wanted but there
| is _no business value_ in doing so up front.
|
| (Incidentally, in the other direction, there are cases where
| I want to tackle problems that promise years of reward only
| after multiple years of effort, and figuring out how to get
| time to work on those problems is actually hard, too.)
|
| We share almost all our code internally and leverage lots of
| open-source code so that we all have the option of
| understanding the code if we need it - but we rely on
| division of labor so that we have the option of building
| something on top of the existing base today, if we need that,
| which is more often what we need.
|
| If you want to work on a hobbyist UNIX-like kernel for
| enjoyment's sake, great, I'm genuinely happy for you. But my
| servers at work are going to keep running a kernel with
| millions of lines of code I've never read, because I need
| those servers to work now.
| akkartik wrote:
| Thanks, I largely agree with that. I just don't think we're
| getting a good deal _for society_ with a year of effort
| from a paid programmer. When I said "reward" I meant
| understanding, not enjoyment. We programmers are privileged
| enough, I'm not advocating to optimize for our fun as well.
| marktani wrote:
| what is an hour of reward? I understand reward in terms of
| some form of a joyful/insightful/... experience, not in terms
| of time
| coldtea wrote:
| Parent talks about how fast you get back results from
| effort (learning, fiddling, etc) spent:
|
| "Most programs today yield insight only after days or weeks
| of unrewarded effort. I want an hour of reward for an hour
| (or three) of effort."
|
| So they want to be rewarded with at least an hour of saved
| time, or more productive time, for one (or few) hours they
| put in, instead of spending weeks to reap rewards.
| akkartik wrote:
| I was just alluding to a subjective sense that you got
| something reasonable for your trouble and efforts. You
| could do something, or you learned something, or you were
| entertained. My argument doesn't depend too much on the
| precise meaning, I think.
| akkartik wrote:
| I think an older version said, "an hour's worth of reward."
| Does that seem clearer? Thanks.
| hinkley wrote:
| I've been coming around to the conclusion that some coding
| patterns, especially overuse of delegation and mutation of
| inputs, make code hard to learn.
|
| I talk occasionally about viewing your code from the debugger
| but that is kind of hand wavy. I wonder if there's a 'linter'
| one could write that looks at coverage or trace reports and
| complains about bad patterns.
| gopher_space wrote:
| I feel like better code visualization would solve a lot of
| my problems. Or at least highlight them.
| hinkley wrote:
| As I've spent more time with flame graphs I realize they
| are, once you get down to brass tacks, the wrong tool for
| perf analysis because it's the _width_ of the flame that
| tells you the most, not the height, and we usually worry
| about height when thinking about actual flames.
|
| However there are all sorts of little subtle costs in
| your system that aren't captured by most of these tools
| due to lack of resolution (I haven't spent a lot of time
| with Intel's hardware solution) and asynchronous costs
| like memory defragmentation. Depth and frequency of calls
| are a useful proxy for figuring out what rocks to look
| under next after you've exhausted the first dozen. For
| this reason the flame graph is a useful fiction so I
| don't poo-poo them where anyone can hear. I can barely
| get people to look at perf data as it is.
|
| But then I think how I'm turned off by some fictions and
| avoid certain fields, like parser writing, and wonder if
| a more accurate model would get more people to engage.
| tlarkworthy wrote:
| yes, lately the Butler Lampson quote echos round my head
| "All problems in computer science can be solved by another
| level of indirection"
|
| The problem is, we have been adding layers of indirection
| for 80 years, and each is leaky. So now it's very difficult
| to do basic stuff, and people are ok with software thick
| with abstractions.
|
| The next stage should be removing unnecessary layers of
| indirection, as, like you said, things are much easier to
| understand and maintain that way.
| randallsquared wrote:
| It's very easy to do basic stuff. It's hard to do basic
| stuff (or anything) _well_ , since you end up having to
| solve problems that should have been solved by layers
| you're building upon.
| tejtm wrote:
| The remainder of the quote Butler quoted matters;
|
| "All problems in computer science can be solved by
| another level of indirection, except for the problem of
| too many layers of indirection." -- David Wheeler
| edgyquant wrote:
| Programming seems to be one of the few fields where we think
| it's a bad thing if our tools aren't made by us (this has
| luckily been waning in recent years, but the push back against
| "bloat" we see on every tech forum is proof that the mindset is
| entrenched.
| analog31 wrote:
| I actually think this applies to the passive analog world too.
| Very few engineers could tell you how steel or concrete are
| made. And we all use abstractions / libraries, for instance
| using tabulated properties of steel alloys rather than knowing
| the underlying physics.
|
| In fact, if put in plainer terms, every engineer would nod in
| agreement with Hyrum's Law. Everybody has a story of a design
| that worked because of an undocumented property of a material,
| that stopped working when Purchasing changed to a new supplier
| of that material.
| xorcist wrote:
| On the contrary. _Every_ engineer is able to learn the
| process when it matters. And it often does. The type of steel
| used, the treatment and the orientation is imporant to the
| outcome. You can not hand wave them away as "implementation
| details".
| Jtsummers wrote:
| I don't see this is a disagreement with the GP. You say
| they (all engineers) can learn it, GP says they (few
| engineers) could tell you how it's done. Those are in
| perfect agreement. At any moment in time, there is likely
| no engineer who can explain fully every process used by
| their domain or object created by others in their domain.
| That doesn't mean that the majority of those engineers
| couldn't learn any arbitrary process or the details of an
| arbitrary object in their domain.
| imbnwa wrote:
| If you're minimizing challenge for your software engineers,
| you're making worse engineers over time
| ramesh31 wrote:
| >If you're minimizing challenge for your software
| engineers, you're making worse engineers over time
|
| Or you're solving your current problems in the most
| efficient means possible, e.g. _engineering_. Minimizing
| one challenge frees mental processing power to worry
| about bigger issues. I couldn 't care less how the memory
| paging in my OS works. I care about building features
| that users find valuable.
| imbnwa wrote:
| Such a needlessly extreme example, sometimes it might
| mean just working on the backend after doing nothing but
| frontend for your career, sometimes it might mean not
| assigning a feature of a certain complexity to the person
| you know likely grasps it instantly and assigning it to
| someone who you're less sure about but wants to do more.
| It can be just as much about figuring out and cultivating
| the potential of your workforce and _that_ can create
| greater efficiency _over time_. There are certainly
| opportunities where this can be accomplished, because if
| everything is mission critical, business itself is doing
| something wrong. The amount of opportunity and the nature
| of such will vary across businesses, and perhaps an
| engineer may need to go elsewhere to search out further
| challenge. Your statement that "Minimizing one challenge
| frees mental processing power to worry about bigger
| issues" presumes a level of uniformity.
| davidgay wrote:
| > Every engineer is able to learn the process when it
| matters.
|
| A civil engineer is going to become an expert material
| scientist, "on need"? I doubt it.
| xorcist wrote:
| Of course not. A basic understanding? Absolutely, yes.
|
| It's even widely accepted to be an important part of the
| education.
|
| Just as Karnaugh maps and parser theory is part of
| computer science engineering curriculums. It's not
| something that's expected to be used a daily basis but
| some general knowledge of how a processor works is
| absolutely necessary to be able to at least talk about
| cache line misses, performance counters, mode swaps etc.
| newshorts wrote:
| I have to agree. The article quotes sound right and wonderful,
| but thinking about the environment in which I work, there's not
| really a practical way to combine our efforts. We simply
| wouldn't get things done in time.
|
| Division of labor ensures that changes can happen reliably and
| quickly. I'm ramping on an internal library right now and it's
| taking a while to understand it's complexities and history. I
| can't imagine making quick changes to this without breaking a
| lot of stuff. It will take time to develop an intuition for
| predicting a change's impact.
|
| Now multiply my ramp up time by every dev on my team every time
| they need to make an update. You can imagine productivity slows
| down and cognitive overhead rises.
| omegalulw wrote:
| > Do you really think we, with a real world distribution of
| programmer abilities, could do all this without massive
| division of labor? How much would be missing if we insisted on
| understanding everything before we use it?
|
| I agree that you don't want to spend time becoming super
| familiar with everything that you use but you should ALWAYS
| have a high level idea of what's happening under the hood. When
| you don't it inevitable leads to garbage in garbage out.
| fuzzfactor wrote:
| >real world distribution of programmer abilities
|
| Way before the arrival of personal computers, it was clear
| for me to realize that I would be highly disappointed if
| average programmer abilities declined half as far as they
| have by now.
|
| Some of the most coherent technical projects are the result
| of a single individual's outstanding vision.
|
| Occasionally, the project can be brought to completion by
| that one individual, in ways that simply could not be
| exceeded alternatively, all the way from from fundamental
| algorithms through coding to UX and design.
|
| Additional engineers could be considered useful or essential
| simply to accelerate the completion or launch.
|
| When the second engineer is brought on board, it may be
| equally performant to have them come fully up to speed on
| everything the first engineer is doing, compared to having
| them concentrate on disparate but complementary needs which
| need to be brought to the table in addition.
|
| If they both remain fully versed, then either one can do
| anything that comes up. Sometimes they can work on the same
| thing together when needed, other times doing completely
| different things to make progress in two areas
| simultaneously.
|
| You're going to end up with the same product either way, the
| second engineer just allows an earlier launch date. But truly
| never within half the calendar time, those man-months are
| legendarily mythical.
|
| For projects beyond a certain size more engineers are simply
| essential.
|
| Then ask yourself what kind of team would you rather have?
|
| All engineers who could contribute to any part of the whole
| enchilada from primordial logic through unexceeded UI/UX?
|
| Or at the opposite end of the spectrum, all engineers who are
| so specialized that the majority of them have no involvement
| whatsoever with things like UI/UX?
|
| Assuming you're delivering as satisfactory a product as can
| be done by the same number of engineers in the same time
| frame.
|
| You're _NOT_ going to end up with the same product either
| way.
|
| Now aren't you glad it's a spectrum so it gives you an
| infinitely greater number of choices other than the optimum?
| nitwit005 wrote:
| Few library and service creators seem to think you can treat
| what they built as a black box. At the very least, they
| generally come with some advice on usage patterns to avoid.
|
| Writing performant code unfortunately tends to require at least
| having a basic working model of how all your dependencies work.
| Otherwise, you tend to find yourself debugging why you ran out
| of file handles in production.
| imbnwa wrote:
| "Division of labor is an extremely mature state for a society.
| Aiming prematurely for it is counterproductive. Rather than try
| to imitate more mature domains, start from scratch and see what
| this domain ends up needing."
|
| This is more the gist of the article than "lets not have
| libraries or abstractions at all", its pointing more towards
| "lets defer those questions until the right time rather than
| from the jump".
|
| I think Enterprise Java and the insane complexity of modern web
| dev are indicative of a consequence.
|
| >Because... whenever someone tries an alternative approach,
| there usually seems to be the hidden unspoken assumptions that
| they actually don't want software to be as big as it is.
|
| I really don't think that OP is making this assumption
| hinkley wrote:
| People's opinions on this stuff contain a lot of unstated
| pain points. Monolith first argues against a coping mechanism
| people use because they hate Big Ball of Mud projects.
|
| Anyone who has ever done repair work can tell you how often
| people wait until it's far too late to do basic maintenance.
| Software is similarly bimodal. People either run it till it
| falls apart, or they baby it like a collector's item.
|
| We have not collectively learned the trick of stopping at the
| first signs of pain or "strange noises" and figuring out
| what's going on. But mostly we have not learned the trick of
| insisting to the business that we are doing our job when we
| do this, instead of plowing on blindly for six to eighteen
| months until there's a break in the schedule. By which time
| everything's a mess and two of the people you hoped would
| help you clean it up have already quit.
| akkartik wrote:
| Totally agreed!
| Frost1x wrote:
| I think most software engineers know when it's time for
| maintenance. The issue is that the people who write their
| paychecks don't want to hear it.
| hinkley wrote:
| I read too much commit history, including my own, to
| agree with that. We are too close to the problem, and we
| enjoy deep dives too much.
|
| I save myself a lot of frustrating work by stopping for a
| cup of coffee or while brushing my teeth or washing my
| hair. "If I change Z I don't need to touch X or Y"
| wrnr wrote:
| In my lifetime Apple has changes hardware platform twice while
| still running on the Next Foundation API. Same thing with
| politicians, they never simplify the complicated laws but instead
| expect software to manage and work around this complexity.
| Nothing to do with the division of labor but more the packing
| order of software engineers.
| ape4 wrote:
| Don't think about how a library works - until you have to
| WJW wrote:
| How do you know when you have to? There might be dozens of
| libraries in just the current project that we could be using
| more effectively if only we'd taken the time to read the
| source.
| Jtsummers wrote:
| We have this wonderful organ called a "brain". It appears to
| be the center of reasoning. We apply that to the situation
| and end up drawing a line where we say, "Beyond this point, I
| will not look. I may learn _what_ it does, but I will not
| dwell on _how_ it does it, neither in principle (conceptual
| model) nor in fact (actual code). Because if I were to
| investigate this, _and every other such system_ , I would
| find that it is turtles all the way down and I will never
| actually complete my objective. Unless I choose to freeze all
| these dependencies at this one moment in time and never
| permit them to change. If, in the course of my work, I find
| that this particular area is a source of trouble or a
| possible source of solution, then I will investigate further.
| On my own, with my team, directing a team member, or
| contacting the responsible party."
|
| It's a noble goal to make systems that are comprehensible,
| and one that should be striven for more often. It's also a
| noble goal to try to fully understand the stack of
| dependencies. But short of freezing time (this subsystem will
| never be permitted to change after I have studied it, except
| by my own hand), spending a lifetime on a more narrow field,
| being in the top 0.0001% of intellects, or working on
| relatively simple systems, it's infeasible in the general
| case. Once you try and do something even as "simple" as build
| out an avionics system for an aircraft, it will quickly
| explode into too many domains for any one person to truly
| comprehend every detail.
| WJW wrote:
| You are certainly much smarter (or more inexperienced in
| aircraft software design) than me if you think avionics
| systems are simple.
|
| But yes, I also have a brain and can reason with it. You
| don't have to restate the obvious as if it is some deep
| wisdom. But parent comment made the point that you should
| only look at library code when you have to and I countered
| that you cannot know when you have to without looking. Your
| counter-counter that you must think about what you have
| seen does not help with deciding when to start looking in
| the first place.
| Jtsummers wrote:
| I put it in quotes for a reason. It's been most of my
| career at this point.
|
| My point is that there is no hard and fast rule for when
| to stop reading or studying the stack of software and
| hardware. But at some point your own judgement enters
| into it because only you know your situation and
| objectives. Asking others when you should stop is inane,
| they cannot answer it for you and you have to use your
| own brain.
|
| If you have the intellect and time to spend reading the
| entire Linux and GCC source. Go for it. Most of us don't,
| so lines get drawn.
| TameAntelope wrote:
| Why do you think there's some algorithm you should run
| that will, in some determinate sense, tell you how to
| behave here?
|
| The comment you're replying to was poking fun at you for
| assuming there would be some objective set of criteria,
| when in reality there isn't anything of the sort.
|
| There are a _lot_ of heuristics I use when picking out a
| 3rd party library, but I 'm an engineer and these
| heuristics are practical, so I doubt you'll enjoy
| thinking in those terms, if your prior comments are any
| indication of what you're looking for here.
| WJW wrote:
| As a fellow engineer, I would indeed like actual reasons
| for why to pick one library over another. Even if that
| reason is as simple as "profiling shows we spend 99% of a
| request in this single library". Falling back to
| experience, heuristics and "just use your brain" can work
| in some circumstances (and with sufficiently experienced
| developers), but it is not how an engineering discipline
| matures.
| TameAntelope wrote:
| ...do you seriously not know how to evaluate 3rd party
| libraries? I can help you if you want, but honestly if
| this is a challenge for you, I'd be worried about whether
| or not this is the right career for you.
|
| Engineers get shit done, first and foremost. If that's
| not your primary concern, you're going to spend a lot of
| time frustrated by your bosses priorities and confused
| about why your peers value things you don't.
| WJW wrote:
| This goes back to the discussion we had on
| https://news.ycombinator.com/item?id=29998868, about
| whether sofware developers are really engineers or not.
| If you think "getting shit done" is the only or even the
| primary concern for an engineer, I think it is best if
| you stay in software (and please far away from any safety
| critical code).
| kortilla wrote:
| > and please far away from any safety critical code
|
| To a rough approximation nobody in the software
| engineering industry works on safety critical code. It
| comes with an onerous set of restrictions and criteria to
| do it well, including fully analyzing all of your
| dependencies for runtime behavior, etc.
|
| If you think applying that to the vast majority of
| software engineering makes any sense, you're going to
| fail hard. It's no different than applying the rigor that
| goes into engineering a fan blade on a GE90 to a toilet
| flush handle on a household toilet.
| TameAntelope wrote:
| I will not hire anyone to my software teams who thinks
| their job is to create art for its own sake. If you can't
| tie your work back to creating value, you have no place
| on my teams, and that includes wasting time rebuilding
| worse versions of libraries that already exist.
| WJW wrote:
| It's pretty sure that I wouldn't want to hire someone as
| argumentative as you either, but I don't see how we went
| from "Don't think about how a library works - until you
| have to" (which I still think is hilariously circular and
| not very useful) to accusing each other of rebuilding
| worse versions of libraries just because. In fact, a few
| comments up we touched on having _actual facts_ as the
| basis for decisions so how you interpreted that as "art
| for it's own sake" is a mystery to me.
|
| In any case, have a good evening good sir and all the
| best in your software engineering endeavors.
| TameAntelope wrote:
| Heh, if you think this is argumentative, you probably
| wouldn't fit on _any_ healthy software team that actually
| tries to work together.
| togaen wrote:
| Sound, sage advice that no one will listen to.
| Msw242 wrote:
| From my reading it is less about being "against" division of
| labor than it is about being against "premature" division of
| labor.
|
| Amazon's API Mandate[0] was arguably incredibly successful, and
| created the necessary conditions for AWS to emerge.
|
| [0]: https://nordicapis.com/the-bezos-api-mandate-amazons-
| manifes...
| imbnwa wrote:
| Yeah I lot of this thread is arguing against a strawman
| axiosgunnar wrote:
| The UX designer doesn't need to know how React works.
|
| The React developer doesn't need to know how TCP/IP networking
| works.
|
| The network expert doesn't need to know how a kernel works.
|
| The kernel developer doesn't need to know how a CPU works.
|
| The CPU engineer doesn't need to know how silicon metallurgy
| works.
|
| And whoever smites metals all day sure as hell does not need to
| know how to draw pretty icons.
|
| etc.
|
| But all put together we are eating the world.
|
| It's beautiful.
| mipsi wrote:
| OSI model of Human Resource Management?
| gtsop wrote:
| This isn't entirely true.
|
| Knowing how react works (or doesn't work) helps a designer
| design a valid experience that can actually be implemented by
| the developers. Ok maybe they don't need to know React, but
| surely they need to know what a browser can and can't do.
|
| A react developer (web developer) needs to know how tcp/ip
| networking works to avoid spending hours of debuging
| communication problems with XYZ service.
|
| And so on.
|
| Surely people can go about doing their jobs without necesserily
| knowing how the adjecent stacks work, but they would be way
| better professionals of they did. I would agree more with your
| argument if you restricted the distance of irrelevant tech-
| stack to let's say 3 levels. Eg, the web developer truly won't
| do any better if they knew more about kernels.
| ItsMonkk wrote:
| I like thinking of it like the Olympic rings. The blue
| ring(all the way to the left) doesn't need to know anything
| about the red ring(all the way to the right), but should
| absolutely know the yellow ring(second ring from the left),
| and possibly bits of the black ring(middle ring). The leakier
| the abstraction, the more you need to reach.
| TameAntelope wrote:
| But it's mostly true, and that's what matters.
|
| Nobody's trying to build a super rigid structure around the
| division of labor, they're just trying to create a general
| set of guidelines that will help answer questions quickly.
| tester756 wrote:
| >The kernel developer doesn't need to know how a CPU works.
|
| >The CPU engineer doesn't need to know how silicon metallurgy
| works.
|
| You're sure about this?
| ginko wrote:
| I'd argue that each of these should have a good understanding
| of the layer below (and maybe above) them.
| TameAntelope wrote:
| It is genuinely frustrating to work with someone who insists on
| building tools that others have built better and actively
| maintain.
|
| I have, in the recent past, told someone if they can't get used
| to using 3rd party libraries, they should seek employment
| elsewhere.
|
| I'm just tired of having the argument. Maybe some shops can
| afford to build everything in house, but nowhere I've ever worked
| can.
| hinkley wrote:
| I simply don't understand why a junior developer would want to
| work at a company that uses as much proprietary software as
| some shops do. What have they learned at the end of their
| tenure that they can apply somewhere else?
| politician wrote:
| If employers hire junior developers for reasons other than
| that they have some capacity to train them (such as cheap
| labor), then, agreed, they are doing them some disservice.
|
| But for folks that are not so junior, their insistence on
| rewriting libraries often is a consequence of a lack of
| confidence or misunderstanding about the purpose of the
| software project.
|
| If you're building a new database for use with your new
| language, yeah, it's worth building it from scratch. But for
| many older shops, simple was made easy long ago through
| selection of libraries that don't need to be rebuilt for
| pedagogical reasons.
| hinkley wrote:
| We had a guy who left (a mess), before he left he asked one
| of the office Nice Guys his opinion on open sourcing a
| framework he wrote. Even the nice guy said it was a bad
| reimplementation of something that already exists.
|
| The logical conclusion is that he was already looking to
| leave and wanted to take his homunculus with him, but my
| head cannon is that he realized his play time was over and
| people weren't going to kiss his butt anymore.
|
| There are distractions I find healthy and ones I find
| unhealthy and most weeks I'm trying to juggle the mix of
| both. Among the ones I don't like are people asking me
| questions about my code that they should be able to answer
| themselves. That means I'm more open to questions from
| junior devs, because they can't, but if people are stuck I
| want to make it easier for the next person. Or me in a year
| when I wander back into a project that was some definition
| of "done" before new requirements came in or my perspective
| on "good" has shifted (these two are not independent
| variables).
| politician wrote:
| Agreed. "Be kind to your future self because time is
| not."
|
| We ought to want to train people up, but those that have
| been trained yet lack discipline are problematic. I'm
| certain that's true across every endeavor.
| TameAntelope wrote:
| I agree, by learning how to use tools they'd have access to
| for the rest of their careers, they've learned how to be
| useful to the rest of the world.
|
| Turning your talent into something that makes people's lives
| better is immensely useful and satisfying.
| akkartik wrote:
| Yeah, I can totally relate.
|
| What I'm trying to point out in OP is a third way beyond "build
| all the things" and "reuse all the things others built" that
| doesn't get explored as often: build/reuse less. Do without.
| You're absolutely right that the tools we have are the best way
| to operate at scale. Do we always need to operate at scale? I
| think we are well-served by constantly questioning that,
| fractally on every decision all the way down.
| ginko wrote:
| So should all projects implement their own kernel and network
| stack?
| throwaway984393 wrote:
| And why aren't web developers writing compilers and linkers?
| HideousKojima wrote:
| I wish they had, instead of the nightmares they created like
| Webpack
| NAR8789 wrote:
| It's interesting to contrast this against the concepts in Team
| Topologies (https://teamtopologies.com/)
|
| - Kartik argues for better mutual understanding and additional
| communication
|
| - Team Topologies states you should match system boundaries to
| team boundaries, _specifically when cognitive and /or
| communications load have grown unmanageable_ due to team size
| and/or software complexity
|
| On the surface, Kartik is arguing against abstractions and Team
| Topologies is arguing for abstractions, but both stances feel
| valid to me. It feels like there's a spectrum of tradeoffs here.
| Is this essentially an analogue to the tradeoffs of monolith vs
| services?
|
| - For small teams, by all means keep abstractions low and
| communication high--monolithic systems and less division of labor
| is ideal and your team will gel better for it. Your team is
| probably not large enough to warrant either services or division
| of labor, so stick with the monolith.
|
| - As an organization scales, you end up with an n^2 communication
| edges problem, and at this point division of labor is necessary
| to limit the cognitive and communications loads.
| redredyupdy wrote:
| As someone who deals with APIs, I understand the point of
| abstraction.
|
| But when you start to deal with performance problems, a high
| level view of the internals can be invaluable for finding the
| problem.
|
| Something like a CRC diagram is invaluable
|
| http://www.agilemodeling.com/artifacts/crcModel.htm
|
| To clarify, this then allows you to look inside of the API (to
| bunch of things talking over more APIs, yay microservices).
| Then you can see which bit might be causing the issue with the
| bit of the larger black box you are talking to
| akkartik wrote:
| _" On the surface, Kartik is arguing against abstractions.."_
|
| Nope. To clarify:
| https://news.ycombinator.com/item?id=30019146#30038853
| ssivark wrote:
| I think there needs to be one more aspect accounted for, and
| that is the _domain_. The domain might naturally push you
| towards larger (or smaller) blobs than you would otherwise
| anticipate, and in such cases I don't think it makes sense to
| fight the domain -- you have to adapt the team and software
| boundaries (map) to the ground reality (terrain), and find
| people who can function effectively with an atypical map -- be
| it extra complexity or communication.
|
| I interpret Kartik's point as stating that it's
| counterproductive to split naturally integrated domains
| (terrain) for the convenience of map making.
| mprovost wrote:
| This sounds like embracing Conway's Law [0] instead of
| observing it as emergent behaviour.
|
| [0] https://en.wikipedia.org/wiki/Conway%27s_law
| beckingz wrote:
| All abstractions are ultimately leaky, but you still need to
| have them to build large things.
| dpark wrote:
| Abstractions are not always leaky. Often the "leaks" come
| from wanting to push past the bounds of the abstraction (for
| performance, additional capabilities, whatever). A good
| abstraction, if used as intended, needn't always leak.
|
| People also often define "leaky" as "can break", which is
| kind of unreasonable. The notion of a data stream across a
| network is an abstraction over many layers of complexity.
| People often say it's leaky because of packet loss and failed
| connections, but that's not a leaky abstraction unless you
| define the abstraction to not include a failure mode.
| [deleted]
| avgcorrection wrote:
| Are all absolutes leaky?
| fatbird wrote:
| Even in my own work, when I'm moving back and forth across the
| code, I'll set up interfaces and contracts where appropriate,
| because it's important to have stability somewhere in the
| evolving mess. One of the problems you can run into while free-
| range programming is chasing the consequences of a change back
| and forth, where a fix in one place requires a fix in another,
| requiring a fix in another...
|
| When I mentor coders one of the core lessons I try to get them to
| internalize is to code from known working state to known working
| state. Always be able to run something, either the code or a unit
| test, so that you know it works as expected. Add some code and
| get to the next known working state. This requires no undefined
| dependencies. If you're suddenly in a position where you're
| saying "I don't know why this doesn't work", you've bitten off
| more change than you can handle and need to back up and stabilize
| by decomposing the working state you're aiming for. Interfaces
| and contracts help immensely with that as long term stable
| scaffolding, as a skeleton for your codebase.
| nyanpasu64 wrote:
| Interfaces and contracts are good. Telling people to not care
| about the code that implements the interface and contract,
| making it difficult to jump into a Cargo crate's source code
| and add debug logging, making it hard to tinker with your
| dependencies, is bad.
| ajuc wrote:
| Abstraction is necessary, but I think OOP as practiced in many
| companies likes abstraction a bit too much.
|
| At the extreme people treat everything except the code they
| currently work on as black boxes and code like this is written:
| //this was written long ago in one class List<Book>
| getBooks(BookRepository bookRepository, String author) {
| var books = bookRepository.getBooksByAuthor(author);
| if (books.isEmpty()) return null; return books;
| } //this too but in a different task and in a
| different class Collection<Book>
| getBookAboutKeyword(String author, String keyword) {
| var books = getBooks(this.bookRepo, author); if (books
| == null) { return null; } //
| let's ignore the fact this should use filtering on db
| return books.stream().filter(Book b ->
| b.about(keyword)).collect(Collections::toList()); }
| //this is written now and there's a NPE List<Book>
| getBooksAndFooBarThem(List<Books> books, String author, String
| keyword) { var books = getBookAboutKeyword(books,
| author, keyword); //foobar them return
| books; }
|
| And of course the solution to NPE was to put ANOTHER null check
| in the getBooksAndFooBarThem function because anything up or down
| the callstack is black magic.
| lkrubner wrote:
| The best counter-argument was given by Alfred North Whitehead
| back in 1912:
|
| "By relieving the brain of all unnecessary work, a good notation
| sets it free to concentrate on more advanced problems, and in
| effect increases the mental power of the race.
|
| It is a profoundly erroneous truism, repeated by all copy-books
| and by eminent people when they are making speeches, that we
| should cultivate the habit of thinking of what we are doing. The
| precise opposite is the case. Civilization advances by extending
| the number of important operations which we can perform without
| thinking about them. Operations of thought are like cavalry
| charges in a battle -- they are strictly limited in number, they
| require fresh horses, and must only be made at decisive moments."
|
| https://en.wikiquote.org/wiki/Alfred_North_Whitehead
| akkartik wrote:
| OP was me annoyingly quoting myself, and here's my refutation
| annoyingly quoting OP:
|
| "Software libraries suck. Here's why, in a sentence: they
| promise to be abstractions, but they end up becoming services.
| An abstraction frees you from thinking about its internals
| every time you use it. A service allows you to never learn its
| internals."
|
| "Confusingly, I've been calling Iverson's notion of
| subordination of detail 'abstraction', and Iverson's notion of
| abstraction 'service' or 'division of labor'. Though lately I
| try to avoid the term "abstraction" entirely. That seems on the
| right track. Regardless of terminology, this is a critical
| distinction."
|
| As I hope these clarify, I have nothing against true
| abstractions as you describe them.
| ramesh31 wrote:
| >Operations of thought are like cavalry charges in a battle --
| they are strictly limited in number, they require fresh horses,
| and must only be made at decisive moments.
|
| I love this analogy. People who aren't accustomed to doing long
| form, extended, in-depth thought about a single problem don't
| seem to understand this. That your attention and mental effort
| is a finite resource, that can be frivolously wasted or put to
| good use. It requires real concerted effort, and it isn't easy.
| Thus, why most people never really think about anything.
| passion__desire wrote:
| You can't do much carpentry with your bare hands and you can't
| do much thinking with your bare brain. (i.e. using appropriate
| Thinking Tools and Intuition Pumps) As Daniel Dennett says, "we
| are productive now because we have downloaded more apps to our
| necktops (i.e. brains)"
|
| Intuition Pumps and Other Tools for Thinking :
| https://www.goodreads.com/en/book/show/18378002
| somethingAlex wrote:
| I feel like all of the downsides given in the quotes can be
| solved by making sure people get a variety of work. Define an
| interface and split the work into chunks. Just make sure people
| are being assigned to different kind of chunks should their skill
| allow it.
|
| This is really just the crux of managing people and bringing out
| their potential. I'm not gonna split a CRUD endpoint or class
| between two people but I've also had two people write the same
| exact function because it was so complex.
| dpark wrote:
| This screams "toy projects" to me. Separation of concerns is a
| hard-won insight. We call code without this "spaghetti" because
| it eventually becomes nearly impossible trace or to reason about.
| Even if you have all your engineers working all over the whole
| codebase, the codebase itself still needs separation of concerns
| to simplify reasoning about behavior. This is considered a good
| engineering practice because the alternative is that you need to
| hold the whole system in your head to understand any behaviors.
| I've seen codebases like this. It's not pretty. Use of
| abstractions to simplify cognitive overhead is not a weakness.
|
| The idea that projects should take source copies instead of
| library dependencies is just kind of nuts, at least for large
| libraries. Yes, the proliferation of tiny libraries is obnoxious
| and adds a ton of pointless overhead. But for large libraries,
| taking a source copy is absolutely crazy. Imagine taking a source
| clone of Chromium into your project because you need a rendering
| engine. Yes, you can probably cut a bunch of code out as a
| result, but you will also now need to deal with security concerns
| and incorporation of new features until the end of time. And this
| is now way more expensive and you can't just do a straightforward
| merge because you changed the code. 5 line package? Sure, just
| copy the code in (assuming licensing is in the clear). 500 dev-
| year codebase? Cloning that code is by far not a win.
| hinkley wrote:
| The thing I tell people when I'm critiquing their tests is that
| nobody gives a shit about your tests until they break, at which
| point they're interacting with your code when they're already
| in a bad mood.
|
| They will associate that feeling with you. So do you want to
| make their bad day worse by writing abstruse code? Or should
| they be simple?
|
| I know I'm "done" with a test when I know why it's red from
| what's in the console. I shouldn't even have to look in the
| source code. Until then, my tests are a work in progress.
|
| We could apply this philosophy a lot more widely.
| akkartik wrote:
| Totally agreed!
|
| Here's a prototype from a few years ago where I tried to make
| this easier: https://github.com/akkartik/mu1#readme (read
| just the first few paragraphs)
|
| I still think the full answer lies in this direction.
| akkartik wrote:
| _" Separation of concerns is a hard-won insight."_
|
| Absolutely. I'm arguing for separating _just_ concerns, without
| entangling them with considerations of _people_.
|
| It's certainly reasonable to consider my projects toy. I
| consider them research:
|
| * https://github.com/akkartik/mu
|
| * https://github.com/akkartik/teliva
|
| _" The idea that projects should take source copies instead of
| library dependencies is just kind of nuts..."_
|
| The idea that projects should take copies seems about symmetric
| to me with taking pointers. Call by value vs call by reference.
| We just haven't had 50 years of tooling to support copies.
| Where would we be by now if we had devoted equal resources to
| both branches?
|
| _"...at least for large libraries. "_
|
| How are these large libraries going for ya? Log4j wasn't
| exactly a shining example of the human race at its best. We're
| trying to run before we can walk.
| hinkley wrote:
| I'm arguing for separating just concerns, without entangling
| them with considerations of people.
|
| Conway's Law says you're going to be disappointed a lot.
| These impedance mismatches lead to throughput problems and
| worse.
| akkartik wrote:
| Can you elaborate on this comment? Based on your comments
| elsewhere I think we're likely to be largely in agreement.
|
| I tend to have a love/hate relationship with Conway's Law,
| spending half my time thinking about how to design stuff
| assuming it, and half my time trying to design stuff
| (unsuccessfully) to make it go away.
| hinkley wrote:
| A lot of pain comes from taking a "concern" that doesn't
| align with the org chart, making simple tasks take weeks
| to accomplish because I need to get my boss to tell your
| boss to do it anyway or we don't make payroll. People
| tend to get precious about things they own. See also the
| legal department or the financial department thinking
| they are the company, as if they had any revenue without
| all of those "cost centers" or legal liabilities.
| akkartik wrote:
| Yeah, I agree with this. There's a larger context here of
| capitalism evolving firms because they're more efficient,
| even though they dilute the price-communication value of
| markets to some extent. As a consequence your boss and my
| boss are talking about something that might affect
| millions of people. That results in an effect akin to
| regulatory capture. So one way to view my case against
| division of labor is as a case for pushing intelligence
| to the edges, as close as possible to the owner of the
| computer running the software.
|
| On this larger level my suggestions are not actually that
| radical. Capitalism has had a long trend to push
| intelligence down to the edges. Kings used to use
| hierarchy to push decisions down. Blitzkrieg was largely
| about creating an organization that pushes _decision-
| making_ down. And we 've been getting better at it. I
| think we (i.e. society) can be even better at it by
| pushing decision-making out of the org and towards the
| people using it.
| dpark wrote:
| > _Absolutely. I 'm arguing for separating just concerns,
| without entangling them with considerations of people._
|
| The separation of people is largely an artifact of scale. At
| some point it doesn't make sense to have everyone touching
| the whole code base. It's untenable to have, e.g., everyone
| who works on Windows be an expert in every part of the code.
| It's also generally unnecessary because in a large code base
| features are often nearly completely contained to one area of
| code.
|
| Some teams are really dysfunctional and won't let people ever
| do work outside "their" area. This is absolutely an
| antipattern.
|
| > _The idea that projects should take copies seems about
| symmetric to me with taking pointers. Call by value vs call
| by reference. We just haven 't had 50 years of tooling to
| support copies. Where would we be by now if we had devoted
| equal resources to both branches?_
|
| We have probably invested dev-millennia into managing copies.
| This is exactly what source control does. This is not a new
| area of investment. Merging is a giant pain in the ass and
| very possibly always will be. Accepting merge pain better
| come with some huge benefits.
|
| > _How are these large libraries going for ya? Log4j wasn 't
| exactly a shining example of the human race at its best.
| We're trying to run before we can walk._
|
| Can you clarify what you see as the alternative? Implementing
| everything from scratch seems absurd and so costly that
| there's no point in considering this an actual option. Plus
| there's no particular reason to believe that you wouldn't
| care your own security bugs. Taking a source drop wouldn't
| protect you if you happened to take one with the bug. And
| you're forever bound to that version unless you want to pay a
| very high cost to reintegrate a new one. What viable option
| do you see that I don't? (Picking on log4j right now is easy
| but are you seriously implementing your own SSH, for
| example?)
|
| For the record, these large libraries are going great. I've
| built numerous products that simply would not have ever been
| created without, say, SQL. The cost would simply have been
| too high. I've built others with libraries that the project
| outgrew, but it was still a big win to start with the
| library.
| akkartik wrote:
| I question the need for scale in 90% of the places where
| the tech industry has cargo-culted it. Clearly I'm still
| failing to articulate this. Perhaps
| https://news.ycombinator.com/item?id=30019146#30040616 will
| help triangulate on what I mean.
|
| > Can you clarify what you see as the alternative?
| Implementing everything from scratch seems absurd and so
| costly that there's no point in considering this an actual
| option.
|
| Not using, reimplementing and copying are the closest thing
| to solutions I have right now. You're right that they're
| not applicable to most people in their current context. I
| have a day job in tech and have to deal with some cognitive
| dissonance every day between my day job and my open source
| research. The one thing I have found valuable to take to my
| scale-obsessed tech job is to constantly be suspicious of
| dependencies and constantly ask if the operational burdens
| justify some new feature. Just switching mindset that way
| from software as asset to software as liability has, I'd
| like to believe, helped my org's decision-making.
|
| > We have probably invested dev-millennia into managing
| copies. This is exactly what source control does. This is
| not a new area of investment. Merging is a giant pain in
| the ass and very possibly always will be. Accepting merge
| pain better come with some huge benefits.
|
| Not all copying is the same. We've learned to copy the
| letter 'e' so well in our writings that we don't even think
| about it. In this context, even if I made a tool to make
| copying easier and merges more reliable, that would just
| cause people to take on more dependencies which defeats the
| whole point of understanding dependencies. So tooling would
| be counter-productive in that direction. The direction I
| want to focus on is: how can we help people understand the
| software they've copied into their applications? _That_ is
| the place where I want tooling to focus. Copying is just an
| implementation detail, a first, imperfect, heuristic coping
| mechanism for going from the world we have today to the
| world I want to move to that has 1000x more forks and 1000x
| more eyeballs looking at source code. You can see some
| (very toy) efforts in this direction at
| https://github.com/akkartik/teliva
|
| > It's untenable to have, e.g., everyone who works on
| Windows be an expert in every part of the code.
|
| It's frustrating to say one thing in response to counter-
| argument A and have someone then bring up counter-argument
| B because I didn't talk about it right there in the
| response to counter-argument A. I think this is what Plato
| was talking about when he ranted about the problems with
| the newfangled technology of writing: https://newlearningon
| line.com/literacies/chapter-1/socrates-.... I'm not saying
| everyone needs to be an expert in everything. I'm saying
| software should reduce the pressure on people to be experts
| so that we can late-bind experts to domains. Not every
| software sub-system should need expertise at the scale at
| which it is used in every possible context. My Linux laptop
| doesn't need to be optimized to the hilt the way Google's
| server farms do. Using the same scheduling algo or whatever
| in my laptop imposes real costs on my ability to understand
| my computer, without giving me the benefits Google gets
| from the algo.
| tomnipotent wrote:
| > It's untenable to have
|
| Absolutely this. There's a wide ranging of programming
| activities that are domain-agnostic, but most of what
| programmers do requires deeper knowledge in a specific
| domain to be useful.
| akkartik wrote:
| _> For the record, these large libraries are going great.
| I've built numerous products that simply would not have
| ever been created without, say, SQL. The cost would simply
| have been too high. I've built others with libraries that
| the project outgrew, but it was still a big win to start
| with the library._
|
| We definitely have great libraries out there. But they're
| the exception rather than the rule. I consider Redis and
| Sqlite in particular to be quite exemplary.
|
| SQL I'm more ambivalent about. Or rather, SQL the abstract
| ideal is great, but I'm more ambivalent about
| implementations like MySQL or PostgreSQL. As I've hopefully
| made clear already, I consider just raw speedup for an
| initial implementation of an app as a poor and incomplete
| metric for the quality of a library. At best it's a good
| quality for something you can use in prototypes, not the
| final version you build after you throw the prototype away.
| (Because we all agree you should throw the prototype away,
| right? :) For non-prototypes you also have to take into
| account the total cost of ownership. I think the ever
| expanding wavefront of "specialization" in roles like DBAs
| is a sign that MySQL and PostgreSQL are a bad trade for
| society. People being deployed to fill in for all the
| numerous gotchas arising from a crappy interface. MySQL and
| PostgreSQL make things more efficient in the short term,
| but they make us less resilient in the long term.
| dpark wrote:
| > _I think the ever expanding wavefront of
| "specialization" in roles like DBAs is a sign that MySQL
| and PostgreSQL are a bad trade for society._
|
| That's a sign of a maturing field. Does the proliferation
| of medical specialties indicate a problem? I would argue
| that it simply arises from the amount of specialized
| knowledge that one can possess and utilize efficiently.
|
| The existence of SQL implementations allows for faster
| _and_ more maintainable implementations. The set of
| engineers who could do what SQL allows without it is
| quite small. If SQL disappeared tomorrow, a thousand
| equivalent implementations would appear tomorrow because
| it delivers real value. Hand rolling queries in a general
| purpose programming language is a very poorly trade off.
| I've seen this sort of stuff before. It's always a bug
| farm and every other engineer asks "why the hell didn't
| you use SQL"?
|
| Give it enough time and you'll see "Redis expert" show up
| as a specialty.
| akkartik wrote:
| _> Does the proliferation of medical specialties indicate
| a problem?_
|
| Ivan Illich certainly thought so:
| http://akkartik.name/illich.pdf. And he said a lot of
| useful stuff in my opinion.
|
| But I don't know enough to have an opinion of medicine. I
| find it useful in my experience. I wasn't saying the
| connection between my observation and conclusion is
| ironclad in all situations. But I do hold the heuristic
| opinion in the one case I mentioned.
|
| _> The set of engineers who could do what SQL allows
| without it is quite small. If SQL disappeared tomorrow, a
| thousand equivalent implementations would appear tomorrow
| because it delivers real value._
|
| How much of this argument is about idealized SQL and how
| much about MySQL or its equivalents? I was careful to
| draw a distinction. SQL the language/mindset is
| incredibly valuable. EF Codd thoroughly deserves his
| Turing Award.
|
| You don't need to persuade me that MySQL lets us do
| things faster. What would really persuade me is an
| argument that it helps us do the right things. I think it
| makes it easy to scale past our level of competence.
| akkartik wrote:
| _> Give it enough time and you'll see "Redis expert" show
| up as a specialty._
|
| No opinion on what happens to Redis after Antirez left it
| (http://antirez.com/news/133). But I have high hopes that
| the versions he created will be fairly timeless and
| usable even a decade from now.
| Jach wrote:
| Lisp machines showed the way decades ago, computing took a
| different path. Here's the system's UDP receive-ip-packet
| method:
| https://twitter.com/RainerJoswig/status/1215728406823886854
|
| The copies vs dependencies thing doesn't seem like a useful
| framework to me -- in order to do anything with software that
| has dependencies, such dependencies need to be copied in some
| form that can be integrated into the final build. So the
| copies are there regardless. Different ecosystems differ on
| how easy it is to work with them. If all you get is a .so,
| then even if it's open source you're going to have to find
| and build your own to make changes. In Java you'll get jars,
| which may or may not contain java files and class files -- in
| the past I've unzipped jars, recompiled a single java file to
| replace a class file, rezipped it back up, and used that jar
| instead of the upstream jar. Not too bad. Some log4j 'fixes'
| involved stripping out some class files from jars, again not
| too bad, just don't accidentally clobber your changes later.
| Some JS projects will have additional instructions (or a bash
| script) that after npm downloads everything into the local
| node_modules, you need to go modify a particular JS file in
| there. It's easy for the changes to get clobbered later,
| but..
|
| Lastly in Lisp land, we have quicklisp, so you point to a
| library dependency like normal and it downloads it locally
| elsewhere (so it only need be downloaded once and not for
| each project). When I'm writing a program using library X,
| and I'm curious how some function works, I can just jump-to-
| source and it'll take me to that local copy. If I want to
| change it, I can, I just edit it, and because Lisp has
| "compile", "compile-file", and "load" all as standard runtime
| functions, not separate programs, I can compile and load my
| changes into the running program and immediately test them.
| Maybe such changes are upstreamable, maybe not. You can
| maintain the changes in the original files, maybe going so
| far as a full source copy locally instead of using the ones
| quicklisp brought down, or just make your own local 'patch'
| file of redefined functions or extended classes that you load
| after loading the original and the patch simply recompiles
| and replaces things you've changed. It's also rather fun to
| do this to inspect and learn about the Lisp implementation
| you're using, what it's doing or how it's implementing
| things, changing stuff to see what happens/fix an edge case
| bug, whatever.
|
| Part of it I think is supported by a notion _Thinking Forth_
| calls a 'lexicon'. See its early section on 'Component
| Programming' but in short 'lexicon' refers to words (symbols)
| of a component that are used outside of a component. "A
| component may also contain definitions written solely to
| support the externally visible lexicon. We'll call the
| supporting definitions 'internal' words." Of course, without
| special effort, in Forth as in Lisp even those internal words
| are not really inaccessibly internal. In Lisp it's the
| difference of referring to an exported symbol by
| namespace:symbol and an un-exported one namespace::other-
| symbol. The other supporting notion here is a that of
| globally available "components", which are just a product of
| decomposition, not some formal thing like separate processes
| or libraries or what have you. "It's important to understand
| that a lexicon can be used by any and all of the components
| at higher levels. Each successive component does _not_ bury
| its supporting components, as is often the case with layered
| approaches to design. Instead, each lexicon is free to use
| all of the commands beneath it. ... An important result of
| this approach is that the entire application employs a single
| syntax, which makes it easy to learn and maintain. This is
| why I use the term 'lexicon' and not 'language.' Languages
| have unique syntaxes."
|
| I'm not sure I agree about the 'easy to learn and maintain'
| bit but at least not making things painfully inaccessible can
| help, especially with re-use. There used to be OS concepts
| where theoretically if I wanted to play a music file, and
| knew I had a popular music player application installed, I
| could just reach in and call a function in that application,
| rather than writing my own or finding some "do one thing..."
| library. All programs on the OS were simultaneously programs
| and DLLs able to be used by other programs.
|
| I don't have many arguments in this space, it's just fun to
| think about, but good luck on your research. You might enjoy
| this quote:
|
| "When I see a top-down description of a system or language
| that has infinite libraries described by layers and layers,
| all I just see is a morass. I can't get a feel for it. I
| can't understand how the pieces fit; I can't understand
| something presented to me that's very complex." --Ken
| Thompson
| akkartik wrote:
| Thank you! My ideas are definitely greatly influenced by my
| time in the Lisp world. Here's the first time I articulated
| them, back in 2011 (so before the first entry in OP):
| http://arclanguage.org/item?id=13263.
|
| > in Lisp land, we have quicklisp, so you point to a
| library dependency like normal and it downloads it locally
| elsewhere (so it only need be downloaded once and not for
| each project). When I'm writing a program using library X,
| and I'm curious how some function works, I can just jump-
| to-source and it'll take me to that local copy. If I want
| to change it, I can, I just edit it, and because Lisp has
| "compile", "compile-file", and "load" all as standard
| runtime functions, not separate programs, I can compile and
| load my changes into the running program and immediately
| test them. Maybe such changes are upstreamable, maybe not.
| You can maintain the changes in the original files, maybe
| going so far as a full source copy locally instead of using
| the ones quicklisp brought down, or just make your own
| local 'patch' file of redefined functions or extended
| classes that you load after loading the original and the
| patch simply recompiles and replaces things you've changed.
|
| Is there an easy/idiomatic way to save your edits, or do
| they get blown away the next time you download the library?
| All my comments are in the context of not having tooling
| for this sort of thing. I wouldn't be surprised to learn
| the Gods at Common Lisp figured all this out years ago. (In
| which case, why is the whole asdf fracas happening?!)
| Jach wrote:
| Heh, that thread brings to mind a bunch of things... One
| advantage of CL that helps with a 'borrowing' aspect that
| give people the jeebies is that the unit of compilation
| is much smaller than the file, so you can also borrow
| much less. Another is that methods are decoupled from
| classes so there's a lot of room for extensibility. (The
| LLGPL interestingly provides an incentive for the open-
| closed principle, i.e. you extend objects you're fine,
| but if you modify the library's then you are subject to
| the LGPL.) If you haven't read Gabriel's _Patterns of
| Software_ book (free pdf on his site), I think you 'd
| enjoy it.
|
| Your edits won't get blown away, at least by default,
| since quicklisp doesn't redownload a system that it knows
| about or check for 'corruption'. The way quicklisp does
| its own versioning also means if you update quicklisp's
| distributions lists and a newer version of the library
| has come out, it'll download that one into its own new
| folder and leave the old one alone. There's a cleanup
| function to clear out old things but I don't know of a
| case where that gets called hidden from you under the
| hood.
|
| Maybe there's some magic and interesting stuff related to
| this for emacs or mezanno OS or in the annals of ancient
| lisp machines but I'm a vim heretic ;) But in any case if
| you want to save stuff, you can just save it in a new or
| existing buffer... So options are basically as I
| described. To give a specific example, I have a script
| that fetches and parses some HTML to archive HN comments
| and I wanted to remove the HTML bits so I'd just have
| text, making it more markdown-y. There are lots of ways
| to do that and I'm pretty sure I chose one of the worst
| ones, but whatever. I was already using the Plump
| library, and after not achieving full success with its
| official extension mechanisms, one method I came across
| and stepped through was https://github.com/Shinmera/plump
| /blob/master/dom.lisp#L428 and I learned I could hijack
| it for my purposes. I started by editing and redefining
| it in place until I got what I wanted, but instead of
| saving my changes over the original, I simply copied it
| over to my script file, modifying slightly to account for
| namespaces e.g. it's "(defmethod plump-dom:text ((node
| plump:nesting-node))", thus redefining and overwriting
| the library implementation when my script is loaded and
| run.
|
| Some possible problems with this approach in general
| include later trying to integrate that script with other
| code that needs the default behavior (though CL support
| for :before/:after/:around auxiliary methods can help
| here, e.g. if I can't just subclass I could insert a seam
| with an :around method branching between my custom
| implementation over the library's without having to
| overwrite the library's; and in the long-term the SICL
| implementation will show the way to first-class
| environments that can allow multiple versions of stuff to
| co-exist nicely). Or the library could update and change
| the protocol, breaking my hack when I update to it. Or in
| other situations there may be more complex changes, like
| if you modify a macro but want the new modifications to
| apply to code using the old def, you need to redefine
| that old code, or if you redefine a class and want/need
| to specially migrate pre-existing instances you need to
| write an "update-instance-for-redefined-class"
| specializer, or if the changes just span across a lot of
| areas it may be infeasible to cherry-pick them into a
| 'patch' file/section of your script, so you're faced with
| choices on how much you want to fork the files of the lib
| and copy them into your own project to load. But anyway,
| all those possible problems are on me.
|
| The asdf noise isn't that big of a deal and I think is
| only a little related here technically since it's a
| rather special library situation. It's more 'interesting'
| socially and as a show of possible conflict from having a
| core piece of infrastructure provided by the
| implementation, but not owned/maintained by the
| implementation. An analogous situation would arise if
| gcc, clang, and visual studio all agreed to use and ship
| versions of musl for their libc with any other libcs
| being obsolete and long forgotten. A less analogous
| situation is the existing one that Linux distributions do
| -- sometimes they just distribute, sometimes they
| distribute-and-modify, whether they are the first place
| to go to report issues or whether they punt it off to
| upstream depends case-by-case.
| alexwennerberg wrote:
| > But for large libraries, taking a source copy is absolutely
| crazy.
|
| The solution is, when possible, to use (and write) smaller
| libraries. Many libraries are far larger and more complex than
| they need to be.
| dpark wrote:
| Depends on what it is you need. If you really have a small
| need, then taking a giant dependency is very possibly a bad
| idea. But I have seen the opposite pretty often. "I don't
| need that library. I just need this one thing. Okay, this
| other thing, and this. And this and this and this." And by
| the time they're done they've reimplemented 75% of the
| library in a far worse way.
|
| The most egregious example I ever saw was reimplementation of
| xml parsing with C++ substrings. They "didn't need" a real
| xml parser. In the end, they found endless bugs with their
| shitty parsing logic and it turns it it was responsible for
| something like 50% of the (very large scale) system resource
| use.
| ramesh31 wrote:
| >The idea that projects should take source copies instead of
| library dependencies is just kind of nuts, at least for large
| libraries.
|
| There's a medium to be found between static linking and dynamic
| dependencies which I think is one of the most crucial tradeoffs
| to understand in software engineering. NPM in my opinion goes a
| bit too far on the modularization of things. Especially in
| large projects where you can have multiple private repo URLs
| and different auth tokens needed for each, it can become a
| nightmare to orchestrate for CI/CD. Something that takes an NPM
| approach, but with a more fleshed out stdlib to eliminate
| things like left-pad is probably the way to go.
| datavirtue wrote:
| Taking source copies is thought to be especially FOR large
| libraries. You take what you need, leaving the "large"
| behind...if you can. You probably can. Though, libraries are
| developed with the assumption that they are to be consumed as
| one massive binary. Making it more difficult to run off with
| just the part(s) you need.
| mjburgess wrote:
| There is something wrong with division of labour in creative
| projects... the story of the service industry seems to be how
| this is relearnt over-and-over. A "silo" is incomprehsible in a
| factory: the pin-machinist isnt silo'd.
|
| Knowledge, skills, vision, creative engagement... these are
| silo'd. And this follows from misconceiving creative work as a
| form of goods-producing labour.
| akkartik wrote:
| Yes! I'd like to reiterate my link to
| https://creators.vice.com/en_uk/article/animator-hand-draws-...
| in OP.
| dpark wrote:
| > _A "silo" is incomprehsible in a factory_
|
| Efficient factories are built on silos. You do your job in your
| area and the next person does theirs. You might collaborate,
| but you absolutely have a clear division of labor.
|
| If you don't have a division of labor and silos, you don't have
| a factory. You have artisans.
| hinkley wrote:
| Because a lot of the profit comes in vertical integration.
|
| I mean that's why you built the factory right? To take
| materials in one end and put something that is greater than
| the sum if its parts out the other?
| akkartik wrote:
| Yes. And that's ok! The world ran on artisans for hundreds of
| years.
|
| It's even better than that right now, because we already have
| factories: the programs. Do we need factories to build the
| factories? I don't think we're there yet, not to mention we
| don't know how to build these meta-factories right.
|
| Quoting myself from OP:
|
| "Libraries with non-trivial functionality take a long time to
| get right, and in the meantime they are produced more like
| guilds of craftsmen than factories. (Even if the products
| themselves permit factory-like operation at scale.) In that
| initial bake-in period we are ill-served by conventional
| metaphors of software components, building blocks, etc. We
| should be dealing more in vertically-integrated self-
| contained systems rather than plug-and-play libraries. More
| OpenBSD, less `gem install`."
|
| Efficiency is great, but it's not the only thing to optimize
| for.
| aspenmayer wrote:
| > Knowledge, skills, vision, creative engagement... these are
| silo'd. And this follows from misconceiving creative work as a
| form of goods-producing labour.
|
| It's also related to the division of decision-making, usually
| between workers, with little or no executive authority, and
| management, with some to all of the executive authority.
| Creative roles are typically closer to management, and their
| knowledge bases are siloed, as is the business's operational
| experience and expertise.
|
| Everyone wants to be close to power, but how many actually
| wield power effectively? What types of hierarchy are amenable
| to diffuse power structures? A lot of business are run like
| petty fiefdoms, and may not benefit from a minor change, and
| yet may not have the stomach for radical reform either. Worker
| owned co-ops and DAOs give me hope for new ways of working.
| codeflo wrote:
| I think Hyrum's Law should be best understood as a warning for
| users rather than as handcuffs for implementors: don't rely on
| non-obvious implementation details.
|
| I've seen software break due to a race condition when an upgrade
| made something faster. Hyrum's Law tells us the performance was a
| hidden part of the interface. Is the solution to never optimize
| anything? Instead, I claim the client code was broken all along.
| fragmede wrote:
| Where client code is something someone outside the organization
| implemented and is using, that makes a certain amount of sense.
| But when the client is another internal team, and the SaaS
| product the company sells broke, the customer doesn't really
| care that the database team and the API team don't get along -
| the product is down and while it's down, no sales are
| happening. The point, thus, of looking into other teams is to
| get a sense for the challenges they face and to have a less
| adversarial working relationship with them.
| kibwen wrote:
| In such a scenario I'd say that the distinction between the
| two teams is essentially imaginary. If management is forcing
| the downstream team to use the new API even if it's broken,
| and then management is forcing the upstream team not to break
| the API for the benefit of the downstream team, then the API
| is an illusion.
___________________________________________________________________
(page generated 2022-01-22 23:00 UTC)