[HN Gopher] The cloudy layers of modern-day programming
___________________________________________________________________
The cloudy layers of modern-day programming
Author : antirez
Score : 108 points
Date : 2022-12-06 10:36 UTC (12 hours ago)
(HTM) web link (vickiboykis.com)
(TXT) w3m dump (vickiboykis.com)
| gilbetron wrote:
| I've been comparing much of modern-day programming to assembling
| a kit from a knockoff Lego brand. It seems like it's easy, but
| then there's a lot of cursing and annoyances as the pieces don't
| quite fit together.
| plugin-baby wrote:
| Knockoff Lego quality is getting very very good these days.
| mvaliente2001 wrote:
| Although I sympathise with the sentiment, that's just half of the
| story. Today, I can deploy a high available cluster in minutes
| using any cloud technology + kubernetes and my favourite web
| framework. True, I need to learn half dozen of frameworks, and
| _it feels_ like I'm not working in the core problem, but had I
| tried to implement an equivalent system without those tools would
| have taken me months, and I would end with an ad-hoc half tested
| non-reusable mess.
| jjav wrote:
| > Today, I can deploy a high available cluster in minutes using
| any cloud technology + kubernetes and my favourite web
| framework.
|
| I think the point is (at least for me it is), when you do that,
| have you done anything new that few or nobody else has done
| before?
|
| No, it's just a well worn path which thousands of organizations
| have done already.
| xxpor wrote:
| Do architects reinvent the I-beam every time they design a new
| building? No, of course not.
|
| The reason society works is because you can _reuse abstractions
| that other people have already invented_. It allows you to scale.
| Without economies of scale, you end up with Baumol 's cost
| disease, which is extremely obvious in the US in industries like
| child and elderly care.
|
| We don't _want_ most software devs to be doing anything because
| gluing stuff together. If they weren 't, we really screwed up
| somewhere.
| CyberDildonics wrote:
| I beams aren't abstractions, they are a well designed standard
| that can consistently meet certain expectations.
| jacoblambda wrote:
| I beams are both an individual abstraction and part of the
| greater abstraction.
|
| Your catalog of beams, bolts, brackets, weld patterns, rebar,
| piles, and concrete pour standards are all abstractions over
| extremely difficult subfields of materials engineering and
| structural engineering.
|
| They exist so that your engineer can focus on building the
| structure using parts and resources with known, standardised
| behavior under the conditions the building will be put in.
|
| Of course you'll see engineers break away from these
| abstractions when they need to for a given structure but
| those abstractions do exist and are commonly used so that a
| given structural engineer doesn't also need a PhD in
| materials engineering and countless other specialized fields.
| dijit wrote:
| I like this allegory because it puts the complexity into a
| visual mindset and brings to light an interesting question
| about the _nature_ of our abstractions.. For example:
|
| Are our abstractions I-beams or Pre-Fabs[0]?
|
| We all know that the rise of pre-fabs is, at its heart, the
| story of cheap developments all lazily (and hastily) thrown
| together in arrangements that are of low quality, mid-to-low
| beauty and do not last for very long.
|
| Skyscrapers of the early 20th century stand today (with
| I-Beams) and are considered by many to be beautiful,
| maintainable etc;
|
| People want pre-fabs, since they're cheap, the economy will
| always be in the pre-fab.
|
| But pre-fabs have a limited shelf-life, and reconstruction is
| more expensive than spending a little extra up front cost.
|
| [0]: This is the sort of pre-fab I am talking about:
| https://en.wikipedia.org/wiki/Prefabricated_building#/media/...
| tester756 wrote:
| If you feel like "modern programming is boring configuration
| mess"
|
| then ask yourself - what have you done in order to have
| interesting job, projects, challenges, etc?
|
| I mean, if you decided at some point that $big_salary for
| throwing JSONs via REST from CRUD app
|
| is what you want to do until you pay your loans (e.g decade),
| then it's fine, but don't be shocked that you aren't doing
| bleeding edge R&D at some fancy place.
|
| Like what prevents you from putting effort for year? two? and
| switching from X to Y?
| mythhouse wrote:
| > Like what prevents you from putting effort for year? two? and
| switching from X to Y?
|
| I don't think it only takes 1 yr to switch to something
| interesting. Any R&D lab wants you have a phd with published
| papers.
| tester756 wrote:
| The first step you can do is: actually start working with
| this on daily basis
|
| Maybe over years you'll gain an expertise to work in
| research, idk.
| nesarkvechnep wrote:
| Every project I worked on was JS/Node hell. Then I decided to
| switch to Elixir/Erlang and now I'm much happier. Of course,
| there are uninteresting or badly engineered projects but
| overall it's much better.
| xcambar wrote:
| There's a true joy in working with some languages. It's like
| using a great pen or driving a well-made car. You get focus
| and flow.
| CyberDildonics wrote:
| _If you feel like "modern programming is boring configuration
| mess"
|
| then ask yourself - what have you done in order to have
| interesting job, projects, challenges, etc?_
|
| These two things don't connect to each other at all
| tester756 wrote:
| How so?
| nijave wrote:
| This seems to be touching more on the complexity of
| productionizing a system than cloud. I think you still run into a
| lot of these same problems getting your code to run somewhere
| else--especially when the management and operation of the system
| is split between many people/teams.
| aaroninsf wrote:
| I don't really understand the problem, or maybe it's more, I
| don't think this problem exists as an existential one for more
| than the set of people for whom it's an existential probem.
|
| Programming and software engineering and their subdomains and
| superdomains are no longer a priesthood or an academic exercise
| or playground for Levy-style oldskool hackers, or rather, no
| longer remotely those things exclusively.
|
| Most of what most people who fit in the supercategory do is work.
| The specifics of the work vary. If they aren't the ones that give
| you joy, change categories is a good and readily available
| solution.
|
| If you want to hack on beautiful code, there are 1000x more ways
| to do so today spanning the purely aesthetic defined as many ways
| as you like (live coding, avocational language design, exercises
| in new tricks for old tools like the demoscene and emulator
| worlds), to the aggressively useful (small tools made by small
| teams of solo devs upon which empires balance).
|
| You don't need to be a mid-level IC at MassiveCorp unless you
| can't give up access to the mana-tap of massive scaling. In which
| case maybe that IS your thing and the complaint is flat.
|
| The joy hasn't left. Maybe it's just harder for the author to
| find? Or the problem is simply that no matter what you do for
| work, it's work, "chase your bliss" being of course a lie,
| because work IS work, and dissatisfaction like water will always
| find its level.
| adra wrote:
| It sounds about right. At the end of the day, most developers
| work to produce features. Features and business domains are far
| less likely to be exciting than technical problems, but the
| business domain is way more important to adding value. Both
| need to be kept in check, but I'm sorry to say, technical
| achievement will not put food on the table or revenue on the
| balance sheet. At the end of the day, you almost always need to
| actually sell what you're doing (in one form or another).
| dongobread wrote:
| It's very unclear to me why the writer ran everything through GCP
| and Colab. Having everything on a single environment with
| near-100% uptime is certainly still possible by renting a VPS,
| which is usually a much cheaper alternative anyways.
|
| There's also many other alternatives that could be used if memory
| efficiency is the goal, e.g. polars/data.table instead of pandas,
| an OLAP database instead of bigquery etc. While I agree that
| pasting together configs is tedious and annoying, a lot of this
| type of work is due to developers themselves trying to replicate
| new trends at $bigco$ instead of optimizing for their own needs.
| rockemsockem wrote:
| I think this problem is almost entirely self-inflicted (by either
| a person or an organization, sorry to people in organizations
| that do this and who recognize the insanity and are powerless to
| stop it).
|
| BigQuery and Colab are fantastic tools largely because they allow
| less technical folks to try things out with very little friction
| and BigQuery scales to ridiculous amounts of data making formerly
| impossible queries possible. The example fits neither of these as
| the author is quite apparently very technical and the data is
| minuscule.
|
| The author could have applied some economics thinking to this and
| correctly concluded that they could have run this exploration on
| a laptop using a regular old file and Jupyter (or just normal
| Python if they want to avoid the IPython infrastructure). Post-
| exploration they can run their application on a properly specced
| VM. IDK if BERT requires a GPU these days or if it can run well
| enough on a CPU, but either way, these resources are easily
| available on GCP without all the vendor-ops the author talks
| about.
|
| To cap things off, the author says at the end that "in no
| previous universe would I be able to try out Stable Diffusion".
| Stable Diffusion requires 6 GB of GPU RAM and ~10 GB of storage.
| That was modest 5 years ago for commodity hardware.
|
| I am not a cloud-decrier by any means, I love the cloud, I use it
| every day, but most of what I use is basic storage (buckets) and
| VMs. With just a little bit of abstraction it's easy to make
| something that runs in the cloud run locally just as easily. That
| also makes it easier to switch clouds if you find a better deal
| somewhere else.
|
| TL;DR: use the right tools for the job
| mattgreenrocks wrote:
| One thing I had to learn to value my sanity here was to simply
| not argue about certain topics, such as software architecture. It
| was like there was this wall of people who emerge from the
| shadows anytime I argue for better architecture, because it
| wasn't directly responsible for Important Enterprise Business
| Features.
|
| Now I realize there's probably more than one flavor of engineer:
|
| 1. software eng: ships features, prefers to use ready-made
| abstractions
|
| 2. infrastructure-ish eng: ships things that enable shipping of
| features, prefers to invent abstractions
|
| The things they value are very different! I suspect HN tends more
| toward SW engineers (because statistics) with some stuff for
| infrastructure engineers.
| agumonkey wrote:
| It's not far from the top-down vs bottom-up too. Having a wide
| enough infra helps adapting at the upper layer faster and
| cleaner.
| mattgreenrocks wrote:
| Very true. So few people talk about this topic is saddens me.
| I have had success switching mindsets when I get stuck on
| one.
| nijave wrote:
| I think in any moderately large business you'll have both.
|
| 1. Is usually implementing business logic for the product 2. Is
| generally working on things #1 uses to work productively
|
| I'm not sure about the composition on HN, but I see a lot of #2
| in "DevOps"/SRE roles. Usually these roles are skipped for
| small startups but at some point they get big enough they need
| dedicated people to take them on
| jakeinspace wrote:
| What scares me about ChatGPT is less so that I'll lose my job
| (though it's possible), but more so that I'll be using language
| models to work at higher and higher levels of abstraction doing
| mainly configuration tweaking. Some of the particular pain points
| expressed in the article should be removed with AI in the loop
| development, but it's another step away from "real programming",
| which is what attracted most of us to this field. Yes, creating
| things is the end goal, but I'm terrified that I won't be able to
| extract nearly as much joy when my job becomes largely prompting
| GPTn to swap out frameworks and UI paradigms for me like magic.
| gryf wrote:
| This.
|
| I already started using ChatGPT to solve problems because I
| can't be arsed to read through ten vendors worth of
| documentation. It wrote me a fairly complete and accurate chunk
| of code the other day to solve a problem.
| agumonkey wrote:
| I have to admit that having it giving short summaries of
| framework docs and models felt like drugs in my veins.
| IanCal wrote:
| I've found it's often like pairing with a junior developer
| who is familiar with whatever problem you're describing and
| types insanely fast, and that's without learning too much how
| best to prompt it. A recent discovery was that you can ask it
| what problems may exist in its code, then ask it to fix them.
| [deleted]
| rr888 wrote:
| Its already like this. My first job in the 90s I wrote our own
| linked list classes, a logging framework and a persistence
| layer. Now it feels like I write css and yaml all day.
| agilob wrote:
| Think about that guy who wrote his own operating system as a
| hobby.
| soulofmischief wrote:
| RIP Terry Davis <3
|
| https://templeos.org/
| therealdrag0 wrote:
| You could switch to a job where you're not writing css and
| yaml?
| [deleted]
| lordgrenville wrote:
| As some other commenters have said, this is really a self-
| inflicted problem. The author has chosen to do an EDA task which
| is very manageable on a laptop via a convoluted stack of cloud
| services - possibly just to illustrate a point. But even if this
| all worked smoothly, the fact that it is far removed from
| "software engineering" has more to do with the fact that it's a
| data analysis project. If it were about about writing firmware
| for audio hardware it would look very different.
|
| Nitpick: ! for shell commands is an IPython feature, not Jupyter.
| It doesn't work with other kernels.
| zubairq wrote:
| Loved this article! Yes, I think this accurately described 90% of
| my consulting engagements!
| somrand0 wrote:
| oh yea.
|
| using a software framework like django for "rapid application
| development" gives me a feeling closer to writing configuration
| files rather than actually "programming" (where "programming" is
| writing hardcore algorithms).
|
| but don't get me wrong, I liked doing that, I got paid to do it.
| But let's call it for what it is: that python code (django app)
| was really django framework config.
|
| this is a simlar phenomenon but worse, at least I could look
| around all of the actual code of the program for which I was
| writing configuration as code.
|
| then, the thing about hardcore algorithms is that they need to be
| written once, and then everybody can use them. this is a giant
| problem. as I think about this, such hardcore algorithms are
| digital artifacts, so they are subject to the same problems all
| other digital artifacts (media, videos) are. a problem also known
| as software piracy. but software has been about composing proven
| algorithms together since day 1. the problem by this point is
| socio-economic. not technical.
|
| I see the whole debate around "who will pay for critical
| infrastructure software" as another instance of "how should
| artists make money in the digital era".
|
| But this comment (which I'm editing) is already at 0 points.
| somebody doesn't like what I'm trying to say, but I knew this
| already.
| agumonkey wrote:
| it's config until you need an option not in the 'language' and
| then you have to get real dirty
| einpoklum wrote:
| > they need to be written once
|
| Well, if you were writing abstract pseudo-code, then maybe. In
| practice, the same algorithms are often reimplemented multiple
| times.
|
| > and then everybody can use them
|
| ah, if only that were true... implementations are often not
| that flexible, nor are programmers that keen on utilizing
| existing implementations.
|
| > this is a giant problem
|
| Problem? To the extent it's true - it's a boon, not a problem.
| Image if whenever you made a chair - suddenly everyone could
| get a chair without taking your own chair or spending any time
| and resources on chair construction. It's a miracle!
|
| > a problem also known as software piracy.
|
| 1. Piracy is when people on ships with guns rob other ships.
| Arrgh, matey!
|
| 2. Sharing and copying software or other media is a good thing,
| not a problem.
|
| PS - I haven't downvoted you. Point taken about django "config-
| programming".
| somrand0 wrote:
| well sure, I completely agree that it's indeed a boon. a HUGE
| boon.
|
| the problem I describe is not a technical one, but a social
| one. and it's only a problem due to the current way society
| works. It's only a problem for some people (e.g me); them who
| are well satisfied by this "status quo" don't see any issue
| beyond lacking enforcement of IP 'rights' and the need for
| better DRM, and copy protections, and other things like that.
|
| you're focusing on the technical aspects (plus you seem to be
| deliberately using the wrong definition of piracy).
|
| I'm talking about the social economic aspects: I'm saying
| that this boon brought about by digital technology is only
| benefiting a select few. I tend to think about this 'boon' as
| potential that we collectively seem to be choosing to forego;
| I think it's my life's mission to do everything I can to
| avoid this "foregoing" of the great potential unleashed by
| digital technology; I feel like I'm swimming against the
| current most of the time.
| einpoklum wrote:
| > I'm saying that this boon brought about by digital
| technology is only benefiting a select few.
|
| If you mean how most people in the world are under stricter
| social control with the advent of technology than they were
| in primitive tribal society (or maybe even middle-age
| agrarian society), then maybe.
|
| But if you mean the benefit from being able to make these
| copies - then definitely disagree: People get to have tons
| of free (libre & gratis) software, and lots of free
| (gratis) cultural products like images, audio, video and
| text. Are you bemoaning what we do with this glut, and
| perhaps the problem of over-abundance, shallowness of
| cultural taste etc.? Otherwise I'm still missing your
| point.
|
| About piracy: I'm using the right definition, it's the
| copyright crooks who are using the wrong one :-P
| slim wrote:
| I think you nailed it with configuration vs programming. modern
| software development is configuration. notice that
| "development" is not programming either. so this trend has been
| going for long time
| midiguy wrote:
| Would you rather do some library gluing, or reinvent a thousand
| wheels with every project? The latter is neat the first few
| times. Whatever your preference, your value as an engineer is
| much higher if you can glue. Imagine if a carpentry workshop
| gives a carpenter a fully-fledged set of industrial power tools,
| but the carpenter insists she can recreate all the other tools
| with just her whittling knife because it's in the 'true spirit of
| carpentry'.
| nijave wrote:
| Ideally a healthy blend of both. Wheels where it relates to
| your companies core competences or there's a gap in the market.
| Glue for everything else (you don't need to invent an
| infrastructure provisioning solution unless you're and
| infrastructure provisioning companies--there's plenty of mature
| solutions). Other places like application libraries--it might
| make sense.
| deathanatos wrote:
| I'd rather re-invent _some_ wheels. The problem with VendorOps
| as I see it is that quite often the vendored "solutions"
| aren't solutions: _they do not meet the requirements!_ Yet ...
| they sort of get like 20% of the way there, so they get adopted
| nonetheless, and the devs toil away on trying to get glue code
| to push it the remaining 80% of the way.
|
| But if we had a system _that we owned_ , then it could be
| adjusted to fit the requirements, elegantly. But we don't, so
| we can't.
|
| The other problem is the "Ops" part: vendor owned systems are
| opaque AF, and when something goes wrong, impossible to debug.
| Then you become a support ticket monkey, praying you can
| convince the powers that be on the other end that a. it is
| truly _their_ stuff that 's broken, not yours and b. we pay for
| it, so yes, you should support it.
|
| When a. or b. fails, then you end up writing yet more glue code
| to try to work around the bugs and outages that your vendor
| just doesn't give a shit about.
| hooverd wrote:
| Wheels aren't licensed rather than owned and charged per
| revolution... yet.
| a1369209993 wrote:
| > Would you rather do some library gluing, or reinvent a
| thousand wheels with every project?
|
| Wheels. Absolutely wheels. Library gluing is what gets us
| garbage like Electron that needs to die in fire.
|
| Imagine if a carpentry workshop gives a carpenter a bunch of
| IKEA kits and tries to conflate wanting to make furniture that
| isn't cost-cut prefabricated crap with insisting she can
| recreate all the other tools with just her whittling knife
| because it's in the 'true spirit of carpentry'. (Honestly, if
| anything, comparing Electron to IKEA is a insult to IKEA -
| there _are_ cases where using IKEA is actually reasonable, they
| just aren 't actually _carpentry_.)
|
| (You use libraries when it makes sense to use libraries, just
| like you use 2x4s when it makes sense to use 2x4s. Sometimes
| you can make the whole thing out of 2x4s, just like you can
| make a whole program out of: tr -cs A-Za-z '\n'
| | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q
|
| but if you're just gluing (screwing?) 2x4s together, you're
| going to get bad results when you need something that's _not_ a
| 2x4.)
| Havoc wrote:
| I suspect this effect is a big part of why Rust keeps winning the
| most loved language polls - because it has some of that old
| school programming vibe that got lost somewhere along the way
| adra wrote:
| Just give it enough time before incentives drive a language
| with large developer adoption to encourage adoption and wide-
| spread appeal vs. a niche language that has little value
| proposition to a company beyond an excuse to keep some key
| developers that are bored happy doing their side-projects.
| gptadmirer wrote:
| I am a fullstack web dev, for about 8 years now.
|
| These days I am learning Arduino and thinking of learning AVR
| programming.
|
| Is there a career path for this kind of intersection?
| Dave_Rosenthal wrote:
| Like many other commenters (of a certain age?), I too have this
| unsatisfied feeling about a particular kind of modern software
| development. The kind where you never really dig down and design
| anything, you just plumb a bunch of stuff together with best
| practices you find on stack overflow.
|
| Many commenters are attributing this problem to the modern high-
| level tools we now have access to. But I don't think this is the
| crux of the issue. You can face the same issue (you're plumbing
| things, not a designing a system) whether you are working with
| low level or high level components.
|
| Heck, you could be working on a hardware circuit, but if the only
| thing you had to do was make sure the right wires, resistors,
| capacitors, etc. were in place between the chips you're still
| just doing plumbing work.
|
| To me, one of the most satisfying things about programming is
| when you can build something great by starting with a concept for
| your lower-level primitives, your tools, and then work up through
| the higher levels of design, ultimately having _the pieces you
| designed_ fit together to form something useful to the world.
|
| This building-things-to-build-things idea is even satisfying in
| other areas. Just gluing a bunch of wood together to make a piece
| of furniture is fine, but building your own jigs and tools to be
| able to do the kind of cuts that enable the end design you
| envision is way more satisfying, and opens up the design space
| considerably.
|
| If I had to lament anything (and perhaps this is what's most in
| alignment with the post) it's that most of the high-level
| primitives you touch these days tend to be sprawling, buggy, not
| focused, and just generally not of high quality or performance.
| It's possible for high-level primitives to avoid these pitfalls
| (e.g. Sqlite, the canonical example) but it does tend to be the
| exception.
|
| I think there is still plenty of interesting and satisfying
| software engineering work to be done when starting with high-
| level libraries and tools. You just need to think about how to
| use their properties and guarantees (along with maybe some stuff
| you build yourself!) to enable the design of _something more_
| than just the (naively-plumbed) sum of the parts.
| jjav wrote:
| Good article, great points. The Knuth quote on lack of creativity
| is right on the money. It's why I've been drifting away from
| hands-on programming even though I'd rather not. I still love
| programming as much as I ever did as a teenager. Late at night
| when everyone is asleep and I can work on my own code, it's as
| much joy as it ever was. What's different is that at $DAYJOB back
| in the 90s and 00s it used to be just as fun all day.
|
| Nowadays when it's just gluing frameworks together and
| configuring AWS services... it doesn't really feel any different
| intellectually than cleaning toilets. Sure the pay is better but
| as a creative challenge they're pretty much on par.
| mythhouse wrote:
| > it doesn't really feel any different intellectually than
| cleaning toilets
|
| There is a big difference. one is cleaning shit other more shit
| creating more shit :)
| thr0wawayf00 wrote:
| > Nowadays when it's just gluing frameworks together and
| configuring AWS services... it doesn't really feel any
| different intellectually than cleaning toilets.
|
| Comparing your six figure white collar job to basic janitorial
| work is pretty damn cringe and pretty objectively untrue.
| jacoblambda wrote:
| You're right. Cleaning toilets is at least a laborious task.
|
| Your standard CRUD applications and web services are largely
| just a rigamarole of reciting the right incantation and duct
| taping bits together. It's immensely non-stimulating work
| when done properly.
|
| This isn't an insult by any means. It's a testament to the
| triumphs of decades of engineering efforts to turn the
| process of orchestrating extremely complex electronic systems
| spanning continents into a largely trivial task for most
| projects.
| pcthrowaway wrote:
| OK, what if we say plumbing then? Same idea, and the pay is
| within an order of magnitude at the median.
| thr0wawayf00 wrote:
| Personally, I think it's pretty hard to make that kind of
| comparison accurately without having professional
| experience in both fields. One could also argue that it's
| as intellectually stimulating as being doctor, but how do
| we actually know that?
|
| I've cleaned toilets professionally, and I'll say once and
| for all: writing software, no matter how monotonous or
| boring, is nothing like cleaning toilets. And I'd be
| willing to bet that someone trying to make such comparisons
| has ever had to do that kind of work.
___________________________________________________________________
(page generated 2022-12-06 23:00 UTC)