[HN Gopher] Earning the privilege to work on unoriginal problems
___________________________________________________________________
Earning the privilege to work on unoriginal problems
Author : donutshop
Score : 84 points
Date : 2023-08-26 12:45 UTC (1 days ago)
(HTM) web link (landmines.substack.com)
(TXT) w3m dump (landmines.substack.com)
| dartos wrote:
| [flagged]
| zerbinxx wrote:
| > Encore automates infrastructure for seamless development, from
| local to your cloud.
|
| > Develop locally with instant infrastructure, preview PRs in
| dedicated environments, and skip tedious Terraform with automatic
| infrastructure setup in your cloud.
|
| If you thought this sounded like a sales pitch for bad
| engineering practices, it's because it is.
|
| I find it very funny that a company hawking their mutant CI
| "solution" talks so much about pmf when what they're selling is
| pretty undifferentiated
| hgsgm wrote:
| [flagged]
| mattgreenrocks wrote:
| Am convinced there exists web tech that enables you to move fast
| enough without requiring a full rewrite once you confirm product
| market fit. Preferably with first class type safety throughout
| (rules out RoR/Django).
|
| In ye olden days, we referred to this as "skill," but I'm more
| sure we're allowed to talk like that now :)
| Zetice wrote:
| Why avoid the rewrite? Why require your work to be "skillful"
| in the first place? Wouldn't it be better to focus all of your
| time and effort on the hardest problem of all, finding
| customers, and leave whatever wisps of brainpower you have left
| to the initial implementation, until you can hire people
| dedicated to that task?
|
| Engineers think a tech startup is mostly tech, but in reality
| it's almost no tech at first. In fact, the more tech you
| introduce early, the worse off you are.
| Kalium wrote:
| It's never product-market fit that requires a rewrite. It's
| bumping into the limits of the tools you chose up front. In my
| experience, that's mostly a function of how technologically
| wrong your guesswork was. If you set out to build cloud cost
| tracking but mostly build AWS accounting, that's pretty close
| and you'll likely do fine. If you set out to build games and
| wind up finding fit with an enterprise chat system, you're
| likely to end of with a full rewrite no matter what web tech
| you started with.
|
| In other words, I don't think there is any web tech that does
| this because it's all about what's between the ears of humans.
| cultofmetatron wrote:
| Phoenix framework is what you're looking for. I can build out
| mvps as quickly as in django/ror but the runtime perf is on par
| with go or rust (for io bound stuff)
|
| Been in operation for 3 years and scaling has come out though
| incremental improvements. no large scale rewrites needed.
|
| 1. out of the bod req/res time is fast
|
| 2. multiclustered websockets come out of the box with channels
| for subscribing to events
|
| 3. easy to spin up a "service" as a genserver
|
| 4. best ecosystem of tooling for building asynchronous and
| parrallel systems
|
| 5. liveview is THE standard for reactive frontend frameworks
| where you are updating in real time from serverside html. (no
| shade thrown on rails but hotwire is a joke by comparison)
| liveview processers are persistent for the live of teh user
| session and can subscribe to server side events so you can have
| a user make a request, push the task into the background and
| update the ui when its completed with very little code.
|
| ok so type safety isn't at ocaml, rust or haskell level but its
| good enough that Its rarely the source of bugs assuming you
| perform even minimal testing.
| eropple wrote:
| Phoenix is quite nice if you're coming to it from a
| dynamically-typed environment, but I'd strongly dispute that
| its type system is "good enough" at large (human, not
| technological) scales. Phoenix itself makes good decisions in
| a lot of places but it relies on things like Ecto for
| database manipulation, which (as just one example) is
| absolutely _not_ enough for substantiative, human-scalable
| development--your "minimal testing" is my, as even a
| comparatively easygoing TypeScript-down-the-stack enjoyer, _a
| whole lot of extra stuff_.
|
| It also faces impedance mismatch with a lot of cloud tooling
| --which is not a disqualifier by itself and BEAM might not
| even be _wrong_ , but most Elixir-in-anger systems I've seen
| end up abandoning much of the benefit of BEAM clustering and
| running a bunch of horizontally scaled web applications
| because they've got containers to manage and it fits the
| overall get-it-out-the-door plan better. This makes things
| like "just add another genserver" not really map to reality
| all that well, though of course genservers are a good layer
| of abstraction and modularization on their own.
|
| LiveView is great, though. I am hopeful for React server
| components and server actions to continue to mature; they're
| promising as it is.
| cultofmetatron wrote:
| > is absolutely not enough for substantiative, human-
| scalable development
|
| you'll have to elaborate there. we have a pretty large
| codebase at this point that serves 3 different apps.
| granted our engineering staff is small but thats exactly
| what makes elxiir so great. It lets you build scalable
| infrastructure with a small team of humans
|
| > most Elixir-in-anger systems I've seen end up abandoning
| much of the benefit of BEAM clustering and running a bunch
| of horizontally scaled web applications because they've got
| containers to manage and it fits the overall get-it-out-
| the-door plan better.
|
| containers and clustering are not mutually exclusive. we
| run a elxiir cluster and take advantage of that with shared
| data and the ability to send mesages between each machine
| in pure elixir. At the same time, each vm is running inside
| a docker container managed with kubernetes. they work well
| together. we had one instance where a vm went down and
| kubernetes immediately brought it back up followed with it
| connecting to its peers.
| jiggawatts wrote:
| I worked at a startup in the very early 2000s that was making
| online games in Java. Think applets with 2D sprite graphics.
|
| They developed their own custom database, a shitty key value
| store that didn't even have transactions because "Oracle doesn't
| scale and it's too expensive". They couldn't make reporting work
| so they copied the data hourly to Oracle anyway. (It turned out
| that Oracle does scale.)
|
| They also invented their own RPC protocol, had their own
| threading library, used an obscure build system, an even more
| obscure SCM, etc...
|
| Nothing was standard. It was all bespoke, _webscale_ , and in-
| house.
|
| When I started there as a junior they had me fix bugs in their
| code. Instead, I would simply delete thousands of lines of
| spaghetti and replace it with a call to a standard library. It
| was like spitting into a volcano to put out the fires of hell
| itself. There was no hope.
|
| They burned $10M and made $250K revenue and collapsed.
|
| Their competition used Oracle, ordinary tooling, and made tens of
| millions in profit.
| tikhonj wrote:
| As a counterpoint, I interned at Jane Street about a decade ago
| --when they were much smaller than now--and they also built
| everything in-house. I'm not just talking about the
| trading/finance-specific systems (I didn't interact with those
| as an intern), but general-purpose internal tools: a code
| review system, a pub-sub system, custom binary and text
| protocols, lots of developer tooling, their own build system,
| their own standard library and async runtime... All with <100
| developers _total_.
|
| And you know what? It worked _amazingly_ well. I 'm not just
| talking about scale, but also about how much any single
| developer seemed to be able to do. Their tools and APIs were
| tastefully designed, tailored to their own needs and fit
| together remarkably well. I remember that they were
| consistently building things with just one or a handful of
| people that would have taken multiple _teams_ at other
| organizations I 've seen.
|
| Looking back, that was a remarkably productive engineering
| environment _because of_ how much they were willing to do
| themselves. The key insight is that building something for
| yourself is qualitatively different from building something for
| external consumption _or_ using something general-purpose that
| already exists. You naturally tailor the code and tools to what
| _you_ need, which is much easier than building something for
| what you think _other_ people _might_ need--and, surprisingly
| often, easier than bending a different tool to your own ends.
|
| But, perhaps more importantly, you develop a shared mental
| model of the system you're developing and how it fits in with
| everything else that you are doing. That shared mental model
| is, ultimately, worth far more than the code by itself... but
| it isn't legible to management, so it's hard to foster and
| preserve without a strong engineering culture backing it up.
|
| The other lesson I took away from that experience (and others!)
| is that the high-level questions people focus on ("buy vs
| build", "monorepo vs polyrepo"... etc) are nowhere near
| determinative. Whether or not any given approach works out
| comes down far more to the little details and nuances of what
| you're doing and how you're doing it, as well as a healthy
| dollop of circumstances beyond your immediate control.
|
| At the end of the day, if _everything_ else was the same, but
| the startup you 're talking about was using perfectly standard
| and popular tools, would you expect the outcome to be any
| different? I've seen far too many Enterprise-grade trash-fires
| to believe that. If the core dynamics that led to a mess don't
| change, you can have a "not invented here" mess or an
| "Enterprise Best Practices" mess or a "buy everything, build
| nothing" mess--but there's no simple strategic direction that
| won't get you _some_ kind of mess.
| walterbell wrote:
| Was Jane Street searching for product-market fit?
|
| _> shared mental model of the system you 're developing and
| how it fits in with everything else that you are doing. That
| shared mental model is, ultimately, worth far more than the
| code by itself... but it isn't legible to management, so it's
| hard to foster and preserve without a really strong
| engineering culture backing it up._
|
| Sounds awesome, did they have engineering leaders with long
| tenure?
| canadianfella wrote:
| Which company was the competition?
| bfeynman wrote:
| Kind of disagree with this, which seems outdated given that it
| becomes easier and easier to implement best practices. Setting up
| CI/CD for a basic MVP is trivial, making it super easy to
| contribute makes iterations faster than just working on "product
| fit"
| [deleted]
| Zetice wrote:
| You'd be surprised by how few people find "Set up CI/CD for a
| basic MVP" a trivial task.
| fijiaarone wrote:
| When something becomes too easy, a more complex, expensive,
| brittle, slower, less flexible way of doing it will be
| invested.
|
| Take GitHub Actions, for example.
| PaulHoule wrote:
| [flagged]
| [deleted]
| konschubert wrote:
| This is basically the same as the boring technology essay, no?
|
| https://mcfunley.com/choose-boring-technology
| leoqa wrote:
| The same is true for operations, if your business scales using
| humans you should avoid complicated tooling until you have
| product market fit. An old CEO curbed my enthusiasm by telling me
| "businesses are built on excel" and he was right.
| kevingadd wrote:
| I've spent a lot of time cleaning up the messes created by
| Experts like the article's author and I'm simply not convinced.
| It's easy to say "product market fit is King!!!" When you aren't
| the one working long nights and weekends reverse engineering code
| after a bug in pmf-hunting payment processing code made all
| purchases free for a day.
|
| Stuff like ci lets you ship safely.
| sanderjd wrote:
| It's interesting to see how most of the comments here are
| either "no, this is how lots of startups end up with a mess
| they have to clean up" or "yes, lots of startups optimize this
| stuff too early".
|
| And I think that's because both things are true! This is one of
| the many hard parts about starting a brand new company,
| figuring out the right balance to strike on this. It's no
| surprise that companies mostly get it wrong, and in both
| directions.
|
| There's no single right answer here. It depends on exactly what
| the company does, the exact path to product market fit, what
| growth looks like afterwards, and how lucky the guesses about
| all that stuff were.
| konschubert wrote:
| You can use off-the shelf components and use boring
| technology without making a big mess out of technical debt.
|
| The alternative to building your own DB isn't to put
| everything into a big json file.
|
| The correct alternative is Postgres.
| svachalek wrote:
| Are there companies that die because they produced something
| people loved and wanted to buy but just couldn't deliver
| because off the shelf components were just too subpar? I
| haven't seen that case. I have seen companies die trying to
| perfect software no one is buying though, quite often.
| brigadier132 wrote:
| Of course companies have died because of quality issues.
| Framing it as "they produce something people love but the
| quality is subpar" is a false framing.
|
| There are no subpar products that people love. First of
| all, "quality" is relative. See chatgpt, it's wrong half
| the time, in a decade if someone were to release a chatbot
| of chat gpt quality we would say it's terrible. But today,
| it's the best we have.
|
| The classic story is how airbnb and stripe launched without
| coding anything, everything was done manually.
|
| Now launch an airbnb competitor today using the same
| strategy. Obviously comparing yourself to airbnb is dumb,
| because back then _all_ software was terrible.
|
| The actually successful modern companies of the past few
| years are openai, tiktok and figma. They all launched with
| _complete_ products and are massively successful. That 's
| what it takes today.
|
| > I have seen companies die trying to perfect software no
| one is buying though, quite often.
|
| Most companies die.
| kev_dev wrote:
| There is a difference between "table-stakes" complexity and
| premature complexity. I'd argue that a simple but sane CI /
| deployment pipeline takes relatively little work to set up and
| falls under "table-stakes" in that even a pre-pmf team will
| have a positive ROI in doing it.
|
| On the flip side I have been the one working long nights and
| weekends reverse engineering code by engineers who prematurely
| built complexity into the system because they wanted to add a
| GraphQL api in addition to a rest API. All while in the pre-pmf
| days, with no value-add to the features that ultimately DID
| find pmf.
|
| I do generally believe that cleaning-up after the pmf-hunting
| phase is itself a privilege that many startups do not get to
| experience, and should be treated as such. I understood the
| author as arguing that we shouldn't chase shiny things and
| should ruthlessly avoid complexity in favor of finding pmf.
| This philosophy is clearly illustrated in the devtools startup
| he is running. I thought there were some cool ideas there.
| kevingadd wrote:
| I simply reject the premise that all problems a startup needs
| to solve are original problems. Your customers have lots of
| ordinary problems too, as do you. Sure, you can't justify
| spending months on building some custom GraphQL
| infrastructure or the perfect CI/CD deployment system, but
| your customers _do_ care about things like "when I download
| and install this software it's not a corrupt build" and "the
| software's updater works" and "when I pay these people money
| for their software I get what I paid for and don't get
| double-billed". These are all unoriginal problems that are
| nontrivial - ideally your startup solves them with off-the-
| shelf solutions to save time, but you still spend engineer
| hours integrating those solutions.
| hgsgm wrote:
| Who is putting forth that straw premise?
| kevingadd wrote:
| The original author?
| tombert wrote:
| I think the overall point for this is fine, but I'd argue that
| both Node.js and and MongoDB have basically crossed over into the
| "boring" realm at this point. I see Node.js used in lots of
| megacorporations, and I feel Mongo is more or less creeping its
| way into megacorps to for "stuff that needs to be fast and I
| don't care about ACID"; I think at this point I'd have to use one
| innovation token to talk some companies _out_ of choosing
| Node.js.
|
| That said, I do kind of hate this mentality; part of the reason
| that crap like Java 8 refuses to die is because companies always
| want to "established" tech, no matter how horrible of a fit it
| is.
| snowstormsun wrote:
| Don't write your own CMS!
| [deleted]
| brigadier132 wrote:
| Out of curiosity, how long do people think it takes to setup e2e
| testing infra and ci/cd? Because for my startup I'm building as a
| solo founder, it took me 2 weeks and I really just can't imagine
| working any other way.
| [deleted]
| theptip wrote:
| I view it more as a "% of resources" and "required number of
| nines uptime" kind of thing. If you're a solo founder pre-PMF
| or even pre-customers, pausing feature work to build infra for
| 2 weeks is IMO often going to be excessive.
|
| If you have a couple engineers and you spend one engineer for
| one sprint to get things running automated for the next
| quarters push, that could be a good spend.
|
| The MVP of CI/CD is a few hours wiring up Gitlab/GH Actions.
| The MVP of e2e tests is more variable but a moving skeleton can
| be hours if no UI, days if including browser testing.
|
| However what makes sense for a first pass is very context-
| dependent.
| quickthrower2 wrote:
| Yeah this startup stuff can feel like "don't waste time buying
| a dishwasher for your cafe you just need to get coffee in
| people's hands!". Just buy the dishwasher you know you will
| need it. Unless the date is 1960 or something.
|
| The trick is: learn CI/CD at your job. When you work on your
| startup it will be a trivial thing. I mean you probably will be
| using Vercel or similar where it takes 120s to setup and deploy
| from nothing. Most of that time is npm install on your machine.
|
| What about other things? k8s for instance?
|
| Simple: if it is second nature to you then you can use it. If
| you can't fix 99% of the issues on the spot then don't. Pick
| something else.
| MaxwellKahn wrote:
| I read the article as a general philosophy to spend more time
| on the product instead of premature scaling optimization. If
| e2e testing + ci/cd is cheap/easy and helps in your situation,
| then that's what you should do.
|
| I feel the author is more criticizing the CTO who prematurely
| decides to rewrite the infra to handle scaling to millions of
| concurrent customers before they have those customers.
| AlotOfReading wrote:
| For my space, it takes months and hundreds of thousands of
| dollars minimum to build new testing infra from scratch. You
| get to choose how many of those dollars are spent on developer
| salaries vs vendor solutions [0].
|
| [0] https://www.dspace.com/en/inc/home/news/engineers-
| insights/b...
| brigadier132 wrote:
| Hardware is a different game.
| vbezhenar wrote:
| For my projects I use containers. My rule is `docker build .`
| in the project directory must build the project and run all the
| necessary tests. It works for all technologies I had to use so
| far.
|
| Making `docker build .` run on push is not a hard task and
| probably could be done in a few hours with cloud services. I
| spent a day to write a custom github workflow to run it on our
| runner, but that's probably not needed for most people.
|
| After your OCI image is built, you can push it into the
| registry.
|
| I guess that counts as CI.
|
| Now E2E tests: I don't know anything simple yet. I guess I can
| hack together some script which will run docker compose and
| invoke that script with github push webhook. Probably would do
| it that way. Should work for not-so-complex projects.
| MattPalmer1086 wrote:
| Last startup I worked at spent a lot of time and money on
| creating a scalable cloud infrastructure. And thought every
| single thing we needed to do had to be written from scratch. And
| then rewritten to be nicer. But no market fit was found, or even
| an actual product at the end of the day.
| malux85 wrote:
| The last one I worked at was like this, due to its founder
| being 19yo and extremely inexperienced.
|
| Raised 6M, no customers, no revenue, no product, a buggy, non-
| functional MVP.
|
| 19yo founder was terrified to relinquish control to senior devs
| because he was scared they would implement something he
| wouldn't understand, so the company sat there burning cash
| making tiny incremental improvements and flip flopping on
| priorities as the new-shiny whims of the founder changed every
| few days.
|
| I have no idea how he raised 6M, I think it was a combination
| of lies and FOMO from latter investors.
|
| I tried my best to guide him and give advice but his ego was
| out of control (but only backed by mediocre skill set) so I had
| to leave when I realised I couldn't help anymore
| rqtwteye wrote:
| It always boggles my mind how these guys are raising that
| much money without any plan.
| jfengel wrote:
| It's better without a plan. Plans have risks and
| foreseeable problems. Handwavey dreams are all upside and
| no downside.
|
| Of course you gotta find the guys with more money than
| sense. They're not exactly thick on the ground but during a
| boom cycle there seem to be quite a few. (Often the winners
| of the last boom cycle, convinced that they were smart
| rather than lucky.)
| cultofmetatron wrote:
| no kidding.. where are these investors??
|
| my company raised a small round e years in. enough to keep us
| engineers comfortable but no where near 6 million. meanwhile
| my cofounders have years of experience in financial analysis
| and running a restaurant while I have 10 years of experience
| as an engineer before stepping in here as cto and first
| technical hire. We actually do have traction and getting new
| customers every day off a product that was self funded. I
| can't understand investing 6 million on a 19 year old with no
| experience with an ego fueled by insecurity. A startup needs
| to be run like a phalanx because there isn't room for
| redundancy and dead weight. And definitely can't afford to
| stick to a ego fueled "vision." I trust my cofounders to do
| their job and they in turn trust me. the "vision" should
| always be about serving the users and thats pretty easy to
| quantify if you are willing to humble yourself and engage
| with them.
|
| the good news is we will have more favorable terms when we do
| a series A. The good thing about starting a startup in a
| recession is that you're more accountable to making your
| business actually be something people want.
| mgkimsal wrote:
| A few years ago, if some angel or 'friend of family' had a
| few hundred $k sitting around, they were getting 0 return
| on it. Investing in speculative startup ideas was at least
| a promise of a return. Now those same startup ideas are
| going to be competing against 5% no risk return. Some of
| the ideas like the one referenced above may not see
| $6m-level funding activity again because of higher rates
| now.
| [deleted]
| drBonkers wrote:
| > A few years ago... a few hundred $k sitting around...
| getting 0 return on it.
|
| People say this with poorly invested money all the time,
| but I must be missing something. You could invest it in
| an index fund, say the S&P 500, and annually yield over
| 10% on average [1].
|
| [1] https://www.officialdata.org/us/stocks/s-p-500/2016?a
| mount=1...
| Archelaos wrote:
| > ... competing against 5% no risk return
|
| Until recently, this was less than inflation. A year ago
| the inflation rate of the US$ was more than 8%. So the
| net result when applying an interest rate of 5% was -3%.
| Less than in most of the period before 2021 when assuming
| an interest rate of 0%. I doubt that investors are not
| aware of this, arn't they?
| canadianfella wrote:
| [dead]
| liquidpele wrote:
| Investors were probably "found" via friends of daddy.
| asdff wrote:
| Certain schools are charging people north of 100k in
| tuition just because they can. Even at the ones that don't
| you see students driving six figure cars somewhat bizarrely
| regularly. It's not hard to imagine finding six people
| whose parents can scrounge up a million in financing,
| especially if you have sold the kid on your idea and they
| seem passionate about it as their first big investment.
| asdff wrote:
| Part of the difficulty is navigating two sets of incentives.
| While yes, you can write fast and loose code that only a mother
| would love and do plenty of things much faster than whatever way
| is perfectly orthodox and correct. However, to be competitive in
| the job hunt, its expected people to have this portfolio of clean
| model code that follows all the best practices. The startup could
| fire you at any time, and you better have hoped you did your due
| diligence to spend part of your time on your own marketability
| instead of devoting it all to the startup, in this all too likely
| situation.
| mrfox321 wrote:
| I think you overestimate the likelihood of being judged on code
| in GitHub.
| vegetablepotpie wrote:
| I believe the term for this is _resume driven development_.
| This also leads developers to focus on learning new
| technologies and integrating them to their current project,
| whereas what the business really needs are stable boring
| solutions. Of course if developers do focus exclusively on
| serving the needs of the business, they're not building up
| their skills for their next job, and the business could still
| fail for circumstances outside of their control.
|
| There's a software law called Teslers law [1], which says that
| complexity cannot be eliminated, the most that can be done is
| to shift it from one part of the system to the next. You can
| make a similar law about _risk_. As a developer, if I ship a
| build system (complete a project successfully), or I build
| something with a new technology, I have a point to put on a
| resume and a successful accomplishment to talk about in my next
| interview. It may be that this is the wrong move for the
| business. It may be that they need a crappy spaghetti code
| abomination to achieve a product market fit in that time. If
| founders know what's going on and they stipulate _boring
| solutions_ the developer accepted risk. They've completely
| invested in the business and are completely tied to the
| decisions of their founders and leadership for their success.
|
| In a perfect world this would start a conversation about
| compensation for accepting or balancing risk, but in my
| experience this absolutely never happens because it gets
| political. Leaders always win because they're better
| politicians than developers. It's easy for leaders to be
| willfully ignorant and dismiss these concepts as too detail
| oriented, and devolves into leaders saying "we're nice people,
| trust us". But the risk _is_ there and someone is accepting it
| and developers respond to this by mitigating it on their end
| deep in the implementation details.
|
| We have optimized at a local maximum, but are globally
| suboptimal. I do not believe we'll ever achieve something more
| optimal because in business climates, everyone is trying to get
| something for very little effort. It takes much time to earn
| trust, but very little time to destroy it.
|
| [1]
| https://en.m.wikipedia.org/wiki/Law_of_conservation_of_compl...
___________________________________________________________________
(page generated 2023-08-27 23:00 UTC)