[HN Gopher] Scalability is overrated
___________________________________________________________________
Scalability is overrated
Author : mooreds
Score : 238 points
Date : 2023-02-04 18:32 UTC (4 hours ago)
(HTM) web link (waseem.substack.com)
(TXT) w3m dump (waseem.substack.com)
| m3kw9 wrote:
| Tdlr: Over optimize is a bet on the unknown, usually the bet
| doesn't pan out.
| Existenceblinks wrote:
| Yes. Most SaaS are B2B and good luck getting 200 customers.
| quadrifoliate wrote:
| Before startups rush to cargo-cult this model, I want to point
| out one important thing - _Adjust according to your market_.
|
| Pilot's customer base probably looks very different than yours.
| The author says in the article that they are a "startup-focused
| accounting firm", so for _their customer base_ which is likely
| startup founders or similar, the experience of talking to a
| fellow CEO probably "helps to 10x the customer experience" as
| the article puts it.
|
| On the other hand if you are selling me some software for a
| boring Enterprise integration, please don't send me calendar
| invites for a conversation with your CEO, I'd rather get great
| API docs and integration guides.
| flurly wrote:
| I completely agree with the idea that scalability is often
| overrated. While it's certainly important for large organizations
| and enterprises, for smaller companies and side projects,
| focusing too much on scalability can actually hinder growth and
| creativity.
|
| It's often more beneficial to focus on solving a problem
| effectively and efficiently, rather than trying to scale it
| before it's even proven to be successful. By starting small and
| iterating based on customer feedback, you can create a better
| product that solves a real problem, which will naturally attract
| more users and drive growth.
|
| That being said, scalability should definitely not be ignored
| altogether. As a project grows, it's important to plan for future
| scalability, but this should only be done once the product has
| proven its worth and a solid foundation has been established.
| simplotek wrote:
| > (...) scalability is often overrated. (...) focusing too much
| on scalability can actually hinder growth and creativity.
|
| I find this blend of comment unproductive and intellectually
| dishonest. "Often" and "too much" are weasel words that remove
| any objective meaning from the verbs they qualify, to the point
| where stating the exact opposite is also valid.
|
| The truth of the matter is that scalability is only not
| important when you don't need it, but it's already disastrously
| too late if you failed to scale when you need to.
| ec109685 wrote:
| "SAS Startups: offer a one on one chat to new customers"
|
| Doesn't have the same ring to it.
| ARandomerDude wrote:
| The problem is if the commenter says "doing X works" instead
| of doing "doing X often works" there will be a HN avalanche
| of "yabut so-and-so didn't do X" and allegations of
| survivorship bias.
| simplotek wrote:
| > (...) there will be a HN avalanche of "yabut so-and-so
| didn't do X" and allegations of survivorship bias.
|
| Isn't it also survivorship bias to claim that scalability
| is overrated because a non-scalable system didn't broke in
| a particular case of a local deployment with residual
| demand?
|
| Meanwhile hikes in demand hitting an unscalable system can
| and do break businesses, and when it happens is already too
| late to rearchitect things.
| JohnBooty wrote:
| That being said, scalability should definitely not be ignored
| altogether.
|
| Yeah. You have to think about what's _possible_ to scale.
|
| You're planning to scale to 1,000,000 or 1,000,000,000 users
| eventually? OK. Well, what's your service? Are you planning the
| next Facebook where everybody can talk to anybody? Then you
| have some challenges you better start figuring out.
|
| Or, is it a SaaS where nobody needs to access anybody else's
| data? OK, then you can probably simply scale vertically for a
| while up to $SOME_MILESTONE, and then scale horizontally via
| some fairly simple sharding or maybe even just database
| partitioning or something. No need to do that now.
| dasil003 wrote:
| > _You 're planning to scale to 1,000,000 or 1,000,000,000
| users eventually? OK. Well, what's your service? Are you
| planning the next Facebook where everybody can talk to
| anybody? Then you have some challenges you better start
| figuring out._
|
| Thinking this way actually leads to one of the most common
| failure modes. Specifically the day-dreamer death where the
| principals become so enamored with a fantasy of what the end-
| state could look like that they lose focus on the next step
| to improve their odds of success. This is under-reported
| because day-dreamers are so common that when they try a
| startup and fail, it's not really an interesting pattern to
| report on. Such founders will tend to cite specific reasons
| that sound better in context (eg. wrong market, product
| flaws, sales funnel flaws, etc). But as someone who's been in
| the web startup space 25 years, I've seen it over and over
| again where founders fail to fully engage with the hard
| problems right in front of them in favor of working on what's
| fun, easy or tied to some dopamine hit of imagined future
| success.
|
| In practice, you should not spend one second thinking about
| the requirements of 1B users until you have at least 10M. You
| will need to completely rewrite everything multiple times
| along the way to that scale anyway, and more importantly, you
| won't know the product requirements to actually get there
| until you achieve the earlier milestones and get the
| necessary user feedback about what's working at each size of
| user base.
| switchbak wrote:
| Precisely. It'd be like trying to make an F22 fighter jet
| right off the bat when you're still in the era of biplanes.
|
| I think people often underestimate the strata of learning,
| and the essential feedback that gets you to the next level.
| Or at least that's how it works for mere mortals like
| myself :)
| switchbak wrote:
| That's exactly right: it depends so much on the context and
| (expected) access patterns. Sometimes you really do need to
| plan ahead, but even that's more about not painting yourself
| into architectural corners rather than specific tech.
|
| What I see also is not just scalability over-engineering, but
| the same in the solution domain. When you don't really
| understand your customers needs sometimes there's a desire to
| create something with ultimate flexibility.
|
| In the days of Xp we used to say YAGNI when people would be
| overly general. I'm trying to bring that back into fashion.
| scott_w wrote:
| > Are you planning the next Facebook where everybody can talk
| to anybody? Then you have some challenges you better start
| figuring out.
|
| Even Facebook didn't start trying to figure that out until
| they had to. Famously you needed an invite for a _very_ long
| time. I'd imagine infrastructure limitations factored into
| this.
|
| You'd be surprised how many successful companies push off
| dealing with scaling problems until they absolutely have to.
| Izkata wrote:
| > Famously you needed an invite for a very long time.
|
| And then it opened to needing a college/university email
| address. Then it opened to everyone.
|
| I got in with my college email address, and at the time it
| was considered a feature, not a limitation: a combination
| of higher quality since people couldn't just make new spam
| accounts whenever, plus a bit of a status symbol compared
| to just being on MySpace.
| JohnBooty wrote:
| That's a great example, thanks.
| [deleted]
| ranting-moth wrote:
| IFIFY: focusing too much on scalability WILL actually hinder
| growth and creativity.
|
| It will hinder growth and creativity because you are solving
| the scaling problem, which is a hard problem. Especially if
| you're early on and don't really have a product or customers.
|
| I've seen multiple projects tank because people working on it
| were solving problems they didn't have and wouldn't have for
| the next 2-3 years _at least._ But they only had a few months
| of funding.
| anyonecancode wrote:
| It's kind of along the lines of "premature optimization is the
| root of all evil." A true quote, that doesn't mean optimizing
| isn't important! The whole trick is in being able to tell
| what's core and what's premature.
|
| Another way of looking at it -- scalability is pretty much the
| entire value proposition of software. It allows people to do
| the same, or more, amount of work without having to linearly
| scale out the number of people working on something. When
| you're first starting out, demonstrating even being able to
| _do_ the work is the first hurdle, but past that you're
| definitely going to need to show you can scale that out
| efficiently.
| [deleted]
| owl57 wrote:
| _> Another way of looking at it -- scalability is pretty much
| the entire value proposition of software_
|
| That sounds like a good way to look. People are Turing-
| complete, although that doesn't mean that they are very good
| at scale without instructions, and "programming" clusters of
| people has its own set of quirks. :)
|
| _> It allows people to do the same, or more, amount of work
| without having to linearly scale out the number of people
| working on something._
|
| But this is not the entire story, but only the throughput
| part of it. And there are entire classes of software where at
| least part of the value lies in latency. That's, for example,
| most of Internet as we know it, from the hypertext browsing
| to videoconferencing, and most software we would call
| "embedded", from watches to spaceships.
| anyonecancode wrote:
| > But this is not the entire story, but only the throughput
| part of it. And there are entire classes of software where
| at least part of the value lies in latency.
|
| Yes, that's a good point!
| wanderingstan wrote:
| After nearly two decades in early stage startups I couldn't agree
| more. Looking back, I know we often built too much too soon, and
| had too much confidence that we were building the right thing.
|
| These days I often advise would be founders to start with doing
| their idea manually with just the minimum amount of tech.
|
| Maybe just a google sheet and answering emails and a handful of
| "customers." If you can't give people a good experience with a
| totally custom personal experience, they're not going to like it
| any more when it's automated and scalable.
| roncesvalles wrote:
| This seems like a big reason why taking on investors too early
| can be harmful. You can't be sitting on runway money and taking
| baby steps with it, even though sometimes that's the right
| thing to do.
| adrr wrote:
| It really depends on the situation. No one at a successful
| startup says, we spent too much time on scalability in the
| beginning. They probably say the opposite. I worked at one
| company that we couldn't rack servers fast enough to keep up
| with our growth. Our competitor who was the market leader had
| to turn off the ability for new customers to signup to prevent
| their site from crashing under load. We ended up surpassing
| them.
|
| Getting a million customers isn't that challenging and really a
| function of money. With $50 CPA, you just need $50m. 1m
| customers will take down most sites unless you do some
| optimizations.
| klabb3 wrote:
| I think you're both right, even if I'd guesstimate the
| situation you encountered is very, very rare.
|
| You should make architectural choices that CAN be scaled
| within reason, on demand and ideally incrementally. That
| doesn't mean you need to build your app on Kubernetes and
| automate welcome emails, surveys, metrics, A/B testing etc.
| But you probably shouldn't build on top of fundamentally non-
| scalable systems either.
|
| This is provided you're in an environment that can reasonably
| scale to 1M users or whatever, which isn't true for all
| domains. Say B2B where that last B is only 10000 people world
| wide.
| jdsully wrote:
| Post product market fit scalability is critical. But a lot of
| effort is spent on things that will never get traction in
| startup land. Before you get traction the times at bat is the
| more important metric.
| avip wrote:
| In other words, founders keep insisting on not reading the lean
| startup, or ignoring all the advice given there.
|
| Ignoring history is a guaranteed path towards repeating it.
| osigurdson wrote:
| Is it possible to generate enough excitement with just a Google
| sheet however? I suppose if it is B2B it would be fine.
| ant6n wrote:
| Well don't tell the public there's a Turk under that AI chess
| board.
| osigurdson wrote:
| What type of business is this? It would be hilarious to
| have a human reading through http requests and writing out
| responses!
| rr808 wrote:
| Steve Blank has talked about this for decades. A startup is
| about validating you have a product that customers want. You
| want to connect with customers and prove you have a market
| before scaling. https://steveblank.com/tag/customer-validation/
| davnicwil wrote:
| I think this kind of concierge / person behind the curtain
| approach only works for a certain type of business though,
| usually services where aspects of the service pipeline can
| either be automated or not.
|
| For purer technology / product businesses, how do you do this,
| fundamentally? How would Google have manually mocked up their
| early product? How would Facebook? Github? Tesla for that
| matter?
|
| Sometimes you really do just unavoidably have to build the
| product out before testing the market, and if it doesn't work,
| just accept the sunk cost and throw it away - and sometimes
| fail completely as a result.
|
| I don't see this as a fundamentally solvable inefficiency, just
| part of how tech product startups work, and the very tradeoff
| that must be made to allow for innovation to happen.
| fragmede wrote:
| > how do you do this, fundamentally? How would
|
| there are still smaller pieces you can MVP to a smaller
| audience before launching it to the world.
|
| > Google have manually mocked up their early product? How
| would
|
| Crawl an intentional community (remember webrings?) or other
| small directed subset of the web and see if you're able to
| get better search results using terms you know exist in the
| corpus, rather than _all of the Internet_.
|
| > Facebook?
|
| They had Myspace as an example so the idea wasn't exactly
| unproven.
|
| > Github?
|
| Kernel developers were already using the software,
| successfully, all (for large values of "all") they did was
| bring it to a wider market with a better UI.
|
| > Tesla for that matter?
|
| People don't get this, but Tesla's biggest achievement to
| date, isn't the cars themselves, but the factory that they're
| built in. There's no way to MVP an entire factory, but
| building a car in a bespoke, pre-assembly fashion is totally
| possible and totally doesn't scale.
|
| If you're asking if electric cars were known to work, the
| first one came out in _1832_. If you 're asking about
| product-market fit, they keep selling cars they haven't made
| yet, just to gauge demand. Aka where's my Cybertruck!?
| HWR_14 wrote:
| > just to gauge demand
|
| The hundreds of millions of USD as an interest free loan
| seemed more important than anything else.
| itake wrote:
| Most startups have a red-ocean indicator in the space to
| point at when telling people about their problem. Most
| startups fail.
| fragmede wrote:
| are those remotely correlated though?
| kirse wrote:
| _too much confidence that we were building the right thing_
|
| SW is weird in that way, we can basically build houses that
| nobody wants to live in. Thing is you probably were building
| the right thing, structurally speaking.
|
| My rule over time has become "structure follows complexity"
| i.e. Anticipate complexity where possible but don't apply the
| engineering structure until the complexity really exists. If
| you built an MVP house, don't add the second bathroom until
| you've got customers complaining about deadlocks on the first.
|
| It's tough because that basically runs counter to the Spirit of
| an Engineer, which is just to imagine the grandest well-
| thought-out + organized thing and march toward creating it.
|
| The bonus of having two decades+ in building SW though is you
| start to see the patterns and realize that the same fundamental
| needs drive everything, so the intersection of building
| something great with customer needs becomes less of a mystery
| and more of a "how fast can we juice this thing". At that point
| I think scalability prep is like compound interest.
| Frost1x wrote:
| >It's tough because that basically runs counter to the Spirit
| of an Engineer, which is just to imagine the grandest well-
| thought-out + organized thing and march toward creating it.
|
| I'm not sure if _Spirit of an Engineer_ is a book, essay, or
| some prior set of principles I 'm unaware of but if it's not
| (and even if it is), I tend to disagree here.
|
| The spirit of of an engineer, in my opinion, is to solve
| problems and problems for people--that's what I and most
| engineers I've spoken with have been drawn to (ignoring those
| purely seeking a high paying profession here since were
| talking about "spirit"). When I was youthful I sought out
| technical complexity and grandiose solutions (I admittedly
| still admire it when it's done correctly). None the less, to
| me, most of it is wasteful and at the end of the day I want
| to build the most _useful_ solution for people to make their
| lives easier.
|
| _Some_ problems require high complexity and glass cathedrals
| in terms of technical solutions, most really don 't, at least
| most of the problems I've been exposed to.
| btilly wrote:
| Actual engineers generally do try to solve problems for
| people. But the engineers understanding of the people that
| they are solving it for is generally rather flawed. And
| therefore their solutions frequently are difficult for the
| intended users for reasons that the engineer can't see.
|
| And yes, engineers really do err on the side of designing
| for future problems and future users that often will never
| exist. This effect is worst on a rewrite of a system - look
| up the "second system effect". We tend to only truly get
| pragmatic tradeoffs the third and later times.
| bcrosby95 wrote:
| I call it "keeping an escape hatch".
|
| The potential need to scale and add features will inform my
| design decisions, but they will not direct them.
| pfarrell wrote:
| "keeping an escape hatch". I like that. A former colleague
| said something similar that also stuck with me. Good
| software architecture allows you to delay decisions.
| alexvoda wrote:
| China proves that building cities worth of actual houses that
| nobody wants to live in is a thing.
| hinkley wrote:
| There's a ninety ten rule with structure that can be hard to
| communicate in code reviews. There's a subtle difference
| between not adding unnecessary structure and writing code
| that is antagonistic to it being added later. Opening a PR
| where someone added a feature or fixed a bug exactly where I
| would have done so is one of the best feelings. Especially
| when they're changing my code. Either they're a very good
| coworker or I've laid the groundwork well. Either of these
| are good news.
|
| Very recently I've come to think of my architecture style as
| Jazz. Like the old trope about listening to the notes he
| isn't playing. But also the improvisational nature and the
| flexibility to adapt to new information.
| maerF0x0 wrote:
| > with just the minimum amount of tech.
|
| as an engineer this part I entirely agree with. Far too often
| CEO driven development leads to pivoting 1Month into 9 month
| projects, 9 months in a row. such that you have nothing
| deployed when you at least could have had 1 thing, but instead
| you have none.
|
| This can be solved as much by discipline to stay on track as by
| choosing to reduce scope to 1 month ideas only.
| dhbradshaw wrote:
| I think this is the right approach.
|
| There is however a pitfall that this approach seems to lead to
| unless you actively prevent it. Roughly the process is 1. You
| hire someone to take care of things manually. 2. As you grow,
| your devs are worrying about other things and so instead of
| automating you hire more and more people to handle the toil
| that should have been automated along the way. 3. Because these
| jobs are handled by one group of people and developers are
| working on customer facing features, the communication and
| prioritization necessary to fix the issues rarely come up.
| StreamBright wrote:
| s/scalability/technical scalability/
|
| Other kinds of scalability (financial, development) are
| underrated.
| SCdF wrote:
| I am aging myself, but:
| http://widgetsandshit.com/teddziuba/2008/04/im-going-to-scal...
|
| The bit that stuck with me (re-reading for the first time since
| 2008, I forgot how aggressive it was): "Scalability is not your
| problem, getting people to give a shit is."
| vrnvu wrote:
| Great post thanks for sharing. I also highlight this one:
| http://widgetsandshit.com/teddziuba/2011/12/process.html
| chaboud wrote:
| That's a great write-up. However, once people _do_ give a shit,
| if you didn 't think about scalability, it becomes an
| _enormous_ problem.
|
| And, funny enough, the people that didn't think about
| scalability, if they're smart, have typically moved on to
| something else.
| switchbak wrote:
| There's also the situation where you've passed your
| scalability limits, you have users, the system is falling
| down and the business has blinders on and simply won't accept
| that they need to invest in a better way.
|
| Being overly speculative and ambitious early on is a great
| way to fail. Building on top of something past its limits and
| ignoring the (developer, customer, velocity) pain is yet
| another.
| cordobaplaza wrote:
| chase demand, and build scalability only when the lack of it
| becomes a problem -- definitely agree
| jaequery wrote:
| I just moved away from GCP to DigitalOcean and went from paying
| over $1k a month to $100 a month.
|
| At this phase of the startup, we aren't missing a beat.
| JanSt wrote:
| I'm also running everything on DO. Dead simple and good
| pricing.
| osigurdson wrote:
| The market wants cloud to be a commodity - I'm certain it will
| get what it wants eventually.
| convolvatron wrote:
| I've been working on and off in supercomputing since the 80s.
|
| every system I've ever worked on has needed work to get to a
| certain scale. and that carries you for a while and if you need
| to get past the next plateau you need to rearchitect again. rinse
| and repeat.
|
| so its not a binary thing.
|
| so unless you have some specific goal in mind, invest to get a
| little past where you need to be today.
| mansoon wrote:
| Unless I have micro services then how scale??????
| taormina wrote:
| Do Things that Don't Scale: http://paulgraham.com/ds.html
| heywhatupboys wrote:
| Went to the pilot.com website. Scroll down.
|
| In huge text: "We partner with the best financial tools in the
| business" and a lot of logos from famous companies! I think, oh
| wow I haven't heard of this company before. But then see the text
| below the large text in small, gray text: "We're fluent in your
| finance tech stack and seamlessly integrate with the tools you
| use."
|
| oh...
| wdaher wrote:
| (Post author here.)
|
| In addition to having deep integrations with these tools, we
| actually _do_ have real partnerships with almost all of them.
|
| "What's a real partnership?" is a great question, but I'd
| mostly think of it principally as "We know them, they know us,
| we have a red phone we can pick up to get things solved for
| each other" (which is good for the customer), and then also
| "And we occasionally do some comarketing together."
| mlhpdx wrote:
| Indeed. The hardest thing for a technology person to do is
| nothing. It's seemly irresistible to "solve" problems fast and
| consider the consequences slowly. My term for the consequences of
| early structure is "calcification" - the loose aggregate of a
| startup becoming an unyielding mass resistant to change. Code,
| processes, culture, whathaveyou.
| pphysch wrote:
| It's easy to say "scalability is overrated" if you've dealt with
| unnecessary k8s deployments.
|
| It's easy to say "scalability is underrated" if you've dealt with
| businesses built on hundreds of standalone PHP/Perl/Python/JS
| scripts with zero unifying architecture, where migrating a data
| schema is virtually impossible because of the state of technical
| anarchy.
|
| Scaling is hard.
| purple_ferret wrote:
| > PHP/Perl/Python/JS scripts with zero unifying architecture,
|
| That's why you use something like a Rails monolith....but oh
| wait, Ruby doesn't scale well!
| ndriscoll wrote:
| That's why you write a monolith on the JVM or CLR, which do
| scale well.
| nine_k wrote:
| As long as you can scale (shard) your persistence layer, I
| don't see why RoR won't scale.
|
| Look at Github, for instance.
| threeseed wrote:
| Not everything is a glorified CRUD app.
|
| If you're doing any computation or highly concurrent
| workloads then you will discover the performance issues
| with Ruby well before you outgrow your persistence layer.
| vidarh wrote:
| I have done both in Ruby, and addressing it was not a big
| problem. E.g. my MSc. involved doing a lot of statistics
| and image processing in Ruby, and solving the performance
| bottlenecks meant rewriting about ~30 lines or so in C
| using Ruby Inline. Later I did map tile rendering on
| demand handling thousands of layers uploaded by customers
| in Ruby. Both using 1.8.x, btw. - far slower than current
| versions.
|
| It took more CPU than if we'd rewritten the core of it in
| something else, but it let us iterate the rendering
| engine itself much faster, and most of the expensive
| computations were done by extensions in C anyway (e.g.
| GDAL and the like).
|
| Of course you can find areas where it's still not viable,
| but my experience is that if you start by prototyping in
| whichever high level language - no matter how slow - that
| is most productive for you, you'll inevitably find you
| need to rewrite far less than you might think to get it
| fast enough. But more importantly: The parts you end up
| rewriting will very often be entirely different parts
| than what you expected, because being able to iterate
| your architecture quickly tends to make you end up in a
| very different place.
| ecshafer wrote:
| TFA isn't talking about that kind of scalability. Its
| scalability in business processes.
| karmakaze wrote:
| Looking at the article title and comments I wasn't able to
| clearly tell. The second heading "do things that don't scale"
| is clearly recognizable and meaningful, but then in
| agreement, I'd have no reason to click.
| hinkley wrote:
| You damned kids never heard campfire stories of being brought
| in to consult on "scaling" a bespoke system for a growing
| company that has been limping along with VBScript in an Excel
| spreadsheet for five years past when it was still tenable to do
| so. The amount of feature parity required to get out of that
| blind alley often killed the project and injured the company.
| Some lived it, the rest of us knew someone who had.
|
| There was a brief moment after Oracle purchased Sun where I
| thought Oracle might try to engineer am Excel competitor on
| Open Office that had an easier migration to a real database
| (Oracle), but that dream died early.
| klabb3 wrote:
| FWIW I have a friend who works with market analysis and his
| excel scripts save an enormous amount of manual labor, today.
| It was the best tool for the job, for them. There are even
| international competitions in excel automation, which is
| kinda funny but also points to how far ahead Excel is for
| actual business uses.
|
| Are there scaling issues? Version control issues? Absolutely!
| But again, that doesn't mean that it's not the best tool for
| the job.
|
| It's easy to mount the highest horse from a technical
| perspective, but as engineers it's also our responsibility to
| be curious about what people are using _voluntarily_ , and
| not just what we think they _should be using_.
| orwin wrote:
| At my last job, my work was basically to read data from
| proprietary APIs and shit an excel table.
|
| I think I was hit by everything. Easy stuff at first: Crlf
| issues, XML Apis, weird rpc apis.
|
| Then, halfway through the project, the results had to change.
| Not only the order, datatype and headers (I actually
| overengineered the first version so those were configurable),
| but the format, duplicate on multiple columns (and empty
| fields counted as duplicate...). Worst job I've ever done.
| I'm also disappointed in myself tbh.
|
| But now I'm a bit of an expert on excel format issue and
| limitations, and that already helped me.
| weego wrote:
| That's conflating scaling with paying off technical debt
| twawaaay wrote:
| Scaling is hard. True.
|
| But the question is, are you trying to make your life miserable
| by scaling before exhausting other options?
|
| Most applications can be optimised for performance by orders of
| magnitude, much easier than trying to scale them by orders of
| magnitude. Any problem is much easier to solve and faster to
| deliver when you can fit it on a single server.
| aleksiy123 wrote:
| I once had a manager tell me scalibility was a good problem to
| have as it meant people where actually using your product.
|
| Obviously with some caveats. That you don't fall over at a small
| amount of users and you are at an early stage.
| IncRnd wrote:
| Your manager was correct. People often overengineeer or
| overarchitect what needs to be done. In reality, you need to
| meet the current needs of your business. Not every single
| action has to account for what might happen - or what you hope
| will happen - if absolutely everything happens perfectly.
| 8note wrote:
| Not only that, you've got more people using your product than
| you expected
| vidarh wrote:
| Cash when you're growing so fast you're running into
| scalability issues is also likely to be far cheaper than cash
| before you have any customers to speak about.
|
| This is more general than scalability: The more of _any_ work
| you can afford to defer without hampering growth, the more of
| your initial capital you can spend on growing, and the more you
| manage to grow the cheaper it will be to raise money to build
| out the rest when it 's time for your next investment.
|
| The hard part is to determine which aspects you can _safely_
| defer.
| freitzkriesler2 wrote:
| I find myself saying "No" more often times than not with founders
| who have scope creep. Scalability is one of them.
|
| Does it work well enough in the geo you want? Yes? You're good to
| go .
| mupuff1234 wrote:
| The time you waste on building an unnecessary scalable system is
| nothing compared to the time and energy you spend on maintaining
| that overly complex system.
| btown wrote:
| This article is about a stellar founder-led onboarding
| experience, with which I wholeheartedly agree! Especially for B2B
| startups, showing that someone isn't just a row in a database is
| vital.
|
| That said, in the current economic climate, the degree to which
| your processes _don 't_ scale for _already-onboarded_
| customers... isn 't necessarily bad, but something to be keenly
| aware of.
|
| If your support (or worse, product/engineering) staff is putting
| out fires that affect single customers, and the solves don't
| scale to other customers... you as a founder need to be _very_
| aware of this and not just sweep it under the rug as part of
| operating expenses. Being diligent about the fully-loaded costs
| of your humans servicing each customer, and comparing that to the
| cash flow (not just lifetime value!) per customer, is vital to
| ensuring your business is sustainable - and ensuring you continue
| to lead it!
| Joel_Mckay wrote:
| Adjust scope to match your teams skill level. That way no one has
| to adapt/train, and can stay in their comfort zone.
|
| Works fine until markets use-case suddenly shift, and a platform
| is hammered out of existence.
|
| Really depends on whether a business plan includes rapid growth
| (i.e. throw more money at the scaling issue), and an apology tour
| in the media. =)
| spamizbad wrote:
| I do agree but I hope people don't take the wrong lesson from
| this.
|
| For example, there's a "right way" to not do scalability and a
| wrong way. The right way would be avoiding patterns like
| microservices early on, while still minimizing human intervention
| and brittle integrations. The wrong way is to just have some
| kludgy solution that causes headaches for your customers.
|
| With our budgets tightening we axed startup vendor we were paying
| $90K/yr to because their shit was slow and kept breaking, pissing
| off our customers. We were able to spin up replacement
| functionality for their product in 6 weeks.
| maerF0x0 wrote:
| A problem many start ups encounter though is that they're only
| sustainable whilst doing non-scalable things.
|
| Using the author's example perhaps it's like they only way they
| actually convert customers is if the CEO calls them.
|
| Or sometimes it's more to do with engineering. Each customer will
| only stay if you implement a feature that costs you more than
| their CLV.
|
| More than once I've worked at a startup, or a company that has
| acquired a startup where this is the case. And when you suggest
| to fix the problem usually people can't quite comprehend it, so
| instead it becomes "you're the problem"
|
| Whenever this is the case it eventually catches up with the.
| Sometimes through burnout and burn and churn employees, sometimes
| because they finally learn you can't sell a dollar for 80cents
| forever.
| throwaway892238 wrote:
| Scalability is unnecessary when you're not growing rapidly. But
| when you _are_ growing rapidly, it 's suddenly very important.
| And if you didn't think about it before, it's already too late.
|
| You don't have to build to scale _right now_. But you should have
| a plan of _when_ you 're going to need to scale, and _how_ you
| 'll do it. Or don't, and wait until your pants are on fire and
| then run around screaming.
| preommr wrote:
| Someone explain the article to me.
|
| > I still personally email every new Pilot customer and offer to
| get on a call with them.
|
| How is this a thing that needs to scale and how is it time
| consuming to write some automation? Just export the contacts, and
| write a small script that loops over it and sends an email using
| some transactional email script if you've got something like ses.
| It shouldn't take more than an hour to do from scratch, and like
| 5 minutes if you look it up using chatgpt3 or even Stackoverflow.
| Scaevolus wrote:
| If you're going to spend 15 minutes in a call with someone,
| automating away a 10 second email is not much of a time
| savings.
| sound1 wrote:
| For me, if you are selling a service, customer service is
| paramount. Everything else boils down from there, just my
| opinion.
| iamgopal wrote:
| Rework is the book that is surprisingly underrated in this
| community which covered all points of comments but a decade ago.
| dboreham wrote:
| "Don't solve problems you don't have"
| eternalban wrote:
| > But more importantly, it locks in an experience before you're
| sure it's right.
|
| This single sentence communicates more than the author intended.
| I see the phenomena all the time: _poorly architected systems_
| that are _brittle_ in face of user requirement changes. In the
| extreme case, a new widget or input on the UI echoes all the way
| to the database, touching and mutating every layer in between.
|
| So, it is a perfectly valid matter to weigh the benefits of
| optimistic and forward looking engineering vs gaining deeper
| understanding of a market faster. This is a valid consideration.
| But if anyone tells you that going down any other path other than
| tacking on bits and pieces on the way "as we learn" is an error
| because of "user experience" (other than, ahem, performance) then
| be certain, certain, that the speaker lacks the necessary
| technical understanding to be offering strategic technical
| advice.
| Spivak wrote:
| I'm not sure I like this because you will always be wrong in
| what ways the code needs to be flexible. The most flexible code
| is soaking WET where each component is self-contained with no
| shared dependencies, no abstractions, and can be thrown out or
| rewritten with absolutely no possibility of it affecting any
| other parts of the service.
|
| Devs _hate_ writing this code, every fiber of your being
| recoils at the thought of just copy /pasting everywhere,
| ignoring anything that seems like a pattern, and having several
| identical API endpoints just to keep everything wet. But you
| will never be so productive in the face of changing
| requirements.
| Scubabear68 wrote:
| I agree with the premise, but would rephrase it as "know what
| you're going to need, when you're going to need it, and have a
| rough idea what it'll take yo get there".
|
| And as the article implies, not just systems but also your
| processes. I recently saw a moderately big startup strangling
| themselves on processes that they won't need for several more
| years. Mid level leadership (in itself a problem) agonized over
| scale, and didn't realize that the products were being built at a
| snails pace and money was being burned with abandon. Late 2022
| was a harsh wake up call.
| beebmam wrote:
| Couldn't disagree more, from my own experience, if performance is
| important for your service.
| twawaaay wrote:
| As they say... when all you have is a hammer, every problem
| starts looking like a nail.
|
| Nowadays, software engineers barely finish learning basics of
| their first programming language and they jump in on scaling
| their first applications they developed while following a
| tutorial.
|
| It is always better to first exhaust other options like improving
| basic efficiency and performance of your applications. A single
| server can do hundreds of thousands or even millions of
| transactions per second. I have many times seen vast farms of
| servers doing each at most hundreds or thousands of very simple
| transactions per second.
|
| A problem is almost always easier to solve and faster to deliver
| when it is only expected to run on a single server.
|
| And don't start arguing on the need of having multiple servers
| for redundancy. Properly automated, new service node can spawn
| within seconds -- maybe couple of your users will see a slight
| hiccup if you handle it correctly.
|
| The goal here is not to say those practices are bad. What I am
| trying to say is that engineering is about understanding
| tradeoffs and making _informed_ decisions.
|
| --
|
| I worked for a large bank which had internal automation platform.
| It consisted of about 140 "microservices" (apostrophes
| appropriate here...), 20 servers running those services and about
| 40 developers.
|
| When I left 3 years later we had 3 servers (bank required hot
| backups, one within same primary DC and one in secondary DC),
| just one monolithic app and only 5 developers.
|
| Our job was easier and we were happier than ever before.
| Reliability improved vastly. And our internal clients started
| getting the functionality they needed.
|
| Previously, any service call from a client had to jump through
| multiple layers of "microservices". Vast majority of the
| application was devoted to communication between nodes, error
| handling, recovery from strange situations, etc. Once everything
| rolled into a single service those vanished because there were no
| longer network calls but method invacations.
|
| And we no longer had to release 140 components and maintain
| versioning and dependencies between all of them. We only had one
| component to take core of.
|
| I made a "new component checklist". Whenever somebody wanted to
| spawn a new component they had to first go through team review
| and prove the requirement would benefit from being separate
| component rather than part of existing monolith. No new component
| was ever created after this was instituted.
|
| Yep, they def did not need microservices...
| tonetheman wrote:
| premature scaling is the root of a lot of wasted time
|
| That said if you know where boundaries are in your design you
| should keep those obvious and clear so that scaling will not be
| hard if you are lucky enough to need to scale.
| FpUser wrote:
| Scalability "requirements" are often misused by the developers so
| they get to play with "cool" and often very slow technologies at
| the owner's expense. I've seen it happen over and over.
| dilyevsky wrote:
| Many experienced devs have ptsd from working themselves into a
| corner on unscalable stack which required death marches to
| resolve. Not surprisingly they often do a complete 180 after
| that. A strong cto / senior eng core would prevent the company
| from going to either extremes but unfortunately not many of
| them have that.
| fsociety wrote:
| I am sure there is truth to this, but from my personal
| experience the death marches I have seen have come from
| overly complex designs which try to handle all future
| problems but then fail miserably. Whereas evolving a simple
| design to be more scalable often works out well.
|
| I am sure there are many situations this isn't true, but I
| have not seen them in my experience.
| ARandomerDude wrote:
| This also becomes a self-creating problem where devs are afraid
| not to have played with the cool tech, lest they fail to have
| the golden resume.
| JohnBooty wrote:
| Yeah. Resume-driven development. "Hey, we need k8s and/or huge
| AWS clusters because scalability" when often something simpler
| would suffice.
|
| At a previous position our devops guy had provisioned a cluster
| of god knows how many AWS Redis instances. I pointed out this
| was a lot more than we needed. He swore up and down we needed
| them _because scalability._
|
| He had no idea how much data was in those things. LESS THAN A
| MEGABYTE of data. In the whole cluster. We had no plans to grow
| it, either. And we were only accessing it a few times per day.
| We didn't even need Redis. We could have just stored it in the
| database directly.
|
| He didn't know any of that, and he didn't care. Because it
| didn't matter. Resume-driven development.
|
| He wasn't stupid. He was very smart. He knew what he was doing.
| "Managed a Redis cluster" looks good on a resume. "Just used
| the database instead" does not.
| quadrifoliate wrote:
| > He wasn't stupid. He was very smart. He knew what he was
| doing. "Managed a Redis cluster" looks good on a resume.
| "Just used the database instead" does not.
|
| Did your company fire him and replace him with a DBA instead?
| If not, then the resume driven development is working!
|
| I hate it when people look down on resume-driven development
| while simultaneously also doing things that promote it by
| letting HR people unrelated to technology screen resumes for
| recruiting.
|
| Here's a hint -- if your company is screening, even
| subconsciously for buzzword (and believe me, most of them do)
| then you are enabling resume-driven development whether you
| like it or not.
| tester756 wrote:
| >He wasn't stupid. He was very smart. He knew what he was
| doing. "Managed a Redis cluster" looks good on a resume.
| "Just used the database instead" does not.
|
| Why not just lie that you did that?
|
| How employer would check that, lol.
| morelisp wrote:
| Then these guys dance off from that to a company that
| actually needs to store tens of billions of k/v pairs, HA,
| and can make a big mess there before anyone figures out
| they're actually clueless beyond `sed s/replicas: 3/replicas:
| 69/ && docker-compose up`.
| ec109685 wrote:
| Another example is how casinos treat their best customers. You
| get an account representative, personal phone number you can text
| message to.
|
| Slack still answers every customer feedback personally which is
| endearing and encourages more ideas.
| jsz0 wrote:
| If you create something so great that demand is off the charts
| chances are your users will stick around long enough for you to
| fix your scalability problems.
| mmcnl wrote:
| Also, scaling issues are not a problem, they're milestones that
| ought to be celebrated. The things you are building are being
| used. You get to solve known unknowns, which are so much better
| than unknown unknowns. The reward is clear. What's not to love?
___________________________________________________________________
(page generated 2023-02-04 23:00 UTC)