[HN Gopher] What software engineers can learn from the rapid col... ___________________________________________________________________ What software engineers can learn from the rapid collapse of Fast Author : gregdoesit Score : 257 points Date : 2022-04-07 17:23 UTC (5 hours ago) (HTM) web link (newsletter.pragmaticengineer.com) (TXT) w3m dump (newsletter.pragmaticengineer.com) | manesioz wrote: | > I found some unsettling information about the founder of Fast, | Dominic Holland. | | This thread [1] touches on what some of that stuff probably was. | Sounds like the CEO was a charismatic scammer. | | [1]: https://nitter.net/jack_raines/status/1511737489190494208 | trollski wrote: | gowld wrote: | > The mock-up of the spreadsheet people who received offers at | Fast had access to. The numbers represent what a senior engineers | with $220K in base salary, and 30,000 options saw as numbers on | their potential compensation value. | | The spreadsheet didn't even have a row for "might not be a huge | success". | | This company was red flags from the start. I can't imagine | someone walking away from a $300+K FAANG job for $240K + obvious | BS equity. | diiaann wrote: | Not having a big company signed is not inherently bad. Going | after small business can be a valid strategy (i.e. Salesforce) | but it sounds like there wasn't healthy growth or the right | relationships to support this path. | everybodyknows wrote: | Consider this: | | > ... engineering directly raising concerns to the CEO, and | suggesting to focus on larger customers, _fewer customizations_ | , and bring in more revenue. Sales, however, wanted the | opposite: close many deals and hit their targets of signups. In | the end, sales got their way, ... | | If the product is customized for nearly every customer, they've | degenerated to a de facto consulting shop. | lupire wrote: | > degenerated to a de facto consulting shop. | | Which is fine, as long it's priced right. Palantir is a | consulting shop. | photochemsyn wrote: | Wow, these numbers are like those from pets.com: | | > "During its first fiscal year (February to September 1999) | Pets.com earned $619,000 in revenue, and spent $11.8 million on | advertising." (Wikipedia) | | > "The company raised $82.5 million in a February 2000 IPO but | filed for bankruptcy nine months later." (Investopedia) | | Fast rasied $102 million in capital and had $600k in revenue that | year... Is Big Finance about to hit the panic button again? | nly wrote: | $100M is a lot less money today than it was in 2000 after | factoring in inflation and low rates. | tomrod wrote: | Yes, but also no. | | Per the CPI, for every $1 in 2000, you need $1.68 today. | 1.68^1/22 - 1 is about 2.39%/year inflation. | | https://data.bls.gov/cgi- | bin/cpicalc.pl?cost1=1&year1=200001... | propter_hoc wrote: | For the purposes of engineering salaries, CPI is not an | accurate representation of the change in what $100M can | buy. | [deleted] | cush wrote: | But the revenue is also inflated, so it all works out in the | wash. | whymauri wrote: | So Pets.com actually drove more revenue than Fast?! Oh my | god. | pphysch wrote: | So is 600k | ben7799 wrote: | This does seem like a 1990s story. | | I'm just barely young enough to have missed it. Friends who | quit school early were in on it. | | I don't know if this was the case at Fast but in a lot of cases | back in the 1990s a lot of the engineers knew everything was | screwed up.. and they just didn't care, the money was good | while it lasted. | | The experience almost always helped people get ahead, no one | ever gets punished for doing good work at a company that fails | due to bad management and investor decisions. | adamsmith143 wrote: | That Startups are fundamentally risky and despite all the talk on | HN you are highly likely to spend years working in a high pace | high stress environment for equity that will ultimately be | worthless or at best match comp at FAANGs all while having | questionable WLB. | nemo44x wrote: | Over a career though you'll be at quite a few of them. Figure | 7-10 or more in some cases. Some will be quick fails (within a | year you know it's just not going to work and you move on) and | others will be OK success (maybe you have an OK exit with | equity but not life changing) over 4 years of service and maybe | 1 or possibly 2 will be really nice. And if you're really lucky | one will be beyond amazing. | AYBABTME wrote: | But working at FAANG is not much fun. I think a mix of both | startups and Big Tech leads to a happy life. | lupire wrote: | What's fun about working at startup that is just cloning a | bad old Amazon feature? Besides getting to call yourself a | "staff+" on Twitter, I mean. | adamsmith143 wrote: | The meteoric title inflation is definitely interesting. I | consistently see people who were Analyst level at Fortune | 500s getting up to Director of X in a year or two at | startups. Seriously makes me question the quality of people | at these places if any random analyst is qualified to be a | manager of managers. | adamsmith143 wrote: | Well that's definitely subjective but in terms of Pay and WLB | I think big tech beats startups generally | ChasingEchoes wrote: | i guess its up to what everyone is looking for. | | I personally avoid start-ups like the plague. To the point | where i dont even bother accepting linkedin requests from | startup people | | Im not that big of a fan of "Big Tech" either, i wouldnt go | for something like FAANG, even if maybe the pay would be good | | What gave me a happy life was "industry" firms. In my case | those were either consultancy/strategy firms (MBB/Big4 ) at | first, which was stressful, but had good advancement chances | (jumping steps based on performance) and very good networking | opportunities. | | Now i've retreated in "an industry". In my case a forbes 100 | company in the automotive sector (they have cloud systems and | develop software too). And life is bliss. | [deleted] | mxuribe wrote: | Oh, did you mean for that to rhyme? Shorten it a bit, and folks | would remember it more easily. ;-) | lifeisstillgood wrote: | Incentives matter. Aligning everyone's correctly matters. So much | of the story sounded "yeah but you can survive that" right up to | the point here: | | >>> Sales, however, wanted the opposite: close many deals and hit | their targets of signups | | If your salespeople are focused on selling to the wrong people | nothing matters. You are either Shopify taking years to build or | you are a rocket ship taking shortcuts - decide | edent wrote: | Can someone explain what Fast's USP was? | | Every ecommerce platform I've used has a PayPal / Google Pay / | Klarna button. | | I click it and my payment & shipping details are there. Seamless. | | So what problem was Fast trying to solve? | bprater wrote: | If you want to use a traditional credit card (inside of Paypal | or GPay), your options are currently limited. Shopify has the | best implementation of this tech with ShopPay ... if you pay at | any Shopify merchant, you have the option for Shopify to | remember your digits. This streamlines checkout to a click or | two. That's the big USP of models like this - speed of | checkout. | dnissley wrote: | How are options limited by PayPal and GPay? What is a | "traditional credit card"? | tough wrote: | New klarna virtual cards for not needing ecommerce implementing | financing directly is genius. I guess with nowadays infra as | code for financing that makes sense in 2022 | toomuchtodo wrote: | Affirm is doing the same [1]. Debit rails are just easier | versus the integration schlep. If you're a fintech, you can | do all sorts of cool presentation and product offerings using | a virtual account attached to a deposit account (BNPL is one, | as is a hybrid credit/deposit account). Debit/virtual card | also empowers the consumer, as you're not beholden to the | merchant or their gateway provider to support a specific | fintech integration. | | [1] https://www.affirm.com/debit | k__ wrote: | I first heard of it today. | miketery wrote: | USP? | | Edit: found it, unique selling proposition. | | And totally agreed, how do you compete with google pay, Apple | Pay, Instagram, etc. it's a losing game when you don't have | network effect. | Apocryphon wrote: | It's in the same category as whatever Bolt is doing. | Animats wrote: | _" Most engineers joining didn't know much about why the one- | click checkout industry has the potential of billions."_ | | So why isn't this a standard feature of every shopping cart | program, including the cheap ones? The complicated part is that | you need "undo", valid for a while after ordering. That's what | makes one-click buy feel safe for customers. This complicates | inventory management. But you really need "undo" for an hour or | so after ordering. | rileyphone wrote: | Amazon had the patent but it expired recently AFAIK. | jollybean wrote: | That's the magic question. | | 1) Carts are slow to evolve but they will. | | 2) Bolt/Fast contain user based information, consumed from | their use on other sites. So there's a 'network effect'. If | they've shopped at ABC.com before your store, then you already | have their CC data ready to go in their car. Sort of like | Single Sign On but for carts. | Animats wrote: | _Bolt /Fast contain user based information, consumed from | their use on other sites. So there's a 'network effect'. If | they've shopped at ABC.com before your store, then you | already have their CC data ready to go in their car. Sort of | like Single Sign On but for carts._ | | Uh oh. That has so much scam potential. | odonnellryan wrote: | Lots of sites do not have a good undo for orders. I ordered | from remarkable and their solution was to deny the package... | cush wrote: | I ordered a piece of furniture from Wayfair, and they had the | same response - deny the package. | | Well the package arrived while I was at work. After calling | them 3 times to come pick it up, due to shipping delays they | were never able to, and eventually told me to keep it for | free. | why-el wrote: | This connects nicely with Dan's article from yesterday [1], if | only tangentially. I think a good habit is to instill a mindset | whereby there is a single metric for cost, possibly associated | with your main product, and keep track of that cost as it relates | to your infrastructure spending. I've seen it before work well. | For instance, if you sell computers, the cost could be, we spend | $100 on infrastructure per computer sold, and engineers can then | argue for spending more effectively. | | [1] https://news.ycombinator.com/item?id=30936189 | sciurus wrote: | Will Larson had a good post on this recently- | https://infraeng.dev/efficiency/ | why-el wrote: | Wonderful, thanks for sharing. | cush wrote: | I have Fast recruiter emails in my inbox from literally last | week... | eweise wrote: | I don't understand why $600K in revenue wasn't a giant red flag. | Maybe they don't tell the employees the revenue numbers? If so, | that is a giant red flag. | tschellenbach wrote: | I tend to tell potential new hires about our revenue, funding | raised, valuation etc. You wonder if there needs to be more | education on equity or if these people just went for the high | base salary. | tschellenbach wrote: | Compare revenue to funding raise/valuation. If those numbers are | out of whack just be aware that your equity is probably not going | to have a good outcome. | d3ntb3ev1l wrote: | con man as CEO is usually a good indicator | lmeyerov wrote: | Oof, just one of our customers pays about the same as their full | revenue, and we're a cockroach team. | | The real thing here is they're not that surprising. A lot of | companies with their kind of funding use enterprise sales to | force say $5-10M ARR, but there's only so much VC money can force | for a leaky funnel, broken product, and overall incorrect market | + fit. I didn't appreciate this until maybe a year or two ago. | Valuation multiples in 50-100X range are super common (and even | wackier numbers in seed/a). Think make believe stories like "well | with another 12-18mo of growth this really just a bit over a 30X | on some future forward revenue multiple...". The cash almost | always leads to overspending, and it's highly unlikely the next | 2-3 raises won't blow up and everyone goes home. I'm actually | super impressed by the Docker team because they've been one of | those rare cases of crawling out of that trap, even if with a lot | less of the team. | | We get job candidates with high competing offers for companies I | know to be rotten inside, yet there's only so much I can say. | "Our new hires are getting paid from customer revenue and with | equity that doesn't have $50M-$500M of investor thumbs on the | scales already cutting you out in 95% of the likely scenarios" | generally doesn't punch through the kool aid. | asadlionpk wrote: | > we're a cockroach team | | is this a reference to | http://paulgraham.com/guidetoinvestors.html#:~:text=nuclear%... | pelagicAustral wrote: | I should be a part of a cockroach team, in my estimation. | mwcampbell wrote: | I'm not familiar with the phrase "cockroach team" in this | context, and the obvious web search didn't help. Can you please | elaborate? | alex_c wrote: | I'm not the op and this is the first time I've seen someone | else use the term, but I've used it too. Aspirationally - | small and impossible to kill. While a unicorn's goal might be | growth at all costs (with associated high chance of failure), | our goal is to survive anything that might get thrown our | way. | jollybean wrote: | small, surviving on crumbs. | lmeyerov wrote: | Hard to kill because they're financially independent. We | believe enough in the technology we're building (gpu-powered | graph ai & visualization) and the missions we're powering | (security, fraud, anti-misinfo, healthcare, etc.) that we | turned down large initial funding as it would have broken us. | So awhile back, being cockroaches meant running as cost- | effectively as we could (and still do, in fact, just not as | extreme) as we figured it out, and now that things are | working, it means prioritizing sustainable growth vs | financial games likely to fail. | | In contrast, popular models prone to failure and largely | fueled by market phenomena like low inflation rates include | ad-based social media ("we'll turn on revenue in 5 years, no, | really!"), blitzscaling ("we'll eventually figure it out!"), | sales-driven growth ("our federal team will make our $ back | in 3-5 years, no really!"), acquihires ("with enough AI | talent we'll just sell our staff!"). | | I like venture capital, such as for deep tech and true | growth, but the reality is most tech VC funding is more about | financial engineering that assumes most of the portfolio | fails. They push for commercial scaling too early and wipe | out, which is fine for the rich VC who just needs 1-3 bets to | win. Fine for them, but sucks for most founders & employees | unless they job hop every 2 years. If we take VC $ again, | it'd be on our terms and with a clear payback/growth return. | | A great way to kill an otherwise great idea & company is | putting VC $ in, which puts a company on overdrive and | financially implodes before it has a chance to figure things | out. They're addicted and likely can't stop relying on VC | without layoffs (or less likely, succeeding), at which point | many of the A players leave as well and hard to recover. | Instead, if a cockroach raises > $2M (so commercialization | phase), that should generate enough sales revenue that they | can keep organically growing in 18-24mo later even if a | follow-on round doesn't happen. You'll hear such cockroach | founders saying they don't even touch the money for months. | Numbers-wise, most Series A companies fizzle out, which is | because of this unicorn-or-bust structure.. and especially | whenever the market is not on a bull tear. With a bit more | time, they probably could have figured it out... but they | blew it. | | Some articles: - Paul Graham's article on this: | http://www.paulgraham.com/badeconomy.html . - | https://medium.com/swlh/how-to-build-a-cockroach-startup- | ins... - https://medium.com/the-mission/be-a-cockroach-not-a- | unicorn-... | diehunde wrote: | Maybe they work at Cockroach Labs | altdataseller wrote: | Cockroaches just care about survival first and foremost. | Unlike unicorns that care about growing huge and taking over | the world | Jemaclus wrote: | > we're a cockroach team | | I think I know what you mean by this, but I'm not sure. Could | you elaborate on what this means? Thanks in advance. :) | munificent wrote: | I infer that it means "small and hard to kill". | PragmaticPulp wrote: | > We get job candidates with high competing offers for | companies I know to be rotten inside, | | I fell for this trick once when I was younger, but never again. | | As it turns out, when companies are bleeding talent and | struggling to hire they suddenly find ways to pay well above | market rate. You can have a fancy title, too! | | We had a competitor do something similar at a prior company. I | would tell candidates that we can't match their compensation | numbers but to come back if things don't work out at the other | company. I'd often get a message about 3-4 months later asking | if the job was still available | meetups323 wrote: | May help explain some Meta offers I've seen... 2 years out of | college? Sure, come as a Senior! That's not enough? Principal | it is! | naiwenwt wrote: | The problem is sometimes this is done for legitimate | reasons and genuinely does lead to stronger outcomes for | both employers and employees. | | It's hard to tell the difference between "stop the | bleeding" title/pay inflation and "aggressively competing | for talent" inflation. | meetups323 wrote: | Differentiator is what other companies will pay. If GAM | all offer X, F offers 3X, and GAM refuse to budge on | their X... either F knows some deep dark secrets about | your capabilities which no one else can appreciate, or | they're bleeding talent. | willsi wrote: | That was fast... | twic wrote: | Informative but depressing looking at the salaries given in the | job listings at the end. | IMTDb wrote: | The key informations are here : | | > Fast offered $200-240K/year in base salaries with full remote | work | | > Sign-on bonuses were common and large. Those who asked for | almost always received sign-on bonuses of $20-50K as a one-off | payment | | > Equity issued was lavish and presented as potentially life- | changing | | > A good part of people are echoing how working at Fast was an | amazing experience. People liked the culture, and how employee | happiness was a priority. | | What's not to like here ? Working for huge amount of money, in a | company whose culture puts employees happiness first, and | provides an amazing working experience. | | Management did "all the right things" : remote ? check ! | competitive salaries ? check ! amazing experience working there ? | check ! employees-first culture ? check !. Why would you even | _need_ to look elsewhere. | | And when it all crashed and burned, the feeling is : | | > There are people who are frustrated and disappointed with | company leadership, and how the bust came out of nowhere. | | Trick is that it did not come out of nowhere, people just chose | to look away. It's easier to blame management (which is | definitely at fault as well !) than to say "I was paid way too | much compared to the actual value I was providing". When you are | paid $200k / year, you should _easily_ be able to justify that | your work generates more than several thousands of USD _alone_ , | if you are unable to do that, you need to have a conversation | with your mirror as well - or just accept the fact that you are | benefitting from a system, which may crash and burn if too many | people are in this situation, or keep on living if you are an | exception. | deltarholamda wrote: | >When you are paid $200k / year, you should easily be able to | justify that your work generates more than several thousands of | USD alone | | Revenue per Employee used to be a real metric that people used. | Of course, P/E ratios used to be sane as well. | paxys wrote: | Is there anything actually new to learn here? Yes all equity is | hypothetical. Yes there is no guaranteed road to an IPO. Yes you | are taking a risk. People who take jobs at these kinds of | companies for the upside do (or at least should) know this. | lupire wrote: | Don't work for a con artist with a long rap sheet. | me_me_mu_mu wrote: | i guess everything was fast | tlogan wrote: | I never heard about Fast - till now. So my opinion is that they | failed in advertising. Maybe they spent too much on engineering | but definitely not enough on advertising. | moffkalast wrote: | They died like they lived - Fast. | bogwog wrote: | When I read the title, I thought this article was about | Netflix's speed test website fast.com, and thought they were | having some catastrophic outage or something. | davidkuennen wrote: | Holy shit. How do you even spend 10M/Month? | danesparza wrote: | It can be done. It's expensive. But it can be done. | waqf wrote: | You have 500 people and you spend 20k/month to employ each of | them. | 0xJRS wrote: | It seems very hard to do but I worked at a startup in the early | 2010s and we were spending >500k/m. We had ~25 employees, 7 of | them execs, all buying new homes and cars, meanwhile we had 1 | single paying customer bringing in 10k/m. I told some of my | close colleagues that we probably had 6-12 months left when I | put in my resignation. | | 12 months later they laid off 10 of the 12 remaining engineers. | shagie wrote: | > For senior software engineers, Fast offered $200-240K/year in | base salaries | | Lets put everyone there. $20k/month. | | > On Monday, 4th April, Fast laid off all of its workforce of | about 450 employees, of which about 150 were software | engineers. | | That 150 at $20k/month is $3M by itself. | | The other 300... if they were paid half of that would easily be | another $3M. | | I'm certain there are other costs - but its easy to point to a | "if they were paying that much, this many employees represents | this much per month" that is a sizable fraction of that | $10M/month. | [deleted] | lordnacho wrote: | Makes no sense at all. Presumably whoever is allowing people to | be hired knows what the revenue figures are? | | I suppose it's possible they thought there was some sort of dam | holding back business, which was about to burst, thus requiring | loads of staff to deal with. | | But most people would just say "we'll cross that bridge when we | get there" and allow a bit of queuing up of customers, rather | than somehow hiring and training a bunch of people in | anticipation. | throwawayboise wrote: | This sounds very much like my experience with a dot-com flameout | in 1999. | | Massive spending of VC investment money on offices, a sales team, | engineers, and a data center and servers to meet their expected | (hoped-for) scale (there was no cloud then). | | No large clients on the platform. Not a lot of small clients | either, TBH. | | One day out of the blue over half the company was let go. Some | effort to downsize and retool, but it seemed perfunctory. All the | rest of the staff was let go after another couple of months. | | If you have no customers, you have no business. | taylodl wrote: | An important lesson I learned while working for a startup - pay | attention to the revenue stream and pay attention to expenses. | The two will not stay out of alignment for long. I've talked to | start-ups not paying any attention whatsoever to their revenue | stream, they seem to think the bonds are the revenue stream. Many | of those startups are quickly gone. When looking at a start-up | make sure they have a good grasp of their revenue, their | expenses, their revenue forecasts, etc. If they start hand-waving | or show numbers that are out of whack then you're better off | passing on their "opportunity." | buf wrote: | Wow, I make the same revenue as Fast as a one-man entrepreneur. | | It really amazes me sometimes the strategies of these heavily | funded companies. Why pile on so much burn so quickly? | kache_ wrote: | Time to ring up some VCs, buf xD | buf wrote: | And end up like Fast?! | sydthrowaway wrote: | What do you do? | sodality2 wrote: | Their profile contains a website with some companies that are | probably related. | coldcode wrote: | I worked at a consulting firm in the late 90's (right before | Dotcom collapse) and we got a new CFO who bragged we were all | going to millionaires shortly. Most of us engineers thought he | was nuts as consulting firms are not exactly money engines. We | survived past the March cliff in 2001 but in mid summer we died | at 4:30PM with 20 minutes notice to leave the office before the | locks were changed. I never trusted a CFO new hire again... | sydthrowaway wrote: | If I was a FAANG hiring manager, I'd tear up any resume from | these get rich quick dollar chaser wannabes | lbrito wrote: | "Inside Fast's Rapid Collapse" | | Can't believe they missed the chance to headline a pun with the | company's name | | "That was one Fast collapse" | hidelooktropic wrote: | I'd assume it was on purpose, given the obviousness of the pun. | mwcampbell wrote: | I think the most applicable warning for engineers when it comes | to the actual work, as opposed to whether one should join a | particular startup, is this: | | > Engineers calculated the load Fast had in needing to serve | their traffic. The Fast button was rendered less than 500,000 | times per day - rarely needed to ever serve more than a few | requests per second. | | > One of the few warning signs engineers noticed is how Fast | spent far more on infrastructure than the scale of the operation | would have called for. Engineers sometimes brought up suggestions | to scale infra down, and save costs - given there was not much | revenue generated. | | Sounds like the whole thing could have run on a single cheap VM, | perhaps with a second one for redundancy. | rockostrich wrote: | This is pretty hilarious to read. | | I did an interview screen with them at the end of last year for | a data engineering position (former coworker was high up on the | design side of things and he gave a referral). The first bit of | the technical question was just "write a SQL query" and boiled | down to doing a group by, count, and limit. The second was | something like "what if we needed to return this query from a | distributed system without going to the database?" without much | context. | | As any sane interviewer would do, I asked for context around | scale, infrastructure, and what "without going to the database" | even meant. The interviewer just reiterated the question. I | started talking about different approaches depending on context | and all the interviewer did was tell me to program a solution. | Turns out all he wanted was a time-windowed hash-map to cache | values and the fact that it's a distributed system didn't | matter at all. | | I think that just about sums up their approach to engineering | solutions. | fourseventy wrote: | I run a company in the e-commerce tech space. We handle ~1M | requests per day and run it all on a few medium sized VMs on | Google cloud for like $1k/mo. | whymauri wrote: | time to raise a $100M and go w e b s c a l e | EinsDueTresFour wrote: | I would add that the "Sales vs engineering and sales winning" | would be a very relevant warning also: | | > [sales] signed up a large number of smaller businesses on the | platform. [...] However, integrating these smaller businesses | was challenging thanks to several customizations needed for | each new customer. | | The ratio of revenue per each small customer vs the total cost | of integration (and very likely on-going maintenance) was | probably at least an amber flag somewhere for those who had | visibility of it. | | But maybe there was on-going hope that they would be able to | sign up a big customer and integrate them before they ran out | of money? | lumost wrote: | At some point managers often fear the "red flag" of "we have | a giant team with no work/customers". Hence, they will often | push for dubious consulting(ish) work to at least ensure | everyone remains busy. | | Unfortunately this can lead to the outcome that the company | just loses more money. | ben7799 wrote: | Management probably really screwed this up if they were | really only making $600k/yr. | | Places I've worked yearly contracts for B2B have been in the | $100k-$5M/yr range. | | The customers paying $100k/yr barely get any integrations and | feature changes... they go for the ride. The customers paying | $1M+/yr get features on the double. | | If Fast was doing integrations and customization for | customers that were paying $1000 or less a year something was | very very wrong. | vosper wrote: | > The ratio of revenue per each small customer vs the total | cost of integration (and very likely on-going maintenance) | was probably at least an amber flag somewhere for those who | had visibility of it. | | At the last company I worked at (won't call them a startup, | just a 10-year old small business that somehow kept raising | funding but has never made a profit) I was astonished to | learn from a new CPO that in our tenth year we didn't know if | we were making money from a customer. | | We had no idea how individual customers used the application, | which was very data intensive, or what that was costing us. | The information was there, but no-one ever looked. Every sale | was considered a success, even if it turned out to cost us 3x | in data costs and customer support. Often customer support | alone would make a sale into a loss. | | We spent a relatively large portion of our revenue running | very expensive servers to respond instantly to searches that | no-one ever performed. I created detailed monitoring to show | this. I talked about them to everyone, including Product and | the CEO. No-one disputed those facts. But instead we had | several people essentially dedicated to downsizing | application servers and deleting unused S3 buckets, trimming | three zeros a month when we could have trimmed five. | | I have learned not to assume that warnings are visible, that | people pay attention to them if they are visible, or that | anyone cares. | | They're still circling the drain, still raising money | (somehow!) and I doubt anything's changed. | mywittyname wrote: | > We had no idea how individual customers used the | application, which was very data intensive, or what that | was costing us. | | Cost can be a difficult problem to solve! I've been working | on a project for about six months trying to identify just | how much it costs to deliver a unit of the thing we sell. | There's just so much variability, one customer might have | widgets that are 40kb in size and rarely ever run widget | analytics workloads, while another customer has 40mb | widgets and can't stop looking at them. | | I feel for the whole "how do customers use the application" | thing too. Product analytics at most places seems to be a | concern long after features are developed. | | "So how many customers have been using that feature we | spent two years and $20MM developing?" | | "Good question." | | "Soooo, mywittyname, can you figure out how many people at | companies with over 200 seats look at dog pictures after | opening an email with a 'C' in the title?" | truffdog wrote: | > We spent a relatively large portion of our revenue | running very expensive servers to respond instantly to | searches that no-one ever performed. | | The last 7 years of my life as an SRE. It's maddening. | vosper wrote: | We also had multiple contractors working full-time for | months on a CI project that _was projected before it | began_ to save us at most a few thousand dollars a year. | In the end they half-delivered a poorly-built system | everyone hated, and I forced the abandonment of the | project and we went back to the old system. | | It was the pet project of another team-lead, that no-one | else wanted. Group decision making is bizarre. | mylons wrote: | this sounds like Ridgeline, a startup most have never heard of, | founded by the founder of Workday and some of its early | employees. | hinkley wrote: | I would hope this problem doesn't exist at startups but I've | seen it at entrenched institutions any number of time: | | If your boss's boss or peers signed a contract for $1m a year | from Oracle, you're going to use Oracle even if Postgres is a | better product. And the only way to use Postgres in this | situation is if they don't know you're running it. Which puts | you in the uncomfortable situation of wanting to say, "But | Oracle sucks, look how successful we've been at using | Postgres?". | | Basically if you don't already know that someone higher up the | food chain is comfortable with saying, "I've made a huge | mistake", you aren't going to 'help' them learn to do it by | pointing out they fucked up. What they're going to hear is that | you're asking them to say, "I just wasted a million dollars of | company money on something everybody hates," which not only are | they not going to do, but they're going to start thinking about | how to shoot the messenger. | | As many conversations go in a couple of my hobbies and talking | to friends and acquaintances about health concerns, the answer | often starts with, "first, get a time machine" and ends with, | "or learn to live with it," often with some useful compromises | in between. | | One of the important things to do early in your tenure at a | place is to bid various people in the management chain and | those who seem to be on a management track to see how open they | are to having their minds changed. Because you want to know | this when the stakes are low (also the blowback on low stakes | things seems to be less problematic). Then you know if or when | a boondoggle is in the making whether you can stop it before it | starts, or amplify all of the problems and kill it along the | way, or whether you should keep your head down and conspire to | keep the 'doggle at arm's length with those who agree with you. | zozbot234 wrote: | > If your boss's boss or peers signed a contract for $1m a | year from Oracle, you're going to use Oracle even if Postgres | is a better product. | | What you'd do in that situation is make the best of that $1M | contract you're stuck with anyway, while being flexible | enough to migrate away from that solution when it actually | makes sense to do so. Just because you've made a huge mistake | doesn't mean that making a different choice after-the-fact is | more sensible. Hindsight is always 20/20. | lolsoftware wrote: | The TailScale folks caught a bunch of flak for their "do things | that don't scale" approach to databases. But, honestly, most | startups would be better off following that approach than what | Fast did. Sounds like the engineers were just entertaining | themselves with shiny toys rather than solving the problems | they actually had. | xen0 wrote: | I wonder, how many engineers had full time jobs just | developing and maintaining the various integrations with all | their small clients? | mywittyname wrote: | I have to imagine they were all stepping on each other | and/or reinventing slightly different shaped wheels too. | | My experience has been that companies who will customize | integrations for you have poorly designed products. By | that, I mean, when we ask for a feature, and they bring on | an engineer who does some requirements gathering then comes | back with some invisible solution. That's a major redflag. | | The success of an integration, in my experience, is | directly proportional to the amount of the work I'm able to | handle as a client. | vimwizard wrote: | >My experience has been that companies who will customize | integrations for you have poorly designed products. By | that, I mean, when we ask for a feature, and they bring | on an engineer who does some requirements gathering then | comes back with some invisible solution. That's a major | redflag. | | I'm feeling a bit called out right now (not an owner but | that's my experience, too) | zozbot234 wrote: | Tailscale's "database from 2022" is neither here nor there. | What you should do in this situation is use the most boring | tech, such as PostgreSQL with a simple HA setup. Thus | avoiding the "play with nifty toys" trap and maximizing the | chance of making appropriate choices. | mwcampbell wrote: | Perhaps one problem with the current backlash against over- | complicated infrastructure is that for some of us, doing | everything in one process on one machine using a local | embedded database is its own form of playing with nifty | toys. | bryanrasmussen wrote: | yeah, I guess, I'm not going to get too specific here but | I do think it would be nifty if I didn't go through a | couple 10 minute periods every day where I have to | basically shut everything down run a number of scripts, | then restart individual processes that are problematic, | to be able to work again. | | Note that this basically works 99% of the time now and is | part of a learning process that had the whole thing | taking 2 - 5 hours every day for several weeks, down to 2 | hours a day for several weeks after that, and now down to | the nearly inescapable 20 - 40 minutes a day I have now. | | Yeah, not doing that would be my own form of playing with | nifty toys. | enra wrote: | > Sounds like the engineers were just entertaining themselves | with shiny toys rather than solving the problems they | actually had. | | This is often the problem when hiring too fast. New employees | don't have context or direction but generally people try to | be productive, so they start inventing work. | | Management team is new too, and not built up either direct or | handle the bandwidth and they also are lacking the direction. | Senior leadership and founders are likely busy hiring more | folks and potentially trying to sort through the chaos | described before. | | IMO, pre-product market fit company shouldn't have 500 | employees, maybe not even 10. These things don't become | easier with more people, usually everything gets harder and | slow. You also shouldn't have massive marketing or sales | force before you actually have product to sell and have | figured out your sales motion. | | It seemed that Fast was just trying to shortcut their way to | be the next Stripe by hiring people instead of actually | working on the fundamentals like the product and business. | jakey_bakey wrote: | I find it absolutely crazy that companies do this. | | My company's strategy is get one extremely solid person | across all our verticals (Android / iOS / Backend / Infra) | to minimise communication overhead and maximise iteration | speed (since we throw out half the code we write anyway). | | I would be very hard pressed to hire more than 2-3 people | in each role even if we were offered 10s of millions. | recuter wrote: | Headcount perversely increases the valuation sometimes. | Hence, big head positions. | ChrisMarshallNY wrote: | _> My company 's strategy is get one extremely solid | person across all our verticals (Android / iOS / Backend | / Infra) to minimise communication overhead and maximise | iteration speed (since we throw out half the code we | write anyway)._ | | but ... but ... _that makes sense_ ... | | I have found that having fewer _good_ engineers is _much_ | better than lots of bad ones. | | That approach is treated as heresy, in today's tech | industry. | mwcampbell wrote: | Perhaps the thinking is that it's easier to hire multiple | good-enough programmers to get dependable productivity, | and to replace one if something goes wrong. In the | extreme case, if a company has one programmer who is very | good when they're working at peak productivity, but then | they have a period of low productivity for whatever | reason, that company is in trouble. I've been that | person. | kevan wrote: | It's not just limited to startups. When my business unit in | Amazon spun up we hired way too fast and ended up having a | harder time shipping v1 of the product than if we would've | started lean and grown organically. We ended up spending a | lot of time on a modularized platform to enable ~5 matrixed | feature teams to contribute across iOS and Android apps. | Looking back, a small dedicated team for iOS and another | team for Android probably could've shipped v1 in less than | half the time with a lot less tech debt. | enra wrote: | Yeah for sure. I think even senior leaders forget the | team culture/context piece, and try to scale up too fast | because they are used to it in their current orgs or | previous companies. | | If you have a 500 person org that has operated for years | you can probably add 10%-20% new people each year without | it breaking too bad. But it can be very hard to build a | complete function from 0 to 100 people within a year | because there is no infra, culture or understanding how | things work. The new people cannot be absorbed/aligned | because there is no central gravity so people easily end | up pulling different directions and spending their time | on internal coordination vs actual useful output. | [deleted] | lmeyerov wrote: | It wasn't "doesn't scale to 10-100X bigger" but "doesn't burn | themselves and their teammates out 6mo from now and | needlessly risk a stressful & prolonged sev0 when one of the | 1-2 people who did weird things gets COVID / goes on a | honeymoon / leaves for netflix". | | Spending time on needless infra (scale you don't need, AI you | don't need, infra you don't need, things could have been | postgres as in this case, ...) both prevents building useful | stuff ($-generating features/experiences/...) and increases | operational cost (maintenance, debugging, ...). Both the lost | revenue growth and the increased costs are _compounding_. | Imagine a maniac steadily piling on a bit more technical debt | every day and eat a bit more of the seed corn every day | instead of growing it. | | A good phrase for this is "playing house": | http://www.paulgraham.com/before.html | [deleted] | nonameiguess wrote: | Staying employed for another few decades without losing | seniority in an industry that expects you to stay current on | all of the latest hottest fads is a problem the engineers | actually had. | wnevets wrote: | > Sounds like the engineers were just entertaining themselves | with shiny toys rather than solving the problems they | actually had. | | Programmers love to add the latest tech to their resume then | move on to a new position within ~18 months | CoastalCoder wrote: | That may be specific to early-mid career focus | | If/when a developer builds deep knowledge of a problem | domain, things like framework-of-the-month may become an | unwelcome distraction. | randmeerkat wrote: | > Sounds like the engineers were just entertaining themselves | with shiny toys rather than solving the problems they | actually had. | | More like, sounds like management was trying to get as much | money as quickly as they could from their VCs before it all | imploded. The engineers aren't at fault that management | didn't have a real product or vision. | mrkurt wrote: | We all know plenty of engineers who want to build things | for inappropriate scale. The company was a disaster, but | part of building a disastrous company is hiring the | engineers who want to make everything webscale and have no | sense of pragmatism. | randmeerkat wrote: | > The company was a disaster, but part of building a | disastrous company is hiring the engineers who want to | make everything webscale and have no sense of pragmatism. | | I mean, the engineers held up their side of the bargain. | They were hired for webscale, the company got webscale... | | Now, if this was a more strategic startup, that had tried | to hire pragmatic engineers, that insisted on webscale, I | would blame the engineers. However, in this case, I think | it's pretty clear that the engineers did exactly what | they were hired to do. | hobs wrote: | Pretty sure the reason they caught that flak is because they | chose to reinvent the wheel instead of do a normal thing that | would work until later - they're on their what - third | rewrite of that now? | | Tootally worth the time /s | [deleted] | lolsoftware wrote: | I don't think that's a fair characterization of what | happened. First, they scaled a simple solution for a long | time and migrated away from it before it exploded | spectacularly. I don't recall them explicitly saying how | much time they spent on it, so how is it possibly to say | whether it was worth the time or not? I also don't remember | if it was here or on Twitter, but one of the engineers | listed the many features they built and shipped _instead_ | messing with their database. I'd say that was a good | business decision - they got more features to their | customers sooner and dealt with a scaling issue before it | impacted customers. | [deleted] | [deleted] | lupire wrote: | > The TailScale folks caught a bunch of flak for their "do | things that don't scale" approach to databases. | | I've seen this repeated on HN, but never saw the alleged | flak. | SkyPuncher wrote: | I've done startup engineering for a while. I've seen exactly | 1 issue that we couldn't solve easily (N+1) or with simply | turning our server count up (often way cheaper than putting | an engineer to a problem). | | Basically, to move development velocity faster we "load | everything". Seems like a really bad idea on the surface. In | practice, we only ran into an issue with a single customer | that was literally 1000x as large as the other customers. It | took about two weeks by 1 engineer to rework a select few | calls that were egregiously slow and we were back at it. | | ---- | | In other words, even with some arguably inefficient technical | decisions, I've rarely if ever seen performance issues at | early stage startups. | Spivak wrote: | And if you think this is undersized for comedic effect I did | ops for a company that went from <1mil to 15mil revenue (with | the same sales strategy of targeting lots of small clients) on | 4 1u white label supermicros. Two app servers and two MySQL | dbs. Never in my life had such reliable infrastructure. | StillBored wrote: | Early in my career when in a discussion with my manager at | the time, he threw out the "planes with two engines have | twice as many failures as single engine planes" line. | | So in my now close to 30 years of experience later, working | on systems expected to run 24/7/365, I can't tell you what | percentage of failures and outages were caused by the | redundancy/fail-over/etc software and hardware layered into a | system, but its definitely a very significant part of the | problem. Frequently because its just that a "layer" someone | bolted on, even if that layer was some software engineer | designing for "web scale". All the edge cases are frequently | not completely thought out, and when you hit one its a lot | harder to recover, than simply restarting a single service | running on a single server. | amluto wrote: | I once managed a thoroughly non-critical server that lived | in a closet. In the couple years I managed it, we had quite | a few failures due to the UPS tripping and no unscheduled | downtime at all from any other reason. | | I've had massive problems with fancy (HPE, IIRC, but it | wasn't called HPE then) raid arrays caused by the | controller being a POS. Using plain mdraid would have been | far more reliable. | | Sometimes I wonder if all the effort people put into | protocols that run Paxos and Raft could be better spent | running a small number of coordinator servers with manual | failover. I'm suspicious that, in many use cases, leader | elections cause failures more often than the leaders | themselves fail. | jjav wrote: | This is a big lesson to heed. Sure, sometimes a startup | really needs extravagantly scalable infrastructure. For the | 99.9% of startup, no, you don't. | | So the odds are you don't. Keep it very simple, very small. I | can't highlight this enough. A few boxes will get you very | far. If you ever actually hit the limits, throw a big party | to celebrate success and only then start to think about | making your infrastructure a bit (just a bit) more complex. | jonas21 wrote: | It would not have made a difference. Maybe they'd be losing | $9.9M a month instead of $10M a month. | whateveracct wrote: | Heh maybe if they also didn't hire a bunch of engineers to | scale out too, it would've helped.. | mwcampbell wrote: | Sure, their primary problem was extravagant hiring, but I | think this is still an important warning for developers | working in fledgling companies that are appropriately | cautious on the hiring side. | mcphage wrote: | > Sure, their primary problem was extravagant hiring | | I think their primary problem was a huge valuation--with | correspondingly huge expectations--but no real revenue. If | you're pulling in $600K revenue, but valued at $500M... | you're gonna have a bad time. | PragmaticPulp wrote: | > Tiny daily sales numbers which all employees received was the | first such warning sign. Internally, Fast was transparent on | sales. Every day, every employee would receive a sales summary | email that listed the number of sales completed with Fast | checkout, and the total sales amount. | | > Fast did less than $300K worth of sales and below $6K in | revenue on most days from January 2022 to April 2022. There were | days with around $2,000 in revenue for Fast. | | I'm actually surprised that everyone was receiving daily updates | of company revenue. | | If you're surrounded by 100s of people at a company known to give | high base pay but you're seeing daily revenue numbers in the | range of $1K to $6K (they had $600K total revenue in 2021, | supposedly) then you have to know that your time is very limited. | | I assume they were led to believe that more investment money was | just around the corner to keep the business going? With a $10 | million monthly burn rate they would have needed a staggering | amount of capital to just continue to exist, let alone execute | any plans to turn the ship around. | meetups323 wrote: | The entire culture of b2b startups IME is "yes we're burning | money now, but just you wait till Moby Dick comes along... just | implement XYZ features marketing/sales says are important and | we'll have him in no time!" | | Of course, this is somewhat tautological: if they weren't | burning money, they'd just be a company. Startup phase | complete. | ceeplusplus wrote: | The difference being that in successful B2B startups you | don't have 600k ARR at 500 employees. | gregdoesit wrote: | Author here. I was wrong on this information and updated the | article - got a correction since. L6 and above employees would | receive this: staff+ engineers, eng leadership, sales etc. | | There are companies where this information does go out to all | employees in the spirit of radical transparency. Skyscanner is | an example where every day, every employee gets the full | revenue breakdown. These numbers are also shown on monitors | across the company. | Twirrim wrote: | >L6 and above employees would receive this: staff+ engineers, | eng leadership, sales etc. | | So all the people that have the authority and capability of | saying "woah, woah, woah, we're building something far too | complicated" had all the information they needed to prove it, | and didn't. That's a fascinating sign of immaturity, and | arguably incompetence, at some level in the org. The article | points out engineers pushing up about scaling down | infrastructure, but it's not clear if that came from the | staff+ level or below (or both?), and leadership pushing | back, or quite what. | encoderer wrote: | Lol. L6. | | Look no further folks. This guy Dominic was clearly LARPing a | startup. | | Leveling frameworks before traction is a joke to me. | mytailorisrich wrote: | That's what happens when you go on a crazy hiring spree. | deepinsand wrote: | I believe startups should implement levels once you hire 2 | engineers. It's hard to retrofit a system, especially if | you're trying to be thoughtful about any pay imbalances. | JumpCrisscross wrote: | > _startups should implement levels once you hire 2 | engineers_ | | I think OP is cracking a joke about a $600k company | having six levels of hierarchy. | deepinsand wrote: | Here's my experience: | | - When you hire, you implicitly put people in a level | which dictates what you're willing to pay them. | | - People will always feel underpaid, and demand more $$. | Without a system, you give them out arbitrarily. | | - You now have enough people to add structure to the | process. Do you base people's levels on their current | salary? On their skill? | | - What happens when people notice the discrepancy in pay | or skill among a level? Especially if it's skewed by | race, gender, etc? | | I think you should have as many levels as pay-bands in | your startup, which might be 6. | mywittyname wrote: | Nobody wants to be a level 1. So the real level 1 was | probably L4. | lupire wrote: | L1 is junior data center tech and junior helpdesk. | bps4484 wrote: | I completely disagree. They were 450 people with 150 | engineers. If I had to guess I would bet the engineers | clamored for leveling and it came from bottoms up requests | and a need to be fair with compensation when hiring. | | Now to be clear they shouldn't have been that big | (clearly), but leveling people was not the problem. | whymauri wrote: | I worked at a company where the founder manufactured an | inane 13 level hierarchy (at seed!!!). We had two | engineers, all level one, lol. | dymk wrote: | So there were 14 employees total, if I had to guess? /s | PragmaticPulp wrote: | Even startups benefit from job titles and hierarchy. | | Even startup employees benefit from a visible promotion | path. Remember, they had hundreds of employees. Not just a | couple people in a small office somewhere. | | Most likely, the levels corresponded to pay bands and | helped determine where people fit into the seniority | hierarchy (such as determining who receives sensitive daily | information updates) | sergiotapia wrote: | Levels before you even have product market fit is a joke. | [deleted] | alar44 wrote: | Yeah that's all fine and good but 6 levels is ridiculous | for a new startup. My company has been around for 30 | years, doing $75 million in sales this year, has 300 | employees and we now have 4 levels, one more than last | year, and even that feels artificial. | encoderer wrote: | You are 100% right -- for late stage startups where you | would normally see 500 people. | | At the Fast stage, the only viable promotion path is ship | work that impacts the business or close deals that bring | in revenue. The promotion levels game is something you | play in larger organizations to engineer a rat race for | people because it turns out they really like that. But | no, a startup does not benefit from hierarchy quite the | opposite in fact. That crap is all overhead it's | necessary as you grow but actually detrimental to your | success. | bps4484 wrote: | they were a company of nearly 500 people. they had 150 | engineers. At that stage people who don't have levels | will feel like they have no direction. | encoderer wrote: | This is just such a big company mentality. | | Yes, you need like Engineer and Senior Engineer and then | when you have some rock stars who actually _move the | business forward_ they are your principles and /or future | directors. | | To try to build this all up ahead of time, show me where | it's ever worked? When Google was that size they were | playing with having _no managers at all_. | xrd wrote: | That's really fascinating information. | | Dashboards are all the rage as a way to get teams aligned | around "critical numbers" (the metrics that matter). | | But, if your dashboards start telling a story of | impossibility, your best people are going to leave earlier | rather than hang on. | | A definite downside to the dashboard cult. | uoaei wrote: | It's only a downside if the company's priorities are so | completely misaligned vs employees' priorities. But at that | point, I would call that misalignment the downside! | nosianu wrote: | Is it? Maybe it's good if the decision to abandon ship is | distributed over more people? | | It hinges on one thing that I don't know: How likely are | already failing companies to recover? Probably with the | added condition that incoming help is not foreseeable, | because if it's failing but people know help is coming that | takes away some of the negative signaling. | | I suspect that most of the times struggling companies won't | get better, or at beast will continue to struggle and never | make it big. | | If that is the case, it will be better for both parties, | not just the employees but also for the founders, if the | life of this failed attempt of a business is cut short. I | know first hand that when you are heavily invested pulling | the plug yourself is really hard. I have no data, but I | would not be surprised if the problem of staying too long - | for founders too - is larger than the opposite, pulling the | plug too early. Not that the latter is easy to show, since | even if you get a large sample, you will never know for | certain for many of the data points what would have | happened if the plug had not been pulled without parallel | mirror universes. | gowld wrote: | > staff+ engineers, eng leadership, | | How does this even exist at a company with essentially no | revenue? This should be at most 1 person in "staff+ | engineers, eng leadership" | gregdoesit wrote: | This is a hallmark of a scaleup which assumes it will | become a massive company in a short amount of time, and | sets up its structure accordingly. | | There are times when a gamble like this pays off. Take how | Uber built their organization and systems similar to how | Google did it, even when they were smaller. | | In the case of Uber, their approach, you could argue, paid | off in the sense that Uber did get traffic that would have | been non-trivial to handle, and complex business use cases | that it could handle easier thanks to it's structure. | | My view is that this approach is an all-or-nothing setup | and it might end up poorly - and spending WAY more money | than needed - than if taking it one step at a time. | | Also note that Uber did operate in an environment when it | could raise ridiculous amounts of money as it scaled up. | This doesn't seem to be the case in the current funding | environment of 2022, which has cooled down considerably | from 2021. | lupire wrote: | Uber had revenue and millions of customers begging for | someone to offer the service. | | Google did not have "staff+" when it was starting out. It | had founders and a couple of hires. | gregdoesit wrote: | Great points on both. Uber, indeed, did not have | engineering levels beyond senior engineer until year 2 or | 3 as far as I know (I joined later, but talked with | several early engineers while there). | | I was trying to play devil's advocate but I have to agree | that pre-product-market-fit, these levels are likely an | overkill and distract from the real problem: validating | product-market fit. | axg11 wrote: | Amongst a sea of negatives, that they continued to share these | numbers with employees is a positive mark for the culture at | Fast. Sure, it's easy to say with hindsight that Fast employees | should have seen the demise coming. On the other hand, they | also watched one of the founders raise $100M+ from Stripe. | Perhaps they thought the burn could continue and eventually the | product would reach traction. | brundolf wrote: | > I'm actually surprised that everyone was receiving daily | updates of company revenue. | | My company posts daily new user signups in #general on Slack. | That's not quite revenue, but you could almost extrapolate | betaby wrote: | I've never heard about Fast before I read the article. I don't | know whether I live in a bubble or SV lives in a bubble. | xxpor wrote: | I thought Netflix had spun off https://fast.com | gowld wrote: | > one-click checkout scaleup Fast | | a what now? | [deleted] | karmasimida wrote: | > the company only generated $600K in revenue in 2021 | | Ain't none of the investors catch this ... ? Losing money is one | thing, but this seems to me as really low growth right? | [deleted] | eatonphil wrote: | Presumably they did catch this because they weren't able to | raise a new round. | karmasimida wrote: | > the company only generated $600K in revenue in 2021 | | Ain't none of the investors catch this ... ? Losing money is one | thing, but this seems to me as really low growth | louthy wrote: | This is what surprised me. It seems the board was asleep at the | wheel, failing in their fiduciary responsibilities | eterm wrote: | > every small business needed custom engineering work to be done, | making integration slow | | This is the killer for SMEs. If you're not selling to enterprise, | find a solution you can whitebox and quickly ship with minimum | customisation. | steve76 wrote: ___________________________________________________________________ (page generated 2022-04-07 23:00 UTC)