[HN Gopher] Why billing systems are a nightmare for engineers
___________________________________________________________________
Why billing systems are a nightmare for engineers
Author : Rafsark
Score : 454 points
Date : 2022-05-18 16:06 UTC (6 hours ago)
(HTM) web link (www.getlago.com)
(TXT) w3m dump (www.getlago.com)
| Melatonic wrote:
| Why XXXXXXX are a nightmare for engineers:
|
| PEOPLE
|
| On a more serious note though billing can be hugely complicated
| and I am really glad I do not have to deal with it
| Rafsark wrote:
| Do you have a dedicated team in charge of it?
| didip wrote:
| This is why Stripe has a big moat. Not many people want to deal
| with this problem.
| jlg23 wrote:
| I've implemented a few billing systems and "the nightmares" were
| never related toengineering but always to specification: If
| bizdev does not know what they want, no outsourced imlementation
| will be able to help them.
| AnhTho_FR wrote:
| From your experience, who should specify the billing system?
| The Product team?
|
| We're building Lago so that the right questions / decisions
| frameworks are asked during the implementation, so it's like a
| forcing power embedded in the product.
|
| Our experience is, unless the Product/Biz team has been exposed
| to billing, they will never specify in a way that is precise
| enough, so the engineers who implement billing will have to
| think/assess/decide themselves.
| retcon wrote:
| >/as Qonto is a 'neobank', we wanted to charge our customers
| directly in their wallet, through a ledger connected to our
| internal billing system. The team started from a duo of full-time
| engineers building a billing system (which is already a
| considerable investment), to currently a dedicated cross-
| functional team called 'pricing'.
|
| Stored procedures (ca. Oracle v6) suddenly sound a walk in the
| park.
| AnhTho_FR wrote:
| They do indeed!
| yobbo wrote:
| One design decision that, for me, seems to simplify things is to
| consider the "business system" a type of state machine that
| records all business events and serves as a "source of truth". If
| the events are not recorded, they have not occurred from the
| business perspective. A ledger-type architecture can be useful.
|
| This means that business events or user operations generate state
| transitions, which eventually are implemented as database
| transactions. The event log can be stored and inspected.
|
| The end user terms are encoded in the state-transitions. Contract
| obligations are encoded in the state of the database.
|
| As for calendar issues - operations need to be performed over
| "real calendars". It might be practical to "materialize" the
| business calendar into discrete units for this purpose.
| Rafsark wrote:
| For sure; I do think an event based solution working as a
| source of truth for the billing is the right solution. However
| it still creates engineering difficulties (making sure you
| don't ingest the event twice for instance). The ledger-type
| architecture can definitely work. When we built the system for
| a fintech, it was actually an event-based architecture
| connected to a ledger (taking the money out of a wallet). I
| think the whole process would be: - Ingesting events for usage
| based feature - Store these events in a database and make sure
| you make the idempotency right - Aggregate the usage of these
| events based on most common aggregate functions - Price this
| usage inside plans or subscriptions - Assign a subscription to
| a customer - Trigger a webhook (used for invoice/payment) at
| the end or beginning of the billing period
| yobbo wrote:
| And that events, as much as possible, are formulated as
| observations or state-changes rather than operations. Events
| in the form of "set x=1" or "y=0 was observed" are
| idempotent, where as "do this with that" is problematic.
|
| In the framework of state-transitions, actions such as "price
| this with that" are implicit in the state matrix, and
| shouldn't be called from APIs. Rather, API calls should be
| used to produce reports which are print-outs of the "current
| state". Webhooks in this framework are attempted state-
| transitions. User interactions generate events, and requests
| for reports (for example invoices), but as little as possible
| in the form "do this with that".
|
| My opinion currently is that there is little value in sharing
| code (libraries/frameworks) for these sorts of solutions
| because the code is trivial, application/language/domain-
| dependent, and coupled to everything else in the system.
|
| But design patterns, templates, and concise examples are
| hugely valuable because they help illustrate the complexity
| and solutions up front.
| templaedhel wrote:
| Great article and excited to see another solution tackling the
| complex problem of billing. I'm an founding engineer at
| https://metronome.com (building usage-based billing
| infrastructure) and definitely echo the sentiments shared here --
| building a billing system that can scale is no small feat and
| anyone who previously built billing in-house can attest to just
| how painful that is.
| johnzim wrote:
| I'm the CTO of a company tackling exactly this problem
| (www.salesbricks.com) and I'd go one further.
|
| The complexity in billing, invoicing, payments etc are _not_ the
| stuff that computers or systems do. The complexity is and always
| will be in the stuff humans do.
| eesmith wrote:
| A classic, from http://www.canonical.org/~kragen/tao-of-
| programming.html
|
| > There was once a programmer who was attached to the court of
| the warlord of Wu. The warlord asked the programmer: ``Which is
| easier to design: an accounting package or an operating system?''
|
| > ``An operating system,'' replied the programmer.
|
| > The warlord uttered an exclamation of disbelief. ``Surely an
| accounting package is trivial next to the complexity of an
| operating system,'' he said.
|
| > ``Not so,'' said the programmer, ``when designing an accounting
| package, the programmer operates as a mediator between people
| having different ideas: how it must operate, how its reports must
| appear, and how it must conform to the tax laws. By contrast, an
| operating system is not limited by outside appearances. When
| designing an operating system, the programmer seeks the simplest
| harmony between machine and ideas. This is why an operating
| system is easier to design.''
|
| > The warlord of Wu nodded and smiled. ``That is all good and
| well, but which is easier to debug?''
|
| > The programmer made no reply.
| buescher wrote:
| Which came first? I am pretty sure it was computer accounting
| systems.
|
| Which is older, the oldest continuously supported operating
| system, or the oldest continuously supported (computer)
| accounting system?
| eesmith wrote:
| The oldest computer accounting system was likely LEO -
| https://en.wikipedia.org/wiki/LEO_(computer), online in 1951
| and doing accounts probably by 1955, if not earlier.
|
| I believe it predate operating systems by a few years.
| (Wikipedia claims the first OS was the GM-NAA I/O, produced
| in 1956.)
| AnhTho_FR wrote:
| OMG love the story, thanks for sharing!
| PaulHoule wrote:
| Applications programming has a complexity of it's own that's
| distinct from the problems in systems programming.
| alex_suzuki wrote:
| Thoughtful, and made me laugh. Thanks!
| bombcar wrote:
| Heh, I'd say the operating system is easier to debug, because
| the accounting package could literally have "bugs" that are
| caused by errors in the tax laws, business processes, etc,
| which cannot be fixed, only worked around.
| teddyh wrote:
| The linked page does not show it, but _The Tao of Programming_
| is actually a book which you can buy:
|
| https://en.wikipedia.org/wiki/The_Tao_of_Programming
|
| I own a copy; it's great.
| epberry wrote:
| Usage based billing is not only hard to implement but hard to
| consume. No one really knows what they are spending and how much
| particular customers or products cost. I did a writeup a week ago
| on CDNs and ended up spending several hours in spreadsheets.
|
| I had felt this pain vaguely before but actually seeing it
| everyday at my current job makes me realize there are engineers
| suffering on both sides of software billing.
| hobo_mark wrote:
| Speaking of usage-based billing, if I intend to charge users a
| usage-based bill at the end of the month, as far as I have
| looked Stripe is the only provider that supports this (they
| call it "metered billing"), is there really no alternative?
| epberry wrote:
| The OG poster website is actually a Stripe alternative. They
| also call out Chargebee which is another alternative that
| should support this.
| Rafsark wrote:
| Stripe offers simple metering, but in the end, you make the job
| of calculating metering charges (aggregate all usage and send
| the value to stripe). It's becoming harder when you are an API
| or a cloud company who want to bill a server usage for
| instance. Calculations are getting heavy, costful and hard to
| maintain. This is why I do think stripe is not a usage based
| solution, and also the reason why I decided to create a company
| trying to solve this pain :)
| kdeldycke wrote:
| This.
|
| Usage-based billing is incomplete without its suite of
| reporting, consolidation and visualization tools. No wonders
| there is a cottage industry of cloud computing consultants and
| SaaS solutions focused on helping you to make sense of your
| AWS/GCP/Azure invoices.
| ab_testing wrote:
| I think companies should not try to reinvent the wheel with
| building billing systems. It is better to implement an ERP to
| take care of billing rather than building out your own. If you
| ever grow large enough, this homegrown system will never be SOX
| compliant and will not stand up to audits.
| jakewins wrote:
| Pro-rated billing is one of the many places outside of database
| query planners where Dynamic Programming is useful! Maybe there's
| a better way to do it, but well, if you give a man a hammer.. I
| wrote up how we did this at Equipmentshare back in 2016:
| https://tech.davis-hansson.com/p/price-wars.html
| pbreit wrote:
| If you try to keep it simple, it's not that hard.
| spchampion2 wrote:
| Just wait until you meet billing's angry roommate: invoicing.
|
| In the US, an invoice is just a weird PDF that you might glance
| at before sending off to your accounts payable team. But in other
| countries, especially those that use VAT style taxing systems, an
| invoice can be a legal document that expresses costs and taxes in
| a legally prescribed way for accounting purposes. Many countries
| have prescriptive rules over how you invoice, who can write
| invoices, when you can invoice, what goes on an invoice, etc. And
| every country does it differently.
|
| Are you building a service that charges incrementally many times
| per period? Or even worse, refunds incrementally? You might be
| sending a truckload of expensive accounting papers to your
| customer every month, one per transaction. And each of those
| pages was printed using government prescribed software from
| oddball vendors you must use or else.
| ozim wrote:
| Yeah - and simply creating an invoice is creating taxable
| income in EU - if customer drags their feet you still owe VAT
| to the taxman.
|
| If you want to change invoice you have to make a special
| "correction invoice" because changing "real invoice" is a
| criminal offense - fun things :)
| johnrgrace wrote:
| And most big companies who agree to pay you net X days, that
| usually is after the presentation of a correct invoice. The
| invoice doesn't get to them, or it isn't right the clock
| doesn't start for them to pay you. Getting invoices wrong can
| quickly result in major cashflow drops.
| AnhTho_FR wrote:
| I'm Anh-Tho, one of Lago's co-founders here. Indeed, from our
| experience people see 'billing or invoicing' as a monolith
| block, whereas there's a whole 'pricing stack', billing >
| payment > invoicing > cash collection + accounting + CPQ +
| sales commission. For cash collection, we mentioned the tool
| used by Lattice called Upflow which helps to manage account
| receivables.
| cm2012 wrote:
| Just one of many reasons its easier to do business in the US
| IAmEveryone wrote:
| The difference is marginal at best. Most of the requirements
| are obvious and never/rarely change, like your address or the
| date. And adding a an incrementing number to a set of
| documents isn't exactly rocket science, either.
| AnhTho_FR wrote:
| Hey cm2012, I'm one of the cofounders of Lago (my business
| partner wrote this post). Actually even if you're
| incorporated in the US, if you serve a client in EU for
| instance, the VAT rules of EU apply to how you're supposed to
| invoice your customer, beyond a certain transaction limit.
| So... unless you're based in the US, and only serving US-
| based users, it gets complex very fast!
| notch656a wrote:
| If you wire your dollars onto US soil, for a service
| performed in the US by US business, good luck getting the
| US legal system to enforce upon that US business whatever
| euro-cucked invoicing scheme is called for.
| pimterry wrote:
| This doesn't matter. For B2B, if you don't show your EU
| customers a valid invoice, they simply cannot pay you.
|
| It's not EU governments or post-purchase issues you have
| to deal, most of the time - it's trying to persuade a
| large accounting department to pay you unaccountable
| money. If you can't follow their invoicing requirements,
| it's just not going to happen.
| notch656a wrote:
| >they simply cannot pay you.
|
| Do people ever stop and wonder WHY in nations like
| Portugal the informal economy is roughly DOUBLE of say
| the US? When every invoice must be available to the
| government, what actually ends up happening is non-
| conforming invoices either get made up by the customer or
| the money magically flows out under some other auspice.
|
| Making it literally illegal to accept an invoice that is
| legal in the country in which you obtained it borders on
| logic even my toddler can understand is absolutely
| begging for bad business climate or tax evasion.
| wewxjfq wrote:
| Do people ever stop and wonder WHY the US is the biggest
| tax haven on earth? Because apparently it's against your
| freedumbs to put an incrementing number on invoices and
| write your address on it.
| notch656a wrote:
| >Do people ever stop and wonder WHY the US is the biggest
| tax haven on earth?
|
| I consider this an advantage of the US, not disadvantage.
| sbierwagen wrote:
| Feature, not a bug. If there's pervasive lawbreaking,
| then the officials can collect more bribes.
| dkjaudyeqooe wrote:
| European bureaucracy is out of control, and that's just
| the west. In the east and former Soviet states, it
| beggars belief.
| AnhTho_FR wrote:
| I'm Anh-Tho, one of a co-founders of Lago. I'm French,
| and I confirm, in B2B, no invoice (that is compliant, not
| a mere receipt) -> no payment.
| notch656a wrote:
| Thank you Anh-Tho for your response.
|
| What is the mechanism for French government to subpoena a
| non-EU business to find this invoice? If the customer
| (instead of vendor) produced an invoice that matched
| their bank statements and non-conforming receipt,
| hypothetically, would anyone really know the difference?
| PeterisP wrote:
| Probably they wouldn't as full audits aren't that common,
| but audits do happen and most accountants won't care
| about your purchase as much to personally commit a felony
| and forge documents for no good reason (accountants do
| have personal responsibility, and every accounting course
| reminds wannabe accountants that "the boss ordered it" is
| not an excuse) so they simply say that they can't/won't
| do it and if the vendor can't send a satisfactory invoice
| then they won't do business with that vendor.
| notch656a wrote:
| How would a French audit uncover anything wrong with a
| conforming format invoice under the name of some random
| American company that exactly matched the bank statement,
| physical goods, and customs paperwork?
| AnhTho_FR wrote:
| Hey notch656a! I'm not a government expert, but the
| challenges would start with:
|
| - I am Frenchy (a French co) - Buying software from Ricky
| (a US co)
|
| If I don't have a proper invoice from Ricky to give to my
| accountant at the end of the year, and I get a tax
| control, then, I'll be fined.
|
| If the tax ministry notices large amounts of similar
| fines, they might look at Ricky.
|
| So the risk is more on Frenchy (risk to be fined). But
| I'd say that Ricky is safe for a while, unless the
| invoiced amount are really huge.
| DocTomoe wrote:
| That american exceptionalism approach will work with
| private citizen customers. With business customers,
| leaning back in the legal framework encouraging tax fraud
| will not fly. They just won't buy from you then.
|
| That's why any US company wanting to do business in
| Europe either has an European subsidiary, or follows
| something that's compatible with EU invoicing schemes.
| notch656a wrote:
| >. With business customers, leaning back in the legal
| framework encouraging tax fraud will not fly.
|
| Yes which is why in nations like Portugal, where any
| business invoice may be subject to accountability to
| government, there totally isn't a massive informal (read:
| tax-evading) economy to deal with the fact that the
| government makes it literally impossible (at least in
| above words) to accept an invoice legal in the
| jurisdiction in which it was issued.
| PeterisP wrote:
| It's not literally impossible - the simple solution to
| standard invoice being unacceptable is to issue a
| different invoice that matches the requirements, and
| almost every vendor is happy to do so in order to make
| the sale.
|
| Also, the informal tax-evading economy relies on personal
| contacts and 'mutual understanding' - it's not easily
| accessible to a foreign vendor, and when foreign vendors
| do want to access that economy, it requires _much more_
| adoption of local customs than just making a different
| format of invoice.
| notch656a wrote:
| Yes literally impossible to take something in a format
| legal in the US, that doesn't match the foreign
| requirements, based on European testimony above (the
| exact truth of which IDK).
|
| Informal tax-evading (when the seller isn't evading, just
| the buyer) doesn't require personal contacts. This is
| just naive thoughtless statement. It happens all the
| time. Example: business owner goes to Panama where they
| know no one. Buys a pallet of llama wool, seller gives a
| non-conforming receipt. Buyer goes back to France,
| imports the pallet as "cotton" and creates a fake invoice
| showing the Panamanian sold cotton. Seller then sells
| llama wool on the streets for cash, marking in the books
| that they sold "cotton" for significantly less. They then
| use the on-the-books "cotton" proceeds to buy a shit-
| wagon car or something, and fix it using the unrecorded
| cash on the side. They sell the now nice shit-wagon and
| record the proceeds as profit. Now all the money for the
| llama wool is accounted for.
|
| Cross country tax-evading is only assured in this case to
| require personal contacts when the tax evading happens on
| _both_ sides. When it only happens on the EU side,
| there's no need for personal contacts on the US side.
| gamblor956 wrote:
| You keep bringing this up and it's simply not true. The
| EU has a single set of VAT invoice requirements that
| applies across the EU (the MOSS invoicing standard). Any
| VAT invoice that satisfies these rules is legal in all EU
| countries (https://eur-lex.europa.eu/legal-
| content/EN/ALL/?uri=CELEX:32..., article 226).
|
| My company does plenty of business with EU businesses and
| we use a single standard EU VAT invoice without issue.
| raverbashing wrote:
| People are mixing up things here
|
| It's not illegal to pay an US company with an US invoice
| in the EU. (HN likes to invent problems where there
| aren't, and companies do that all the time). It is not
| even limited to the US, most other countries can't/won't
| follow the format.
|
| However there might be extra steps in adding those
| invoices to their accounting. For example, when import,
| the buyer will be charged the import tax (instead of
| having it added to the invoice)
|
| The informal economy of Portugal has nothing to do with
| the above btw since that has nothing to do with American
| invoices (and you can bet German rules are not much
| simpler) - not saying the bureaucracy isn't, most of the
| time, stupid. The US also has it fair share of stupid
| crap that people have to deal as well but it is
| "transparent" to most Americans.
| orf wrote:
| There isn't a massive tax evasion economy in the USA?
| notch656a wrote:
| Yes. It would be even larger if US companies couldn't
| accept foreign invoices without being in a special
| format. Does tax evasion in the US somehow disprove tax
| evasion in Europe?
| yupper32 wrote:
| That's their point, they said "do business in the US".
| You're saying the same thing.
| mitchdoogle wrote:
| Protecting consumers is always bad for business
| baronswindle wrote:
| In this particular case, do you think non-US governments
| could protect the interests of consumers in their
| jurisdictions in a way that is less unfriendly to the
| businesses that want to sell to them?
| wonton53 wrote:
| I work on a payment system that invoices consumers on behalf of
| other companies and pays out the money to these companies by
| splitting the payout on multiple partners (luckily I do not
| handle the support and follow-up). The system have ability to
| refund invoices and customers some times end up paying more
| than or just parts of the invoice. On top og all this it
| integrates with several old 90s systems with poor datetime
| handling and poor uptime (some times a windows xp box in the
| cudtomers offices). It also handles card payments and over all,
| by far the hardest thing to get right is the invoicing. Its
| just so extremely fuzzy and time sensitive.
| trollied wrote:
| Yup, it's a nightmare. Invoices needing to be in specific
| number sequences. Any corrections need to be dealt with by
| reissuing a special "corrective invoice". Don't even get me
| started on geocoding for taxes. Urgh. Also, when you've got the
| invoicing right, you've got to work out how to do the feeds to
| the accounting systems so that it all posts to the right place.
| trollied wrote:
| I'll add that the accounting side can get complicated really
| quickly. You end up with lots of different scenarios. Earned
| but unbilled: out of bundle call charges, for example. Billed
| but unearned: a paid in advance monthly fee, for example -
| this may need to be recognised as a percentage per day. Etc
| etc. So much to take into account when feeding to ledgers.
| radicalbyte wrote:
| A credit note. How the ** do you handle it in the US without
| issuing credit notes in your bookkeeping?
| indymike wrote:
| We issue a credit note*
|
| * Except (these aren't the right thing to do, but I've seen
| it in the wild many times):
|
| - When someone in sales issues an invoice with a negative
| amount on line 3. Hey, if I take payments with invoices,
| why not give back?
|
| - When someone agrees to settle up by providing free
| services and throws it on the next bill.
| Rafsark wrote:
| Yep, fortunately in Europe we can issue credit notes, making
| our job easier. I do think giving the possibility to send
| charges information to a webhook is a great way to start:
| this data can be plugged to any invoicing system (or any tax,
| payment providers) who are trying to solve those things. The
| hardest part for me, when working in that fintech, was to
| build the whole logic to trigger the invoice, not so much the
| invoice itself. But this company was a EU company, probably
| easier to work with invoices
| codegeek wrote:
| And then just wait until you meet Invoicing's annoying cousin
| Purchase Orders (PO). THere is a PO. Can I pay this in multiple
| invoices ? Sorry the PO is not approved yet. We cannot accept
| an invoice. Sorry the Invoice doesn't have reference to our PO.
| Can you fix that please ?
| AnhTho_FR wrote:
| Hey @codegeek! I'm Anh-Tho, one of the co-founders of Lago.
|
| We're working on making the 'CPQ' : Configure Price Quote
| journey simpler, and this includes the PO dead end. We're on
| a mission to make the pricing stack more open and
| customizable for companies (hence the open-source angle), and
| we're starting with the billing system. Join the journey,
| we'll share our repo soon, and hopefully we can address these
| painpoints faster with the community contributions too!
| vidarh wrote:
| Many years ago I led a billing team at Yahoo! responsible for
| billing for premium services in European markets.
|
| My team existed for the sole reason that the European business
| did _not_ trust that the US payment services team understood
| European needs, and the "invoice is a legal document" thing
| was one of them. I spent so many meetings repeating to the US
| team that no, we could not switch to their invoicing as long as
| a new software release might retroactively change the template
| used to show already issued invoices (e.g. the registered
| office might change, or the VAT number).
|
| We didn't have to deal with printed invoices thankfully, but we
| did ensure we produced each invoice once and stored it, so that
| the European finance team could sleep at night.
|
| At the time Yahoo! had _8_ different payment platforms. Some of
| those were due to weirdness around acquisitions and Japan
| (which was a joint venture), but apart from mine I believe two
| others also existed because of local weirdness that the local
| businesses decided it was safest not to let Yahoo! US mess
| with.
|
| At the time I left, after 3 years of that, we'd managed to get
| agreements to migrate a few bits and pieces to the US after
| they'd shown the understood requirements for specific features,
| but it was like pulling teeth.
| AnhTho_FR wrote:
| I'm Anh-Tho, one of a co-founders of Lago, thanks for your
| comment! I was the country manager of France for a US
| Sequoia-backed company before, and I never ever managed to
| convince HQ to work on EU invoicing, I think they just did
| not want to open the pandora box you just described!
| christkv wrote:
| I definitively am interested in a couple of months. I
| rather not build our own if we don't have too.
|
| Do you have any early guides or api docs to take a look at.
| Our use case.
|
| 1. Base service fixed packages (based on users with volume
| discounts)
|
| Extra costs 2. Extra storage space usage 3. Add on services
| (fixed or usage billing) 4. Invoicing third parties
| (reselling content)
| 908B64B197 wrote:
| > You might be sending a truckload of expensive accounting
| papers to your customer every month, one per transaction. And
| each of those pages was printed using government prescribed
| software from oddball vendors you must use or else.
|
| I guess that's good for employment numbers? There's really two
| ways to create jobs, innovate or regulate. The US and the EU
| have apparently chosen very different paths. With every country
| having it's own special snowflakes laws, there's a reason EU
| founders always try to come to the US first.
| dawkins wrote:
| Invoicing gets even better in countries like Portugal, where
| you have to send to the tax authorities every invoice that you
| generate.
| giomasce wrote:
| In Italy too. They specifically developed an XML format for
| that, and each time you issue an invoice you have to (well,
| with some exceptions, but they are gradually removing all of
| them) send a copy to the revenue agency, which will forward
| it to the recipient (and keep a copy).
|
| While it is a bit inconvenient, though, I don't think it is a
| bad idea. I am pretty sure it helps a lot making the life
| harder for people evading taxes, and to some extent the
| availability of a standard machine-readable format makes it
| easy to do accounting. My wife sells videocourses online and
| I developed a thing that automatically issues invoices as
| soon as people buy a product (with some human supervision,
| mainly to avoid the machine doing havoc is some exceptional
| situation happens; normally it is just "click a button to
| send the invoice"). The invoices are then automatically
| available to the accountant for doing accountant things.
|
| You can find the XSD here:
| https://www.fatturapa.gov.it/en/norme-e-
| regole/documentazion...
| bombcar wrote:
| And even worse - your biggest customers won't even _smell_ your
| invoice unless you enter it line by line into some ancient SAP
| system developed 20 years ago, where everything cloud related
| is classified as _telephony_ except storage space which must be
| classified as _filing cabinets (movable)_ or your invoice will
| be (very slowly) rejected.
|
| And if it is rejected you have to enter it all again by hand;
| _editing_ isn 't a feature.
| deltarholamda wrote:
| I said, over and over again, that moving away from goats and
| salt as currency was a mistake, but nobody listened to me.
| D-Coder wrote:
| But the different conversion rates for each
| kingdom/duchy/province/parish were a nightmare even then.
| Rafsark wrote:
| Wow, haven't lived this pain fortunately... but totally
| understand how hard it is to try building a << modern pricing
| stack >> but being forced to create invoices in old softwares
| like SAP...
| folmar wrote:
| I don't get what you're talking about. Our biggest customer
| reluctantly agreed to accept preliminary invoices for
| acceptance over fax instead of signed-for mail, provided that
| we also ship a paper version with a red stamp promptly.
| [deleted]
| Rafsark wrote:
| Hi! We decided to aggregate all costs of a "billable period".
|
| Imagine you bill your customers monthly (the billable period),
| all the charges (usage-based features + subscription) will
| appear as line items of a single invoice.
|
| This enables you to gather all the fees of a period into a
| total invoice, but still be able to provide granularity to your
| customers (breakdown of all the fees to be paid).
| onion2k wrote:
| Hi, customer here. We'd like to change our billing period to
| start on the 31st of _every_ month, and we 'd like to pay 90
| days in arrears, and we'd also like you to invoice us ahead
| of the billing month. Oh, and we're big enough that losing
| our business will end your startup, so don't get it wrong.
| Thanks!
| AnhTho_FR wrote:
| I'm Anh-Tho, one of Lago's co-founders here. Thanks for
| sharing! What did you do then? I'm curious!
| bckr wrote:
| Hey, Anh-Tho, just as a point of feedback, since you're
| leaving a lot of comments in this thread, it's not
| necessary to introduce yourself in each one. Most folks
| will be scanning the whole thread and seeing your
| introduction gets repetitive and breaks the
| conversational tone and makes it overly commercial.
| Thanks.
| burnished wrote:
| I fear if they did not they'd be accused of something
| else. I agree with your observation but I don't think
| there is an obvious solution as you imply.
| bckr wrote:
| They can disclose their interests without the repetitive
| intro. But for the above comment, just asking a question,
| even this is unnecessary.
| AnhTho_FR wrote:
| I agree it was repetitive, but indeed, I did not want
| people to think I was trying to pretend to be unrelated
| to the post.
|
| Thanks for the constructive feedback!
| bckr wrote:
| you're welcome
| vincentmarle wrote:
| Wait, who are you again?
| AnhTho_FR wrote:
| Noted thanks! It's my 1st post on HN, so I appreciate the
| feedback.
| marcosdumay wrote:
| Yep. Invoice software is country specific. One can sell
| invoicing software to several countries, it just can not be the
| same software. And no, you can't just abstract the differences,
| because the invoices focus changes widely.
|
| At least we are in a situation where the seller is almost only
| subject to the laws of the seller's country. There are some
| exceptions, but one can mostly deal with those (the EU has
| plenty, the good news is that if the seller is not also on the
| EU, most do not apply).
|
| Also, I haven't seen a country accounting system that made
| accounting papers inherently expensive. Some make refunds very
| expensive, but not the papers. It's more a matter of badly
| designed software and processes.
| duxup wrote:
| Not even legal issues.
|
| In my experience invoicing, and really all 'printable'
| documents is the land of "oh someone did something wrong put X
| on the document". And then again and again and again... And
| "this is two pages long that's too much" and on and on ...
|
| It's bike shedding insanity with no ideal or end in sight.
| Every change is arbitrary with no real measure of success.
|
| I swear the only time anyone LOOKs at these documents is to
| bitch about them.
|
| Oh man and just as if I summoned it someone just sent me a
| ticket about something on an invoice.
| gamblor956 wrote:
| Just because you don't understand invoicing doesn't mean that
| the concerns that appear trivial to you are actually trivial.
|
| From the perspective of someone who works with the people who
| process the invoices, putting "X on the document" and "two
| pages long" can be a huge deal when you have to deal with
| hundreds of invoices or when "X" can mean that an invoice
| must proceed down a different processing path. These aren't
| bike shedding concerns; they affect the actual processing of
| the invoice. In many cases, these concerns affect the
| validity of the invoice, meaning that the recipient's
| obligation to pay (or the associated deadline) is not
| triggered.
|
| And doing something wrong on an invoice is _never
| bikeshedding_. An invoice is a legal contract, and in the
| E.U., an invoice is an _unmodifiable document_.
| duxup wrote:
| They're trivial... when you make the same change to add
| something, remove it, add it, remove it and hear that it's
| all being done "because X" and "X" never stops, it's
| clearly not working and the changes are trivial.
| gamblor956 wrote:
| No, the changes aren't trivial.
|
| It sounds like you simply don't understand why they're
| doing it and you just assume that they're trivial because
| you have no knowledge of the domain.
|
| Off the top of my head, I can thing of a dozen reasons
| for why they'd require something to be added, then
| removed, then added, etc., from one invoice to the next.
| duxup wrote:
| > It sounds like you simply don't understand why they're
| doing it
|
| I don't know why you would think that considering I work
| with them and you do not...
| gamblor956 wrote:
| I work with the A/R and A/P teams all the time. It's a
| large part of doing tax in-house.
| [deleted]
| msluyter wrote:
| I rewrote an invoicing system (originally in .NET) in Python and
| yeah, I totally feel this. All the system did was generate a CSV
| file that could be input into another system (Quickbooks, iirc).
| The original .NET version was nonsensical at times, replete with
| bugs, and I spent a lot of time trying to figure out what the
| original intent was (there were no docs, of course, and finance
| often had conflicting/incorrect ideas about what the system
| should do.) Version 1 had to simply replicate the csv generated
| by the original system exactly, which required purposefully
| writing some bugs that I discovered along the way.
|
| But yeah, there were all sorts of nasty complexities -- this was
| a usage based system, but there were (optional) tiers (pay less
| for higher tiers), as well as certain time segments that were
| discounted. And then there were cases where multiple accounts
| could share a set of data and get a discounted rate and/or only
| one of the accounts would be billed. And on and on....
|
| One lesson I drew from this experience was that, when discussing
| things with finance to try to figure out what the requirements
| were, it _really_ helped to have pictures. I would create these
| elaborate usage /time graphs, and color in portions to reflect
| different billing/pricing scenarios. These made it much easier to
| discuss the current/future behavior of the system. Because saying
| "really, this is simply the integral of a usage function over
| time" didn't fly.
| elietoubi wrote:
| Hi! I co-led the billing stack re-write at Uber circa 2017 and it
| was an absolute nightmare. Some of the unexpected hard things
| that are not mentioned in this article: - Manage promotion was
| very complex - Modeling money (we used to model it as cents ie
| 1/100 of the common denomination ... turns out it broke a lot of
| countries like Japan where the Japanese Yen doesn't have cents) -
| Timezone is a hellhole
|
| One interesting open source project was https://killbill.io/
|
| PS: Since I built a billing and invoicing startup .. check it
| out: zenbill.com
| 0des wrote:
| Hey thanks for showing me this. What lies beyond the starter
| plan in concrete terms?
| [deleted]
| nickdothutton wrote:
| The reason billing systems are hard is because they tend to just
| accumulate features/behaviours/requirements. Forever. Although
| billing is not my specialty I have had to involve myself in it as
| product owner and business unit manager. Try and imagine the
| amount of logic, complexity, rules, and exceptions involved in a
| system that has accumulated perhaps 30-40 years of customer sign-
| ups. Umpteen contracts, umpteen iterations of each contract, and
| perhaps 50 or 100 products, each sold in 10 or 100 countries. I
| now know more about the tax system in South America than I ever
| wanted to.
| Rafsark wrote:
| Yep, totally right. Same at the previous company we worked for
| and built the billing logic. You start doing it in house
| because the first version is simple. It's getting complicated
| when you keep adding paying features to a product. Often see
| product managers becoming experts in taxes and accounting :)
| r00fus wrote:
| I see building or implementing these kind of systems as
| "discovering" use cases for given scenarios/customers/markets.
| That's why "billing" is almost too generic - instead "billing
| for telco in South American markets" - is more useful because
| those verticals have specific sets of use-cases and challenges.
| Brystephor wrote:
| I've worked at two companies in their payment systems. This post
| does a good job of mentioning many of the billing system
| challenges. I think it understates how difficult and important
| idempotency is to maintain at scale with multiple teams though.
|
| Additionally, it didn't mention financial regulation changes.
| India had changes sometime in the last few years which required
| whole new systems built that were specific to India customers.
| One example of complexity is this: does the system apply to
| customers who are in India or does it apply to businesses who are
| in India (the owners might not be)?
|
| He also did not mention fraud, which is basically uncollected
| spend that results in charge backs typically. If you get too many
| chargebacks, you'll get fined. If you get even more chargebacks,
| you'll get kicked off the card network and will no longer be able
| to accept cards within that network.
|
| Post pay (customer gets the product and pays afterwards) models
| are subject to more fraud, uncollected charges, etc.
|
| There's also the fun of dealing with downstream payment
| processors. I haven't worked with a processor or bank yet that
| was very reliable. Sometimes they'd return 500 http codes and
| then process a payment, requiring manual intervention. Sometimes
| that happens with files of payments because most banks deal with
| files for transactions.
| tablespoon wrote:
| > Additionally, it didn't mention financial regulation changes.
| India had changes sometime in the last few years which required
| whole new systems built that were specific to India customers.
| One example of complexity is this: does the system apply to
| customers who are in India or does it apply to businesses who
| are in India (the owners might not be)?
|
| IIRC, I've read this is one of the reasons that systems like
| SAP are so popular. They might be an ugly monster from a
| technical perspective and seem overly complex, but they provide
| thoroughly tested implementations that handle all that stuff.
| r00fus wrote:
| It's not like SAP/Oracle/etc have some magical power. They
| just pay attention to these things, build the product, THEN
| release the product. And initial releases are buggy just like
| most software.
|
| So more agile players can get involved more quickly and
| entrench position.
| AnhTho_FR wrote:
| Hey @tablespoon, I'm one of the cofounders of Lago here.
|
| Products like SAP do seem complex because of (i) the
| complexity of the billing problem and all the edge cases,
| (ii) they were not built with an API-first or engineer-first
| mindset
|
| We're only at the beginning of the journey, but that's one of
| the reasons why we thought of open-sourcing our billing API,
| so that the work could be forked and tweaked to address edge
| cases if needed (no need to wait for SAP to update the
| product: and they will never prioritize small edge cases).
| Also, it seemed to be no middle ground between 'Stripe
| Billing' (it's great for B2C Subscriptions) and billing
| systems that could address the millions of nuances of pricing
| between 'subscription' and 'usage-based' (most companies are
| a mix of both).
| _glass wrote:
| I am the last one to defend SAP, but they do have an API
| first approach since some time, check this out:
| https://api.sap.com/products/SAPS4HANACloud/apis/all Also
| edge cases are taken care of, sometimes before you even
| small updates, via notes, mostly tax regulation changes. So
| yeah, for these things SAP is really made for. If you have
| a SAP account, you can check out, just to see how many
| changes are coming in: https://launchpad.support.sap.com/#/
| solutions/notesv2/?q=pol...
| mirchiseth wrote:
| SAP and Oracle are always easy targets. Not sure about
| Oracle but SAP has a relatively newer (10 yr old)
| subscription and usage billing product -
| https://www.sap.com/products/subscription-billing.html
| jll29 wrote:
| In the SAP R/3 family of products, there is also a
| component Factory Calendar (IMG -> SAP NetWeaver -> General
| Settings -> Maintain Calendar, transaction /SCAL) - a
| component that knows about all the various calendar systems
| incl. public holidays of the world and the dreaded leap
| years. If I remember correctly, it's customizable for each
| customer premise with respect to whether each day of the
| year is salaried or not.
|
| This is a component classified as "basis technology" relied
| upon by many modules for billing (e.g. should a daily
| salaray be calculated in a country for a particular person?
| Should an invoice be considered "overdue" yet, based on the
| number of business days passed since issue? etc.).
| AnhTho_FR wrote:
| Hey @Brystefor, I'm @Rafsark's co-founder. We could have spent
| more time on idempotency, indeed. We have not deep dived into
| the 3 other aspects you mentioned indeed, we're actually
| planning to 1/ open-source the articles and add additional
| challenges as they are mentioned 2/ open our roadmap based on
| these inputs as well Thanks for sharing! Will ping you back
| here when it's live PS: we had not thought of financial
| regulation changes indeed!
| Brystephor wrote:
| Oh, I didn't even realize this was a blog for an open source
| project. I assumed it was someones personal blog.
|
| Two other things that you may want to consider:
|
| 1. PCI compliance. If you're involved in the storage of
| payment information, you may want to look into these
| regulations.
|
| 2. Regarding point 3 of your mission "...only integrating
| with specific partners to lock you in (e.g., Stripe Billing
| only working with Stripe Payments)...". I think there are
| many companies out there who want to integrate with more than
| a single payment provider (eg Braintree, Stripe, Adyen, etc)
| but run into this problem. It increases the cost of payments
| due to the "rent-seeker" charge method they use. Providing a
| design framework for how a single company could integrate
| with multiple providers could be beneficial. This is
| something I've considered trying to build, but I think it
| could fit with this initiative.
| AnhTho_FR wrote:
| Hi @Brystephor! I'm Anh-Tho, one of the cofounders of Lago.
|
| We're building an open-source alternative to Stripe Billing
| indeed, but we want to share our personal experience, hence
| the personal blog post.
|
| We'll open our repo very soon and our public roadmap too.
| Thanks for adding the 2 points, we're working on 'a design
| framework for how a single company could integrate with
| multiple providers could be beneficial.' indeed!
|
| Will ping you back here when we're live!
| robot wrote:
| Problem is it still requires an API to integrate. Stripe has all
| these capabilities but API integrations are needed to use them.
| If you are looking to build a micro SaaS without needing to deal
| with billing API integrations check out our software:
| https://saasbox.net. Built for completely eliminating any billing
| related SW development. It doesn't handle all the corner cases
| mentioned in the article, but some of them are handled, such as
| plan upgrade / downgrades with pro-rating, editing plans on the
| fly, migrating users across plans, notifying your application on
| those changes.
| Rafsark wrote:
| Why is it a problem to use an API?
| vikeri wrote:
| My understanding is that Stripe billing can be managed
| completely no-code from the Stripe dashboard? And I think it
| includes automatic tax, prorations etc.
| pizzaknife wrote:
| first sentence. homie rip
| 0xbadcafebee wrote:
| Don't spend time, effort and money on things that do not bring
| business value. Any time I see a company with engineers building
| something they could buy off the shelf [and the thing they're
| building is not the product] I know their priorities are whack.
| I've worked for several companies that had more money than
| brains. They'd spend _years_ , and millions of dollars in
| salaries, to build shit they could have bought and implemented in
| two months. And not only that, but multiple iterations of
| _failing_ to build said thing, by _multiple teams_.
|
| Just because a tractor is used for farming doesn't mean a farmer
| should build her own tractor.
| afarrell wrote:
| At some companies, it takes more effort to go through the
| procurement process than to build the thing.
| kdeldycke wrote:
| The essence of MVP I guess.
|
| Now, back to the article's topic, the real question is: is
| there an off-the-shelf billing product available out there?
| indymike wrote:
| > Is there an off-the-shelf billing product available out
| there?
|
| There are lots and lots of OTS billing systems. They might
| work the way you want or need. Billing is the classic "I
| wanted wheel, except I wanted a round one" kind of problem.
| Right now, I'm looking for a system that will let sales
| invoice whatever they put on paper, and be good a structured,
| recurring billing that is easy to hook up to my SaaS
| platform. Oh, and Bob in sales wants to be able to edit the
| invoices after they are issued...
| janci wrote:
| I was working on a billing system for utility company and it was
| a nightmare but not for the reasons from the article.
|
| Dates are no problem, everything is billed at the end of month,
| quarter or year.
|
| Upgrades are downgrades are simplified, customer can legally
| change tariff only on predefined dates/events.
|
| Usage - yes, but exampled in the article do not even scratch the
| complexity of this
|
| Idempotency is not really needed, the billing process is not
| continuous, it is a batch job.
|
| Cash collection is out of scope of billing software.
|
| Taxing was clear. Yes, there were multiple billable items with
| different tax rates, but not too complex. All customers were
| local.
|
| The nightmare part was:
|
| - customers can have multiple consumption locations, location can
| have multiple meters and customer can request to split the bill
| on multiple invoices or combine multiple bills to single invoice
| as they please
|
| - meters can be replaced at any time
|
| - meter readings are available on dates totally independent from
| billing cycle. Most of consumption data are mere forecasts.
|
| - when the actual consumption data is available, everything must
| be recalculated AND compensated (on a correction invoice showing
| the difference)
|
| - actual consumption data can be wrong and may be corrected at a
| later date, even multiple times
|
| - consumption points can be added or removed or moved to
| different customer at any time, but this information is only
| available after the fact
|
| - the prices can change mid-billing cycle for some customers but
| the consumption data is not available with that granularity
|
| - customer legal information (company name, address) can change
| mid-cycle
| politician wrote:
| Did the company use a relational database to deal with the
| billing and subsequent adjustments? I've been there: it can be
| more straightforward to use an event streaming approach that
| recalculates billing as new events arrive.
| janci wrote:
| Yes, almost everything ended as temporal or bitemporal model.
| Pain to work with. Even for simple questions like "What is
| the customer #1234 company name?" the system needs to know
| when are you asking.
| gnat wrote:
| By "temporal model" do you mean relational tables of
| "customer X used Y amount of this resource at price Z from
| timestamp A to timestamp B" or ... ?
| benwilson-512 wrote:
| In field metering situations you can have the timeline of
| each meter, when it records data points, and then you
| have the different timeline of the backend service, when
| it actually _receives_ data points.
|
| Bonus points if the meters can send data out of order.
| gnat wrote:
| Ah, thanks. OMFG and I thought our billing was hard.
| You're in my thoughts and prayers!
| GrumpyNl wrote:
| Designed and build the system meter readings and calculations
| for the Dutch market, all mysql, worked like a dream.
| AnhTho_FR wrote:
| Thanks janci for sharing your experience!
|
| Indeed, we only scratched the surface of the complexity of
| metered billing, we'll do a deep dive soon on this topic (it
| does deserve its own post).
|
| I think for the'nightmares' you mention: - some might be
| specific to utility (not applicable to an online business),
| such as '- meter readings are available on dates totally
| independent from billing cycle'. - Some topics might be simpler
| for a utility co: tax for instance (you know where your users
| are, and they are limited to a geographic region)
|
| - But some nightmares such as '- the prices can change mid-
| billing cycle for some customers but the consumption data is
| not available with that granularity' do really resonate for
| online businesses too, indeed
|
| Thanks for sharing, great insights for our future post!
| formerkrogemp wrote:
| For online businesses as well, selling above a certain amount
| of revenue or product can trigger what's known as a nexus
| event in taxation wherein the business becomes liable for
| collecting sales or use tax. Now online businesses are on the
| hook for collecting sales tax in every jurisdiction above
| wildly varying threshold values. Should the business fail to
| collect the sales tax for selling from Montana to California
| for instance, then the business will be liable for sales tax
| not collected for products or services sold to California
| residents online even though the business might physically
| reside in Montana.
| _glass wrote:
| I was also consulting for a utility company. Billing was easy,
| we used SAP (lol). But CRM (where we also used SAP) was a
| nightmare. The different processes where a nightmare. I mean
| utilities are amazingly complex logistically. New metering for
| a single house build, when it's ordered, installed, etc.
| cletus wrote:
| Pretty much everything is more complicated than people (including
| engineers) think.
|
| Take a simple example: selling clothing in a retail store. Sounds
| simple right? You have an inventory of items and someone pays for
| it. Should be a simple transaction, right? Well, now you're
| dealing with:
|
| - How do you determine the price? There might be a retail price
| but there are sales, discounts, overrides for various reasons (eg
| defective goods), etc;
|
| - You have to handle different payment options eg cash, credit
| card, debit card, gift cards, store credit. Payment methods can
| be mixed;
|
| - You have to handle returns. These generally have to go back to
| the original form of payment. Some goods might not be eligible
| for a return. What if the payment is split between store credit
| and a credit card? Where does the balance go?
|
| - Identity of the person making the sale or handling the return;
|
| - Authorization procedures for returns and price overrides.
|
| Everything is complicated.
|
| As an aside, this is one reason why I'm so bearish on Web3 and
| smart contracts in general. The edge cases are so complex and
| possibly unknowable that codifying these, particularly on an
| immutable blockchain, to remove the need for human intervention
| seems doomed to failure.
|
| Take Web3 identity. If you lose your password there are methods
| for recovering your account. The first is a reset password
| option. That may be insufficient and there'll sometimes be a
| human avenue to recover your account. Now consider a Web3
| identity. I've seen various incarnations of these such as
| authorizing other identities who with consensus can recover your
| account. Well, those identities can be compromises so you've just
| added an attack vector. Alterantively that recovery mechanism may
| become insufficient as people lose identities themselves.
|
| If these problems weren't complex they'd be solved problems.
| AnhTho_FR wrote:
| Hey Cletus! I'm Anh-Tho, one of the cofounders of Lago.
|
| Totally agree with you on web3 identity and its edge cases,
| it's just billing ^10000 in terms of complexity.
|
| I think for 'billing', usually people (especially non-tech
| people), don't understand the different blocks of the pricing
| stack: authorization, pricing grid, billing itself, invoicing
| (they often only see end-result: a receipt or invoice), payment
| gateways, dunnings, accounting etc). This post barely scratches
| the surface of it, but we intend to write and deep-dive more on
| each of these building blocks.
| logicalmonster wrote:
| > As an aside, this is one reason why I'm so bearish on Web3
| and smart contracts in general. The edge cases are so complex
| and possibly unknowable that codifying these, particularly on
| an immutable blockchain, to remove the need for human
| intervention seems doomed to failure.
|
| I do think the potential for errors is currently real (people
| mess up simple wallet transfers all of the time) but it's
| reasonable to assume that big things and making them easy for
| normal people to use is going to take time.
|
| It's easy to forget how ridiculously early into crypto humans
| still are. We're only about 14 years into the invention of
| crypto. We're only about 7 years since the release of Ethereum.
| The general public barely understands what crypto is other than
| having some conception of it as some kind of Internet money.
| Most people cannot discuss what a blockchain is, what smart
| contracts are, or why oracles are important. And forget the
| general public for a minute: even smart technical people on HN
| can barely give a good description of how all of these pieces
| work and fit together.
| trompetenaccoun wrote:
| >If these problems weren't complex they'd be solved problems.
|
| That's an axiomatic and therefor meaningless statement. Knowing
| something is hard to solve doesn't mean new tech won't bring
| improvements regarding the problem.
|
| It doesn't follow that having an immutable ledger means it has
| to be used for every single application. That would be like
| claiming trains are useless because they can't climb stairs.
| They don't have to do that, we keep walking the stairs just
| like we've always done.
|
| Immutability is great in situations where you want to record
| that something has taken place and don't want disputes about
| this. Some of the things you mentioned can be automated to some
| degree. Take returns as an example: You sell the item with a
| unique token that makes it identifiable and everything else can
| be automated, including chargebacks. We already have this to
| some extent with bar codes and RFID tags, but it can be taken
| to the next level with a non-fungible and non-falsifiable token
| that's connected to a smart contract which automatically
| returns money, does the accounting, updates the inventory,
| reports the tax data and so forth if authorized by the right
| person. As the owner you can easier monitor and analyze the
| flow of money, better detect fraud and come up with a myriad of
| improvements. Let's assume we have autonomous delivery trucks
| soon, then the entire supply process could be automated,
| including the billing as soon as the product is accepted. With
| something like the Helium Network shipments can be tracked, so
| there can be no dispute over where they are located.
|
| For company owners who really doesn't want to spend much time
| at the office you could go a step further and even set
| conditions for disputes like contracts triggering a certain
| action after a suppliers products have been rejected too many
| times because of quality issues, or vice versa is a customer
| keeps ordering and returning products from a factory. The
| contract could switch to automatically ordering from a
| competitor.
| [deleted]
| ben7799 wrote:
| I agree with thesis but there are entire aspects of this that he
| didn't even touch upon. There is a multiverse of different
| billing nightmares for engineers.
|
| I worked on a piece of software that bridged billing data from
| IBM mainframes out to the web. The hoops that had to be jumped
| through to get data out of the mainframes and the number of hacks
| and kludges involved was legendary. Likely everyone here has paid
| many bills in their life on websites that used that software. At
| one point I knew the vast majority of my personal bills had gone
| through it. Credit Cards, Utilities, Gas cards, etc.. Stuff
| everyone had to pay with paper before the web.
| colejohnson66 wrote:
| Could you expand on what kind of hacks/kludges were required?
| ben7799 wrote:
| Basically the mainframes were almost incapable of any data
| exchange that was meaningful in the 21st century. Instead of
| IBM doing something same the market just built software to
| scrape bills off the mainframes printer output, which you
| could intercept in electronic form. But then every accounts
| bill files were different. And sometimes the mainframes moved
| data around on the page, and so on and so forth.
| schwinn140 wrote:
| This is some brilliant, next-level, content marketing here.
| Nicely done.
| nico75 wrote:
| Solid job unearthing the numerous billing system challenges
| (almost triggering PTSD from prior experiences interacting with
| such systems...). While I haven't fully checked out the details
| of the solution you offer, I'm excited to see builders embarking
| on building better solutions :huggingface
| spookthesunset wrote:
| I occasionally get PTSD flashbacks from my time maintaining a
| home brew billing system.
|
| That shit is _hard_. There are so many ways to fuck it up
| because you didn't know what you were doing. The list of
| unknown unknowns for a billing system are huge unless you are
| an actual expert in billing, which you aren't and neither is
| anybody in your team.
| perlgeek wrote:
| I work at a b2b company with a 25+ years history, and OH MY
| GOD...
|
| * Sales is payed (partially) off commissions, so they do their
| very best to sell whatever the customer wants, not what the
| engineers know how to bill, or that product management has
| prepared. It's really hard to push back against a contract with
| revenue 10x your yearly salary "just" because the new features to
| bill it don't fit together with existing features.
|
| * It's very hard to keep track of which customer pays for what,
| exactly, and is entitled to which service. Easy for standard
| products, hard for bespoke products, nearly impossible for shared
| services (it took us >9 months to sunset a shared HTTP proxy that
| was in use by just 4 customers that didn't even pay for it...)
|
| * Legacy contracts: imagine 25 years iterating on product design,
| and keeping the old contracts running (except when the customer
| leaves on their own). Some of the old contracts are hard or
| impossible to find, nobody wants to spend their time rummaging
| through them to find out what exactly a customer is entitled to.
| A year ago we still had a customer that still pays for 8EUR per
| GB HTTP traffic (web hosting), because that was the going rate
| back in the days
|
| * Usage-based billing: half of our customers want to be flexible
| and only pay for what they use, and the other half uses SAP and
| for them, invoice amounts changing from month to month are the
| pure horror. So clever sales people invent hybrid models ("pay
| for what you use, but fixed price until a certain threshold,
| we'll notify you when you get close to reaching the threshold").
| Another source of complexity.
|
| * Usage-based billing: it's basically a topic for the whole
| company, most departments provide services that needs metering,
| but metering is usually an afterthought to the architecture, hard
| to verify for somebody on the outside
|
| * Usage-based billing: for big customers you cannot put all the
| details on the invoice (our mailing service provider is limited
| to 99 pages per invoice...), so you need separate reporting to
| drill into the details. Fire and brimstone will rain down if
| reporting and invoice disagree on some numbers...
|
| This just scratches the surface. Customers that want to split
| their bills along cost centers that don't align with access
| control tenants, different electricity prices based on location,
| the list is endless.
| Rafsark wrote:
| The article just scratches the surface too. I could have
| written a book to describe it!
|
| Reporting for revenue is a hard game, indeed. Spent a lot of
| time trying to reconcile revenue streams in the same company we
| ended up building the whole billing system. It's also not easy
| because metering is not something that is predictable nor easy
| to make it fit into a finance report.
| christophilus wrote:
| This has been my life for the past few months. Billing and dates
| / times are the worst part of the job.
| AnhTho_FR wrote:
| Have you looked at existing solutions? Why did not they fit?
| gen220 wrote:
| Not the OP, but anybody who has built a deep integration with
| any billing provider (inclusive of Stripe) knows there are
| some bodies buried, although you typically don't discover
| them until you're late in the game.
|
| 90% of your use cases are covered, and with excellent UX and
| docs to boot. But that sweet 10%!
|
| ---
|
| For a concrete example, Stripe has a very nice "subscription"
| abstraction that makes it easier to operate a SaaS business.
| They also offer a nice "tax" product, that doesn't do filing
| for you, but does offer tax computation if you code your
| products appropriately. Sounds great!
|
| But if you want to sell 2 subscriptions to the same customer,
| and those 2 subscriptions are fulfilled in 2 distinct taxable
| regimes (say, you're shipping widgets to NYC once per month
| and to a vacation home in NH once per quarter), you're
| screwed because Stripe only allows 1 shipping address for
| customer for tax purposes, and each subscription references
| this address when computing tax.
|
| We went through support, which filtered through to
| engineering, and the eventual answer was "no plan to change
| this anytime soon", which is difficult to interpret any other
| way than "good luck". :)
|
| There is no clean or even acceptably-dirty solution to this
| problem, tragically. So as a business we can't allow
| customers to transact with us this way. Oh well.
|
| There's another, similar, story around creating draft
| invoices to give people quotes before checking out. The
| dashboard API has a function to modify the line items of a
| draft invoice, but it's not a public-facing API and there's
| no short-term intention to make it public-facing, so one has
| to build an ugly workaround.
|
| This effectively means one needs to maintain internal models
| for subscriptions, products, taxes, invoices, invoice items,
| products, shipping addresses, on and on. And Stripe becomes a
| somewhat-dumb processor for these models (although the state
| machine for subscriptions is admittedly very valuable).
|
| I'd love to "offshore" everything to Stripe, status as the
| source of truth for data included, but it's not feasible in a
| surprising frequency of scenarios.
|
| Caveat that _every_ payment provider is like this, Stripe is
| much better and actually improves over time.
| Rafsark wrote:
| I think it's easier for billing systems to have 1
| subscription for 1 customer. The reason behind this is
| mainly because you can apply several prices to the same
| feature belonging to two different subscriptions. Stripe
| does not work on it because they might consider this as a
| edge case.
|
| When you look at usage-based billing, even if a billed
| feature is linked to consumption, it needs to be related to
| a subscription given the fact that this feature can be
| priced differently for several plans. Every company has a
| singular pricing model in mind and it's hard to have an
| all-in-one model fitting all the needs. Even when we built
| internally the whole system for a fintech back in 2016, we
| had some edgecases creating conflict inside the company.
| gen220 wrote:
| It definitely is an edge case.
|
| But I'd push back on the 1 subscription per customer
| idea. Consider any physical fulfillment business, where
| you have an optional annual membership (amazon prime,
| costco, whatever) and N subscriptions of varying
| frequencies for physical products.
|
| It doesn't make sense to roll these into a single
| subscription on the backend (unless the periodicity
| happens to line up perfectly, and even then there's a
| heated debate, since you still probably want them to be
| separate charges).
|
| You can go down the route of creating N "virtual"
| customers for each "real" customer, but it doesn't really
| solve the problems, because these N customers have a
| complex relationship with each other that you need to
| encode and manage.
|
| Hence, the name of this post :). It's definitely not
| easy!
| christophilus wrote:
| This is what I'm doing, but then you don't get a clean
| history in Stripe, which is annoying to folks (e.g. to
| our support team). So, it's something I'm thinking about
| changing.
| phphphphp wrote:
| A lot of the time there's trade-offs that can be made that
| engineers don't even consider because they're not pure. For
| example, why do you need a one-to-one relationship between
| your customers and Stripe customers? I am guilty of it
| myself, where an 80% solution is unacceptable from an
| engineering perspective but, actually, when balanced
| against the cost of building out a perfect solution, the
| business may well find it very acceptable.
|
| One of the most important things I've learned about working
| with third-party systems is that there has to be some
| amount of surrender to their way of doing things.
| Essentially, the price of using Stripe is not being able to
| use invoices as a quote, and that has to be articulated to
| the business. If that's not acceptable, then the business
| should be considering the requirements in aggregate when
| evaluating a solution, rather than deciding on Stripe and
| then gradually re-implementing parts of Stripe internally
| because they aren't right for the business.
|
| (Obviously that's easier said than done, and for engineers
| it's a fun challenge to make something impossible possible,
| but it's almost always a mistake and as engineers it's our
| responsibility to articulate the burden of owning
| workarounds.)
| gen220 wrote:
| The example you give is a smart one that we did explore!
|
| Effectively, relationship is no longer user -> stripe
| customer, but user -> shipping_address -> stripe
| customer.
|
| We decided the cost of implementing and maintaining this
| solution wasn't worth the benefit to the business. A
| credit to your later points.
|
| We came up with a solution that allows us to ship one-off
| things to arbitrary addresses with correct tax
| calculation, and if a customer really wanted to setup a
| subscription to a distinct address, we could implement it
| in the future, hackily, with this primitive.
| dqv wrote:
| >One of the most important things I've learned about
| working with third-party systems is that there has to be
| some amount of surrender to their way of doing things.
| Essentially, the price of using Stripe is not being able
| to use invoices as a quote, and that has to be
| articulated to the business.
|
| We have this same limitation on another payment platform.
| We just change the concept on our end to "payers" and
| allow an account to have multiple payers. When the user
| adds a payment method, they can either select that
| existing payer or, if they need to change the name or
| address, they create a new one.
|
| The provider we use had an invoicing product, but it has
| several misfeatures that just make the product
| unworkable. We absolutely have to be able to enforce
| first-in-first-out invoice payments. Users _will_ pay the
| May invoice before the April invoice and pretend like
| they are all paid up. I 'm not sure why they didn't
| consider this with a subscription-invoice product...
| users paying the invoices out of order was a big problem
| because users saw it as an exploitable vulnerability they
| could use to get a free month here and there.
| lfittl wrote:
| > There's another, similar, story around creating draft
| invoices to give people quotes before checking out. The
| dashboard API has a function to modify the line items of a
| draft invoice, but it's not a public-facing API and there's
| no short-term intention to make it public-facing, so one
| has to build an ugly workaround.
|
| There is a similar "not a public-facing API" limitation
| with configuring multiple invoice email addresses for
| Stripe customers (works in the dashboard, but blocked in
| the public API, where you can only set a single email per
| customer).
|
| I'm actually quite surprised Stripe does this, since their
| initial reason for being was developer-friendly payments.
| But oh well, maybe I should take a look at an alternative
| (like Lago) sometime.
| AnhTho_FR wrote:
| We're working on making the quoting process simpler
| indeed. What we've learnt that the only companies that
| can afford a proper quoting system are enterprises: they
| implement Salesforce CPQ (configure price quote) or
| Zuora. They are mostly sales-driven, so the investment is
| worth it.
|
| But it does not make sense for PLG companies, who start
| with self-serve and layer sales afterwards: the
| investment is not worth it, and it creates 2 different
| systems: self-serve and sales-driven billings, not great
| for the user experience (and for the internal processes!)
| my_usernam3 wrote:
| Reading this article gives me a smidgen of sympathy for Xfinity.
| They have so many plan options/timed promotional deals, and seem
| to run into so many issues every seemingly minor change, now it
| makes sense to me why.
|
| But in the same breadth still ... F** Xfinity
| AnhTho_FR wrote:
| Anh-Tho, one of the co-founders of Lago, here! yes totally,
| each time I open a pricing page, I have full sympathy for the
| engineering team who made it happen! I used to be the one
| tweaking with pricing plans on a spreadsheet and handing it
| over to engineers in a 'just make it happen' fashion!
| bcrosby95 wrote:
| I've had to deal with subscription based billing, and in my
| experience it wasn't a nightmare. There are some corner cases,
| but, those exist in every domain. And the stakes are definitely
| raised since you're dealing with people's money.
|
| Usage based billing sounds like a nightmare though.
|
| Taxes in the USA are a nightmare too. I have a friend that lives
| on a street, and for one side the tax rate is 7.5%, and the other
| side its 9.25%. Literally no company we tested got this right.
| Even Amazon.
| Rafsark wrote:
| For simple subscription-based billing, it's somehow not this
| hard, you are right. It also depends on how much plans you
| provide to your customers, and you still need to scratch your
| head for upgrades and downgrades.
|
| I truly believe in usage-based billing, so I do think pricing
| are getting more and more complex over time.
|
| Taxes are a nightmare for everyone also (in Europe too...)
| spookthesunset wrote:
| Dont forget price changes. Gotta deal with that. Then
| discounts, trials, weird durations, and a bunch of other shit
| I forget.
|
| My advice for anybody thinking of building their own
| subscription management system is to stop and go look at
| something like Zuora. You do _NOT_ want to get stuck
| maintaining some home grown subscription management system--
| it will never be better than Zuora and you 'll always be
| playing catchup, leaving your company unable to quickly react
| to the market. All the time spent adding functionality you
| can buy from a third party is money that you could have
| invested in your actual product. And worse, the fact you
| couldn't quickly react to the market because some key feature
| was missing in your homebrew thing is literally leaving money
| on the table.
| awillen wrote:
| Once upon a time I was the first product person at a now-
| decacorn, and my first task was fixing the billing system. It was
| quite the monster, and we ended up implementing a combination of
| Zuora and an internal system, as there were some parts of the
| billing model Zuora couldn't handle.
|
| I came away from this with one big lesson - if you're considering
| a complex billing model, consider the engineering implications
| first. With most products, engineering feedback gets taken into
| account - often product proposes something, engineering breaks it
| down, product realizes that feature x is vastly more complicated
| than they thought and not worth the effort, and the requirements
| are changed to simplify or remove it.
|
| The one thing that never seems to be true of is pricing models -
| that decision gets made at the very top and handed down with no
| chance for feedback. I think that if it the folks designing the
| billing system realized the costs, they might simplify things. If
| the complexity of your billing system means that 3% of your
| engineering team (plus additional folks in support and finance)
| is going to be working on it forever, but if you simplify it a
| bit you could keep 90% of it and only have 1% of engineering
| working on it, that might be a good tradeoff - after all, that
| leaves you more engineers to build features, which should drive
| additional sales. Unfortunately, that analysis never seems to get
| done up front and the cost is only understood after the billing
| system is deeply integrated into everything and would take an
| unpalatable amount of effort to change.
| Rafsark wrote:
| Couldn't agree more on this. Even if finance, sales, marketing
| or other deps are involved, billing is an engineering thing!
| Back in 2016, while building the billing system for qonto.com
| (european fintech unicorn), we were surrounded by people
| willing to add complexity to a thing they don't understand. A
| team of 2 engineers building it ended up in a whole squad
| called "Pricing Cross Functional Team"... even the name of this
| team was complex :)
| PeterisP wrote:
| Yes, this is a big thing - there needs to be a clear model of
| how variable pricing and discounts are going to work in this
| company, with the sales team being able to apply the actual
| amounts of any prices and discounts, but not arbitrarily
| changing the model.
|
| It doesn't work when the model is too simple or restrictive,
| which will simply result in the model being violated; you do
| need at least the ability to customize pricing for individual
| customers, and for specific invoices, but other than that the
| decisions (e.g. whether you apply invoice discounts as
| percentages or specific amount or both) can vary but you just
| need to pick any reasonable option and stick with it.
| awillen wrote:
| Yeah, this is tough because when the VP of Sales knows that
| one particular change will close the deal, they're going to
| push for it no matter what the cost. In reality it may be
| better from a big picture perspective to give a concession
| that is actually better for the customer and worse for the
| company than x as it relates to that particular deal, because
| the larger concession fits within the current billing system
| with no custom work, so the overall cost to the business is
| less.
| simonjgreen wrote:
| There's few things worse in B2B than encountering a system where
| engineers have reinvented general accounting principles from
| scratch.
| kdeldycke wrote:
| The only worthwhile re-invention of late might have been
| triple-accounting.
| awinter-py wrote:
| billing is a chronological typesystem and is only hard to build
| because most languages don't offer time-safe types
|
| (and because municipalities don't expose their tax rules as
| types)
| asciimike wrote:
| Nit: there's a lot of noise being made in comments about it being
| an open source product, but I haven't found the source (maybe I
| have to speak to an expert to get it?). github.com/getlago has
| the docs, which is nice (and gives a lot more info than is
| available on the website about how the product works), but not
| quite the same.
|
| Also note:
|
| > Our open-core version is forever free. We will introduce paying
| add-ons in the future, with a consumption-based approach.
| AnhTho_FR wrote:
| It's coming soon, we're in the final steps of QA, but we could
| not wait to share this post! Thanks for the comments on the
| documentation, by the way!
| asciimike wrote:
| Awesome, glad to hear, would love to take a look whenever
| it's available! Looking at the docs it seems like the current
| state of the product is much more about subscriptions than
| usage based billing (e.g. no aggregation or rating). Any
| plans to add those in?
| bhawks wrote:
| 'Tax logic' is an oxymoron and a trap for smart technologists to
| fall into. Tax codes are not logically consistent or clearly
| defined. They're a bunch of illfitting laws,
| beauracratic/regulatory guidance and interpretations of judicial
| decisions.
|
| I tell folks working in this area to accept the illogical nature
| of it all and to be ready for all sorts of arbitrary last minute
| changes. As the article points out understanding time plays an
| important role here too - when do different tax treatments take
| effect is an important question to answer.
| MarcinMas wrote:
| Yeah I can agree. I made custom integration with Chartmogul and
| it was a nightmare. But we did it! And now I feel like real
| engineer
| lelandbatey wrote:
| Billing is a nightmare, and if I had one piece of advice for
| people building a pay-for-what-you-use system like most SaaS,
| it's this:
|
| DO NOT bill against your business logic entities. They change,
| and doing a COUNT at the end of the month won't catch all things
| which changed or which cost you money _during_ the month.
| Instead, figure out what you bill for and and when you do that
| thing, record a row in a DB or into a steam of that "billable
| event".
|
| Reconciling billable events is much easier to do, and it's
| tolerant of all the weird edge cases (such as a support person
| hard deleting data when they should soft delete, which would
| otherwise throw off your end of month counts).
|
| There's a reason AWS (in general) can produce a log of everything
| you did which cost you money. It's painful, but it's less painful
| than the alternative.
| Rafsark wrote:
| Couldn't agree more! I do think the best way to do it is to log
| ingested events in a db, and decide which way you want to
| aggregate a billable feature at a defined billable period. This
| way lets you (i) keep track of everything, (ii) invoice complex
| behaviors and (iii) provide great granularity to your customers
| inside a final invoice.
| theptip wrote:
| Sound advice. Anything to do with accounting, you'll probably
| want to treat as an append-only log of events. (Note, you can
| also use event-sourcing and have your domain entities _be_
| events, in a billing bounded context that might make sense. Not
| usually the first approach I'd recommend though.)
|
| On a similar note, make sure you think about bitemporality as
| well. In other words, "effective_at" vs. ""created_at". You
| might learn that due to a bug, you didn't record a billable
| event, and you need to insert it into the history (say, to put
| it in the correct billing period). But setting the "created_at"
| in the past brings a bunch of issues; it's confusing and you'll
| soon hit issues where lying about created-time trips you up.
| (Migrations, reasoning about what version of the code applied
| what events, etc).
|
| Fowler has good further reading here:
| https://martinfowler.com/articles/bitemporal-history.html.
| bob1029 wrote:
| I think event sourcing / logging is _the_ way to solve this
| kind of thing.
|
| You have an obvious start/stop checkpoint in the log,
| processing can be done in batch after hours, no information
| ever gets destroyed, etc.
|
| Mutable database rows that are used for accounting purposes
| should sound dangerous to anyone trusted with these systems.
| bombcar wrote:
| And depending on how you do it, it's not as hard as it sounds -
| for example, the first minute of "cloud server V1" can get a
| row in the db, and each "period" that row gets _updated_ with
| the current minutes.
|
| Or you can log them into tables that get processed into other
| invoice tables.
|
| This also lets you keep grandfathered customers around as long
| as you want.
| flappyeagle wrote:
| If you squint, this is similar to how e-commerce billing is
| done. You have stock keeping units that represent the logical
| item being sold, and every transaction clones that row as part
| of the order record.
|
| This way, if the price changes, the item description changes,
| the images on the item change, etc, you can still have a record
| of what was _actually_ sold and how much to refund a return
| etc.
| rsanaie wrote:
| I'm getting PTSD reading this article
| Rafsark wrote:
| Ahah so sorry. Hope this is because of the substance, not about
| the form ;)
| api wrote:
| Any kind of business or accounting system tends to be an endless
| long tail of "can the invoicing system send the invoice in
| Esperanto using only 16-bit Unicode characters on Tuesdays but
| only while it is raining and only for customers whose last names
| end with E and who signed up more than one year ago according to
| the Chinese calendar?" type features.
|
| That's why attempts at "no-code" systems where biz/accounting
| people can design their own system are a perennial fad, but as
| these systems are developed they always just end up evolving into
| weird domain specific programming languages.
|
| It's also why any sufficiently large company ends up implementing
| their own in-house accounting system and/or customizing some
| byzantine monstrosity like SAP which is really just another way
| of implementing your own.
| nikanj wrote:
| The #1 reason those systems are moderately successful: When
| your sales person needs to write the logic for billing customer
| X, they'll try to make the customer contract more reasonable.
|
| Normally sales people are deal-driven, and if promising the
| customer "14% off on the first three transactions on every
| rainy Tuesday, unless Monday was also rainy" gets the deal
| closed, then that's going in the contract.
|
| No better way to ensure the contracts are reasonable, than
| making sure the person negotiating the deal feels the pain of
| billing it
| Rafsark wrote:
| Agreed! this triggers also complex behaviors because sales
| people want to handle this in their env (Salesforce, hubspot
| or whatever); You also have to build the logic to update
| subscriptions and pricing directly from it, in a two-way
| sync. Things getting crazier ;)
| biermic wrote:
| Absolutely agree with the poster. This is my life at the moment.
| Just to take money for a small piece of software I built.
|
| Multiple currencies, trial periods, when in the process do you
| ask for the creditcard, country specific taxes,... And the worst:
| invoices.
|
| While paddle.com takes care of many of those things - I was
| shocked after realizing how much work this is going to be.
|
| Please think carefully about those things before you quit your
| job to work on your "Micro SaaS" or "unicorn startup".
| Rafsark wrote:
| Agreed! But I do prefer to build it for me rather than someone
| else's startup, though
| dand31 wrote:
| Have you not looked at Stripe? Their product is fairly straight
| forward to set up
| herdrick wrote:
| Have you looked at Outseta? https://www.outseta.com/billing and
| https://www.outseta.com/demo . I haven't used it but it looks
| great. I'd love to hear what you think.
| akrymski wrote:
| Have also been considering Paddle. Would love to know what you
| found it's missing?
| micromacrofoot wrote:
| Ok now integrate billing with flight booking and timezones... the
| devil's stack
| dboreham wrote:
| Note OP makes a billing product (OSS, granted) so is motivated to
| characterize the field as complex and hard. In reality billing is
| just like many other processes humans engage in. For sure it's
| more complex than computing prime numbers, but hey try writing
| software to capture medical services processes :) In the end this
| is exactly what engineers are supposed to do: figure out ways to
| model/capture/represent complex processes and data.
| spookthesunset wrote:
| Or better, buy a billing system instead of inventing one
| inhouse. Billing is hard to do correctly. For one thing it
| interconnects with so many different platforms in your company.
| For another thing, the cost of fucking something up is high--
| people don't exactly like seeing mistakes in their bill. Hell,
| think about tax... you wanna deal with that shit?
|
| To do it "correct" you've got to have people who truly
| understand accounting concepts like double entry bookkeeping,
| financial reporting, cash vs. accrual, financial regulations,
| auditors, etc. Unless your company's product is billing, good
| luck hiring an engineer who knows this shit.
|
| But trust me from experience. Don't build your own billing
| system. There are several good ones out there that will grow
| with your organization. Buy one.
| piva00 wrote:
| Similar experience in some companies in my CV. Billing
| systems aren't fun to maintain, develop nor debug. They take
| a non-trivial amount of time when done wrong and if you are
| building one in-house you'll inevitably run into some
| mistake.
|
| Buy a billing system, it'll save engineering hours having to
| deal with it and if you choose right they'll scale with your
| company.
| jdenquin wrote:
| I totally agree with you, I had to work on the Qonto's
| billing system (that Raffi is talking in the blog post) and
| it wasn't fun to maintain. I remember the pain it was to
| change anything without breaking the whole system, not
| because the system was bad, but because it's complex and
| when you build it in-house, you will certainly take some
| shortcuts that makes it not so flexible!
| spookthesunset wrote:
| Not only will you get mistakes, but much worse in my
| opinion is the opportunity costs involved. You will always
| be playing catchup to add new (very basic) features that
| the real packages offer right out of the box.
|
| For example if your homebrew thing doesn't handle price
| changes, that might take you quite some time to build in
| functionality. That time spent is time where you couldn't
| quickly react to market changes, which is money left on the
| table. Had you bought something that supports a simple use
| case like "change the price", you'd hit a couple buttons
| and boom... your product now sells for a different price.
|
| A companies billing system is one of the most important
| systems in the company. It is literally how money gets into
| your pocket. If you build it yourself, you will sign up for
| pissing away a ton of time writing code that literally will
| let you collect _more_ money from your customers. Had you
| bought something, you 'd just fucking go do the change
| right away and collect _more money from your customers_
| almost instantly.
| jermaustin1 wrote:
| Could not agree more. Even building your own billing system
| atop something like stripe is super complicated. Rebill
| dates, proration, upgrade/downgrade, cancel, updating card on
| file, refunds, charge backs. These are all things you end up
| patching as you come across them.
|
| There was a really nice and simple to use billing software
| that was built on stripe. I used it for a handful of products
| after I decided to never roll my own again. It is gone now.
| GoDaddy bought them and shut them down.
|
| I think Stripe checkout is a valid solution now if you don't
| mind sending your customers away from your site, but I
| haven't played with it since they rolled it out.
| gopher_space wrote:
| "How to realize when you're about to build your own billing
| system" would be a more useful discussion.
| vishnugupta wrote:
| As you say; billing system is tip of an iceberg. There's
| reconciliation, settlement, payouts, tax, financial reporting
| and what not. For a small mom-and-pop store, sure build it.
| But for any half-complex business billing is a rabbit hole.
| dhosek wrote:
| This feels like a good moment to mention my most embarrassing
| bug which was, of course, in a billing system. I had written
| code to handle the overdue billing of customers' usage on an
| internet telephony service. The code seemed to work great in QA
| so we went ahead and pushed it to production.
|
| A couple days later we got an angry call from a customer whose
| card we'd maxed out. The final stage of the process was to
| adjust their bill by the amount that they paid, but there was a
| sign error in that adjustment so instead of lowering their
| balance by the amount paid, we increased it. As a consequence,
| for a user with, say, a $50 balance, we kept doubling the
| amount that we charged them each day.
|
| Exponential growth is a powerful thing.
| alfalfasprout wrote:
| I'm not sure this comment is exactly in good faith though-- OP
| does make a billing product but as a result is also uniquely
| qualified to provide insights into what can make it hard.
|
| This isn't OP trying to claim it's the most difficult problem
| to solve from an engineering standpoint. Merely why it's hard.
|
| Of course engineers solve problems, hard or not. That's not
| really being debated here is it?
| RockRobotRock wrote:
| I think this is a great article and sales pitch for the
| company, since a lot of the comments have been anecdotes of how
| its actually even HARDER than the author leads on.
| [deleted]
| [deleted]
| AnhTho_FR wrote:
| Hi dboreham, I'm one of the co-founders of Lago here (the OP).
|
| I completely agree with you, billing is not the most complex
| process of all, medical services may be of higher complexity.
|
| Our post was meant to highlight the discrepancy between the
| perceived low complexity (lots of teams who have never done it
| think it's simple) and the reality, with our own experience
| building a fintech.
|
| I think for some processes (medical services? I'm not an expert
| in this field to be honest) people might suspect it's going to
| be challenging.
|
| That was our intention. Basically we think no B2B saas should
| be building billing themselves, unless they are 300% sure their
| pricing will always remain subscription based only, and very
| simple (same amount every month).
|
| Hope I helped clarify!
| [deleted]
| zippergz wrote:
| I have no vested interest in selling billing products, and my
| experience is that billing is somewhat unique in the delta
| between how hard people who have never done it THINK it is, and
| how hard it actually is. There are many hard things that appear
| simple, but time and time again I have seen inexperienced
| engineers be caught off guard by just how much harder billing
| is than it looks.
| Aeolun wrote:
| If it's as annoying as translating human processes into code, I
| can see how they'd want to standardize it so they'd never have
| to do that particular part again.
|
| Then again, if you make it too customizable, you end up with
| current single sign on solutions.
| Andrew_nenakhov wrote:
| As someone, who has built a billing system not once in my life,
| but twice (one for an internet privider I worked for, which
| counted _the amount of traffic_ , and another one for a SaaS
| project(, I fully sympathize with the post. Billing is an
| unbelievable can of worms even before you get to taxes. Add in
| all the things the marketing people want from billing (trials,
| discounts, per-seat per usage pricing, etc), and you have enough
| tasks to last till retirement, no matter how young you are.
| Rafsark wrote:
| Yep, do agree that non tech team created a lot of pricing
| complexities, sometimes without any reasons. Remember a
| marketing team updating plans on webflow and telling the
| product team << see, not that hard >> ;) Ended up with a whole
| squad of engineers called << the pricing cross functional team
| >> ...
| londons_explore wrote:
| Random idea:
|
| Don't build a billing system. Or at least not a precise one.
|
| Have a price list as normal, but just guestimate the user's bills
| based on any easily accessible information. Eg. For a hosting
| company I might just count how many VM's they have active when
| the billing script runs . If the user isn't happy with the total,
| give them a button to correct the total and pay that.
|
| Spot check what the user's pay, and warn then ban any users which
| are clearly deliberately paying substantially less than list
| price for your services.
| CharlesW wrote:
| How does auditing work if billing is non-deterministic?
| AnhTho_FR wrote:
| I love it, sounds like the Swedish shops with no cashier
| https://mashable.com/article/swedish-store-cashiers Has it been
| tried before? How were disputes managed?
| Hackbraten wrote:
| You'd still have to apply the right tax rate for every line
| item. If you allow the user to edit the total, then which of
| the tax amounts are you going to recalculate?
| __alexs wrote:
| This is a great way to be bankrupted by crypto currency miners.
| recursive wrote:
| This sounds like trouble at several levels.
|
| Someone's going to reverse engineer your heuristic and min-max
| it. At that point, you'll need to defend your heuristic against
| edge cases. And then you may as well just make it official.
|
| Also, B2B customers are allergic to "pay what you feel"-type
| schemes.
| bokohut wrote:
| As a co/founder and core architect of multiple PCI level 1
| compliant payment processing companies in the last two decades I
| will add as others have that billing and payments is exactly like
| anything else which is "difficult". Once you spend enough time in
| the trenches dealing with the challenging technical issues your
| experience and creativity leads to the proper solutions that
| further enhance the system. Everything is just inputs and outputs
| yet it is what one does in between those "puts" which matters and
| thus where experience can reduce such challenges.
| walnutclosefarm wrote:
| Always has been. Marketing and sales create complexity on the
| front end, devising all kinds of products, plans, discount
| schemes and related complexity; jurisdiction, taxes, and
| accounting rules add complexity on the backend. In the middle,
| measuring what you're billing for is the hook around which all
| that complexity chaotically orbits.
|
| Decades ago - the computer involved was an honest to goodness
| System 360 (which is to say, not a 370, Z, or any of the the code
| compatible 360 successors) - I spent an evening trying to figure
| out the service charge on my bank account. I couldn't get at all
| close to the figure on the statement by applying the bank's
| schedule of charges. I took it into the bank, and got bumped by
| the teller to a department manager, and thence to a VP (small
| town; small bank), whom I told, "I will pay the service charge if
| you can explain to me what I'm being charged for." 2 hours later
| he gave up, having not gotten any where near an explanation,
| grinned, and said, "I can fix this." He set some flag on my
| account, and I never again paid service charges at that bank. Yep
| - that flag remained set on my account for nearly 20 years,
| through multiple software and hardware upgrades, until the bank
| was acquired by a big, expanding regional bank, and I left for a
| different, entirely local bank.
| Rafsark wrote:
| The marketing team of my previous company created a whole fake
| price plan that never existed in the code base. Somehow it
| works to drag acquisition, but it's a pain in the ** to explain
| why it's not as easy as updating webflow ;)
| [deleted]
| hammock wrote:
| Double-entry accounting is tough for a lot of people. It's a
| different domain and trying to fit it into traditional "math"
| will only cause headaches.
| tabtab wrote:
| An alternative to DE is "Assume Balance". See near the bottom
| of http://wiki.c2.com/?AccountingModeling
| kdeldycke wrote:
| Still, someone tried in the "Algebraic Models for Accounting
| Systems" book: https://www.amazon.com/Algebraic-Accounting-
| Systems-Salvador...
|
| Source: https://github.com/kdeldycke/awesome-billing#finance
| daxaxelrod wrote:
| Poked around their site a bit, they claim to be open source but I
| don't see a link to a git repo anywhere. Also searched GitHub a
| bit but might have missed it.
| AnhTho_FR wrote:
| Hey @daxaxelrod! I'm one of the cofounders of Lago, the lib
| will be opened very soon, we're in the final steps of QA,
| that's why. I will make sure to ping you back here when it is.
| We wanted to share this post about our first-hand experience
| with billing a bit ahead of this. Thanks for your patience!
| daxaxelrod wrote:
| Hi! Cool, can't wait to poke around. I've used killbill
| https://github.com/killbill previously and would be
| interested in your offering.
| AnhTho_FR wrote:
| Will ping you asap!
| fatnoah wrote:
| My very first startup (wayyyy back in 2000) built a billing SaaS.
| I fully agree that rating, billing, invoicing, and all of the
| related things are hard. I learned a lot in that role about
| dealing with complexity...and date/time.
| Rafsark wrote:
| What are the top 3 things you learned that every company should
| be aware of? Could be interesting to have an experienced view
| on this :)
| aerovistae wrote:
| > To solve this problem at scale, we adopted a radical sharing
| approach. We've started building an Open-Source Alternative to
| Stripe Billing (and Chargebee, and all the equivalents).
|
| This is so great. I personally haven't had to deal with this
| problem, but I've worked at a number of organizations (and heard
| about many others!) where this sort of business logic had to be
| implemented. It's just reinventing the wheel. I shudder to think
| how many companies have implemented a system for managing
| recurring subscriptions.
|
| We have things like Ruby-on-Rails, Django, Laravel, and many
| others to take the bite out of building web applications. They
| keep us from having to reinvent the wheel.
|
| We need similar open source frameworks for common business use-
| cases - billing, subscriptions, order/purchase management, and so
| on.
|
| Sometimes it's weird to remember we're in the stone age of
| technology - all of this is a few decades old at most, and even
| early predecessors don't go back more than 70 years. Human
| history goes back tens of thousands of years. There's so much yet
| to come.
| ptsneves wrote:
| This is where solutions like WooCommerce shine. WooCommerce is
| an amazing piece of software that is many times used for free,
| but generates thousands of dollars.
| intuxikated wrote:
| > We need similar open source frameworks for common business
| use-cases - billing, subscriptions, order/purchase management,
| and so on.
|
| You mean like an ERP? There's 2 open source ones I can think
| of:
|
| - Odoo, which is the bigger one, mostly Open-source (CRM /
| sales / subscriptions / invoices / webshop / inventory /
| purchase) but some enterprise-only modules (Rental, Field
| Services), I work with this software daily as an Odoo Developer
| (Customizing Odoo for customers' needs).
|
| - Erp-Next, which is completely open source as far as I can
| tell, through my limited testing it seems to be less advanced
| than Odoo currently.
|
| EDIT: you can even check runbot.odoo.com for some test-
| environments which are automatically built, where you can
| test/experiment, login is always admin:admin
| aerovistae wrote:
| I hadn't heard of that! Maybe that is what I meant.
| sandworm101 wrote:
| Do not worry. Blockchain will solve or at lease simplify all of
| these problems.
|
| It is the way.
| StevenWaterman wrote:
| Look, Poe's law!
| Rafsark wrote:
| Even if with blockchain and crypto you need to define a logic
| to bill your customer. This logic is pre-transactional and
| define how much you need to price the transaction.
| anticensor wrote:
| Adopt a _nothing is fungible, everything is serialised_
| approach and maintain real equivalence classes instead of
| assuming fungibility as a proxy of replaceability.
| oznog wrote:
| It's missing when you find out that business people don't know
| how the business must be executed with total precision.
| kache_ wrote:
| nightmare, or job creator?
| bloodyplonker22 wrote:
| nightmare job creator.
| Rafsark wrote:
| Job creation comes always because of hard-things to build; When
| getting easier, it cuts job!
| giantg2 wrote:
| I work on a team collecting fees at a financial company. It is
| tedious and boring. There is a lot of complexity. I've often
| asked the business if they had ever thought about a different fee
| model that would be less complex. They just want to stick the
| legacy business model into the new tech...
| AnhTho_FR wrote:
| I'm Anh-Tho, one of a co-founders of Lago, thanks for your
| comment! Financial and cloud infrastructure companies are the
| ones with the most complexity. I've been on the business side,
| and sometimes you do want to change the pricing but there's so
| many implications: - Maybe existing users will churn and/or
| revenue will decrease as a result - If you change the billing
| system, there's a risk of bugs/errors and complains - If you
| spend engineering time on this, then you need to deprioritize
| other projects So... business teams often end up giving up, and
| that's a shame because iterating on pricing is a very powerful
| lever of revenue growth.
| welder wrote:
| I implemented billing at WakaTime with Stripe, Braintree, and
| Coinbase as swappable payment providers. It's definitely a source
| of edge case bugs, but overall it works pretty well. It supports
| prorating subscriptions across providers, monthly & yearly
| billing, trails, discounts.
|
| * We don't remit taxes for customers because that would be
| impossible to build in-house, but everything else was achievable.
|
| * We have tons of logging, and generate transient Quotes on our
| end, then check that the expected charge matches what the payment
| provider actually charged.
|
| * Each Quote is reproducible, so if any bugs come up we can give
| the "create quote" code the same input and see what went wrong.
|
| * What helped when building was hourly tasks running in the
| background to make sure customers who should be billed were being
| charged, and make sure we didn't double charge any customers, and
| then alerts so we could find and fix those bugs.
|
| I would do it again to support both PayPal and Stripe, but it's
| definitely a source of complexity for the code base.
| hedgehog wrote:
| Can corroborate, my experience working on an upgrade to an
| existing multi-platform (iOS IAP + Strip for web product)
| subscription service was certainly a bit painful.
| dekhn wrote:
| Basically everything about enterprise is a nightmare for
| engineers. What amazes me (after having worked at some of the
| most profitable companies in the world) is just how little
| intelligence the leadership has about their own revenue or costs,
| beyond "wow huge amounts of money is coming in or going out". And
| how many critical processes are implemented manually, by
| individuals, with personal spreadsheets, on their laptops.
| cdkmoose wrote:
| Just wait until you have to do this under government controlled
| (tariff) environments. Years ago I worked on a tele-conference
| platform for large international telecom. We built the software
| to run all aspects of their platform including billing.
|
| Government tariffs covered billing rules for each leg of a call
| including rates based on caller's local time of day. Converting
| bridge time to local time via area/country codes(only data
| available) and time-zones is quite a mess. Add DST calculations
| to that per locale to reconcile negative duration legs or legs
| that should be 1.5 hours but show up as .5 hours just makes it
| cloudier.
|
| If your system repeatedly gets it wrong, you could be in trouble
| for billing fraud or tariff violation. No joy in that job.
| chasd00 wrote:
| I spent a lot of time right out of college working on the
| equivalent of an invoicing system for a pharmacy chain (rx drug
| pricing, electronic carrier submission, and reconciliation).
|
| Account receivables is also another nightmare. We would get
| checks that randomly show up at corporate for no reason from
| insurance carriers but then when the carrier realized their error
| instead of handling it as a separate process they would deduct
| the check amount from whatever invoice ( or even across a range
| of invoices! ) from us they felt like. We literally had a bucket
| called "magic money" these random deposits went into that we
| would use to fill in the A/R gaps from insurance carrier
| insanity. There was no connection between magic money and
| whatever invoice they decided to short change us on so it was
| just a hope-for-the-best process.
| po1nt wrote:
| Implementing and maintaining billing is my daily job for the last
| 6 years.
|
| I would also mention topics like: - Reporting (various data
| aggregation and audit reports) - High reliability, nothing hurts
| company more than unability to bill their customers - Rounding,
| this can be seen as a subset of "taxes" but it's much more
| complex - Locating user, also can be seen as a subset of "taxes"
| but user can have country A set in profile configurations,
| country B on credit card and country C geolocated from IP address
| - Timezones, issue everywhere but when we talk accounting it's
| super important - Talking to moronic payment gateway providers.
| This is the biggest ones. I would love to just flip Apple and
| Google like Linus flipped of nVidia. Proprietary, poorly
| documented, "find out yourself by weeks of experimenting"
| bullshit with no easy way of getting technical support even when
| you bring those companies millions per year. Things don't work,
| deprecate monthly or weekly and expect you to be always ready to
| make changes. Some implementation make zero sense at all with
| complete paradigm shift of handling payments like between Google
| in-app pay and Google subscriptions.
|
| But as my SO says "Don't cry, it pays well"
| AnhTho_FR wrote:
| Wow, thanks for sharing! We found out 'billing engineers' are
| very rare and really demanded, as it requires to be very detail
| oriented, technical, and business processes oriented. But even
| if there is high demand (and I guess: corresponding high pay,
| as you mention), very few engineers are up to the challenge!
| Rafsark wrote:
| Yep you are right, indeed. We could have mentioned
| Reporting/Analytics for revenue, which is always a huge pain
| for companies!
| mc4ndr3 wrote:
| "Should we build it in house?"
|
| No. No, spelled F-U-C-K-N-O.
|
| Should you compare and contrast off the shelf solutions, test one
| or two, and slowly expand its use to cover more of your business?
| By all means, yes.
|
| Should you develop an in-house custom variant after that? Still
| no. For billing or anything else.
|
| Even a pure tech company like Google should not vertically
| integrate everything. You quickly end up wasting time making
| horrid systems instead of generating your main business revenue.
|
| "Oh, but our needs don't align with the way the industry standard
| third party systems work." That's a symptom of the disease. You
| won't cure it by enabling the virus to proliferate even further.
|
| People who don't execute I.T. low level changes mistake computers
| for magic. Yes, computers can do a lot. But even the tiniest
| feature takes an incredible amount of focus, cost, development
| time, maintenance time, headaches, and gnashing of teeth to
| accomplish. The magic is that it looks easy, from the user's
| point of view.
|
| Let's put it bluntly: We can draw a direct line from third party
| systems with multiple businesses as customers, with cost savings.
| It's far more likely to be a complete waste of resources to in-
| house a new software system.
|
| Maybe you don't have a name for the kind of system you want or
| need. That's okay. Keep assuming it's probably already out there,
| and continually search for it. Googling is cheaper than building.
|
| "But we can build it better than them because..." Please, that's
| the fallacy of exceptionalism. Remember, every company wants to
| believe they're truly pioneering, talented, and logistically
| capable of doing all the things. Don't spread yourself so thin.
| Keep the lights on. Grow your customer base. Everything else is a
| pipe dream.
|
| "But the third party systems suck."
|
| Yes, they very much suck. And yet the cheapest path to success is
| pushing them to un-suck their product. Not starting from scratch
| to create a second, even more immature product.
|
| Frankly, computer systems are already quite flexible. You can
| often bend an existing one to fit your needs. This happens with
| Jira, for example. They customize it until it can't function at
| all anymore. But this sad extension situation is still cheaper
| and more effective for conducting business than invention.
|
| If you truly believe a new kind of system needs to be built, go
| and start a new software company. Or pay a software vendor to do
| it. (Software vendors, just take their money, smile, and nod.)
| But for Warren Buffet's sake, don't spend a dime on a new line of
| code if at all possible.
| einpoklum wrote:
| > Why billing systems are a nightmare for engineers
|
| My guess: Because basically nobody wants to develop one of these
| as a hobby FOSS project. The whole idea is collecting people's
| money, which when you write your own code for public benefit, you
| basically don't do :-)
|
| > We quickly decided to add plans, and 'pay-as-you-go' features
|
| So, the author talks about things getting more complex, beyond
| their expectations. But - why did they not expect this? I mean,
| they know that companies pricing changes over time; that payment
| models change, including rates, cadence, use of internal credit
| etc. We all know this without ever having worked on a billing
| system.
|
| > Qonto is a 'neobank' ... Fintech unicorn
|
| So they were expecting billing to be easy for a bank-like
| endeavor? Hmm.
|
| > #1 - Dates
|
| The date problems per se are quite mundane and any non-trivial
| software system which involves timed activity deals with them.
| The billion period problems are not really date problems and are
| a recap of what OP had already written about. They are also
| problems most of us are well aware of from our personal
| experience as people in modern society ("Does my bus pass expire
| a day after I bought it, i.e. 24 hours later, or does it expire
| at the turn of the day = at midnight?")
|
| etc.
| CSMastermind wrote:
| I just want to say that this article is a great intro to the
| topic and I credit a lot of my growth and development as an
| engineer to working on billing systems in my first job out of
| college.
| thibo_skabgia wrote:
| Love this comprehensive article. I've seen that pain in many a
| company I freelanced for. And it's much deeper than they even
| know.
| hkon wrote:
| Depends on your requirements I would say. If your target
| customers are global, then yes, you'll have to deal with a lot of
| rules and regulations. But the engineering side of things I don't
| think is that hard.
|
| Perhaps when you first enter the domain it seems like a lot
| because you are unfamiliar with it, but such is it with all
| things.
|
| Also, these guys have an economical incentive of convincing you
| it's hard. So keep that in mind.
| fhrow4484 wrote:
| Since founders are commenting here. getlago.com webpage is
| strange:
|
| "Lago is backed by Y Combinator"
|
| "The Open Source Stripe Billing Alternative"
|
| Why would YC invest in both Stripe and a competitor of Stripe?
| abraae wrote:
| To avoid the innovator's dilemma?
___________________________________________________________________
(page generated 2022-05-18 23:00 UTC)