[HN Gopher] Exiting the Vietnam of programming: our journey in d... ___________________________________________________________________ Exiting the Vietnam of programming: our journey in dropping the ORM in Go Author : henvic Score : 54 points Date : 2021-11-26 16:45 UTC (6 hours ago) (HTM) web link (alanilling.medium.com) (TXT) w3m dump (alanilling.medium.com) | [deleted] | marcus_holmes wrote: | So frustrating. You got like 80% of the way there, and then went | "nope, too much work" and diverted to add more complexity. | | The answer is to write the SQL yourself, and the scan methods | yourself. Code generation is better than ORM, but still a | wrapper, still adds complexity, and still brings problems. | | Yes it's a pain in the arse to write all that boilerplate in one | go (pun intended). But if you'd started without an ORM you'd have | written them one at a time as you needed them and barely noticed | it. | | Keeping your code aligned with your database schema is very | little effort - database schema changes are usually rare and | significant. | | I write a view for each access method (so I can change the schema | without worrying about changing every access method), and a | function for each update/insert/delete (for the same reasons). It | takes maybe 20 mins to write the whole set each time I have to | add a feature with new data, which is a rounding error in the | time it takes to write all the rest of it. | | The point is that the database schema is optimised to storing | data in the best way possible. The middle layer is optimised for | processing data in the best way possible, and the front end is | optimised for displaying data in the best way possible. None of | these three things are equal. Use an interface between each of | them. The interface is important and needs to be carefully | considered. | irq-1 wrote: | Putting all SQL into the database gets you two huge advantages: | First, all relations and dependencies are visible for DDL | changes, and what the code uses is easy to see without looking | through the code. Second, a test database can use (near) live | data with the new DDL and new code, making testing much better | especially for non-development staff who can spot issues based | on the data or UX. | e12e wrote: | > I write a view for each access method (so I can change the | schema without worrying about changing every access method), | and a function for each update/insert/delete (for the same | reasons). | | Database view and stored procedures? | marcus_holmes wrote: | yeah. Postgres has functions, SQL Server has sprocs. | 2StepsOutOfLine wrote: | I'm shocked more people arent talking about SQLBoiler in | threads like these. It solves this exact problemset. You write | the SQL schema and it generates all the scan and helper | functions for you. We've had a great experience with it at work | after running into similar woes as OP with ORM's. | | https://github.com/volatiletech/sqlboiler | mattcwilson wrote: | What are your thoughts on what the article author has to say | about SQLBoiler? | butMyside wrote: | What if there are no layers and you can't think outside | hierarchy? | | There's no reason for anyone else to take your view as | sacrosanct since many many many apps and services are working | just fine without ORMs. | | You might accomplish something interesting today if you set | aside protecting unimportant ephemera you talked yourself into | believing. | 5e92cb50239222b wrote: | > I write a view for each access method (so I can change the | schema without worrying about changing every access method), | and a function for each update/insert/delete (for the same | reasons). | | God how I hate supporting such architecture. It loses edit | history unless you're very careful (which many developers are | not) and do absolutely every piece of DDL in a versioned | migration, and makes DDL extremely painful because you | frequently have to drop two dozen functions and views (which | depend on each other recursively) and then recreate them. | aidos wrote: | We've started using views and friends a lot more recently | (now I've recovered from my PTSD from a lifetime ago when I | worked in a poorly managed sql server shop). As you point | out, they're often not first class citizens. | | On the flip side, we've just moved ours to files on disk (one | per object) and hacked in some tooling to autogenerate the | migrations from them (we use alembic / sqlalchemy which | already does a good job here). | | It's kinda the promised land. Edit code on disk, commit and | review as usual. | marcus_holmes wrote: | this. Don't write your SQL into the database. Write them as | scripts (starting with "drop XYX", then "create XYZ", check | them into git, and treat them as code. | | Migrations are for schema changes. Views and functions are | not schema. | languagehacker wrote: | We will never be able to separate this fine argument from the | fact that gorm has been terribly immature for quite some time. | ORMs take a long time to grow. Look at Arel and ActiveRecord. | Sufficiently mature ORMs provide addordances to work around the | fundamental N+1 and abstraction issues that always crop up, but | they take time, dedication, and a broad community. Go is an | interesting case in which there are enough useful applications | that don't comprise "web apps" that we might not be in at the | point of ORM maturity. I've used both gorm and go pg, and have | definitely encountered frustrating limitations coming from both | ends of the problem. | cjdell wrote: | In NodeJS Prisma is really good for just building statically | typed CRUD-like queries. It gives you result types inferred from | the schema so you're less likely to make mistakes. | | Would still use real SQL for serious stuff, but in my experience | most queries are CRUD anyway so I don't see lightweight ORMs as a | problem as long as they're not doing magic caching in the | background and lazy fetching. Let me decide how I want to do | that. | topicseed wrote: | Using MikroORM in NodeJS and works wonders. Has the "raw" | option for queries I want to handle myself, or even the "knex" | builder for something in between | cjdell wrote: | Hmm... Looks like a "active record" pattern (like Hibernate). | This is exactly what I don't want. Essentially I only use | ORMs as query builders. No magic behinds the scenes. | dekhn wrote: | This article would have been better without the unnecessary war | analogy and history lesson. | dmos62 wrote: | The funny thing is that it references Vietnam, while the | Afghanistan exit happened just a few months ago. | lioeters wrote: | Indeed the analogy has not aged well. It's in bad taste, and | somewhat offensive, to hear the term "Vietnam of programming". | [deleted] | tommek4077 wrote: | Why did it not age well? Are there any new findings regarding | Vietnam? | quesera wrote: | To most people in the world, "Vietnam" is not a war, and is | not merely a symbol of absurdly misconceived incompetence. | hinkley wrote: | There's two reasons a problem stays unsolved. Perhaps three if | you believe Jim Highsmith (we are managing paradoxes). One, it's | unsolvable, like the Halting Problem. Two, it's an XY problem and | nobody is asking the right question. | | I'd been thinking recently that you could build a stupid simple | ORM if you set up your SQL results to have the exact same names | as your target object/struct and fed the type in as a parameter. | Push these results into this box. | | But that's not the functionality that sells ORMs. It's the object | graph. Mapping relational data to graphs, ORM or not, is quite | messy. Maybe that says we should be using graph databases for | this sort of data instead. | dvt wrote: | The actually story here is having the hubris to think you can | build a _good_ ORM (in Golang no less, lol) with a handful of | engineers at a small company that 's not even a _technology_ | company. How did this guy become a CTO again? Complete failure of | leadership. | | Mature ORMs (that still often suck) are built by literally | thousands of open-source contributors. | nauticacom wrote: | Never understood the hate for ORMs. I need to map from my data | storage to my domain model somehow, why write all that code | myself | the_af wrote: | > _Never understood the hate for ORMs. I need to map from my | data storage to my domain model somehow, why write all that | code myself_ | | This is addressed in the article the title of this one refers | to, "The Vietnam of Computer Science" [1]. I really recommend | reading it in order to understand why the "obvious" solution is | not always the best fit, but the very short summary is: due to | the Object-Relational Impedance Mismatch [2], using an ORM can | give you headaches and unwanted surprises. | | [1] http://blogs.tedneward.com/post/the-vietnam-of-computer- | scie... | | [2] | https://en.wikipedia.org/wiki/Object%E2%80%93relational_impe... | makeitdouble wrote: | Went to most of the article, and it seemed to boil down to | this kind of reflection: | | > But as time progresses, it's only natural that a well- | trained object-oriented developer will seek to leverage | inheritance in the object system, and seek ways to do the | same in the relational model. | | Basically, things are not perfect, and people try to blindly | apply layers upon layers, thus getting deeper and deeper into | a quagmire. | | But the proposed solution of applying no automated layer | doesn't compute. If he can restrain himself from using any | ORM at all, why can't he restrain himself from adding | inheritance in ORM models or other "let's OOP this to death" | blind approach ? | | This feels like the "I sometimes get drunk so I'll build a | whole support structure to stop me from drinking", and at no | point someone steps in to say "just be moderate and it will | be fine" | avgcorrection wrote: | Maybe the original author never read Chomsky. | | https://chomsky.info/198210__/ | the_af wrote: | Thanks for posting it. I knew Chomsky's position (that the | US involvement in Vietnam was actually a war _against_ | South Vietnam, with North Vietnam as the excuse) but had | never taken the time to actually read this interview in | detail. | | Yes, this makes the whole ORM analogy fall apart, because | it's based on a misconception. But I think the analogy | still works when using the popular understanding of that | war as the comparison. | nauticacom wrote: | It's no wonder this article was written in 2006. One of the | problems describes _DBAs_! Has anyone heard of a DBA in the | last decade? | | Most of these other problems are just... solved? Any decent | language will let you override the meaning of equality | between two objects, if you don't use a ton of inheritance | the inheritance problem won't bite you, good libraries let | you describe the schema once and either generate SQL or | generate classes, etc. | the_af wrote: | Two years ago I worked at a company with a DBA team. In my | current job, there are no database experts and people don't | understand things like indexes. | | In my experience, most developers don't understand | databases (of any kind). | | So I'd say none of the problems from that article are | really solved. | henvic wrote: | I never understood the love for them. | | I recently posted this about my experience in migrating to | PostgreSQL without anything of the sort of ORM... | https://henvic.dev/posts/go-postgres/ | | We achieved a really great result doing so. In comparison, at | my previous workplace, I was forced to use ORM, and everything | was slower, awful, and hard to maintain for no benefit | whatsoever. I'm not saying that it's 100% fault of ORMs, but at | least 60%, for sure it was. | Daishiman wrote: | The love comes when you're a single maintainer at a standard | line-of-business app that manages a few hundred tables. | | Most apps don't need to be performant. Most business logic | _is_ object-centric and relation-centric. Leveraging the best | features of highly-integrated frameworks like Django and | Rails mean that the framework has to understand and talk to | objects, not database rows. | | I have maintained several Django projects with a couple | hundred tables by myself or another team member. I cannot | imagine maintaining such a gigantic heap of equivalent SQL | for trivial business logic. | hello_moto wrote: | Nobody "loves" them. It's one of many tools just like | generic, OOP, functional, imperative, "nullsafe" features. | | Long list of tools with usefulness. | | I find people who tend to develop extreme on either end | aren't that pragmatic. | | I read your blog and I don't see any problems caused by ORM. | Your product data model should fit in ORM (I get it, you're | just using it as a simple example). The blog only mentioned | "I used ORM in the past and had bad experience" without | explaining the problems. | | > In comparison, at my previous workplace, I was forced to | use ORM, and everything was slower, awful, and hard to | maintain for no benefit whatsoever | | That seems to be the problem: your previous workplace, not | the ORM itself. | karmasimida wrote: | Is writing SQL query templates too much? | | The only valid advantage of ORM is to prevent SQL injection, | which can be solved with prepared statement. | nauticacom wrote: | Okay so I write a prepared statement, send it to my database | with some parameters, and get back some rows. Now what? How | do I turn those rows into structs or objects or whatever that | I can pass around for business logic? Maybe I write a | function that takes a row and maps it to an object, I call | that for every row I get back, whatever. Oh but now I'm | joining a belongs-to relationship and I want that joined row | represented as an object, too, so I have to write a function | to convert that row and oh what if the fields have | conflicting names okay I can always prefix them with the name | of the table and wait sometimes I'm not querying with this | related object so I have to check if the related id is | included in the row and... on and on and on for every | possible way I could make a different query. | | Great, now I've basically reinvented an ORM but it's worse | because I have to manually maintain all this mapper code and | it only ever accounts for the cases I've considered. If I | make it fully generic, then I've _really_ just reinvented an | ORM. | drran wrote: | My compiler cannot type check SQL code, so I need to run SQL | to check it correctness. It's the same problem as with | JavaScript, Python, Perl, shell, etc. | | For JavaScript, the problem is solved (partially) with | TypeScript. | rob74 wrote: | Because relational databases are very good at (surprise!) | relations. So in SQL you can query data from several joined | tables, query only certain columns, aggregate columns etc. etc. | And _all of these things_ are made needlessly complicated by | using ORM. Even with the best ORM, you still have to learn an | arcane new syntax for stuff you could write in 30 seconds when | using SQL... | hello_moto wrote: | ORM has escape hatch so you can write SQL for the complex | queries (and hopefully don't open yourself to SQL injection | along the process) and still benefit from ORM for the simple | to moderate case. | | > So in SQL you can query data from several joined tables | | Can do that in your ORM too. | | > query only certain columns | | So far no issue with ORM | nauticacom wrote: | Right, but for most of that stuff I can just write a one-off | query, munge it into the ORM's objects and pass that off to | my view for rendering. But probably 80% of my queries are | "get this object by ID" or "get these objects and some | related ones by a simple condition" or _maybe_ a slightly | complex condition. In these cases an ORM works wonders and | has almost zero downsides | pionar wrote: | The hate isn't in the mapping. It's perfectly reasonable to | abstract away mapping of relational data to objects. | | It's the translation of objects to SQL queries that's the | problem. ORMs typically output inefficient and mostly | obfuscated SQL, when a human can do a better job of hand- | writing the SQL, knowing the data usage patterns needed. | njitbew wrote: | ORMs could (and most do) provide some escape hatch, where you | can write the query yourself and reuse the hydration layer, | or reuse the query generator and customize the hydrator, or a | combination. Or you can just bail out completely for the few | performance critical queries. | | Honestly, ORMs are just an abstraction. They come at a cost | and they're not a silver bullet, just like most abstractions. | I believe the hate for ORMs in many cases is due to a lack of | understanding/wrong expectations. | switchbak wrote: | It's also the problem of leaky abstractions. Now I need to | know a lot about SQL, as well as the ORM, and in particular | how the ORM maps to my conceptual model of the underlying | SQL. | | The cognitive load (with some more than others) of this can | be extreme. I'm thinking of Hibernate I'm particular. | | I think there's a good underlying drive for this: type safe, | composable, declarative language-centric queries. Especially | helpful with bulk data management. I think if SQL was a more | natural mapping to this model, it wouldn't be so bad, but SQL | was not created for programmatic interaction (it was created | to be hand written), and can be challenging to create | abstractions around. | | These days I'll lean on simple ORM-light tools, but quickly | prefer an escape hatch when things get even a little complex. | Mapping the output is something that I'm happy to delegate to | a framework though. | | Some tools can even provide compile time type checking of | your queries, I think that's getting somewhere more | interesting. | topicseed wrote: | Speaking for Typescript and Mikro-ORM. I use the ORM | extensively for regular operations and queries, but usually | for searches with more complex queries, I do it by hand (get | PKs), and then load the entities by PK (I could skip that but | it's to let the ORM handle the mapping). | | ORMs generally have a "raw" escape route where you can just | do it yourself? Or maybe it's just the one I use.. | abraae wrote: | It's not just yours, it's common to most (virtually all?) | ORMs, which makes these ideological arguments somewhat | moot. | | Just use an ORM for mapping and for simple queries and hand | crank the complex queries. | FpUser wrote: | I avoided this Vietnam completely. It was clear to me from the | beginning that data model optimal for relational storage and the | one optimal for internal application state (I usually write | stateful servers in C++) are 2 different things. | dominotw wrote: | i thought we'd all agreed 15 yrs ago to not use vietnam analogy | to discuss orms. | the_af wrote: | The article this title refers to [1] is very insightful, and I | don't think it was ever improved by something better (analogy- | wise, or more descriptive). | | [1] http://blogs.tedneward.com/post/the-vietnam-of-computer- | scie... | dominotw wrote: | thats the article i was referring to. | | i remember a lot of consternation about that analogy | | followp | | http://blogs.tedneward.com/post/thoughts-on-vietnam- | commenta... | the_af wrote: | Well, in my mind Ted Neward's thoughts in that followup | explain why it's still an illustrative analogy. | ChrisMarshallNY wrote: | I am uncomfortable with the analogy. I understand it, but there | are other analogies we could use (some are worse. People used | to use the term "tar baby" to describe this kind of situation, | and I find that term rather disturbing, as well). | gunfighthacksaw wrote: | We have a new, shiny Afghanistan analogy to use! | rjsw wrote: | The last reference to Vietnam that I saw in the MSM was that it | had done a good job of handling COVID-19. Maybe the article is | describing a good use of an ORM. | taeric wrote: | I think that ship eventually left harbor. :( Scanning their | charts, July and August spiked heavy. | geodel wrote: | I think you agreed with yourself while thinking aloud in empty | room 15 yrs ago. | [deleted] | pixelgeek wrote: | What sort of ghoul compares a programming issue with something | like the conflict in Vietnam? When he started writing about 15 | million people dying you think that would have made him stop | and rethink the metaphor. | erect_hacker4 wrote: | This is standard fare on HN... | | Just remember that HN is not Reddit, that's the most | important thing. | erect_hacker5 wrote: | This is standard fare for HN. | | Just don't mention that HN is like Reddit - that's off | limits. | kevmo314 wrote: | As a junior dev I had no idea what an ORM was. | | As a mid-level dev I discovered them and wanted to use them | everywhere. | | As a senior dev I've gone back to manually writing queries. | | Such is life. | VWWHFSfQ wrote: | As with everything, it really depends on what you're doing. Do | I want to implement an entire CMS and core business backend | directly in SQL? No, of course I don't. I want to use Django's | ORM because then I get all the nice QuerySet API, integrations | with forms and filters, the nice Admin UI, etc. | | But do I want my end-users to hit my Django ORM APIs 10,000 | times per second? Absolutely not. So you write hand-optimized | SQL for those and maybe run them in a service a little bit | faster than Python/Django/Gunicorn. | | It's the same database, it's the same "API". But you get the | value of everything that Django's ORM provides and you just | cheat a little bit by dropping down to the lower level for | things that need it, and not everything needs it. | | Don't throw out the baby with the bath water! | dagmx wrote: | As a senior dev, I use an ORM unless I have a reason not to, | and I'd know why or not fairly early. | | Why do I use an ORM? Because they're way more maintainable for | a wider range of devs, with fewer accidental footguns, and the | good ones let me do direct queries too, so there's little | downside. | | So I default to using an ORM like sqlalchemy , because I can | hand it off to a junior dev afterwards, and they can easily | keep it going without needing to be writing good SQL directly. | | The other advantage is that I can abstract over multiple | databases easily. So I can have an sqlite db for quick local | testing, and a postgres DB for more thorough testing and | another for production. | karmasimida wrote: | ORM is that one layer that fails to abstract anything | substantial yet whose existence is intrusive enough to annoy | people. | | The worst example of leaky abstraction. | toast0 wrote: | ORM makes the easy things easier and the hard things harder. | No thanks. | cageface wrote: | So choose an ORM that lets you easily drop down to plain | SQL in the rare cases where it's necessary and you get the | best of both worlds. | devoutsalsa wrote: | 100%. You don't have to trash your 5 passenger car | because you occasionally need to transport 6 people. | aidos wrote: | Can you provide some examples? I use sqlalchemy, and the | amount of work that it does for me when it comes to | handling mutating data is pretty remarkable. | toast0 wrote: | No real specifics, but I usually run into ORM in the | context of I'm the person on the team with experience | running MySQL and someone has a slow query. When they're | hand built queries, I can usually provide a query or | sequence of queries that provides the same data without | knocking the server over and it gets put into production | within the same day. When they're ORM, it takes days to | find the query, and then more days to bend my query | sequence to fit the ORM. | | Sometimes you can add indexes to fix things, but | sometimes you really just need to do a simple query | followed by a large number of single row queries unioned | together, but that doesn't seem to be a feature of the | ORMs I've been adjacent to. | aidos wrote: | Yeah, I remember now the pain of using bad orms. That | feeling that you know what you want, but it's going to | load the data via configuration that you can't bend into | shape. | | Sqlalchemy is not that kind of orm. If anything is | running slow in production I can find it from the logs | and then track it to the code quickly - then you can do | any sort of query you want in its place. For me it's | never the orm loading stuff that's slow, it's always the | more complicated queries we've done for reporting or | whatever. | GordonS wrote: | And when you eventually ascend to true mastery, you'll be able | to enter a zen-like trance, in which you'll be able to tell | when to use one or not, or indeed whether to use both (e.g. ORM | for simple CRUD stuff, hand-written queries for everything | else). | diveanon wrote: | This is the way. | | ORM's can be a huge time saver and simplify your code base | for the day to day CRUD, nobody should be writing that | anymore IMO. | brodo wrote: | The problem is that most apps start out as simple CRUD apps. | Also each dependency is a liability. People really | underestimate how pulling in dependencies can lower your | software quality. Most people have security bugs in mind, but | there are also performance issues, memory leaks and logging- | issues, needles abstractions and incompatibilities. | m0zg wrote: | Real fun begins if your team has all three stages of seniority. | Typically it's somewhere in the middle, so your default is to | use ORM, even though as a senior person you know everyone will | profoundly regret this in the long run (and you will regret it | immediately). | robertlagrant wrote: | Some small points: | | > The ORM was a natural culprit: it's easy to see that many ORMs | will make use of object introspection to build SQL statements, | hydrate results, or both. Gorm's memory footprint is extreme | although sadly not uncommon. Bridge started with Python on the | backend and we used Django's ORM to interact with the database | which had similar problems. | | I believe these are both active record ORMs, which are not as | good for various reasons (including default memory usage) as good | data mappers such as SQLAlchemy and Hibernate. | | As for where enums etc are "mastered" - this is still a decision | to make with or without ORMs. | | _I_ think the main tricky bit of ORMs is that objects are trees | and relations are graphs; this causes problems. Not so much | anything about encapsulation etc. | aitchnyu wrote: | I use this pattern in Django and I feel without ORM or query | builder this will need 4 raw queries. | | ``` results = Foo.objects()... if a: results = | results.filter(a=a) if b: results = results.filter(a=b) ``` | hk1337 wrote: | I feel like the biggest problem with an ORM is how people try and | use them. It's like an all or nothing approach. You hit a wall | where using an ORM for that sucks and so now using an ORM for | everything sucks. | | I never would have thought of using an ORM in golang, it doesn't | seem like that kind of language. Even an ORM in python is | debatable. | bigmattystyles wrote: | I use them for the really simple stuff, where I basically know | what the query to be generated is. What I don't understand | though, what is wrong with writing a sprocket or sql function | and invoking it as a method from your oo layer? That gets to | 98% of my usage and it's fine. | Ciantic wrote: | The hate should be directed at SQL builder part of most ORMs. | Which gets complicated pretty quickly, and it's like learning a | new language when SQL works usually much better. | | If you write your SQL, and some tool, like code generation, then | maps it to object, you are golden. It's the part you'd have to | write anyway. However if the code generation goes bust (like gets | abandoned like most hobby mappers), you should make sure you can | take the code and continue like you never used the generator in | the first place. | | It looks like the authors suggestion SQLC is pretty close to | that. It seems to generate a lot of code though, so it might be | an issue if you don't want to maintain SQLC, test it out in here: | https://play.sqlc.dev/ | hn_throwaway_99 wrote: | I've been a huge fan of Slonik, | https://github.com/gajus/slonik, which makes it easy and safe | to write direct SQL queries. The author of Slonik had a good | article about the downsides of query builders, previously | discussed on HN, that I totally agree with: | https://gajus.medium.com/stop-using-knex-js-and-earn-30-bf41... | meitham wrote: | On the contrary, I hate dealing with the various escaping | characters for SQL, and to me SQLAlchemy expression is a | life-saver. It gives me back a textual sql expression as a | string and I can choose exactly how to run it. I do avoid the | ORM part as I like to be in total control of the transaction, | and I am often in the python async world, where ORM don't | play nicely. | hn_throwaway_99 wrote: | > I hate dealing with the various escaping characters for | SQL | | I don't understand that bit, with something like Slonik you | don't use escape characters, it does it transparently for | you by taking advantage of JavaScript's string template | literals. | makeitdouble wrote: | I guess that there's not many people going to write a long essay | on how they use ORMs moderately, knowing the limitations, and | it's perfectly fine. It wouldn't gather a whole crowd either. | | What feels wrong to me in this post is the lack of nuance on how | they used their ORM. For instance the whole part on how the ORM | generated queries they wish were done in a different manner: why | didn't they bother writing either lower level code or just raw | SQL to get their data as they wished ? | | It looks to me they could have kept their ORM for the 80% cases | it works great, and hand tune the 20% that could be problematic. | Just like any other optimization problem. | | There would be still the performance part where some overhead | would still be there, but it's usually not an issue if the ORM | option was on the table in the first place. ___________________________________________________________________ (page generated 2021-11-26 23:00 UTC)