[HN Gopher] Richard Hipp Speaks Out on SQLite (2019) [pdf]
___________________________________________________________________
Richard Hipp Speaks Out on SQLite (2019) [pdf]
Author : pncnmnp
Score : 91 points
Date : 2022-12-11 16:00 UTC (7 hours ago)
(HTM) web link (sigmodrecord.org)
(TXT) w3m dump (sigmodrecord.org)
| clusterhacks wrote:
| There are great bits of discussion about SQLite being maintained
| and enhanced by a very small team - three people.
|
| While I very much appreciate the safety of being a very small cog
| in a larger machine, I also wonder how my career might be more
| enjoyable and satisfying if I had chosen to limit my options to
| small teams where every single person on the team had a critical
| impact on the success of a project.
| jerrygenser wrote:
| Is there a succession plan in place given the bus factor [1]
| with such a small team? Is anyone else concerned about
| maintenance if either a normal everyday medical issue came up
| or in the worst case something unimaginable?
|
| [1] https://en.wikipedia.org/wiki/Bus_factor
| xiphias2 wrote:
| It's not really a problem when it's a 100% well tested
| software.
| bob1029 wrote:
| From my perspective, SQLite is stable enough to be considered
| "done". For our product, we could go back to builds from 2020
| and none of our customers would be able to detect a
| difference.
|
| SQLite is not something we wire up and expose bare-ass to the
| public internet. We don't worry about 0-day security issues
| in this context. It's more like a wrapper our product uses
| for structured disk access. Anything that would warrant
| urgent patching is in .NET vendor or in-house software piles.
|
| If something horrible happened and the entire SQLite dev team
| was wiped off the face of the planet, I'd make a note to look
| at alternatives by ~2025.
| rwbt wrote:
| Yes, Dr. Hipp mentioned it once that there is a foundation in
| place to take care of things and increase the bus factor. I
| think that was one of the conditions before Nokia signed
| their agreement to license SQLite for Symbian.
| richardlblair wrote:
| There are organizations, even large ones, that prefer to work
| this week. Trying to set the maximum number of team members to
| 6 people including a PM and a designer. As you move through
| your career I highly recommend adding a question during the
| interview process to address this. It really does make a big
| difference on your quality of life, and you'll make some great
| friends along the way.
| ilyt wrote:
| That kinda entirely depends with what people you're stuck with.
| In 6-10 person team you can just avoid working with that one
| colleague you just can't gel with (even if they might be
| perfectly fine from competency standpoint).
|
| I think medium-sized projects (say a team of max 10, or maybe 2
| smaller ones) seems to be the most fun ones. You have enough
| manpower to do some bigger reorganizing if required but it
| isn't as big that rewrite of any part is years-long saga.
| qbasic_forever wrote:
| More recently he's been on the Changelog podcast in the last year
| and it's a great listen: https://changelog.com/podcast/454
| johnboiles wrote:
| I've been trying to use SQLite3 in an embedded (ESP32 -- dual
| core 240MHz w/ qspi NOR flash) project and the performance has
| been surprisingly bad. Seems like the minimum amount of time to
| make a query (even for example, selecting from a table that does
| not exist, takes a minimum of 5ms). For a simple table with 10
| short columns I'm getting 10s of milliseconds per record to
| SELECT. I can parse whole JSON files with the same data to RAM
| faster then I can query SQLite which seems wrong. SQLite is
| performing ~10x worse than I'd expected it would.
|
| Anyone had experience running SQLite on low-power embedded
| platforms? Is this expected performance?
| fbdab103 wrote:
| If you have the RAM to load up a toy dataset, what happens if
| you utilize a :memory: database instead of disk? If the
| performance is the same, you know the problem exists outside of
| the storage.
| bob1029 wrote:
| Could you share some source code around your implementation?
| Selecting from a non-existing table should be a very quick
| operation. I suspect you might be doing something like creating
| a new connection per command if you are seeing this behavior.
| phonon wrote:
| Have you benchmarked the storage? That might be the bottleneck.
| andrewl wrote:
| This is interesting. I wonder if he's actually working on it:
|
| Q: If you magically had enough extra time to do one additional
| thing at work that you're not doing now, what would it be?
|
| A: Oh, I've got a long list. But my #1 thing right now I think
| would be a new version control system which the working name is
| Fit, it's a combination of Fossil with Git. Uses the Fossil user
| interface but it uses the low-level file format of Git. So that
| then you can work with Fossil's interface but push and pull to
| legacy Git users. And I think that would be huge.
| mdaniel wrote:
| > Uses the Fossil user interface but it uses the low-level file
| format of Git. So that then you can work with Fossil's
| interface but push and pull to legacy Git users.
|
| I'm surprised that's the #1 magic-wand project for anyone. I'm
| not trying to yuck his yum, but am just surprised.
|
| But the larger elephant in the room for any such effort is that
| Fossil's mental model is, by choice, _way_ different from git.
| If they just want a new ui that isn 't as "git sux," surely a
| new cli to kid-gloves git would be less work than trying to
| reinvent a whole new version control system, and that goes
| 1000x for trying to maintain wire compat with _every_ git
| server in the world
| ilyt wrote:
| I don't think Git wire format changes all that often and old
| clients work with new servers just fine so I don't think that
| would be a problem.
|
| > But the larger elephant in the room for any such effort is
| that Fossil's mental model is, by choice, way different from
| git.
|
| How? I've been glancing at docs but I don't see much
| difference, still a DAG of commits that are named after its
| hash like almost every other DVCS
|
| From onlooker perspective the killer feature seems to be
| integration with project management tools vs "just" being a
| VCS
| dahart wrote:
| > I'm surprised that's the #1 magic-wand project for anyone.
|
| It seems like Dr. Hipp is frustrated that Fossil hasn't had
| mainstream adoption, and that git has, he's talked about it
| quite a bit in Fossil docs and here on HN. Given that SQLite
| is very stable and adoption is an absolutely runaway success,
| I'm not too surprised his wish list now turns to Fossil.
|
| > Fossil's mental model is, by choice, way different from
| git.
|
| Is that accurate? It does come with a ton of extra features,
| but seems like the core version control model is quite
| similar conceptually, and differs more on implementation
| (SQLite backing instead of file system) and workflow dogma
| (Fossil's moralizing of rebase and commit graph edits). The
| Fossil docs even start with "The feature sets of Fossil and
| Git overlap in many ways." Fossil was released after but
| within a year of git being released, the timeline and docs
| and comments from the authors generally suggest it's a
| reaction to git, taking the good parts and trying to, in the
| authors' eyes, improve and fix their perceived problems with
| git.
| mdaniel wrote:
| I don't mean DAG versus "lolol," because I'm not saying
| Fossil isn't an SCM, I'm pointing out the design choices
| which presumably someone who wants to use Fossil buys into
| and thus would be a culture clash with trying to
| participate with the git upstream. I am cognizant this is
| quoting a bunch from the same page but pull-quotes the
| anti-git stance that makes me question whether this is a
| _technical_ problem
|
| We'll start with Fossil not tolerating `git rebase` or `git
| commit --amend`: https://www.fossil-
| scm.org/home/doc/trunk/www/fossil-v-git.w... and the number
| of repos I've seen where PRs don't land without someone
| chirping about "please rebase your branch"
|
| "Fossil was designed for [...] just four programmers, and
| 64% of it is from the lead developer alone. [They] know
| each other well and interact daily." https://www.fossil-
| scm.org/home/doc/trunk/www/fossil-v-git.w... makes one
| wonder what kind of awesomeness would ensue if running `fit
| clone https://github.com/chromium/chromium.git`
|
| and its related topic: "Fossil is not designed for drive-by
| contributions" and "Where Git encourages siloed
| development, Fossil fights against it.":
| https://www.fossil-scm.org/home/doc/trunk/www/fossil-v-
| git.w... -- I would be really sad to have all my working
| changes auto-pushed by the tool, although the
| https://github.com/martinvonz/jj#readme is also in that
| spirit, where almost every operation is a `git add . && git
| commit -mwip`
|
| Fossil is `git push --mirror` by default:
| https://www.fossil-scm.org/home/doc/trunk/www/fossil-v-
| git.w... versus `git push origin HEAD` which is what most
| people think of with "git push"
|
| Then, as one of the obvious actual technical issues: Fossil
| is SHA-3: https://www.fossil-
| scm.org/home/doc/trunk/www/fossil-v-git.w...
| dahart wrote:
| Yeah I agree, this isn't a technical problem. The stated
| goal is to build the Fossil interface using the git
| engine, so it's not Fossil, it's just an implementation
| of git that is compatible with Fossil's workflow. It
| seems like the assumption there is maybe people want
| Fossil's workflow but are stuck having to use git because
| it's entrenched.
|
| BTW, Fossil's comparisons to git (especially rebase and
| rewriting) have always been hyperbolic. It's improving
| slowly over the years, however the fundamental problem is
| still there: the Fossil team is presumptuously stating
| git's intentions under Fossil's overtly moral framework.
| Git devs have never stated any intention "to record what
| the development of a project should have looked like had
| there been no mistakes." That's the Fossil team framing
| things to sell the idea behind Fossil. Ironically, the
| Fossil team is attempting to rewrite the history of git
| development by using such language. I think the reason
| behind Fossil's lack of adoption isn't because git is
| entrenched, I suspect it's because very few people care
| about having the ability to record in stone every single
| commit no matter how messy, most people _enjoy_ having
| the ability to clean up after the fact. (Not to mention
| that the idea of capturing and preserving an exact
| "history" is misguided and fundamentally impossible; it's
| already up to me what I put in a commit in the first
| place, and Fossil is providing dis-incentives to commit
| early and often, undermining the safety reasons to have
| version control in the first place.)
| robertlagrant wrote:
| The counterpoint is that Git's enables multiple
| workflows, especially branching and merging, rather than
| just trunk-based development. Not to say you can't do the
| latter in Git, but people might struggle to adapt.
|
| I think trunk-based, warts and all, is probably the best
| method for a team that isn't extremely async. Which, I
| think, makes it useful for a large number of projects.
| tiffanyh wrote:
| Super interesting given the purposeful differences between
| Fossil & Git.
|
| https://www.fossil-scm.org/home/doc/trunk/www/fossil-v-git.w...
| fbdab103 wrote:
| I think it is just being practical. Much to my chagrin, git
| has consumed the world. The Fossil tooling offers a
| reasonably slick (well, not the CSS) out of the box
| experience which would be nice to have elsewhere.
| Cyberdog wrote:
| Git's UI is difficult, yes, but I'm sure far more people are
| familiar with it at this point than Fossil. It would be far
| more inefficient for me to learn Fossil's commands for what I
| need Git to do than just continuing to use Git, at least in the
| short term (and with no guarantees it'll improve in the long
| term). If anything, a tool which goes the other way around -
| allows me to interact with Fossil using familiar Git commands -
| seems more useful to me.
| eptcyka wrote:
| This is the case for a lot of tools. There was a time that
| for a majority of people on the planet, handwriting a letter
| would be faster than typing it up too.
| _a_a_a_ wrote:
| That would seem to be just a new porcelain for git.
| eesmith wrote:
| Fossil also supports bug tracking, wiki, forum, email alerts,
| chat, and technotes.
|
| I don't think it's right to call that "just a new porcelain".
|
| I have no clue how Hipp thinks to implement it, much less in
| a way that allows push and pull to legacy Git users.
| ilyt wrote:
| I'd imagine legacy users would just see a unconnected
| branch with a bunch of textfiles. fit-tickets for tickets,
| fit-wiki for wiki, fit-docs for docs etc.
| marmetio wrote:
| Isn't that the point of both projects?
|
| Git is primarily a file format. There's also a canonical set
| of low-level command-line utilities. But you can create
| whatever interface you want. Most programmers and GUIs happen
| to use the canonical command-line tools, but that's hardly
| necessary. Fit would keep the necessary part of Git (the file
| format), and provide a different human interface.
| vouaobrasil wrote:
| That would be really nice. I think there is definitely room for
| version control for people who are less in the programming
| realm or who work on personal projects that have a more
| creative art flavour (writers, desingers, etc). The user
| interface of Git is too frustrating for these types. Even as a
| mathematician and somewhat-programmer I dislike Git a lot.
| Rebase, blame, stash...ugghhh. Although Git is nice I liked
| Subversion better.
| eternityforest wrote:
| Non-programmers aren't going to want _any_ CLI version
| control, and git-cola plus meld is perfectly fine for simple
| use cases as long as nobody force pushes or uses a submodule
| or something like that.
|
| Although usually what seems to happen is people only use one
| linear history and never branch anything. I'm not sure if
| that's because Git is hard, or just because nobody wants to
| learn about branches and stuff in any system if they don't
| have to.
|
| In any case it works perfectly fine and is an amazing step up
| compared to no VCS.
| tokamak-teapot wrote:
| Have you used Mercurial? I found it a lot more user friendly,
| especially with TortoiseHg for Windows. Not sure if that's
| still around as I don't use Windows any more but even though
| there are many similarities, git always feels like it's
| designed to solve problems I don't want to use source control
| to solve.
| muxator wrote:
| TortoiseHg is still maintained and works both on windows
| (no direct experience) and linux.
|
| The development happens on its mailing list, but there is
| also an Heptapod (a gitlab fork modified to also support
| mercurial) instance here:
| https://foss.heptapod.net/mercurial/tortoisehg
|
| Interesting enough, almost all my work nowadays consists of
| interacion with git remotes. But my local clone has always
| been mercurial via hg-git.
| leeoniya wrote:
| the main reason i'm not all-in on Fossil is because you cannot
| squash commits; my WIP branch history is very messy, e.g.
| "looks to be working", "refactor", "typo", "optimize", "undo
| checkpoint". i don't need any of that junk on the `main` branch
| once the feature branch is merged.
|
| it's really too bad, everything else in Fossil looks very
| compelling.
| zie wrote:
| You can hide those commits, you just can't delete/squash
| them. It's really not as big of a deal as everyone claims.
| andrewl wrote:
| In this conversation from July 2021 he said he's going to write
| his own email server, which I would be very interested in:
|
| https://corecursive.com/066-sqlite-with-richard-hipp/
|
| _I'm going to write my own mail server. I was making notes on
| that even as we were setting up this call. That's a big problem,
| and that's at least as difficult if not more difficult than
| writing a database engine, but I don't want to be beholden to
| Gmail. I don't want them controlling my destiny. I don't want
| them controlling the record of all of my conversations. I want to
| control that myself, and so I'm going to go through a lot of pain
| and a lot of work and a lot of effort to come up with some
| solution that I can control myself. I can go out and lease a
| virtual machine out there in the cloud and run it myself and not
| depend on a third party to control my email._
| mdaniel wrote:
| > I'm going to write my own mail server. I don't want to be
| beholden to Gmail.
|
| Wowzers, that's some RMS-"Web Over Email"-level "I have privacy
| concerns so I'm going to reinvent $foo"
| https://stallman.org/stallman-computing.html
| eesmith wrote:
| How do you go from "control" (used several times) to
| "privacy"?
|
| Hipp not only wrote SQLite, which uses his own parser
| generator (Lemon) but he also wrote his own version control
| system (Fossil), and his own editor.
|
| Perhaps it's more likely that he wants to be in control of
| the things which most closely affect him, rather than this
| being a privacy issue?
___________________________________________________________________
(page generated 2022-12-11 23:00 UTC)