[HN Gopher] Ask HN: Why are Git submodules so bad?
___________________________________________________________________
Ask HN: Why are Git submodules so bad?
I have been a git user for a long time, but I've never used
Subversion or any other VCS more than a little. I also hardly use
Git submodules, but when I do, I don't struggle. Yet people talk
about Git submodules as though they are really hard. I presume I'm
just not using them as much as other people, or that my use case
for them happens to be on their happy path. So why are Git
submodules so bad?
Author : ghoward
Score : 100 points
Date : 2022-06-18 18:11 UTC (4 hours ago)
| armchairhacker wrote:
| Git submodules are fine and can be really useful, but they are
| really hard. I've run into problems like:
|
| 1. Git clone not cloning submodules. You need `git submodule
| update` or `git clone --recursive`, I think
|
| 2. Git submodules being out-of-sync because I forgot to pull them
| specifically. I'm pretty sure `git submodule update` doesn't
| always work with this but maybe only when 3)
|
| 3. Git diff returns something even after I commit, because the
| submodule has a change. I have to go into the submodule, and
| either commit / push that as well or revert it. Basically every
| operation I do on the git main I need to also do on the submodule
| if I modified files in both
|
| 4. Fixing merge conflicts and using git in one repo is already
| hard enough. The team I was working on kept having issues with
| using the wrong submodule commit, not having the same commit /
| push requirements on submodules, etc.
|
| All of these can be fixed by tools and smart techniques like
| putting `git submodule update` in the makefile. Git submodules
| aren't "bad" and honestly they're an essential feature of git.
| But they are a struggle, and lots of people use monorepos instead
| (which have their own problems...).
| lelandfe wrote:
| Agreed with all of this!
|
| In git parlance, the submodule porcelain is hard to use (but
| the plumbing is good)
| 42e6e8c8-f7b8-4 wrote:
| That's the entirety of git. Extremely fast. Bummer UI. It's
| status quo and no changing. Despite using git for 10+ years,
| I frequently have to look up commands and then I end up
| scratching my head as to why the CLI UI is like that.
| oneplane wrote:
| I suppose the biggest problem is that the concept of
| SCM/VCS is just not simple enough to make both easy and
| useful/advanced at the same time.
|
| You can have a 'pull, merge, push'-only system, but at that
| point we're re-inventing subversion. So making it more
| advanced would mean we also need to have the knowledge and
| skills to do other activities correctly and that means the
| tooling can't make as many choices for you because there
| simply isn't a default way that works all the time.
|
| Most efforts at git-alternatives run in to the same
| problems and either they'll be just as advanced and have
| the samen benefits and downsides or they end up less
| advanced but now it's not equally useful and you can't
| really make it work right.
| howinteresting wrote:
| Pull, merge, rebase, push with local commits is what
| almost everyone cares about.
| kjeetgill wrote:
| I think you're really, really undervaluing branches. The
| fact that patches are mostly shared as branches changes
| everything.
|
| It's what enables three-way merges, and makes rebasing
| much more manageable.
|
| The ability to traverse and jump to some other point in
| history is really missing here too.
| pan69 wrote:
| > but at that point we're re-inventing subversion
|
| So? Maybe Subversion is all most developers need?
| abirch wrote:
| Git makes sense for the Linux kernel and merging patches
| at scale. 99% of software development would be fine with
| subversion
| dopidopHN wrote:
| Have you use subversion ? If yes, do you remember how
| slow it was? How cutting a branch and then merge it was
| seen as something for senior dev to handle ?
| lambda_lord wrote:
| Git completely replaced Subversion so quickly because the
| benefits were apparent even at a small scale. Subversion
| was centralized and slow whereas Git branches were cheap
| and fast. It turns out the distributed model is just a
| lot better. Even my college project teams benefited from
| the superior experience of Git.
| bsder wrote:
| Except that everybody changed to _Github_ not _git_. And
| effectively recreated Subversion with caching.
| dopidopHN wrote:
| No, I worked with gitlab, bitbucket, custom git server
| installs in the last 3 years alone.
| PaulDavisThe1st wrote:
| "Subversion with caching" is not subversion.
|
| I used subversion for a long time and was resistant to
| moving ardour.org to use git instead. 24hours after we
| switched (we never use 3rd party git hosting as our
| canonical repo), I was already convinced it was not the
| right choice, but an excellent choice.
| dalyons wrote:
| Solo devs maybe. Merging and branching are so much worse
| in svn that it's not good enough for "most developers",
| aka those working on professional projects with a team of
| developers. Sure we used to make do with svn, but I have
| no desire to go back.
| mnd999 wrote:
| Honestly I'd take CVS over subversion. Merging was just
| bad on subversion and I found the IDE integrations to be
| confusing, obtuse and buggy.
|
| Sure, CVS was limited but it was reliable and
| straightforward.
| akavel wrote:
| Mercurial covers generally the same concepts as git and
| is thus also not trivial to learn for someone
| uninitiated; yet its interface was like day vs. night
| when compared to git since their very early days. It
| proves one can design a decent interface if one actually
| tries to care about usability and friendliness. How I
| remember the past days, git won the rivalry squarely due
| to GitHub becoming popular (though I assume there were
| some reasons why GH chose git over hg).
| fanf2 wrote:
| Switching branches in a repository with submodules is a huge
| pain, especially if (like the Ansible repo) some branches have
| the subdirectory in the same repo like normal, and some
| branches have the same subdirectory in a submodule.
| Phurist wrote:
| That sounds like a problem that exists between the chair and
| the keyboard.
| onion2k wrote:
| In 25 years of software dev, and most of that in developing
| user interfaces to things, I have never found a commonly
| repeated error that could be attributed to user error. It's
| _always_ badly designed software that ignores the user 's
| mental model that's developed based on using the software.
|
| In this case, "git clone" clones a repo 99% of the time,
| because 99% of repos are shallow and simple. For "git clone"
| to only clone the top level when you have su modules instead
| of prompting to ask if you want a deep clone, or doing a deep
| clone by default because that's the expected behavior, is
| pretty poor design IMO.
| fburnaby wrote:
| I agree with all of this. Submodules aren't easy but they
| perform a useful job. It's hard to see how they could be made
| significantly easier. Where else in software is dependency
| management easy and convenient?
| pm215 wrote:
| "Make 'git checkout' of the top level repo also set the
| submodules to the contents they should have for that top
| level commit hash" is probably the main change I'd want. The
| current setup means that checking out a branch or doing a git
| bisect gives you an inconsistent source tree, which seems
| like a really unhelpful behaviour.
| dtgriscom wrote:
| `git submodule update --init --recursive` is the magic phrase.
|
| And, yes: submodules are really useful, as well as a PIA.
| acadapter wrote:
| And you can also --recurse-submodules when cloning
| andi999 wrote:
| What tools fix this?
| abathur wrote:
| Using something like Nix to specify the dependencies instead.
| agilob wrote:
| There are git options for managing these difficulties like:
|
| git config --global submodule.recurse true
|
| https://git-scm.com/book/en/v2/Git-Tools-Submodules search for
| "git config"
| elcritch wrote:
| Thanks! That one option simplifies like 2/3 of the pain of
| using submodules. Doing a pull or checkout would just be
| regular commands.
|
| Though it'd be nice if `git commit' supported it too and just
| did a `git submodule foreach git commit ...`.
| anonymoushn wrote:
| If you work on blorp which contains openresty which contains
| luajit, and you have a patch to luajit that you are making
| because it enables some work in the end product, you need to make
| a commit to luajit, make a commit to openresty with your changes,
| make another commit openresty to change the luajit version, make
| a commit to blorp with your changes, and make another commit to
| blorp to change the openresty version. You will create 3 code
| reviews none of which actually contain all of your changes
| together. Your coworkers have decent odds of not being able to
| build their software because they don't have a keybind for `git
| submodule update --init --recursive` yet.
| tommyjl wrote:
| I recently started using git subtree[0] instead of dealing with
| all the problems with git submodules, and have been very happy
| with the experience so far. It does copy every file into your
| repository, though.
|
| [0]:
| https://github.com/git/git/blob/master/contrib/subtree/git-s...
| TrianguloY wrote:
| I don't use submodules, but I do use git repositories inside
| other git repositories, and let IntelliJ manage them both
| simultaneously as if they were two different projects. Works
| fine.
|
| I once enabled it as submodule to test. It adds nothing and from
| that moment any change in the child creates a change in the
| parent, which for my use case is totally unnecessary (I want both
| of them to be independent, even if hierarchically one is inside
| the other).
|
| Submodules are probably a good option to have libraries that you
| rarely touch, so you can update/modify them as with a
| maven/gradle project. For most other user cases submodules make
| more problems that advantages.
| andix wrote:
| I always ran into issues when switching branches, merging or
| rebasing. And then you have to figure out what's going on. If
| you're not used to work with submodules, that's the moment where
| you have to learn it. And a lot of people get overwhelmed then.
| grogenaut wrote:
| This is what killed it for me. Changes to submodules don't seem
| to do well across commits/rebases, or bisect type operations.
| After fixing it on so many people's computers when using
| modules I just had us give up and move the stuff out.
| howinteresting wrote:
| The problem with submodules is that they're read-write. Read-only
| submodules would be completely fine.
| d_watt wrote:
| Git submodules aren't bad in that they're buggy, they do what the
| documentation suggests.
|
| I think they're difficult to use, because it breaks my mental
| model of how I expect a repository to work. It creates a new
| level of abstraction for figuring out how everything is related,
| and what commands you need to be able to keep things in sync (as
| opposed to just a normal pull/branch/push flow). It creates a
| whole new layer to the way your VCS works the consumer needs to
| understand.
|
| The two alternatives are
|
| 1. Have a bunch of repositories with an understanding of what an
| expected file structure is, ala ./projects/repo_1,
| ./projects/repo_2. You have a master repo with a readme
| instructing people on how to set it up. In theory, there's a
| disadvantage here in that it puts more work on the end user to
| manually set that up, but the advantage is there's a simpler
| understanding of how everything works together.
|
| 2. A mono repo. If you absolutely want all of the files to be
| linked together in a large repo, why not just put them in the
| same repo, rather than forking everything out across many repos.
| You lose a little flexibility in being able to mix and match
| branches, but nothing a good cherry-pick when needed can't fix.
|
| Either of these strategies solve the same problem sub-modules are
| usually used to solve, without creating a more burdensome mental
| model, in my opinion. So the question becomes why use them and
| add more to understand, if there are simpler patterns to use
| instead.
| aslilac wrote:
| You completely missed the problem that submodules are actually
| supposed to solve though. Using them for either of those cases
| would almost definitely be the wrong choice.
|
| What they're really for, is vendoring someone else's code into
| yours. They're still not great even at that, but sometimes
| they're the best option.
| d_watt wrote:
| Interesting. When you say that's the problem they're intended
| to solve, do you have a link to that as their intended use
| case.
|
| IE is that "a" usecase or "the" usecase? I've never seen
| submodules used for that, only for internal dep management,
| so if there's content about "what they're really for," I'd
| love to read more.
| grogenaut wrote:
| When I worked in games we did exactly this but with
| perforce. All of our libraries were in a vendor tree and
| all in source. We slapped our own build over the top of
| them and checked in the build artifacts (perforce). If we
| needed an update, we updated their code and maybe our build
| script.
|
| It'd be nice to use submodules for this but I gave up years
| ago.
|
| The other big use is where you have your own libraries and
| you'd like to be able to share them across projects. My
| friend does game jams and has his own simple engine, he
| versions the engine and adds abilities and uses it across
| game projects.
| theduder99 wrote:
| check the first paragraph on this page: https://git-
| scm.com/book/en/v2/Git-Tools-Submodules
| ddlsmurf wrote:
| This i. Also subtree is an interesting relevant tool too.
| jayd16 wrote:
| The security model seems to be "terrible UX defaults that you
| turn to unsafe defaults instead."
|
| You end up with a lot of gotchas instead of them just working.
|
| The mental model of juggling multiple repos in a non-atomic way
| also violates the rule of least astonishment. Working with read
| only submodules smooths this part out at least.
|
| GUI support is slowly getting better at least.
| pengaru wrote:
| > The security model seems to be "terrible UX defaults that you
| turn to unsafe defaults instead."
|
| Except practically nothing in git is unsafe, it's like Plan9's
| Venti in the sense that you can only add content to it. You
| never lose data, but you can easily lose your bearings.
|
| I presume a major reason why there's not much effort put into
| UX guard rails preventing losing one's bearings in the day-to-
| day work is a product of the fundamental fact that the
| underlying committed data is always there. People just need to
| at least familiarize themselves with `git reflog` to help lose
| the anxiety.
| layer8 wrote:
| > the underlying committed data is always there
|
| That's not always true, due to reflog expiration and GC.
|
| This is an important difference to e.g. Subversion where you
| can delete branches/tags just to hide them, but the history
| always remains 100% there and nothing is ever really deleted.
| SahAssar wrote:
| In git if a commit is within the history of a branch or tag
| it won't be GC:ed, right?
| layer8 wrote:
| That's correct, but commits from that history can also be
| removed, for example by squashing a branch. The Git
| history is quite mutable in that sense and doesn't
| necessarily reflect the actual commit history.
| normaltool wrote:
| I don't think most people have a complete idea about git
| submodules. but I think it's easier to say bad directly because
| everyone around him is vilifying it.
| TillE wrote:
| It's really annoying that submodules give you a detached head by
| default, so working on a submodule within a project is prone to
| mistakes. Otherwise they've been fine for me.
| FrenchyJiby wrote:
| Beyond how hard to use they may or may not be, my personal hatred
| of git submodules is about bypassing your normal dependency
| management system. See 12 Factors on Dependencies[1].
|
| I've not seen many uses of submodules that weren't better served
| by adding the package from pypi/npm/crates/...
|
| [1]: https://12factor.net/dependencies
| SahAssar wrote:
| Why would using pypi/npm/crates be better? If I'm already using
| git (which I will be) then using something else for package
| management needs to be a huge step up, especially if the other
| systems you mentioned are language specific and a git solution
| would probably be language agnostic.
| gitgud wrote:
| > _bypassing your normal dependency management system_
|
| You can use release tags when using Git submodules, which makes
| it closer to a "normal dependency management system".
|
| It's better than using the commit hash or branch, but still not
| that great without flexible semver dependencies...
| withinboredom wrote:
| I've been reading a lot of research papers lately, sometimes
| with POC repos. When compiling the code, it would often fail
| because dependencies have changed over the years (we're talking
| about C/C++ code which doesn't have a package manager like
| you're probably used to). Most of these repos would fail to
| compile because it expected libraries installed on the system
| to not have changed in the past 10 years or so. In exactly one
| of these repos did the author use a git submodule so I was
| linking directly to the correct version of the library.
| Granted, those libraries _did not_ do this, so it still failed
| to compile 7 years later...
|
| If everyone used git submodules for deps, you'd end up with a
| (block) chain (if everyone's commits were signed) for deps.
| Groxx wrote:
| Submodules are just complicated because Git makes _no decisions
| at all_ about how they should behave, beyond "never make a
| decision that could lose data".
|
| So you have to understand the tradeoffs and make every decision
| at every step. It's the safe option.
|
| Like, what happens if you remove a submodule between revisions?
| Git won't remove the files, you could have stuff in there. So it
| just dangles there, as a pile of changed files that you now have
| to separately, manually remove or commit, because it's no longer
| tracked as a submodule. And then repeat this same kind of "X
| could sometimes be risky, so don't do it" behavior for dozens of
| scenarios.
|
| All of which is in some ways reasonable, and is very much "Git-
| like behavior". But it's annoying, and if you don't really
| understand it all it seems like it's just getting in your way all
| the time for no good reason. Git has been very very slowly
| improving this behavior in general, but it's still taking an
| _extremely_ conservative stance on everything, so it 'll probably
| never be streamlined or automagic - making one set of decisions
| implicitly would get in the way of someone who wants different
| behavior.
| dustingetz wrote:
| mutable pointers in the middle of your immutable lineage.
| completely broken model
| WanderPanda wrote:
| Whats up with git submodules refusing to clone/update/checkout
| submodules while having all their files showing up as deleted? I
| encounter this quite a lot and the solution seems to be a git
| submodule sync --recursive (or something like that) but I don't
| get why I run into this in the first place? Probably related to
| forgetting --recurse-submodules when cloning but what do I know?
| sto_hristo wrote:
| They have their uses and misuses. People misuse them a lot, get
| burned, they blame them, and then they hate them.
|
| If you lack a dependency mgmt system, work mostly solo or in very
| small and tightly coordinated team and just need some githubbed
| project to make yours work, submodules may be the right tool for
| you.
| throwawaylala1 wrote:
| Exactly! Submodules are a tool within a toolkit. Learn the
| toolkit, learn the tool, and then decide whether or not it'll
| solve your problem.
| Dylan16807 wrote:
| But this is such a non-answer to what's good or bad about
| them.
| throwawaylala1 wrote:
| It was a response to a comment, not the OP.
|
| There is nothing "bad" about submodules. They do what they
| were intended to do very well. They are often misused
| because people don't take the time to understand how things
| work (event at a conceptual level) before diving into it.
|
| I've found submodules most useful when I'm tracking an
| external dependency that doesn't change often -and- I don't
| have a dependency management system to lean on.
|
| For example, I have a repository of 100+ various bash
| utility scripts. One of the scripts depends on another
| repository with 500MB of data. I don't use it often so I
| don't need that large repo cloned alongside my bash scripts
| every time. In this case:
|
| 1) The dependency doesn't change often and I like being
| deliberate when using a new version of that dependency.
|
| 2) The dependency is optional and can simply be ignored
| when using Git's default clone behavior for my repo.
|
| 3) Since it's just a bunch of bash scripts, I don't have a
| package management system to lean on. I use these scripts
| on multiple *nix distributions.
|
| Could I make it work without Git submodules? Yes. I would
| have added the dependency to .gitignore and wrote a script
| to `git clone` the dependency and set it to a specific
| revision. Submodules just makes that process a lot easier.
| Dylan16807 wrote:
| > It was a response to a comment, not the OP.
|
| You were elaborating on the comment, weren't you? And
| that comment was quite vague too.
|
| > There is nothing "bad" about submodules.
|
| The difference between "what are uses and misuses" and
| "what is good and bad" is not enormous. I think you're
| nitpicking. Plus, there _are_ things that are flat-out
| bad. It 's a human-made program under constant
| development, after all.
| throwawaylala1 wrote:
| > You were elaborating on the comment, weren't you? And
| that comment was quite vague too.
|
| What was vague about that comment?
|
| > The difference between "what are uses and misuses" and
| "what is good and bad" is not enormous.
|
| This is incorrect. The definition of "bad" is something
| of low quality. Misusing a tool and then calling it low
| quality is dumb.
|
| > Plus, there are things that are flat-out bad. It's a
| human-made program under constant development, after all.
|
| What about git submodules is "flat-out bad"?
|
| It's ironic that you're criticizing others' comments on
| being vague and providing non-answers while you've
| contributed nothing to the discussion. Maybe educate
| yourself on technical things and you won't need to waste
| your time having meta discussions on internet forums.
| Looks to me like you want to contribute but have nothing
| of value to say.
| JoshTriplett wrote:
| The biggest reason I find git submodules painful: a "commitlink"
| object in a git tree _does not count as a reference to that
| commit or anything that commit references_ , for the purposes of
| garbage collecting the repository or pushing and pulling changes.
| You can't have the _only_ reference in your repository to a given
| commit be a commitlink within another tree.
|
| I'd like to jettison the entire model of "reference another
| repository that you may or may not have", along with the
| `.gitmodules` file as anything other than optional metadata, and
| instead have them be a fully integrated thing that necessarily
| comes along with any pull/push/clone.
| Izkata wrote:
| They act almost the same as pinned-revision svn externals, which
| people don't really seem to have a problem with. The biggest
| difference I can think of is needing a special command to pull in
| the submodules, where svn pulls its externals automatically.
| sascha_sl wrote:
| Submodules aren't bad. But in a world where I have to explain
| that running revert on the 250 megs of jar files someone comitted
| isn't a fix, and where people often just delete and re-clone
| entire repos because they don't know what's going on, they incur
| a heavy support burden on the few people that know how to use
| them.
|
| You know, the people that already carry everyone else through
| their job.
| seabrookmx wrote:
| I feel your pain. I don't expect everyone to be an expert on
| everything but given how important version control is in this
| profession, the lack of understanding (or desire to
| understand!) it's fundamentals by many in our industry is
| downright sad.
| Pathogen-David wrote:
| As many others in this thread have stated, the main issue is they
| have fairly poor UX and if you aren't used to them they can be
| pretty annoying. They especially have quirks when they're removed
| from (or moved within) an existing Git repository.
|
| One thing I haven't seen mentioned in this thread though is that
| they force an opinion of HTTPS vs SSH for the remote repository.
|
| If a developer usually uses SSH, their credential manager might
| not be authenticated over HTTPS (if they even have one configured
| at all!) If they usually use HTTPS, they might not even have an
| SSH keypair associated with their identity. If they're on Windows
| setting up SSH is generally even higher friction than it is on
| Linux.
|
| For someone just casually cloning a repository this is a lot of
| friction right out of the gate, and they haven't even had to
| start dealing with deciphering your build instructions yet!
|
| -------
|
| Personally I still use Git submodules despite their flaws because
| they mesh well with my workflow. (This is partially due to the
| fact that when I transitioned from Hg to Git it was at a job that
| used them heavily.)
|
| The reality is every solution to bringing external code into your
| project (whether it's using submodules, subtrees, tools like
| Josh, scripts to clone separately, IDE features for multi-repo
| work, ensuring dependencies are always packages, or just plain
| ol' copy+pasting) all have different pros and cons. None of them
| are objectively bad, none are objectively best for all
| situations. You need to determine which workflow makes the most
| sense for you, your project, and your collaborators.
| al2o3cr wrote:
| A couple things off the top of my head:
|
| * some folks had to deal with a LOT of submodules back in the
| day; for instance, it wasn't uncommon to have a dozen+ in your
| "vendor/plugins" directory in a Rails 1.x app. More submodules,
| more problems
|
| * sometimes submodules get used to "decompose a monolith" into a
| confusing tangle of interdependent repos. Actually changing stuff
| winds up requiring coordination across multiple repos (and
| depending on the org, multiple teams) and velocity drops to zero.
| Eventually somebody declares "submodules SUCK!!!!!one!!!" and
| fixes things by re-monolithing...
| dathinab wrote:
| They are unergonomic!
|
| Annoying to setup and keep in sync "correctly" (for given
| project, EDIT: especially edits).
|
| Sure, this depends a bit on the other tooling and for what reason
| you use.
|
| This doesn't meant they are hard or complicated, but the moment
| the defaults do not do what most times is needed and you can't
| change the default (in the project, instead of user settings)
| i.e. need to do additional manual steps all the time some people
| will hate it many will dislike it.
| tkuraku wrote:
| Git submodules aren't perfect, but they can definitely be useful.
| I use them all the time and for my use case they are a good
| solution. They do take some getting used to though.
| zbuf wrote:
| If submodules were really slick, they'd become the primary way to
| manage build-time dependencies.
| mkl95 wrote:
| Git submodules aren't so bad, but they are a pretty leaky
| abstraction.
| truffdog wrote:
| Pointers are hard.
| withinboredom wrote:
| If you think git submodules are bad/confusing, wait until you see
| subtrees.
| arjvik wrote:
| What's the mental model for the use of a git submodule?
|
| I've always thought of them as a way to "vendor" a git
| repository, i.e. declare a dependency on a specific version of
| another project. I thought they made sense to use only when
| you're not actively developing the other project (at least within
| the same mental context). If you did want to develop the other
| project independently, I thought it best to clone it as a non-
| submodule somewhere else, push any commits, then pull them down
| into the submodule.
___________________________________________________________________
(page generated 2022-06-18 23:00 UTC)