[HN Gopher] Eden
___________________________________________________________________
Eden
Author : tosh
Score : 332 points
Date : 2022-04-12 17:48 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| psuedo_uh wrote:
| [deleted]
| gregwebs wrote:
| A Mercurial compatible SCM (not sure if it is a fork) built for
| their workflow (monorepo) and scale (enormous, git is not usable
| at their scale, at least for a monorepo). Uses Python and Rust.
| Designed for efficient centralization rather than
| decentralization.
| markstos wrote:
| The docs still refer to the the tool as Mecurial/hg:
|
| https://github.com/facebookexperimental/eden/tree/main/eden/...
| tux1968 wrote:
| It says it was originally based on Mercurial, but is no longer
| a distributed source code control system. Are you sure it's
| still compatible?
| deanCommie wrote:
| So if you are at Facebook's or Google's scale, and also run a
| monorepo, this will be great for you.
|
| Which is to say, this is a product for one company - Facebook.
| speedgoose wrote:
| It's interesting that they prefer to develop such a tool
| rather than giving up on the monorepo concept.
| sulam wrote:
| One very counterintuitive truth of large scale software
| development is that as you scale to multiple services, you
| are gravitationally pulled into a monorepo. There are
| different forces at work but the strongest at this scale
| are your data models and API definitions.
| gubatron wrote:
| riyadparvez wrote:
| Dan Luu wrote about monorepo. It's worth a read
| https://danluu.com/monorepo/
| simias wrote:
| Is it? I'm working for a small company and even at our
| scale, when the workflow is centralized, I find git a bit
| painful at times. I mean it's still an amazing tool, don't
| get me wrong, but when you have to deal with several sub-
| projects that you have to keep in sync and need to evolve
| together, I find that it gets messy real fast.
|
| I think the core issue with git is that the submodule thing
| is obviously an afterthought that's cobbled together on top
| of the preexisting SCM instead of something that was taken
| into account from the start. It's better than nothing, but
| it's probably the one aspect of git that sometimes makes me
| long for SVN.
|
| At the scale of something like Facebook you'd either have
| to pay a team to implement and support your split repo
| framework, or you'd have to pay a team to implement and
| support your monorepo framework. I don't have enough
| experience with such large codebases to claim expertise,
| but I would probably go for a monorepo as well based on my
| personal experience. Seems like the most straightforward,
| flexible and easier to scale approach.
| gen220 wrote:
| If your company is small, I don't think you should be
| using git submodules at all.
|
| My last place was about 10 years young, 150 engineers,
| and was still working within a single git repo without
| submodules.
|
| There is a non-zero amount of discoverable config that
| goes into managing a repo like that, but it's trivial
| compared to the ongoing headaches of managing submodules,
| like you suggest.
| ssl232 wrote:
| Do Facebook and Google literally have repos with everything
| they write in there available to everyone that works there
| (modulo privileged stuff)?
| xrikcus wrote:
| Not quite, but almost.
| klodolph wrote:
| You need good tooling to work with large monorepos, you
| need good tooling to work with large multirepos. Neither
| option is easy at that scale.
| pedrogpimenta wrote:
| Or Google, as per your first sentence :)
| onlyrealcuzzo wrote:
| Google already has its own [=
| dijit wrote:
| Would kill to have piper open source though.
| criddell wrote:
| Why wouldn't it be great for other companies that run a
| monorepo? Would it be nuts to go from some other monorepo
| (like Subversion) to this?
| madeofpalk wrote:
| I wouldn't think there would be many companies running
| repos as large as Facebook.
|
| If you use a tool like this you would basically be on your
| own. If you "stick to git" (or mercurial or whatever) at
| least you have all that momentum behind you, and you almost
| definitely won't be the first people to encounter a
| problem.
| voidfunc wrote:
| Can't wait for the "We switched to Eden from Git" posts in 6
| months.
| zelphirkalt wrote:
| Which will result in lots of people thinking "FB does it!
| It must be the future! We have to do this as well, or we
| will be left behind!" _sigh_.
| bin_bash wrote:
| tl;dr: automatic sparse checkouts for massive monorepos
| tazjin wrote:
| Also check out josh: https://github.com/josh-project/josh
| tosh wrote:
| I wonder if Eden is a reference to
| https://en.wikipedia.org/wiki/Eden_(2021_TV_series)
| inanutshellus wrote:
| The project's name goes back to 2016. Granted, the README
| called the project a "filesystem" back then not an SCM, but...
| still, the name seems to predate the series by quite a bit.
| NobodyNada wrote:
| Given that your link says the show came out in 2021, and the
| GitHub repo is at least 3 years old, probably not.
| benatkin wrote:
| I think it's probably from Facebook's culture of pretending
| it's a force for good when it's a mixed bag like every other
| Megacorp. Here's an example of what Zuck has said:
|
| > I believe the most important thing we can do is work to bring
| people closer together. It's so important that we're changing
| Facebook's whole mission to take this on.
|
| No wonder they have a major project named Yoga and now this...
| tekknolagi wrote:
| Pretty sure it's called yoga because it is a _flexible_
| layout engine. A joke.
| benatkin wrote:
| Here's another one: https://facebook.github.io/prophet/
|
| It's used to predict things, but still strikes me as an odd
| name, just like Yoga.
| pc86 wrote:
| The comment you're replying to is a perfectly good reason
| why it's named Yoga, not sure why it'd still seem "odd"
| with that context. Not particularly witty, but it
| definitely makes sense.
| [deleted]
| KerrAvon wrote:
| > it's a mixed bag like every other Megacorp
|
| No, you can't simply dismiss the fundamental problems with
| Facebook/Meta with whataboutism. Google and Apple and
| Microsoft are standard mixed bags. Not Facebook.
|
| Facebook is pure evil. It has queued up the complete
| obliteration of western democracy, which even Rupert Murdoch
| couldn't quite manage on his own. You can absolutely find
| non-directly-evil things to at Facebook/Meta, but it all
| supports the evil in the end.
|
| https://www.theatlantic.com/magazine/archive/2022/05/social-.
| ..
| loeg wrote:
| Nah, it's older than 2021.
| benatkin wrote:
| If Eden the TV series was based on a book, the book could be
| older, but it isn't.
|
| It's also possible, but not likely, and not the case here,
| that a TV show could be a sensation long before the first
| episode comes out. I can't think of a time when this happened
| except for a prequel or sequel like Better Call Saul, where
| it was much awaited, but I'm sure there are instances of that
| occurring.
|
| Edit: from another comment, there's this, which came out in
| 1997 but isn't mentioned in the Wikipedia article for the TV
| series so I'm not sure it's related: https://en.m.wikipedia.o
| rg/wiki/Eden:_It%27s_an_Endless_Worl... There is a Manga
| mentioned in the article for the TV series but it came out
| the same year as the TV show.
| Lammy wrote:
| https://bindingofisaacrebirth.fandom.com/wiki/Eden
| layer8 wrote:
| If anything, rather
| https://en.m.wikipedia.org/wiki/Eden:_It%27s_an_Endless_Worl...
| hestefisk wrote:
| _Proceeds to book edenhub.com_
| tzury wrote:
| they use git (and github) to maintain their own source control
| management system?
| tazjin wrote:
| No, they have a system similar to Google's copybara which
| exports this repository out of their internal monorepo.
| Barrin92 wrote:
| nothing would be wrong with it if they did. A control
| management system suited for gigantic monorepos isn't itself
| necessarily a gigantic monorepo.
| Ancapistani wrote:
| I think it's a tall order for another SCM to challenge git. I
| can't imagine how it could be any more entrenched in the
| industry.
|
| Further, I'm happy with git. I played with Mercurial years ago,
| long enough to work with it day-to-day, and just didn't find any
| relevant advantages versus git.
|
| I love that people are still out there trying to improve things,
| and certainly don't want that to stop, but it's difficult for me
| to imagine switching at this point.
| mountainriver wrote:
| I would just love it if git could handle large files, it causes
| a lot of hacky solutions right now
| jchw wrote:
| If you go to work for Google or Facebook, it quickly becomes
| apparent that switching SCMs is much cheaper than trying to use
| ordinary Git or Mercurial at scale. (Though it is clear that
| Google, Facebook and Microsoft are all trying to maximize the
| amount of familiarity and not reinvent the wheel too much; they
| all have been working on tools either building on, utilizing,
| or based on already existing SCM tools.)
| optymizer wrote:
| I used Meta's Mercurial, having previously used primarily git
| (and SVN, and CVS before that). It has a number of very cool
| improvements over git, and it's well integrated into the rest
| of their infrastructure.
|
| You know the feeling of having to use SVN after using Git? This
| is what it feels like to use Git after getting used to Meta's
| Mercurial. I wish I could go into the details, but I don't know
| how much of it was ported back to Mercurial.
| qbasic_forever wrote:
| I don't think it's trying to compete with git, it's not
| decentralized or meant to support big distributed open source
| project development. This looks like a nice tool for Big
| Company to manage its internal, private code repositories.
| z3t4 wrote:
| The decentralized part of Git and Mercurial is nice (eg. no
| need for Github et.al), but I think most software projects
| using Git or Mercurial _do_ have a centralized server /hub...
| efsavage wrote:
| The biggest differentiation for me between Git and Mercurial is
| that Mercurial is _far_ better for code reviews because it
| manages stacks of "as small as possible" changes much easier.
| The git workarounds I've tried to replicate 'hg histedit' and
| 'hg absorb' are ... not good.
|
| Similarly, I think Git(hub) has succeeded in open source
| because bundling more complete changes into PRs works well for
| unevenly distributed contributions.
| avgcorrection wrote:
| This looks like it's supposed to be more appropriate for very,
| very big repos. Which current Git doesn't support and isn't
| fundamentally designed to support.
|
| So rather than use Git + Git Annex or something like that
| (maybe more), you'll just use this alternative SCM.
|
| (I keep hearing about how Git will eventually support big
| repos, but it's still not on the horizon. Big repos with Git
| still seems to be limited to big corps who have the resources
| to make something bespoke. Personally I don't use big repos so
| I have no skin in this game.)
| m12k wrote:
| I'm also happy with git, but there's 3 main things that could
| improve on git IMO:
|
| 1) Better handling of large files than git-lfs. As in 10+ GB
| repos. This is needed for game development (currently they tend
| to use Perforce or PlasticSCM)
|
| 2) Sparse checkout via file system integration (like Eden has)
|
| 3) Build system integration, so unchanged files and modules
| don't even need to be fetched to be compiled, because cached
| builds can be fetched from a build server instead (requires
| proper modularization, so e.g. C++ macro expansion doesn't just
| prevent anything from being cacheable)
|
| These are all features that primarily have value for repos that
| push the limits on size, like big monorepos (with a huge amount
| of files) or game development (with big asset files). But get
| it right, and you could massively cut down the time it takes to
| check out a branch and build it.
| thomascgalvin wrote:
| ClearCase (claimed to) support points two and three way back
| in the 90s. It never really worked right; it was always
| trying to "wink-in" someone else's binary, despite the fact
| that it was built using a different version of the source.
| 411111111111111 wrote:
| > _Build system integration_
|
| It would be a game changer if any SCM actually manages to
| achieve this without needing multiple developer teams just
| for maintenance.
|
| That would probably break gits current dominance within a
| relatively short timeframe... But I don't think it'll happen
| within the foreseeable future,
| avar wrote:
| If you want an SCM with direct build system integration
| there's always GNU make with its support for RCS :)
|
| More seriously, can you describe what "build system
| integration" would look like? Basically like what GNU make
| does with RCS? I.e. considering the SCM "files" as sources.
|
| How would such a system build a dumb tarball snapshot?
| webmaven wrote:
| _> These are all features that primarily have value for repos
| that push the limits on size, like big monorepos (with a huge
| amount of files) or game development (with big asset files).
| But get it right, and you could massively cut down the time
| it takes to check out a branch and build it._
|
| Would this be a good fit for large machine learning data
| sets?
|
| Every time a new model comes out that touts having been
| trained on something like a quarter of a billion images, I
| ask myself "how the heck does someone manage and version a
| pile like that?"
| kvnhn wrote:
| This is by no means a perfect match for your requirements,
| but I'll share a CLI tool I built, called Dud[0]. At the
| least it may spur some ideas.
|
| Dud is meant to be a companion to SCM (e.g. Git) for large
| files. I was turned off of Git LFS after a couple failed
| attempts at using it for data science work. DVC[1] is an
| improvement in many ways, but it has some rough edges and
| serious performance issues[2].
|
| With Dud I focused on speed and simplicity. To your three
| points above:
|
| 1) Dud can comfortably track datasets in the 100s of GBs. In
| practice, the bottleneck is your disk I/O speed.
|
| 2) Dud checks out binaries as links by default, so it's super
| fast to switch between commits.
|
| 3) Dud includes a means to build data pipelines -- think
| Makefiles with less footguns. Dud can detect when outputs are
| up to date and skip executing a pipeline stage.
|
| I hope this helps, and I'd be happy to chat about it.
|
| [0]: https://github.com/kevin-hanselman/dud
|
| [1]: https://dvc.org
|
| [2]: https://github.com/kevin-hanselman/dud#concrete-
| differences-...
| bastardoperator wrote:
| Not handling large binary files is a feature from my
| perspective. Git is for source code and treating it like a
| disk or a catch all for your project is how people get into
| scaling trouble. I don't see any reason why versioned
| artifacts can't be attached to a build, or better, in a warm
| cache. I get that it's easier to keep everything together,
| but we can look at the roots of git and it becomes fairly
| obvious, Linus wasn't checking in 48GB 4K mpeg files for his
| next FPS.
| markstos wrote:
| Coming from Darcs, Git has a horrible user interface. Something
| like Jujutsu has a chance to disrupt Git.
|
| It can use the Git data store, so developers can in theory
| start using it without the whole team adopting it. Then it
| addresses the big problem with Git, the user interface:
|
| https://github.com/martinvonz/jj
|
| I'm not suggesting that "jj" in particular will disrupt Git,
| but I think eventually the a tool which supports the git data
| store with a better user interface could take hold.
| tick_tock_tick wrote:
| Git usage for most developer is 3-4 commands or once in a
| blue moon when they fuck up badly save a copy of your changes
| and reset hard. There aren't enough user interface
| improvements possible to get people to switch.
| shashashasha___ wrote:
| it solves scale issues that git can't solve at the moment. fb
| monorepos are huge. so for most people/companies this issue is
| not critical to solve and git is great.
| ithkuil wrote:
| A SCM like https://pijul.org/ would be a challenge to git. Eden
| is more a challenge for things like perforce.
| noobermin wrote:
| I don't think them putting it out is an attempt to compete with
| anything, they're just putting it out to put it out.
|
| If they are then I totally agree.
| cproctor wrote:
| OK, slightly off-topic but maybe the right minds are here. We
| have been developing an introductory CS curriculum committed to
| thinking-with-powerful-tools, including the command line, real
| programming languages, and git. It's great until it isn't. We
| intentionally maintain a simplified workflow, but still get the
| occasional merge conflict or local state blocking a pull. I
| keep thinking there must be a simplified wrapper over git which
| maintains the conceptual power while avoiding the sharp edges,
| even if at the cost of robustness. I'd be more interested in an
| abstraction than a GUI, but would be interested to hear
| whatever others have come up with.
| avar wrote:
| Can you show a step by step list of git commands where you
| find the interface to be bad?
| gabrielrc wrote:
| Unfortunately their open source support is very little to non
| existent.
|
| It's great, and it doesn't work outside Facebook.
| bjackman wrote:
| I'm actually curious what the strategy is here. To my knowledge
| only FB, Google and MS do megascale monorepo, and Google and MS
| already have a solution. Are there now other companies
| outgrowing Git that Facebook is hoping to build a community
| with?
| travisd wrote:
| Facebook has a terrible track record when it comes to open-
| sourcing their internal tools. See: Phabricator, HHVM, Flow,
| Jest, ...
|
| Even React, which is their most popular library, is not
| actually "open source." They're very transparent about the fact
| that their priorities are Facebook's needs -- even if they do
| take community input.
|
| None of this is per-se _bad_ , but you should definitely treat
| an open-source project out of Facebook with skepticism when it
| comes to adopting it for your own use cases (possibly making
| sure you're not too locked in when an incompatible v2 comes out
| with virtually no warning after FB's internal implementation
| drifts).
| peterhunt wrote:
| This is not a fair characterization of React. The tech lead
| of the project doesn't even work at FB anymore.
| hn_throwaway_99 wrote:
| What's wrong with Flow, Jest or Graphql? I think these are
| all fantastic projects. I mean, Flow "lost out" to
| Typescript, but, it's usual for one winner to emerge from
| competing frameworks.
| paulhodge wrote:
| Graphql is great. With Jest, that project feels a little
| abandoned because the Typescript support (ts-jest) is
| pretty janky and has bad performance. Meanwhile in the
| ecosystem in 2022, it's becoming the norm to have first-
| class Typescript support.
| folkrav wrote:
| > Even React, which is their most popular library, is not
| actually "open source."
|
| How do you define "open source"? It typically simply means
| the source code is available. By any definition I can think
| of, React is definitely both free and open source. How they
| design the software or if they take contributors isn't really
| relevant.
| xcambar wrote:
| nowadays, it is expected that open source is also a synonym
| for "community-driven", which is a very bold assumption.
|
| I don't know how we got to this point but it's interesting
| to notice that the terminology drifted.
| picardo wrote:
| By that definition, any software project that is driven
| by a BDFL wouldn't be open source, including Linux.
|
| The terminology hasn't drifted that much. I think there
| is a small and vocal group of people who are trying to
| take back "open source" by reframing what it means, so as
| to exclude corporate projects. It doesn't make sense to
| me.
| naikrovek wrote:
| > How do you define "open source"? It typically simply
| means the source code is available.
|
| that is not what that means. "source is available" and
| "open source" are _very_ different.
| colinmhayes wrote:
| Disagree. We need to stop trying to cram meaning into
| phrases that are already defined. Open source means the
| source code is freely available. Adding anything else
| requires a different name.
| TheDong wrote:
| > How do you define "open source"? It typically simply
| means the source code is available.
|
| I agree with you that react is definitely open source, but
| I'd also encourage you to use more specific wording around
| what "open source" means.
|
| I think wikipedia gets this right:
| https://en.wikipedia.org/wiki/Open-source_software
|
| "Open-source software (OSS) is computer software that is
| released under a license in which the copyright holder
| grants users the rights to use, study, change, and
| distribute the software and its source code to anyone and
| for any purpose"
|
| The license is the key bit.
|
| On the other hand, there's "source available" software
| (also on wikipedia https://en.wikipedia.org/wiki/Source-
| available_software ), which is what your definition equates
| to, and I personally don't want to see confused with open
| source or free software.
| folkrav wrote:
| React is MIT, which is pretty much one of the least
| restrictive ones, granting basically no rights to the
| original publisher.
|
| That Wikipedia definition sounds more like what typically
| gets described as "Free and Open-Source Software"/FOSS,
| no?
| TheDong wrote:
| I'm aware React is MIT and of the various licenses etc.
|
| As I see it commonly defined, "open source software",
| FOSS, and FLOSS all mean the same thing more or less.
| That the project uses an OSI approved license, or one
| very close to it, whether it's MIT, Apache2, or GPL.
|
| "Free software" is the only of the phrases that I see
| having two competing common definitions, the "free as in
| money", and "free as in Free Software Foundation's
| definition of free software". This seems pretty
| understandable, since "free" is overloaded.
|
| I only infrequently see people mixing up "open source"
| and "source available", and that's the specific thing I'm
| trying to discourage people from mixing up. I think
| keeping those terms clear, and especially calling out
| "source available" software as _not_ being "open source"
| (i.e. not granting you the freedom to modify it or run
| your own copy in some cases) is important.
| kodah wrote:
| There's three domains where people usually use the term
| "open source".
|
| - Freely licensed software (eg: MIT, GPL, etc)
|
| - Code visibility (eg: ForgeRock)
|
| - Community focus/contributions
|
| Not all "open source" implements each of these, and just
| because they implement one and not the other doesn't mean
| they're not expressly open source. Just some ecosystems
| are _more_ open than others.
| Lammy wrote:
| Free Software == my project's code, to others
|
| Open Source == other peoples' contributions, to my
| project's code
| TheDong wrote:
| I have not seen this distinction commonly.
|
| Can you link to a reference?
|
| As I understand it, "Free Software" is the term the FSF
| and general hacker community settled on for licenses that
| preserve user's freedom to modify and redistribute source
| code.
|
| O'Reilly etc shifted to the term Open Source as part of
| making the idea less associated with "hacker culture",
| and more associated with businesses (as described here:
| https://en.wikipedia.org/wiki/Open-
| source_software#End_of_19... )
|
| From there, I see "Open source" being slightly more often
| associated with companies or younger developers, and
| "free software" more often being associated with the GNU
| project, copyleft projects, etc.
|
| I'm curious if you have references or more explanation
| about the difference you're trying to draw, since it's
| one I haven't seen before.
| Lammy wrote:
| > I'm curious if you have references or more explanation
| about the difference you're trying to draw, since it's
| one I haven't seen before.
|
| My distinction between the two is whether outside
| contributions make it back into the original project.
| Free Software is about the rights of end users to inspect
| the code and make and distribute their own modifications,
| but then Open Source takes it a bit further by explicitly
| soliciting contributions with the ostensible aim of
| building a better project through cooperative labor than
| an individual programmer could build alone.
|
| In practice though "Open Source" has turned into unpaid
| project management work for billion-dollar corporations,
| bitter disputes between contributors over conflicting
| standards of morality, technical visions in constant flux
| as contributors come and go, and endless bikeshedding
| about semantic version numbers / code style guides /
| other things that don't matter. For years I thought I was
| totally burned out on Free Software and walked away from
| all of it, but what I was actually burned out on is Open
| Source and have been able to love programming again by
| working on things that are explicitly "Free Software but
| not Open Source".
|
| The `actix-web` drama a few years ago is a perfect
| example, when a huge crowd of onlookers felt morally
| justified excoriating a popular project's creator /
| maintainer for not managing their project to the crowd's
| standards: https://steveklabnik.com/writing/a-sad-day-
| for-rust
| naoqj wrote:
| >Open-source software (OSS) is computer software that is
| released under a license in which the copyright holder
| grants users the rights to use, study, change, and
| distribute the software and its source code to anyone and
| for any purpose
|
| ??? Isn't React licenced under the MIT licence? It seems
| to me that it tickes all boxes?
| TheDong wrote:
| As I wrote at the top of my comment. "I agree with you
| that react is definitely open source" (and no, I did not
| edit that in, it was there when you read it)
|
| I'm aware react is licensed under the MIT license. I was
| just talking about how the parent comment chose to define
| "open source".
| renewiltord wrote:
| Everyone is going on tangents, but yes React is as open-
| source as they come. What people are conflating are the
| notion of community-driven and open-source.
|
| React is a successful Facebook OSS project. An example of
| one which went poorly is Thrift. Facebook open-sourced it
| and then internally used fbthrift which diverged
| drastically. OSS Thrift isn't that popular these days any
| more.
| lala45 wrote:
| Free Software is an established term, and React certainly
| is not Free Software, due to its patent. I very much doubt
| that React is Open Source either, for the same reason.
| sneak wrote:
| No, open source means that the software is free software.
|
| The source code to Windows is available.
| tptacek wrote:
| React is MIT-licensed. It is, of course, open source. If you
| don't like their decisions, fork it.
| lala45 wrote:
| React is patented and Facebook is actively choosing not to
| offer a patent grant, so unfortunately that's not the whole
| story.
|
| A fork may not be an option. Perhaps a given organization
| may not even be allowed to use React, if Facebook decides
| against it for some reason. Jurisdictions can differ as
| well.
|
| In my opinion, the best thing would be if Facebook simply
| made the terms clear by using the Apache license or
| similar. But hey, it's Facebook, so I'm not expecting
| much...
| bjt wrote:
| What patents does Facebook have on React?
| aliveli wrote:
| Stay tuned ;)
| ripphab wrote:
| For what, another Phabricator, that I'll inevitably have to
| migrate my company away from again?
|
| So if history serves the next announcement to watch for is
| your departure from Facebook and the launch of Edenity, which
| will be sunset and abandoned inside a decade once it fails to
| IPO. Am I close?
| epage wrote:
| A bit skeptical when there are relatively recent commits like
|
| > Re-sync with internal repository
| joatmon-snoo wrote:
| Open sourcing an internal repository with extensive,
| ongoing work on it is always a difficult affair, because
| you're creating a second source of truth. (It isn't just
| how you manage external contributions, but also workflows
| like releases and CI.)
|
| I wouldn't consider this to be a problem.
| epage wrote:
| It means they haven't (yet?) transitioned to open-first
| and it'll require proving themselves that they'll do
| open-first development before trusting them. Not willing
| to bet my work on a product where the governance isn't
| open but everything is driven by and for a single
| company's needs.
| bogwog wrote:
| Does this do anything special for handling binary data, or is it
| mostly for text like git? I've heard that Perforce (another
| centralized SCM) does a good job with large binaries.
|
| I love git just as much as the next guy, but git-lfs sucks.
| martincmartin wrote:
| I worked at Meta and used Eden. I remember it as a virtual,
| FUSE based file system for a huge monorepo. Basically, if you
| have GBs of data in your monorepo, and any individual user
| accesses < 1% of it, then there's no point in having each "hg
| update" update the other 99%.
|
| But we were explicitly dissuaded from having binary data. I
| worked on image processing, and wanted to store test images for
| unit tests in hg, but the consensus was, that was a bad idea.
| So we stored them in a separate data store, and had a make rule
| to fetch them as part of the build.
| maxmcd wrote:
| Is there a FUSE filesystem for Git that takes a similar strategy
| to edenFS?
|
| Only setting up files and folders as they are requested might be
| very helpful with various git monorepo access patterns. Maybe
| there is something inherit to Git's design that makes this less
| practical.
| zellyn wrote:
| https://github.com/microsoft/VFSForGit
|
| Which seems to have been superseded by
| https://github.com/microsoft/scalar according to the README.
| ctur wrote:
| The SCM ecosystem at Facebook is tremendously powerful and the
| result of some of the best minds at Facebook working on those
| systems for many years. From the scaling of the monorepos to the
| code review workflows, nothing really matches it. The ergonomics
| of most of the tooling was simply top notch (which it needed to
| be... engineers, particularly at Meta, are an opinionated lot who
| don't tolerate poor tools).
|
| It's great to see this out in the wild now.
| Yeroc wrote:
| Do you know how this compares to Microsoft's efforts to scale
| git via the Scalar [https://github.com/microsoft/scalar]
| project?
| jjcm wrote:
| FWIW this is also true on the design side. I've researched a
| lot of design teams and their custom tooling (I worked on
| design tools at Atlassian and now am working on them at Figma).
| Facebooks is leagues above the rest. I've been blown away at
| what they do there. Some really amazing engineering and design
| thinking happening on Facebook's internal tooling.
| influx wrote:
| It's powerful, but fbcode is sloooooooow.
| martincmartin wrote:
| How so? I never thought of it as slow.
|
| Source: Was a Meta infra engineer until last month, working
| in CDN & LogDevice, among other teams.
| influx wrote:
| What else have you used? Source: Meta infra PE until last
| month, but have used build and source system at Amazon and
| found it faster.
| martincmartin wrote:
| Yeah the fbcode build system was hella slow. A minute
| plus just to build the "action graph." I did marvel at
| all the lost productivity, especially since an minute
| plus is enough time for me to context switch to another
| task, or at least go get (another) cup of tea...
|
| Mercurial / Eden were pretty fast though. Never had
| complaints about them.
| tehlike wrote:
| As a facebook employee, and a former google employee, I prefer
| google's SCM system and tooling instead..
|
| Cider, Piper, CitC, Blaze, etc make a really complete system.
| Nothing beats it.
|
| This is kind of also surprising because product speed at Fb is
| much better :)
| peppertree wrote:
| After doing a round of onsite with fb tools team, I got the
| impression there were lots of bright engineers that wanted fb
| pay without having to touch fb products.
| jfengel wrote:
| A lot of programmers would rather write tools for other
| programmers than for non-programmer end users. It's the
| target market that they know best, and it's most prestigious.
|
| I wish more of those bright engineers would try to solve
| problems for other users rather than re-re-re-re-re-
| optimizing the life of their colleagues. But writing code for
| users involves, among other things, knowing something about
| users, which is more bogged down and less fun.
| mym1990 wrote:
| Their colleagues ARE users, they are just a different type,
| and a type that engineers understand more.
|
| I would rather a good engineer put their skills towards
| helping others use their skills more effectively than put
| that same engineer in a place where they don't like the
| work and produce below par results.
| xiphias2 wrote:
| For some of us it's really hard to understand / solve
| problems of people whose life is spent on browsing
| instagram most of the time - when it's not TikTok.
| jackomelon wrote:
| Keep the holier-than-thou judgmental stuff to yourself.
| Your comment adds nothing.
| eyelidlessness wrote:
| You're getting downvoted, and that's not undeserved
| because this _is_ very judgmental on its face. But I get
| the sense there's a sincere answer to a few sincere
| questions worth at least asking: have you asked them
| (users you've categorized this way) that direct question?
| Or asked how work you're invested in isn't addressing
| their problems /goals? If so, and if you found their
| answers unrelatable, did you ask further questions which
| might clarify for you what they want?
|
| I know _all_ of that is a lot to ask of anyone. But you
| might find you learn things you have in common with
| people you don't expect to. And it might make your job
| less frictionful too.
| xiphias2 wrote:
| It's not like I don't understand their problems/goals.
| Just yesterday the person I was spending time with showed
| a cat video from Facebook, and as I seemed half
| interested only, she asked me if I like animals.
|
| I love animals in real life, but that doesn't mean that I
| love watching videos of animals on a screen, for me it's
| not the same experience.
|
| The main goal that I see social networks addressing with
| many people is entertainment, but I feel that it takes
| away from my social life with people, that's why I have
| aversion to short videos for example.
|
| Facebook didn't have these short videos forced on me, but
| just a few months ago it started to show video reels for
| me, and I couldn't stop myself watching them.
|
| After bringing the settings back to just showing what's
| happening with my friends, the next time I opened it, it
| started showing addictive videos to me again.
|
| Youtube disabled showing downvotes, which was the best
| signal for me to see if it's worth to see a 15 minute
| video, or if it's just a spam talking about nothing for
| 15 minutes just to optimize content length for the
| algorithm. And shorts are not a solution, as they don't
| have any depth either. Also it started spamming me with
| new channels that reupload videos of interesting people
| from years ago just to make me think that it's new
| content.
|
| The main problem for me working on social networks is
| that making profit can't be separated from making people
| addicted to the lowest forms of entertainment (I'm a
| Xoogler).
| hobs wrote:
| Start by removing a heap of condescension and recognizing
| that there's interesting problems all around you.
| chrisseaton wrote:
| This is an unkind comment. Why condescend like this?
| xiphias2 wrote:
| I'm not sure why is it condescending. I have met people
| who do this all the time while I'm trying to just have a
| drink/dinner or get to know a person, but I'm just not
| able to connect with heavy instagram/facebook users. I
| prefer not to see them again, as for me it's very boring,
| and I see them as being unkind, when in all fairness they
| are probably just social network addicts.
| chrisseaton wrote:
| It's condescending because you're focusing on the media
| that people use rather than what they have to say in
| life. There are (obviously) millions of interesting,
| passionate, intelligent, creative people using both of
| these apps.
| OJFord wrote:
| But on the contrary.. many of the things well-built for the
| majority of 'non-programmer end users' are not the way I
| (as a 'programmer end user', whether the particular thing
| is a programming-adjacent tool or not) would ideally like
| them to be.
|
| So I'd selfishly adjust it to 'solve other problems for the
| same users'! Asahi, for example, awesome stuff - I've lost
| count how many times I've had to explain (to both
| colleagues and non-engineer family/friends/etc.) 'no no,
| absolutely agree, love Mac _hardware_ [...] '.
| seanmcdirmid wrote:
| > I got the impression there were lots of bright engineers
| that wanted fb pay without having to touch fb products.
|
| I chose a job at Google Engprod (a tooling/developer
| productivity org) because I specifically like to work on dev
| tooling and I'm much less interested in working on products
| aimed at end users. Surely, many engineers at FB feel the
| same way.
| LightG wrote:
| I'm sorry, Meta has only just been born. These are Facebook
| engineers.
|
| It turns my stomach to read that name which is just a cloak for
| the dagger ...
| 0xbadcafebee wrote:
| Hm. I can see why they'd build some of these features, but
| there's some significant downsides. The VFS in particular will
| end up a poor experience when a transitory network problem causes
| apps to hang when pulling code. What happens if you 'grep -r', or
| if 'mlocate' indexes it?
|
| On the build side.... holy jesus, are they really compiling 40
| different dependencies from scratch every time they push code?
| This build has been running for 5 minutes and it's still just
| compiling dependencies:
| https://github.com/facebookexperimental/eden/runs/5997101905...
| Come on, ya'll. You're supposed to be the "advanced FAANG
| people". Cache some build deps, will ya?
| hammycheesy wrote:
| A similar effort was made by Microsoft a while back to scale up
| Git in order to use with the Windows monorepo:
| https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...
| 3a2d29 wrote:
| Junior engineer here, so got isn't used for the windows repo? I
| assumed git was the holy tool or all version control.
| wincy wrote:
| I thought this a few years ago when I was more junior as
| well. In open source circles git has been used for quite
| awhile but my company just recently moved one of our repos
| from TFVC to git. A Fortune 500 company I worked for was
| still using TFVC for some legacy products. Another product
| from a acquisition used Subversion and the migration to git
| (even with the company using GitHub Enterprise) still took
| almost three years.
|
| Getting the inertia amongst developers to migrate to a
| different SCM can be quite the challenge.
|
| Edit: initially said migrating to git was a challenge. But
| really it's migrating from any SCM to another that's
| challenging.
| actuallyalys wrote:
| There are a lot of advantages to using git, in part because
| it was developed for a massive project (the Linux kernel) so
| it's very thoroughly tested, in part because it's exploded in
| popularity so a lot of a tools, integrations, and
| documentation are available for it. However, it's not the
| alpha and omega of version control. Some people prefer the
| interface of Mercurial, some people use Fossil because it
| integrates issues and wikis, others have stuck to older tools
| like Subversion, and still others are experimenting with new
| approaches like Pijul.
| dharmaturtle wrote:
| git was created in 2005. No idea what MS uses, though some
| history is here: https://en.wikipedia.org/wiki/Microsoft_Visu
| al_SourceSafe#Mi...
| criddell wrote:
| Microsoft used their own fork of Perforce for a long time.
| Not sure what they use today.
| strongpigeon wrote:
| Source Depot is what it was called. IIRC, Windows was
| moving to Git (through some virtual file system [0]) and
| had some major teams actively using it, but that was a
| while ago too.
|
| [0] https://github.com/microsoft/VFSForGit
| xiphias2 wrote:
| It looks like Google's Perforce/Piper, I think that would be a
| better comparision than Git, but I don't see any data on how the
| two compares. Anyways, Eden being open source is a great
| advantage already.
| zelphirkalt wrote:
| Never forget though, that being open source doesn't mean that
| much, when there is only _one_ party deciding, what goes into
| the code base.
| josephg wrote:
| Being opensource does mean it's free. And can dig in and read
| the code. And if you want to, fork it and add your own
| features.
| bb88 wrote:
| It's GPL 2.0, on github, which makes it ridiculously easy to
| fork, tear apart, debug, etc, given your time and ability.
|
| That said, I would like to see it self hosted. I think then
| and only then will I give it a serious look.
| krashidov wrote:
| Facebook/Meta's dev tooling, libraries, and infrastructure has
| always impressed me. React, Hack, jest, and now this. I am very
| surprised they haven't tried to monetize this aspect of their
| business like Microsoft and Amazon did.
|
| They could be a very strong contender in that space instead of
| being known as a terrible time suck for humanity.
___________________________________________________________________
(page generated 2022-04-12 23:00 UTC)