[HN Gopher] Launch HN: Infield (YC W20) - Safer, faster dependen...
___________________________________________________________________
Launch HN: Infield (YC W20) - Safer, faster dependency upgrades
Hi HN, we're Steve, Allison, and Andy, founders of Infield
(https://infield.ai). Infield makes it clear which open source
dependencies you should upgrade next, and how to do so safely. We
do this by using a LLM to read every changelog. Here's a short
demo video: https://www.youtube.com/watch?v=diCGmtMUeRU We're
launching today with support for Ruby packages. If you've ever run
`bundle outdated` or upgraded a Rails app, Infield is for you. You
can try it on your own project at https://app.infield.ai/hn. Upload
your Gemfile and Gemfile.lock (no email/name/cc required) and we'll
show you Infield on your code. I (Steve) have been building open
source software and commercial web apps for more than a decade. I
spent the last year personally upgrading Rails apps by hand for
companies in order to research this problem. I'm convinced that
every company is re-inventing the wheel and doing by hand a bunch
of toilsome work that can be done with software. As one example, I
was working as a consultant upgrading an app to Rails 7. This
company was using the attr_encrypted gem to encrypt information at
the database level. Rails 7 brings built-in support for encryption
in a way that's incompatible with this gem. Having hit this same
problem at two other companies I already knew how to handle the
migration; but if I hadn't, they could have risked their most
sensitive customer data (this is what you tend to encrypt). After
that project I started building a personal database of "upgrade
experiences" and before long felt sure we could make useful
software. The time-consuming part of package upgrades is not
coding--it's mostly risk assessment, research, and project
planning. If I'm on a maintenance rotation and have half a day to
pay down some technical debt, which package upgrades should I look
at? I might end up spending that time trying to upgrade something
only to get blocked and give up. Worse, many breaking changes are
subtle and won't be caught by CI. I've brought down production only
to find an issue was buried in a changelog I didn't read! Infield
scans all of your dependencies to prioritize upgrades based on
effort (how much work is this? Is it risky?) and impact (will
upgrading fix a security issue? will it get me onto a supported
branch?). We can do this because we use GPT to read the changelog
for every package you rely on. Changelogs are broken apart into
discrete changes and classified according to the keepachangelog.com
standard. Then a human expert reviews the output. We can spend more
time researching each package than you because we're going to re-
use this work for every future customer doing the same upgrade.
Sometimes you want to do a complex upgrade like Rails that might be
blocked on other packages being upgraded first. For this case we
run an optimization based on the PubGrub algorithm to solve for a
suggested target version of all your dependencies that will be
compatible with the next version of Rails. We group and order these
blockers into an "Upgrade path" you can follow. Most of the
existing work in this space is security monitoring software like
Dependabot or Snyk. These tools are primarily sold to security
teams to let you know about CVEs that affect your dependencies.
They're reactive, a way to let developers know when they need to
upgrade something but not how to do it. Our goal with Infield is to
make it so easy to keep dependencies up to date that you're always
on the latest versions. Infield is $60/mo/repo and we're launching
today with support for Ruby. Javascript and Python are probably
next, but we're very interested to hear which language you feel
this pain in most acutely. Ruby is first since the consolidation
around Rails allows us to really nail the experience for a focused
set of packages. You can try Infield on your own codebase at
https://app.infield.ai/hn. With the paid plan we'll hook into your
codebase to continuously scan your dependencies as you merge PRs.
That works via a Github app, or you can use our CLI tool to send us
just the files we need as part of your CI pipeline. Please give it
a try and comments are welcome! We'd love to hear everything you
hate (or love, or just think) about dependency management and how
we can make it better!
Author : stevepike
Score : 40 points
Date : 2023-06-08 14:07 UTC (8 hours ago)
| aorth wrote:
| Sounds a bit like the Renovate bot https://docs.renovatebot.com/,
| but with LLM changelog parsing.
|
| I started using Renovate on a few project recently and I am
| impressed. Not quite yet to the point of letting it merge
| dependency updates for me, but for unearthing them and creating a
| pull request (after which my tests run automatically) is already
| a big help.
| ADLenehan wrote:
| Andy here (Product at Infield). I've also been impressed with
| what I've seen from Renovate - particularly on the
| configuration side. How has it worked for you on major /
| breaking upgrades?
| setgree wrote:
| Very cool! Is the LLM-on-the-backend a recent addition or have
| y'all been thinking about that potential since 2020? Would be
| curious to hear a bit more about your journey to market
| stevepike wrote:
| Thanks! We actually pivoted to working on this problem at the
| beginning of last year. We were previously called Syndetic and
| were building software for data providers. As we considered
| pivoting I spent some time thinking about the biggest problems
| I've faced maintaining software and started doing personal
| consulting. I upgraded a bunch of Rails and React apps for
| various companies over the course of 2022.
|
| Toward the middle of last year I re-connected with Andy (our
| third cofounder) who I went to school with and have wanted to
| work on a company with for a long time. We did a bunch of
| customer discovery / product work to figure out how to take
| what I learned doing this by hand and turn it into a software
| product. That was exciting enough that we decided to bring Andy
| on as a third co-founder and pivot our YC company.
|
| Allison and my background is in building data businesses.
| Before Infield and Syndetic we worked at a startup in the
| beverage industry where we standardized inventory data for
| every alcohol product sold in the US. As we got into building
| Infield we didn't expect to use LLMs at all. We imagined a
| similar human-in-the-loop expert system to what we've built
| before.
|
| I've been extremely impressed with recent language model's
| ability to handle unstructured changelog text. For example,
| consider the following snippet of a changelog:
| Security: - Address an issue with password validation
| Breaking change: - The `foo?` method now returns a
| boolean instead of int
|
| Language models can carry the context through, so we are able
| to not just parse this apart into discrete changes (which I
| could figure out how to do with a regex) but also bring in
| context and categorize them. It can do this generically and
| really feels like something new.
| setgree wrote:
| Very cool, I love hearing founder journeys. Best of luck to
| you all.
| xwowsersx wrote:
| Maybe it's just me but this wording seems weird to me, not sure
| why exactly:
|
| > World-class Engineering teams use Infield to make dependency
| maintenance time twice as productive.
|
| Specifically, "to make dependency maintenance time twice as
| productive." Feel like it ought to say "to cut the time they
| spend maintaining dependencies in half". I know that's wordier
| and maybe that's not quite it, but the wording as is feels a bit
| tortured to me.
| ADLenehan wrote:
| Andy (Infield co-founder) here - agreed that wording was a bit
| clunky. Changed now. Appreciate the feedback!
| ZeroCool2u wrote:
| Ruby is nice, but if you were to actually solve this problem for
| Python you'd print money. The deluge of knowledge required to
| actually navigate Python dependency management across all major
| OS's for the most complex use cases (typically ML stuff with deep
| compiler/tool chain deps) is nothing short of a nightmare.
| Especially on Windows, the rage I feel towards MSVC knows no
| bounds. venv, pip, (mini)conda, mamba, poetry, apt, yum...
| Obviously the last two are good to know anyways, but ugh the
| hours I've lost and will never get back to the rest.
|
| Anyways, god speed my dudes.
| stevepike wrote:
| (founder here). The interaction between high level language
| package managers and system binaries or compiled extensions is
| definitely on our roadmap. For example the ruby net-ssh gem
| relies on specific versions of openssl being compiled on the
| machine. Do you think this is particularly difficult in ML?
|
| Is your frustration mostly from needing compile all this non-
| python stuff across environments?
| ZeroCool2u wrote:
| Yeah, you're right on the money. This is basically insane in
| ML, but _especially_ in corporate environments, think F500,
| where sudo is disabled in containers and experimentation (in
| terms of trying different package versions) is quite
| challenging due to security restrictions.
|
| Frankly, it's one of the reasons that we're seeing a lot of
| excitement around stuff written in Rust, but with Python
| bindings[1][2]. Cargo makes it so easy to build across
| various platforms and then just produce statically compiled
| platform specific python wheels. If PyTorch and TF had been
| written using Rust, any other other ultra high perf language
| with a tool-chain as reliable as cargo, a lot of folks lives
| would be easier.
|
| [1]: https://www.getdaft.io/
|
| [2]: https://www.pola.rs/
| esafak wrote:
| Do you work at a company forcing you to develop on Windows? You
| couldn't pay to do to that.
| jchonphoenix wrote:
| I don't think you'd print money just by telling people if a
| dependency is hard or easy to install though. You'd need to
| actually solve the problem of getting the dependency installed
| or suggesting some alternative solution that gets the code
| written.
|
| E.g. if I'm banging my head against a wall trying to upgrade to
| the newest version of PyTorch for some graph embeddings feature
| and everything breaks, knowing it's high risk doesn't help me.
| I still need to get the thing installed so I can use the
| feature.
| ZeroCool2u wrote:
| Okay yeah, totally agree. I already know it's hard to
| install. If they gave me a path they were confident would
| work in terms of deps/versions to upgrade to, that I would
| pay for if it worked well.
| wodow wrote:
| What I would love to be standard in all product launches like
| this would be to have a mailing list per integration or, as with
| this, programming language.
|
| I want to be able to drop my email address and only hear back
| when support for $MY_FAVE_LANG is available.
| stevepike wrote:
| It's a bit buried, but if you go to https://app.infield.ai/hn
| and click "I don't use Ruby" we'll ask for your email/language
| and put you on our list.
| SkyPuncher wrote:
| I clicked into this because I was excited, but everything feels
| like it's missing the mark for me. And, that pricing. Woof.
| That's steep for being a fancy changelog interpreter.
|
| ----
|
| My biggest pain isn't with the plan. It's with the actual upgrade
| process. Things _always_ break. Every single time we do a major
| upgrade, there's this long tail of things we need to fix. Even
| worse is the undocumented changes that break. This gives me
| nightmares.
|
| The problem with your product is four parts:
|
| * It needs to solve much more of the problem. It's not really
| valuable to know that I need to change stuff. I can figure that
| out by running my package managers update tool and seeing what
| breaks.
|
| * When things break, it's unbelievably time consuming to fix
| them. Even worse is the person tasked with doing the upgrade is
| likely not the subject matter expert on broken code.
|
| * I need to know what's broken that wasn't documented. This is by
| far the hardest part of our upgrade cycles. It keeps me up at
| night knowing a major upgrade could break things that aren't
| covered by unit tests.
|
| * Because dependency upgrades are so hard, we tend to do them in
| large swoops. We set aside a week or two of pain and power
| through them. Given you're charging $60/m, this suggests:
| * Your model is more aligned with small, continuous upgrades.
| Once you're "close" to the most recent version, it tends to be
| easier to upgrade. * Your model needs a better way to
| "gain pace" with the latest releases. If I can incrementally
| upgrade one package on every PR until I'm up to date, that'd be
| amazing.
|
| -----
|
| I think this tool would be extremely value to me (likely
| $60/m/repo) if it:
|
| * It ran the repo's test suite against progressively newer
| versions of dependencies. Showing when and where unit tests fail.
|
| * Provide some means of "intelligent E2E" testing. Fire up our
| API/application. Tell us what changes when dependencies are
| upgraded.
|
| * Fan out broken code to the subject matter expert. If an upgrade
| from X to Y is broken, look at the `git blame` to figure out who
| knows how to fix the code. Ping them asking them for help.
| stevepike wrote:
| Thanks for the feedback. We know we're tackling a big problem
| here. Our ultimate goal is that if you use Infield we keep you
| on the latest version of every dependency, automatically. The
| road to get there can go a number of ways.
|
| > My biggest pain isn't with the plan. It's with the actual
| upgrade process. Things _always_ break. Every single time we do
| a major upgrade, there's this long tail of things we need to
| fix. Even worse is the undocumented changes that break. This
| gives me nightmares.
|
| Planning is one way to make those major upgrades go wrong less
| often. Our idea is that we can break large upgrades down into
| incremental changes that are individually safe and mixed in to
| your regular dev cycles, so that when you go to do the large
| upgrade you're not making such a big change all at once. This
| means things like fixing every deprecation ahead of time and
| making sure all the other dependencies you use are compatible
| with the version you're upgrading to.
|
| We're calling this "upgrade paths" in our tool. Something like
| a major rails version upgrade is terrifying because of all the
| moving parts. Often you may have a dozen other gems that need
| to be upgraded in order to upgrade Rails. You said "If I can
| incrementally upgrade one package on every PR until I'm up to
| date, that'd be amazing.". We want to make that the flow for
| major framework upgrades, so you're merging in incremental
| improvements in the blocking dependencies ahead of time.
|
| Undocumented breakages are for sure on our roadmap. We have two
| types in mind today: - Changes that are missing from the
| changelog - Incompatibilities across gems (for instance there
| was a time recently where if you had both datadog and newrelic
| installed and used elsaticsearch upgrading the newrelic gem
| would break prod)
|
| We have some ideas on how to figure this out automatically
| (like reading code diffs with GPT rather than just changelogs),
| but the best way is to see upgrades out in the wild. As we see
| more and more upgrade experiences from our customers we'll be
| able to catch these issues and build this dataset.
|
| I'd love to hear other ideas for making major framework
| upgrades safer.
|
| > * It ran the repo's test suite against progressively newer
| versions of dependencies. Showing when and where unit tests
| fail.
|
| Would this be something like `git bisect` for upgrades? We run
| your CI to figure out where you can upgrade to without
| something breaking? We're trying to figure this out statically
| by reading changelogs and building up a database of breaking
| changes. Our roadmap is to make this database better over time
| by pulling in more and more undocumented changes (by seeing
| customer upgrade experiences and by sourcing information from
| places like github issues). In my experience it's more common
| (and much more dangerous) that things break when your
| dependency changes in a way that wouldn't be caught by CI.
| ilikebits wrote:
| How are you computing upgrade paths? This seems impossible to do
| accurately, especially for Ruby, since you can't simulate the
| user's build's dependency resolution given that Gemfiles are
| dynamic.
| stevepike wrote:
| Can you expand a little? Here's some technical background on
| what we're doing:
|
| We have our own database of every version of every rubygems
| package alongside its runtime dependencies (like you see at
| https://rubygems.org/gems/pundit).
|
| Then we parse your Gemfile and Gemfile.lock. We use the Gemfile
| to figure out gem group and pinned requirements (we run turn
| your Gemfile into a ruby AST since Gemfiles can be arbitrary
| ruby code; we use bundler's APIs to parse your Gemfile.lock).
| This gives us all of the dependencies your rely on.
|
| Then we let you choose one or more package that you want to
| upgrade and the version you want to target (let's say Rails
| 7.0.4.3).
|
| Now we have [your dependencies and their current versions],
| [target rails version], [all of the runtime dependency
| constraints of these gems]. We run this through a dependency
| resolution algorithm (pubgrub). If it resolves then you're good
| to upgrade to that version of Rails without changing anything.
|
| If this fails to resolve, it's because one or more of your
| current dependencies has a runtime restriction on rails (or
| another indirect gem being pulled in by the new rails version).
| This is where the optimization part comes in. The problem
| becomes "what is the optimal set of versions of all your
| dependencies that would resolve with the next version of
| Rails". Currently we solve for this set trying to optimize for
| the fewest upgrades. As our dataset of breaking changes gets
| better we'll change that to optimizing for the "lowest effort".
|
| Happy to elaborate.
| Alifatisk wrote:
| Interesting, might try it out!
| [deleted]
| funerr wrote:
| Cool idea! I thought about it too a few years ago while doing
| some upgrades. Do you support more languages? And do you have it
| in CLI form?
| stevepike wrote:
| Only Ruby right now. We want to make it work really well for
| one language before expanding. Which language would you want it
| in?
|
| We have a command line tool you can use to send your lockfile
| to our server so we can process it, but you have to use our web
| app to view the results.
___________________________________________________________________
(page generated 2023-06-08 23:01 UTC)