[HN Gopher] Flyte: A Cloud Native Machine Learning and Data Proc...
___________________________________________________________________
Flyte: A Cloud Native Machine Learning and Data Processing Platform
Author : mgrover
Score : 54 points
Date : 2020-01-07 18:11 UTC (4 hours ago)
(HTM) web link (eng.lyft.com)
(TXT) w3m dump (eng.lyft.com)
| minimaxir wrote:
| How does this differ from Airflow? https://airflow.apache.org/
|
| Looking at the docs for both, Flyte has similar functionality to
| Airflow, except less mature and a more functional Task
| specification syntax. Airflow has the same data ETL operators as
| well, plus a few more.
| flytehub wrote:
| Here is a blog post I wrote a few weeks ago that describes how
| Flyte's structured workflow specification allows for using
| open-source workflows with no code.
|
| https://medium.com/@flytehub/introducing-flytehub-open-sourc...
| kumare3 wrote:
| Great question, I am working on a follow up blog that will
| explain the differences in more detail. Flyte does take some
| inspiration from airflow, but it has a lot of important
| differences - Flyte natively understands data flow between
| tasks. This is achieved using its own type system created in
| protobuf - Flyte tasks are first class citizens and hence can
| be shared, reused and are always associated with an interface
| declaration - Flyte is container and kibernetes native. It is
| also multi tenant. - Flyte corn scheduler, control plane api
| and the actual execution engine are decoupled. Each workflow
| can be independently executed on a different execution engine -
| Flyte workflows are purely specification - defined in protobuf
| and Flyte tasks also - Flyte provides an event stream of the
| execution - since Flyte is aware of the data, it comes with
| built in memorization and auto cataloging - like airflow Flyte
| can have plugins in python, but it supports a richer plugin
| interface - Flyte is written in Golang and on top of
| kuberenetes It is definitely less mature in the open source, so
| please help us make it better. But it has been battle tested at
| Lyft for more than 3 years in production.
| roberto wrote:
| Any plans to support Python type annotations, instead of using
| the @inputs and @outputs decorators?
| matthewphsmith wrote:
| That is something to which the SDK was designed to be extended
| --in fact, the SDK was designed specifically to be extended to
| different programming models besides the rudimentary one we
| provide out-of-box. However, we have not had the time to take
| on the implementation ourselves as the existing one serves our
| purposes at Lyft well and we are but a small team. So we do
| hope that someone in the community with a knack for syntax can
| help improve things!
___________________________________________________________________
(page generated 2020-01-07 23:00 UTC)