[HN Gopher] The Unbundling of Airflow ___________________________________________________________________ The Unbundling of Airflow Author : gorkemyurt Score : 25 points Date : 2022-02-15 19:32 UTC (3 hours ago) (HTM) web link (blog.fal.ai) (TXT) w3m dump (blog.fal.ai) | gorkemyurt wrote: | High Resolution Version of the diagram if anyone is interested | | https://drive.google.com/file/d/1btZ0yck9SdgsUdNom0WXgHcSQvO... | DevKoala wrote: | I love Airflow. Plenty of data businesses I've built are nothing | more than just one DAG. | | As for the article, I don't think we are yet at the point in | which a competing stack comprised of individual specialized | components do things better since Airflow is more than the sum of | its parts imho. | fdgsdfogijq wrote: | This post is hard to follow. But I'll give my unsolicited opinion | on airflow: | | Its too complex to run as a single team and there are far better | tools out there for scheduling. Airflow only makes sense when you | need complex logic surrounding when to run jobs, how to backfill, | when to backfill, and complex dependency trees. Otherwise, you | are much better off with something like AWS step functions. | DevKoala wrote: | I run Airflow even for my local trading setup. For large teams, | I often go with managed solutions like Astronomer. | zukzuk wrote: | Everyone's context is different, but I've found the exact | opposite to be true. Airflow is simple and dumb enough that it | can be easily understood and managed by a small team, but it's | also flexible and powerful enough that we can't come up with a | good enough reason to switch to anything else. | samdjstephens wrote: | Interesting, I wouldn't say that I've found it difficult to run | in even a small team. | | The problem I've always had with Airflow has been with non- | cron-like use cases, for example data pipelines kicked off when | some event occurs. Sensors were often an awkward fit and the | HTTP API was quite immature back when I was using it | ajoseps wrote: | do you have recommendations on alternatives that are not tied | to a cloud provider? | jonpon wrote: | We are trying to build something like this at | https://www.magniv.app/. | | Would love to have you join our beta if you are interested! | fmakunbound wrote: | It's ok, but seems to be a bit too complex for what it does. It | was pretty janky running it locally (pegged the CPU), and now | that we have it in MWAA we've got several support issues on it | with AWS for unkillable task instances and scheduler problems. ___________________________________________________________________ (page generated 2022-02-15 23:00 UTC)