[HN Gopher] Red Engine: modern scheduling framework for Python a...
___________________________________________________________________
Red Engine: modern scheduling framework for Python applications
Author : jonbaer
Score : 112 points
Date : 2022-07-03 17:27 UTC (5 hours ago)
(HTM) web link (red-engine.readthedocs.io)
(TXT) w3m dump (red-engine.readthedocs.io)
| daenz wrote:
| As cool as it is, scheduled jobs (and jobs in general) should
| really be isolated from the rest of your system and from each
| other to limit the blast radius. Jobs are notorious for crashing
| and backing up, so you really don't want that impacting other
| systems on the same machine.
|
| Clouds have managed systems for scheduled jobs now (AWS
| Eventbridge, GCP Cloud Scheduler) which handle this for you.
| wmichelin wrote:
| How does it handle state and restarts? What happens if a job is
| scheduled to run "before 10am", then the entire server restarts
| at 9:55am, will it try to run that same job again when it boots
| back up?
| Closi wrote:
| It will run it if the scheduler is called before 10am according
| to the docs - at runtime the conditions must be met.
| angrais wrote:
| If it took less than 5 minutes to boot then I can't see why it
| wouldn't work.
|
| How would that work for other schedulers? Also, if a server
| reboots that's quite bad all round anyway. Hopefully you'd be
| notified directly.
| vore wrote:
| I think on the contrary, it's important to build software
| that's resilient to random reboots: you can't control things
| like losing power or your CPU failing, so the less toil you
| have to manage around that the better.
| mplewis wrote:
| Other schedulers have a durable database of attempted runs.
| This doesn't seem to have anything like that.
| dr_kiszonka wrote:
| Would you have a recommendation for an easy to use Python
| scheduler with such a feature for a personal project?
| nrjames wrote:
| Is this meant for use instead of something like Luigi?
| krastanov wrote:
| This fits my use case so perfectly! I have a very small internal
| app taking care of organizing seminar talks, calendars, email
| announcements, recordings of the talks, and signups. It is a
| single python file of less than 1500 lines and an sqlite
| database. This library is so perfect for taking care of scheduled
| events. Everything else I have found is ridiculously over-
| complicated of a solution.
| hungryroark wrote:
| Can you share repo to your application?
| bckr wrote:
| Yeah, I'll definitely try this for little server-based tasks
| vorpalhex wrote:
| If you're going to build a framework like this, please please put
| your consistency behavior and retry behavior at the top of your
| docs.
| ancieque wrote:
| Yes, 100% this. Too many frameworks claim that they are "easy"
| just to find out they left out things that other older
| frameworks have already solved.
|
| While we are on this: Do you know of any task scheduler
| framework that is similar to celery, but has better guarantees
| around task execution than what acks_late= True gives you?
|
| I always find myself building a system that stores the really
| important Tasks in Postgres so that I can recover from anything
| in the broker or celery crashing. What i use celery for is just
| scheduling these Tasks by creating a celery job with the
| Postgres Job ID as the parameter.
|
| Then, to detect if something went horribly wrong, I have a
| sweeping job that checks if any job in Postgres has not run in
| celery for some reason. If that is the case, we just re-queue
| the job.
| ancieque wrote:
| After reading the linked article in more detail, I might have
| misinterpreted things a bit, but my concerns are still
| related :)
| vorpalhex wrote:
| If your tasks are idempotent, Dramatiq is intended for your
| case.
|
| https://dramatiq.io/
| nerdponx wrote:
| > Clean: Scheduling is just plain English
|
| Ugh, no thanks.
|
| First of all, English itself is not clean; it's a messy
| amalgamation of special cases and inconsistent spelling rules.
|
| Second, it isn't actually English anyway. It might look like
| English, but it's actually a DSL that happens to correspond to
| English a lot of the time. English text is meant to be
| interpreted by humans, who understand context & connotation, and
| can resolve ambiguities by making educated guesses or discussing
| the text with other humans. But your English-like DSL can only be
| interpreted by a computer program, which cannot (and arguably
| should not) do such things. Ergo, the benefits of using natural
| language are lost, and you are left with the same strict
| interpretation rules as any other programming language, but
| without any of the syntactic rigor that would normally help you
| construct programs/expressions that are both syntactically
| correct and also do what you intended them to do. Finally, the
| passing similarity to another language is a newbie trap and it
| makes teaching more difficult. See also: SQL, Python.
|
| Worse still, it's represented in code as a string literal. It
| cannot be reasonably syntax-highlighted or otherwise
| statistically analyzed, nor can it be easily constructed
| dynamically if needed, nor can scheduling primitives be combined
| or composed. It is the worst of all worlds, and you have no way
| to check if your program is valid other than to run it and see if
| it crashes. And you have to re-learn operator precedence /
| associativity rules, because they probably won't be identical to
| the rules in Python itself.
|
| I'm sorry if there is a really high quality scheduling engine
| underneath this DSL, but I absolutely would never want to use
| something like this in production code.
|
| (I'm sure you can guess how I feel about BDD frameworks and
| "expect.foo.to.be.equal.to" style test APIs).
| klysm wrote:
| The mistake of trying to make programming languages look like
| English looks like it will be repeated forever. English is not
| a good language for expressing things specifically to computers
| or other humans for that matter.
| OJFord wrote:
| It's also just annoyingly redundant, and breaks searching,
| completion, etc.: @app.cond('is foo')
| def is_foo():
|
| IMO @app.cond def is_foo():
|
| And 'losing' that you can omit the underscore to make it look a
| bit like English is much better. Plus then I can jump from its
| use back to this definition without any DSL-aware tooling (that
| probably doesn't exist).
| New_California wrote:
| You nailed it.
| Volker_W wrote:
| > See also: SQL, Python
|
| I do not see what is wrong with SQL and I definitely do not see
| what is wrong with Python.
| Groxx wrote:
| @app.task('daily & is foo', execution="process")
|
| followed by @app.task("after task
| 'do_daily'")
|
| Yeah, I did a hard turn towards "nope" right there.
|
| Similar but not quite Python combined with similar but not
| quite English does not make a tasty dish. It makes yet another
| pointless one-off thing to learn and struggle with.
| nerdponx wrote:
| These in particular remind me of "Baba Is You", which is
| probably not a good thing for an API.
| elcapitan wrote:
| Just got hard AppleScript PTSD.
| aeyes wrote:
| For some reason they chose to put the concept which is
| hardest to understand on the front page. 'is foo' is showing
| how to define and use a custom condition, the conditions in
| the rest of the documentation are easy to understand and make
| sense to me.
|
| The only way I could see me struggle with this would be to
| piece together a large call graph in my head. I can
| understand why the author says that Airflow is better suited
| for this case because you get a visualization.
| mft_ wrote:
| It's plain English like SQL is plain English. It uses English
| words that's kind-of make sense to a novice, but that doesn't
| stop it being a new programming language.
| nomdep wrote:
| Looks nice but very limited.
|
| I don't see any mention to serialization, so I guess this is
| single-server and memory only. I also couldn't find any mention
| to error handling or retries.
| cercatrova wrote:
| Not to be confused with REDengine by CD Projekt Red, used to make
| the Witcher and Cyberpunk 2077 games [0]. They're moving to
| Unreal Engine 5 though.
|
| [0] https://witcher.fandom.com/wiki/REDengine
| djvdq wrote:
| Yeah, when I read the title I was like: "wtf, CDPR released
| Python version of their engine?"
| daemin wrote:
| I too first thought this had something to do with RED Engine
| 4 from CDPR.
|
| Unfortunately CDPR's latest RED Engine 4 will not be released
| as open source in the forseeable future, it's basically dead
| and locked away, and will probably never see the light of
| day. Maybe it would be open sourced but in something like 20
| years time.
| efxhoy wrote:
| from redengine import RedEngine app = RedEngine()
| @app.task('daily') def do_things(): ...
| if __name__ == "__main__": app.run()
|
| > We initialized the RedEngine application, created one task
| which _runs every 10 seconds_ and then we started the app.
| https://red-engine.readthedocs.io/en/stable/tutorial/quick_s...
|
| Might wanna fix that.
| wodenokoto wrote:
| Off topic but,
|
| Many scheduling system I've worked with have this weird tendency
| to run when deployed and then every time it's time for it to run.
|
| I've had this happen with kubernetes, Scheduled Quries in GCP big
| query and a few other systems.
|
| That seems like absolute madness. Why would anything do that?
| qwertox wrote:
| I'm using APScheduler for most of my scheduling tasks.
|
| Does Red Engine integrate with asyncio? When searching the docs
| for this keyword no hits showed up.
| bergundy wrote:
| Going to shamelessly plug Temporal's Python SDK which was
| designed for asyncio.
|
| https://github.com/temporalio/sdk-python
|
| Disclaimer: I work for Temporal
| swagonomixxx wrote:
| Using Python decorators is a very strange choice for constructing
| computational graphs. Then again, I don't really use Python for
| this kind of thing, I roll my own solutions (of just functions
| being composed together).
|
| One big issue I have with the proposed approach is that it's very
| difficult for me to see at a glance the actual compute graph. I
| suppose you can build some tools to visualize it from the DSL in
| the decorator call, but I'd much rather be able to see this
| directly in code, with no weird magic, so that I can very easily
| interpret and update it if need be.
| gigatexal wrote:
| What makes this true?
|
| "Red Engine is not meant to be the scheduler for enterprise
| pipelines, unlike Airflow, but it is fantastic to power your
| Python applications."
| timcavel wrote:
| Wronnay wrote:
| It sounds like the author doesn't think enterprise apps can be
| Python applications
___________________________________________________________________
(page generated 2022-07-03 23:00 UTC)