[HN Gopher] Flow-Based Programming
___________________________________________________________________
Flow-Based Programming
Author : hypomnemata
Score : 127 points
Date : 2021-01-20 16:42 UTC (6 hours ago)
(HTM) web link (jpaulm.github.io)
(TXT) w3m dump (jpaulm.github.io)
| blackrock wrote:
| Fascinating. This is how I build my programs. I never knew it had
| a particular name for the style. I just assumed it was
| functional-style programming.
|
| This allows me to build ever greater programs, that does very
| complicated things, and has thousands of lines of code and logic,
| but everything gets boiled down to small bite-sized pieces.
|
| It's a more data oriented approach. And it seems to follow an
| assembly line model.
|
| I liken it to building code pyramids. Where I keep stacking and
| chaining one function to another, to build even bigger pyramids.
|
| At the end, all the code is heavily unit tested, and I have a
| high degree of trust in the fidelity of the codebase.
| Geminidog wrote:
| This type of programming is actually a subset of functional
| programming called point free programming. It's equivalent to
| programming using only combinators as the fundamental unit.
|
| https://en.wikipedia.org/wiki/Tacit_programming
|
| All programs are actually pipelines of data flowing from a low
| entropy state to high entropy state with IO and state as the
| endpoints of the pipes-. Using the point free style or flow based
| programming makes this entropy and the pipelined nature of all
| programs more explicit.
|
| Using OOP or regular programming the pipeline nature of data
| flowing to and from state and IO becomes less evident and more
| convoluted.
| bergie wrote:
| Nice to see this here, as it has been the inspiration for quite a
| few years of my work: https://noflojs.org/
| leetrout wrote:
| NoFlo really is impressive. Great work.
| jes5199 wrote:
| neat to see this here! I've recently become convinced Flow Based
| Programming is the ideal that other programming paradigms are
| reaching for. But in practice, there's a lot that's not obvious
| to me how it would actually work-- so it's great to have a bunch
| of new reading suggestions!
| akavel wrote:
| Mandatory fanboi mention: https://luna-lang.org/ - now Enso:
| https://medium.com/@enso_org/enso-dev-blog-18th-december-202...
| pantsforbirds wrote:
| Luna is one of the coolest projects I've seen. I've really
| enjoyed watching the progress.
|
| Curious to see if it (or concepts from it) gets picked up for
| programming education one day.
| samuell wrote:
| I'm super intrigued by luna/enso. Only that every time I've
| tried it, the editor has had serious stability problems. I
| wonder if a more traditional tooling and editor support
| wouldn't be more successful for wide adoption?
| devmunchies wrote:
| Similar concept to what I do when programming in F#/OCaml by
| default. You create your data types and then they "flow" through
| functions. Each function is `input -> output` but bigger workflow
| are also input -> output.
| adamnemecek wrote:
| The main difference is that with data flow, the flows happen
| conceptually at the same time.
| voldacar wrote:
| This is a good article, in the history or multithreading sections
| it would have been nice to mention the various old hardware
| implementations of this model, such as the transputer or
| connection machine
| fahrrad34 wrote:
| Do I understand it correct when I think that clojure's core.async
| is an example implementation?
| dustingetz wrote:
| not quite, but this seems close:
| https://github.com/leonoel/missionary
| adamkl wrote:
| Rich Hickey gave a good talk on core.async and it does seem
| pretty similar in concept:
|
| https://youtu.be/drmNlZVkUeE
| teknopurge wrote:
| https://nodered.org/ - great project for all sorts of needs.
| jcims wrote:
| I recently used nodered for some process automation and it was
| absolutely fantastic for quickly prototyping, dashboarding,
| doing sensor integration, etc. Highly recommended!!!
| analog31 wrote:
| It might not tick all of the boxes for being a complete software
| development tool, but Excel strikes me as being a dataflow
| programming model. I wonder if it's a reason why it's easy for
| laypeople to learn.
| pantsforbirds wrote:
| You can write pretty interesting data pipelines using akka
| streams with a very similar idea to FBP. It's probably not
| technically FBP, but the whole reactive stream implementation is
| similar in thought but allows for things like disparate data
| source speeds without blowing out part of the flow.
| smartmic wrote:
| A nice and powerful C++ flow-based programming framework is
| DSPatch [1]. It is definitely worth to mention here, check out
| also the examples in the audio domain.
|
| [1] http://flowbasedprogramming.com/docs/html/index.html
| jgraettinger1 wrote:
| We're building a tool, Estuary Flow, which seeks to be an end-to-
| end realization of practical, configuration driven, and scale-out
| flow-based programming -- with an important twist.
|
| The central concept is a "collection", an append-only set of
| schematized documents, which can be _captured_ and _materialized_
| into other systems (e.x. pub /sub, S3 buckets, etc).
| "Derivations" are collections defined in terms of source
| collections, and stateful transformations/joins/aggregations
| which are applied to them.
|
| A key twist is that collections are simultaneously a batch
| dataset (backed by cloud-storage) and also a real-time stream.
| They unify the current dichotomy of "historical" vs "streaming"
| data into a single addressed concept. Declare a new derivation,
| and it automatically back-fills over history right from S3, then
| seamlessly transitions to live data.
|
| If this sounds interesting, check out our docs [0]. We're early,
| but love feedback!
|
| [0] https://estuary.readthedocs.io/en/latest/README.html
| hpoe wrote:
| As I was reading this I was enthusiastically agreeing with the
| idea, but something felt super familiar about it. Then I realized
| this is basically the same concept as Unix pipes.
|
| I have my little individual programs, grep, awk, sed, jq, etc,
| and then I can endlessly mix and match those different components
| to do what I want.
|
| The limitation that I have seen with Unix pipes isn't often with
| the ability to process or manage the data it is that it only
| works if all the data is setup the proper way. As I have been in
| the industry longer it seems to me that most code that gets
| written isn't about actual computing but just centered around
| importing and transforming data that is expressed in different
| ways.
|
| Is there something that makes reading in disparate data records
| easier so I can focus on the computing part of computer program
| and less time on parsing data?
| adamnemecek wrote:
| Unix pipes make it hard to have a large graph. Like most pipes
| take one thing and produce one thing. You don't really have
| complex data flow graphs.
| taeric wrote:
| Though, this really kind of fits with the metaphor. Pipes
| typically just send liquid.
|
| You could have a conduit connector to join a bunch of
| wires/signals, I suppose. But, realistically, that is a mess
| always.
| adamnemecek wrote:
| Right but you generally have a complex pipe system even
| with liquids. Sewers send waste to a particular processing
| facility.
|
| A node editor makes it somewhat less messy.
| ledauphin wrote:
| I don't mean to derail the conversation, but this really does
| remind me of the game Factorio, though sort of in reverse.
|
| In Factorio, you build a larger and larger factory out of pre-
| established functional components (assemblers, labs, chemical
| plants, etc) that take in a limited set of inputs and produce
| (usually) a single output. Your challenge is not to define the
| functional core processes, but instead to wire together those
| functional components by connecting their inputs and outputs in
| ever-more-automated fashion, starting by hand, then using simple
| belts (pipes) that eventually allow arbitrary load-balancing via
| "splitters", and eventually through to trains (the forking and
| load balancing happening via backpressure in the train system)
| and robots (where everything is managed essentially as a single
| state database of requests, and backpressure is provided by
| output limitations, usually per functional component).
|
| Naively, I think that someday a decent chunk of programming might
| actually look like this, and parts even be represented visually
| (though in my opinion likely still defined formally as text).
| Only I think programmers will continue to write the functional
| components themselves, unlike in Factorio. They'll just live on
| different levels of the "codebase", and the "pipes" level will
| likely be a lot more abstracted than it is in Factorio.
|
| As a software developer, I find this paradigm to map very well to
| serverless architectures, because you generally want to think a
| level higher than the per-machine basis. It does require a
| willingness to forgo handy and well-established tools like the
| filesystem and Unix pipes in favor of higher level abstractions
| around transfer and storage of data.
| freeqaz wrote:
| You have, from first principles, reconstructed a huge portion
| of my thought process for building https://refinery.io
|
| Factorio and Minecraft automation mods are a big inspiration!
| Check out InfiniFactory too :)
|
| Bridging existing applications to the Serverless paradigm is
| far from simple. That's one of the biggest struggles I've
| experienced trying to build a Flow-based software platform.
|
| Learn more every day though. Thank you for the interesting
| comment!
| mikewarot wrote:
| I took a look at your site (refinery.io), and the "watch
| demo" is really a "read introduction"... I was expecting to
| sit back and let you show me 10-15 minutes of video that
| sells the idea.
|
| It looks very professionally done, but reading screens isn't
| as easy as it used to be for me, a video is better.
|
| Good luck!
| freeqaz wrote:
| Thank you for the thoughts! I really should sit down and
| record a video. I'll figure out how to get that put
| together soon :)
| dgb23 wrote:
| Another video game that works similarly is Oxygen not Included.
|
| One of the most compelling arguments for a data-flow/flow-based
| programming is the mental model and the visualization aspect of
| it. This opens up opportunities for monitoring, visual, data-
| driven programming and it is white board friendly.
|
| In Elements of Clojure[0], the author discusses the concept of
| "principled components and adaptive systems". And a flow based
| design reminds me of exactly that. The semantics of composition
| and communication are well-defined and universal, but
| internally the components can (should) be specific and
| concrete.
|
| Similar can be said about Small Talk as well. A primary aspect
| of its design was the mental model, understanding and learning.
| The core idea was that learners (especially children)
| understand things in terms of their operational semantics.
|
| > I don't mean to derail the conversation, but this really does
| remind me of the game Factorio, though sort of in reverse.
|
| So no, I don't think this is a derailment, but likely one of
| the most important aspects of paradigms like this.
|
| [0]https://elementsofclojure.com/
| ledauphin wrote:
| thank you for making this connection - I had recently been
| reading Elements of Clojure and noticed the concept of
| principled components, and I had been slowly working my way
| towards this realization/analogy myself.
| u678u wrote:
| Its great but surely its more like OO programming perhaps without
| inheritance.
| amelius wrote:
| Flow-based programming is cool, until the flow starts altering
| the flow.
| cjohnson318 wrote:
| I think at that point it's a complex dynamic system.
| galaxyLogic wrote:
| In Dataflow -programming (or something like that) how do you
| program a decision that depends on the result of some component?
|
| It would seem like I need to suspend my computation then "ask"
| the result from some component, get the result back, and then
| alter my computation based on that result.
|
| Pure flow-forward would not seem to support this easily. Or can
| it? Or does it come down to that we always will need BOTH sync-
| and async- functions? (unless of course we limit the problem
| domain)
| dustingetz wrote:
| I think you are talking about Self-Adjusting Computation, so
| the answer is yes dataflow can do that
| https://blog.janestreet.com/breaking-down-frp/
| DonaldFisk wrote:
| Yes, it can.
|
| The simplest way is with when/unless vertices which take a
| boolean input and some other inputs, and which either output
| the other inputs if the boolean input is true/false, or output
| nothing.
| erichocean wrote:
| Spreadsheets are a variant of Dataflow programming, so however
| they do it.
| phreeza wrote:
| Apache Beam seems to be an implementation of this idea. It works
| well when the things you have to do matches the logic, but it
| gets tricky if you need to do stuff iteratively or recursively.
| samuell wrote:
| Beam has some overlap, but in my understanding it has a rather
| involved syntax for defining the data flows, quite far from the
| simple list of connections between in and out ports that you
| see in FBP systems following J P Morrisons principles more
| closely.
|
| Apache NiFi comes quite a lot closer, with the main difference
| that they only have a single in-port, instead of separate named
| ones. Also it seems to be among the more heavy and complex
| implementations (for good and bad).
| jpaulmorrison wrote:
| "They" meaning "FBP-like" or "FBP-inspired systems", to use
| Joe Witt's terminology... They are usually control-flow
| oriented, and to my mind do not yield the productivity and
| maintainability advantages of the "classical" FBP paradigm
| shift. I recently asked Node-RED's Nick O'Leary why Node-RED
| only allows one input port, and the answer was something like
| "because we have never run into the requirement"... a) this
| is not something that can easily be added after the fact,
| without totally rethinking the product architecture, and b)
| trying to build complex systems without that feature would
| be, to me, like trying to hang wallpaper with only one hand!
| One litmus test I use is that of concatenating data streams
| using the two different paradigms - I tried to describe this
| in an article a year or so ago, recently updated:
| https://jpaulm.github.io/fbp/concat.html Cheers!
| samuell wrote:
| The core of the FBP principles are the holy grail of true
| componentized function architecture.
|
| Instead of losing yourself in ever more complex syntax
| convolutions as has happened in a lot of functional programming,
| you make the components (even long running stateful ones) self
| contained with ports for data input and output as the sole means
| of communicating with them, over buffered channels to allow
| asynchronous computation, and most importantly keep the network
| definition separate.
|
| Just this idea in itself is just brilliant. Hats off to Mr
| Morrison for that!
|
| It allows to decouple complex software into reusable components,
| without clever FP syntax.
|
| (Though, FP is perfect for implementing the components
| themselves. It just doesn't really scale all too well for whole
| program architecture, in my experience).
|
| One point to note: The visual component of many fbp systems is
| completely optional, as is the idea of using novel DSLs. You can
| as well define your networks and components in pure code. See
| GoFlow https://github.com/trustmaster/goflow) and my own little
| experiment FlowBase (https://flowbase.org) for examples of that,
| in Go.
|
| I successfully built a rather complex little app to convert from
| Semantic web RDF format to (semantic) mediawiki XML dump format,
| in two weeks straight, of linear development time: for each
| component (of ca 7), implement, test, go to the next component
| (See: https://github.com/rdfio/rdf2smw)
|
| The same implementation in procedural PHP took months, and still
| doesn't have all bugs and strange behaviours filed out.
| codetrotter wrote:
| Your flowbase.org domain redirects to a different domain. I
| thought that perhaps you had a typo in the URL or forgot to
| renew your registration of the domain, but upon a closer look I
| think it's probably just a misconfiguration in your webserver
| config causing it to redirect to another one of your projects.
|
| My guess is that you forgot to put the definition of the domain
| sans www in your webserver config.
|
| If my guess is correct then this link should work:
|
| https://www.flowbase.org
|
| Edit: doesn't work either. Perhaps you don't have a TLS cert
| for that domain? In which case maybe this link will work
| instead:
|
| http://www.flowbase.org
|
| Edit 2: Without https it works. And like the person responding
| to this comment said it redirects to the GH repo.
| samuell wrote:
| Ouch, sorry, and thanks! Wrote on mobile on the go and didn't
| manage to double check the link.
| wizzwizz4 wrote:
| The link's trying to point you here:
| https://github.com/flowbase/flowbase
| samuell wrote:
| Thank you!
| the_duke wrote:
| > you make the components self contained with ports for data
| input and output as the sole means of communicating with them,
| over buffered channels to allow asynchronous computation
|
| This concept sounds exactly like actor systems like Erlang/OTP
| and Akka, only with a different set of terminology.
|
| The submitted site and your comment don't mention those
| anywhere.
|
| Are there appreciable differences between actor systems and
| FBP?
| megameter wrote:
| FBP is most analogous to the unit record machines of yore:
|
| https://en.m.wikipedia.org/wiki/Unit_record_equipment
|
| What you are programming is the graph of how these machines
| connect and flow data(keypunch cards) through each other,
| while the machines themselves and their function are
| abstracted out. Data does not stay "at rest", it's presumed
| to reach a terminating point where it exits the graph. A
| buildup of unprocessed data in a machine's inbox results in
| an overflow.
|
| It's a very useful model for making a debuggable asynchronous
| system since it imposes static constraints everywhere that
| you can map to your real hardware resources, versus the
| emphasis on dynamism seen in the actor model(actors hold
| private data, modify their state, create new actors - all
| explicitly disallowed in FBP).
| samuell wrote:
| Actor systems differ in their means of communication. While
| actor systems use the actors as the main units that you
| connect, with fire and forget messaging to each actor's
| inbox, FBP proposes communication between named ports over
| channels with bounded buffers, very much like the CSP
| implementation in Go (just that you have long running
| components as in actor systems - something that can easily be
| implemented in Go with structs with go channels in fields for
| the ports, as shown in the FlowBase library I mentioned in
| another comment).
|
| The FBP model provides implicit backpressure, because of the
| bounded buffers on the channels. The actor model on the other
| hand is more loosely coupled and flexible.
|
| This actually means FBP systems are slightly more optimal for
| efficient within-one-node, in-memory parallellisation,
| whereas actor systems shine more for distributed systems.
|
| They shine on different scales, in other words.
|
| I wrote a post many years ago on this, that seems to have
| aligned with the experience of a number of people, based on
| the reactions in the comments:
| https://rillabs.com/posts/flowbased-vs-erlang-message-
| passin...
| bcrosby95 wrote:
| > The FBP model provides implicit backpressure, because of
| the bounded buffers on the channels. The actor model on the
| other hand is more loosely coupled and flexible.
|
| Note that this is why people recommend using GenServer's
| "call" instead of "cast" in Elixir/Erlang by default - it
| applies a sort of backpressure because call waits for the
| process to return a result rather than being fire and
| forget.
|
| It's not exactly the same thing, or as configurable as Go
| channels, but it is an option.
| Nullabillity wrote:
| Akka Streams adds named ports and defined backpressure, as
| well as a graphical-ish DSL for connecting everything.[0]
|
| [0]: https://doc.akka.io/docs/akka/current/stream/stream-
| graphs.h...
| samuell wrote:
| That is nice to see! I was evaluating Akka a bit some
| years ago.
| dustingetz wrote:
| FP decouples the AST (symbolic functions with input ports and
| an output port) from the evaluation context, which might be
| async, or sync, backpressured, stateful, exceptional,
| incremental/reactive ... mix-n-match whatever behaviors you
| want, all for the same _abstract_ AST
| samuell wrote:
| I actually didn't want to make this too much of an FP-bashing
| comment, as I find great use of many FP concepts, daily.
|
| I have yet to see an FP concept for composition though, that
| is as simple and generic in its implementation, as the FBP
| principles.
|
| I have sometimes thought FBP networks provide roughly the
| same function (pun not intended) as a monad, although I never
| seem to fully grasp what a monad is, so I can't tell for sure
| :o)
| dustingetz wrote:
| Yes, dataflow with dynamic topologies (self-adjusting
| computation) forms a monad. If the "flowchart" is merely
| static, that forms an applicative.
| https://blog.janestreet.com/breaking-down-frp/ In my
| opinion, understanding the monad is essential to
| understanding how to fully separate an AST from its
| evaluation behavior. That is basically what a monad _is_ ,
| the monad is the computational structure that _does that_.
| samuell wrote:
| Thanks, that clarifies a lot!
| dmbarbour wrote:
| I think most FP languages don't decouple AST from evaluation
| context. E.g. I cannot perform abstract interpretation on a
| function's AST, nor rewrite a function's AST for incremental
| computation, nor perform explicit partial evaluation during
| function composition by composing the ASTs. I only can access
| opaque `Input -> Output`.
|
| Also, FP algorithms developed for one evaluation context
| cannot easily be ported to another. They are implicitly
| entangled with assumptions. For example, I cannot evaluate a
| lazy algorithm in an eager evaluation context without paying
| a huge price. Adding back-pressure for list processing where
| a program was not designed for it would easily result in
| deadlock. Adding exceptions without adding unwind (or
| bracket, try/finally, etc.) to the existing program
| expressions will easily result in buggy code.
|
| I think your assertion might be true for some specific
| integrations of FP nodes into FBP. Is this what you mean?
| FigmentEngine wrote:
| Sounds like XProc 3.0 https://xproc.org/
| megameter wrote:
| Having studied and implemented FBP systems in the past, one
| major takeaway I've gleaned is that most automation problems
| start off as a linear sequence of processes, and so the
| branching-graph of FBP looks unwieldy and superfluous. But this
| is deceptive; you probably don't want to have a huge number of
| branches in the design, but you will want them as an
| optimization step or a way to combine inputs.
|
| So it's useful to design with FBP in mind but with a linear
| interface as the entry point.
|
| Another aspect of this is that FBP graphs are static but you
| may have a need to reconfigure them frequently; that is, you
| may want to have a graph compilation step drawn from a source
| language, rather than manually wiring it up.
|
| A way of making that graph compilation more than a syntax is to
| include a formal constraint solver: Excel, for example, flows
| the data after determining a solution for how cells relate to
| each other. The power of the spreadsheet paradigm really lies
| in these combinations of concepts.
|
| Lastly, there isn't really magic in the
| algorithmic/implementation aspects of FBP. It grew out of
| 1960's mainframe types of problems, and so it can be
| implemented in a low level way with static pools of memory and
| pieces of assembly code. But it remains conceptually just as
| relevant to today's massive distributed systems.
| hexo wrote:
| This reminds me of reactive programming, especially functional
| reactive programming.
| mikewarot wrote:
| Side effects are forbidden by structure, flows could be monitored
| in a GUI/Debugger, and as a result components can be tested as a
| unit, instead of a whole system. I love it!
|
| It is easier to design digital circuits when you have a whole
| catalog of 7400 and 4000 series gates, than it is using
| individual transistors. It is easier to wire a house when you're
| not making wires and switches with a hammer and a forge.
|
| I welcome this new higher level of abstraction, and am willing to
| pay the cost in terms of CPU and Memory to get there, just as I'm
| willing to waste transistors or copper wire to have something
| done and working.
| dang wrote:
| If curious see also
|
| from 2015, on the same article:
| https://news.ycombinator.com/item?id=8867584
|
| Related:
|
| 2019 (a bit) https://news.ycombinator.com/item?id=20215592
|
| 2019 (another bit) https://news.ycombinator.com/item?id=19203642
|
| 2019 https://news.ycombinator.com/item?id=18859019
|
| 2015 (not that good)
| https://news.ycombinator.com/item?id=10755250
|
| 2015 (a bit better) https://news.ycombinator.com/item?id=9718868
|
| 2015 https://news.ycombinator.com/item?id=8992281
|
| 2013 https://news.ycombinator.com/item?id=6264657
|
| 2012 https://news.ycombinator.com/item?id=4239274
| homieg33 wrote:
| Very useful list. Thanks for providing.
___________________________________________________________________
(page generated 2021-01-20 23:01 UTC)