[HN Gopher] Show HN: Streamdal - an open-source tail -f for your...
___________________________________________________________________
Show HN: Streamdal - an open-source tail -f for your data
Hey there! This is Dan and Ustin (@uzarubin), and we want to share
something cool we've been working on for the past year - an open-
source _`tail -f`_ for your data, with a UI. We call it _"
Streamdal"_ which is a word salad for streaming systems (because we
love them) and DAL or data access layer (because we're nerds).
Here's the repo: https://github.com/streamdal/streamdal Here's the
site: https://streamdal.com And here's a live demo:
https://demo.streamdal.com (github repo has an explanation of the
demo) -- -- -- THE PROBLEM We built this because the current
observability tooling is not able to provide real-time insight into
the actual data that your software is reading or writing. Meaning
that it takes longer to identify issues and longer to resolve them.
That's time, money, and customer satisfaction at stake. Want to
build something in-house? Prepare to deploy a team, spend months of
development time, and tons of money bringing it to production. Then
be ready to have engineers around to babysit your new monitoring
tool instead of working on your product. -- -- -- THE BASIC FLOW
So, wtf is a "tail -f for your data". What we mean is this: 1. We
give you an SDK for your language, a server, and a UI. 2. You
instrument your code with _`StreamdalSDK.Process(yourData)`_
anytime you read or write data in your app. 3. You deploy your
app/service. 4. Go to the provided UI (or run the CLI app) and be
able to peek into what your app is reading or writing, like with
_`tail -f`_. And that's basically it. There's a bunch more
functionality in the project but we find this to be the most
immediately useful part. Every developer we've shown this to has
said "I wish I had this at my gig at $company" - and we feel
exactly the same. We are devs and this is what we've always wanted,
hundreds of times - a way to just quickly look at the data our
software is producing in real-time, without having to jump through
any hoops. _If you want to learn more about the "why" and the
origin of this project - you can read about it here:
https://streamdal.com/manifesto_ -- -- -- HOW DOES IT WORK? The
SDK establishes a long-running session with the server (using gRPC)
and "listens" for commands that are forwarded to it all the way
from the _UI - > server -> SDK_. The commands are things like: _"
show me the data that you are currently consuming"_, _" apply these
rules to all data that you produce"_, _" inspect the schema for all
data"_, and so on. The SDK interprets the command and either
executes Wasm-based rules against the data it's processing or if
it's a _`tail`_ request - it 'll send the data to the server, which
will forward it to the UI for display. The SDK _IS_ part of the
critical path but it does not have a dependency on the server. If
the server is gone, you won 't be able to use the UI or send
commands to the SDKs, but that's about it - the SDKs will continue
to work and attempt to reconnect to the server behind the scenes.
-- -- -- TECHNICAL BITS The project consists of a lot of
"buzzwordy" tech: we use gRPC, grpc-Web, protobuf, redis, Wasm,
Deno, ReactFlow, and probably a few other things. The server is
written in Go, all of the Wasm is Rust and the UI is Typescript.
There are SDKs for Go, Python, and Node. We chose these languages
for the SDKs because we've been working in them daily for the past
10+ years. _The reasons for the tech choices are explained in
detail here:_ https://docs.streamdal.com/en/resources-support/open-
source/ -- -- -- LAST PART OK, that's it. What do you think? Is
it useful? Can we answer anything? - If you like what you're
seeing, give our repo a star:
https://github.com/streamdal/streamdal - And If you _really_ like
what you 're seeing, come talk to us on our discord:
https://discord.gg/streamdal Talk soon! - Daniel & Ustin
Author : dsies
Score : 97 points
Date : 2023-10-31 15:31 UTC (7 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| dsies wrote:
| Hey peeps - Dan here - ready to answer any questions you've got!
| caledhwa wrote:
| This is fabulous work! Really excited for this release! Can't
| wait to get into all the details and see how I can use it in my
| organization.
| qwidjib0 wrote:
| Grats on the launch!
| od0 wrote:
| Awesome work! Love this
| tzumby wrote:
| Congrats on the lunch Dan and Ustin!
| uzarubin wrote:
| It was tasty : )
| iporollo wrote:
| Love the implementation here, much needed monitoring approach on
| data pipelines
| kensavage wrote:
| Node support - sweet!! Is there any way to interact with the data
| as it comes in real-time?
| jacobheric wrote:
| Hey there! I wrote the node sdk for this, you can use the node-
| sdk to to execute wasm-based rules on your data and interact
| with the data in real time. With the wasm rules you can do
| things like detect and mask PII, etc. The node-sdk is here:
| https://github.com/streamdal/node-sdk. There are some minimal
| examples of these pipelines in the readme and examples/sandbox
| directories.
| openquery wrote:
| Congrats on the launch! Been waiting for this.
| ollybee wrote:
| Is there not a literal "tail -f" type client to send data from
| log files?
| uzarubin wrote:
| We live more on the preventive side where we tap into the real-
| time data flow before it reaches logs or data stores.
| Therefore, enabling you to be preventive vs reactive.
| bradbrewer wrote:
| I'd like to use it but I have strict requirements for data not
| leaving our network
| dsies wrote:
| Good news then :) Everything stays on your network. Actually,
| in most situations - everything stays completely client-side.
| Because the rules that the client executes are Wasm modules,
| all data inspections and transformations occur in the client
| itself.
|
| There is a server component (that you host) - but it is only
| used for pushing rules/Wasm down to the SDKs and for
| facilitating tail - that's it.
| tim-- wrote:
| That console UI looks beautiful!
| uzarubin wrote:
| Thanks Tim! We're working toward having visually appealing UI,
| especially for open source!
___________________________________________________________________
(page generated 2023-10-31 23:00 UTC)