[HN Gopher] Why We're Building Observable ___________________________________________________________________ Why We're Building Observable Author : petulla Score : 196 points Date : 2020-11-20 15:34 UTC (7 hours ago) (HTM) web link (observablehq.com) (TXT) w3m dump (observablehq.com) | silentsea90 wrote: | Extremely dumb question: how is this different from tableau | and/or jupyter notebooks? I don't use visualizations and don't | understand this space :) | tracyhenry wrote: | Tableau offers templated visualizations. You generally do not | need to program to create stuff. Observable is a notebook | environment where you can write JS (mostly) to create and tweak | _highly customized_ visualizations. It 's similar to Jupyter on | a very high level (they all have reactive cells for example), | but for JS and visualizations. | silentsea90 wrote: | That makes sense! Thank you! | | Some more questions: 1) Assuming the bulk of folks use | jupyter notebooks for analysis, why not integrate with | jupyter or use d3.js in jupyter directly? I assume there | would be a learning curve for data manipulation on js where | analysts/scientists would prefer python 2) Assuming that this | isn't even meant for the jupyter audience, who is it meant | for within a company? | tracyhenry wrote: | I mean this is my question as well: how are they making | money? As of right now Observable seems like a mix of use | cases like Medium-like explanation articles, vis | debugging/authoring and interactive journalism that don't | seem to generate much money (ofc I can be wrong). | | I totally agree with you that the DS community is much much | larger than the vis community. With the new funding round I | think they should start doing something with DS. | carls wrote: | I'm a big fan of Observable, having used it to prototype and | learn a number of different visualizations. Note that you don't | just have to use D3, but can use other visualization libraries as | well (i.e. Vega-Lite, Highcharts). | | I also want to shout out Mike Bostock, one of the company | founders (and creator of D3). I emailed him randomly to ask for | some help with a d3 package and he replied the next morning. | | Busy creators who nevertheless still make themselves available to | engaging with the community always impress me! | jeffbee wrote: | Bostock jumps into random Twitter threads to answer questions | from Observable users. He's really helpful and generous. | breck wrote: | He responded to me too, almost a decade ago when I was first | getting into the field and looking for mentors. He had some | great tips too (basically which was to __not __do what I was | doing privately emailing devs asking for mentorship and instead | get involved with the public open communities on via GitHub, | Stackoverflow, sharing demos, etc). Obvious advice in | hindsight, but took me a while to internalize that. | [deleted] | anentropic wrote: | Skimming the docs... the tutorials are really nicely done but | it's hard to find a clear description of what exactly is | possible. | | Can you install libraries (from npm?) | | Can you build and share custom components, not just notebooks? | | Can you connect to a web server? | tracyhenry wrote: | You can install npm libraries. You can connect to DBMS servers | (not sure about arbitrary web servers). I believe they offer | iframes to facilitate embedding but not frontend components. | | Interactive vis is not very well integrated with the frontend | ecosystem IMHO. Integrating a very interactive D3 vis with | React, for example, can be painful. | SamBam wrote: | I'm a little confused about what the announcement is. Are they | moving out of beta? | carls wrote: | It's a bit buried, but if you click through the link about how | today is a big day for Observable, you'll notice it links to an | article announcing that they raised their Series A today: | $10.5M from Sequoia and Acrew. | ParadiseEclipse wrote: | I wonder why it's buried... the comments on this thread don't | seem to be on a single topic, so it looks like most people | have struggled to find the point too | tlarkworthy wrote: | I am a huge fan of Observable and I think it has massive | potential beyond visualizations. To me it's the holy grail of a | development environment, there is no separation between | development and output. There is no toolchain separating source | and binary, infact, the binary is _partially_ recomputed on | changes. | | On top, it is forkable for others to build upon. It support | comments, its reactive. When I explain it to people I say its | like a combination excel + vscode + google docs + github. There | is nothing like it! | | I been trying to see how far I can push it for non- visualization | software. Turns out... pretty far.... | | Firebase + Stripe => | https://observablehq.com/embed/@tomlarkworthy/saas-tutorial | | On Demand Minecraft Servers => | https://observablehq.com/@tomlarkworthy/minecraft-servers | | Zero deploy serverless cells => | https://observablehq.com/@tomlarkworthy/serverside-cells | | With serverless cells I can serve HTTP endpoints from a notebook! | E.g. custom HTML like: | | https://serversidecells-ibyw6dtm4q-ey.a.run.app/notebooks/@t... | is generated from a notebook! | | I am so productive in Observable because there is no context | switch between development, using the product and debugging, it's | all just one thing. Amazing, utterly amazing. I hope the | Observable team can accommodate my use cases, as its clearly not | what they had in mind. | xipho wrote: | Observable is fantastic, but as others have hinted at, IMO, I | think it's trying to lock you in. I want the raw JS, in one | click, so that I can build out a visualization locally, just | hacking on HTML/JS. Steps to doing this are highly obfuscated | when they could be a single button. | | Overall you can see its evolution from prior sites- someone spent | a huge amount of time demonstrating cool visualizations, then | tried to make it all "proprietary". Nothing wrong with that, but | why not make people addicted to the service by giving them the | raw drug, rather than making them hate you for dancing through | hoops? | | Again- huge fan of access to many cool visualizations that have | greatly improved what I can express visually for presentations | etc., but very unhappy that I can't do this so much easier for my | particular workflow. I'd love to be shown otherwise, but I've | also spent a lot of time looking for answers, reading forum | responses, etc., and they seem to all reflect wanting you to go | through them, not along for the ride with them. | mistidoi wrote: | I kind of thought that too, but somewhere around the start of | the pandemic, I was able to quickly develop an interactive | dataviz on Observable | (https://observablehq.com/@mistidoi/coronavirus-tests- | positiv...) and drop it into a blog post trivially | (https://relevant.healthcare/blog/2020-04-02-covid-state- | visu...). Beyond the work on the viz itself, it took something | like 15 minutes to get it into the blog post, and it's more or | less seamless. | | Confession though, I might have been powered by Mike Bostock | fanboy juice. d3 is part of what made me fall in love with | programming. | carls wrote: | Can you share how you were able to embed the notebook into | your blog post? | protoduction wrote: | I agree that Observable is fantastic, but also wish it was more | open. | | I'm building something similar called Starboard Notebook[0] | that has a different set of trade-offs. It ends up being | something in between Jupyter and Observable: | | * One of the goals is to build Jupyter how it would have been | if it was designed for the web (only). | | * It's open source [1], plays nice with git (the format is | plaintext), and supports local viewing & editing [2] | | * Because of that you can host it yourself, put it on your blog | / github pages, anywhere. | | * There is little magic, it actually is just Javascript at it's | base. This means you can use standard browser APIs and HTML, | and when you are ready to "graduate" the notebook | implementation that should be straightforward. In my eyes | notebooks are only for the first 20% of the work that does 80% | of the job for small applications/experimentation (which is | often where it ends anyway). | | * You can "build the ship as you sail": you can load new cell | types dynamically at runtime. This is also how Python is | supported (through WebAssembly). | | * You can have interop with Python and Javascript which is | really powerful. Example: Create a drag an drop form using | HTML+JS, then process the dropped CSV file using Pandas and | visualize using matplotlib. | | [0]: https://starboard.gg [1]: | https://github.com/gzuidhof/starboard-notebook [2]: | https://github.com/gzuidhof/starboard-cli | Jugurtha wrote: | Playing with this a bit. It appears I can't use `pip` with | `pyodide` and have to use `micropip` with: | micropip.install('torch') | | Which fails with a message: Couldn't find a | pure Python 3 wheel for 'torch' | | Would you agree that this is a substitute for "pencil and | paper" to express ideas around data? | RobinL wrote: | I don't think that's entirely fair. | | For instance, I host content on my personal website [0] that is | written in using Observable, and in fact the site provides | guides to allow anyone to export content[1] and embed elsewhere | online. | | My content can be entirely under my control if I save my | libraries out of the Observable API as a tarball[2]. | | I genuinely think the differences between Observable and 'pure' | javascript are driven by the vision for a more powerful way of | using javascript for rapid prototyping, rather than any attempt | to make it proprietary. I think these ideas are revolutionary | and Observable is one of my favourite working environments as a | data scientist. | | I also think that Mike Bostock has shown good faith, and | dedication to the open source community through decades of | work, which he's always given away for free. | | With all that said, if ObservableHQ and its API were to | disappear overnight, the lack of an editing environment would | make it difficult to update existing content and so I think | there is space for a offline version of the editor. As far as I | understand it, nothing's stopping a third party building this. | | [0]https://www.robinlinacre.com/energy-usage/ | [1]https://observablehq.com/@observablehq/downloading-and- | embed... | [2]https://github.com/RobinL/robinl.github.io/tree/dev/tgz | xipho wrote: | Thanks for the links, I completely concur with the goodness | of everything you've raised. | | To clarify where I'm coming from, my use-case is (fully) | offline. I want to mock the visualization in the service then | get the raw, static JS offline to be used as a template. I'll | write a Ruby wrapper around the data I'm exploring and | gradually parameterize the template as I better understand | it, and what my data looks like in it. I don't want to dance | with details of the JS, I want to dance with details of the | data and parameters, so get me out of requiring online hooks | ASAP. I _know_ this is possible because prior versions of | Observable let you do exactly what I want, but now I get some | super friendly /not friendly interfaces that obfuscates what | I'm familiar with. | | As a separate issue not fully related- Ultimately my goal is | a static visualization for scientific publications or | presentations (I can export SVG from Observable, so my prior | arguments doesn't hold here). Science has a very long shelf- | life, in my field we routinely reference papers and their | figures 200+ years old. While I absolutely love the idea of | "living publications" I also fear life that depends on | other's services. If the big people can't place nice | (coughing in your direction Google services gone dark, I can | think of various examples that have screwed over scientists), | then you can see my caution in adopting Observable's all-in- | dynamic approach. | golemiprague wrote: | I might be deficient in some way but I just feel a bit dumbstruck | with a lot of the visualisations and I can't make sense of them. | Would rather just read a table with simple numbers. Except from | very simple graphs everything else looks to me like a cognitive | overload which I can't decipher. | kowlo wrote: | You can't self-host with Observable. For many, that means it's | not a feasible replacement or improvement over an existing | literate programming environment. | | It is a bit silly to downvote this for not being true, when it is | factual. | j-pb wrote: | Not true. | | The Observable runtime is open source, and trivial to integrate | into your existing react/whatever webapp. | | There is also a vscode plugin that allows for live editing and | rendering. It doesn't give you the ObservableHQ interface, but | tbh I like working with files more. It also allows me to git | everything. | | https://marketplace.visualstudio.com/items?itemName=GordonSm... | ParadiseEclipse wrote: | It sounds like you confirmed that it can't be self-hosted - | are you sure about it not being true? | j-pb wrote: | Hosting the notebook and hosting the development | environment are two different things. | | It's like saying git (observable) can't be self hosted | because github (observablehq) is not open source. | kowlo wrote: | Just to jump in and say my comment that you said wasn't | true was "... improvement over an existing literate | programming environment.". | j-pb wrote: | I don't argue that you might find a different notebook | environment more suitable. That's totally up to taste, | preferences, and silly do argue about. | | But observable (the reactive notebook technology) can do | everything a jupyter notebook can do, you just need to | write a bit of communication code to hook up a websocket. | (For me sandboxing the ui is a feature not a bug.) | | And you can also self host it on a machine of your | choice, and edit it locally. | | We do that internally for all our dashboards. | kowlo wrote: | It is true. | | You can't self-host the same way you can self-host the | Jupyter Notebook or Jupyter Lab environment. | | Or has that changed? Can I work on something offline without | communicating with their servers? | j-pb wrote: | Check the link provided. It's just running inside of | vscode, with no communication to their servers whatsoever. | kowlo wrote: | I did, it's not the same literate programming environment | like I said. Are we misunderstanding each other? | j-pb wrote: | "You can't self-host with Observable." | | You can self host observble, but you can't self host | observablehq. | | Literate programming just means the combination of code | and documentation/visualisation. | | Observablehq provides a nice inline code editor, but | that's incidental to what observable is. | | Git can also be used with a GUI but that GUI is not | essential to what git is. | | Simularly observable notebooks can just as well be | created without the GUI, in a quite convenient way. | kowlo wrote: | I can see the ambiguity in what you've quoted, but it's | clarified in the sentence that immediately follows it. It | was referring to the "literate programming environment". | mrkwse wrote: | I really like Observable a lot, and have made a couple of brief, | messy visualisations with it (messy in terms of code at least). | | I hope at some point they can offer some sort of offline support. | The focus on ease of sharing is great, and the volume and breadth | of content has really helped me to get more proficient with d3. | The inability to open and edit notebooks offline/locally is a | real inconvenience, however. | | I have a small personal site on Netlify (with Next.js) and going | forward I'm looking at hosting the sort of stuff I have on | Observable there, simply as it's easier to work to when offline. | The quality of the community and resources is a really valuable | thing, but the need to be online is a real drawback at times. | kowlo wrote: | Indeed having the environment offline would be great. But this | doesn't seem to fit their model for profit. | ontouchstart wrote: | You can host __production __notebooks offline /locally or on | your own SSG/CDN. | | Here is a project that we built last year (by its nature, it | had a limited shelf life so it made no sense to have long term | cloud support.) | | https://spike150.mocanyc.org/ | | If you inspect the source in the browser, you will see that the | notebooks are hosted on the same server. | | P.S. If you want online interactive notebook on your own site, | I guess that is their business model either as a viral sharing | of free notebooks or paid private team sharing. (I was on a | team plan during the development of the project. But once the | project is done, there is no need for us to continue on team | plan and be interactive. I am glad that they offer support of | offline production notebook.) | | P.P.S. If Observable Team is listening, here is a business idea | for them: observable playground as a service. It would be less | a technical challenge than a business development challenge. | rgovostes wrote: | Coincidentally, I tinkered with my first Observable notebook last | night (https://observablehq.com/@rzg/prosthaphaeresis -- don't | judge too harshly). | | One of the improvements on the computational notebook concept is | that it seems to compute a data flow graph so it knows what cells | to update when the data changes, without you needing to re-run | cells in a particular sequence. The cells can also be out of | order. | | The editor interface is pretty slick, and I wish it were open | source so I could use it in other projects. | | (Mike Bostock, one of the founders, is also the creator of D3.js | and the information visualization gallery bl.ocks.org. Here's my | favorite: https://bl.ocks.org/mbostock/31bd072e50eaf550d79e) | breck wrote: | Wow, never heard of Prosthaphaeresis before. So cool! Really | enjoyed playing with your notebook. So concise and clear. | thanks! | mrkwse wrote: | The data flow is really well implemented with Observable. The | fact you don't need to go through and repeat the execution of a | number of downstream cells when you make a change is really | useful. | | Another great advantage they have is the data map on the right- | hand margin. It's a good way of seeing how complex a notebook | is getting as well as being a really efficient way of | navigating through the data flow. | billconan wrote: | An issue of visualization on the web I don't know how to solve is | how to load large data. | RobinL wrote: | Very much depends on how large you're talking, but check out | Apache Arrow: | https://observablehq.com/@theneuralbit/introduction-to-apach... | https://observablehq.com/@uwdata/arquero-and-apache-arrow | catacombs wrote: | That depends on the job. Is the data constantly updating? | tracyhenry wrote: | We are building a data vis tool | (https://github.com/tracyhenry/kyrix) to solve this exact | problem many web vis tools have. | | The idea is not surprising: using database indexes and a | backend to send the frontend necessary data to render _on | demand_. DB and the backend are containerized so the | installation, data loading and authoring are all one command. | | For creators, we offer D3 and JSON-based declarative primitives | that enable creation of pan/zoom like visualizations very | easily (e.g. 10s of lines of JSON for a 15-zoom-level vis). You | can check out an interactive demo here, which visualizes 1.88 | million wildfires: http://wildfire.kyrixdemo.live/ | | Since big data vis is very use case dependent, I'd like to | reference two other tools that might be useful: | | Falcon: https://github.com/uwdata/falcon NanoCubes: | https://nanocubes.net/ | | Disclaimer: I'm still in academia so I'm apparently biased | towards academic works. But fortunately they are all open | source. | breck wrote: | I saw you mention Falcon and NanoCubes so had to check our | kyrix. I think this is a really important space. I used to | assume Palantir had solved this but now that they're public | it seems to be more of a smoke and mirrors startup. | | 3 questions: | | 1. Why isn't there a demo I can just click and visit? | | 2. Why not TypeScript? (I think this would be a worthy short | term and long term investment --for this project and beyond-- | to make the upgrade) | | 3. Why Docker? (This one I'm probably wrong about and am just | a curmudgeonly old gray beard). Every time I see Docker I say | "nope". I get it's use for massive clusters, but for running | on one machine I dislike it (at that point I'd prefer just to | run an image on a cloud VM). | tracyhenry wrote: | Thanks for the feedback! | | 1. In the original comment there is a link to a demo | created by Kyrix: http://wildfire.kyrixdemo.live/ Are you | looking for a different type of demo? | | 2. Honest answer: we are very understaffed academics who | also need to write papers and theses. We want to convert to | TS, that is just one item on the wish list. | | 3. We want everyone to be able to spin up a Kyrix app _on | their laptop_ using 3 CLI commands (you can try, | instructions are in the README). With a backend and a | database comes the cost of complex installation. Docker | helps make everyone 's dev environment consistent so it's | easier to troubleshoot. | breck wrote: | 1. Ah! Sorry I missed that and went to the GitHub. Might | be helpful to also add that link to the "website" field | of the GitHub? | | 2. I get it. I never had the "pleasure" of being an | academic, but got the chance to be a software engineer | working alongside grad students for a couple of years and | it felt like I imagine what taking a tour of a sweatshop | factory floor would be like. (a huge exaggeration, but I | saw a lot of low pay and lots of time on pdfs and not | enough on code) | | 3. Yeah, I get it, but maybe allowing everyone to run it | on their own VM would be better (and providing a one | click "click here to get your own droplet on digital | ocean running kyrix" sort of thing). I've found that | Docker suffers from the XKCD problem (10 different | environments to support--what if there was just 1....11 | different environments to support!). I've almost never | had a pleasant experience using Docker (and anything that | requires downloading a 1GB+ image I don't consider | pleasant). But again, I'm probably just being a grump and | the other grads I worked with all seems to like it. | Another idea is could use do this with SQLite? Have a | more slimmed down version that didn't require Docker + | those dependencies | | Thanks for the answers! Very cool stuff and it's a very | interesting problem. | tracyhenry wrote: | Again, thanks for the feedback. Your comments are great | food for thought! | count wrote: | Generally Docker lets you say 'here, this works', rather | than 'follow these 800 steps to get your environment in the | right shape'. | domoritz wrote: | Vega-Lite and Falcon author here. Many people ask for | scalable visualizations and I think declarative visualization | approaches are really promising since systems can | transparently optimize data movement and rendering. I have | done some more experiments to scale Vega such as | https://github.com/vega/scalable-vega but I think what we | need is a system that automatically offloads heavy | computations to a backend such as | https://github.com/omnisci/jupyterlab-omnisci. | chatmasta wrote: | Observable supports direct DB connections [0], but these only | work in private notebooks at the moment. | | At Splitgraph [1] we're building a "data delivery network" | (DDN, like a CDN but for databases) that looks like a big | Postgres database. It works really well with the Observable | Postgres client in private notebooks -- you configure it like | you would for any Postgres connection. For public notebooks, | you can use our HTTP API for sending SQL queries directly to | the DDN. Here's an example [2] using that SQL-over-HTTP API to | plot some Covid data. | | [0] https://observablehq.com/@observablehq/connecting-to- | databas... | | [1] https://www.splitgraph.com | | [2] https://observablehq.com/@mildbyte/splitgraph-ddn- | oxcovid19-... | breck wrote: | I've been watching splitgraph for a while. __Very | __interesting what you are doing! | chatmasta wrote: | Thanks! That means a lot :) Much more to come. | breck wrote: | I've found 10MB (gzipped) to be a good rule of them. Anything | bigger, query a backend. Storing bigger things in ephemeral tab | memory can be painful if you refresh browser a lot. | brandnamehq wrote: | Great domain. We like hq domains too :) | petulla wrote: | My gripe with Observable is I can't fit into a Python-based | workflow. I.e., a machine learning or gpu-training worfklow. It | is a nice place to create explainers, though. I almost think of | it as being like a Medium alternative. | catacombs wrote: | > My gripe with Observable is I can't fit into a Python-based | workflow. | | Observable is for JavaScript ecosystem. I wouldn't count on | them implementing anything for Python. | hobofan wrote: | With the growth expectations that come with the funding | amount they just collected, I'd count on it. There is just so | much more demand for data visualization and notebook-style | projects in the Python community compared to the JS | community. | Jugurtha wrote: | Could you say more about your workflow? Our platform[0] may | suit you: | | - Collaborative no-setup notebooks: different images with the | most popular libraries pre-installed. See each other's cursors, | very useful for troubleshooting, pair programming. | | - Long-running notebooks: GPU ready. You can schedule notebooks | right from the notebook view so you don't context switch. You | can close your browser, disconnect from the internet, or shut | your laptop and still see the notebook's execution. | | - Automatic params/metrics/model tracking: no boilerplate. | | - One click deployment: clicking a button deploys a model and | gives you a nice "REST" API endpoint you can invoke. | | If you're doing anything related to ML and use notebooks, you | probably had the above problems. We're building around Jupyter | notebooks because in our experience, our machine learning | projects were not held back for lack of slick stylesheets. | | - [0]: https://iko.ai | pea wrote: | For Python, Datapane may solve your use-case (I am one of the | founders). It's an open source framework for creating HTML | reports from text, dataframes, plots | (altair/bokeh/plotly/folium), and files. We provide a free | public platform for sharing reports on the web, and you can | check out some of the things our users are building here: | https://datapane.com/gallery/ | | It's different to something like Observable in that you import | it into your existing Python environment (Colab, Jupyter, | Airflow, etc.), build the reports from your assets, and push | them up -- as opposed to having to do the analysis itself in | our interface. | domoritz wrote: | Take a look at https://www.streamlit.io/. | ilaksh wrote: | Can you run arbitrary libraries like Tensorflow.js? | TeMPOraL wrote: | You can run just about anything that can be run in the | browser (all code in an Observable notebook executes in your | browser). Though depending on the flavour of packaging/module | system a given package uses, you may need to figure out a | correct magic incantation to import it. | fractionalhare wrote: | This is also the main thing holding me back from using it. The | Observable experience feels more polished than Jupyter for me. | petulla wrote: | My hope is eventually to be able to run it locally and | integrate it into other language, like Tensorboard. | darksaints wrote: | I really like Observable and have used it quite a bit, but I | absolutely hate that it is the default documentation form for D3, | because D3 is a javascript library, and Observable is not | javascript. At the very least it would be nice if there was a way | to take an example and press a button to get the same thing in | javascript. | carls wrote: | A lot of the code you can write does look like JS, though. You | can use the libraries that may be common to a JS dev. It's not | perfect, but there's enough overlap and the differences are | understandable and documented[1]. | | Also having learned a lot of D3 by reading through Observable | tutorials, I'm curious what you've seen that doesn't seem to | work when you port it to JS? | | One thing I'm aware of is that you have to move a lot of the | plotting code into the .then() method of promise when you're | loading data. | | ------ | | [1] Observable's not JavaScript: | https://observablehq.com/@observablehq/observables-not-javas... | macNchz wrote: | Yeah I pulled out d3 for a project recently, having used it | before-quite a bit at one point-but not for several years. I | just wanted to browse some basic examples for the current | version, to refresh my memory and familiarize myself with the | current API, but the reliance on Observable drove me up the | wall. Even a lot of the old examples from bl.ocks.org just | redirect to Observable now. | | The presentation is one more layer to understand between | "here's the code" and "here's what the code produces". I do | generally like the notebook format for exploratory data work, | but it's not at all ideal as part of a workflow for writing | code that will fit into a larger application, in my experience. ___________________________________________________________________ (page generated 2020-11-20 23:00 UTC)