[HN Gopher] Elementary Audio: a modern platform for writing high... ___________________________________________________________________ Elementary Audio: a modern platform for writing high performance audio software Author : davidbarker Score : 92 points Date : 2022-04-11 16:14 UTC (6 hours ago) (HTM) web link (www.elementary.audio) (TXT) w3m dump (www.elementary.audio) | Archit3ch wrote: | Hey Nick! I remember you showcasing this in the audio dev | community. It's a paradigm shift in the way audio software is | written. Congratulations on your launch and I wish you success! | realYitzi wrote: | the shell examples are not working for me. `zsh: command not | found: elementary` after following install instructions. | AdmiralAsshat wrote: | Just a heads-up that I could see both the project name and | specifically the domain causing some confusion with the pre- | existing ElementaryOS project (whose domain is | https://elementary.io). | chaosprint wrote: | Very interesting project. I really like your website design and | the documentation. | | For the audio part, I think it would be very interesting to | compare it with my project Glicol (https://glicol.org) from many | aspects. | | From Web Audio aspect, Glicol recommands users to add only two | lines of code to the project: | | ```html | | <script src="https://cdn.jsdelivr.net/gh/chaosprint/glicol@latest | /js/src/... | | ``` | | Then: | | ```js | | run(out: `sin 440`) | | ``` | | For params updating, just call the `run` again: | | ```js | | run(out: `sin 220`) | | ``` | | And Glicol engine will use LCS to analyse the difference, and | avoid updating the whole graph. | | Since Glicol uses Rust, WASM and SharedArrayBuffer, there is no | GC, and support sample level control. | | I haven't checked the source code of your project, but since I | know CodePen quite well, it looks like that you need to add SAB, | although it requires CORS during deployment. | | Also, I think it may need more consideration to promote it with | "Write once, run anywhere" at this stage. For example, how about | Teensy? For Glicol, although it has POC to show that it can run | as VST or on Bela board (https://github.com/chaosprint/glicol), | there are still a long way to go (for both of us). I need to have | Rust no_std from some of Glicol's dependencies. Yet I don't know | how JS can be used in Teensy/STM32. But this is definitely not | the end of the world. Sometimes, to have a better scope is | actually a good thing for a project. If I would be to promote | Elementary Audio, I would call it a Tone.js alternative. It would | still be a significant contribution, since Tone.js has been so | influential in Web Audio, especially for beginners. How do you | think? | | I think for Tone.js, the time management and scheduling is not | very satisfying, perhaps you can introduce more on this topic for | Elementary? | | All being said, I think it is very exciting to see a new project | in audio community. Good luck and please feel free to discuss | more. | poisonarena wrote: | So with this I could make some fun js audio stuff? I dont really | get it, why should I use this instead of howler js, or are they | even comparable. either way seems pretty interesting and there is | a free/lite version | a-dub wrote: | the webpages require a little digging, but they have an | examples page. | | looks like it's a set of dsp functions where the filter graph | is constructed by composing the functions in functional style | and then passing it to a render function at the end. | | so kinda like big data style composition of functions (like the | scala stuff people often write to run on spark clusters) but | with dsp/audio building blocks running in the browser instead. | poisonarena wrote: | thank you | reggieband wrote: | What a crazy coincidence. I just spent the last couple of days | doing a survey of audio engines with a mind to find those that | could run in wasm. | | Did you write your own engine or are you built on top of csound, | supercollider, ChucK or something else? How do you compare | against the higher level or pattern based implementations like | Sonic Pi, Tidal Cycles, Overtone and Cabbage Audio? How about | libraries like LibPD or even Max/MSP? | | I was even considering this space myself and on my list of todo's | was checking out the licenses of the above to see what the | ramifications of deploying a commercialized version of them would | be. However, I must admit that the market for audio software is | incredibly split. On the one end are the high-end audio effects | with eye-watering prices. On the other OSS end many of the devs | are literally begging for scraps on Patreon barely making enough | to survive. There doesn't seem to be an in-between. I am very | curious what you believe your market will be. | chaosprint wrote: | You should check Glicol: https://github.com/chaosprint/glicol | For the marketing part, frankly speaking, I don't know. Since | most of these projects you mentioned are based on academics, | including myself. | reggieband wrote: | I definitely did checkout Glicol, sorry I didn't call it out | along with the others but I wanted to focus one the "uses | existing old-school C/C++ audio engine compiled to wasm as a | server" question. I understand you wrote your own custom | engine in Rust. I saw your post in the comment section | related to Overtone and was actually going to reply saying to | you that I appreciated your work. I think I watched all of | your Youtube video demos, even the ones on QuaverSeries. | | The other small project that was interesting to me (that | seems to have partially disappeared) was Sporth. The webpage | for that has an unfortunate message saying the project was | taken down to respect csound community. | chaosprint wrote: | Wow, thanks for your interest! I haven't maintained | QuaverSeries for a while as I kinda feel that there are | some issues on Tone.js that cannot be fixed in the | foreseeable future. | | For the topic you are interested (C++ audio engine as | WASM), you should check these two: | https://github.com/frantic0/sema-engine | https://github.com/micknoise/Maximilian | | For those language you mentioned csound, supercollider, | Chuck (should add Faust here), I think these languages | focus more on different music programming philosophy. Of | course, SC is probably the most famous for reusing its | audio engine for other high-level langs like the Tidal, | Sonic Pi you mentioned. Also, SuperCollider's scsynth can | run in browsers now: https://scsynth.org/t/webassembly- | support/3037 | | If you are interested, https://strudel.tidalcycles.org/ is | an ongoing work to port Tidal to browsers. Perhaps at some | point it can be connected with Glicol audio engine. | reggieband wrote: | That supercollider/scsynth wasm thing was actually what I | was thinking about. Unfortunately the implementation is | still in a fork and hasn't been merged upstream. [1],[2] | | Thanks for the other links, sema-engine was new to me and | not in my original investigation. I think I skimmed over | Maximilian but it is probably worth a closer look. | | One of the benefits of csound/supercollider in my mind | are their existing communities and the number of | available open source modules. I've seen entire repos | full of .scd definitions which would give any project | starting with supercollider as its engine a big head | start. The main downside is the esoteric languages these | projects tend to use, which is why I think there are so | many projects that build on top of it. This is one of the | benefits of the client/server model I suppose. | | I also agree that this whole area is full of different | philosophies. At the low end you have straight DSP kind | of stuff which is mostly floating point buffer | manipulation, FFTs/convultions, etc. and at the other you | have sequencers, scale pattern generators and composition | tools. And of course the live performance aspect comes | into play as well as interfacing/syncing with devices/DAW | using OSC/Midi. I'm very excited to see where it all is | going to end up. | | 1. https://github.com/supercollider/supercollider/issues/ | 5224 | | 2. | https://github.com/supercollider/supercollider/pull/5571 | PaulDavisThe1st wrote: | > At the low end you have ... and at the other you have | ... | | Totally missing out on the actual DAWs where most people | do most of this stuff. | | Yes, there is some interest in and use of standalone | tools. But for all kinds of reasons, the majority of new | audio software development occurs with the assumption | that it will run within a DAW (or at least, a plugin | host). | | One of the reasons for that is to offload almost all of | the architectural issues to the host, and leave the | developer to pay attention to the sorts of things you | describe in the paragraph I was quoting from. | reggieband wrote: | For my personal interest (probably no where near the | majority) I am interested in these libraries primarily | for use outside of a DAW. I mean, I already have a wealth | of plugins for most conceivable purposes to use in one of | the several DAWs I have installed. I am also in no hurry | to move the kind of things I do in a DAW to a web based | app. And frankly, none of the examples I've seen so far | come within a country mile of the audio quality of the | higher-end plugins I have. | | I'm interested in these libraries to see how we can | expand on musical creativity outside of DAWs. The first | thing that comes to my mind was the cute HTML 5 game | Friday Night Funkin'[1]. Or games such as Lumines [2] | where the rhythmic music is integral to the gameplay. And | other kinds of interactive media where advanced control | of synthesized audio might enhance the experience. I | think it would be interesting to give people a | simplified/gamified interface to a suite of audio tools | with the intent to unlock creativity, especially aimed at | a very young age (e.g. toddlers and pre-teens). This is a | place I don't think traditional DAWs will ever attempt to | target and where web-first experiences tend to do | extremely well. Being able to quickly develop and deploy | "good-enough" audio effects that have an element of | interactivity for those kind of products/experiences | would be nice to have. | | 1. https://ninja-muffin24.itch.io/funkin | | 2. https://www.youtube.com/watch?v=hv4sCM5X96o | PaulDavisThe1st wrote: | Excellent and interesting response! The idea of | responsive audio within a browser-based game had slid | right by my old-fart brain. Will be interesting to see | how that turns out. | chaosprint wrote: | How about Faust? I haven't studied it in detail but | apparently they are very active porting it to different | platforms including wasm | tomduncalf wrote: | I have been playing with Faust and WebAudio lately. It | works really quite well - e.g. the online Faust IDE [1] | and playground [2] use the WebAudio/WASM implementation | to run in the browser. The Faust team are currently | working on moving to an improved WebAudio architecture | (with support for WebAudio Modules etc.) but the current | stuff should work well enough for most cases. | | It's a cool language too, very functional based. I'm a | total newcomer to DSP but it was pretty straightforward | to figure out how to build a simple synth in not much | code at all after reading a few docs and looking at | examples. There are loads of high quality implementations | of effects etc. in the standard library and examples too | so you can get something sounding pretty good without too | much work. | | [1] https://faustide.grame.fr/ [2] | https://faustplayground.grame.fr/ | aaaaaaaaaaab wrote: | >high performance audio software | | >runs in browser | | doubt.gif | iainctduncan wrote: | Maybe I'm missing something from my quick read, but the idea of | using this for plugins seems like a real misfire. I'm curious | whether the devs come from the pro audio world at all. It's one | area where prioritizing for development time by using high level | garbage collected languages just isn't done because your user is | _always_ concerned with how many instances they can run and what | the lowest latency they can get is, and that latency constraint | is going to be applied across your entire system. So if there is | one plugin that hoses the latency by needing a time slice big | enough to run a GC in the audio dsp routine, pro audio buyers won | 't touch it. | | It looks like a nice way to do interactive web audio work, but | plugins are a different beast all together. If you want a place | where old school skating-with-razor-blades pointer coding is | still common practice, it's DSP code. | replygirl wrote: | my friend is running an independent studio, i'm a bedroom | musician, and we're planning some elementary plugins together. | we tried some stuff with max and it wasn't expressive enough, | we tried the steinberg sdks but they were too much of a lift, | elementary is good enough for our workflows and in that perfect | zone with enough control and enough familiarity that we feel we | can be productive. we'll see in due time if it's too good to be | true, but we're excited! | trypwire wrote: | Hey, Nick from Elementary Audio here. | | You're totally right that in this domain you have to be | extremely careful with performance and latency. Elementary | takes great care to deliver that; it's a JavaScript "frontend" | API but all of the actual handling of audio is done natively | with high quality realtime constraints. The core engine is all | native C/C++, and we run in the web by compiling to wasm and | running inside a web worker. For delivering a desktop app or | audio plugin, the native engine is compiled directly into the | target binary. | | So, while JavaScript and garbage collectors can bring a | performance concern, we're only using JavaScript there for | handling the lightweight virtual representation of the | underlying engine state, and for that role JavaScript is plenty | fast enough and the garbage collector quite helpful! | iainctduncan wrote: | Ah cool, after I posted I was wondering if this was the case. | I was just thinking "maybe the renderer is in WASM"? | | That's pretty cool, it will be interesting to watch. I do | something very vaguely similar in my computer music work | where Scheme orchestrates C level objects. | | Personally, I wouldn't want to use JS as the high level | orchestrator myself as it's such a dog's breakfast of a | language, but I can see that many might. | FractalHQ wrote: | > such a dog's breakfast of a language | | That's why we have supersets like Typescript and Svelte! | iainctduncan wrote: | You should really fix the initial impressions you make from | the site to get that front and centre though. Experienced | audio devs are going to dismiss this unless you make it clear | very quickly. | trypwire wrote: | 100%. I'm actively learning exactly this, haha :). It's | helpful feedback, thank you! | danuker wrote: | > by compiling to wasm | | TIL WASM bypasses JS completely, including its memory | management. | bricklebrack wrote: | This is my new favorite comment for illustrating the perilous | future of general computing and how it needs to be taken away | from JavaScript if we have any chance of survival. Electron, | Webassembly fetishism, the pursuit of language synchrony at | the expense of common sense, it all gets you this. This | comment. Right here. This is the future of software and it | should scare the shit out of you. | | Let me get this straight: you realized latency was a concern, | so you wrote in C/C++ (which, exactly?), then turned it into | wasm so you can run it in a Web worker? What the hell was the | point of that? That's like saying you bought an M6 and | converted it to a foot-powered bicycle. What exactly do you | think wasm does? You seem to be implying that you think the | native engineering you invested in continues to matter, in | the same way, after you do that. You also imply heavily that | you understand the wasm you're executing to still be native. | Do you think that? Do you understand what you're giving up in | the Web worker? As in, directly tied to latency and real-time | requirements, your _whole reason_ to go native in the first | place? | | Whatever your response is, deep down, you and I both know | it'll be justification to do Web things for Web's sake. I | know this because everyone I've had this discussion with has | played the same notes (imagine the deployment model!) while | failing to understand that they're justifying their | preference. The only people who build Web stuff want to build | Web stuff. In the high performance sphere, of which DSP is a | very mature practice, this looseness with the fundamentals of | software is going to put you out of the game before you're | even started. | hamasho wrote: | I'm a web person defending web things but providing | something on the web has a significant advantage. I know | you've heard this again and again, but I can't emphasize it | enough. | | You can use the site from most platforms, including PCs and | mobiles. You don't have to install software, a single click | is enough. Of course, browsers have considerable | limitations, and serious users will eventually choose other | tools, but providing such an accessible software is a | really huge advantage for me. | | edit: words | [deleted] | moron4hire wrote: | Do _you_ understand how WASM and Web workers work? Do you | understand that low-enough-latency audio doesn 't take a | super computer anymore? Yeah, if you were working on DSP | stuff in the 1990s, you were a hot shit programmer. | Nowadays, it doesn't really say much at all. And it | certainly doesn't justify talking about it as if it were a | moral failure to not treat DSP with respect. | PaulDavisThe1st wrote: | > Do you understand that low-enough-latency audio doesn't | take a super computer anymore | | It never did. Low latency audio has almost nothing to do | with CPU power. Here's a summary of some of the issues | faced on modern general purpose computers: | | https://manual.ardour.org/setting-up-your-system/the- | right-c... | | I know how WASM and Web workers work. Since nothing you | can do in WASM or a web worker has anything to do with | either (a) realtime scheduling priority (b) actual audio | hardware i/o, they don't have much to do with solving the | hard parts of this. Browsers in general do not solve it: | they rely on relatively large amounts of buffering | between them and the platform audio API. Actual music | creation/pro-audio software requires (sometimes, not | always) much lower latencies than you can get routing | audio out of a browser. | iainctduncan wrote: | And even when it doesn't _require_ it, we always _want | it_. :-) | jancsika wrote: | And even when we set it, we don't get it, because we | blithely read a "latency" label in a GUI instead of | measuring the round-trip latency on the specific device | in question. | PaulDavisThe1st wrote: | That wouldn't be correct either, at least half the time. | Problem is that "latency" is used with different | meanings, at least two: | | 1. time from an acoustic pressure wave reaching a | transducer (microphone), being converted to a digital | representation, being processed by a computer, being | converted back to an analog representation and finally | causing a new acoustic pressure wave care of another | transducer (speaker). | | 2. time between when someone uses some kind of physical | control (mouse, MIDI keyboard, touch surface, many | others) to indicate that they would like something to | happen (a new note, a change in a parameter) and an | acoustic pressure wave emerging somewhere that reflects | that change. | | The first one is "roundtrip" latency; the second one is | playback latency. | Flankk wrote: | That's great on paper but how do I actually use it? If I want | to play back a wavetable you have el.table() but it's missing | the oversampling and decimation tools required for | antialiasing. The el.lowpass() is a biquad which is not | suited well for modulation. How can this compete with JUCE | when the documentation and features are so sparse? | ushakov wrote: | exactly | | users often have to bear the costs for developer productivity | | with audio software the performance aspect is critical to user | and thus the cost of increased developer productivity is | exceeding the reward | | you can't make any compromise, because otherwise users won't be | buying the product and you go bankrupt | | this is one thing i wish has happened to Electron | PaulDavisThe1st wrote: | It's hard to know what to say. | | > The conventional approach to writing audio software is an | challenging process requiring expertise in a wide breadth of | topics- C++, multi-threading and thread safety, lock-free | programming, and realtime thread safety come to quickly to mind, | and we haven't even mentioned digital signal processing itself. | Moreover, the tooling, workflow, and developer experience that we | have in this approach pales in comparison to that of the web | software industry, for example. All together, the reality of | these complications is that the conventional approach for writing | audio software is difficult and time consuming. We believe it | doesn't have to be that way. | | This is then followed by a screenshot of some code that looks | like a SuperCollider script from 15 years ago. | | There are at least a dozen audio "scripting" languages that free | someone from all the issues mentioned above. Many have been | mentioned in the thread already. The only things that Elementary | Audio appears to have in its favor are that it leverages JS and | it runs in the browser (sort of). | | For some people, those two alone will be interesting. Have at it, | people! | | But for people who write plugins designed to run inside DAWs, or | modules inside Rack, and in particular for the people who write | DAWs or Rack (or Rack-like things), this is of approximately zero | interest. It's a web-centric version of SuperCollider, which is | cool but ultimately so bounded by the limitations of the platform | that for many things, SuperCollider or something similar remains | more powerful and more flexible. | Archit3ch wrote: | > SuperCollider or something similar remains more powerful and | more flexible | | SuperCollider's license effectively stops you from designing | commercial products. Even the free version of Elementary allows | that. | PaulDavisThe1st wrote: | Well, if you plan to release a commercial product based on | writing JS, compiling that to WASM, running somehow inside or | alongside a C/C++ core engine also compiled to WASM, both | running in some standalone WASM environment ... good luck! | | Elementary looks like a nice playground for audio if you have | a web-leaning bent. Just like SuperCollider is a nice | playground for audio if you are familiar with some its | fundamental assumptions. | | Neither of them seem appropriate to be the basis of products | I would charge someone for, other than maybe as a one-off | custom tool of some sort. | abetusk wrote: | What are you talking about? SuperCollider is licensed under | GPLv3 [1]. | | [0] https://github.com/supercollider/supercollider#license | iainctduncan wrote: | That was my impression too, it looks very much like an SC style | computer music language. Which begs the question, who is this | for? Someone who writes plugins doesn't need or want the high | level SC style language. Someone seriously into computer | music/audio isn't going to want the limitations, and has a lot | of options already to choose from that are mature, if | relatively unknown outside music circles. | | If it were my company, I would be marketing this squarely at | people with the need for high level procedural audio in a web | context. This looks like it could be great for browser-based | game devs, for example. I do know that Pure Data and Andy | Farnell's great sound design book for it were a hit in the game | dev world of procedural audio, FWTW. | aaaaaaaaaaab wrote: | It's probably for people who are smart enough to know DSP, | but only know JavaScript and refuse to learn anything else. | If you squint it looks like an empty set. | bfrog wrote: | Another day, another pile of JavaScript eating into an area I | care about with a language I despise. | moron4hire wrote: | One of the biggest problem I have with WebAudio API is managing | nodes. The API is extremely verbose, when you consider that | anything interesting will need to be composed of dozens of basic | nodes combined in different ways. | | You're given a very basic set of primitive nodes, from which you | can construct pretty much anything else. But those things you | construct will never be "nodes" themselves. You can't create | synthetic nodes or subclass node types. But then many of the | basic nodes are useful on their own. So you end up in a bind of | having two different provenance branches of audio "node-like" | things and need to know which is which before you can connect any | one of Column A to any other of Column B, or vice versa. | | You're also on your own in terms of keeping track of those | connections. Once a node is connected to another node, there is | no in-API way of telling what nodes a node is connected to, or | what nodes connect to a given node. There's a whole, huge, | potentially cyclical graph structure sitting around in memory | that you can manipulate but never inspect. | | With how hard everything is to keep track of what is where, you | can easily end up in a situation where old, now-disconnected or | unused nodes stay resident in memory and never get garbage | collected. Some developers then punt on tracking connections at | all and expect the user to reload the page if there is a major | change in audio setup ( _coughcoughGoogleOmnitonecoughcough_ ). | | So when people talk about how Web-based gaming really hasn't | taken off or replaced the haunting specter of Adobe Flash, and | then point at WebGL as the problemm, I kind of laugh. WebGL is | the least of your worries. | | Anyway. | | Elementary Audio seems to end-run around all that by A) only | letting you declaritively define your audio setup, and B) running | everything in a single Audio Worklet, thereby bypassing the | majority of the Web Audio API. | trypwire wrote: | I couldn't agree with this more. Even beyond the web audio API, | we have similar graph APIs in the low level/native C++ world | that become unmanageable in their complexity simply due to an | inability to compose and manage transitions with ease. | | This is exactly one of the difficulties that Elementary aims to | address, whether you're running in the browser (where indeed, | Elementary uses almost none of the actual web audio api), or | natively. | [deleted] | _bohm wrote: | Heads up (if the OP is related to the project) the docs page for | audio plugins 404s | trypwire wrote: | Thanks! The docs just underwent a significant change for v1.0; | the new plugins page is here: | https://www.elementary.audio/docs/packages/plugin-renderer | ushakov wrote: | so this is basically a paid Web Audio API wrapper? | TheOtherHobbes wrote: | It does say you can make plugins, but the link doesn't work. | | It would be huge if you could make cross-platform AU/VST*/AAX | without the costs of JUCE, but I suspect that isn't on offer | here. | mxmilkiib wrote: | Support for LV2 is landing in JUCE 7 apparently. | jcelerier wrote: | Slowly getting there with https://github.com/celtera/avendish | :) | fxtentacle wrote: | Last time I checked, a perpetual JUCE license was priced | similar to 3 days from a C++ freelancer. I've never heard it | being called out on price before. | TheOtherHobbes wrote: | It's changed since I last looked. J V4 Used to be $999 for | solo devs (2016), now J V6 is $800 with up to $500k annual | revenue. | | And there's a free tier that allows up to $50k annual | revenue. | | That is indeed pretty affordable. | | J V3 used to be more than that. I remember looking at it | and writing it off for cost reasons, but that was quite | some time ago. | moron4hire wrote: | Their intro docs page says Windows _and_ macOS, but actually | going to the documentation on plugins says "macOS only" | https://www.elementary.audio/docs/packages/plugin-renderer | ushakov wrote: | still not convinced | | you can make cross-platform plugins without JUCE by using | iPlug, Faust or even Rust | | the selling point of this seems to allow js/web developers to | build plugins | | and we already know what happened when web developers were | allowed to build desktop apps (Electron) | | luckily given real-time and cpu constraints this might never | happen to Audio Plug-ins | TheOtherHobbes wrote: | You can make plugins in Faust, but it's not a super smooth | process. iPlug is the nicest option. | | I agree about the js though. Even if it's a reinvention of | web audio with a C++ backend, js and DSP just don't feel | like a good fit. | | And I would be surprised if the UI side wasn't limited in | the way that full native isn't. | jerrysievert wrote: | have you looked at iplug2? | | https://iplug2.github.io | | since I was already making my own knobs and panels for | vcvrack, it was an easy fit for building out vst3/au plugins. | trypwire wrote: | That is indeed on offer :) | | The new docs structure broke the website links, I'll get that | fixed asap. See here in the mean time: | https://www.elementary.audio/docs/packages/plugin-renderer | | Also, as another user here has already pointed out, the | "plugin development kit" is still macos only, though I | anticipate releasing the windows version within hopefully a | week or two. | iainctduncan wrote: | I find it hard to imagine people the desktop/commercial audio | world wanting to write plugins in JS. At a glance, this seems | like an interesting webaudio project that is trying to be too | much. Plugins are used in a very different situation where | low latency and high performance trump pretty much | everything. Reconciling that with web dev priorities seems | optimistic. | sharklazer wrote: | Yeah. I got lost at JS. Buffers and buffer manipulation are | everything in audio processing, I don't see anything | stealing C/C++'s thunder here except Rust. | Severian wrote: | Is this for Web audio only? | trypwire wrote: | Nope! You can run the same Elementary code in the web browser, | at the command line, offline (for file->file processing), or in | a native audio plugin ___________________________________________________________________ (page generated 2022-04-11 23:00 UTC)