[HN Gopher] Transcending Posix: The End of an Era?
___________________________________________________________________
Transcending Posix: The End of an Era?
Author : jsnell
Score : 126 points
Date : 2022-09-10 09:44 UTC (13 hours ago)
(HTM) web link (www.usenix.org)
(TXT) w3m dump (www.usenix.org)
| throwaway892238 wrote:
| I agree. We need a standard specification for distributed
| computing. Categorize all the different aspects of a distributed
| system, then write standard APIs for them all.
|
| The whole point behind POSIX was so people didn't need to
| constantly reinvent the wheel and glue things together to do
| simple things on different platforms. Today if you need to do
| anything in the cloud, you must "integrate" the pieces, which
| means to take out scissors and glue and toothpicks and a text
| editor and literally arts-and-crafts the pieces together. Want to
| add something to your system? More arts and crafts! Rather than
| waste all that time, if everything just used a common interface,
| everything would just work together, the way it (mostly) does
| with POSIX.
|
| This is bad for the companies that make all their money off
| proprietary/incompatible interfaces - lock-in devices that are
| advertised as a plus because they can be "integrated" with enough
| time and money. But it's better for our productivity and the
| advancement of technology if we didn't have to integrate at all.
| Just run the same thing everywhere.
| user3939382 wrote:
| My answer to the title is "no" for the same reason we're still
| using email protocols designed in 1985. I guess the counterpoint
| would be systemd.
|
| TFA says "Therefore, we argue that the POSIX era is over, and
| future designs need to transcend POSIX." So they're actually not
| answering their own question, they're saying "the answer should
| be x" which is subtly different. There are a lot of should-be's
| in computing.
| pjmlp wrote:
| Since I moved into managed languages I never cared about POSIX
| any longer, as long as the runtime supports the underlying
| hardware, OS or hypervisor, I am good.
|
| Ironically, while I am an heavy email user, at work it is
| mostly used for internal newsletters, all the communication
| happens over Slack.
| hsghfhfjshsg wrote:
| wake me up when you can signup to a tiktok account using
| slack.
|
| slack replaced your internal, HR monitored, internal IM. Not
| email. Just so happens that lots of places were using email
| as their IM solution.
| pjmlp wrote:
| I don't use tiktok either way, so I guess you will be
| sleeping for a long time.
|
| In case you missed part of my comment I mention the small
| detail of being an avid email user, but of course you
| needed to do some tiktok marketing.
| throw0101c wrote:
| Hadn't heard of "Dennard scaling" before:
|
| > _These services cannot expect to run faster from year to year
| with increasing CPU clock frequencies because the end of Dennard
| scaling circa 2004 implies that CPU clock frequencies are no
| longer increasing at the rate that was prevalent during the
| commoditization of Unix._
|
| Definition:
|
| > _Dennard scaling, also known as MOSFET scaling, is a scaling
| law which states roughly that, as transistors get smaller, their
| power density stays constant, so that the power use stays in
| proportion with area; both voltage and current scale (downward)
| with length.[1][2] The law, originally formulated for MOSFETs, is
| based on a 1974 paper co-authored by Robert H. Dennard, after
| whom it is named.[3]_
|
| * https://en.wikipedia.org/wiki/Dennard_scaling
|
| The article then mentions Moore's Law.
| notacoward wrote:
| It's interesting that this is almost entirely about the
| filesystem _API_ , but barely touches on all the other things
| that the object-store proponents and other "POSIX is dead"
| advocates usually focus on. Hierarchical directory structure,
| byte addressability, write in place, permissions model,
| persistence and consistency requirements, and so on. If everybody
| switched to a new API today, POSIX would still be very much
| alive. I'm not going to argue about whether the rest of POSIX
| should die or not, having played an active role in such
| discussions for about twenty years, but for that to happen it has
| to be a _lot_ more than just changing how requests and responses
| are communicated.
| pjmlp wrote:
| Cloud computing runtimes and serverless.
|
| They can be running bare metal on top of a type 1 hypervisor,
| on top of a UNIX, Windows, mainframe, micro, or whatever.
|
| Android and ChromeOS, which although based on the Linux kernel,
| don't expose a POSIX userspace to regular app developers as
| public API, only OEMs get to play with it, or placing the
| devices into developer mode like Crostini.
|
| It doesn't matter at all on the XBox and Nintendo, but it does
| on PlayStation.
|
| Its relevancy depends pretty much how deep one still is into
| classical UNIX like workflows, but like COBOL and Fortran it
| isn't going away anytime soon.
|
| Unysis still sells Burroughs as well.
| [deleted]
| garaetjjte wrote:
| >which although based on the Linux kernel, don't expose a
| POSIX userspace to regular app developers as public API
|
| No, Android apps can contain native code that uses Linux
| syscalls.
| pjmlp wrote:
| Do so at your own peril, they aren't part of the public
| API.
|
| "NDK C library"
|
| https://developer.android.com/ndk/guides/stable_apis#c_libr
| a...
|
| "Improving Stability with Private C/C++ Symbol Restrictions
| in Android N"
|
| https://android-
| developers.googleblog.com/2016/06/improving-...
|
| "Android changes for NDK developers"
|
| https://android-developers.googleblog.com/2016/06/android-
| ch...
|
| Termux developers also think they could do whatever they
| felt like on Android, guess what, they can't.
| mwcampbell wrote:
| > However, contemporary applications rarely run on a single
| machine. They increasingly use remote procedure calls (RPC), HTTP
| and REST APIs, distributed key-value stores, and databases,
|
| I'm seeing an increasing trend of pushback against this norm. An
| early example was David Crawshaw's one-process programming notes
| [1]. Running the database in the same process as the application
| server, using SQLite, is getting more popular with the rise of
| Litestream [2]. Earlier this year, I found the post "One machine
| can go pretty far if you build things properly" [3] quite
| refreshing.
|
| Most of us can ignore FAANG-scale problems and keep right on
| using POSIX on a handful of machines.
|
| [1]: https://crawshaw.io/blog/one-process-programming-notes
|
| [2]: https://litestream.io/
|
| [3]; https://rachelbythebay.com/w/2022/01/27/scale/
| oefrha wrote:
| > Running the database in the same process as the application
| server, using SQLite, is getting more popular with the rise of
| Litestream.
|
| As someone who uses SQLite a lot, I'm suspicious of this claim.
| Litestream is strictly a backup tool, or, as its author puts
| it, disaster recovery tool. It gives you a bit more peace of
| mind than good old periodic snapshots, but it does _not_ give
| you actual usable replication,* so I doubt it meaningfully
| increased SQLite adoption in the RDBMS space (compared to the
| application data format space where it has always done well).
|
| * There was a live read replica beta which has since been
| dropped. Author did mention a separate tool they're working on
| which will include live replication.
| https://github.com/benbjohnson/litestream/issues/8#issuecomm...
| tekacs wrote:
| For folks' context, the new tool that's being discussed in
| the thread mentioned by the parent here is litefs [0], as
| well as which you can also look at rqlite [1] and dqlite [2],
| which all provide different trade-offs (e.g. rqlite is 'more
| strongly consistent' than litefs).
|
| [0]: https://github.com/superfly/litefs
|
| [1]: https://github.com/rqlite/rqlite
|
| [2]: https://github.com/canonical/dqlite
| bryanlarsen wrote:
| Cranshaw mentions "tens of hours of downtime" as a drawback.
| Given that downtime sometime corresponds with times of high
| load aka, the really important times, that's going to be a deal
| killer for most.
|
| But his architecture does seem to be consistent with a "minutes
| of downtime" model. He's using AWS, and has his database on a
| separate EBS volume with a sane backup strategy. So he's not
| manually fixing servers, and has reasonable migration routes
| for most disaster scenarios.
|
| Except for PBKAC, which is what really kills most servers. And
| HA servers are more vulnerable to that, since they're more
| complicated.
| mattarm wrote:
| > Cranshaw mentions "tens of hours of downtime" as a
| drawback. Given that downtime sometime corresponds with times
| of high load aka, the really important times, that's going to
| be a deal killer for most.
|
| I posit that this kind of "deal killer" is most often a wish
| list item and not a true need. I think most teams without a
| working product think these kinds of theoretical reliability
| issues are "deal killers" as a form of premature
| optimization.
|
| I worked at a FANG doing a product where we thought
| availability issues caused by sessions being "owned" by a
| single server design was a deal killer. I.e. that one machine
| could crash at any time and people would notice, we thought.
| We spent a lot of time designing a fancy fully distributed
| system where sessions could migrate seamlessly, etc. Spent
| the good part of a year designing and implementing it.
|
| Then, before we finished, a PM orchestrated purchase of a
| startup that had a launched product with similar
| functionality. Its design held per-user session state on a
| single server and was thus _much_ simpler. It was almost
| laughably simple compared to what we were attempting. The
| kind of design you 'd write on a napkin over a burrito lunch
| as minimally viable, and quickly code up -- just what you'd
| do in a startup.
|
| After the acquisition we had big arguments between our team
| and those at the startup about which core technology the FANG
| should go forward with. We'd point at math and theory about
| availability and failure rates. They'd point at happy users
| and a working product. It ended with a VP pointing at the
| startup's launched product saying "we're going with what is
| working now." Within months the product was working within
| the FANG's production infrastructure, and it has run almost
| unchanged architecturally for over a decade. Is the system
| theoretically less reliable than our fancier would-be system?
| Yes. Does anybody actually notice or care? No.
| thayne wrote:
| It is a deal killer for anyone who has SLAs specified in
| contracts. Which is pretty common in B2B
| macintux wrote:
| Maybe. In that example, if the service has run for over a
| decade, it seems plausible that whatever contractual
| penalties they would have had to pay out for occasional
| downtimes would be far less than the initial and ongoing
| development time required to implement a far more complex
| solution, not to mention the additional hardware/cloud
| costs.
| [deleted]
| mike_hearn wrote:
| If you have an application server then you still have RPCs
| coming from your user interface, even if you run the whole DB
| in process. And indeed POSIX has nothing to say about this.
| Instead people tend to abuse HTTP as a pseudo-RPC mechanism
| because that's what the browser understands, it tends to be
| unblocked by firewalls etc.
|
| One trend in OS research (what little exists) is the idea of
| the database OS. Taking that as an inspiration I think there's
| a better way to structure things to get that same simplicity
| and in fact even more, but without many of the downsides. I'm
| planning to write about it more at some point on my company
| blog (https://hydraulic.software/blog.html) but here's a quick
| summary. See what you think.
|
| ---
|
| In a traditional 3-tier CRUD web app you have the RDBMS, then
| stateless web servers, then JavaScript and HTML in the browser
| running a pseudo-stateless app. Because browsers don't
| understand load balancing you probably also have an LB in there
| so you can scale and upgrade the web server layer without user-
| visible downtime. The JS/HTML speaks an app specific ad-hoc RPC
| protocol that represents RPCs as document fetches, and your web
| server (mostly) translates back and forth between this protocol
| and whatever protocol your RDBMS speaks layering access control
| on top (because the RDBMS doesn't know who is logged in).
|
| This approach is standard and lets people use web browsers
| which have some advantages, but creates numerous problems. It's
| complex, expensive, limiting for the end user, every app
| requires large amounts of boilerplate glue code, and it's
| extremely error prone. XSS, XSRF and SQL injection are all bugs
| that are created by this choice of architecture.
|
| These problems can be fixed by using "two tier architecture".
| In two tier architecture you have your RDBMS cluster directly
| exposed to end users, and users log in directly to their RDBMS
| account using an app. The app ships the full database driver
| and uses it to obtain RPC services. Ordinary CRUD/ACL logic can
| be done with common SQL features like views, stored procedures
| and row level security [1][2][3]. Any server-side code that
| isn't neatly expressible with SQL is implemented as RDBMS
| server plugins.
|
| At a stroke this architecture solves the following problems:
|
| 1. SQL injection bugs disappear by design because the RDBMS
| enforces security, not a highly privileged web app. By
| implication you can happily give power users like business
| analysts direct SQL query access to do obscure/one-off things
| that might otherwise turn into abandoned backlog items.
|
| 2. XSS, XSRF and all the other escaping bugs go away, because
| you're not writing a web app anymore - data is pulled straight
| from the database's binary protocol into your UI toolkit's data
| structures. Buffer lengths are signalled OOB across the entire
| stack.
|
| 3. You don't need a hardware/DNS load balancer anymore because
| good DB drivers can do client-side load balancing.
|
| 4. You don't need to design ad-hoc JSON/REST protocols that
| e.g. frequently suck at pagination, because you can just invoke
| server-side procedures directly. The DB takes care of
| serialization, result streaming, type safety, access control,
| error reporting and more.
|
| 5. The protocol gives you batching for free, so if you have
| some server logic written in e.g. JavaScript, Python, Kotlin,
| Java etc then it can easily use query results as input or
| output and you can control latency costs. With some databases
| like PostgreSQL you get server push/notifications.
|
| 6. You can use whatever libraries and programming languages you
| want.
|
| This architecture lacks popularity today because to make it
| viable you need a few things that weren't available until very
| recently (and a few useful things still aren't yet). At
| minimum:
|
| 1. You need a way to distribute and update GUI desktop apps
| that isn't incredibly painful, ideally one that works well with
| JVM apps because JDBC drivers tend to have lots of features.
| Enter my new company, stage left (yes! that's right! this whole
| comment is a giant ad for our product). Hydraulic Conveyor was
| launched in July and makes distributing and updating desktop
| apps as easy as with a web app [4].
|
| 2. You're more dependent on having a good RDBMS. PostgreSQL
| only got RLS recently and needs extra software to scale client
| connections well. MS SQL Server is better but some devs would
| feel "weird" buying a database (it's not that expensive
| though). Hosted DBs usually don't let you install arbitrary
| extensions.
|
| 3. You need solid UI toolkits with modern themes. JetBrains has
| ported the new Android UI toolkit to the desktop [5] allowing
| lots of code sharing. It's reactive and thus has a Kotlin
| language dependency. JavaFX is a more traditional OOP toolkit
| with CSS support, good business widgets and is accessible from
| more languages for those who prefer that; it also now has a
| modern GitHub-inspired SASS based style pack that looks great
| [6] (grab the sampler app here [7]). For Lispers there's a
| reactive layer over the top [8].
|
| 4. There's some smaller tools that would be useful e.g. for
| letting you log into your DB with OAuth, for ensuring DB
| traffic can get through proxies.
|
| Downsides?
|
| 1. Migrating between DB vendors is maybe harder. Though, the
| moment you have >1 web server you have the problem of doing a
| 'live' migration anyway, so the issues aren't fundamentally
| different, it'd just take longer.
|
| 2. Users have install your app. That's not hard and in a
| managed IT environment the apps can be pushed out centrally.
| Developers often get hung up on this point but the success of
| the installed app model on mobile, popularity of Electron and
| the whole video game industry shows users don't actually care
| much, as long as they plan to use the app regularly.
|
| 3. To do mobile/tablet you'd want to ship the DB driver as part
| of your app. There might be oddities involved, though in theory
| JDBC drivers could run on Android and be compiled to native for
| iOS using GraalVM.
|
| 4. Skills, hiring, etc. You'd want more senior devs to
| trailblaze this first before asking juniors to learn it.
|
| [1] https://www.postgresql.org/docs/current/ddl-
| rowsecurity.html
|
| [2] https://docs.microsoft.com/en-us/sql/relational-
| databases/se...
|
| [3]
| https://docs.oracle.com/database/121/TDPSG/GUID-72D524FF-5A8...
|
| [4] https://hydraulic.software/
|
| [5] https://www.jetbrains.com/lp/compose-mpp/
|
| [6] https://github.com/mkpaz/atlantafx
|
| [7]
| https://downloads.hydraulic.dev/atlantafx/sampler/download.h...
|
| [8] https://github.com/cljfx/cljfx
| agumonkey wrote:
| what about the ux side of things ? I love everything you
| mention about perf but I never ran into a single 90s era app
| that had interesting ergonomics. Web can be shitty or worse
| but there are some valuable UX aspects now.
| cgh wrote:
| Desktop apps that query/update the db directly were a thing
| back in the '90s. They were an example of what we called
| "client/server". They were swiftly superseded by the web,
| which sort of hijacked the client/server architecture. As you
| noted, the basic reason is desktop app distribution and
| updating is hard. If your company can beat this problem, then
| great, because removing the intermediate web layer makes a
| lot of sense in certain cases (eg, enterprise deployments).
| mcculley wrote:
| I have built a proprietary Swing desktop app that we use
| inside the tugboat company I run that does this. The
| wrinkle is that instances of this app are only
| intermittently connected to the Internet and our central
| instance of PostgreSQL. The Swing app uses a local SQLite
| instance and synchronizes it in the background when a
| connection is available. The users never experience any
| latency.
|
| Careful schema design to support synchronization without
| collisions is the only real difference between this kind of
| app and CRUD apps that expect to always be able to reach
| the Internet.
| mike_hearn wrote:
| Yep. OS vendors dropped the ball on distribution, browsers
| picked up the slack. But browsers also dropped the ball in
| a lot of ways. E.g. browsers do a lot of useful stuff but
| none of that is available for servers or CLI apps. There's
| lots of scope to do better. Also, on the desktop you have
| POSIX or POSIX-ish features at least :)
|
| BTW, Conveyor is free for open source projects, and
| currently free for commercial too. The current versions
| focus on solving the basics really well. Deployment/update
| is like for a website statically generated from Markdown.
| You can build fully signed-or-self-signed and self-updating
| packages from cross-platform artifacts on whatever OS you
| happen to use with one command. So, you can package up
| Electron and JVM apps given just inputs like JS or JAR
| files, from your dev laptop or Linux CI box, and you get
| packages for every OS. You also get a simple download HTML
| page that detects the user's OS and CPU.
|
| To do a new release you just re-build and re-upload the
| site. Clients will start updating immediately. On macOS it
| uses Sparkle, on Windows the OS will do the updates in the
| background and Linux users get packages.
|
| It does native apps too but then of course you need to
| compile the binaries for each OS yourself.
|
| One possibility we're researching is to page code in from
| the database on demand. That way you only have to push
| updates to the client occasionally, like when refreshing
| the core runtimes. For changing business logic the client
| would use SQL queries to speculatively load code based on
| what other clients have been requesting. If it works it
| means you can get rid of minification, bundling, all the
| other hacks web devs do to reduce requests and round-
| tripping, whilst keeping the "instant" deployment browsers
| give you.
| whartung wrote:
| I only have a couple of points regarding this.
|
| First, simply, I don't know anyone that puts their DB
| connections "on the internet". That live, raw, database "SQL"
| socket to be poked, prodded, hammered, and cajoled by
| complete strangers (or their automatronic minions).
|
| Second, is DB latency. Sending "coarse" grained service
| requests across the planet is one thing compared to the
| potential volume of random SQL commands that folks do.
|
| Mind, much of that can be mitigated if you build a stored
| procedure service layer. But classic "2 tier" "client/server"
| work didn't do that exclusively, just just threw out SQL
| willy nilly as the need dictated.
|
| As old school as I am, even I tend to shy away from the
| "monolithic DB". You think your app was a monolith before,
| wait until it's all baked into Oracle. I've always found the
| DB to be a "bad citizen" when it comes to things like
| versioning, source code control, etc.
|
| Even if I were doing a desktop app, I still think I would
| prefer a middle tier ("application server") managing service
| endpoints than cram it all into the DB, especially today.
| mike_hearn wrote:
| Shodan knew at least 600,000 PostgreSQLs listening on the
| open internet when I last looked. Presumably quite a few
| are mistakes, of course. But people do it and the sky
| doesn't fall. Same for SSH or many other types of server.
| Of course the web ecosystem has 30 years of accumulated
| work so yes, you'd be missing stuff like Cloudflare,
| reCAPTCHA etc. Better for more controlled contexts than
| something like HN.
|
| Latency is easy to screw up whether you do web apps or
| direct SQL connections. You have to be conscious of what a
| request costs, and you can easily batch SQL queries. Yes,
| you have to watch out for frameworks that spam the DB but
| those are bad news anyway, and of course there are lots of
| web frameworks that generate inefficient code. Not sure
| it's so different.
|
| Your app will have to deal with DB versioning whether it's
| a web app or not. Tools like Flyway help a lot with linking
| your DB to version control and CI.
|
| Nonetheless, I totally understand where you're coming from.
| Thanks for the thoughts.
| coldtea wrote:
| >* Shodan knew at least 600,000 PostgreSQLs listening on
| the open internet when I last looked. Presumably quite a
| few are mistakes, of course.*
|
| A few? I'd say most are accidental and the rest are just
| bad ideas...
|
| > _But people do it and the sky doesn 't fall._
|
| Well, the same is true for playing Russian roulette too.
| Most of the times you're winning!
| mike_hearn wrote:
| We don't know either way, but a standard Postgres install
| doesn't let remote connections do much. You still have to
| authenticate before anything is allowed. It's not much
| different to sshd in this regard. A typical web server is
| far more promiscuous, with a massive surface area exposed
| to unauthenticated connections. There have been way more
| disasters from buggy web frameworks/apps that get
| systematically popped by crawlers, than from people
| running RDBMS.
| pjmlp wrote:
| Basically back to the VB/Delphi glory days with stored
| procedures, or even better Oracle Forms.
| mike_hearn wrote:
| Yeah. Never used Oracle Forms but did use Delphi a lot.
| Borland never tried to solve distribution any better than
| Microsoft did. These firms were born in the era when
| "shipping" was meant literally and showed no interest in
| the problem of how to change software more than once every
| few years. Then people realized you could iterate a web app
| every day if you wanted to, the way they worked gave you
| total insight into what users were actually doing, you
| could use scripting languages better than VB and more.
| Businesses wanted the agility, devs wanted to use UNIX and
| Perl instead of VB/Delphi and the web was off to the races.
|
| There were other issues of course, it wasn't just about
| distribution. Too bad so many downsides came along with the
| upsides. The real goal for OS research should be IMHO to
| find ways to combine what people like about web dev with
| what people like about desktop/mobile dev. All the action
| is above the POSIX layer.
| mwcampbell wrote:
| > the way they worked gave you total insight into what
| users were actually doing
|
| How do you suggest achieving this in desktop apps? Some
| kind of log streaming?
| mike_hearn wrote:
| Most RDBMS have auditing features these days. Unless
| there's a lot of complex logic on the client that's
| sufficient. Otherwise yes, there are plenty of libraries
| and services for collecting and analyzing logs these
| days. I've looked at streaming Java Flight Recorder logs
| too, but it's probably overkill.
| mwcampbell wrote:
| > devs wanted to use UNIX and Perl instead of VB/Delphi
|
| What do you think drove this? Presumably plenty of people
| in the dark mass of 9-to-5 devs were happy with
| VB/Delphi. Jonathan Edwards has written [1] that VB came
| from "a more civilized age. Before the dark times...
| before the web." Did nerdy devs like me, with our
| adolescent anti-Microsoft attitude (speaking for myself
| anyway; I was born in 1980), ruin everything?
|
| [1]: https://alarmingdevelopment.org/?p=865
| mike_hearn wrote:
| The language and libraries situation on Windows wasn't
| great during this time.
|
| Delphi was good but a compiled language with manual
| memory management. It was _very_ easy to write code that
| crashed, which would nuke the user 's state leaving
| little evidence of what happened. It also had a lot of
| legacy quirks like not allowing circular dependencies
| between compilation units, networking support was poor
| (iirc TCP, HTTP and other classes required you to buy a
| third party library). The VCL was a wrapper around Win32,
| which had some great strengths but also really
| frustrating weaknesses e.g. extremely poor/non-existent
| layout management, poor support for typography and no
| support for styling or branding. There were many positive
| aspects of course.
|
| Microsoft gave you VB or C++, both with Win32 again. The
| C++ developer experience was horrid. VB was at least a
| scripting language with garbage collection, but, it was
| also constrained by the fact that B stood for "Beginners"
| so Microsoft were very reluctant to fix any of its legacy
| or add more powerful features.
|
| Compared to that situation, scripting languages and
| especially Perl had some massive advantages:
|
| 1. Ran on UNIX/Big Iron which is where all the best
| hardware and databases could be found. Lots of devs liked
| UNIX because it was CLI and developer oriented.
|
| 2. Unashamedly designed for experts with tons of powerful
| features, quality of life stuff like integrated regex,
| garbage collection, proper stack traces, error logs you
| could view via telnet within seconds etc.
|
| 2. CPAN provided an ever growing repository of open
| source libraries, instantly accessible, for free! On
| Windows there were very few libraries, they were mostly
| quite expensive and closed source, no easy way to
| discover them (pre-Google) and only C++/Delphi devs could
| write them. VB was sort of a consumer-only language. Open
| source culture started with RMS at the MIT AI lab and so
| was very UNIX centric for a long time. Arguably it still
| is.
|
| Really, it's hard to overstate how revolutionary proper
| garbage collection + CPAN was. GC is a force multiplier
| and CPAN is the granddaddy of all the open source library
| repositories we take for granted today. Imagine how
| unproductive you'd be without them.
|
| The big downside was that Perl had no UI libraries and
| didn't really run on Windows. So how do you use it to
| write apps for normal people? Then Netscape started
| adding interactivity features to the web and it was all
| totally text based. Text was Perl's forte! Add the <form>
| tag, CGI, HTTP and now you're cooking with gas. Bye bye
| hateful 0xC00005 Access Violation errors and useless bug
| reports like "I clicked a button and the app
| disappeared".
|
| The web was a huge step back _for users_ , who went from
| having pretty sophisticated GUIs with fast table views,
| menus, shortcut keys, context menus, Office integration,
| working copy/paste, instant UI response etc to ... well,
| the web. But often users will suffer through that if it
| makes their devs more productive because all they really
| care about are features and competitive advantage. The
| web was crude but it let people escape the Windows
| ecosystem to one with open source + GC + pro languages +
| big iron UNIX.
| AtlasBarfed wrote:
| "a more civilized age. Before the dark times... before
| the web."
|
| There is revisionist history, and then there is that
| statement. That statement ... is almost offensive.
|
| ... Why did the "golden age" end? Because Microsoft
| sucked in so so so so so so so many ways. That statement
| is a bald faced lie, a whitewashing attempt to forgive
| Microsoft from inflicting millions of man years in damage
| to the IT industry over three decades of anticompetitive
| practices to keep their substandard software in a
| position of market dominance.
|
| But for anyone beyond the MCSE factory programmer (aka
| NOT the thought leaders of the industry) aside from those
| profiting handsomely from the Microsoft evil empire, did
| not like Microsoft.
|
| In the days of DOS, you had DOS or UNIX. Which was
| better?
|
| In the days of Windows in the 3.1 and even pretty much in
| the windows 95, you didn't have preemptive multitasking
| (something that non-windows had for 20 years at that
| point). It crashed CONSTANTLY, had no security, required
| restarts for practically anything that was installed.
|
| Meanwhile the UNIX people would brag about machines not
| being rebooted for years. This was before the constant
| patch cycle was a thing.
|
| Microsoft's apis were slapdash and disorganized, and
| frequently went out of favor.
|
| During this time Microsoft would constantly crush and
| screw over competitors and engage in constant
| anticompetitive behavior. If you didn't suck on the teat
| of Microsoft, your platform was under unrelenting
| assault, not by pure technical achievement, but by the
| full smorgasborg of corporate dirty tricks, FUD,
| bribery/kickbacks, lobbying, lies, secret apis, golf
| schmoozing with nontechnical CIOs to close deals, no or
| undermined standards, etc.
|
| The graveyard is long: Sybase, Lotus 123, Netscape,
| Novell.
|
| The Microsoft times were a time of being stuck with one
| option: and OS that crashed constantly or is utterly
| porous to attackers. A browser that has incompatible web
| apis and a disbanded developer team (IE 6) that gets
| corporate mandated and is a thorn in your side in the
| entire IT stack for two decades. Databases stolen from
| companies (Sybase signed SUCH a bad deal, it astonishes
| me to this day) running on platforms that can't stay up.
| Office software with inaccessible file formats and
| byzantine and closed programmatic apis for accessing it.
| A substandard desktop UI.
|
| If you used Microsoft software with an ounce of
| historical knowledge or awareness, you could see the
| prison. You had no practical choices. All the executives
| in your company were bought and paid for. Microsoft had
| sales forces that tracked non-Microsoft systems and
| targeted them within companies by any means necessary.
| Every new piece of software written in MIcrosoft had to
| pay the "sociopath management tax" and go through
| extensive reviews on how it could be used to further or
| maintain Microsoft's empire and control.
|
| Their software was deliberately dumped in beta form onto
| the market to crowd out the competitors.
|
| None of this is an "adolescent" attitude. I'll repeat
| myself: MILLIONS OF MAN HOURS OF DAMAGE. You know,
| probably billions. Decades x tens of millions of workers.
|
| This isn't just IT programmer frustration. This is bad
| applications forced on non-programmer users. This is
| better companies, better software, better IT industry
| denied proper funding and profits. Instead, Microsoft
| took trillions of dollars in revenues from them. This is
| undermining a free market, free ideas, and freedom for
| Microsoft's profit.
| mike_hearn wrote:
| Both stances are too extreme. Yes, the web didn't bring
| "dark times", it was adopted for sound reasons. But your
| view of Windows vs UNIX is equally far out. Until the web
| got good Windows vs UNIX was no contest for anything
| meant for ordinary users. To the extent MS was anti-
| competitive it was hurting Netscape and Apple, not UNIX:
|
| * Until Linux, UNIX didn't even run on PCs at all, only
| expensive proprietary hardware. Windows ran on cheap
| machines made by a competitive market. Businesses wanted
| PCs for lots of reasons (e.g. Office).
|
| * UNIX wasn't a single OS, it was many forks that were
| not compatible. You couldn't just write an app for UNIX,
| put it on CDs and sell it.
|
| * Anything GUI related was _terrible_ compared to Win32.
| Motif was the standard but cave man tech compared to what
| you could do on the client with Windows.
|
| * UNIX was wildly more expensive.
|
| * PCs had a thriving gaming culture. It meant people used
| the same tech at home as at work, creating a self-
| training workforce. UNIX vendors didn't care about games,
| they were too 'serious' for that. Their loss.
|
| * Windows 95 did do pre-emptive multi-tasking, by the
| way. Only Win16 apps that hadn't been ported were
| cooperatively multi-tasked. Those apps didn't stick
| around very long because porting to Win32 was easy and
| Windows 95 was an absolute monster success. People queued
| up at midnight around the blocks of PC stores to buy it,
| it was so popular.
|
| * Windows apps crashed a lot compared to UNIX apps
| because the average Windows machine ran way more apps
| than the average UNIX machine, ran on way lower quality
| hardware that was often flaky, had way more diversity of
| peripherals and configurations, and apps were deployed in
| places where it was hard to get crash logs (no networks).
|
| * Windows machines didn't have uptimes of years because
| nobody cared. They were workstations. You turned them off
| at the end of the day when you went home because
| otherwise they'd burn their CRTs in and their lifetime
| was reduced. The culture of leaving non-server machines
| on all the time and never rebooting them only began once
| ACPI and other power management tech started to become
| common (something non-Apple UNIX struggles with to this
| day). And once it was useful to do so, Microsoft had a
| version of Windows that could have uptimes of months or
| years, no big deal.
| mek6800d2 wrote:
| I agree with you about the massive damage caused by
| Microsoft, but ... it was there. In the DOS times, there
| was no choice between DOS and Unix on PCs -- there was
| effectively just DOS. Aside from niche things like
| Coherent (IIRC), Unix was only available on workstations
| and up, too expensive for small businesses and consumers.
|
| Also, VMS and other OSes were the ones that ran for years
| without rebooting. Unix at the time was not so stable.
| Before Tcl, John Ousterhout wrote a log-structured file
| system for Unix because their Unix systems crashed often
| enough and took so long to boot that a possible loss of
| data but fast boot-up was deemed better than the
| lengthier downtimes with the existing file system.
|
| So the PC market went with Microsoft and its encompassing
| environment, much to everyone's detriment. Fortunately,
| we've all moved on to the web and JavaScript and
| everything is now sunshine and roses. :-)
| mwcampbell wrote:
| Interesting approach.
|
| Web applications can come close to directly accessing the
| database by using GraphQL with something like Hasura or
| PostGraphile. PostGraphile even uses Postgres's row-level
| security. A colleague and I once did a project using Hasura
| with a SPA-style JavaScript front-end and a separate back-end
| service driven by Hasura webhooks for doing the actual
| computation, and we ended up being unhappy with that
| approach. Some of our problems were related to the SPA
| architecture, but some were related to our use of GraphQL and
| Hasura.
|
| We ended up starting over with a typical server-rendered web
| application, where the server itself accesses the database
| and communicates with the computation-heavy back-end service
| over gRPC, using a minimum of client-side JavaScript. I
| remain happy with that architecture, though I continue to
| explore different ways of integrating modest amounts of
| client-side JavaScript for interactivity and real-time
| updates. And to bring it back to the topic of my previous
| comment, if we assume that there has to be a server rendering
| HTML anyway, then I think it often makes sense to reduce
| complexity by bringing the database into that server process.
| I haven't yet tried that in production, though.
|
| I think my preference is to use HTTP primarily as it was
| originally intended, for fetching HTML, as well as for form
| submissions. For finer-grained interactivity, I think it's
| better to use WebSockets as opposed to REST-ish requests and
| responses. I'm not dogmatic on that, though.
|
| On web apps versus packaged desktop apps, I'm still inclined
| to go with web whenever feasible, and only develop a packaged
| desktop app if a web app just won't work. Being able to use
| an application without an installation step is really
| powerful for pre-sales demos or trials, for onboarding new
| users, and for applications that may only be used
| occasionally by any given user. Even for an application that
| you use all the time, a web app can be fine, as the
| popularity of Google Docs demonstrates. For example, if you
| just want to get the browser chrome out of the way, desktop
| browsers support "installing" a web app as an OS-level app
| with no browser chrome. IMO, Hydraulic's Eton demo app could
| just as well be a web app.
|
| I look forward to your blog post, though even your
| preliminary HN comment offers a lot to think about.
| mike_hearn wrote:
| The Eton Notes app is just a way to show what the
| download/install UX looks like when combined with a Compose
| app. It's mostly just a mockup with no real functionality.
|
| Yes, for transient apps that are only used occasionally or
| where the user isn't committed e.g. social networks, the
| combination of sandboxing + no integration/unintegration
| steps, automatic deletion from the cache etc is really
| useful. Of course there's no rule that says only _web_
| browsers can supply these features. Other kinds of browser-
| like thing can do so too.
|
| It's also worth thinking a bit outside the box. Although we
| claim web apps don't have installation steps, that's not
| really true most of the time. The lack of any explicit
| integration step ("install") means the browser doesn't know
| if the user values anything they do on the site. So you
| have to provide data persistence, and that in turn means
| you need a signup / account creation flow. It also means
| you're on the hook to store, replicate and back up any data
| any user ever creates, even if they only used it once and
| then never return.
|
| Well, a lot of users really hate creating _yet another_
| account, especially if they aren 't committed yet. It's
| tedious, they think you'll spam them with marketing emails
| and they're probably right, plus they don't want to make
| another password. Or they make one, abandon for a year,
| then can't remember how to log in.
|
| You might think that's so fundamental it can't be any other
| way, but it's really just a side effect of how browsers
| have evolved. Think about how you might do demos outside
| the browser. You could just have a trial mode where the app
| spins up a local in-process RDBMS like H2 that writes the
| data into the app's private data area (on Windows) or home
| directory on macOS/Linux. No accounts necessary - just one
| or two clicks to download+trigger app install, and you're
| done. If/when the user decides to graduate, _then_ they
| create an account and the app uploads the data from the
| local DB to the real remote DB. If they abandon it and don
| 't care, it's not your problem, it costs you nothing. If
| they run low on disk space the OS will suggest they delete
| old unused apps at that point and you'll get tossed along
| with the rest.
|
| Mostly though, this is about making developers more
| productive. If the primary determinant of your success is
| feature throughput and not shaving a few seconds off your
| onboarding, e.g. you're making specialised apps, internal
| apps, enterprise software, then optimizing for better dev
| environments can make sense. Installation just isn't that
| bad.
| bigdict wrote:
| > The primary use case for Unix was to multiplex storage (the
| filesystem) and provide an interactive environment for humans
| (the shell) [2], [3]. In contrast, the primary use case of many
| contemporary POSIX-based systems is a service running on a
| machine in a data center, which have orders of magnitude lower
| latency requirements.
|
| And yet the most pleasant development environment for producing
| the latter is going to be the former.
| phendrenad2 wrote:
| Why would POSIX need to be replaced? Sure, POSIX is a bit clunky
| to use, but who uses it directly? And there aren't any
| performance benefits you can get from replacing POSIX. This is
| just one of those questions that CS academics ask themselves
| because they ran out of algorithms to discover.
| dis-sys wrote:
| It was a nice read. Thanks for sharing the article on HN.
| However, I feel the fellowing claim in the article is not backed
| by any detailed facts/analysis.
|
| > Virtual memory is considered to be a fundamental operating
| system abstraction, but current hardware and application trends
| are challenging its core assumptions.
|
| What are specifically challenging the vm assumptions?
| chasil wrote:
| Maybe that expensive ram in low quantities and rotational
| storage on a 32-bit system are no longer relevant as design
| constraints?
| mike_hearn wrote:
| The overhead of the TLB and page walks have been increasing
| over time as memory gets larger - 4kb pages made sense in some
| contexts but not others so page sizes are now increasing, and
| this is complicating APIs and kernels. The complexities of huge
| pages on Linux being a case in point.
|
| I suspect he may also be thinking of the prevalence of one
| virtual machine:one app in the cloud, where you can argue (and
| the Seastar guys do) that the process abstraction just slows
| you down because the unit of isolation is the VM anyway.
| generalizations wrote:
| > POSIX provides abstractions for writing applications in a
| portable manner across Unix-like operating system variants and
| machine architectures. However, contemporary applications rarely
| run on a single machine. They increasingly use remote procedure
| calls (RPC), HTTP and REST APIs, distributed key-value stores,
| and databases, all implemented with a high-level language such as
| JavaScript or Python, running on managed runtimes. These managed
| runtimes and frameworks expose interfaces that hide the details
| of their underlying POSIX abstractions and interfaces.
| Furthermore, they also allow applications to be written in a
| programming language other than C, the language of Unix and
| POSIX. Consequently, for many developers of contemporary systems
| and services, POSIX is largely obsolete because its abstractions
| are low-level and tied to a single machine.
|
| > Nevertheless, the cloud and serverless platforms are now facing
| a problem that operating systems had before POSIX: their APIs are
| fragmented and platform-specific, making it hard to write
| portable applications. Furthermore, these APIs are still largely
| CPU-centric, which makes it hard to efficiently utilize special-
| purpose accelerators and disaggregated hardware without resorting
| to custom solutions. For example, JavaScript is arguably in a
| similar position today as POSIX was in the past: it decouples the
| application logic from the underlying operating system and
| machine architecture. However, the JavaScript runtime is still
| CPU-centric, which makes it hard to offload parts of a JavaScript
| application to run on accelerators on the NIC or storage devices.
| Specifically, we need a language to express application logic
| that enables compilers and language run times to efficiently
| exploit the capabilities of the plethora of hardware resources
| emerging across different parts of the hardware stack. At the
| same time, it would be an an interesting thought experiment to
| ponder how different would the hardware design of these devices
| be without the CPU-centrism in POSIX.
|
| It sounds like the author discovered the motivation behind the
| last time the POSIX standard was obsoleted, and some of the
| reasons why Plan9 was developed.
| pjmlp wrote:
| And after Plan 9 came Inferno with even less POSIX.
| gumby wrote:
| > Unix was the first operating system written by programmers for
| programmers
|
| This is of course far from true; CTSS and especially ITS (which
| was so programmer friendly that rather than what is called today
| a "shell", it's user interface was a debugger) long predated it
| and both Thompson and Ritchie were exposed to them while at MIT
| to work on Multics. The supreme programming environments for
| programmers, the MIT Lispms and the D-machines at PARC are the
| ultimate in programmer-friendly OSes that are, for various
| important reasons, unmatched today.
|
| Also, in their early papers about Unix they talked about how they
| didn't need all that fancy stuff that mainframe systems like
| Multics needed, like VM and IO modules (Unix just does everything
| through the kernel in a simple fashion). Well, it was for a small
| computer system. There were good reasons that mainframes were
| structured the way they were and by the mid 80s that model
| started migrating to workstations (e.g. Britton-Lee accelerator
| and graphics subsystems) and by the 90s that model was already
| quite common in microcomputers too (for example with mpus in disk
| controllers and network interfaces).
|
| But the Unix cultists ignored that revolution. They came late to
| the party with regards to networking; even when developing the
| sockets interface they relied heavily on the PDP-10 approach, but
| seemingly didn't learn anything else from that.
|
| Glad they are finally noticing that the world has changed.
| 323 wrote:
| There was a great talk I can't find right now about how the share
| of computer hardware that it's directly controlled by the OS
| kernel keeps on shrinking.
|
| In a modern smartphone, the OS directly controls just one of a
| dozen chips. The rest are running their own OSes which do their
| own thing independently of the main OS.
| pid-1 wrote:
| Could be this one? https://youtu.be/tCMs6XqY-rc
| chalst wrote:
| I'm guessing it's Tim Roscoe's keynote on how most
| fundamental questions about what the hardware is actually
| doing are invisible to traditional OS abstractions:
|
| https://www.youtube.com/watch?v=36myc8wQhLo
|
| Roscoe's talk is fairly long (the video is > 1h), but the
| basic thesis was taken up in a segment of Bryan Cantrill's 20
| min OSFF talk:
|
| https://www.youtube.com/watch?v=XbBzSSvT_P0
|
| Both talks are very good; I recommend watching both, in
| either order.
| 323 wrote:
| Yes, it was Tim Roscoe's one.
| stjohnswarts wrote:
| How can you claim posix is dead when you will need some 10-100x
| better to get it adopted?
| Icathian wrote:
| This falls right in line with the recent "hardware offload for
| all" article posted on HN, and the talks around Hubris that Bryan
| Cantrill has given. It's an exciting time for computing, I think.
|
| I've often lamented that I was born too late to watch Unix/POSIX
| be born and grow up. That's always felt like a golden time in
| computing. I also wasn't in the industry yet for the early 2000s
| and the rise of web-scale, though I was at least aware. It's
| exciting to think about what systems will do over the next twenty
| years. I'm looking forward to it.
| jeff-davis wrote:
| Posix used to feel like a nice compromise that worked for high
| level applications, one-off tasks, and low-level system software.
| I remember the first time I created an ISO file using "cat
| /dev/cdrom > some_file.iso", and I thought it was amazing that
| you didn't need a separate application for that (like you do on
| windows).
|
| Now the posix APIs feels like the worst of both worlds.
|
| Most applications shouldn't really store data on the filesystem,
| and probably not even configuration. Most applications use a
| database for a zillion reasons (e.g. better backup, restore,
| consistency, replication, transactions, avoiding random writes,
| ...). Yet the filesystem is also not a great interface if you're
| building a database: you want more control over flushing and
| buffering and atomic file operations. The filesystem is mainly
| good at storing compiled code.
|
| The "everything is a text file" interface is not great anymore,
| either. Simple manipulation of the output of basic commands (like
| text 'ps') requires error-prone parsing and quite some skill with
| things like grep, sed, awk, sort, etc. That's OK for experienced
| sysadmins working on individual machines, but doesn't integrate
| very well in larger systems where you have lots of automated
| access to that information. I have some experience doing
| sysadmin-like work, but I still run into problems with simple
| things like spaces in filenames and getting the quoting and
| escaping right in a regex. One option is for it to be more
| database-like, but that's not the only option. Erlang provides
| nice programmatic access to processes and other system
| information while still feeling very free and ad-hoc.
|
| Of course, Linux is still great in many ways. But it feels like
| some of the core posix APIs and interfaces just aren't a great
| fit, and a lot of the interesting stuff is happening in non-posix
| APIs. And applications/services are being built on higher-level
| abstractions that aren't tied to single machines.
| Animats wrote:
| > Most applications use a database for a zillion reasons
|
| Is it time to look at database-first operating systems again?
| There have been a few. Tandem's Guardian was very well regarded
| in its day. Unfortunately, Tandem was acquired by DEC/Compaq,
| which tried to move it to the Alpha around 1997. Then, after HP
| acquired what was left of Compaq, HP tried to move it to
| Itanium. (There was a MIPS version in there somewhere.) After
| all those bad decisions, in 2014 it was finally ported to x86,
| by which time few cared. There's still an HP offering.[1]
|
| In Guardian, the database system owned the raw disks. There
| were no "files". If you wanted something that looked like a
| file, it was just a blob in a database entry. Everything was
| transaction-oriented, like SQL. That was the whole point.
| Everything was a redundant, recoverable transaction.
|
| The OS was cluster oriented. Adding or replacing servers was
| normally done without shutdown. Databases were replicated, and
| you could add new disks, wait for them to come into sync, and
| remove the old ones.
|
| All this in the 1970s. They were way ahead of their time.
|
| [1] https://www.hpe.com/us/en/servers/nonstop.html
| stjohnswarts wrote:
| Until someone gives me a system 10x more productive and useful
| I'll stick to posix.
| mustache_kimono wrote:
| > Now the posix APIs feels like the worst of both worlds.
|
| I think, like anything, it depends on what you're doing, and
| how you're doing it.
|
| > The "everything is a text file" interface is not great
| anymore, either.
|
| Text doesn't seem markedly different from a JSON API? Perhaps
| "worse is better" more often when you are composing several
| programs/apps to make one system? Even complex distributed
| systems.
|
| > But it feels like some of the core posix APIs and interfaces
| just aren't a great fit
|
| And I think these APIs can thrive on top of (the good parts of)
| POSIX. I'm not so certain we need to fundamentally rethink this
| layer, because this layer seems to work pretty well at what it
| does well. It should be pruned occasionally, but I'm not sure
| we need a whole new abstraction.
|
| FWIW, I think this what the article is saying. Let's create
| some new interfaces where the old interfaces don't model the
| hardware well[0].
|
| [0]: https://queue.acm.org/detail.cfm?id=3212479
| dc-programmer wrote:
| There is the alternative of an object based interface like
| Windows offers.
|
| I thought it was an interesting idea until I actually got
| around to experimenting with Powershell recently. The
| ergonomics are more akin to Python than any of the GNU tools.
| I can't help but wonder why not just use a language like
| Python rather than a heavyweight scripting language if it's a
| similar LOE (speaking in general, I'm aware of CLR)? The
| benefit of the text based interface to me is the expediency
| of performing one-off CLI actions. I inclined think an object
| interface actually impedes this goal.
| carapace wrote:
| I've come around to the opinion that (POSIX-style) filesystems
| are no longer a great idea. Instead, I think something more like
| Git (plumbing, not porcelain) would be preferable as the
| underlying storage abstraction.
| gjvc wrote:
| AS/400 ?
| mike_hearn wrote:
| What exactly are you thinking of? Git manages files after all.
|
| If you mean the underlying data structures, that's basically
| what modern filesystems are. XFS, APFS, BTRFS etc are all copy-
| on-write file systems that use git-like structures underneath.
| In the same way that git branches are "instant" because no data
| is actually copied, so too can these file systems clone files
| or whole trees instantly without copying any real data. You can
| easily "branch" a directory tree, rsync the results to a remote
| machine, etc.
| carapace wrote:
| The thought is somewhat inchoate still. I'm working with a
| pure functional language (Joy https://joypy.osdn.io/ ) and
| when it came time to add filesystem support I balked.
| Instead, I'm trying out immutable 3-tuples of (hash, offset,
| length) to identify sequences of bytes (for now the "backing
| store" is just a git repo.) Like I said, it's early days but
| so far it's very interesting and useful.
|
| I get what you're saying about modern filesystems, and I
| agree. I guess from that POV I'm saying we could stand to
| remove some of the layers of abstraction?
| mike_hearn wrote:
| Well, Git still uses mutable state stored in files. You
| can't avoid it - the world is mutable. The question is how
| to expose and manage the mutations.
|
| At any rate you might be interested in a few different
| projects:
|
| 1. BlueStore: https://ceph.io/en/news/blog/2017/new-
| luminous-bluestore/
|
| 2. The DAT or IPFS protocols, which are based on the idea
| of immutable logs storing file data, identified by hashes,
| with public keys and signatures to handle mutability.
| comex wrote:
| For one thing, it would be nice if every directory had a hash
| that covered all its contents (recursively), like Git tree
| objects. That way, all sorts of tools that need to check
| which files in a directory tree have changed - including `git
| diff`, `make`, file sync tools, and indexed search tools -
| could immediately skip directories with no changes, without
| needing a separate fsmonitor tool. The cost would be higher
| overhead recalculating hashes when files are constantly being
| updated.
|
| It would also be nice to support filesystem transactions,
| somewhat analogous to Git commits. POSIX file APIs make it
| difficult to avoid race conditions in the best circumstances,
| and extremely difficult if there's a security boundary
| involved. You can never check something about a path (e.g.
| "is this a symlink?") and then rely on that thing being true,
| because it could have been concurrently modified between the
| check and whatever you do next. So you have to rely on the
| limited atomic semantics available - for example, instead of
| explicitly checking for a symlink, you can use O_NOFOLLOW -
| but those are not always sufficient, depending on what you're
| trying to do. It shouldn't be this way. I should be able to
| take a frozen snapshot of the filesystem, inspect it to my
| heart's content, make the changes I want, and finally
| atomically commit or abort.
|
| Regarding copy-on-write clones, can any of those filesystems
| actually clone arbitrary directories? In APFS's case, you can
| make copy-on-write clones of files, and you can make copy-on-
| write snapshots of entire volumes, but you can't just take
| some random directory and make a copy-on-write clone of it
| (without individually cloning all the files under the
| directory). I believe the same limitation exists for some or
| all of the modern Linux filesystems.
| jra_samba wrote:
| I'm giving a (slightly updated) version of my talk at the
| Storage Network Industry Association Storage Developer's
| Conference (2022) in Freemont, CA next thursday:
|
| https://storagedeveloper.org/events/sdc-2022/agenda/2022-09
| -...
|
| "Symbolic links Considered Harmful"
|
| Might be relevant to readers :-).
| mike_hearn wrote:
| Filesystems do have timestamps, but they aren't propagated
| to the root for performance reasons - normally you want
| file IO to be fast more than you want to be able to quickly
| test if a large directory tree changed and propagation
| would make all writes contend on a single lock (for the
| root dir hash/ts).
|
| Agreed about fs transactions. Not sure about cloning
| directory trees. I thought btrfs could do it but I never
| used that feature. You might well be right.
| sylware wrote:
| Richard Stallman POSIX now will have to do one of the hardest
| stuff in the software realm: resist planned obsolescence while
| replacing some interfaces with "in-the-end better" interfaces,
| that on the long run. This is extremely though: often, replacing
| "in-the-end better" interfaces is actually planned obsolescence.
|
| For instance event based programming, "epoll" should replace
| "select". Synchronous programming, signalfd, timerfd, etc... but
| then "signals" should be more accurately classified, for instance
| in monothreaded applications segfault won't be delivered from a
| "signalfd" in a "epoll" syscall... and why not keep only the
| "realtime signal" behavior?
|
| If POSIX only "add" new "options", in the end, you'll get
| gigantic spec just being the mirror image of the very few
| implementations since its size will make it unreasonable for a
| new implementation from scratch.
|
| The "leaner POSIX" route seems the right way here, but it is
| really easier to say than to do.
| bitwize wrote:
| No, I/O completion ports (which Linux only recently got a form
| of) should replace epoll and select.
|
| fork() should be ditched.
| theteapot wrote:
| > fork() should be ditched.
|
| Why? In favor of what?
| bitwize wrote:
| Something like windows CreateProcess or posix_spawn().
|
| fork() plays havoc with threads. If you want to start a new
| process, specify a fresh process image.
| 10000truths wrote:
| Embryonic processes. Basically:
|
| 1. Call a function that creates an empty child process in a
| suspended state.
|
| 2. Call functions that: map/write memory into the child
| process's address space; add file descriptors to the child
| process; set the child process's initial register values;
| and so on.
|
| 3. Call a function to unsuspend and run the child process.
| lgg wrote:
| fork() is terrible, but embryonic processes also have a
| lot of performance issues and prevent a number of
| valuable security mitigations. In general a spawn() style
| mechanism seems like a better approach (despite the
| deficiencies of specific examples like posix_spawn()).
| aseipp wrote:
| Because fork() was very simple and conceptually "easy" to
| do when it first was introduced, and is now massively
| complex and has huge implications on every part of the
| system. It's not compositional, isn't thread safe, insecure
| by default (inherits env/fds), and it's also slow with all
| the state it must copy. And at a conceptual level it
| doesn't work in environments where the nature of a
| "process" and "address space" aren't synonymous. For
| instance if your application uses a hardware accelerator
| (NIC, GPU, whatever) fork() isn't ever viable or sensible,
| since the hardware resources can't be duplicated safely.
| And it'll never work in a design like WebAssembly (just an
| example of this idea, but WASM isn't the only one), again
| "process" and "virtual memory address space" are not the
| same. Consider that posix_spawn can make reasonable sense
| in WebAssembly at a first guess ("launch this wasm
| module"), but fork() in contrast is much more difficult
| when it implies COW semantics.
|
| The reality is fork() is pretty much exclusively used to
| launch new processes these days, outside a few specific
| cases. Today, it's a poor fit for that problem. And the
| answer is what Windows has been doing (and POSIX has now
| had) for a long time: explicitly launching processes by
| giving a handle/pathname to an executable like posix_spawn.
| That's the first solution, anyway; a better one would be
| more capability-oriented design where you have to supply a
| new address space with all its resources yourself.
|
| This HotOS paper is a pretty good detailed coverage of the
| argument; I find it very convincing. If fork() went away, I
| honestly wouldn't miss it, I think.
| https://www.microsoft.com/en-
| us/research/uploads/prod/2019/0...
| hkgjjgjfjfjfjf wrote:
___________________________________________________________________
(page generated 2022-09-10 23:00 UTC)