[HN Gopher] Gnu/Hurd strikes back: How to use the legendary OS i... ___________________________________________________________________ Gnu/Hurd strikes back: How to use the legendary OS in a (somewhat) practical way Author : yarapavan Score : 197 points Date : 2023-08-07 16:38 UTC (6 hours ago) (HTM) web link (mhatta.medium.com) (TXT) w3m dump (mhatta.medium.com) | mhaberl wrote: | > The GNU Hurd's hardware support is poor, so trying to run it on | modern physical machines is suicide. | | > ...one reason for using GNU/Hurd is that the Linux kernel has | become too huge | | By any chance, are these two statements related? Is it smaller | because it lacks the drivers? | Hizonner wrote: | The drivers aren't part of the microkernel and run with limited | privilege. So, yeah, OK, if you wrote all those drivers you'd | have a lot of code, but not all of it would have total control | of the system. You can't really make a direct comparison. | | Linux has a hell of a lot of system calls and weird bells and | whistles in non-driver code, too. | dale_glass wrote: | I also recall it having some ridiculously low limit for | partitions, like 1GB or something. | | Apparently because it mmaps the entire disk, and under 32 bit | CPUs there's not enough address space. | | Which is a ridiculous technical limitation. You could have | bigger disks than that by the mid 90s. | | Come think of it, my Ryzen 9 3950X supports 43 bits of address | space which works out to 8TB. So if that limit applies, even | the 64 bit version would be limited on modern hardware. | monocasa wrote: | A 3950x has 48 bits of virtual address space, or 256TiB. | dale_glass wrote: | I'm honestly not sure which counts for this purpose, | physical or virtual? And which fraction do you get to use | on HURD? | | But still, 256 TiB is just a dozen drives these days. Not a | very consumer-like configuration, but very much a limit you | can run against. | preinheimer wrote: | They lost me with the GNU explanation. | phant0mas wrote: | You can try using it through Guix :) | slim wrote: | how? | sourcepluck wrote: | Really nice article, it's cool to see people working on GNU/Hurd. | chillbill wrote: | Medium is such a hostile platform. | OfSanguineFire wrote: | I'm surprised to see that Mach is still the microkernel for Hurd. | When I last (very casually) followed Hurd development two decades | ago, Mach was generally regarded as obsolete and there were hopes | to base Hurd on a new microkernel like seL4. | marcosdumay wrote: | Oh, interesting, people created a GNU clone of Mach, removing | the licensing problem that all but killed the project early on | its life. | | For the ones not in the know, Mach is an OS designed after the | idea of "the network is your environment, not the computer" | where it should make little difference if the resources you are | accessing is on your computer or not. And "resource" here is | really generic, meaning things like files, login information, | printers and other periferals, CPU, whatever. | | For some reason (maybe related to costs and licensing), people | never adopted the idea... and insisted on reimplementing it | with unreliable hacks and lots and lots of problems on top of | every other OS out there instead of just having it on the OS | itself. | cesarb wrote: | > For the ones not in the know, Mach is an OS designed after | the idea of "the network is your environment, not the | computer" where it should make little difference if the | resources you are accessing is on your computer or not. And | "resource" here is really generic, meaning things like files, | login information, printers and other periferals, CPU, | whatever. For some reason (maybe related to costs and | licensing), people never adopted the idea | | To me, the reason is obvious: what happens when someone | unplugs the network cable? This is #1 in the list of | distributed computing fallacies (https://en.wikipedia.org/wik | i/Fallacies_of_distributed_compu...): "the network is | reliable". The other fallacies on that list might also be | relevant. | marcosdumay wrote: | > what happens when someone unplugs the network cable? | | About the same thing that happens when you unplug your | corporate Windows machine. You only have cached and local | resources. | | People insist on using all of that functionality. People | also insist on implementing it by hacks everywhere with | nothing talking with each other and each part failing for | its own reasons. | anyfoo wrote: | > an OS designed after the idea of "the network is your | environment, not the computer" where it should make little | difference if the resources you are accessing is on your | computer or not. [..] For some reason (maybe related to costs | and licensing), people never adopted the idea... | | I personally think it's a pipe dream, and the way history | worked out seems to confirm that. | | _Truly_ transparent networking never really worked out | anywhere. The better RPC solutions in use, for example, make | it very obvious that an RPC is an RPC, instead of looking | like a local function call. In file transfer, it has been | common to e.g. replace NFS with something HTTP based for | quite some time. "Cloud" file systems seem to operate at a | much higher layer, preferring external syncing and special | high level APIs with only some actual OS support sprinkled | in. | | For Operating Systems, a lot of Mach breaks down when you | want to do things fast and securely. | | It's a testament to APIs better being shaped by locality. | Roughly and informally spoken, the closer you are to the | actual CPU, the more "lightweight" and "direct" your API can | be. The further you move away, the more resilient you need to | become, and the more control and insight you need to give the | caller about the actual transport. | | To pick two very real extremes as an example: When writing to | a register on a local bus, you often just assume it will | work. If it doesn't, a panic is sensible (sometimes even | worse, like a lockup leading to a watchdog timeout). When | trying to invoke a function on another server, obviously | that's not acceptable. Then you want to be able to timeout, | retry, inspect transport failures, fall back to other | servers, configure endpoints, be picky and explicit about | payload (and result), and so on and so forth. | | The immense cruft necessary for the latter case becomes | unnecessary and expensive the more local you are. It affects | security, and can even pose chicken and egg problems. | m463 wrote: | Isn't macos Mach? | rayiner wrote: | HURD (the parts that aren't the rest of the GNU OS) is mainly a | set of servers for Mach. These are closely coupled to how mach | handles IPC, how it handles virtual memory, etc. Apparently, | there were also shortcomings in L4 itself, or at least with | respect to what HURD wants to do with its microkernel: | https://www.gnu.org/software/hurd/history/port_to_another_mi... | naasking wrote: | > Apparently, there were also shortcomings in L4 itself, or | at least with respect to what HURD wants to do with its | microkernel | | Yes, that was the issue as I recall. But seL4 is an | excellent, secure high-performance microkernel. If something | is painful to do in L4, it will likely not be faster or more | secure running under any other microkernel, and so you should | arguably question the wisdom of what you're trying to do. | pshirshov wrote: | Is it really alive? No SMP, no stable x64 port. No ARM port. | pengaru wrote: | I used to idle in the hurd irc channel and there was definitely | someone working on smp support, but it was just one person's | pet project apparently. They seemed to have things working in | an experimental setting, but I don't think anything landed | upstream. | | It was clear there's a severe lack of manpower on the project. | I think it'd have to be rewritten in rust or something hip to | get new blood today. | johnny22 wrote: | Redox seems like it's a bit farther along than hurd in some | repects https://www.redox-os.org/ | cmrdporcupine wrote: | It's very cool, but the MIT license on that means it will | never fit the niche (mainly ideological) that the Hurd was | aiming at. | | I actually think GPL is key in long-term adoption and is | part of at least the early part of Linux's success. To | prevent fragmentation. To coax commercial entities to | contribute. And to generally encourage community | development, rather than forks (since forks require | distributing source anyways). | | But others disagree, and these days Apache/MIT type | licenses proliferate. | bpye wrote: | I'm not convinced that the license is the biggest | motivator today for vendors getting code to upstream | Linux. Look at LLVM - Sony has spent significant effort | landing support for PlayStation targets there, the | license doesn't require it, but it means they aren't | chasing such a moving goalpost with some internal tree. | cmrdporcupine wrote: | Kinda wonder if the best approach for them at this point would | be to target RISC-V and worry only really about running in a | hypervisor, or very narrowly targeted open hardware, instead of | futzing with implementing a pile of device drivers. Make it | useful in that environment, and then go from there. | | Hell, wouldn't it be cool if there was an official GNU GPL'd | RISC-V board, running GNU Hurd? | rekado wrote: | See https://lists.gnu.org/archive/html/bug- | hurd/2023-06/msg00038... | | > * APIC, SMP, and 64bit support was improved a lot: they now | do boot a complete Debian system, but some bugs remain to be | fixed. | Hizonner wrote: | Can anybody compare the state of Hurd with, or point to good | comparisons with, similar projects like Genode? | Analemma_ wrote: | Even rms gave up on Hurd and it stopped being a core priority | of the GNU Project a long time ago. The current | maintainers/developers are doing it as a solo hobby. Which is | fine, but that's what you should keep in mind as far as support | if you decide to use it. | nerdponx wrote: | Is Hurd (in theory) a good fit for lightweight VMs? | jacquesm wrote: | Once upon a time the Hurd was the future. But that time is long | in the past and nowadays the Hurd is a relic and I don't see any | chance of a revival. Something entirely new probably stands a | better chance at arousing some interest than the Hurd. | AdmiralAsshat wrote: | https://miro.medium.com/v2/resize:fit:4800/format:webp/1*HuH... | | The GNU/Hurd default terminal can display a JPEG inside the | terminal? | msla wrote: | Same with Linux: | | https://en.wikipedia.org/wiki/Linux_framebuffer | | XTerm can as well, if built with Sixel support: | | https://github.com/saitoha/libsixel | | https://en.wikipedia.org/wiki/Sixel | arccy wrote: | where is the JPEG XL support? | ratmice wrote: | Also in that same screenshot, hurd used to drop you into a | shell as the nobody user where you could run the login | binary... Curious if that was changed as part of debian, or | changed for some other reason. | taeric wrote: | Kind of weird to see linux called out as too big, and then the | ack that writing a ton of device drivers is hard and will take | time. Isn't that a large part of the source tree? Realizing I | haven't even tried compiling a kernel in a long long time. Feels | oddly sad to say. | cbsmith wrote: | With a microkernel model, most of those drivers would run in | user space, not kernel space. So it's possible for the device | drivers to be large but the GNU HURD kernel itself to be much | smaller than the Linux kernel. | badrabbit wrote: | Hurd is a microkernel so it is lean by design. I suspect that | statement was a dig on perceived architectural flaw of Linux. | taeric wrote: | I get that. But it is still an unfair dig when a large chunk | of the code that is being pointed at is the code that they | will have to write? (Granted, they can put an easier line in | the sand on how far back they go on what they support | hardware wise, but still...) | pengaru wrote: | It's not about the code existing, it's about the code being | an integral part of a monolithic architecture vs. "servers" | facilitating a microkernel design like hurd/mach. | taeric wrote: | But that is silly? If I have to audit the code for my | system running, I would have to audit the code for each | of the "servers" as well? If not, why not? | | To that end, how different is it, actually? And what are | the other tradeoffs? | tlb wrote: | That's the big unknown question with microkernels. It was | (in the 90s) a reasonable theory that a microkernel could | be much more robust and secure and easy to develop than a | monolithic kernel. | | It would be more robust because individual services could | crash and restart. That could really work for, say, WiFi | drivers. It doesn't quite work for disk drivers or the | file system, since how do you even restart something | without a file system? | | It would be more secure because drivers would live in | isolated memory spaces. So a non-critical piece like a | printer driver couldn't read memory from something more | critical like a file system. Part of what changed since | microkernels were originally proposed is that drivers are | often split into low- and high-level parts. So a printer | driver in Linux consists of a standard USB or network | driver in the kernel, plus a user-level driver that can | be worked on like a microservice. And the WiFi driver is | a low-level driver plus wpa_supplicant running at user | level. The USB and network drivers are shared with | critical services, so you can't easily restart them | anyway. | | It would be easier to develop because you can restart | just the piece you're working on without rebooting the | whole kernel. That's a plausible argument, but somewhat | undermined by HURD taking 30 years to develop. But there | were many other reasons for that. | | I guess the way to convince yourself one way or another | is to try writing a driver for both. | taeric wrote: | Right, I remember parts of that history. Memory | protection schemes in other operating systems didn't | exactly stay static, though. And a ton of the | optimizations that monolithic kernels (mostly looking at | windows) did went away just with the lack of need for | them. Heck, at the time, Mac wasn't even a preemptive | multitasking operating system. | | And "drivers" could just restart is laughable with a role | play of it. Ok, your wifi driver restarted. What are the | protocols to get everyone caught up between that driver | and your application for what messages were lost and the | messages that are now arriving? This is effectively no | different from any distributed application where similar | things are unsurprisingly hard to achieve well. | | I'll state that I think this is a laudable goal. And I'm | happy to see people working on it. Would be thrilled to | be shown I'm wrong. | strbean wrote: | > Ok, your wifi driver restarted. What are the protocols | to get everyone caught up between that driver and your | application for what messages were lost and the messages | that are now arriving? | | TCP? | taeric wrote: | Doesn't fully help? Could actually be more difficult if | you let the driver do the TCP ack. And you are likely | going to back yourself into a situation where you need | TCP between the application and the driver... Remember, | at that point, it is just a message pass, as well. | | There is a good article somewhere about how you have to | do end to end for any of this to really work. | klodolph wrote: | If your WiFi driver crashes and restarts, at the minimum, | I think it could be treated as if you lost network | connectivity for a short time. This is a situation that | applications should be able to handle anyway, since it | happens all the time on laptops. | | There are plenty of applications out there that just | don't respond to configuration changes gracefully. For | example, if you have a music program running, and then | you plug a MIDI keyboard in, is the new MIDI keyboard | recognized by your music program? Maybe so, and you can | use it immediately. Maybe not, and you have to restart | the application. | | If your MIDI driver crashes, maybe the music program | doesn't handle it gracefully, and you have to quit and | relaunch the program. But that is miles better than | crashing the system and rebooting. | taeric wrote: | It really depends on what all is done "in the driver," | though? Since, per the application, the data may have | been passed off to the driver with a successful message | pass. And if I have to have secrets in the driver for it | to be able to connect to access points and such, it seems | dubious how protected from all of my secrets the system | can be between all parties. | | MIDI is actually an easy example, all told. If I unplug | and replug a keyboard in, I expect it to probably not | handle any persistent state of the keyboard correctly. | (For fun, it is common for the "at rest" value of peddles | to be discovered at connect time. Good luck with that if | the driver crashes.) | klodolph wrote: | > Since, per the application, the data may have been | passed off to the driver with a successful message pass. | | The application _already_ knows that the data is not | guaranteed to be delivered, or even sent at all. That's | just the nature of network programming--it's inherently | unreliable. | | > And if I have to have secrets in the driver for it to | be able to connect to access points and such, it seems | dubious how protected from all of my secrets the system | can be between all parties. | | To make an analogy--locking your front door doesn't | protect thieves from stealing your plastic flamingoes and | garden gnomes, but it does make it harder to steal the | $50,000 in cash you have hidden under the mattress. | | The point of isolation is to reduce the effect of | failures. If you have a buggy WiFi driver, in a | monolithic kernel, that could be really bad--it could | even be a remotely exploitable vulnerability in the | kernel itself. In other words, you have no front door, | and the thieves can steal the $50,000 hidden under your | mattress. If your WiFi driver is unprivileged, then the | thieves have only broken through your front gate, and | they can only steal your garden gnomes and plastic | flamingoes. They need to combine the vulnerability with a | local privilege escalation vulnerability, or be happy | with the value of plastic flamingoes. | | > For fun, it is common for the "at rest" value of | peddles to be discovered at connect time. | | The keyboard itself does this, either when it's turned | on, or through the configuration menu (or not at all). | It's not something that has anything to do with drivers | or connecting the keyboard. | pengaru wrote: | The servers are run in isolated address spaces like | processes, taking advantage of MMU protection for more of | the low-level facilities. | | Ask yourself why it's safer to use FUSE filesystems on | Linux when mounting untrusted block devices/image files | than the in-kernel filesystem drivers and maybe it'll | become clear why this is advantageous. | | Or if you're familiar with containers and why that's a | good thing, you can think of it as containerizing the | kernel's subsystems/drivers. | taeric wrote: | This... didn't answer my question? For many that are | concerned with containers and such, they also have to | audit the container process code. Such that, fine, try to | use that to inflate the numbers on Linux. | | Pointing at literally decades of device drivers as why | the size of the code is bad, while acknowledging that it | will take time to write device drivers so this will work | on a respectable amount of hardware just feels weird. | | That all said, please don't take this as a criticism of | any of these ideas. I confess seeing Hurd in the | headlines gave me a smile. Is nice to see people can | still work on many ideas out there. I am only questioning | the "size of code bad!" right next to "it will take us a | long time to replicate a large portion of the code we | just pointed at." | strbean wrote: | Code that will have to be written, but won't be part of the | kernel. I think that's on the cusp of fair/unfair. | taeric wrote: | I agree it is defensible. But, it is code that has to run | for me to use my computer. I'm not clear that I have | reason to care about the kernel distinction, at this | level. Would help the defense to lean in and show legit | and not theoretical examples of it. | [deleted] | weberer wrote: | Yes, its mostly drivers. Here's a breakdown of the Linux kernel | by lines of code: | | https://upload.wikimedia.org/wikipedia/commons/f/f5/Sankey_D... | winter_blue wrote: | Wow, I'm surprised by how small the file systems are. | | On the smaller side, Ext4 at just 21,515 lines, and on the | larger side, Btrfs at 55,758 lines. | | It almost makes me think that writing a new file system in | Rust, might actually be a tractable project. | PaulDavisThe1st wrote: | [flagged] | meepmorp wrote: | It's a good thing that you warned them off, or they | might've accidentally gone and done something interesting | or enjoyable. | PaulDavisThe1st wrote: | Yep, because everyone who is going to do something | interesting or enjoyable starts with the question "should | I use language X because it's cool?". | | Look, I'm as into cool FS-hacks as anyone, but writing a | new filesystem has 99+ problems and the language used is | either the last of them, or it isn't even in the set. | | Granted, you could (on Linux) write a little pseudo- | filesystem (e.g. /proc or /sys) and not face any of the | issues involved in extN or btrfs etc. I didn't get the | sense that this was what the OP intended, but I could be | wrong. | serf wrote: | I think it's a great project to tackle. | | Here are my criticisms : | | 1) LoC doesn't indicate complexity. When you're looking | at a code base and you see that it's 65000 lines, that by | no means should be an indicator of whether or not a re- | write is easily feasible -- especially with a filesystem. | | 2) Language shouldn't be a motivator for a re-write | without considering why the language offers advantages, | the end users don't care about which file extension is | littered across the source tree. | | Both of those tendencies (one to re-write everything in | your favorite language, and two to come across with an | idea that you understand how complex a mechanism is | simply from LoC) indicate to me, anecdotally of course, a | certain lack of experience; and I have a hard time | imagining someone with those concerns tackling a re-write | of a filesystem that gains 'the audience' any benefits. | | If it's just for the sake of personal toys and fooling | around with your own machine I feel as if those | motivations are fine, but I have concerns if 'a new | filesystem in Rust' comes from what I view as shaky | initial motivations. | Nereuxofficial wrote: | I also thought that'd be a fun project. BTRFS seems to have | the largest feature set although many features are | experimental but maybe there would be a place for something | smaller that scales well and maybe has some of the new | features like deduplication | JohnFen wrote: | Filesystems tend not to be all that complex to implement. | lutoma wrote: | I'm writing a hobby kernel/operating system and one of the | things that surprised me was how simple ext2 is. It really | isn't much different from fat32 in many ways, just a bit | more flexible and designed in a more future-proof way. | | Obviously ext2 is a very outdated filesystem by today's | standards, but considering ext3 and ext4 are essentially | just some extensions slapped onto ext2, I still didn't | expect it. | | Another thing I found really impressive was just how | resilient e2fsck is. During early development of my ext2 | code I frequently mangled the file system in a myriad of | weird ways, but no matter how much I broke things, e2fsck | was almost always able to restore the file system into a | reasonable state. | | (For the curious, there is some excellent documentation on | the ext2 file system at | https://www.nongnu.org/ext2-doc/ext2.html that I mostly | used for my implementation of it. Sadly no such thing seems | to really exist for ext3/ext4). | bee_rider wrote: | That's a neat way of visualizing it. | | It is interesting that the "remainder" chunk of the drivers | is so big. Wonder if they need some more categories. | stavros wrote: | I was expecting a block chart, kind of disappointed it | isn't. | dwheeler wrote: | I agree, it's odd. | | I don't have anything recent, but back in 2004, the _majority_ | of the Linux kernel code was in its drivers: | https://dwheeler.com/essays/linux-kernel-cost.html I expect | that most of the current Linux kernel code is also for handling | hardware (that is, drivers + the code to handle various | architectures). | AdmiralAsshat wrote: | Could one not make the case that maybe device driver support | doesn't belong in the kernel itself? | boricj wrote: | NetBSD has architectured their drivers to be basically | static archives reusable anywhere. It's part of the NetBSD | rump kernel. | einpoklum wrote: | What do you mean by "reusable everywhere"? I mean, where | could they be used other than in a NetBSD kernel? | boricj wrote: | Anywhere. There are lots of examples on the topic ; | running NetBSD driver code inside NetBSD's user-land is | the simplest use-case | (https://www.netbsd.org/docs/rump/sptut.html), but that | code is portable without modifications to just about any | context possible. | | Check out Antti Kantee's _The Design and Implementation | of the Anykernel and Rump Kernels_ if you want details on | the architecture. | rjsw wrote: | The drivers are not used in that way in NetBSD, they are | either compiled into the kernel or built as modules just | like in Linux, the graphics drivers even use the source | code from Linux. | heavyset_go wrote: | The Linux kernel driver API/ABI isn't stable so there's no | permanent/versioned interface you can program against that | would make out-of-tree drivers non-headache inducing: | https://www.kernel.org/doc/Documentation/process/stable- | api-... | | > _You think you want a stable kernel interface, but you | really do not, and you don 't even know it. What you want | is a stable running driver, and you get that only if your | driver is in the main kernel tree. You also get lots of | other good benefits if your driver is in the main kernel | tree, all of which has made Linux into such a strong, | stable, and mature operating system which is the reason you | are using it in the first place._ | ye-olde-sysrq wrote: | Isn't this this mono vs micro kernel debate all over again? | pessimizer wrote: | Isn't that the subject of this entire thread? | anon25783 wrote: | Things are perhaps a bit different now than when Torvalds | and Tanenbaum debated the issue on comp.os.minix over | thirty years ago. | dpe82 wrote: | What, specifically, is different? | anon25783 wrote: | The machines are different. Multicore 64-bit chips are | now standard for consumer PCs. RAM and persistent storage | are faster and much more abundant. The architecture of | the modern x86-64 is much more sophisticated than that of | the 386 for which the earliest Linux was written. | Vectorization, predictive branching, and asynchronous | code are all front and center in the modern programmer's | ecosystem. | | In short, hardware is more capable, and so perhaps now we | can afford to take more opportunities to trade a little | bit of overhead for abstractions that are more modular | and robust. | taeric wrote: | In a very real sense, the hardware has given a | realization of microservice ideas, but at the hardware | level. My WIFI card, the common and popular example, | probably has more processing power than some of the early | desktop computers back in the day. Certainly SSDs are | getting much more complicated. | | They aren't general purpose, but I presume microkernel | "services" would also not be general purpose? | lmm wrote: | Often they are general-purpose. There's a blog from maybe | 10 years ago of someone launching Linux on a hard drive | (it refuses to fully boot because the hard drive doesn't | have an MMU, but they get some familiar log output). | mike_hearn wrote: | Being a microkernel doesn't automatically make you more | robust. If you look at their docs, Hurd has only one | feature that's different to Linux and that's a sort of | souped up FUSE. But as they realized later, allowing | arbitrary unprivileged programs to extend the filesystem | like that doesn't mesh well with the UNIX security model. | You can write a "translator" that redirects one part of | the filesystem tree to another, so you get the same | issues as with chroot or symlink attacks. Their proposed | solution (never implemented?) is to stop programs paying | attention to translators by default if they are running | as a different user. There are other cases where | translators can DoS apps or do other undesirable things. | | The basic problem is that the kernel boundary is a trust | boundary. You assume that anything in the kernel is legit | and won't try to attack you, which simplifies things a | lot. In the microkernel world the original idea was that | all the servers would be just ordinary programs you could | swap in and out at will. But then the threat model | becomes muddied and unclear. Is it OK to run a server | that's untrusted? If so, to what extent? | | The Hurd seems under-designed as a consequence. To make | that vision work you'd need to be willing to depart from | UNIX way more, which really means being a research OS. | winternewt wrote: | RAM is faster but CPU is even faster. The latency chasm | between CPU and RAM has increased. | josefx wrote: | > Vectorization, predictive branching, and asynchronous | code are all front and center in the modern programmer's | ecosystem. | | Not to forget all the hardware based security bypasses | that let any random application read and even modify any | memory location it wants to. | praptak wrote: | Do we have more hardware isolation? There was a pretty | strong argument against microkernels - a driver running | in userspace can still bork the whole machine if the | hardware being controlled can write to memory at | arbitrary addresses. | j16sdiz wrote: | we have more hardware isolation. | | they are seldom implemented correctly on | firmware/mainboard level. | bonzini wrote: | Yes, there's IOMMUs. | | On the other hand, datacenters have become so large that | a 1% performance improvement can amount to millions of | dollars in hardware and energy savings, so the extra cost | of a microkernel might not be very welcome outside | consumer devices. | RoseRose wrote: | > In short, hardware is more capable, and so perhaps now | we can afford to take more opportunities to trade a | little bit of overhead for abstractions that are more | modular and robust. | | We could afford that before, it's just that Linus didn't | see the value and put his foot down. Which is definitely | a choice. | ssrc wrote: | I think it's monorepo vs multi-repo, and since Linux | maintainers update the drivers (I believe) when internal | APIs change, and the internal APIs are not stable, | monorepo seems more practical. | actionfromafar wrote: | Maybe I'm missing the joke, but in or out of kernel | context is orthogonal to how the source code is stored. | jonhohle wrote: | The author is the same (Linus), so it makes sense that he | would design a source control system that supported the | monorepo he created over the prior 15-ish years. | PaulDavisThe1st wrote: | "how the source code is stored" is what leads to "wow, | the linux kernel is really big". Nobody's measuring | kernel size by the number of resident pages it takes up. | davidw wrote: | There's something clever to say about the author of Linux | vs the author of Git, but it's not coming to mind right | now... | emmericp wrote: | 66% was drivers in 2019. | jasoneckert wrote: | And given what the modern Linux kernel does, alongside the | power of modern systems (even embedded ones), it's also | difficult to argue that the Linux kernel is too large by any | stretch of the imagination. | genewitch wrote: | my entire kernel, including all drivers needed for my system | except the nvidia blob is 11MB - 10876160 bytes. i don't | optimize for size, necessarily, i'm sure i could get it down | a lot more if i disabled most of the "default enabled" stuff | in a modern kernel that assures it will boot on a majority of | systems. | darksaints wrote: | I understand what you're saying, but both are true. | | 1. Linux is too big. Even if we exclude the drivers, which are | a majority of the kernel codebase, linux is still a massive | kernel. Most microkernels are small enough to fit in the L3 | cache, some are small enough to fit in the L2 cache. Linux, | even if we could exclude the drivers, doesn't even come close. | This inability to fully cache the kernel ends up negating the | primary benefit that monolithic kernels have, which is | performance. Linux spends more time bouncing around the cache | than a modern microkernel takes to do an extra syscall. | | To add to this, unless you are compiling your kernel | specifically for your machine, your kernel is going to be a | bloated compromise of the set of drivers that your system is | most likely to see. That means that you will have dozens, if | not hundreds of drivers compiled into your kernel which will | never need to be used. | | It's also too big in the sense that there are now far too many | lines of code to be able to effectively monitor for | vulnerabilites. That codebase size won't change when having a | microkernel with the same number of drivers, but what will | change is the ability of a single vulnerability to compromise | your whole system. You have effectively compartmentalized | vulnerabilities to their own address space. So in that sense, | big codebases are a huge problem for monolithic kernels, but | only a small problem for microkernels. | | 2. It will take time to write a ton of device drivers, simply | because most of them aren't written yet. To that extent, Linux | most definitely has an incumbent advantage. But that isn't to | say that microkernels also have an advantage here. Anyone who | has written a driver knows how much the ability to iterate | quickly is helpful. Drivers in userspace can be written | directly on the system that they will run on, with no | virtualization or kernel reboots necessary, and with incredibly | fast start/restart/stop. Userspace drivers are dramatically | simpler to write, simpler to build, simpler to monitor, simpler | to test, and simpler to distribute. | | With all that being said, I definitely don't think Hurd is the | future. But what could change the world is already out there | but not really ready to use at the moment. One possibility | would be an open-sourcing of QNX which is an amazing and | incredibly mature microkernel OS, possibly the most mature in | the world. If there are any bored billionaires out there | reading this, please buy QNX from Blackberry and do that. | Another possibility would be to put a shit ton of effort behind | the SeL4 userspace, which is just too damn hard to do anything | practical with at the moment. | taeric wrote: | I'm very open to that debate, but then show it. Exclude the | drivers from the numbers. To do otherwise feels very | dishonest. | | Process isolation and selinux are things that exist. You even | mention it. Turns out, having complete isolation between | basically everything in your system is hard. I don't see how | microkernels will make that difficulty disappear? | darksaints wrote: | But that is the entire point of the comparison. I get that | you think an apples to apples comparison is the most honest | way to portray this, but the whole point is to portray that | this is an apples to oranges comparison. Linux has millions | of lines of code running in ring 0, a micro kernel does | not. They may end up having the same lines of code for | comparable total system size, but that isn't the point of | the comparison. | | Process isolation actually isn't fundamentally hard...it's | just hard on a system that wasn't designed for it. And | SELinux doesn't do what microkernels do. Even with a | tightly locked down SELinux configuration, a driver | vulnerability can lead to root access. SELinux can protect | your userland the same way that microkernels do, but it | can't put your drivers in userland. | taeric wrote: | Fair. I'm not a fan of apples to oranges comparisons | without leaning into more of it. Especially not when the | obvious driver for difference between two things is that | one is old and has a ton of the code you are currently | behind on writing. | | I do question if process isolation isn't fundamentally | hard. Separate processes for separate things entirely is | not fundamentally hard. That, I agree. But, so much of | what we do is integrated together. Is why early attempts | to make it so that some programs can't see my whole hard | drive are bonkers annoying. I specifically want access to | the picture I just saved in that editing program so that | I can email it. | edgyquant wrote: | >obvious driver for difference between two things is that | one is old and has a ton of the code you are currently | behind on writing. | | In this case they're both old though, as Hurd predates | Linux by a year. | tredre3 wrote: | > I'm very open to that debate, but then show it. Exclude | the drivers from the numbers. To do otherwise feels very | dishonest. | | With drivers: --------------------------- | ----------------------------------------------------------- | - Language files | blank comment code --------------- | ----------------------------------------------------------- | ------------- C | 32699 3332747 2610014 17218820 | C/C++ Header 23690 712617 | 1373965 7109290 Assembly | 1336 48534 101884 234314 | | Without drivers: ------------------------ | ----------------------------------------------------------- | ---- Language files | blank comment code --------------- | ----------------------------------------------------------- | ------------- C | 13200 1031558 951267 5356551 | C/C++ Header 13760 300309 | 535100 1643611 Assembly | 1321 47500 100899 230638 | | Seven million lines without drivers. Linux is huge. I'm | sure you'll move the goalposts further though, so what | should I do next? Delete most stuff in arch/ and samples/ ? | | Numbers obtained by running cloc 1.96 on current kernel | master. | taeric wrote: | This seems to just support my criticism, though? Even | without the drivers, it is still a big number, but is | literally less than half the lines of code shown before? | Why inflate what you show just to get an absurd | comparison? | | I think many are misunderstanding my criticism here. I | offer it more as a way to strengthen the message than I | do to question it entirely. | | To that end, the only "goal post shift" I would add is to | ask how many of the 17 million lines of driver code have | to be done for Hurd to support hardware that would make | it more relevant for a ton of users? | kelnos wrote: | > _Delete most stuff in arch / and samples/ ?_ | | Probably? If you really want an apples-to-apple | comparison, delete everything but x86_64 support from | both Linux and Hurd. Remove sample code. | | linux/net isn't strictly "drivers", but implements the | base of the networking stack. That's 900k lines. Is that | networking base code included with the Hurd figures? If | not, delete it (or include in your comparison Hurd's base | networking code). | | Beyond that, this is getting a bit ridiculous. Who cares | about the LoC comparisons? They're very different kernels | with very different levels of maturity and number of | features. I'm not even convinced you can call one | "better" or "worse" based on this metric at all. | klodolph wrote: | > I'm very open to that debate, but then show it. Exclude | the drivers from the numbers. To do otherwise feels very | dishonest. | | You can ask for numbers without casting aspersions like | that. | | This is HN. It is something like a casual conversation. You | are free to ask for numbers, other people are free to | provide or not provide them, and we should be able to have | this conversation without leveling accusations of | dishonesty (which the word "feels" does not really temper). | taeric wrote: | I didn't say they were being dishonest, I said it feels | dishonest. Consider it a critique of the message with my | reasoning given. | | I was actually trying to find the numbers myself, and it | was surprisingly more involved than I'd care for. I | didn't want to give someone else more work, but happy to | see the numbers expanded. I stand by my criticism, | though. It feels off. If you want to say it is justified, | spend more effort justifying it. | kelnos wrote: | > _To add to this, unless you are compiling your kernel | specifically for your machine, your kernel is going to be a | bloated compromise of the set of drivers that your system is | most likely to see. That means that you will have dozens, if | not hundreds of drivers compiled into your kernel which will | never need to be used._ | | These days most Linux distros build pretty much any driver as | a module that can be built as one. Boot-critical modules | (like NVMe drivers) are included in the initramfs so they're | available before the disk is mounted. | | I agree that Linux is still larger than a microkernel, but | very few users have a bloated kernel in RAM due to lack of | customizations. | account-5 wrote: | What are your thoughts on Fuchsia? Not QNX but definitely got | billions behind it (until Google kill it that is). | darksaints wrote: | Fuschia is definitely interesting but I don't really see a | lot of non-Google work being done with it and while google | is worth billions, it doesn't seem like they're investing | billions into fuschia specifically. I'd say caution is | warranted but it's certainly a possibility if there were | more non-google players in the ecosystem. | dralley wrote: | >This inability to fully cache the kernel ends up negating | the primary benefit that monolithic kernels have, which is | performance. | | OK then, so, which microkernel provides performance on-par | with or better than Linux? | | This is a strong claim and some evidence is warranted. | darksaints wrote: | Linux IPC syscall latencies benchmark in the 10s of | microseconds typically. SeL4 has worst case execution | guarantees that are better than that, even counting the | fact that you have to do two syscalls for every one Linux | syscall. We're talking less than a thousand cycles total, | which would be less than a single microsecond. | | https://www.sel4.systems/About/Performance/home.pml | dralley wrote: | Does SeL4 have all of of the speculative execution | mitigations that Linux has? | | Syscall latency also is not the end-all-be-all of | performance. Does that _actually_ translate to faster | disk and socket I /O on average? | mschuster91 wrote: | > Syscall latency also is not the end-all-be-all of | performance. | | That depends on what you're doing. If you're running some | digital signage display or whatever, no one cares about a | missing display frame or whatever. But if you're running | a car's HUD or, worse, its steering system, you | _absolutely_ need the execution time guarantees or your | code won 't make it past certification. | dralley wrote: | I'm not saying it's not an important space, because it | is. But the majority of use cases are not that latency- | sensitive. And if you're not that latency-sensitive, it's | not obvious that microkernels like SeL4 and QNX are | faster than Linux, for the relevant definitions of | faster. | | And likely it's not even close when you take into | consideration the capability gap between Linux and | competitors, that is to say, the "fatness" comes from | legitimately useful functionality, or compatibility, that | any alternate kernel may need to implement to replace | Linux for its own use cases, and not because something is | wrong with Linux. | darksaints wrote: | I do believe there are speculative execution mitigations, | yes...and from what I remember there was a significant | performance impact when they were implemented, and | especially so on processors without dedicated code | caches. | | Syscall latency is definitely not the end-all-be-all of | performance, but it is all there is to compare between a | monolithic kernel and a microkernel. In both cases, | driver code needs to run, and that driver code can be | good or shitty just like everything else. | | When it comes down to it, the syscall latency that is | comparable is the context switch between privileged and | unprivileged mode. The holy grail of microkernels isn't | to be faster than macrokernels, it is to be just as fast, | but a hell of a lot more modular, usable, and secure. | toast0 wrote: | > To add to this, unless you are compiling your kernel | specifically for your machine, your kernel is going to be a | bloated compromise of the set of drivers that your system is | most likely to see. That means that you will have dozens, if | not hundreds of drivers compiled into your kernel which will | never need to be used. | | I imagine there's two pools of Linux users: those that | compile their own kernel, probably with only the drivers they | use; and those that use distribution kernels, which have a | small set of compiled in drivers, and a huge initrd with all | the drivers. Neither one of those groups ends up with all of | the drivers loaded into memory: the compile their own have a | small set, the distro kernel people end up with only the | drivers they need loaded; both groups have a relatively small | kernel footprint. | Dah00n wrote: | Can you use vulnerabilities in a driver that's there but | not currently loaded? | toast0 wrote: | Not unless you can trigger the system to load it. | mschuster91 wrote: | No, and this is what saved a lot of people's arses many | times over. A _lot_ of the CVEs for the Linux kernel turn | out to only be exploitable under very specific | circumstances. | emptysongglass wrote: | On the topic of the Linux kernel, why is it that modules and | config options are so poorly documented, often not at all? I | remember when I was compiling my own kernel, many options and | modules I'd just have to guess based on my hardware. I | understand Torvalds to be an incredibly demanding BDFL. I'd | assume he'd insist on clear helper docs for any modules that | wish to merge. | Affric wrote: | You can get 32MB L2 cache on an Intel i9 13000k and maximum | size of a compiled Linux Kernel appears to be around 8MB. | | I am not going to pretend I have any idea because I have just | googled these numbers but it seems to me that an amount of | cache could be reserved for the kernel without too much of a | hit to applications on what are currently high end machines. | twic wrote: | > Most microkernels are small enough to fit in the L3 cache, | some are small enough to fit in the L2 cache. Linux, even if | we could exclude the drivers, doesn't even come close. This | inability to fully cache the kernel ends up negating the | primary benefit that monolithic kernels have, which is | performance. Linux spends more time bouncing around the cache | than a modern microkernel takes to do an extra syscall. | | This makes no sense. The comparison that matters isn't kernel | vs microkernel, its kernel vs microkernel _plus all the | userland code needed to implement system services_. If my | program wants to read a file and write to the network, then | the code to do that has to be brought to the CPU to run | before my program can finish doing that, and it doesn 't | matter whether that is in the kernel or userland. | cesarb wrote: | > This makes no sense. The comparison that matters isn't | kernel vs microkernel, its kernel vs microkernel _plus all | the userland code needed to implement system services_. | | And also, what matters for performance is the size of the | "hot" code. A lot of code, both on a monolithic kernel and | on the userland code implementing system services for a | microkernel, is going to be "cold" code which executes | rarely. This includes initialization code, shutdown code, | most of the error handling code, code to react to uncommon | hardware events, and so on. | | > If my program wants to read a file and write to the | network, then the code to do that has to be brought to the | CPU to run before my program can finish doing that, and it | doesn't matter whether that is in the kernel or userland. | | And given that the essential complexity is the same, it's | very likely that the size of the relevant code is similar. | darksaints wrote: | You're right...that is the part that doesn't matter. But | the kernel still has to coordinate and schedule that. The | kernel has an inherent overhead for execution and context | switching, and will always have some amount of code that is | hot. | | The reason microkernels typically benchmark on syscall | (typically IPC) latency is because that is the only truly | apples to oranges part that you can't compare between the | two. Monolithic kernels have a single context switch, | microkernels have two. But microkernels, being micro, are a | lot friendlier to caches (less to load and flush), and | especially so with dedicated code caches (which do not need | to flush on context switch). | AtlasBarfed wrote: | On modern computer systems still virtual paged? I don't think | it really matters if the entire OS fits in the l3 cache | because if it's paged as i believe it is then the most | relevant pages will end up in cache. | | So even if you have a really large monokernal the parts of it | that actually get used will be in cache in the parts that | aren't used and won't be in cache. | | Has other people stated how Michael Colonel is still going to | need a lot of user modules to do most of the work that's | normally in the monolithic kernel, so it'll probably end up | being about the same amount of code that needs to be in cash | first, not in cash. The differences is really come down to | just call boundaries as I understand it | mcv wrote: | > 1. Linux is too big. | | Too big for what exactly? Because it's clearly not too big to | run. It's been running very well for decades, and Hurd has | not. | | I've known for decades that on paper, microkernels and Hurd | should be superior, and I've wondered for nearly as long why | then they aren't taking off. If they're so good, they should | at least be thriving in some niche where that performance | matters, shouldn't they? | | And now I hear that even after 3 whopping decades, it still | doesn't run on real hardware! Entire operating systems have | been born and died in that time. | | What is the problem that's making Hurd still not work? | | > 2. It will take time to write a ton of device drivers, | simply because most of them aren't written yet. To that | extent, Linux most definitely has an incumbent advantage. | | But why is Linux the incumbent when Mach and Hurd were | conceived 3 years earlier? | | > But that isn't to say that microkernels also have an | advantage here. | | I would love to believe that, but then why has that advantage | not paid off for 30 years? | | > With all that being said, I definitely don't think Hurd is | the future. A Why not? And is this why nobody has been | writing those drivers? | | I don't mean to be shitting on Hurd, but it's been really | promising for so long, and not delivering, that it's starting | to feel like fusion power: always in the future, nearly | there, but never quite here. | darksaints wrote: | > I've known for decades that on paper, microkernels and | Hurd should be superior, and I've wondered for nearly as | long why then they aren't taking off. If they're so good, | they should at least be thriving in some niche where that | performance matters, shouldn't they? | | They (microkernels) thrive in Automobiles, Aircraft, | Spacecraft, Security systems (such as hardware encryption | devices), Defense Munitions, Embedded Devices, Maritime | devices, radar systems, and many other areas. The common | thread here is that they are not just resource constrained | and performance sensitive, they are also typically very | specialized and mostly write-from-scratch. | | That last part is the key to understanding why general | purpose operating systems mostly use legacy kernels that | are decades old. We want to preserve as much as we can from | what came before. | | > But why is Linux the incumbent when Mach and Hurd were | conceived 3 years earlier? | | Because Linux worked earlier. Nobody in the GNU ecosystem | knew how to develop a microkernel that worked, let alone | drivers for a constantly changing target OS. | | > I would love to believe that, but then why has that | advantage not paid off for 30 years? | | Again, they have. Almost all RTOSes are microkernels, and | the most popular embedded OSes are microkernels. When | you're writing something from scratch, it is always going | to be easier with a microkernel. Don't mistake Hurd with | the concept of a microkernel. Hurd sucks because it has | always sucked. It's bloated, poorly documented, relies on a | very janky GNU ecosystem that has already mostly adapted to | Linux out of practicality concerns, and there is no | ecosystem tooling around it. | | But the advantages of microkernels _in general_ are very | real. The best one out there in terms of maturity is QNX, | and while practically every modern car on the planet runs | on it, it 's licensing pretty much precludes it from being | used as a general purpose OS. | bityard wrote: | > Most microkernels are small enough to fit in the L3 cache | | Is that important, though? As soon as you start processing | any meaningful amount of data, won't your OS be evicted from | the cache anyway? | | > unless you are compiling your kernel specifically for your | machine, your kernel is going to be a bloated compromise of | the set of drivers that your system is most likely to see. | That means that you will have dozens, if not hundreds of | drivers compiled into your kernel which will never need to be | used. | | I'm not sure this is true. Device drivers are typically | compiled as modules, which are only loaded into the kernel if | needed. Common wisdom among Linux power users for the last | decade or so has been that compiling your own kernel for any | tangible performance/memory improvement is basically futile. | | Are there exceptions to this that you are aware of, and can | elaborate on? | darksaints wrote: | > Is that important, though? As soon as you start | processing any meaningful amount of data, won't your OS be | evicted from the cache anyway? | | Writable caches yes. However many (most?) modern processors | also contain a code-only cache which can be used for OS | caching without any need to flush. The raspberry pi 4 can | fit many modern microkernels in their tiny immutable cache, | so it gains a massive caching benefit by being small. | | > Are there exceptions to this that you are aware of, and | can elaborate on? | | You're probably right, I was unaware of the extent of the | kernel module usage. Regardless, the smallest common kernel | in use that I know about is for alpine Linux and that is | still at least 100MB, which is dramatically larger than the | <1MB microkernels that are typically available. | sterlind wrote: | how different is the Hurd driver model from Linux's? is it a | straightforward but tedious process to port drivers, or does | it require major rethinking? I wonder if machine translation | is possible, or even LLM-assisted automatic ports as a | starting point. disastrous to think about, as careful as | kernel code has to be, but maybe with enough static analysis | and guardrails it's doable? | darksaints wrote: | It is quite different, and porting drivers from Linux would | be very hard to do en masse. However, other open source | kernels have drivers that are much easier to port. NetBSD, | for example, has a whole host of drivers that can and are | used almost unchanged in other microkernels. | irdc wrote: | > Most microkernels are small enough to fit in the L3 cache | | Have you looked at the size of L3 caches lately? You could | fit a whole 'nother OS in one of those. | darksaints wrote: | Okay, so there are a handful of massive $4k+ chips that can | fit the Linux kernel in the L3 cache. Meanwhile, you could | fit 5 copies of the SEL4 kernel in the L2 cache of a | raspberry pi. | kelnos wrote: | Which is fine for a program that does essentially | nothing. But once a program wants to talk to the network | or disk, or display something on the screen, it needs | more system services that need to come from somewhere. | Can you fit all of those in the caches too? | marcosdumay wrote: | > Isn't that a large part of the source tree? | | For Linux, yes. For a microkernel, those are different trees of | userspace software. | | Not to say, Hurd itself is userspace software, that runs over a | microkernel (included on the distribution). | uxp8u61q wrote: | > For Linux, yes. For a microkernel, those are different | trees of userspace software. | | If that were the argument, then the comparison with Linux | would be meaningless. Hurd is much bigger than freertos, | should everyone switch? | muixoozie wrote: | Can get a sense of how long it takes to compile here | https://openbenchmarking.org/test/pts/build-linux-kernel | surprisingly fast with modern processors. | taeric wrote: | My memory was spending as much time learning on the different | options as anything else. Though, I think most of my memory | was all of the config for FreeBSD. Compiling Firefox was the | beast that I remember. Only thing I still compile is Emacs, | and that is silly fast on modern computers. | | Really, the speed of compilation is such that I understand | why so many source distribution methods are as popular as | they are now. Feels funny to have come to that point so | effectively. | genewitch wrote: | I used to compile kernels on a 2012 HP quad-CPU, 80 thread | machine. It would finish in a minute and a half or so. I got | a ryzen 5850 to replace it, it also takes a minute and a half | or so - but uses 1/4th the power at the wall. 32 threads vs | 80, 250W vs 1000+W | stusmall wrote: | I love how much things have progressed. I remember I used to | have to set up a clean build to run while I was at school so | it could be done when I got back. Now it is done within | seconds. | gattilorenz wrote: | Heh, not a particulary in-depth article in any way, to be | honest. | | "the GNU/Hurd is Unix(POSIX)-compatible, so most things | work"... well that's to be said for MacOS, QNX, etc. In | practice, where is the "65% of the Debian archive can be built | for the Hurd" figure coming from? I'm sure it can be built in | theory, how much does it require source modifications? | rekado wrote: | Here's a list of failing packages with reasons for failure: | https://people.debian.org/~sthibault/failed_packages.txt | | And here's the overview on how many packages have | successfully been built vs how many have failed: https://buil | dd.debian.org/status/architecture.php?a=hurd-i38... | | The number of packages that can be built _have_ been built; | it is not a theoretical number. | gattilorenz wrote: | thanks for the pointer, it's actually quite interesting. | Apparently even (most of) X11 should work then? | molticrystal wrote: | LoC is a metric for manageability of the code base and compile | time, and you might need other metrics to judge those two | factors. | | For real world use wheen comparing OS you would generally go | for other metrics like resource usage, multi threading | bottlenecks, throughput of network packet and their processing, | context switches(or equivalent) per second, and dozens of other | things that would be paragraphs in length before you get to | LoC. | janneke wrote: | Rumpdisk on a GNU Guix childhurd: | https://toot.aquilenet.fr/@civodul/110848429561223704 | | just one simple `guix system reconfigure' away! | | Guix Hurd on my thinkpad x60: | https://todon.nl/@janneke/110451493405777898 | | yeah, that's real (old, but not ancient) iron. | | Enjoy! | [deleted] | davidw wrote: | An idea from the 'Crossing the Chasm' book is to find a niche | where whatever characteristics of the product you're working on | can win. You are not going to win going head to head with the | entrenched competitors. | | What does that look like for Hurd? | harry8 wrote: | >You are not going to win going head to head with the | entrenched competitors. | | Linux, (not big and professional like gnu), is not going to win | but could find a niche somewhere. Hm, ok. | davidw wrote: | Yep, it found a "niche": people could actually hack on it and | get it to do stuff, rather than waiting around for the other | thing, which wasn't even really a thing yet anyway. | | Linux has a large, large established user base at this point. | If you want to attract people to a different platform, you | need a compelling reason. | | For people just looking to look/experiment/learn/play, | 'something new' that's more easily understood because it's | small might be enough, but that's unlikely to be a large user | base. | mjhay wrote: | GNU/Hurd is legendary, but not for good reasons. | Tao3300 wrote: | I love the Loch Ness monster comparison at the beginning. | Brings it back to earth from the clickbaity headline fast. | Thrownhurd wrote: | Mayhaps egregious? | AceJohnny2 wrote: | i.e. "notorious" | hinkley wrote: | It's in-famous. | | It's gone beyond famous and now it's IN... famous. ___________________________________________________________________ (page generated 2023-08-07 23:00 UTC)