[HN Gopher] Systemd: The Good Parts ___________________________________________________________________ Systemd: The Good Parts Author : pcr910303 Score : 242 points Date : 2021-05-16 18:42 UTC (4 hours ago) (HTM) web link (christine.website) (TXT) w3m dump (christine.website) | theteapot wrote: | Cool .---------. .---------. | SystemD | |------>| Service | '---------' '---------' | Not Cool .---------. .---------. | SystemD | |<------| Service | '---------' '---------' | dale_glass wrote: | I don't get it? | mwvr wrote: | A furry named Xe promoting systemd, just another day in the Linux | twilight zone. | ohazi wrote: | I actually like using systemd. I took the time to learn how to | use it, and as a desktop Linux user, it's honestly quite nice. | | But the one thing that still really pisses me off about systemd- | the-project was the fact that they ate udev-the-project. In my | view, that decision was unnecessary and was done for purely anti- | competitive reasons. | | If you're not familiar, udev is basically the user-facing device | manager for Linux. It's what allows you to easily configure rules | and permissions for all the USB devices that you plug into your | computer. These devices still need kernel drivers, but udev is | how you tell your system "Please let user ohazi use this device, | and also give it a convenient name like /dev/someDevice0" | | By devouring the udev project, the systemd maintainers have | guaranteed that dealing with USB devices on non-systemd systems | was going to be a giant pain. | | Then came the forks -- Gentoo maintains eudev, which is a | systemd-free fork of udev. But really, this shouldn't be a fork. | Udev should be independent and available on all systems. If | systemd wants to do something special with devices, they should | use udev APIs like everybody else. In my view, this is what | finally allowed systemd to win the init war, despite all the | protest. Ideological arguments about how best to start daemons is | one thing, but you simply can't use a modern system without a | sane approach to USB. | | Edit: It seems that some of my concerns were overblown, e.g. you | apparently can run udev without systemd as pid 1. Udev has a | build-time dependency on systemd, but not a run-time dependency. | The eudev fork removes that build-time dependency, and (maybe?) | also papers over some other inconsistencies. I'm pleased that | this is the case. I'm annoyed that the discourse around | systemd/udev has been muddy enough to lead me to incorrect | conclusions (and I'm aware that the original version of this | comment likely added fuel to this particular fire). Oh well... | live and learn. | egberts1 wrote: | systemd still hasn't properly replaced ISC dhclient so I dumped | the entire systemd and went both Gentoo and Denuvan (systemd- | free Debian). | edoceo wrote: | I love Gentoo and really like they chose to make eudev and a | elogind so I can continue to use OpenRC, which has been working | great for me since forever. | ohazi wrote: | Don't get me wrong -- I too am extremely grateful to the | Gentoo devs for maintaining eudev. It's also the obvious | choice for highly memory constrained embedded Linux systems | that can't fit systemd. I just think it's absolutely bonkers | that it has to be a fork... eudev should really just be udev. | unilynx wrote: | Wasn't it because Kay Sievers was a major developer on both | udev and systemd? | | I can image someone thinking then "I'm implementing everything | twice, both for udev and systemd. I'll just merge them" | zxzax wrote: | According to Lennart that's correct, the reason it was done | was to reduce the amount of duplicate code: | https://lwn.net/Articles/490441/ | ohazi wrote: | I realize that I don't really have any right to complain | about this. I don't contribute to either project, and I | think that maintainers should feel free to do whatever they | think is best for their projects, including things that | help reduce burdensome things that they find annoying. | | However, as a heavy user of both projects, I see what those | decisions have led to in practice, and have opinions about | what this might mean for the community in the medium/long | term. The reality on the ground today is that you either | run systemd+udev, or you run OpenRC/something else + eudev. | And having observed other projects that went down the "we | forked, but merge regularly to try and keep the forks in | sync" rabbit hole (e.g. ffmpeg and libav), this almost | always ends badly. | | The goal may be to keep the two separate projects | compatible, but inevitably environment differences, bugs, | refactors, etc. cause the two projects to diverge. This | sucks for everybody, but it sucks _a lot more_ for people | using the less popular fork, and this can eventually kill | the fork. I worry about eudev long-term. | zxzax wrote: | So that really seems to be the core of the problem to me: | nobody really wants to seriously maintain udev as an | individual project. It's not a fun or glamorous project | to work on, it's boring, nobody will notice it unless it | breaks, and nobody really seems to care that it depends | on libsystemd. For most people this is just an | implementation detail. For an embedded distro I can't see | why you would use udevd or eudev, you probably want a | much simpler device manager without the giant hwdb. | cyberdelica wrote: | That's what code libraries were intended for. | zxzax wrote: | Not sure why you're saying that -- moving it to a library | is basically what happened. The libudev code got moved to | libsystemd, and now libudev is a wrapper around that. | cyberdelica wrote: | > Not sure why you're saying that -- moving it to a | library is basically what happened. The libudev code got | moved to libsystemd, and now libudev is a wrapper around | that. | | Oh, so now one has to include "libsystemd" if one wants | to use "udev" - is that what you're writing? Hence the | original comment: | | > By devouring the udev project, the systemd maintainers | have guaranteed that dealing with USB devices on non- | systemd systems was going to be a giant pain. | zxzax wrote: | I don't understand, is this not what you were asking for | initially? libsystemd.so is just a shared library, it's | not any more of a giant pain than any other library. | cyberdelica wrote: | Why not just include libudev, in systemd instead? | zxzax wrote: | Because that would not reduce code duplication. I don't | know if you have ever worked on any of the low-level | Linux libraries written in C, but the amount of | unnecessary code duplication across them is awful. The | standard library features provided by glibc are extremely | inadequate for modern applications, every non-trivial C | project I've seen starts to include their own private | implementations of various C++ things like hashmaps, | binary trees, dynamically sized strings, unicode support, | async event loops, etc. | | For utilities that have to interact with low level kernel | APIs it's even worse, every library seems to have to | reimplement their own parsing of various other random | things like netlink, or in the various pseudo filesystems | like sysfs, procfs, cgroupfs, etc etc, the situation is | really way out of hand. I don't know how to solve this in | a reasonable manner beyond what systemd is already doing. | Yes people will complain that they have a systemd | dependency now but what else can you do? This is the | exact reason the BSDs update the kernel, libc, init and | core utils in tandem and consider a lot of the kernel API | to be private, Linux was just slow to catch on in that | regard. | cyberdelica wrote: | > Because that would not reduce code duplication. | | I don't want to call you out for making a disingenuous | argument, but it was either subsumed by systemd for code | duplication reasons, or it wasn't?! | | Duplicate code, could be refactored out to a shared | library, that could then be incorporated in both udev, | and systemd. That would mean, anyone looking to | incorporate udev into a system, could do so without | depending on libsystemd. | | Instead, it would seem udev code, has been subsumed by | libsystemd (in your own words) - which would appear to | the sceptical eye, as a power play on the part of red hat | - to force other distributions into using libsystemd, | which would logically end with them also using systemd | itself. | | > I don't know if you have ever worked on any of the low- | level Linux libraries written in C [...] | | Yes, I have. | jhasse wrote: | > Duplicate code, could be refactored out to a shared | library, that could then be incorporated in both udev, | and systemd. | | That's basically what has been done. The library was | named libsystemd. | | You seem to have a problem with the name "libsystemd". | cyberdelica wrote: | How do you compile udev, without "libsystemd", or without | glibc? | | > You seem to have a problem with the name "libsystemd". | | Ah, the ad hominem. Do you have a problem with the Unix | Philosophy? | cwyers wrote: | No, you just categorically don't understand. libsystemd | is not systemd the init system, it doesn't require | systemd to be running, it is simply the name of "the | shared library that everything developed under the | systemd project uses for common code." Now, you can argue | until the cows come home that you'd be happier if udev, | systemd, and libredhathatesyou (or some other name for | libsystemd that doesn't include systemd in it) were all | in different Git repos, but since you can run udev | without a dependency on systemd running on the system, | that seems like a pretty low-stakes dispute. | cyberdelica wrote: | > No, you just categorically don't understand. | | Of course I understand, as do the rest of the people not | employed by red hat upvoting me. | | Time and time again, I've seen systemd advocates making | slippery, disingenuous, and outright false arguments. | When they're called out on it, the goalposts magically | move, a rotation of usernames appear to downvote and | brigade which can be ascertained through downvote timing | correlation. When they can't win an argument through | facts, then they make bogus arguments that one doesn't | get it, or some such nonsense - or claim they're a red | hat conspiracy monger. | | Seeing it over and and over again is lame, and played | out. | | Now, why don't you answer the question - if udev was | subsumed by "libsystemd" as is claimed due to "code | duplication" - then why did they not just include | "libudev" as a dependency for "libsystemd"? | | Of course, the question will never be answered, as it'll | reveal the truth. | zxzax wrote: | The code savings from including udev in systemd turned | out to be greater, as there was more re-usable code | already written in systemd. Please be the better person | and don't revive this flamewar, it's not helpful, let's | stick to the technical facts and work together to find | the answers we seek -- for example you can look at the | git logs to see all the shared functionality and the code | that was changed around: | https://github.com/systemd/systemd/tree/main/src/udev | zxzax wrote: | The issue is not with the actual udev device event logic | itself, but with all the other bits I talked about. Those | are the things that would need to be duplicated. | | >to force other distributions into using libsystemd, | which would logically end with them also using systemd | itself. | | As I said elsewhere, libsystemd is just a library with | some generic functions provided for convenience. This is | like saying that installing python libraries on your | system logically means that the PSF is trying to take | over your system and forcibly rewrite everything in | python, it doesn't make any sense. | mixmastamyk wrote: | I'd expect the opposite, if someone said they moved | something to a library. | TacticalCoder wrote: | > By devouring the udev project... | | That's the main issue (not udev specifically but the overall | philosophy of trying to make systemd mandatory for as many | things as possible). | | Any distro adopting systemd is basically forever giving up any | hope of moving away from systemd in the future. | | I'm still on Debian (Debian user since "Buzz") but I already | tried Devuan (Debian fork without systemd) once and may switch | to it at some point. | | I simply don't like the overall mindset of those pushing | systemd. | jamal-kumar wrote: | I had a hard time trying Devuan because all the package | mirrors are really far away from me unfortunately, but I | don't have any issues at all just removing it from whenever I | am working with debian servers. It all used to work through | some shell script called systemd-shim, I'm not entirely sure | now though, maybe someone else has an idea off hand | | https://sysdfree.wordpress.com/2020/07/02/319/ | nine_k wrote: | I cannot shake off the feeling that the core developers | standing behind systemd, pulseaudio, Gnome desktop spend a lot | of time using macOS, and like it better than Linux. | | This would explain why they imitate or copy many of its | features (and misfeatures), and seemingly tend to replace the | Unix ways with entirely different approaches. Effectively | forcing things on users, as it was with pulseaudio and systemd, | is also a very macOS thing to do. | | I'm not opposed to replacing Unix with different approaches, as | long as they keep playing on its key strengths, and do not | remove modularity and composability. That is, Plan 9 is fine | with me. Much of what Red Hat does, sadly, no. | zxzax wrote: | If plan 9 is what you want, I would encourage you to use | 9front. I can't see why you would use Linux if you didn't | like Red Hat, they have been a key contributor in Linux land | for a long time, employing many long-time kernel developers. | zxzax wrote: | >that decision [...] was done for purely anti-competitive | reasons. | | This is not correct, I don't know where you got this idea. The | systemd-udevd daemon still runs without systemd. The code has | moved into the systemd repository, and it has a build | dependency on libsystemd, but otherwise it has no runtime | dependency; your distro should be able to package it separately | if it wants. | ohazi wrote: | Can you point to a distro that does this vs. using eudev? I | mean, you might be right, but I haven't found any, and I | think the reason for this is that there are subtler inter- | dependencies that require additional workarounds: | | > > Also, AFAIR the connection between systemd and udev | doesn't really go deeper than the fact that they're sharing | the same upstream tarball. It is still possible to build and | use udev without systemd | | > But that's not really the case operationally, and it's why | eudev was forked away from it for Gentoo. | | https://lwn.net/Articles/778547/ | zxzax wrote: | The udevd package in gentoo does exactly this, which is | supported as an alternative to eudev: | https://packages.gentoo.org/packages/sys-fs/udev | | I believe the reason for eudev is to avoid some of those | other unwanted changes from upstream udev that broke udev | scripts. Which to me is a fine technical justification, but | not really related to another udev implementation having a | build dependency on libsystemd. The udevd package seems | maintained so I really don't understand what operational | issues that comment is getting at. | ohazi wrote: | Interesting. I didn't realize this was a supported | option, but the Gentoo wiki also seems to agree with you | [1], so I may need to walk back some of my frustration | around this. | | I still have concerns that having a shared codebase makes | it easier for interdependencies to materialize later, but | if they've managed to keep it as decoupled as is claimed | here, then I'm pleasantly surprised, and also annoyed | that the communication around this issue has been | confusing enough to lead me to an apparently incorrect | conclusion. | | [1] https://wiki.gentoo.org/wiki/Gentoo_Without_systemd#T | he_udev... | Conan_Kudo wrote: | Most of the systemd components actually _work_ without | systemd as the init. I 've used udev and nspawn without | systemd-init. | | As long as you don't have a problem with libsystemd | existing, most things work fine without systemd-init. | Mainly systemd-journal and systemd-oomd both require | systemd-init. | dralley wrote: | >and it has a build dependency on libsystemd | | That does seem kind of unfortunate for anyone that wants to | bootstrap a systemd-less distibution. | zxzax wrote: | Libsystemd is just a shared library with some common code | used by various systemd daemons (an epoll-based event loop, | the dbus protocol implementation, uuid generation, parsers | for the binary logs, etc). I can understand why people | don't want to run systemd as pid1, but having some other | external daemon depend on this library doesn't make it any | harder to bootstrap that type of distribution. | salawat wrote: | Why take the risk of a maintainer saying fuckit and | pulling in the rest of systemd since you're already using | their utils? | | It's just one of those things where it's better to stay | away. | zxzax wrote: | That doesn't really make any sense, libsystemd is just a | library with some fairly generic functionality. | devenblake wrote: | I have a feeling if this was as simple as you say, distro | maintainers would be doing it. | JoshTriplett wrote: | They do. The few distributions that still support non- | systemd init systems _do_ split out udev: | https://packages.debian.org/sid/udev | cbmuser wrote: | Which is not that uncommon, FWIW. | | binutils and gdb were also moved into a common repository and | gcc merged some repositories as well. | makomk wrote: | In practice I think systemd-udevd probably still runs without | systemd, but as I understand it that's not an officially | supported solution and the systemd developers have said they | reserve the right to make changes that break it at any time. | (The main reason why they haven't yet is most likely that | their attempts to integrate dbus into the kernel and systemd | more tightly were rejected by the kernel devs; if I remember | rightly that was expected to be the point at which you could | not use it without systemd.) | | Using the kernel interfaces that udev relies on without using | udev itself is also unsupported and the developers consider | it within their rights to break those as well. | zxzax wrote: | It has been supported for at least the last 9 years now, I | don't see why they would have any reason to break it. | | >The main reason why they haven't yet is most likely that | their attempts to integrate dbus into the kernel and | systemd more tightly were rejected by the kernel devs; if I | remember rightly that was expected to be the point at which | you could not use it without systemd. | | That seems quite dubious, udev has never depended on dbus, | and there really would be no reason for it to ever do that. | | >Using the kernel interfaces that udev relies on without | using udev itself is also unsupported and the developers | consider it within their rights to break those as well. | | This I know is true, the netlink stuff varies really wildly | from driver to driver. The alternative would be to put | udevd and the other userspace bits into the kernel, which I | really doubt anyone wants to do. | mjw1007 wrote: | What particularly grated was that they managed to introduce | bugs into udev and then say they didn't have the "expertise or | understanding" to fix them: | https://github.com/systemd/systemd/issues/7157#issuecomment-... | | (That bug was causing udev to give multiple disks the same name | in /dev/disk/by-path, which I think is pretty much core udev | functionality, given that udev was intended to supersede | devfs.) | 867-5309 wrote: | the systemd stars-to-issues ratio is worrying to say the | least | TheDong wrote: | What do you imagine that ratio shows? Different projects | use issues differently, so that number isn't super | comparable... | | And I've also noticed that low-level systems projects | generally attract relatively few stars (i.e. a javascript | library or go library will have more stars than a well used | c library or systemd). My theory on that is that the people | involved in projects like systemd, gnu, etc, see github as | mostly just a git repository host, and don't bother with | stars etc. | | I guess from my perspective, I put zero weight into the | stars on a project, and issue count only matters in the | context of how the project deals with issues. What do you | see as "worrying" in there? Do you see more value in those | numbers than me? | Iolaum wrote: | lolz - i 'll be keeping an eye on that from now on :) (on | some of my fav projects) | zxzax wrote: | I'm just curious, how would you suggest they debug and | maintain support for that hardware, without even owning that | particular device? It seems like anything they do would just | be a wild guess as to whether it actually works or not. In my | experience with this specialized datacenter hardware, if | people want any of this stuff fixed (in userspace or in the | kernel) it usually falls on the hardware vendor to hire some | developers to fix it and upstream it, or at the very least | donate some hardware to the upstream project so they can test | properly. | mjw1007 wrote: | If they felt they needed information from, say, the Linux | scsi maintainers then they could have gone out and asked | for it. If they thought that in order for udev to do a | proper job with these devices they needed the sg3-utils | maintainers to write something new, they could have gone | and asked for that. As it was they just stuck a "needs-new- | home" label on the bug and ignored it. | | I think useful action would have been more likely if udev | had still been maintained by a team who believed they'd | taken on the responsibility to make /dev/disk/by-path work, | rather than maintained as a small part of a project whose | maintainers are mostly interested in other things. | | (Also, IIRC from what I had to do to work around it, they | could have fixed that bug by reverting the change that | introduced it. It wasn't a case of existing udev code not | working with new hardware; it was caused by someone making | changes without understanding the consequences.) | zxzax wrote: | That's what I mean though, all of that seems to be just | wild guessing as to what will actually work and sending | random emails out to random people who may or may not be | able to fix it (Disclaimer: I don't know anything about | this specific bug or the specific hardware). Wouldn't | someone who actually owns the device and knows who to | contact on the kernel side be a better person to lead the | effort on that? | mjw1007 wrote: | This was a bug in udev. | | The right people to lead the effort on fixing bugs in | udev are the udev maintainers. | | If people did not wish to have the responsibility for | leading the effort to fix bugs in udev, they had the | option of not taking over the maintenance of udev. | | I agree with ohazi, above, that the world would be a | better place if the systemd people had availed themselves | of that option. | | (To be fair, it looks like this issue was fixed a few | months later by someone who knew what they were doing. So | the main lesson here may be something more like | "@poettering should not be triaging udev bugs".) | zxzax wrote: | I see people saying this kind of thing often in open | source and I think you are missing the point here: who is | going to fix this bug? You can say systemd developers | shouldn't triage it, and then you're left with nobody to | triage the bug at all. So what will you do? The point is, | they don't own the hardware, they can't fix the bug. You | have to find somebody who actually does own the hardware | who knows what is going on, which seems to be exactly | what happened. But if you get unlucky and if zero of | those people are contributing to udev or any of its | forks, then the bug probably won't ever get fixed. | 3np wrote: | I think their point is: with smaller project scope, this | bug wouldn't have existed in the first place. | zxzax wrote: | That is throwing the baby out with the bathwater though, | people who want large scale projects will continue to | make projects with large scopes. | unanswered wrote: | This wasn't a hardware bug and it didn't materialize only | when some particular hardware was plugged in. Were they | just supposed to open a phone book, pick a random hard | drive manufacturer, and say "hey this is your problem | now"? | zxzax wrote: | No, it seems the specific hardware causing the issue was | mentioned in the bug report. | mjw1007 wrote: | For what it's worth, I didn't mean that the systemd | people shouldn't triage the bug. | | I meant that I suspect @poettering was mistaken when he | said the current maintainers didn't collectively have | "the expertise and understanding to maintain this | properly." | | As far as I can make out, there was nothing particularly | hardware-specific about the bug. The bug was that udev | was assuming that there could be no more than one disk | per SATA host node, which turns out not to be the case. | | `udevadm info` output would have been enough to see what | was going on, which I'm sure the reporter would have | happily supplied. | zxzax wrote: | From what you were saying earlier, from the perspective | of the bug triager, it sounds like there were multiple | areas that maybe this could have been fixed in, not | necessarily udev. So still that comes back to: the | correct people to triage that bug really would have been | the distro maintainers, who have total visibility over | the whole system and who are better equipped to pinpoint | where the bug should be fixed. And then from there they | can pass it off to a systemd person or an OEM person, or | both, or whatever (Again disclaimer, I don't know | anything about this particular SATA hardware or whether | this is normal behavior for a host node, this is just my | experience from trying to triage this type of bug). | Valmar wrote: | > But the one thing that still really pisses me off about | systemd-the-project was the fact that they ate udev-the- | project. In my view, that decision was unnecessary and was done | for purely anti-competitive reasons. | | The developers of udev at the time agreed to merge to the | project into systemd, as it made a lot of sense to them. | | There was no "devouring", no "anti-competitive" bullshit. | dijit wrote: | Copying my reply from Lobste.rs because there's more people here | who can pick me apart: | | _FD: I'm not a large supporter of SystemD after using it when it | was first introduced in Fedora and watching it take over the | ecosystem, I do use it and as a sysadmin/SRE type I'm heavily | versed in it and how it functions, so, everything with a pinch of | salt._ | | I gave the video a good shot, half the time is spent going over | some systemd-ecosystem basics; | | The author then makes the case that Alpine is a less attractive | development/production-deployment platform due to lack of this | ecosystem (though they make effort to avoid directly saying that | systemd would be a good solution itself.) | | I couldn't disagree more honestly. I love the concept of Alpine: | as small as possible and no smaller. | | The attention to minimise dependencies makes it the distro of | choice in containers; I've even considered (strongly) running it | on some machines where systemd would have been a lot of needless | complexity. | | I think deviating from this path for alpine is a grave error, | there will always be a place for a niche system that is small; | especially when it's commonly paired with some other log | managment/service orchestrator like Alpine is. | | Additionally, it gives people like me an option to run without | init systems that could be considered heavy and intrusive with | heavy dependency chains. | | So, kindly, Alpine: please don't. | AshamedCaptain wrote: | I have a system where systemd decides, about half the times I | boot it, to immediately unmount all of the filesystems listed in | fstab literally right after having finished mounted them as part | of the regular boot process. All of this with other daemons | starting in parallel. | | It will fail to unmount most of the filesystems, since they are | of course busy, but often it will succeed in unmounting /var, | /tmp, /home and others. Then it will continue starting on further | services as if nothing happened and even proceed to gdm. But of | course without /home I can't even login. | | No idea how to even start debugging it. | sigg3 wrote: | I'd start by looking at the systemd unit files in | /etc/systemd/system. Beware of service wants. | | I had straight up errors in a network unit file that caused | weird behavior (IP address disappearing). This was a long time | ago though. Sometimes moving a unit from one target (graphical) | to an earlier (multiuser) solves the issue. | | You might also have "defaults" in /usr/lib/systemd/ to compare | with. | | systemctl and friends are really good tools for troubleshooting | but takes a little while to get used to. Duck duck go is your | friend. | | Finally, some volume technologies like VDO requires a | x-systemd.requires statement in fstab. | jraph wrote: | I would run systemd-analyze critical-chain and then systemd- | analyze blame to see if there is a service that could do this. | And then I would look at the detailed boot logs. If nothing can | be found, I'd look for some way to increase the log level. | | If you are not curious, reinstalling everything could be quite | fast too, depending on what you have installed. | AshamedCaptain wrote: | The logs are mostly useless and random. There is nothing that | consistently appears between the point where it finishes | mounting (lots of "Mouting...", then "Reached target Local | File Systems") and where it starts unmounting ("Stopped | target Local File Systems", then lots of "Unmouting ..."). | Sometimes these two lines are less than one second apart. But | I haven't found any message which is consistently between | these two points that would point to a suspect. | | And this is not a problem that happens on every boot, so it | doesn't show up on analysis... not to mention that most of | the analysis is designed on trying to reduce boottime, which | obviously is no help (thought it is rather fast already). | | It is rather easy to guess why a service is starting, but not | why it decides to stop 'orderly', much less why a .target | decides to stop... | jraph wrote: | Most certainly you can only analyze things on boots where | the problem appears, unless logs are persisted across | boots. | | Indeed systemd-analyze focuses on boot times, though I find | it is useful to get a list of services that are run. | | If you are adventurous you could probably replace the | umount (resp. the systemd-umount) binary by a script that | runs umount (resp. systemd-umount), and also prints the | process tree (and its PID) in some file so you can get | insight on what ran it. | teddyfrozevelt wrote: | Systemd generates mount units for the filesystems listed in | fstab. Maybe check those with systemctl status? (Like | `systemctl status -- -.mount` for my root partition) | AshamedCaptain wrote: | Sadly no. The problem is that the entire local-fs.target is | stopped 'cleanly' but with no clear reason. i.e. | $ systemctl status local-fs.target Active: inactive | (dead) since [...] 21:12:56 ... systemd[1]: Reached | target Local File Systems. 21:12:57 ... systemd[1]: | Stopped target Local File Systems. | | Yes, the log just shows it stopping one second after being | started, nothing happening inbetween, and no reason given for | it being stopped. | | Incredibly enough, local-fs.target is a dependency for | graphical.target, so the system should not have continued | booting. But not only it has continued booting, systemd even | thinks that it finished booting all-OK. State is "running" | (not "degraded" as it would be if any service/mount failed), | with 0 failed services. Even though both graphical.target and | local-fs.target are 'dead'. | dijit wrote: | Candidly: that command looks like the least discoverable | command I've seen in the last year. | heavyset_go wrote: | The command systemctl status *.mount | | Just shows the status of all mount units. | | It's the same command you'd use to check any other systemd | unit, for example, if you want to check Docker's status: | systemctl status docker.service | | Seems consistent and discoverable as long as you have a | surface understanding of systemd units. | teddyfrozevelt wrote: | Fair. I think it's because systemd lets you use slashes in | unit names, but you can't represent those as files so you | have to use a dash instead. This is the worst for the root | filesystem mounted at / since that's the entire name. In | systemd's defense, it warns you when you try systemctl | status the normal way. Hint: to specify | units starting with a dash, use "--": systemctl | [OPTIONS...] COMMAND -- -.mount ... | | I found my mount names by running systemctl list-units, so | it's not something I had to look up. | azornathogron wrote: | It is at least built in a logical way and once you | understand that logic then _related_ commands will be | discoverable. | | `systemctl` is the systemd command line tool for inspecting | and manipulating services/units. `systemctl status` is the | command for showing the status of a service or unit. | | The `--` part is not specific to systemd at all, it's used | in _many_ commands to separate flags from positional | arguments. This lets you use positional arguments that | might happen to start with a dash without them being | interpreted as flags. [1]. | | systemd manages mounts, they're named according to the | mount point, with a transformation to turn it into a | filename (mostly converting slashes to dashes, plus a | .mount suffix). Hence, the root mount ('/') is named | '-.mount'. | | [1] See, e.g., | https://unix.stackexchange.com/questions/11376/what-does- | dou... | unilynx wrote: | The annoying part is that now that '-' stands for '/', | you need to use '\x2d' for '-'. Which is pretty annoying | when manipulating mount names during the shell. | | I wish they had picked ':' for '/' instead of '-'. That | one would be pretty uncommon to find in a mount path, and | has a bit of precedent on OSX. | | (Who would ever mount anything on a path with a colon | anyway? That would be as silly as using slashes instead | of dashes for options) | salawat wrote: | C:\ much? | heavyset_go wrote: | If you use a wildcard operator, you can list all the mount | statuses: systemctl status *.mount | viraptor wrote: | You can check in journal which service startup does the | unmounting align with. Maybe generate a bootchart | http://manpages.ubuntu.com/manpages/bionic/man1/systemd-boot... | | You could also replace umount with a wrapper that logs the | parent process tree on execution. | | And of course you can open an issue on GitHub. | pengaru wrote: | > And of course you can open an issue on GitHub. | | Only do this if you've reproduced what seems to be a bug in | one of the latest two releases, otherwise contact your | distro. | | The systemd github issue tracker is _not_ a support forum, | you 'll find this stated in the issue submission form. | kristjansson wrote: | Fair advice, and they're probably right to redirect most | users that would file a bug with them. | | But on a normative level, perhaps a project that doesn't | want to support users shouldn't have subsumed a huge swath | of user land things? | stevenhuang wrote: | If that happens again I'd try opening a virtual terminal, login | as root and check logs. | Twirrim wrote: | > If that happens again I'd try opening a virtual terminal, | login as root and check logs. | | If the system has unmounted all the drives, there won't _be_ | any logs. This happens often enough with systemd / journald | to be a big source of annoyance. | | The only option you get is to tell systemd to boot in debug | mode _but_ that fundamentally alters the timing and behaviour | of so much that it often stops the weird behaviour from ever | happening (similar fun can be had with `dracut` where | enabling debug tends to stop any race condition from ever | happening) | AshamedCaptain wrote: | Note systemd does dump its own log to the kernel ring | buffer until journald starts (the last message I see on the | kernel ring buffer is Started Journal Service). It's no | help though when (as in my case) journald starts but then | has /var unmounted below its feet. I can write a script | that keeps /var busy though, preventing it from being | unmounted. | isodude wrote: | We had that kind of problem because we relied on /dev/disks/by- | label | heavyset_go wrote: | Can you tell us more about this? How did you determine that | that was the problem? What was your solution? | | I ask because I use /dev/disks/by-label and have never had | this problem, and if I ever had to experience it, it would | drive me insane. | sigg3 wrote: | I've solved a similarly sounding issue by using the UUID. | Get it with blkid, then use UUID="<uuid goes here>" in | fstab instead of LABEL="<label>". | | But this was a long time ago, on a much older systemd (like | Fedora 17 or 18). | dharmab wrote: | This is a common error even back in the Sysvinit days. | /dev/disks/by-label is not guaranteed to map the name | labels to disks on every boot, and it can change with boot | timing based on the storage controller behavior. Use | /dev/disks/by-uuid instead. | | /dev/disks/by-label is a legacy from the old days of | systems with static disks wired in a specific, unchanging | configuration. | heftig wrote: | You've confused by-label with the kernel's device nodes | (e.g. /dev/sda1). Kernel names are assigned in order. | Labels are just like UUIDs and part of the filesystem. | | However, unlike UUIDs they don't necessarily exist, are | not as likely to be unique (you should only get duplicate | UUIDs by cloning filesystems) and are easier to change. | This makes them less suitable for mount configuration. | corty wrote: | Deactivate graphical boot screens (often just by pressing Esc), | increase the log level[0] and try to read the logs somehow. | Maybe you can still log in as root. If you cannot log in as | root even, but networking works, configure rsyslog to send your | logs to another host and read them there. | | [0] https://freedesktop.org/wiki/Software/systemd/Debugging/ | jraph wrote: | if you can't login as root, you can try to put bash as init | (with init=/bin/bash in the kernel boot command line) to | inspect the system. Might be difficult on some systems (with | SELinux?) | megous wrote: | if you have access to kernel command line to add init= you | can also disable selinux there. | pengaru wrote: | That's super annoying. I've experienced similarly frustrating | boot problems when I had some FS corruption causing /var fsck | to fail @ boot. | | Despite dropping me to a rescue shell, systemd entered a | continuous loop of retrying fscking /var while I was attempting | to identify and fsck the problematic filesystem. | | Few things are more infuriating than having your system drop | you to a rescue shell, while it continues to endlessly change | the state of the system - and not just any state, but the very | area you're in the rescue shell to investigate and resolve. It | just kept making /var busy while I kept attempting to manually | fsck it... | [deleted] | bfrog wrote: | Yeah it's all pretty darn good. People forget the cluster of bash | scripts, various cronz, inetd, log, and other supervisor type | daemons all to do what systemd does better with a simpler config. | | Now ask me about pulseaudio and there's a project with nothing | but issues, looking forward to the new pipewire overlords. | tediousdemise wrote: | > Now ask me about pulseaudio and there's a project with | nothing but issues | | Don't forget Bluetooth, WiFi, and graphics ( _cough_ Nvidia). | Or any peripheral, for that matter. | | Until vendors stop forcing us to play games with their broken | blobs, Linux will forever be just a grand experiment for | frustrated hobbyists and unfortunate professionals around the | globe. A digital tragedy of the commons. | techrat wrote: | >Linux will forever be just a grand experiment for frustrated | hobbyists and unfortunate professionals around the globe. | | Linux is the most widely used kernel in the entire world. If | you consider 'Linux' to be the OS, it is #1. If you consider | all distributions that use the Linux kernel, it is also #1, | collectively and individually (Android.) | | If you look at the ranked list of Supercomputers, Linux has | the entire top 500. | | What the hell are you talking about? | OhSoHumble wrote: | Probably just referring to desktop usage. | kllrnohj wrote: | > and individually (Android.) | | Which, worth noting, does its own thing for init & services | ( https://cs.android.com/android/platform/superproject/+/ma | ste... ). | | I think it's fair to assume they were talking about Linux | as a desktop OS, which for some reason absolutely refuses | to learn lessons from the other 99.9% of the market. And is | also infatuated with internal fragmentation despite its | tiny usage. That makes it really fun as a hobby for sure, | and powerfully flexible for datacenter usages (from the | mundane to the supercomputers). But also pretty terrible as | a "I just want my laptop to _work reliably & well_" OS. | GekkePrutser wrote: | I don't agree with her sentiment that alpine should adopt | something similar. The whole reason I use alpine is because it | doesn't use systemd. | | I like alpine because it uses a simpler, lighter approach. It's | not for everyone, and I'm pretty sure alpine maintainers don't | intend it to be for everyone. It serves its niche really well as | it is. For those who like systemd there's many distro choices | already. | | Edit: As detaro mentioned in another comment ( | https://news.ycombinator.com/item?id=27176391 ) they are actually | working on something, but they aim to solve the various issues | around alpine that don't fit well with alpine's philosophy. It | sounds like a good story and I'm open to this idea. The main | reasons I don't like systemd are its complexity and its dbus | reliance, basically bringing too many desktop paradigms into all | Linux systems, even servers and containers. | | So about my above point: I'm open to it adopting a service | manager but not too similar to systemd :) | [deleted] | yuuta wrote: | I'm using split horizon DNS with WireGuard on my systems (I mean | using special DNS servers for some domain names and the public | ones for the rest. Correct me if I'm wrong). I used to try | systemd-networkd and systemd-resolved because they seem to fit | together. I found systemd-resolved is extremely buggy with | systemd-networkd managed WireGuard. My WireGuard endpoints use | domain names and systemd-networkd fails to resolve them after a | manual restart. I couldn't figure out why. | | I also could not find a way to NOT combine split horizon DNS with | network interfaces (like most documents online only include | scenarios like resolving corporate domain names using a VPN | interface but that wasn't my case). I have multiple VPN | connections to the same network (runs BGP basically) and I did | not want the split DNS to bind to some specified interfaces. | | Moreover, systemd-resolved seems not able to do split DNS without | systemd-networkd. It needs systemd-networkd managed interfaces to | do that. What? I just want separate DNS servers for separate | domains. My WireGuard interfaces are managed by wg-quick. Correct | me if it is possible to use systemd-resolved standalone in this | case. | | The hard systemd-resolved dependency of systemd-networkd also had | issues. Sometimes, I just want systemd-networkd to configure | resolv.conf using DHCP DNS but it looks like I HAD TO START | systemd-resolved AND LINK /run/systemd/stub-resolv.conf to | /etc/resolv.conf in order to achieve that (correct me if there's | a way to not use resolved completely). Excuse me? | | systemd-networkd itself was buggy, too. I once encountered | https://github.com/systemd/systemd/issues/18108 and my friends | reported they faced this issue as well. I don't mean that | software should not have bugs but encounter them is really | inconvenient. | | In conclusion, I mean, systemd components seem to fit together | and help you do a lot of things, but they did't do the things | well, compared to separate programs for their specified tasks. | Moreover, you have to deal with various components of systemd and | figure out their relationships in order to achieve some specified | tasks. For split DNS, I'm using dnsmasq now and it's working | pretty well. | jph wrote: | > By not having something competitive Alpine is less and less | attractive for newer production deployments. | | Alpine uses OpenRC (https://wiki.gentoo.org/wiki/OpenRC). IMHO | Alpine and OpenRC tend to make production deployments more | predictable and reliable, because of deliberately simple init | ordering, as well as a smaller surface area as compared to | systemd. | coolreader18 wrote: | I think given these lines in the article (and the fact that she | mentions OpenRC): | | > I have been an Alpine user for almost a decade and it's one | of my favorite linux distributions. | | > This talk is intended to show how green the grass is on the | other side and how Alpine can benefit from these basic ideas. | | She's probably familiar with it. | detaro wrote: | There is some work on a replacement that does more while | fitting the requirements and philosophy of alpine: | https://skarnet.com/projects/service-manager.html | GekkePrutser wrote: | Interesting, I have a lot more faith in the alpine team | implementing something like this than in RedHat. Even when | RedHat made consumer OSes (before spinning them off to | Fedora) I didn't like them. | | What they're doing looks like a great alternative. I also | hope they will not have any reliance on dbus as systemd does. | Reliance on dbus is one of the main issues I have with | systemd (besides its size and complexity). | mendelmaleh wrote: | Good read, thank you! | xena wrote: | Author of the talk here. If you have questions for me please feel | free to reply to this comment with them. | Hackbraten wrote: | No question, just thank you for an excellent article. I've | found it difficult to come across articles that focus on the | good parts of systemd. | Traster wrote: | Hey, thanks for the great talk, I often type something like | "journalctl -xfe" and just get completely lost, you mentioned | that's out of scope for this talk but do you have a | recommendation for some hints about how to use journalctl | effectively? | dale_glass wrote: | A quick guide: | | * Want logs since boot? journalctl -b. The first boot is -b | 0, the previous boot is -b -1. | | * Want logs for smartd? journalctl -u smartd | | * Want warnings and above? journalctl -p warning | | * Want logs since yesterday? journalctl -S yesterday | | * Want logs on April 22 between 10 and 11? journalctl -S | "2021-04-22 10:00" -U "2021-04-22 11:00" | | * Want logs with microsecond timestamps? journalctl -o short- | precise | | * Or in JSON for parsing? journalctl -o json | | * Want tail -f? journalctl -f | themulticaster wrote: | To add some of my own favorite commands: | | * Want only kernel logs? journalctl -k or journalctl | --dmesg | | * Want to filter the output right away? journalctl | --grep=some_regex | | * You booted a live environment, but want to look at the | installed OS's journal: journalctl | --root=/path/to/mounted_root or journalctl | --directory=/path/to/journal | | * Want to look at the logs from a container you booted | using systemd-nspawn (and friends)? Just use journalctl | --machine=container_id | | * What time intervals do the boot indices (for -b) | correspond to? Look no further than journalctl --list-boots | | I'd say journalctl really has a lot of generally useful | flags. | jen20 wrote: | I did a talk at HashiConf EU a few years ago with the same | title - interesting to see where our ideas of "the good parts" | overlap! | xena wrote: | Wow, I think that was completely by coincidence. I've never | seen your talk before. Where did we overlap? | fghorow wrote: | "This page intentionally left blank." | | (Yeah, I'm a dinosaur who remembers dead tree IBM manuals. Sue | me.) | ape4 wrote: | 1. You don't have to clone then modify a 100 line bash script for | a normal daemon's /etc/rc.d/init.d script. | jude- wrote: | The OpenBSD init system and OpenRC init system haven't had this | problem for years -- even before systemd became a thing. | CameronNemo wrote: | Similarly runit, s6, and Upstart have never had that issue. | It is the oddest complaint to come up with when considering | life without systemd. | unanswered wrote: | I for one have great sympathy for people who are placed in | the position of having to give good reasons for using | systemd. "All your distro are belong to us" was the | original plan but it didn't quite work out. The shills had | to scramble but they managed to make the best of a bad | situation. | jraph wrote: | Upstart was good and probably paved the way to systemd. | cbmuser wrote: | No, Upstart's design was so bad that even Scott James | Remnant (the creator of Upstart) admitted it's design was | inherently flawed. | | The biggest problem with Upstart was that it put the | dependency chain upside down and just started a service | whenever it's dependencies were fulfilled, no matter | whether it actually made sense to start the service. | ahartmetz wrote: | I remember reading about it and thought it seemed wrong, | but presumably the author did his research and knew what | he was doing. Oh well. His attitude and the readability | of Upstart code were very good, though. | | The Upstart design was probably easier to implement. With | the systemd design, the dependency tree needs to be | traversed bottom up (figure out what needs to be started | to start the targeted set of services), then top down | (start dependees when dependencies are ready). With | Upstart, top down was enough. Or so it seems. | agumonkey wrote: | I wonder if systemd is influencing a new init right now | moshmosh wrote: | OpenRC is the only init system I've ever used on Linux that I | actually _like_ , rather than just tolerating. | dbcurtis wrote: | This | [deleted] | fiddlerwoaroof wrote: | Instead you have to write a couple-hundred-line bash script and | then modify an under-documented ini-file to reference it | merb wrote: | you never used systemd? | tgv wrote: | There's a bit of truth in that. The file format is | underdocumented. You've got to look everywhere to find | snippets of information, and then there are these weird | incompatibilities between versions (Service.Type bit me | once, and either After or Wants simply didn't do what the | scant information suggested). And it often starts a script | or a dedicated tool, e.g. for backing up, or watching | directories. | | However, it's easier to deploy and keep track of your own | services, sudo and chdir are built in, there's a watch dog, | easy journaling. That's nothing to sneeze at. | fiddlerwoaroof wrote: | I've written a handful of systemd units now and, while | the simple cases are nice, anything more complicated is | really hard to figure out by reading manpages or other | official sources of documentation. And, additionally, | I've found it's often easier to write a shell script to | be invoked by systemd than to handle certain issues in | the unit file itself. | | As far as the watchdog but things go there's plenty of | good enough solutions that don't really on systemd: | daemonize, DJB's daemontools, etc. | iso1210 wrote: | I have 90 scripts in my /etc/init.d directory. 51 are less than | 100 lines, the most recent one I created is actually a perl | script which is 29 lines long. | fiddlerwoaroof wrote: | Yeah, with the #!-lines, sysv init scripts can be written in | just about anything and, as long as you implement the | expected command line interface, the init system will handle | it correctly. | Arch-TK wrote: | I've not seen even sysvinit based systems with this problem. | Every service manager I know of solves this problem. This is as | much an argument in favour of systemd as "don't have to clean | up horse excrement" is an argument in favour of owning a fiat | 500. | viraptor wrote: | Most init scripts are in that ballpark. See openssh for | example (88 lines) https://github.com/openssh/openssh- | portable/blob/master/open... | | And yet, it still tries to use pidfiles (which systemd has a | better alternative for), and notes "We really should check if | the service is already going" which they don't do because | it's an actual non-trivial issue if you're writing a | standalone init script. | Arch-TK wrote: | Does anyone use these init scripts? Every popular sysvinit | based init system uses short declarative service scripts | for most services. Every non-sysvinit init system seems to | mostly manage with 2 line scripts. | | And yes pidfiles are crap and I don't like them either (a | genuine criticism of sysvinit and advantage of systemd) but | systemd isn't the first init system to do away with | pidfiles so it's hardly a unique strength. | viraptor wrote: | Could you show an example of a declarative sysv-init | script? To be honest, I'm not aware of it being possible | and ddg doesn't bring up anything. | | > Does anyone use these init scripts? | | By default every popular distro before systemd/upstart | did. Many services still use them anyways (systemd just | generates a shim to run the start/stop/status operations | on them) | ChuckMcM wrote: | Okay, I'll admit I was hoping for a blank web page :-). Not a | fan. | | I always ask ""What problem is this trying to solve?" and then | "Is this a good solution for that?" | | I appreciate the "startup dependency problem" and when I was in | the Systems Group at Sun we worked with AT&T on their solution | which was the whole "run level" thing. | | And I appreciate the whole "shell scripts can do _anything_ " | problem that just using a naming scheme on shell scripts to run | can lead to unexpected behaviors. | | From an architectural taste perspective, I prefer systems which | are more comprehensible in parts and thus don't try to abstract | out semantics to the point of incomprehension. | | So at the end of the day, something with a better model, better | visibility into its operation and configuration, with a way to | validate and debug dependencies would be good. But I haven't been | impressed with systemd. | Arch-TK wrote: | >I appreciate the "startup dependency problem" - except systemd | doesn't actually solve this problem. | | Good to see people down-voting a completely factually correct | statement that they don't believe and don't want to personally | investigate. Go read the systemd sources or read any article | analyzing how systemd does dependency based startup and see for | yourself. | addicted wrote: | SystemD solved enough problems that such a large number of | distro owners switched to it that one of the biggest complains | against it is how prevalent it is. | dale_glass wrote: | It solves a whole lot of problems to me. | | * It gets rid of a system cobbled together from init, inittab | (which people rarely remember exists), monit, cron, and | inetd/xinetd. | | * It removes a bunch of horrible hacks in init scripts such as | calls to sleep. | | * It removes the need for pid files. | | * It removes the need for a lot of stupid boilerplate, like the | whole start-stop-daemon stuff in scripts. | | * It allows precisely tracking what belongs to each service. I | can easily find out what a random process belongs to. | | * It captures all the log output of each process | | * It drastically improves the upgrade process -- I never need | to look at a 3-way diff of /etc/init.d/apache2 again | | * As a developer it means I don't need to write a slightly | different script for every distro | | * It makes logging a whole lot pleasant | | * It allows having user services without having to hack around | it with cron | | * It allows a whole bunch of settings to be applied to any | random service, without having to edit a shell script and | figure out how to stick it in there. | | * It means services run identically when started by hand, | without the confusion of a service inheriting my local | environment. | | * It makes my VM servers boot faster. And if you're going to | pipe up with "I reboot my servers once a year", that doesn't | cut it for me. I boot my servers on demand, when they're | needed. They adapt to the load. | A4ET8a8uTh0 wrote: | "I hope Alpine ends up with something similar!" | | Direct quote from the website. | | Eh, no. There is already a short list of viable non-systemd | candidates. We.. well.. I don't need less. It already feels like | I am transgressing some law by typing this on PopOS. | rzzzt wrote: | There is some work going on around replacing OpenRC in Alpine, | this might be the thing the author is alluding to: | https://skarnet.com/projects/service-manager.html | gforce_de wrote: | whoever maintains the comparison of init-systems at | https://skarnet.com/projects/service-manager.html should | include OpenWRT's 'procd' | A4ET8a8uTh0 wrote: | Thank you for sharing this. I was not aware of it. | darthrupert wrote: | To each their own, but I won't touch Alpine (outside of docker | use) until they get systemd or something fairly similar. | efiecho wrote: | > To each their own, but I won't touch Alpine (outside of | docker use) until they get systemd.... | | Luckily that won't happen as systemd doesn't support musl | which Alpine uses for libc. | 2pEXgD0fZ5cF wrote: | I'm very glad for that too. | lixtra wrote: | The same design goals that make them a good docker | distribution may have made them to stay away from systemd. | | Meanwhile others claim that systemd makes docker redundant | anyway[1]. | | [1] https://news.ycombinator.com/item?id=26473581 | cerved wrote: | I can't weigh in on the pros/cons of systemd because that's out | of my depth. But I am somewhat fascinated by the hate it seems to | endure. Like this | | https://suckless.org/sucks/systemd/ | | I guess it's a bit of a watershed | Arch-TK wrote: | For something which the authors seem intent on forcing down | everyone's throat whether they want it or not (genuinely you | can find quotes of Lennart claiming he wants to make it more | difficult to not use systemd or that he wants to "gently push" | people towards it) it's not really surprising. If their only | goal was to gain adoption based on the merits of their software | while not intentionally making themselves intrusive nobody | would really care. | exabrial wrote: | Things that are terrible: logging. Why did we have to screw this | up? Just writing stuff to a file was fine. | | Things that are good: simple syntax, easy to use utils | | Hidden gems: cgroups, file systems containment, etc. Systemd | could completely eliminate the need for stuff like docker by | totally isolating a process. | moshmosh wrote: | The nice thing about docker is that you can use the same | tooling on lots of operating systems, not just that it wraps | some Linux functionality. Sure, it may involve using a VM, | depending on the system, but I still get to just run "docker" | and "docker-compose" and such, as if it were native. A systemd | solution would need a VM running Linux w/ systemd _even on | Linux_ , in some cases. | | Really, though, if I care about isolation and don't want to use | docker, I'm gonna reach for FreeBSD + jails anyway, and I don't | think systemd's gonna change that for me any time soon. If I | want containers with better tooling outside the target OS, I'll | use docker. Systemd falls in an awkward middle that I don't | have much interest in. | vially wrote: | > A systemd solution would need a VM running Linux w/ systemd | even on Linux, in some cases. | | But that's exactly what happens with docker anyway (e.g.: | Docker is still running in the Linux VM when running it from | a different operating system). And considering the widespread | systemd adoption throughout the Linux ecosystem, it's far | more likely for systemd to be already installed in the VM | than docker anyway. | comboy wrote: | It's just status quo. How would you feel about introducing | having to log things into file after all you needed was stdout? | | Btw, you still can do that nothing is stopping you, but I don't | think each and every service needs to decide (and let user | configure) where to put its logs, how to separate them (maybe | by errors, maybe by users/domains) and how to rotate them. | | I'm still not a big fan of systemd but it does make some things | easier (but why wouldn't they think for a moment how easy | "systemctl restoptart foo" will be to type, tab complete and | issue different commands one after another) | dale_glass wrote: | The logging is amazing. Transparent compression, easy search, | output in JSON if you want to parse logs, timestamps with | microseconds if you need them. | | No, writing stuff to a file was not fine. It's only fine if all | you do is to 'tail' and 'grep' it, but any time you want to do | something more advanced, it's a bloody mess. | | Why parse a text date, when it started its life as an unix | timestamp? Why write a bunch of regular expressions trying to | get basic data such as an HTTP response code, when it could be | a field in the log that you could just search for? Why deal | with parsing while the log may rotate or truncate under you, | when the log system could be making your life easier? | | I wrote a whole bunch of log parsing stuff in my day, and so I | have no nostalgia whatsoever regarding this "just write stuff | to a file" approach. | Arch-TK wrote: | > The logging is amazing. | | Amazing for you maybe. If you have a slower spinning HDD | journalctl -eu <service> takes 10-30 seconds (not an | exaggeration). Meanwhile cat or tail -f of a log file is | almost instant somehow. | | > Why write a bunch of regular expressions trying to get | basic data such as an HTTP response code, when it could be a | field in the log | | I mean, this is a problem which doesn't require systemd to | solve, you just need a better log format. Someone could have | standardised a line based logging format with fields and then | someone else could have written libraries to write it and | libraries to parse it. | | > Why deal with parsing while the log may rotate or truncate | under you | | That's a log daemon problem that I haven't found to exist | with any log daemon I have used. | dale_glass wrote: | > If you have a slower spinning HDD journalctl -eu | <service> takes 10-30 seconds (not an exaggeration). | | That's because you're asking for a log of a service since | logging began, which may well have been months worth of | logging. You're saying "give me everything that happened | ever to this service in a pager, then scroll down to the | bottom". | | If you want tail -f, use a -f: 'journalctl -f -u | <service>'. That's going to be a lot faster. You can also | use '-b' to make it return a log of what happened since | boot (worth looking into the manual for the details of that | one), or specify a start date with -S, like '-S yesterday'. | | > Someone could have standardised a line based logging | format | | And what were they waiting for? In the end, systemd did it, | but it's a pretty recent invention. There was plenty time | to get there first. | | > That's a log daemon problem that I haven't found to exist | with any log daemon I have used. | | Because log daemons classically don't care about this at | all -- it's a problem for whatever reads the files that the | log daemon generates afterwards. | Arch-TK wrote: | > That's because you're asking for a log of a service | since logging began, which may well have been months | worth of logging. You're saying "give me everything that | happened ever to this service in a pager, then scroll | down to the bottom". | | journalctl -fu is just as slow. I don't keep months of | logs and the amount of logs I do keep easily should fit | within a minimal amount of space. Somehow systemd manages | to screw this up completely and I really don't think this | has anything to do with user error. | | > And what were they waiting for? In the end, systemd did | it, but it's a pretty recent invention. There was plenty | time to get there first. | | I don't know what people were waiting for. But I think | what systemd did could have been done in a more | transparent and decoupled fashion that didn't involve | binary logs. Just because they did something right | doesn't mean that the way they went about achieving it | isn't inherently flawed. I get that logs suck but fixing | that shouldn't require me to change my entire system | management stack. | | > Because log daemons classically don't care about this | at all -- it's a problem for whatever reads the files | that the log daemon generates afterwards. | | The log daemon I use (svlogd) first closes the current | file, renames it and then opens a new empty current file. | If it finds that there's now too many old logfiles it | deletes the oldest one. I don't see how this could | possibly break any tool which is reading the file except | for cutting the stream short when tailing during a | rotation. Moreover I don't think that's a problem since | if you need to do some kind of processing of the logs | then you should just put that in front of or after the | instance of svlogd in the pipeline (svlogd can be asked | to forward logs to stderr which then means you can chain | it with something else). | ATsch wrote: | > Just writing stuff to a file was fine. | | Haha, I wish. Then that file becomes too large. So you want to | truncate the start. However, removing bytes from the start of | an open file is a bad idea. So you open a new log file every | day instead. You really want to compress the old log files too, | to save space. So now you can no longer just grep the logs as | before. You're now also duplicating code and configuration for | every daemon. So you write a daemon called `logrotate` that | does this centrally instead. Into which you then copy the log | paths configured for every program on your system and how long | the logs should get kept. However programs don't much like it | when you move the log files out from under them. So you patch | every program respond to a signal by checking whether it's log | files have been moved. Which then breaks for multi-process | daemons. You come up with a whole buffet of ugly hacks to make | that work most of the time. | | Then you install a program which doesn't come with a logrotate | config. Then you decide you'd really want a more structured way | to analyze logs than writing fragile regexes. Then you want to | run something as a non-root user. Then you have an ephermeral | batch job. Then you want to ship off all of your logs to | another location and really don't want to duplicate the log | paths yet again. Then you want to see what happened just before | the server crashed last boot. Then you run two of a daemon and | give them different log paths. Then you want to look at your | kernel and application logs together. Then you want to delete | just last month's logs to free up some space. Also wouldn't it | be handy if there was some kind of index to make things faster? | ... | magicalhippo wrote: | > However, removing bytes from the start of an open file is a | bad idea. | | A digression, but I keep wondering why filesystems never | optimized this scenario. Being able to trim the start of the | file is really useful in certain cases, and without support | options are pretty bad. | | It also seems fairly easy to support at the cost of a few | bytes in the file entry (a start-of-data index into first | sector), and resizing a file by seeking past end is already a | thing. | | I must be missing something... | Arch-TK wrote: | All of this seems solved by svlogd honestly. | ATsch wrote: | I have never used svlogd but at it looks like it does read | logs from it's stdin, which is a solid step forward as it | avoids much of the pitfalls caused by applications writing | directly into log files. | | Collection is just the first part of a log system though of | course. | ex_amazon_sde wrote: | JournalD is excellent. Indexing structured data like logs is | the right thing to do. Log messages have a lot of useful | metadata to do filtering on. | | > Systemd could completely eliminate the need for stuff like | docker by totally isolating a process. | | It does: nspawn is much leaner than docker and it's available | on all Linux systems with SystemD. | stock_toaster wrote: | > JournalD is excellent | | This literally made me laugh out loud. | | Over the years I have historically had several production | system outages due to journald corrupting its own data store, | being unable to log, and services hanging due to being unable | to log (process stdout buffer full, journald not reading it, | so process stuck trying to write output). | | It certainly has improved over the years, and it has been a | long time since this issue has happened (on one of the | systems I manage at least), but I have accrued some pretty | deep seated feelings about it at this point due to historical | pain. | porsager wrote: | That's interesting. Do you have a link to more info about | isolating a process like that? | comboy wrote: | Arch wiki is usually a good place to learn something in a | succinct way (I've never even used Arch): | https://wiki.archlinux.org/title/cgroups | memco wrote: | > If you need to reboot the server right now for some reason, | will that service come back up on reboot? | | I didn't see this discussed more after it was asked, but I could | use some pointers on how to answer this question since I have a | service I wrote that I think should start on reboot, but has to | be started manually. It would be nice to know when I have fixed | it without having to reboot after each change to just to test. | Anyone know how I can do that? | viraptor wrote: | The answer is there in the status output: | Loaded: loaded ... ; enabled ... | | If it's enabled and starts from the service file currently, you | can expect that it will start after a reboot. Of course you can | never be sure it's successful, but at least it will attempt to. | corty wrote: | Problem with systemd is, startup is not deterministic. systemd | starts everything it can start at the same time, leading to | subtle race conditions between services and the network. If you | have any dependency that isn't explicitly spelled out or any | service that is falsely reported as ready, you will get funny | system states on some boots, whereas it will work on others. | systemd-udev does its part to make things even worse there, | because it also introduces more races against kernel devices. | So with systemd, even when you tested the system, you can not | always be sure that it will start everything properly. | | Compared to sysv-init this is a regression, because init | scripts usually ran sequentially and also didn't involve | unreliable systemd-udev events. | | Compared to Solaris SMF this is pitiful. SMF takes into account | service dependencies from its declarative service descriptions | and computes a deterministic sequential startup order. If it | has booted once, it will always boot again, in the same order, | reliably. Best of all worlds. | viraptor wrote: | > Compared to sysv-init this is a regression, because init | scripts usually ran sequentially and also didn't involve | unreliable systemd-udev events. | | This goes both ways - compared to sysv-init we finally get a | good way to manage services in response to device changes. | corty wrote: | It depends on the application. For laptops, sure, | hotplugging stuff is important. For servers, not so much, | reliability is far more important than getting your USB- | stick automatically recognized. systemd is heavily skewed | towards the unreliable but fast and dynamic laptop usecase. | viraptor wrote: | A server won't have many udev messages happening after | boot, (Past the drive and network cards) so it shouldn't | hold up and services either. Have you got specific | situations in mind that are not reliable? | corty wrote: | Network cards aren't simply just cards anymore. Often you | get multiple interfaces depending on each other, e.g. | bridges depending on multiple ethernet interfaces which | wait for STP delay to be over until they are really | forwarding packets. VLAN interfaces that are dependent on | their parent ethernet interface. VPN and other tunnels. | So you get stuff like "start wpa_suplicant for 802.1x, | take the ethernet interfaces up. start the bridge | interface. check if DNS works for the VPN. then start the | VPN daemon and the VPN interface. then start network | services listening on the VPN interface". That would be a | typical setup for a server in a remote office somewhere. | | Another fun situation is a server with LVM, RAID and | encrypted disks (also typical for a remote office | somewhere). SAS disks will come online (after spinup | time), generate an event. LVM and raid detection will | run, generating more events, usually dependent on one | another. Device naming is in some cases dependent on the | event order, so you also have to rely on systemd-udev not | fucking up by-id or by-name links. Then you have | cryptsetup, which decrypts the disk, generating another | event, but only upon an automatic or manual all-clear and | password from the (hopefully working) network. Then you | arrive at the actual filesystems in fstab, after a chain | of udev events, each of which (and especially the LVM and | cryptsetup ones) sometimes fails. Same "fun" for SAN | devices. | | Oh, and another fun problem is systemd exhausting | randomness for too many UUIDs, so cryptsetup hangs for | swap devices because those get a key from /dev/random. | | Of course those bugs get often fixed over time, but only | until the next RedHat release arrives with yet another | systemd-somethingd, networkmanager, systemd-udev-bug that | introduces another new problem in the above. Usability | has gone up since RH7, but reliability is definitely | down. | viraptor wrote: | This is a long list of complexities, but I'm not sure how | they relate to udev/systemd. Yes, setting up aggregate | interfaces, VPNs and fancy drive configurations is | complex and can fail due to various delays. That's not a | problem with udev reliability if an underlying system | fails. We had the same issues before systemd/udev was a | thing. | | > cryptsetup hangs for swap devices because those get a | key from /dev/random | | You can choose urandom (third column of crypttab). | Systemd doesn't enforce anything here. | klodolph wrote: | At best, ordered startup should be optional, since boot | otherwise takes too long on workstations. | | It would be nice to have hard errors when dependencies are | not explicitly declared. For example, if you don't declare | dependency on the network, your service gets put in a sandbox | where it can't access the network at all. In practice, that | kind of sandboxing sounds difficult. | | My personal experience is that it is fairly straightforward | to debug races in startup. Some service fails, you look in | the log and see that it has a dependency that hasn't started, | and then add that missing dependency to the service's | declaration. Just speaking from personal experience--this | doesn't come up especially often and doesn't require a large | amount of time to fix, and the benefits of fast startup save | you much more time than the extra debugging costs you. | drewg123 wrote: | I reboot my workstation about once a month, and I couldn't | care less how long it takes. What I _do_ care about is that | my workstation actually reboots predictably. If it needs | manual intervention when I 'm traveling, I'm screwed. | aidenn0 wrote: | My POST takes longer than my serialized init scripts on my | workstation. | iso1210 wrote: | I rarely boot my workstation, certainly no more than once a | month (I don't bother with every new kernel update) | acdha wrote: | > Compared to sysv-init this is a regression, because init | scripts usually ran sequentially and also didn't involve | unreliable systemd-udev events. | | I find this really odd because we spent decades with that | exact class of problems on SysV init, in part due to the | primitive dependency model and the lack of standard | mechanisms to launch daemons or react to events meant so many | different reinventions of that particular wheel (need Apache | to wait for NFS mounts? Here's another shell kludge to | maintain!). | | Switching to systemd gave a single, simple answer for all of | that: use the init system. I don't know what services you | manage but for the mostly Debian/RHEL environments I've | worked on your description more fits the before state than | after in my experience. | corty wrote: | Apache waiting for NFS mounts is still not solved. Now | Apache may depend on a file system, but it still cannot | depend on "the LDAP is available so you can resolve the | UIDs on that NFS". So we are a tiny step ahead maybe. But | the problem isn't solved at all, and back in the day we | just solved it by reordering Apache manually all to the | back, maybe with a "check that UIDs are resolvable"-script | in front. systemd can't do that (except if you allow "evil" | shell-scripts in bootup again...). | acdha wrote: | 1. We didn't "solve" that back in the SysV days by | reordering - mostly we just lived with the occasional | race conditions and restarted it. | | 2. You could use systemd's health check or pre-exec to | handle prerequisite checks like this if you can't use the | dependency mechanism for some reason (there's no reason | why you couldn't have a custom type=oneshot service set | as a requirement for Apache). | | There's nothing "evil" about scripting like this - the | thing which makes me tired of SysV is the limited | standard feature set (daemonization, logging, restarts, | dependencies, event handling, etc.) and challenges | coordinating across distributions for things like | overrides. The wheels fall off shell scripting when you | need to deal with that much complexity and do so in a | portable manner with a bunch of shell code you don't | control - a health check is far less tedious to deal | with, especially since you have a ton of functionality in | systemd which avoids the need to handle the hard parts. | zxzax wrote: | I don't see why you couldn't do that in systemd (or any | other init), it's not hard to make a small unit that | simply runs a shell script. | rav wrote: | In the systemctl status output, it says "Loaded: loaded (...; | enabled; ...)" - "enabled" means that the service starts on | boot. Use "systemctl enable" to "enable" a service. It creates | a symlink in /etc/systemd/system/multi-user.target.wants | assuming the service's unit file says to enable it through the | "multi-user" target. By listing that directory you can see what | services load on boot. | CameronNemo wrote: | Not exactly. Socket units can pull in services that are not | otherwise enabled. You also have to search through all of | your udev rules to check if there are SYSTEMD_WANTS | directives pulling services in. | Arnavion wrote: | rav (correctly) said that enabled services are started | automatically on boot. They didn't say that _non_ -enabled | services are _not_ started automatically on boot. | nemetroid wrote: | They also said: | | > By listing that directory you can see what services | load on boot. | Arnavion wrote: | Yes, and I didn't read that sentence as disallowing other | ways to autostart services either. | nemetroid wrote: | To me it's equivalent to "To see what services load on | boot, you can list that directory.". | ratsmack wrote: | >systemd is a set of building blocks that you can use to make a | linux system. | | A more accurate description would be "Systemd is a set of | building blocks that you can use to make Linux more like | Windows". | dale_glass wrote: | systemd is incredibly Linux centric. In fact it intentionally | doesn't even try to be remotely portable, because an explicit | design aim was to expose all the cool stuff that Linux has, but | wasn't getting enough use. | trasz wrote: | Sure, but design-wise it follows Windows principles: it's | tightly coupled, and not particularly tweakable. | [deleted] | dale_glass wrote: | These days I see it as a bonus. I don't _want_ to mess | around with the boot process. I don 't _want_ to write some | sort of glue layer in between loosely coupled parts. And I | certainly don 't want somebody else to do it either, | because that too often tends to result in some kind of | surprise that one has to deal with at 3 AM. | | What I want is a solid system that does what it's supposed | to -- start stuff -- in a maximally boring fashion. | Valmar wrote: | This is far from true, lmao. It's only tightly coupled to | depending on the Linux kernel. | | systemd was in fact heavily inspired by MacOS's launchd. At | worst, it can be said that complex boot managers will begin | to converge on a design that actually works ~ MacOS did, | Windows did, and now Linux has with systemd. The three have | plenty of differences, though systemd is still closer to | launchd than not. | | And in actuality, it's very tweakable. Did you notice that | the source code is open, for a start...? | viseztrance wrote: | I ignored systemd because I thought it didn't affect me, as I | only run linux (fedora) on my desktop, and hardly had any issues | at that. | | But recently, when I changed my fstab file to mount a drive at | boot, I made a typo. My system wouldn't boot at all and it | wouldn't even drop to a shell so I can chroot the file system | like you would with init.d. | | I thought that there's some arcane command I don't know about. | But there wasn't. You're locked off, and expected to boot off | something else to fix your system. Horrible. | | Having this said, I honestly expected way more on the section | about benefits to users. | zaarn wrote: | Atleast on Arch, Systemd should drop you into the Emergency | shell if it cannot mount a device that isn't defined as | optional. IIRC it might require a root password to be set, | however. | | Otherwise, edit the kernel commandline to add | "systemd.unit=emergency.target" to the end, which triggers | systemd to straight boot into this console without trying | anything that it doesn't need to bring up the console. | viseztrance wrote: | It tried when the mounting failed, saying it would drop me to | a shell and then said "it fail to open up a shell". | zaarn wrote: | That usually indicates something is very wrong and might | need an init=/bin/bash instead, as a last resort, to find | out why it couldn't open the shell (usually, because there | is no root password or it couldn't find any shell binary to | run). | otterlicious wrote: | Or if you're blocked by the root password prompt add | `init=/bin/bash` and make your filesystem writable if | necessary with `mount -oremount,rw /` | zaarn wrote: | This will block systemd however. The nice thing about the | emergency shell is that you can use systemd things like | trying to mount filesystems. And you can later isolate | multi-user to continue the boot when you finished your fix. | /bin/bash would be atleast useful to try to get a root | password set. | ajross wrote: | > But recently, when I changed my fstab file to mount a drive | at boot, I made a typo. My system wouldn't boot at all | | I don't know exactly what went wrong in your case, but killing | Unix systems by messing with fstab has been a rite of passage | for more than four decades. Systemd certainly didn't invent | that. Hell, I broke a chromebook not 48 hours ago messing with | the boot setup. | | But FWIW: managing a recovery image is, amusingly, _not_ | historically a job systemd has tried to take on. This is what | your live image is for. | viseztrance wrote: | I've been using linux on my desktop for almost 20 years. This | was the first time I couldn't get a shell to fix my problem. | ATsch wrote: | It sounds like what you ran into are the effects of the general | trend of doing more in early boot, as | https://freedesktop.org/wiki/Software/systemd/separate-usr-i... | elaborates on, which is independent of systemd | viseztrance wrote: | Thanks for sharing this. | kbenson wrote: | I think systemd brought a lot of good stuff to the table, but I | also think it's a sprawling mess with poorly named and documented | (as in there are behaviors implied by statements in different | config directive sections that make it hard to reason about how | stuff works). | | I'm looking forward to the successor of systemd, or at a minimum, | them rolling out config syntax 2.0 which breaks compatibility in | an effort to make stuff more sanely named and documented. | | I know that would cause huge churn. I know it will never happen. | But I can dream, because I don't think anyone can look at how | systemd configs define dependencies and say that they both | understand the nuances and that those directives are are named | sanely in a way that normal people can understand, and that's | indicative of a lot of stuff with systemd. | | Even though it was probably designed by some of the most | knowledgeable people WRT init systems, it grew organically as | they discovered new things they needed, and it needed a lot | (which I think just goes to show how little was really understood | by anyone about how to tightly couple these things). | Decabytes wrote: | I just learned about Vtubers this week. I would have never | thought this would have crossed into the Linux world but here we | are! | efiecho wrote: | Poetteringware seems to be highly praised by mostly young people | who would like Linux to be a Windows clone. People who | understands Unix concepts tends to dislike it. | StreamBright wrote: | Well yes and no. I like the good parts of SystemD (not the | parts this video is talking about). | | For me there are 3 things that SystemD gets right: | | - no shell scripts for service configurations | | - no forking services (no need for daemon mode anymore) | | - logging and service management is tightly coupled (this is | arguably good with some bad parts, required CPU for logging | with systemd is not great) | | And the rest is just pure madness. | | - DNS (but whyt?) | | - NTP (but why????) | | - system limits (seriously, da f???) | | - pid 1 should be a very simple service (More: | https://ewontfix.com/14/) | | I have spent 25 years on Unix, so I guess I understand the | concepts. | larschdk wrote: | DNS + NTP are completely optional, but you get really good | integration with systemd-networkd's DHCP client if you use | them. Switching network is completely seamless, and the | resolver updates fully automatically (direct DNS and NTP are | blocked in many networks, sadly). Also, you get something | that works out of the box, with essentially 0 configuration. | You can't even install a system today over the internet | without setting the time, and systemd-timesyncd does it with | 0 configuration. | Arch-TK wrote: | > - no shell scripts for service configurations | | Other than "sysvinit based init had horrific shell scripts | with 100 lines of repeated nonsense", given that almost every | alternative to systemd which still uses shell scripts needs | around 2 lines of shell per service (including the shebang) | what exactly is the issue with shell scripts (if all they do | is run a wrapper which sets up the environment of a program | or the program itself)? | | > - no forking services | | When I used archlinux these were extremely common and I used | archlinux as recently as last year. | | > - logging and service management is tightly coupled (this | is arguably good with some bad parts, required CPU for | logging with systemd is not great) | | So runit handles spawning a log daemon before the service is | started to ensure all log entries are kept and comes with a | really simple yet extremely powerful log daemon (but | obviously you can use almost anything else including cat if | you want). It manages to integrate logging tightly into | service supervision while being maximally flexible and | decoupled. What do you think about this? | 1vuio0pswjnm7 wrote: | I use daemontools instead of runit. What am I missing. | | multilog is the simplest (smallest) and most versatile log | rotating daemon I have found. | Arch-TK wrote: | A few extra features, but you probably don't need them. I | use svlogd because I use runit and I use runit because it | happens to be supported by my distro. | StreamBright wrote: | OK I bite. Nothing is wrong with shell scripts. However, I | would like to know if something _wont_ work. Shell scripts | do not have this feature. You know if something does not | work when you run it. There might be external dependencies | missing or other very typical problems. If we have a way to | push all of these to "compile time" it is an improvement. I | also spent many years in the land mines of implicit | overrides in shell script hells and thank you but no. As a | good example you can have how people try to set JVM | parameters in a big data project using shell scripts. I it | is absolute insanity. Same thing in init scripts. Did you | implement reload or restart? Is stop actually doing | something or a noop. And so on. | | If there is only one thing that I could keep from systemd, | it would be the use of unit files instead of shell scripts. | | The rest is your observations, nothing to disagree or agree | with. | tristan957 wrote: | No it is the old people that don't understand the systemd | project is exactly the Unix philosophy | dijit wrote: | "Old people", I guess I meet the definition _now_ but When I | first encountered systemd and used it as my daily driver I | was 21. | | I'm 31 now. ___________________________________________________________________ (page generated 2021-05-16 23:00 UTC)