[HN Gopher] The real realtime preemption end game
___________________________________________________________________
The real realtime preemption end game
Author : chmaynard
Score : 317 points
Date : 2023-11-16 14:47 UTC (8 hours ago)
(HTM) web link (lwn.net)
(TXT) w3m dump (lwn.net)
| andy_ppp wrote:
| What do other realtime OS kernels do when printing from various
| places? It almost seems like this should be done in hardware
| because it's such a difficult problem to not lose messages but
| also have them on a different OS thread in most cases.
| EdSchouten wrote:
| Another option is simply to print less, but expose more events
| in the form of counters.
|
| Unfortunately, within a kernel that's as big as Linux, that
| would leave you with many, many, many counters. All of which
| need to be exported and monitored somehow.
| taeric wrote:
| This seems to imply you would have more counters than
| messages? Why would that be?
|
| That is, I would expect moving to counters to be less
| information, period. That not the case?
| nraynaud wrote:
| My guess is that each counter would need to have a
| discovery point, a regular update mechanism and a
| documentation, while you can send obscure messages willy-
| nilly in the log? And also they become an Application
| Interface with a life cycle while (hopefully) not too many
| people will go parse the log as an API.
| taeric wrote:
| I think that makes sense, though I would still expect
| counters to be more dense than logs. I'm definitely
| interested in any case studies on this.
| ajross wrote:
| It's just hard, and there's no single answer.
|
| In Zephyr, we have a synchronous printk() too, as for low-level
| debugging and platform bringup that's usually _desirable_ (i.e.
| I 'd like to see the dump from just before the panic please!).
|
| For production logging use, though, there is a fancier log
| system[1] designed around latency boundaries that essentially
| logs a minimally processed stream to a buffer than then gets
| flushed from a low priority thread. And this works, and avoids
| the kinds of problems detailed in the linked article. But it's
| fiddly to configure, expensive in an RTOS environment (you need
| RAM for that thread stack and the buffer), depends on having a
| I/O backend that is itself async/low-latency, and has the
| mentioned misfeature where when things blow up, it's usually
| failed to flush the information you need out of its buffer.
|
| [1] Somewhat but not completely orthogonal with printk. Both
| can be implemented in terms of each others, mostly. Sometimes.
| vlovich123 wrote:
| What if the lower priority thread is starved and the buffer
| is full? Do you start dropping messages? Or overwrite the
| oldest ones and skip messages?
| ajross wrote:
| It drops messages. That's almost always the desired
| behavior: you never want your logging system to be doing
| work when the system is productively tasked with other
| things.
|
| I know there was some level of argument about whether it's
| best to overwrite older content (ring-buffer-style,
| probably keeps the most important stuff) or drop messages
| at input time (faster, probably fewer messages dropped
| overall). But logging isn't my area of expertise and I
| forget the details.
|
| But again, the general point being that this is a
| complicated problem with tradeoffs, where most developers
| up the stack tend to think of it as a fixed facility that
| shouldn't ever fail or require developer bandwidth. And
| it's not, it's hard.
| xenadu02 wrote:
| In many problem spaces you can optimize for the common success
| and failure paths if you accept certain losses on long-tail
| failure scenarios.
|
| A common logging strategy is to use a ring buffer with a
| separate isolated process reading from the ring. The vast
| majority of the time the ring buffer handles temporary
| disruptions (eg slow disk I/O to write messages to disk) but in
| the rare failure scenarios you simply overwrite events in the
| buffer and increment an atomic overwritten event counter.
| Events do not get silently dropped but you prioritize forward
| progress at the cost of data loss in rare scenarios.
|
| Microkernels and pushing everything to userspace just moves the
| tradeoffs around. If your driver is in userspace and blocks
| writing a log message because the log daemon is blocked or the
| I/O device it is writing the log to is overloaded it does the
| same thing. Your realtime thread won't get what it needs from
| the driver within your time limit.
|
| It all comes down to CAP theorem stuff. If you always want the
| kernel (or any other software) to be able to make forward
| progress within specific time limits then you must be willing
| to tolerate some data loss in failure scenarios. How much and
| how often it happens depends on specific design factors, memory
| usage, etc.
| TeeMassive wrote:
| It's kind of crazy that a feature necessitated 20 years of active
| development to be somewhat called complete.
|
| I hope it will be ready soon. I'm working in a project that has
| strict serial communication requirements and it has caused us a
| lot of headaches.
| eisbaw wrote:
| Zephyr RTOS.
| worthless-trash wrote:
| Can you expand on this, as I'm a little naive in this area, say
| you isolated the cpus (isolcpus parameter) and then taskset
| your task onto the isolated cpu, would not the scheduler no
| longer be involved, and your task be the only thing serviced by
| that CPU ?
|
| Is it other interrupts on the CPU that break your process out
| of the "real time" requirement, I find this all so interesting.
| TeeMassive wrote:
| It's an embedded system with two logical cores with at least
| 4 other critical processes running. Doing that will only
| displace the problem.
| loeg wrote:
| Synchronous logging strikes again! We ran into this some at work
| with GLOG (Google's logging library), which can, e.g., block on
| disk IO if stdout is a file or whatever. GLOG was like, 90-99% of
| culprits when our service stalled for over 100ms.
| cduzz wrote:
| I have discussions with cow-orkers around logging;
|
| "We have Best-Effort and Guaranteed-Delivery APIs"
|
| "I want Guaranteed Delivery!!!"
|
| "If the GD logging interface is offline or slow, you'll take
| downtime; is that okay?"
|
| "NO NO Must not take downtime!"
|
| "If you need it logged, and can't log it, what do you do?"
|
| These days I just point to the CAP theorem and suggest that
| logging is the same as any other distributed system. Because
| there's a wikipedia article with a triangle and the word
| "theorem" people seem to accept that.
|
| [edit: added "GD" to clarify that I was referring to the
| guaranteed delivery logging api, not the best effort logging
| API]
| loeg wrote:
| I read GD as "god damn," which also seems to fit.
| rezonant wrote:
| aw you beat me to it
| msm_ wrote:
| Interesting, I'd think logging is one of the clearest
| situations when you want best effort. Logging is, almost by
| definition, not the "core" of your application, so failure to
| log properly should not prevent the core of the program from
| working. Killing the whole program because logging server is
| clearly throwing the baby out with the bathwater.
|
| What people probably mean is "logging is important, let's
| avoid losing log messages if possible", which is what "best"
| in "best effort" stands for. For example it's often a good
| idea to have a local log queue, to avoid data loss in case of
| a temporary log server downtime.
| cduzz wrote:
| People use logging (appropriately or inappropriately; not
| my bucket of monkeys) for a variety of things including
| audit and billing records, which are likely a good case for
| a guaranteed delivery API.
|
| People often don't think precisely about what they say or
| want, and also often don't think through corner cases such
| as "what if XYZ breaks or gets slow?"
|
| And don't get me started on "log" messages that are 300mb
| events. Per log. Sigh.
| insanitybit wrote:
| If you lose logs when your service crashes you're losing
| logs at the time they are most important.
| tux1968 wrote:
| That's unavoidable if the logging service is down when
| your server crashes.
|
| Having a local queue doesn't mean logging to the service
| is delayed, it can be sent immediately. All the local
| queue does is give you some resiliency, by being able to
| retry if the first logging attempt fails.
| insanitybit wrote:
| If your logging service is down all bets are off. But by
| buffering logs you're now accepting that problems _not_
| related to the logging service will also cause you to
| drop logs - as I mentioned, your service crashing, or
| being OOM 'd, would be one example.
| andreasmetsala wrote:
| No, you're losing client logs when your logging service
| crashes. Your logging service should probably not be
| logging through calls to itself.
| tremon wrote:
| But if your service has downtime because the logs could
| not be written, that seems strictly inferior. As someone
| else wrote upthread, you only want guaranteed delivery
| for logs if they're required under a strict audit regime
| and the cost of noncompliance is higher than the cost of
| a service outage.
| insanitybit wrote:
| FWIW I agree, I'm just trying to be clear that you are
| choosing one or the other, as the grandparent was
| stating.
| linuxdude314 wrote:
| It's not the core of the application, but it can be the
| core of the business.
|
| For companies that sell API access logs in one form or
| another are how bills are reconciled and usage metered.
| wolverine876 wrote:
| Logging can be essential to security (to auditing). It's
| your record of what happened. If an attacker can cause
| logging to fail, they can cover their tracks more easily.
| deathanatos wrote:
| To me audit logs aren't "logs" (in the normal sense),
| despite the name. They tend to have different
| requirements; e.g., in my industry, they must be
| retained, by law, and for far longer than our normal
| logs.
|
| To me, those different requirements imply that they
| _should_ be treated differently by the code, probably
| even under distinct flows: synchronously, and ideally to
| somewhere that I can later compress like hell and store
| in some very cheap long term storage.
|
| Whereas the debug logs that I use for debugging? Rotate
| out after 30 to 90d, ... and yeah, best effort is fine.
|
| (The audit logs might also end up in one's normal logs
| too, for convenience.)
| wolverine876 wrote:
| While I generally agree, I'll add that the debug logs can
| be useful in security incidents.
| fnordpiglet wrote:
| It depends.
|
| Some systems the logs are journaled records for the
| business or are discoverable artifacts for compliance. In
| highly secure environments logs are not only durable but
| measures are taken to fingerprint them and their ordering
| (like ratchet hashing) to ensure integrity is invariant.
|
| I would note that using disk based logging is generally
| harmful in these situations IMO. Network based logging is
| less likely to cause blocking at some OS level or other
| sorts of jitter that's harder to mask. Typically I develop
| logging as an in memory thing that offloads to a remote
| service over the network. The durability of the memory
| store can be an issue in highly sensitive workloads, and
| you'll want to do synchronous disk IO for that case to
| ensure durability and consistent time budgets, but for
| almost all application disk less logging is preferable.
| shawnz wrote:
| If you're not waiting for the remote log server to write
| the messages to its disk before proceeding, then it seems
| like that's not guaranteed to me? And if you are, then
| you suffer all the problems of local disk logging but
| also all the extra failure modes introduced by the
| network, too
| fnordpiglet wrote:
| The difference is that network IO can be more easily
| masked by the operating system than block device IO. When
| you offload your logging to another thread the story
| isn't over because your disk logging can interfere at a
| system level. Network IO isn't as noisy. If durability is
| important you might still need to wait for an ACK before
| freeing the buffer for the message which might lead to
| more overall memory use, all the operations play nicely
| in a preemptable scheduling system.
|
| Also, the failure modes of _systems_ are very tied to
| durable storage devices attached to the system and very
| rarely to network devices. By reducing the number of
| things that need a disk (ideally to zero) you can remove
| disks from the system and its availability story. Once
| you get to fully disk less systems the system failure
| modes are actually almost nothing. But even with disks
| attached reducing the times you interact with the disk
| (especially for chatty things like logs!) reduces the
| likelihood the entire system fails due to a disk issue.
| lmm wrote:
| > If you're not waiting for the remote log server to
| write the messages to its disk before proceeding, then it
| seems like that's not guaranteed to me?
|
| Depends on your failure model. I'd consider e.g.
| "received in memory by at least 3/5 remote servers in
| separate datacenters" to be safer than "committed to
| local disk".
| ReactiveJelly wrote:
| If it's a journaled record for the business then I think
| I'd write it to SQLite or something with good
| transactions and not mix it in the debug logs
| fnordpiglet wrote:
| There are more logs than debug logs, and using SQLite as
| the encoding store for your logs doesn't make it not
| logging.
| supriyo-biswas wrote:
| The better way to do this is to write the logs to a file or
| an in-memory ring buffer and have a separate thread/process
| push logs from the file/ring-buffer to the logging service,
| allowing for retries if the logging service is down or slow
| (for moderately short values of down/slow).
|
| Promtail[1] can do this if you're using Loki for logging.
|
| [1] https://grafana.com/docs/loki/latest/send-data/promtail/
| insanitybit wrote:
| But that's still not guaranteed delivery. You're doing what
| the OP presented - choosing to drop logs under some
| circumstances when the system is down.
|
| a) If your service crashes and it's in-memory, you lose
| logs
|
| b) If your service can't push logs off (upstream service is
| down or slow) you either drop logs, run out of memory, or
| block
| hgfghui7 wrote:
| You are thinking too much in terms of the stated
| requirements instead of what people actually want: good
| uptime and good debugability. Falling back to local
| logging means a blip in logging availability doesn't turn
| into all hands on deck everything is on fire. And it
| means that logs will very likely be available for any
| failures.
|
| In other words it's good enough.
| mort96 wrote:
| "Good uptime and good reliability but no guarantees" is
| just a good best effort system.
| insanitybit wrote:
| Good enough is literally "best effort delivery", you're
| just agreeing with them that this is ultimately a
| distributed systems problem and you either choose CP or
| AP.
| kbenson wrote:
| Yeah, what the "best effort" actually means in practice
| is usually a result of how much resources you want to
| throw at the problem. Those give you runway on how much
| of a problem you can withstand and perhaps recover from
| without any loss of data (logs), but in the end you're
| usually still just buying time. That's usually enough
| though.
| o11c wrote:
| Logging to `mmap`ed files is resilient to service
| crashes, just not hardware crashes.
| sroussey wrote:
| We did something like this at Weebly for stats. The app
| sent the stats to a local service via UDP, so shoot and
| forget. That service aggregated for 1s and then sent off
| server.
| laurencerowe wrote:
| Why UDP for a local service rather than a unix socket?
| rezonant wrote:
| > "If the GD logging interface is offline or slow, you'll
| take downtime; is that okay?"
|
| > [edit: added "GD" to clarify that I was referring to the
| guaranteed delivery logging api, not the best effort logging
| API]
|
| i read GD as god-damned :-)
| salamanderman wrote:
| me too [EDIT: and I totally empathized]
| Zondartul wrote:
| I have some wishfull thinking ideas on this, but it should be
| possible to have both at least in an imaginary, theoretical
| scenario.
|
| You can have both guaranteed delivery and no downtime if your
| whole system is so deterministic that anything that normally
| would result in blocking just will not, cannot happen. In
| other words it should be a hard real-time system that is
| formally verified top to bottom, down to the last transistor.
| Does anyone actually do that? Verify the program and the
| hardware to prove that it will never run out of memory for
| logs and such?
|
| Continuing this thought, logs are probably generated
| endlessly, so either whoever wants them has to also guarantee
| that that they are processedand disposed of right after being
| logged... or there is a finite ammount of log messages that
| can be stored (arbitrary number like 10 000) but the user (of
| logs) has to guarantee that they will take the "mail" out of
| the box sooner than it overfills (at some predictable,
| deterministic rate). So really that means even if OUR system
| is mathematically perfect, we're just making the downtime
| someone elses problem - namely, the consumer of the infinite
| logs.
|
| That, or we guarantee that the final resources of our self-
| contained, verified system will last longer than the finite
| shelf life of the system as a whole (like maybe 5 years for
| another arbitrary number)
| morelisp wrote:
| PACELC says you get blocking or unavailability or
| inconsistency.
| tuetuopay wrote:
| We had prod halt once when the syslog server hanged. Logs were
| pushed through TCP which propagated the blocking to the whole
| of prod. We switched to UDP transport since: better to lose
| some logs than the whole of prod.
| tetha wrote:
| Especially if some system is unhappy enough to log enough
| volume to blow up the local log disk... you'll usually have
| enough messages and clues in the bazillion other messages
| that have been logged.
| deathanatos wrote:
| TCP vs. UDP and async best-effort vs. synchronous are
| _completely_ orthogonal...
|
| E.g., a service I wrote wrote logs to an ELK setup; we logged
| over TCP. But the logging was async: we didn't wait for logs
| to make it to ELK, and if the logging services went down, we
| just queued up logs locally. (To a point; at some point, the
| buffer fills up, and logs were discarded. The process would
| make a note of this if it happened, locally.)
| tuetuopay wrote:
| > TCP vs. UDP and async best-effort vs. synchronous are
| completely orthogonal...
|
| I agree, when stuff is properly written. I don't remember
| the exact details, but at least with UDP the asyncness is
| built-in: there is no backpressure whatsoever. So poorly
| written software can just send udp to heart's end.
| lopkeny12ko wrote:
| I would posit that if your product's availability hinges on +/-
| 100ms, you are doing something deeply wrong, and it's not your
| logging library's fault. Users are not going to care if a
| button press takes 100 more ms to complete.
| fnordpiglet wrote:
| 100ms for something like say API authorization on a high
| volume data plane service would be unacceptable. Exceeding
| latencies like that can degrade bandwidth and cause workers
| to exhaust connection counts. Likewise, even in humans
| response space, 100ms is an enormous part of a budget for
| responsiveness. Taking again authorization, if you spend
| 100ms, you're exhausting the perceptible threshold for a
| humans sense of responsiveness to do something that's of no
| practical value but is entirely necessary. Your UI developers
| will be literally camped outside your zoom room with virtual
| pitch forks night and day.
| loeg wrote:
| Yes, and in fact the service I am talking about is a high
| volume data plane service.
| hamandcheese wrote:
| Not every API is a simple CRUD app with a user at the other
| end.
| kccqzy wrote:
| Add some fan out and 100ms could suddenly become 1s, 10s...
| oneepic wrote:
| Oh, we had this type of issue ("logging lib breaks everything")
| with a $MSFT logging library. Imagine having 100 threads each
| with their own logging buffer of 300MB. Needless to say it
| _annihilated_ our memory and our server crashed, even on the
| most expensive sku of Azure App Service.
| pests wrote:
| Brilliant strategy.
|
| Reminds me a litte of the oldtimers trick of adding a
| sleep(1000) somewhere so they could later come back and have
| some resources later, or if they needed a quick win with the
| client.
|
| Now cloud companies are using malloc(300000000) it to fake
| resource usage. /s
| RobertVDB wrote:
| Ah, the classic GLOG-induced service stall - brings back
| memories! I've seen similar scenarios where logging, meant to
| be a safety net, turns into a trap. Your 90-99% figure
| resonates with my experience. It's like opening a small window
| for fresh air and having a storm barrel in. We eventually had
| to balance between logging verbosity and system performance,
| kind of like a tightrope walk over a sea of unpredictable IO
| delays. Makes one appreciate the delicate art of designing
| logging systems that don't end up hogging the spotlight (and
| resources) themselves, doesn't it?
| tyingq wrote:
| I wonder if this being fixed will result in it displacing some
| notable amount of made-for-realtime hardware/software combos.
| Especially since there's now lots of cheap, relatively low power,
| and high clock rate ARM and x86 chips to choose from. With the
| clock rates so high, perfect real-time becomes less important as
| you would often have many cycles to spare for misses.
|
| I understand it's less elegant, efficient, etc. But sometimes
| commodity wins over correctness.
| foobarian wrote:
| Ethernet nods in vigorous agreement
| tuetuopay wrote:
| The thing is, stuff that require hard realtime cannot satisfy
| with "many cycles to spare for misses". And CPU cycles is not
| the whole story. A badly made task could lock down the kernel
| not doing anything useful. The point of hard realtime is
| "nothing cannot prevent this critical task from running".
|
| For automotive and aerospace, you really want the control
| systems to be able to run no matter what.
| tyingq wrote:
| Yes, there are parts of the space that can't be displaced
| with this.
|
| I'm unclear on why you put "many cycles to spare for misses"
| in quotes, as if it's unimportant. If a linux/arm (or x86)
| solution is displacing a much lower speed "real real time"
| solution, that's the situation...the extra cycles mean you
| can tolerate some misses while still being as granular as
| what you're replacing. Not for every use case, but for many.
| bee_rider wrote:
| It is sort of funny that language has changed to the point
| where quotes are assumed to be dismissive or sarcastic.
|
| Maybe they used the quotes because they were quoting you,
| haha.
| tuetuopay wrote:
| it's precisely why I quoted the text, to quote :)
| archgoon wrote:
| I'm pretty sure they were just putting it in quotes because
| it was the expression you used, and they thus were
| referencing it.
| tuetuopay wrote:
| You won't be saved from two tasks deadlocking with
| cycles/second. _this_ is what hard realtime systems are
| about. However, I do agree that not all systems have a real
| hard realtime requirements. But those usually can handle a
| non-rt kernel.
|
| As for the quotes, it was a direct citation, not a way to
| dismiss what you said.
| tremon wrote:
| I don't think realtime anything has much to do with mutex
| deadlocks, those are pretty much orthogonal concepts. In
| fact, I would make a stronger claim: if your "realtime"
| system can deadlock, it's either not really realtime or
| it has a design flaw and should be sent back to the
| drawing board. It's not like you can say "oh, we have a
| realtime kernel now, so deadlocks are the kernel's
| problem".
|
| Actual realtime systems are about workload scheduling
| that takes into account processing deadlines. Hard
| realtime systems can make guarantees about processing
| latencies, and can preemptively kill or skip tasks if the
| result would arrive too late. But this is not something
| that the Linux kernel can provide, because it is a system
| property rather than about just the kernel: you can't
| provide any hard guarantees if you have no time bounds
| for your data processing workload. So any discussion
| about -rt in the context of the Linux kernel will always
| be about soft realtime only.
| tuetuopay wrote:
| much agreed. I used deadlocks as an extreme example
| that's easy to reason about and straight to the point of
| "something independent of cpu cycles". something more
| realistic would be IO operations taking more time than
| expected. you would not want this to be blocking
| execution for hard rt tasks.
|
| In the case of the kernel, it is indeed too large to be
| considered hard realtime. Best case we can make it into a
| firmer realtime than it currently is. But I would place
| it nowhere near avionics flight calculators (like fly-by-
| wire systems).
| nine_k wrote:
| How much more expensive and power-hungry an ARM core would
| be, if it displaces a lower-specced core?
|
| I bet there are hard-realtime (commercial) OSes running on
| ARM, and the ability to use a lower-specced (cheaper,
| simpler, consuming less power) core may be seen as an
| advantage enough to pay for the OS license.
| lmm wrote:
| > How much more expensive and power-hungry an ARM core
| would be, if it displaces a lower-specced core?
|
| The power issue is real, but it might well be the same
| price or cheaper - a standard ARM that gets stamped out
| by the million can cost less than a "simpler"
| microcontroller with a smaller production run.
| zmgsabst wrote:
| What's an example of a system that requires hard real time
| and couldn't cope with soft real time on a 3GHz system having
| 1000 cycle misses costing 0.3us?
| lelanthran wrote:
| > What's an example of a system that requires hard real
| time and couldn't cope with soft real time on a 3GHz system
| having 1000 cycle misses costing 0.3us?
|
| Any system that deadlocks.
| LeifCarrotson wrote:
| We've successfully used a Delta Tau real-time Linux motion
| controller to run a 24 kHz laser galvo system. It's
| ostensibly good for 25 microsecond loop rates, and pretty
| intolerant of jitter (you could delay a measurement by a
| full loop period if you're early). And the processor is a
| fixed frequency Arm industrial deal that only runs at 1.2
| GHz.
|
| Perhaps even that's not an example of such a system, 0.3
| microseconds is close to the allowable real-time budget,
| and QC would probably not scrap a $20k part if you were off
| by that much once.
|
| But in practice, every time I've heard "soft real time"
| suggested, the failure mode is not a sub-microsecond miss
| but a 100 millisecond plus deadlock, where a hardware
| watchdog would be needed to drop the whole system offline
| and probably crash the tool (hopefully fusing at the tool
| instead of destroying spindle bearings, axis ball screws,
| or motors and gearboxes) and scrap the part.
| imtringued wrote:
| Sure, but this won't magically remove the need for dedicated
| cores. What will probably happen is that people will tell the
| scheduler to exclusively put non-premptible real time tasks on
| one of the LITTLE cores.
| binary132 wrote:
| I get the sense that applications with true realtime
| requirements generally have hard enough requirements that they
| cannot allow even the remote possibility of failure. Think
| avionics, medical devices, automotive, military applications.
|
| If you really need realtime, then you _really need_ it and
| "close enough" doesn't really exist.
|
| This is just my perception as an outsider though.
| calvinmorrison wrote:
| If you really need realtime, and you really actually need it,
| should you be using a system like Linux at all?
| refulgentis wrote:
| ...yes, after realtime support lands
| lumb63 wrote:
| A lot of realtime systems don't have sufficient resources
| to run Linux. Their hardware is much less powerful than
| Linux requires.
|
| Even if a system can run (RT-)Linux, it doesn't mean it's
| suitable for real-time. Hardware for real-time projects
| needs much lower interrupt latency than a lot of hardware
| provides. Preemption isn't the only thing necessary to
| support real-time requirements.
| skyde wrote:
| what kind of Hardware is considered to have "lower
| interrupt latency"? Is there some kind of Arduino board I
| could get that fit those lower interrupt latency required
| for real-time but still support things like Bluetooth?
| lumb63 wrote:
| Take a look at the Cortex R series. The Cortex M series
| still has lower interrupt latency than the A series, but
| lower processing power. I imagine for something like
| Bluetooth that an M is more than sufficient.
| refulgentis wrote:
| Sure but that was already mentioned before the comment I
| was replying to. Standard hardware not being great for
| realtime has nothing to do with hypothetical realtime
| Linux.
| rcxdude wrote:
| realtime just means execution time is bounded. It doesn't
| necessarily mean the latency is low. Though, in this
| sense RT-linux should probably be mostly thought of as
| low-latency linux, and the improvement in realtime
| guarantees is mostly in reducing the amount of things
| that can cause you to miss a deadline as opposed to
| allowing you to guarantee any particular deadline, even a
| long one.
| tyingq wrote:
| I'm guessing it's not that technical experts will be
| choosing this path, but rather companies. Once it's "good
| enough", and much easier to hire for, etc...you hire non-
| experts because it works _most_ of the time. I 'm not
| saying it's good, just that it's a somewhat likely outcome.
| And not for everything, just the places where they can get
| away with it.
| froh wrote:
| nah. when functional safety enters the room (as it does
| for hard real time) then engineers go to jail if they
| sign off something unsafe and people die because of that.
| since the challenger disaster there is an awareness that
| not listening to engineers can be expensive and cost
| lifes.
| epcoa wrote:
| And nothing stupid and easily preventable ever happened
| in aviation again
|
| https://en.wikipedia.org/wiki/Maneuvering_Characteristics
| _Au...
| synergy20 wrote:
| no you don't, you use a true RTOS instead.
|
| linux RTOS is at microseconds granularity but it still can
| not 100% guarantee it, anything in cache nature (L2 cache,
| TLB miss) are hard for hard real time.
|
| a dual kernel with xenomai could improve it, but it is not
| widely used somehow, only used in industrial controls I
| think.
|
| linux RT is great for audio, multimedia etc as well, where
| real-time is crucial, but not a MUST.
| froh wrote:
| > anything in cache nature (L2 cache, TLB miss) are hard
| for hard real time
|
| yup that's why you'd pin the memory and the core for the
| critical task. which, alas, will affect performance of
| the other cores and all other tasks. and whoosh there
| goes the BOM...
|
| which again as we both probably are familiar with leads
| to the SoC designs with a real time core microcontroller
| and a HPC microprocessor on the same package. which leads
| to the question how to architect the combined system of
| real-time microcontroller and compute power but soft real
| time microprocessor such that the overall system remains
| sufficiently reliable...
|
| oh joy and fun!
| snickerbockers wrote:
| Pretty sure most people who think they need a real-time
| thread actually don't tbh.
| rcxdude wrote:
| really depends on your paranoia level and the consequences
| for failure. soft to hard realtime is a bit of a spectrum
| in terms of how hard of a failure missing a deadline
| actually is, and therefore how much you try to verify that
| you will actually meet that deadline.
| cptaj wrote:
| Unless its just music
| itishappy wrote:
| It may not be safety critical, but remember that people can
| and will purchase $14k power chords to (ostensibly) improve
| the experience of listening to "just music".
|
| https://www.audioadvice.com/audioquest-nrg-dragon-high-
| curre...
| cwillu wrote:
| FWIW, a power chord is a _very_ different thing than a
| power cord.
| itishappy wrote:
| LOL, what a typo! Good catch!
| binary132 wrote:
| what if your analog sampler ruins the only good take you
| can get? What if it's recording a historically important
| speech? Starting to get philosophical here...
| duped wrote:
| Unless that music is being played through a multi kW
| amplifier into a stadium and an xrun causes damage to the
| drivers and/or audience (although, they should have hearing
| protection anyway).
| beiller wrote:
| Your talk of xrun is giving me anxiety. When I was
| younger I dreamed of having a linux audio effects stack
| with cheap hardware on stage and xruns brought my dreams
| crashing down.
| robocat wrote:
| xrun definition:
| https://unix.stackexchange.com/questions/199498/what-are-
| xru...
|
| (I didn't know the term, trying to be helpful if others
| don't)
| dripton wrote:
| You can divide realtime applications into safety-critical and
| non-safety-critical ones. For safety-critical apps, you're
| totally right. For non-critical apps, if it's late and
| therefore buggy once in a while, that sucks but nobody dies.
|
| Examples of the latter include audio and video playback and
| video games. Nobody wants pauses or glitches, but if you get
| one once in a while, nobody dies. So people deliver these on
| non-RT operating systems for cost reasons.
| binary132 wrote:
| This kind of makes the same point I made though -- apps
| without hard realtime requirements aren't "really realtime"
| applications
| duped wrote:
| The traditional language is "hard" vs "soft" realtime
| binary132 wrote:
| RTOS means hard realtime.
| pluto_modadic wrote:
| I sense that people will insist on their requirements
| being hard unnecessarily... and that the bug is the fault
| of it being on a near-realtime system instead of it being
| faulty even on a realtime one.
| tremon wrote:
| No -- soft realtime applications are things like video
| conferencing, where you care mostly about low latency in
| the audio/video stream but it's ok to drop the occasional
| frame. These are still realtime requirements, different
| from what your typical browser does (for example): who
| cares if a webpage is rendered in 100ms or 2s? Hard
| realtime is more like professional audio/video recording
| where you want hard guarantees that each captured frame
| is stored and processed within the alotted time.
| lll-o-lll wrote:
| > You can divide realtime applications into safety-critical
| and non-safety-critical ones.
|
| No. This is a common misconception. The distinction between
| a hard realtime system and a soft realtime system is simply
| whether missing a timing deadline leads to a) failure of
| the system or b) degradation of the system (but the system
| continues to operate). Safety is not part of it.
|
| Interacting with the real physical world often imposes
| "hard realtime" constraints (think signal processing).
| Whether this has safety implications simply depends on the
| application.
| jancsika wrote:
| Your division puts audio _performance_ applications in a
| grey area.
|
| On the one hand they aren't safety critical.
|
| On the other, I can imagine someone getting chewed out or
| even fired for a pause or a glitch in a professional
| performance.
|
| Probably the same with live commercial video compositing.
| wongarsu wrote:
| There is some amount of realtime in factory control where
| infrequent misses will just increase your reject rate in QA.
| abe_m wrote:
| Having worked on a number of "real time" machine control
| applications:
|
| 1) There is always a possibility that something fails to run
| by its due date. Planes crash sometimes. Cars won't start
| some times. Factory machinery makes scrap parts sometimes. In
| a great many applications, missing a real time deadline
| results in degraded quality, not end of life, or regional
| catastrophy. The care that must be taken to lower the
| probability of failure needs to be in proportion to the
| consequence of the failure. Airplanes have redundant systems
| to reduce (but not eliminate) possibility of failure, while
| cars and trucks generally don't.
|
| 2) Even in properly working real time systems, there is a
| tolerance window on execution time. As machines change modes
| of operation, the amount of calculation effort to complete a
| cycle changes. If the machine is in a warm up phase, it may
| be doing minimal calculations, and the scan cycle is fast.
| Later it may be doing a quality control function that needs
| to do calculations on inputs from numerous sensors, and the
| scan cycle slows down. So long as the scan cycle doesn't
| exceed the limit for the process, the variation doesn't cause
| problems.
| mlsu wrote:
| That is true, but generally not acceptable to a regulating
| body for these critical applications. You would need to
| design and implement a validation test to prove timing in
| your system.
|
| Much easier to just use an RTOS and save the expensive
| testing.
| vlovich123 wrote:
| But you still need to implement the validation test to
| prove that the RTOS has these requirements...
| blt wrote:
| How is your point 2) a response to any of the earlier
| points? Hard realtime systems don't care about variation,
| only the worst case. If your code does a single multiply-
| add most of the time but calls `log` every now and then,
| hard realtime requirement is perfectly satisfied if the
| bound on the worst-case runtime of `log` is small enough.
| abe_m wrote:
| I suppose it isn't, but I bristle when I see someone
| tossing around statements like "close enough doesn't
| really exist". In my experience when statements like that
| start up, there are people involved that don't understand
| variation is a part of every real process. My point is
| that if you're going to get into safety critical systems,
| there is always going to be some amount of variation, and
| there is always a "close enough", as there is never an
| "exact" in real systems.
| moffkalast wrote:
| I feel like at this point we have enough cores (or will soon,
| anyway) that you could dedicate one entirely to one process
| and have it run realtime.
| KWxIUElW8Xt0tD9 wrote:
| That's one way to run DPDK processes under LINUX -- you get
| the whole processor for doing whatever network processing
| you want to do -- no interruptions from anything.
| ajross wrote:
| > Think avionics, medical devices, automotive, military
| applications.
|
| FWIW by-device/by-transistor-count, the bulk of "hard
| realtime systems" with millisecond-scale latency requirements
| are just audio.
|
| The sexy stuff are all real applications too. But mostly we
| need this just so we don't hear pops and echos in our video
| calls.
| binary132 wrote:
| Nobody thinks Teams is a realtime application
| ajross wrote:
| No[1], but the people writing the audio drivers and DSP
| firmware absolutely do. Kernel preemption isn't a feature
| for top-level apps.
|
| [1] Actually even that's wrong: for sure there are teams
| of people within MS (and Apple, and anyone else in this
| space) measuring latency behavior at the top-level app
| layer and doing tuning all the way through the stack. App
| latency excursions can impact streams too, though ideally
| you have some layer of insulation there.
| lmm wrote:
| Like many binary distinctions, when you zoom in on the
| details hard-versus-soft realtime is really more of a
| spectrum. There's "people will die if it's late". "The line
| will have to stop for a day if it's late". "If it's late,
| it'll wreck the part currently being made". Etc.
|
| Even hard-realtime systems have a failure rate, in practice
| if not in theory - even a formally verified system might
| encounter a hardware bug. So it's always a case of tradeoffs
| between failure rate and other factors (like cost). If
| commodity operating systems can push their failure rate down
| a few orders of magnitude, that moves the needle, at least
| for some applications.
| JohnFen wrote:
| When I'm doing realtime applications using cheap, low-power,
| high-clockrate ARM chips (I don't consider x86 chips for those
| sorts of applications), I'm not using an operating system at
| all. An OS interferes too much, even an RTOS. I don't see how
| this changes anything.
|
| But it all depends on what your application is. There are a lot
| of applications that are "almost real-time" in need. For those,
| this might be useful.
| eisbaw wrote:
| Great to hear. However even if Linux the kernel is real-time,
| likely the hardware won't be due to caches and internal magic CPU
| trickery.
|
| Big complex hardware is a no-no for true real-time.
|
| That's why AbsInt and WCET tools mainly has simple CPU
| architectures. 8051 will truly live forever.
|
| btw, Zephyr RTOS.
| wholesomepotato wrote:
| Features of modern CPUs don't really prevent them from real
| time usage, afaik. As long as something is bounded and can be
| reasoned about it can be used to build a real time system. You
| can always assume no cache hits and alikes, maximum load etc
| and as long as you can put a bound on the time it will take,
| you're good to go.
| synergy20 wrote:
| mlock your memory, test with cache miss and cache
| invalidation scenarios will help, using no heap for memory
| allocation, but it's a bit hard
| jeffreygoesto wrote:
| Right. But still possible.
|
| https://www.etas.com/en/applications/etas-middleware-
| solutio...
| SAI_Peregrinus wrote:
| Exactly. "Real-time" is a misnomer, it should be called
| "bounded-time". As long as the bound is deterministic, known
| in advance, and guaranteed, it's "real-time". For it to be
| useful it also must be under some application-specific
| duration.
|
| The bounds are usually in CPU cycles, so a faster CPU can
| sometimes be used even if it takes more cycles. CPUs capable
| of running Linux usually have higher latency (in cycles) than
| microcontrollers, but as long as that can be kept under the
| (wall clock) duration limits with bounded-time it's fine.
| There will still be cases where the worst-case latency to
| fetch from DRAM in an RT-Linux system will be higher than a
| slower MCU fetching from internal SRAM, so RT-Linux won't
| take over all these systems.
| bloak wrote:
| So the things that might prevent you are:
|
| 1. Suppliers have not given you sufficient information for
| you to be able to prove an upper bound on the time taken.
| (That must happen a lot.)
|
| 2. The system is so complicated that you are not totally
| confident of the correctness of your proof of the upper
| bound.
|
| 3. The only upper bound that can prove with reasonable
| confidence is so amazingly bad that you'd be better off with
| cheaper, simpler hardware.
|
| 4. There really isn't a worst case. There might, for example,
| be a situation equivalent to "roll the dice until you don't
| get snake eyes". In networking, for example, sometimes after
| a collision both parties try again after a random delay so
| the situation is resolved eventually with probability one but
| there's no actual upper bound. A complex CPU and memory
| system might have something like that? Perhaps you'd be happy
| with "the probability of this operation taking more than 2000
| clock cycles is less than 10^-13" but perhaps not.
| dooglius wrote:
| System management mode is one example of a feature on modern
| CPUs that prevents real-time usage https://wiki.linuxfoundati
| on.org/realtime/documentation/howt...
| nraynaud wrote:
| I think it's really useful on 'big' MCU, like the raspberry pi.
| There exists an entire real time spirit there, where you don't
| really use the CPU to do any bit banging but everything is on
| time as seen from the outside. You have timers that receive the
| quadrature encoders inputs, and they just send interrupt when
| they wrap, the GPIO system can be plugged to the DMA, so you
| can stream the memory to the output pins without involving the
| CPU (again, interrupts at mid-buffer and empty buffer). You can
| stream to a DAC, stream from a ADC to memory with the DMA. A
| lot of that stuff bypasses the caches to get a predictable
| latency.
| stefan_ wrote:
| Nice idea but big chip design strikes again: on the latest
| Raspberry Pi, GPIO pins are handled by the separate IO chip
| connected over PCI Express. So now all your GPIO stuff needs
| to traverse a shared serial bus (that is also doing bulk
| stuff like say raw camera images).
|
| And already on many bigger MCUs, GPIOs are just separate
| blocks on a shared internal bus like AHB/APB that connects
| together all the chip IP, causing unpredictable latencies.
| 0xDEF wrote:
| >Big complex hardware is a no-no for true real-time.
|
| SpaceX uses x86 processors for their rockets. That small drone
| copter NASA put on Mars uses "big-ish" ARM cores that can
| probably run older versions of Android.
| ska wrote:
| Does everything runs on those CPUs though? Hard realtime
| control is often done on much simpler MCU at the lowest
| level, with oversight/planning for a high level system....
| zokier wrote:
| In short, no. For Ingenuity (the Mars2020 helicopter) the
| flight computer runs on pair of hard-realtime Cortex R5
| MCUs paired with a FPGA. The non-realtime Snapdragon SoC
| handles navigation/image processing duties.
|
| https://news.ycombinator.com/item?id=26907669
| ska wrote:
| That's basically what I expected, thanks.
| SubjectToChange wrote:
| _Big complex hardware is a no-no for true real-time._
|
| There are advanced real time cores like the Arm Coretex-R82. In
| fact many real time systems are becoming quite powerful due to
| the need to process and aggregate ever increasing amounts of
| sensor data.
| Aaargh20318 wrote:
| What does this mean for the common user? Is this something you
| would only enable in very specific circumstances or can it also
| bring a more responsive system to the general public?
| stavros wrote:
| As far as I can understand, this is for Linux becoming an
| option when you need an RTOS, so for critical things like
| aviation, medical devices, and other such systems. It doesn't
| do anything for the common user.
| SubjectToChange wrote:
| The Linux kernel, real-time or not, is simply too large and
| too complex to realistically certify for anything safety
| critical.
| ska wrote:
| For the parts of such systems that you would need an RTOS for
| this isn't really a likely replacement because the OS is way
| too complex.
|
| The sort of thing it could help with is servicing hardware
| that _does_ run hard realtime. For example, you have an RTOS
| doing direct control of a robot or medical device or
| whatever, and you have a UI pendant or the like that a user
| is interacting with. If linux on that pendant can make some
| realtime latency guarantees, you may be able to simplify
| communication between the two without risking dropping bits
| on the floor.
|
| Conversely, for the common user it could improve things like
| audio/video streaming, in theory but I haven't looked into
| details or how much trouble there is currently.
| ravingraven wrote:
| If by "common" user you mean the desktop user, not much. But
| this is a huge deal for embedded devices like industrial
| control and communication equipment, as their devs will be able
| to use the latest mainline kernel if they need real-time
| scheduling.
| fbdab103 wrote:
| My understanding is that real-time makes a system _slower_. To
| be real-time, you have to put a time allocation on everything.
| Each operation is allowed X budget, and will not deviate. This
| means if the best-case operation is fast, but the worst case is
| slow, the system has to always assume worst case.
| dist-epoch wrote:
| It could allow very low latency audio (1-2 ms). Not a huge
| thing, but nice for some audio people.
| andrewaylett wrote:
| RT doesn't necessarily improve latency, it gives it a fixed
| upper bound for _some_ operations. But the work needed to allow
| RT can definitely improve latency in the general case -- the
| example of avoiding synchronous printk() calls is a case in
| point. It should improve latency under load even when RT isn 't
| even enabled.
|
| I think I'm right in asserting that a fully-upstreamed RT
| kernel won't actually do anything different from a normal one
| unless you're actually running RT processes on it. The reason
| it's taken so long to upstream has been the trade-offs that
| have been needed to enable RT, and (per the article) there
| aren't many of those left.
| rcxdude wrote:
| the most common desktop end-user that might benefit from this
| is those doing audio work: latency and especially jitter can be
| quite a pain there.
| knorker wrote:
| I just want SCHED_IDLEPRIO to actually do what it says.
| deepsquirrelnet wrote:
| What a blast from the past. I compiled a kernel for Debian with
| RT_PREEMPT about 17-18 years ago to use with scientific equipment
| that needed tighter timings. I was very impressed at the
| latencies and jitter.
|
| I haven't really thought about it since then, but I can imagine
| lots of used cases for something like an embedded application
| with raspberry pi where you don't quite want to make the leap
| into a microcontroller running an RTOS.
| 0xDEF wrote:
| What do embedded real-time Linux people use for bootloader, init
| system, utilities, and C standard library implementation? Even
| Android that does not have real-time constraints ended up using
| Toybox for utilities and rolling their own C standard library
| (Bionic).
| rcxdude wrote:
| You aren't likely to need to change a lot of these: the whole
| point is basically making it so that all that can run as normal
| but won't really get in the way of your high-priority process.
| It's just that your high-priority process needs to be careful
| not to block on anything that might take too long due to some
| other stuff running. In which case you may need to avoid
| certain C standard library calls, but not replace it entirely.
| salamanderman wrote:
| I had a frustrating number of job interviews in my early career
| where the interviewers didn't know what realtime actually was.
| That "and predictable delay" concept from the article frequently
| seemed to be lost on many folks, who seemed to think realtime
| just meant fast, whatever that means.
| mort96 wrote:
| I would even remove the "minimum" part altogether; the point of
| realtime is that operations have predictable upper bounds. That
| might even mean slower average cases than in non-realtime
| systems. If you're controlling a car's braking system, "the
| average delay is 50ms but might take up to 80ms" might be
| acceptable, whereas "the average delay is 1ms but it might take
| arbitrarily long, possibly multiple seconds" isn't.
| ska wrote:
| The old saying "real time" /= "real fast". Hard vs "soft"
| realtime muddies things a bit, but I think it's probably the
| majority of software developers don't really understand what
| realtime actually is either.
| biorach wrote:
| Thomas Gleixner looks like a farmer.
| NalNezumi wrote:
| Slightly tangential, but does anyone know good learning material
| to understand real-time (Linux) kernel more? For someone with
| rudimentary Linux knowledge.
|
| I've had to compile&install real-time kernel as a requirement for
| a robot arm (franka) control computer. It would be nice to know a
| bit more than just how to install the kernel.
| ActorNightly wrote:
| https://www.freertos.org/implementation/a00002.html
|
| Generally, having experience with Greenhills in a previous job,
| for personal projects like robotics or control systems I would
| recommend programming a microcontroller directly rather than
| dealing with SoC with RTOS. Modern STM32s with Cortex chips
| have enough processing power to run pretty much anything.
| alangibson wrote:
| Very exiting news for those of us building CNC machines with
| LinuxCNC. The end of kernel patches is nigh!
| Tomte wrote:
| OSADL runs a cool QA farm: https://www.osadl.org/OSADL-QA-Farm-
| Real-time.linux-real-tim...
| Animats wrote:
| QNX had this right decades ago. The microkernel has upper bounds
| on everything it does. There are only a few tens of thousands of
| lines of microkernel code. All the microkernel does is allocate
| memory, dispatch the CPU, and pass messages between processes.
| Everything else, including drivers and loggers, is in user space
| and can be preempted by higher priority threads.
|
| The QNX kernel doesn't do anything with strings. No parsing, no
| formatting, no messages.
|
| Linux suffers from being too bloated for real time. Millions of
| lines of kernel, all of which have to be made preemptable. It's
| the wrong architecture for real time. So it took two decades to
| try to fix this.
| vacuity wrote:
| For a modern example, there's seL4. I believe it does no
| dynamic memory allocation. It's also formally verified for
| various properties. (Arguably?) its biggest contribution to
| kernel design is the pervasive usage of capabilities to
| securely but flexibly export control to userspace.
| adastra22 wrote:
| Capabilities are important, but I don't think that was
| introduced by seL4. Mach (which underlies macOS) has the same
| capability-based system.
| vacuity wrote:
| I didn't say seL4 introduced capabilities. However, to my
| knowledge, seL4 was the first kernel to show that
| _pervasive_ usage of capabilities is both feasible and
| beneficial.
| bregma wrote:
| The current (SDP 8) kernel has 15331 lines of code, including
| comments and Makefiles.
| gigatexal wrote:
| QNX is used in vehicle infotainment systems no? Where else?
|
| I'm not bothered by the kernel bloat. There's a lot of dev time
| being invested in Linux and while the desktop is not as much of
| a priority as say the server space a performant kernel on
| handhelds and other such devices and the dev work to get it
| there will benefit the desktop users like myself.
| bkallus wrote:
| I went to a conference at GE Research where I spoke to some
| QNX reps from Blackberry for a while. Seemed like they were
| hinting that some embedded computers in some of GE'S
| aerospace and energy stuff relies on QNX.
| lmm wrote:
| > QNX is used in vehicle infotainment systems no? Where else?
|
| A bunch of similar embedded systems. And blackberry, if
| anyone's still using them.
| tyfon wrote:
| It was used in my old toyota avensis from 2012. The
| infotainment was so slow you could measure performance in
| seconds pr frame instead of frames pr second :)
|
| In the end, all I could practically use it for was as a
| bluetooth audio connector.
| gigatexal wrote:
| > QNX had this right decades ago. The microkernel has upper
| bounds on everything it does. There are only a few tens of
| thousands of lines of microkernel code. All the microkernel
| does is allocate memory, dispatch the CPU, and pass messages
| between processes. Everything else, including drivers and
| loggers, is in user space and can be preempted by higher
| priority threads.
|
| So much like a well structured main method in a C program or
| other C like language where main just orchestrates the calling
| of other functions and such. In this case main might initialize
| different things where the QNX kernel doesn't but the idea or
| general concept remains.
|
| I'm no kernel dev but this sounds good to me. Keeps things
| simple.
| vacuity wrote:
| Recently, I've been thinking that we need a microkernel
| design in applications. You have the core and then services
| that can integrate amongst each other and the core that
| provide flexibility. Like the "browser as an OS" kind of
| things but applied more generally.
| galdosdi wrote:
| Yes! This reminds me strongly of the core/modules
| architecture of the apache httpd, as described by the
| excellent O'Reilly book on it.
|
| The process of serving an HTTP request is broken into a
| large number of fine grained stages and plugin modules may
| hook into any or all of these to modify the input and
| output to each stage.
|
| The same basic idea makes it easy to turn any application
| concept into a modules-and-core architecture. From the day
| I read (skimmed) that book a decade or two ago this pattern
| has been burned into my brain
| the8472 wrote:
| For an example how far the kernel goes to get log messages out
| even on a dying system and how that's used in real deployments:
|
| https://netflixtechblog.com/kubernetes-and-kernel-panics-ed6...
| rwmj wrote:
| About printk, the backported RT implementation of printk added to
| the RHEL 9.3 kernel has deadlocks ...
| https://issues.redhat.com/browse/RHEL-15897 &
| https://issues.redhat.com/browse/RHEL-9380
| w10-1 wrote:
| There is no end game until there are end users beating on the
| system. That would put the 'real' in 'real-time'.
|
| But who using a RTOS now would take the systems-integration
| cost/risk of switching? Would this put Android closer to Metal
| performance?
___________________________________________________________________
(page generated 2023-11-16 23:00 UTC)