[HN Gopher] How to add eBPF observability to your product
___________________________________________________________________
How to add eBPF observability to your product
Author : mrry
Score : 102 points
Date : 2021-07-03 16:50 UTC (6 hours ago)
(HTM) web link (brendangregg.com)
(TXT) w3m dump (brendangregg.com)
| nyellin wrote:
| This works great if you control the servers or have a uniform
| fleet. I used to work at a security vendor which sold ebpf based
| software that ran as a daemonset on customer kubernetes clusters.
| Compiling ebpf bytecode on the customer's hosts wasn't an option.
|
| Iirc the real challenge was with writing kprobe ebpf functions
| that access native structs. I don't think we ever found a good
| solution for that because you need kernel headers for each
| machine which we didn't have.
|
| Of course if I'm missing something obvious, do tell!
|
| (I'm the same person who left this comment on Brendan's site as
| well. Not a random copy paster)
| boomskats wrote:
| I've been on and off playing with the same scripts described in
| this article for the last few months (installable as bcc-tools
| under Fedora). While some of them are a little unstable or need a
| bit of Python version handholding in fc34, I've been blown away
| with the data I've been able to get with seemingly negligible
| overhead. I was also really surprised to find them in the default
| repos, and how relatively few people reference them when
| discussing other eBPF tooling (e.g. Pixie).
|
| As someone who has been building observability software for the
| last few years I'm ridiculously excited about eBPF. Looking
| forward to seeing what the Netflix dashboards mentioned in the
| post look like & the data pipelines that support them.
| alexchamberlain wrote:
| According to Wikipedia, BPF stands for [Berkeley Packet
| Filter][1] and can be used to observe packets sent by a process.
|
| [1]: https://en.m.wikipedia.org/wiki/Berkeley_Packet_Filter
| ww520 wrote:
| eBPF is more capable of BPF. It's a general hooking mechanism
| in the Linux kernel. Think Aspect-Oriented Programming, but in
| the kernel. The biggest benefit I think is that it can
| intercept and modify function calls in many places, not just
| the kernel. Intercepting/filtering bad data at the network
| driver level is much more efficient than letting the data
| reaching the kernel code.
| yjftsjthsd-h wrote:
| Technically, yes, but in practice not really; eBPF started as
| an extension of the packet filtering framework and then got
| generalized to be a nearly-universal
| debugging/monitoring/tracing system that's only sometimes used
| for packets. https://ebpf.io/ is a decent starting point for
| reading.
| hn_throwaway_99 wrote:
| My biggest pet peeve is when authors don't just start with a
| 1-2 sentence outline of the topic that they're discussing,
| and instead go directly into acronym soup. And I've been in
| software development for a long time and have never heard of
| BPF/eBPF before.
|
| Thank you for your explanation.
| mhh__ wrote:
| Brendan Gregg literally wrote the book on this kind of
| stuff, if you search for any of this tracing stuff it comes
| up with his website anyway.
| enedil wrote:
| Brendan Gregg is the author of eBPF, and he writes on his
| blog. It is fair to assume a basic knowledge of his works
| when reading his website.
| bch wrote:
| Maybe I'm misunderstanding what you mean by "author", but
| BPF was invented by Steven McCanne, and Van Jacobson in
| 1992[0], and tweaked over the years by various developers
| and operating systems. It looks[1] like the Linux _e_ BPF
| work was initiated by Alexei Starovoitov, and came to be
| exposed to users c. 2014.
|
| [0] https://en.wikipedia.org/wiki/Berkeley_Packet_Filter
|
| [1] https://lwn.net/Articles/740157/
| Ristovski wrote:
| Brendan Gregg is the author of most of the utilities, not
| bpf/ebpf itself. You can find a non-exhaustive list here:
| http://www.brendangregg.com/ebpf.html
| bch wrote:
| I get that, and appreciate he's productive and sharing
| his skill. I wasn't sure though if the long history of
| other people's hard work wasn't being swept up in
| appreciation for 'brendangregg
| alexchamberlain wrote:
| Thanks for explaining that - I was totally missing the point
| to be honest.
| yjftsjthsd-h wrote:
| Yeah, the name is kinda unfortunate in the modern context,
| but in fairness it's easy to not expect that a packet
| filtering framework will turn out to be the basis for a
| universal kernel tracing system...
| puptelpete wrote:
| For me, BPF is the most exciting development in the kernel
| world for ages.
|
| Brendan Gregg, the author of the posted blog entry, works with
| BPF on observability at Netflix and delivered a keynote at
| UbuntuMasters 2019. The video is on his blog is a great intro.
| [0]
|
| I've been watching BPF for a couple of years now, and it seems
| slow on the uptake, but I hope Gregg is right that people will
| eventually start writing new drivers, firewalls, observability
| and security tools, loaded from userspace, but all running
| safely in a kernel vm, maybe even written in new async-first
| programming languages.
|
| [0] https://brendangregg.com/blog/2019-12-02/bpf-a-new-type-
| of-s...
| WillDaSilva wrote:
| And eBPF stands for extended Berkeley Packet Filter:
|
| > eBPF is an extended BPF JIT virtual machine in the Linux
| kernel
|
| It's really quite an amazing tool, especially since you can use
| it to run sandboxed programs in kernel without changing kernel
| source or loading a kernel module. These programs can be
| written using a limited dialect of C.
|
| Some examples of its use can be found here:
| https://github.com/iovisor/bcc/tree/master/examples
___________________________________________________________________
(page generated 2021-07-03 23:00 UTC)