# Interview with Ilan Rabinovitch by Seth Kenlon Ilan Rabinovitch is well-known to anyone who has done the conference circuit around Southern California. He's a helpful and friendly guy who I met once at a [BarCamp](http://barcamp.org/w/page/402984/FrontPage) years ago, and more than once encountered on IRC, and often ended up getting great tech tips from him by sheer proximity. He's speaking at this year's [Linux Fest NorthWest](https://www.linuxfestnorthwest.org/2016/sessions/monitoring-101-finding-signal-noise), training attendees on system monitoring. I spoke with him about what he'd be covering: **Anyone starting out in System Administration hears a lot about "monitoring". Broadly speaking, what is "monitoring" and how's it done?** As systems administrators and application developers, we build and deploy services that our colleagues or users depend on. Monitoring is the practice of observing and recording the behavior of these services to ensure they are functioning as we expect. Monitoring allows us to receive meaningful, automated alerts for potential problems and ongoing failures. It also allows us to quickly investigate and get to the bottom of issues or incidents once they occur. **What should I be monitoring? Is it just my network? or are there other things that I should be looking at? and will your talk address those things?** In short, monitor everything. Collecting data is cheap, but not having it when you need it can be expensive, so you should instrument everything, and collect all the useful data you reasonably can. So yes, you want data about your network, but also about your operating systems, your applications, and even business metrics. Individually, they might be interesting. Combined, they can help you tell the full story of why an incident occurs in your environment. You never want to be in a situation where you cannot explain why you experienced an outage or service degradation. That being said, the specifics of what to monitor and alert on will differ from environment to environment. What's important to you and your application may not be critical for me. During the session at Linuxfest NorthWest, we'll review some frameworks and methods for categorizing your metrics, and identifying what is important to you and your business. **A focus of your talk at Linux Fest Northwest is separating "signal from noise". The more you monitor, the more noise you find you have to sift through. Without copy-pasting your entire talk, what are some of the tactics you propose a sys admin takes to mitigate that?** It's important to distinguish between monitoring and alerting. Just because you collect a piece of data for trending or investigating purposes doesn't mean you need to page a team member every time it moves. If we're paging someone, it should be urgent and actionable every single time. As soon as you start questioning the validity of alerts and whether they require a response you are going to start picking and choosing when to respond and investigate alerts from your monitoring. This is called *pager fatigue* and significantly undermines your monitoring goals. You primarily want to focus your alerting around [work metrics](https://www.datadoghq.com/blog/monitoring-101-alerting): metrics that quantify the work or useful output from your environment. These represent the top-level health of your systems and services and are the true indicator of whether your systems are performing correctly. Your users will never call you to say “CPU is high”, but they might complain about slowing responses on your APIs or the service being down entirely. So why are you waking up engineers about high cpu usage? **Your talk is not tool-specific, but what are some of your personal favourite monitoring tools?** I'd have to say [Datadog](https://www.datadoghq.com) is my favorite tool at the moment! Not because they're my employer (it's actually the other way around. The reason I joined Datadog this past summer was due to how much I loved using their products as a customer). We're a mix of a hosted service and open source agent that runs on your servers to collect metrics. We tend to focus on environments with dynamic infrastructure (containers, cloud and auto-scaling or scheduled workloads), as well as aggregating metrics from a number of sources including other monitoring systems. The open source world has seen some great developments and improvements in our toolsets in recent years. While Nagios and Cacti or Ganglia have been the workhorses of the open source monitoring stack for the better part of the last 20 years, we now have a number of new tools such as Graphite and Graphana for time series data, ELK for log management, and much more. Which tool you pick to form your monitoring stack will depend on your environment, staff, and scaling needs. The [monitoringsucks project](https://github.com/monitoringsucks/tool-repos) offers a great overview of available tools. Brendan Gregg also has some amazing resources for understanding Linux system performance on his blog, I especially like [the Linux Observability slide](http://www.brendangregg.com/Perf/linux_observability_tools.png). **One thing that intimidates people from actively monitoring systems, I think, is the setup required, and learning the new tools. How long did it take you to get comfortable with monitoring, and the tools that go along with it?** I’ve been working with some form of monitoring for over 15 years, but I wouldn’t say I’ve mastered it. Monitoring, like other focuses of engineering, is an ongoing learning experience. Each time I build a new environment or system, it has different requirements either because of new technologies or new needs of the business. With that in mind, with each project I get to re-evaluate what metrics are most important and how best to collect them. **Another thing that intimidates people from monitoring is the fear that they won't know how to diagnose an issue when they find it. What do you do when you see an issue that raises a red flag, but you have no idea how to go about solving it (or even understanding what needs solving)?** Burying one’s head in the sand is never a solution. Even if you don’t know how to respond, it’s much better to be aware that your service or systems have failed, than to find out out it when irate users or customers call. With that in mind: start simple. Find the metrics and checks that tell you if your service is online and performing the way your users expect. Once you’ve got that in place, you can expand your coverage to other types of metrics that might deepen visibility into how your services interact, or of their underlying systems. **How do you keep from falling into a false sense of security once your monitoring system is up and running?** Post-mortems and incident reviews are great learning opportunities. Use them to learn from any mistakes, as well as to identify areas in your environment where you would benefit from more visibility. Did your monitoring detect the issue or did a user? Is there a leading indicator that might have brought your attention to the issue before it impacted our business? **What's your background as a professional geek? how did you get started in the tech business, and what do you do?** I got my start tinkering with a hand-me-down computer my parents gave my siblings when they upgraded the computer they used for their small business. I did quite a bit of tinkering over the years, but my interests got a jump start when I started playing with Linux with the goal of building a home router. I quickly found myself attending user group meetings with groups like [SCLUG](www.sclug.org), [UCLA LUG](http://linux.ucla.edu), and others. The skills I picked up at LUGs and in personal projects helped me pick up consulting / entry level sysadmin work. Since then, I’ve spent many years as SysAdmin, then later leading infrastructure automation and tooling teams for large web operations like Edmunds.com and Ooyala. Most recently, I’ve had the opportunity to combine my interest in large scale systems with my open source community activities as Director of Technical Community at Datadog. **You clearly do a lot with open source, and for the open source community. Why's open source important to you?** The Open Source community and Linux helped me get my start in technology. The open nature of Linux and other software made it possible me to dive in and learn about how everything fit together. Then LUGs and other community activities were always there to help if I ever got stuck. It’s important to me to stay active the community and give back so that others can have similar experiences. **Wait a minute, aren't you that one guy from SCaLE? How did you get involved with SCaLE and how's it going?** Indeed! I'm one of the co-founders and the current conference chair of [SCaLE](https://www.socallinuxexpo.org). SCaLE started back in 2002 as a 1 day event focused on open source and free software at the University of Southern California. We started SCALE at time where there were very few tech events in the Southern California area. Additionally the available FOSS/Linux focused events were primarily organized as a commercial ventures (eg Linux World). Between geography, cost and in some cases age requirements we found it was quite difficult to see developer lead sessions about the open source projects we were interested in. In the spirit of open source we decided to scratch our own itch and build a place where we could attend the sessions we interested in seeing. Naively, we thought "how hard could it be to start a conference?" Thus SCALE was born. We are now launching planning for our 15th year of SCALE, which will be held March 2-5, 2017 at the Pasadena Convention Center. While the event has grown from just a few hundred our first year to 3200 attendees, we've made a strong effort to keep our community feel. We like to think we've succeeded in meeting our original goals of bringing community, academia, and business together under a single, affordable, and accessible conference. **In that case, I have to ask: what distribution of Linux do you use, and what's your current desktop or Window Manager?** My home machine runs Ubuntu, so Unity on the desktop. My personal servers tend to be Debian based. At Datadog, we primarily use Ubuntu across our infrastructure, although we work with a wide set of Linux distros to ensure our open source agent works well across the board. Our docker containers start off from a Debian image.