Title: Using Netdata on NixOS and connecting to Netdata cloud
       Author: Solène
       Date: 16 September 2022
       Tags: nixos monitoring netdata cloud
       Description: In this article, you will learn about Netdata monitoring
       tool, how to use it on NixOS and how to connect it to the free Netdata
       privacy friendly cloud solution
       
       # Introduction
       
       I'm still playing with monitoring programs, and I've been remembered
       about Netdata.  What an improvement over the last 8 years!
       
       This tutorial explains how to get Netdata installed on NixOS, and how
       to register your node in Netdata cloud.
       
 (HTM) Netdata GitHub project page
 (HTM) Netdata live demo
       
       # What's Netdata?
       
       This program is a simple service to run on a computer, it will
       automatically gather a ton of metrics and make them easily available
       over the local TCP port 19999.  You just need to run Netdata and
       nothing else, and you will have every metrics you can imagine from your
       computer, and some explanations for each of them!
       
       That's pretty cool because Netdata is very efficient, it draws nearly
       no CPU while gathering a few thousands metrics every few seconds, and
       is memory efficient and can be constrained to a dozen of megabytes.
       
       While you can export its metrics to something like graphite or
       Prometheus, you lose the nice display which is absolutely a blast
       compare to Grafana (in my opinion).
       
       Update: as pointed out by a reader (thanks!), it's possible to connect
       Netdata instances to only one used for viewing metrics.  I'll
       investigate this soon.
       
 (HTM) Netdata documentation about streaming.
       
       Netdata also added some machine learning anomaly detection, it's simple
       and doesn't use many resources or require a GPU, it only builds
       statistical models to be able to report if some metrics have an unusual
       trend.  It takes some time to gather enough data, and after a few days
       it's starting to work.
       
       # Installing Netdata on NixOS
       
       As usual, it's simple, add this to your NixOS configuration and
       reconfigure the system.
       
       ```
         services.netdata = {
           enable = true;
       
           config = {
             global = {
               # uncomment to reduce memory to 32 MB
               #"page cache size" = 32;
       
               # update interval
               "update every" = 15;
             };
             ml = {
               # enable machine learning
               "enabled" = "yes";
             };
           };
         };
       ```
       
       You should have Netdata dashboard available on http://localhost:19999 .
       
       ## Streaming mode
       
       Here is a simple configuration on NixOS to connect a headless node
       without persistency to send all on a main Netdata server storing data
       but also displaying them.
       
       You need to generate an UUID with uuidgen, replace UUID in the text
       with the result.  It can be per system or shared by multiple Netdata
       instances.
       
       My networks are 10.42.42.0/24 and 10.43.43.0/24, I'll allow everything
       matching 10.* on the receiver, I don't open port 19999 on a public
       interface.
       
       ### Senders
       
       ```
         services.netdata.enable = true;
         services.netdata.config = {
             global = {
                 "default memory mode" = "none"; # can be used to disable local data storage
             };
         };
         services.netdata.configDir = {
           "stream.conf" = pkgs.writeText "stream.conf" ''
             [stream]
               enabled = yes
               destination = 10.42.42.42:19999
               api key = UUID
             [UUID]
               enabled = yes
           '';
         };
       ```
       
       ### Receiver
       
       ```
         networking.firewall.allowedTCPPorts = [19999];
         services.netdata.enable = true;
         services.netdata.configDir = {
           "stream.conf" = pkgs.writeText "stream.conf" ''
             [UUID]
               enabled = yes
               default history = 3600
               default memory mode = dbengine
               health enabled by default = auto
               allow from = 10.*
           '';
         };
       ```
       
       
       # Netdata cloud
       
       Netdata company started a "cloud" offer that is free, but they plan to
       keep it free but also propose more services for paying subscribers. 
       The free plan is just a convenience to see metrics from multiple nodes
       at the same place, they don't store any metrics apart metadata (server
       name, OS version, kernel, etc..), when you look at your metrics, they
       just relay from your server to your web browser without storing the
       data.
       
       The free cloud plan offers a correlating feature, but I still didn't
       have the opportunity to try it, and also email alerting when an alarm
       is triggered.
       
 (HTM) Netdata cloud website
 (HTM) Netdata cloud data privacy information
       
       ## Adding a node
       
       The official way to connect a Netdata agent to the Netdata cloud is to
       use a script downloaded on the internet and run it with some parameter.
       
 (HTM) Connecting a Linux agent
       
       I strongly dislike this method as I'm not a huge fan of downloading
       script to run as root that are not provided by my system.
       
       When you want to add a new node, you will be given a long command line
       and a token, keep that token somewhere.  NixOS Netdata package offers a
       script named `netdata-claim.sh` (which seems to be part of Netdata
       source code) that will generate a pair of RSA keys, and look for the
       token in a file.
       
 (HTM) Netdata data page: Add a node
       
       Once you got the token, we will claim it to associate it to a node:
       
       1. create /var/lib/netdata/cloud.d/token and write the token in it
       2. run nix-shell -p netdata --run "netdata-claim.sh" as root
       3. your node should be registered in Netdata cloud
       
       # Conclusion
       
       Netdata is really a wonderful tool, ideally I'd like it to replace all
       the Grafana + storage + agent stack, but it doesn't provide persistent
       centralized storage compatible with its dashboard.  I'm going to
       experiment with their Netdata cloud service, I'm not sure if it would
       add value for me, and while they have a very correct data privacy
       policy, I prefer to self-host everything.