# Dynamic Nodes

## Overview

Starting in Slurm 22.05, nodes can be dynamically added and removed from Slurm.

## Dynamic Node Communications

For regular, non-dynamically created nodes, Slurm knows how to communicate with nodes by reading in the slurm.conf. This is why it is important for a non-dynamic setup that the slurm.conf is synchronized across the cluster. For dynamically created nodes, The controller automatically grabs the node's NodeAddr and NodeHostname for dynamic slurmd registrations. The controller then passes the node addresses to the clients so that they communicate, and even fanout, to other nodes.

## Slurm Configuration

MaxNodeCount=#
  Set to the number of possible nodes that can be active in a system at a time. See the slurm.conf man page for more details.
SelectType=select/cons_tres
  Dynamic nodes are only supported with cons_tres.

### Partition Assignment

Dynamic nodes can be automatically assigned to partitions at creation by using the partition's nodes ALL keyword or NodeSets and specifying a feature on the nodes.

e.g.

    Nodeset=ns1 Feature=f1
    Nodeset=ns2 Feature=f2

    PartitionName=all  Nodes=ALL Default=yes
    PartitionName=dyn1 Nodes=ns1
    PartitionName=dyn2 Nodes=ns2
    PartitionName=dyn3 Nodes=ns1,ns2

## Creating Nodes

Nodes can be created two ways:

1. Dynamic slurmd registration

     Using the slurmd -Z and --conf options a slurmd will register with the controller and will automatically be added to the system.

     e.g.

         slurmd -Z --conf "RealMemory=80000 Gres=gpu:2 Feature=f1"

2. scontrol create NodeName= ...

     Create nodes using scontrol by specifying the same NodeName line that you would define in the slurm.conf. See slurm.conf man page for node options. Only State=CLOUD and State=FUTURE are supported. The node configuration should match what the slurmd will register with (e.g. slurmd -C) plus any additional attributes.

     e.g.

         scontrol create NodeName=d[1-100] CPUs=16 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=31848 Gres=gpu:2 Feature=f1 State=cloud

## Deleting Nodes

Nodes can be deleted using scontrol delete nodename=<nodelist>. Nodes can only be deleted if they have no jobs running on them and aren't part of a reservation.

## Limitations

1. When dynamic nodes are added to Slurm they will potentially be alphabetically out of order internally — leading to suboptimal job allocations if node names represent topology of the nodes — until a reconfigure or a restart is performed. If using a Topology Plugin, the topology will also be out of date until a reconfigure/restart is performed.
2. The route/toplogy plugin is not supported since dynamic nodes require that the Fanout/Treewidth be disabled.

Last modified 15 August 2023