Dynamic node groups with Salt

Salt is a great choice for a configuration management system. I have found that it scales well, both in terms of number of nodes, and complexity of configurations. However, I have found one aspect of Salt that remains under-developed: Salt doesn’t have great support for custom groupings of minions. Minion groups (which could also be called node groups, host groups, instance groups, etc.) allow you to target a predefined group of minions.

Use Case

In order for an information technology architecture to be considered scalable and resilient, critical services must be provided by clusters of resources. No single point of failure should take a service offline. Sometimes, it’s possible to use naming conventions to denote resources a cluster, and use Salt’s shell-style globbing to match the minions in the cluster. For example, to apply states to Staging vs Production PHP servers:

salt 'stgphp-*' saltutil.sync_all
salt 'php-*' saltutil.sync_all

Name globbing fails in more complex situations. For example, many cloud providers can dynamically scale a group of resources, deleting resources when demand is low, and creating resources when demand rises. You may not be able to control the naming of these auto-created resources. The resources may be created across availability zones or regions, and these groupings may not be explicit in resource naming. We need a way to dynamically assign a Salt minion to a group that can be targeted with Salt commands.

Node Groups

Salt includes a feature called nodegroups, but it has several limitations that make it less than useful. First, you have to define node groups in /etc/salt/master and restart the master daemon for changes to take effect. Here’s an example:

nodegroups:
  production-web-servers-east:
    - php-5hvd.c.my-project.internal
    - php-c5j9.c.my-project.internal
    - php-ok99.c.my-project.internal
    - php-ckhj.c.my-project.internal

This has to be a manual process. If you had an automated process that could return information about host groupings, you wouldn’t want to give it permission to write to your Salt master file. Further, shortcuts like name globbing don’t work in the master file, so you have to spell out the exact minion ID. I conclude that node groups are onlyh useful for infrastructure that is manually managed.

Grain Targeting

Salt allows you to target minions based on grains. Grains are key:value pairs which are set on each minion. Many useful grains are predefined by Salt, and you can add your own. To see the grains defined on a minion:

salt 'minion-id' grains.items

If your cloud provider or virtualization infrastructure supports resource metadata, and that metadata contains the group for a resource, you might be able to find a predefined grain that contains the group name. For example, if you have a grain called instance_group, you can target minions by that grain:

salt -G 'instance_group:production-web-servers-east' test.version

If that grain doesn’t exist, you can define it like this:

salt 'php-5hvd.*' grains.append instance_group production-web-servers-east

It’s still a manual step, but much more amenable to automation than editing the Salt master configuration file. In a future post, I’ll show you my next step towards automating the grain definitions to define groups of minions.