Managing remote servers with salt-ssh

I love using Salt to configure and manage servers at scale. Typically, a Salt master server manages an army of servers which are on the same private network. A salt-minion daemon runs on each client server and communicates with the master. Sometimes you need to manage a server which is outside of your internal network, such as a marketing web server running on a service like AWS LightSail, Linode, or Digital Ocean. That’s when salt-ssh comes in handy.

Master Configuration

On most distributions, you will need to install package salt-ssh on the master server, even if you have salt-master installed.

Edit /etc/salt/roster and add the host config as shown below. We’ll use the master’s default ssh key, which is stored in /etc/salt/pki/master/ssh/ You can specify a custom key pair in another path by adding option priv in the roster file.

some-host-name:
  host: 203.0.113.1
  user: salt
  sudo: True
  tty: True

Client Configuration

The client does NOT need to have a minion installed, so I will use the term “client” instead of “minion.” This is one of the strengths of salt-ssh. You do need to add a user that can connect via SSH between the master and client. Run the following commands as a superuser:

useradd salt
sudo -u salt mkdir /home/salt/.ssh
sudo -u salt vi /home/salt/.ssh/authorized_keys
# Paste in public key
sudo -u salt chmod -R og-rwx /home/salt/.ssh/

You also need to give the user permission to run sudo commands without a password:

usermod -G wheel salt
visudo -f /etc/sudoers.d/salt

Enter the following line, and save the file:

salt ALL=(ALL) NOPASSWD:ALL

Testing SSH and salt-ssh Access

Test basic connectivity, sudo access, and finally the salt-ssh configuration:

ssh -i /etc/salt/pki/master/ssh/salt-ssh.rsa salt@18.216.140.168
ssh -i /etc/salt/pki/master/ssh/salt-ssh.rsa salt@18.216.140.168 sudo ls -l /root/
salt-ssh -t -w 'lkm-fw-01' test.ping

Next Steps

Add the client to the Topfile, using the hostname that you specified in the roster file. Now, you should be able to apply Salt states. Note that you might need to use some more sophisticated matching within the topfile if some of your default states assume that the host is on an internal network. You should expect salt-ssh to run more slowly than when running a minion on the client.

Issues with Client Versions

There can be issues when using salt-ssh to manage a host that has a very different operating system from the salt-master. For example, you get the following error when trying to manage older Debian and Ubuntu hosts (including Ubiquiti EdgeRouters running EdgeOS, which is based on Debian 7.11) from a CentOS/RedHat 7 master:

      File "/var/tmp/.salt_be47b3_salt/py2/tornado/netutil.py", line 38, in <module>
         import certifi
ImportError: No module named certifi

The error occurs because the system Python interpreter on the target doesn’t have the required package called certifi. The workaround is to install the certifi package on the RedHat/CentOS 7 master so that it can be bundled with the Python environment that is shipped to the client:

yum install python2-certifi
salt-ssh -t -w 'lkm-fw-01' test.ping

Be sure to use the -t and -w flags used with salt-ssh, which force it to clear the cache on both master and client, or you will continue to get the same error because it isn’t rebuilding the cached files.

References

  1. https://docs.saltstack.com/en/latest/topics/ssh/
  2. https://www.linode.com/docs/applications/configuration-management/configure-and-use-salt-ssh/