Manage SSL using saltstack
Keeping track of where all of your SSL certificates are installed can be a time consuming task, and one that can get you into some hot water if not done properly or in a timely fashion. Causing a production outage on account of letting a certificate expire is never good. Neither is having to replace your certificate on a large group of servers due to mishandling of the private key (lost, stolen, etc.). When you do have to replace certificates, though, how do you go about it? Is it a manual task? How are you sure you "got them all"?
This post aims to provide a solution for environments where Apache is handling SSL, and Saltstack is being used to manage IT infrastructure. I will provide an overview on the use of custom salt grains and modules that will facilitate collecting SSL certificate information and automating the process of updating certificates that stand in need of replacement.
Grains are a great way to get relatively static information from a host. You can create your own grains and leverage saltstack's ability to target minions that match specific filters, e.g
Saltstack doesn't come out of the box with a grain for SSL certificate data, and there are some technical reasons for this - one being that the tools commonly used to gather certificate data aren't normally installed with most operating systems by default, particularly in the case of Windows. SSL certificate info also isn't the type of 'static' os-related details that the core grain typically returns. As we'll see later in this post, the targeting system with saltstack can leverage grain data in a way that is very useful in rapidly replacing outdated certificates or updating certs where the private key has been compromised.
I wrote a custom grain, located on github, that you can put into your _grains folder in your salt filesystem. Sync the grain to your minions:
Make sure your minions have pyOpenSSL installed, as it is a dependency for the grain.
Test that the grain works:
If you don't have an environment with SSL certs and Apache ready to test with, follow the guide at https://github.com/solutionreach/tls_grain_demo to follow along.
You'll see output similar to the following:
We can now target minions who match very specific ssl certificate criteria. Let's find minions with a cert whose fingerprint is EA:DC:1C:F6:A2:44:D9:AA:B4:96:A5:C7:5C:37:BB:E3:FB:A7:48:DE:
Now that we can target minions matching specific ssl certificate criteria, we are able to execute states to update the minion to use the correct or updated ssl certificate and corresponding private key. The following state (.sls) file should be modified to match your environment (file paths, file names, etc).
Assuming your salt file_roots base is /srv/salt, create a directory named ssl under /srv/salt. We'll be storing the certificate you want to push to your minions in a subdirectory of ssl, called files. Create /srv/salt/ssl/files/. Copy the the new certificate file, and, if necessary, the bundle (certificate chain) file, to the files subdirectory. Create a pillar to store the private key in. For example, create /srv/pillar/ssl.sls, and add contents similar to the following to it (make sure to replace the private key block with your own):
Create a state file, /srv/salt/ssl/update_cert.sls, and add the following contents, making sure to use correct file names for your environment:
The first line looks for a pillar key called 'conf', and if it is missing, calls the execution module ssl.default. This is a custom module that will parse the loaded Apache virtual hosts and find which configuration file is handling the default virtual host for port 443. Create /srv/salt/_modules/ssl.py, and add the following contents to it:
Sync the custom module to your minions:
This module will parse the results of 'httpd -S'. If you want to manually specify the path to the conf file you want to update the certificate and key information for, pass in a pillar to the state execution:
The result is that your new or updated certificate file will be put into the correct location on the minion, the private key contents will be copied securely via the pillar system, the specified Apache conf file handling SSL will be updated to point to the new files, and the httpd service will be restarted.
Running "salt <target> grains.get cert" should now show the updated certificate information.
Using the grain targeting functionality of saltstack, you can now target multiple minions needing an updated certificate, for example, those using the certificate with a certain fingerprint, or a certain CN (i.e *.yourdomain.com), and update their certificate info in a single command:
This post aims to provide a solution for environments where Apache is handling SSL, and Saltstack is being used to manage IT infrastructure. I will provide an overview on the use of custom salt grains and modules that will facilitate collecting SSL certificate information and automating the process of updating certificates that stand in need of replacement.
Grains are a great way to get relatively static information from a host. You can create your own grains and leverage saltstack's ability to target minions that match specific filters, e.g
1 | salt -G os:CentOS |
Saltstack doesn't come out of the box with a grain for SSL certificate data, and there are some technical reasons for this - one being that the tools commonly used to gather certificate data aren't normally installed with most operating systems by default, particularly in the case of Windows. SSL certificate info also isn't the type of 'static' os-related details that the core grain typically returns. As we'll see later in this post, the targeting system with saltstack can leverage grain data in a way that is very useful in rapidly replacing outdated certificates or updating certs where the private key has been compromised.
I wrote a custom grain, located on github, that you can put into your _grains folder in your salt filesystem. Sync the grain to your minions:
1 | salt TARGET saltutil.sync_grains |
Make sure your minions have pyOpenSSL installed, as it is a dependency for the grain.
Test that the grain works:
1 | salt TARGET grains.get cert |
If you don't have an environment with SSL certs and Apache ready to test with, follow the guide at https://github.com/solutionreach/tls_grain_demo to follow along.
You'll see output similar to the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | cert:
----------
expired:
False
ports:
----------
443:
----------
cn:
localhost.localdomain
end:
Sep 04 22:51:09 2016
expired:
False
fingerprint:
EA:DC:1C:F6:A2:44:D9:AA:B4:96:A5:C7:5C:37:BB:E3:FB:A7:48:DE
serial:
1818
start:
Sep 05 22:51:09 2015
subject:
/C=--/ST=SomeState/L=SomeCity/O=SomeOrganization/OU=SomeOrganizationalUnit/
CN=localhost.localdomain/emailAddress=root@localhost.localdomain
|
We can now target minions who match very specific ssl certificate criteria. Let's find minions with a cert whose fingerprint is EA:DC:1C:F6:A2:44:D9:AA:B4:96:A5:C7:5C:37:BB:E3:FB:A7:48:DE:
1 | salt -G cert:ports:443:fingerprint:EA:DC:1C:F6:A2:44:D9:AA:B4:96:A5:C7:5C:37:BB:E3:FB:A7:48:DE test.ping |
Now that we can target minions matching specific ssl certificate criteria, we are able to execute states to update the minion to use the correct or updated ssl certificate and corresponding private key. The following state (.sls) file should be modified to match your environment (file paths, file names, etc).
Assuming your salt file_roots base is /srv/salt, create a directory named ssl under /srv/salt. We'll be storing the certificate you want to push to your minions in a subdirectory of ssl, called files. Create /srv/salt/ssl/files/. Copy the the new certificate file, and, if necessary, the bundle (certificate chain) file, to the files subdirectory. Create a pillar to store the private key in. For example, create /srv/pillar/ssl.sls, and add contents similar to the following to it (make sure to replace the private key block with your own):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | ssl:
keys:
localhost: |
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAtS1YoxGjponlMCWBB2jkdUI/ShsH4cuiV38oflsPVmRFMkJn
5Qux90DqtjMS4+jjyqACML73ivZ4Xbd+oQ5vs6rsliIYljtATCuts+lr3RYqj6/M
GPtnsFj+aQFIAjcl7HZGYK6d8Kz4hzZw5msHZQ4fXhFLy4OkKYmbNAaBzTuhbBav
T01bJFOd0fQcqOjjsFS7pNds5sM3emlG3goOxUjKmz5dQZyTmssyKHuvJ8R08WTz
IOiUzmdKEtvgOYOg5kcpnZSwKtp2pTniV8qo13so8Hd+N/lNv+6pxNwlNAEn9nlG
k53NjMkyNtwmjjQkKSXOTH7RyQo4H+2j4QAb3QIDAQABAoIBAAg15+Br+jif4y0N
Zg2J6WCPAgM/ulm3OsIwVwty4P0PSjt+2up8XzJPVNOw+NgvQ7N5EOBYpys7BDVr
DWgLGXDQ/CvJm8ejI8TP7e1LVMIOMRuV4e/35LkPL2he0H6ZNTTEH2heQXYYwKKQ
CwnGK+2eeDlxGzg73QKs48W1zcgZKjBdNVM8KVO2Eq/6ZfNVihVfXE+PS/OmegJ1
tzAVlA0ULsTm9ZKEO979EwCmtzt9wZVfv+QAXyl+utWLaw9Eq2qqOtFHLEX/LMsi
bQDlrqu19onC/eZETzeUxn6FfS0kFAhOqOwAvuWucLdnn+mjZe3TZ3vgxe5E77pg
xXVMJ4ECgYEA2NwODhB337m3kSuZBxlMwo+T0lRjMNHGn76pmiFWvrfepWP32DVe
ZKs+fRRL79WYxEnQ9+l4MNU6Qip022RXsWHEjyrAOhmZrBpvZDLZIA0V0NssmfRb
bh7pHxgac348ghHTuqwsUajRMjfdsPT8JJu/EInQaQBXi/Wo8OPcjY0CgYEA1eCX
kYUKh6rKY1ud+lipSAJ18uOJexhlx7/o6A6FGbFWgYYU1JCn00p9iaj73ik6mGwp
KZxv8gjfbT/0f9xDUlrSdvqAi7AShbtMzuGm7Syal1/OSFCHKZgO2CNwbWqFSyR9
ciknGyR/+1rZSCajb1QNgJ+p8ZJ+F/fjI+YSa5ECgYAGpB6OEbUKFe7oNDSYgg7W
unzlaQ4slZAGnlklTjYQ2yKnX/tcFK5SWOgt/mwg6SKniDctEGpM3IrPMeuoOFdz
KDJTzBRc8yHAooKcx+3cTGpJnhysjk4qfinXeO8/HOLxt5j7U4B787aMWieg3q/G
Ezrzr7TBMnB76ccFsYS0lQKBgEji1wRScHo3a+1cRYQRPu85V3TEsg9vCxB9iCO8
/wL4emB9jw+5lFffJNUK73qQVlWnVofFtiineWzDxNDmAVNJfDvrtm7kPAuy/sPq
BncBIlW/4o6mUsmL9lMWELRY/r/S2aVT/O21DKBeH3oFIOFJpahVwRaAM9R7N77+
IvzhAoGAbMvQDlcSt/r+ph2IW9JwtE7mWVvp4gBRHndJFguPtTNu95JsMSrBolOl
IvxFdbHF7ClvxYzAX6FJn+TIhu1AqGagw652hfudsvcb/T0W9Yf0yHz9zgXF64b9
NmbIiazrkmvxfytzr0iPuegktCg7GhZzFwKlrfUdqPKEyG56VPE=
-----END RSA PRIVATE KEY-----
|
Create a state file, /srv/salt/ssl/update_cert.sls, and add the following contents, making sure to use correct file names for your environment:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | {% set ssl_conf = salt['pillar.get']('conf', salt['ssl.default']()) %}
replace_ssl_key:
file.replace:
- name: {{ ssl_conf }}
- pattern: |
SSLCertificateKeyFile.*$
- repl: SSLCertificateKeyFile /etc/pki/tls/private/localhost.key\n
replace_ssl_cert:
file.replace:
- name: {{ ssl_conf }}
- pattern: |
SSLCertificateFile.*$
- repl: SSLCertificateFile /etc/pki/tls/certs/localhost.crt\n
replace_ssl_bundle_ca:
file.replace:
- name: {{ ssl_conf }}
- pattern: |
SSLCACertificateChainFile.*
- repl: SSLCACertificateChainFile /etc/pki/tls/certs/gd_bundle.crt\n
replace_ssl_bundle:
file.replace:
- name: {{ ssl_conf }}
- pattern: |
SSLCertificateChainFile.*
- repl: SSLCertificateChainFile /etc/pki/tls/certs/gd_bundle.crt\n
/etc/pki/tls/certs/localhost.crt:
file.managed:
- source: salt://ssl/files/localhost.crt
- user: root
- group: root
- mode: 644
/etc/pki/tls/certs/gd_bundle.crt:
file.managed:
- source: salt://ssl/files/gd_bundle.crt
- user: root
- group: root
- mode: 644
/etc/pki/tls/private/localhost.key:
file.managed:
- user: root
- group: root
- mode: 644
- contents_pillar: ssl:certificates:private:localhost
restart_httpd:
module.run:
- name: service.restart
- m_name: httpd
|
The first line looks for a pillar key called 'conf', and if it is missing, calls the execution module ssl.default. This is a custom module that will parse the loaded Apache virtual hosts and find which configuration file is handling the default virtual host for port 443. Create /srv/salt/_modules/ssl.py, and add the following contents to it:
1 2 3 4 5 6 7 8 9 10 11 | import subprocess
def default():
p1 = subprocess.Popen(['httpd', '-S'], stdout=subprocess.PIPE)
p2 = subprocess.Popen(['grep', '-Po', '_default_:443.*'], stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen(['awk', '{print$3}'], stdin=p2.stdout, stdout=subprocess.PIPE)
p4 = subprocess.Popen(['sed', 's/:[0-9]*//'], stdin=p3.stdout, stdout=subprocess.PIPE)
p5 = subprocess.Popen(['sed', 's/)//'], stdin=p4.stdout, stdout=subprocess.PIPE)
p6 = subprocess.Popen(['sed', 's/(//'], stdin=p5.stdout, stdout=subprocess.PIPE)
p7 = subprocess.Popen(['perl', '-p', '-e', 's/\n//' ], stdin=p6.stdout, stdout=subprocess.PIPE)
path = p7.communicate()[0]
return path
|
Sync the custom module to your minions:
1 | salt '*' saltutil.sync_modules |
This module will parse the results of 'httpd -S'. If you want to manually specify the path to the conf file you want to update the certificate and key information for, pass in a pillar to the state execution:
1 | salt TARGET state.sls ssl.update_cert pillar='{cert: /etc/httpd/conf.d/ssl.conf}'
|
The result is that your new or updated certificate file will be put into the correct location on the minion, the private key contents will be copied securely via the pillar system, the specified Apache conf file handling SSL will be updated to point to the new files, and the httpd service will be restarted.
Running "salt <target> grains.get cert" should now show the updated certificate information.
Using the grain targeting functionality of saltstack, you can now target multiple minions needing an updated certificate, for example, those using the certificate with a certain fingerprint, or a certain CN (i.e *.yourdomain.com), and update their certificate info in a single command:
1 | salt -G cert:ports:443:cn:*.yourdomain.com state.sls ssl.update_cert pillar='{cert: /etc/httpd/conf.d/ssl.conf}'
|
Comments
Post a Comment