Splunk will automatically generate an internal server certificate on first run. This certificate is used by default for SSL secured communication from splunkd over port 8089. It is also used as part of the mongdb initialisation on the KVStore.
This certificate is auto generated on first run, and has an expiry date two years into the future.
Now, under normal use this isn’t an issue, as upgrades to the Splunk platform will ensure to renew the certificate as required (another reason to keep things up to date)
In the event it expires (I’ve seen this happen multiple times due to environments sitting on the same version for two years) you will see very odd behaviour from Splunk. Most noticeably, the KVStore will refuse to start using the expired certificate. Now, it will happily continue to run under the expired certificate, so everything is peachy until the instance restarts, at which point mongodb will exit on start due to said expired cert…
How do I fix this?
Actually, it’s quite simple. To force Splunk to re-issue the certificate, simply move the current server.pem out of $SPLUNK_HOME/etc/auth and restart Splunk.
A new certificate will be generated to replace the missing one, with a new expiry date two years into the future.