I was in the audience when Amazon announced the AWS Secrets
Manager at the AWS Summit San Francisco. My first thought was
that we already have a way to store secrets in SSM Parameter
Store. In fact, I tweeted:
“Just as we were all working out the details of using SSM Parameter Store to manage our secrets…
Another tool to help build securely (looking forward to learning about it).
#AWSsummit San Francisco” – @esh 2018-04-04 11:10
When I learned the AWS Secrets Manager pricing plan, my questions
So, I started poring over the AWS Secrets Manager documentation and
slowly strarted to gain possible enlightenment.
I have archived below three stream of consciousness threads that I
originally posted to Twitter.
Thoughts 1: Secret Rotation Is The Value In AWS Secrets Manager
After reading the new AWS Secrets Manager docs, it looks like there is
a lot of value in the work Amazon has invested into the design of
There are a number of different ways systems support secrets, and
various failure scenarios that must be accounted for.
Though RDS secret rotation support is built in to AWS Secrets Manager,
customers are going to find more value in the ability to plug in
custom code to rotate secrets in any service–using AWS Lambda,
Customers write the code that performs the proper steps, and AWS
Secrets Manager will drive the steps.
It almost looks like we could take the secret rotation framework
design and develop AWS Step Functions and CloudWatch Events Schedule
to drive rotation for secrets in SSM Parameter Store,
but for such a critical piece of security infrastructure execution, it
makes sense to lean on Amazon to maintain this part and drive the
rotation steps reliably.
There are ways to create IAM policies that are fine-tuned with just
the AWS Secrets Manager permissions needed, including conditions on
custom tags on each secret.
When designing AWS Secrets Manager, I suspect there were discussions
inside Amazon about whether ASM itself should perform the steps to
move the current/pending/previous version labels around during secret
rotation, to reduce the risk of customer code doing this
I think this may have required giving the ASM service too much
permission to manipulate the customer’s secrets, so the decision seems
right to keep this with the customer’s AWS Lambda function, even
though there is some added complexity in development.
The AWS Secrets Manager documentation is impressively clear for
creating custom AWS Lambda functions for secret rotation, especially
for how complex the various scenarios can be.
Here’s a link to the AWS Secrets Manager Use Guide.
Thoughts 2: Rotating Secrets Frequently Is Important
I’m starting to understand that it’s “AWS Secrets Manager” not “AWS
Secrets Store”, and that the biggest part of that management seem to
be the automated, transparent, reliable secret rotation.
Now that I can see a smooth path to regular secret rotation with AWS
Secrets Manager, I’m starting to feel like I’ve been living in
primitive times letting my database passwords sit at the same value
for months (ok, years).
AWS Secrets Manager documentation shows an example of rotating a
password every 10 days, but now that I’ve seen the light, I’m
thinking, “Why not every hour?!”
In fact, why not rotate the secret using AWS Secret Manager shortly
after the longest time we expect it to take to use that secret?
If a database client is likely to take less than 30 seconds to connect
to the server after requesting the password, we could rotate that
secret every minute (using the two-user method described in the AWS
Secret Manager docs).
This moves in the direction of single-use secrets that are handed out
when a client has been authorized with the current IAM permissions.
I can imagine that some third party services might not appreciate
credential/token rotation requests at super-frequent rates.
The ideal would be to have every service authenticate clients/requests
live with IAM directly, but until then, frequent secret rotation with
AWS Security Manager seems worth investigating.
Thoughts 3: Rotation Functions Could Be Shared Among Customers
Writing AWS Lambda “Rotation Functions” to perform secret
rotation with AWS Secrets Manager is straight-forward, but not
trivial, and any code related to security needs to be clean, reliable,
Each integration with AWS Secrets Manager will need to deal with the
rotation “scenario” (as described in the docs), the method of changing
secrets (database connection, API, web session), and the peculiarities
of how the secrets are used with that product or service.
There are going to be a lot of different products and services that
need these written.
There are a lot of AWS Secrets Manager customers who will need these
There doesn’t need to be one implementation per AWS Secrets Manager
customer per product/service.
A well written AWS Lambda Rotation Function for AWS Secrets Manager
should be usable by any AWS customer who uses the same product or
No secrets or account information should be stored in the AWS Lambda
Rotation Functions themselves, but would reside in the AWS Secrets
We need a way for developers to publish high quality AWS Secrets
Manager Lambda Rotation Functions, and for AWS Secrets Manager
customers to find and use them.
Could the AWS Serverless Application Repository be used to host and
share AWS Lambda Rotation Functions for the AWS Secrets Manager?
I have concerns about the visibility/auditability of the AWS Lambda
Rotation Function code in the SAR, especially when it comes to
security code that will be provided customer secrets, but perhaps
these can be addressed somehow.
I have not used the AWS Secrets Manager yet, so all of these thoughts
should be taken as musings by somebody who is not familiar with the
product except through reading the documentation. Feedback and
corrections are welcomed in the comments here or on Twitter.
Original article and comments: https://alestic.com/2018/04/aws-secrets-manager-thoughts/