ACM.48: Manually creating a Lambda function to retrieve secrets from secrets manager
This is a continuation of my series of posts on Automating Cybersecurity Metrics.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Interlude: Still waiting for copyrighted materials to be removed from of these sites. I added information on how to report copyright infringement to Google’s legal team here:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this post we are going to create a Lambda function manually in the AWS console to test our role and see if we can access our secret in Secrets Manager.
Because we put the secret in secrets manager, we don’t need to hard code it into our code. In fact, no human even needs to see those credentials as I wrote about in this prior post:
We can retrieve them using some pretty simple code in a Lambda function. I often talk to clients on IANS Research consulting calls on how easy this is to do. Now I’ll show you. This code was pretty much part of my cloud security class before but I’m going to revise it to simplify it a bit. In class I demonstrated the bad ways as well as the good ways to handle secrets. Here I just want to get it done so we’re going to focus on what you should be doing.
The Lambda Batch Job Trigger Role
If you recall we created a role earlier to trigger batch jobs. I put the policy with the Lambda code.
I have since refactored the Lambda role and policy to trigger a batch job in a similar method to how I refactored code in prior posts. Check those out for the refactored directory layout and code.
Secret in Secrets Manager
We created a secret in Secrets Manger. This code also was refactored to support our new naming convention but essentially we’ve got similar secret encrypting the credentials we created and stored in Secrets Manager. We want to test retrieving and decrypting that secret in a Lambda function.
Manually getting a feel for AWS Lambda Functions
Although I highly recommend automation for security and if possible complete automation, there are times when it is easier to start in the AWS or other cloud console to get a feel for how things work. That’s what we’re going to do in this post to create a simple Lambda function to test out our role. As it turns out, simple is not always so simple when it comes to KMS, Secrets Manager, IAM, and a thoughtful security architecture.
Retrieving Secrets with a Lambda Function
If you ever took my original cloud security class you already did what I’m going to do in this blog post in one of the labs. We added credentials to AWS Secrets Manager and retrieved it with some Python code in a Lamba function.
Since I already wrote that code once before I’ll pull it out of the lab, but we’ll look at some additional security we can apply to our code and deployment. Often in classes you are performing actions piecemeal. In this series of blog posts I’m architecting a system end-to-end with layered security controls to try to protect our secrets and data. Here you’ll get to see how a lambda that retrieves secrets fits into the larger picture of a cloud architecture.
I’m not guaranteeing it’s perfect or has no security problems, but you can actually see how all the pieces of different parts of cloud security and application security work together. Hopefully, you can understand the thought process that goes into architecting a secure solution.
Creating a Lambda function in the AWS Console
The first thing we want to do is test that our role has the correct permissions to obtain our secret from secrets manager and decrypt it. The easiest way to get started with AWS Lambda is to drop some code into the Lambda UI in the AWS Console.
Navigate to Lambda in the AWS Console and click “Create function.”
On the next screen, enter a name, choose the latest version of python and arm64. Why? Check out the architecture information; arm costs less, but if your code uses libraries that require x86 then you’ll need to use that. We can use arm in this case.
Next we can choose a role, but here’s where we get stuck. If you used my original role created in the prior blog post, you won’t find the role we created in the list. Why not?
We need to grant the AWS Lambda service permissions to assume our role in the AssumeRolePolicyDocument (also know as Trust Policy) in our Cloud Formation template. As you can imagine, we can’t require MFA for this role assumption since Lambda is not a user in our account.
Let’s modify that role template to allow Lambda to assume the role. We need to modify the Principal in the trust policy for the role to the Lambda function. That change looks like this:
Deploy the role.
./deploy.sh
Check to see the change exists in the Trust Policy for the role:
Return to your Lambda function and click the refresh button.
Now you can select your role from the list:
Note that you can also return to this setting by clicking Configuration on your Lambda dashboard or a particular function and edit the Execution role:
We’ll look at some of the advanced functions later. For now click “Create Function.”
Now we have a function with some sample code. Click Test.
You can pass in arguments to a Lambda function just as we have been passing arguments into our command line scripts and parameters into our CloudFormation templates. Here’s where you can set up mock inputs to test your function as if it were part of a larger system. For now, just click TestEvent and click Save at the bottom.
In the above, we’re passing in three values when we call our Lambda which we don’t really need but you can just leave them for now.
Your test event is saved.
Click Test again. Your source code is successfully executed.
Return to the tab with the Lambda
Now I’m going to add a simplified version of the code from my cloud security class that accesses a secret, but I’m going to modify it to access the secret we’ve created in this blog post series.
import os
import boto3
def lambda_handler(event,context):client=boto3.client("secretsmanager")
secret='BatchJobAdminCredentials'
response=client.get_secret_value(SecretId=secret)#normally we don't want to print out secrets but this
return {
#is a temporary test to ensure our role works. Then
#we will delete this code. It will not get checked in
#to source control
'body': response,
'headers': {
'Content-Type': 'text/plain'
},
'statusCode': 200
}
On our first attempt to execute this code we’ll get the following error:
Test Event Name
TestEventResponse
{
"errorMessage": "An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::xxxxx:assumed-role/BatchJobTriggerRole/TestSecretsManagerAccess is not authorized to perform: secretsmanager:GetSecretValue on resource: BatchJobAdminCredentials because no identity-based policy allows the secretsmanager:GetSecretValue action",
"errorType": "ClientError",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 5, in lambda_handler\n response=client.get_secret_value(SecretId='BatchJobAdminCredentials')\n",
" File \"/var/runtime/botocore/client.py\", line 391, in _api_call\n return self._make_api_call(operation_name, kwargs)\n",
" File \"/var/runtime/botocore/client.py\", line 719, in _make_api_call\n raise error_class(parsed_response, operation_name)\n"
]
}
As you can see from the error message above our Lambda Function can’t access the secret because it doesn’t have the GetSecretValue permission. Let’s go back and take a look at our role.
In my first implementation of the role, there were no permissions associated with your role.
I have since added them if you’re looking at the latest code in the GitHub repo. But let’s walk through how you would add them potentially.
Zero trust roles for Lambda functions
I showed you how to create an IAM role from CloudTrail in a prior post:
In this post we’re going to use the trial-and-error method of coming up with a zero trust policy because I don’t think we need many permissions. Let’s start by adding the GetSecret permission:
One one of the things I needed to do was specify which secret this role can access. For the purposes of this post we’re going to just add one particular secret ARN as a resource. This may be problematic later when we have multiple operators but we’ll keep it simple for now. I want to reference the output with the secret ARN from the appropriate stack — only I never added any outputs.
Let’s add the secret ARN to our outputs in the template used to deploy that stack. If you followed along in my prior posts, you can see the path to the template we need to change in the AWS Console. Click on the stack that created the secret and then click on Template. Note again that I refactored so the directory structure and file names are different now but here’s where you can find the latest.
Add the output we need to that template:
We can run our batch job test script and verify that the outputs appear:
Great! Now go back to our lambda policy and reference the export in the resources section to allow the GetSecret action on this particular secret.
OK now we’ve got a working role template.
I made sure to delete the old role with a different name so I wouldn’t be confused later. Just like a great chef in the kitchen, it’s good to keep your workspace clean.
Now I want to change my lambda function to use this new role. Return to the Lambda console. Navigate to your function. Click Configuration.
Caveat on Working with the Lambda Console with an SSO User
Notice that it appears that the role that was assigned has been deleted. Click Edit.
In my case, something is wrong. The new role isn’t showing up here:
Let’s revisit our BatchTriggerRole. Do you remember what the problem is? But wait, our Lambda Function does have a proper trust policy. As it turns out, AWS SSO timed out. I had to refresh my login via the AWS SSO page and then I got the roles to show up:
Resolving Bugs (Typos) in Lambda Functions
Now we can click Test and Test again. This is where I notice that I misspelled secretSmanager above and I added the wrong action name:
Tip: To avoid my fate…always copy and paste when you can.
Next we get the unhelpful message, “Access to KMS is not allowed.” I’ve written about this a few times. Here’s a summary. It seems like a KMS bug to me or at least something that could be more user-friendly.
I explain why this error is problematic in that post but often it is because you edited a role and AWS magically changed your policy, and CloudFormation does not detect the change. See the above blog post for more details.
What we can see in the following message is that it
In order to figure out what permissions we need to add we can look in CloudTrail manually to figure out what permissions we need. Use the Lambda function name as the user name to search.
Scroll over and look at the Error code column we looked at in a prior post. Look for a failure and click on that item to see the details.
The first error isn’t exactly helpful:
Click on the next item that says AccessDenied.
Here we get the strange cipher text message I’ve written about before that doesn’t really make sense to me. But essentially this usually means that the person trying to decrypt something doesn’t have permission to use the KMS key used to encrypt the object.
The action was invoked by Secrets Manager and the event was on KMS. The denied action was Decrypt. So we can start by allowing the KMS decrypt action in our policy.
The question is, does this action apply to the Secrets Manager secret, or the KMS key? The error message doesn’t say but since this action is in relation to KMS it seems like the action would be taken on the KMS key. So if we add the action to the part of the policy where the resource is a secret it may fail. Let’s try it to find out.
There’s another problem here I’ve mentioned before. We had to add Decrypt to this policy. We also had to add Decrypt to the policy that creates and stores the credentials. Does that make sense? An AWS customer should be able to segregate the Encrypt and the Decrypt actions to protect AWS secrets. This seems like a bug and a security problem to me that I’ve written about before. I hope it is that gets fixed but I’m sure it causes complications for backwards compatibility.
If there was some reason the above needed to be Decrypt then I would expect the opposite action to require an Encrypt action (but that doesn’t really make sense does it?) This seems like an improper implementation. I already requested a fix on #awswishlist but please feel free to add your vote.
With this policy we get the same error:
Move the KMS statement into its own section in the policy with access to resource * (something I will fix in a future post). You’ll still get the KMS error if you followed along from the start. Why?
We changed the name of our Lambda role for one thing. Let’s take a look at the key used to encrypt our secret. First take a look at the secret.
Here we can see concretely that AWS has magically turned the role we deleted into some other characters:
It seems to me that there should be a more appropriate error message here and AWS should not be altering customer policies as already explained in the related blog post and that is where I explain how to force CloudFormation to redeploy the correct encrypt and decrypt ARNs.
Redeploy Your Function After Changing Code
Let’s try to test the Lambda function again once we resolve that. Well, I had a typo so I fixed that, clicked deploy, then test. Make sure you wait for the deploy function to finish and get this message or your Lambda function may end up in a non-fucntional state.
Next this:
I know that has something to do with what gets returned in our secret and it’s not a string so I’ll just wrap it in a str function. This is only test and we’d never leave it like this we’re just making sure we can actually retrieve the secret at this point:
response=str(client.get_secret_value(SecretId=secret))
Success. We have used a few simple lines of code in our Lambda function to retrieve a secret from Secrets Manager.
We have a few different things to do next. We need to automate our Lambda function deployment. We need to figure out how we are going to execute commands with this credentials using a role that requires MFA. Follow for updates as we work through it.
Teri Radichel
If you liked this story please clap and follow:
Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research
© 2nd Sight Lab 2022
All the posts in this series:
____________________________________________
Author:
Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.
Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts
