Rollback of SettingsResolversStack

[ptortiz]

The root cause is below. The config is for AWS_Organizations. Any insight would be appreciated.

Embedded stack arn:aws:cloudformation:us-west-2:533267120214:stack/workload-discovery-SettingsResolversStack-4O5HJV3KFX9F/31271720-0d68-11ef-b32c-06a1847ad5ab was not successfully created: The following resource(s) failed to create: [PerspectiveSettingsLambdaRole].

[svozza]

Are you deploying this solution in a private subent of an already existing VPC? If so, go into the SettingsResolversStack nested stack that failed and check if the error is that a custom resource timed out. The reason for this is that this custom resource runs in a VPC and in order for a custom resource to signal to CloudFormation that it has either succeeded or failed it must write to an S3 bucket: if there is no NAT gateway or S3 endpoint in the VPC then there is no way for this request to get to S3. There is documentation to verify if the VPC you are deploying to has the necessary configuration:

https://docs.aws.amazon.com/solutions/latest/workload-discovery-on-aws/prerequisites.html#verify-your-vpc-configuration

Bear in mind, if you do not have a NAT gateway in your VPC then an S3 endpoint if only sufficient to deploy the solution, you will need VPC endpoints for every service listed in the documentation below in order for the Discovery process to work:

https://docs.aws.amazon.com/solutions/latest/workload-discovery-on-aws/aws-apis.html

[svozza]

I need the actual error from the nested stack before I can tell what's going on. It just seems very odd that it would fail on creating an IAM role. If the cfn root cause feature in the console telling you it's that stack?

[ptortiz]

This is the actual error:

Resource handler returned message: "Invalid policy (Service: Iam, Status Code: 400, Request ID: ee4a83d0-db64-47d3-ae1a-951802be1692)" (RequestToken: c3a3a452-9eb7-4567-5ab6-48bca4ecb1ec, HandlerErrorCode: InvalidRequest)

[ptortiz]

This stack also fails: WdGlobalResourcesStackSet. All other resources are successfully provisioned.

Resource handler returned message: "Account used is not a delegated administrator (Service: CloudFormation, Status Code: 400, Request ID: a16b2cae-349d-4d0a-ba12-60cb0bd7739f)" (RequestToken: a51d0eb0-66e4-ceaf-06ba-9a67aee3902d, HandlerErrorCode: InvalidRequest)

[tindotuckshop]

I am also getting same error below when creating the SettingsResolversStack

2024-05-08 12:50:27 UTC-0700 |
SettingsResolversStack |
CREATE_FAILED |

Embedded stack arn:aws:cloudformation:us-east-1:593074725626:stack/workload-discovery-SettingsResolversStack-1K3FN19567UDC/2a860b90-0d74-11ef-a05f-0e45b15e00cd was not successfully created: The following resource(s) failed to create: [PerspectiveSettingsLambdaRole].

If I check in CloudTrail, the role is created with no error, but then it is immediately deleted.

I am also deploying in a new VPC. Could it be that the stack is being created before NAT gateway is created for this VPC to get access to S3?

"eventTime": "2024-05-08T19:50:25Z",
"eventSource": "iam.amazonaws.com",
"eventName": "CreateRole",
"awsRegion": "us-east-1",
"sourceIPAddress": "cloudformation.amazonaws.com",
"userAgent": "cloudformation.amazonaws.com",
"requestParameters": {
"path": "/",
"roleName": "workload-discovery-Settin-PerspectiveSettingsLambda-WQMFmqDMrgsQ",
"assumeRolePolicyDocument": "{"Version":"2012-10-17","Statement":[{"Action":["sts:AssumeRole"],"Effect":"Allow","Principal":{"Service":["lambda.amazonaws.com"]}}]}",
"description": ""
},
"responseElements": {
"role": {
"path": "/",
"roleName": "workload-discovery-Settin-PerspectiveSettingsLambda-WQMFmqDMrgsQ",
"roleId": "AROAYUFQCEL5IGV6WRPNN",
"arn": "arn:aws:iam::593074725626:role/workload-discovery-Settin-PerspectiveSettingsLambda-WQMFmqDMrgsQ",
"createDate": "May 8, 2024 7:50:25 PM",
"assumeRolePolicyDocument": "%7B%22Version%22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Action%22%3A%5B%22sts%3AAssumeRole%22%5D%2C%22Effect%22%3A%22Allow%22%2C%22Principal%22%3A%7B%22Service%22%3A%5B%22lambda.amazonaws.com%22%5D%7D%7D%5D%7D"
}
},
"requestID": "ceb71927-5d37-4541-805e-4e96b5e3db37",
"eventID": "7fb8eee1-ceab-4db8-972c-04244759d157",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "593074725626",
"eventCategory": "Management",
"sessionCredentialFromConsole": "true"

[svozza]

This stack also fails: WdGlobalResourcesStackSet. All other resources are successfully provisioned.

Resource handler returned message: "Account used is not a delegated administrator (Service: CloudFormation, Status Code: 400, Request ID: a16b2cae-349d-4d0a-ba12-60cb0bd7739f)" (RequestToken: a51d0eb0-66e4-ceaf-06ba-9a67aee3902d, HandlerErrorCode: InvalidRequest)

Ah, so this is likely the issue. You need to deploy in a delegated admin account that has the ability to do stack sets and cross account AWS Config. There's links in the docs on how to enable those features in a delgated admin account:

https://docs.aws.amazon.com/solutions/latest/workload-discovery-on-aws/choosing-the-deployment-account.html

[ptortiz]

This stack also fails: WdGlobalResourcesStackSet. All other resources are successfully provisioned.
Resource handler returned message: "Account used is not a delegated administrator (Service: CloudFormation, Status Code: 400, Request ID: a16b2cae-349d-4d0a-ba12-60cb0bd7739f)" (RequestToken: a51d0eb0-66e4-ceaf-06ba-9a67aee3902d, HandlerErrorCode: InvalidRequest)

Ah, so this is likely the issue. You need to deploy in a delegated admin account that has the ability to do stack sets and cross account AWS Config. There's links in the docs on how to enable those features in a delgated admin account:

https://docs.aws.amazon.com/solutions/latest/workload-discovery-on-aws/choosing-the-deployment-account.html

Yes, I did that and it resolved the quoted error, but not the SettingsResolverStack error. Thanks.

[svozza]

Ok, I thought those two errors we from the same run. Once CloudFormation starts rolling back things you can see weird errors in unrelated stacks as changes get unwound. I've also just realised that I misread SettingsResolverStack as SearchResolverStack. The latter is the one that has a custom resource in the VPC so ignore everything I said about that.

I have never seen a failure in the SettingsResolverStack regarding an IAM role. You can see in the CloudFormation that the role is very simple and there's nothing in there that looks invalid:

workload-discovery-on-aws/source/cfn/templates/settings-resolvers.template

Line 64 in 91e36ff

Policies:

. Are these the only other errors in any of the other nested stacks?

[ptortiz]

I am only seeing the one error creating the IAM role in the SettingResolverStack. All other resources provisioned successfully. No errors in any of the other nested stacks. Thanks.

[svozza]

This is a bit of shot in the dark but I wonder if there's a chance this was an intermittent error from IAM. It seems like you've provisioned the stack using the Preserve successfully provisioned resources option. What happens if you just ask it to retry? As I said, I've never seen this error before and as you can see from the policy the code base there, it's not malformed. If it fails again, can you send me a screenshot of the all the the events that occurred after the retry in the CFN console for that nested stack?

[ptortiz]

Here are the events associated with the retry. Thanks.

[svozza]

I am also getting same error below when creating the SettingsResolversStack

2024-05-08 12:50:27 UTC-0700 | SettingsResolversStack | CREATE_FAILED |

Embedded stack arn:aws:cloudformation:us-east-1:593074725626:stack/workload-discovery-SettingsResolversStack-1K3FN19567UDC/2a860b90-0d74-11ef-a05f-0e45b15e00cd was not successfully created: The following resource(s) failed to create: [PerspectiveSettingsLambdaRole].

If I check in CloudTrail, the role is created with no error, but then it is immediately deleted.

I am also deploying in a new VPC. Could it be that the stack is being created before NAT gateway is created for this VPC to get access to S3?

"eventTime": "2024-05-08T19:50:25Z", "eventSource": "iam.amazonaws.com", "eventName": "CreateRole", "awsRegion": "us-east-1", "sourceIPAddress": "cloudformation.amazonaws.com", "userAgent": "cloudformation.amazonaws.com", "requestParameters": { "path": "/", "roleName": "workload-discovery-Settin-PerspectiveSettingsLambda-WQMFmqDMrgsQ", "assumeRolePolicyDocument": "{"Version":"2012-10-17","Statement":[{"Action":["sts:AssumeRole"],"Effect":"Allow","Principal":{"Service":["lambda.amazonaws.com"]}}]}", "description": "" }, "responseElements": { "role": { "path": "/", "roleName": "workload-discovery-Settin-PerspectiveSettingsLambda-WQMFmqDMrgsQ", "roleId": "AROAYUFQCEL5IGV6WRPNN", "arn": "arn:aws:iam::593074725626:role/workload-discovery-Settin-PerspectiveSettingsLambda-WQMFmqDMrgsQ", "createDate": "May 8, 2024 7:50:25 PM", "assumeRolePolicyDocument": "%7B%22Version%22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Action%22%3A%5B%22sts%3AAssumeRole%22%5D%2C%22Effect%22%3A%22Allow%22%2C%22Principal%22%3A%7B%22Service%22%3A%5B%22lambda.amazonaws.com%22%5D%7D%7D%5D%7D" } }, "requestID": "ceb71927-5d37-4541-805e-4e96b5e3db37", "eventID": "7fb8eee1-ceab-4db8-972c-04244759d157", "readOnly": false, "eventType": "AwsApiCall", "managementEvent": true, "recipientAccountId": "593074725626", "eventCategory": "Management", "sessionCredentialFromConsole": "true"

@tindotuckshop I've just realised that I misread SettingsResolverStack as SearchResolverStack in the initial post. The latter is the one that has a custom resource in the VPC so ignore everything I said about that. (The settings stack doesn't even have a custom resource.)

[ptortiz]

Ok, I thought those two errors we from the same run. Once CloudFormation starts rolling back things you can see weird errors in unrelated stacks as changes get unwound. I've also just realised that I misread SettingsResolverStack as SearchResolverStack. The latter is the one that has a custom resource in the VPC so ignore everything I said about that.

I have never seen a failure in the SettingsResolverStack regarding an IAM role. You can see in the CloudFormation that the role is very simple and there's nothing in there that looks invalid:

workload-discovery-on-aws/source/cfn/templates/settings-resolvers.template

Line 64 in 91e36ff

Policies:

. Are these the only other errors in any of the other nested stacks?

I am only seeing the one error creating the IAM role in the SettingResolverStack. All other resources provisioned successfully. No errors in any of the other nested stacks. Thanks.

[svozza]

@ptortiz @tindotuckshop It looks like there's been a change with how IAM interprets ARNs in the Resource section of polices and that's what's causing the issue. We are working on a fix that we aim to release soon. I will update this ticket to let you know when the release has come out.

[svozza]

We have released a patch (v2.1.7) today to change the affected IAM policy and prevent this error from occurring.

[ptortiz]

This fixed the problem. Thanks.

[svozza]

Good stuff!