Testing Microsoft Sentinel detection coverage - Case AWS

Mikko Koivunen


Watchlist Insight

SIEM users should consider performing testing to validate that their detections actually work, especially if they are doing actual detection engineering (instead of just relying on vendor-provided rules).

Building a testing methodology is a very organization specific task. My goal for this article is to demonstrate some basic principles and one specific example for detection testing in Microsoft Sentinel.

If we compare the SIEM world to generic testing methodologies, we could come up with the following taxonomy:

Since we are basically borrowing terminology from another field (software development), I imagine there will be other opinions on these as well.

There would also be a lot to talk about testing methodologies when it comes to SIEM and detection engineering in general, or Microsoft Sentinel in particular. I see many opportunties for future articles around this topic.

In order to get at least something published before my summer holidays, I am now focusing in one end-to-end testing scenario: validating Amazon Web Services CloudTrail coverage in Sentinel, using a public testing tool that can simulate multiple known attack patterns against AWS.

AWS & Sentinel configuration

My test environment is a multi-account AWS environment integrated to Microsoft Sentinel:

In Sentinel I have enabled all the Microsoft provided rules that target the Amazon Web Services data source as can be seen in the screenshot. This also includes GuardDuty and VPC based rules, but as those data sources are not being ingested it does not really make a difference here.

Analytics Rules

Testing tool (Stratus)

For the testing tool I chose Stratus Red Team.

Stratus has a pretty good AWS adversary simulation coverage. Also as a big benefit for me Stratus does not require existing infrastructure, as it deploys the test resources with Terraform when running the tests.

You can see all tests available currently in Stratus here: https://stratus-red-team.cloud/attack-techniques/list/

For this testing sprint I chose to limit the number of tests to cover two ATT&CK Tactics, Credential Access and Defense Evasion:

| TECHNIQUE NAME                                         | MITRE ATT&CK TACTIC  |
| Retrieve EC2 Password Data                             | Credential Access    |
| Steal EC2 Instance Credentials                         | Credential Access    |
| Retrieve a High Number of Secrets Manager secrets      | Credential Access    |
| Retrieve And Decrypt SSM Parameters                    | Credential Access    |
| Delete CloudTrail Trail                                | Defense Evasion      |
| Disable CloudTrail Logging Through Event Selectors     | Defense Evasion      |
| CloudTrail Logs Impairment Through S3 Lifecycle Rule   | Defense Evasion      |
| Stop CloudTrail Trail                                  | Defense Evasion      |
| Attempt to Leave the AWS Organization                  | Defense Evasion      |
| Remove VPC Flow Logs                                   | Defense Evasion      |

Running the tests

To get easily reportable test results, I decided to run the Stratus tests one-by-one, making sure each test has enough time to complete the end-to-end scenario:

  1. Perform the test (“Detonating” in Stratus terminology).
  2. Wait for the activities to be logged in CloudTrail.
  3. Wait for the Sentinel data connector polling interval to run.
  4. Wait for the scheduled Analytics Rules to run.

Running the Stratus tests is simple. You need AWS CLI configured and authenticated on the test machine. There are some things to understand about Stratus, so be sure to read the documentation, but in practice the tests are very simple to run. Example test detonation and results below:

~ stratus detonate aws.credential-access.ec2-get-password-data
2022/06/08 14:18:44 Checking your authentication against AWS
2022/06/08 14:18:44 Warming up aws.credential-access.ec2-get-password-data
2022/06/08 14:18:44 Initializing Terraform to spin up technique prerequisites
2022/06/08 14:18:49 Applying Terraform to spin up technique prerequisites
2022/06/08 14:18:58 Running ec2:GetPasswordData on 30 random instance IDs

Collecting the results

For each test I recorded the related events in CloudTrail, verified that the event is found in Log Analytics and checked if any of the built-in Analytics Rules detected the activity.

Some initial comments for a couple of suprising findings were also recorded as soon as discovered. Further analysis will be done later, detections that do not work as expected will be fixed as part of normal detection engineering processes. Some things will end up as not beight detectable at all without too high false positive penalties.

I kept the reporting pretty simple for now, as you can see:


For this article I was mainly focusing in building this methodology and testing practices, so some individual tests do not have comments or development plans. But as you can see, it looks like there is some work to do for improving AWS detection coverage in this environment.


Building a testing program for your Security Operations can be a good way to improve your detection coverage. It sounds like a thing we could take for granted, but in reality it is not as common as could be expected.

There are a few things that need to be considered if you want to start a similar testing process:

I am looking forward to continuing this topic by running the rest of the AWS tests for all ATT&CK Tactics, and also to evaluate how to further develop the testing process itself.

One practical development item here is how to integrate testing into detection engineering; when a new rule is being developed, how to design the unit test at the same time. Stay tuned.