perf(misconf): High memory usage (> 10 GB) on some repos

[nikpivkin]

Discussed in #6958

^{Originally posted by david-nascimento-form3 June 18, 2024}

Description

Seems related to:

• perf(misconf): High memory usage (9.5 GB) and long scan time (45 min) on some repos #6557

We noticed high memory usage when using the misconfig scanner terraform, it easily reaches the 24GB of memory when a high number of terraform resources are present [>10000].

[!Note]

This profiling information was not from the code sample below

Desired Behavior

To have a steady memory usage, if possible close to tfsec.

Actual Behavior

A high memory usage that leads to a container being killed.

Reproduction Steps

This is an issue with a high number of resources, the following sample easily reach 20GB of memory:


locals {
  team_repos = [ for i in range(1000): "repo-${i}"]
  teams = [ for i in range(10): "team-${i}"]
  repositories = merge([for team_id in local.teams : { for repo in local.team_repos : "${team_id}-${repo}" => team_id}]...)
}

resource "aws_ecr_repository" "ecr-repository" {
  for_each = local.repositories

  name                 = each.key
  image_tag_mutability = "IMMUTABLE"
  tags = {
    "Team" : each.value
  }
}


To note that when analysed with `tfsec` the memory did not exceed `1GB`.

Target

Filesystem

Scanner

Misconfiguration

Output Format

JSON

Mode

Standalone

Debug Output

$ trivy config --misconfig-scanners terraform test-tf --debug

2024-06-18T16:46:54+01:00       DEBUG   Parsed severities       severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
2024-06-18T16:46:54+01:00       DEBUG   Cache dir       dir="/Users/***********/Library/Caches/trivy"
2024-06-18T16:46:54+01:00       INFO    Misconfiguration scanning is enabled
2024-06-18T16:46:54+01:00       DEBUG   Policies successfully loaded from disk
2024-06-18T16:46:54+01:00       DEBUG   Enabling misconfiguration scanners      scanners=[terraform]
2024-06-18T16:46:54+01:00       DEBUG   [nuget] The nuget packages directory couldn't be found. License search disabled
2024-06-18T16:46:54+01:00       DEBUG   Scanning files for misconfigurations... scanner="Terraform"
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.073583000 terraform.scanner                Scanning [&{%!s(*mapfs.file=&{ [] {. 256 2147484096 {13948924422273498744 333823334 0x10d8f81a0} <nil>} {{{0 0} {[] {} 0x14002f1dcc0} map[test.tf:0x140027e5bf0] 0}}}) test-tf}] at '.'...
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.075550000 terraform.scanner.rego           Overriding filesystem for checks!
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.076185000 terraform.scanner.rego           Loaded 3 embedded libraries.
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.105253000 terraform.scanner.rego           Loaded 191 embedded policies.
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.146610000 terraform.scanner.rego           Loaded 194 checks from disk.
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.146888000 terraform.scanner.rego           Overriding filesystem for data!
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346109000 terraform.parser.<root>          Setting project/module root to '.'
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346135000 terraform.parser.<root>          Parsing FS from '.'
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346163000 terraform.parser.<root>          Parsing 'test.tf'...
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346434000 terraform.parser.<root>          Added file test.tf.
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346517000 terraform.scanner                Scanning root module '.'...
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346521000 terraform.parser.<root>          Setting project/module root to '.'
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346523000 terraform.parser.<root>          Parsing FS from '.'
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346529000 terraform.parser.<root>          Parsing 'test.tf'...
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346605000 terraform.parser.<root>          Added file test.tf.
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346613000 terraform.parser.<root>          Evaluating module...
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346645000 terraform.parser.<root>          Read 2 block(s) and 0 ignore(s) for module 'root' (1 file[s])...
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346651000 terraform.parser.<root>          Added 0 variables from tfvars.
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346686000 terraform.parser.<root>          Working directory for module evaluation is "/Users/***********/trivy"
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346725000 terraform.parser.<root>.evaluator Filesystem key is '**********************************************'
2024-06-18T16:46:54+01:00       DEBUG   [misconf] 46:54.346728000 terraform.parser.<root>.evaluator Starting module evaluation...
2024-06-18T16:49:52+01:00       DEBUG   [misconf] 49:52.125958000 terraform.parser.<root>.evaluator Expanded block 'aws_ecr_repository.ecr-repository' into 10000 clones via 'for_each' attribute.
2024-06-18T16:49:52+01:00       DEBUG   [misconf] 49:52.131119000 terraform.parser.<root>.evaluator Starting submodule evaluation...
2024-06-18T16:49:52+01:00       DEBUG   [misconf] 49:52.131450000 terraform.parser.<root>.evaluator All submodules are evaluated at i=0
2024-06-18T16:49:52+01:00       DEBUG   [misconf] 49:52.131456000 terraform.parser.<root>.evaluator Starting post-submodule evaluation...
2024-06-18T17:01:01+01:00       DEBUG   [misconf] 01:01.298179000 terraform.parser.<root>.evaluator Finished processing 0 submodule(s).
2024-06-18T17:01:01+01:00       DEBUG   [misconf] 01:01.299317000 terraform.parser.<root>.evaluator Module evaluation complete.
2024-06-18T17:01:01+01:00       DEBUG   [misconf] 01:01.329185000 terraform.parser.<root>          Finished parsing module 'root'.
2024-06-18T17:01:01+01:00       DEBUG   [misconf] 01:01.329385000 terraform.executor               Adapting modules...
2024-06-18T17:01:01+01:00       DEBUG   [misconf] 01:01.413986000 terraform.executor               Adapted 1 module(s) into defsec state data.
2024-06-18T17:01:01+01:00       DEBUG   [misconf] 01:01.414046000 terraform.executor               Using max routines of 9
2024-06-18T17:01:01+01:00       DEBUG   [misconf] 01:01.414613000 terraform.executor               Initialized 486 rule(s).
2024-06-18T17:01:01+01:00       DEBUG   [misconf] 01:01.414619000 terraform.executor               Created pool with 9 worker(s) to apply rules.
2024-06-18T17:01:01+01:00       DEBUG   [misconf] 01:01.851145000 terraform.scanner.rego           Scanning 1 inputs...
2024-06-18T17:01:02+01:00       DEBUG   [misconf] 01:02.970171000 terraform.executor               Finished applying rules.
2024-06-18T17:01:02+01:00       DEBUG   [misconf] 01:02.970229000 terraform.executor               Applying ignores...
2024-06-18T17:01:05+01:00       DEBUG   OS is not detected.
2024-06-18T17:01:05+01:00       INFO    Detected config files   num=2
2024-06-18T17:01:05+01:00       DEBUG   Scanned config file     path="."
2024-06-18T17:01:05+01:00       DEBUG   Scanned config file     path="test.tf"

Operating System

Sonoma 14.5, [Container image linux/amd64]

Version

Version: 0.52.2

Also tested with latest version of master:

Version: dev
Vulnerability DB:
  Version: 2
  UpdatedAt: 2022-05-26 12:06:58.288892667 +0000 UTC
  NextUpdate: 2022-05-26 18:06:58.288892267 +0000 UTC
  DownloadedAt: 2022-05-26 13:22:42.722024668 +0000 UTC
Check Bundle:
  Digest: sha256:cfb65621a1f55d9d099c4c28931b252716fcda8bba5081eb43f1001668e79d85
  DownloadedAt: 2024-06-18 14:19:01.638403 +0000 UTC

Checklist

• Run trivy image --reset
• Read the troubleshooting