Add gitlab ci diff cli / pr comment #27153

CelianR · 2024-06-28T13:27:25Z

What does this PR do?

This adds a task that prints gitlab ci modifications as shown below.
It also comments on PRs.

Motivation

Additional Notes

This adds:

An invoke task to show / post diff
A job calling this task
A pr commenter task
A new SSM parameter for the full api token
A new way to get a gitlab token (same way as in the original pr-commenter)
Unit tests

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

agent-platform-auto-pr · 2024-06-28T13:43:51Z

[Fast Unit Tests Report]

On pipeline 38129343 (CI Visibility). The following jobs did not run any unit tests:

Jobs:

tests_deb-arm64-py3
tests_deb-x64-py3
tests_flavor_dogstatsd_deb-x64
tests_flavor_heroku_deb-x64
tests_flavor_iot_deb-x64
tests_rpm-arm64-py3
tests_rpm-x64-py3
tests_windows-x64

If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help

pr-commenter · 2024-06-28T14:22:13Z

Regression Detector

Regression Detector Results

Run ID: 02600f9b-ae99-4ecc-8e2f-d2bf0b202e7e Metrics dashboard Target profiles

Baseline: 671f875
Comparison: a067f23

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	links
➖	pycheck_1000_100byte_tags	% cpu utilization	+1.75	[-3.20, +6.70]	Logs
➖	basic_py_check	% cpu utilization	+1.30	[-1.45, +4.05]	Logs
➖	idle	memory utilization	+0.70	[+0.67, +0.74]	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	+0.57	[-12.04, +13.18]	Logs
➖	file_tree	memory utilization	+0.30	[+0.26, +0.34]	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.01, +0.01]	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.00	[-0.00, +0.00]	Logs
➖	otel_to_otel_logs	ingress throughput	-0.13	[-0.94, +0.68]	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-1.04	[-1.92, -0.15]	Logs

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

KSerrania

Great work 🎉. Some general comments after testing for a bit:

We should show the contents of added jobs - we want to be able to quickly read what a new job does in PR review,
I'd make the naming of the parameters of inv -e notify.gitlab-ci-diff more consistent, eg. "before" vs. "after",
~~For local use, I'd like to be able to say "use my local (uncommitted) config" as one of the comparison points - right now, you need to commit changes to be able to evaluate them,~~ already done
If the linting API fails (because the Gitlab config is invalid), the task should not immediately crash, and instead show that error in the PR comment.

CelianR · 2024-07-02T11:54:07Z

Example Gitlab CI Configuration Changes

Modified Jobs

notify_failure_summary_daily

  notify_failure_summary_daily:
    dependencies: []
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    resource_group: notification
    rules:
    - if: $BUCKET_BRANCH != "nightly" && $BUCKET_BRANCH != "oldnightly" && $BUCKET_BRANCH
        != "dev"
      when: never
    - if: $DEPLOY_AGENT == "true" || $DDR_WORKFLOW_ID != null
      when: always
    script:
    - source /root/.bashrc
    - export SLACK_API_TOKEN=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $SLACK_AGENT_CI_TOKEN_SSM_NAME)
    - export GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $GITLAB_READ_API_TOKEN_SSM_NAME)
    - export DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $API_KEY_ORG2_SSM_NAME)
    - python3 -m pip install -r requirements.txt -r tasks/libs/requirements-notifications.txt
-   - weekday="$(date --utc '+%A')"
+   - weekday="$(date --utc '+%A')" ah
?                                  +++
    - "if [ \"$weekday\" = \"Sunday\" ] || [ \"$weekday\" = \"Monday\" ]; then\n  echo\
      \ \"Skipping daily summary on $weekday\"\n  exit\nfi\n"
    - inv -e notify.failure-summary-send-notifications --daily-summary
    - "if [ \"$weekday\" = \"Friday\" ]; then\n  echo 'Sending weekly summary'\n  inv\
      \ -e notify.failure-summary-send-notifications --weekly-summary\nfi\n"
    stage: notify
    tags:
    - arch:amd64
    timeout: 15 minutes

Added Jobs

notify_gitlab_ci_changes

notify_gitlab_ci_changes:
  before_script:
  - git fetch origin main
  - git checkout main
  - git checkout -
  image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
  needs: []
  rules:
  - if: $CI_PIPELINE_SOURCE != "push"
    when: never
  - changes:
      compare_to: main
      paths:
      - .gitlab-ci.yml
      - .gitlab/**/*.yml
  script:
  - source /root/.bashrc
  - python3 -m pip install -r tasks/libs/requirements-github.txt
  - 'export GITHUB_KEY=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $GITHUB_PR_COMMENTER_APP_KEY_SSM_NAME)

    export GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $GITHUB_PR_COMMENTER_INTEGRATION_ID_SSM_NAME)

    export GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh
    $GITHUB_PR_COMMENTER_INSTALLATION_ID_SSM_NAME)

    export GITHUB_TOKEN=$(inv github.get-token-from-app --pkey-env GITHUB_KEY --installation-id
    GITHUB_INSTALLATION_ID --no-base64)

    export GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $GITLAB_FULL_API_TOKEN_SSM_NAME)

    '
  - inv -e notify.gitlab-ci-diff --pr-comment
  stage: notify
  tags:
  - arch:amd64

Renamed Jobs

notify_github -> notify_github_renamed

Changes

Removed	Modified	Added	Renamed
0	1	1	1

pr-commenter · 2024-07-02T12:03:21Z

Gitlab CI Configuration Changes

Added Jobs

notify_gitlab_ci_changes

notify_gitlab_ci_changes:
  before_script:
  - git fetch origin main
  - git checkout main
  - git checkout -
  image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
  needs: []
  rules:
  - if: $CI_PIPELINE_SOURCE != "push"
    when: never
  - changes:
      compare_to: main
      paths:
      - .gitlab-ci.yml
      - .gitlab/**/*.yml
  script:
  - source /root/.bashrc
  - python3 -m pip install -r tasks/libs/requirements-github.txt
  - 'export GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $GITHUB_PR_COMMENTER_APP_KEY_SSM_NAME
    | base64)

    export GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $GITHUB_PR_COMMENTER_INTEGRATION_ID_SSM_NAME)

    export GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh
    $GITHUB_PR_COMMENTER_INSTALLATION_ID_SSM_NAME)

    export GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $GITLAB_FULL_API_TOKEN_SSM_NAME)

    '
  - inv -e notify.gitlab-ci-diff --pr-comment
  stage: notify
  tags:
  - arch:amd64

Changes

Removed	Modified	Added	Renamed
0	0	1	0

amenasria · 2024-07-02T12:41:48Z

👏 praise: ‏Wtf this is so clean omg thank you for this !!!!

amenasria

LGTM !

KSerrania · 2024-07-02T13:27:43Z

.gitlab/notify/notify.yml

+      export GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $GITHUB_PR_COMMENTER_INTEGRATION_ID_SSM_NAME)
+      export GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $GITHUB_PR_COMMENTER_INSTALLATION_ID_SSM_NAME)
+      export GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/aws_ssm_get_wrapper.sh $GITLAB_FULL_API_TOKEN_SSM_NAME)
+    - inv -e notify.gitlab-ci-diff --pr-comment


Could there be a problem if CI config changes have been made on main that aren't present on the branch? Would the diff also show these changes?

KSerrania · 2024-07-02T13:28:47Z

tasks/notify.py

+    """
+
+    pr_comment_head = 'Gitlab CI Configuration Changes'
+    job_url = os.environ['CI_JOB_URL']


This makes the task hard to use locally, you have to set the CI_JOB_URL locally, even though it's only used in the case where a PR comment is made.

CelianR added 15 commits June 27, 2024 14:37

[gitlab-lint] Add get-full-gitlab-ci task

07486f9

[gitlab-lint] Prettier print

25cf9e3

[gitlab-lint] Flatten

90ff1c7

[gitlab-lint] Code cleaning

32697d8

[gitlab-lint] Testing

45440db

[gitlab-lint] Sort job keys

cd0e442

[gitlab-lint] Updated filter + clean

884656c

wip

0d634b0

differ: Added

c46b8bf

gitlab ci diff added

8d56198

Renamed to print-gitlab-ci, moved functions to notify.yml

18fec0a

Merge branch 'main' into celian/gitlab-lint-acix-311

09928b4

Moved to notify

f6e24fd

merge

02d2915

[gitlab-config-diff] Works in CLI

09345ba

CelianR added changelog/no-changelog qa/no-code-change Skip QA week as there is no code change in Agent code team/agent-devx-infra labels Jun 28, 2024

CelianR self-assigned this Jun 28, 2024

CelianR requested review from a team as code owners June 28, 2024 13:27

KSerrania reviewed Jun 28, 2024

View reviewed changes

CelianR marked this pull request as draft June 28, 2024 14:39

CelianR added 3 commits June 28, 2024 16:51

[gitlab-config-diff] Applied suggestions

17c5d9d

[gitlab-config-diff] Added pr commenter

b75a17a

[gitlab-config-diff] Fixed pr commenter, made pr comment format + tests

d516213

CelianR changed the base branch from main to celian/gitlab-lint-acix-311 July 1, 2024 11:16

[gitlab-config-diff] PR comments on error + updated format

a59fcde

CelianR added 11 commits July 1, 2024 14:36

[gitlab-config-diff] Notify job

19f4faf

[gitlab-config-diff] Notify job

ffe9caf

[gitlab-config-diff] Notify job

fe73eb2

[gitlab-config-diff] Notify job

27e4102

[gitlab-config-diff] Notify job

56b230c

[gitlab-config-diff] Notify job

ccbf115

[gitlab-config-diff] Notify job

3fdf6d2

[gitlab-config-diff] Notify job

9107885

[gitlab-config-diff] Notify job

0e894a2

[gitlab-config-diff] Notify job

7c81eab

Updated format

b9fc4a4

Base automatically changed from celian/gitlab-lint-acix-311 to main July 2, 2024 08:33

CelianR added 4 commits July 2, 2024 11:12

[gitlab-config-diff] Perms

89763a7

[gitlab-config-diff] Cleaned code, enable pr comment

4a2efe8

[gitlab-config-diff] Updated before commit

30c7c14

[gitlab-config-diff] Updated cli display

9f8e0c1

DataDog deleted a comment from pr-commenter bot Jul 2, 2024

CelianR added 3 commits July 2, 2024 13:21

[gitlab-config-diff] Little fixes

de87c9b

merge

6c5d1d7

[gitlab-config-diff] Cleaned code, updated pr comment format

335cf3d

[gitlab-config-diff] Cleaned code

29662d0

CelianR marked this pull request as ready for review July 2, 2024 12:09

amenasria approved these changes Jul 2, 2024

View reviewed changes

CelianR added 3 commits July 2, 2024 15:04

[gitlab-config-diff] Removed github token task

80c6378

[gitlab-config-diff] Cancelled auth changes

580f418

[gitlab-config-diff] Cleaned code according to reviews

a067f23

KSerrania reviewed Jul 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add gitlab ci diff cli / pr comment #27153

Add gitlab ci diff cli / pr comment #27153

CelianR commented Jun 28, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Jun 28, 2024 •

edited

Loading

pr-commenter bot commented Jun 28, 2024 •

edited

Loading

Fine details of change detection per experiment

Explanation

KSerrania left a comment •

edited

Loading

CelianR commented Jul 2, 2024 •

edited

Loading

notify_failure_summary_daily

notify_gitlab_ci_changes

pr-commenter bot commented Jul 2, 2024 •

edited

Loading

notify_gitlab_ci_changes

amenasria commented Jul 2, 2024

amenasria left a comment

KSerrania Jul 2, 2024

KSerrania Jul 2, 2024

Add gitlab ci diff cli / pr comment #27153

Are you sure you want to change the base?

Add gitlab ci diff cli / pr comment #27153

Conversation

CelianR commented Jun 28, 2024 • edited Loading

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

agent-platform-auto-pr bot commented Jun 28, 2024 • edited Loading

pr-commenter bot commented Jun 28, 2024 • edited Loading

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Fine details of change detection per experiment

Explanation

KSerrania left a comment • edited Loading

Choose a reason for hiding this comment

CelianR commented Jul 2, 2024 • edited Loading

Example Gitlab CI Configuration Changes

Modified Jobs

notify_failure_summary_daily

Added Jobs

notify_gitlab_ci_changes

Renamed Jobs

Changes

pr-commenter bot commented Jul 2, 2024 • edited Loading

Gitlab CI Configuration Changes

Added Jobs

notify_gitlab_ci_changes

Changes

amenasria commented Jul 2, 2024

amenasria left a comment

Choose a reason for hiding this comment

KSerrania Jul 2, 2024

Choose a reason for hiding this comment

KSerrania Jul 2, 2024

Choose a reason for hiding this comment

CelianR commented Jun 28, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Jun 28, 2024 •

edited

Loading

pr-commenter bot commented Jun 28, 2024 •

edited

Loading

KSerrania left a comment •

edited

Loading

CelianR commented Jul 2, 2024 •

edited

Loading

pr-commenter bot commented Jul 2, 2024 •

edited

Loading