Fix the recordings to allow promptflow runs. #3477

nick863 · 2024-06-28T01:09:25Z

Description

The promptflow creates a lot of temporary files on-the -ground, which are being transferred to the cloud. In this PR I have added the code to sanitize the run names and added other fixes to allow run recordings.

The run name and experiment name are now sanitized as well as other ID-s.
Added the sanitized tracking URI in the recordings
PF run first checks if the run with the same name exists and if it does it gets 404. If we continue run, we are getting two different responses to the same run name request: first 404 and then 200 (when the run was already created). These requests are the same and if failed with 404 request will be recorded, the second i.e. continuing run will fail. I have added the code to overwrite only failed runs (code >= 400)
The resulting cassettes were re recorded.

See work item 3305909

All Promptflow Contribution checklist:

The pull request does not introduce [breaking changes].
CHANGELOG is updated for new features, bug fixes or other significant changes.
I have read the contribution guidelines.
Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: suggested workflow.

General Guidelines and Best Practices

Title of the pull request is clear and informative.
There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

Pull request includes test coverage for the included changes.

…irovins/fix_pf_recordings

github-actions · 2024-06-28T01:13:34Z

promptflow SDK CLI Azure E2E Test Result nirovins/fix_pf_recordings

4 files 4 suites 4m 18s ⏱️
244 tests 204 ✅ 40 💤 0 ❌
976 runs 816 ✅ 160 💤 0 ❌

Results for commit a75fe00.

♻️ This comment has been updated with latest results.

github-actions · 2024-06-28T01:16:35Z

promptflow-evals test result

9 files 9 suites 2h 42m 28s ⏱️
54 tests 49 ✅ 5 💤 0 ❌
486 runs 441 ✅ 45 💤 0 ❌

Results for commit a75fe00.

♻️ This comment has been updated with latest results.

github-actions · 2024-06-28T01:34:54Z

SDK CLI Test Result nirovins/fix_pf_recordings

4 files 4 suites 1h 8m 30s ⏱️
786 tests 763 ✅ 23 💤 0 ❌
3 144 runs 3 052 ✅ 92 💤 0 ❌

Results for commit a75fe00.

♻️ This comment has been updated with latest results.

…irovins/fix_pf_recordings

zhengfeiwang · 2024-07-03T01:44:45Z

src/promptflow-recording/promptflow/recording/azure/utils.py

+    return value
+
+
+def sanitize_pf_run_ids(value: str) -> str:


just curious why we need such a utility here? is there any limitation that randstr cannot support the scenario?

in pf-azure tests, it works well so that we don't need to apply sanitization, so i'd like to know the limitation here.

zhengfeiwang · 2024-07-03T01:46:40Z

src/promptflow-recording/promptflow/recording/azure/utils.py

    # Sanitize telemetry event
    if isinstance(body_dict, list) and "Microsoft.ApplicationInsights.Event" in body:
        body_dict = SanitizedValues.FAKE_APP_INSIGHTS
+    # Sanitize the artifact paths


code missing here, or reluctant comment line?

zhengfeiwang · 2024-07-03T01:48:37Z

...ests/recordings/azure/test_metrics_upload_TestMetricsUpload_test_writing_to_run_history.yaml

-        "parentRunUuid": null, "rootRunUuid": "a5390638-bc06-4129-b24f-2a96292f0df3",
-        "lastStartTimeUtc": "2024-06-18T18:21:00.126+00:00", "currentComputeTime":
-        "00:00:00", "computeDuration": null, "effectiveStartTimeUtc": "2024-06-18T18:21:00.126+00:00",
+        "token": "eyJhbGciOiJSUzI1NiIsImtpZCI6IkZBMTRBODAyQjhDRTQ0MjQxRjE0NkU1RDgyNUQwNzJDODE3NjM0ODQiLCJ0eXAiOiJKV1QifQ.eyJyb2xlIjoiQ29udHJpYnV0b3IiLCJzY29wZSI6Ii9zdWJzY3JpcHRpb25zL2IxNzI1M2ZhLWYzMjctNDJkNi05Njg2LWYzZTU1M2UyNDc2My9yZXNvdXJjZUdyb3Vwcy9uaXJvdmlucy1yZy1lYXN0dXMvcHJvdmlkZXJzL01pY3Jvc29mdC5NYWNoaW5lTGVhcm5pbmdTZXJ2aWNlcy93b3Jrc3BhY2VzL25pcm92aW5zLWFpLXByb2plY3QiLCJhY2NvdW50aWQiOiIwMDAwMDAwMC0wMDAwLTAwMDAtMDAwMC0wMDAwMDAwMDAwMDAiLCJ3b3Jrc3BhY2VJZCI6ImY2YTcyYTc0LWY1NWEtNGRiOS04NmNmLTBiNTc3ZTZkYWQzZCIsInByb2plY3RpZCI6IjAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCIsImRpc2NvdmVyeSI6InVyaTovL2Rpc2NvdmVyeXVyaS8iLCJ0aWQiOiI3MmY5ODhiZi04NmYxLTQxYWYtOTFhYi0yZDdjZDAxMWRiNDciLCJvaWQiOiI3ODc1YmFhNi0wMjhkLTQ5YzgtYmQ0ZC1hNzcyYWE0ZGMwMTMiLCJwdWlkIjoiMTAwM0JGRkRBOEE0RDBFNyIsImlzcyI6ImF6dXJlbWwiLCJhcHBpZCI6Ik5pa29sYXkgUm92aW5za2l5IiwiZXhwIjoxNzIxMzUwNjYyLCJhdWQiOiJhenVyZW1sIn0.STZiicI55Jz8o0PSqzP3641BU7wgxpe5z68nnRlQbtE2UNEu_MawQl156nHHFe8On7h-_9T2aphGlou_iEO1dQfG9TkckNV86Hbl-qwKB_vDusdd9FEM6u87EiXZYJ6m9YOqQBTs0DSzYaEOn30mW0lRXvdvb-bZCKYs4ICSkR_JMEkvs8qdHAIWxe9puMC8KGKm_VHVP0zuklsFwOw6XJ05CZ06H8mf9cN948kPKgDBlTXN9TMzS02qoVHra_Vnyt84QRMVBCAfdrLghc4BrmEN75S0o9upc2VeTaY_8JhXo51UQP3qBpRohlNRU0vx1Bv_kytu8uNPuq5FiXuEWQ",


maybe it's better to sanitize this

nick863 added 2 commits June 27, 2024 17:57

Fix recordings to be able to record the promptflow runs.

b4de485

Merge branch 'main' of https://github.com/microsoft/promptflow into n…

36ed95e

…irovins/fix_pf_recordings

nick863 requested review from a team as code owners June 28, 2024 01:09

nick863 temporarily deployed to internal June 28, 2024 01:09 — with GitHub Actions Inactive

github-actions bot added the promptflow-evals label Jun 28, 2024

nick863 added 2 commits June 28, 2024 10:02

Fix

48e6c18

Merge branch 'main' of https://github.com/microsoft/promptflow into n…

c84cb66

…irovins/fix_pf_recordings

nick863 temporarily deployed to internal June 28, 2024 17:13 — with GitHub Actions Inactive

nick863 added 2 commits July 1, 2024 16:06

Fix the absense of cosmos DB request

60aa654

Merge branch 'main' of https://github.com/microsoft/promptflow into n…

3228684

…irovins/fix_pf_recordings

nick863 temporarily deployed to internal July 1, 2024 23:06 — with GitHub Actions Inactive

nick863 had a problem deploying to internal July 1, 2024 23:06 — with GitHub Actions Failure

nick863 temporarily deployed to internal July 1, 2024 23:06 — with GitHub Actions Inactive

Add temporary logging

9da7d8f

nick863 temporarily deployed to internal July 1, 2024 23:52 — with GitHub Actions Inactive

Allow tracing on live test

2158350

nick863 had a problem deploying to internal July 2, 2024 16:51 — with GitHub Actions Failure

nick863 temporarily deployed to internal July 2, 2024 17:27 — with GitHub Actions Inactive

nick863 had a problem deploying to internal July 2, 2024 17:27 — with GitHub Actions Failure

nagkumar91 approved these changes Jul 2, 2024

View reviewed changes

nick863 temporarily deployed to internal July 2, 2024 17:51 — with GitHub Actions Inactive

nick863 had a problem deploying to internal July 2, 2024 17:51 — with GitHub Actions Failure

nick863 temporarily deployed to internal July 2, 2024 18:16 — with GitHub Actions Inactive

nick863 had a problem deploying to internal July 2, 2024 18:16 — with GitHub Actions Failure

nick863 temporarily deployed to internal July 2, 2024 18:41 — with GitHub Actions Inactive

Merge branch 'main' of https://github.com/microsoft/promptflow into n…

a75fe00

…irovins/fix_pf_recordings

nick863 temporarily deployed to internal July 2, 2024 19:39 — with GitHub Actions Inactive

zhengfeiwang reviewed Jul 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix the recordings to allow promptflow runs. #3477

Fix the recordings to allow promptflow runs. #3477

nick863 commented Jun 28, 2024

github-actions bot commented Jun 28, 2024 •

edited

Loading

github-actions bot commented Jun 28, 2024 •

edited

Loading

github-actions bot commented Jun 28, 2024 •

edited

Loading

zhengfeiwang Jul 3, 2024 •

edited

Loading

zhengfeiwang Jul 3, 2024 •

edited

Loading

zhengfeiwang Jul 3, 2024

Fix the recordings to allow promptflow runs. #3477

Are you sure you want to change the base?

Fix the recordings to allow promptflow runs. #3477

Conversation

nick863 commented Jun 28, 2024

Description

All Promptflow Contribution checklist:

General Guidelines and Best Practices

Testing Guidelines

github-actions bot commented Jun 28, 2024 • edited Loading

promptflow SDK CLI Azure E2E Test Result nirovins/fix_pf_recordings

github-actions bot commented Jun 28, 2024 • edited Loading

promptflow-evals test result

github-actions bot commented Jun 28, 2024 • edited Loading

SDK CLI Test Result nirovins/fix_pf_recordings

zhengfeiwang Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

zhengfeiwang Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

zhengfeiwang Jul 3, 2024

Choose a reason for hiding this comment

github-actions bot commented Jun 28, 2024 •

edited

Loading

github-actions bot commented Jun 28, 2024 •

edited

Loading

github-actions bot commented Jun 28, 2024 •

edited

Loading

zhengfeiwang Jul 3, 2024 •

edited

Loading

zhengfeiwang Jul 3, 2024 •

edited

Loading