*: add API concurrency metrics #7541

CabinfeverB · 2023-12-13T08:12:05Z

What problem does this PR solve?

Issue Number: ref #7167
should be merged after #7239

What is changed and how does it work?

Enable concurrency monitoring for all APIs

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

Code changes

Has the configuration change
Has HTTP API interfaces changed (Don't forget to add the declarative for the new API)
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tiup:
Need to cherry-pick to the release branch

Release note

None.

Signed-off-by: Cabinfever_B <[email protected]>

Signed-off-by: Cabinfever_B <[email protected]> address comment Signed-off-by: Cabinfever_B <[email protected]>

Signed-off-by: Cabinfever_B <[email protected]>

ti-chi-bot · 2023-12-13T08:12:07Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

disksing
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

rleungx · 2023-12-13T08:17:05Z

metrics/grafana/pd.json

+          "dashLength": 10,
+          "dashes": false,
+          "datasource": "${DS_TEST-CLUSTER}",
+          "description": "The concurrency number ofeach kind of gRPC commands",


Suggested change

"description": "The concurrency number ofeach kind of gRPC commands",

"description": "The concurrency number of each kind of gRPC commands",

pkg/ratelimit/concurrency_limiter.go

Signed-off-by: Cabinfever_B <[email protected]>

rleungx · 2023-12-14T09:12:31Z

server/grpc_service.go

@@ -678,7 +744,7 @@ func (s *GrpcServer) GetStore(ctx context.Context, request *pdpb.GetStoreRequest
 	} else if rsp != nil {
 		return rsp.(*pdpb.GetStoreResponse), err
 	}
-
+	time.Sleep(120 * time.Millisecond)


Sorry, I forgot to delete this when addressing the comment. I can't count concurrency without this during manual testing.

need remove it.

rleungx · 2023-12-14T09:55:04Z

pkg/ratelimit/controller.go

+				label := key.(string)
+				// Due to not in hot path, no need to save sub Gauge.
+				if con := limiter.getConcurrencyLimiter(); con != nil {
+					l.concurrencyGauge.WithLabelValues(l.apiType, label).Set(float64(con.getCurrent()))


Maybe we can find another way to show the statistics, the current implementation only captures the instant value every 15s.

Signed-off-by: Cabinfever_B <[email protected]>

rleungx · 2023-12-20T08:42:57Z

pkg/ratelimit/concurrency_limiter.go

@@ -68,3 +74,13 @@ func (l *concurrencyLimiter) getCurrent() uint64 {

 	return l.current
 }
+
+func (l *concurrencyLimiter) getMaxConcurrency() uint64 {
+	l.mu.RLock()


Should be Lock?

Signed-off-by: Cabinfever_B <[email protected]>

HuSharp · 2023-12-21T03:25:50Z

/retest

HuSharp · 2023-12-21T03:26:46Z

/test

ti-chi-bot · 2023-12-21T03:26:48Z

@HuSharp: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

/test build

The following commands are available to trigger optional jobs:

/test pull-integration-copr-test
/test pull-integration-realcluster-test

Use /test all to run the following jobs that were automatically triggered:

tikv/pd/ghpr_build

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

HuSharp · 2023-12-21T03:27:04Z

/test build

Signed-off-by: Cabinfever_B <[email protected]>

codecov · 2023-12-21T03:52:34Z

Codecov Report

Merging #7541 (c34d2b1) into master (cee6e63) will decrease coverage by 0.08%.
The diff coverage is 54.15%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7541      +/-   ##
==========================================
- Coverage   74.87%   74.80%   -0.08%     
==========================================
  Files         459      459              
  Lines       50668    50975     +307     
==========================================
+ Hits        37939    38130     +191     
- Misses       9385     9482      +97     
- Partials     3344     3363      +19

Flag	Coverage Δ
unittests	`74.80% <54.15%> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Signed-off-by: Cabinfever_B <[email protected]>

nolouch

rest lgtm

nolouch · 2023-12-27T07:26:28Z

server/grpc_service.go

@@ -678,7 +744,7 @@ func (s *GrpcServer) GetStore(ctx context.Context, request *pdpb.GetStoreRequest
 	} else if rsp != nil {
 		return rsp.(*pdpb.GetStoreResponse), err
 	}
-
+	time.Sleep(120 * time.Millisecond)


need remove it.

Signed-off-by: Cabinfever_B <[email protected]>

CabinfeverB · 2023-12-28T05:53:08Z

/merge

ti-chi-bot · 2023-12-28T05:53:10Z

@CabinfeverB: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-12-28T05:53:12Z

This pull request has been accepted and is ready to merge.

Commit hash: c34d2b1

ref tikv#7167 Signed-off-by: Cabinfever_B <[email protected]> Signed-off-by: pingandb <[email protected]>

CabinfeverB added 11 commits October 19, 2023 14:19

refactor ratelimit pkg

b8be11c

Signed-off-by: Cabinfever_B <[email protected]>

refactor ratelimit pkg

5c242b8

Signed-off-by: Cabinfever_B <[email protected]>

refactor test

b8cde6f

Signed-off-by: Cabinfever_B <[email protected]>

address comment

753c9d4

Signed-off-by: Cabinfever_B <[email protected]>

Merge branch 'master' into rate_limit/refactor

9fb345f

address comment

8c6a990

Signed-off-by: Cabinfever_B <[email protected]>

address comment

12d9529

Signed-off-by: Cabinfever_B <[email protected]>

address comment

a815282

Signed-off-by: Cabinfever_B <[email protected]> address comment Signed-off-by: Cabinfever_B <[email protected]>

address comment

4748cf5

Signed-off-by: Cabinfever_B <[email protected]>

concurrency metrics

5514e9b

Signed-off-by: Cabinfever_B <[email protected]>

fix conflict

4d70445

Signed-off-by: Cabinfever_B <[email protected]>

ti-chi-bot bot added the release-note-none label Dec 13, 2023

ti-chi-bot bot requested review from disksing and rleungx December 13, 2023 08:12

CabinfeverB removed the request for review from disksing December 13, 2023 08:12

ti-chi-bot bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Dec 13, 2023

CabinfeverB requested a review from nolouch December 13, 2023 08:12

rleungx reviewed Dec 13, 2023

View reviewed changes

CabinfeverB added 3 commits December 13, 2023 17:49

fix conflict

8da0b38

Signed-off-by: Cabinfever_B <[email protected]>

address comment

ed2de59

Signed-off-by: Cabinfever_B <[email protected]>

address comment

c620463

Signed-off-by: Cabinfever_B <[email protected]>

rleungx reviewed Dec 14, 2023

View reviewed changes

CabinfeverB added 2 commits December 18, 2023 15:37

Merge branch 'master' into rate_limit/concurrency_limit

31d416d

address comment

12faadd

Signed-off-by: Cabinfever_B <[email protected]>

rleungx reviewed Dec 20, 2023

View reviewed changes

CabinfeverB added 2 commits December 21, 2023 09:32

address comment

d76914e

Signed-off-by: Cabinfever_B <[email protected]>

fix test

ce33540

Signed-off-by: Cabinfever_B <[email protected]>

CabinfeverB added 2 commits December 21, 2023 11:42

fix test

b52c4f6

Signed-off-by: Cabinfever_B <[email protected]>

Merge branch 'master' into rate_limit/concurrency_limit

e1ef4d2

CabinfeverB added 2 commits December 21, 2023 11:57

fix

469005b

Signed-off-by: Cabinfever_B <[email protected]>

merge master

170acf1

Signed-off-by: Cabinfever_B <[email protected]>

disksing approved these changes Dec 26, 2023

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 26, 2023

merge master

a1f8fd0

Signed-off-by: Cabinfever_B <[email protected]>

nolouch reviewed Dec 27, 2023

View reviewed changes

address comment

bf14f9a

Signed-off-by: Cabinfever_B <[email protected]>

rleungx approved these changes Dec 27, 2023

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 27, 2023

CabinfeverB added 4 commits December 28, 2023 09:40

Merge branch 'master' into rate_limit/concurrency_limit

62bc1d8

fix

b3311b7

Signed-off-by: Cabinfever_B <[email protected]>

fix

d8cab6c

Signed-off-by: Cabinfever_B <[email protected]>

address comment

c34d2b1

Signed-off-by: Cabinfever_B <[email protected]>

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 28, 2023

ti-chi-bot bot merged commit b7b4537 into tikv:master Dec 28, 2023
24 of 26 checks passed

pingandb pushed a commit to pingandb/pd that referenced this pull request Jan 18, 2024

*: add API concurrency metrics (tikv#7541)

1139800

ref tikv#7167 Signed-off-by: Cabinfever_B <[email protected]> Signed-off-by: pingandb <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: add API concurrency metrics #7541

*: add API concurrency metrics #7541

CabinfeverB commented Dec 13, 2023 •

edited

Loading

ti-chi-bot bot commented Dec 13, 2023 •

edited

Loading

rleungx Dec 13, 2023

rleungx Dec 14, 2023

CabinfeverB Dec 14, 2023

nolouch Dec 27, 2023

rleungx Dec 14, 2023

rleungx Dec 20, 2023

HuSharp commented Dec 21, 2023

HuSharp commented Dec 21, 2023

ti-chi-bot bot commented Dec 21, 2023

HuSharp commented Dec 21, 2023

codecov bot commented Dec 21, 2023 •

edited

Loading

nolouch left a comment

nolouch Dec 27, 2023

CabinfeverB commented Dec 28, 2023

ti-chi-bot bot commented Dec 28, 2023

ti-chi-bot bot commented Dec 28, 2023

	"description": "The concurrency number ofeach kind of gRPC commands",
	"description": "The concurrency number of each kind of gRPC commands",

*: add API concurrency metrics #7541

*: add API concurrency metrics #7541

Conversation

CabinfeverB commented Dec 13, 2023 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Dec 13, 2023 • edited Loading

rleungx Dec 13, 2023

Choose a reason for hiding this comment

rleungx Dec 14, 2023

Choose a reason for hiding this comment

CabinfeverB Dec 14, 2023

Choose a reason for hiding this comment

nolouch Dec 27, 2023

Choose a reason for hiding this comment

rleungx Dec 14, 2023

Choose a reason for hiding this comment

rleungx Dec 20, 2023

Choose a reason for hiding this comment

HuSharp commented Dec 21, 2023

HuSharp commented Dec 21, 2023

ti-chi-bot bot commented Dec 21, 2023

HuSharp commented Dec 21, 2023

codecov bot commented Dec 21, 2023 • edited Loading

Codecov Report

nolouch left a comment

Choose a reason for hiding this comment

nolouch Dec 27, 2023

Choose a reason for hiding this comment

CabinfeverB commented Dec 28, 2023

ti-chi-bot bot commented Dec 28, 2023

ti-chi-bot bot commented Dec 28, 2023

CabinfeverB commented Dec 13, 2023 •

edited

Loading

ti-chi-bot bot commented Dec 13, 2023 •

edited

Loading

codecov bot commented Dec 21, 2023 •

edited

Loading