initial draft (to be reviewed) for API def. #2

kmrdhruv · 2023-04-18T04:41:24Z

Initial API draft for tenant/topic/subscription entity mgmt, produce and dlq consume.

…ersion)

gauravAshok · 2023-04-24T08:28:16Z

docs/api.yaml

+          example: 1500
+        message_size:
+          type: integer
+          description: "Maximum message size in KBs."


i think avg msg size is more important metric. But depends on, what do we intend to use this info for?

this i just added for reference/discussion. We should keep either one of them.

gauravAshok · 2023-04-24T08:29:10Z

docs/api.yaml

+          example: 10
+        max_storage:
+          type: integer
+          description: "Maximum size in MB"


if we have avg msg size & throughput & retention policy. Will this info be useful? It looks to be inferable.

i was more inclined towards max_storage than msg size as msg size might vary and tough to guess avg size. Also max storage might help in force purge or billing aspect.

gauravAshok · 2023-04-24T08:31:56Z

docs/api.yaml

+      type: object
+      properties:
+        max_throughput:
+          type: integer


should we make it more human friendly, like
10Kb, 1024B, 1MB. The units can be assumed to be per sec. and we only ask for string that can be parsed as unit of some bytes.

or we reduce the scale?. just ask KB or B. Mb might be too large.

merge both retention and capacity, also only throughput.

gauravAshok · 2023-04-24T12:11:44Z

docs/api.yaml

+        multiplier:
+          type: integer
+          description: "step for iteration"
+          example: 10
+        retry_attempts:
+          type: integer
+          description: "maximum retry attempts allowed."
+          maximum: 3


so there are "small" / frequent retries that we we can do inline (i.e. during consumption itself), but there are longer retries that we can do using topics.

How do configs map to the above behaviour?

As discussed, multiplier is just that, its description may not be be correct.
So, now the current policy will be backed by topics.
From implementation point of view, there may be scope of doing some initial retries locally only, before we park such messages in to RQ. do we need to make that explicit as part of policy.

gauravAshok · 2023-04-25T08:09:26Z

docs/api.yaml

+        dlq_from_recovery:
+          type: number
+          example: 0.5


I am not too inclined on "dlq_from_recovery" ratio. Makes things slightly complicated, too fine grained which users probably can't manage.

since messages in RQ & DLQ are "failed" messages, users might want to prioritize one over the other, dependeing on their usecase. Cant we just say, "recovery_allocation" generally applies to recovery of failed messages, doen't matter RQ / DLQ. By default RQ messages will be prioritized.
If need be we can just provide the toggle to prioritize RQ or DLQ over the other, (as this probably will be business usecase specific).

Also, just to point out the behaviour, one nuance that a plain "0.5" ratio doesn't capture, is that it is max allocation. if there are not enough RQ / DLQ messages, the unused allocation is just kind of used for main messages. Similar behaviour when prioritizing RQ over DLQ, if there are not enough messages/parallelism in RQ, then DLQ will be consumed from pending parallelism quota.

gauravAshok · 2023-04-25T08:13:20Z

docs/api.yaml

+        dlq_from_recovery:
+          type: number
+          example: 0.5
+        max_error_threshold:


In case of grouped subscription, will the error_rate be calculated in terms of failed groups, instead of messages failing to RQ / DLQ. right?

I was thinking of later.. as primarily MTTR and also rq/dq size will be determined by number of messages.

gauravAshok · 2023-04-25T08:56:54Z

docs/api.yaml

+              type: integer
+              description: "wait in seconds before attempting next increase/decrease step."
+              example: 10
+            ping_seconds:


how is ping achieved?

may be needs renaming...
post complete throttle, consumer needs to continuously check at regular interval if service has come back and start recovery process. This regular interval is being called as ping_seconds.
Think of it as attempt to deliver the message continuously where failure doesn't move the msg to different queue.

gauravAshok · 2023-04-25T09:16:13Z

docs/api.yaml

+          $ref: '#/components/schemas/dlq_policy'
+        consumption_policy:
+          $ref: '#/components/schemas/consumption_policy'
+        version:


when we say version, we mean schema version right?

This was object versioning and not schema versioning.
Shall we cover schema version using API versioning ?

gauravAshok · 2023-04-25T09:16:52Z

docs/api.yaml

+          description: "Not used for now. Intended for message deduplication semantics."
+        properties:
+          $ref: '#/components/schemas/property_bag'
+        groupkey:


Suggested change

groupkey:

group_key:

gauravAshok · 2023-04-25T09:56:24Z

docs/api.yaml

+          type: integer 
+
+  parameters:
+    tenanat_param:


Suggested change

tenanat_param:

tenant_param:

gauravAshok · 2023-04-25T11:00:29Z

docs/api.yaml

+          $ref: '#/components/responses/error_response'
+
+
+  /v1/tenants/{tenant_name}/subscriptions/{sub_name}/status:


how is this status different from state.

This state vs status, led me to search the difference between them. I think I align with this https://softwareengineering.stackexchange.com/questions/219351/state-or-status-when-should-a-variable-name-contain-the-word-state-and-w

I can understand the active, inactive (/ blocked), running, stopped. being the status of the subscription. Similarly, active, inactive (/blocked) for the topic status.

Is consumer_status for internal details of the consumer, lag and other stuff? Internal Stats?

intresting.. I was thinking of it as "state" being more like state-machine state, status being more detailed running info. kind of single value and multi-value attribute.
Both info as of now intended for external consumption.

stats/info instead of status.

gauravAshok · 2023-04-25T11:43:13Z

docs/api.yaml

+      - $ref: '#/components/parameters/from_param'
+      - $ref: '#/components/parameters/to_param'
+      - $ref: '#/components/parameters/msg_id_param'
+      - $ref: '#/components/parameters/group_id_param'


group_id vs group_key. we can stick to one.

aligned .. any preferences ?

gauravAshok · 2023-04-25T11:47:40Z

docs/api.yaml

+
+    put:
+      description: >-
+        Unsideline messages from DLT for further consumption.


How will this unsideline api interact with normal consumer that is running for all messages present in MQ,RQ,DLQ.
If a filter is provided, will that stop normal DLQ consumption and only consume matching messages (without moving pointer)?

As path includes "dlt", not applicable for MQ/RQ.
get is returning msgs w/o moving the pointer .. kind of idempotent.
put is modifying the dlt internal state by accepting the unsideline request i.e. moving the pointer on consumption.

Request tracking mechanism needs to be figured out.

* adding a basic authz layer. routeDefinitions describe the required permission, and the authz handler checks that against the authz provider. * adding LICENSE

initial draft (to be reviewed) for API def.

abdb592

kmrdhruv requested a review from gauravAshok April 18, 2023 04:41

Updating with latest definition (previous checkin somehow had stale v…

8e78d3c

…ersion)

kmrdhruv requested a review from anvinjain April 20, 2023 08:07

gauravAshok reviewed Apr 24, 2023

View reviewed changes

gauravAshok reviewed Apr 25, 2023

View reviewed changes

docs/api.yaml Outdated

type: integer

parameters:

tenanat_param:

Copy link

Collaborator

gauravAshok Apr 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

tenanat_param:

tenant_param:

gauravAshok reviewed Apr 25, 2023

View reviewed changes

gauravAshok and others added 5 commits May 3, 2023 18:09

Adding a basic authz layer. (#3)

d4b3075

* adding a basic authz layer. routeDefinitions describe the required permission, and the authz handler checks that against the authz provider. * adding LICENSE

draft update with few alternative url formats

df15191

Marking status as draft

04f670a

adding swagger-ui on git hub pages

cec0a4d

rename as per github pages

8ffdda3

kmrdhruv temporarily deployed to github-pages May 11, 2023 16:25 — with GitHub Pages Inactive

marking api specification status as draft

dbed9cc

kmrdhruv temporarily deployed to github-pages May 11, 2023 17:05 — with GitHub Pages Inactive

kmrdhruv temporarily deployed to github-pages May 11, 2023 17:11 — with GitHub Pages Inactive

Dhruv Kumar added 2 commits May 22, 2023 11:31

basic entity creations for Topic creation flow.

9287123

removing changes to messaging project.

904fb8a

kmrdhruv temporarily deployed to github-pages May 22, 2023 06:07 — with GitHub Pages Inactive

initial version of topic create

c883423

kmrdhruv temporarily deployed to github-pages May 29, 2023 09:04 — with GitHub Pages Inactive

minor readme formatting changes

163cbba

kmrdhruv temporarily deployed to github-pages May 29, 2023 09:36 — with GitHub Pages Inactive

Update README.md

17feb30

kmrdhruv had a problem deploying to github-pages May 29, 2023 10:27 — with GitHub Pages Failure

kmrdhruv merged commit dd5db9e into master May 29, 2023
2 of 3 checks passed

kmrdhruv had a problem deploying to github-pages June 1, 2023 05:11 — with GitHub Pages Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial draft (to be reviewed) for API def. #2

initial draft (to be reviewed) for API def. #2

kmrdhruv commented Apr 18, 2023

gauravAshok Apr 24, 2023 •

edited

Loading

kmrdhruv Apr 25, 2023 •

edited

Loading

gauravAshok Apr 24, 2023

kmrdhruv Apr 25, 2023

gauravAshok Apr 24, 2023 •

edited

Loading

kmrdhruv Apr 26, 2023

gauravAshok Apr 24, 2023

gauravAshok Apr 25, 2023

gauravAshok Apr 25, 2023 •

edited

Loading

gauravAshok Apr 25, 2023

kmrdhruv Apr 26, 2023

gauravAshok Apr 25, 2023

kmrdhruv Apr 25, 2023

gauravAshok Apr 25, 2023

kmrdhruv Apr 25, 2023

gauravAshok Apr 25, 2023

gauravAshok Apr 25, 2023

gauravAshok Apr 25, 2023 •

edited

Loading

kmrdhruv Apr 25, 2023

kmrdhruv Apr 26, 2023

gauravAshok Apr 25, 2023

kmrdhruv Apr 25, 2023

gauravAshok Apr 25, 2023

kmrdhruv Apr 25, 2023 •

edited

Loading

kmrdhruv Apr 26, 2023

		$ref: '#/components/responses/error_response'


		/v1/tenants/{tenant_name}/subscriptions/{sub_name}/status:

initial draft (to be reviewed) for API def. #2

initial draft (to be reviewed) for API def. #2

Conversation

kmrdhruv commented Apr 18, 2023

gauravAshok Apr 24, 2023 • edited Loading

Choose a reason for hiding this comment

kmrdhruv Apr 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gauravAshok Apr 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gauravAshok Apr 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gauravAshok Apr 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmrdhruv Apr 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gauravAshok Apr 24, 2023 •

edited

Loading

kmrdhruv Apr 25, 2023 •

edited

Loading

gauravAshok Apr 24, 2023 •

edited

Loading

gauravAshok Apr 25, 2023 •

edited

Loading

gauravAshok Apr 25, 2023 •

edited

Loading

kmrdhruv Apr 25, 2023 •

edited

Loading