Skip to content

Commit

Permalink
Update HTTP guide with more details, add in GIF showing run as well
Browse files Browse the repository at this point in the history
  • Loading branch information
@pflooky
pflooky committed Nov 10, 2023
1 parent d886e15 commit d7452e0
Show file tree
Hide file tree
Showing 8 changed files with 123 additions and 67 deletions.
Binary file added docs/diagrams/http_generation_run.gif
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
45 changes: 27 additions & 18 deletions docs/setup/guide/data-source/http.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ description: "Automatically generate data for OpenAPI/Swagger to HTTP endpoints.

Creating a data generator based on an [OpenAPI/Swagger](https://spec.openapis.org/oas/latest.html) document.

![Generate HTTP requests](../../../diagrams/http_generation_run.gif)

## Requirements

- 10 minutes
Expand Down Expand Up @@ -145,11 +147,12 @@ Then it gets used as a path parameter in the DELETE and GET requests.
To link them all together, we must follow a particular pattern when referring to request body, query parameter or path
parameter columns.

| HTTP Type | Column Prefix | Example |
|-----------------|---------------|------------------|
| Request Body | `bodyContent` | `bodyContent.id` |
| Path Parameter | `pathParam` | `pathParamid` |
| Query Parameter | `queryParam` | `queryParamid` |
| HTTP Type | Column Prefix | Example |
|-----------------|---------------|----------------------|
| Request Body | `bodyContent` | `bodyContent.id` |
| Path Parameter | `pathParam` | `pathParamid` |
| Query Parameter | `queryParam` | `queryParamid` |
| Header | `header` | `headerContent_Type` |

Also note, that when creating a foreign field definition for a HTTP data source, to refer to a specific endpoint and
method, we have to follow the pattern of `{http method}{http path}`. For example, `POST/pets`. Let's apply this
Expand All @@ -159,9 +162,9 @@ knowledge to link all the `id` values together.

```java
var myPlan = plan().addForeignKeyRelationship(
foreignField("my_http", "POST/pets", "bodyContent.id"),
foreignField("my_http", "GET/pets/{id}", "pathParamid"),
foreignField("my_http", "DELETE/pets/{id}", "pathParamid")
foreignField("my_http", "POST/pets", "bodyContent.id"), //source of foreign key value
foreignField("my_http", "DELETE/pets/{id}", "pathParamid"),
foreignField("my_http", "GET/pets/{id}", "pathParamid")
);

execute(myPlan, conf, httpTask);
Expand All @@ -171,9 +174,9 @@ knowledge to link all the `id` values together.

```scala
val myPlan = plan.addForeignKeyRelationship(
foreignField("my_http", "POST/pets", "bodyContent.id"),
foreignField("my_http", "GET/pets/{id}", "pathParamid"),
foreignField("my_http", "DELETE/pets/{id}", "pathParamid")
foreignField("my_http", "POST/pets", "bodyContent.id"), //source of foreign key value
foreignField("my_http", "DELETE/pets/{id}", "pathParamid"),
foreignField("my_http", "GET/pets/{id}", "pathParamid")
)

execute(myPlan, conf, httpTask)
Expand Down Expand Up @@ -247,31 +250,37 @@ docker logs -f docker-http-1
172.21.0.1 [06/Nov/2023:01:45:57 +0000] GET /anything/pets/ID20618951 HTTP/1.1 200 Host: host.docker.internal}
```

Great! Now we have replicated a production-like flow of HTTP requests.
Great! Now we have replicated a production-like flow of HTTP requests.

### Ordering

If you wanted to change the ordering of the requests, you can alter the order from within the OpenAPI/Swagger document.
This is particularly useful when you want to simulate the same flow that users would take when utilising your
application (i.e. create account, query account, update account).

### Rows per second

By default, Data Caterer will push requests per method and endpoint at a rate of around 5 requests per second. If you
want to alter this value, you can do so via the below configuration.
By default, Data Caterer will push requests per method and endpoint at a rate of around 5 requests per second. If you
want to alter this value, you can do so via the below configuration. The lowest supported requests per second is 1.

=== "Java"

```java
var httpTask = http("my_http")
import com.github.pflooky.datacaterer.api.model.Constants;

...
var httpTask = http("my_http", Map.of(Constants.ROWS_PER_SECOND(), "1"))
...
.count(count().records(2));
```

=== "Scala"

```scala
val httpTask = http("my_http")
import com.github.pflooky.datacaterer.api.model.Constants.ROWS_PER_SECOND

...
val httpTask = http("my_http", options = Map(ROWS_PER_SECOND -> "1"))
...
.count(count.records(2))
```

Check out the full example under `AdvancedHttpPlanRun` in the example repo.
Binary file added site/diagrams/http_generation_run.gif
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion site/search/search_index.json
View file

Large diffs are not rendered by default.

40 changes: 26 additions & 14 deletions site/setup/guide/data-source/http/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1925,6 +1925,7 @@ <h1 id="http-source">HTTP Source</h1>
<p>Generating data based on OpenAPI/Swagger document and pushing to HTTP endpoint is a paid feature. Try the free trial <a href="../../../../get-started/docker/">here</a>.</p>
</div>
<p>Creating a data generator based on an <a href="https://spec.openapis.org/oas/latest.html">OpenAPI/Swagger</a> document.</p>
<p><img alt="Generate HTTP requests" src="../../../../diagrams/http_generation_run.gif" /></p>
<h2 id="requirements">Requirements</h2>
<ul>
<li>10 minutes</li>
Expand Down Expand Up @@ -2057,6 +2058,11 @@ <h3 id="foreign-keys">Foreign keys</h3>
<td><code>queryParam</code></td>
<td><code>queryParamid</code></td>
</tr>
<tr>
<td>Header</td>
<td><code>header</code></td>
<td><code>headerContent_Type</code></td>
</tr>
</tbody>
</table>
<p>Also note, that when creating a foreign field definition for a HTTP data source, to refer to a specific endpoint and
Expand All @@ -2066,19 +2072,19 @@ <h3 id="foreign-keys">Foreign keys</h3>
<div class="tabbed-content">
<div class="tabbed-block">
<div class="highlight"><pre><span></span><code><span class="kd">var</span><span class="w"> </span><span class="n">myPlan</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">plan</span><span class="p">().</span><span class="na">addForeignKeyRelationship</span><span class="p">(</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;POST/pets&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;bodyContent.id&quot;</span><span class="p">),</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;GET/pets/{id}&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;pathParamid&quot;</span><span class="p">),</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;DELETE/pets/{id}&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;pathParamid&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;POST/pets&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;bodyContent.id&quot;</span><span class="p">),</span><span class="w"> </span><span class="c1">//source of foreign key value</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;DELETE/pets/{id}&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;pathParamid&quot;</span><span class="p">),</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;GET/pets/{id}&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;pathParamid&quot;</span><span class="p">)</span>
<span class="p">);</span>

<span class="n">execute</span><span class="p">(</span><span class="n">myPlan</span><span class="p">,</span><span class="w"> </span><span class="n">conf</span><span class="p">,</span><span class="w"> </span><span class="n">httpTask</span><span class="p">);</span>
</code></pre></div>
</div>
<div class="tabbed-block">
<div class="highlight"><pre><span></span><code><span class="kd">val</span><span class="w"> </span><span class="n">myPlan</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">plan</span><span class="p">.</span><span class="n">addForeignKeyRelationship</span><span class="p">(</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;POST/pets&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;bodyContent.id&quot;</span><span class="p">),</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;GET/pets/{id}&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;pathParamid&quot;</span><span class="p">),</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;DELETE/pets/{id}&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;pathParamid&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;POST/pets&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;bodyContent.id&quot;</span><span class="p">),</span><span class="w"> </span><span class="c1">//source of foreign key value</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;DELETE/pets/{id}&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;pathParamid&quot;</span><span class="p">),</span>
<span class="w"> </span><span class="n">foreignField</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;GET/pets/{id}&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;pathParamid&quot;</span><span class="p">)</span>
<span class="p">)</span>

<span class="n">execute</span><span class="p">(</span><span class="n">myPlan</span><span class="p">,</span><span class="w"> </span><span class="n">conf</span><span class="p">,</span><span class="w"> </span><span class="n">httpTask</span><span class="p">)</span>
Expand Down Expand Up @@ -2140,24 +2146,30 @@ <h3 id="custom-metadata">Custom metadata</h3>
<span class="m">172</span>.21.0.1<span class="w"> </span><span class="o">[</span><span class="m">06</span>/Nov/2023:01:45:57<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>GET<span class="w"> </span>/anything/pets/ID55185420<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span><span class="w"> </span>Host:<span class="w"> </span>host.docker.internal<span class="o">}</span>
<span class="m">172</span>.21.0.1<span class="w"> </span><span class="o">[</span><span class="m">06</span>/Nov/2023:01:45:57<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>GET<span class="w"> </span>/anything/pets/ID20618951<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span><span class="w"> </span>Host:<span class="w"> </span>host.docker.internal<span class="o">}</span>
</code></pre></div>
<p>Great! Now we have replicated a production-like flow of HTTP requests. </p>
<p>Great! Now we have replicated a production-like flow of HTTP requests.</p>
<h3 id="ordering">Ordering</h3>
<p>If you wanted to change the ordering of the requests, you can alter the order from within the OpenAPI/Swagger document.</p>
<p>If you wanted to change the ordering of the requests, you can alter the order from within the OpenAPI/Swagger document.
This is particularly useful when you want to simulate the same flow that users would take when utilising your
application (i.e. create account, query account, update account).</p>
<h3 id="rows-per-second">Rows per second</h3>
<p>By default, Data Caterer will push requests per method and endpoint at a rate of around 5 requests per second. If you
want to alter this value, you can do so via the below configuration.</p>
<p>By default, Data Caterer will push requests per method and endpoint at a rate of around 5 requests per second. If you
want to alter this value, you can do so via the below configuration. The lowest supported requests per second is 1.</p>
<div class="tabbed-set tabbed-alternate" data-tabs="5:2"><input checked="checked" id="__tabbed_5_1" name="__tabbed_5" type="radio" /><input id="__tabbed_5_2" name="__tabbed_5" type="radio" /><div class="tabbed-labels"><label for="__tabbed_5_1">Java</label><label for="__tabbed_5_2">Scala</label></div>
<div class="tabbed-content">
<div class="tabbed-block">
<div class="highlight"><pre><span></span><code><span class="kd">var</span><span class="w"> </span><span class="n">httpTask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">http</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">)</span>
<div class="highlight"><pre><span></span><code><span class="kn">import</span><span class="w"> </span><span class="nn">com.github.pflooky.datacaterer.api.model.Constants</span><span class="p">;</span>

<span class="p">...</span>
<span class="kd">var</span><span class="w"> </span><span class="n">httpTask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">http</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">Map</span><span class="p">.</span><span class="na">of</span><span class="p">(</span><span class="n">Constants</span><span class="p">.</span><span class="na">ROWS_PER_SECOND</span><span class="p">(),</span><span class="w"> </span><span class="s">&quot;1&quot;</span><span class="p">))</span>
<span class="w"> </span><span class="p">...</span>
<span class="w"> </span><span class="p">.</span><span class="na">count</span><span class="p">(</span><span class="n">count</span><span class="p">().</span><span class="na">records</span><span class="p">(</span><span class="mi">2</span><span class="p">));</span>
</code></pre></div>
</div>
<div class="tabbed-block">
<div class="highlight"><pre><span></span><code><span class="kd">val</span><span class="w"> </span><span class="n">httpTask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">http</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">)</span>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="nn">com</span><span class="p">.</span><span class="nn">github</span><span class="p">.</span><span class="nn">pflooky</span><span class="p">.</span><span class="nn">datacaterer</span><span class="p">.</span><span class="nn">api</span><span class="p">.</span><span class="nn">model</span><span class="p">.</span><span class="nc">Constants</span><span class="p">.</span><span class="nc">ROWS_PER_SECOND</span>

<span class="p">...</span>
<span class="kd">val</span><span class="w"> </span><span class="n">httpTask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">http</span><span class="p">(</span><span class="s">&quot;my_http&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">options</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">Map</span><span class="p">(</span><span class="nc">ROWS_PER_SECOND</span><span class="w"> </span><span class="o">-&gt;</span><span class="w"> </span><span class="s">&quot;1&quot;</span><span class="p">))</span>
<span class="w"> </span><span class="p">...</span>
<span class="w"> </span><span class="p">.</span><span class="n">count</span><span class="p">(</span><span class="n">count</span><span class="p">.</span><span class="n">records</span><span class="p">(</span><span class="mi">2</span><span class="p">))</span>
</code></pre></div>
</div>
</div>
Expand Down
35 changes: 35 additions & 0 deletions site/setup/guide/data-source/open-metadata-source/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1097,6 +1097,17 @@
</span>
</a>

</li>

<li class="md-nav__item">
<a href="#data-validation" class="md-nav__link">
<span class="md-ellipsis">

Data validation

</span>
</a>

</li>

</ul>
Expand Down Expand Up @@ -2179,6 +2190,30 @@ <h3 id="custom-metadata">Custom metadata</h3>
</code></pre></div>
<p>Great! Now we have the ability to get schema information from an external source, add our own metadata and generate
data.</p>
<h3 id="data-validation">Data validation</h3>
<p>Another aspect of OpenMetadata that can be leveraged is the definition of data quality rules. These rules can be
incorporated into your Data Caterer job as well by enabling data validations via <code>enableGenerateValidations</code> in
<code>configuration</code>.</p>
<div class="tabbed-set tabbed-alternate" data-tabs="4:2"><input checked="checked" id="__tabbed_4_1" name="__tabbed_4" type="radio" /><input id="__tabbed_4_2" name="__tabbed_4" type="radio" /><div class="tabbed-labels"><label for="__tabbed_4_1">Java</label><label for="__tabbed_4_2">Scala</label></div>
<div class="tabbed-content">
<div class="tabbed-block">
<div class="highlight"><pre><span></span><code><span class="kd">var</span><span class="w"> </span><span class="n">conf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">configuration</span><span class="p">().</span><span class="na">enableGeneratePlanAndTasks</span><span class="p">(</span><span class="kc">true</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="na">enableGenerateValidations</span><span class="p">(</span><span class="kc">true</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="na">generatedReportsFolderPath</span><span class="p">(</span><span class="s">&quot;/opt/app/data/report&quot;</span><span class="p">);</span>

<span class="n">execute</span><span class="p">(</span><span class="n">conf</span><span class="p">,</span><span class="w"> </span><span class="n">jsonTask</span><span class="p">);</span>
</code></pre></div>
</div>
<div class="tabbed-block">
<div class="highlight"><pre><span></span><code><span class="kd">val</span><span class="w"> </span><span class="n">conf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">configuration</span><span class="p">.</span><span class="n">enableGeneratePlanAndTasks</span><span class="p">(</span><span class="kc">true</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">enableGenerateValidations</span><span class="p">(</span><span class="kc">true</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">generatedReportsFolderPath</span><span class="p">(</span><span class="s">&quot;/opt/app/data/report&quot;</span><span class="p">)</span>

<span class="n">execute</span><span class="p">(</span><span class="n">conf</span><span class="p">,</span><span class="w"> </span><span class="n">jsonTask</span><span class="p">)</span>
</code></pre></div>
</div>
</div>
</div>
<p>Check out the full example under <code>AdvancedOpenMetadataSourcePlanRun</code> in the example repo.</p>


Expand Down
Loading

0 comments on commit d7452e0

Please sign in to comment.