Generator takes following parameters:
num_of_shards
, which defines number of vectors that will be generated - it's identical to number of shards located in total on cloud.num_of_samples
, which defines nomber of periods that will be considered when generating tasks.new_period
, which defines the amount of time related to one element of load vector. Default value is5.0
shape
, which defines α parameter of gamma distribution from which number of tasks per period will be drawn.size
, which defines β parameter of gamma distribution from which number of tasks per period will be drawn.
Generate requests and load vectors using following function from generator
module:
def generator(num_of_shards, num_of_samples, new_period, shape, scale)
Tasks are drawn from gamma distribution, but to be precise - as far as shape
parameter is scalar - it's Erlang distribution, whose mean and standard deviation can be calulated with following equations:
μ = α * β
σ = β * √α
Shard allocator is able to allocate shards on nodes according to one of three algorithms:
random
- evenly distribute shards on nodes with random ordersequential
- evenly distribute shards on nodes with sequential orderSALP
- Shards Allocation based on Load Prediction
def shard_allocator(shards, nodes, algorithm_name):
shards
- number of shardsnodes
- number of nodesalgorithm_name
- one of listed above algorithmrandom
/sequential
/SALP
def simulator(parallel_requests, new_period):
Simulator takes two parameters:
parallel_requests
- number of requests that could be processed on node at once,new_period
, which defines the amount of time related to one element of load vector. Default value is5.0
.
Simulator algorithm:
generator(5, 10, 5, 2, 2)
Example of requests
id | timestamp | shard | load |
---|---|---|---|
0 | 1.56 | 1 | 12.221 |
1 | 2.895 | 1 | 8.789 |
2 | 3.765 | 3 | 4.164 |
3 | 10.0 | 3 | 2.007 |
4 | 10.22 | 5 | 3.824 |
5 | 10.45 | 1 | 4.14 |
6 | 10.605 | 2 | 7.647 |
7 | 12.44 | 1 | 3.452 |
8 | 14.29 | 5 | 3.069 |
9 | 16.88 | 3 | 3.781 |
10 | 19.63 | 5 | 5.0 |
11 | 20.91 | 2 | 8.126 |
12 | 23.275 | 4 | 3.159 |
13 | 23.665 | 5 | 3.319 |
14 | 25.32 | 4 | 6.323 |
15 | 25.91 | 4 | 0.938 |
16 | 25.95 | 1 | 12.248 |
17 | 26.415 | 1 | 4.473 |
18 | 26.57 | 5 | 0.577 |
19 | 27.775 | 2 | 5.004 |
20 | 28.315 | 5 | 5.905 |
21 | 30.065 | 4 | 4.648 |
22 | 30.555 | 2 | 5.702 |
23 | 31.135 | 5 | 15.114 |
24 | 31.195 | 4 | 3.907 |
25 | 31.505 | 3 | 0.329 |
26 | 31.62 | 3 | 3.506 |
27 | 34.13 | 4 | 4.986 |
28 | 36.065 | 2 | 9.23 |
29 | 37.085 | 1 | 0.613 |
30 | 39.42 | 2 | 2.097 |
31 | 41.39 | 5 | 0.963 |
32 | 41.42 | 4 | 4.456 |
33 | 44.585 | 4 | 1.779 |
34 | 46.305 | 5 | 2.526 |
35 | 47.535 | 2 | 2.493 |
36 | 49.105 | 5 | 2.03 |
Example of load vectors
[1] | [2] | [3] | [4] | [5] | [6] | [7] | [8] | [9] | [10] | [11] |
---|---|---|---|---|---|---|---|---|---|---|
3.109 | 0.0 | 4.921 | 0.178 | 0.0 | 2.527 | 0.178 | 2.211 | 0.0 | 0.0 | 0.0 |
0.0 | 0.0 | 0.879 | 0.65 | 0.818 | 1.252 | 1.445 | 2.154 | 0.303 | 1.982 | 0.006 |
0.247 | 0.586 | 0.401 | 0.624 | 0.132 | 0.0 | 0.742 | 0.025 | 0.0 | 0.0 | 0.0 |
0.0 | 0.0 | 0.0 | 0.0 | 0.345 | 1.411 | 2.194 | 0.843 | 0.799 | 0.448 | 0.0 |
0.0 | 0.0 | 0.907 | 0.546 | 1.193 | 0.849 | 2.617 | 1.0 | 0.193 | 3.152 | 0.227 |
shard_allocator(shards, nodes, algorithm_name):
Example of shards allocated with random algorithm
shard | node |
---|---|
1 | 2 |
2 | 2 |
3 | 1 |
4 | 1 |
5 | 2 |
simulator(3, 5)
Example of output from simulation
id | timestamp | shard | load | expected_end_time | actual_end_time | delay |
---|---|---|---|---|---|---|
0 | 1.56 | 1 | 12.221 | 13.781 | 13.781 | 0.0 |
1 | 2.895 | 1 | 8.789 | 11.684 | 11.684 | 0.0 |
2 | 3.765 | 3 | 4.164 | 7.929 | 7.929 | 0.0 |
3 | 10.0 | 3 | 2.007 | 12.007 | 12.007 | 0.0 |
4 | 10.22 | 5 | 3.824 | 14.044 | 14.044 | 0.0 |
6 | 10.605 | 2 | 7.647 | 18.252 | 18.252 | 0.0 |
5 | 11.684 | 1 | 4.14 | 14.59 | 15.824 | 1.234 |
7 | 12.44 | 1 | 3.452 | 15.892 | 15.892 | 0.0 |
8 | 14.29 | 5 | 3.069 | 17.359 | 17.359 | 0.0 |
9 | 16.88 | 3 | 3.781 | 20.661 | 20.661 | 0.0 |
10 | 19.63 | 5 | 5.0 | 24.63 | 24.63 | 0.0 |
11 | 20.91 | 2 | 8.126 | 29.036 | 29.036 | 0.0 |
12 | 23.275 | 4 | 3.159 | 26.434 | 26.434 | 0.0 |
13 | 23.665 | 5 | 3.319 | 26.984 | 26.984 | 0.0 |
14 | 25.32 | 4 | 6.323 | 31.643 | 31.643 | 0.0 |
15 | 25.91 | 4 | 0.938 | 26.848 | 26.848 | 0.0 |
17 | 26.434 | 1 | 4.473 | 30.888 | 30.907 | 0.019 |
18 | 26.57 | 5 | 0.577 | 27.147 | 27.147 | 0.0 |
16 | 26.848 | 1 | 12.248 | 38.198 | 39.096 | 0.898 |
19 | 27.775 | 2 | 5.004 | 32.779 | 32.779 | 0.0 |
20 | 28.315 | 5 | 5.905 | 34.22 | 34.22 | 0.0 |
22 | 30.555 | 2 | 5.702 | 36.257 | 36.257 | 0.0 |
21 | 30.907 | 4 | 4.648 | 34.713 | 35.555 | 0.842 |
25 | 31.643 | 3 | 0.329 | 31.834 | 31.972 | 0.138 |
24 | 31.972 | 4 | 3.907 | 35.102 | 35.879 | 0.777 |
23 | 32.779 | 5 | 15.114 | 46.249 | 47.893 | 1.644 |
26 | 35.555 | 3 | 3.506 | 35.126 | 39.061 | 3.935 |
27 | 35.879 | 4 | 4.986 | 39.116 | 40.865 | 1.749 |
28 | 36.065 | 2 | 9.23 | 45.295 | 45.295 | 0.0 |
29 | 39.061 | 1 | 0.613 | 37.698 | 39.674 | 1.976 |
30 | 39.42 | 2 | 2.097 | 41.517 | 41.517 | 0.0 |
32 | 41.42 | 4 | 4.456 | 45.876 | 45.876 | 0.0 |
31 | 41.517 | 5 | 0.963 | 42.353 | 42.48 | 0.127 |
33 | 44.585 | 4 | 1.779 | 46.364 | 46.364 | 0.0 |
34 | 46.305 | 5 | 2.526 | 48.831 | 48.831 | 0.0 |
35 | 47.535 | 2 | 2.493 | 50.028 | 50.028 | 0.0 |
36 | 49.105 | 5 | 2.03 | 51.135 | 51.135 | 0.0 |