Bug in NGSIElasticsearchSink?

[santcorc]

Hi to everyone, I do not know if this could be consider a bug or a different way of implementation adapted to other scenarios, due to according to implementation it is a searched feature. I am talking about the Index convention number 6

append the created <date> such as yyyy.mm.dd.

In the implementation I made I had to modify the code to eliminate this because in long-term deployments Elasticsearch would crash due to the maximum number of indexes is reached. According to the default options of elasticsearch, each elasticsearch node have 1000 shards and each index need 2, so in a single-node-cluster only 500 indexes could be created. This, following the expected behaviour would be filled in a short amount of time, for example having 20 entities being updated everyday would take less than one month to not let create more indexes. Even if increasing the maximum number of shards which is not recommended by elasticsearch or adding more nodes, creating a multi-node-cluster would only be a temporal fix and is not scalable in long term scenarios.

If neccessary, I could send a pull request with the modified NGSIElasticsearchSink that is being used in my actual deployment, eliminating the addition of the date at the end of the created indexes.

[AlvaroVega]

Thanks @santcorc Any pull request that you can do in order to fix a bug is welcome!

[santcorc]

Thanks for the fast reply @AlvaroVega But the thing is that according to the documentation https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/flume_extensions_catalogue/ngsi_elasticsearch_sink.html it is a searched feature. If this is changed that documentation should also be modified.

[fgalan]

Your PR could include also modifications in the documentation. Thanks!