Merge branch 'master' into feature/4004_jexl_expressions

scripts/entities_consistency/README.md → scripts/oriondb_consistency/README.md

@@ -1,7 +1,7 @@
-The entities consistency script analyze the contents of the entities database in orion DBs and
+The database consistency script analyze the contents of the Orion database (entities, subscriptions, etc.) and
 check several consistency rules, reporting violations found.
 
-Ref: [entity document database model]([../../doc/manuals/admin/database_model.md#entities-collection])
+Ref: [Orion database model](../../doc/manuals/admin/database_model.md)
 
 ## Requirements
 
@@ -10,23 +10,26 @@ Usage of virtual env is recommended.
 
 ## Usage
 
-Run `entities_consistency.py -h` for arguments details.
+Run `oriondb_consistency.py -h` for arguments details.
 
 ## Rules
 
-* Rules 1x: DB inconsistencies (use to be severe problems)
-* Rules 2x: Syntax restrictions
-* Rules 9x: Usage of legacy features
+* Rules for entities
+  * Rules E1x: DB inconsistencies (use to be severe problems)
+  * Rules E2x: Syntax restrictions
+  * Rules E9x: Usage of legacy features
+* Rules for subscriptions
+  * Rules S9x: Usage of legacy features
 
-### Rule 10: `_id` field consistency
+### Rule E10: `_id` field consistency
 
 Each entity in DB has an `_id` field with three subfields:
 
 * `id`
 * `type`
 * `servicePath`
 
-### Rule 11: mandatory fields in entity
+### Rule E11: mandatory fields in entity
 
 The following fields are mandatory:
 
@@ -36,7 +39,7 @@ The following fields are mandatory:
 
 It is not an exhaustive check of every field in the database model, but some entities created/updated with old Orion versions may be missing them.
 
-### Rule 12: mandatory fields in attribute
+### Rule E12: mandatory fields in attribute
 
 The following subfields are mandatory for every attribute:
 
@@ -46,23 +49,23 @@ The following subfields are mandatory for every attribute:
 
 It is not an exhaustive check of every field in the database model, but some entities created/updated with old Orion versions may be missing them.
 
-### Rule 13: `attrNames` field consistency
+### Rule E13: `attrNames` field consistency
 
 For each item in `attrNames` array there is a corresponding key in `attrs` object and the other way around.
 
-### Rule 14: `mdNames` field consistency
+### Rule E14: `mdNames` field consistency
 
 For every attribute, for each item in `mdNames` array there is a corresponding key in `md` object and the other way around.
 
-### Rule 15: not swapped subkeys in `_id`
+### Rule E15: not swapped subkeys in `_id`
 
 In MongoDB JSON objects are stored taking order into account, so DB allows to have a document with
 `_id` equal to `{"id": "E", "type": "T", "servicePath": "/"}` and at the same time have another document with `_id`
 equal to `{"type": "T", "id": "E", "servicePath": "/"}` without violating `_id` uniqueness constraint.
 
 This rule checks that this is not happening in the entities collection.
 
-### Rule 16: `location` field consistency
+### Rule E16: `location` field consistency
 
 Check that location in consistent. In particular:
 
@@ -73,41 +76,41 @@ Check that location in consistent. In particular:
 
 This rule is for location inconsistencies. For usage of deprecated geo types, there are additional rules in the 9x group.
 
-### Rule 17: `lastCorrelator` existence
+### Rule E17: `lastCorrelator` existence
 
 Check if `lastCorrelator` is included.
 
 This field was introduced in [Orion 1.8.0](https://github.com/telefonicaid/fiware-orion/releases/tag/1.8.0) (released in September 11th, 2017).
 
-### Rule 20: entity id syntax
+### Rule E20: entity id syntax
 
 Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.
 
-### Rule 21: entity type syntax
+### Rule E21: entity type syntax
 
 Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.
 
-### Rule 22: entity servicePath syntax
+### Rule E22: entity servicePath syntax
 
 Check [servicePath documentation](https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/orion-api.md#entity-service-path)
 
-### Rule 23: attribute name syntax
+### Rule E23: attribute name syntax
 
 Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.
 
-### Rule 24: attribute type syntax
+### Rule E24: attribute type syntax
 
 Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.
 
-### Rule 25: metadata name syntax
+### Rule E25: metadata name syntax
 
 Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.
 
-### Rule 26: metadata type syntax
+### Rule E26: metadata type syntax
 
 Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.
 
-### Rule 90: detect usage of `geo:x` attribute type where `x` different from `json`
+### Rule E90: detect usage of `geo:x` attribute type where `x` different from `json`
 
 Check usage of deprecated geo-location types, i.e:
 
@@ -123,13 +126,13 @@ Suggested action is to:
 
 Note this rule doesn't check location consistency for this case (e.g. more than one geo-location attribute in the same entity). That's done by another rule in the 1x group.
 
-### Rule 91: detect usage of more than one legacy `location` metadata
+### Rule E91: detect usage of more than one legacy `location` metadata
 
 Check usage of `location` in more than one attribute of the same entity.
 
 Note this rule doesn't check location consistency for this case (that's done by another rule in the 1x group).
 
-### Rule 92: detect legacy `location` metadata should be `WGS84` or `WSG84`
+### Rule E92: detect legacy `location` metadata should be `WGS84` or `WSG84`
 
 The value of the `location` metadata should be `WGS84` or `WSG84`.
 
@@ -138,7 +141,7 @@ Additional consideration:
 * Entities attributes may have `location` metadata with values different from `WGS84` or `WSG84` if created using NGSIv2. That would be a false positive in this rule validation 
 * This rule doesn't check location consistency for this case (that's done by another rule in the 1x group).
 
-### Rule 93: detect usage of redundant legacy `location`
+### Rule E93: detect usage of redundant legacy `location`
 
 Checks usage of redundant `location` metadata, i.e. when at the same time a `geo:` type is used in the
 same attribute.
@@ -150,7 +153,7 @@ Additional, considerations:
  * This rule assumes only one `location` is in the entity (i.e. Rule 91 is not violated). If that doesn't occur, only the first occurrence is taken into account. 
  * This rule doesn't check location consistency for this case (that's done by another rule in the 1x group).
 
-### Rule 94: detect usage of not redundant legacy `location`
+### Rule E94: detect usage of not redundant legacy `location`
 
 Checks usage of not redundant `location` metadata, i.e. when at the same time the type of the attribute is nog `geo:`.
 same attribute.
@@ -166,11 +169,33 @@ Additional, considerations:
 * This rule assumes only one `location` is in the entity (i.e. Rule 91 is not violated). If that doesn't occur, only the first occurrence is taken into account.
 * This rule doesn't check location consistency for this case (that's done by another rule in the 1x group).
 
+### Rule S90: detect usage of legacy notification format
+
+Check usage of legacy notification format in subscriptions (i.e. [`notification.attrsFormat`](../../doc/manuals/orion-api.md#subscriptionnotification) set to `legacy`).
+
+Suggestion action is to:
+
+* Change `notification.attrsFormat` to one of the supported formats (e.g. `normalized`).
+
+Autofix action:
+
+* `notification.attrsFormat` is set to `normalized`.
+
+## Autofix mode
+
+If `--autofix` is spefified in the command line arguments, the script implements automatic fixing of documents
+violating the rules.
+
+**WARNING: the usage of `--autofix` will do DB modification**. Use it with care. It is strongly recommended to
+do a backup of your data before using it.
+
+It only works for some rules. Check specific rules documntation for details (look for "Autofix action").
+
 ## Testing
 
-You can test the `entities_consistency.py` script this qy:
+You can test the `oriondb_consistency.py` script this qy:
 
 1. Populate `orion-validation` DB with testing document. To do so, copy-paste the content of the `validation_data.js` in `mongosh`
-2. Run `test_entities_consistency.py` test and check the log output
+2. Run `test_oriondb_consistency.py` test and check the log output
 
-You can also run `test_entities_consistenct.py` under coverage to check every rule is covering all the possible validation cases.
+You can also run `test_orion_consistenct.py` under coverage to check every rule is covering all the possible validation cases.