Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improvements in Orion DB consistency script #4535

Merged
merged 7 commits into from
Apr 10, 2024
Merged

Improvements in Orion DB consistency script #4535

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cca7d63
FIX rename entities_consistency.py script to more general oriondb_con…
fgalan Apr 9, 2024
7039520
ADD Rule S90
fgalan Apr 9, 2024
a88563f
FIX autofix framework added
fgalan Apr 9, 2024
21ec49f
FIX autofix updates in db
fgalan Apr 10, 2024
b7ad3ac
FIX minor fix
fgalan Apr 10, 2024
4dbc1bf
FIX minor fix
fgalan Apr 10, 2024
95a95a1
FIX minor fix
fgalan Apr 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 54 additions & 29 deletions scripts/entities_consistency/README.md → scripts/oriondb_consistency/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
The entities consistency script analyze the contents of the entities database in orion DBs and
The database consistency script analyze the contents of the Orion database (entities, subscriptions, etc.) and
check several consistency rules, reporting violations found.

Ref: [entity document database model]([../../doc/manuals/admin/database_model.md#entities-collection])
Ref: [Orion database model](../../doc/manuals/admin/database_model.md)

## Requirements

Expand All @@ -10,23 +10,26 @@ Usage of virtual env is recommended.

## Usage

Run `entities_consistency.py -h` for arguments details.
Run `oriondb_consistency.py -h` for arguments details.

## Rules

* Rules 1x: DB inconsistencies (use to be severe problems)
* Rules 2x: Syntax restrictions
* Rules 9x: Usage of legacy features
* Rules for entities
* Rules E1x: DB inconsistencies (use to be severe problems)
* Rules E2x: Syntax restrictions
* Rules E9x: Usage of legacy features
* Rules for subscriptions
* Rules S9x: Usage of legacy features

### Rule 10: `_id` field consistency
### Rule E10: `_id` field consistency

Each entity in DB has an `_id` field with three subfields:

* `id`
* `type`
* `servicePath`

### Rule 11: mandatory fields in entity
### Rule E11: mandatory fields in entity

The following fields are mandatory:

Expand All @@ -36,7 +39,7 @@ The following fields are mandatory:

It is not an exhaustive check of every field in the database model, but some entities created/updated with old Orion versions may be missing them.

### Rule 12: mandatory fields in attribute
### Rule E12: mandatory fields in attribute

The following subfields are mandatory for every attribute:

Expand All @@ -46,23 +49,23 @@ The following subfields are mandatory for every attribute:

It is not an exhaustive check of every field in the database model, but some entities created/updated with old Orion versions may be missing them.

### Rule 13: `attrNames` field consistency
### Rule E13: `attrNames` field consistency

For each item in `attrNames` array there is a corresponding key in `attrs` object and the other way around.

### Rule 14: `mdNames` field consistency
### Rule E14: `mdNames` field consistency

For every attribute, for each item in `mdNames` array there is a corresponding key in `md` object and the other way around.

### Rule 15: not swapped subkeys in `_id`
### Rule E15: not swapped subkeys in `_id`

In MongoDB JSON objects are stored taking order into account, so DB allows to have a document with
`_id` equal to `{"id": "E", "type": "T", "servicePath": "/"}` and at the same time have another document with `_id`
equal to `{"type": "T", "id": "E", "servicePath": "/"}` without violating `_id` uniqueness constraint.

This rule checks that this is not happening in the entities collection.

### Rule 16: `location` field consistency
### Rule E16: `location` field consistency

Check that location in consistent. In particular:

Expand All @@ -73,41 +76,41 @@ Check that location in consistent. In particular:

This rule is for location inconsistencies. For usage of deprecated geo types, there are additional rules in the 9x group.

### Rule 17: `lastCorrelator` existence
### Rule E17: `lastCorrelator` existence

Check if `lastCorrelator` is included.

This field was introduced in [Orion 1.8.0](https://github.com/telefonicaid/fiware-orion/releases/tag/1.8.0) (released in September 11th, 2017).

### Rule 20: entity id syntax
### Rule E20: entity id syntax

Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.

### Rule 21: entity type syntax
### Rule E21: entity type syntax

Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.

### Rule 22: entity servicePath syntax
### Rule E22: entity servicePath syntax

Check [servicePath documentation](https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/orion-api.md#entity-service-path)

### Rule 23: attribute name syntax
### Rule E23: attribute name syntax

Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.

### Rule 24: attribute type syntax
### Rule E24: attribute type syntax

Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.

### Rule 25: metadata name syntax
### Rule E25: metadata name syntax

Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.

### Rule 26: metadata type syntax
### Rule E26: metadata type syntax

Check [identifiers syntax restrictions](../../doc/manuals/orion-api.md#identifiers-syntax-restrictions) for this case.

### Rule 90: detect usage of `geo:x` attribute type where `x` different from `json`
### Rule E90: detect usage of `geo:x` attribute type where `x` different from `json`

Check usage of deprecated geo-location types, i.e:

Expand All @@ -123,13 +126,13 @@ Suggested action is to:

Note this rule doesn't check location consistency for this case (e.g. more than one geo-location attribute in the same entity). That's done by another rule in the 1x group.

### Rule 91: detect usage of more than one legacy `location` metadata
### Rule E91: detect usage of more than one legacy `location` metadata

Check usage of `location` in more than one attribute of the same entity.

Note this rule doesn't check location consistency for this case (that's done by another rule in the 1x group).

### Rule 92: detect legacy `location` metadata should be `WGS84` or `WSG84`
### Rule E92: detect legacy `location` metadata should be `WGS84` or `WSG84`

The value of the `location` metadata should be `WGS84` or `WSG84`.

Expand All @@ -138,7 +141,7 @@ Additional consideration:
* Entities attributes may have `location` metadata with values different from `WGS84` or `WSG84` if created using NGSIv2. That would be a false positive in this rule validation
* This rule doesn't check location consistency for this case (that's done by another rule in the 1x group).

### Rule 93: detect usage of redundant legacy `location`
### Rule E93: detect usage of redundant legacy `location`

Checks usage of redundant `location` metadata, i.e. when at the same time a `geo:` type is used in the
same attribute.
Expand All @@ -150,7 +153,7 @@ Additional, considerations:
* This rule assumes only one `location` is in the entity (i.e. Rule 91 is not violated). If that doesn't occur, only the first occurrence is taken into account.
* This rule doesn't check location consistency for this case (that's done by another rule in the 1x group).

### Rule 94: detect usage of not redundant legacy `location`
### Rule E94: detect usage of not redundant legacy `location`

Checks usage of not redundant `location` metadata, i.e. when at the same time the type of the attribute is nog `geo:`.
same attribute.
Expand All @@ -166,11 +169,33 @@ Additional, considerations:
* This rule assumes only one `location` is in the entity (i.e. Rule 91 is not violated). If that doesn't occur, only the first occurrence is taken into account.
* This rule doesn't check location consistency for this case (that's done by another rule in the 1x group).

### Rule S90: detect usage of legacy notification format

Check usage of legacy notification format in subscriptions (i.e. [`notification.attrsFormat`](../../doc/manuals/orion-api.md#subscriptionnotification) set to `legacy`).

Suggestion action is to:

* Change `notification.attrsFormat` to one of the supported formats (e.g. `normalized`).

Autofix action:

* `notification.attrsFormat` is set to `normalized`.

## Autofix mode

If `--autofix` is spefified in the command line arguments, the script implements automatic fixing of documents
violating the rules.

**WARNING: the usage of `--autofix` will do DB modification**. Use it with care. It is strongly recommended to
do a backup of your data before using it.

It only works for some rules. Check specific rules documntation for details (look for "Autofix action").

## Testing

You can test the `entities_consistency.py` script this qy:
You can test the `oriondb_consistency.py` script this qy:

1. Populate `orion-validation` DB with testing document. To do so, copy-paste the content of the `validation_data.js` in `mongosh`
2. Run `test_entities_consistency.py` test and check the log output
2. Run `test_oriondb_consistency.py` test and check the log output

You can also run `test_entities_consistenct.py` under coverage to check every rule is covering all the possible validation cases.
You can also run `test_orion_consistenct.py` under coverage to check every rule is covering all the possible validation cases.
Loading
Loading