Merge branch 'master' into task/notify_cmd

telefonicaid · Jun 20, 2024 · 5b11591 · 5b11591
2 parents ae08bd6 + 80e19ab
commit 5b11591
Show file tree

Hide file tree

Showing 27 changed files with 3,007 additions and 1,149 deletions.
diff --git a/CHANGES_NEXT_RELEASE b/CHANGES_NEXT_RELEASE
@@ -1,5 +1 @@
-- Add: extend handleNotificationNgsi2 to allow receive commands from CB notifications (#1455)
-- Fix: reduce information showed handling errors to just config flags (#1594)
-- Upgrade pymongo dep from 4.3.3 to 4.6.3
-- Upgrade express dep from 4.18.1 to 4.19.2
-- Add: allow devices with the same device_id in the same service and subservice but different apikey (#1589)
+- Add: extend handleNotificationNgsi2 to allow receive commands from CB notifications (#1455)
diff --git a/config.js b/config.js
@@ -76,7 +76,8 @@ var config = {
     subservice: '/gardens',
     providerUrl: 'http://192.168.56.1:4041',
     deviceRegistrationDuration: 'P1M',
-    defaultType: 'Thing'
+    defaultType: 'Thing',
+    expressLimit: '1Mb'
 };
 
 module.exports = config;
diff --git a/doc/admin.md b/doc/admin.md
@@ -260,13 +260,13 @@ the `mongob` section (as described bellow). E.g.:
 
 It configures the MongoDB driver for those repositories with 'mongodb' type. If the `host` parameter is a list of
 comma-separated IPs, they will be considered to be part of a Replica Set. In that case, the optional property
-`replicaSet` should contain the Replica Set name. If the database requires authentication, username (`user`),
-password (`password`) and authSource (`authSource`) can be set. If the database requires TLS/SSL connection but any
-validation of the certificate chain is not mandatory, all you need is to set the ssl (`ssl`) option as `true` to connect
-the database. If you need to add more complex option(s) such as `retryWrites=true` or `w=majority` when connection
-database, extraArgs (`extraArgs`) can be used to perform it. For The MongoBD driver will retry the connection at startup
-time `retries` times, waiting `retryTime` seconds between attempts, if those attributes are present (default values are
-5 and 5 respectively). E.g.:
+`replicaSet` should contain the Replica Set name. If the database requires authentication, username (`user`), password
+(`password`) and authSource (`authSource`) can be set. If the database requires TLS/SSL connection but any validation of
+the certificate chain is not mandatory, all you need is to set the ssl (`ssl`) option as `true` to connect the database.
+If you need to add more complex option(s) such as `retryWrites=true` or `w=majority` when connection database, extraArgs
+(`extraArgs`) can be used to perform it. For The MongoBD driver will retry the connection at startup time `retries`
+times, waiting `retryTime` seconds between attempts, if those attributes are present (default values are 5 and 5
+respectively). E.g.:
 
 ```javascript
 {
@@ -421,6 +421,14 @@ characters (such as semi-colons) which are
 specification. When provisioning devices, it is necessary that the developer provides valid `objectId`-`name` mappings
 whenever relaxed mode is used, to prevent the consumption of forbidden characters.
 
+#### `expressLimit`
+
+IotAgents, as all Express applications that use the body-parser middleware, have a default limit to the request body
+size that the application will handle. This default limit for ioiotagnets are 1Mb. So, if your IotAgent receives a
+request with a body that exceeds this limit, the application will throw a “Error: Request entity too large”.
+
+The 1Mb default can be changed setting the `expressLimit` configuration parameter (or equivalente `IOTA_EXPRESS_LIMIT` environment variable).
+
 ### Configuration using environment variables
 
 Some of the configuration parameters can be overriden with environment variables, to ease the use of those parameters
@@ -482,6 +490,7 @@ overrides.
 | IOTA_EXPLICIT_ATTRS                  | `explicitAttrs`                 |
 | IOTA_DEFAULT_ENTITY_NAME_CONJUNCTION | `defaultEntityNameConjunction`  |
 | IOTA_RELAX_TEMPLATE_VALIDATION       | `relaxTemplateValidation`       |
+| IOTA_EXPRESS_LIMIT                   | `expressLimit`                  |
 
 Note:
 

diff --git a/doc/api.md b/doc/api.md
@@ -11,7 +11,7 @@
         -   [Uniqueness of groups and devices](#uniqueness-of-groups-and-devices)
     -   [Special measures and attributes names](#special-measures-and-attributes-names)
     -   [Entity attributes](#entity-attributes)
-    -   [Multientity support)](#multientity-support)
+    -   [Multientity support](#multientity-support)
     -   [Metadata support](#metadata-support)
         -   [NGSI LD data and metadata considerations](#ngsi-ld-data-and-metadata-considerations)
     -   [Advice on Attribute definitions](#advice-on-attribute-definitions)
@@ -32,6 +32,7 @@
         -   [Measurement transformation order](#measurement-transformation-order)
         -   [Multientity measurement transformation support (`object_id`)](#multientity-measurement-transformation-support-object_id)
     -   [Timestamp Processing](#timestamp-processing)
+    -   [Multimeasure support](#multimeasure-support)
     -   [Overriding global Context Broker host](#overriding-global-context-broker-host)
     -   [Multitenancy, FIWARE Service and FIWARE ServicePath](#multitenancy-fiware-service-and-fiware-servicepath)
     -   [Secured access to the Context Broker](#secured-access-to-the-context-broker)
@@ -157,8 +158,8 @@ parameters defined at device level in database, the parameters are inherit from
 
 Group service uniqueness is defined by the combination of: service, subservice and apikey
 
-Device uniqueness is defined by the combination of: service, subservice, device_id and apikey. Note that several
-devices with the same device_id are allowed in the same service and subservice as long as their apikeys are different.
+Device uniqueness is defined by the combination of: service, subservice, device_id and apikey. Note that several devices
+with the same device_id are allowed in the same service and subservice as long as their apikeys are different.
 
 ## Special measures and attributes names
 
@@ -286,32 +287,76 @@ e.g.:
 
 ```json
 {
-     "entity_type": "Lamp",
-     "resource":    "/iot/d",
-     "protocol":    "PDI-IoTA-UltraLight",
-..etc
-     "commands": [
-        {"name": "on","type": "command"},
-        {"name": "off","type": "command"}
-     ],
-     "attributes": [
-        {"object_id": "s", "name": "state", "type":"Text"},
-        {"object_id": "l", "name": "luminosity", "type":"Integer",
-          "metadata":{
-              "unitCode":{"type": "Text", "value" :"CAL"}
-          }
+    "entity_type": "Lamp",
+    "resource": "/iot/d",
+    "protocol": "PDI-IoTA-UltraLight",
+    "commands": [
+        { "name": "on", "type": "command" },
+        { "name": "off", "type": "command" }
+    ],
+    "attributes": [
+        { "object_id": "s", "name": "state", "type": "Text" },
+        {
+            "object_id": "l",
+            "name": "luminosity",
+            "type": "Integer",
+            "metadata": {
+                "unitCode": { "type": "Text", "value": "CAL" }
+            }
         }
-     ],
-     "static_attributes": [
-          {"name": "category", "type":"Text", "value": ["actuator","sensor"]},
-          {"name": "controlledProperty", "type": "Text", "value": ["light"],
-            "metadata":{
-              "includes":{"type": "Text", "value" :["state", "luminosity"]},
-              "alias":{"type": "Text", "value" :"lamp"}
+    ],
+    "static_attributes": [
+        { "name": "category", "type": "Text", "value": ["actuator", "sensor"] },
+        {
+            "name": "controlledProperty",
+            "type": "Text",
+            "value": ["light"],
+            "metadata": {
+                "includes": { "type": "Text", "value": ["state", "luminosity"] },
+                "alias": { "type": "Text", "value": "lamp" }
             }
-          },
-     ]
-   }
+        }
+    ]
+}
+```
+
+Metadata could also has `expression` like attributes in order to expand it:
+
+e.g.:
+
+```json
+{
+    "entity_type": "Lamp",
+    "resource": "/iot/d",
+    "protocol": "PDI-IoTA-UltraLight",
+    "commands": [
+        { "name": "on", "type": "command" },
+        { "name": "off", "type": "command" }
+    ],
+    "attributes": [
+        { "object_id": "s", "name": "state", "type": "Text" },
+        {
+            "object_id": "l",
+            "name": "luminosity",
+            "type": "Integer",
+            "metadata": {
+                "unitCode": { "type": "Text", "value": "CAL" }
+            }
+        }
+    ],
+    "static_attributes": [
+        { "name": "category", "type": "Text", "value": ["actuator", "sensor"] },
+        {
+            "name": "controlledProperty",
+            "type": "Text",
+            "value": ["light"],
+            "metadata": {
+                "includes": { "type": "Text", "value": ["state", "luminosity"], "expression": "level / 100" },
+                "alias": { "type": "Text", "value": "lamp" }
+            }
+        }
+    ]
+}
 ```
 
 ### NGSI-LD data and metadata considerations
@@ -421,6 +466,8 @@ mappings of the provision. If `explicitAttrs` is provided both at device and con
 precedence. Additionally `explicitAttrs` can be used to define which measures (identified by their attribute names, not
 by their object_id) defined in JSON/JEXL array will be propagated to NGSI interface.
 
+Note that when `explicitAttrs` is an array or a JEXL expression resulting in to Array, if this array is empty then `TimeInstant` is not propaged to CB.
+
 The different possibilities are summarized below:
 
 Case 1 (default):
@@ -525,8 +572,8 @@ expression. In all cases the following data is available to all expressions:
 -   `subservice`: device subservice
 -   `staticAttributes`: static attributes defined in the device or config group
 
-Additionally, for attribute expressions (`expression`, `entity_name`) and `entityNameExp` measures are avaiable in the
-**context** used to evaluate them.
+Additionally, for attribute expressions (`expression`, `entity_name`), `entityNameExp` and metadata expressions
+(`expression`) measures are available in the **context** used to evaluate them.
 
 ### Examples of JEXL expressions
 
@@ -972,32 +1019,162 @@ Will now generate the following NGSI v2 payload:
 
 ## Timestamp Processing
 
-The IOTA processes the entity attributes looking for a `TimeInstant` attribute. If one is found, for NGSI v2, then it
-adds a `TimeInstant` attribute as metadata for every other attribute in the same request. With NGSI-LD, the Standard
-`observedAt` property-of-a-property is used instead.
+Timestamp processing done by IOTA is as follows:
 
-If a `TimeInstant` arrives as measure but not follows [ISO_8601](https://en.wikipedia.org/wiki/ISO_8601) then measure is
-refused.
+-   An attribute `TimeInstant` is added to updated entities
+-   In the case of NGSI-v2, a `TimeInstant` metadata is added in each updated attribute. With NGSI-LD, the Standard
+    `observedAt` property-of-a-property is used instead.
 
 Depending on the `timestamp` configuration and if the measure contains a value named `TimeInstant` with a correct value,
 the IoTA behaviour is described in the following table:
 
-| `timestamp` value | measure contains `TimeInstant` | Behaviour                                              |
-| ----------------- | ------------------------------ | ------------------------------------------------------ |
-| true              | Yes                            | TimeInstant and metadata updated with measure value    |
-| true              | No                             | TimeInstant and metadata updated with server timestamp |
-| false             | Yes                            | TimeInstant and metadata updated with measure value    |
-| false             | No                             | TimeInstant and metadata updated with server timestamp |
-| Not defined       | Yes                            | TimeInstant and metadata updated with measure value    |
-| Not defined       | No                             | TimeInstant and metadata updated with server timestamp |
+| `timestamp` conf value | measure contains `TimeInstant` | Behaviour                                              |
+| ---------------------- | ------------------------------ | ------------------------------------------------------ |
+| true                   | Yes                            | TimeInstant and metadata updated with measure value    |
+| true                   | No                             | TimeInstant and metadata updated with server timestamp |
+| false                  | Yes                            | TimeInstant and metadata updated with measure value    |
+| false                  | No                             | TimeInstant and metadata updated with server timestamp |
+| Not defined            | Yes                            | TimeInstant and metadata updated with measure value    |
+| Not defined            | No                             | TimeInstant and metadata updated with server timestamp |
 
-The `timestamp` value used is:
+The `timestamp` conf value used is:
 
 -   The one defined at device level
 -   The one defined at group level (if not defined at device level)
 -   The one defined at [IoTA configuration level](admin.md#timestamp) / `IOTA_TIMESTAMP` env var (if not defined at
     group level or device level)
 
+Some additional considerations to take into account:
+
+-   If there is an attribute which maps a measure to `TimeInstant` attribute (after
+    [expression evaluation](#expression-language-support) if any is defined), then that value will be used as
+    `TimeInstant, overwriting the above rules specified in "Behaviour" column. Note that an expression in the could be
+    used in that mapping.
+-   If the resulting `TimeInstant` not follows [ISO_8601](https://en.wikipedia.org/wiki/ISO_8601) (either from a direct
+    measure of after a mapping, as described in the previous bullet) then it is refused (so a failover to server
+    timestamp will take place).
+
+## Multimeasure support
+
+A device could receive several measures at the same time.
+
+For example:
+
+```json
+[
+    {
+        "vol": 0
+    },
+    {
+        "vol": 1
+    },
+    {
+        "vol": 2
+    }
+]
+```
+
+In this case a batch update (`POST /v2/op/update`) to CB will be generated with the following NGSI v2 payload:
+
+```json
+{
+    "actionType": "append",
+    "entities": [
+        {
+            "id": "ws",
+            "type": "WeatherStation",
+            "vol": {
+                "type": "Number",
+                "value": 0
+            }
+        },
+        {
+            "id": "ws",
+            "type": "WeatherStation",
+            "vol": {
+                "type": "Number",
+                "value": 1
+            }
+        },
+        {
+            "id": "ws",
+            "type": "WeatherStation",
+            "vol": {
+                "type": "Number",
+                "value": 1
+            }
+        }
+    ]
+}
+```
+
+Moreover if a multimeasure contains TimeInstant attribute, then CB update is sorted by attribute TimeInstant:
+
+For example:
+
+```json
+[
+    {
+        "vol": 0,
+        "TimeInstant": "2024-04-10T10:15:00Z"
+    },
+    {
+        "vol": 1,
+        "TimeInstant": "2024-04-10T10:10:00Z"
+    },
+    {
+        "vol": 2,
+        "TimeInstant": "2024-04-10T10:05:00Z"
+    }
+]
+```
+
+In this case a batch update (`POST /v2/op/update`) to CB will be generated with the following NGSI v2 payload:
+
+```json
+{
+    "actionType": "append",
+    "entities": [
+        {
+            "id": "ws",
+            "type": "WeatherStation",
+            "vol": {
+                "type": "Number",
+                "value": 2
+            },
+            "TimeInstant": {
+                "type": "DateTime",
+                "value": "2024-04-10T10:05:00Z"
+            }
+        },
+        {
+            "id": "ws",
+            "type": "WeatherStation",
+            "vol": {
+                "type": "Number",
+                "value": 1
+            },
+            "TimeInstant": {
+                "type": "DateTime",
+                "value": "2024-04-10T10:10:00Z"
+            }
+        },
+        {
+            "id": "ws",
+            "type": "WeatherStation",
+            "vol": {
+                "type": "Number",
+                "value": 0
+            },
+            "TimeInstant": {
+                "type": "DateTime",
+                "value": "2024-04-10T10:15:00Z"
+            }
+        }
+    ]
+}
+```
+
 ## Overriding global Context Broker host
 
 **cbHost**: Context Broker host URL. This option can be used to override the global CB configuration for specific types

diff --git a/doc/requirements.txt b/doc/requirements.txt
@@ -1,4 +1,4 @@
 mkdocs==1.2.4
 Pygments==2.15.0
 Markdown==3.3.4
-jinja2==3.1.3
+jinja2==3.1.4