Merge branch 'master' into task/notify_cmd

CHANGES_NEXT_RELEASE

@@ -1 +1,7 @@
-- Add: extend handleNotificationNgsi2 to allow receive commands from CB notifications (#1455)
+- Add: extend handleNotificationNgsi2 to allow receive commands from CB notifications (#1455)
+- Add: openmetrics-compatible `/metrics` endpoint in nortbound API (#1627)
+- Remove: push-based stats (including stats section in config file)
+- Fix: service header to use uppercase in case of update and delete (#1528)
+- Fix: Allow to send to CB batch update for multimeasures for NGSI-LD (#1623)
+- Add: new JEXL transformations for including into an array keys that have a certain value: valuePicker and valuePickerMulti
+- Add: attribute metadata and static attributes metadata added to jexl context (#1630)

doc/README.md

@@ -4,7 +4,7 @@
 
 -   [Getting started](getting-started.md)
 -   [IoT Agent API](api.md)
--   [Installation and administration manual](admin.md)
+-   [Administration manual](admin.md)
 -   [Deprecated features](deprecated.md)
 -   [Roadmap](roadmap.md)
 

doc/admin.md

@@ -5,7 +5,6 @@
         -   [loglevel](#loglevel)
         -   [contextBroker](#contextbroker)
         -   [server](#server)
-        -   [stats](#stats)
         -   [authentication](#authentication)
         -   [deviceRegistry](#deviceregistry)
         -   [mongodb](#mongodb)
@@ -126,7 +125,7 @@ allowing the computer to interpret the rest of the data with more clarity and de
 ```
 
 Under mixed mode, **NGSI v2** payloads are used for context broker communications by default, but this payload may also
-be switched to **NGSI LD** at service group or device provisioning time using the `ngsiVersion` field in the
+be switched to **NGSI LD** at group or device provisioning time using the `ngsiVersion` field in the
 provisioning API. The `ngsiVersion` field switch may be added at either group or device level, with the device level
 overriding the group setting.
 
@@ -159,17 +158,6 @@ support nulls or multi-attribute requests if they are encountered.
 }
 ```
 
-#### `stats`
-
-It configures the periodic collection of statistics. Use `interval` in milliseconds to set the time between stats
-writings.
-
-```javascript
-stats: {
-    interval: 100;
-}
-```
-
 #### `authentication`
 
 Stores the authentication data, for use in retrieving tokens for devices with a trust token (required in scenarios with
@@ -318,7 +306,7 @@ added `agentPath`:
 
 #### `types`
 
-See **Type Configuration** in the [Configuration API](#configurationapi) section below.
+This parameter includes additional groups configuration as described into the [Config group API](api.md#config-group-api) section.
 
 #### `service`
 
@@ -356,7 +344,7 @@ if this flag is activated:
 
 #### `defaultResource`
 
-default string to use as resource for the registration of new Configurations (if no resource is provided).
+default string to use as resource for the registration of new config groups (if no resource is provided).
 
 #### `defaultKey`
 

doc/deprecated.md

@@ -25,6 +25,7 @@ A list of deprecated features and the version in which they were deprecated foll
 -   Support to legacy expressions (finally removed in 3.2.0)
 -   Bidirectinal pluging (finally removed in 3.4.0)
 -   appendMode configuration (`IOTA_APPEND_MODE` env var) (finally removed in 3.4.0)
+-   `config.stats` section, and push-mode statistics.
 
 The use of Node.js v14 is highly recommended.
 
@@ -57,3 +58,4 @@ The following table provides information about the last iotagent-node-lib versio
 | Support to Legacy Expressions                         | 3.1.0                                                 | April 25th, 2023              |
 | bidirectional plugin                                  | 3.3.0                                                 | August 24th, 2023             |
 | appendMode configuration (`IOTA_APPEND_MODE` env var) | 3.3.0                                                 | August 24th, 2023             |
+| push-mode stats                                       | 4.5.0                                                 | June 11th, 2024               |

doc/devel/architecture.md

@@ -4,31 +4,6 @@ The following section defines the architecture and message flow which is common
 
 ### Device to NGSI Mapping
 
-Each Device will be mapped as an Entity associated to a Context Provider: the Device ID will be mapped by default to the
-entity ID and the type of the entity will be selected by the IoT Agent in a protocol-dependent way (e.g: with different
-URLs for different types). Both the name and type will be configurable by the user, either by type configuration or with
-the device preprovisioning.
-
-Each of the measures obtained from the device should be mapped to a different attribute. The name and type of the
-attribute will be configured by the user (globally for all the types in the IoT Agent configuration or in a per device
-basis preprovisioning the devices). Device measures can have three different behaviors:
-
--   **Active attributes**: are measures that are pushed from the device to the IoT agent. This measure changes will be
-    sent to the Context Broker as updateContext requests over the device entity. NGSI queries to the context broker will
-    be resolved in the Broker database.
-
--   **Lazy attributes**: some sensors will be passive, and will wait for the IoT Agent to request for data. For those
-    measures, the IoT Agent will register itself in the Context Broker as a Context Provider (for all the lazy measures
-    of that device), so if any component asks the Context Broker for the value of that sensor, its request will be
-    redirected to the IoT Agent (that behaves as a NGSI10 Context Provider). This operation will be synchronous from the
-    customer perspective: the Context Broker won't return a response until the device has returned its response to the
-    IoT Agent.
-
--   **Commands**: in this case, the interaction will begin by setting an attribute in the device's entity, for which the
-    IoT Agent will be regitered as CP. The IoT Agent will return an immediate response to the Context Broker, and will
-    be held responsible of contacting the device to perform the command itself, updating special `status` and `info`
-    attributes in the entity as soon as it has any information of the command progress.
-
 The following sequence diagram shows the different NGSI interactions an IoT Agent makes with the Context Broker,
 explained in the following subsections (using the example of a OMA Lightweight M2M device).
 
@@ -38,7 +13,7 @@ Be aware that the IoT Agents are only required to support NGSI10 operations `upd
 standard formats (currently in JSON format; XML deprecated) but will not answer to NGSI9 operations (or NGSI convenience
 operations of any kind).
 
-#### Configurations and Device provisioning information
+#### Config groups and Device provisioning information
 
 In order for a device to connect to the IoT Agent, the device should be provisioned (although there may be occasions
 where this registration is not needed). The provision process is meant to provide the IoT Agent with the following
@@ -58,27 +33,15 @@ information:
 In order to provide this information, the IoT Agent Northbound API provides two resources: Device and Configuration
 provisioning.
 
-**Configurations** may be used when a set of similar devices will be connected to the IoT Agent, to avoid provisioning
-the same set of information for every device. Custom APIKeys can be only provided with the use of Configurations for
+**Config groups** may be used when a set of similar devices will be connected to the IoT Agent, to avoid provisioning
+the same set of information for every device. Custom APIKeys can be only provided with the use of config group for
 device groups. When a device is provisioned, it is assigned to a configuration _if there is one that matches its type,
 its service and its subservice_. In that case, all the default information in the Configuration is merged with the
 device information to create the definitive Device object that will be stored in the system.
 
-Particular IoT Agents _may_ support autoregistration of devices into configurations, if enough information is given from
+Particular IoT Agents _may_ support autoregistration of devices into config groups, if enough information is given from
 the Southbound.
 
-#### Configurations and subservices
-
-Configurations are meant to be a mean of simplifying the device provisioning for groups of very similar devices.
-Considering that different groups of devices may be created in the same subservice that may require different
-configurations, multiple configurations are allowed for each subservice. Considering the key in the association between
-Device and Configuration was the triplet (service, subservice, type), all of these elements are considered mandatory.
-
-This statement doesn't hold true for older IoT Agents, though. In older versions of the IoT Agents, each device
-configuration was assigned to a particular subservice and just one configuration was allowed per subservice, so the
-relation between a Device and a Configuration didn't need the type to discriminate between Configurations. That's why
-for those agents, type was not a mandatory parameter.
-
 #### Registration
 
 Whenever a device is registered, the IoT Agent reads the device's entity information from the request or, if that
@@ -103,7 +66,7 @@ response to the caller, transparently.
 
 **IMPORTANT NOTE:** at the present moment, commands (both push and poll) are supported only in the case of explicitly
 provisioned agents. For autoprovisioned agents commands are not currently supported, although
-[an issue](https://github.com/telefonicaid/iot-agent-node-lib/issues/572) has been created about this functionality.
+[an issue](https://github.com/telefonicaid/iotagent-node-lib/issues/572) has been created about this functionality.
 
 Commands are modelled as updates over a lazy attribute. As in the case of the lazy attributes, updates over a command
 will be forwarded by the Context Broker to the IoT Agent, that will in turn interact with the device to perform the
@@ -153,52 +116,6 @@ taking too much to pick them up. See the configuration section for details.
 The library does not deal with protocol transformation or South Bound communications for neither of the command types
 (that's the task for those specific IoT Agents using the library).
 
-##### NGSI-LD `datasetId` support
-
-Limited support for parsing the NGSI-LD `datasetId` attribute is included within the library. A series of sequential
-commands for a single attribute can be sent as an NGSI-LD PATCH payload as follows:
-
-```json
-{
-    "lampColor": [
-        {
-            "type": "Property",
-            "value": { "color": "green", "duration": "55 secs" },
-            "datasetId": "urn:ngsi-ld:Sequence:do-this"
-        },
-        {
-            "type": "Property",
-            "value": { "color": "red", "duration": "10 secs" },
-            "datasetId": "urn:ngsi-ld:Sequence:then-do-this"
-        }
-    ]
-}
-```
-
-This results in the following sequential array of attribute updates to be sent to the `UpdateHandler` of the IoT Agent
-itself:
-
-```json
-[
-    {
-        "name": "lampColor",
-        "type": "Property",
-        "datasetId": "urn:ngsi-ld:Sequence:do-this",
-        "metadata": {},
-        "value": { "color": "green", "duration": "55 secs" }
-    },
-    {
-        "name": "lampColor",
-        "type": "Property",
-        "datasetId": "urn:ngsi-ld:Sequence:then-do-this",
-        "metadata": {},
-        "value": { "color": "red", "duration": "10 secs" }
-    }
-]
-```
-
-The equivalent for NGSI-v2 is not required since `datasetId` syntax is not supported by NGSI-v2.
-
 #### Active attributes
 
 Whenever a device proactively sends a message to the IoT Agent, it should transform its data to the appropriate NGSI
@@ -243,50 +160,3 @@ The following figure offers a graphical example of how a COAP IoT Agent work, or
 device to a command update to the device.
 
 ![General ](../img/iotAgentLib.png 'Architecture Overview')
-
-### The `TimeInstant` element
-
-As part of the device to entity mapping process the IoT Agent creates and updates automatically a special timestamp.
-This timestamp is represented as two different properties of the mapped entity::
-
--   With NGSI-v2, an attribute metadata named `TimeInstant` (per dynamic attribute mapped, which captures as an ISO8601
-    timestamp when the associated measurement (represented as attribute value) was observed. With NGSI-LD, the Standard
-    `observedAt` property-of-a-property is used instead.
-
--   For NGSI-v2 only, an additional attribute `TimeInstant` is added to the entity which captures as an ISO8601
-    timestamp when the last measurement received from the device was observed.
-
-If no information about the measurement timestamp is received by the IoT Agent, the arrival time of the measurement will
-be used to generate a `TimeInstant` for both the entity attribute and the attribute metadata.
-
-Take into account that:
-
--   the timestamp of different attributes belonging to the same measurement record may not be equal.
--   the arrival time and the measurement timestamp will not be the same in the general case.
--   if `timezone` field is defined as part of the provisioning of the device or group, timestamp fields will be
-    generated using it. For instance, if `timezone` is set to `America/Los_Angeles`, a possible timestamp could be
-    `2025-08-05T00:35:01.468-07:00`. If `timezone` field is not defined, by default Zulu Time Zone (UTC +0) will be
-    used. Following the previous example, timestamp could be `2015-08-05T07:35:01.468Z`.
-
-E.g.: in the case of a device that can take measurements every hour of both temperature and humidity and sends the data
-once every day, at midnight, the `TimeInstant` reported for each measurement will be the hour when that measurement was
-observed (e.g. 4:00 PM), while all the measurements will have an arrival time around midnight. If no timestamps were
-reported with such measurements, the `TimeInstant` attribute would take those values around midnight.
-
-This functionality can be turned on and off through the use of the `timestamp` configuration flag (described in the
-configuration), as well as 'timestamp' flag in device or group provision.
-
-### Implementation decisions
-
-Given the aforementioned requirements, there are some aspects of the implementation that were chosen, and are
-particularly under consideration:
-
--   Aside from its configuration, the IoT Agent Lib is considered to be stateless. To be precise, the library mantains a
-    state (the list of entities/devices whose information the agent can provide) but that state is considered to be
-    transient. It's up to the particular implementation of the agent to consider whether it should have a persistent
-    storage to hold the device information (so the internal list of devices is read from a DB) or to register the
-    devices each time a device sends a measure. To this extent, two flavours of the Device Registry has been provided: a
-    transient one (In-memory Registry) and a persistent one (based in MongoDB).
--   The IoT Agent does not care about the origin of the data, its type or structure. The mapping from raw data to the
-    entity model, if there is any, is a responsibility of the particular IoT Agent implementation, or of another third
-    party library.