Merge pull request #4564 from telefonicaid/remove/legacy-subs

REMOVE subs legacy format

CHANGES_NEXT_RELEASE

@@ -6,3 +6,4 @@
 - Fix: lighter operation to get databases list from MongoDB (#4517)
 - Hardening: compile code using C++14 standard
 - Remove: `-dbhost`, `-rplSet`, `-dbTimeout`, `-dbuser`, `-dbAuthMech`, `-dbAuthDb`, `-dbSSL` and `-dbDisableRetryWrites` CLI parameters along with associated env vars, already deprecated in Orion 3.12.0 (use `-dbURI` instead`)
+- Remove: legacy subscription format

doc/manuals/admin/database_model.md

@@ -304,8 +304,7 @@ Fields:
     fields: q, mq, georel, geometry and/or coords (optional)
 -   **count**: the number of notifications sent associated to
     the subscription.   
--   **format**: the format to use to send notification, possible values are **JSON**
-    (meaning JSON notifications in NGSIv1 legacy format), **normalized**, **keyValues**, **simplifiedNormalized**, **simplifiedKeyValues** and **values** (the last five used in NGSIv2 format).
+-   **format**: the format to use to send notification, possible values are **normalized**, **keyValues**, **simplifiedNormalized**, **simplifiedKeyValues** and **values**.
 -   **status**: either `active` (for active subscriptions), `inactive` (for inactive subscriptions) or
     `oneshot` (for [oneshot subscriptions](../orion-api.md#oneshot-subscriptions)). Note that Orion API consider additional states (e.g. `expired`)
     but they never hit the DB (they are managed by Orion).

doc/manuals/admin/logs.md

@@ -422,15 +422,6 @@ time=2024-01-11T13:57:13.595Z | lvl=WARN | corr=527c0912-b089-11ee-bb8c-080027cd
 time=2024-01-11T13:57:13.624Z | lvl=WARN | corr=52808938-b089-11ee-9835-080027cd35f1 | trans=1704981432-655-00000000010 | from=127.0.0.1 | srv=s1 | subsrv=/A | comp=Orion | op=postUpdateContext.cpp[163]:updateForward | msg=Deprecated usage of legacyForwarding mode in update forwarding operation (regId: 659ff3b9691855f16d00ec5a)
 ```
 
-* Usages of [`"attrsFormat": "legacy"`](../orion-api.md#subscriptionnotification). For instance:
-
-```
-time=2024-01-11T16:23:24.646Z | lvl=WARN | corr=be709034-b09d-11ee-b5d1-080027cd35f1 | trans=1704990203-652-00000000012 | from=127.0.0.1 | srv=s1 | subsrv=/A | comp=Orion | op=MongoCommonSubscription.cpp[598]:setFormat | msg=Deprecated usage of notification legacy format in subscription creation (subId: 65a015fcda947708d30425eb)
-time=2024-01-11T16:23:24.675Z | lvl=WARN | corr=be74dc98-b09d-11ee-b2d6-080027cd35f1 | trans=1704990203-652-00000000013 | from=127.0.0.1 | srv=s1 | subsrv=/A | comp=Orion | op=mongoGetSubscriptions.cpp[217]:setNotification | msg=Deprecated usage of notification legacy format detected in existing subscription (subId: 65a015fcda947708d30425eb)
-time=2024-01-11T16:23:24.701Z | lvl=WARN | corr=be709034-b09d-11ee-b5d1-080027cd35f1 | trans=1704990203-652-00000000012 | from=127.0.0.1 | srv=s1 | subsrv=/A | comp=Orion | op=MongoCommonSubscription.cpp[598]:setFormat | msg=Deprecated usage of notification legacy format in subscription modification (subId: 65a015fcda947708d30425eb)
-time=2024-01-11T16:23:24.716Z | lvl=WARN | corr=be7ae5ac-b09d-11ee-98c8-080027cd35f1 | trans=1704990203-652-00000000015 | from=127.0.0.1 | srv=s1 | subsrv=/A | comp=Orion | op=Notifier.cpp[680]:buildSenderParams | msg=Deprecated usage of notification legacy format in notification (subId: 65a015fcda947708d30425eb)
-```
-
 * Usages of `geo:point`, `geo:line`, `geo:box` or `geo:line`.
 
 ```

doc/manuals/admin/statistics.md

@@ -57,7 +57,6 @@ The counter block provides information about counters for the times a particular
     "deprecatedFeatures": {
       "geoFormat": 2,
       "ngsiv1Forwarding": 4,
-      "ngsiv1NotifFormat": 4,
       "ngsiv1Requests": 4
     },
     "invalidRequests": 2,

doc/manuals/deprecated.md

@@ -39,6 +39,7 @@ A list of deprecated features and the version in which they were deprecated foll
         * `POST /NGSI10/updateContext`
         * `POST /v1/queryContext`
         * `POST /NGSI10/queryContext`
+    * NGSIv1 format in subscription notifications (`notification.atttrsFormat` set to `legacy`) removed in Orion 4.0.0
 * `attributes` field in `POST /v2/op/query` is in Orion 1.15.0. It is a combination of `attrs` (to select
   which attributes to include in the response to the query) and unary attribute filter in `q` within
   `expression` (to return only entities which have these attributes). Use them instead.

doc/manuals/orion-api.md

@@ -1993,40 +1993,7 @@ If `attrsFormat` is `values` then values partial entity representation mode is u
 }
 ```
 
-If `attrsFormat` is `legacy` then subscription representation follows  NGSIv1 format. This way, users
-can benefit from the enhancements of Orion subscriptions (e.g. filtering) with NGSIv1 legacy notification receivers.
-
-Note that NGSIv1 is deprecated. Thus, we don't recommend to use `legacy` notification format any longer.
-
-```json
-{
-	"subscriptionId": "56e2ad4e8001ff5e0a5260ec",
-	"originator": "localhost",
-	"contextResponses": [{
-		"contextElement": {
-			"type": "Car",
-			"isPattern": "false",
-			"id": "Car1",
-			"attributes": [{
-				"name": "temperature",
-				"type": "centigrade",
-				"value": "26.5",
-				"metadatas": [{
-					"name": "TimeInstant",
-					"type": "recvTime",
-					"value": "2015-12-12 11:11:11.123"
-				}]
-			}]
-		},
-		"statusCode": {
-			"code": "200",
-			"reasonPhrase": "OK"
-		}
-	}]
-}
-```
-
-Notifications must include the `Ngsiv2-AttrsFormat` (expect when `attrsFormat` is `legacy`)
+Notifications must include the `Ngsiv2-AttrsFormat`
 HTTP header with the value of the format of the associated subscription, so that notification receivers
 are aware of the format without needing to process the notification payload.
 
@@ -4712,7 +4679,7 @@ A `notification` object contains the following subfields:
 |--------------------|-------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | `attrs` or `exceptAttrs` |          | array | Both cannot be used at the same time. <ul><li><code>attrs</code>: List of attributes to be included in notification messages. It also defines the order in which attributes must appear in notifications when <code>attrsFormat</code> <code>value</code> is used (see [Notification Messages](#notification-messages) section). An empty list means that all attributes are to be included in notifications. See [Filtering out attributes and metadata](#filtering-out-attributes-and-metadata) section for more detail.</li><li><code>exceptAttrs</code>: List of attributes to be excluded from the notification message, i.e. a notification message includes all entity attributes except the ones listed in this field. It must be a non-empty list.</li><li>If neither <code>attrs</code> nor <code>exceptAttrs</code> is specified, all attributes are included in notifications.</li></ul>|
 | [`http`](#subscriptionnotificationhttp), [`httpCustom`](#subscriptionnotificationhttpcustom), [`mqtt`](#subscriptionnotificationmqtt) or [`mqttCustom`](#subscriptionnotificationmqttcustom)| ✓                 | object | One of them must be present, but not more than one at the same time. It is used to convey parameters for notifications delivered through the transport protocol. |
-| `attrsFormat`          | ✓                 | string | Specifies how the entities are represented in notifications. Accepted values are `normalized` (default), `simplifiedNormalized`, `keyValues`, `simplifiedKeyValues`, `values` or `legacy`.<br> If `attrsFormat` takes any value different than those, an error is raised. See detail in [Notification Messages](#notification-messages) section. |
+| `attrsFormat`          | ✓                 | string | Specifies how the entities are represented in notifications. Accepted values are `normalized` (default), `simplifiedNormalized`, `keyValues`, `simplifiedKeyValues`, or `values`.<br> If `attrsFormat` takes any value different than those, an error is raised. See detail in [Notification Messages](#notification-messages) section. |
 | `metadata`         | ✓                 | string  | List of metadata to be included in notification messages. See [Filtering out attributes and metadata](#filtering-out-attributes-and-metadata) section for more detail.            |
 | `onlyChangedAttrs` | ✓                 | boolean | If `true` then notifications will include only attributes that changed in the triggering update request, in combination with the `attrs` or `exceptAttrs` field. (default is `false` if the field is omitted)) |
 | `covered`          | ✓                 | boolean | If `true` then notifications will include all the attributes defined in `attrs` field, even if they are not present in the entity (in this, case, with `null` value). (default value is false). For further information see [Covered subscriptions](#covered-subscriptions) section |

src/lib/common/RenderFormat.cpp

@@ -41,7 +41,6 @@ const char* renderFormatToString(RenderFormat format, bool noDefault, bool useLe
 {
   switch (format)
   {
-  case NGSI_V1_LEGACY:              return useLegacyWord ? "legacy" : "JSON";
   case NGSI_V2_NORMALIZED:          return "normalized";
   case NGSI_V2_KEYVALUES:           return "keyValues";
   case NGSI_V2_VALUES:              return "values";
@@ -71,8 +70,6 @@ const char* renderFormatToString(RenderFormat format, bool noDefault, bool useLe
 */
 RenderFormat stringToRenderFormat(const std::string& s, bool noDefault)
 {
-  if (s == "JSON")         { return NGSI_V1_LEGACY;        }  // DB content for NGSIv1 rendering due to legacy reasons
-  if (s == "legacy")       { return NGSI_V1_LEGACY;        }
   if (s == "normalized")   { return NGSI_V2_NORMALIZED;    }
   if (s == "keyValues")    { return NGSI_V2_KEYVALUES;     }
   if (s == "values")       { return NGSI_V2_VALUES;        }

src/lib/common/RenderFormat.h

@@ -44,14 +44,13 @@
 typedef enum RenderFormat
 {
   NO_FORMAT              = 0,
-  NGSI_V1_LEGACY         = 1,
-  NGSI_V2_NORMALIZED     = 2,
-  NGSI_V2_KEYVALUES      = 3,
-  NGSI_V2_VALUES         = 4,
-  NGSI_V2_UNIQUE_VALUES  = 5,
-  NGSI_V2_CUSTOM         = 6,
-  NGSI_V2_SIMPLIFIEDNORMALIZED = 7,
-  NGSI_V2_SIMPLIFIEDKEYVALUES  = 8
+  NGSI_V2_NORMALIZED     = 1,
+  NGSI_V2_KEYVALUES      = 2,
+  NGSI_V2_VALUES         = 3,
+  NGSI_V2_UNIQUE_VALUES  = 4,
+  NGSI_V2_CUSTOM         = 5,
+  NGSI_V2_SIMPLIFIEDNORMALIZED = 6,
+  NGSI_V2_SIMPLIFIEDKEYVALUES  = 7
 } RenderFormat;
 
 

src/lib/common/errorMessages.h

@@ -62,7 +62,7 @@
 #define ERROR_DESC_BAD_REQUEST_INVALID_JTYPE_ENTID    "Invalid JSON type for entity id"
 #define ERROR_DESC_BAD_REQUEST_INVALID_JTYPE_ENTTYPE  "Invalid JSON type for entity type"
 #define ERROR_DESC_BAD_REQUEST_INVALID_JTYPE_SCOPE    "invalid JSON type for scope value: must be string"
-#define ERROR_DESC_BAD_REQUEST_INVALID_ATTRSFORMAT    "invalid attrsFormat, accepted values: legacy, normalized, simplifiedNormalized, keyValues, simplifiedkeyValues, values"
+#define ERROR_DESC_BAD_REQUEST_INVALID_ATTRSFORMAT    "invalid attrsFormat, accepted values: normalized, simplifiedNormalized, keyValues, simplifiedkeyValues, values"
 #define ERROR_DESC_BAD_REQUEST_INVALID_STATUS         "status is not valid: it has to be either active, inactive or oneshot"
 #define ERROR_DESC_BAD_REQUEST_INVALID_RANGE          "ranges only valid for equal and not equal ops"
 #define ERROR_DESC_BAD_REQUEST_INVALID_LIST           "lists only valid for equal and not equal ops"

src/lib/common/statistics.cpp

@@ -152,7 +152,6 @@ int noOfSimulatedNotifications   = -1;
 // Deprecated features
 int noOfDprNgsiv1Request         = -1;
 int noOfDprLegacyForwarding      = -1;
-int noOfDprLegacyNotif           = -1;
 int noOfDprGeoformat             = -1;
 
 

src/lib/mongoBackend/mongoCreateSubscription.cpp

@@ -198,15 +198,6 @@ std::string mongoCreateSubscription
     return "";
   }
 
-  if (sub.attrsFormat == NGSI_V1_LEGACY)
-  {
-    __sync_fetch_and_add(&noOfDprLegacyNotif, 1);
-    if (logDeprecate)
-    {
-      LM_W(("Deprecated usage of notification legacy format in subscription creation (subId: %s)", subId.c_str()));
-    }
-  }
-
   reqSemGive(__FUNCTION__, "ngsiv2 create subscription request", reqSemTaken);
 
   return subId;

src/lib/mongoBackend/mongoGetSubscriptions.cpp

@@ -210,17 +210,7 @@ static void setNotification(Subscription* subP, const orion::BSONObj& r, const s
   nP->lastSuccessCode   = r.hasField(CSUB_LASTSUCCESSCODE)?  getIntOrLongFieldAsLongF(r, CSUB_LASTSUCCESSCODE)  : -1;
 
   // Attributes format
-  subP->attrsFormat = r.hasField(CSUB_FORMAT)? stringToRenderFormat(getStringFieldF(r, CSUB_FORMAT)) : NGSI_V1_LEGACY;
-
-  if (subP->attrsFormat == NGSI_V1_LEGACY)
-  {
-    __sync_fetch_and_add(&noOfDprLegacyNotif, 1);
-    if (logDeprecate)
-    {
-      LM_W(("Deprecated usage of notification legacy format detected in existing subscription (subId: %s)", subP->id.c_str()));
-    }
-  }
-
+  subP->attrsFormat = r.hasField(CSUB_FORMAT)? stringToRenderFormat(getStringFieldF(r, CSUB_FORMAT)) : NGSI_V2_NORMALIZED;
 
   //
   // Check values from subscription cache, update object from cache-values if necessary

src/lib/mongoBackend/mongoUpdateSubscription.cpp

@@ -370,15 +370,6 @@ std::string mongoUpdateSubscription
   if (subUp.notifyOnMetadataChangeProvided) setNotifyOnMetadataChange(subUp, &setB);
   if (subUp.attrsFormatProvided)   setFormat(subUp, &setB);
 
-  if (subUp.attrsFormat == NGSI_V1_LEGACY)
-  {
-    __sync_fetch_and_add(&noOfDprLegacyNotif, 1);
-    if (logDeprecate)
-    {
-      LM_W(("Deprecated usage of notification legacy format in subscription modification (subId: %s)", subUp.id.c_str()));
-    }
-  }
-
   // Description is special, as "" value removes the field
   if (subUp.descriptionProvided)
   {

src/lib/ngsi10/SubscribeContextRequest.cpp

@@ -180,7 +180,7 @@ void SubscribeContextRequest::toNgsiv2Subscription(Subscription* sub)
   // description and expression are not touched, so default empty string provided by constructor will be used
   sub->status                         = STATUS_ACTIVE;
   sub->descriptionProvided            = false;
-  sub->attrsFormat                    = NGSI_V1_LEGACY;
+  sub->attrsFormat                    = NGSI_V2_NORMALIZED;
   sub->notification.blacklist         = false;
   sub->notification.httpInfo.custom   = false;
 

src/lib/ngsiNotify/Notifier.cpp

@@ -151,20 +151,7 @@ static bool setPayload
     ncr.subscriptionId  = subscriptionId;
     ncr.contextElementResponseVector.push_back(&cer);
 
-    if (*renderFormatP == NGSI_V1_LEGACY)
-    {
-      __sync_fetch_and_add(&noOfDprLegacyNotif, 1);
-      if (logDeprecate)
-      {
-        LM_W(("Deprecated usage of notification legacy format in notification (subId: %s)", subscriptionId.c_str()));
-      }
-
-      *payloadP = ncr.toJsonV1(false, attrsFilter, blacklist, metadataFilter);
-    }
-    else
-    {
-      *payloadP = ncr.toJson(*renderFormatP, attrsFilter, blacklist, metadataFilter);
-    }
+    *payloadP = ncr.toJson(*renderFormatP, attrsFilter, blacklist, metadataFilter);
 
     *mimeTypeP = "application/json";
   }
@@ -681,22 +668,7 @@ SenderThreadParams* Notifier::buildSenderParams
 
     ci.outMimeType = JSON;
 
-    std::string payloadString;
-    if (renderFormat == NGSI_V1_LEGACY)
-    {
-      __sync_fetch_and_add(&noOfDprLegacyNotif, 1);
-      if (logDeprecate)
-      {
-        LM_W(("Deprecated usage of notification legacy format in notification (subId: %s)", subId.c_str()));
-      }
-
-      bool asJsonObject = (ci.uriParam[URI_PARAM_ATTRIBUTE_FORMAT] == "object" && ci.outMimeType == JSON);
-      payloadString = ncr.toJsonV1(asJsonObject, attrsFilter, blacklist, metadataFilter);
-    }
-    else
-    {
-      payloadString = ncr.toJson(renderFormat, attrsFilter, blacklist, metadataFilter);
-    }
+    std::string payloadString = ncr.toJson(renderFormat, attrsFilter, blacklist, metadataFilter);
 
     /* Parse URL */
     std::string  host;

src/lib/serviceRoutines/statisticsTreat.cpp

@@ -93,7 +93,6 @@ static void resetStatistics(void)
 
   noOfDprNgsiv1Request         = -1;
   noOfDprLegacyForwarding      = -1;
-  noOfDprLegacyNotif           = -1;
   noOfDprGeoformat             = -1;
 
   statisticsTime = getCurrentTime();
@@ -221,7 +220,6 @@ std::string renderCounterStats(bool fullCounters)
   JsonObjectHelper jsDeprecated;
   renderUsedCounter(&jsDeprecated, "ngsiv1Requests", noOfDprNgsiv1Request, fullCounters);
   renderUsedCounter(&jsDeprecated, "ngsiv1Forwarding", noOfDprLegacyForwarding, fullCounters);
-  renderUsedCounter(&jsDeprecated, "ngsiv1NotifFormat", noOfDprLegacyNotif, fullCounters);
   renderUsedCounter(&jsDeprecated, "geoFormat", noOfDprGeoformat, fullCounters);
 
   std::string deprecation = jsDeprecated.str();