diff --git a/CHANGES_NEXT_RELEASE b/CHANGES_NEXT_RELEASE index 0fb2c7d025..93817432aa 100644 --- a/CHANGES_NEXT_RELEASE +++ b/CHANGES_NEXT_RELEASE @@ -1,3 +1,4 @@ +- Add: alarm for notification queue overpassing a given threshold (new CLI `-notifAlarmThreshold` / env var `ORION_NOTIF_ALARM_THRESHOLD`) (#4113) - Add: servicePath field to builtin attributes (#2877) - Add: notification.mqtt.retain and notification.mqttCustom.retain flag for MQTT retain in notifications (#4388) - Fix: logDeprecate not working correctly (`geo:json` wrongly considered as deprecated) diff --git a/doc/manuals/admin/cli.md b/doc/manuals/admin/cli.md index 0899b45255..57613e77e1 100644 --- a/doc/manuals/admin/cli.md +++ b/doc/manuals/admin/cli.md @@ -191,6 +191,7 @@ The list of available options is the following: to the `-k` or `--insecure` parameteres of the curl command. - **-mqttMaxAge**. Max time (in minutes) that an unused MQTT connection is kept. Default: 60 - **-logDeprecate**. Log deprecation usages as warnings. More information in [this section of the documentation](../deprecated.md#log-deprecation-warnings). Default is: false. It can be changed after Orion startup with the [log admin REST API](management_api.md#log-configs-and-trace-levels), with the `deprecated` field +- **-notifAlarmThreshold**. Maximum threshold for notification queue alarms, as a percentage of the maximum queue size, default 0 (meaning no queue alarms are used) ## Configuration using environment variables @@ -270,3 +271,4 @@ Two facts have to be taken into account: | ORION_NGSIV1_AUTOCAST | ngsiv1Autocast | | ORION_MQTT_MAX_AGE | mqttMaxAge | | ORION_LOG_DEPRECATE | logDeprecate | +| ORION_NOTIF_ALARM_THRESHOLD | notifAlarmThreshold | diff --git a/doc/manuals/admin/logs.md b/doc/manuals/admin/logs.md index 192e835dbe..5fd63ef899 100644 --- a/doc/manuals/admin/logs.md +++ b/doc/manuals/admin/logs.md @@ -233,6 +233,7 @@ Alarm conditions: | 5 | WARNING | The following WARN text appears in the 'msg' field: "Raising alarm NotificationError ``: ``". | The following WARN text appears in the 'msg' field: "Releasing alarm NotificationError ``", where `` is the same one that triggered the alarm. Orion prints this trace when it successfully sent a notification to that URL. | Notification Failure. The ``text contains the detailed information. | Orion is trying to send the HTTP notification to a given receiver and some problem has occurred. It could be due to a problem with the network connectivity or on the receiver, e.g. the receiver is down. In the second case, the owner of the receiver of the notification should be reported. No specific action has to be performed at Orion Context Broker service. | 6 | WARNING | The following WARN text appears in the 'msg' field: "Raising alarm ForwardingError ``": ``". | The following WARN text appears in the 'msg' field: "Releasing alarm ForwardingError ``", where `` is the same one that triggered the alarm. Orion prints this trace when it successfully interact with ContextProvider to that URL.| Forwarding Error. The ``text contains the detailed information. | Orion is trying to interact with ContextProvider and some problem has occurred. It may be due to context provider response for forwarded query or update is empty. No specific action has to be performed at Orion Context Broker service. | 7 | WARNING | The following WARN text appears in the 'msg' field: "Raising alarm MqttConnectionError ``": ``". | The following WARN text appears in the 'msg' field: "Releasing alarm MqttConnectionError ``", where `` is the same one that triggered the alarm. Orion prints this trace when it successfully interact with ContextProvider to that URL.| Error connection to MQTT broker. The ``text contains the detailed information. | Orion is trying to connecto to an MQTT broker (associated to a subscription) and some problem has occurred. It may be due to several reasons: MQTT broker in unreachable, user/pass is wrong, etc. No specific action has to be performed at Orion Context Broker service, but maybe in the MQTT broker configuration or in the associated subscription. +| 8 | WARNING | The following WARN text appears in the 'msg' field: "Raising alarm NotificationQueue ``: ``". | The following WARN text appears in the 'msg' field: "Releasing alarm NotificationQueue ``", where `` is the same one that triggered the alarm. Orion prints this trace when notification queue goes back below the threshold. | The notification queue associated to the service (or "default" for default queue) has overpassed the alarm threshold. The `` text described the particular threshold. | No specific action has to be performed at Orion Context Broker service, but the update flow causing the notification on that service (or default queue) should be lowered in order to reduce pressure on queue. Another possible problem is due to malfunctioning notification receivers, if they are slow processing notifications and responding to Orion. By default, Orion only traces the origin (i.e. raising) and end (i.e. releasing) of an alarm, e.g: diff --git a/src/app/contextBroker/contextBroker.cpp b/src/app/contextBroker/contextBroker.cpp index bf81abec8f..050b86cfca 100644 --- a/src/app/contextBroker/contextBroker.cpp +++ b/src/app/contextBroker/contextBroker.cpp @@ -207,6 +207,7 @@ unsigned long fcMaxInterval; int mqttMaxAge; bool logDeprecate; +int notifAlarmThreshold; @@ -277,6 +278,7 @@ bool logDeprecate; #define NGSIV1_AUTOCAST_DESC "automatic cast for number, booleans and dates in NGSIv1 update/create attribute operations" #define MQTT_MAX_AGE_DESC "max time (in minutes) that an unused MQTT connection is kept, default: 60" #define LOG_DEPRECATE_DESC "log deprecation usages as warnings" +#define NOTIF_ALARM_THRESHOLD_DESC "maximum threshold for notification queue alarms, as a percentage of the maximum queue size, default 0 (meaning no queue alarms are used)" #define DBURI_DESC "complete URI for database connection" @@ -336,10 +338,8 @@ PaArgument paArgs[] = { "-connectionMemory", &connectionMemory, "CONN_MEMORY", PaUInt, PaOpt, 64, 0, 1024, CONN_MEMORY_DESC }, { "-maxConnections", &maxConnections, "MAX_CONN", PaUInt, PaOpt, 1020, 1, PaNL, MAX_CONN_DESC }, { "-reqPoolSize", &reqPoolSize, "TRQ_POOL_SIZE", PaUInt, PaOpt, 0, 0, 1024, REQ_POOL_SIZE }, - { "-inReqPayloadMaxSize", &inReqPayloadMaxSize, "IN_REQ_PAYLOAD_MAX_SIZE", PaULong, PaOpt, DEFAULT_IN_REQ_PAYLOAD_MAX_SIZE, 0, PaNL, IN_REQ_PAYLOAD_MAX_SIZE_DESC }, { "-outReqMsgMaxSize", &outReqMsgMaxSize, "OUT_REQ_MSG_MAX_SIZE", PaULong, PaOpt, DEFAULT_OUT_REQ_MSG_MAX_SIZE, 0, PaNL, OUT_REQ_MSG_MAX_SIZE_DESC }, - { "-notificationMode", ¬ificationMode, "NOTIF_MODE", PaString, PaOpt, _i "transient", PaNL, PaNL, NOTIFICATION_MODE_DESC }, { "-notifFlowControl", ¬ifFlowControl, "NOTIF_FLOW_CONTROL", PaString, PaOpt, _i "", PaNL, PaNL, FLOW_CONTROL_DESC }, { "-simulatedNotification", &simulatedNotification, "DROP_NOTIF", PaBool, PaOpt, false, false, true, SIMULATED_NOTIF_DESC }, @@ -371,6 +371,8 @@ PaArgument paArgs[] = { "-logDeprecate", &logDeprecate, "LOG_DEPRECATE", PaBool, PaOpt, false, false, true, LOG_DEPRECATE_DESC }, + { "-notifAlarmThreshold", ¬ifAlarmThreshold, "NOTIF_ALARM_THRESHOLD", PaInt, PaOpt, 0, PaNL, PaNL, NOTIF_ALARM_THRESHOLD_DESC }, + PA_END_OF_ARGS }; @@ -673,6 +675,15 @@ static void contextBrokerInit(void) pNotifier = new Notifier(); } + if (notifAlarmThreshold < 0) + { + LM_X(1, ("Fatal Error (notifAlarmThreshold negative value not allowed)")); + } + if (notifAlarmThreshold > 100) + { + LM_X(1, ("Fatal Error (notifAlarmThreshold value is greater than 100)")); + } + /* Set notifier object (singleton) */ setNotifier(pNotifier); diff --git a/src/lib/alarmMgr/AlarmManager.cpp b/src/lib/alarmMgr/AlarmManager.cpp index da84e66b79..fd2a2226e1 100644 --- a/src/lib/alarmMgr/AlarmManager.cpp +++ b/src/lib/alarmMgr/AlarmManager.cpp @@ -58,6 +58,8 @@ AlarmManager::AlarmManager() badInputResets(0), notificationErrors(0), notificationErrorResets(0), + notificationQueues(0), + notificationQueueResets(0), forwardingErrors(0), forwardingErrorResets(0), mqttConnectionErrors(0), @@ -66,6 +68,7 @@ AlarmManager::AlarmManager() dbErrorResets(0), dbOk(true), notificationErrorLogAlways(false), + notificationQueueLogAlways(false), forwardingErrorLogAlways(false), mqttConnectionErrorLogAlways(false), badInputLogAlways(false), @@ -82,6 +85,7 @@ AlarmManager::AlarmManager() int AlarmManager::init(bool logAlreadyRaisedAlarms) { notificationErrorLogAlways = logAlreadyRaisedAlarms; + notificationQueueLogAlways = logAlreadyRaisedAlarms; badInputLogAlways = logAlreadyRaisedAlarms; dbErrorLogAlways = logAlreadyRaisedAlarms; mqttConnectionErrorLogAlways = logAlreadyRaisedAlarms; @@ -246,6 +250,21 @@ void AlarmManager::notificationErrorGet(int64_t* active, int64_t* raised, int64_ } +/* **************************************************************************** +* +* AlarmManager::notificationQueueGet - +* +* NOTE +* To read values, no semaphore is used. +*/ +void AlarmManager::notificationQueueGet(int64_t* active, int64_t* raised, int64_t* released) +{ + *active = notificationQ.size(); + *raised = notificationQueues; + *released = notificationQueueResets; +} + + /* **************************************************************************** * @@ -589,6 +608,74 @@ bool AlarmManager::forwardingErrorReset(const std::string& url) +/* **************************************************************************** +* +* AlarmManager::notificationQueue - +* +* Returns false if no effective alarm transition occurs, otherwise, true is returned. +*/ +bool AlarmManager::notificationQueue(const std::string& service, const std::string& details) +{ + semTake(); + + std::map::iterator iter = notificationQ.find(service); + + if (iter != notificationQ.end()) + { + iter->second += 1; + + if (notificationQueueLogAlways) + { + LM_W(("Repeated NotificationQueue %s: %s", service.c_str(), details.c_str())); + } + else + { + // even if repeat alarms is off, this is a relevant event in debug level + LM_T(LmtNotifierQueue, ("Repeated NotificationQueue %s: %s", service.c_str(), details.c_str())); + } + + semGive(); + return false; + } + + ++notificationQueues; + + notificationQ[service] = 1; + semGive(); + + LM_W(("Raising alarm NotificationQueue %s: %s", service.c_str(), details.c_str())); + + return true; +} + + + +/* **************************************************************************** +* +* AlarmManager::notificationQueuesResets - +* +* Returns false if no effective alarm transition occurs, otherwise, true is returned. +*/ +bool AlarmManager::notificationQueueReset(const std::string& service) +{ + semTake(); + + if (notificationQ.find(service) == notificationQ.end()) // Doesn't exist + { + semGive(); + return false; + } + + notificationQ.erase(service); + ++notificationQueueResets; + semGive(); + + LM_W(("Releasing alarm NotificationQueue %s", service.c_str())); + + return true; +} + + /* **************************************************************************** * * AlarmManager::badInputReset - diff --git a/src/lib/alarmMgr/AlarmManager.h b/src/lib/alarmMgr/AlarmManager.h index 685eba17ed..3bd7f9201e 100644 --- a/src/lib/alarmMgr/AlarmManager.h +++ b/src/lib/alarmMgr/AlarmManager.h @@ -44,6 +44,8 @@ class AlarmManager int64_t badInputResets; int64_t notificationErrors; int64_t notificationErrorResets; + int64_t notificationQueues; + int64_t notificationQueueResets; int64_t forwardingErrors; int64_t forwardingErrorResets; int64_t mqttConnectionErrors; @@ -53,11 +55,13 @@ class AlarmManager bool dbOk; std::map notificationV; + std::map notificationQ; std::map forwardingErrorV; std::map badInputV; std::map mqttConnectionErrorV; bool notificationErrorLogAlways; + bool notificationQueueLogAlways; bool forwardingErrorLogAlways; bool mqttConnectionErrorLogAlways; bool badInputLogAlways; @@ -80,6 +84,9 @@ class AlarmManager bool notificationError(const std::string& url, const std::string& details); bool notificationErrorReset(const std::string& url); + bool notificationQueueReset(const std::string& service); + bool notificationQueue(const std::string& service, const std::string& details); + bool forwardingError(const std::string& url, const std::string& details); bool forwardingErrorReset(const std::string& url); @@ -93,6 +100,7 @@ class AlarmManager void dbErrorsGet(bool* active, int64_t* raised, int64_t* released); void badInputGet(int64_t* active, int64_t* raised, int64_t* released); void notificationErrorGet(int64_t* active, int64_t* raised, int64_t* released); + void notificationQueueGet(int64_t* active, int64_t* raised, int64_t* released); void forwardingErrorGet(int64_t* active, int64_t* raised, int64_t* released); void mqttConnectionErrorGet(int64_t* active, int64_t* raised, int64_t* released); diff --git a/src/lib/logMsg/traceLevels.h b/src/lib/logMsg/traceLevels.h index d0df21e423..30e1b5aec3 100644 --- a/src/lib/logMsg/traceLevels.h +++ b/src/lib/logMsg/traceLevels.h @@ -102,6 +102,7 @@ typedef enum TraceLevels LmtNotificationRequestPayload, LmtNotificationResponsePayload, LmtMqttNotif, + LmtNotifierQueue, /* Input/Output payloads (180-199) */ LmtServiceInputPayload = 180, diff --git a/src/lib/logSummary/logSummary.cpp b/src/lib/logSummary/logSummary.cpp index d566f122c0..165c00d034 100644 --- a/src/lib/logSummary/logSummary.cpp +++ b/src/lib/logSummary/logSummary.cpp @@ -114,6 +114,7 @@ static void* logSummary(void* vP) alarmMgr.dbErrorsGet(&deActive, &deRaised, &deReleased); alarmMgr.badInputGet(&biActive, &biRaised, &biReleased); alarmMgr.notificationErrorGet(&neActive, &neRaised, &neReleased); + alarmMgr.notificationQueueGet(&neActive, &neRaised, &neReleased); alarmMgr.forwardingErrorGet(&fwdActive, &fwdRaised, &fwdReleased); alarmMgr.mqttConnectionErrorGet(&mceActive, &mceRaised, &mceReleased); diff --git a/src/lib/ngsiNotify/QueueNotifier.cpp b/src/lib/ngsiNotify/QueueNotifier.cpp index 71a8e47ae2..d10f0951f7 100644 --- a/src/lib/ngsiNotify/QueueNotifier.cpp +++ b/src/lib/ngsiNotify/QueueNotifier.cpp @@ -177,6 +177,7 @@ void QueueNotifier::sendNotifyContextRequest } bool enqueued = sq->try_push(paramsP); + if (!enqueued) { QueueStatistics::incReject(1); @@ -186,5 +187,23 @@ void QueueNotifier::sendNotifyContextRequest return; } + extern int notifAlarmThreshold; + + if (notifAlarmThreshold != 0) + { + std::string details = ("notification queue reached maximum threshold"); + + long unsigned int threshold = queueSize(service)*notifAlarmThreshold/100; + + if (threshold >= queueSize(service)) + { + alarmMgr.notificationQueue(queueName.c_str(), details.c_str()); + } + } + else + { + alarmMgr.notificationQueueReset(queueName.c_str()); + } + QueueStatistics::incIn(1); } diff --git a/test/functionalTest/cases/0000_cli/bool_option_with_value.test b/test/functionalTest/cases/0000_cli/bool_option_with_value.test index 157a33b97a..6b529f831d 100644 --- a/test/functionalTest/cases/0000_cli/bool_option_with_value.test +++ b/test/functionalTest/cases/0000_cli/bool_option_with_value.test @@ -104,5 +104,6 @@ Usage: contextBroker [option '-U' (extended usage)] [option '-ngsiv1Autocast' (automatic cast for number, booleans and dates in NGSIv1 update/create attribute operations)] [option '-mqttMaxAge' ] [option '-logDeprecate' (log deprecation usages as warnings)] + [option '-notifAlarmThreshold' ] --TEARDOWN-- diff --git a/test/functionalTest/cases/0000_cli/command_line_options.test b/test/functionalTest/cases/0000_cli/command_line_options.test index 20a639175b..48bf8eca4f 100644 --- a/test/functionalTest/cases/0000_cli/command_line_options.test +++ b/test/functionalTest/cases/0000_cli/command_line_options.test @@ -93,5 +93,6 @@ Usage: contextBroker [option '-U' (extended usage)] [option '-ngsiv1Autocast' (automatic cast for number, booleans and dates in NGSIv1 update/create attribute operations)] [option '-mqttMaxAge' ] [option '-logDeprecate' (log deprecation usages as warnings)] + [option '-notifAlarmThreshold' ] --TEARDOWN-- diff --git a/test/functionalTest/cases/0000_cli/tracelevel_without_logLevel_as_DEBUG.test b/test/functionalTest/cases/0000_cli/tracelevel_without_logLevel_as_DEBUG.test index cf566803ce..d3fb989278 100644 --- a/test/functionalTest/cases/0000_cli/tracelevel_without_logLevel_as_DEBUG.test +++ b/test/functionalTest/cases/0000_cli/tracelevel_without_logLevel_as_DEBUG.test @@ -94,5 +94,6 @@ Usage: contextBroker [option '-U' (extended usage)] [option '-ngsiv1Autocast' (automatic cast for number, booleans and dates in NGSIv1 update/create attribute operations)] [option '-mqttMaxAge' ] [option '-logDeprecate' (log deprecation usages as warnings)] + [option '-notifAlarmThreshold' ] --TEARDOWN-- diff --git a/test/functionalTest/cases/3658_env_vars/env_vars.test b/test/functionalTest/cases/3658_env_vars/env_vars.test index 495c359dec..df51ec4272 100644 --- a/test/functionalTest/cases/3658_env_vars/env_vars.test +++ b/test/functionalTest/cases/3658_env_vars/env_vars.test @@ -139,6 +139,7 @@ Extended Usage: contextBroker [option '-U' (extended usage)] [option '-ngsiv1Autocast' (automatic cast for number, booleans and dates in NGSI] ORION_NGSIV1_AUTOCAST FALSE /FALSE/ [option '-mqttMaxAge' /dev/null +done +echo +echo + + +echo "05. Create/update entity in serv2 3 times (update #2 raises alarm)" +echo "==================================================================" +for i in {1..3} +do + echo $i + orionCurl --url '/v2/entities?options=upsert,forcedUpdate' --payload "$payload" --tenant serv2 > /dev/null +done +echo +echo + + +echo "06. Create/update entity in serv3 (default) 7 times (update #4 raises alarm)" +echo "============================================================================" +for i in {1..7} +do + echo $i + orionCurl --url '/v2/entities?options=upsert,forcedUpdate' --payload "$payload" --tenant serv3 > /dev/null +done +echo +echo + + +echo "07. Grep log for notificationQueue alarm" +echo "========================================" +egrep 'Raising alarm|Releasing alarm|Repeated' /tmp/contextBroker.log | awk -F'|' '{ print $10 }' | awk -F= '{ print $2 }' +echo +echo + + +accumulatorStop $LISTENER_PORT + + +--REGEXPECT-- +01. Subscribe serv1 to the accumulator endpoint that never responses +==================================================================== +HTTP/1.1 201 Created +Date: REGEX(.*) +Fiware-Correlator: REGEX([0-9a-f\-]{36}) +Location: /v2/subscriptions/REGEX([0-9a-f]{24}) +Content-Length: 0 + + + +02. Subscribe serv2 to the accumulator endpoint that never responses +==================================================================== +HTTP/1.1 201 Created +Date: REGEX(.*) +Fiware-Correlator: REGEX([0-9a-f\-]{36}) +Location: /v2/subscriptions/REGEX([0-9a-f]{24}) +Content-Length: 0 + + + +03. Subscribe serv3 to the accumulator endpoint that never responses +==================================================================== +HTTP/1.1 201 Created +Date: REGEX(.*) +Fiware-Correlator: REGEX([0-9a-f\-]{36}) +Location: /v2/subscriptions/REGEX([0-9a-f]{24}) +Content-Length: 0 + + + +04. Create/update entity in serv1 5 times (update #3 raises alarm) +================================================================== +1 +2 +3 +4 +5 + + +05. Create/update entity in serv2 3 times (update #2 raises alarm) +================================================================== +1 +2 +3 + + +06. Create/update entity in serv3 (default) 7 times (update #4 raises alarm) +============================================================================ +1 +2 +3 +4 +5 +6 +7 + + +07. Grep log for notificationQueue alarm +======================================== +Raising alarm NotificationQueue serv1: notification queue reached maximum threshold +Raising alarm NotificationQueue serv2: notification queue reached maximum threshold +Raising alarm NotificationQueue default: notification queue reached maximum threshold + + +--TEARDOWN-- +brokerStop CB +dbDrop ${CB_DB_NAME} serv1 +dbDrop ${CB_DB_NAME} serv2 +dbDrop ${CB_DB_NAME} serv3 diff --git a/test/functionalTest/cases/4113_alarm_for_notification_queue_overpassing_a_given_threshold/alarm_for_notification_queue_overpassing_a_given_threshold_relog.test b/test/functionalTest/cases/4113_alarm_for_notification_queue_overpassing_a_given_threshold/alarm_for_notification_queue_overpassing_a_given_threshold_relog.test new file mode 100644 index 0000000000..dc37e8a61c --- /dev/null +++ b/test/functionalTest/cases/4113_alarm_for_notification_queue_overpassing_a_given_threshold/alarm_for_notification_queue_overpassing_a_given_threshold_relog.test @@ -0,0 +1,240 @@ +# Copyright 2023 Telefonica Investigacion y Desarrollo, S.A.U +# +# This file is part of Orion Context Broker. +# +# Orion Context Broker is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, either version 3 of the +# License, or (at your option) any later version. +# +# Orion Context Broker is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero +# General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Orion Context Broker. If not, see http://www.gnu.org/licenses/. +# +# For those usages not covered by this license please contact with +# iot_support at tid dot es + +# VALGRIND_READY - to mark the test ready for valgrindTestSuite.sh + +--NAME-- +alarm for notification queue overpassing a given threshold + +--SHELL-INIT-- +dbInit ${CB_DB_NAME} serv1 +dbInit ${CB_DB_NAME} serv2 +dbInit ${CB_DB_NAME} serv3 +brokerStart CB 0 IPv4 -multiservice -relogAlarms -notifAlarmThreshold 50 -notificationMode threadpool:6:2,serv1:4:2,serv2:2:2 +accumulatorStart --pretty-print localhost $LISTENER_PORT + +--SHELL-- + +# +# 01. Subscribe serv1 to the accumulator endpoint that never responses +# 02. Subscribe serv2 to the accumulator endpoint that never responses +# 03. Subscribe serv3 to the accumulator endpoint that never responses +# 04. Create/update entity in serv1 5 times (update #3 raises alarm, update #4 and #5 cause repeated log) +# 05. Create/update entity in serv2 3 times (update #2 raises alarm, update #3 cause repeated log) +# 06. Create/update entity in serv3 (default) 7 times (update #4 raises alarm, updates #5, #6 and #7 cause repeated log) +# 07. Grep log for notificationQueue alarm +# + + +echo '01. Subscribe serv1 to the accumulator endpoint that never responses' +echo '====================================================================' +payload='{ + "description": "serv1 subscriptions", + "subject": { + "entities": [ + { + "id": "E", + "type": "T" + } + ] + }, + "notification": { + "http": { + "url": "http://localhost:'$LISTENER_PORT'/waitForever" + } + } +}' +orionCurl --url /v2/subscriptions --payload "$payload" --tenant serv1 +echo +echo + + +echo '02. Subscribe serv2 to the accumulator endpoint that never responses' +echo '====================================================================' +payload='{ + "description": "serv2 subscriptions", + "subject": { + "entities": [ + { + "id": "E", + "type": "T" + } + ] + }, + "notification": { + "http": { + "url": "http://localhost:'$LISTENER_PORT'/waitForever" + } + } +}' +orionCurl --url /v2/subscriptions --payload "$payload" --tenant serv2 +echo +echo + + + +echo '03. Subscribe serv3 to the accumulator endpoint that never responses' +echo '====================================================================' +payload='{ + "description": "serv3 subscriptions", + "subject": { + "entities": [ + { + "id": "E", + "type": "T" + } + ] + }, + "notification": { + "http": { + "url": "http://localhost:'$LISTENER_PORT'/waitForever" + } + } +}' +orionCurl --url /v2/subscriptions --payload "$payload" --tenant serv3 +echo +echo + + + +echo "04. Create/update entity in serv1 5 times (update #3 raises alarm, update #4 and #5 cause repeated log)" +echo "=======================================================================================================" +payload='{ + "id": "E", + "type": "T", + "A": { + "value": 1, + "type": "Number" + } +}' +for i in {1..5} +do + echo $i + orionCurl --url '/v2/entities?options=upsert,forcedUpdate' --payload "$payload" --tenant serv1 > /dev/null +done +echo +echo + + +echo "05. Create/update entity in serv2 3 times (update #2 raises alarm, update #3 cause repeated log)" +echo "================================================================================================" +for i in {1..3} +do + echo $i + orionCurl --url '/v2/entities?options=upsert,forcedUpdate' --payload "$payload" --tenant serv2 > /dev/null +done +echo +echo + + +echo "06. Create/update entity in serv3 (default) 7 times (update #4 raises alarm, updates #5, #6 and #7 cause repeated log)" +echo "======================================================================================================================" +for i in {1..7} +do + echo $i + orionCurl --url '/v2/entities?options=upsert,forcedUpdate' --payload "$payload" --tenant serv3 > /dev/null +done +echo +echo + + +echo "07. Grep log for notificationQueue alarm" +echo "========================================" +egrep 'Raising alarm|Releasing alarm|Repeated' /tmp/contextBroker.log | awk -F'|' '{ print $10 }' | awk -F= '{ print $2 }' +echo +echo + + +accumulatorStop $LISTENER_PORT + + +--REGEXPECT-- +01. Subscribe serv1 to the accumulator endpoint that never responses +==================================================================== +HTTP/1.1 201 Created +Date: REGEX(.*) +Fiware-Correlator: REGEX([0-9a-f\-]{36}) +Location: /v2/subscriptions/REGEX([0-9a-f]{24}) +Content-Length: 0 + + + +02. Subscribe serv2 to the accumulator endpoint that never responses +==================================================================== +HTTP/1.1 201 Created +Date: REGEX(.*) +Fiware-Correlator: REGEX([0-9a-f\-]{36}) +Location: /v2/subscriptions/REGEX([0-9a-f]{24}) +Content-Length: 0 + + + +03. Subscribe serv3 to the accumulator endpoint that never responses +==================================================================== +HTTP/1.1 201 Created +Date: REGEX(.*) +Fiware-Correlator: REGEX([0-9a-f\-]{36}) +Location: /v2/subscriptions/REGEX([0-9a-f]{24}) +Content-Length: 0 + + + +04. Create/update entity in serv1 5 times (update #3 raises alarm, update #4 and #5 cause repeated log) +======================================================================================================= +1 +2 +3 +4 +5 + + +05. Create/update entity in serv2 3 times (update #2 raises alarm, update #3 cause repeated log) +================================================================================================ +1 +2 +3 + + +06. Create/update entity in serv3 (default) 7 times (update #4 raises alarm, updates #5, #6 and #7 cause repeated log) +====================================================================================================================== +1 +2 +3 +4 +5 +6 +7 + + +07. Grep log for notificationQueue alarm +======================================== +Raising alarm NotificationQueue serv1: notification queue reached maximum threshold +Repeated NotificationQueue serv1: notification queue reached maximum threshold +Raising alarm NotificationQueue serv2: notification queue reached maximum threshold +Repeated NotificationQueue serv2: notification queue reached maximum threshold +Raising alarm NotificationQueue default: notification queue reached maximum threshold +Repeated NotificationQueue default: notification queue reached maximum threshold + + +--TEARDOWN-- +brokerStop CB +dbDrop ${CB_DB_NAME} serv1 +dbDrop ${CB_DB_NAME} serv2 +dbDrop ${CB_DB_NAME} serv3