Running EdgeDriver on a Different Core

[Hieromon]

This topic is derived from the Internet Transparency checks that have been discussed in this thread.

@FRANAIRBUS Your trial inspired me to do the following:

Wouldn't it be useful to be able to poll using a ping loop on a different core? But this technique can only be used with ESP32 with multiple cores.
So I coded a simple evaluation sketch. If this idea is really useful, I'm ready to enhance EdgeUnifed so that EdgeDriver can run on a different core. It would take the place of semaphore control and task creation in the following evaluation sketch.

#include <Arduino.h>
#include <WiFi.h>
#include <WebServer.h>
#include <freertos/task.h>
#include <ping/ping_sock.h>
#include <AutoConnect.h>
#include <EdgeUnified.h>

#ifndef LED_BUILTIN
#pragma message("Warning, LED_BUILTIN is undefined. Assumes Pin #2.")
#define LED_BUILTIN 2
#endif

static const uint32_t PING_DESTINATION = 0x08080808UL;  // 8.8.8.8

typedef struct {
  esp_ping_handle_t hdl;
  uint32_t  destination = PING_DESTINATION;
  uint32_t  count = 1;
  uint32_t  interval = 0;
  uint32_t  timeout = 1000;
  bool  hasTimeout = false;
} PING_t;

void startPing(void);
void processPing(void) {}
void endPing(void);
void pollingByPing(void*);

EdgeDriver<PING_t>  ping(startPing, processPing, endPing);

TaskHandle_t  taskPing;
static SemaphoreHandle_t  pingSession = NULL;

void startPing() {
  Serial.println("Starting PING");
  vSemaphoreCreateBinary(pingSession);
  xTaskCreatePinnedToCore(pollingByPing, "pollingByPing", 4096, NULL, 1, &taskPing, CONFIG_ARDUINO_RUNNING_CORE != pdPASS);
}

void endPing() {
  esp_ping_delete_session(ping.data.hdl);
}

static void pingOnSuccess(esp_ping_handle_t hdl, void* args) {
  (void)(hdl);
  (void)(args);
  ping.data.timeout = false;
  Serial.println("success");
}

static void pingOnTimeout(esp_ping_handle_t hdl, void* args) {
  (void)(hdl);
  (void)(args);
  ping.data.timeout = true;
  Serial.println("failed");
}

static void pingOnEnd(esp_ping_handle_t hdl, void* args) {
  (void)(hdl);
  (void)(args);
  xSemaphoreGive(pingSession);
}

void pollingByPing(void* param) {
  (void)(param);
  ip_addr_t dest;
  esp_ping_config_t configPing = ESP_PING_DEFAULT_CONFIG();
  esp_ping_callbacks_t  cbs = {
    .cb_args = NULL,
    .on_ping_success = pingOnSuccess,
    .on_ping_timeout = pingOnTimeout,
    .on_ping_end = pingOnEnd
  };

  dest.type = 0;
  dest.u_addr.ip4.addr = ping.data.destination;
  configPing.count = ping.data.count;
  configPing.interval_ms = ping.data.interval;
  configPing.timeout_ms = ping.data.timeout;
  configPing.target_addr = dest;

  xSemaphoreTake(pingSession, portMAX_DELAY);
  esp_ping_new_session(&configPing, &cbs, &ping.data.hdl);
  xSemaphoreGive(pingSession);
  
  while (true) {
    Serial.printf("Ping %s - ", IPAddress(dest.u_addr.ip4.addr).toString().c_str());
    esp_ping_start(ping.data.hdl);
    vTaskDelay(1000 / portTICK_PERIOD_MS);
  }
}

typedef struct {
  int     pin = LED_BUILTIN;
  unsigned long cycle = 500;
} GPIO_t;

void startGPIO(void);
void processGPIO(void);
void endGPIO(void);

EdgeDriver<GPIO_t>  gpio(startGPIO, processGPIO, endGPIO);

void startGPIO() {
  Serial.println("Starting GPIO");
  pinMode(gpio.data.pin, OUTPUT);
  digitalWrite(gpio.data.pin, LOW);
  gpio.setEdgeInterval(gpio.data.cycle);
}

void processGPIO() {
  int blink = ping.data.hasTimeout ? LOW : !digitalRead(gpio.data.pin);
  digitalWrite(gpio.data.pin, blink);
}

void endGPIO() {
  digitalWrite(gpio.data.pin, LOW);
}

AutoConnect portal;

void setup() {
  delay(1000);
  Serial.begin(115200);
  Serial.printf("Core %d is running\n", xPortGetCoreID());

  Edge.attach(gpio);
  
  if (portal.begin()) {
    Edge.attach(ping);
  }
}

void loop() {
  Edge.process(portal);
  portal.handleClient();
}

[FRANAIRBUS]

Hello @Hieromon

It is impressive how he writes a new code in a very short time and it seems like a very good idea to me, but I have some doubts...

so you mean that autoconnect and edgedriver can work from core 1 or core 0 with a single option or API? each independent?

or is it that autoconnect aux and menus can work on one core and the process driver on another?

this last case would solve the previous problem of blocking http requests to custom autoconnect menu pages

notice at the time of your ping example when i ask to configure new ap as it hangs the process of led_ping for about 3 seconds until it loads the webpage

the behavior is that if I request the personalized web page the led or led process stops with the http request.... this is an expected behavior I imagine because I cannot respond to a request and at the same time send another ping
In the previous problem the same thing happened but in reverse. the blocking of http was caused by the blocking of the connection process with the mqtt, mysql etc. database that prevented the loading of custom pages and the movement of the loop for more active processes

As for esp8266, it's been a while since we fell a bit short, a shame but the future holds many more esp32

On the other hand, your example as a single file has loaded well in my program as a single (ping.h) but when trying to separate them into .cpp and .h I could not. I have a mistake.
Compilation error: 'esp_ping_handle_t' does not name a type; did you mean 'esp_eth_handle_t'?
surely I'm wrong in some collocation

I think we should open another discussion for the realization of a standard for the creation of the EdgeDriver programs because I am going crazy sometimes with small things that I don't even know where to put them

example:
your pin code at:
src/edge_ping/edge_ping.cpp
src/edge_ping/edge_ping.h

or in examples but define a standard to always use of good practices such as not putting only gpio. and rather use ping_gpio or led_ping

Last but not least.
you have to include it. I love the idea of ​​putting the probe reader processes together with the sending data processes (mqtt, mysql) in one core and the processes (thermostat, relay, ping, even homekit) in another core. It would give us much more game.

The question is how to separate custom pages from processes so that processes such as database requests do not paralyze the loading of custom web pages or menus.

in case the separation is very difficult. there is the possibility to stop processes with !ping as we do now. although for good practices we must stop them anyway

[FRANAIRBUS]

Hello @Hieromon

i like more

static void pingOnSuccess(esp_ping_handle_t hdl, void* args) {
  (void)(hdl);
  (void)(args);

  ping.data.is_ok = true;
  ping.data.timeout = false;
  //Serial.print("success ");

    esp_ping_get_profile(hdl, ESP_PING_PROF_REQUEST, &transmitted, sizeof(transmitted));
    esp_ping_get_profile(hdl, ESP_PING_PROF_REPLY, &received, sizeof(received));
    esp_ping_get_profile(hdl, ESP_PING_PROF_DURATION, &total_time_ms, sizeof(total_time_ms));
    printf("Success %d packets transmitted, %d received, time %dms\n", transmitted, received, total_time_ms);

}

I have put a bool is true for the control but maybe you already had it?

ping.data.timeout = false;

it is right? or is it

ping.data.hasTimeout = false;

recommends me to use
transmitted, received, total_time_ms in typedef struct {} or out?

[Hieromon]

@FRANAIRBUS Ah, I guess I lacked an explanation of my intent. My apologies.
I originally came up with the idea of running EdgeDriver on another core because how can I seamlessly check Internet connectivity in your use case? This is because I was considering the following:

1. Avoid blocking due to waiting for a timeout when connecting to a server.
2. Therefore, you want to eliminate the conditions under which WiFiClient::connect can fail.
3. The method we are currently using to implement this is to detect ping responses in advance.
4. However, Ping also has a delay, and its impact on the event loop of other EdgeDrivers cannot be completely eliminated.

Then it occurred to me, ”If we keep detecting ping responses in the background, we can seamlessly check the Internet connection".

So, I implemented the code for evaluation that I posted at the beginning of this topic. It implemented the counterpart EdgeDriver as a simple GPIO for better logic visibility.
This GPIO EdgeDriver stops blinking the LED when a ping response fails, but does not have a ping session for that purpose. LED blinking is determined based on ping.data.hasTimeout, the value of which is derived by the PING EdgeDriver. And the PING EdgeDriver continues polling the ping response on another core. In other words, by rewriting this part of the code as follows, the validity of the server connection can be detected without waiting:

void LH_MySQL_LOOP() {
...
  if (!ping.data.hasTimeout) {
    HTTPClient http;
    http.begin(wifiClient, mysql.data.serverNameMySQL);
    ...
  }
...
}

And if the above strategy works, EdgeUnified could provide an API for EdgeDriver to run on another core, allowing users to delegate xTaskCreatePinnedToCore related processing to Edge.

The above is my intention. With this in mind, below is my response to your question.

so you mean that autoconnect and edgedriver can work from core 1 or core 0 with a single option or API? each independent?

Yes, I think it is possible. Of course, the entire EdgeUnified structure would need to be revised.
However, ESP-IDF's TCP/IP stack does not strictly support multitasking. It is thread-safe but merely event-driven and blocks incoming tasks. So, the HTTP I/O of AutoConnect and the HTTP I/O of EdgeDriver collide on the same port. It may be possible to avoid the conflict by separating the ports, but I have not verified this. PING session polling is seamless because it is ICMP.

the behavior is that if I request the personalized web page the led or led process stops with the http request.... this is an expected behavior I imagine because I cannot respond to a request and at the same time send another ping
In the previous problem the same thing happened but in reverse. the blocking of http was caused by the blocking of the connection process with the mqtt, mysql etc. database that prevented the loading of custom pages and the movement of the loop for more active processes

Migrating the LED GPIO EdgeDriver to a different core may avoid the phenomenon of blocking blinking. Similarly, sensor dropouts could be prevented. However, verification is required.

I think we should open another discussion for the realization of a standard for the creation of the EdgeDriver programs because I am going crazy sometimes with small things that I don't even know where to put them

Yes, that's right. Essentially I have to document the description of the EdgeDriver programming model. But I can not yet follow up on that.
Please post the individual PING EdgeDriver code in a separate topic, consisting of edge_ping.cpp and edge_ping.h? I will try to modify that code in the posted topic.

The question is how to separate custom pages from processes so that processes such as database requests do not paralyze the loading of custom web pages or menus.

After all, AutoConnect is a RequestHandler of the WebServer class. So I believe it can be separated from other physical I/O.
The question is how to make it available as a simple API to users who expect it from the EdgeUnified library, but I believe that EdgeUnified can internally separate the execution core of AutoConnect and EdgeDriver. I will look into this a bit more.
Most importantly, the EdgeDriver running on another core forms another event loop from the EdgeUnified running on the core. We need something like a parent instance that calls EdgeDriver::process, which runs in a different core from EdgeUnified. It is similar to a fork() in Unix. I will have to try it to see if I can implement it successfully.

[Hieromon]

@FRANAIRBUS Regarding the following question you posted:

I have put a bool is true for the control but maybe you already had it?

ping.data.timeout = false;

it is right? or is it

ping.data.hasTimeout = false;

The ESP-IDF Ping service is itself a session. Refs: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/icmp_echo.html#create-a-new-ping-session
The count of pings treated as a session, the interval, and timeout control are all handled by esp_ping.

pingOnSuccess is called each time a PING response is detected. To determine Internet transparency at the fastest possible time, it should be determined that the ping is established only after the first ping is responded to. It does not make sense if a PING session has more than two transmissions or a timeout greater than the GPIO sampling rate. It is no different than WiFiClient:connect waiting for a timeout.

Therefore, the following code is the best way to check Internet transparency in the fastest possible time. If you are measuring the success rate of a ping, it makes no sense to poll the ping. If even one ping fails, it is sufficient to cancel the transmission of that outgoing turn. Otherwise it will be slower than WiFiClient::connect with an adjusted timeout value.
In this example, the optimal ping count is 1, interval is 0, and timeout is 1000 milliseconds as GPIO blinking period which is turn-off to turn-off.

static void pingOnSuccess(esp_ping_handle_t hdl, void* args) {
  (void)(hdl);
  (void)(args);
  ping.data.hasTimeout = false;
}

[FRANAIRBUS]

@Hieromon
ok then that's correct, I didn't realize your mistake until creating "ping.data.is_ok = true;"
I mean that you had an error in your example with that value
error here. #4

Migrating the LED GPIO EdgeDriver to a different core may avoid the phenomenon of blocking blinking. Similarly, sensor dropouts could be prevented. However, verification is required.

I don't mind the blinking of the LED so much, I like the idea that it indicates operation.
I care more about the ability to change autoconnect to not block http requests.

don't worry, I'm already studying the use of cores in esp32 because they were new concepts for me, as well as the use of the semaphore that I must learn to use to avoid that the serial impressions do not join me.

12:01:15.261 -> Ping 8.8.8.8 - Starting Process SQLC
12:01:15.397 -> Success 1 packets transmitted, 1 received, time 134ms
12:01:16.274 -> Ping 8.8.8.8 - Success 1 packets transmitted, 1 received, time 32ms
12:01:16.357 -> Complex Data Inserted.

static void pingOnSuccess(esp_ping_handle_t hdl, void* args) {
  (void)(hdl);
  (void)(args);

  //ping.data.is_ok = true;
  ping.data.hasTimeout = false; // ping.data.timeout = false; //error example
  //Serial.print("success ");

    esp_ping_get_profile(hdl, ESP_PING_PROF_REQUEST, &transmitted, sizeof(transmitted));
    esp_ping_get_profile(hdl, ESP_PING_PROF_REPLY, &received, sizeof(received));
    esp_ping_get_profile(hdl, ESP_PING_PROF_DURATION, &total_time_ms, sizeof(total_time_ms));
    printf("Success %d packets transmitted, %d received, time %dms\n", transmitted, received, total_time_ms);

    xSemaphoreGive(pingSession);  ??????????????

}

**

the last line avoids the overlap of the serial output??????

**

I understand that (transmitted, received, total_time_ms) better inside the structure

ping.data.packets
ping.data.received
ping.data.total_time_ms

ok I understand that it is not worth polling the ping response time because it would slow it down, so in my first example I only sent a packet, I find the response time interesting to have as a value for other functions or better understand the cases of use of a client or check the fiber optic or adsl line of the physical client, in case the ping to (dns google) is greater than 500ms then timeout for (Mysql connect) should not be configured to 500ms otherwise it will never send data

but even if I try it later I would like to know if the last line
xSemaphoreGive(pingSession); ??????????????
would avoid overlapping Serial.Print();

[FRANAIRBUS]

hello @Hieromon

I need to convert the String of ip of the menu in uint32_t
could you help me on that? I can't find the formula and I've tried countless possibilities and I can't find it

menu.data.PING_IP.c_str();

Thanks a lot

[Hieromon]

@FRANAIRBUS

I mean that you had an error in your example with that value
error here. #4

Thank you for finding my mistake. In order not to get lost when reviewing that code at a later date, the correct code for Ping EdgeDriver is:

static void pingOnSuccess(esp_ping_handle_t hdl, void* args) {
  (void)(hdl);
  (void)(args);
  // ping.data.timeout = false;  // It is wrong.
  ping.data.hasTimeout = false;
  Serial.println("success");
}

static void pingOnTimeout(esp_ping_handle_t hdl, void* args) {
  (void)(hdl);
  (void)(args);
  // ping.data.timeout = true; // It is wrong.
  ping.data.hasTimeout = true;
  Serial.println("failed");
}

[Hieromon]

@FRANAIRBUS

I need to convert the String of ip of the menu in uint32_t

They are all operations using the IPAddress class member functions provided by the ESP32 or ESP8266 Arduino core. The codes to interconvert the type of IP address instance held by the IPAddress class are as follows:

String to IPAddress:

IPAddress ip;
ip.fromString("8.8.8.8");

IPAddress to uint32_t

IPAddress ip(8, 8, 8, 8);
uint32_t ipAddress = (uint32_t)ip;

Convert IP address in String format to uint32_t

String ipStr("8.8.8.8");
IPAddress ip;
ip.fromString(ipStr);
uint32_t ipAddress = (uint32_t)ip;

or reduction coded

String ipStr("8.8.8.8");
uint32_t ipAddress = [&ipStr]() -> uint32_t { IPAddress _ip; _ip.fromString(ipStr); return (uint32_t)_ip; }();

[FRANAIRBUS]

hello
Thank you very much, without your confirmation I would never have succeeded.
i think he was actually putting it in a bad spot

In the end I think it will be better to create a configuration page for this driver (ping) with its
ip:
timeout:
etc....
and call it from my ac_menu with a button to avoid having to modify the code of this driver once I add it to the repository.

{
      "name": "PING_SET",
      "type": "ACSubmit",
      "value": "PING SETTING",
      "uri": "/ping_setting"
    },

instead of including this line because if the user does not have it defined it will not work

void startPing() {
  Serial.println("Starting PING");
    
  IPAddress ip;
  ip.fromString(menu.data.PING_IP);
  ping.data.destination = (uint32_t)ip;

  vSemaphoreCreateBinary(pingSession);
  xTaskCreatePinnedToCore(pollingByPing, "pollingByPing", 4096, NULL, 1, &taskPing, CONFIG_ARDUINO_RUNNING_CORE != pdPASS);
}

Or use some kind of definition

#define PING_DESTINATION_IP
#if PING_DESTINATION_IP
  //IPAddress ip;
  //ip.fromString(menu.data.PING_IP);
  //uint32_t PING_DESTINATION = (uint32_t)ip;
  uint32_t PING_DESTINATION = 0x04040808UL;  // 8.8.8.8
#else
  static const uint32_t PING_DESTINATION = 0x08080808UL;  // 8.8.8.8
#endif

[Hieromon]

My personal opinion is that it is better to isolate such system configuration settings from normal user access. Because they contain contents related to system behavior guarantees, it's generally better to bind configuration files at build time than to access them in the UI via a web browser.

However, since your requirement seems to be to deploy by distributing just one binary file, you may not be able to simply adopt this method.

[FRANAIRBUS]

@Hieromon Thank you very much for giving me your opinion.

However, since your requirement seems to be to deploy by distributing just one binary file, you may not be able to simply adopt this method.

There is another possibility that would be to load .json files with the parameters in the OTA update or by receiving .bin files with different configurations.

the problem in this case is that the ip configuration. ping should be able to be changed remotely or on the physical client because some clients will use local databases and others remote databases.

for a closed product then you should not even be able to change the server address or the GPIO configuration or almost any type of data in the form of configuration by web browser.

is that what you mean?

[FRANAIRBUS]

@Hieromon

Or do you mean the first case?

if we create an Edge Driver called ping
then the Edge Driver installation manual (ping)
it will be
insert to main:

#include <AutoConnect.h>
#include "EdgeUnified.h"

#include "LH_PING.h"
#define PING_DESTINATION_IP (8.8.8.8)

void setup() {
delay(1000);
Serial.begin(115200);

LH_PING_SETUP();

server.on("/", handleRoot);
portal.begin();


if (portal.begin()) {   
    Edge.attach(ping);
  }
}

void loop() {

  Edge.process(portal);
  mDNSUpdate();
  portal.handleClient();
}

LH_PING_SETUP(); It's because I'm inserting all the ........ inside the .cpp file itself

example:

void LH_SQLC_SETUP(){
  sqlc.autoRestore(true);
  sqlc.serializer(sqlcSerialize, sqlcDeserialize, 512);
  Edge.attach(sqlc);
  Edge.join(FPSTR(SETTINGS_SQLC), auxSQLCSetting);
  Edge.join(FPSTR(START_SQLC), auxSQLCStart);
  Edge.join(FPSTR(CLEAR_SQLC), auxSQLCClear);
  Edge.join(FPSTR(STOP_SQLC), auxSQLCStop);
  }

Another thing is that you want to integrate "ping" within the "EdgeUnified.h" library itself.

that in that we are waiting for your great work...