Streaming ESP32 camera feed to Flask web app usingFastAPI and python-socketio as middleware

[Balaji-Ganesh]

• Python will be like the middleware between both ESP32 and web app.
• In the next iterations, the middleware can be configured to make some image processing.

[Balaji-Ganesh]

Issues faced to implement

• First, getting the feed from ESP32 in async manner.
  • By grace, solved using asyncio.get_event_loop().run_until_complete(get_cam_feed())), with camera websockets created in __init__.py file -- without any middleware in between.

Issues facing to implement

• By using a middleware in between get_cam_feed() and the asyncio.get_eve... call.
  • This middleware is needed to apply image processing algorithms.
  • Finding no way to fix this.

[Balaji-Ganesh]

Few solutions found on stack overflow.

• Why sending a message to a web socket does not yield control to the event loop?
• asyncio gather yielding results as they come in
• How to use asynchronous coroutines like a generator?
• Delegating to async subgenerators

Try figuring out solution from these. Or via any new source.

[Balaji-Ganesh]

An Idea.💡 ..

• Why to communicate only with yield, is there not any other option???

this lead to......

One alternative, seems to be workable..

• The web app too creates a socket communication using socket.io JS library -- Just like how HashIncludeElectronic's way..
• It connects to the python's Flask server, and the exchange happens.
• Hoping that, this also makes the communication like - toggling features, and other too.

Now going to perform experiment on this, and will report the status.

As a next update, instead of having flask's template, can go with React web-app, can use MUI and can design better UI. Any way, that's for later time.

[Balaji-Ganesh]

Could able to achieve till here with the help of these references..

1. Basic Chat Application | Chat Application using Flask, Socket.IO & mongoDB (Part-1) -- Gave the 💡 idea of Real-time Chat application.
2. Sending images - Flask-SocketIO send images
   • Answer_1 - Sending from python
   • Answer-2 - Receiving, decoding and displaying in webpage

Heartful thanks to them.

[Balaji-Ganesh]

• Initial trials failed when tried to experiment based on previous knowledge, but got strucked in loop of failing.

• At last by grace, found a repo by AhmedBhati - video-streaming-with-flask-socketio, who made a repo on the task of streaming webcam data to web using flask and socketIO. Adopted that code, now could able to stream web cam data.

• Thanks to the author of that repo.

• Now restructuring the files as per their need.

• Next, the final task -- testing with ESP32 cam feed.

[Balaji-Ganesh]

• This comment is regarding the immediate above commit.

• As with multiprocessing not working, tried with multithreading.

• Couldn't fix error with multiprocessing, but with multithreading via

  • At last with monkey patching - from Stack Overflow issue Python - Flask-SocketIO send message from thread: not always working, solution worked.

[Balaji-Ganesh]

Error getting with multiprocessing: - Ran out of input error.

Log

PS D:\4-2_sem\MajorProject\SelfDrivingCar> python .\main.py
[DEBUG] main: About to establish communications
[DEBUG] main: Communications established
[DEBUG] main: Starting the application...
 * Restarting with stat
[DEBUG] main: About to establish communications
[DEBUG] main: Communications established
[DEBUG] main: Starting the application...
 * Debugger is active!
 * Debugger PIN: 119-550-104
(15972) wsgi starting up on http://127.0.0.1:5000
(15972) accepted ('127.0.0.1', 53906)
[DEBUG] web: Client connected successfully
127.0.0.1 - - [14/Jun/2023 12:07:52] "GET /socket.io/?EIO=3&transport=polling&t=OYud_hl HTTP/1.1" 200 421 0.008005
(15972) accepted ('127.0.0.1', 53907)
[DEBUG] web: client's ack:  {'msg': 'CONN_ESTABLISHED'}
127.0.0.1 - - [14/Jun/2023 12:07:52] "POST /socket.io/?EIO=3&transport=polling&t=OYud_iR&sid=c930cb0509244948aa51c21bfeff7cf9 HTTP/1.1" 200 219 0.000000
127.0.0.1 - - [14/Jun/2023 12:07:52] "GET /socket.io/?EIO=3&transport=polling&t=OYud_ia&sid=c930cb0509244948aa51c21bfeff7cf9 HTTP/1.1" 200 183 0.000000
[DEBUG] web: client's ack:  {'msg': 'CONN_TERMINATED'}
[DEBUG] web: Client disconnected successfully
127.0.0.1 - - [14/Jun/2023 12:07:53] "GET /socket.io/?EIO=3&transport=websocket&sid=c930cb0509244948aa51c21bfeff7cf9 HTTP/1.1" 200 0 0.719858
(15972) accepted ('127.0.0.1', 53911)
[DEBUG] web: Client connected successfully
127.0.0.1 - - [14/Jun/2023 12:07:58] "GET /socket.io/?EIO=3&transport=polling&t=OYue1Ck HTTP/1.1" 200 421 0.000000
(15972) accepted ('127.0.0.1', 53913)
[DEBUG] web: client's ack:  {'msg': 'CONN_ESTABLISHED'}
127.0.0.1 - - [14/Jun/2023 12:07:59] "POST /socket.io/?EIO=3&transport=polling&t=OYue1Up&sid=618f280ac8fa4e24ba4061fb0beba363 HTTP/1.1" 200 219 0.005010
127.0.0.1 - - [14/Jun/2023 12:07:59] "GET /socket.io/?EIO=3&transport=polling&t=OYue1Ut&sid=618f280ac8fa4e24ba4061fb0beba363 HTTP/1.1" 200 183 0.000000
[DEBUG] web: (stream event) client sent: {'msg': 'start streaming'}
[[[[[[[[[[[[[[[[[[[[[[[[[[[STREAMING BEGINS]]]]]]]]]]]]]]]]]]]]]]]]]]]
[DEBUG] web: Creating proceses to begin streaming
[DEBUG] web: about to start spawned process for streaming
Exception in thread Thread-9 (_handle_event_internal):
Traceback (most recent call last):
  File "C:\Users\balga\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "C:\Users\balga\AppData\Local\Programs\Python\Python310\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\balga\AppData\Local\Programs\Python\Python310\lib\site-packages\socketio\server.py", line 682, in _handle_event_internal
    r = server._trigger_event(data[0], namespace, sid, *data[1:])
  File "C:\Users\balga\AppData\Local\Programs\Python\Python310\lib\site-packages\socketio\server.py", line 711, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "C:\Users\balga\AppData\Local\Programs\Python\Python310\lib\site-packages\flask_socketio\__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "C:\Users\balga\AppData\Local\Programs\Python\Python310\lib\site-packages\flask_socketio\__init__.py", line 713, in _handle_event
    ret = handler(*args)
  File "D:\4-2_sem\MajorProject\SelfDrivingCar\app\middleware\communication\web_communicator.py", line 58, in handle_stream
    camProcess.start()
  File "C:\Users\balga\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "<string>", line 1, in <module>
  File "C:\Users\balga\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\balga\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
[DEBUG] main: Thanks for utilizing the application.

Current Resolution:
As working with multithreading, proceeding further with that.

[Balaji-Ganesh]

This is related to the above immediate commit.
From asyncio 'The future belongs to a different loop than the one specified as the loop argument'.

🗝 Point:

First of all, if you want to run a webserver and also serve websockets, you rather use an ASGI server such as FastAPI (https://fastapi.tiangolo.com/).

Note also that Flask is a WSGI server and will handle every request in a separate thread. If you are using an event-loop, the event-loop is local to the thread. Thus be careful when combining multi-threading and event-loops.

• Figured out that, FastAPI uses Starlette ASGI framework.
• Now need to learn FastAPI and build the application.

[Balaji-Ganesh]

By all grace - now 🌉 bridge between ESP32 and Web-app is built.

• Need to change the stack thought initially.

Current Stack

• ESP32 - C++ with ArduinoWebSocket
• middleware - FastAPI and python-socket
• web - (Plain) Flask and Bootstrap.

Screenshot

Flow used to get the feed in web

Initial Setup

1. Get into the middleware by cd middleware and run uvicorn main:app --reload. This runs on address: http:127.0.0.1:8000. This runs the FastAPI app, which is the main bridge.
2. Now take another terminal instance, go to web folder and run python app.py, this runs the flask app, which could display the web pages. This runs on 127.0.0.2:5000.

Getting the feed in web

1. Establish the communication between ESP32 and middleware as follows:
   • Go to the browser/postman/thunder-client and query as http://127.0.0.1:8000/connection/establish, this outputs {"message":"Connection to ESP32 established."}, if successful.
   • If would like to terminate in case, then use 127.0.0.1:8000/connection/terminate.
   • To have a test feed, query for 127.0.0.1:8000/camera-feed/start this opens a cv2 window showing the feed, this outputs: {"message":"Camera task started."}.
   • If would like to stop use http://127.0.0.1:8000/camera-feed/stop, this outputs: {"message":"Camera task stopped."}. Don't press Esc key on the feed display window, this abruptly quits the feed. For safe quit, use the query way.
   • Note the point that, the address 127.0.0.1, is of the server which we got above.
2. Get the feed on web-app.
   • Open the flask URL : http://127.0.0.2:5000/ and click on Manual Mode button to goto http://127.0.0.2:5000/manual_mode.
   • For debugging purpose, open up the developer tools, to know socketio connection status and server's replies.
   • If the socketio connection is success, console window displays: Connected to server.
   • Now click on the Tweakable Features section, click on ☑ Camera Streaming , this opens a feed in Feed section, like above screenshot.

• That's it.

[Balaji-Ganesh]

Missed to add the resource taken to resolve CORS error. Stackoverflow - Connect A Backend Python FastAPI with or without SocketIO Server with a Frontend React Client

[Balaji-Ganesh]

Inherently also fixed the streaming collision data from ESP32 to web.

• For testing use queries:
  • After connection establishment as in comment -> Streaming ESP32 camera feed to Flask web app usingFastAPI and python-socketio as middleware #1 (comment)
  • Use http://127.0.0.1:8000/collision-distance/start to start the feed. This will print the feed on the middleware's terminal.
  • Use http://127.0.0.1:8000/collision-distance/stop to stop the feed.