Expressions Language support

[fgalan]

Allow calculated expressions in Orion would be a very useful feature. For instance, notify an attribute which is the sum of another two (of numeric type) or the concatenation of another two (of string type).

Ideally, a language like the JEXL (used by IOTAs) would be desirable. Alternatives in C/C++ need to be analized. The following table include several alternatives:

Library	Pros	Cons
TinyExpr	Easy to use	Potentially immature (e.g. no version releasing implemented). Only covers mathematical operations (e.g. concat strings is not supported)
CEP
ECL for LISP expressions
JEXL embedding the V8		Too heavy?
Tcl

(Analysis in progress. More entries will be added to the table as we go. Feedback highly welcomed!)

[mapedraza]

Another alternative is achieve this is to use Common Expression Language

It is available a C++ library implementing the language

Pros: Maintained by Google, available in different languages
Cons: Different language compared to IoTA

[fgalan]

Possible alternatives:

1. Implement an expression language inside CB (as this issue originally proposes)
2. Implement a side proxy (eg. nodejs with JEXL) that will do the calculations. That proxy will be included within the CB container, so from a "black box" perspective, it would be as if the functionality has been included in CB itself.
3. Use IOTA to do the calculations, using a feedback loop to generate the notification.

[mapedraza]

Another possible alternative is using the same JEXL used in IoTAs through the python library
https://github.com/mozilla/pyjexl

Python exposes a C/C++ in order to execute python code inside C code:
https://docs.python.org/3/extending/embedding.html

[fgalan]

Chap GPT prompt

can you show me a C or C++ code that embeds a Python interpreter that in sequence loads de jexl module to run jexl expressions?

Response:

#include <Python.h>

int main(int argc, char *argv[]) {
    // Initialize the Python interpreter
    Py_Initialize();

    // Import the jexl module
    PyObject* jexl_module = PyImport_ImportModule("jexl");

    if (jexl_module != NULL) {
        // The jexl module is successfully imported

        // You can now execute Jexl expressions using the Python API

        // For example, let's create a Jexl expression and evaluate it
        const char* jexl_expression = "2 + 2";
        PyObject* result = PyObject_CallMethod(jexl_module, "eval", "(s)", jexl_expression);

        if (result != NULL) {
            // The expression is successfully evaluated
            printf("Result of Jexl expression: %s\n", PyUnicode_AsUTF8(result));

            // Don't forget to decrement the reference count
            Py_DECREF(result);
        } else {
            // There was an error in evaluating the expression
            PyErr_Print();
        }

        // Decrement the reference count for the jexl module
        Py_DECREF(jexl_module);
    } else {
        // There was an error in importing the jexl module
        PyErr_Print();
    }

    // Finalize the Python interpreter
    Py_Finalize();

    return 0;
}

To compile:

g++ your_program.cpp -o your_program -lpython3.x

(replace 3.x by actual Python version)

Note: Ensure that the Jexl module is installed in your Python environment before running the program (pip install jexl).

[fgalan]

Based on the previous example, considering this code (in Python):

from pyjexl import JEXL

# Create a JEXL engine
jexl_engine = JEXL()

# Define a JEXL expression
expression = 'x + y'

# Create a context with variables
context = {'x': 4, 'y': 11}

# Evaluate the expression using the context
result = jexl_engine.evaluate(expression, context)

# Print the result
print(f'Result: {result}')

Now, the same embedded in C++:

#include <Python.h>

int main(int argc, char *argv[]) {
    // Initialize the Python interpreter
    Py_Initialize();

    // Import the jexl module
    PyObject* jexl_module = PyImport_ImportModule("pyjexl");

    // Create JEXL engine
    PyObject* jexl_engine = PyObject_CallMethod(jexl_module, "JEXL", NULL);
       
    // Create expression and context
    PyObject* expression = Py_BuildValue("s", "x + y");
    PyObject* context = Py_BuildValue("{s:i, s:i}", "x", 4, "y", 11);

    // Call evaluate method
    PyObject* result = PyObject_CallMethod(jexl_engine, "evaluate", "OO", expression, context);

    // Print result 
    PyObject* repr = PyObject_Repr(result);
    const char* result_str = PyUnicode_AsUTF8(repr);
    printf("Result: %s\n", result_str);

    // Free resources
    Py_XDECREF(repr);
    Py_XDECREF(result);
    Py_XDECREF(context);
    Py_XDECREF(expression);
    Py_XDECREF(jexl_engine);
    Py_XDECREF(jexl_module);

    // Finalize the Python interpreter
    Py_Finalize();
    
    return 0;
}

It can be compiled using:

g++ example.cpp -o example -I/usr/include/python3.11/ -lpython3.11

It works as expected:

$ ./example 
Result: 15

[fgalan]

Useful ref: https://docs.python.org/3/c-api/call.html#c.PyObject_CallMethod

[fgalan]

After some brainstorming with @manucarrace and @mrutid

• JEXL calculations could be used in two places:
  • Calculated attributers (upon a attribute updates, some other attributes are calculated based in JEXL in some place, ie. side attributes, metadata, etc.)
  • Enhance the ${...} syntax in custom notifications (which currently only support direct attribute replacement) to support also JEXL expressions (e.g. ${temperature/100}.
• After some discussion it seems the feature 2 is more useful considering the platform e2e, so we are going to prioritize that case

[fgalan]

PR #4511

[fgalan]

In PR #4511 until commit 2cc2424 implementation has been based in Python.

However, after some investigation on the Python multi-thread model (see this post at SOF) JavaScript V8 seems to be a better alternative. We are going to test it.

[fgalan]

First test with V8 standalone program (a hello world example):

// compile: g++ -std=c++14 simplest.cpp -g -o simplest -I/usr/include/v8 -L/path/to/v8/lib -lv8 -lv8_libplatform -pthread

#include <iostream>
#include <libplatform/libplatform.h>
#include <v8.h>

using namespace v8;

int main(int argc, char* argv[]) {
    // Initialize V8.
    V8::InitializeICUDefaultLocation(argv[0]);
    V8::InitializeExternalStartupData(argv[0]);
    std::unique_ptr<Platform> platform = platform::NewDefaultPlatform();
    V8::InitializePlatform(platform.get());
    V8::Initialize();

    // Create a new Isolate and enter it.
    Isolate::CreateParams create_params;
    create_params.array_buffer_allocator = v8::ArrayBuffer::Allocator::NewDefaultAllocator();
    Isolate* isolate = Isolate::New(create_params);
    {
        Isolate::Scope isolate_scope(isolate);
        HandleScope handle_scope(isolate);

        // Create a new context.
        Local<Context> context = Context::New(isolate);

        // Enter the created context for compiling and
        // running the JavaScript code.
        Context::Scope context_scope(context);

        // Create a string containing the JavaScript source code.
        Local<String> source = String::NewFromUtf8(
            isolate, "'Hello, ' + 'world!'",
            NewStringType::kNormal).ToLocalChecked();

        // Compile the source code.
        Local<Script> script = Script::Compile(context, source).ToLocalChecked();

        // Run the script to get the result.
        Local<Value> result = script->Run(context).ToLocalChecked();

        // Convert the result to a string and print it.
        String::Utf8Value utf8(isolate, result);
        printf("%s\n", *utf8);
    }

    // Dispose the isolate and tear down V8.
    isolate->Dispose();
    V8::Dispose();
    V8::ShutdownPlatform();
    return 0;
}

[fgalan]

At the end, we are going to use a libcjexl.a library.

[fgalan]

More transformations added in PR #4561