Spike - OpenSearch Observability plugin and Wazuh metrics assessment

[asteriscos]

Description

For the next major release of Wazuh, we want to incorporate metrics and traces about the different components of Wazuh in the dashboard.

To achieve this, we want to leverage the OpenSearch Observability plugin, as it provides a framework to work with metrics and traces. This framework is works with OpenTelemetry, which will be used in other Wazuh components.

The goal of this issue is to identify the capabilities and restrictions of the OpenSearch Observability plugin to generate Wazuh metrics and traceability reports. Within the observability plugin lies the Notebooks application which allows the enhancement of standard dashboards with code snippets, live visualizations, and narrative text.
These Notebooks can be used to generate complex reports.

We need to:

• determine if we can use the observability plugin it to manage the metrics and traces of Wazuh.
• evaluate and design the events format, as well as dashboards and visualizations covered by the plugin.
• evaluate the capabilities of the notebooks plugin to explore and report Wazuh observability metrics

References:

• Observability plugin documentation
• Notebooks documentation
• OpenSearch and OpenTelemetry
• OpenTelemetry documentation
• https://medium.com/@erikstrm_11611/demystify-opentelemetry-with-opensearch-as-a-single-plane-of-glass-919f884eb568
• Observability plugin documentation
• Notebooks documentation

Functional Requirements

• Generate PDF reports: The user must be able to generate PDF reports of engine and agent comms metrics.
• Initial threat detection and posture status: Threat detection and posture status will be regularly sent to users via email based on Wazuh dashboard initial startup configuration.

Implementation Restrictions

• Use OpenSearch features: Ensure we use as many native features as possible to achieve the requirements.
• Pre-configured reports: We must be able to pre-configure a set of reports in the initial setup of the application.
• Stateless reports: The generated reports should be stateless to make the docker deployment easier.

Plan

• Analysis
  • Prepare the dev environment to have the Observability plugin
  • Detect restrictions and features to be developed in wazuh-dashboard-plugins
  • Identify RBAC permissions to restrict operations in the Observability plugin
  • Identify events format requirements and limitations
  • Identify a way to have initial dashboards in Wazuh dashboards initial setup
• PoC
  • Generate mock metrics using the open telemetry protocol
  • Create a dashboard in the Notebooks app using the mocked metrics
  • Generate PDF reports

[jbiset]

Update 2024-07-03

The code of the Observability plugin is investigated and analyzed focusing on the idea of using the Notebook
Below is a rendering flowchart to get to the Notebooks CRUD page.

Something to note that may be interesting is that they use DashboardContainerByValueRenderer, similar to the current rendering of Dashboards

[jbiset]

Update 2024-07-04

Notebook rendering tracking is deepened.
Below is the flow and format of the Input of the DashboardContainerByValueRenderer that ends up being used in a Notebook when it is a visualization.

Flowchart from the Notebook main

OutputBody VISUALIZATION code case

case 'VISUALIZATION':
        let from = moment(visInput?.timeRange?.from).format(dateFormat);
        let to = moment(visInput?.timeRange?.to).format(dateFormat);
        from = from === 'Invalid date' ? visInput.timeRange.from : from;
        to = to === 'Invalid date' ? visInput.timeRange.to : to;
        return (
          <>
            <EuiText size="s" style={{ marginLeft: 9 }}>
              {`${from} - ${to}`}
            </EuiText>
            <DashboardContainerByValueRenderer
              key={key}
              input={visInput}
              onInputUpdated={setVisInput}
            />
          </>
        );

visInput

{
    "viewMode": "view",
    "panels": {
        "1": {
            "gridData": {
                "x": 0,
                "y": 0,
                "w": 48,
                "h": 20,
                "i": "1"
            },
            "type": "visualization",
            "explicitInput": {
                "id": "1",
                "savedObjectId": "c6182e90-3a46-11ef-9824-2bce77daa33d"
            }
        }
    },
    "isFullScreenMode": false,
    "filters": [],
    "useMargins": false,
    "id": "ie5b34eb1-3a46-11ef-b2f0-61dbd60e329b",
    "visSavedObjId": "c6182e90-3a46-11ef-9824-2bce77daa33d",
    "timeRange": {
        "to": "2024-07-04T20:49:10.429Z",
        "from": "2024-06-04T20:49:10.429Z"
    },
    "title": "embed_viz_ie5b34eb1-3a46-11ef-b2f0-61dbd60e329b",
    "query": {
        "query": "",
        "language": "lucene"
    },
    "refreshConfig": {
        "pause": true,
        "value": 15
    }
}

Note

Although in this case inputVis uses a savedObjectId, in theory it should also support the definition of a visualization as currently used in the different dashboards.