HOW TO: Write values from Home Assistant controls to ebusd (Idiot's guide (by an idiot))

[JonesGW-MGD]

Like many I suspect, I am delighted with HA and ebusd integration - my recently acquired v5 adapter now gives me a wealth of data from my newly installed Vaillant boiler and SensoCOMFORT (VRC720rf) controller.

I don’t really have any need to set parameters on the system from HA but I was curious to try it in a limited fashion. I did assiduously read the ebusd wiki, GitHub discussion pages and HA forums but I found I ended up very confused. This is no doubt because, while I am technically-minded, I am not an IT guy let alone a dev guy and I really got stumped with what many will probably find the easiest of things - like HA payload / template formatting etc.

For ages it seemed like I could update MQTT topics only for them to reset when I did a ebusctl read for example - generally because I was formatting the value wrong. But with a lot of perseverance, trial and error (mostly error), the aforementioned resource pages, and ChatGPT (which is surprisingly knowledgable about ebusd despite a Jan 2022 context for version 3.5), I succeeded!

I thought it might help others to provide a step-by-step ‘idiots guide’. Maybe this helps people that are in a similar position to me?

My caveat is this - this may not be the only, or the best or even a good way of doing it - but it worked at least for me. I’d obviously welcome any pointers about what I could have done better in this thread.

My initial assumption is you have something similar in set-up to me (at least in end result - the specific kit/tools don’t really matter):

I have a RaspPi with the ebusd v5 Adapter USB-connected. I have Mosquitto MQTT, and the ebusd add-on installed. (I also have ‘Advanced SSH and Web Terminal’, and ‘Studio Code Server’ add-ons - you will need something similar but you don't have to have these local to the home assistant instance of course, I just found it easier in my case). I found it helpful to have MQTT Explorer on my workstation (not Pi) - you can probably do without but I found it helpful to see the MQTT topics from outside of HA.

I git-cloned the configuration files so they were installed locally and have my mqtt-hassio.cfg file local too. (I had spent some time editing entries for the files that loaded - mostly bai.308523.inc in my case - to correct some values to work for my boiler which is an 835 Ecotec IQ Exclusive).

(To git clone locally, submit this from your config folder:  

git clone https://github.com/john30/ebusd-configuration.git

This will put the files in /config/ebusd-configuration. Then you can edit your ebusd config path parameter in the addon to read: /config/ebusd-configuration/latest/en/ ... Or /de/ of course.)

My set up works well and reliably - HA auto-discovered lots of sensors and I then subsequently built out a pretty dashboard using Mushroom UI with the main sensors I wanted to monitor, keeping the HA-generated dashboard for debugging etc.

Please don’t try the following if you’re not 100% happy with the general basic working of the ebusd/ebus devices/HA set up - this will only end up in an bottom-less rabbit hole otherwise!

Throughout the following steps, if you’ve got an MQTT Explorer client session going, it really helps to watch the changes.

You can clearly skip some of these steps if you think the granular testing is a bit tedious (I.e. testing the raw value, template value, service call etc.)

Step 1: Make sure that the config is able to write values not just read.

Writing values is (sensibly) not enabled by default.

You can do this by editing mqtt-hassio.cfg and remming out line 118 and enabling line 120:

# include only messages having the specified direction ("r", "w", "u", or "uw". partial match, alternatives and wildcard supported).
# HA integration: writable messages excluded for now
# filter-direction = r|u
# HA integration: for including writable messages, use this line or overwrite with '--mqttvar=filter-direction=r|u|^w'.
filter-direction = r|u|^w

But you might find it more helpful to do with an override on the ebusd add-on config - using

--mqttvar=filter-direction=r|u|^w

on the ‘custom command line options’ line. You can then confirm when you next start the ebusd add-on from the logs that you should be able to write successfully.

(Obviously you need to re-start the ebusd add-on)

For reference, my ebusd configuration from the add-on page looks like this:

(Ignore --mtqqvar="filter-name=" parameter - this is a misunderstanding on my part. -if you want to avoid the mqtt-hassio.cfg file from filtering using command line override, you should enter instead -- mqttvar=filter-name= <- no quotes! I have subsequently corrected this.

Step 2: Decide your candidate parameters to change

First, establish from the .csv config files that the element is writable, not all are - and some you think might be, aren't, like diagnostic code 5 - FlowTempDesired because that's a result of other elements that are writable.

For example I choose ‘MaintenanceDate’ and ‘Hc1HeatCurve’. These are both controller values - I.e. you’ll find them (or something similar for your config) in ’15’ type files. I choose them because the first is ‘totally harmless’ if I screwed up and the second was something I might usefully want to change from HA because that changes the flow temperature.

(On my Dashboard, I have a sensor for ‘Hc1HeatCurve’, I don’t display ‘MaintenanceDate’ as that’s boring so also why it’s useful to have MQTT Explorer.)

Henceforth, my steps are going to use these parameters - you can obviously experiment and substitute your own.

Step 3: Do a test ‘write’

Helpful here to have a couple of browser tabs open - First a terminal session onto your HA instance, second, HA>Developer Tools>Services

Go to your terminal session, type:

 docker exec -it `docker ps | grep ebusd | awk '{print $1}'` /bin/bash

This gets you into the ebusd container. (I appreciate there are other ways and maybe better/safer ways of doing this - like using puTTY from other threads I’ve read, but this worked for me, and I ‘stuck to the script’ so wasn’t too concerned I’d mess up)

Type:

ebusctl read MaintenanceDate

And note the default date. (Mine was the Vaillant default of 01.01.2019 but yours may have been set properly by your installer - so worth noting it down)

Go to your HA>Developer Tools>Services session, and enter the following (it’s easier for me to show the yaml representation but you can do this in the UI):

service: mqtt.publish
data:
  qos: "1"
  retain: true
  topic: ebusd/basv/MaintenanceDate/set
  payload: 05.10.2024

Note that the topic is suffixed with ‘/set’ — This stumped me for a long time but maybe because I’m an idiot…. (Obviously choose whatever date you want as payload of course but do not elaborate the format of the payload with quotes or double quotes- or use ‘Payload template’ - I spent a lot of trial and ERROR time here.

Go back to your ebusd docker instance session. Repeat the read: ebusctl read MaintenanceDate

If it doesn’t display the date in the Service request - STOP - and backtrack.

Assuming it’s updated the entry you’re on the right track! And safe to proceed…. At this point it might be worth reverting the MaintenanceDate back to it’s original state.

Step 4: Second test ‘write’

MaintenanceDate is useless as an editable parameter, so let’s affect the Heat Curve which will be useful/critical/dangerous! (For context, if you have a heat source - boiler, heat pump, etc. that is set for what in the UK at least we call ‘Weather Compensation’ then the heat source determines what the flow temperature is (i.e. Hc output/ aka diagnostic code d.40 flow temp) by applying a curve where the flow temp is the bisect of the curve and the outside temp.). Changing the heat curve will change the efficiency of your heat source (well, or rather the output of it…. )

Same process from the ebusd container: ebusctl read Hc1HeatCurve, and note value.

HA>Developer Tools>Service:

service: mqtt.publish
data:
  qos: "1"
  retain: true
  topic: ebusd/basv/Hc1HeatCurve/set
  payload: 1.65

Go back to your ebusd docker instance session. Repeat the read: ebusctl read Hc1HeatCurve

(Obviously in case it’s not obvious - messing with your heat curve will definitely affect your boiler / heat source performance - my recommendation is alter it by no more than five hundredth while you experiment - I.e. from say 1.55 to 1.60 (or say 0.50 to 0.55 if you’re lucky enough to have a heat pump rather than a boiler!))

Assuming the value changed, you’re good to go.

Step 5: Build an HA control

So now we can demonstrably control the value we’re interested in, we need to build a UI control that allows us to alter it. (And also an Automation)

Step 5A: Create an Input_number

(Using an input_number isn't the only way to do this of course you can pick your poison)

I did this by editing my configuration.yaml:

input_number:
    ebusd_set_heatcurve:
      name: "Set Heat Curve"
      min: 01
      max: 03
      step: 0.05
      mode: slider

My Min/Max make sense for a gas boiler - they won’t for a heat pump - adjust accordingly.

If you ‘Check configuration’ under Development Tools (always do this after editing any yaml file) and then reload your yaml (Restart>Quick reload)- you should see the following appear under Settings>Devices & Services>Helpers



Step 5B: Create an HA UI control based on this Input_number

Now I have an entity ‘Set Heat Curve’, I can create a control in HA.

Because I’m using Mushroom UI, YMMV, but I created a card on my Dashboard that is basically a slider. Just pick an appropriate control/card in Lovelace-UI if you’re not using Mushroom

- type: custom:mushroom-number-card
    entity: input_number.ebusd_set_heatcurve
    name: Set Heat Curve
    display_mode: slider

Step 5C: Test the control

Go to Developer Tools>Template and paste this:


{{ states('input_number.ebusd_set_heatcurve') | float}}

(I Don't think you actually need to pipe the states value to float - as I think input_number is a number not a string, but at this point I'd really not got my head around HA default values/templates etc.)

Change the slider on the dashboard you published it to and observe the result on state change. Assuming it changes correctly, proceed with Developer Tools>Service and paste this:

service: mqtt.publish
data:
  qos: "1"
  retain: true
  topic: ebusd/basv/Hc1HeatCurve/set
  payload: "{{ states('input_number.ebusd_set_heatcurve') | float}}"

Go back to your ebusd docker instance session. Repeat the read: ebusctl read Hc1HeatCurve

Assuming the values are what you expect, proceed…

Step 6: Create a HA automation to change value

We now need an automation that replicates the ebusd ‘change on state’ action scenario we proved in step 5:

Create an automation: Settings>Automations & Scenes>Create Automation>Create New Automation> add ‘state’ trigger on your input_number entity, add action ‘call service’ mqtt:Publish, select the right topic, and paste payload:

"{{ states('input_number.ebusd_set_heatcurve') | float}}"   

I found the editor complains about this saying you can't use the UI for a template (and this might the root cause of my next issue) but that’s fine - save that. I also found when I looked at the automations.yaml it appears to add quotes/escaped characters - the result is that if you run this it will send the value as a string to the topic and not a value so this won’t update.

This is where someone might explain in simple terms what I’m doing wrong - I don't know why the exact same payload line that you can test in the service tools saves differently in the automations editor - quite possibly user-error on my part so just be aware.... but anyway, I manually amended the payload by trial and error to work (by amending the entry in automations.yaml).

Your automations.yaml file should have an entry that looks something like this:

- id: '1706869489422'
  alias: Set_Heat_Curve
  description: publish value of slider to change heatcurve
  trigger:
  - platform: state
    entity_id:
    - input_number.ebusd_set_heatcurve
  condition: []
  action:
  - service: mqtt.publish
    data:
      qos: 0
      retain: false
      topic: ebusd/basv/Hc1HeatCurve/set
      payload: "{{ states('input_number.ebusd_set_heatcurve') | float}}"
  mode: single

Step 7: Change the slider, and ebusd updates the value - done!

(and you can obviously verify on the physical device(s) or by confirming with ebusctl read Hc1HeatCurve)

Just as a final note - I have both a sensor and a slider - these could obviously be combined into a single control of course.

For reference, an extract of my boiler dashboard looks this:

Final final - 'exit' from the ebusd docker instance returns you to the home assistant instance

[lildadou]

I'm an IT guy and despite this, it took me a few hours to :

1. learn that it was necessary to publish on /set
2. to publish the payload not as JSON but as CSV.

(I came across your post only a few minutes after discovering all this. I'm experiencing my greatest frustration; the Nutella jar is going to suffer)

[GuyHarg]

I have an almost identical Vaillant + eBusd setup, and am an idiot as well. So far this has been super helpful thanks. When you saved and updated mqtt-hassio.cfg locally where did you save it and what adjustment did you make to the eBusd add-on config setup. Your MQTT integrated CFG file path looks unchanged?

[JonesGW-MGD]

Hi there - I'm away from my setup currently, but I don't think I did amend mqtt-hassio.cfg, instead I overrode on the custom command line options of the plug-in. (My 'Step 1', above). I think..... can check in a day or so.

[GuyHarg]

Thx that's what it looked like. You also had --mqttvar="filter-name=" in the custom command line what was that for please? Thx in advance.

[JonesGW-MGD]

There is a line in mqtt-hassio.cfg that filters out some of the sensor/message types by default. If you run with this, some of your ebusd data will not auto-discover in HA. How much that matters depends on what you are trying to acquire I guess, but to remove the filter, either comment out the line in the file, or override it on the command line as I have done.

[GuyHarg]

Thanks again. Weirdly when I added the filter removal command some of my existing data eg DisplayedOutsideTemp became unavailable in my basic HA dashboard so I guess I have a lot of fiddling around to get this right. Running MQTT Explorer as you suggest is very helpful. In MQTT Exp Global/updatecheck I get:

"version 23.3 available, broadcast.csv: different version available, memory.csv: different version available, vaillant/08.hmu.csv: different version available, vaillant/15.basv.csv: different version available, vaillant/broadcast.csv: different version available, vaillant/errors.inc: different version available, vaillant/general.csv: different version available, vaillant/hcmode.inc: different version available"

I assume since your setup localises a lot of these files once I go live I fork away from them permanently? Is there a risk some of these updates are critical and I have to go back scratch with a new GitHub clone?

Also Vaillant sent me this https://opensource.myvaillant.com do you think something in here is of any help?

[JonesGW-MGD]

Re mqttvar; that's odd, maybe I've got the format wrong, will check.

(It's possible it should be --mqttvar=filter-name= without the quotes. I may have misread the section under the add-on's documentation tab>top tips!

"If some polled readings do not show up in Home Assistant it might be because mqtt-hassio.cfg is configured to filter them out. Try setting to mqttvar to "filter-name=" and this will remove any filters so you can debug the issue".

Alternatively, comment out of the file and host locally - think it's line 106:

filter-name = status|temp|yield|count|energy|power|runtime|hours|starts|mode|curve|^load$|^party$

Yes, local setup means you're forked, you can periodically update them - it depends if you've modified any of the files in any significant way. For my boiler configuration, I found changing certain values better suited my specific boiler (that is, I got some messages populating that otherwise had said undefined for the default boiler file I'm using.) so I"D probably be quite cautious about what I synced.

The configuration files don't change that often - there is quite a lengthy fork, add, merge process with quite a lot of testing - looks like you have a heat pump? The 08.hmu.csv last had a commit 4 months ago.

That link is just Vaillant's way of complying with open source requirements that if any code uses open source, you need to provide a listing and link to where the source is obtainable - in this case it's a tonne of open source stuff in your VR920 internet gateway. It's the same with any internet router, skybox or other set top box or literally any bit of current code will leverage some persons/team's work for free! , etc. etc. So no, of no use!

[GuyHarg]

Yes removing the quotes worked and has added 40+ more data names. These OEMs like Vaillant really just do the bare minimum required it's super frustrating. I don't need a lot of variables just the basics - I'm running aroTHERM+ 5kW + sensoCOMFORT RF + 2x Sunamp heat batteries for hot water + MVHR based heating and cooling controlled with an Ecobee smart thermostat so I want to know what the heat pump is doing at any given time and hope to automate some seasonal factors. Once I've worked out what to modify in the csv etc files I'll have to keep track to easily update things if required. Again, many thanks for the help.

[JonesGW-MGD]

Thanks for the call-out on the filter-name option - I had changed my start-up and have now changed the notes, above.

Good luck in your endeavours! Your set-up sounds interesting! (MVHR is quite rare in the UK where I'm at - so interested in your locale!) The Sunamp heat batteries appeal to me - are they basically water-to-water thermal stores?

[GuyHarg]

I retrofitted our Oxford mid terrace to passivhaus enerPHit certification it's been quite the journey. The Sunamps get charged by either 65º ASHP water or via an immersion heater in some circumstances and then provide mains instant hot water. Work well and good for smaller terraces. The Sunamp product is very slave-like so I've had to set up some third party Homekit based automation for those as well. Like you I'm quite tech oriented but not a coder (well not since the 70s / 80s haha!).

[GuyHarg]

A bit stuck on the config files and all the data streaming into my MQTT explorer. Blanking the filter-name I think is giving me everything on the ebus coming into MQTT explorer under ebusd and $SYS headers.

Ignoring $SYS, ebusd subheaders are:
global
hmu
base
vr_71
broadcast

However one morning I woke to a whole lot more subheaders under ebusd:
scan.08
scan.15
scan.26
scan.76
scan.f6

and then to confuse this idiot even more I got a new major header called homeassistant with about 250 config= lines like

config = { "unique_id":"ebusd_global_signal", "device":{ "identifiers":"ebusd", "manufacturer":"ebusd.eu", "name":"ebusd", "sw_version":"23.2", "suggested_area":"Heating" }, "state_topic":"ebusd/global/signal", "name":"ebusd signal", "device_class":"connectivity", "payload_on":"true", "payload_off":"false" }

And now after a reboot the homeassistant header has not appeared for a few days.

Firstly, am I overcomplicating this as I don't want to write to my aroTHERM+ at all - should I just be looking at the devices/entities being recognised by HA automatically or do I need to check what I want from MQTT explorer and modify my config files to get it in HA? At this point the data coming into the entities looks like it's missing things I need tbh.

Secondly what might have happened when the homeassistant header turned up in MQTT?

Apologies in advance for the nube questions.

[JonesGW-MGD]

I probably can't give you an explanation as I simply accepted what I saw in both MQTT and HA!

But for comparison, I also have named topics such as 'broadcast', 'bai' (for my boiler, instead of 'hmu'), etc. and then separate scan. topics as you have. In HA, the scan. topics are basically auto-discovered and create a device each with typically three sensors (HA>Settings>Devices & Services>MQTT>Devices) - these simply identify the ID, HW and SW levels, whereas under the say 'ebusd hmu' device, you see a sensor for all/most message definitions.

'Fraid I didn't really question that or note when stuff started to appear. I do know that if I stop and start Explorer you can see that some topics take time to appear as presumably they are not regularly polled?

In terms of guidance as to what to do/focus on. - I can offer a combination of what I did and didn't do but would do this time(!):

1. Don't worry about the fact you may have hundreds of entities/sensors that you aren't interested in for now - I'm not sure they are creating any significant overhead and you can tidy this up later
2. Instead, now you have an end-to-end proof point of data flowing from ebus into HA via ebusd and MQTT, concentrate on identifying the useful data you want from within HA, including those that might be missing
3. I found it helpful to create a new dashboard view and copy the stuff I was interested in to that, while keeping the default 'Overview' dashboard much like a 'scratch-pad'. (I happened to use the 'Mushroom-ui' extension for this dashboard as it looked aesthetically nicer to me than the default 'Lovelace-ui'. but that's obviously not critical. Took me about 2-3 hours to get to grips with Mushroom using various YouTube vids).
4. Once you have what you need (or the extent of what's available) you can then do tidy-up if you want by honing the original files and maybe resetting to read only etc.

Just on the 08.hmu.csv file. I don' t know much about heat pumps and ebusd, but back in Feb 24, quite a large number of users started to complain on this forum that a recent update to the configuration files had caused erratic behaviour to their Arotherm Pluses - this was after a commit that attempted to add the reading of a lot more values from 'LiveMonitor'. It seemed the way to read LiveMonitor required a write first - and this subsequently caused issues on the bus, causing the HP to 'crash' or cycle. See #1205. and referred posts.

The updates were retro-ed out I believe. But it may mean that data you expect to be able to 'read' may not currently be accessible - you can probably determine more by following various HP related threads on the two forums (ebusd and ebusd configuration files).

Hope this helps a little.