[Bug]: app_abfallplus_de returns names with numbers instead of the types

[OnSive]

I Have A Problem With:

A specific source

What's Your Problem

https://community.home-assistant.io/t/waste-collection-schedule-framework/186492/542

The attribute of my sensor do only contain names with numbers.

Source (if relevant)

app_abfallplus_de / ZAK Abfall App

Logs

N/A

Relevant Configuration

Sensor (you can use the following wizard: custom_components/waste_collection_schedule/waste_collection_schedule/wizard/app_abfallplus_de.py)

waste_collection_schedule:
    sources:
    - name: app_abfallplus_de
      args:
        app_id: de.k4systems.abfallappzak
        city: Fischen
        bezirk: Ortsteile

Checklist Source Error

• Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
• Checked that the website of your service provider is still working
• Tested my attributes on the service provider website (if possible)
• I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

Checklist Sensor Error

• Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

Required

• I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
• I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.

[specularius]

Same here with these settings

waste_collection_schedule:
    sources:
    - name: app_abfallplus_de
       args:
          app_id: de.k4systems.lkemmendingen
          city: Biederbach
          strasse: Alle Straßen

[5ila5]

That's quite a neat bug.
Did some initial digging:
The app does return the correct data, but the reverse engineered way returns this super weird data (I do not know who's names these are and why they are returned here).

Debugging this is pretty annoying, as it's often not quite clear what went wrong exactly. I already did some trial and error but without success. I will take a deeper look in the coming days.

[OnSive]

@5ila5 can you give me a hint where to start debugging? Couldn't find a starting point as first time user of ha and waste collection schedule.

Btw. I hope that those are not real people as there would be way to many less then 18 year old people in a waste collector database imo...

[5ila5]

Generally debugging sources:

1. Cloning the reposetory
2. modifing the files (sources are placed in custom_components/waste_collection_schedule/waste_collection_schedule/source/ some sources have a corresponding Service script (app_abfallplus_de has one), which is called by the source script. it should be located at custom_components/waste_collection_schedule/waste_collection_schedule/service/
3. You can test the modified source using the test script documented here

app_abfallplus_de

To compare the calls made by the source with the request of the real app, you need: an android (vm?) or IOS (don't know hot this works) device where you can inspect the traffic made by the phone (I use Http Toolkit to inspect the traffic made by my android VM running inside Android Studio)

Some changes like the User-Angent and client ID were made with an IPhone (as far as I can tell). But I tried to change them (Andoird has 2 different User-Agents for different requests and client_id must be 5 separate sections instead of one long integer string) and it did not solve anything, so I think the error is probably somewhere else.

[OnSive]

@5ila5 thanks that's helpful, maybe i can find a hint to the bug. Got the dev env running 👍

[5ila5]

I put together the requests that are made by the app.
https://gist.github.com/5ila5/205672118616e30a2eaccbc89db42df4
Running this returns the XML with the names instead of the bin types, the app gets another XML (with the correct bin types), I do not find the difference between these requests and those made by the app.

[HenryLoenwind]

BTW, I suggest you complain to your local council (that is responsible for this public service) about this data being gated behind a private company that actively tries to prevent citizens from accessing it.

[Tigra16v]

The same Problem with these settings:

waste_collection_schedule:
sources:
- name: app_abfallplus_de
args:
app_id: de.k4systems.abfallinfocw
city: Egenhausen
strasse: Alle Straßen

two days ago everything was still going.

[Fireblade900rr]

Also the same for my city.

waste_collection_schedule:
sources:
- name: app_abfallplus_de
args:
app_id: de.k4systems.abfallinfoapp
city: Weilerswist

[coarsy]

And here the same too. Waste collection schedulde is not running anymore:

waste_collection_schedule:
sources:
- name: app_abfallplus_de
args:
app_id: de.k4systems.zakb
city: Heppenheim
strasse: Schubertweg
hnr: 6

[MatzzeF]

@HenryLoenwind pretty sure that this is a issue with the HA Plugin and not the data provider. The apps from the waste companies runs pretty fine here and shows the right data.

[x4N70pHyLL]

Same issue for Rastatt (Abfallplus). Getting Names with Numbers

[mngsps]

My sensors are all "unknown" since 29th of june...

waste_collection_schedule:
    sources:
    - name: app_abfallplus_de
      args:
        app_id: de.k4systems.abfallappbh
        city: Au
        strasse: Alle Straßen

[roninf]

Same here for:

- name: app_abfallplus_de args: app_id: de.k4systems.abfallappbh

Looks to me like birthdays....

[boun]

@ReneNulschDE in my region de.k4systems.zawdw I did not get strange names, but calendar was empty (sometimes a strange date for a single event). I changed the code before the self.get_collections() call to set self._client to an id, that I sniffed from the app and sure enough, the calendar was filled up again. So there seems to be a problem in the set-up phase. I tested your changes but they did not fix the problem.

[garionth]

My sensors are all "unknown" since 29th of june...

waste_collection_schedule:
    sources:
    - name: app_abfallplus_de
      args:
        app_id: de.k4systems.abfallappbh
        city: Au
        strasse: Alle Straßen

In your case i would suggest to check the log.
I had the same (sensors going to unknown) a couple of days ago. In the logs i could see that my provider (also app_abfallplus_de) changed the required parameters for the config.
The log said "Alle Straßen" is not a valid street name. And then gave a list of all the streetnames of my city.

btw thanks who ever is responsible for putting that return into the log. Very helpfull.

It was like in your case

city: real_city_name
strasse: Alle Straßen (all streets)

and is now
city: real_city_name
strasse: real_street_name
hsnr: real_number

[garionth]

I don't know if it helps, but from the looks of things on my end, abfallplus is still changing things atm.

2 days ago i did not have the name problem like the OP, but fantasy dates for the pickup. (between 60 and 200+ days in the future) on my sensors.

Now they are all unknown.

And as i said in the above post. 2-3 weeks ago the config for my city street changed. But this of course, could be a "normal" update on their side.

[mngsps]

My sensors are all "unknown" since 29th of june...

waste_collection_schedule:
    sources:
    - name: app_abfallplus_de
      args:
        app_id: de.k4systems.abfallappbh
        city: Au
        strasse: Alle Straßen

In your case i would suggest to check the log. I had the same (sensors going to unknown) a couple of days ago. In the logs i could see that my provider (also app_abfallplus_de) changed the required parameters for the config. The log said "Alle Straßen" is not a valid street name. And then gave a list of all the streetnames of my city.

btw thanks who ever is responsible for putting that return into the log. Very helpfull.

It was like in your case

city: real_city_name strasse: Alle Straßen (all streets)

and is now city: real_city_name strasse: real_street_name hsnr: real_number

Thank you for the input!
In their App its still "Alle Straßen" (all streets) in my case.
Furthermore, nothing noticable in the logs:

2024-06-30 23:07:14.640 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration waste_collection_schedule which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-06-30 23:07:22.509 INFO (MainThread) [homeassistant.setup] Setting up waste_collection_schedule
2024-06-30 23:07:32.575 WARNING (MainThread) [homeassistant.setup] Setup of waste_collection_schedule is taking over 10 seconds.
2024-06-30 23:07:44.093 INFO (MainThread) [homeassistant.setup] Setup of domain waste_collection_schedule took 21.58 seconds
2024-06-30 23:07:44.095 INFO (MainThread) [homeassistant.components.sensor] Setting up waste_collection_schedule.sensor
2024-06-30 23:07:44.116 INFO (MainThread) [homeassistant.components.sensor] Setting up waste_collection_schedule.sensor
2024-06-30 23:07:44.119 INFO (MainThread) [homeassistant.components.sensor] Setting up waste_collection_schedule.sensor
2024-06-30 23:07:44.131 INFO (MainThread) [homeassistant.components.sensor] Setting up waste_collection_schedule.sensor
2024-06-30 23:07:55.478 INFO (MainThread) [homeassistant.components.calendar] Setting up waste_collection_schedule.calendar

[AKGitH]

Yesterday I wrote an e-mail to [email protected] and explained the API problem getting person names back.

But till now I got no feedback, but it looks that someone is playing arround with the data, today I get "unkown" back.
Have a look to the data update date/time:

[Tigra16v]

Same for me. After I tried it with street and house number and then undid it, I no longer have any names with numbers, just "unknown".

[anderl78]

For me too.... I get only "unknown"

[Fireblade900rr]

Crazy, but after a restart for other reasons its working as expected, witout changes.

waste_collection_schedule:
sources:

• name: app_abfallplus_de
  args:
  app_id: de.k4systems.abfallinfoapp
  city: Weilerswist

[Tigra16v]

I found the solution. At least in my case it works. I noticed that the sensor calendar.apps_by_abfall was present multiple times. calendar.apps_by_abfall_1 calendar.apps_by_abfall_2 and so on. After I deleted all but one and restarted home assistant everything works again.

(In Settings -> Devices and services -> Entities)

[roninf]

My calendar is empty meanwhile without any entry...

[anderl78]

Crazy, but after a restart for other reasons its working as expected, witout changes.

waste_collection_schedule: sources:

• name: app_abfallplus_de
  args:
  app_id: de.k4systems.abfallinfoapp
  city: Weilerswist

Doesn't work for me... after reboot still "unknow"

[anderl78]

I found the solution. At least in my case it works. I noticed that the sensor calendar.apps_by_abfall was present multiple times. calendar.apps_by_abfall_1 calendar.apps_by_abfall_2 and so on. After I deleted all but one and restarted home assistant everything works again.

(In Settings -> Devices and services -> Entities)

Thank you. I got the clones too. Deleted them too, followed by a reboot - but still the same problem : state of my sensors is still "unknown"

[mngsps]

I found the solution. At least in my case it works. I noticed that the sensor calendar.apps_by_abfall was present multiple times. calendar.apps_by_abfall_1 calendar.apps_by_abfall_2 and so on. After I deleted all but one and restarted home assistant everything works again.
(In Settings -> Devices and services -> Entities)

Thank you. I got the clones too. Deleted them too, followed by a reboot - but still the same problem : state of my sensors is still "unknown"

Same for me. No change after removing duplicates...

[anderl78]

Oh, I just tested the abfallplus-app on my phone - it works without any problem... I even changed street and Number to see if it really gets new data...changed back, got my data back. Even checked there website - works well, all data is present there.

[Tigra16v]

I found the solution. At least in my case it works. I noticed that the sensor calendar.apps_by_abfall was present multiple times. calendar.apps_by_abfall_1 calendar.apps_by_abfall_2 and so on. After I deleted all but one and restarted home assistant everything works again.

(In Settings -> Devices and services -> Entities)

Thank you. I got the clones too. Deleted them too, followed by a reboot - but still the same problem : state of my sensors is still "unknown"

The sensor that is left must not have a number at the end. Otherwise it must be renamed. It must be called calendar.apps_by_abfall.

[mngsps]

The sensor that is left must not have a number at the end. Otherwise it must be renamed. It must be called calendar.apps_by_abfall.

I did exactly this. No change, still "unknown"

[Tigra16v]

The sensor that is left must not have a number at the end. Otherwise it must be renamed. It must be called calendar.apps_by_abfall.

I did exactly this. No change, still "unknown"

I'm sorry. It helped me. Then there are probably other reasons. I wish you continued luck

[ReneNulschDE]

I spend some more hours on this topic. Here is a code that can be checked. (The code is not final but a starting point.)

Changes:

• Custom request header structure for the first request
• Not more than one request per second

Testing source app_abfallplus_de ...
  found 84 entries for de.k4systems.abfallappnf Ahrenviöl alle Straßen
  found 26 entries for de.albagroup.app Braunschweig Hauptstraße 7A  
  found 70 entries for de.k4systems.bonnorange Auf dem Hügel
  found 139 entries for de.ucom.abfallavr Brühl Habichtstr. 4A
  found 45 entries for de.k4systems.abfallappwug Bergen hauptstr. 1
  found 78 entries for de.k4systems.abfallappcux Wurster Nordseeküste Aakweg Alle Hausnummern
  found 40 entries for de.abfallwecker Mutzschen, Am Lindigt 1
  found 48 entries for de.k4systems.leipziglk Brandis Brandis
  found 46 entries for de.k4systems.leipziglk Machern Machern
  found 64 entries for de.k4systems.lkgoettingen, Abfallwirtschaft Altkreis Göttingen,  Adelebsen, Alle Straßen

manual testing (abfallappzak - Fischen, lkemmendingen - Biederbach, abfallinfoapp - Weilerswist)

[anderl78]

I spend some more hours on this topic. Here is a code that can be checked. (The code is not final but a starting point.)

Changes:

• Custom request header structure for the first request
• Not more than one request per second

Testing source app_abfallplus_de ...
  found 84 entries for de.k4systems.abfallappnf Ahrenviöl alle Straßen
  found 26 entries for de.albagroup.app Braunschweig Hauptstraße 7A  
  found 70 entries for de.k4systems.bonnorange Auf dem Hügel
  found 139 entries for de.ucom.abfallavr Brühl Habichtstr. 4A
  found 45 entries for de.k4systems.abfallappwug Bergen hauptstr. 1
  found 78 entries for de.k4systems.abfallappcux Wurster Nordseeküste Aakweg Alle Hausnummern
  found 40 entries for de.abfallwecker Mutzschen, Am Lindigt 1
  found 48 entries for de.k4systems.leipziglk Brandis Brandis
  found 46 entries for de.k4systems.leipziglk Machern Machern
  found 64 entries for de.k4systems.lkgoettingen, Abfallwirtschaft Altkreis Göttingen,  Adelebsen, Alle Straßen

manual testing (abfallappzak - Fischen, lkemmendingen - Biederbach, abfallinfoapp - Weilerswist)

Hello!

Thank you very much! Seems to fix the Problem for me!

[Lackii]

I spend some more hours on this topic. Here is a code that can be checked. (The code is not final but a starting point.)

Thanks a lot, this fixes the issue also for me. Before i have had the strange entries with names and since monday an empty schedule. Clearing the double calendar.apps_by_abfall entries did not changed anything.

[AKGitH]

After HA restart, I got this errors:

Moving back to the original file, no errors showed, but still get "unkown".

[mngsps]

I spend some more hours on this topic. Here is a code that can be checked. (The code is not final but a starting point.)

Changes:

• Custom request header structure for the first request
• Not more than one request per second

Solved the problem in my case as well! Thank you!!

[ReneNulschDE]

@AKGitH : Your screenshot does not show the error message. Please share the log entry and your config.

[AKGitH]

Hi "ReneNulschDE",
I have downloaded the py-file again and now it works.
Great Job!

For your information: I use:
-name: app_abfallplus_de
-app_id: de.ucom.abfallavr

Many thanks!

[anderl78]

Btw, in case you need the information, Im using:
name: app_abfallplus_de
app_id: de.k4systems.zawdw

[lorddagobert]

@ReneNulschDE

Solved the problem also in my case as well! Thank you!!

name: app_abfallplus_de
app_id: de.idcontor.abfalllu

Many thanks again

[tobibachert]

I spend some more hours on this topic. Here is a code that can be checked. (The code is not final but a starting point.)

Changes:

• Custom request header structure for the first request
• Not more than one request per second

Testing source app_abfallplus_de ...
  found 84 entries for de.k4systems.abfallappnf Ahrenviöl alle Straßen
  found 26 entries for de.albagroup.app Braunschweig Hauptstraße 7A  
  found 70 entries for de.k4systems.bonnorange Auf dem Hügel
  found 139 entries for de.ucom.abfallavr Brühl Habichtstr. 4A
  found 45 entries for de.k4systems.abfallappwug Bergen hauptstr. 1
  found 78 entries for de.k4systems.abfallappcux Wurster Nordseeküste Aakweg Alle Hausnummern
  found 40 entries for de.abfallwecker Mutzschen, Am Lindigt 1
  found 48 entries for de.k4systems.leipziglk Brandis Brandis
  found 46 entries for de.k4systems.leipziglk Machern Machern
  found 64 entries for de.k4systems.lkgoettingen, Abfallwirtschaft Altkreis Göttingen,  Adelebsen, Alle Straßen

manual testing (abfallappzak - Fischen, lkemmendingen - Biederbach, abfallinfoapp - Weilerswist)

THAT solved it for me.
Thanks a lot @ReneNulschDE

[Tigra16v]

After the last core update I have the problem again. (sensor status unknown) Since I'm still a complete beginner and my English isn't great, I honestly don't understand what exactly solved the problem for you. Can someone please explain to me what exactly I need to do?

Or is there anything wrong with explaining it to me in German? As far as I can see, everyone here is German, right?

[coarsy]

@Tigra16v You have to replace the content of the file AppAbfallplusDe.py with this: 6051222

[Tigra16v]

ok. and how exactly do I do that? I can't copy the text in the link (in the github app).

[thoduh]

Press the Link, then Press the tree Points:

Press "View File"
Press "Copy raw file"
Paste in Homeassistant...

[thoduh]

@Tigra16v You have to replace the content of the file AppAbfallplusDe.py with this: 6051222

worked. thank you!

[Tigra16v]

OK. I got it working. But via the browser and not in the app. But now nothing works. The sensors are no longer available and I get the following error message

Logger: homeassistant.setup
Quelle: setup.py:485
Erstmals aufgetreten: 17:22:14 (5 Vorkommnisse)
Zuletzt protokolliert: 17:22:14

Unable to prepare setup for platform 'waste_collection_schedule.sensor': Unable to set up component.

[chma73]

@Tigra16v I think your AppAbfallplusDe.py is not correct, are you sure that you did the copy and replace correct.

[chma73]

@ReneNulschDE thanks for your effort and support, your update works fine for me

[Tigra16v]

At least I'm not aware of what I did wrong. I also get the following error message:Logger: homeassistant.setup
Quelle: setup.py:404
Erstmals aufgetreten: 18:25:35 (1 Vorkommnisse)
Zuletzt protokolliert: 18:25:35

Error during setup of component waste_collection_schedule
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/setup.py", line 404, in _async_setup_component
result = await task
^^^^^^^^^^
File "/config/custom_components/waste_collection_schedule/init.py", line 113, in async_setup
await hass.async_add_executor_job(
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/waste_collection_schedule/init.py", line 201, in add_source_shell
new_shell = SourceShell.create(
^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py", line 204, in create
source_module = importlib.import_module(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 995, in exec_module
File "", line 488, in _call_with_frames_removed
File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/app_abfallplus_de.py", line 1, in
import waste_collection_schedule.service.AppAbfallplusDe as AppAbfallplusDe
File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/service/AppAbfallplusDe.py", line 541
for a in soup.find_all(
^
SyntaxError: '(' was never closed

[Tigra16v]

@Tigra16v bist du noch in einem anderen Forum aktiv, bei dem man dir direkt schreiben kann?

Leider nicht. Aber bin offen für Vorschläge

[ReneNulschDE]

@Tigra16v : I strongly recommend to wait until the code has been merged into the official repo, especially if you have not yet gained so much experience. (My code example was just for testing for "experienced users" and to validate my tests).

If you can't/won't wait, click on this link and copy the complete content into the service/AppAbfallplusDe.py. Then restart HA.

Translated with DeepL.com (free version)

@Tigra16v : Ich empfehle dringend zu warten bis der Code in das offizielle Repo eingeflossen ist, speziell wenn Du noch nicht so viel Erfahrung gesammelt hast. (Mein Code Beispiel war nur zum Testen für "erfahrene User" und zur Validierung meiner Arbeit.)

Wenn Du nicht warten kannst/willst, klicke auf diesen Link und kopiere den kompletten Inhalt in die service/AppAbfallplusDe.py. Starte danach HA neu.

[Tigra16v]

It's done. I've got it working now. I copied the file into the directory again using the command line. Now everything works. Great 👍 Thank you for the help!