Configuring Zigbee2MQTT for use with Adaptive Lighting

[th3w1zard1]

Last updated: October 2024

This guide serves to configure Zigbee2MQTT's configuration.yaml for use with Adaptive Lighting.
This guide is meant for a supervised HA install with the use of the Zigbee2MQTT addon. I cannot promise it will be relevant to users not using the HA addon.

See the bottom of this post for a quick copy paste of all required/recommended options.

Please note the device_options in the configurations below if defined in configuration.yaml will set these values for all devices, which you may not want. Everything in device_options are correct config options you can set in devices.yaml to be used for specific devices.

Required options for Home Assistant Addon specifically:

Note: some of these options are already default.

homeassistant:
  discovery_topic: 'homeassistant'
  status_topic: 'homeassistant/status'
mqtt:
  force_disable_retain: false
advanced:
  cache_state: true
  legacy_api: false
device_options:
  disabled: false # default
  homeassistant: true

Required options for Adaptive Lighting specifically

homeassistant:
mqtt:
advanced:
  cache_state_persistent: true
  cache_state_send_on_startup: false
  report: true # default
device_options:
  optimistic: true
  filtered_cache: ["brightness", "rgb_color", "hs_color", "xy_color", "color_temp_kelvin", "color_temp", "color", "color_mode"]

You also must configure your light's power_on_state to match the last known state.

Required Adaptive Lighting options: Why?

cache_state_persistent: true and cache_state_send_on_startup: false

From Zigbee2MQTT docs:

Optional: persist cached state, only used when cache_state: true (default: true)
Optional: send cached state on startup, only used when cache_state_persistent: true (default: true)

We want Zigbee2MQTT to store all of the last known states for restarts etc, but we do not want these sent to Home Assistant on startup. In future updates, Adaptive Lighting will assume whenever brightness/color info is provided in the service data, the light should be set to manual control.

report: true (default)

From Zigbee2MQTT docs:

Optional: Enables report feature (default true), this feature is DEPRECATED since reporting is now setup by default when binding devices. Docs can still be found here: https://github.com/sjorge/zigbee2mqtt.io/blob/4d8f358dade21b2a382fd18aec182b9bda633db1/docs/information/report.md

Adaptive Lighting needs to be able to listen to state change events in Home Assistant in order to properly detect manually controlled events.

filtered_cache

From Zigbee2MQTT docs:

This will ignore the unchanged attributes whenever a state update is pushed to HASS. Needed in future updates when Adaptive Lighting will assume brightness/color info in light.turn_on are manually controlled events.

Additional Options you may need:

retrieve_state: true

From Zigbee2MQTT docs:

(DEPRECATED) Retrieves the state after setting it. Should only be enabled when the reporting feature does not work for this device.

optimistic: true

From Zigbee2MQTT docs:

Publish optimistic state after set, e.g. when a brightness change command succeeds Zigbee2MQTT assumes the brightness of the device changed and will publish this (default true)

For me, this option is broken on my lights which I've reported here. When it's disabled, state changes are never reported to Home Assistant. This needs to be true if you're having that issue.

Both Required and Recommended Options (copy/paste version)

homeassistant:
  legacy_entity_attributes: true
mqtt:
  force_disable_retain: false
advanced:
  cache_state: true
  cache_state_persistent: true
  cache_state_send_on_startup: false
  report: true
  legacy_api: false
device_options:
  disabled: false
  homeassistant: true
  optimistic: true
  filtered_cache: ["brightness", "rgb_color", "hs_color", "xy_color", "color_temp_kelvin", "color_temp", "color", "color_mode"]

Resources:
Configuration | Zigbee2MQTT
Devices and Groups | Zigbee2MQTT
FAQ | Zigbee2MQTT

Further Configuration

Please see this top tier reply further down in this thread.

[fbloemhof]

Nice! I implemented this yesterday, did not have all these settings in place yet.

The report: true setting causes that a restart of Z2M is very slow, it takes ages before the webinterface is available because it seems very busy (re)configuring the reporting for all devices?

I had some weird behavior with unpredictable state changes etc., same as described in #829. Then I found this topic.

[djurny]

As a tip, for reporting, you can set these options per device as well, to make sure not all devices start reporting. Same goes for most device_options, they can be set per device.
Groetjes,

[djcrawleravp]

Is this still working?

[basnijholt]

I'm using zigbee2mqtt without any problems.

[djcrawleravp]

I'm stuck in ZTM version 1.33.1 because, when automating my lights using Node-RED, this issue occurs:

Upon detecting absense, the lights are supposed to enter sleep mode for 30 seconds and then turn off. However,

when presence is detected again, the lights turn on in sleep mode and then switch back to adaptive mode.

In later versions, the lights turn on in sleep mode but fail to transition to adaptive mode (sometimes they succeed after a couple of seconds, sometimes they just stay in sleep mode).

And that issue only occurr on versions greater than 1.33.1

THis is how my config look like:

mqtt:
  base_topic: zigbee2mqtt
  keepalive: 60
  reject_unauthorized: true
  force_disable_retain: false
  version: 4
advanced:
  transmit_power: 5
  homeassistant_legacy_entity_attributes: false
  legacy_api: false
  legacy_availability_payload: false
  log_syslog:
    app_name: Zigbee2MQTT
    eol: /n
    host: localhost
    localhost: localhost
    path: /dev/log
    pid: process.pid
    port: 514
    protocol: udp4
    type: '5424'
  channel: 25
  log_level: error
  cache_state: true
  cache_state_persistent: true
  cache_state_send_on_startup: false
availability: true
device_options:
  disabled: false
  homeassistant: true
  optimistic: true
  filtered_cache: ["brightness", "rgb_color", "hs_color", "xy_color", "color_temp_kelvin", "color_temp", "color", "color_mode"]
homeassistant:
  legacy_entity_attributes: true

[R3Cardoso]

This options don't work any more.
Got my zigbee2mqtt on version 1.40.1-1 and lights only adapt after turning on, not before.

[djcrawleravp]

I just solved my problem, It wasn't Adaptive Lightning.

You can check this issue

[djurny]

Hi there,
My whole setup is based on controlling my lighting with Adaptive Lighting, with zigbee2mqtt being the main light source handler. For me it was not just about setting up zigbee2mqtt with AL, but the lights controlled by zigbee2mqtt required some quirks to make sure that it all worked nicely together.

Make sure the AL interval is big enough to make steps that your lights can handle.

I had my interval set to 30 seconds, but got a lot of spurious manual_control events from Ikea Zigbee bulbs. After some digging into why this is happening it seemed like AL was trying to set a color temperature that did not translate to a mired value difference for my Ikea bulbs. This led AL to believe the lights were manually controlled and AL stopped adapting the lights.
After reading the documentation and increasing the interval to 90 seconds (which leads to a bigger change in color_temp in mireds), the spurious manual_control events have not popped up anymore.

Make sure to stop any ongoing transition if you have Ikea bulbs.

Be sure to read the documentation on how to setup AL for Ikea bulbs (..yeah yeah I know).
In my setup, I use some generic script (per AL entity) to turn lights on/off and dim/undim them. Some other HASS automations deal with dimming AL controlled bulbs when the area is vacant for some time (based on occupancy). When occupancy is detected, AL controlled bulbs are undimmed.
Every now and then, I would see that the Ikea bulbs would not respond to an 'off' or 'undim' command. Based on some HAFO (hacking around and finding out), it seemed that sending a color_temp slightly off from the current (target) value would stop transitions for E14 Ikea bulbs. This seemed to work for a while, but using the guide actually works best.
Make sure that if you turn AL controlled bulbs off, or undim them, to stop any ongoing transactions to make sure your lights react.

Have proper color transitions for your lights.

After starting out with WiZ connected lights, I was used to lights transitioning from off to on with the color temperature already set properly.
This changed with the Zigbee bulbs from both Ikea TRÅDFRI and Philips Hue. The starting color temperature seemed to be whatever it was set to when the bulb was turned off. This is not a biggie if you turn off your bulbs in the late evening and flip them back on in the early morning (both have low color_temps), but this started to annoy me.
The solution from @sjorge worked well (see ref), for all the Zigbee bulbs I had in my setup. Enable the execute_when_off option for color_options for your bulbs and send the color_temp you want before you turn on the light (#1022 (comment)). In my configuration I decided to enable execute_when_off after a bulb is turned off.

zigbee2mqtt's transition and AL's initial_transition and adapt_delay.

I've had some odd off-to-on transition effects on some of my bulbs, where it would transition slowly at first then quicklly to the 'setpoint'.
This was resolved by making sure to either use an initial_transition in AL that is equal to the transition in zigbee2mqtt, or to use an AL adapt_delay that is slightly longer then z2m's transition.
What also seemed to be the case, is that disabling manual_control will result in a change of light settings, with - I think - the transition from the AL switch configuration, whilst interrupting any transition that is taking place.

With the caveats above, my AL works well with zigbee2mqtt and both Ikea and Hue bulbs.

Hope this all helps someone out there.
Groetjes,

Ref Koenkk/zigbee2mqtt#13156 (reply in thread) (@sjorge)
Ref #1022 (comment)

[R3Cardoso]

Thanks for the help, but I wasn't able to resolve anything.
I'm not sure where I need to put the code? should be on all automations?
//snip - repeat: for_each: '{{ state_attr(entity_light, "entity_id") }}' sequence: - service: mqtt.publish data_template: topic: 'zigbee2mqtt/{{state_attr(repeat.item, "friendly_name")}}/set/color_options' payload: | { "execute_if_off": "true" } - service: mqtt.publish data_template: topic: 'zigbee2mqtt/{{state_attr(repeat.item, "friendly_name")}}/set' payload: | { "color_temp": "{{turn_on_quirk_color_temp_mired_current}}", "transition": "0" } //snip

About transition I got 0 on zigbee2mqtt and adapt_delay, on initial_transition got 1 because it helps disguise the change.

[djurny]

Hi there,
I think you are referring to your issue Zigbee2mqtt to fast for adapt?

To 'prime' the color_temp of your lights, which means set them to the wanted color_temp before you turn them on, you would have to make a script in homeassistant. You can read up here on what that entails.

See here for a tested automation that sets the color_temp of zigbee lights that are controlled by adaptive lighting through the mqtt integration.

There is no simpler way to do this a.f.a.i.k.

Hope that helps,
Groetjes,

[R3Cardoso]

Thanks for the explanation, but I don't have the same light name on zigbee2mqtt and HA(this was easy fix) but for AL I got only some entity to control several lights. For exemple got a entity call bedrooms that control all the lights on bed rooms.

And as far I also understand, I would need to have a script to every single light I got. :/ a lot of work.

I will wait for zigbee2mqtt are able to do light Philips Hue box was able to do it. If philips hue box do it, zigbee2mqtt sooner or later will be able too.

[djurny]

To keep this topic to the point, I have posted a comment in your issue here: #996 (comment).

[wise-rice]

I feel like this post need an update for the latest Version of Zigbee2mqtt 2
It removed many of the mentioned (legacy) settings

What is the current set of correct z2m settings to enable a smooth AdaptiveLighting experience?

@RubenKelevra @basnijholt - maybe you could add the necessary setting to the readme page of AdaptiveLighting, so it is visible from within the integration?

[cipheredsyntax]

I agree with @wise-rice. I just updated my Z2M to the latest version and a lot of settings have changed. It would be nice to have an updated guide on how to make Adaptive Lighting work properly with Z2M. I've temporarily uninstalled the integration due to how many issues I'm running into.

Maybe someone else can share a working configuration for AL/Z2M?

[basnijholt]

I personally use Zigbee2MQTT with just default settings. Which legacy settings are in the README that @wise-rice is referring to?

[RubenKelevra]

Pre 2.0 of z2m there were some legacy options, which are now (mostly) removed IIRC. So the options are either ignored or you need to remove them prior to updating to 2.0.

These here are the settings in question:

Koenkk/zigbee2mqtt#24198

[fbloemhof]

I am also running on Z2M with default config and AL is working really good on it.

[cipheredsyntax]

I have spent more hours on trying to get adaptive lighting to work than any other integration in Home Assistant by far. It's pretty frustrating how changing a single thing will break the entire configuration. I don't think I have ever gotten AL to work properly in my house and I cannot figure out why. Some of the things I keep running into are:

• Lights flashing their old color before adapting (even when setting the color temp before turning them on). I am not a fan of setting a delay of 250ms to wait for my light to turn on with the right color.
• Lights randomly flashing when turning them.
• Manual control not working. Lights keep adapting.
• Transitions not working properly. Some lights turn on instantly, some have a smooth fade-in and fade-out.

And like many other people, I use smart IKEA bulbs. I have looked into getting Philips HUE bulbs but I'm reading reports of how many others are having the same issues as me with with HUE bulbs so that's not an option either.

Hence why I asked if people are willing to share how they've set up their configuration to work with AL because at this point I cannot wrap my head around it anymore. The default Z2M configuration does not work by any means. I like the integration a lot but it's just too broken in too many ways (at least for me).

[djurny]

Hi @jncasey,
It does not look pretty but it seems to work:

    enabled_adaptive_lighting_entities: |-
      {%set ns=namespace(entities=[])%}
      {%for _ in integration_entities("adaptive_lighting") |
              select("match", "^switch\.adaptive_lighting_adapt_color_.*") |
              expand |
              selectattr("state", "==", "on") |
              map(attribute="entity_id") |
              list%}
        {%set ns.entities=ns.entities+[_|replace("adapt_color_", "")]%}
      {%endfor%}
      {{ns.entities}}      

Seems that replace() expects and produces strings, so feeding it a list will break down the list into a string.

Will add your color_temp check to the gist later today,

Thanks a lot for the suggestions!
Grt,

[djurny]

Hi @cipheredsyntax,

Also, one of the caveats of your configuration: when turning a light off after you've set it to a different color (temperature) or brightness, it will turn back on with that same previous color (temperature) or brightness if it's within those 5 minutes of the automation updating every light that's off.

The easiest way would be to just increase the frequency of the automation time trigger. This is not a serious solution of course. Adding a trigger for every adaptive lighting light that is turned off might be an option - will have to do some homework there.

Main reason for making the automation was to take away or at least minimize the transition effect observed when turning on any adaptive lighting controlled lights.

Also, to have it work in the most user configuration agnostic way as possible, so anyone should be able to just add it and have it work for them.

If you ask me, without changing adaptive lighting, a script providing "ON" and "OFF" for every adaptive lighting instance would be preferred. But, that cannot be made easily for any configuration out there and would require any light.turn_on/turn_off anywhere to be replaced with that script. (Button on_tap actions in dashboards etc.)

I don't know if you have one readily available, but I tried creating one myself although to no avail as I am too unexperienced in this. I got as far as this:

triggers:
  - platform: state
    entity_id:
        "{{ expand(state_attr('switch.adaptive_lighting_inside', 'configuration').lights) | map(attribute='entity_id') | list }}"
    to: 'off'
actions:
  - variables:
      adaptive_lighting: >-
        {{ integration_entities("adaptive_lighting") |
           select("match", "^switch\.adaptive_lighting_.*") |
           reject("match", "^switch\.adaptive_lighting_adapt_.*") |
           reject("match", "^switch\.adaptive_lighting_sleep_mode_.*") |
           list | first }}
      turned_off_light: "{{ trigger.entity_id }}"

I know that the entity_id trigger doesn't work this way, but hopefully it explains to you what I'm trying to achieve.

I'll have a think about how this can be added to the automation.
Thanks for the suggestion!
Grt,

[djurny]

Hi @cipheredsyntax,
Seems that the state trigger does not accept templates in the entity_id. I thought about making some template sensor that would accept some template to build a list of lights to watch, but according to the documentation these type of template sensors will be evaluated every minute, which makes sense.

For my setup just added the light groups that run adaptive lighting explicitly:

    - trigger: state
      entity_id:
        - light.entrance
        - light.family_room
        - light.living_room
        - light.work_room
        - light.work_room_table
        - light.hallway
        - light.master_bedroom
        - light.sofa_overhead
      to:
        - 'off'

Adding these to the automation as trigger will also run all actions whenever one of my light groups turns off. I do not want this to trigger if any of the lights are controlled outside of the light group on purpose.

Perhaps someone else has a bright idea here to make the automation as generic as possible.

Grt,

[fbloemhof]

Is there something special I need to do to use this?

I checked multiple AL switches but "state_attr(adaptive_lighting_entity, "configuration").lights" is empty (the whole configuration part is emtpy).

Yes, need to enable include_config_in_attributes.

[djurny]

Hi @fbloemhof,
Added some actions to enable include_config_in_attributes.
(Decided to move the gist to a fork of adaptive lighting, feel free to comment/discuss further there.)
Grt,

[djurny]

Hi @wise-rice,

Does it actually send a command while the lamp is in "off" state?

Yes, yes it seems to do just that. You can check the zigbee2mqtt logging (set to info or lower) and see the messages be sent.
Grt,

[R3Cardoso]

About that code you are working on, I got two problems.
1º lights name wtih space gives error.
2º I got two lights that turn on when I run the automation. No sure if others also turn on as they give error.

error 2025-03-20 19:07:05z2m: Entity 'Wall' is unknown
error 2025-03-20 19:07:05z2m: Entity 'Celling' is unknown
info 2025-03-20 19:07:06z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/Garden', payload '{"brightness":222,"color":{"x":0.477,"y":0.4137},"color_mode":"color_temp","color_options":{"execute_if_off":"true"},"color_temp":400,"effect":null,"linkquality":96,"state":"OFF","update":{"installed_version":23,"latest_version":23,"state":"idle"}}'
error 2025-03-20 19:07:06z2m: Entity 'Wall' is unknown
info 2025-03-20 19:07:06z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/Gardenl', payload '{"brightness":222,"color":{"x":0.477,"y":0.4137},"color_mode":"color_temp","color_options":{"execute_if_off":"true"},"color_temp":400,"effect":null,"linkquality":100,"state":"ON","update":{"installed_version":23,"latest_version":23,"state":"idle"}}'
error 2025-03-20 19:07:06z2m: Entity 'Wall' is unknown
error 2025-03-20 19:07:06z2m: Entity 'Celling' is unknown
error 2025-03-20 19:07:06z2m: Entity 'Wall' is unknown
error 2025-03-20 19:07:06z2m: Entity 'Celling' is unknown
error 2025-03-20 19:07:07z2m: Entity 'Wall' is unknown