I noticed that too - The screen get brighter in a matter of nanoseconds but when I am in a dark environment it takes a minute or two and sometimes it never turns the brightness down. Hope they'll fix that with an update or something soon.
Anyway, what I did to go past that is turning down the brightness and then setting it back to auto brightness when I am in a dark environment.
Current device: Black Samsung Galaxy S5 GM-900F
Previous: Black mist Samsung Galaxy S4 i9500 and Pebble blue Samsung Galaxy S3 I9300.