I am having the same problem. Also a nexus 5 with custom rom. Maybe this is a nexus issue?

Which kernel are you running on the phone.