I am having the same problems on mine. I went into Verizon about it. I figured it was my phone. The guy called the IT department and was able to speak to them. Now, I'm telling you truths here, but we all know that when you're talking to a VZW Rep you get random stuff.
They did some things, really just asked me what I tried to do, and then they were making sure I wasn't rooted and stuff, but in the end said that there is a known issue with the IR Blaster. I'm actually quite disappointed in this. But, they said... here's where you get what you want to out of this... they are releasing a patch that will solve it. I was even given a date of January 15th. Now, I'm not going to be thumping around anything on anything about a Jan 15 update other than telling you that I got the information today and I'm taking it as a very small bit of information and I'm not going to bet the farm on this date and information, so you do what you feel you need to.
WHAT WE DO NEED!!! Get your phones out, point it at a TV and shoot that bastard until you find out if your phone works. Please, we need you guys to pitch in and find out if this is actually all Note 4's, some random ones or just a couple. I don't want to be stuck in a week past my 14 day window getting some jerks phone that bought it to play with, took it clubbing, spilled a drink on it and turned it in and now I'm stuck with that piece of _____. I want a new phone if I need a new phone as mine is 8 days old. So, please please get out there and help us.