Conclusion: It's not a phone defect.
I have done some testing, and concluded (at least for me) it is software/hardware combo as said previously (heating up). Since I have the phone set up exactly how I want it (rooted and all), and I really don't want to return it since the phone itself is perfect (no physical defects with buttons, screen, etc), I kept a really good watch on when it does it, and it only does it after I have the screen on for like an hour or so, and the phone itself gets warm/hot. If the temp stays below 95F the brightness stays constant. Only if it gets up in the 100s I start to see it, so I do believe it's auto-dimming to cool off the phone. I can live with that since it doesn't get that hot all the time. Mostly playing games, or heavy browsing on the web using desktop mode, and when it does do it the change is slight. I only really notice it on white screens. I was worried since I thought it was a defect, but that doesn't seem to be the case. I'm super happy with this beast. Best phone I ever owned.
On a related note I do remember my GSII doing it on TW. After I put CM on it I never saw it.
Moto Defy Green Lens - CM 7.2 / Android 2.3.7
Moto Defy Red Lens - CM 7.2 / Android 2.3.7
Nexus One - CM 7.2 / Android 2.3.7
Galaxy S II - CM 10.2 / Android 4.3
HTC Desire HD - Stock GB / Android 2.3.6
Galaxy Note 3 - Stock Rooted JB / Android 4.3