UPDATE: I'm fairly sure I figured out what's happening when the phone slows down.
From the restart, up until the phone goes into deep sleep, it works normally. Once I turn the screen off and put it aside for a minute or two, that's when it slows down. I basically let the screen on for half an hour and ran an app in the background for microphone so the phone wouldn't deep sleep for an hour and the phone was still fast/geekbench was still reporting high scores. As soon as I closed that app down and turned the screen off for a minute, woke the phone back up, it started being slow.
Can anyone confirm this? What can be causing this strange problem? Can I help diagnose this? I hope I'm not the only one with this problem cause I really like this rom otherwise