Turns out the neodymium magnets I had near my desk actually latched onto my phone and magnetized the NFC antenna. I tested this by removing the cover and putting the magnet near the antenna and it sticks. :P Without the cover on, the compass worked much better.
I've mostly demagnetized my NFC antenna now by doing some magnet sweeps and leaving it near the heater, but I've learned to keep the magnets away.
If anyone is having problems with the compass overall accuracy, removing the back cover may help.