Well in my experience it certainly isn't just you.
I got the Zagg because the ATT store had it. Then discovered the Speck case liked to lift the edges of it. Ordered the ArmorSuit. In the meantime I messed with the S-pen... noticed the marks just like you describe. About 4 days later ArmorSuit showed up, tried and failed (both at ATT store and direct to Zagg) to get refund. Bit the bullet. A couple days later I noticed the same markings with the ArmorSuit. I dealt with it for a few weeks, but finally couldn't take it anymore and ordered a Glas-t. About a week now with the Glas-t and no S-pen marks or scratches. Had I gotten the Glas-t originally I'd have saved myself 40 or so bucks. Live and learn.
Glas-t, while thicker, fits in the three cases I currently have with no issues.
Proud owner of an S4 Active. Not overly proudly on AT&T.