I use a cloth pouch for my phone and haven't got any scratches since I got it (launch)
Like crzycrkr said, sand and minerals/stones in general are kryptonite for Gorilla Glass, whereas coins/keys/metal are no problem.
So, in my opinion, if you keep the phone in a pouch, you shouldn't face any serious problems.
Did my post help you? Please hit Thanks! Use the search function!