Originally Posted by mattcooksey1
Is there a [permanent downside to using one with too few amps? Or just the obvious one that the phone won't charge fast (if at all)
No, it will just be less current. In some cases this may mean the phone won't charge at all (for example, a low current power adapter while you are using a hotspot would most likely still drain your battery slowly). Low current won't cause damage but your phone may take longer to be able to turn on when it fully depletes and has to recharge.
Real world example: my phone ran out of battery while I was in my car and I tried to use a cigarette port charger and then immediately power it back on as I had done numerous times at home with the A/C adapter. Unfortunately the car adapter was lower current so it took about 10 minutes for my phone to charge back to a point where it would power on. During this time I was blindly driving around haha.