If I plan on using it AND I know I'm going out where there is potentially no USB to charge with, I bring my portable USB battey charger and/or one of my other phones. However, I don't mind not having a phone with me at all times, so if it dies, it dies. Still, I just got it to last 4 days, so if you don't watch videos or play games frequently, charging every three days is unheard of for a smartphone!
The battery's health relies on charging it properly, and I value keeping this up over having it charged when convenient for me. Battery saver kicks in at 20% I think, and with battery saver, you'll get almost another day with light usage, so charging when battery saver turns on is probably really hurting your battery.
I generally unplug my phone when I notice the LED turns green (even if I'm going to bed for the night I'll unplug it). I have never had such amazing battery life with any other phone, but I also have never taken such lengths to improve battery life either.