Charging at 3.7 or so means you're only using about half the battery capacity. Overall, it supposedly quadruples your cell recharges, so it doubles cell life (since you're charging at 0.5 capacity).
But that's for optimal cell life, not what the cell can work with--and is designed to work with.
While I generally try to limit usage to 3.7, if I'm out and about and don't want to carry a passel of batteries, they get drained right down until the mod stops working (3.2-ish in my case). The battery is here to serve me, not the other way around. That time, it just has to bear up and deal with it.
As mentioned above, check your specs, but 3.0 is OK on the very vast majority of batteries I've ever seen. 2.5V is OK on many INR batteries. However, at the lowest voltages you should charge your batteries as soon as possible.