from my experience as a diver, I can say that it is not so much about the duration in water but about the depth and the pressure. The deeper you go into water, the higher is the pressure. For every 10 meter, the pressure doubles. From 1 bar to 2 bar and so on. What this means, is, if you go beyond the depth for which the phone is specified and tested, the protective covers might be too weak to hold steady against the pressure of the water around it. Then they'll give in and bulge inside, thus water comes in -> phone's dead!
This may not mean that if you go 1,1m that the phone will die immediately. It could withstand 2 m, maybe even 3 or 5. But I wouldn't bet or try it out. It may survive these depths, but only for a very short time.
To conclude, I would assume that going to swim for 1 hour in up to 1 m should be fine. But going down to 3m for 5 minutes - not so good idea.