Also if you load due to connection problems or because of a mistake made or any situation where you dont need to remember anything, then you get to much time. And these situations happen pretty often.
On the other hand, its a problem that time runs if you actually need time to remember the map.
A solution could be that you can pause the time if both players press a certain button or you can choose a certain time that is added in case of loading.