It's not a problem; it's one of the most interesting parts of the game and also the shortest.
Added after 8 minutes
If a timer were added to battles between players, the game would at least not become worse. The timer would be for both players, so everyone would be on equal footing.