Judging by the numerous applications for the Blitz-Jeb tournament, not everyone is relaxing during their opponent's turn. And a large number of participants isn't necessary here.
In terms of time, one XL+U game can take longer than the entire tournament. So, I don't think this is a limiting factor.
Auto-translated
