Why is it not possible to connect to all game servers?

We tried it – and I’m creating the server – he can’t connect to me, and he’s creating one – I can’t connect to him. And if he connects to someone else, does it work normally, or when I connect to someone or they connect to me? It works, but not always – sometimes I can’t join others either, but I always thought it was just players declining my request. But I want to specifically play with this player, but we can’t manage it. And does this bug happen often?!