Yes, this happens when launching in multiplayer mode, but with the second player controlling the game.
So, do I understand correctly that when night falls for me, it's still daytime for the second player, which causes this desynchronization?
Tomorrow I'll try playing with a friend, see how it works, and let you know.