Speed wasn't just meant for combat; the Tower is generally a slow castle in terms of tempo – very often, until the end of week 2, everyone is running at a speed of 6, with a maximum native speed of 11, and even that requires upgrading the Djinns, which isn't always done. This leads to slower movement on the map; many castles switch to 7-9 speed by week 1, or even start with 9 speed.
After all, there's the option to upgrade Gargoyles to 11 speed, so everyone can run at 9 speed.