The outer wall, being unmanned, serves as an excellent shelter for the crossbowmen. I moved about 120 of them right next to the wall, then built a siege tower to let them use the towers. This allowed me to wipe out with minimal losses everything except the ballista in the farthest tower on the hill.