Because :
1) you didn't use the Battle Calculator to simulate your chances before launching the attack
2) if you can't break the walls before your morale drops below 50, there is a chance for you to flee the battle
It is generally assumed, that in absence of professional soldirs, you still have to break into the castle, defended by the refugee villagers, if you want to pillage its ressources
Have fun,
