A fort cannot hurt your soldiers (it's just a building for crying out loud!!
An ordinary one, not a Stephen King building!), but your soldiers might not be able to destroy the walls so they might flee from the battle. Use the battle calculator to simulate battles. And read the manual