I think so too. Non-violence is considered more than just physical; its the behavior of being non-violent in thought speech and deed, in absolutely every way possible.
Just found something on wiki that may be relevant;
Virodhinī hiṃsā or Self defence - Virodhini hiṃsā is committed for self-defence of self, property, family or country against violent attackers, robbers, or dacoits. A householder tries to avoid hiṃsā at all cost, but in such cases it may be unavoidable and hence should be non-vindictive and kept to barest minimum.