Admins should only be looking at who downvotes posts when there is some sort of history of vote-abuse happening. Somebody voicing their opinion with their votes is a perfectly valid use of the upvote/downvote system.

PTB