As we've said before, the number of downvotes is tiny compared to the up votes. Also, we are all human so it is inevitable that there will always be odd or questionable individual votes, so we're talking about a small selection of a small number of votes.
In most cases the aggregate result is usually about right. For example a post that attracts an unusual down vote will usually get a subsequent upvote from a member who thinks "hang on a second, that post is correct". The end result is a neutral outcome.