I use downvoting for comments that are incorrect, or are in violation of the guidelines. I'll also flag when appropriate, but sometimes flagging feels overly aggressive.
People are right to have both downvoted and flagged your comment - downvoting for disagreement is a perfectly normal part of HN and has always been and complaining about votes is something the guidelines ask you not to do (same with generic community-directed swipes, comparisons to reddit, etc)
