Even the Census Bureau views fathers this way- when fathers are the primary caregiver, the Census considers it a "childcare arrangement". There's a petition to change this: http://www.change.org/petitions/dads-don-t-babysit

I am going to be a father in about 2 months. A friend of mine, who is a father of 3, has responded with this some line when confronted with the "babysitting" comment, and I am already preparing myself for the same.

The sad thing is that he is probably one of the best fathers I know, so it just seems extra tragic to me that people assume the opposite about him.

Oh man, please let me steal that response from you. That's fantastic.

It's all yours!

I'm home with our daughter semi-regularly, and I use the "babysitting" line, albeit somewhat ironically, as every day is babysitting in this context to a greater or lesser extent. Still, my wife's coworkers express surprise that I'm at home alone with the kid and am not calling her every five minutes. You definitely do get a different reaction as a guy alone with kids.

As a non-native speaker, what is the difference and why is the former (I'm guessing) mocking/derogatory?

Babysitting is a term that means paying someone else (usually a young teenager) to look after your children while you go out. To say that a father looking after his child is "babysitting" is to imply that it's not normal and expected for a father to look after his child.