That's interesting. Gabriel, does DuckDuckGo's crawler have a distinct user agen... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

Matt_Cutts on Nov 22, 2012 | parent | context | favorite | on: Google competitor DuckDuckGo says it's getting shu...

That's interesting.

Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact