Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Last I checked, robots.txt excludes all crawlers from at least several major classifications, including housing listings (/hhh):

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /cgi-secure
    Disallow: /forums
    Disallow: /search
    Disallow: /res/
    Disallow: /post
    Disallow: /email.friend
    Disallow: /eaf
    Disallow: /reply
    Disallow: /?flagCode
    Disallow: /ccc
    Disallow: /hhh
    Disallow: /sss
    Disallow: /bbb
    Disallow: /ggg
    Disallow: /jjj
    Disallow: /*rss$
http://www.craigslist.org/robots.txt

'jjj' is jobs, 'ggg' is gigs, 'bbb' is services, 'sss' is for sale, 'ccc' is community, 'res' is resumes.



Right...but not apartment listings (aap)

http://newyork.craigslist.org/aap/ http://newyork.craigslist.org/roo/ http://newyork.craigslist.org/sub/

http://www.google.com/search?client=safari&rls=en&q=...


Point. I mistakenly assumed /aaa covered all of housing.

Looks like the same principle applies to a number of other subcategories. So while overview indices aren't spiderable, subindices are.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: