User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /search/ Disallow: /survey/ Disallow: /sharedcontent/ Disallow: /management/workforce/compendium/viewLog.cfm Disallow: /management/workforce/compendium/_runverity.cfm # Some crawlers support this. The value is the number of seconds between requests. Crawl-delay: 10 ####### extended standards ######## # I doubt any crawlers support these "extended standards" directives. # maximum rate is one page every 10 seconds Request-rate: 1/10 # only visit between ~11:00 (1500=15:00 UTC/GMT) and 16:00 (2000=20:00 UTC/GMT) Visit-time: 1500-2000