Bots

Information for Webmasters

What is Linkdex?

Linkdex is an enter­prise-class plat­form that com­bines team man­age­ment soft­ware with search engine opti­miza­tion tools. Some­times we crawl web­sites to under­stand back­link pro­files or gath­er infor­ma­tion for our users so they can improve their web­sites and SEO strate­gies.

Why have you crawled my website?

Unlike some crawlers, our bots don’t ran­dom­ly trawl the Inter­net for infor­ma­tion. We only crawl pages which our users deem rel­e­vant.

If your site has been crawled, our bots will appear in your analytics/server logs with the fol­low­ing details:

  • linkdexbot v2.0 checks for links between Web pages. This helps us build a fresh, accu­rate pic­ture of links between web­sites.
    Mozilla/5.0 (com­pat­i­ble; linkdexbot/2.0; +https://www.linkdex.com/en-us/about/bots/)
  • linkdexbot v2.1 is our more gen­er­al site crawler that helps mar­keters detect tech­ni­cal issues with a website’s pages, and works in 2 ver­sions: a stan­dard Web Search and a Mobile Search con­fig­u­ra­tion.
    Mozilla/5.0 (com­pat­i­ble; linkdexbot/2.1; +https://www.linkdex.com/en-us/about/bots/)
    Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (com­pat­i­ble; linkdexbot-mobile/2.1; +https://www.linkdex.com/en-us/about/bots/)
  • linkdexbot v2.2 checks the con­tent of pages that are ranked by search engines. This helps us iden­ti­fy who are the influ­en­tial web­sites, brands and authors writ­ing about dif­fer­ent top­ics.
    Mozilla/5.0 (com­pat­i­ble; linkdexbot/2.2; +https://www.linkdex.com/en-us/about/bots/)
  • Ear­li­er ver­sions of the bots had the name linkdex.com/v2.0 and linkdex.com/v2.1

Responsible Crawling

Linkdex will respect the crawl fre­quen­cy lim­i­ta­tion you have out­lined in your robots.txt file. We also crawl rel­a­tive­ly slow­ly, often wait­ing up to 5 sec­onds between pages. This ensures your website’s per­for­mance is not affect­ed in any way.

Contact information

If you would like to dis­cuss our site crawlers, please email us at [email protected] with Site Crawler in the sub­ject line.