Ozserf wrote:Funny I also saw another bot I wanted to post about but I'm not sure I remember.... something like MAGIC-12 [bot]
Magestic-12[bot] ?
Yea thats a good not evil one although i have no clue what it dose.
Ozserf wrote:Funny I also saw another bot I wanted to post about but I'm not sure I remember.... something like MAGIC-12 [bot]
Googlebot
From Wikipedia, the free encyclopedia
Jump to: navigation, search
Googlebot Original author(s) Google
Type Web crawler
Website GoogleBot FAQ
Googlebot is the search bot software used by Google, which collects documents from the web to build a searchable index for the Google Search engine.
If a webmaster wishes to restrict the information on their site available to a Googlebot, or another well-behaved spider, they can do so with the appropriate directives in a robots.txt file,[1] or by adding the meta tag <meta name="Googlebot" content="nofollow" /> to the web page.[1] Googlebot requests to Web servers are identifiable by a user-agent string containing "Googlebot" and a host address containing "googlebot.com".[2]
Currently, Googlebot follows HREF links and SRC links.[1] There is increasing evidence Googlebot can execute JavaScript and parse content generated by Ajax calls as well.[3] Googlebot discovers pages by harvesting all of the links on every page it finds. It then follows these links to other web pages. New web pages must be linked to other known pages on the web in order to be crawled and indexed or manually submitted by the webmaster.
A problem which webmasters have often noted with the Googlebot is that it takes up an enormous amount of bandwidth.[citation needed] This can cause websites to exceed their bandwidth limit and be taken down temporarily. This is especially troublesome for mirror sites which host many gigabytes of data. Google provides "Webmaster Tools" that allow website owners to throttle the crawl rate.[4]
techgump wrote:Exabot is for: http://www.exalead.com/search/
Users browsing this forum: No registered users and 6 guests