FEWBot – What Is It and Why Is It Coming to My Website?

FEWBot is the generic name for ForEveryWoman.org’s web crawler. It crawls the web looking for organizations and resources for women facing an unexpected pregnancy. This information is compiled into our database which you can view by clicking here.

How FEWBot accesses your website

For most sites, FEWBot shouldn’t access your site more than once every few seconds on average. However, due to delays it’s possible that the rate will appear to be slightly higher over short periods.

FEWBot was designed to be run simultaneously by multiple machines to improve performance and scale as the web grows. Also, to cut down on bandwidth usage. As such, your logs may show visits from several IP addresses, all with the FEWBot user agent. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server. 

FEWBot crawls primarily from IP addresses in the United States.

Blocking FEWBot from your website

It’s almost impossible to keep a web server secret by not publishing links to it. For example, as soon as someone follows a link from your “secret” server to another web server, your “secret” URL may appear in the referrer tag and can be stored and published by the other web server in its referrer log. Similarly, the web has many outdated and broken links. Whenever someone publishes an incorrect link to your site or fails to update links to reflect changes in your server, FEWBot will try to crawl an incorrect link from your site.

If you want to prevent FEWBot from crawling content on your website, you can update your robots.txt file as follows:

User-Agent: FEWBot
Disallow: *

Alternatively, FEWBot respects User-Agent: * for directives.