Overview

Huckabot is Huckabuy’s main crawler which is utilized by almost all of Huckabuy’s products. The primary purpose of Huckabot is to crawl and index a customer’s website, which is then rendered and optimized with our Dynamic Rendering Product. Several of the Page Speed product boosters, such as Fold Prioritization, also leverage Huckabot in order to optimize and improve a website’s performance.

When Huckabot performs any outgoing requests, such as crawling a website, it uses a Proxy so that all requests use a common user-agent (huckabot-) as well as a standard IP Address (34.134.146.126). This information gives customers the view specific requests from Huckabuy using either the user-agent or IP Address.

Whitelisting

If your website uses any kind of firewall or request filtering, make sure to add the Huckabot user-agent (huckabot-) as well as the proxy IP Address (34.134.146.126) to your firewall’s whitelist so that Huckabuy’s products are able to fully crawl and access your website, which is needed for Structured Data, Page Speed and Dynamic Rendering.

Also, during Huckabuy Cloud’s onboarding process, if you are using either the Standard, Enterprise, or Cloudflare API integration methods, a firewall rule is automatically created to allow Huckabot to bypass any firewall rules in place.