Skip to main content
Version: Stable (v5.x)

DocSearch x Algolia Crawler

Use this page to resolve common crawler questions. If you need more help, contact us on Discord or through Algolia support.

For more troubleshooting guidance, read the Crawler FAQ:

For questions related to the DocSearch program, please see our DocSearch program FAQ.

How often will you crawl my website?​

Crawls are scheduled at a random time once a week. You can configure this schedule from the config file or trigger one manually from the Crawler interface.

Why do I have duplicate content in my results?​

This can happen when you have more than one URL pointing to the same content, for example with ./docs, ./docs/ and ./docs/index.html.

Configure canonical URLs on your website. For more information, read Google's Consolidate duplicate URLs guide.

Use exclusionPatterns to exclude duplicate URL patterns.

Are the docsearch-scraper and docsearch-configs repositories still maintained?​

The legacy scraper infrastructure is deprecated. You can still run a legacy scraper instance and use its index with the historical DocSearch v3 UI. For current frontend integrations, choose a DocSearch v5 package. The scraper or crawler version doesn't determine the frontend package version.

How do I migrate?​

Every owner should have received a migration email from Algolia with the details. If you were not part of the previous index owners, or the maintainer has changed, you can request access via our support page.

Use the migration email to complete these steps:

  • Join the Algolia application with the invite included in the email
  • Update your frontend integration with the credentials received in the email. For v5, follow the JavaScript or React guide.
Search.tsx
<DocSearch
  appId="YOUR_NEW_ALGOLIA_APP_ID"
  apiKey="YOUR_NEW_ALGOLIA_SEARCH_API_KEY"
  indices={['YOUR_INDEX_NAME']} // The index name doesn't change.
/>

What should I do with my legacy config and credentials?​

Use the credentials for your current Algolia application. Don't use legacy credentials for new frontend integrations.

You should use the dedicated web interface to make any changes to your index.

Why do I see two Algolia apps in my dashboard?​

We did not remove access to the legacy DocSearch application (BH4D9OD16A) to give you the time to get familiar with our new infrastructure. BH4D9OD16A will remain available until the migration has been completed for all the DocSearch users.

Search yields no results​

If your search does not yield any results, but there is no error in your browser developer tools, there might be an issue with your index.

Check the following settings:

  1. Confirm that your crawler configuration matches your website structure.

    Use a configuration template for your website generator as a starting point. Test your selectors with the URL tester.

  2. Confirm that your index settings are current. If they aren't, the search preview displays a banner.

    The crawler applies index settings only when it creates an index, which keeps the Algolia dashboard as the source of truth. If you changed your configuration significantly or moved to a different website generator, delete the index from the Algolia dashboard before starting a new crawl.

Can I delete my crawler?​

Don't delete the default crawler associated with your DocSearch application. Creating a replacement might not preserve the program configuration.

What if I delete my DocSearch Crawler?​

Contact us on Discord or email us at the address below.

Can I use the crawler on password-protected sites?​

The crawler included with DocSearch applications can't access password-protected sites that require a login. To crawl these sites, choose an Algolia plan and add a crawler. Pay-as-you-go plans include a crawler, but the free tier has usage limits.