How We Test Search Engines for AI
Updated
What counts as AI
We count generative AI a person can see on the results page:
- AI answers, summaries, and Overviews that appear on their own
- AI Mode, chat, or an assistant panel
- AI-generated images in the results
We do not count machine learning used only to rank results or fight spam, spelling correction, autocomplete, or knowledge panels.
An Ask AI button that runs only after a click is an optional entry. We record it in caveats. It does not count as AI by default.
Test queries
Every test uses the same query set. The desk will lock the wording before the first round. Later edits go in the changelog.
| Type | Query |
|---|---|
| knowledge | why is the sky blue |
| how-to | how to remove a stripped screw |
| health | symptoms of low iron |
| product | best budget noise cancelling headphones |
| person | who was Ada Lovelace |
| news | inflation news |
| comparison | baking soda vs baking powder |
| image | golden retriever puppy (image search) |
Devices and browsers
- Desktop: Chrome, Firefox, and Edge, on the current stable version. We record the version number.
- Phones: Chrome on Android and Safari on iOS.
- Default state is signed out. The test location is a US IP address.
- If a device was not tested, that cell stays pending. We do not infer it.
What we record
For each query we record:
- Whether AI appeared, and what kind
- Whether it appeared by default
- Whether it can be turned off
- Whether the setting is still there after the browser is closed and opened again
- What happens on a phone
- A screenshot
Those notes map to the fields in the comparison: default generative answers, can disable AI, setting persists, AI images, result source, login, price, privacy, and mobile support.
Status labels
- No AI With default settings at the tested address, no generative AI appeared in any of our test queries.
- AI optional Generative AI appears by default on some queries, but an official setting turns it off.
- Workaround There is no official off switch. AI can be avoided only with an unofficial method that may stop working.
- Pending test We haven't finished a hands-on test yet. No results are shown until we do.
How often we retest
- Once a month, during the first week.
- Again when a reader reports a change.
- Each month we also re-read Google's help page. That page currently says "We're currently updating settings for Google Search services," so the official wording can change.
Evidence
Each finished test cites a screenshot and, where the claim depends on it, an official document. Both are dated. Screenshots are hosted on this site.
Independence
No ads, no affiliate links, and no paid placements. A search engine cannot pay to change its place in the comparison.
Report a change
We retest when a reader reports a change. A report address is not published on this site.
Changelog
No changes recorded yet.