AI crawler tracking

AI assistants and search engines read your pages with crawlers that never run JavaScript, so no analytics script can see them. Wazlytics records them on your server instead and shows them in the AI crawlers panel on your dashboard.

What the three tabs mean

• Indexing — search and answer engines building their index (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Applebot and more).
• AI answers — a crawler fetched your page live to answer someone’s question (ChatGPT, Claude, Perplexity, Le Chat, DuckAssist, Meta AI).
• Training — crawlers collecting content for AI model training (GPTBot, ClaudeBot, Amazonbot, Common Crawl and more).

WordPress

Nothing to set up. Once tracking is on for your site, the connected Wazlytics plugin reports crawler visits automatically — within a day, or right away when you open the Wazlytics settings page in wp-admin. The same switch covers WordPress and API reporting.

Any other backend

Send one request per crawler hit. Create a token under Settings → Integration. Hits are recorded only while tracking is on for your site. Only send requests whose user agent looks like a bot — Wazlytics ignores anything it doesn’t recognise, but there is no need to send human traffic.

curl -X POST https://wazlytics.com/api/v1/crawler-hits \
  -H "Content-Type: application/json" \
  -H "X-Wazlytics-Domain: example.com" \
  -H "X-Wazlytics-Site-Token: YOUR_TOKEN" \
  -d '{"path": "/pricing", "user_agent": "Mozilla/5.0 (compatible; GPTBot/1.1)", "ip": "20.171.206.42", "status": 200}'

path, user_agent and ip are required; status defaults to 200. The API always answers 202 for a valid request, so you never need to handle the result — send it in the background after your response. Limits: path up to 2048 characters, user_agent up to 1024, status 100–599. If a quick curl test answers 401 or 404, the domain or token is wrong.

Laravel

// In a terminable middleware, after the response is sent:
Http::withHeaders([
    'X-Wazlytics-Domain' => 'example.com',
    'X-Wazlytics-Site-Token' => config('services.wazlytics.token'),
])->timeout(1)->post('https://wazlytics.com/api/v1/crawler-hits', [
    'path' => $request->getPathInfo(),
    'user_agent' => (string) $request->userAgent(),
    'ip' => $request->ip(),
    'status' => $response->getStatusCode(),
]);

Node.js

// Express middleware — send after the response finishes, so res.statusCode is final:
app.use((req, res, next) => {
  res.on('finish', () => {
    fetch('https://wazlytics.com/api/v1/crawler-hits', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-Wazlytics-Domain': 'example.com',
        'X-Wazlytics-Site-Token': process.env.WAZLYTICS_TOKEN,
      },
      body: JSON.stringify({
        path: req.originalUrl.split('?')[0],
        user_agent: req.get('user-agent'),
        ip: req.ip,
        status: res.statusCode,
      }),
    }).catch(() => {});
  });
  next();
});

Fake crawlers

Anyone can pretend to be Googlebot or GPTBot. For operators that publish their crawler IP addresses (OpenAI, Google, Microsoft, Perplexity, Apple), Wazlytics checks each hit against that list and hides the ones that don’t match. The panel shows how many were hidden. The check uses the visitor IP your server or host passes along, so on a site that isn’t behind a proxy or CDN a determined spoofer could slip past it with a forged forwarded-IP header.

What is stored

The page path (without its query string), which crawler it was, its category, whether its IP checked out, and the HTTP status. The IP address and full user agent are used for the checks and then discarded — they are never stored. AI crawler tracking is included in every plan and does not count toward your monthly events.

For how crawler tracking fits alongside visitors from ChatGPT and other AI tools, see AI traffic analytics.