Monetize AI Crawlers with x402 Instead of Blocking Them

Block or charge?

When AI training crawlers show up in your logs, the default reaction is to add Disallow: / to robots.txt and move on.

That's a reasonable choice — but it leaves money on the table. Training data is genuinely valuable. The companies building foundation models have budgets. The question is whether there's a frictionless way to collect payment from an automated process.

x402 provides exactly that.

---

What is x402?

x402 is an open protocol for HTTP micropayments, built on EIP-3009 (signed USDC transfers). When a crawler hits an x402-protected endpoint, the server responds with HTTP 402 Payment Required and a machine-readable payment challenge. The crawler signs a USDC transfer, includes it in the next request, and the server verifies and processes it atomically.

No API keys. No billing accounts. No human in the loop. The entire flow is a two-request exchange — challenge then payment — and it works with any x402-compatible client.

Payments settle on Base mainnet in seconds. Your wallet receives USDC directly.

---

Setting up .well-known/x402

The .well-known/x402 file is a JSON discovery endpoint that tells x402 clients which paths are monetized and how much they cost. Place it at:

https://yourdomain.com/.well-known/x402

A minimal configuration:

{
  "version": "1",
  "monetizedPaths": [
    {
      "path": "/*",
      "pricing": [
        {
          "scheme": "exact",
          "currency": "USDC",
          "network": "base-mainnet",
          "amount": "1000",
          "decimals": 6
        }
      ]
    }
  ],
  "paymentAddress": "0xYOUR_WALLET_ADDRESS"
}

This charges $0.001 USDC per crawled page (amount: 1000 with 6 decimal places = 0.001 USDC). Adjust the path pattern and amount to match your content.

For a Next.js app using the x402-next middleware, the setup is a few lines in middleware.ts:

import { withPaymentRequired } from 'x402-next'

export const middleware = withPaymentRequired({
  facilitatorUrl: 'https://x402.org/facilitator',
  paymentRequiredResponse: {
    maxAmountRequired: '1000',
    asset: '0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913', // USDC on Base
    payTo: process.env.PAYMENT_ADDRESS,
  },
})

export const config = {
  matcher: ['/((?!_next|favicon.ico).*)'],
}

---

Tracking your revenue with paylog.dev

Once x402 is live, you can track inbound payments with the paylog API:

# See all x402 payments received on Base
GET https://paylog.dev/api/v1/x402/report?wallet=YOUR_ADDRESS

# Combined MPP + x402 report
GET https://paylog.dev/api/v1/report?wallet=YOUR_TEMPO_ADDRESS&chain=all

The response breaks down payments by sender address and service, with daily totals. This tells you not just the total received, but which crawlers are paying and at what frequency.

You can also use the CLI:

EVM_PRIVATE_KEY=0x... npx @kakedashi/paylog report --chain base

---

Revenue estimates

x402 crawl revenue depends on your traffic and how many crawlers support the protocol. A rough estimate for a content site:

Monthly crawls	Price per crawl	Monthly revenue
1,000	$0.001	$1
10,000	$0.001	$10
100,000	$0.001	$100
1,000,000	$0.001	$1,000

High-traffic sites (news, documentation, code repositories) can see crawler volumes in the millions. At $0.001 per crawl, that's real revenue without any additional operational cost.

For higher-value content — paid research, proprietary data, curated datasets — charging $0.01–$0.10 per page is not unreasonable. The right price depends on what the content is worth to a model trainer versus what they'd pay through a licensing deal.

---

The practical picture today

x402 client adoption is still early. Most crawlers today don't implement the protocol, so they'll either ignore the 402 response or back off. This means:

• Training crawlers that do support x402 will pay.
• Training crawlers that don't can be blocked via robots.txt as a fallback.
• AI assistants (ClaudeBot, ChatGPT-User) can be allowed for free — they drive traffic, not training data extraction.

A combined strategy:

# Allow AI assistants for free
User-agent: ClaudeBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
Allow: /

# Training crawlers: charge via x402 (fallback: block)
User-agent: GPTBot
User-agent: anthropic-ai
User-agent: Google-Extended
Disallow: /

With .well-known/x402 in place, x402-capable clients will pay instead of being blocked. Non-capable clients see the Disallow and (should) respect it.

Check your current configuration at paylog.dev/score — it shows whether .well-known/x402 is detected and flags training bots that are neither charged nor blocked.