Make your website ready for AI agents—without chasing a magic file.

1. Make access an explicit policy

A robots.txt file manages automated crawler access; it is not a universal privacy control or proof of indexing. OpenAI also documents separate purposes for OAI-SearchBot, GPTBot, and user-triggered ChatGPT-User requests. Decide which uses you permit, then configure and test those rules deliberately.

• Return a useful 200 response. The canonical page should not depend on a login, consent wall, or broken client-side render.
• Publish intentional crawler rules. Do not copy a blanket allow or deny policy without understanding the user agents.
• Use canonical URLs and a sitemap. Keep the important public pages stable and linked from human-visible navigation. If a concise source map would help, build it with the free llms.txt generator.
• Protect private material with access control. Do not treat robots.txt as authentication.

Primary references: OpenAI crawler documentation and Google's robots.txt guide.

2. Put the facts in the page—not only in metadata

Visible copy should answer the questions a careful buyer or assistant would ask. Structured data can restate those facts in a machine-readable form, but it should agree with the page and never invent missing proof.

Schema.org's potentialAction property can describe a possible action, but markup does not replace an understandable interface.

3. Remove decision-making guesswork

“Get started” is not enough when the visitor still has to infer the price, who qualifies, what happens next, or whether a purchase renews. Put material decision details next to the relevant action. If facts vary, explain the rule or range instead of omitting them.

4. Make the action path boring and reliable

Prefer semantic links, buttons, labels, and forms that expose their purpose. Name the outcome—“Book a 20-minute call” or “Download the report”—rather than relying on “Learn more.” Preserve keyboard access, visible focus, validation messages, and a confirmation state.

• One primary action per decision point, with a descriptive accessible name.
• Labels and instructions that remain available after a field receives focus.
• Errors that say what failed and how to recover without discarding valid input.
• A confirmation page or message that explains completion, delivery, and support.
• Human confirmation for sensitive or irreversible actions.

Where WebMCP fits

WebMCP is a proposed, experimental web standard for exposing structured tools to browser agents. Chrome describes it as a progressive enhancement that can make agent actions more reliable than guessing through interface controls. It is promising for complex forms and workflows, but it is not required for a clear public website—and it needs security review and task-specific testing before production use.

Current status and limitations: Chrome's WebMCP overview and tool security guidance.

Give an external agent a bounded readiness tool

ActableSite also publishes a read-only website audit MCP server. It accepts public targets only, exposes no account or mutation tools, and returns commercial offer details as confirmation-gated metadata rather than taking a purchase action.

A 10-minute readiness check

1. 1

   Open the page in a private window. Can you identify the operator and offer in one screen?

2. 2

   View the page source. Are the core facts present in useful HTML?

3. 3

   Check /robots.txt, the canonical tag, and sitemap for intentional, consistent URLs.

4. 4

   Find price, delivery, eligibility, commitment, and limits without opening a chat.

5. 5

   Follow the primary action using only the keyboard. Confirm the result is explicit and recoverable.

6. 6

   Compare visible facts with structured data. Remove contradictions and unsupported claims.