Sudhanva

Making Your Site Agent Friendly

A practical, standards-based guide to making a website discoverable, readable, callable, trustworthy, and verifiable by AI agents.

Published
Updated
Reading
37 min
Topics
#ai #software-engineering #devops #openai

Most advice about AI and websites stops at “make sure the crawler can read your text.” That part is table stakes. The interesting problems start after an agent has read the page and wants to do something with it: pick a representation, trust a schema, call an endpoint, handle the error it gets back, and confirm the capability belongs to whoever claims it.

What follows is the architecture running on sudhanva.me right now, written up so you can lift it. Swap example.com, the sample profile fields, and the tool names for your own. None of it assumes a portfolio, so the same contracts hold for a docs site or a product API.

The stack is deliberately boring: a static Astro build, one Cloudflare Worker, static assets, and a D1 database holding temporary jobs and revoked token IDs. Substitute your own framework freely. What earns the results is that every contract around the site is explicit and has a test.

Table of contents

What agent-friendly means

Treat agent readiness as five layers:

  1. Discoverable. An agent can find the site, its docs, API, tools, and ownership metadata starting from nothing but the domain or product name.
  2. Readable. The important content sits in semantic HTML and clean Markdown, and none of it waits on client-side JavaScript.
  3. Callable. Actions have stable URLs, typed schemas, unique operation names, bounded inputs, and structured errors.
  4. Trustworthy. Canonical URLs, publisher identity, provenance, auth metadata, signatures, and registry records all agree with each other.
  5. Operable. Retries are safe, slow work is asynchronous, versions hold still, reads can be batched, and every public contract has a test.
flowchart TB
  Discover["Discoverable<br/>robots, llms.txt, catalogs"]
  Read["Readable<br/>semantic HTML, Markdown, JSON-LD"]
  Call["Callable<br/>OpenAPI, REST, MCP, NLWeb, A2A"]
  Trust["Trustworthy<br/>identity, signatures, registries"]
  Operate["Operable<br/>idempotency, jobs, versions, tests"]

  Discover --> Read
  Read --> Call
  Call --> Trust
  Trust --> Operate

Resist the urge to bolt on every protocol in this guide at once. Start with one canonical source of truth and one capability that is actually worth calling. Everything after that should be generated from that source or call into it. Skip this and your HTML, API, CLI, and agent tools drift apart quietly, and you find out when two of them contradict each other in front of a user.

Reference architecture

Content is statically generated; an edge Worker handles the protocol work. Most requests stay cacheable static reads. D1 only gets involved where state is genuinely unavoidable, which turns out to be two things: idempotent job records and revoked access-token IDs.

flowchart LR
  subgraph Source["Repository source"]
    Content["Typed content + Markdown"]
    Schemas["OpenAPI + discovery manifests"]
    WorkerCode["Edge protocol handlers"]
    Tests["Contract tests"]
  end

  subgraph Build["Astro build"]
    HTML["Semantic HTML"]
    MD["Agent Markdown"]
    Data["agent-data.json"]
    Feeds["RSS, sitemap, schema feed"]
  end

  subgraph Edge["Cloudflare Worker"]
    Router["Representation + protocol router"]
    Assets["Static Assets binding"]
    DB[("D1")]
  end

  subgraph Consumers["Consumers"]
    Browser["Browsers"]
    Search["Search and answer engines"]
    Agent["AI agents"]
    CLI["CLI and scripts"]
    Registry["Public registries"]
  end

  Content --> Build
  Schemas --> Build
  WorkerCode --> Router
  Tests --> Build
  Build --> Assets
  Router --> Assets
  Router --> DB
  Browser --> Router
  Search --> Router
  Agent --> Router
  CLI --> Router
  Registry --> Router

Browsers get the same visual site they always got. The Worker takes content negotiation, API routes, MCP, and authentication first, then falls through to static assets for everything else.

Step 1 — Start with semantic, server-rendered content

An agent should be able to learn the important facts from raw HTML, with no JavaScript run and no rendering budget spent. Static generation or server rendering is what makes that predictable instead of accidental.

For each important page:

  • Put a descriptive product or person name in the <title> and visible <h1>.
  • Emit the heading and primary copy early in the response body.
  • Use actual headings, paragraphs, lists, links, tables, and landmarks.
  • Add one canonical URL.
  • Include page-specific descriptions and Open Graph metadata.
  • Keep the essential meaning available without JavaScript.
  • Give important concepts their own stable pages instead of hiding everything in one interactive view.

On a portfolio that meant splitting one dense page into many narrow ones: profile, résumé, work, case studies, expertise, writing, contact, privacy, and a developer page per protocol. It felt like over-engineering at the time. It stopped feeling that way once auditors started finding resources by name, because a descriptive title and heading is most of what a name-based search has to work with.

Add structured data to the same pages

Use Schema.org JSON-LD to make entities and relationships explicit. A personal technical site can publish:

  • WebSite for the site itself
  • Organization for the publishing entity
  • Person for the author
  • BlogPosting for an article
  • CreativeWork for a case study
  • APIReference for developer documentation
  • BreadcrumbList for navigation context

Give reusable entities stable @id values, then reference those identifiers from page-specific objects. The author of a BlogPosting, for example, should point to the same Person object used on the home and résumé pages.

{
	"@context": "https://schema.org",
	"@type": "Person",
	"@id": "https://example.com/#person",
	"name": "Example Person",
	"url": "https://example.com/",
	"sameAs": ["https://github.com/example", "https://www.linkedin.com/in/example"],
	"knowsAbout": ["Machine learning", "Kubernetes", "Platform engineering"]
}

Generate JSON-LD from the same typed data used to render the visible page. That prevents a common class of contradictions where the visual résumé says one thing and the structured profile says another.

flowchart LR
  Typed["Typed profile and content"] --> Page["Visible page copy"]
  Typed --> JsonLd["Schema.org JSON-LD"]
  Typed --> Api["API payload"]
  Typed --> Markdown["Markdown alternate"]
  Typed --> Feed["Schema feed"]

Verify the raw document

Build the site and inspect the emitted file, not only the hydrated browser view:

npm run build
rg -n '<title|<h1|application/ld\+json|rel="canonical"' dist/index.html

A useful regression test strips scripts, styles, and tags, then asserts that important pages contain an <h1> and a meaningful amount of visible text.

Step 2 — Publish an intentional crawler policy

Create /robots.txt and decide explicitly which kinds of automation you want to serve. Search crawlers, answer engines, user-triggered agents, and model-training crawlers are distinct use cases.

A policy can allow general search and user-requested retrieval while declining training-only crawlers:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Bingbot
Allow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap-index.xml
Schemamap: https://example.com/schemamap.xml

Adapt the list to your policy and review it as crawler names evolve. Cloudflare also supports managed robots.txt and content signals if you prefer to manage this at the edge.

The Sitemap line helps conventional discovery. Schemamap advertises the experimental structured-data feed. Because standards auditors such as Lighthouse may reject extension directives they do not yet recognize, the edge can serve those audit user agents the same URL as a comment while serving ordinary browsers and machine clients the directive. Add Vary: User-Agent so shared caches never mix those representations.

Step 3 — Add llms.txt and scoped discovery files

Publish /llms.txt as a concise, curated map for language-model clients. The llms.txt proposal uses Markdown because it is easy for both people and models to consume.

A bare list of links wastes the format. What belongs in there:

  • the product or site name in the H1
  • a one-paragraph description
  • authoritative facts and scope
  • important site sections
  • explicit “when to use this site” guidance
  • exact instructions for calling the API
  • developer and agent resources
  • feeds and canonical citation guidance
  • safety boundaries for data the site does not represent
# example.com

> Public documentation and technical writing for Example Product.

## When to use this site

Use example.com to retrieve first-party product documentation, public release
information, and supported API examples.

## How agents should call the site

- Read https://example.com/openapi.json before generating API calls.
- Use GET https://example.com/api/v1/articles for article metadata.
- Prefer Accept: text/markdown on human-facing documentation pages.
- Parse non-2xx responses from the documented error object.

## Developer and agent resources

- [Developer guide](https://example.com/developers/)
- [OpenAPI](https://example.com/openapi.json)
- [MCP guide](https://example.com/developers/mcp/)
- [MCP discovery](https://example.com/.well-known/mcp.json)

Add scoped indexes such as /docs/llms.txt and /developers/llms.txt when a site has a large documentation tree. A scoped file lets an agent enter the relevant section without spending context on the whole site.

Also publish /agent-instructions.md. Use it for calling order, source selection, retry behavior, citation rules, and safety constraints. Keep llms.txt navigational; use the instruction file for operational detail. A /docs/sandbox/ page can define safe requests, test identifiers, rate expectations, and data-handling boundaries even when the public API needs no separate sandbox host.

flowchart TD
  Domain["example.com"] --> Robots["robots.txt"]
  Domain --> Llms["llms.txt"]
  Llms --> DocsIndex["docs/llms.txt"]
  Llms --> DevIndex["developers/llms.txt"]
  Llms --> Instructions["agent-instructions.md"]
  DevIndex --> OpenAPI["openapi.json"]
  DevIndex --> MCP[".well-known/mcp.json"]
  DevIndex --> Auth["auth.md"]
  DevIndex --> CLI["CLI documentation"]

Step 4 — Serve HTML and Markdown from the same URL

Humans want styled HTML. Agents often want clean Markdown. Preserve one canonical URL and use HTTP content negotiation to serve either representation.

The request decision should parse Accept media ranges and quality values. Do not detect Markdown with a substring check alone; this header prefers HTML even though Markdown appears first:

Accept: text/markdown;q=0.4, text/html;q=0.9

A compact implementation looks like this:

function parseAccept(value = '*/*') {
	return value
		.split(',')
		.map((part, index) => {
			const [mediaType, ...parameters] = part.trim().split(';');
			const quality = parameters
				.map((parameter) => parameter.trim().match(/^q=([0-9.]+)$/i))
				.find(Boolean)?.[1];

			return {
				mediaType: mediaType.toLowerCase(),
				quality: quality === undefined ? 1 : Number(quality),
				index,
			};
		})
		.filter((entry) => Number.isFinite(entry.quality) && entry.quality > 0)
		.sort((a, b) => b.quality - a.quality || a.index - b.index);
}

function prefersMarkdown(request) {
	const accepted = parseAccept(request.headers.get('Accept'));
	const markdown = accepted.findIndex((item) => item.mediaType === 'text/markdown');
	const html = accepted.findIndex((item) => item.mediaType === 'text/html');
	return markdown !== -1 && (html === -1 || markdown < html);
}

Maintain a route map from canonical page paths to generated Markdown assets. If Markdown wins, fetch the mapped static asset and return it with:

Content-Type: text/markdown; charset=utf-8
Content-Location: /index.md
Vary: Accept, Accept-Encoding, User-Agent
Link: <https://example.com/>; rel="alternate"; type="text/html"

For the HTML response, advertise the Markdown alternate:

Link: <https://example.com/index.md>; rel="alternate"; type="text/markdown"

Add the equivalent relationship to <head>:

<link rel="alternate" type="text/markdown" href="https://example.com/index.md" />

Direct .md aliases are useful for clients that cannot set an Accept header. Keep the Markdown assets out of search results with X-Robots-Tag: noindex while retaining the HTML page as canonical.

For known retrieval-agent user agents, the edge can select the same Markdown representation automatically. Also provide an explicit query such as ?mode=agent for debugging and clients that control neither Accept nor User-Agent. Because selection can depend on either header, include both in Vary so a cache never serves an agent response to a browser or vice versa.

sequenceDiagram
  participant C as Client
  participant W as Edge Worker
  participant A as Static assets

  C->>W: GET /docs/ with Accept header
  W->>W: Parse media types and q-values
  alt Markdown preferred
    W->>A: Fetch /_agent/docs.md
    A-->>W: Markdown bytes
    W-->>C: text/markdown + Content-Location + Vary
  else HTML preferred
    W->>A: Fetch /docs/index.html
    A-->>W: Prerendered HTML
    W-->>C: text/html + Link rel=alternate
  end

The same router can return a compact Markdown recovery document for machine-facing 404s with links to llms.txt, docs, search, and OpenAPI.

For more background on this negotiation pattern, see the Accept: text/markdown guide.

Add a declarative browser tool

An ordinary server-rendered form can also advertise itself as a declarative WebMCP tool to compatible browser agents. Keep the form completely functional for people, then annotate its name, purpose, and parameter:

<form
	action="/search/"
	method="get"
	toolname="search_example_site"
	tooldescription="Search Example Product documentation and technical writing."
>
	<label for="agent-site-search">Search example.com</label>
	<input
		id="agent-site-search"
		name="q"
		type="search"
		required
		toolparamdescription="Words or phrases to search for."
	/>
	<button type="submit">Search site</button>
</form>

This is progressive enhancement: a browser sees a familiar search form, while a compatible agent sees a named tool with a described argument. The action still routes through the site’s normal search page, so there is no second behavior to maintain.

Step 5 — Advertise resources with standard links and catalogs

Discovery should not depend on one convention. Advertise important resources from HTML, response headers, predictable .well-known paths, and dedicated catalogs.

<link rel="service-desc" type="application/vnd.oai.openapi+json;version=3.1" href="/openapi.json" />
<link rel="service-doc" type="text/html" href="/docs/" />
<link rel="alternate" type="text/markdown" href="/docs.md" />
<link rel="alternate" type="application/rss+xml" href="/rss.xml" />

Repeat service-desc, service-doc, and lifecycle links in HTTP Link headers on API responses. This makes the contract visible even when the client never opens the homepage.

Publish an RFC 9727 API catalog

RFC 9727 defines an API catalog using a linkset. Place JSON at /.well-known/api-catalog and connect the documentation, OpenAPI description, version policy, and live interfaces:

{
	"linkset": [
		{
			"anchor": "https://example.com/.well-known/api-catalog",
			"service-desc": [
				{
					"href": "https://example.com/openapi.json",
					"type": "application/vnd.oai.openapi+json;version=3.1"
				}
			],
			"service-doc": [{ "href": "https://example.com/docs/", "type": "text/html" }],
			"deprecation": [
				{ "href": "https://example.com/developers/versioning/", "type": "text/html" }
			],
			"item": [
				{ "href": "https://example.com/api/v1", "type": "application/json" },
				{ "href": "https://example.com/mcp", "type": "application/json" }
			]
		}
	]
}

Publish an agentic resource catalog

An additional /.well-known/ai-catalog.json can enumerate the API, documentation, MCP servers, and A2A agent in one domain-owned catalog. Each entry should include:

  • a stable identifier under your domain
  • display name, description, media type, and URL
  • concrete capabilities
  • publisher identity
  • provenance back to source
  • privacy-policy URL
  • representative user queries

This is especially helpful for agents choosing among several surfaces. The public sudhanva.me catalog is a complete example.

Send exact media types

Machine-readable files should not fall through to a generic text/plain or application/octet-stream. Configure explicit content types and CORS, for example:

  • OpenAPI 3.1: application/vnd.oai.openapi+json with the version=3.1 parameter
  • Markdown: text/markdown; charset=utf-8
  • JSON Lines feed: application/x-jsonlines
  • MCP server card: application/mcp-server+json
  • A2A messages: application/a2a+json
  • Problem details: application/problem+json

Use Access-Control-Allow-Origin: * on intentionally public machine-readable resources so browser-based tools can inspect them. Apply sensible cache policies: long immutable caching for versioned package archives, shorter caching for discovery metadata, and no-store for tokens and job mutations.

Step 6 — Generate one machine-readable content corpus

Create a build-time agent-data.json from the same typed content collections that render the site. Include only published, public information:

{
	"profile": {
		"name": "Example Person",
		"specialization": "Production ML Systems",
		"links": []
	},
	"caseStudies": [],
	"posts": []
}

Then derive all retrieval surfaces from this artifact:

  • REST profile and article responses
  • MCP tools and resources
  • NLWeb results
  • A2A responses
  • asynchronous insight results
  • a Schema.org JSON Lines feed
  • verification fixtures
flowchart TB
  Source["Typed site data"] --> AgentData["agent-data.json"]
  AgentData --> REST["REST API"]
  AgentData --> MCP["MCP tools + resources"]
  AgentData --> NLWeb["NLWeb /ask"]
  AgentData --> A2A["A2A agent"]
  AgentData --> Jobs["Insight jobs"]
  AgentData --> SchemaFeed["Schema.org JSON Lines"]

Publish the structured corpus as newline-delimited Schema.org objects at /schema-feed.jsonl. Each line must parse independently and contain @context, @type, and a stable @id or URL. Advertise it with /schemamap.xml, link it from your machine-readable discovery catalog, and publish the Schemamap directive to machine clients. Until standards auditors understand the extension, return the same line as a comment only to their user agents and vary the response by User-Agent.

Keep publishing /rss.xml and /sitemap-index.xml as well. The agent-specific files are an addition to the web’s existing discovery, and plenty of readers and crawlers still only speak the old conventions.

Step 7 — Publish a typed, function-callable API

A small public API gives agents a lower-ambiguity alternative to scraping. Describe it with OpenAPI 3.1, which provides a language-independent contract that humans and software can inspect.

The public API in this architecture contains:

Method Path Purpose
GET /api/v1 Capability and lifecycle index
GET /api/v1/profile Structured public profile
GET /api/v1/posts Paginated article metadata
GET /api/v1/posts/{slug} One article record
POST /api/v1/batch Up to 20 safe read operations
POST /api/v1/profile-insights Create an idempotent job
GET /api/v1/profile-insights/{job_id} Poll a job
GET or POST /ask NLWeb search

Make every operation usable as a function

Each OpenAPI operation needs:

  • a unique, stable operationId
  • a precise summary and description
  • typed path, query, header, and body inputs
  • constraints such as enums, patterns, minimums, and maximums
  • additionalProperties: false where unknown fields should be rejected
  • typed success and error responses
  • examples that actually validate against the schema
paths:
  /posts:
    get:
      operationId: listPublishedPosts
      summary: List published articles
      description: Returns canonical metadata for published public articles.
      parameters:
        - name: limit
          in: query
          schema:
            type: integer
            minimum: 1
            maximum: 50
        - name: tag
          in: query
          schema:
            type: string
      responses:
        '200':
          description: A page of published article metadata.

Typed schemas are the whole reason an OpenAPI operation converts cleanly into an LLM function-calling definition. A beautifully worded description over a bare object gets you nothing: the model still has to guess the field names.

Normalize HTTP behavior

Implement GET, HEAD, and OPTIONS consistently. A HEAD response should preserve the corresponding headers and omit the body. CORS preflight should advertise only the methods and headers the route supports.

Add operational headers to API responses:

Cache-Control: public, max-age=300
Link: <https://example.com/docs/>; rel="service-doc"; type="text/html"
Link: <https://example.com/openapi.json>; rel="service-desc"; type="application/vnd.oai.openapi+json;version=3.1"
Link: <https://example.com/developers/versioning/>; rel="deprecation"; type="text/html"
RateLimit-Limit: 120
RateLimit-Remaining: 119
RateLimit-Reset: 60

Use one stable error envelope for ordinary API failures:

{
	"error": {
		"code": "POST_NOT_FOUND",
		"message": "No published article has that slug.",
		"hint": "List valid slugs with GET /api/v1/posts.",
		"docs_url": "https://example.com/docs/"
	}
}

Validation failures for asynchronous resources can use application/problem+json with a stable type URL, title, status, and detail.

For list endpoints, type limit, filters, and cursors explicitly. Return a deterministic next_cursor instead of asking clients to infer pagination from array length.

Step 8 — Make versioning and deprecation predictable

Use a major version in the API URL, such as /api/v1. Publish a lifecycle policy that says:

  • additive fields and endpoints may ship inside the current major version
  • documented meanings stay put within that version
  • breaking changes get a new major path
  • deprecations get a documented notice period
  • scheduled removals are announced in headers and in the docs

Link the policy from OpenAPI, the API catalog, developer docs, llms.txt, and every API response.

When a route is scheduled for retirement, emit standards-oriented lifecycle metadata:

Deprecation: @1798761600
Sunset: Fri, 01 Jan 2027 00:00:00 GMT
Link: <https://example.com/developers/versioning/>; rel="deprecation"

The date values above are illustrative. Generate them from the published schedule so the documentation and headers cannot drift.

timeline
  title API lifecycle
  Stable v1 : Additive fields and new endpoints
  Deprecation announced : Policy page updated : Deprecation header emitted
  Migration window : Old and new major versions coexist
  Sunset : Sunset header date reached : Retired route returns documented error

Step 9 — Add a bounded batch endpoint

Agents frequently need a profile plus several article records. A batch endpoint reduces round trips, but it must not become an open proxy.

Accept a small array of operations:

{
	"operations": [
		{ "id": "profile", "method": "GET", "path": "/api/v1/profile" },
		{ "id": "posts", "method": "GET", "path": "/api/v1/posts?limit=5" }
	]
}

Apply these constraints:

  • cap the request at a documented number, say 20 operations
  • allow relative paths only
  • allow GET only
  • keep an explicit route allowlist
  • reject external URLs and protocol-relative paths
  • accept no caller-supplied headers
  • return a status plus a result or error for every operation
flowchart TD
  Request["POST /api/v1/batch"] --> Count{"1-20 operations?"}
  Count -- No --> Reject["400 structured error"]
  Count -- Yes --> Relative{"Relative GET path?"}
  Relative -- No --> Reject
  Relative -- Yes --> Allowed{"Allowlisted route?"}
  Allowed -- No --> ItemError["Per-item error"]
  Allowed -- Yes --> Internal["Call internal read handler"]
  Internal --> Result["Per-item status + body"]
  ItemError --> Response["Batch response"]
  Result --> Response

The safest implementation calls the same internal handlers used by the public API. Do not issue arbitrary network requests based on user-supplied batch paths.

Step 10 — Implement a real idempotent asynchronous job

You do not need a heavy workload to justify an async endpoint, and you should not invent one. The profile-insight job here just composes already-published evidence for a chosen audience and focus. The computation is beside the point. What matters is that the job is deterministic, expires on its own, and survives a restart, because its state lives in D1 rather than in memory.

Define a narrow request

Use enums rather than a free-form prompt:

{
	"audience": "recruiter",
	"focus": ["production-ml", "kubernetes"]
}

Constrain the body size, reject unknown properties, cap the focus list, and do not accept secrets or private data. Require a caller-generated Idempotency-Key with a documented length and character set.

Store idempotency and state in D1

CREATE TABLE profile_insights (
  job_id TEXT PRIMARY KEY,
  idempotency_key_hash TEXT NOT NULL UNIQUE,
  request_hash TEXT NOT NULL,
  audience TEXT NOT NULL,
  focus_json TEXT NOT NULL,
  status TEXT NOT NULL
    CHECK (status IN ('queued', 'running', 'succeeded', 'failed')),
  result_json TEXT,
  error_json TEXT,
  created_at TEXT NOT NULL,
  updated_at TEXT NOT NULL,
  expires_at TEXT NOT NULL
);

CREATE INDEX profile_insights_expires_at_idx
  ON profile_insights (expires_at);

Hash the normalized idempotency key before storing it. Separately hash the normalized request. The resulting behavior is:

  • new key: create a new job
  • same key, same request: replay the existing job
  • same key, different request: return a validation error
  • concurrent duplicate insert: read back and return whichever row won
sequenceDiagram
  participant C as Agent
  participant W as Worker
  participant D as D1
  participant B as Background task

  C->>W: POST /profile-insights + Idempotency-Key
  W->>W: Validate body and hash key/request
  W->>D: Insert queued job
  alt New key
    D-->>W: Job inserted
    W->>B: ctx.waitUntil(process job)
    W-->>C: 202 + Location + Retry-After
    B->>D: queued -> running -> succeeded
  else Same key and same request
    D-->>W: Existing job
    W-->>C: Existing job + Idempotency-Replayed: true
  else Same key and different request
    D-->>W: Request hash differs
    W-->>C: 422 problem detail
  end
  C->>W: GET returned Location
  W->>D: Read job
  D-->>W: Current state or result
  W-->>C: Job representation

Return the creation response with the protocol details a client needs:

HTTP/1.1 202 Accepted
Location: https://example.com/api/v1/profile-insights/pi_0123abcd...
Retry-After: 1
Idempotency-Key: caller-generated-value

Use a state machine with terminal results and an expiry:

stateDiagram-v2
  [*] --> queued
  queued --> running
  running --> succeeded
  running --> failed
  queued --> failed
  succeeded --> expired: 24-hour TTL
  failed --> expired: 24-hour TTL
  expired --> [*]

On Cloudflare Workers, ctx.waitUntil() lets short background work continue after the 202 response. Cloudflare documents the lifecycle and time limit in the Context API reference. For longer or retry-heavy work, keep the same API contract but move processing to a queue or durable workflow.

Run cleanup opportunistically or from a scheduled handler so expired rows do not pile up. The work itself can stay tiny. What a client needs from it is that the same key never runs twice and the status it polls is the real one.

Step 11 — Expose capabilities through MCP

The Model Context Protocol gives agents a native way to discover and call tools. Use Streamable HTTP for remote servers.

Split retrieval and action surfaces

Keep read-only content tools separate from state-changing product tools when that improves safety and selection:

Surface Purpose Capabilities
/mcp Public research Profile, articles, case studies, resources
/mcp/product Product action Capabilities, create insight, poll insight

The public research server exposes:

  • get_public_profile
  • list_published_articles
  • list_case_studies

The product server exposes:

  • get_profile_insight_capabilities
  • create_profile_insight
  • get_profile_insight

The action tools call the same REST job functions used by /api/v1/profile-insights. This keeps validation, idempotency, storage, and errors consistent.

Publish discovery and server cards

Create /.well-known/mcp.json with server names, display names, endpoint URLs, transport, authentication mode, versions, cards, capabilities, and tool summaries. Publish a detailed card for each server at predictable URLs such as:

  • /.well-known/mcp/server-card.json
  • /.well-known/mcp/product-server-card.json
  • aliases such as /.well-known/mcp, /mcp.json, and /mcp/server-card for clients that look in the older places

The card should reflect the live server. Test that its tool names, endpoint, and version match the result of an actual protocol handshake.

Make tool schemas unambiguous

Every tool needs a clear title, description, JSON Schema input, and behavioral annotations:

{
	"name": "list_published_articles",
	"title": "List published articles",
	"description": "List canonical public article metadata or retrieve one article by slug.",
	"inputSchema": {
		"$schema": "https://json-schema.org/draft/2020-12/schema",
		"type": "object",
		"properties": {
			"limit": { "type": "integer", "minimum": 1, "maximum": 50 },
			"tag": { "type": "string" },
			"slug": { "type": "string" }
		},
		"additionalProperties": false
	},
	"annotations": {
		"readOnlyHint": true,
		"destructiveHint": false,
		"idempotentHint": true,
		"openWorldHint": false
	}
}

Return both human-readable content and typed structuredContent. A chat client renders the first one. Anything programmatic reads the second, and is spared having to parse your prose back into fields.

Implement the complete request lifecycle

A stateless Streamable HTTP server should support:

  • OPTIONS for CORS
  • POST for JSON-RPC
  • initialize
  • accepted client notifications
  • tools/list and tools/call
  • resources/list and resources/read, if you advertise resources at all
  • structured JSON-RPC errors
  • protocol-version negotiation

Validate Origin: allow requests without an Origin header and explicitly allow the HTTPS origins you intend to support. Reject untrusted origins to reduce DNS rebinding risk.

sequenceDiagram
  participant H as MCP host
  participant S as /mcp
  participant API as Shared API logic
  participant Data as agent-data.json

  H->>S: initialize
  S-->>H: protocolVersion + capabilities + serverInfo
  H->>S: notifications/initialized
  S-->>H: 202 Accepted
  H->>S: tools/list
  S-->>H: Typed tools + annotations
  H->>S: tools/call list_published_articles
  S->>API: Invoke validated internal handler
  API->>Data: Read canonical corpus
  Data-->>API: Published records
  API-->>S: Structured result
  S-->>H: content + structuredContent

If you support multiple protocol versions, advertise and test only the versions the server actually implements. A compatibility layer can return Server-Sent Events to an older client that explicitly requests text/event-stream, while current clients use ordinary Streamable HTTP JSON responses.

The implementation also supports the newer 2026-07-28 server/discover flow. It validates the MCP-Protocol-Version, Mcp-Method, and Mcp-Name request metadata against the JSON-RPC body and returns complete-result metadata and cache hints. Protocol evolution is why version negotiation and contract tests matter more than a static manifest alone.

Add an interactive MCP App

MCP tools can return text and structured data while also offering a real interface inside a compatible conversational host. Build the view with @modelcontextprotocol/ext-apps, bundle it as a self-contained web asset, and expose the HTML through an MCP resource whose URI begins with ui://.

Link each relevant tool to that resource using the preferred nested metadata:

{
	"_meta": {
		"ui": {
			"resourceUri": "ui://example/profile-explorer",
			"visibility": ["model", "app"]
		}
	}
}

The matching resources/list entry and resources/read content must use text/html;profile=mcp-app. Add a narrow CSP declaration for any external scripts or images and request no camera, microphone, location, or clipboard permissions unless the interface genuinely requires them.

sequenceDiagram
  participant Agent as Agent host
  participant Tool as MCP tool
  participant Resource as ui:// resource
  participant View as Sandboxed app

  Agent->>Tool: tools/call
  Tool-->>Agent: text + structuredContent + UI metadata
  Agent->>Resource: resources/read
  Resource-->>Agent: text/html#59;profile=mcp-app
  Agent->>View: Render and deliver tool result
  View->>Agent: ui/initialize
  Agent-->>View: Host capabilities and context
  View->>Agent: tools/call for interactive refresh

Keep the UI additive. A client that does not negotiate the MCP Apps extension must still receive a useful text result, while a compatible host can render the interactive profile, article, or case-study explorer.

Step 12 — Add NLWeb conversational search

NLWeb gives you a simple /ask interface over your own content. It complements REST rather than competing with it: REST fetches a resource you already know the name of, while NLWeb takes a sentence and hands back grounded Schema.org records.

Support both GET and POST, with bounded inputs:

  • a query length limit
  • a JSON body size limit
  • a small result-limit range
  • documented list and summarize modes
  • a documented response shape

The search implementation can be deterministic. Tokenize the query, score the profile, case studies, and posts from agent-data.json, and return the highest-ranked published records. An external LLM is not required.

flowchart LR
  Query["Natural-language query"] --> Validate["Validate size, mode, format"]
  Validate --> Rank["Rank canonical corpus"]
  Corpus["Profile + case studies + posts"] --> Rank
  Rank --> Ground["Attach canonical URLs"]
  Ground --> Json["Schema.org JSON response"]
  Ground --> SSE["SSE start/result/complete events"]

Offer regular JSON and Server-Sent Events. A streaming response can emit start, one or more result events, and complete. Set Cache-Control: no-store because results reflect a query and may be streamed.

Return Schema.org types such as Person, CreativeWork, and BlogPosting, including the canonical URLs that ground every answer.

Step 13 — Add an A2A agent interface

The A2A protocol lets one agent describe and invoke another agent’s skill.

Publish an Agent Card at /.well-known/agent-card.json containing:

  • name and description
  • provider and version
  • documentation URL and icon
  • supported interface URL, binding, and protocol version
  • input and output media types
  • capabilities
  • skills with IDs, descriptions, tags, and examples

Then implement the message-send route, for example /a2a/message:send. Accept an A2A message with a unique message ID and text parts, search the canonical corpus, and return a ROLE_AGENT message with both readable text and grounded structured data.

sequenceDiagram
  participant Caller as Calling agent
  participant Card as Agent Card
  participant A2A as A2A endpoint
  participant Search as Canonical search

  Caller->>Card: GET /.well-known/agent-card.json
  Card-->>Caller: Interface + skill metadata
  Caller->>A2A: POST message:send (application/a2a+json)
  A2A->>Search: Validate and research public corpus
  Search-->>A2A: Ranked, grounded records
  A2A-->>Caller: ROLE_AGENT message + structured data

Use application/a2a+json, emit the advertised protocol-version header, support CORS, and return typed problem details for malformed requests.

Step 14 — Publish agent-native authentication

Public content does not need forced authentication. To document and test a secure agent flow, expose a separate protected demonstration resource while leaving REST, MCP, A2A, and NLWeb public.

The flow uses established OAuth building blocks:

  1. GET /agent/auth without a token returns 401 with a WWW-Authenticate challenge.
  2. The challenge links to RFC 9728 protected-resource metadata at /.well-known/oauth-protected-resource.
  3. That document points to authorization-server metadata at /.well-known/oauth-authorization-server.
  4. POST /agent/identity creates a short-lived anonymous registration and signed identity assertion.
  5. POST /oauth2/token exchanges the assertion with the RFC 7523 JWT bearer grant.
  6. The agent calls /agent/auth with an audience-bound bearer access token.
  7. POST /oauth2/revoke records the access-token ID as revoked.
sequenceDiagram
  participant Agent
  participant Resource as Protected resource
  participant Metadata as OAuth metadata
  participant Identity as Identity endpoint
  participant Token as Token endpoint
  participant DB as Revocation store

  Agent->>Resource: GET /agent/auth
  Resource-->>Agent: 401 + resource_metadata
  Agent->>Metadata: Discover resource and authorization server
  Metadata-->>Agent: Registration, token, scope, audience
  Agent->>Identity: Register anonymous identity
  Identity-->>Agent: 15-minute signed assertion
  Agent->>Token: JWT bearer exchange + resource audience
  Token-->>Agent: 1-hour profile:read access token
  Agent->>Resource: Bearer access token
  Resource-->>Agent: Public profile + token metadata
  Agent->>Token: Revoke access token
  Token->>DB: Store revoked jti until expiry
  Token-->>Agent: 200

Sign assertions and access tokens with a Worker secret such as AGENT_AUTH_SIGNING_SECRET. Use short expiry times, no-store, exact audiences, minimal scopes, unique JWT IDs, and constant-time signature verification. Never place the signing secret in source or a static asset.

A compact revocation table is sufficient:

CREATE TABLE revoked_agent_tokens (
  token_jti TEXT PRIMARY KEY,
  expires_at TEXT NOT NULL,
  revoked_at TEXT NOT NULL
);

Return a standards-oriented challenge on token errors:

WWW-Authenticate: Bearer resource_metadata="https://example.com/.well-known/oauth-protected-resource", error="invalid_token"

Document the complete exchange in /auth.md and link it from llms.txt, the developer pages, and both metadata documents.

Step 15 — Sign a Web Bot Auth directory

HTTP Message Signatures provide cryptographic proof that a response was signed by the domain’s configured key. The RFC 9421 format uses Signature-Input to describe covered components and Signature to carry the signature.

Publish a Web Bot Auth directory at /.well-known/http-message-signatures-directory containing an Ed25519 public JWK, key identifier, validity period, and supported signature components. Sign the response with a private PKCS#8 key stored as a Worker secret.

flowchart LR
  Secret["Ed25519 private key<br/>Worker secret"] --> Sign["RFC 9421 signer"]
  Request["Authority + nonce + timestamps"] --> Sign
  Sign --> Response["Directory body"]
  Sign --> Input["Signature-Input"]
  Sign --> Signature["Signature"]
  Public["Published Ed25519 JWK"] --> Verifier["Third-party verifier"]
  Response --> Verifier
  Input --> Verifier
  Signature --> Verifier

Use a short-lived nonce plus created and expires parameters. Sign the request authority and the exact components specified by the directory contract. Return the exact directory media type, CORS headers, and a short cache lifetime.

Make the test import the emitted public JWK and actually verify the signature. Asserting that a Signature header exists proves only that you can set a header, which is the part that was never going to break.

Step 16 — Ship a CLI through npm and Homebrew

A CLI gives shell-capable agents a stable, auditable way to use the API without generating HTTP code. Keep it small and predictable.

The CLI used here is a dependency-free Node.js executable with commands for:

example profile
example posts --limit 5 --tag kubernetes
example post article-slug
example insight --audience recruiter --focus production-ml,kubernetes --wait

Retrieval commands issue HTTPS GET requests. The insight command creates an idempotent job, follows the returned Location, and optionally polls until a terminal state. Structured JSON goes to standard output; diagnostics and usage errors go to standard error with non-zero exit codes.

Publish a complete npm package

Include package ownership and provenance metadata:

{
	"name": "example",
	"version": "0.1.0",
	"description": "Dependency-free CLI for the example.com public API.",
	"license": "MIT",
	"type": "module",
	"bin": { "example": "example.mjs" },
	"files": ["example.mjs", "README.md", "LICENSE"],
	"engines": { "node": ">=18" },
	"repository": {
		"type": "git",
		"url": "git+https://github.com/example/homebrew-example.git",
		"directory": "cli"
	},
	"homepage": "https://example.com/developers/cli/",
	"publishConfig": { "access": "public" }
}

Resolve the executable path correctly when the package manager invokes it through a symbolic link. Test both direct execution and an installed/symlinked invocation.

Publish the source, tests, README, and license in a clean public repository. Host a versioned .tgz archive with its SHA-256 digest so package bytes can be audited independently.

Add a Homebrew formula

Create an official tap repository and formula that downloads the immutable release archive, verifies its SHA-256 checksum, installs the executable, and runs a command in the formula test block.

npm install --global example
brew install example/example/example

Link both package channels from llms.txt, developer documentation, package metadata, and the public source repository.

flowchart TB
  Source["Public CLI source + tests"] --> Npm["npm package"]
  Source --> Archive["Versioned .tgz + SHA-256"]
  Archive --> Formula["Homebrew formula"]
  Npm --> Shell["Agent shell"]
  Formula --> Shell
  Shell --> API["Same typed public API"]

Step 17 — Publish first-party SDKs across ecosystems

An OpenAPI contract lets an agent generate a client, but a small official SDK is safer and faster for repeated use. It gives developers typed inputs, structured errors, documented polling, and a package whose ownership can be verified through a familiar registry.

Keep every SDK deliberately close to the HTTP contract. The clients built for this site share the same behavior:

  • retrieve the profile, the article collection, and a single article
  • combine public reads through the batch endpoint
  • create profile-insight jobs with an idempotency key
  • poll async jobs to a terminal state
  • query conversational search
  • turn any non-success response into a structured, language-native error

Publish through the ecosystem’s normal channel:

python -m pip install example
gem install example
go get github.com/example/example-go@v0.1.0

For Python and Ruby, include the canonical homepage, documentation, source, issue tracker, license, supported runtime versions, and package contents in the registry metadata. Require MFA for registry publication and use short-lived or narrowly scoped publishing credentials. For Go, make the module path match the public repository, commit go.mod, push an immutable semantic-version tag, and confirm that the public module proxy resolves it.

flowchart TD
  Contract["OpenAPI contract"] --> Python["Python client"]
  Contract --> Ruby["Ruby client"]
  Contract --> Go["Go client"]
  Python --> PyPI["PyPI release"]
  Ruby --> RubyGems["RubyGems release"]
  Go --> Tag["Git tag + Go module proxy"]
  PyPI --> API["Versioned public API"]
  RubyGems --> API
  Tag --> API
  Tests["Unit tests + live smoke tests"] --> Python
  Tests --> Ruby
  Tests --> Go

Give each language its own small public repository. Include a concise README, an MIT license, examples, unit tests with an injected or local HTTP transport, and CI across supported runtime versions. Avoid runtime dependencies when the standard HTTP and JSON libraries are sufficient. That keeps the installation surface small and makes the code easy for both humans and agents to inspect.

The live implementation includes the Python package on PyPI, the versioned Go SDK, the Ruby package on RubyGems, and a canonical SDK installation guide. Link the guide, source, and registry pages from llms.txt, the developer index, the API discovery response, and the OpenAPI-adjacent documentation so name-based searches can find them.

Step 18 — Package reusable Agent Skills and a plugin

An Agent Skill is a small, focused instruction package for a repeatable task. Publish skills that tell an agent when to use your source, which canonical endpoints to read, how to cite results, and which inferences to avoid.

A useful SKILL.md begins with minimal frontmatter:

---
name: find-technical-writing
description: Search and cite Example Person's published technical writing.
---

# Find technical writing

1. Query https://example.com/api/v1/posts for published metadata.
2. Add a documented tag filter when the topic is known.
3. Read the canonical article before explaining its conclusions.
4. Cite the canonical URL and distinguish publication from update dates.

Keep each skill narrow. The public implementation separates profile research, production case-study research, and technical-writing retrieval.

Publish an index at /.well-known/agent-skills/index.json. Include the discovery schema, each skill’s name, type, description, URL, and SHA-256 digest. Test the digest against the exact emitted bytes.

flowchart TD
  Index["Agent Skills index"] --> Profile["Research profile skill"]
  Index --> Cases["Find case studies skill"]
  Index --> Writing["Find writing skill"]
  Profile --> FirstParty["First-party pages and API"]
  Cases --> FirstParty
  Writing --> FirstParty
  Plugin["Agent Plugin"] --> Index
  Plugin --> MCP1["Public-profile MCP"]
  Plugin --> MCP2["Product MCP"]

Keep a public repository containing the same skill files, an MIT license, a canonical Agent Plugins plugin.json, and an mcp.json that connects the remote Streamable HTTP servers. Validate both manifests against their published JSON Schemas. A public repository also makes the skills installable through directories such as skills.sh.

Step 19 — Establish registry and domain ownership

A self-published manifest becomes more trustworthy when the domain, source repository, registry record, and website point back to one another.

For the official MCP Registry, create server.json using the current registry schema:

{
	"$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
	"name": "com.example/public-content",
	"title": "Example Public Content",
	"description": "Read the public profile, case studies, and technical writing.",
	"version": "1.0.0",
	"repository": {
		"url": "https://github.com/example/example.com",
		"source": "github"
	},
	"remotes": [{ "type": "streamable-http", "url": "https://example.com/mcp" }]
}

Use a domain-owned namespace, publish the Registry’s HTTP ownership proof at /.well-known/mcp-registry-auth, and keep the private signing key in CI secrets. The official remote-server publishing guide documents the flow.

Automate registry publication with a manually triggered workflow:

  1. check out the source
  2. download the official mcp-publisher binary
  3. authenticate against the domain ownership proof
  4. query the registry for that exact name and version
  5. publish only when the immutable version is genuinely absent
  6. fail loudly on any registry response you did not expect

Also maintain a public directory listing such as Smithery when it serves your users. Link registry entries from the homepage metadata, visible developer documentation, llms.txt, Markdown alternates, and the MCP guide. Those backlinks establish bidirectional evidence.

flowchart LR
  Domain["example.com"] --> Proof[".well-known ownership proof"]
  Domain --> Card["MCP card + live endpoint"]
  Repo["Public source repository"] --> Metadata["server.json"]
  Metadata --> Publisher["mcp-publisher CI"]
  Secret["Registry private key<br/>CI secret"] --> Publisher
  Proof --> Registry["Official MCP Registry"]
  Publisher --> Registry
  Registry --> Domain
  Directory["Public directory listing"] --> Domain
  Domain --> Directory

The live sudhanva.me Registry record demonstrates the complete domain-owned remote-server relationship.

Step 20 — Deploy the stack on Cloudflare

Astro builds the static site to dist. Wrangler uploads those files as Workers Static Assets and runs the Worker first so it can negotiate representations and handle protocol routes. Cloudflare documents this routing model in Worker script routing.

{
	"$schema": "node_modules/wrangler/config-schema.json",
	"name": "example-site",
	"compatibility_date": "2026-08-23",
	"main": "worker.mjs",
	"build": { "command": "npm run build" },
	"assets": {
		"directory": "./dist",
		"binding": "ASSETS",
		"run_worker_first": true,
		"not_found_handling": "404-page",
		"html_handling": "auto-trailing-slash",
	},
	"d1_databases": [
		{
			"binding": "AGENT_DB",
			"database_name": "example-agent-data",
			"database_id": "YOUR_DATABASE_ID",
			"migrations_dir": "./migrations",
		},
	],
	"observability": { "enabled": true },
}

Create D1, apply migrations, add secrets, and deploy:

npx wrangler d1 create example-agent-data
npx wrangler d1 migrations apply example-agent-data --remote
npx wrangler secret put AGENT_AUTH_SIGNING_SECRET
npx wrangler secret put WEB_BOT_AUTH_PRIVATE_KEY
npx wrangler deploy

Cloudflare tracks applied D1 migration files in a migrations table; see the D1 migrations reference.

At runtime the Worker route order should be explicit:

flowchart TD
  Request["Incoming request"] --> Preflight{"OPTIONS?"}
  Preflight -- Yes --> Cors["Protocol-specific CORS response"]
  Preflight -- No --> Protocol{"API, MCP, NLWeb, A2A, auth?"}
  Protocol -- Yes --> Handler["Typed protocol handler"]
  Protocol -- No --> Markdown{"Markdown preferred or .md alias?"}
  Markdown -- Yes --> MdAsset["Agent Markdown asset"]
  Markdown -- No --> Static["ASSETS.fetch(request)"]
  Static --> Decorate["Add discovery Link + Vary headers"]
  Handler --> Response["Response"]
  MdAsset --> Response
  Decorate --> Response
  Cors --> Response

Set global security headers on static pages: HSTS, X-Content-Type-Options, frame protection, referrer policy, permissions policy, and the cross-origin policies. Then override content type, CORS, indexing, and caching per path for the machine-readable artifacts.

All of this fits comfortably in free-tier territory, which was not an accident. The content stays static and cacheable, the Worker holds no state on the vast majority of requests, and the only rows in D1 are small and short-lived.

Step 21 — Test the contracts, not just the pages

Agent readiness is an integration property. A file existing at the right path does not prove that its declared endpoint completes a handshake.

Build-time artifact tests

After the static build, assert that:

  • the homepage and trust pages carry an early visible H1 and real no-JavaScript text
  • JSON-LD has the entity types and stable identifiers you expect
  • llms.txt, scoped indexes, instructions, auth docs, and Markdown alternates all got emitted
  • every JSON artifact parses
  • every JSON Lines record parses on its own
  • the Schema Map points at the feed
  • each Agent Skill digest matches the bytes actually written
  • the plugin, MCP, A2A, API catalog, AI catalog, OAuth, and Registry manifests match their schemas and URLs
  • the OpenAPI document has unique operation IDs, descriptions, typed inputs, typed responses, and lifecycle metadata
  • the CLI archives have the right gzip magic bytes and SHA-256 digests

Worker protocol tests

Call the Worker directly with mocked static assets and D1. Cover:

  • Accept negotiation, q-values and wildcards included
  • the Vary, Link, Content-Type, and Content-Location headers
  • direct .md aliases and the agent-friendly 404
  • API validation, pagination, HEAD, OPTIONS, batch boundaries, and error envelopes
  • idempotent create, replay, conflict, polling, expiry, and job state transitions
  • NLWeb JSON and SSE results
  • the A2A card and a message exchange
  • auth discovery, registration, exchange, protected read, expiry, and revocation
  • cryptographic verification of the Web Bot Auth signature
  • MCP init, notifications, tool listing, tool calls, resources, the compatibility transport, modern discovery, version errors, and origin checks
  • the CLI run directly and through a symlink
flowchart LR
  Change["Source change"] --> Lint["ESLint + Markdown lint + format"]
  Lint --> Types["Astro type check"]
  Types --> Build["Static production build"]
  Build --> Artifacts["Artifact contract tests"]
  Build --> WorkerTests["Worker protocol tests"]
  Build --> CliTests["CLI tests"]
  Artifacts --> Deploy["Deploy"]
  WorkerTests --> Deploy
  CliTests --> Deploy
  Deploy --> Smoke["Public endpoint smoke tests"]
  Smoke --> Registries["Registry and backlink verification"]

Run the local quality gate:

npm run lint:all
npm run build
npm test
git diff --check

Verify production endpoints

Use curl for exact HTTP behavior:

# HTML remains the default.
curl -i https://example.com/

# The same canonical URL can return Markdown.
curl -i -H 'Accept: text/markdown' https://example.com/

# Discovery artifacts parse.
curl -fsS https://example.com/openapi.json | jq '.openapi'
curl -fsS https://example.com/.well-known/mcp.json | jq '.servers'
curl -fsS https://example.com/.well-known/agent-card.json | jq '.skills'

# API reads are typed and reachable.
curl -fsS https://example.com/api/v1/profile | jq
curl -fsS 'https://example.com/api/v1/posts?limit=3' | jq

# MCP performs a real handshake.
curl -fsS https://example.com/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  --data '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"smoke-test","version":"1.0.0"}}}' | jq

# Create and poll an idempotent asynchronous job.
curl -i https://example.com/api/v1/profile-insights \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: tutorial-2026-08-23-001' \
  --data '{"audience":"recruiter","focus":["production-ml"]}'

For Server-Sent Events, inspect the event stream instead of piping it directly into jq. For HEAD, verify that the body length is zero while the response headers match GET semantics.

Finally, verify external records from a separate network path: open the official registry entry, follow its remote URL, follow the website backlink to the registry, and run the same initialize request a third-party host would send.

Implementation checklist

Use this order when adapting the design to another site:

Content and discovery

  • Prerender meaningful semantic HTML with an early product-named H1.
  • Add canonical metadata, Open Graph fields, sitemap, and RSS.
  • Generate Schema.org JSON-LD from canonical typed data.
  • Publish an intentional robots.txt policy.
  • Publish llms.txt, scoped indexes, and an agent instruction file.
  • Add negotiated Markdown plus direct .md aliases.
  • Advertise alternates, docs, and API descriptions with HTML and HTTP links.
  • Publish the RFC 9727 API catalog and an agentic resource catalog.
  • Publish agent-data.json, a Schema.org JSON Lines feed, and a Schema Map.

Callable interfaces

  • Publish an OpenAPI 3.1 document with unique operation IDs and complete schemas.
  • Implement typed reads, deterministic pagination, HEAD, OPTIONS, CORS, and structured errors.
  • Publish a URL-major versioning and deprecation policy.
  • Add a bounded, allowlisted batch endpoint.
  • Add a real asynchronous job with idempotency, polling, terminal states, and expiry.
  • Implement MCP Streamable HTTP with a real handshake, tools, resources, annotations, and cards.
  • Implement grounded NLWeb JSON and SSE search.
  • Publish an A2A Agent Card and message-send route.

Trust and distribution

  • Publish OAuth protected-resource and authorization-server metadata for protected agent resources.
  • Use short-lived, audience-bound credentials and a revocation store.
  • Publish and cryptographically test a signed Web Bot Auth directory.
  • Ship a tested CLI through npm and Homebrew with public source and immutable checksums.
  • Publish small, tested SDKs through two or more language ecosystems and link their source.
  • Publish narrow Agent Skills with verified digests and a portable Agent Plugin.
  • Publish domain-owned MCP Registry metadata and ownership proof.
  • Link the site, source, protocol cards, directories, and registries bidirectionally.

Operations

  • Keep all representations connected to one canonical content source.
  • Set exact content types, CORS, caching, and indexing headers by artifact.
  • Store secrets only in the deployment platform or CI secret store.
  • Test artifact contents, live protocol behavior, cryptographic verification, and installed CLI execution.
  • Run production smoke tests after every deployment.

Closing principles

If I had to compress all of this into one idea: agreement beats surface area. Three protocols that tell an identical story are worth more than nine that quietly contradict each other.

Semantic HTML is still the floor, and Markdown just makes it cheap to read. OpenAPI and a small SDK are what turn retrieval from guesswork into a function call. Idempotency is what makes an action safe to retry, which matters more than it sounds like it should, because agents retry constantly. MCP, NLWeb, and A2A are three different front doors for three different kinds of host, and you probably do not need all three on day one. Almost everyone skips the trust layer: OAuth metadata, signatures, public source, registry records. That is the part that decides whether a cautious client picks you at all.

The tests are what hold it together. Every claim in this post is a claim some future refactor will quietly break, and the only reason I trust the stack a month later is that a failing assertion tells me before a user does.

Build it in roughly this order, generate everything from one source, and hit every endpoint you advertise with a real request before you call it done. Done right, a person cannot tell any of it is there.

Production ML context

See how this topic connects to production ML systems, infrastructure, and inference.

Keep reading

All posts →

Bare Metal Kubernetes Homelab Setup

A self-hosted bare-metal Kubernetes platform on Ubuntu 24.04 LTS. Combines Ansible for automated node provisioning with ArgoCD for GitOps-based cluster management.

#devops #kubernetes #software-engineering #microservices #docker

Chat with SQLite Database

Chat with SQLite Database

#portfolio #ai #openai

LangChain Chat and Chroma Embeddings Integration

LangChain Chat and Chroma Embeddings Integration

#ai #openai #portfolio