Making Your Site Agent Friendly · Sudhanva Narayana

Most advice about AI and websites stops at “make sure the crawler can read your text.” The interesting engineering problems start after an agent reads the page and wants to act on it, such as negotiating the right representation, validating a schema, invoking a tool or API endpoint, interpreting structured error recovery hints, and cryptographically verifying that the capability actually belongs to the domain claiming it.

This guide documents the exact agent-readiness architecture running in production on sudhanva.me today. I wrote it as a reference blueprint you can lift directly into your own stack. Swap out example.com, the sample profile fields, and the tool names for your own domain. None of the patterns assume a personal portfolio; the exact same contracts apply whether you are running a developer documentation portal, a SaaS marketing site, or a public product API.

The underlying infrastructure uses standard components: a static Astro build, a single Cloudflare Worker at the edge, static assets, and a lightweight D1 SQLite database used exclusively for short-lived asynchronous job state and revoked OAuth token IDs. The real engineering work lies in making every machine-facing contract explicit, deterministic, and continuously tested.

What agent-friendly means
Reference architecture
Step 1: Start with semantic, server-rendered content
Step 2: Publish an intentional crawler policy
Step 3: Add llms.txt and scoped discovery files
Step 4: Serve HTML and Markdown from the same URL
Step 5: Advertise resources with standard links and catalogs
Step 6: Generate one machine-readable content corpus
Step 7: Publish a typed, function-callable API
Step 8: Make versioning and deprecation predictable
Step 9: Add a bounded batch endpoint
Step 10: Implement a real idempotent asynchronous job
Step 11: Expose capabilities through MCP
Step 12: Add NLWeb conversational search
Step 13: Add an A2A agent interface
Step 14: Publish agent-native authentication
Step 15: Sign a Web Bot Auth directory
Step 16: Ship a CLI through GitHub Packages and Homebrew
Step 17: Publish first-party SDKs across ecosystems
Step 18: Package reusable Agent Skills and a plugin
Step 19: Establish registry and domain ownership
Step 20: Deploy the stack on Cloudflare
Step 21: Test the contracts and the pages
Implementation checklist

What agent-friendly means

It helps to think of agent readiness as five progressive layers of capability:

Discoverable. An agent can locate the site, its documentation, API schemas, callable tools, and domain ownership proofs starting from nothing more than the domain or product name.
Readable. Every core fact lives in clean, semantic HTML and negotiated Markdown, with none of it locked behind client-side JavaScript hydration or layout shifts.
Callable. Programmatic actions expose stable URLs, strict JSON Schema or OpenAPI 3.1 input contracts, unique operation IDs, bounded payloads, and machine-parseable error envelopes.
Trustworthy. Canonical URLs, publisher metadata, provenance links, OAuth discovery documents, HTTP message signatures, and public registry records all cross-reference and agree with one another.
Operable. Retries are idempotent, long-running work uses an idempotent asynchronous job state machine, API versions hold still, clients can batch read requests, and automated contract tests guard every surface on every deploy.

flowchart TB
  Discover["Discoverable<br/>robots, llms.txt, catalogs"]
  Read["Readable<br/>semantic HTML, Markdown, JSON-LD"]
  Call["Callable<br/>OpenAPI, REST, MCP, NLWeb, A2A"]
  Trust["Trustworthy<br/>identity, signatures, registries"]
  Operate["Operable<br/>idempotency, jobs, versions, tests"]

  Discover --> Read
  Read --> Call
  Call --> Trust
  Trust --> Operate

Do not try to bolt on every protocol in this guide at once. Start with a single canonical source of truth in your repository and one capability that is worth calling programmatically. Generate every downstream artifact (HTML pages, Markdown mirrors, JSON-LD, OpenAPI payloads, and MCP responses) from that single source of truth. If you maintain separate hand-edited copies for your website, API, CLI, and agent tools, they will drift apart, and you will only find out when an agent cites a stale contract back to a user.

Reference architecture

At build time, Astro compiles all content and discovery manifests into static files; at runtime, a single Cloudflare Worker sits in front of those static assets to handle content negotiation and live protocol endpoints. Because almost every read request resolves to a compiled static asset or in-memory corpus lookup, latency stays low and edge caching works naturally. The Worker touches D1 only where mutable state is required (tracking idempotent asynchronous job records and storing revoked OAuth access-token IDs (jti)).

flowchart LR
  subgraph Source["Repository source"]
    Content["Typed content + Markdown"]
    Schemas["OpenAPI + discovery manifests"]
    WorkerCode["Edge protocol handlers"]
    Tests["Contract tests"]
  end

  subgraph Build["Astro build"]
    HTML["Semantic HTML"]
    MD["Agent Markdown"]
    Data["agent-data.json"]
    Feeds["RSS, sitemap, schema feed"]
  end

  subgraph Edge["Cloudflare Worker"]
    Router["Representation + protocol router"]
    Assets["Static Assets binding"]
    DB[("D1")]
  end

  subgraph Consumers["Consumers"]
    Browser["Browsers"]
    Search["Search and answer engines"]
    Agent["AI agents"]
    CLI["CLI and scripts"]
    Registry["Public registries"]
  end

  Content --> Build
  Schemas --> Build
  WorkerCode --> Router
  Tests --> Build
  Build --> Assets
  Router --> Assets
  Router --> DB
  Browser --> Router
  Search --> Router
  Agent --> Router
  CLI --> Router
  Registry --> Router

Human visitors in a browser get the exact same fast, visual site as before. When a request arrives at the edge, the Worker inspects the path and headers first (handling Accept: text/markdown negotiation, REST API routes, MCP JSON-RPC calls, NLWeb queries, A2A messages, and OAuth token exchanges) before falling through to the static asset binding for normal page loads.

Step 1: Start with semantic, server-rendered content

An agent fetching your URL with a plain HTTP client should be able to extract every important fact from the raw HTML response without executing a single line of JavaScript or spinning up a headless browser. Static site generation (SSG) or server-side rendering (SSR) makes that guarantee structural rather than accidental.

On every important page across your site:

Put an unambiguous product, organization, or person name in both <title> and the visible <h1>.
Emit the primary heading and lead summary early in the HTML document body rather than burying them beneath megabytes of inline state.
Use native semantic elements (<main>, <article>, <section>, <h2> and <h3>, <p>, <ul>, <table>, <nav>) instead of generic <div> soup.
Declare a single authoritative <link rel="canonical"> URL.
Provide page-specific meta descriptions and Open Graph tags.
Ensure all primary text, tables, and links remain present in the initial HTML payload without client-side scripts.
Give distinct concepts their own dedicated, linkable URLs rather than hiding everything behind tabs or modals on a single page.

When refactoring my own site, I split a dense single-page portfolio into dedicated, focused routes for the profile, resume, work index, individual production case studies, domain expertise, technical writing, contact, privacy policy, and per-protocol developer pages. At first that felt like overkill for a personal site, until I watched automated evaluators and retrieval agents resolve entity queries by matching page titles and <h1> headings directly to URLs.

Add structured data to the same pages

Pair your semantic HTML with Schema.org JSON-LD so entity types and relationships are machine-unambiguous. Depending on the page, a technical site can emit:

WebSite for site-wide identity and search actions
Organization for the publishing entity
Person for the author or subject
BlogPosting or TechArticle for engineering posts
CreativeWork for architecture case studies
APIReference or WebAPI for developer documentation
BreadcrumbList for hierarchical navigation context

Assign stable @id URIs to core entities (such as https://example.com/#person or https://example.com/#org) and reference those @id nodes from every child page so an agent or knowledge-graph parser sees a connected graph rather than disconnected duplicate strings:

{
	"@context": "https://schema.org",
	"@type": "Person",
	"@id": "https://example.com/#person",
	"name": "Example Person",
	"url": "https://example.com/",
	"sameAs": ["https://github.com/example", "https://www.linkedin.com/in/example"],
	"knowsAbout": ["Machine learning", "Kubernetes", "Platform engineering"]
}

Always generate your JSON-LD payloads from the exact same TypeScript/content modules that render the visible HTML page. That single habit prevents a common failure mode where a visual resume or pricing table gets updated in a redesign while a forgotten JSON-LD block in the layout header continues serving last year’s data.

flowchart LR
  Typed["Typed profile and content"] --> Page["Visible page copy"]
  Typed --> JsonLd["Schema.org JSON-LD"]
  Typed --> Api["API payload"]
  Typed --> Markdown["Markdown alternate"]
  Typed --> Feed["Schema feed"]

Verify the raw document

After building the site, inspect the emitted static HTML directly on disk rather than relying on DevTools in a hydrated browser tab:

npm run build
rg -n '<title|<h1|application/ld\+json|rel="canonical"' dist/index.html

In our test suite, a build-time check strips all <script>, <style>, and HTML tags from every key route and asserts that the remaining plain text contains both an <h1> and a substantial character count of readable prose.

Step 2: Publish an intentional crawler policy

Publish an explicit /robots.txt that distinguishes between the different classes of automated traffic hitting your domain, including traditional search indexers, AI answer engines, user-initiated retrieval agents, and bulk foundation-model training scrapers.

Rather than leaving everything to a blanket wildcard, name the specific search, assistant, and training user agents you want to welcome, and explicitly disallow the bulk scrapers you don’t. For example, this policy welcomes OpenAI, Anthropic, Perplexity, Google, Apple, and Bing bots while blocking Common Crawl (CCBot):

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Bingbot
Allow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap-index.xml
# Schema Feeds map (experimental): https://example.com/schemamap.xml

Revisit your user-agent list periodically as new agent crawlers launch. If you manage multiple properties at the edge, Cloudflare also provides managed robots.txt policies and content signals.

One practical detail with robots.txt: while the experimental Schemamap specification proposes a top-level Schemamap: directive in robots.txt alongside Sitemap:, strict robots.txt validators (including Lighthouse’s SEO audit) flag unknown directives as syntax errors and drop your score below 100. Keeping the schema map URL as a comment in robots.txt while linking /schemamap.xml explicitly from llms.txt and /.well-known/ai-catalog.json gives schema-aware agents the pointer they need without tripping strict robots.txt parsers or requiring Vary: User-Agent branching.

Step 3: Add llms.txt and scoped discovery files

Publish /llms.txt at the root of your domain as a concise, high-signal entrypoint for language-model clients. The llms.txt specification uses standard Markdown because both human engineers and LLMs can parse its hierarchy at a glance with minimal token overhead.

Dumping a raw sitemap of 500 bare URLs into llms.txt wastes the format. Instead, structure /llms.txt around the decisions an agent actually needs to make:

the canonical product, organization, or person name in the # H1
a tight one-paragraph summary of what the site is
authoritative ground-truth facts and scope boundaries
links to primary content sections and their .md mirrors
explicit “when to use this site” routing guidance
exact rules for calling the public REST API and MCP servers
links to OpenAPI, MCP, A2A, NLWeb, CLI, and SDK documentation
machine-readable feeds (rss.xml, sitemap-index.xml, schema-feed.jsonl) and citation preferences
clear safety boundaries stating what private or employer-confidential data the site does not hold

# example.com

> Public documentation and technical writing for Example Product.

## When to use this site

Use example.com to retrieve first-party product documentation, public release
information, and supported API examples.

## How agents should call the site

- Read https://example.com/openapi.json before generating API calls.
- Use GET https://example.com/api/v1/articles for article metadata.
- Prefer Accept: text/markdown on human-facing documentation pages.
- Parse non-2xx responses from the documented error object.

## Developer and agent resources

- [Developer guide](https://example.com/developers/)
- [OpenAPI](https://example.com/openapi.json)
- [MCP guide](https://example.com/developers/mcp/)
- [MCP discovery](https://example.com/.well-known/mcp.json)

If your site has deep documentation or developer subtrees, publish scoped discovery files alongside the root index, such as /docs/llms.txt and /developers/llms.txt. Scoped indexes let an agent working on an API integration load only the developer tree without burning context window tokens on unrelated blog posts or marketing pages.

Pair llms.txt with /agent-instructions.md. While llms.txt answers “what exists on this domain and where is it?”, /agent-instructions.md defines operational norms such as preferred call order (for example, checking structured endpoints before scraping HTML), retry and idempotency rules, canonical date attribution, and strict guardrails against fabricating unstated facts. Similarly, publishing a /docs/sandbox/ page gives automated evaluators and cautious agents explicit safe test inputs, rate expectations, and confirmation that read and demo endpoints require no destructive cleanup.

flowchart TD
  Domain["example.com"] --> Robots["robots.txt"]
  Domain --> Llms["llms.txt"]
  Llms --> DocsIndex["docs/llms.txt"]
  Llms --> DevIndex["developers/llms.txt"]
  Llms --> Instructions["agent-instructions.md"]
  DevIndex --> OpenAPI["openapi.json"]
  DevIndex --> MCP[".well-known/mcp.json"]
  DevIndex --> Auth["auth.md"]
  DevIndex --> CLI["CLI documentation"]

Step 4: Serve HTML and Markdown from the same URL

Human readers want styled, interactive HTML; language-model agents almost always do better with clean, token-efficient Markdown stripped of navigation chrome and layout markup. Instead of forcing agents onto a separate subdomain, keep a single canonical URL and use standard HTTP content negotiation (Accept: text/markdown) to serve the right representation.

Make sure your edge router parses Accept media ranges and q quality weights properly rather than running a naive .includes('text/markdown') substring check. For instance, a client sending this header explicitly prefers HTML (q=0.9) over Markdown (q=0.4), even though text/markdown appears first in the string:

Accept: text/markdown;q=0.4, text/html;q=0.9

A compact implementation looks like this:

function parseAccept(value = '*/*') {
	return value
		.split(',')
		.map((part, index) => {
			const [mediaType, ...parameters] = part.trim().split(';');
			const quality = parameters
				.map((parameter) => parameter.trim().match(/^q=([0-9.]+)$/i))
				.find(Boolean)?.[1];

			return {
				mediaType: mediaType.toLowerCase(),
				quality: quality === undefined ? 1 : Number(quality),
				index,
			};
		})
		.filter((entry) => Number.isFinite(entry.quality) && entry.quality > 0)
		.sort((a, b) => b.quality - a.quality || a.index - b.index);
}

function prefersMarkdown(request) {
	const accepted = parseAccept(request.headers.get('Accept'));
	const markdown = accepted.findIndex((item) => item.mediaType === 'text/markdown');
	const html = accepted.findIndex((item) => item.mediaType === 'text/html');
	return markdown !== -1 && (html === -1 || markdown < html);
}

Keep a route map from canonical page paths to generated Markdown assets. If Markdown wins, fetch the mapped static asset. Return it with these headers:

Content-Type: text/markdown; charset=utf-8
Content-Location: /index.md
Vary: Accept, Accept-Encoding, User-Agent
Link: <https://example.com/>; rel="alternate"; type="text/html"

For the HTML response, advertise the Markdown alternate:

Link: <https://example.com/index.md>; rel="alternate"; type="text/markdown"

Add the equivalent relationship to <head>:

<link rel="alternate" type="text/markdown" href="https://example.com/index.md" />

Also expose direct .md path aliases (like /about.md or /blog/posts/slug.md) for lightweight tools and curl scripts that cannot customize request headers. On those .md responses, emit X-Robots-Tag: noindex and a <link rel="canonical"> header pointing back to the HTML URL so search engines index only the primary page.

For known retrieval-agent User-Agent strings, the edge Worker can also default to the Markdown representation automatically when the client sends a generic Accept: */*, while supporting an explicit ?mode=agent query parameter for quick browser inspection. Because the representation decision can depend on either Accept or User-Agent, always emit Vary: Accept, Accept-Encoding, User-Agent so CDN and browser caches never serve raw Markdown to a human tab or full HTML to a Markdown client.

sequenceDiagram
  participant C as Client
  participant W as Edge Worker
  participant A as Static assets

  C->>W: GET /docs/ with Accept header
  W->>W: Parse media types and q-values
  alt Markdown preferred
    W->>A: Fetch /_agent/docs.md
    A-->>W: Markdown bytes
    W-->>C: text/markdown + Content-Location + Vary
  else HTML preferred
    W->>A: Fetch /docs/index.html
    A-->>W: Prerendered HTML
    W-->>C: text/html + Link rel=alternate
  end

When an agent hits a 404 Not Found with Accept: text/markdown (or an agent User-Agent), return a concise Markdown recovery document pointing to /llms.txt, /docs/, /ask, and /openapi.json instead of a decorative HTML 404 page. For deeper background on content negotiation for agents, see the Accept: text/markdown guide.

Add a declarative browser tool

Standard server-rendered HTML <form> elements can also advertise themselves to browser-based agents as declarative WebMCP tools. Keep the form 100% functional as a normal HTML GET/POST submission for humans, and layer on the toolname, tooldescription, and toolparamdescription attributes:

<form
	action="/search/"
	method="get"
	toolname="search_example_site"
	tooldescription="Search Example Product documentation and technical writing."
>
	<label for="agent-site-search">Search example.com</label>
	<input
		id="agent-site-search"
		name="q"
		type="search"
		required
		toolparamdescription="Words or phrases to search for."
	/>
	<button type="submit">Search site</button>
</form>

Because this is pure progressive enhancement, regular browsers render a normal accessible search box while WebMCP-aware browser agents discover a typed tool backed by the exact same /search/ route. (On sudhanva.me, I register site search through the imperative navigator.modelContext WebMCP API instead, so I omit the duplicate form attributes to avoid presenting two identically purposed tools to the same browser agent.)

Step 5: Advertise resources with standard links and catalogs

Discovery should not depend on one convention. Advertise important resources from HTML, response headers, predictable .well-known paths, and dedicated catalogs.

Add discovery links to HTML

<link rel="service-desc" type="application/vnd.oai.openapi+json;version=3.1" href="/openapi.json" />
<link rel="service-doc" type="text/html" href="/docs/" />
<link rel="alternate" type="text/markdown" href="/docs.md" />
<link rel="alternate" type="application/rss+xml" href="/rss.xml" />

Repeat service-desc, service-doc, and lifecycle links in HTTP Link headers on API responses. Then the contract is visible even if the client never opens the homepage.

Publish an RFC 9727 API catalog

RFC 9727 defines an API catalog as a linkset. Put the JSON at /.well-known/api-catalog. Connect it to the documentation, OpenAPI description, version policy, and live interfaces:

{
	"linkset": [
		{
			"anchor": "https://example.com/.well-known/api-catalog",
			"service-desc": [
				{
					"href": "https://example.com/openapi.json",
					"type": "application/vnd.oai.openapi+json;version=3.1"
				}
			],
			"service-doc": [{ "href": "https://example.com/docs/", "type": "text/html" }],
			"deprecation": [
				{ "href": "https://example.com/developers/versioning/", "type": "text/html" }
			],
			"item": [
				{ "href": "https://example.com/api/v1", "type": "application/json" },
				{ "href": "https://example.com/mcp", "type": "application/json" }
			]
		}
	]
}

Publish an agentic resource catalog

An additional /.well-known/ai-catalog.json can list the API, documentation, MCP servers, and A2A agent in one domain-owned catalog. Each entry should include:

a stable identifier under your domain
display name, description, media type, and URL
concrete capabilities
publisher identity
provenance back to source
privacy-policy URL
representative user queries

The catalog helps an agent choose among several surfaces. The public sudhanva.me catalog is a complete example.

Send exact media types

Machine-readable files should not fall through to a generic text/plain or application/octet-stream. Set explicit content types and CORS. For example:

OpenAPI 3.1: application/vnd.oai.openapi+json with the version=3.1 parameter
Markdown: text/markdown; charset=utf-8
JSON Lines feed: application/x-jsonlines
MCP server card: application/mcp-server-card+json
A2A Agent Card: application/a2a-agent-card+json, the type the AI catalog uses for agent cards
A2A messages: application/a2a+json
Problem details: application/problem+json

Use Access-Control-Allow-Origin: * on public machine-readable resources, so that browser-based tools can inspect them. Apply sensible cache policies, with long immutable cache lifetimes for versioned package archives, shorter lifetimes for discovery metadata, and no-store for tokens and job mutations.

Step 6: Generate one machine-readable content corpus

Create a build-time agent-data.json from the same typed content collections that render the site. Include only published, public information:

{
	"profile": {
		"name": "Example Person",
		"specialization": "Production ML Systems",
		"links": []
	},
	"caseStudies": [],
	"posts": []
}

Then derive all retrieval surfaces from this artifact:

REST profile and article responses
MCP tools and resources
NLWeb results
A2A responses
asynchronous insight results
a Schema.org JSON Lines feed
verification fixtures

flowchart TB
  Source["Typed site data"] --> AgentData["agent-data.json"]
  AgentData --> REST["REST API"]
  AgentData --> MCP["MCP tools + resources"]
  AgentData --> NLWeb["NLWeb /ask"]
  AgentData --> A2A["A2A agent"]
  AgentData --> Jobs["Insight jobs"]
  AgentData --> SchemaFeed["Schema.org JSON Lines"]

Publish the structured corpus as newline-delimited Schema.org JSON objects at /schema-feed.jsonl, where every line parses independently and carries @context, @type, and a canonical @id or url. Point to that feed from /schemamap.xml, link it in your discovery catalogs, and include a comment pointer in /robots.txt. Alongside those newer agent feeds, continue generating /rss.xml and /sitemap-index.xml; agent discovery should layer cleanly on top of existing web standards rather than replacing them.

Step 7: Publish a typed, function-callable API

Giving agents a small, well-typed REST API eliminates the fragility of scraping DOM trees. Describe the entire surface with OpenAPI 3.1 at /openapi.json so both human developers and LLM tool-calling runtimes can inspect a single language-neutral contract.

In our reference architecture, the public API surface stays compact and purposeful:

Method	Path	Purpose
`GET`	`/api/v1`	Capability and lifecycle index
`GET`	`/api/v1/profile`	Structured public profile
`GET`	`/api/v1/posts`	Paginated article metadata
`GET`	`/api/v1/posts/{slug}`	One article record
`POST`	`/api/v1/batch`	Up to 20 safe read operations
`POST`	`/api/v1/profile-insights`	Create an idempotent job
`GET`	`/api/v1/profile-insights/{job_id}`	Poll a job
`GET` or `POST`	`/ask`	NLWeb search

Make every operation usable as a function

For an OpenAPI 3.1 operation to compile cleanly into an OpenAI, Anthropic, or Gemini function-calling tool definition, every route needs:

a unique, camelCase or snake_case operationId that never changes across patch releases
a crisp summary and behavioral description
explicit JSON Schema types for all path, query, header, and request-body parameters
tight validation bounds (enum, pattern, minimum, maximum, maxItems, maxLength)
additionalProperties: false on request bodies so hallucinated parameter names fail fast with a clear validation message
typed 2xx and 4xx/5xx response schemas
concrete examples that validate against those schemas

paths:
  /posts:
    get:
      operationId: listPublishedPosts
      summary: List published articles
      description: Returns canonical metadata for published public articles.
      parameters:
        - name: limit
          in: query
          schema:
            type: integer
            minimum: 1
            maximum: 50
        - name: tag
          in: query
          schema:
            type: string
      responses:
        '200':
          description: A page of published article metadata.

Strict parameter schemas are what make an OpenAPI spec callable by an agent without guesswork. Writing prose descriptions over an untyped type: object forces the model to guess field names and types, and sooner or later it will guess wrong.

Normalize HTTP behavior

Support GET, HEAD, and OPTIONS consistently across every read endpoint: HEAD should return the exact same headers as GET with an empty body, and OPTIONS preflight responses should advertise the exact Allow and Access-Control-Allow-Methods sets supported by that route.

Include discovery and rate-limit policy headers on API responses:

Cache-Control: public, max-age=300
Link: <https://example.com/docs/>; rel="service-doc"; type="text/html"
Link: <https://example.com/openapi.json>; rel="service-desc"; type="application/vnd.oai.openapi+json;version=3.1"
Link: <https://example.com/developers/versioning/>; rel="deprecation"; type="text/html"
RateLimit-Policy: "public";q=120;w=60

Emit RateLimit-Policy to communicate your advisory quota window, and only emit a live RateLimit remaining-tokens header if your edge actually tracks per-client counters. Returning a hardcoded remaining=119 on every request misleads adaptive clients; if you don’t maintain per-IP counters in Durable Objects or rate-limiting bindings, publish only the policy header.

When a request fails, return a structured error object that tells the agent how to recover:

{
	"error": {
		"code": "POST_NOT_FOUND",
		"message": "No published article has that slug.",
		"hint": "List valid slugs with GET /api/v1/posts.",
		"docs_url": "https://example.com/docs/"
	}
}

For input validation errors on state-changing or asynchronous routes, RFC 9457 application/problem+json (type, title, status, detail) works equally well. And on paginated list endpoints, always return an explicit next_cursor field (null when exhausted) so callers never have to guess whether another page exists from the array length.

Step 8: Make versioning and deprecation predictable

Use a major version in the API URL, such as /api/v1. Publish a lifecycle policy that says:

additive fields and endpoints can ship in the current major version
documented meanings do not change in that version
breaking changes get a new major path
deprecations get a documented notice period
headers and docs announce scheduled removals

Link the policy from OpenAPI, the API catalog, developer docs, llms.txt, and every API response.

When you schedule a route for retirement, emit standards-oriented lifecycle metadata:

Deprecation: @1798761600
Sunset: Fri, 01 Jan 2027 00:00:00 GMT
Link: <https://example.com/developers/versioning/>; rel="deprecation"

Generate the dates from the published schedule, so that the documentation and headers cannot drift apart.

timeline
  title API lifecycle
  Stable v1 : Additive fields and new endpoints
  Deprecation announced : Policy page updated : Deprecation header emitted
  Migration window : Old and new major versions coexist
  Sunset : Sunset header date reached : Retired route returns documented error

Step 9: Add a bounded batch endpoint

Agents frequently need a profile and several article records. A batch endpoint decreases round trips. But it must not become an open proxy.

Accept a small array of operations:

{
	"operations": [
		{ "id": "profile", "method": "GET", "path": "/api/v1/profile" },
		{ "id": "posts", "method": "GET", "path": "/api/v1/posts?limit=5" }
	]
}

Apply these constraints:

limit the request to a documented number, for example 20 operations
allow relative paths only
allow GET only
keep an explicit route allowlist
reject external URLs and protocol-relative paths
accept no caller-supplied headers
return a status plus a result or error for every operation

flowchart TD
  Request["POST /api/v1/batch"] --> Count{"1-20 operations?"}
  Count -- No --> Reject["400 structured error"]
  Count -- Yes --> Relative{"Relative GET path?"}
  Relative -- No --> Reject
  Relative -- Yes --> Allowed{"Allowlisted route?"}
  Allowed -- No --> ItemError["Per-item error"]
  Allowed -- Yes --> Internal["Call internal read handler"]
  Internal --> Result["Per-item status + body"]
  ItemError --> Response["Batch response"]
  Result --> Response

The safest implementation calls the same internal handlers that the public API uses. Do not send arbitrary network requests from user-supplied batch paths.

Step 10: Implement a real idempotent asynchronous job

You don’t need an expensive GPU workload to justify an asynchronous job endpoint, and you shouldn’t bolt on a slow, ungrounded LLM call just to fake latency. On sudhanva.me, POST /api/v1/profile-insights synthesizes structured evidence from the canonical corpus for a requested audience (recruiter, Principal-engineer, founder, etc.) and focus areas. The synthesis itself is fast and deterministic, and the endpoint implements a genuine, database-backed idempotent asynchronous job lifecycle with caller-supplied Idempotency-Key deduplication, conflict detection when a key is reused with different parameters, 202 Accepted with Location and Retry-After headers, state transitions persisted in Cloudflare D1 across restarts, and automatic 24-hour TTL expiration.

Define a narrow request

Constrain the request schema to validated enums rather than accepting arbitrary free-form text prompts:

{
	"audience": "recruiter",
	"focus": ["production-ml", "kubernetes"]
}

Cap the request byte length, reject unknown fields (additionalProperties: false), bound the focus array length, refuse any personally identifiable or secret payloads, and require an Idempotency-Key header matching a strict length and character pattern.

Store idempotency and state in D1

CREATE TABLE profile_insights (
  job_id TEXT PRIMARY KEY,
  idempotency_key_hash TEXT NOT NULL UNIQUE,
  request_hash TEXT NOT NULL,
  audience TEXT NOT NULL,
  focus_json TEXT NOT NULL,
  status TEXT NOT NULL
    CHECK (status IN ('queued', 'running', 'succeeded', 'failed')),
  result_json TEXT,
  error_json TEXT,
  created_at TEXT NOT NULL,
  updated_at TEXT NOT NULL,
  expires_at TEXT NOT NULL
);

CREATE INDEX profile_insights_expires_at_idx
  ON profile_insights (expires_at);

By hashing the normalized Idempotency-Key and hashing the canonicalized JSON request body separately, the Worker guarantees four clean behaviors:

Fresh key: inserts a new queued row, schedules background completion via ctx.waitUntil(), and returns 202 Accepted.
Same key + identical request payload: returns the existing job record (with Idempotency-Replayed: true) without creating duplicate work.
Same key + mutated request payload: rejects the collision immediately with 422 Unprocessable Entity (application/problem+json).
Concurrent race with the same key: catches the UNIQUE constraint violation on idempotency_key_hash, reads back the winning row from D1, and verifies the request hash before replaying.

sequenceDiagram
  participant C as Agent
  participant W as Worker
  participant D as D1
  participant B as Background task

  C->>W: POST /profile-insights + Idempotency-Key
  W->>W: Validate body and hash key/request
  W->>D: Insert queued job
  alt New key
    D-->>W: Job inserted
    W->>B: ctx.waitUntil(process job)
    W-->>C: 202 + Location + Retry-After
    B->>D: queued -> running -> succeeded
  else Same key and same request
    D-->>W: Existing job
    W-->>C: Existing job + Idempotency-Replayed: true
  else Same key and different request
    D-->>W: Request hash differs
    W-->>C: 422 problem detail
  end
  C->>W: GET returned Location
  W->>D: Read job
  D-->>W: Current state or result
  W-->>C: Job representation

Return the creation response with the protocol details that a client needs:

HTTP/1.1 202 Accepted
Location: https://example.com/api/v1/profile-insights/pi_0123abcd...
Retry-After: 1
Idempotency-Key: caller-generated-value

Use a state machine with terminal results and an expiry:

stateDiagram-v2
  [*] --> queued
  queued --> running
  running --> succeeded
  running --> failed
  queued --> failed
  succeeded --> expired: 24-hour TTL
  failed --> expired: 24-hour TTL
  expired --> [*]

On Cloudflare Workers, ctx.waitUntil() lets short background work continue after the 202 response. Cloudflare documents the lifecycle and time limit in the Context API reference. For longer or retry-heavy work, keep the same API contract. Move the work to a queue or durable workflow.

Run cleanup opportunistically or from a scheduled handler, so that expired rows do not pile up. The work itself can stay tiny. But the same key must never run twice, and the status that a client polls must be the real one.

Step 11: Expose capabilities through MCP

The Model Context Protocol gives agents a native way to discover and call tools. Use Streamable HTTP for remote servers.

Split retrieval and action surfaces

Keep read-only content tools separate from state-changing product tools when that improves safety and selection:

Surface	Purpose	Capabilities
`/mcp`	Public research	Profile, articles, case studies, resources
`/mcp/product`	Product action	Capabilities, create insight, poll insight

The public research server exposes:

get_public_profile
list_published_articles
list_case_studies

The product server exposes:

get_profile_insight_capabilities
create_profile_insight
get_profile_insight

The action tools call the same internal job handlers as /api/v1/profile-insights. This keeps validation, idempotency, storage, and errors consistent.

Publish discovery and server cards

Create /.well-known/mcp.json with server names, display names, endpoint URLs, transport, authentication mode, versions, cards, capabilities, and tool summaries. Publish a detailed card for each server at predictable URLs such as:

/.well-known/mcp/server-card.json
/.well-known/mcp/product-server-card.json
aliases such as /.well-known/mcp, /mcp.json, and /mcp/server-card for clients that look in the older places

The card should match the live server. Test that its tool names, endpoint, and version match the result of a real protocol handshake.

Make tool schemas unambiguous

Every tool needs a clear title, description, JSON Schema input, and behavioral annotations:

{
	"name": "list_published_articles",
	"title": "List published articles",
	"description": "List canonical public article metadata or retrieve one article by slug.",
	"inputSchema": {
		"$schema": "https://json-schema.org/draft/2020-12/schema",
		"type": "object",
		"properties": {
			"limit": { "type": "integer", "minimum": 1, "maximum": 50 },
			"tag": { "type": "string" },
			"slug": { "type": "string" }
		},
		"additionalProperties": false
	},
	"annotations": {
		"readOnlyHint": true,
		"destructiveHint": false,
		"idempotentHint": true,
		"openWorldHint": false
	}
}

Return both human-readable content and typed structuredContent. A chat client renders the first one. Programmatic clients read the second. They do not have to parse your prose back into fields.

Implement the complete request lifecycle

A stateless Streamable HTTP server should support:

OPTIONS for CORS
POST for JSON-RPC
initialize
accepted client notifications
tools/list and tools/call
resources/list and resources/read, if you advertise resources at all
structured JSON-RPC errors
protocol-version negotiation

Validate Origin to decrease the risk of DNS rebinding. A private or authenticated server should allow only the HTTPS origins that it intends to support. A public, unauthenticated, read-only server like this one can be looser. It accepts requests with no Origin, from any https:// origin, and from http://localhost or http://127.0.0.1 for local inspectors. It returns 403 for all other requests. Whichever rule you pick, write it down in your MCP docs.

sequenceDiagram
  participant H as MCP host
  participant S as /mcp
  participant API as Shared API logic
  participant Data as agent-data.json

  H->>S: initialize
  S-->>H: protocolVersion + capabilities + serverInfo
  H->>S: notifications/initialized
  S-->>H: 202 Accepted
  H->>S: tools/list
  S-->>H: Typed tools + annotations
  H->>S: tools/call list_published_articles
  S->>API: Invoke validated internal handler
  API->>Data: Read canonical corpus
  Data-->>API: Published records
  API-->>S: Structured result
  S-->>H: content + structuredContent

If you support multiple protocol versions, advertise and test only the versions that the server actually implements. A compatibility layer can return Server-Sent Events to an older client that explicitly requests text/event-stream. Current clients use ordinary Streamable HTTP JSON responses.

The implementation also supports the newer 2026-07-28 server/discover flow. It validates the MCP-Protocol-Version, Mcp-Method, and Mcp-Name request metadata against the JSON-RPC body. It returns complete-result metadata and cache hints.

Add an interactive MCP App

MCP tools can return text and structured data. They can also offer a real interface inside a compatible conversational host. Follow these steps:

Build the view with @modelcontextprotocol/ext-apps.
Bundle it as a self-contained web asset.
Expose the HTML through an MCP resource whose URI begins with ui://.

Link each relevant tool to that resource with the preferred nested metadata:

{
	"_meta": {
		"ui": {
			"resourceUri": "ui://example/profile-explorer",
			"visibility": ["model", "app"]
		}
	}
}

The matching resources/list entry and resources/read content must use text/html;profile=mcp-app. Add a narrow CSP declaration for any external scripts or images. Do not request camera, microphone, location, or clipboard permissions unless the interface needs them.

sequenceDiagram
  participant Agent as Agent host
  participant Tool as MCP tool
  participant Resource as ui:// resource
  participant View as Sandboxed app

  Agent->>Tool: tools/call
  Tool-->>Agent: text + structuredContent + UI metadata
  Agent->>Resource: resources/read
  Resource-->>Agent: text/html#59;profile=mcp-app
  Agent->>View: Render and deliver tool result
  View->>Agent: ui/initialize
  Agent-->>View: Host capabilities and context
  View->>Agent: tools/call for interactive refresh

Keep the UI additive. A client that does not negotiate the MCP Apps extension must still receive a useful text result. A compatible host can render the interactive profile, article, or case-study explorer.

Step 12: Add NLWeb conversational search

NLWeb gives you a simple /ask interface over your own content. It complements REST. REST fetches a resource when you already know its name. NLWeb takes a sentence and hands back grounded Schema.org records.

Support both GET and POST, with bounded inputs:

a query length limit
a JSON body size limit
a small result-limit range
documented list and summarize modes
a documented response shape

The search implementation can be deterministic, with these steps:

Tokenize the query.
Score the profile, case studies, and posts from agent-data.json.
Return the highest-ranked published records.

You do not need an external LLM.

flowchart LR
  Query["Natural-language query"] --> Validate["Validate size, mode, format"]
  Validate --> Rank["Rank canonical corpus"]
  Corpus["Profile + case studies + posts"] --> Rank
  Rank --> Ground["Attach canonical URLs"]
  Ground --> Json["Schema.org JSON response"]
  Ground --> SSE["SSE start/result/complete events"]

Offer regular JSON and Server-Sent Events. A streaming response can emit start, one or more result events, and complete. Set Cache-Control: no-store, because the results depend on a query and the server can stream them.

Return Schema.org types such as Person, CreativeWork, and BlogPosting. Include the canonical URLs that ground every answer.

Step 13: Add an A2A agent interface

The A2A protocol lets one agent describe and invoke another agent’s skill.

Publish an Agent Card at /.well-known/agent-card.json that contains:

name and description
provider and version
documentation URL and icon
supported interface URL, binding, and protocol version
input and output media types
capabilities
skills with IDs, descriptions, tags, and examples

Then implement the message-send route, for example /a2a/message:send. The route should do these steps:

Accept an A2A message with a unique message ID and text parts.
Search the canonical corpus.
Return a ROLE_AGENT message with both readable text and grounded structured data.

sequenceDiagram
  participant Caller as Calling agent
  participant Card as Agent Card
  participant A2A as A2A endpoint
  participant Search as Canonical search

  Caller->>Card: GET /.well-known/agent-card.json
  Card-->>Caller: Interface + skill metadata
  Caller->>A2A: POST message:send (application/a2a+json)
  A2A->>Search: Validate and research public corpus
  Search-->>A2A: Ranked, grounded records
  A2A-->>Caller: ROLE_AGENT message + structured data

Use application/a2a+json and emit the advertised protocol-version header. Support CORS, and return typed problem details for malformed requests.

Step 14: Publish agent-native authentication

Public content does not need forced authentication. To document and test a secure agent flow, expose a separate protected demonstration resource. Keep REST, MCP, A2A, and NLWeb public.

The flow uses established OAuth building blocks:

GET /agent/auth without a token returns 401 with a WWW-Authenticate challenge.
The challenge links to RFC 9728 protected-resource metadata at /.well-known/oauth-protected-resource.
That document points to authorization-server metadata at /.well-known/oauth-authorization-server.
POST /agent/identity creates a short-lived anonymous registration and signed identity assertion.
POST /oauth2/token exchanges the assertion with the RFC 7523 JWT bearer grant.
The agent calls /agent/auth with an audience-bound bearer access token.
POST /oauth2/revoke records the access-token ID as revoked.

sequenceDiagram
  participant Agent
  participant Resource as Protected resource
  participant Metadata as OAuth metadata
  participant Identity as Identity endpoint
  participant Token as Token endpoint
  participant DB as Revocation store

  Agent->>Resource: GET /agent/auth
  Resource-->>Agent: 401 + resource_metadata
  Agent->>Metadata: Discover resource and authorization server
  Metadata-->>Agent: Registration, token, scope, audience
  Agent->>Identity: Register anonymous identity
  Identity-->>Agent: 15-minute signed assertion
  Agent->>Token: JWT bearer exchange + resource audience
  Token-->>Agent: 1-hour profile:read access token
  Agent->>Resource: Bearer access token
  Resource-->>Agent: Public profile + token metadata
  Agent->>Token: Revoke access token
  Token->>DB: Store revoked jti until expiry
  Token-->>Agent: 200

Never put the signing secret in source or a static asset. Sign assertions and access tokens with a Worker secret such as AGENT_AUTH_SIGNING_SECRET. Use short expiry times, no-store, exact audiences, minimal scopes, unique JWT IDs, and constant-time signature verification.

A compact revocation table is sufficient:

CREATE TABLE revoked_agent_tokens (
  token_jti TEXT PRIMARY KEY,
  expires_at TEXT NOT NULL,
  revoked_at TEXT NOT NULL
);

Return a standards-oriented challenge on token errors:

WWW-Authenticate: Bearer resource_metadata="https://example.com/.well-known/oauth-protected-resource", error="invalid_token"

Document the complete exchange in /auth.md. Link it from llms.txt, the developer pages, and both metadata documents.

Step 15: Sign a Web Bot Auth directory

HTTP Message Signatures provide cryptographic proof that a response or key directory was signed by the domain owner’s private key. Built on RFC 9421, Signature-Input declares which HTTP components and metadata parameters (@authority, created, expires, nonce, keyid, alg) are covered, while Signature carries the raw Ed25519 signature bytes.

To support Web Bot Auth verification, publish an Ed25519 key directory at /.well-known/http-message-signatures-directory containing your public JWK, key thumbprint/identifier, validity window, and supported signature parameters, signed dynamically at the edge using a PKCS#8 private key stored in a Cloudflare Worker secret (WEB_BOT_AUTH_PRIVATE_KEY).

flowchart LR
  Secret["Ed25519 private key<br/>Worker secret"] --> Sign["RFC 9421 signer"]
  Request["Authority + nonce + timestamps"] --> Sign
  Sign --> Response["Directory body"]
  Sign --> Input["Signature-Input"]
  Sign --> Signature["Signature"]
  Public["Published Ed25519 JWK"] --> Verifier["Third-party verifier"]
  Response --> Verifier
  Input --> Verifier
  Signature --> Verifier

Generate a fresh cryptographic nonce alongside short-lived created and expires timestamps, sign the canonicalized @authority signature base with Web Crypto’s Ed25519 implementation, and return the directory with application/http-message-signatures-directory+json and CORS headers.

Write a protocol test that imports the public JWK returned in the response body, reconstructs the RFC 9421 signature base string from Signature-Input, and runs crypto.subtle.verify('Ed25519', ...) against the Signature header. A test that merely checks expect(res.headers.get('Signature')).toBeTruthy() proves only that you can concatenate strings; actual signature verification catches base-string formatting bugs immediately.

Step 16: Ship a CLI through GitHub Packages and Homebrew

Many coding agents (Claude Code, Codex, Cursor, Aider) have direct shell access and strongly prefer running a well-behaved CLI with --help and JSON output over writing ad-hoc curl pipelines.

The sudhanva CLI is a zero-dependency Node.js (>=22) executable wrapping the public API:

example profile
example posts --limit 5 --tag kubernetes
example post article-slug
example insight --audience recruiter --focus production-ml,kubernetes --wait

Read commands issue clean HTTPS GET requests, while insight generates a fresh UUID Idempotency-Key, posts to /api/v1/profile-insights, follows the Location header, and optionally polls until the job reaches succeeded or failed. Clean, parseable JSON goes to stdout, while human diagnostics and usage errors go to stderr with non-zero exit codes.

Publish a complete package

Publish the npm package to GitHub Packages under your GitHub user or organization scope (@example/example), including explicit repository, homepage, engines, and publishConfig metadata:

{
	"name": "@example/example",
	"version": "0.1.0",
	"description": "Dependency-free CLI for the example.com public API.",
	"license": "MIT",
	"type": "module",
	"bin": { "example": "example.mjs" },
	"files": ["example.mjs", "README.md", "LICENSE"],
	"engines": { "node": ">=22" },
	"repository": {
		"type": "git",
		"url": "git+https://github.com/example/homebrew-example.git",
		"directory": "cli"
	},
	"homepage": "https://example.com/developers/cli/",
	"publishConfig": { "registry": "https://npm.pkg.github.com", "access": "public" }
}

A release workflow can publish with its own GITHUB_TOKEN and packages: write permission, so you do not store a registry secret. GitHub links the package to the repository that repository names.

When the package manager invokes the executable through a symbolic link, resolve the path correctly. Test both a direct run and an installed, symlinked run.

Publish the source, tests, README, and license in a clean public repository. An install from GitHub Packages needs a token with read:packages, even for a public package. So also host the versioned .tgz archive on your own site, with its SHA-256 digest next to it (for example /cli/sudhanva-0.2.0.tgz.sha256). Then anyone can install without a token and audit the package bytes independently.

Add a Homebrew formula

Create an official tap repository and a formula. The formula downloads the immutable release archive and verifies its SHA-256 checksum. It installs the executable and runs a command in the formula test block.

brew install example/example/example
npm install --global @example/example # after adding @example:registry and a read:packages token to .npmrc

Link every install channel from llms.txt, developer documentation, package metadata, and the public source repository.

flowchart TB
  Source["Public CLI source + tests"] --> Pkg["GitHub Packages package"]
  Source --> Archive["Versioned .tgz + SHA-256"]
  Archive --> Formula["Homebrew formula"]
  Pkg --> Shell["Agent shell"]
  Archive --> Shell
  Formula --> Shell
  Shell --> API["Same typed public API"]

Step 17: Publish first-party SDKs across ecosystems

An OpenAPI contract lets an agent generate a client, but a small official SDK is safer and faster for repeated use. It gives developers typed inputs, structured errors, and documented polling. It also gives them a package with ownership that they can verify through a familiar registry.

Keep every SDK close to the HTTP contract. The clients for this site all have the same behavior:

retrieve the profile, the article collection, and a single article
combine public reads through the batch endpoint
create profile-insight jobs with an idempotency key
poll async jobs to a terminal state
query conversational search
turn any non-success response into a structured, language-native error

Publish through the ecosystem’s normal channel:

python -m pip install example
gem install example
go get github.com/example/example-go@v0.1.0

For Python and Ruby, include the canonical homepage, documentation, source, issue tracker, license, supported runtime versions, and package contents in the registry metadata. Require MFA for registry publication. Use short-lived or narrowly scoped credentials to publish.

For Go, do these steps:

Make the module path match the public repository.
Commit go.mod.
Push an immutable semantic-version tag.
Confirm that the public module proxy resolves it.

flowchart TD
  Contract["OpenAPI contract"] --> Python["Python client"]
  Contract --> Ruby["Ruby client"]
  Contract --> Go["Go client"]
  Python --> PyPI["PyPI release"]
  Ruby --> RubyGems["RubyGems release"]
  Go --> Tag["Git tag + Go module proxy"]
  PyPI --> API["Versioned public API"]
  RubyGems --> API
  Tag --> API
  Tests["Unit tests + live smoke tests"] --> Python
  Tests --> Ruby
  Tests --> Go

Give each language its own small public repository. Include a concise README, an MIT license, examples, unit tests with an injected or local HTTP transport, and CI across supported runtime versions. Do not add runtime dependencies when the standard HTTP and JSON libraries are sufficient. Then the installation surface stays small, and both humans and agents can inspect the code easily.

The live implementation includes:

the Python package on PyPI
the versioned Go SDK
the Ruby package on RubyGems
the Rust crate on crates.io
the JavaScript and TypeScript package on GitHub Packages
a canonical SDK installation guide

Link the guide, source, and registry pages from llms.txt, the developer index, the API discovery response, and the OpenAPI-adjacent documentation. Then name-based searches can find them.

Step 18: Package reusable Agent Skills and a plugin

An Agent Skill is a small, focused instruction package for a repeatable task. Publish skills that tell an agent when to use your source, which canonical endpoints to read, how to cite results, and which inferences to avoid.

A useful SKILL.md starts with minimal frontmatter:

---
name: find-technical-writing
description: Search and cite Example Person's published technical writing.
---

# Find technical writing

1. Query https://example.com/api/v1/posts for published metadata.
2. Add a documented tag filter when the topic is known.
3. Read the canonical article before explaining its conclusions.
4. Cite the canonical URL and distinguish publication from update dates.

Keep each skill narrow. The public implementation separates profile research, production case-study research, and technical-writing retrieval.

Publish an index at /.well-known/agent-skills/index.json. Include the discovery schema. For each skill, include its name, type, description, URL, and SHA-256 digest. Test the digest against the exact emitted bytes.

flowchart TD
  Index["Agent Skills index"] --> Profile["Research profile skill"]
  Index --> Cases["Find case studies skill"]
  Index --> Writing["Find writing skill"]
  Profile --> FirstParty["First-party pages and API"]
  Cases --> FirstParty
  Writing --> FirstParty
  Plugin["Agent Plugin"] --> Index
  Plugin --> MCP1["Public-profile MCP"]
  Plugin --> MCP2["Product MCP"]

Keep a public repository with the same skill files, an MIT license, and a canonical Agent Plugins plugin.json. Also add an mcp.json that connects the remote Streamable HTTP servers. Validate both manifests against their published JSON Schemas. A public repository also lets users install the skills through directories such as skills.sh.

Step 19: Establish registry and domain ownership

A self-published manifest becomes more trustworthy when the domain, source repository, registry record, and website point back to one another.

For the official MCP Registry, create server.json with the current registry schema:

{
	"$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
	"name": "com.example/public-content",
	"title": "Example Public Content",
	"description": "Read the public profile, case studies, and technical writing.",
	"version": "1.0.0",
	"repository": {
		"url": "https://github.com/example/example.com",
		"source": "github"
	},
	"remotes": [{ "type": "streamable-http", "url": "https://example.com/mcp" }]
}

Use a domain-owned namespace. Publish the HTTP ownership proof for the Registry at /.well-known/mcp-registry-auth. Keep the private signing key in CI secrets. The official remote-server publishing guide documents the flow.

Automate registry publication with a manually triggered workflow:

check out the source
download the official mcp-publisher binary
authenticate against the domain ownership proof
query the registry for that exact name and version
publish only when that immutable version is absent
fail loudly on any registry response you did not expect

If it serves your users, also keep a public directory listing such as Smithery. Link registry entries from the homepage metadata, visible developer documentation, llms.txt, Markdown alternates, and the MCP guide. With links in both directions, a client can confirm the relationship from either side.

flowchart LR
  Domain["example.com"] --> Proof[".well-known ownership proof"]
  Domain --> Card["MCP card + live endpoint"]
  Repo["Public source repository"] --> Metadata["server.json"]
  Metadata --> Publisher["mcp-publisher CI"]
  Secret["Registry private key<br/>CI secret"] --> Publisher
  Proof --> Registry["Official MCP Registry"]
  Publisher --> Registry
  Registry --> Domain
  Directory["Public directory listing"] --> Domain
  Domain --> Directory

The live sudhanva.me Registry record is a working example of the domain-owned remote-server relationship.

Step 20: Deploy the stack on Cloudflare

Astro builds the static site to dist. Wrangler uploads those files as Workers Static Assets. Cloudflare runs the Worker first, so the Worker can negotiate representations and handle protocol routes. Cloudflare documents this routing model in Worker script routing.

{
	"$schema": "node_modules/wrangler/config-schema.json",
	"name": "example-site",
	"compatibility_date": "2026-08-23",
	"main": "worker.mjs",
	"build": { "command": "npm run build" },
	"assets": {
		"directory": "./dist",
		"binding": "ASSETS",
		"run_worker_first": true,
		"not_found_handling": "404-page",
		"html_handling": "auto-trailing-slash",
	},
	"d1_databases": [
		{
			"binding": "AGENT_DB",
			"database_name": "example-agent-data",
			"database_id": "YOUR_DATABASE_ID",
			"migrations_dir": "./migrations",
		},
	],
	"observability": { "enabled": true },
}

The commands below do these steps:

Create D1.
Apply the migrations.
Add the secrets.
Deploy.

npx wrangler d1 create example-agent-data
npx wrangler d1 migrations apply example-agent-data --remote
npx wrangler secret put AGENT_AUTH_SIGNING_SECRET
npx wrangler secret put WEB_BOT_AUTH_PRIVATE_KEY
npx wrangler deploy

Cloudflare records the applied D1 migration files in a migrations table. For details, see the D1 migrations reference.

At runtime, the Worker route order should be explicit:

flowchart TD
  Request["Incoming request"] --> Preflight{"OPTIONS?"}
  Preflight -- Yes --> Cors["Protocol-specific CORS response"]
  Preflight -- No --> Protocol{"API, MCP, NLWeb, A2A, auth?"}
  Protocol -- Yes --> Handler["Typed protocol handler"]
  Protocol -- No --> Markdown{"Markdown preferred or .md alias?"}
  Markdown -- Yes --> MdAsset["Agent Markdown asset"]
  Markdown -- No --> Static["ASSETS.fetch(request)"]
  Static --> Decorate["Add discovery Link + Vary headers"]
  Handler --> Response["Response"]
  MdAsset --> Response
  Decorate --> Response
  Cors --> Response

Set global security headers on static pages: HSTS, X-Content-Type-Options, frame protection, referrer policy, permissions policy, and the cross-origin policies. Then, for the machine-readable artifacts, override the content type, CORS, indexing, and caching for each path.

All of this fits comfortably in free-tier territory, and that was not an accident. The content stays static and cacheable. The Worker holds no state for almost all requests. The only rows in D1 are small and short-lived.

Step 21: Test the contracts and the pages

Agent readiness is an integration property. If a file exists at the right path, that does not prove that its declared endpoint completes a handshake.

Build-time artifact tests

After the static build, assert that:

the homepage and trust pages carry an early visible H1 and real no-JavaScript text
JSON-LD has the entity types and stable identifiers you expect
llms.txt, scoped indexes, instructions, auth docs, and Markdown alternates all got emitted
every JSON artifact parses
every JSON Lines record parses on its own
the Schema Map points at the feed
each Agent Skill digest matches the bytes actually written
the plugin, MCP, A2A, API catalog, AI catalog, OAuth, and Registry manifests match their schemas and URLs
the OpenAPI document has unique operation IDs, descriptions, typed inputs, typed responses, and lifecycle metadata
the CLI archives have the right gzip magic bytes and SHA-256 digests

Worker protocol tests

Call the Worker directly with mocked static assets and a mocked D1. Cover these cases:

Accept negotiation, including q-values and wildcards
the Vary, Link, Content-Type, and Content-Location headers
direct .md aliases and the agent-friendly 404
API validation, pagination, HEAD, OPTIONS, batch boundaries, and error envelopes
idempotent create, replay, conflict, polling, expiry, and job state transitions
NLWeb JSON and SSE results
the A2A card and a message exchange
auth discovery, registration, exchange, protected read, expiry, and revocation
cryptographic verification of the Web Bot Auth signature
MCP init, notifications, tool listing, tool calls, resources, the compatibility transport, modern discovery, version errors, and origin checks
the CLI run directly and through a symlink

flowchart LR
  Change["Source change"] --> Lint["ESLint + Markdown lint + format"]
  Lint --> Types["Astro type check"]
  Types --> Build["Static production build"]
  Build --> Artifacts["Artifact contract tests"]
  Build --> WorkerTests["Worker protocol tests"]
  Build --> CliTests["CLI tests"]
  Artifacts --> Deploy["Deploy"]
  WorkerTests --> Deploy
  CliTests --> Deploy
  Deploy --> Smoke["Public endpoint smoke tests"]
  Smoke --> Registries["Registry and backlink verification"]

Run the local quality gate:

npm run lint:all
npm run build
npm test
git diff --check

Verify production endpoints

Use curl for exact HTTP behavior:

# HTML remains the default.
curl -i https://example.com/

# The same canonical URL can return Markdown.
curl -i -H 'Accept: text/markdown' https://example.com/

# Discovery artifacts parse.
curl -fsS https://example.com/openapi.json | jq '.openapi'
curl -fsS https://example.com/.well-known/mcp.json | jq '.servers'
curl -fsS https://example.com/.well-known/agent-card.json | jq '.skills'

# API reads are typed and reachable.
curl -fsS https://example.com/api/v1/profile | jq
curl -fsS 'https://example.com/api/v1/posts?limit=3' | jq

# MCP performs a real handshake.
curl -fsS https://example.com/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  --data '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"smoke-test","version":"1.0.0"}}}' | jq

# Create and poll an idempotent asynchronous job.
curl -i https://example.com/api/v1/profile-insights \
  -H 'Content-Type: application/json' \
  -H "Idempotency-Key: $(uuidgen)" \
  --data '{"audience":"recruiter","focus":["production-ml"]}'

For Server-Sent Events, inspect the event stream. Do not pipe it directly into jq. For HEAD, make sure that the body length is zero and that the response headers match GET semantics.

Finally, verify external records from a separate network path:

Open the official registry entry.
Follow its remote URL.
Follow the website backlink to the registry.
Send the same initialize request that a third-party host sends.

Implementation checklist

Use this order when you adapt the design to another site:

Content and discovery

Prerender meaningful semantic HTML with an early product-named H1.
Add canonical metadata, Open Graph fields, sitemap, and RSS.
Generate Schema.org JSON-LD from canonical typed data.
Publish an intentional robots.txt policy.
Publish llms.txt, scoped indexes, and an agent instruction file.
Add negotiated Markdown plus direct .md aliases.
Advertise alternates, docs, and API descriptions with HTML and HTTP links.
Publish the RFC 9727 API catalog and an agentic resource catalog.
Publish agent-data.json, a Schema.org JSON Lines feed, and a Schema Map.

Callable interfaces

Publish an OpenAPI 3.1 document with unique operation IDs and complete schemas.
Implement typed reads, deterministic pagination, HEAD, OPTIONS, CORS, and structured errors.
Publish a URL-major versioning and deprecation policy.
Add a bounded, allowlisted batch endpoint.
Add a real asynchronous job with idempotency, polling, terminal states, and expiry.
Implement MCP Streamable HTTP with a real handshake, tools, resources, annotations, and cards.
Implement grounded NLWeb JSON and SSE search.
Publish an A2A Agent Card and message-send route.

Trust and distribution

Publish OAuth protected-resource and authorization-server metadata for protected agent resources.
Use short-lived, audience-bound credentials and a revocation store.
Publish and cryptographically test a signed Web Bot Auth directory.
Ship a tested CLI through GitHub Packages and Homebrew with public source and immutable checksums.
Publish small, tested SDKs through two or more language ecosystems and link their source.
Publish narrow Agent Skills with verified digests and a portable Agent Plugin.
Publish domain-owned MCP Registry metadata and ownership proof.
Link the site, source, protocol cards, directories, and registries bidirectionally.

Operations

Keep all representations connected to one canonical content source.
Set exact content types, CORS, caching, and indexing headers by artifact.
Store secrets only in the deployment platform or CI secret store.
Test artifact contents, live protocol behavior, cryptographic verification, and installed CLI execution.
Run production smoke tests after every deployment.

Closing principles

Three lessons stood out from building and operating this stack:

Idempotency is the foundation of safe agent actions. Agents retry aggressively on network blips, timeouts, and context-window Compaction. Requiring an Idempotency-Key and persisting state in D1 turns retries from duplicate side effects into harmless replays.
Different hosts speak different protocols, so share one core. Browser agents look for WebMCP or negotiated Markdown, IDE and desktop hosts speak MCP Streamable HTTP, conversational search clients hit NLWeb /ask, and multi-agent orchestrators inspect A2A Agent Cards. You rarely need all of them on day one, but routing every protocol handler through the same validated functions and canonical agent-data.json corpus makes adding a new surface a 100-line adapter instead of a rewrite.
Trust is a graph of cross-references. Anyone can serve a JSON file at /.well-known/mcp.json. Cautious clients and registries verify whether your domain, GitHub repository, OAuth metadata, RFC 9421 Web Bot Auth signatures, package registries (PyPI, crates.io, RubyGems, Go proxy, GitHub Packages, Homebrew), and official MCP Registry ownership proofs all point back to one another.

Every machine-readable contract in this post is something a well-meaning future refactor could break. The only reason I trust all twenty-one layers a month later is that the build and protocol test suite fails loudly in CI before a broken contract ever reaches production.