How I automated internal linking with AI

Two articles can both mention Astro and have almost nothing useful to say to each other’s readers. One explains an animated avatar. The other explains a newsletter. Automatically linking the word “Astro” between them gives you another internal link, but what does the reader get from clicking it?

That’s the problem I want an internal linking script to solve. Finding similar articles is only the beginning. The harder part is deciding whether a particular sentence gives someone a reason to leave the page.

This site has a TypeScript script that does both. It compares passages using embeddings, asks an LLM whether a link would help, and wraps existing words in Markdown link syntax. The model never returns a rewritten article.

The SEO term is contextual internal linking: links within the body of a page that point to relevant content on the same site. Google’s link guidance recommends descriptive anchor text and linking to resources that help readers understand the current page. That’s a useful standard to build against. This walkthrough makes no claim about measured ranking improvements.

Give the model a small editing job

I already have a semantic recommendation engine for related content. An inline link needs a more specific reason than a related-post card does. Its anchor text makes a promise about what the destination explains.

The implementation lives in scripts/add-internal-links.ts. It reads published posts, work pages, and newsletters from the repository. It runs as an explicit authoring command, outside the normal build:

pnpm run inlinks

It uses the Vercel AI SDK through AI Gateway, with AI_GATEWAY_API_KEY available to the script. The implementation currently selects openai/text-embedding-3-small for embeddings and openai/gpt-6-luna for summaries and link decisions. Those are configuration choices in this repository, not requirements of the approach.

The command writes back to source files. There is no dry-run flag, so use a clean Git checkpoint and review the resulting diff. I’ll return to a cleanup limitation before the end, because I wouldn’t copy that part unchanged.

1. Extract prose without treating MDX as plain text

An MDX file contains paragraphs, code samples, imports, and components. A phrase inside a code sample can look like a perfect anchor to a text search. Turning it into a Markdown link would damage the example.

The script parses the body into an abstract syntax tree, or AST, using packages already in the site:

const mdxProcessor = unified().use(remarkParse).use(remarkMdx);

function parseMarkdown(content: string): Root {
  return mdxProcessor.parse(content) as Root;
}

It collects paragraph text while skipping code, inline code, JSX, imports, exports, and MDX expressions. It then groups paragraphs into chunks targeting 600 characters. Longer paragraphs are split around sentence boundaries. That is a target size, not a hard limit: an individual long sentence can exceed it.

This gives the search a useful unit. A long post might discuss deployment in one section and caching in another. Comparing those passages separately gives us a better place to look for a link than comparing one representation of the entire article.

Keep the original Markdown too. The extracted prose is for retrieval; the original source and its AST positions are what make the eventual edit possible.

2. Give each chunk enough context to search

Before embedding a chunk, the script prefixes it with the article’s title, description, summary, tags, and up to twelve headings. A passage about “the cache” gets some context about which cache the author means.

The summary is generated from metadata and the first 3,000 characters of extracted body text. That makes it a compact introduction, not a reliable account of everything the article covers. Later, the link decision must rely on an actual destination passage.

Here’s the batch embedding call from the script:

const { embeddings: newEmbeds } = await embedMany({
  model: EMBED_MODEL,
  values: uncachedChunks.map((c) => c.chunk),
  maxParallelCalls: 2,
  maxRetries: 2,
});

An embedding represents text as a vector. The SDK’s embedding utilities also provide cosine similarity for comparing those vectors. The script keeps the vectors in memory and caches them in a local JSON file. There is no vector database.

For each eligible source chunk, it compares the vector against chunks in other documents. It skips the source document itself and destinations already linked from that source. Matches scoring at least 0.65 become candidates, sorted by similarity. Only the strongest match for each destination survives.

That threshold is a retrieval setting. A score of 0.65 does not mean a link has a 65% chance of being useful.

The script also limits each page to six detected internal links, including existing ones. This is an editorial cap, not an SEO prescription. Google explicitly says there is no ideal link count. A page already above the cap doesn’t get trimmed down; the script simply adds nothing.

There’s a deliberate consequence of how the shortlist works: candidates are capped before model review. If a page has room for two links and the model rejects both candidates, the script doesn’t keep searching until it fills those slots. Zero additions is an acceptable result.

3. Ask for evidence, not a plausible explanation

The most important instruction in the prompt is short:

Default to shouldLink: false. Zero links is a successful result.

The model receives the current source Markdown, a local window around the matching chunk, and a corresponding window from the destination article. Each window includes the matching chunk and up to one neighboring chunk on either side. Titles and summaries provide background.

It must return four fields:

const analysisSchema = z.object({
  shouldLink: z.boolean(),
  anchorText: z.string(),
  sourcePassage: z.string(),
  targetEvidence: z.string(),
});

sourcePassage must copy the source sentence containing the anchor. targetEvidence must copy a supporting sentence from the destination excerpt. The prompt asks whether that passage explains, demonstrates, or substantiates the specific idea in the anchor’s sentence. Merely sharing a tool, topic, or audience is insufficient.

For example, the repository’s test fixture uses this source sentence:

Use database indexes to speed up queries.

Its target evidence is:

Database indexes let SQLite find rows without scanning the entire table.

That is a defensible destination for “database indexes.” A project case study that happens to mention SQLite would need more evidence.

The script checks that the source passage belongs to the local context and that the quoted evidence actually appears in the destination excerpt. It also rejects generic anchors and limits them to two through six words, or one specific term of at least eight characters.

These checks catch invented quotations and unsuitable anchor shapes. They cannot prove that a real quotation supports the link. That judgment still depends on the model and the person reviewing the diff. Structured output makes a response parseable; it doesn’t make its reasoning correct.

4. Insert the link at the approved occurrence

Suppose “database indexes” appears twice. One sentence recommends them. An earlier sentence says they were removed from an example. Replacing the first occurrence would put the link somewhere the model didn’t approve.

The script locates the exact source passage first. It rejects a passage that appears more than once, then requires exactly one word-bounded occurrence of the anchor within it.

Next, it checks the AST. The entire anchor must fit inside one plain-text node. It cannot sit inside a heading, code, an existing link, or an MDX component. It also excludes the opening paragraph. These are enforced in code, even though the prompt already asks for them.

Once those checks pass, the edit itself is just string slicing:

const before = doc.slice(0, targetIndex);
const after = doc.slice(anchorEnd);
const linkedText = `[${anchorText}](${url})`;

return before + linkedText + after;

The destination URL comes from the loaded document, not from model output. The model selects existing words; code supplies the punctuation and address.

The resulting change in the test fixture is easy to review:

-Use database indexes to speed up queries.
+Use [database indexes](/blog/sqlite-indexes) to speed up queries.

An additional validation step checks for exactly one link to the expected URL and compares the prose after stripping links and normalizing whitespace. That comparison is a secondary check, not proof of byte-for-byte preservation. The slicing operation is what confines the body edit to the selected anchor. Writing the file also serializes its frontmatter again, which can produce formatting changes in the diff.

5. Cache the expensive preparation

There are two caches worth separating:

Cache	What makes an entry reusable
Article summary	The raw document hash and chat model match
Chunk embedding	The chunk index and hash of its full prefixed text match

If a summary request fails, the script falls back to the article description. Link-analysis errors skip that candidate. An embedding failure can stop the run.

Caching reduces repeated preparation work, but it doesn’t make reruns free. Link decisions aren’t cached. Changing a summary changes the text sent for every chunk in that document, which can invalidate all its embeddings. Even adding a link changes the raw document hash used for the summary cache.

One limitation to fix when adapting this code: the embedding cache doesn’t store the embedding model identifier. Clear it when changing models, or include the model in the cache key. Vectors produced by different models shouldn’t be mixed just because their source text matches.

6. Test the editor independently of the model

The existing tests mock the model response. That lets them check the edit without a paid API request or a different answer on each run:

pnpm exec tsx --test tests/internal-links.test.ts

They cover invented or missing evidence, repeated passages, partial-word matches, generic anchors, formatting boundaries, and protected Markdown and MDX. They also check that an already-linked destination is skipped and that the selected sentence gets the link even when the same words appear earlier.

That’s the distinction I want in the tests. Whether a model chooses a useful destination needs editorial evaluation. Whether my code links the wrong occurrence should have a deterministic answer.

Review removals as carefully as additions

The current command starts with a separate cleanup pass that removes links to URLs absent from its published-content inventory. This pass is less careful than the insertion code: it uses a regular expression and compares complete URLs against posts, work pages, and newsletters.

That means it can mistake a valid /about link, or a post URL with a fragment, for a dead link. It can also match Markdown-looking text in code samples. I would narrow that cleanup to known content routes, normalize URLs, and make it AST-aware before reusing it on another site. You can adopt the insertion approach without adopting this cleanup pass.

The command also writes files as it goes. If a later API call fails, earlier edits may already be on disk. Review the whole diff after a failed run too.

For a small repository, comparing chunks directly keeps the setup manageable. The search work grows roughly quadratically with the total number of chunks, so a large archive may eventually need indexed retrieval. I haven’t supplied a corpus-size benchmark here; measure your own run before deciding you need another service.

What I would keep at any size is the separation between proposing and applying a link. Give the model passages to compare and require it to name the exact edit. Then make the code check the location, preserve the writing, and leave you a diff you can actually review.