setQuery(e.currentTarget.value)}
class="search-input"
/>
{loading &&
);
}
```
The client initialization uses your public search key, making it safe to expose. The SDK handles typo tolerance automatically - searching for "javascirpt" will still find "javascript" posts.
## Advanced Features You Probably Don't Need
### Typo Tolerance
Meilisearch handles typos out of the box. No configuration needed. It just works. This is genuinely impressive - searching for "astr" finds "astro", "javascirpt" finds "javascript", and so on.
For most blogs, this is overkill. Your readers can spell.
### Faceted Search and Filtering
Want to filter by tags or date ranges? Meilisearch makes it easy:
```typescript
const results = await index.search(query, {
filter: 'tags = "astro" AND date > 2024-01-01',
limit: 10,
});
```
But honestly, for a blog, a simple tag page works fine. You don't need real-time faceted filtering.
### Vector Search and Hybrid Search
Meilisearch ships with a vector store, and recent releases made the newer store the default for newly created indexes. You can do semantic search by generating embeddings and mixing them with keyword relevance:
```python
# Example from Meilisearch docs
results = client.index('books').search(query, opt_params={
'hybrid': {
'semanticRatio': 0.7,
'embedder': 'openai'
},
'limit': 4
})
```
The `semanticRatio` controls the balance between keyword and semantic search. A value of 0.7 means 70% semantic, 30% keyword matching.
But here's the catch - you need to generate embeddings for all your content. This requires an OpenAI API key, adds complexity to your indexing pipeline, and costs money for the API calls.
For a small blog, this is complete overkill.
### Multimodal Search
Meilisearch added multimodal embeddings support, letting you search across text and images in newer releases. It's impressive, but it also increases operational complexity and cost, so it's hard to justify for a small site.
## The Honest Assessment
After setting all this up, here's my take on when you should actually use Meilisearch.
### When Meilisearch Makes Sense
Use Meilisearch if you have:
- 1000+ documents that need search
- Real-time content updates without rebuilds (like a CMS)
- Complex filtering requirements (e-commerce, job boards)
- Multiple languages requiring sophisticated typo tolerance
- The infrastructure to maintain a production search server
Good use cases include large documentation sites, e-commerce platforms, job boards, or any application where search is a core feature.
### Why It's Overkill for Small Blogs
On a sub-50-post blog, Meilisearch mostly adds a server to babysit, recurring infra cost, and a lot of configuration surface area while delivering little beyond what static search already gives you. If you rebuild on deploy and only need simple tag filtering, Pagefind or Orama stay simpler, cheaper, and good enough.
## Conclusion
Meilisearch is powerful, impressive software that I genuinely enjoy using. But for a static blog, it violates the principle of "keeping it simple."
By choosing a static library like Pagefind or Orama, you eliminate an entire class of maintenance tasks and security risks. You get search that is fast, smart, and effectively free.
Build this setup to learn. But deploy the static solution to production.
---
## What's next?
This is part of a series of posts on implementing search for static sites:
- [The Right Way to Add Orama Search to Astro](/blog/orama-astro) -- simple, zero-config search for small to medium sites
- [Why I Switched from Orama to Pagefind](/blog/pagefind-astro) -- chunked index for better scalability
- Meilisearch is the Best Search You'll Never Need (**_you are here_**) -- server-side search with advanced features
- [Why I Didn't Use Google Programmable Search](/blog/google-custom-search-astro) -- the hidden costs and indexing delays that make it impractical
- [I Tried 4 Search Engines So You Don't Have To](/blog/astro-search-comparison) -- comprehensive comparison from a small blog perspective
All with practical examples from a real production blog.
---
---
title: Why I Switched from Orama to Pagefind
description: Why I use Pagefind: A deep dive into chunked indexing, bandwidth scaling, and setting up a production-grade search for Astro.
type: blog
url: https://sarthakmishra.com/blog/pagefind-astro
date: 2026-01-13
tags: ['astro', 'search', 'pagefind', 'performance', 'typescript']
author: Sarthak Mishra
excerpt: Orama is great, but downloading a 2MB JSON index is not. Here is how Pagefind solves the scalability problem with binary chunks and zero-config ranking.
---
# Why I Switched from Orama to Pagefind
I previously wrote about [building search with Orama](/blog/orama-astro). It's a fantastic piece of engineering—an in-memory database running entirely in the browser.
But it has a fatal flaw for content sites: **Scalability.**
Orama (and Fuse.js/Lunr) are **monolithic**. To search _anything_, the user must download _everything_. If you have 500 posts, the user downloads 500 posts' worth of metadata before they can type a single letter in the search bar.
**Pagefind** flips this model. Instead of sending the database to the user, it splits the index into thousands of tiny binary "chunks." The browser only downloads the chunks relevant to the specific search query.
Here is how I implemented Pagefind in Astro, covering the parts the docs skip, like custom ranking, complex filtering, and fixing the "Dev Mode" headache.
## Why I love the chunking approach
When you build your site, Pagefind scans your HTML and generates a static "API" in your `dist/` folder.
1. **`pagefind.js` (30KB):** The tiny client entry point.
2. **`pagefind-entry.json` (4KB):** A map of all words to "index chunks."
3. **Index Chunks (10-20KB each):** Binary files containing lists of documents that match specific word hashes.
**The Flow:**
1. User types "React".
2. `pagefind.js` hashes "React" and looks up which _chunk_ contains "R" words.
3. It fetches _only_ `index_chunk_a1b2.pf_index`.
4. It finds the matches and fetches the _fragments_ for those specific URLs to display the snippet.
**Total bandwidth:** ~50KB.
**Orama bandwidth:** ~600KB (for the same dataset).
## 1. The Setup
The `npx pagefind --site dist` command works in prod, but it breaks your local dev workflow because `astro dev` doesn't generate a `dist` folder.
We need a setup that works in both environments.
**Installation:**
```bash
pnpm add -D pagefind
```
**The Config:**
Create a `pagefind.yml` in your root. This keeps your CLI commands clean.
```yaml
# pagefind.yml
site: dist
exclude_selectors:
- 'nav'
- 'footer'
- '.related-posts'
glob: '**/*.html'
```
**The "Dev Mode" Fix:**
Since Pagefind needs static HTML to crawl, it _cannot_ index your site while it's in hot-reloading dev mode. The workaround is to build the index once, and then tell Astro to serve it as a static asset.
Add this to `package.json`:
```json
"scripts": {
"dev": "pnpm build:search && astro dev",
"build": "astro build && pnpm build:search",
"build:search": "pagefind --site dist"
}
```
Now, we need to allow Astro to serve the `pagefind.js` file from the `dist` folder during dev. Update `astro.config.mjs`:
```javascript
export default defineConfig({
vite: {
server: {
// Allow serving files from the dist directory during dev
fs: {
allow: ['dist'],
},
},
},
});
```
_Note: In dev, the search results will be "stale" (from the last build). This is the trade-off._
## 2. The Schema: Decorating HTML
Pagefind doesn't use a JSON schema. It uses your DOM. You control the index using **Data Attributes**.
### The Basics
Wrap your main content in a `data-pagefind-body` attribute and add metadata attributes to the elements you want to index:
```html
Searching...
}
{error && {error}
}
{results.length > 0 && (
{results.map((result) => (
)}
{!loading && query && results.length === 0 && (
{result.title}
{result.description}
))}No results found for "{query}"
)}
{description}
{tags.map(tag => ( ))} ``` Now, searching for "Astro" in the intro paragraph ranks higher than "Astro" in the footer. To learn more, checkout the [weighting](https://pagefind.app/docs/weighting/) and [metadata](https://pagefind.app/docs/metadata/) docs. ## 3. The Search UI Since `pagefind.js` is a generated asset, we can't `import` it normally. We must use a dynamic import pointing to the URL. ```typescript export default function SearchModal() { const [pagefind, setPagefind] = useState(null); const [results, setResults] = useState([]); // Lazy load on open useEffect(() => { async function load() { // 1. Dynamic import from the public URL const lib = await import(/* @vite-ignore */ '/pagefind/pagefind.js'); await lib.init(); setPagefind(lib); } load(); }, []); const handleSearch = async (query) => { if (!pagefind || !query) return; // 2. Run Search const search = await pagefind.search(query); // 3. Load data for top 5 results // Pagefind returns "pointers" first, then we fetch the data const topResults = await Promise.all( search.results.slice(0, 5).map((r) => r.data()), ); setResults(topResults); }; // ... Render UI } ``` ## Where Pagefind Shines (and Fails) ### Where it is Extremely Effective 1. **Long-Form Documentation:** Pagefind supports **Sub-Result Anchors**. If you search "Permissions", and you have a massive guide called "Linux Basics", Pagefind can return a direct link to the `## Permissions` H2 inside that page (`/linux-guide#permissions`). Orama struggles with this without massive index bloat. _Enable it with `await pagefind.init({ showSubResults: true })`._ 2. **Multi-Site Search:** If you have a Blog (Astro) and a Documentation site (Starlight), Pagefind can **merge indexes**. You can run one search bar that queries both `blog.com/pagefind` and `docs.com/pagefind` seamlessly in the browser. ### Where it Fails 1. **Typo Tolerance:** This is the biggest trade-off. Orama uses [Levenshtein distance](https://github.com/oramasearch/orama/blob/main/packages/orama/src/components/levenshtein.ts) to match, for example, "astor" to "astro." Pagefind **does not**. [Pagefind](https://pagefind.app/docs/multilingual/) is a _stemming_ engine. It knows "run" matches "running." But if you type "rnning", it finds nothing. It relies on prefix matching, so typing "prog" finds "programming", but "pgramming" finds zero. 2. **Dev Experience:** There is no way around it: having to run a build to update the search index is annoying. If you write a new post, you won't see it in search until you restart the server with a fresh build. ## Conclusion Pagefind is the "Adult" choice for static search. It isn't as flashy as Orama, and it doesn't do vector embeddings or any other AI magic. But it scales from 10 pages to 10,000 pages without changing a line of code. --- ## What's next? This is part of a series of posts on implementing search for static sites: - [The Right Way to Add Orama Search to Astro](/blog/orama-astro) -- simple, zero-config search for small to medium sites - Why I Switched from Orama to Pagefind (**_you are here_**) -- chunked index for better scalability - [Meilisearch is the Best Search You'll Never Need](/blog/meilisearch-astro) -- server-side search with advanced features - [Why I Didn't Use Google Programmable Search](/blog/google-custom-search-astro) -- the hidden costs and indexing delays that make it impractical - [I Tried 4 Search Engines So You Don't Have To](/blog/astro-search-comparison) -- comprehensive comparison from a small blog perspective All with practical examples from a real production blog. --- --- title: The Right Way to Add Orama Search to Astro description: Stop using the Orama Astro plugin. Here is how to build a type-safe, metadata-rich search engine that actually works in Dev Mode. type: blog url: https://sarthakmishra.com/blog/orama-astro date: 2026-01-12 tags: ['astro', 'search', 'orama', 'preact', 'typescript', 'ui-design'] author: Sarthak Mishra excerpt: The official Orama plugin is great for demos, but it breaks in dev mode and sucks at metadata. Here is how I built a production-grade, zero-config search engine using Astro Endpoints. --- # The Right Way to Add Orama Search to Astro I love static sites. They are fast, secure, and cheap to host. But **search** has always been the weak point. You usually have two options: 1. **External Services:** Algolia or similar self-hosted services like Meilisearch. 2. **Client-Side Libraries:** Orama, Pagefind, Lunr or Fuse. I was looking for something that runs **entirely in the browser**, supports **fuzzy matching**, handles **typos**, and doesn't cost me a dime. Enter [Orama](https://orama.com/) (formerly Lyra). It's an immutable, in-memory, full-text search engine written in TypeScript. It's fast. However, most tutorials (including the official docs) tell you to use the `@orama/plugin-astro`. **Do not do this.** The plugin has three major problems: 1. **No Dev Mode:** It only generates the index during `astro build`. If you run `pnpm dev`, your search bar 404s. 2. **Weak Metadata:** It scrapes your HTML output. If you want to display a "Thumbnail" or "Author" in your search results, you have to build awkward side-channel maps. 3. **No Control:** You can't easily boost specific fields (like `title` > `body`) without fighting the config. Here is how to implement Orama the _right_ way using Astro Endpoints. ## The Architecture Instead of scraping HTML, we generate the index directly from **Astro Content Collections**. 1. **Build Time:** We create a `.json.ts` endpoint that fetches all posts, builds an Orama DB, and serializes it to JSON. 2. **Client Time:** The browser fetches this JSON file (lazy-loaded) and hydrates a local Orama instance. This solves the Dev Mode problem immediately because Astro treats `.json.ts` files as live routes. ## 1. The Index Endpoint Create a file at `src/pages/search-index.json.ts`. This acts as our "Search API." Defining a strict schema is important. Orama performs best when it knows exactly what fields to expect. By specifying fields like `title`, `slug`, `date`, and `description`, you optimize the index for performance and ensure each search result contains everything needed to render its card. ```typescript export async function GET() { const posts = await getCollection('blog', ({ data }) => !data.draft); // 1. Create the DB instance const db = await create({ schema: { title: 'string', slug: 'string', description: 'string', date: 'string', // Store as string for easy display tags: 'string[]', }, }); // 2. Insert Data const records = posts.map((post) => ({ title: post.data.title, slug: `/blog/${post.slug}`, description: post.data.description, date: post.data.date.toISOString(), tags: post.data.tags, })); await insertMultiple(db, records); // 3. Serialize to JSON const index = await save(db); return new Response(JSON.stringify(index), { headers: { 'Content-Type': 'application/json', }, }); } ``` ## 2. The Search Component (Client-Side) On the client, we fetch this JSON and "hydrate" Orama. I use Preact here to save bundle size, but the logic is framework-agnostic. We use `load` to restore the database state. This is much faster than re-indexing data on the client. ```typescript // Define the same schema for type safety const SCHEMA = { title: 'string', slug: 'string', description: 'string', date: 'string', tags: 'string[]', } as const; export default function SearchModal() { const [db, setDb] = useState` tags and hoping for the best. We use the Content Collection schema directly. The search result object contains exactly what we need to render the UI:
```typescript
// The `hit.document` is fully typed
{
title: "Building Static Search...",
slug: "/blog/building-static-search",
date: "2026-01-12T00:00:00.000Z",
tags: ["astro", "search"]
}
```
### 3. Relevance Tuning
Notice the `boost: { title: 2 }` in the search query? If a user searches for "Astro", a post with "Astro" in the title should rank higher than a post that just mentions it in the footer. The official plugin abstracts this away. Here, we have full control.
## When NOT to use it
I've talked Orama up, but let's be honest about the trade-offs.
### The Bandwidth Issue
Orama is an **In-Memory** search engine. This means the _entire_ index must be loaded into the user's browser RAM.
- **50 Posts:** Index is ~30KB (Gzipped). **Perfect.**
- **500 Posts:** Index is ~300KB. **Manageable.**
- **5,000 Posts:** Index is ~3MB+. **Unusable.**
If you have thousands of pages, forcing the user to download a 3MB JSON file just to search is bad UX.
**The Alternative:** For larger sites, use **Pagefind**.
Pagefind splits the index into small "chunks." It only downloads the chunks relevant to the search query. It scales to 100k+ pages while keeping initial bandwidth low.
In my implementation above, I indexed the `description`, not the full body.
Why? **Size.** Indexing the full body content of 100 posts bloats the JSON file significantly. For a blog, searching titles, descriptions, and tags is usually 95% of what users need. If you _must_ search the full body text, be prepared for the file size to triple.
### Solving the Bandwidth Problem
The `src/pages/search-index.json.ts` architecture solves the dev mode and metadata issues, but as your blog grows, you will hit the **Bandwidth Wall**.
If you index the full body of 200 posts, your JSON file could easily exceed 2MB. Here is how to solve this using **Schema Optimization** and **AI-Powered Semantic Compression**.
#### 1. Schema Optimization (The Low-Hanging Fruit)
The fastest way to bloat your index is to index fields you don't actually search.
**Bad Schema:**
```typescript
const schema = {
title: 'string',
slug: 'string',
body: 'string', // 🚨 DANGER: Indexing 5,000 words per post
author: 'string',
tags: 'string[]',
image: 'string', // 🚨 DANGER: Why are you indexing a URL string?
};
```
**Optimized Schema:**
```typescript
const schema = {
title: 'string',
slug: 'string',
summary: 'string', // Index the description or a generated summary
tags: 'string[]',
// Store metadata (image, date) but DO NOT index it for search
};
```
**Tip:** Orama allows you to store data without indexing it. In our JSON generation script, we only feed Orama the _searchable_ text. The metadata needed for the UI (like `image` or `date`) can be stored in the `document` payload but ignored by the indexing engine.
#### 2. AI-Powered "Semantic Compression"
This is the interesting part. Instead of indexing your raw Markdown (which is full of noise like "the", "and", code blocks, and imports), use an LLM at build time to generate a **Search Profile** for each post.
**The Concept:**
1. **Read** the full blog post content during the build.
2. **Pass** it to a cheap LLM (like GPT-4o-mini or a local Llama 3 via Ollama).
3. **Generate** a dense, keyword-rich summary specifically designed for search.
4. **Index** that summary instead of the body.
**Why this works:**
A 3,000-word tutorial on "Setting up Nginx" might only contain 50 words of unique, searchable signal. By indexing only the signal, you reduce your index size by ~90% while _improving_ relevance because you remove the noise.
**The Build Script (`scripts/generate-search-embeddings.ts`):**
```typescript
// This runs ONLY at build time. No client-side cost.
async function generateSearchIndex() {
const posts = await getCollection('blog');
const searchIndex = [];
for (const post of posts) {
// Check if we have a cached summary to save API costs
let searchContent = post.data.search_summary;
if (!searchContent) {
const { text } = await generateText({
model: openai('gpt-4o-mini'),
prompt: `
Analyze this blog post about "${post.data.title}".
Extract the core technical concepts, libraries mentioned, problems solved,
and key takeaways.
Output a single paragraph of dense, keyword-rich text optimized for a search engine.
Do not use filler words.
Content: ${post.body.substring(0, 8000)}
`,
});
searchContent = text;
}
searchIndex.push({
title: post.data.title,
slug: `/blog/${post.slug}`,
// Index this AI-generated "dense" content instead of the body
content: searchContent,
tags: post.data.tags,
});
}
// Save to JSON...
}
```
#### 3. Chunking Long Content
If you have massive posts (like a "Complete Guide to Linux" with 10 chapters), a single search result pointing to the top of the page is annoying. The user wants to jump to the specific _section_ (like "Permissions").
**The Strategy:**
Don't treat one Post as one Document. Treat one **H2 Section** as one Document.
1. **Split** your markdown by `## H2` headers.
2. **Create** an Orama document for each section.
3. **Add** an anchor hash to the slug (e.g., `/blog/linux-guide#permissions`).
```typescript
// Transform 1 Post into N Search Documents
const records = [];
posts.forEach((post) => {
const sections = splitByH2(post.body); // Helper function
sections.forEach((section) => {
records.push({
title: `${post.data.title} > ${section.heading}`,
slug: `/blog/${post.slug}#${section.slug}`,
content: section.text.substring(0, 1000), // or use AI summary of section
});
});
});
```
This increases the number of documents (which increases index size slightly), but it improves the user experience for long-form content.
#### Summary of Techniques
| Technique | Implementation Difficulty | Index Size Reduction | Relevance Impact |
| :-------------------------- | :------------------------ | :------------------------ | :---------------- |
| **Schema Optimization** | Low | High (20-40%) | Neutral |
| **Stop Word Removal** | Low (Orama config) | Moderate (10-15%) | Neutral |
| **AI Semantic Compression** | High (Requires API/LLM) | **Massive (80-90%)** | **Positive** |
| **Section Chunking** | Moderate | Negative (Increases Size) | **Positive (UX)** |
For a personal blog, **Technique #2 (AI Compression)** is the sweet spot. It keeps your search index tiny (sub-50KB) while making your search feel "smart" because it matches concepts, not just exact keywords.
## Conclusion
For a developer blog, portfolio, or documentation site under 500 pages, **Orama + Astro Endpoints** is the best option. You get instant, app-like search without the infrastructure headache.
But if you are building the next Wikipedia, don't make your users download the whole database.
---
## What's next?
This is part of a series of posts on implementing search for static sites:
- The Right Way to Add Orama Search to Astro (**_you are here_**) -- simple, zero-config search for small to medium sites
- [Why I Switched from Orama to Pagefind](/blog/pagefind-astro) -- chunked index for better scalability
- [Meilisearch is the Best Search You'll Never Need](/blog/meilisearch-astro) -- server-side search with advanced features
- [Why I Didn't Use Google Programmable Search](/blog/google-custom-search-astro) -- the hidden costs and indexing delays that make it impractical
- [I Tried 4 Search Engines So You Don't Have To](/blog/astro-search-comparison) -- comprehensive comparison from a small blog perspective
All with practical examples from a real production blog.
---
---
title: Honey, I Shrunk my Frontend Bundle Size!
description: How I replaced Lottie with Canvas sprites, migrated to Preact, and removed heavy dependencies to cut over 300KB from my bundle.
type: blog
url: https://sarthakmishra.com/blog/optimizing-astro-bundle-size
date: 2026-01-11
tags: ['performance', 'preact', 'astro', 'optimization', 'typescript', 'canvas']
author: Sarthak Mishra
excerpt: I audited my site's bundle and found a 200KB monster hiding in the sidebar. Here is how I replaced Lottie with Canvas sprites, migrated to Preact, and eliminated blocking scripts to make the site fly.
---
# Honey, I Shrunk my Frontend Bundle Size!
We love dropping big libraries on small problems. Need an animated icon? Install `lottie-react`. Need a toast? Install `sonner`. Need a hover effect? Install `react-spring`.
It works until you look at the bundle.
After adding `rollup-plugin-visualizer`, I found my sidebar icons alone were pulling ~200KB of JavaScript. React added another ~40KB. Time to trim.
(null);
// Pure Canvas animation
return ;
}
```
## Migration 1: React to Preact (~40KB saved)
For a [content-heavy site](/blog/building-perfect-toc-component), I do not need full React. Preact gives a near-identical API at ~3KB. I began with compat mode, then moved components to Preact primitives over time.
```javascript
// astro.config.mjs
export default defineConfig({
integrations: [preact({ compat: true })],
vite: {
resolve: {
alias: [
{ find: 'react-dom/server', replacement: 'preact-render-to-string' },
{
find: 'preact/compat/server',
replacement: 'preact-render-to-string',
},
],
},
},
});
```
## Migration 2: Ditching react-spring (~20KB saved)
I was using `react-spring` for simple hover transitions. A physics engine for a 50px move is wasteful. Native CSS transitions with a custom cubic-bezier did the job.
**Before:**
```typescript
const style = useSpring({ transform: `translateY(${position}px)` });
```
**After:**
```typescript
```
## Migration 3: Custom toast (~20KB saved)
`sonner` is great, but overkill here. I replaced it with ~120 lines of vanilla TypeScript that manipulates the DOM directly.
```typescript
toast.error('Something went wrong');
```
Here's the full implementation:
```typescript
/**
* Standalone toast notifications for use in Astro pages and scripts
* This is a vanilla TS implementation that doesn't depend on React
*/
type ToastType = 'error' | 'warning' | 'success' | 'info';
const styleConfig = {
error: {
iconColor: '#DC2626',
iconPath:
'',
},
warning: {
iconColor: '#F59E0B',
iconPath:
'',
},
success: {
iconColor: '#16A34A',
iconPath:
'',
},
info: {
iconColor: '#2563EB',
iconPath:
'',
},
};
let toastContainer: HTMLDivElement | null = null;
function getToastContainer(): HTMLDivElement | null {
if (typeof document === 'undefined') return null;
if (!toastContainer || !document.body.contains(toastContainer)) {
toastContainer = document.createElement('div');
toastContainer.className = 'toast-container'; // Fixed positioning, top-right
document.body.appendChild(toastContainer);
}
return toastContainer;
}
function createToast(type: ToastType, message: string): void {
const container = getToastContainer();
if (!container) return;
const config = styleConfig[type];
const wrapper = document.createElement('div');
wrapper.className = 'toast-wrapper'; // Slide-in animation
wrapper.innerHTML = `
${message}
`;
container.appendChild(wrapper);
// Animate in
requestAnimationFrame(() => {
wrapper.classList.add('visible');
});
const removeToast = () => {
wrapper.classList.remove('visible');
setTimeout(() => wrapper.remove(), 200);
};
const closeBtn = wrapper.querySelector('.toast-close');
closeBtn?.addEventListener('click', removeToast);
// Auto-dismiss after 5 seconds
setTimeout(removeToast, 5000);
}
export const toast = {
error: (message: string) => createToast('error', message),
warning: (message: string) => createToast('warning', message),
success: (message: string) => createToast('success', message),
info: (message: string) => createToast('info', message),
};
```
Same UX, zero dependencies, works outside the component tree.
## Optimizing the critical path
PageSpeed showed **2,769ms** max critical path latency. The main offenders: analytics and my [pixelation effect](/blog/building-pixelation-hover-effect) loading too early.
### 1) Defer analytics
Load analytics in `requestIdleCallback`, fall back to `load` if unavailable.
```html
```
### 2) Smarter pixelation
- Disable on mobile (performance over aesthetics)
- Batch reads/writes with `requestAnimationFrame`
- Use `offsetWidth` when possible to leverage cached layout
## The results
- Lottie removal: ~230KB saved
- React → Preact: ~40KB saved
- Spring + sonner: ~40KB saved
- **Total:** ~310KB (gzipped)
Critical path latency dropped from ~2.7s to non-blocking. The site feels snappier. Animations stay smooth on low-end devices because they are CSS-native or Canvas-based.
): Promise {
// Convert AVIF/SVG to PNG base64 for the API
const imageBase64 = await convertImageToPngBase64(imagePath);
const { text } = await generateText({
model: openrouter.chat('google/gemini-3-flash-preview'),
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: 'Generate a concise, descriptive alt text for this image. Max 10-15 words. Focus on the main subject. Return only the text.',
},
{
type: 'image',
image: imageBase64,
},
],
},
],
});
return text.trim();
}
```
One technical hurdle: APIs rarely accept optimized formats like AVIF. I had to include a step using `sharp` to convert images to PNG buffers on the fly before sending them out.
## Smart Caching
[Processing hundreds of images](/blog/automating-image-placeholders) on every build would be slow and expensive. I implemented a [content-addressable cache](/blog/semantic-recommendations) using SHA256 hashing.
The script hashes the file content directly to detect changes:
```typescript
function getFileHash(filePath: string): string {
const content = fs.readFileSync(filePath);
return crypto.createHash('sha256').update(content).digest('hex');
}
```
If the hash matches what is in my `image-metadata.json`, we skip the API call. This means subsequent builds are nearly instant, and I only pay when I add or modify an image.
**One detail to watch out for:** If you update an image's content but run the script without the AI flag (e.g., just to update placeholders), you must ensure you don't save the new hash with the old text. My script explicitly clears the cached alt text if the hash changes but AI generation is skipped, preventing stale descriptions from persisting.
## Integration with Astro
The output of the script is a simple JSON file mapping filenames to metadata:
```json
{
"sarthak-photo.avif": {
"altText": "Black and white selfie of a smiling man wearing sunglasses on a beach.",
"hash": "22150ba448..."
}
}
```
Now, instead of manually passing `alt` props every time, my components look it up automatically.
### The PostImage Component
In my `PostImage.astro` component, I import the metadata and perform a lookup based on the filename.
```astro
---
const { thumbnail, alt } = Astro.props;
// Lookup metadata by filename
const filename = thumbnail;
const metadata = filename
? (imageMetadata as Record)[filename]
: null;
// Allow manual override, otherwise use AI text
const altText = metadata?.altText || alt || '';
---
```
This pattern extends to my Carousels and Project Cards. If I forget to add an alt prop manually, the AI coverage kicks in.
## Handling Conflicts
Since I map metadata by filename, duplicates in different directories were a risk. I added a pre-validation step that scans the filesystem and kills the build if it detects ambiguous filenames.
```typescript
if (conflicts.length > 0) {
console.error('❌ Filename conflicts found:');
conflicts.forEach(c => console.error(` ${c}`));
process.exit(1);
}
```
It's a strict rule, but it keeps the lookup logic fast and simple.
## The Result
I added a script to my `package.json` to run this before the main build.
```json
"scripts": {
"alt-text": "tsx scripts/generate-image-metadata.ts --alt-texts",
"build": "tsx scripts/generate-image-metadata.ts --alt-texts && astro build"
}
```
The result is comprehensive accessibility coverage with zero runtime overhead. The site remains a static build, the images remain optimized, but now screen readers actually get useful context instead of filenames or silence.
It is a small automation, but it solves a persistent problem. And honestly, the AI is often better at describing the images than I am.
---
---
title: Building a Smooth Pixelation Reveal Effect with Canvas
description: How I built a high-performance, interactive pixelation effect for image transitions using HTML5 Canvas and custom easing interactions.
type: blog
url: https://sarthakmishra.com/blog/building-pixelation-hover-effect
date: 2026-01-09
tags: ['javascript', 'canvas', 'typescript', 'animation', 'ui-design', 'performance']
author: Sarthak Mishra
excerpt: CSS filters weren't giving me the control I needed. Here is how I used the Canvas API to built a smooth, 60fps pixelation reveal effect that works perfectly with Astro view transitions.
---
# Building a Smooth Pixelation Reveal Effect with Canvas
I wanted to add a specific interaction to my blog cards: images that start as a retro, blocky mess and snap into high-fidelity focus when you hover over them.
CSS `backdrop-filter` or `image-rendering` properties can get close, but they lack the granular control needed for a smooth transition animation.
Here is how I built a performant, framework-agnostic pixelation effect.
## The Concept
The logic behind pixelation is surprisingly simple. You don't need complex shaders or WebGL. You just need to abuse the `drawImage` method.
1. **Shrink it:** Draw the image onto a canvas at a tiny fraction of its size (e.g., 5% width).
2. **Enlarge it:** Draw that tiny version back onto the full-size canvas.
3. **Important:** Disable image smoothing.
When you scale an image up with smoothing disabled, the browser renders hard edges between the data points (Nearest Neighbor interpolation), creating the perfect pixelated look.
## The Implementation
I wrapped the logic in a TypeScript class to handle state, events, and the animation loop.
### The Draw Loop
The core algorithm calculates a `pixelFactor` based on the current animation progress.
```typescript
// 1. Turn off smoothing to get that blocky look
this.ctx.imageSmoothingEnabled = false;
// 2. Calculate the 'tiny' dimensions based on the pixelation level
// level = 1.0 (full quality) -> 0.01 (giant pixels)
const w = Math.floor(this.width * level);
const h = Math.floor(this.height * level);
// 3. Draw down to the tiny size (offscreen or just logic)
this.ctx.drawImage(this.img, 0, 0, w, h);
// 4. Draw back up to full size
this.ctx.drawImage(
this.canvas, // Source is the canvas itself (containing the tiny image)
0,
0,
w,
h, // Source rect
0,
0,
this.width,
this.height // Dest rect
);
```
### Adding Physics (Easing)
Linear animations feel robotic. To make the reveal feel satisfying, I used a custom easing function. The goal is to spend more time in the "semi-pixelated" state and snap quickly into full focus at the end.
I track a `progress` variable from 0 to 1. When the user hovers, I animate progress to 1. When they leave, I animate it back to 0.
```typescript
// Cubic ease-in-out for that snappy feel
function easeInOut(p: number) {
return p < 0.5 ? 4 * p * p * p : 1 - Math.pow(-2 * p + 2, 3) / 2;
}
// In the animation loop:
const eased = easeInOut(this.progress);
const pixelFactor = 1 + (MAX_PIXEL_SIZE - 1) * (1 - eased);
const level = 1 / pixelFactor;
this.draw(level);
```
### Handling High DPI Screens
One "gotcha" with Canvas is that it looks blurry on Retina/High-DPI displays by default. You have to manually account for `window.devicePixelRatio`.
I set the internal canvas dimensions (the buffer) to `width * dpr`, but style it with CSS to the original width.
```typescript
const dpr = window.devicePixelRatio || 1;
this.canvas.width = rect.width * dpr;
this.canvas.height = rect.height * dpr;
this.ctx.scale(dpr, dpr);
```
## Integrating with Astro (or React)
Since I'm using Astro, I wanted this to work without [heavy framework overhead](/blog/optimizing-astro-bundle-size). I attach the effect to any container with a `data-pixelate` attribute.
The markup is simple:
```html
```
And the initialization script runs on page load. Because I'm using [Astro's View Transitions](/blog/building-animated-sprite-hero) with client-side navigation, I have to make sure to clean up old instances or re-initialize on navigation.
```typescript
document.addEventListener('astro:page-load', () => {
const containers = document.querySelectorAll('[data-pixelate]');
containers.forEach(container => new PixelateEffect(container));
});
```
## Performance & Optimization
Canvas operations are generally fast, but doing them on hover for multiple cards simultaneously is a bit much.
**1. Lazy Rendering:** I only run the `drawImage` loop if the visual state actually changed. If the animation is idle (fully clear or fully pixelated), we stop the `requestAnimationFrame` loop.
**2. [Image Preloading](/blog/automating-image-placeholders):** You can't draw an image that hasn't loaded. The class checks `img.complete`. If it's false, it waits for the `onload` event before attempting to draw.
**3. Offscreen Buffering:** For the intermediate "tiny" image, using a separate offscreen canvas can sometimes be cleaner than drawing to the main canvas and overwriting it, though modern browsers are quite good at optimizing the draw-on-self pattern.
## The Result
The result is a highly performant interaction that adds character without distracting from the content. It works smoothly at 60fps and gracefully degrades if the canvas fails (the original image is just CSS-hidden behind it).
Sometimes the native web APIs are still the best tool for the job.
---
---
title: How to generate blurry placeholders for your images
description: How I eliminated layout shift and improved perceived performance by generating 20px base64 placeholders automatically at build time.
type: blog
url: https://sarthakmishra.com/blog/automating-image-placeholders
date: 2026-01-08
tags: ['astro', 'performance', 'automation', 'typescript', 'ux-engineering', 'image-optimization']
author: Sarthak Mishra
excerpt: Waiting for images to load is bad user experience. Watching the layout jump around is worse. Here is how I automated the generation of tiny, base64 placeholders to fix both problems without manual work.
---
# How to generate blurry placeholders for your images
When building a portfolio site heavily reliant on visuals, you hit a specific performance wall: images take time to load.
On a fast connection, you might not notice it. But on a spotty mobile network, the user experience degrades fast. You get empty white boxes, or worse, massive layout shifts as images pop in and push text around.
The standard solution is lazy loading or setting explicit aspect ratios. It helps with the layout shift, but it doesn't solve the empty box problem.
I wanted something better: instant visual feedback. A [blurry, low-resolution version](/blog/building-pixelation-hover-effect) of the image that loads immediately with the HTML, then cross-fades into the real image. This is a pattern used by Medium, Unsplash, and Next.js.
Here is how I built an automated pipeline to handle it.
## The Strategy: 20px Base64 Thumbnails
The logic is straightforward:
1. Take every image in the project.
2. Generate a tiny (20px) version of it.
3. Convert that tiny version to a base64 string.
4. Inject that string directly into the component's CSS.
Because the data is inline (base64), it requires zero extra HTTP requests. Because it is 20px, the payload is negligible.
## The Build Script
I needed a script that runs before the build, scans my assets, and generates a metadata file. I used a two-tier approach for generation: `ffmpeg` for speed and quality, falling back to `sharp` if that fails.
Here is the core generation logic:
```typescript
async function generatePlaceholder(imagePath: string): Promise {
try {
// Try ffmpeg first (better quality/speed)
const base64String = execSync(
`ffmpeg -nostdin -i "${imagePath}" -vf "scale=20:-1" -f image2pipe -c:v mjpeg - 2>/dev/null | base64 -w 0`,
{ encoding: 'utf8', maxBuffer: 10 * 1024 * 1024 }
);
return `data:image/jpeg;base64,${base64String}`;
} catch (error) {
// Fallback to sharp
try {
const buffer = await sharp(imagePath)
.resize(20, 20, { fit: 'inside', withoutEnlargement: true })
.jpeg({ quality: 50 })
.toBuffer();
const base64 = buffer.toString('base64');
return `data:image/jpeg;base64,${base64}`;
} catch (sharpError) {
console.error(`Failed: ${imagePath}`, sharpError);
return null;
}
}
}
```
This output is saved to `src/data/image-metadata.json`. It maps filenames to their base64 placeholders and a content hash (more on that later).
```json
{
"sarthak-photo.avif": {
"placeholder": "data:image/jpeg;base64,/9j/4AAQSkZJRg...",
"hash": "22150ba448a81f2ea638a5f2a980268b061cb32758e1a818eb2c43cda8a76db4"
}
}
```
## Integrating with Astro Components
To render the placeholders, I updated my `PostImage` component to look up these placeholders.
The trick is using the placeholder as a CSS `background-image` on the container div. This ensures it displays immediately, occupying the exact same space the final image will fill.
```astro
---
const { thumbnail, image, ...props } = Astro.props;
// Lookup metadata by filename
const filename = thumbnail;
const metadata = filename
? (imageMetadata as Record)[filename]
: null;
const placeholder = metadata?.placeholder || '';
---
```
When the high-res `Picture` loads, it simply covers the background placeholder.
## Optimizing the Build Process
Running image processing on every build is slow. To keep my deployment times under control, I implemented a caching system using file hashing.
Before processing an image, the script calculates a [SHA256 hash](/blog/automating-image-alt-text) based on the file's modification time, size, and a sample of its content.
```typescript
function getFileHash(filePath: string): string {
const stats = fs.statSync(filePath);
const content = fs.readFileSync(filePath);
return sha256(
`${stats.mtime.getTime()}-${stats.size}-${content.toString('hex').substring(0, 1000)}`
);
}
```
If the hash in `image-metadata.json` matches the file on disk, we skip it. This cache hit rate is usually around 95%+, meaning sub-2-second processing times for the entire image library.
## Handling Collisions
One real-world edge case I hit immediately: filename conflicts. If you have `header.jpg` in two different blog folders, a flat JSON map will break.
I added a pre-validation step to the script. It scans all directories and throws a hard error if it detects duplicate filenames, forcing me to name files descriptively (which is better for SEO anyway).
```typescript
const conflicts: string[] = [];
for (const [filename, paths] of filenameMap.entries()) {
if (paths.length > 1) {
conflicts.push(`Duplicate: ${filename} in ${paths.join(', ')}`);
}
}
if (conflicts.length > 0) {
process.exit(1); // Fail the build
}
```
## The Result
The impact on user experience is immediate.
1. **Zero Layout Shift:** Because the container has the background image immediately, the aspect ratio is reserved before the browser even parses the `img` tag.
2. **Perceived Speed:** The user sees _something_ instantly. The page feels responsive, even if the actual heavy assets are still coming down the pipe.
3. **Developer Experience:** I don't have to manually create thumbnails or configure lazy loading plugins. I drop an image in the folder, reference it in my markdown, and the build system handles the rest.
This is the kind of automation I love: high impact on the frontend, zero friction in the workflow.
---
---
title: How to generate animated pixel art with AI and Python
description: How I built a pixel-perfect animated hero section using Midjourney, a custom Python processing pipeline, and Astro View Transitions.
type: blog
url: https://sarthakmishra.com/blog/building-animated-sprite-hero
date: 2026-01-07
tags: ['astro', 'python', 'design', 'midjourney', 'automation', 'web-animation']
author: Sarthak Mishra
excerpt: I wanted a retro, pixel-perfect animation for my hero section. Here is how I turned a Midjourney video into an optimized sprite sheet using Python and canvas.
---
# How to generate animated pixel art with AI and Python
I recently decided to overhaul my site's hero section. I wanted something that felt personal and retro--specifically, an animated pixel art profile.
The easy way would be to find a GIF and slap it in an `![]()
` tag. But GIFs are bulky, hard to control, and don't scale well without artifacts. I wanted full control over the frame rate, the color palette, and the rendering sharpness.
So I built a pipeline that takes an AI-generated video, processes it with Python into an optimized sprite sheet, which is then rendered inside Astro with a canvas element.
## The Pipeline
The workflow looks like this:
1. **Creation:** Text-to-Image (ChatGPT) → Polish (Photoshop) → Animation (Midjourney).
2. **Processing:** Python script to extract frames, quantize colors, and fix jitter.
3. **Rendering:** HTML Canvas.
## Step 0: How to generate the video
Getting consistent pixel art from AI is harder than it looks. Most models try to add "detail" that ruins the 8-bit aesthetic, or they hallucinate a grid that doesn't align.
I started with ChatGPT to get the base composition. It took a bit of prompt engineering to force it into a strict low-res style.
```text
First attempt (with an image of myself for reference):
Please generate an image in the style of Retro 8-bit pixel art; pixel avatar; indie 2d game character art; bright orange flat background; for the given reference photograph. The image should be 1:1 aspect ratio.
Edit (with the same reference image and the failed attempt):
Try again; but enlarge the pixel blocks, and make the image flatter; make the background yellow
```
{newsletterTitle} #{newsletterNumber}
```
**SVGs to PNGs:** Every icon, logo, and graphic had to be converted to PNG and hosted statically. I put everything in Astro's `public/emails/` folder so they can be referenced with absolute URLs.
**Tailwind limitations:** While react-email supports Tailwind, many utilities don't work in email contexts. I ended up writing custom CSS for mobile responsiveness:
```css
@media (max-width: 520px) {
.main-container {
padding-left: 24px !important;
padding-right: 24px !important;
}
.button {
width: 100% !important;
text-align: center !important;
}
}
```
Here are the three final templates I ended up with:
` component lets you define custom styles for every element:
```tsx
{message}
```
This gives me the flexibility to write newsletter content in familiar markdown while ensuring it renders beautifully across email clients. Headers, lists, code blocks, blockquotes - everything just works.
{
const response = await fetch(`${this.baseUrl}/values/${key}`, {
headers: { Authorization: `Bearer ${this.apiToken}` },
});
return response.status === 404 ? null : await response.text();
}
}
```
**2. MDX File Parsing:** The script scans `src/data/newsletter/` for MDX files, parses frontmatter, and finds the latest non-draft issue:
```typescript
function getLatestNewsletterIssue(): ParsedNewsletter | null {
const files = fs
.readdirSync(newsletterDir)
.filter(file => file.endsWith('.mdx') && file.startsWith('issue-'))
.sort();
// Skip drafts and find highest issue number
for (const file of files) {
const { frontmatter } = parseFrontmatter(content);
if (frontmatter.draft === true) continue;
// Return latest by issue number
}
}
```
**3. Smart Scheduling Logic:** Calculates the next Thursday at 10am PDT, accounting for timezone conversion and edge cases. Note that this implementation is simplified for this article; a more robust solution would use a library like `date-fns-tz` to handle daylight saving changes automatically.
```typescript
function getNextThursdayPDT(): Date {
const nowPDT = new Date(
new Date().toLocaleString('en-US', {
timeZone: 'America/Los_Angeles',
})
);
const currentDay = nowPDT.getDay();
const currentHour = nowPDT.getHours();
let daysUntilThursday: number;
if (currentDay < 4) {
daysUntilThursday = 4 - currentDay;
} else if (currentDay === 4 && currentHour < 10) {
daysUntilThursday = 0; // Today, but before 10am
} else {
daysUntilThursday = 7 - currentDay + 4; // Next week
}
// Convert back to UTC for Resend
return new Date(nextThursday.getTime() + 7 * 60 * 60 * 1000);
}
```
### Individual Email Scheduling
One of Resend's limitations is that batch emails cannot be scheduled - only individual emails can use the `scheduledAt` parameter. The script handles this by creating individual scheduling requests:
```typescript
const emailPromises = subscribers.map(async email => {
const unsubscribeUrl = await getUnsubscribeUrl(email, kv, baseUrl);
return await resend.emails.send({
from: 'Sarthak Mishra ',
to: [email],
subject: `${newsletter.title} #${newsletter.issueNumber.toString().padStart(3, '0')}`,
scheduledAt: scheduledDate.toISOString(),
headers: {
'List-Unsubscribe': `<${unsubscribeUrl}>`,
'List-Unsubscribe-Post': 'List-Unsubscribe=One-Click',
'X-Entity-Ref-ID': `newsletter-${newsletter.issueNumber}-${Date.now()}`,
},
react: Newsletter({
/* ... */
}),
});
});
await Promise.all(emailPromises);
```
### Duplicate Prevention & Status Tracking
The script maintains comprehensive state in KV to prevent duplicate sends:
```typescript
// Check status before scheduling
const status = await checkNewsletterStatus(issueNumber, kv);
if (status.sent || status.scheduled) {
console.log('Newsletter already processed');
return;
}
// Mark as scheduled after successful scheduling
await kv.put(
`newsletter:scheduled:${issueNumber}`,
JSON.stringify({
scheduledAt: new Date().toISOString(),
scheduledFor: scheduledDate.toISOString(),
})
);
```
### Deployment Integration
The script runs as part of my deployment pipeline. After pushing a new newsletter MDX file, the deployment process automatically:
1. Builds the site with the new newsletter archive page
2. Runs the scheduling script to check for new issues
3. Schedules emails if a new non-draft issue is found
4. Deploys everything to Cloudflare Pages
This means I can write a newsletter in MDX, commit it, and the entire scheduling happens automatically. The script validates everything: Resend's 30-day scheduling limit, subscriber status, duplicate prevention, and proper unsubscribe token generation.
## The Result
I now have a newsletter system that:
- Costs $0 (Cloudflare Workers & KV + Resend - Free tiers)
- Handles subscription, confirmation, and unsubscribe flows automatically
- Creates SEO-friendly archive pages for each issue
- Schedules newsletters automatically on deployment
Writing a newsletter feels exactly like writing a blog post. I can focus on content instead of fighting with tools.
P.S. This is my first [newsletter](/newsletter/issue-1).
---
---
title: Setup your Ubuntu Server in Under 10 Minutes
description: Skip the manual server setup with a three-stage Ansible approach that gets you from fresh Ubuntu install to production-ready in under 10 minutes.
type: blog
url: https://sarthakmishra.com/blog/setup-ubuntu-server-in-10-minutes
date: 2025-07-29
tags: ['ubuntu', 'server-setup', 'ansible', 'automation', 'devops', 'linux', 'infrastructure']
author: Sarthak Mishra
excerpt: Most Ubuntu server guides take 45+ minutes and leave you with an insecure setup. Here's my three-stage automation that consistently delivers production-ready servers in under 10 minutes - including all the security hardening everyone forgets.
---
# Setup your Ubuntu Server in Under 10 Minutes
I've deployed dozens of Ubuntu servers over the last couple of years, and I can tell you this: most "quick setup" guides aren't actually quick when you factor in all the security hardening, service configuration, and inevitable do-overs when something breaks.
After countless hours of trial and error, I've distilled my server setup process into three automated stages that consistently get a [production-ready Ubuntu server](/blog/setting-up-traefik-from-local-to-production) running in under 10 minutes.
> This is a follow-up to my [previous article](/blog/self-hosting-done-right) on self-hosting with Ansible and Docker, focusing specifically on the initial server preparation that everyone seems to gloss over.
## The Reality Check: Why Most Quick Setup Guides Fail
Most Ubuntu server tutorials follow the same pattern: install the OS, run a few commands, maybe configure SSH, and call it done. But in practice, you're missing crucial pieces:
- **No proper user management** - still logging in as root or ubuntu user
- **Incomplete security hardening** - SSH is wide open, no firewall rules
- **Manual Docker installation** - copy-pasting curl commands from the internet
- **No network isolation** - everything exposed to the world
- **Zero automation** - when something breaks, you start from scratch
The tutorials that do cover these aspects take 45+ minutes and require you to stay glued to your terminal.
## My Three-Stage Approach
Instead of one massive script that tries to do everything, I break server setup into three distinct phases:
1. **Stage 1: Bootstrap** (2 minutes) - Secure SSH access and admin user creation
2. **Stage 2: System Hardening** (5 minutes) - Security, networking, and Docker installation
3. **Stage 3: Production Ready** (3 minutes) - Service deployment and monitoring
Each stage builds on the previous one, with clear validation points and the ability to resume from any stage if something goes wrong.
## Stage 1: Bootstrap - Getting Secure Access (2 minutes)
The bootstrap stage solves the chicken-and-egg problem of server access. You start with a fresh Ubuntu server accessible via password authentication, and end up with proper SSH key access and a dedicated admin user.
Here's what happens behind the scenes:
```yaml
- name: Create admin user for final access
ansible.builtin.user:
name: '{{ admin_username }}'
shell: /bin/bash
groups: sudo
append: true
state: present
- name: Configure passwordless sudo for admin user
ansible.builtin.lineinfile:
path: /etc/sudoers.d/{{ admin_username }}
line: '{{ admin_username }} ALL=(ALL) NOPASSWD: ALL'
create: true
validate: 'visudo -cf %s'
mode: '0440'
- name: Install SSH public key for admin user
ansible.builtin.authorized_key:
user: '{{ admin_username }}'
key: '{{ admin_ssh_public_key }}'
state: present
- name: Disable SSH password authentication
ansible.builtin.lineinfile:
path: /etc/ssh/sshd_config
regexp: '{{ item.regexp }}'
line: '{{ item.line }}'
validate: 'sshd -t -f %s'
loop:
- { regexp: '^#?PasswordAuthentication', line: 'PasswordAuthentication no' }
- { regexp: '^#?PubkeyAuthentication', line: 'PubkeyAuthentication yes' }
notify: restart sshd
```
**Critical insight**: Most people make the mistake of disabling password authentication before testing SSH key access. My bootstrap stage keeps both users accessible via SSH keys initially, giving you a safety net.
Run this stage with:
```bash
ansible-playbook -i inventory/stage1-bootstrap.yml playbooks/bootstrap.yml
```
After 2 minutes, you have an admin user with SSH keys and password authentication disabled - but you can still access the server if something goes wrong.
## Stage 2: System Hardening - Production Security (5 minutes)
This is where most DIY approaches fall apart. You need to coordinate multiple security changes without locking yourself out. My second stage handles:
- **SSH hardening** with proper timeout and connection limits
- **UFW firewall** configuration with Docker integration
- **Tailscale VPN setup** for secure remote access
- **Docker installation** with security configurations
The key here is the [UFW-Docker](https://github.com/chaifeng/ufw-docker) integration. Standard Ubuntu setups have a massive security hole: Docker bypasses UFW rules entirely, potentially exposing container ports to the internet.
```yaml
- name: Add UFW-Docker security rules
ansible.builtin.blockinfile:
path: /etc/ufw/after.rules
block: |
# BEGIN UFW AND DOCKER
*filter
:DOCKER-USER - [0:0]
-A DOCKER-USER -j ufw-user-forward
-A DOCKER-USER -j RETURN -s 10.0.0.0/8
-A DOCKER-USER -j RETURN -s 172.16.0.0/12
-A DOCKER-USER -j RETURN -s 192.168.0.0/16
# Block external access to Docker containers
-A DOCKER-USER -j DROP
```
**Tailscale installation is also automated**, giving you secure VPN access even if the direct IP becomes unreachable:
```yaml
- name: Install Tailscale
ansible.builtin.apt:
name: tailscale
state: latest
- name: Authentication with Tailscale
ansible.builtin.command:
cmd: 'tailscale up --authkey={{ tailscale_auth_key }} --hostname={{ server_hostname }} --accept-risk=lose-ssh'
- name: Reboot the server to apply updates and hardening
ansible.builtin.reboot:
```
Run this stage with:
```bash
ansible-playbook -i inventory/stage2-hardened.yml playbooks/system-setup.yml
```
After this stage completes, the server reboots automatically. You can then access it via either its direct IP or Tailscale hostname.
## Stage 3: Production Ready - Service Deployment (3 minutes)
The final stage focuses on getting your server ready for actual workloads:
- **Storage configuration** (if using multiple drives)
- **Service deployment** with Docker Compose
- **Monitoring setup** with proper log aggregation
- **Backup system configuration**
For most self-hosted setups, this means launching your core services like Portainer, Traefik, monitoring tools, etc.
```bash
ansible-playbook -i inventory/stage3-production.yml playbooks/services.yml
```
## The Gotchas I Learned the Hard Way
**1. Docker's UFW Bypass**
Docker creates its own iptables rules that completely bypass UFW. Without proper configuration, your containers might be exposed to the internet even with UFW enabled. The fix requires modifying `/etc/ufw/after.rules` to force Docker traffic through UFW's chain.
**2. SSH Configuration Testing**
Always test SSH configuration syntax with `sshd -t` before restarting the service. I've locked myself out of servers because of typos in `sshd_config`. The playbook snippet in Stage 1 includes a `validate` step to prevent this.
**3. Tailscale Authentication**
The `--authkey` is essential for non-interactive setup, and `--accept-risk=lose-ssh` is crucial when setting up Tailscale on a remote server. Without them, the command waits for an interactive confirmation that never comes. The Stage 2 playbook example now includes both.
**4. UFW Default Policies**
Set UFW policies before enabling rules. The order matters:
```bash
ufw default deny incoming
ufw default allow outgoing
ufw enable
```
**5. Admin User Sudo**
Don't forget to add your admin user to the `sudo` group AND configure passwordless sudo. Otherwise, automation breaks on privilege escalation. The Stage 1 playbook example now handles this for you.
## Performance Insights
**Total time breakdown:**
- Stage 1 (Bootstrap): ~2 minutes
- Stage 2 (Hardening): ~5 minutes
- Automated reboot: ~1 minute
- Stage 3 (Services): ~3 minutes
**Network transfer:** ~200MB for all packages (Docker, Tailscale, system updates)
**What takes the longest:** Docker installation and system package updates. Using a local apt cache or mirror can cut this in half.
## Why This Approach Works
**1. Idempotent Operations**
Every Ansible task can be run multiple times safely. If something fails, just re-run that stage.
**2. Clear Validation Points**
Each stage ends with connectivity verification. You know immediately if something went wrong.
**3. Rollback Safety**
SSH configuration changes include syntax validation and backup creation. Firewall rules are reset to known-good states.
**4. Scalable**
Need to set up 10 servers? Change the inventory file and run the same playbooks.
## Common Alternatives and Why They Fall Short
**Cloud-init scripts**: These run once during instance creation. If they fail, you're rebuilding from scratch.
**Docker containers for the OS**: Adds complexity and doesn't address the underlying security hardening needs.
**Manual setup scripts**: Work once, break when Ubuntu versions change or packages get updated.
**Configuration management alternatives (Puppet, Chef)**: Overkill for initial server setup, primarily designed for ongoing configuration drift management.
## Getting Started
The complete automation is available in my [self-hosted repository](https://github.com/SarthakMishra/self-hosted). To adapt it for your environment:
1. **Fork and customize the inventory files** with your server IPs and user details
2. **Update group_vars/all.yml** with your domain and network settings
3. **Create group_vars/vault.yml** with your SSH keys and Tailscale auth key
4. **Run the three stages** in order
The key is understanding that automation isn't just about speed - it's about repeatability and confidence. When your server setup is fully automated, you can experiment freely, knowing you can always get back to a known-good state in under 10 minutes.
---
In my next post, I'll cover the service deployment patterns I use to get applications running reliably on this hardened foundation, including proper secrets management, backup automation, and zero-downtime updates.
> Have questions about the setup process or want to share your own automation approaches? - Let me know on [Twitter](https://twitter.com/MishraMishry).
---
---
title: Self-hosting done right with Ansible & Docker
description: Learning infrastructure automation the hard way - from multi-node cluster-f*ck to a production-ready, single-node setup with Ansible.
type: blog
url: https://sarthakmishra.com/blog/self-hosting-done-right
date: 2025-07-23
tags: ['self-hosting', 'infrastructure', 'ansible', 'devops', 'learning']
author: Sarthak Mishra
excerpt: I thought a multi-node Docker Swarm cluster was the answer to my self-hosting dreams. I was wrong. Here's how I learned Infrastructure as Code by making every mistake possible.
---
# Self-hosting done right with Ansible & Docker
Like most developers, I've spent years using cloud services. AWS, DigitalOcean, Vercel—they're convenient, predictable, and someone else's problem when things break.
But there's an undeniable appeal to owning your infrastructure. Full control, no vendor lock-in, and the satisfaction of building something from the ground up.
So, naturally, I decided to [build my own self-hosted infrastructure](/blog/setup-ubuntu-server-in-10-minutes). How hard could it be?
Spoiler alert: **Very f\*cking hard.**
> TL;DR: the complete, up-to-date configuration for my entire setup is available in this [GitHub repo](https://github.com/SarthakMishra/self-hosted). I'll be updating it continuously as I learn and discover new things about self-hosting.
## Some Context
This wasn't my first attempt at self-hosting. I've deployed [Coolify](https://coolify.io) on DigitalOcean for quick app deployments and set up [Mailcow](https://mailcow.email) on Hetzner for email campaigns. I was already running most of the services I'll mention in my local Docker setup.
But Docker on Windows/WSL is a pain. The networking quirks, file system performance issues, and random crashes pushed me to find a more robust solution.
The dream was simple: **reliable, automated infrastructure that I could scale and maintain with ease.**
## Mistake #1: Over-Engineering with Kubernetes and Docker Swarm
Like any developer who's read too many articles about microservices, I started with what seemed like the "right" solution: [K3s](https://k3s.io), a lightweight Kubernetes distribution.
${message}
|
|