Why is my Next.js page HTML so large?

Open view-source on any App Router page and scroll to the bottom. After the HTML you will find self.__next_f.push([1,"..."]) scripts. That is the RSC payload, the serialized component tree React uses to hydrate the page and to navigate without a full reload. Every element a server component renders appears in it as data, props included.

So a server component that renders an inline <svg> 49 times puts that path in the document 98 times. On my page today the payload is 261 KB of the 460 KB document.

Compression hides this. The live page is 33 KB with brotli and 45 KB with gzip, because repeated strings are what compressors handle best. That is why the problem never showed up as a slow download. The phone still has to decompress, parse and hold every byte, and that cost lands on the main thread of the slowest device that opens the page.

How do I find what is making the page heavy?

Weigh it by parts before changing anything. My first guess was the usual one, large props passed into client components and duplicated into the payload. Every component on /projects was a server component, so that guess was wrong, and work planned around it would have saved nothing.

This script fetches a page and prints the HTML size, the RSC payload, the JSON-LD, and the long attribute values that repeat most:

// weigh.mjs: node weigh.mjs https://example.com/page
const html = await (await fetch(process.argv[2])).text()
const kb = (n) => `${(n / 1024).toFixed(0)} KB`

const flight = [...html.matchAll(/self\.__next_f\.push\(\[1,"([\s\S]*?)"\]\)<\/script>/g)]
  .reduce((sum, m) => sum + m[1].length, 0)
const jsonLd = [...html.matchAll(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/g)]
  .reduce((sum, m) => sum + m[1].length, 0)
console.log(`HTML ${kb(html.length)}, RSC payload ${kb(flight)}, JSON-LD ${kb(jsonLd)}`)

// Long SVG paths and class strings, in the HTML (d="…") and the payload (\"d\":\"…\").
const counts = new Map()
for (const m of html.matchAll(/(?:\bd|class|className)\\?"?[=:]\\?"([^"\\]{80,})/g)) {
  counts.set(m[1], (counts.get(m[1]) ?? 0) + 1)
}
;[...counts]
  .map(([value, n]) => ({ n, bytes: n * value.length, value: value.slice(0, 50) }))
  .sort((a, b) => b.bytes - a.bytes)
  .slice(0, 5)
  .forEach((r) => console.log(`${kb(r.bytes).padStart(7)}  ${String(r.n).padStart(3)}x  ${r.value}…`))

On the 854 KB version the top line was a single 2,115-character path, the GitHub mark, 98 times.

Fix 1: draw repeated icons once with an SVG sprite

The GitHub link on each row rendered its own <svg> with the full path. I moved the paths into one hidden sprite rendered by the root layout, and every icon now points at it with <use>:

const SYMBOLS = {
  github: { viewBox: '0 0 438.549 438.549', d: 'M409.132 114.573c…' },
} as const

function symbolIcon(id: keyof typeof SYMBOLS) {
  return function SymbolIcon(props: React.SVGProps<SVGSVGElement>) {
    return (
      <svg viewBox={SYMBOLS[id].viewBox} fill="currentColor" {...props}>
        <use href={`#icon-${id}`} />
      </svg>
    )
  }
}

export function IconSprite() {
  return (
    <svg aria-hidden="true" width="0" height="0" style={{ position: 'absolute', overflow: 'hidden' }}>
      {Object.entries(SYMBOLS).map(([id, { viewBox, d }]) => (
        <symbol key={id} id={`icon-${id}`} viewBox={viewBox}>
          <path fill="currentColor" d={d} />
        </symbol>
      ))}
    </svg>
  )
}

<IconSprite /> goes once in app/layout.tsx. Each use is now under 200 bytes instead of 2.1 KB, and currentColor still follows the link's text colour. /projects went from 854 KB to 676 KB.

The sprite is not free. It adds about 3 KB to every page, so a page with one GitHub link got slightly heavier. That trade only pays off for icons that repeat. A single logo stays inline.

Fix 2: JSON-LD on a list page should link, not copy

The page's ItemList structured data held a full SoftwareApplication for each of the 49 projects: name, description, image, author, operating system. Each project's own page already publishes that same entity. The list was 40 KB of JSON-LD, shipped twice.

Google's carousel documentation describes this case as a summary page: list items that point at detail pages carry only position and url. The helper now returns the short form whenever a detail page exists:

export function projectListItem(position: number, detailUrl: string | undefined, item: object) {
  return detailUrl
    ? { '@type': 'ListItem', position, url: detailUrl }
    : { '@type': 'ListItem', position, item }
}

The JSON-LD went from 40 KB to 10 KB, and no entity is described in two places that could drift apart.

Fix 3: name the class strings every row repeats

Each row carried long Tailwind strings for its grid, thumbnail, title, counter, tag and links, and they shipped twice like everything else. In Tailwind v4 a string that repeats down a long list can become a named utility, variants included:

@utility feed-row {
  @apply relative flex flex-col gap-3 border-b border-slate-300/10 py-6 last:border-none
    sm:grid sm:grid-cols-[200px_1fr] sm:gap-6 sm:py-7 lg:grid-cols-[240px_1fr];
}

One catch: if the element's classes go through cn() (tailwind-merge), keep colours out of the utility. tailwind-merge can't see inside feed-link, so it won't know that a caller's text-teal-300 should replace the colour inside it. My feed-link holds layout only, and the colours stay on the element.

To prove nothing changed visually, I compared computed styles for every row part between a local next start build and production. They matched except for a rounding difference in one oklab channel.

Fixes 2 and 3 together took the page from 658 KB to 459 KB. (A rebuild between the two passes measured 658 KB, not the 676 KB from the sprite step.)

Fix 4: prefetch links on intent, not on scroll

This one doesn't change the HTML. Next.js prefetches every <Link> as it enters the viewport. On a page with a link per row, one visit to /projects fired 36 route prefetches, about 1 MB decoded, and on Vercel each one is a CDN request plus ISR reads.

The Next.js docs suggest a hover-only pattern for this. I added focus and touchstart so keyboard and phone users still get a head start:

'use client'
import Link from 'next/link'
import { useState } from 'react'

export function IntentLink(props: React.ComponentProps<typeof Link>) {
  const [intent, setIntent] = useState(false)
  return (
    <Link
      {...props}
      prefetch={intent ? null : false}
      onMouseEnter={() => setIntent(true)}
      onFocus={() => setIntent(true)}
      onTouchStart={() => setIntent(true)}
    />
  )
}

null is the default prefetch behaviour; false turns it off until there is intent. The <a href> in the HTML is identical, so crawlers see the same links.

How do I keep the page from growing back?

Set a budget that fails. My production check reads every URL in the sitemap and fails any page over 520 KB of uncompressed HTML:

const HTML_BUDGET_KB = 520
const kb = Math.round(html.length / 1024)
if (kb > HTML_BUDGET_KB) fail(`${url}: ${kb} KB of HTML, budget ${HTML_BUDGET_KB} KB`)

And pin each fix with a test. The sprite test renders each repeated icon and fails if one use is over 200 bytes or doesn't reference its symbol.

The budget alone wasn't enough. Writing this article, I ran the weighing script again and found a link arrow drawn inline 82 times on the page, about 11 KB, still under the budget and so never flagged. It is a symbol now too, and the page is 448 KB. So I re-run the weigh script whenever a list page changes, not only when the budget fails.

It is one of the ten phBot plugins I maintain. The original was written by JellyBitz; my version adds commands, walk-script commands and a new tab. Every argument and default below is read from the source of xControl 1.9.1, not from memory.

How do I set up xControl?

1. Download xControl.py from the plugin repo and put it in the Plugins folder inside your phBot folder.
2. Restart phBot. The plugin gets its own tab, named xControl.
3. On each bot, type the leader's character name into the box under Party Leaders and press Add. The list is saved per character, so you do this once.
4. Log the leader in and type a command in chat.

Three rules decide whether a command runs:

• Only leaders are obeyed. A message from anyone not on that bot's leader list is ignored, whoever sends it.
• Commands are uppercase. The plugin compares text exactly, so START works and start does nothing.
• The chat channel picks who reacts. Party chat reaches every bot in the party. A private message reaches one bot. All chat reaches every bot close enough to see it.

Arguments are separated by spaces. Below, <name> is required and [name] is optional.

Which commands start, stop and move the bots?

How do I change where the bots train?

How do I teleport, return or revive the party?

Which commands help in PvP and job runs?

EQUIP trader followed by START on a caravan party is the whole job-suit swap in two messages.

What about items, party and chat?

Can xControl run from a walk script?

Yes. xControl adds five commands that phBot walk scripts (.txt) can call. They use commas, like every script command:

A script that gathers a party before a run looks like this:

pinvite,Alpha,Bravo,Charlie
wait_party,4,90000
chat,party,START

The bot that runs it sends START in party chat, and every party bot with that character in its leader list starts.

Why does a command do nothing?

Check these in order:

• The sender is not on that bot's leader list. The list belongs to each bot. Adding the leader on one character does not add it on the others.
• The command was typed in lowercase, or with extra text before it. The message has to start with the command.
• The bot cannot hear the channel. A private message reaches one bot. Party chat only reaches bots in your party.
• The item or NPC name does not match. EQUIP and USE search by part of the name; RECALL and TP <destination> need the NPC near the bot.

The phBot log shows a line for most commands, including the failures (Plugin: ...), so read it first. If you are writing your own plugin on top of this, the phBot Python API guide covers the handle_chat hook xControl is built on.

Why does Google crawl opengraph-image at all?

In the App Router, an opengraph-image.tsx file next to a page becomes a route that returns a PNG, and Next.js writes its URL into the page's head:

<meta property="og:image" content="https://www.mohamedgado.com/projects/quizz-app/opengraph-image">

Googlebot follows URLs it finds, and this is a plain absolute URL on your own domain. It fetches the image, sees that it isn't a page, and doesn't index it as one. Search Console reports that outcome as "Crawled – currently not indexed". That label covers "we looked and passed", and for a social card that is the right result.

So if the only rows in that bucket are image routes, feeds and icons, nothing is broken. The rows worth reading are the HTML pages. For those, the status usually means Google judged the content thin or too close to another page. I wrote about that case in Discovered – currently not indexed.

Why only some of my files had noindex

I already had a rule from April that sent noindex on the root files:

// next.config.js, headers()
{
  source:
    '/:file(favicon\\.ico|icon\\.svg|apple-touch-icon\\.png|opengraph-image|twitter-image)',
  headers: [{ key: 'X-Robots-Tag', value: 'noindex' }],
}

/:file(...) matches one path segment. /opengraph-image matches it, and /projects/quizz-app/opengraph-image doesn't, so every per-page card went out with no robots header at all:

$ curl -sI https://www.mohamedgado.com/projects/quizz-app/opengraph-image | grep -i x-robots
$ curl -sI https://www.mohamedgado.com/opengraph-image | grep -i x-robots
X-Robots-Tag: noindex

The fix is two more rules: one for the dynamic route folders, one for the feeds:

{
  source: '/:dir(projects|writing)/:slug/:file(opengraph-image|twitter-image)',
  headers: [{ key: 'X-Robots-Tag', value: 'noindex' }],
},
{
  source: '/:feed(feed\\.xml|writing/feed\\.xml)',
  headers: [{ key: 'X-Robots-Tag', value: 'noindex' }],
},

I named the two folders instead of writing /:path*/opengraph-image. A pattern that open is one typo away from matching a page, and a noindex header on a page is far worse than a missing one on a PNG.

A test that runs real paths through the matcher

Reading the config by eye is how the gap got in. The test compiles each rule's source with the same path-to-regexp build that Next.js ships, then checks paths in both directions:

const { pathToRegexp } = load('next/dist/compiled/path-to-regexp')
const covered = (path: string) =>
  noindex.some((r) => pathToRegexp(r.source, [], { strict: true, delimiter: '/' }).test(path))

for (const path of ['/projects/quizz-app/opengraph-image', '/writing/feed.xml' /* ... */])
  assert.ok(covered(path), `${path} is indexable`)
for (const path of ['/', '/projects/quizz-app', '/writing'])
  assert.ok(!covered(path), `${path} is a page but sends noindex`)

Before the fix it failed on /projects/quizz-app/opengraph-image is indexable. The second loop matters as much as the first. It is what stops a later, broader rule from quietly deindexing the pages.

Does noindex clear the Search Console report?

Not necessarily, and this is the part I didn't expect. My root /opengraph-image and /favicon.ico have sent noindex since April, and they were still listed under "Crawled – currently not indexed" in November. The report shows Google's last verdict on a URL. For a file it has already decided not to index, a new header may never change which bucket the row sits in.

So the header doesn't tidy the report. What it does:

• It makes the decision explicit instead of leaving it to Google's judgment, and it keeps the cards out of Google Images, where a text card on a dark background is worth nothing.
• It doesn't block anyone. It isn't a robots.txt disallow, so Googlebot can still fetch the image, and so can Facebook, X, Slack and LinkedIn when they build a link preview. The card keeps working everywhere it is meant to.

What I'd do on a new Next.js site

1. Ignore image, feed and icon rows in "Crawled – currently not indexed". Read the HTML rows.
2. If you want the files marked, send X-Robots-Tag: noindex on every depth of opengraph-image and twitter-image, not only the root.
3. Never disallow those routes in robots.txt. A crawler that can't fetch the image can't show your card.
4. Pin the rule with a test that also asserts the pages are not matched.

Originally published at mohamedgado.com.

Five clients, one message

Most people bot more than one character, and every phBot client loads its own copy of the plugin. A global chat line or a unique spawn notice reaches all of them at once. Without coordination, five clients means five identical Telegram messages.

The clients share nothing in memory, but they share a disk. Before sending a shared event, each copy tries to create the same marker file, named after a hash of the event and its text:

SHARED_EVENTS = {'global', 'notice', 'unique', 'all_chat'}
SHARED_WINDOW = {'global': 30.0, 'notice': 30.0, 'unique': 120.0, 'all_chat': 15.0}

def _claim_shared(key, window):
    h = hashlib.md5(key.encode('utf-8', 'replace')).hexdigest()
    path = os.path.join(_DEDUP_DIR, h)
    try:
        if time.time() - os.stat(path).st_mtime < window:
            return False          # someone sent this inside the window
    except FileNotFoundError:
        pass
    try:
        fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
        os.close(fd)
        return True               # we created it, we send
    except FileExistsError:
        return False              # another client won the race

O_CREAT | O_EXCL is the whole trick. The operating system creates the file only if it does not exist, and does so atomically, so exactly one client gets True even when all five arrive in the same millisecond. No lock server, no sockets between clients. (The real function also re-claims an expired marker by touching it, and returns True on any unexpected error, so a disk problem costs a duplicate, never a lost alert.)

The window is per event type. A unique spawn gets 120 seconds because the same notice repeats; all-chat gets 15 because people really do say the same thing twice. A cleanup pass deletes markers older than an hour, at most once every five minutes, so the folder never grows.

A cooldown for everything else

Per-character events, like a party invite or a trade request, don't need cross-client dedup, but they can still fire in bursts. Those go through a per-event cooldown on a monotonic clock, 60 seconds by default:

def _throttled(event_type):
    now = time.monotonic()
    with _throttle_lock:
        if now - _last_sent.get(event_type, 0) < config.get('cooldown_sec', 60):
            return True
        _last_sent[event_type] = now
        return False

time.monotonic() instead of time.time() means a clock change can't freeze or flood the alerts. Private messages pass force=True and skip the cooldown. Nobody wants the second PM from a friend held back for a minute.

Fire on the edge, not the level

"HP is 0" stays true for as long as the character lies dead. Checking it on every tick would send a message every tick until the cooldown caught it. The plugin keeps the previous value and fires only on the change from alive to dead:

if _prev_hp is not None and _prev_hp > 0 and hp == 0:
    notify('death', 'Character died', extras=[('Last HP', _prev_hp)])
_prev_hp = hp

Potions work the same way: one message when the count drops from some to none, not one per check while the bag stays empty. The is not None guard matters on login, when there is no previous value yet and a fresh character at 0 potions shouldn't count as running out.

Never send from the game thread

phBot calls handle_chat and event_loop on its own thread. A Telegram request that takes three seconds there freezes the bot for three seconds. So notify() only puts a small dict on a queue.Queue, and one daemon thread does the network work, with three tries per channel and a short back-off between them:

def notify(event_type, text, force=False, extras=None):
    if not config.get('events', {}).get(event_type, False):
        return
    if event_type in SHARED_EVENTS:
        if not _claim_shared(event_type + '|' + (text or ''), SHARED_WINDOW.get(event_type, 30.0)):
            return
    elif not force and _throttled(event_type):
        return
    _send_q.put({'type': event_type, 'text': text or '', 'extras': extras or []})

notify is also the public API. Another plugin can import xNotify and call xNotify.notify('trade_start', 'Trade opened') to reuse the same channels, dedup and cooldowns.

Why Discord said 403 and Telegram didn't

Telegram worked on the first try: a form POST to https://api.telegram.org/bot<token>/sendMessage with chat_id and text. The Discord webhook, with a correct URL, failed with HTTP 403 and Cloudflare error code 1010.

Error 1010 means Cloudflare blocked the request based on the client's browser signature. Python's urllib sends User-Agent: Python-urllib/3.x, and Discord's edge rejects it. Setting any honest custom agent fixed it:

def _http_post(url, data_bytes, headers):
    # Cloudflare (Discord) blocks the default urllib User-Agent with 403/1010.
    headers = dict(headers)
    headers.setdefault('User-Agent', 'xNotify/%s (+phBot plugin)' % pVersion)
    req = urllib.request.Request(url, data=data_bytes, headers=headers, method='POST')

The plugin uses only the standard library because phBot's embedded Python makes installing requests awkward for users. If your own webhook calls from Python get a 403 with a correct URL, check the User-Agent before the token.

Telling a real unique from a champion

The first version flagged every champion and giant near the character as a unique, because it matched several rarity codes. phBot's mob rarity field uses 0 for general, 1 champion, 3 giant, 4 party champion, 5 elite and 6 unique. Only 6 is a real unique, so that is the default unique_types, with a name and keyword list as a fallback for server-specific bosses. The plugin README had said 8 until this week. The code was right, the docs weren't.

What I'd keep for any game-bot notifier

1. Dedup across processes with an exclusive file create when the clients share a disk. It's simpler than any IPC.
2. Fire on transitions, not states.
3. Keep network calls off the thread that runs the game.
4. Set a User-Agent on every urllib request you send to a Cloudflare-fronted API.

The code is in Vette1123/phbot-plugins, along with nine other plugins.

Originally published at mohamedgado.com.

Why did http://mohamedgado.com take two redirects?

This started as a crawl finding. Typing http://mohamedgado.com took two hops:

http://mohamedgado.com/   308 -> https://mohamedgado.com/
https://mohamedgado.com/  308 -> https://www.mohamedgado.com/

Vercel upgrades every HTTP request to HTTPS on the same host first, with a 308. Their encryption docs say this redirect is an industry standard and cannot be disabled. Then the apex redirects to www. You can't merge the two hops, and you shouldn't try: the HSTS preload list requires that a site listening on port 80 redirects to HTTPS on the same host before it goes anywhere else.

What you can remove is the HTTP request itself. A browser that has the domain on its built-in HSTS preload list never sends http:// at all. It rewrites the URL to https:// before the first byte leaves the machine. The Vercel docs make the same point in their HSTS note: a site on the list is reached over HTTPS right away instead of following the redirect the CDN issues.

What does Vercel send for HSTS by default?

The docs give two defaults. The .vercel.app deployment URLs get the full preload header, and the whole vercel.app domain is already preloaded. Custom domains get HSTS for that one hostname only:

Strict-Transport-Security: max-age=63072000;

hstspreload.org rejects that. Its API returned two errors for my domain: No includeSubDomains directive and No preload directive. The docs also say you can change the header with custom response headers in your project, so I did.

Setting the header in next.config.js

The header goes in the rule that covers every route, next to the rest of the security headers:

// next.config.js
async headers() {
  return [
    {
      source: '/:path*',
      headers: [
        {
          key: 'Strict-Transport-Security',
          value: 'max-age=63072000; includeSubDomains; preload',
        },
        // Content-Security-Policy, X-Content-Type-Options, ...
      ],
    },
  ]
}

Before you add includeSubDomains, list every hostname in your DNS that serves web traffic. The directive tells browsers to refuse plain HTTP on all of them, including names you forgot about. I read the records at my registrar. There were five web hosts: the apex, www, and short subdomains for my apps Masareef, Rafiq and Nafis. All were on Vercel and all on HTTPS. The rest were TXT and MX records for email, which HSTS doesn't affect. One HTTP-only subdomain, such as an email click-tracking CNAME, would have broken the moment a browser saw the new header.

After the deploy, www was correct:

$ curl -sI https://www.mohamedgado.com | grep -i strict
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
$ curl -sI https://mohamedgado.com | grep -i strict
Strict-Transport-Security: max-age=63072000

The apex had not changed.

Why the apex redirect ignored headers()

In the Vercel dashboard, the apex was set to "Redirect to Another Domain", 308, to www. That is the setup Vercel suggests when you add a domain, and it works. But the redirect is answered at the domain level, before any request reaches your deployment's routing. Your headers(), redirects() and middleware never run for it, so the response carries Vercel's default HSTS and nothing from your config.

Preload checks the apex, because the list entry is the registrable domain. So the header has to be on the response the apex actually sends, and here that response is a redirect.

Moving the apex redirect into next.config.js

The fix is to let the deployment answer the apex and redirect it in code, where headers() applies. I added a host-guarded rule at the top of redirects():

// next.config.js
async redirects() {
  return [
    {
      source: '/:path*',
      has: [{ type: 'host', value: 'mohamedgado.com' }],
      destination: 'https://www.mohamedgado.com/:path*',
      permanent: true, // 308, same as the domain-level redirect
    },
    // ...the rest
  ]
}

One thing to check before you trust that has rule: an unanchored pattern mohamedgado.com would also match www.mohamedgado.com and loop forever. Next.js compiles has values as new RegExp(`^${value}$`) (you can read it in prepare-destination.js), so the match is exact.

Then the order of operations matters:

1. Deploy the code redirect first. It does nothing yet, because the domain-level redirect still answers the apex before the deployment sees it.
2. In Vercel, open Domains, edit the apex, and switch it from "Redirect to Another Domain" to "Connect to an environment: Production".

Done in that order, the apex redirects to www the whole time. Done the other way round, there is a window where the apex serves the whole site as a duplicate of www.

How do I check it worked?

The apex now answers through the deployment, so it carries the full header and keeps the path and query:

$ curl -sI "https://mohamedgado.com/projects/masareef?x=1"
HTTP/1.1 308 Permanent Redirect
Location: https://www.mohamedgado.com/projects/masareef?x=1
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload

hstspreload.org has an API that runs the same checks as its form:

$ curl -s "https://hstspreload.org/api/v2/preloadable?domain=mohamedgado.com"
{ "errors": [], "warnings": [] }

I also pinned both rules with tests that read next.config.js. One asserts that the every-route header has a max-age of at least a year plus includeSubDomains and preload. The other asserts that the apex has exactly one 308 to www. A later edit can't quietly drop either one.

Should you submit to the preload list?

Getting the domain eligible was the useful part. The header now reaches every host, and the apex takes one redirect, not two. Submitting is a separate decision, and for a portfolio I didn't submit.

The list does nothing for search rankings. Google already crawls the HTTPS URLs. A returning visitor's browser already remembers the HSTS header for two years after the first visit. Most browsers now try HTTPS first when someone types a bare domain. What preload adds is protection on the very first visit from someone who types http:// on a network that strips TLS. The cost is a commitment: every future subdomain has to serve HTTPS, and getting off the list takes months.

If you run a login or payments on the domain, submit. If not, the header alone covers almost everything.

This post walks the demo's real code from the 2023 version to the current one, and explains why the check moved.

What did the 2023 middleware do?

This is the whole of the demo's auth wiring, on next 13.4.7 and @clerk/nextjs 4.21:

// middleware.ts
import { authMiddleware } from '@clerk/nextjs'

export default authMiddleware({
  publicRoutes: ['/api/:path*'],
})

export const config = {
  matcher: ['/((?!.*\\..*|_next).*)', '/', '/(api|trpc)(.*)'],
}

authMiddleware was deny by default. Every path the matcher covered needed a session unless publicRoutes named it. The dashboard at / and the profile page at /profile had no auth code of their own:

// app/(dashboard)/profile/page.tsx
import { UserProfile } from '@clerk/nextjs'

export default function ProfilePage() {
  return (
    <main className='flex space-y-4 p-4'>
      <UserProfile />
    </main>
  )
}

The sign-in and sign-up pages are optional catch-all routes, app/(auth)/(routes)/login/[[...sign-in]]/page.tsx and register/[[...sign-up]], each rendering Clerk's <SignIn /> or <SignUp />. Two environment variables point Clerk at them:

NEXT_PUBLIC_CLERK_SIGN_IN_URL=/login
NEXT_PUBLIC_CLERK_SIGN_UP_URL=/register

That part has not changed. The catch-all segment is still how Clerk's components handle their own sub-steps (verification, factor-two) under one route.

Where did that setup leak?

Deny by default felt safe, and it mostly was. Two holes were sitting in that file anyway.

The matcher skipped any path with a dot in it. /((?!.*\\..*|_next).*) excludes a path when anything after the first slash contains a ., which was meant for favicon.ico and image files. A dynamic page like /u/[handle] visited as /u/john.doe also contains a dot, so the middleware never ran and the page rendered for anyone. Nothing in the demo had such a route, but nothing stopped the next one from being added. Clerk's current matcher skips only paths ending in a known static extension (css, js, png, woff2 and so on).

publicRoutes: ['/api/:path*'] made every API route public. That was deliberate, so webhooks could reach the app. But any Route Handler added later under /api that read user data had to check the session itself, and nothing reminded anyone to.

Both holes have the same shape. The protection was a list of URL patterns in one file, and the code it protected lived somewhere else. Clerk's migration guide names a third case that a path list cannot cover: Server Functions (including Server Actions) are called by ID, not by path. A Server Action defined in a protected folder and called from a public page arrives at the middleware as a request for the public page.

What replaced authMiddleware?

@clerk/nextjs v5 introduced clerkMiddleware(), imported from @clerk/nextjs/server. Clerk's upgrade guide states the primary change: it "does not protect any routes by default". A straight port of the demo looked like this, using createRouteMatcher() to list the protected paths:

// middleware.ts, @clerk/nextjs v5 and v6
import { clerkMiddleware, createRouteMatcher } from '@clerk/nextjs/server'

const isProtected = createRouteMatcher(['/', '/profile(.*)'])

export default clerkMiddleware(async (auth, req) => {
  if (isProtected(req)) await auth.protect()
})

Note the inversion. The 2023 file listed public routes; this one lists protected routes. A new page that nobody adds to the list ships public, which is the failure deny by default was meant to prevent.

How should a Clerk app protect routes on Next.js 16?

Clerk's current docs (last updated 10 November 2026) deprecate createRouteMatcher(). It still works and logs a one-time warning in development, and it will be removed in the next major version of @clerk/nextjs. The recommendation is to protect each server-side resource where it runs.

Next.js 16 renamed Middleware to Proxy, so the file is proxy.ts. The behaviour is the same and only the name changed. It keeps clerkMiddleware() with no auth logic, because Clerk still needs it to read the session:

// proxy.ts (middleware.ts on Next.js 15 and earlier)
import { clerkMiddleware } from '@clerk/nextjs/server'

export default clerkMiddleware()

export const config = {
  matcher: [
    // Skip Next.js internals and static files, unless found in search params
    '/((?!_next|[^?]*\\.(?:html?|css|js(?!on)|jpe?g|webp|png|gif|svg|ttf|woff2?|ico|csv|docx?|xlsx?|zip|webmanifest)).*)',
    // Always run for API routes
    '/(api|trpc)(.*)',
    // Clerk's frontend API proxy path
    '/__clerk/(.*)',
  ],
}

The check moves into the page:

// app/(dashboard)/profile/page.tsx
import { UserProfile } from '@clerk/nextjs'
import { auth } from '@clerk/nextjs/server'

export default async function ProfilePage() {
  await auth.protect()

  return (
    <main className='flex space-y-4 p-4'>
      <UserProfile />
    </main>
  )
}

Route Handlers and Server Functions get the same line at the top. auth.protect() answers per request type: a signed-out page request is redirected to the sign-in URL, a Server Action gets a 401, and any other non-document request, such as a Route Handler, gets a 404.

Two details catch people on the way.

• Put the check in the page, not only the layout. A layout does not re-render on client-side navigation between its children, and Route Handlers and Server Functions never pass through it. The page is where the data is read, so that is where the check belongs.
• Add a loading.tsx next to protected pages. With Cache Components, which new Next.js apps enable by default, a page that reads request data such as the session needs a Suspense boundary, and loading.tsx provides one. It changes the status code: a signed-out request then gets HTTP 200 with the redirect streamed in, instead of a 307. I hit the same streaming rule from the other side when notFound() returned 200 behind loading.tsx.

Can I keep the redirect in proxy.ts?

Yes, as a speed-up and not as the guard. Clerk's guide allows an early redirect for signed-out users after every resource has its own check:

// proxy.ts
import { clerkMiddleware } from '@clerk/nextjs/server'

export default clerkMiddleware(async (auth, req) => {
  // Not an auth guarantee: it only saves signed-out users a round trip.
  const { pathname } = req.nextUrl
  if (pathname === '/profile' || pathname.startsWith('/profile/')) {
    const { isAuthenticated, redirectToSignIn } = await auth()
    if (!isAuthenticated) return redirectToSignIn()
  }
})

To check that the early redirect is not doing the real work, delete it and test signed out. Every protected page should still redirect, and every protected Route Handler should still refuse. If deleting a line in proxy.ts exposes data, that line was the guard and belongs in the resource.

Clerk also ships @clerk/eslint-plugin with a require-auth-protection rule. It flags App Router pages, Route Handlers and Server Functions under folders you mark protected that never call an auth check. That restores what deny by default gave the 2023 demo: a new page that forgets its check fails lint instead of shipping public.

The migration in four steps

1. Upgrade @clerk/nextjs and replace authMiddleware from @clerk/nextjs with clerkMiddleware from @clerk/nextjs/server.
2. On Next.js 16, rename middleware.ts to proxy.ts and replace the matcher with Clerk's current one.
3. Add await auth.protect() to every page, Route Handler and Server Function that reads user data, with a loading.tsx beside protected pages.
4. Remove the path checks from proxy.ts, test the app signed out, and turn on the lint rule so new routes cannot skip the check.

The demo's source and live dashboard still show the 2023 wiring, which is the starting point for step one.

What does a minimal phBot plugin look like?

Here is a complete plugin. It adds a tab with a checkbox, and when it is ticked, it answers "ping" in party chat with "pong".

from phBot import *
import QtBind

pName = 'xHello'
pVersion = '1.0.0'

gui = QtBind.init(__name__, pName)
QtBind.createLabel(gui, 'Replies "pong" when a party member types "ping".', 10, 10)
cbxEnabled = QtBind.createCheckBox(gui, 'cbxEnabled_clicked', 'Enabled', 10, 40)
enabled = False

def cbxEnabled_clicked(checked):
    global enabled
    enabled = checked

def handle_chat(t, player, msg):
    if enabled and t == 4 and msg.strip().lower() == 'ping':
        phBotChat.Party('pong')

log('Plugin: ' + pName + ' v' + pVersion + ' loaded')

Save it as xHello.py in Plugins/, then restart phBot or use Reload Plugins. A new xHello tab appears.

Three things in it apply to every plugin:

• pName and pVersion are module-level names. My plugins also set pAuthor and pUrl, the raw GitHub URL of the file.
• QtBind.init(__name__, pName) creates the plugin's tab and returns a handle. Every UI call takes that handle first.
• Callbacks are passed by name, as a string: 'cbxEnabled_clicked'. phBot looks the function up in your module when the checkbox changes, so the name has to match exactly.

Which functions does phBot call in a plugin?

You opt in to an event by defining a function with the right name. These are the ones my plugins use:

The return value of the two packet hooks is easy to get wrong. Every hook in my plugins ends with an explicit return True; a hook that falls off the end returns None, and you should not have to find out what phBot does with that.

The chat types in handle_chat are numbers. xControl maps them like this: 1 all, 2 private, 3 guild, 4 party, 5 global, 6 note, 7 stall, 11 union. Union messages arrive with the guild name in front, so xControl strips everything up to the first ": " before parsing.

How do I read the character and the world?

The phBot module exposes getters that return plain dicts and lists:

• get_character_data(): name, server, region, position, HP, MP and more. My xNotify plugin reads hp, mp, name, region and x/y from it.
• get_inventory(), get_pets(), get_pet_inventory() and get_item(model_id) for items.
• get_party(), get_monsters() and get_position() for the world around you.
• get_config_dir(): the folder where phBot keeps its configs. Plugins store their own JSON there, for example get_config_dir() + pName + '.json'.

Treat every getter as able to return nothing. Right after login, or during a teleport, get_character_data() can be empty, so xNotify reads it as get_character_data() or {} inside a try.

How do I make the bot do something?

• log(text) writes to phBot's log tab. It is the first debugging tool you have.
• start_bot() and stop_bot() toggle botting. set_training_position(region, x, y, z) moves the training area, and set_training_script(path) swaps the walk script.
• move_to(x, y, z) walks the character. xControl's follow command calls it every loop tick.
• phBotChat.All, .Party, .Guild, .Union, .Global and .Stall send chat. phBotChat.Private(name, msg) and phBotChat.Note(name, msg) take a recipient first.
• inject_joymax(opcode, data, encrypted) sends a raw packet to the server, and inject_silkroad sends one to the client. data is a bytearray.

Packet injection is how most of the interesting plugins work. My Magic Pop spinner, the lightstone crafter and the Jupiter hunter all build packets by hand and send them with inject_joymax. To find the opcodes in the first place, I use xPacketTool, which logs client and server packets with an opcode filter.

How do I run plugin code from a walk script?

Any plugin function that takes one args list can be called from a phBot training script by its name. xPacketTool defines inject(args), so a script line like this greets:

inject,3091,0

phBot splits the line on commas and passes the list, so args[0] is the command name itself, and the values start at args[1]. The function returns 0, as every script command in my plugins does. xControl uses the same mechanism for its chat,party,<message> command.

What goes wrong most often?

• Blocking the loop. event_loop runs on phBot's own thread. A time.sleep or a slow HTTP call there freezes the bot. xNotify, which sends Telegram and Discord alerts, queues them for a background threading.Thread instead.
• Reacting to your own messages. handle_chat also fires for what your character says. xControl remembers each message it sent and skips it once when it comes back.
• UI method names that differ between builds. Some QtBind methods are not present on every phBot build. My plugins look a method up with getattr and skip it when it is missing, rather than crashing the whole plugin on load.

The full plugins, including xControl's party commands and the Caravan runner with its own dashboard, are on the phBot plugins page, with the source on GitHub.

What does each Lemon Squeezy status mean for access?

cancelled is the one people misread. It does not mean "gone". The customer has paid to the end of the period, and Lemon Squeezy moves the subscription to expired when that period ends. Cutting access on the subscription_cancelled webhook takes away days the customer paid for.

How did expired subscribers keep Pro?

Freeby's webhook stored two fields per subscription: the mapped status, and a plan. The plan was derived like this:

const plan: 'pro' | 'free' = status === 'free' ? 'free' : 'pro'

Every paid status, expired included, stored plan: "pro". That alone is fine, since the row records which product was bought. The bug was on the reading side. The client limit and the monthly invoice limit both looked only at the plan:

const limit = LIMITS[entitlement.plan].maxInvoicesPerMonth

entitlement.plan came straight from the row, so an expired subscriber had Infinity invoices and Infinity clients, and always would. The entitlement also had an isActive flag computed from the status, but only the email-sending check used it. Two checks used one field, one check used the other, and nothing forced them to agree.

How did one failed payment cut access?

The status mapping and the access list disagreed about past_due:

// mapLsStatus: the webhook side
case 'past_due':
  return { status: 'past_due', isActive: true } // grace period

// getEntitlement: the read side
export const ACTIVE_STATUSES = ['active', 'on_trial']

The webhook mapper was written with a grace period in mind. The read side, which decides what the app actually does, left past_due out. A customer whose renewal failed for an ordinary reason, such as an expired card, lost Pro while Lemon Squeezy was still retrying the charge.

What is the fix?

Make one pure function decide what a user gets right now, and make plan mean the effective plan, so the 11 places that read it are right without being touched:

export const ACTIVE_STATUSES = ['active', 'on_trial', 'past_due']

export function deriveEntitlement(row, now: Date): Entitlement {
  const inPaidPeriod =
    row.status === 'canceled' &&
    row.currentPeriodEnd !== null &&
    row.currentPeriodEnd > now
  const isActive =
    row.plan === 'pro' && (ACTIVE_STATUSES.includes(row.status) || inPaidPeriod)
  return {
    plan: isActive ? 'pro' : 'free',
    status: row.status,
    isActive,
    cancelAtPeriodEnd: row.cancelAtPeriodEnd || inPaidPeriod,
    currentPeriodEnd: row.currentPeriodEnd,
  }
}

The database read calls it with new Date(). The tests call it with a fixed date, which is why now is a parameter instead of a Date inside the function.

How do I stop the two lists drifting apart again?

The past_due bug existed because the same fact lived in two places. A test now walks every Lemon Squeezy status through the webhook mapper and checks the answer against the access list:

for (const ls of ['active', 'on_trial', 'past_due', 'unpaid', 'paused',
  'cancelled', 'cancelled_immediately', 'expired']) {
  const r = mapLsStatus(ls)
  expect(ACTIVE_STATUSES.includes(r.status)).toBe(r.isActive)
}

Run against the old list, it fails on past_due. Five more tests pin the effective plan: active Pro gets Pro limits, expired Pro gets Free limits, cancelled Pro keeps access until its period ends and loses it after, and paused Pro is Free.

What should I check in my own billing code?

• Find every place that gates a feature, and check whether it reads the stored plan or the current status. If both exist, one of them is wrong somewhere.
• Treat cancelled as "still paid until ends_at", not as "gone".
• Decide on past_due on purpose. A grace period is the usual choice, because most failed renewals are expired cards that the customer fixes.
• Keep the status-to-access decision in one pure function with the clock passed in, so you can test the end of a billing period without waiting a month.

What does minified React error #418 mean?

React 19 reports it as "Hydration failed because the server rendered HTML didn't match the client." During hydration React walks the server HTML and the first client render side by side. If one has a node the other lacks, React cannot reuse the markup. It discards the server tree and renders the whole app again on the client.

In production the message is minified, so all you get is #418 and a link. It names no component and no node. On my site it fired on every production page load, and nothing looked broken, because the client re-render produced a correct page. The cost was invisible: the server HTML was thrown away, and every visitor paid for a full client render. I think this is what my earlier performance audit recorded as "mobile lab bound by react-dom hydration", but I have no before and after measurement to prove that link.

Why does reading matchMedia in useState cause a hydration mismatch?

The server has no window. A hook that wants to run on both sides has to pick a value for the server, and the natural choice is false. The mistake is reading the browser value during the first client render.

This is the shape of my broken hook. It is simplified, not the exact file I had:

import { useEffect, useState } from 'react'

// Broken: the first client render reads the browser, the server cannot.
export function useMediaQuery(query: string): boolean {
  const [matches, setMatches] = useState(() =>
    typeof window === 'undefined' ? false : window.matchMedia(query).matches
  )

  useEffect(() => {
    const mql = window.matchMedia(query)
    const onChange = () => setMatches(mql.matches)
    mql.addEventListener('change', onChange)
    return () => mql.removeEventListener('change', onChange)
  }, [query])

  return matches
}

On the server the initializer returns false. In a desktop browser the same initializer runs during the first client render and returns true. Hydration compares the two trees at that moment, so the value is already different before any effect runs.

The caller made it visible. My spotlight component draws a cursor-following overlay only when the device has a fine pointer:

const hasFinePointer = useMediaQuery('(pointer: fine)')

// ...
{hasFinePointer && (
  <div ref={overlay} className="pointer-events-none fixed inset-0 z-30 ..." />
)}

The server rendered no overlay. The desktop client rendered one extra child, so every sibling after it shifted position. That is enough for #418.

How do I find which component mismatches?

Production will not tell you. Run pnpm dev (or your package manager's dev command) and load the page. The development build of React prints the full, non-minified error, including a component stack with + and - lines that mark the exact mismatched node. The + side is what the client rendered and the - side is what the server sent. For my bug that diff pointed at the spotlight overlay div.

Two things to check once you have the component:

1. Does it read window, document, localStorage, matchMedia or Date.now() while rendering, or in a useState initializer?
2. Does it branch the JSX on that value?

If both are yes, you have found it. Other common causes of #418 are invalid HTML nesting and browser extensions that modify the DOM, but those did not apply here.

A second clue was sitting in my own code. A mounted guard already existed in a command palette trigger, with a comment explaining that the media query mismatched. The spotlight had no such guard. A per-caller workaround for a hook-level bug is the signal to fix the hook.

How do I fix it with useSyncExternalStore?

useSyncExternalStore takes three arguments: subscribe, getSnapshot and getServerSnapshot. From the React docs, getServerSnapshot is used during server rendering and during hydration on the client. That is the property that matters here. React does not call your client getSnapshot for the hydrating render, so the first client render sees the same false the server saw. Right after hydration finishes, React reads the real snapshot and re-renders if it differs. The docs describe this behaviour, and I did not trace it through React's source.

This is the hook as it is now in my repo:

import { useCallback, useMemo, useSyncExternalStore } from 'react'

export function useMediaQuery(query: string): boolean {
  // One MediaQueryList per query, reused by both subscribe and getSnapshot.
  const mql = useMemo(
    () => (typeof window === 'undefined' ? null : window.matchMedia(query)),
    [query]
  )

  const subscribe = useCallback(
    (onStoreChange: () => void) => {
      mql?.addEventListener('change', onStoreChange)
      return () => mql?.removeEventListener('change', onStoreChange)
    },
    [mql]
  )

  return useSyncExternalStore(
    subscribe,
    () => mql?.matches ?? false,
    () => false
  )
}

Three details in it are worth copying:

• The MediaQueryList is created once per query in useMemo. getSnapshot runs on every render, so calling matchMedia inside it would repeat work for no reason.
• subscribe is wrapped in useCallback keyed on the list. A new function identity each render would make React unsubscribe and resubscribe every time.
• The server snapshot is a constant false. That is the contract for callers: the first paint always has false, and the real value arrives in the commit right after.

Do I still need a mounted guard?

No. Once the hook returns false on the first client render, a mounted state flag adds nothing. I deleted the guard in the palette trigger. That component now hides itself on desktop with the Tailwind class sm:hidden, which is CSS and needs no JavaScript value at all.

This points at the cheaper option for many cases. If a media query only changes layout or visibility, use CSS. Keep useMediaQuery for the cases where JavaScript has to act on the result, such as attaching a mouse handler, as my spotlight does.

The trade-off is a one-frame flash. With a false server snapshot, a desktop user gets the touch layout first and the desktop layout after hydration. For my overlay that is invisible, since it is a faint gradient. For a layout that changes the page structure, pick a server default that suits your most common visitor, or use CSS.

How do I confirm the fix worked?

In production, the console on a fresh page load no longer shows #418; the only errors left there come from browser extensions. If it ever comes back, pnpm dev is where to look, since it prints the full hydration warning with its component stack.

One trap while checking: counting JSON-LD tags with a bare grep for application/ld+json double-counts, because the RSC payload embeds the component tree as an escaped string. Count real tags with grep -o '<script type="application/ld+json"' instead, or document.querySelectorAll in the browser.

The rule I took from it: never read window, matchMedia or localStorage in a useState initializer in a component that renders on the server. Use useSyncExternalStore with a server snapshot, or use CSS.

The site this came from is described in the portfolio case study. Another rendering bug from the same site, a 200 status on unknown slugs, is covered in Next.js soft 404: why notFound() returned 200 behind loading.tsx.

The report that started this was short: the images on the Masareef, Rafiq and Nafis project pages looked pixelated and bad. The files on disk were fine. Every screenshot was 1200px wide and sharp. The browser was blurring them on the way to the screen.

Why does will-change: transform make things blurry?

will-change: transform tells the browser to promote the element to its own compositor layer. That layer is drawn into a bitmap, and from then on the compositor moves, rotates and scales the bitmap on the GPU without repainting the content.

That is fast, and it is also the problem. A layer that is rasterised once, upright, has to be resampled to be turned. A bitmap that is resampled at an angle cannot keep a one-pixel edge crisp, and small text is made of one-pixel edges. My notes from the fix put it plainly: a rotated raster of small type is never sharp at any angle, and the angle only decides how much.

Scaling has the same cost. In my case the hover state scaled the card by 1.045 on that same layer. The layer was not re-rasterised at the new size, so the card under the cursor was an upscaled copy of its own bitmap. The one card the visitor was looking at was the softest on the page.

How did I confirm the cause?

I read computed style straight from production in the browser the report came from, a 1512px wide window at a device pixel ratio of 2. One evaluation returned three things: the transform matrix matrix(0.999877, -0.0157…), will-change: transform, and an image request for w=640&q=75. That was the diagnosis before I changed a line.

The matrix is what a rotation of about 0.9 degrees looks like: the cosine is close to 1 and the sine is about 0.0157. It is easy to dismiss that as "almost not rotated". The raster does not care how small the angle is.

The screenshots were part of a fanned deck. I had added the tilt earlier with a comment that kept it under a degree, because past that the captions baked into the images read as crooked. That earlier work measured the caption tops to within 3px. Nobody zoomed into the pixels on a 2x screen, so the blur was designed in and defended by a comment.

What did the CSS look like before and after?

Before, every card was a fanned, overlapping layer:

.shot {
  flex: 0 0 clamp(11rem, 44vw, 15rem);
  margin: 0 -1.5rem 0 0; /* overlap */
  transition: transform 0.28s cubic-bezier(0.2, 0.7, 0.3, 1);
  will-change: transform;
}
.shot:nth-child(odd)  { transform: rotate(-0.9deg); }
.shot:nth-child(even) { transform: rotate(0.9deg); }

.shot:hover,
.shot:focus-within {
  transform: rotate(0deg) translateY(-0.6rem) scale(1.045);
}

After, the cards are flat. No tilt, no scale, no will-change, and the hover lift is a plain translate that lands on whole pixels:

.shot {
  flex: 0 0 clamp(12rem, 58vw, 16.5rem);
  margin: 0;
}

.shot a.image-zoom {
  transition: translate 0.32s cubic-bezier(0.2, 0.8, 0.2, 1);
}

/* Lift, never scale: a whole-pixel translate is the one transform that does
   not resample the screenshot once it lands. */
.shot a.image-zoom:hover,
.shot a.image-zoom:focus-visible {
  translate: 0 -0.375rem;
}

The rule I wrote down afterwards: never rotate or scale raster content that carries small type, and never put will-change: transform on anything that scales. If a card needs to lift, translate it by a whole number of pixels.

Was it only the CSS?

No. There were three more causes, and the CSS was the biggest.

The images were AVIF at quality 75, which left ringing around small UI type. Quality 90 was clean for 6 KB more per card, 25 KB to 31 KB at 640px wide. The cards were also 240px wide, a phone screen shown at a fifth of its size, and clicking one opened the raw file in a new tab. I changed the cards to 264px on desktop and 58vw on a phone, cut the srcset to 384, 640 and 828 at quality 90, and opened a full-height viewer on click.

Because Next.js only serves widths and qualities listed in next.config.js, I added q=90 to images.qualities, and a test now reads that config and fails if any rendered image URL asks for a width or quality the optimizer would reject.

Why were the cover banners blurry as well?

A second report came the same day: "I meant this", with a screenshot of the cover banner at the top of the Masareef page, not the gallery. I had fixed the wrong images first. The lesson there was to look at the first viewport before deciding which images a report means.

The banner problem was about device pixels. The source art was a 1024x500 store graphic. The hero rendered 1086 CSS pixels wide, which is 2172 device pixels on a 2x screen, so the browser stretched the file by about 2.1 times. An earlier "hi-res" version at 2560px wide was an upscale of the same 1024 pixels, so it was soft too, only larger. The file size had been the clue: 75 KB for 2560 pixels of supposed detail.

The check I keep now is to compare a raster's intrinsic width with the rendered width multiplied by the device pixel ratio, in device pixels and not CSS pixels. I redrew the covers from SVG at 3x, which gave a 3072x1500 source, and fixed the sizes attribute, which had assumed a 768px column when the header was 1086px wide.

When should text be an image and when should it be live text?

Redrawing the banner at 3x fixed the 2x screen and exposed the next problem a few hours later. A screenshot of the hero at about 1x showed the wordmark and tagline soft next to the real text above them. At 1x the browser was shrinking 3072 pixels into 768, and shrunk type is filtered just like stretched type.

No file size fixes that, because the words were pixels. I had answered "pixelated" with more pixels twice. The question to ask on the first report is whether the thing is text.

The hero of the five mobile app pages is now live HTML in components/projects/app-banner.tsx: the name, the Arabic name, an accent rule and the tagline are real text over a CSS background of two radial glows and a light SVG grain to avoid banding. The app icon is the only image. The frame is a query container, so the type scales with its own width and one rule covers both the 768px column and a 310px phone column.

The rule that came out of it: text a reader has to read is live text. A raster of type belongs only where the page cannot draw text, such as link-preview cards and store listings. The same lesson says to check sharpness at the pixel density of the screenshot you were sent. My test rig had emulated a 2x laptop, where the 3x cover looked fine, and the report came from a screen at about 1x.

A checklist for blurry images and text in CSS

1. Search the stylesheet for will-change: transform and for any rotate() or scale() on elements with images or text. Remove them from anything that carries small type.
2. Read the computed transform in devtools. A matrix that is not the identity on a card you thought was flat is a lead.
3. Compare the image's intrinsic width with rendered width times device pixel ratio.
4. Test at 1x and at 2x. A fix that is sharp at one density can be soft at the other.
5. If the blurry thing is text, make it text.

You can see the finished pages on the Masareef, Rafiq and Nafis case studies. The site they live on is described in the portfolio write-up.

I found this on my own site, mohamedgado.com. /now, /projects and all 55 case studies were shipping without og:site_name, og:locale, twitter:site and twitter:creator. The root layout set all four. Nothing in the browser looked wrong, and nothing errored.

Does Next.js merge openGraph metadata from the layout?

No, not field by field. The Next.js 16 docs for generateMetadata say that metadata objects exported from several segments of one route are shallowly merged, and that duplicate keys are replaced by the segment closest to the page. They also say it directly for nested fields: openGraph defined in an earlier segment is overwritten by the last segment that defines it.

The docs give the example. The layout sets openGraph.title and openGraph.description. The blog page sets only openGraph.title. The output has the page's og:title and no og:description, because the page's whole openGraph object replaced the layout's.

The same page does inherit when it does not set openGraph at all. A subpage with only a title still gets every Open Graph tag from the layout. The trouble starts the moment a page adds openGraph to change one field, such as the title or the image. That is the normal thing to do on a case study or a blog post, so it hits most real sites.

What did the broken setup look like?

My root layout carried the full set:

// app/layout.tsx
export const metadata: Metadata = {
  openGraph: {
    locale: siteConfig.openGraph.locale,
    siteName: siteConfig.name,
    url: siteConfig.links.websiteUrl,
    type: 'website',
    title: `${siteConfig.name} — ${siteConfig.tagline}`,
    description: siteConfig.description,
  },
  twitter: {
    card: 'summary_large_image',
    creator: siteConfig.twitterTag,
    site: siteConfig.twitterTag,
  },
}

A subpage then set what it needed, which looks harmless. This is a simplified version of what the case-study page did before the helper:

// app/projects/[slug]/page.tsx (before)
return {
  title: project.title,
  openGraph: {
    title: `${project.title} | Mohamed Gado`,
    description,
    url: `/projects/${slug}`,
    images: [ogImage],
  },
  twitter: {
    card: 'summary_large_image',
    title: `${project.title} | Mohamed Gado`,
    description,
  },
}

This page's openGraph has no siteName and no locale, so neither tag is rendered. Its twitter has no site or creator, so those are gone too. The layout's values were replaced, not combined.

How do I check the rendered tags?

Read the HTML the server sends, not the code. A dev tool that shows the metadata object tells you what you wrote, and the question is what crawlers receive. I curl the page and grep for the tag:

curl -s https://www.mohamedgado.com/projects/masareef | grep -o '<meta[^>]*og:site_name[^>]*>'
curl -s https://www.mohamedgado.com/projects/masareef | grep -oE '<meta[^>]*(og:locale|twitter:site|twitter:creator)[^>]*>'

An empty result means the tag is missing. I ran this against production before writing any code, because I wanted to know the gap was real on every subpage and not something I had inferred from reading the merge docs. It was.

What is the fix for Next.js openGraph not inheriting?

The docs suggest pulling shared fields into a variable and spreading it into each page's openGraph. That works, but it leaves it to every page author to remember the spread. I had 55 case studies built from one template plus a few static pages, so I put the whole thing in a function in lib/social-meta.ts:

import type { Metadata } from 'next'
import { siteConfig } from '@/config/site'

type SocialMetaInput = {
  title: string
  description: string
  /** Root-relative path, resolved against the layout's `metadataBase`. */
  path: string
  images: NonNullable<Metadata['openGraph']>['images']
  /** Case studies are articles; set this and `og:type` becomes `article`. */
  article?: { modifiedTime?: string }
}

export function buildSocialMeta({
  title,
  description,
  path,
  images,
  article,
}: SocialMetaInput): Pick<Metadata, 'openGraph' | 'twitter'> {
  const shared = {
    siteName: siteConfig.name,
    locale: siteConfig.openGraph.locale,
    title,
    description,
    url: path,
    images,
  }
  const twitter: Metadata['twitter'] = {
    card: 'summary_large_image',
    site: siteConfig.twitterTag,
    creator: siteConfig.twitterTag,
    title,
    description,
  }
  if (!article) {
    return { openGraph: { ...shared, type: 'website' }, twitter }
  }
  return {
    openGraph: {
      ...shared,
      type: 'article',
      authors: [siteConfig.links.websiteUrl],
      ...(article.modifiedTime ? { modifiedTime: article.modifiedTime } : {}),
    },
    twitter,
  }
}

A page spreads the result into its metadata:

// app/projects/[slug]/page.tsx (after)
return {
  title: project.title,
  alternates: { canonical: `/projects/${slug}` },
  ...buildSocialMeta({
    title: `${project.title} | Mohamed Gado`,
    description,
    path: `/projects/${slug}`,
    images: [ogImage],
  }),
}

Because buildSocialMeta returns both objects complete, the shallow merge no longer matters: the page's openGraph replaces the layout's, and the replacement already contains everything the layout had. The site name, locale and handles come from siteConfig, the same source the layout reads, so they cannot drift apart.

The helper also gave me a single place to add article fields. Case studies now pass article, which sets og:type to article and adds article:modified_time and article:author.

One detail in the helper: I leave twitter.images unset on purpose. My code comment says Next falls back to the openGraph images, with alt text and dimensions included, so the image is declared once.

How do I stop it coming back?

A convention in a README does not survive the next new page. I added a check to the postbuild step. scripts/check-index-parity.ts reads every prerendered page and asserts that it carries og:site_name, og:locale and both Twitter handles. Before the fix it reported 57 failures. After, 0.

The rule written down in my lessons file is short: a page that sets openGraph or twitter goes through buildSocialMeta, and the build fails otherwise. Because the check runs on built HTML, it also catches pages I did not think of, such as a new route someone adds later and forgets to wire up.

What else should I check on pages that override metadata?

Other nested fields behave the same way. The docs name robots next to openGraph as a field that is overwritten wholesale. If a layout sets robots: { index: true, follow: true, googleBot: {...} } and one page sets robots: { index: false }, the page does not keep the googleBot block. Anything that is an object in the Metadata type is a candidate: alternates, icons, twitter, appLinks and so on.

For each one, ask whether the layout sets it and whether any page sets it again. If both are true, the page's version has to be complete. A helper function, a spread of a shared object, or a build-time check on the output are all workable. I picked the helper and the check together, since the helper prevents the mistake and the check proves it.

The change was small, and nothing on the site looked different afterwards. What changed was the HTML that link previews and search engines read for 57 pages. If you set openGraph on any page below your root layout, curl one of those pages today and look for og:site_name.

Why does notFound() return 200 in the Next.js App Router?

I found this during a Search Console pass. Unknown project slugs were not 404s. The Next.js docs describe the mechanism in the loading.js reference: streaming page responses begin with a 200 status code, and the initial response headers fix the status for the rest of the response. Next.js can still stream notFound() after that, so the browser shows your not-found UI, but the status line stays 200.

The docs also say when the stream starts. The response body begins streaming when a Suspense fallback renders, and they give loading.tsx as the example, or when a Server Component suspends under a <Suspense> boundary. Calling notFound() before that point preserves the 404. After it, the status cannot change.

A loading.tsx file is a Suspense fallback wrapped around the segment's page. For a dynamic route, that means the fallback can be sent before your page code has a chance to look up the slug and decide it does not exist.

What does a soft 404 look like for Google?

A soft 404 is a page that tells the user "not found" but tells the crawler 200 OK. Next.js softens the damage: the docs say it injects <meta name="robots" content="noindex" /> into a streamed 404, so search engines are told to keep the URL out of the index even though the status is 200. They also note that some crawlers may label these responses as soft 404s.

On my site the result was messier. The root layout already set index, follow, and the streamed not-found added noindex. The page carried both tags with a 200 status. I did not want to find out which one a crawler would trust, and I did not want Search Console reporting junk URLs as soft 404s.

How do I reproduce it?

Here is the shape of the broken segment. The names are simplified from my project route:

// app/projects/[slug]/page.tsx
import { notFound } from 'next/navigation'

export default async function Page({
  params,
}: {
  params: Promise<{ slug: string }>
}) {
  const { slug } = await params
  const project = findProject(slug)
  if (!project) notFound() // too late: loading.tsx already streamed a 200

  return <ProjectView project={project} />
}

// app/projects/[slug]/loading.tsx
export default function Loading() {
  return <ProjectSkeleton />
}

Check the status with curl:

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/projects/nope

You want 404. In my case it printed 200. To see the duplicate robots tags, fetch the body:

curl -s https://example.com/projects/nope | grep -o '<meta name="robots"[^>]*>'

What fixed it: dynamicParams = false

Every project on my site is already prerendered through generateStaticParams. The fix was one line in the page file, with a comment I left for my future self:

// app/projects/[slug]/page.tsx

// Every project is prerendered below, so an unknown slug is a 404 at routing.
// Left on, it rendered on demand and hit notFound() after [slug]/loading.tsx
// had already streamed a 200: a soft 404 carrying both index and noindex.
export const dynamicParams = false

export async function generateStaticParams() {
  const slugs = await listProjectSlugs()
  return slugs.map((slug) => ({ slug }))
}

The reference for dynamicParams says that with the default of true, segments not returned by generateStaticParams are generated at request time, and with false they return a 404. An unknown slug never reaches your page component, so loading.tsx never streams for it, and the response is a real 404 from routing.

After the change, an unknown slug is a real 404 from routing. The curl check against /projects/nope on my live site now prints 404.

I left loading.tsx in place. At one point I suspected both loading files of bloating the DOM and removed them to test it. The DOM went from 951 to 949 nodes, so they went back. The soft 404 was their only real cost, and dynamicParams fixed that without touching them.

When is dynamicParams = false the wrong fix?

It fits when you know every valid slug at build time. If your slugs come from a database that changes after deploy, false would 404 a page that was added after the build, until you rebuild or revalidate.

The docs also list a restriction: dynamicParams is not available when Cache Components is enabled. I did not test that configuration, so check your own setup if you use it.

For a route that has to render on demand, the docs give a different answer. Run the check before the body streams. For a 404, you can check the slug in Proxy, then rewrite to a not-found route or return a 404 response, and the docs say to keep Proxy checks fast and avoid fetching full content there. I use a proxy for a related case on this site: malformed UTF-8 in a URL, such as /projects/%C0%AF, used to be a 500 on decode. proxy.ts, which is the Next 16 name for middleware, now returns a 404 for those. Its matcher only runs on dynamic segments that contain a %, so no other request pays for it.

The other option is to call notFound() before any await on a slow path and before any <Suspense> boundary. The streaming guide recommends exactly that placement to get a real status code.

How do I check every dynamic route for this?

The rule I wrote down after this: any dynamic route with a loading.tsx must 404 without rendering. That means dynamicParams = false, or a proxy check.

To audit a site:

1. List the routes with a dynamic segment and a loading.tsx next to them.
2. Request a nonsense slug on each one from a production build and read the status code with curl.
3. Anything that prints 200 with a not-found page is a soft 404.
4. Fix it with dynamicParams = false if the slug set is known at build time, or with a proxy check if it is not.

The site this came from is documented in the portfolio case study, and the project pages that carry the fix are listed on /projects.

What does "Discovered – currently not indexed" mean?

Google found the URL, usually through a sitemap or a link, and put it in a queue. It has not crawled it. That differs from "Crawled – currently not indexed", where Google fetched the page and decided not to keep it.

So the status says nothing about your page's content yet, because Google never looked at it. It says Google chose not to spend a request on it. Two things push that decision: how worth crawling the site as a whole looks, and how many URLs showed up at once.

Why were 50 of my pages stuck there?

The 2026-08-09 Search Console export showed the not-indexed count jumping from 6 to 56 on 2026-07-25. Indexed pages had been frozen at 7 since 2026-07-01. The deploy on 2026-07-25 had cross-linked every project page, so all 53 appeared in front of Google at the same time.

About half of those pages were synced from the READMEs of old tutorial repositories. With code blocks stripped, they had between 22 and 190 words of prose, and several were byte-identical to each other. Google read the batch as thin, near-duplicate filler. It spent no crawl budget on any of it, including the projects that had real content.

A site's good pages get judged alongside its bad ones. If you add dozens of weak URLs at once, the strong ones queue behind them.

How do I fix pages stuck in Discovered – currently not indexed?

I fixed it in two passes.

1. Pages under a word floor, MIN_INDEXABLE_WORDS, set to 200, render noindex, follow and drop out of the sitemap and every other discovery surface. That cut the set I was asking Google to index from 53 to 32.
2. I wrote a hand-authored overview for the stub pages. It survives my README sync script, renders above the synced README, and counts toward the threshold. Twenty sidecars got one. The final state was 53 indexable pages and 0 noindex.

My first pass was only the first step. I noindexed 21 pages and reported the problem as solved. That relieved the crawl-budget symptom, but it also gave up on a third of the portfolio. Writing real content was the option that raised the indexed count (52 versus 32), and I should have offered it together with the threshold.

What does the "fewer, better pages" rule look like in code?

One predicate answers the question, and every surface imports it. The sitemap, the page's robots meta, llms.txt and the IndexNow submission all ask it:

export const MIN_INDEXABLE_WORDS = 200

export function isIndexableSidecar(sidecar: {
  overview?: string
  body: string
  seoIndex?: boolean
}): boolean {
  // An explicit override wins in either direction.
  if (typeof sidecar.seoIndex === 'boolean') return sidecar.seoIndex
  const words = countProseWords(`${sidecar.overview ?? ''}\n\n${sidecar.body}`)
  return words >= MIN_INDEXABLE_WORDS
}

listIndexableProjects() filters the project list through that function. Anything that promotes pages to Google reads that list, so no caller can disagree with the page's own robots tag.

The single predicate earned its place the hard way. For a while the word-count arithmetic was inlined in two files, and IndexNow counted the README body but not the new overview. It silently dropped three pages the sitemap listed. A build step that compares sitemap membership against each page's rendered <meta robots>, llms.txt and the IndexNow set caught it on its first run. Without that check, it would have turned into a Search Console mystery weeks later.

You can see the resulting pages on /projects, and the site itself is described in the portfolio case study.

Should I click "Request indexing" on every URL in the report?

No, and this is where I wasted the most time. Request Indexing has a small daily quota, about 10 a day, and the Pages report that lists your stuck URLs lags the live index.

On 2026-11-03 the Pages report showed 49 indexed and 24 not: 12 "Discovered", 6 "Crawled, not indexed", 4 redirects and 2 alternate canonicals. Live URL Inspection told a different story:

• Of the 12 "Discovered" URLs, 11 were already on Google or had been requested the night before. Only /projects/portfolio had never been crawled.
• Of the 6 "Crawled, not indexed" URLs, 5 were OG images and the favicon, which I noindex on purpose. The sixth, /projects/rafiq, was on Google.
• The redirects were the http and apex variants of the home page, which is expected.

I had spent a request on /projects/angular-todo straight from the report's list, and it was already on Google. The Pages report can be days to weeks behind.

The same thing happened on a second site of mine. Six translated pages were listed as "Discovered, not indexed". Live inspection showed five of the six were already on Google, so one request was enough.

My rule now: inspect each URL live first, and request indexing only for the ones Google reports as unknown.

Does the Request Indexing quota reset at midnight?

I assumed it did and told myself the queue would be free the next day. One request went through at the start of the day, and then "Quota exceeded" came back. The quota frees as each request from the previous day ages out, so it works as a rolling window. Spread the queue across the day.

The quota is also per Google account, shared by every property you own. My other site hit "Quota exceeded" after 2 requests because the portfolio had used the rest.

On 2026-10-17 I requested indexing for 11 pages before the 12th returned "Quota exceeded".

What is the difference between "Something went wrong" and "Quota exceeded"?

"Quota exceeded" is the only quota message. On the seventh request of one session I got "Oops! Something went wrong ... Please try again later." and read it as the quota again. A single retry after dismissing it went through. Retry once before assuming you are out of quota.

What does "No referring sitemaps detected" mean?

For a page Google has never seen, the inspection can say "No referring sitemaps detected". I took that for a missing sitemap entry. The live sitemap had the URL, and Google had not re-read the sitemap since I added the page. Check your live sitemap before changing code.

What changed in the numbers?

Search Console between 2026-10-09 and 2026-10-17, after the fixes above:

By 2026-11-03 the Pages report showed 49 indexed. The redirects, a ?ref canonical and the noindexed images in the remaining rows were working as designed.

What should I check first on my own site?

1. Open each URL in URL Inspection and read the live status. The Pages report is a lagging summary.
2. Count the words of real prose on the pages in the bucket. If a large share are stubs or copies of each other, that is your answer.
3. Decide per page: write real content for it, or mark it noindex, follow and take it out of your sitemap and internal link lists.
4. Keep one function that decides what is indexable, and have the sitemap, the robots tag and any submission scripts all call it.
5. Request indexing only for URLs that inspection reports as unknown to Google, and spread the requests across days.

I also checked the site before blaming Google. A crawl of all 59 sitemap URLs on production with a Googlebot user agent returned a 200 on every page, each with a self-canonical, index, follow and one <h1>, and no duplicate titles or descriptions. When the site is clean and the bucket is still full, the cause is usually page quality or volume.