Pause search hydration once a page's worth of matches is found (AI)

loadSearch now scans stubs in small batches and stops as soon as it
has enough matches for the requested page (plus one, to know whether
a next page exists), instead of hydrating every article up front.
Scan progress is kept per-query so paging forward resumes without
re-checking, and paging backward needs no scanning at all.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXSVPS3UmmASiiprPfcnUd

README.md

@@ -373,10 +373,10 @@ any further request per article and without ever fetching an article's actual Ma
 **Hydration** is the separate, deferred step that fetches a specific article's content (title,
 description, cover image, reading time) — it only ever runs for articles about to be rendered:
 the current page's cards (`hydrateArticles()` in `cmjs/js/discovery.js`), a single visited
-article, or (since full-text search needs real content) every result of a search. A large site's
-list is therefore fast to show regardless of how many articles it has — indexing costs one
-request per author, full stop, and only ~20 articles' worth of content is ever fetched per page
-view.
+article, or (since full-text search needs real content) as many search results as the page being
+viewed actually needs. A large site's list is therefore fast to show regardless of how many
+articles it has — indexing costs one request per author, full stop, and only ~20 articles' worth
+of content is ever fetched per page view.
 
 - **Cache** — hydrated articles are cached per post directory in IndexedDB (`cmjs/js/cache.js`),
   keyed by directory path and invalidated by comparing the live `mtime` against the cached one —
@@ -399,12 +399,16 @@ view.
 - **Prefetching** — once the current page has fully rendered, the next page's articles are
   hydrated in the background (fire-and-forget), so clicking "next" is usually an instant cache
   hit rather than a fresh round-trip.
-- **Progressive search** — search hydrates every indexed article (see above — it needs real
-  text, unlike the list view), but does so incrementally rather than in one blocking batch:
-  matches appear and the result list grows as hydration resolves, with a "still indexing…" note
-  until every article has been checked. Results paginate at the same `PAGE_SIZE` as the article
-  list, and the page count can shift while hydration is still filling in matches, settling once
-  it's done.
+- **Paused/resumable search** — search needs real text (title, description), unlike the list
+  view, so it hydrates articles rather than just reading stubs — but only as many as the page
+  being viewed needs, in the same newest-first order as everything else (`loadSearch()` in
+  `cmjs/js/views.js`). It scans stubs in small batches, stops as soon as it's found enough matches
+  to fill the current page (plus one, just to know whether a next page exists), and resumes
+  exactly where it left off — no re-checking — if the user pages further into the *same* query.
+  Paging back needs no scanning at all, since matches only ever accumulate. This keeps a search on
+  a large site to a handful of requests instead of hydrating the entire article set up front. The
+  page count is exact once the whole set's been scanned, and a lower bound (at least one more
+  page) until then.
 - **Crawl health** — if every author's directory listing fails outright, or very few of the
   known post directories parsed successfully (a malformed directory name, or a total breakage
   like the wrong origin or CORS misconfigured), a small warning banner is shown above the article

cmjs/js/discovery.js

@@ -266,25 +266,13 @@ export async function hydrateArticle(stub) {
     return stub;
 }
 
-// Concurrency-limited batch hydration. For a page's worth of stubs this
-// just resolves once; `onProgress`, if given, is called after each item
-// resolves with (resolvedCount, total) — lets a caller hydrating a much
-// bigger batch (search hydrates every result, not just one page) show
-// results incrementally instead of one long wait for all of them.
-//
-// Processing order follows `stubs`' own array order (each worker below
-// just claims the next unclaimed index) — since discoverArticles() already
-// returns stubs sorted newest-first by the authoritative `date` field, that
-// means hydration (and thus which search results fill in first) is already
-// newest-first too, for free.
-export function hydrateArticles(stubs, onProgress) {
-    let resolved = 0;
-    return mapWithConcurrency(stubs, MAX_CONCURRENT_REQUESTS, async stub => {
-        const article = await hydrateArticle(stub);
-        resolved += 1;
-        onProgress?.(resolved, stubs.length);
-        return article;
-    });
+// Concurrency-limited batch hydration. Processing order follows `stubs`'
+// own array order (each worker below just claims the next unclaimed index)
+// — since discoverArticles() already returns stubs sorted newest-first by
+// the authoritative `date` field, that means hydration (and thus which
+// search results fill in first) is already newest-first too, for free.
+export function hydrateArticles(stubs) {
+    return mapWithConcurrency(stubs, MAX_CONCURRENT_REQUESTS, hydrateArticle);
 }
 
 async function runCrawl() {

cmjs/js/views.js

@@ -350,6 +350,20 @@ export async function loadAuthorView(author, page = 1) {
 export const loadCategoryView = (category, page = 1) => loadArticleList({ type: 'category', value: category }, page);
 export const loadTagView = (tag, page = 1) => loadArticleList({ type: 'tag', value: tag }, page);
 
+// How many stubs to hydrate-and-check per scanning round (see loadSearch
+// below) — small enough that a search which only needs page 1 doesn't fire
+// a big burst of simultaneous requests, large enough to keep the number of
+// rounds (and their per-round render) reasonable on a big site.
+const SEARCH_SCAN_BATCH = PAGE_SIZE;
+
+// Scan progress for the current search query — lets paging forward resume
+// exactly where the last call left off instead of re-checking articles
+// already ruled in/out, and lets paging *backward* (or re-rendering the
+// same page) skip scanning entirely, since `matches` only ever grows and
+// stub order never changes. Reset whenever the query text changes; holds
+// only the most recent query, so a new search simply replaces it.
+let searchState = null;
+
 export async function loadSearch(query, page = 1) {
     showListView();
     window.scrollTo(0, 0);
@@ -372,28 +386,37 @@ export async function loadSearch(query, page = 1) {
         if (generation !== currentGeneration) return;
         renderFilterNavigation(stubs);
 
-        // Unlike the list view, search needs every article's real title and
-        // description — text that only exists in its markdown content — so
-        // it hydrates the whole set rather than just a page's worth. Most
-        // of that is typically already cached from earlier browsing/search.
-        // Hydrating is progressive (like indexing) so results appear as
-        // they're found instead of one long wait for the whole site's
-        // content — hydrateArticle() mutates stubs in place, so checking
-        // each stub's own `hydrated` flag is enough to know what's ready,
-        // without tracking a separately-growing list. Pagination applies to
-        // whatever's matched so far, the same page math as the list view —
-        // the page count can shift while hydration is still in progress,
-        // settling once it's done.
+        if (!searchState || searchState.query !== query || searchState.stubs !== stubs) {
+            searchState = { query, stubs, cursor: 0, matches: [] };
+        }
+        const state = searchState;
+
+        // Only scan far enough to fill the requested page, plus one extra
+        // match so pagination knows whether a next page exists — never the
+        // whole (possibly huge) stub set. Jumping straight to a deep page
+        // (typed/bookmarked link) just scans further up front, but still
+        // only as far as *that* page needs.
+        const target = page * PAGE_SIZE + 1;
+
         const renderResults = () => {
             if (generation !== currentGeneration) return;
-            const hydratedSoFar = stubs.filter(stub => stub.hydrated);
-            const done = hydratedSoFar.length === stubs.length;
-            const matches = hydratedSoFar.filter(matchesQuery);
-            const pages = Math.max(1, Math.ceil(matches.length / PAGE_SIZE));
+            const scanDone = state.cursor >= state.stubs.length;
+            const haveEnough = state.matches.length >= target;
+            // Once either is true, nothing left running will change what's
+            // shown for this page — that's what `done` gates downstream
+            // (crawl-health check, hiding the "still working" footer).
+            const done = scanDone || haveEnough;
+            // Exact once the whole set's been scanned; otherwise it's only a
+            // lower bound (what's already been found, or at least one more
+            // page than requested, whichever is bigger) — same "can still
+            // shift until done" contract loadArticleList already uses for
+            // its own page count.
+            const knownPages = Math.max(1, Math.ceil(state.matches.length / PAGE_SIZE));
+            const pages = scanDone ? knownPages : Math.max(knownPages, page + 1);
             const clampedPage = Math.max(1, Math.min(page, pages));
-            const pageMatches = matches.slice((clampedPage - 1) * PAGE_SIZE, clampedPage * PAGE_SIZE);
+            const pageMatches = state.matches.slice((clampedPage - 1) * PAGE_SIZE, clampedPage * PAGE_SIZE);
             renderArticleListBody(view, {
-                total: matches.length,
+                total: state.matches.length,
                 pageArticles: pageMatches,
                 page: clampedPage,
                 pages,
@@ -404,11 +427,24 @@ export async function loadSearch(query, page = 1) {
                 loadingKey: 'searching',
             });
         };
+        renderResults();
 
-        await hydrateArticles(stubs, resolvedCount => {
-            if (resolvedCount === 1 || resolvedCount % 5 === 0 || resolvedCount === stubs.length) renderResults();
-        });
-        renderResults(); // guaranteed-final render — also covers the empty-site case, where the loop above never calls back at all
+        // Scan forward in small batches, stopping as soon as `target` is
+        // met — resuming this same loop (via a later loadSearch call
+        // reusing `state`) picks up right after the last-scanned stub.
+        // The generation check guards against a superseded call's loop
+        // (user navigated away, or to a further page) mutating shared
+        // `state` after the fact — only the most recently started call is
+        // allowed to keep scanning or rendering.
+        while (state.cursor < state.stubs.length && state.matches.length < target) {
+            if (generation !== currentGeneration) return;
+            const batch = state.stubs.slice(state.cursor, state.cursor + SEARCH_SCAN_BATCH);
+            await hydrateArticles(batch);
+            if (generation !== currentGeneration) return;
+            state.cursor += batch.length;
+            batch.forEach(article => { if (matchesQuery(article)) state.matches.push(article); });
+            renderResults();
+        }
     } catch (error) {
         if (generation === currentGeneration) view.innerHTML = `<p class="error">${t('searchError', { message: error.message })}</p>`;
     }