cmkiss/.claude/nginx-sitemap-crawler-plan.md

# Nginx Sitemap Crawler — Design Plan

_Last updated: 2026-09-10_

## Tech stack

- **Language:** pure JavaScript, running **client-side inside a
  webpage** (no Node, no server-side execution anywhere in this
  project — nginx serves static listings only).

## Goal

Maintain a browsable, near-real-time sitemap of a two-level nginx-served
directory tree, using only stock nginx config (no server-side scripts on
the nginx host), minimizing redundant requests over time, and surfacing
the most recently changed content first.

## Constraints

- No server-side execution on the nginx host — plain `nginx.conf` only.
- Client-side script/app is where all logic (crawling, sorting,
  pagination, caching) lives.
- Tree is two levels deep (root → level-1 dirs → level-2 dirs).

---

## 1. Nginx setup

Enable machine-readable directory listings on the relevant locations:

```nginx
location /files/ {
    autoindex on;
    autoindex_format json;
    autoindex_localtime off;
}
```

- `autoindex_format json;` — makes listings parseable instead of scraping HTML.
- `autoindex_localtime off;` — keeps timestamps in GMT, consistent and
  unambiguous for comparison logic (avoids DST/timezone drift issues).

Each listing entry includes `name`, `type` (`file`/`directory`), `size`,
and `mtime`.

---

## 2. Wave-based crawling

Directory names at level 2 aren't known until level 1 is fetched, so the
crawl naturally happens in sequential **waves**, each internally
**concurrent**:

- **Wave 1 (root)** — always fetched live on every run. Single request,
  cheapest possible cost, and the sole source of truth for what
  currently exists and what's changed.
- **Wave 2 (level-1 → level-2)** — fetched only as needed (see §5
  caching) and only for directories relevant to the current page (see
  §4 pagination). Requests within a wave are fired concurrently, not
  sequentially, to keep latency low regardless of directory count.

---

### 2a. Concurrency limits — **decision: configurable, not hardcoded**

Rather than a fixed worker count baked into the logic, concurrency is
exposed as a **single tunable setting** (e.g. `maxConcurrentRequests`)
that governs how many wave-2 requests are in flight at once within a
wave. Rationale and shape:

- **Why configurable:** the right concurrency value depends on factors
  that vary by deployment and can't be known in advance — network
  distance to the nginx host (same-region vs. cross-region), how many
  directories are typically in play per page (tied to §4a page size),
  how many browser tabs/users might be hitting the same nginx instance
  at once, and how conservative the operator wants to be about log
  noise or perceived "scanning" behavior (as noted in earlier
  discussion of request bursts).
- **Where it lives:** a single config value/constant near the top of
  the client script (or exposed as a UI setting/advanced option),
  rather than scattered magic numbers through the crawl logic.
- **Sensible default:** start conservative (e.g. 10) — safe for both
  same-region and cross-region cases, and already shown earlier in
  this plan to keep even a full ~500-directory cold-start crawl under
  a few seconds. Users/operators who know their network is fast and
  local, or who page through smaller slices (page size 10, see §4a),
  can raise it; those on slower/cross-region links or shared/production
  nginx instances can lower it.
- **Interaction with pagination:** since wave-2 is now scoped to just
  the current page's directories (§4a), the concurrency setting mostly
  matters relative to page size — e.g. at page size 10 with
  concurrency 10, an entire page's wave-2 fetches resolve in a single
  batch (~1 RTT); at page size 30 with concurrency 10, the same page
  takes ~3 batches. Setting concurrency ≥ page size effectively
  guarantees single-batch page loads, which is a reasonable rule of
  thumb for operators tuning this value.
- **No hard ceiling imposed by nginx** — as established earlier,
  nginx's default `worker_connections` comfortably absorbs dozens to
  low hundreds of concurrent client connections, so the practical
  upper bound is really about being a good citizen (log noise, shared
  hosting) rather than anything nginx itself will choke on.

---

## 3. Recency-based prioritization

- A directory's `mtime` reflects changes to its **direct** children only
  (additions, removals, renames) — not deeper nested edits. This is
  sufficient for this use case, since the concern is new content
  appearing, not deep nested edits (see "Known limitation" below).
- After wave 1 resolves, sort level-1 entries by `mtime` descending,
  with directory name ascending as a stable tie-breaker (see R5).
  This ordering drives both crawl priority and display order.

---

## 4. Pagination

- Slice the sorted level-1 list into fixed-size pages (default 10,
  user-selectable — see §4a).
- Only fire wave-2 requests for the directories in the **currently
  viewed page** — not the whole tree.
- Page navigation reuses the already-fetched, already-sorted wave-1
  list in memory; no repeated root requests needed when paging.
- Optional: prefetch the next page's wave-2 requests in the background
  while the current page is being viewed, so navigation feels instant.

---

### 4a. Page size — **decision: user-selectable, default 10**

- Default page size: **10** items.
- User-selectable alternatives: **15, 20, 30**.
- Changing page size resets pagination to page 1 (avoids ambiguous
  partial pages / confusing item overlap when the slice boundaries
  shift).
- Since wave 1 is already fully fetched and sorted in memory (see §2,
  §3), changing page size is a **pure client-side re-slice** of the
  existing sorted list — no new wave-1 request needed.
- Wave-2 fetch behavior (§5, cache-aware) is unaffected by page size
  itself — only which/how many level-1 directories fall into the
  current slice changes, which in turn determines which cache
  entries are read or fetched.

---

## 5. Sitemap cache

Persist a cache entry per directory:

```json
{
  "path": "/files/project-a/",
  "mtime": "Wed, 10 Sep 2026 14:32:01 GMT",
  "children": [ "...level-2 entries..." ],
  "children_fetched_at": "2026-09-10T15:00:00Z"
}
```

### 5a. Storage mechanism — **decision: IndexedDB**

This runs purely client-side in a webpage — no filesystem access, so
the real choice is between browser storage APIs: **localStorage** vs.
**IndexedDB**. IndexedDB was chosen:

- **Storage ceiling** — localStorage is capped around 5-10MB
  (browser-dependent) and shared across the whole origin; IndexedDB's
  ceiling is far higher (typically hundreds of MB+), removing a
  constraint that could otherwise bite as the tree grows.
- **Per-record updates** — IndexedDB is an object store keyed by
  record (naturally: path → cache entry), so updating one directory's
  entry doesn't require re-serializing/rewriting the entire cache, the
  way localStorage (full blob, `JSON.stringify` every write) would.
- **Non-blocking** — IndexedDB is async by nature, which fits a cache
  that's written to progressively as crawl waves resolve, without
  freezing the UI thread. localStorage is synchronous and would block
  rendering on every read/write as the cache grows.
- **No manual serialization** — IndexedDB stores structured JS objects
  directly; no need to `JSON.stringify`/`parse` on every access.
- **Still zero external dependencies** — native browser API, no
  library required (though a thin promise-wrapper like `idb` is a
  reasonable, optional ergonomics improvement later if the raw API
  proves too verbose).

**Practical shape:**
- One object store, e.g. `directoryCache`, keyed by directory `path`.
- Each record holds `{ path, mtime, children, children_fetched_at }`
  as described below.
- Reads/writes are async (`Promise`-based or wrapped in one), which
  aligns naturally with the already-async wave-2 fetches.

On each run, compare each level-1 directory's **live** `mtime` (from
wave 1) against the **cached** `mtime`:

- **Match** → skip wave 2 for that directory; serve `children` from
  cache.
- **Newer or missing** → fetch wave 2 live, then update the cache entry
  with the new `mtime` and `children`.

Additional cache maintenance:

- **Pruning** — after wave 1, any cached level-1 entries no longer
  present in the live listing should be removed (handles deletions).
- **Wave 1 is never cached** — it has no parent to diff against, and
  it's already the cheapest possible request (1 per run, regardless of
  tree size).
- **Cold start** — first run has no cache, so it pays the full crawl
  cost once. Every subsequent run should be dramatically cheaper,
  bounded roughly by how much actually changed since the last run.

---

## 6. Why this combination works well together

- Sorting by `mtime` descending means the directories most likely to
  have a **stale cache** (recent changes) are exactly the ones
  prioritized to the front — so page 1 concentrates the real crawl
  cost, while later pages are almost pure cache hits.
- Over repeated runs, total request cost trends toward being
  proportional to **how much changed**, not the size of the tree.
- Progressive/skeleton rendering (page renders immediately with cached
  or partial data, updates in place as fresher wave-2 results arrive)
  keeps the experience responsive even while cache misses are being
  resolved in the background.

---

## Known accepted limitation

Directory `mtime` only reflects direct-child changes, not deeper nested
edits (e.g. a file modified inside an existing level-2 directory won't
bump the level-1 directory's `mtime`). Accepted as a non-issue here,
since the priority signal needed is "new subdirectories appearing,"
which this correctly captures.

---

## Longevity Considerations

_This project is intended as "deploy and forget" for a ~30 year
horizon. That's a materially different constraint than "get this
working now," and shapes several decisions beyond what pure
functionality would require._

### L1. IndexedDB eviction — not a concern
Some browsers (e.g. Safari ITP) may evict IndexedDB data for
infrequently-visited origins. **Explicitly accepted as a non-issue**:
R6's uncached fallback mode already handles any IndexedDB
unavailability gracefully (falls back to live fetching), so eviction
simply means the cache benefit doesn't materialize on that visit — not
a failure. No action needed beyond what's already in R6.

### L2. HTML fallback if JSON listing breaks — **decision: included**
`autoindex_format json` depends on nginx config staying intact over
the project's lifetime. Configs can drift — a server migration,
reconfiguration, or someone disabling `autoindex_format json` (while
leaving `autoindex on;` itself in place) is realistic over 30 years.

To survive this, the crawler includes an **HTML fallback parser**:

- On each request, if the response isn't valid JSON (parse fails, or
  `Content-Type` isn't JSON), fall back to treating the response as
  nginx's **default HTML autoindex output** instead of failing the
  request outright.
- Parse the HTML using the browser's native `DOMParser` (built-in, no
  dependency) — extract entries from the `<a href="...">` listing,
  using the link `href` for name/type (trailing `/` = directory) and
  the surrounding text for `mtime`/size where present.
- This is strictly a **fallback path**, not the primary path — JSON
  remains preferred when available, since it's more reliable to parse
  and already carries structured `mtime`. The HTML fallback exists
  purely so the tool degrades to "still works, just via a different
  parse path" rather than "breaks entirely" if `autoindex_format json`
  is ever lost from the config.
- This directly follows the same silent-skip philosophy as R3: prefer
  degraded function over hard failure, wherever degrading is possible
  without violating the zero-dependency principle.

### L3. Total-failure visibility
Silent-skip (R3) is correct for individual transient request failures,
but should not extend to masking a *total* breakage (e.g. CORS
misconfigured after a migration, autoindex disabled entirely, wrong
origin). The crawler tracks a simple "N of M directories loaded"
count; if that ratio is very low (e.g. near-zero) after a crawl
completes, a minimal, dependency-free visible indicator (a status line
in the UI, no alerting infrastructure) should communicate that
something is broken, so the project doesn't fail silently for years
before a human notices.

### L4. Zero dependencies, deliberately
No npm packages, no build step, no framework — stated explicitly as a
guiding principle, not just an incidental outcome of earlier
decisions. Any dependency introduces a 30-year liability: registries
can vanish, packages can go unmaintained, and even a "small" library
can break under a future browser change with nobody left to patch it.
Future contributors should treat this as a hard constraint, not a
preference to be relaxed for convenience.

### L5. Prefer long-stable JS/browser APIs over new syntax
Favor well-established, broadly-supported features (`fetch`,
`Promise`, `async`/`await`, basic IndexedDB, `DOMParser`) over newer or
experimental JS/browser APIs. The newer something is today, the more
likely it is to be deprecated, changed, or need a polyfill by the time
this project is 15-20 years old. Nothing currently behind a flag or
"experimental" label should be used, even if convenient now.

### L6. Keep this plan attached to the code
For a deploy-and-forget project, the largest real risk isn't
technical — it's a future maintainer with zero context. This document
should ship alongside the implementation (same repository, or
inlined as a comment block) rather than live separately, so the
reasoning behind each decision (why JSON over HTML, why IndexedDB,
why silent-skip, etc.) survives together with the code itself.

### L7. nginx version/config drift
`autoindex_format json` requires a reasonably modern nginx (long
available, but not guaranteed present on every install). Document a
minimum nginx version alongside the deployment instructions, and treat
"response isn't valid JSON" as an expected, handled case (see L2)
rather than an exceptional one — since over 30 years, drift in the
server config is a "when," not an "if."

---

## Risks & Edge Cases

### R1. CORS
If the webpage is served from a different origin than the nginx
directory listing, browser `fetch()` calls will be blocked without
proper CORS headers. Requires an `add_header Access-Control-Allow-Origin
...;` (or equivalent) on the relevant nginx `location` block, unless
the page and the file server share an origin. Needs to be decided/set
up alongside the nginx config in §1.

### R2. Trailing slashes on directory requests
nginx's autoindex only serves a listing cleanly when the request URL
ends in `/`. Requesting a directory path without the trailing slash
typically triggers a redirect (e.g. 301) instead of returning the
listing directly, adding an extra round-trip. The crawler must
consistently append `/` when building wave-2 request URLs from the
`name` field returned in wave-1's JSON.

### R3. Error handling for failed requests — **decision: silent skip**
If a wave-2 request fails (timeout, 404, host briefly unreachable),
it is silently skipped: no retry, no visible error marker. The
directory in question falls back to whatever is already in the
IndexedDB cache (stale-but-present data), or is simply omitted from
that page's render if nothing is cached yet. Accepted trade-off for
simplicity — a failed fetch does not block or degrade the rest of the
page.

### R4. Race conditions from rapid navigation
Changing page size or flipping pages quickly can cause in-flight
requests from a previous page/size to resolve *after* newer ones,
overwriting current state with stale data. Mitigated with a
generation/token counter: each navigation action increments a
"current generation" value, and any response that arrives tagged with
an older generation than the current one is discarded rather than
rendered.

### R5. `mtime` tie-breaking
autoindex JSON timestamps are second-precision, so directories touched
within the same second (e.g. a batch of subdirectories created
together by a script) can tie on `mtime`. The sort in §3 uses `mtime`
descending as the primary key with **directory name ascending** as a
stable secondary key, so ordering — and therefore pagination — stays
consistent across runs instead of jittering on ties.

### R6. IndexedDB availability — **decision: uncached fallback mode**
IndexedDB may be unavailable or restricted (private/incognito modes in
some browsers) or subject to storage eviction under pressure. The
crawler treats any IndexedDB access failure (open error, blocked
transaction, etc.) as a signal to fall back to an **uncached mode**:
every wave-2 request is fetched live for the duration of that session,
with no persistence attempted. This keeps the sitemap fully functional
— just without the performance benefit of cache hits on repeat visits
— rather than erroring out or blocking the page. Detection happens
once at startup (attempt to open the DB; on failure, set an in-memory
`cacheAvailable = false` flag that the rest of the logic checks before
any cache read/write).

---

## Open items / future considerations

_All resolved — no open items remain._

Ruled out: WebDAV / `PROPFIND` (`ngx_http_dav_ext_module`) as a
single-request full-tree alternative — requires a non-stock nginx
build/module, which conflicts with the "ordinary nginx only" and
simplicity constraints. Wave-based crawling with caching (§2, §5)
remains the approach.