# Nginx Sitemap Crawler — Design Plan _Last updated: 2026-09-10_ ## Tech stack - **Language:** pure JavaScript, running **client-side inside a webpage** (no Node, no server-side execution anywhere in this project — nginx serves static listings only). ## Goal Maintain a browsable, near-real-time sitemap of a two-level nginx-served directory tree, using only stock nginx config (no server-side scripts on the nginx host), minimizing redundant requests over time, and surfacing the most recently changed content first. ## Constraints - No server-side execution on the nginx host — plain `nginx.conf` only. - Client-side script/app is where all logic (crawling, sorting, pagination, caching) lives. - Tree is two levels deep (root → level-1 dirs → level-2 dirs). --- ## 1. Nginx setup Enable machine-readable directory listings on the relevant locations: ```nginx location /files/ { autoindex on; autoindex_format json; autoindex_localtime off; } ``` - `autoindex_format json;` — makes listings parseable instead of scraping HTML. - `autoindex_localtime off;` — keeps timestamps in GMT, consistent and unambiguous for comparison logic (avoids DST/timezone drift issues). Each listing entry includes `name`, `type` (`file`/`directory`), `size`, and `mtime`. --- ## 2. Wave-based crawling Directory names at level 2 aren't known until level 1 is fetched, so the crawl naturally happens in sequential **waves**, each internally **concurrent**: - **Wave 1 (root)** — always fetched live on every run. Single request, cheapest possible cost, and the sole source of truth for what currently exists and what's changed. - **Wave 2 (level-1 → level-2)** — fetched only as needed (see §5 caching) and only for directories relevant to the current page (see §4 pagination). Requests within a wave are fired concurrently, not sequentially, to keep latency low regardless of directory count. --- ### 2a. Concurrency limits — **decision: configurable, not hardcoded** Rather than a fixed worker count baked into the logic, concurrency is exposed as a **single tunable setting** (e.g. `maxConcurrentRequests`) that governs how many wave-2 requests are in flight at once within a wave. Rationale and shape: - **Why configurable:** the right concurrency value depends on factors that vary by deployment and can't be known in advance — network distance to the nginx host (same-region vs. cross-region), how many directories are typically in play per page (tied to §4a page size), how many browser tabs/users might be hitting the same nginx instance at once, and how conservative the operator wants to be about log noise or perceived "scanning" behavior (as noted in earlier discussion of request bursts). - **Where it lives:** a single config value/constant near the top of the client script (or exposed as a UI setting/advanced option), rather than scattered magic numbers through the crawl logic. - **Sensible default:** start conservative (e.g. 10) — safe for both same-region and cross-region cases, and already shown earlier in this plan to keep even a full ~500-directory cold-start crawl under a few seconds. Users/operators who know their network is fast and local, or who page through smaller slices (page size 10, see §4a), can raise it; those on slower/cross-region links or shared/production nginx instances can lower it. - **Interaction with pagination:** since wave-2 is now scoped to just the current page's directories (§4a), the concurrency setting mostly matters relative to page size — e.g. at page size 10 with concurrency 10, an entire page's wave-2 fetches resolve in a single batch (~1 RTT); at page size 30 with concurrency 10, the same page takes ~3 batches. Setting concurrency ≥ page size effectively guarantees single-batch page loads, which is a reasonable rule of thumb for operators tuning this value. - **No hard ceiling imposed by nginx** — as established earlier, nginx's default `worker_connections` comfortably absorbs dozens to low hundreds of concurrent client connections, so the practical upper bound is really about being a good citizen (log noise, shared hosting) rather than anything nginx itself will choke on. --- ## 3. Recency-based prioritization - A directory's `mtime` reflects changes to its **direct** children only (additions, removals, renames) — not deeper nested edits. This is sufficient for this use case, since the concern is new content appearing, not deep nested edits (see "Known limitation" below). - After wave 1 resolves, sort level-1 entries by `mtime` descending, with directory name ascending as a stable tie-breaker (see R5). This ordering drives both crawl priority and display order. --- ## 4. Pagination - Slice the sorted level-1 list into fixed-size pages (default 10, user-selectable — see §4a). - Only fire wave-2 requests for the directories in the **currently viewed page** — not the whole tree. - Page navigation reuses the already-fetched, already-sorted wave-1 list in memory; no repeated root requests needed when paging. - Optional: prefetch the next page's wave-2 requests in the background while the current page is being viewed, so navigation feels instant. --- ### 4a. Page size — **decision: user-selectable, default 10** - Default page size: **10** items. - User-selectable alternatives: **15, 20, 30**. - Changing page size resets pagination to page 1 (avoids ambiguous partial pages / confusing item overlap when the slice boundaries shift). - Since wave 1 is already fully fetched and sorted in memory (see §2, §3), changing page size is a **pure client-side re-slice** of the existing sorted list — no new wave-1 request needed. - Wave-2 fetch behavior (§5, cache-aware) is unaffected by page size itself — only which/how many level-1 directories fall into the current slice changes, which in turn determines which cache entries are read or fetched. --- ## 5. Sitemap cache Persist a cache entry per directory: ```json { "path": "/files/project-a/", "mtime": "Wed, 10 Sep 2026 14:32:01 GMT", "children": [ "...level-2 entries..." ], "children_fetched_at": "2026-09-10T15:00:00Z" } ``` ### 5a. Storage mechanism — **decision: IndexedDB** This runs purely client-side in a webpage — no filesystem access, so the real choice is between browser storage APIs: **localStorage** vs. **IndexedDB**. IndexedDB was chosen: - **Storage ceiling** — localStorage is capped around 5-10MB (browser-dependent) and shared across the whole origin; IndexedDB's ceiling is far higher (typically hundreds of MB+), removing a constraint that could otherwise bite as the tree grows. - **Per-record updates** — IndexedDB is an object store keyed by record (naturally: path → cache entry), so updating one directory's entry doesn't require re-serializing/rewriting the entire cache, the way localStorage (full blob, `JSON.stringify` every write) would. - **Non-blocking** — IndexedDB is async by nature, which fits a cache that's written to progressively as crawl waves resolve, without freezing the UI thread. localStorage is synchronous and would block rendering on every read/write as the cache grows. - **No manual serialization** — IndexedDB stores structured JS objects directly; no need to `JSON.stringify`/`parse` on every access. - **Still zero external dependencies** — native browser API, no library required (though a thin promise-wrapper like `idb` is a reasonable, optional ergonomics improvement later if the raw API proves too verbose). **Practical shape:** - One object store, e.g. `directoryCache`, keyed by directory `path`. - Each record holds `{ path, mtime, children, children_fetched_at }` as described below. - Reads/writes are async (`Promise`-based or wrapped in one), which aligns naturally with the already-async wave-2 fetches. On each run, compare each level-1 directory's **live** `mtime` (from wave 1) against the **cached** `mtime`: - **Match** → skip wave 2 for that directory; serve `children` from cache. - **Newer or missing** → fetch wave 2 live, then update the cache entry with the new `mtime` and `children`. Additional cache maintenance: - **Pruning** — after wave 1, any cached level-1 entries no longer present in the live listing should be removed (handles deletions). - **Wave 1 is never cached** — it has no parent to diff against, and it's already the cheapest possible request (1 per run, regardless of tree size). - **Cold start** — first run has no cache, so it pays the full crawl cost once. Every subsequent run should be dramatically cheaper, bounded roughly by how much actually changed since the last run. --- ## 6. Why this combination works well together - Sorting by `mtime` descending means the directories most likely to have a **stale cache** (recent changes) are exactly the ones prioritized to the front — so page 1 concentrates the real crawl cost, while later pages are almost pure cache hits. - Over repeated runs, total request cost trends toward being proportional to **how much changed**, not the size of the tree. - Progressive/skeleton rendering (page renders immediately with cached or partial data, updates in place as fresher wave-2 results arrive) keeps the experience responsive even while cache misses are being resolved in the background. --- ## Known accepted limitation Directory `mtime` only reflects direct-child changes, not deeper nested edits (e.g. a file modified inside an existing level-2 directory won't bump the level-1 directory's `mtime`). Accepted as a non-issue here, since the priority signal needed is "new subdirectories appearing," which this correctly captures. --- ## Longevity Considerations _This project is intended as "deploy and forget" for a ~30 year horizon. That's a materially different constraint than "get this working now," and shapes several decisions beyond what pure functionality would require._ ### L1. IndexedDB eviction — not a concern Some browsers (e.g. Safari ITP) may evict IndexedDB data for infrequently-visited origins. **Explicitly accepted as a non-issue**: R6's uncached fallback mode already handles any IndexedDB unavailability gracefully (falls back to live fetching), so eviction simply means the cache benefit doesn't materialize on that visit — not a failure. No action needed beyond what's already in R6. ### L2. HTML fallback if JSON listing breaks — **decision: included** `autoindex_format json` depends on nginx config staying intact over the project's lifetime. Configs can drift — a server migration, reconfiguration, or someone disabling `autoindex_format json` (while leaving `autoindex on;` itself in place) is realistic over 30 years. To survive this, the crawler includes an **HTML fallback parser**: - On each request, if the response isn't valid JSON (parse fails, or `Content-Type` isn't JSON), fall back to treating the response as nginx's **default HTML autoindex output** instead of failing the request outright. - Parse the HTML using the browser's native `DOMParser` (built-in, no dependency) — extract entries from the `` listing, using the link `href` for name/type (trailing `/` = directory) and the surrounding text for `mtime`/size where present. - This is strictly a **fallback path**, not the primary path — JSON remains preferred when available, since it's more reliable to parse and already carries structured `mtime`. The HTML fallback exists purely so the tool degrades to "still works, just via a different parse path" rather than "breaks entirely" if `autoindex_format json` is ever lost from the config. - This directly follows the same silent-skip philosophy as R3: prefer degraded function over hard failure, wherever degrading is possible without violating the zero-dependency principle. ### L3. Total-failure visibility Silent-skip (R3) is correct for individual transient request failures, but should not extend to masking a *total* breakage (e.g. CORS misconfigured after a migration, autoindex disabled entirely, wrong origin). The crawler tracks a simple "N of M directories loaded" count; if that ratio is very low (e.g. near-zero) after a crawl completes, a minimal, dependency-free visible indicator (a status line in the UI, no alerting infrastructure) should communicate that something is broken, so the project doesn't fail silently for years before a human notices. ### L4. Zero dependencies, deliberately No npm packages, no build step, no framework — stated explicitly as a guiding principle, not just an incidental outcome of earlier decisions. Any dependency introduces a 30-year liability: registries can vanish, packages can go unmaintained, and even a "small" library can break under a future browser change with nobody left to patch it. Future contributors should treat this as a hard constraint, not a preference to be relaxed for convenience. ### L5. Prefer long-stable JS/browser APIs over new syntax Favor well-established, broadly-supported features (`fetch`, `Promise`, `async`/`await`, basic IndexedDB, `DOMParser`) over newer or experimental JS/browser APIs. The newer something is today, the more likely it is to be deprecated, changed, or need a polyfill by the time this project is 15-20 years old. Nothing currently behind a flag or "experimental" label should be used, even if convenient now. ### L6. Keep this plan attached to the code For a deploy-and-forget project, the largest real risk isn't technical — it's a future maintainer with zero context. This document should ship alongside the implementation (same repository, or inlined as a comment block) rather than live separately, so the reasoning behind each decision (why JSON over HTML, why IndexedDB, why silent-skip, etc.) survives together with the code itself. ### L7. nginx version/config drift `autoindex_format json` requires a reasonably modern nginx (long available, but not guaranteed present on every install). Document a minimum nginx version alongside the deployment instructions, and treat "response isn't valid JSON" as an expected, handled case (see L2) rather than an exceptional one — since over 30 years, drift in the server config is a "when," not an "if." --- ## Risks & Edge Cases ### R1. CORS If the webpage is served from a different origin than the nginx directory listing, browser `fetch()` calls will be blocked without proper CORS headers. Requires an `add_header Access-Control-Allow-Origin ...;` (or equivalent) on the relevant nginx `location` block, unless the page and the file server share an origin. Needs to be decided/set up alongside the nginx config in §1. ### R2. Trailing slashes on directory requests nginx's autoindex only serves a listing cleanly when the request URL ends in `/`. Requesting a directory path without the trailing slash typically triggers a redirect (e.g. 301) instead of returning the listing directly, adding an extra round-trip. The crawler must consistently append `/` when building wave-2 request URLs from the `name` field returned in wave-1's JSON. ### R3. Error handling for failed requests — **decision: silent skip** If a wave-2 request fails (timeout, 404, host briefly unreachable), it is silently skipped: no retry, no visible error marker. The directory in question falls back to whatever is already in the IndexedDB cache (stale-but-present data), or is simply omitted from that page's render if nothing is cached yet. Accepted trade-off for simplicity — a failed fetch does not block or degrade the rest of the page. ### R4. Race conditions from rapid navigation Changing page size or flipping pages quickly can cause in-flight requests from a previous page/size to resolve *after* newer ones, overwriting current state with stale data. Mitigated with a generation/token counter: each navigation action increments a "current generation" value, and any response that arrives tagged with an older generation than the current one is discarded rather than rendered. ### R5. `mtime` tie-breaking autoindex JSON timestamps are second-precision, so directories touched within the same second (e.g. a batch of subdirectories created together by a script) can tie on `mtime`. The sort in §3 uses `mtime` descending as the primary key with **directory name ascending** as a stable secondary key, so ordering — and therefore pagination — stays consistent across runs instead of jittering on ties. ### R6. IndexedDB availability — **decision: uncached fallback mode** IndexedDB may be unavailable or restricted (private/incognito modes in some browsers) or subject to storage eviction under pressure. The crawler treats any IndexedDB access failure (open error, blocked transaction, etc.) as a signal to fall back to an **uncached mode**: every wave-2 request is fetched live for the duration of that session, with no persistence attempted. This keeps the sitemap fully functional — just without the performance benefit of cache hits on repeat visits — rather than erroring out or blocking the page. Detection happens once at startup (attempt to open the DB; on failure, set an in-memory `cacheAvailable = false` flag that the rest of the logic checks before any cache read/write). --- ## Open items / future considerations _All resolved — no open items remain._ Ruled out: WebDAV / `PROPFIND` (`ngx_http_dav_ext_module`) as a single-request full-tree alternative — requires a non-stock nginx build/module, which conflicts with the "ordinary nginx only" and simplicity constraints. Wave-based crawling with caching (§2, §5) remains the approach.