Flatten article storage: one file per article, shared author media folder (AI)

Articles no longer live in their own per-post directory — each is now a
single `<slug>~<DDMMYYYY>~<tag1>+<tag2>.md` file directly inside its
author's directory, alongside that author's bio file and a shared
`media/` folder for images. This lets multiple articles reference the
same picture without duplicating it per article.

discovery.js: buildStub/resolveStub parse the article's own file name
instead of a wrapping directory's; indexing's per-author listing now
filters to `.md` files (excluding the bio file) rather than
subdirectories, and the author's own media/ folder is simply not a
`.md` file, so it's excluded for free. The article file's path is now
also its own cache key (the `root` concept is gone).

cache.js: bumped CACHE_SCHEMA_VERSION (cache key shape changed) and
renamed the object store from the now-inaccurate "slugDirs".

Migrated all existing test content (10,200 articles) to the new flat
layout; no code changes needed to the articles' own content.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXSVPS3UmmASiiprPfcnUd
This commit is contained in:
biosiasandClaude Sonnet 5 committed 2026-09-12 13:25:04 +02:00
1 parent 5c716a3f9b
commit 997768e2ae
4 files changed
+148 -128

No files matched your search

+63 -54
View File
@@ -4,18 +4,20 @@ A minimal CMS where articles are Markdown files rendered in the browser. No buil
## How it works
1. The browser traverses `cmjs/articles/` via a server directory index — discovering authors and their post directories; nginx JSON autoindex is preferred, with HTML autoindexes supported as a fallback.
2. Sorting, filtering, and publish status all come from the post directory's own *name*
(`slug~DDMMYYYY~tags`) — knowable straight from the listing that discovers it, with no
1. The browser traverses `cmjs/articles/` via a server directory index — discovering authors and
their article files; nginx JSON autoindex is preferred, with HTML autoindexes supported as a
fallback.
2. Sorting, filtering, and publish status all come from the article file's own *name*
(`slug~DDMMYYYY~tags.md`) — knowable straight from the listing that discovers it, with no
further request per article. A `.md` file's actual content (title, description, image) is
only fetched once that article is about to be shown — the current page's cards, a single
article, or a search — not for the whole site up front. See "Adding an article" and "Article
discovery & caching" below.
3. Clicking an article fetches the Markdown file and renders it with [marked.js](https://marked.js.org/).
4. Hash-based routing handles navigation — back/forward works without page reloads.
5. Once an article's content has been fetched, it's cached in IndexedDB, keyed by post directory
and invalidated by directory `mtime`, so repeat visits skip re-fetching unchanged content —
see "Article discovery & caching" below.
5. Once an article's content has been fetched, it's cached in IndexedDB, keyed by the article
file's own path and invalidated by that file's `mtime`, so repeat visits skip re-fetching
unchanged content — see "Article discovery & caching" below.
## Features
@@ -58,7 +60,7 @@ cmjs/
│ ├── frontmatter.js # parseFrontmatter() — pure YAML-front parser
│ ├── templates.js # loadTemplates(), templates cache
│ ├── discovery.js # discoverArticles(), fetchJson(), article cache
│ ├── cache.js # IndexedDB cache for discovered articles, keyed by post directory
│ ├── cache.js # IndexedDB cache for discovered articles, keyed by article file path
│ ├── concurrency.js # mapWithConcurrency() — caps simultaneous per-article fetches
│ ├── i18n.js # t(), loadStrings(), getLocale() — UI string lookup
│ ├── views.js # loadArticleList, loadArticle, loadAuthorView, loadCategoryView, loadSearch
@@ -76,24 +78,25 @@ cmjs/
└── articles/
└── author/ # One directory per author
├── author.md # Author profile (optional)
├── avatar.jpg # Optional — referenced by author.md's `avatar:` field
└── slug~DDMMYYYY~tag1+tag2/ # One directory per post — see "Adding an article"
├── slug.md # Article content + frontmatter
└── cover.jpg # Optional — referenced by slug.md's `image:` field
├── media/ # Shared images for this author — see "Adding an article"
│ ├── avatar.jpg # Referenced by author.md's `avatar:` field
│ └── cover.jpg # Referenced by one or more articles' `image:` field
└── slug~DDMMYYYY~tag1+tag2.md # One file per article — see "Adding an article"
```
## Adding an article
An article's directory *name* is where its publish date, tags, category, and draft status all
live — not in separate files, and not in frontmatter. The format is:
An article is a single Markdown file placed directly in its author's directory. Its *file name*
is where its publish date, tags, category, and draft status all live — not in separate files, and
not in frontmatter. The format is:
```
<slug>~<DDMMYYYY>~<tag1>+<tag2>+…
<slug>~<DDMMYYYY>~<tag1>+<tag2>+….md
```
- `<slug>` — the stable identifier used in the article's URL (`#post/author/slug`). Treat this
as permanent: retagging or correcting the date later just means renaming the directory (see
below), but changing the slug changes the article's URL.
as permanent: retagging or correcting the date later just means renaming the file (see below),
but changing the slug changes the article's URL.
- `<DDMMYYYY>` — the publish date, 8 digits, no separators (e.g. `16062026` for 16 June 2026).
- `<tag1>+<tag2>+…` — zero or more tag ids, joined with `+`. A tag id matching `expa-YYYY` is
treated as the article's "year"/category, shown as a badge linking to `#category/expa-YYYY`;
@@ -104,32 +107,35 @@ The separator is `~` (tilde), not a hyphen — deliberately, since a slug is exp
kebab-case (single hyphens only) and `~` isn't something you'd type by accident while writing
one, unlike an extra `-`.
A leading underscore on the whole directory name — `_<slug>~...` — marks the article as a
A leading underscore on the whole file name — `_<slug>~....md` — marks the article as a
**draft**: it's excluded from the list, search, and pagination, but still reachable directly by
URL. Remove the underscore to publish it.
Images (a cover photo, inline pictures) live in a `media/` folder shared by all of that author's
articles, not next to each individual article file — so the same picture can be reused across
several articles without keeping a duplicate copy per article. Reference one from frontmatter or
markdown with a path relative to the author directory, e.g. `image: media/photo.jpg`.
Steps:
1. Create the author directory if it doesn't exist: `cmjs/articles/author/`
2. Create the post directory using the naming convention above, e.g.
`cmjs/articles/author/my-article~16062026~kometa+expa-2026/`
3. Create the Markdown file inside, named after the slug: `my-article.md`
4. Optionally add a cover image beside it too, e.g. `photo.jpg`, and reference it from
frontmatter with `image: photo.jpg`. Images live next to the article that uses them, not in a
shared folder.
5. Add a frontmatter block at the top of the Markdown file
6. Write your content — it will appear in the list automatically
2. Create the Markdown file directly inside it, using the naming convention above, e.g.
`cmjs/articles/author/my-article~16062026~kometa+expa-2026.md`
3. Optionally add a cover image to that author's shared `cmjs/articles/author/media/` folder
(creating it if needed), and reference it from frontmatter with `image: media/photo.jpg`.
4. Add a frontmatter block at the top of the Markdown file
5. Write your content — it will appear in the list automatically
To change an article's date, tags, or draft status later, rename its directory — the article's
`slug.md` filename and content don't need to change. `author` is derived from the directory path
— no frontmatter field needed for it either. Because every one of these is encoded in the
directory name, discovering an article's metadata never requires fetching anything beyond the
directory listing that already reveals it — no further per-article request at all, which is what
makes the "Article discovery & caching" section below possible.
To change an article's date, tags, or draft status later, rename its file — its content doesn't
need to change. `author` is derived from the containing directory — no frontmatter field needed
for it either. Because every one of these is encoded in the file name, discovering an article's
metadata never requires fetching anything beyond the directory listing that already reveals it —
no further per-article request at all, which is what makes the "Article discovery & caching"
section below possible.
### Tag labels
A tag in the directory name is just an id (e.g. `kometa`) — the human-readable label shown on
A tag in the file name is just an id (e.g. `kometa`) — the human-readable label shown on
tag pills comes from `cmjs/blog-config/tags.json`:
```json
@@ -160,7 +166,7 @@ Your markdown content here...
|-------|----------|-------|
| `title` | yes | Shown in the list and article header |
| `description` | no | Short summary shown under the title in the list |
| `image` | no | Filename of an image file placed in this same post directory (e.g. `photo.jpg`); shown as card thumbnail and article hero |
| `image` | no | Path to an image in the author's shared `media/` folder (e.g. `media/photo.jpg`); shown as card thumbnail and article hero |
Reading time is computed automatically from the article word count — no frontmatter field
needed. `date`, `tags`, and `draft` frontmatter fields, if present, are ignored — superseded by
@@ -195,7 +201,7 @@ Short bio rendered as markdown.
| Field | Notes |
|-------|-------|
| `title` | Display name shown in the profile card. Falls back to the author directory name. |
| `avatar` | Filename of an image file placed in this same author directory — displayed as a circular thumbnail. |
| `avatar` | Path to an image, relative to the author directory — e.g. `media/avatar.jpg` if kept in the shared media folder alongside article images, or a bare filename if kept directly beside `{author}.md`. Displayed as a circular thumbnail. |
| `github` / `twitter` / `linkedin` / `mastodon` / `website` | Any subset of these renders as pill links below the bio. |
The body text below the frontmatter is rendered as markdown and shown as the bio.
@@ -214,7 +220,7 @@ The body text below the frontmatter is rendered as markdown and shown as the bio
| `#category/expa-2026/page/2` | Same, page 2 |
| `#tag/kometa` | All articles tagged "kometa", page 1 |
| `#tag/kometa/page/2` | Same, page 2 |
| `#post/alice/my-article` | Single article at `cmjs/articles/alice/my-article/my-article.md` |
| `#post/alice/my-article` | Single article at `cmjs/articles/alice/my-article~DDMMYYYY~tags.md` |
Author names and category tags on each card and article are clickable links to their respective filter views. A search box in the header navigates to `#search/query`.
@@ -390,8 +396,10 @@ not built into stock nginx — only worth it if you're already building custom n
## Article discovery & caching
Discovery happens in two phases. **Indexing** crawls `cmjs/articles/` client-side to build a
lightweight "stub" for every article — the author directories, then each author's post
directories along with their `mtime`. That's it: because a post directory's *name* already
lightweight "stub" for every article — the author directories, then each author's own directory
listing (their article files, bio file, and shared `media/` folder together) along with each
file's `mtime`. Non-article entries (the author's own `{author}.md`, the `media/` folder) are
simply filtered out of that same listing. That's it: because an article file's *name* already
encodes its slug, date, tags, and draft status (see "Adding an article" above), that single
listing is enough to sort, filter, paginate, and populate the sidebar's tag/year counts, without
any further request per article and without ever fetching an article's actual Markdown.
@@ -403,20 +411,20 @@ viewed actually needs. A large site's list is therefore fast to show regardless
articles it has — indexing costs one request per author, full stop, and only ~20 articles' worth
of content is ever fetched per page view.
- **Cache** — hydrated articles are cached per post directory in IndexedDB (`cmjs/js/cache.js`),
keyed by directory path and invalidated by comparing the live `mtime` against the cached one —
once *any* visit renders an article, every later visit skips its content fetch too, until its
directory is renamed (a rename changes the cache key, so a retagged/redated article is treated
as new — see "Adding an article" above) or its `.md` file's own mtime changes. Indexing reads
the entire cache once up front (`getAllCacheEntries()`, kicked off in parallel with the
author/post-directory requests) so a warm cache doesn't cost an IndexedDB read per article. If
IndexedDB is unavailable (private browsing, disabled storage), discovery transparently falls
back to fetching everything live on every visit.
- **Cache** — hydrated articles are cached per article file in IndexedDB (`cmjs/js/cache.js`),
keyed by the file's own path and invalidated by comparing its live `mtime` against the cached
one — once *any* visit renders an article, every later visit skips its content fetch too, until
the file is renamed (a rename changes the cache key, so a retagged/redated article is treated as
new — see "Adding an article" above) or its own mtime changes. Indexing reads the entire cache
once up front (`getAllCacheEntries()`, kicked off in parallel with the author/directory-listing
requests) so a warm cache doesn't cost an IndexedDB read per article. If IndexedDB is unavailable
(private browsing, disabled storage), discovery transparently falls back to fetching everything
live on every visit.
- **Cache schema version** — the cached records store the full derived article shape, not raw
content, so a code change to how articles are derived (in `cmjs/js/discovery.js`) can make old
cached entries stale in *structure*, not just content — and a directory's `mtime` won't catch
that. Bump `CACHE_SCHEMA_VERSION` in `cmjs/js/cache.js` whenever that shape changes; it's baked
into the IndexedDB database name, so a bump invalidates every cached entry on the next deploy.
cached entries stale in *structure*, not just content — and a file's `mtime` won't catch that.
Bump `CACHE_SCHEMA_VERSION` in `cmjs/js/cache.js` whenever that shape changes; it's baked into
the IndexedDB database name, so a bump invalidates every cached entry on the next deploy.
- **Concurrency** — `MAX_CONCURRENT_REQUESTS` in `cmjs/js/discovery.js` (default `10`) caps how
many hydration (content) fetches run at once — the only per-article requests left, since
indexing no longer needs one; raise it for a fast/local nginx, lower it for a slow or shared/
@@ -435,17 +443,18 @@ of content is ever fetched per page view.
page count is exact once the whole set's been scanned, and a lower bound (at least one more
page) until then.
- **Crawl health** — if every author's directory listing fails outright, or very few of the
known post directories parsed successfully (a malformed directory name, or a total breakage
like the wrong origin or CORS misconfigured), a small warning banner is shown above the article
list instead of silently rendering an empty page (see `getCrawlStats()` /
known candidate files parsed successfully (a malformed file name, or a total breakage like the
wrong origin or CORS misconfigured), a small warning banner is shown above the article list
instead of silently rendering an empty page (see `getCrawlStats()` /
`renderCrawlHealthWarning()` in `cmjs/js/discovery.js` / `cmjs/js/views.js`).
- **Design rationale** — the original crawler design (wave-based crawling with a per-article
listing fetch, recency prioritisation, HTML-autoindex fallback, long-term "deploy and forget"
constraints) is documented in
[`.claude/nginx-sitemap-crawler-plan.md`](.claude/nginx-sitemap-crawler-plan.md), which ships
alongside the code. The per-article listing fetch it describes no longer exists (metadata moved
into the directory name instead), but the rest of that reasoning — the caching model, the
longevity constraints, the HTML fallback — still applies.
into the file name instead, and articles no longer live in a directory of their own), but the
rest of that reasoning — the caching model, the longevity constraints, the HTML fallback — still
applies.
## Running locally
+14 -13
View File
@@ -1,18 +1,19 @@
// Persistent cache for discovered articles, keyed by slug-directory path.
// Raw IndexedDB (no wrapper library — see .claude/nginx-sitemap-crawler-plan.md
// L4/L5 on the zero-dependency, long-stable-API requirement). If IndexedDB is
// unavailable (private browsing, old browser, blocked), every operation below
// silently becomes a no-op so discovery.js gets live-fetch behaviour for free
// without needing its own availability branching (plan doc R6).
// Persistent cache for discovered articles, keyed by each article file's own
// path. Raw IndexedDB (no wrapper library — see
// .claude/nginx-sitemap-crawler-plan.md L4/L5 on the zero-dependency,
// long-stable-API requirement). If IndexedDB is unavailable (private
// browsing, old browser, blocked), every operation below silently becomes a
// no-op so discovery.js gets live-fetch behaviour for free without needing
// its own availability branching (plan doc R6).
// Bump this whenever discovery.js's derived article record shape changes.
// The version is baked into the DB name so a code deploy invalidates every
// cached entry immediately, instead of stale-shaped records lingering until
// each directory's mtime happens to change — which, on a "deploy and
// forget" site, could be years.
const CACHE_SCHEMA_VERSION = 2;
// each file's mtime happens to change — which, on a "deploy and forget"
// site, could be years.
const CACHE_SCHEMA_VERSION = 3;
const DB_NAME = `cmjs-cache-v${CACHE_SCHEMA_VERSION}`;
const STORE_NAME = 'slugDirs';
const STORE_NAME = 'articleFiles';
let cacheAvailable = true;
let dbPromise = null;
@@ -58,9 +59,9 @@ export function isCacheAvailable() {
// Returns every cached { path, mtime, article } record as a Map keyed by
// path — one bulk read instead of a separate IndexedDB round-trip per
// directory, which matters once there are hundreds of them: discovery.js
// reads this once up front rather than doing a point lookup per directory
// during indexing. Empty Map on a miss or if the cache is unavailable.
// article file, which matters once there are hundreds of them: discovery.js
// reads this once up front rather than doing a point lookup per file during
// indexing. Empty Map on a miss or if the cache is unavailable.
export async function getAllCacheEntries() {
const records = await runRequest('readonly', store => store.getAll());
return new Map((records || []).map(record => [record.path, record]));
+70 -60
View File
@@ -7,10 +7,10 @@ import { t } from './i18n.js';
// Caps simultaneous markdown content fetches during hydration (see
// hydrateArticle()/hydrateArticles()) — the only per-article requests left,
// since indexing no longer needs a per-article listing fetch at all (every
// stub field is encoded in its directory's own name — see buildStub()). Raise
// it for a fast/local nginx, lower it for a slow or shared/production
// instance — see .claude/nginx-sitemap-crawler-plan.md §2a for the original
// rationale (written when this also governed indexing requests).
// stub field is encoded in its file's own name — see buildStub()). Raise it
// for a fast/local nginx, lower it for a slow or shared/production instance
// — see .claude/nginx-sitemap-crawler-plan.md §2a for the original rationale
// (written when this also governed indexing requests).
const MAX_CONCURRENT_REQUESTS = 10;
const MONTH_ABBREVIATIONS = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];
@@ -58,11 +58,14 @@ export async function fetchJson(url) {
throw new Error(`Expected a directory listing at ${url}`);
}
// Images live alongside whatever references them — an article's cover/inline
// images sit in its own slug directory, an author's avatar in their own
// author directory — so `articlePath` (the referencing .md file's own URL)
// is the resolution base, not a shared folder. A bare filename in
// frontmatter therefore always means "next to this file."
// Images live alongside whatever references them: every article file sits
// directly in its author's own directory (see buildStub()), next to that
// author's shared `media/` folder — so `articlePath` (the referencing .md
// file's own URL) is the resolution base. A relative path in frontmatter
// like `media/cover.jpg` resolves against the article file's own directory,
// which is the author directory itself, landing it in that shared folder —
// letting multiple articles reference the same image without duplicating it
// per article.
export function resolveMediaUrl(value, articlePath = '') {
if (!value) return '';
let path = String(value).trim();
@@ -129,8 +132,8 @@ function notifyProgress(done) {
progressListeners.forEach(listener => listener(snapshot, done));
}
// How many of the last crawl's slug directories actually produced an article
// vs. how many were candidates, plus how many authors' listings failed
// How many of the last crawl's candidate files actually produced an article
// vs. how many were listed, plus how many authors' listings failed
// outright — lets the UI show a "something's badly wrong" indicator instead
// of silently rendering an empty/near-empty list forever.
export function getCrawlStats() {
@@ -149,19 +152,21 @@ function excerpt(content) {
return `${text.slice(0, 277).replace(/\s+\S*$/, '')}…`;
}
// A post directory's own name is the sole authoritative source of its
// slug, publish date, tags/category, and draft status — see the README's
// "Adding an article" section. Format: `<slug>~<DDMMYYYY>[~tag1+tag2+…]`,
// An article file's own name is the sole authoritative source of its slug,
// publish date, tags/category, and draft status — see the README's "Adding
// an article" section. Format: `<slug>~<DDMMYYYY>[~tag1+tag2+…].md`,
// optionally prefixed with `_` for draft. `~` (rather than e.g. "--") is
// deliberate: a slug is expected to be plain kebab-case, and "~" isn't a
// character anyone would type by accident while writing one — unlike a
// double-hyphen separator, which collides with an easy slug typo (an extra
// "-") and silently drops the whole article instead of erroring. Returns
// null for any directory name that doesn't match — callers treat that as a
// silent skip, e.g. a stray directory that isn't a post at all.
function parseSlugDirName(name) {
const draft = name.startsWith('_');
const core = draft ? name.slice(1) : name;
// null for any file name that doesn't match — callers treat that as a
// silent skip, e.g. the author's own bio file or a stray unrelated file.
function parseArticleFileName(fileName) {
if (!fileName.endsWith('.md')) return null;
const base = fileName.slice(0, -3);
const draft = base.startsWith('_');
const core = draft ? base.slice(1) : base;
const [slug, dateSegment, tagsSegment] = core.split('~');
if (!slug || !dateSegment || !/^\d{8}$/.test(dateSegment)) return null;
const day = dateSegment.slice(0, 2);
@@ -176,18 +181,21 @@ function parseSlugDirName(name) {
}
// Everything an article card/filter/sort needs, parsed entirely from its
// directory's own name and its listing entry's mtime — no fetch of any
// kind. This is what runs for every article at index time; the (expensive)
// file's own name and its listing entry's mtime — no fetch of any kind.
// This is what runs for every article at index time; the (expensive)
// content fetch is deferred to hydrateArticle(), which only ever runs for
// articles actually being rendered. Returns null if the directory name
// doesn't match the naming convention (see parseSlugDirName) — the caller
// articles actually being rendered. Returns null if the file name doesn't
// match the naming convention (see parseArticleFileName) — the caller
// treats that as a silent skip, matching the rest of this module's
// approach.
function buildStub(author, dirName, mtime, authorRoot) {
const parsed = parseSlugDirName(dirName);
// approach. `path` (the article file's own URL) doubles as its cache key —
// see cache.js — since every article now lives directly in its author's
// directory, alongside that author's bio file and shared `media/` folder,
// rather than in a directory of its own.
function buildStub(author, fileName, mtime, authorRoot) {
const parsed = parseArticleFileName(fileName);
if (!parsed) return null;
const { slug, date, tags, draft } = parsed;
const root = `${authorRoot}${encodeURIComponent(dirName)}/`;
const path = `${authorRoot}${encodeURIComponent(fileName)}`;
// "Category" isn't a separate concept anymore — it's just whichever tag
// id (if any) looks like an expedition year.
const category = tags.find(id => /^expa-20\d{2}$/i.test(id)) || null;
@@ -196,9 +204,8 @@ function buildStub(author, dirName, mtime, authorRoot) {
author,
slug,
hash: `${author}/${slug}`,
root,
path,
mtime,
path: `${root}${encodeURIComponent(slug)}.md`,
category,
expa: category ? Number(category.replace(/^expa-/i, '')) : null,
date,
@@ -208,19 +215,19 @@ function buildStub(author, dirName, mtime, authorRoot) {
};
}
// Cache-aware: a directory whose mtime matches its cached entry's mtime
// reuses that entry directly — worthwhile only when it's already hydrated
// Cache-aware: a file whose mtime matches its cached entry's mtime reuses
// that entry directly — worthwhile only when it's already hydrated
// (skipping a real content re-fetch); a merely-parsed stub is free to
// rebuild, so there's no point caching one just to skip re-parsing a
// string. `cacheSnapshot` is the whole cache read once up front (see
// runCrawl) — an in-memory lookup here instead of a separate IndexedDB read
// per directory, which mattered when this also had to guard a listing
// fetch; kept anyway since it's still the same one cheap lookup.
function resolveStub(author, dirName, mtime, authorRoot, cacheSnapshot) {
const root = `${authorRoot}${encodeURIComponent(dirName)}/`;
const cached = cacheSnapshot.get(root);
// per file, which mattered when this also had to guard a listing fetch;
// kept anyway since it's still the same one cheap lookup.
function resolveStub(author, fileName, mtime, authorRoot, cacheSnapshot) {
const path = `${authorRoot}${encodeURIComponent(fileName)}`;
const cached = cacheSnapshot.get(path);
if (cached && mtime != null && cached.mtime === mtime && cached.article?.hydrated) return cached.article;
return buildStub(author, dirName, mtime, authorRoot);
return buildStub(author, fileName, mtime, authorRoot);
}
// Fields only derivable from an article's actual markdown content (not its
@@ -261,7 +268,7 @@ export async function hydrateArticle(stub) {
// already reflect this same mutation.)
const authoritative = { ...stub };
Object.assign(stub, meta, authoritative, deriveContentFields(meta, content, stub.path), { hydrated: true });
await setCacheEntry(stub.root, { mtime: stub.mtime, article: stub });
await setCacheEntry(stub.path, { mtime: stub.mtime, article: stub });
} catch { /* leave stub unhydrated — retried next time it's rendered */ }
return stub;
}
@@ -289,49 +296,52 @@ async function runCrawl() {
const authors = authorEntries.filter(entry => entry.type === 'directory' && !entry.name.startsWith('.'));
// Wave 2 — always fetched live, in full, for every author (not just the
// current page): each author's post directories, with their mtimes.
// This is what builds the flat, cross-author list that drives global
// sorting and pagination — see the plan doc's "keep the flat feed"
// decision. It's also now the *only* per-directory network request
// current page): each author's own directory listing, with per-file
// mtimes. This is what builds the flat, cross-author list that drives
// global sorting and pagination — see the plan doc's "keep the flat
// feed" decision. It's also the *only* per-author network request
// indexing ever needs: every stub field (slug, date, tags, category,
// draft status) is parsed straight out of each directory's own name
// (see buildStub/parseSlugDirName) — there's no more separate
// per-article listing fetch the way there used to be. One author's
// listing failing (network blip, deleted mid-crawl) is silently
// skipped rather than failing every other author's list too, but is
// still counted, so a *total* breakage (wrong origin, CORS) remains
// visible via getCrawlStats() instead of just silently producing an
// empty article list.
// draft status) is parsed straight out of each article file's own name
// (see buildStub/parseArticleFileName) — there's no per-article fetch
// at index time. The author's own bio file (`<author>.md`) and shared
// `media/` folder live in the same listing and are simply filtered out
// here — only `.md` files other than the bio file are candidate
// articles. One author's listing failing (network blip, deleted
// mid-crawl) is silently skipped rather than failing every other
// author's list too, but is still counted, so a *total* breakage (wrong
// origin, CORS) remains visible via getCrawlStats() instead of just
// silently producing an empty article list.
let authorsFailed = 0;
const authorDirLists = await Promise.all(authors.map(async author => {
const authorFileLists = await Promise.all(authors.map(async author => {
const authorRoot = `${BASE_URL}articles/${encodeURIComponent(author.name)}/`;
const bioFileName = `${author.name}.md`;
try {
const entries = await fetchJson(authorRoot);
return entries
.filter(entry => entry.type === 'directory' && entry.name !== '_history')
.map(entry => ({ author: author.name, dirName: entry.name, mtime: entry.mtime ?? null, authorRoot }));
.filter(entry => entry.type === 'file' && entry.name.endsWith('.md') && entry.name !== bioFileName)
.map(entry => ({ author: author.name, fileName: entry.name, mtime: entry.mtime ?? null, authorRoot }));
} catch {
authorsFailed += 1;
return [];
}
}));
const allDirs = authorDirLists.flat();
const allFiles = authorFileLists.flat();
// Every stub is parsed synchronously from data already in hand — no
// per-directory fetch, so no need for mapWithConcurrency here (that's
// still used by hydrateArticles(), which does real network fetches).
// per-file fetch, so no need for mapWithConcurrency here (that's still
// used by hydrateArticles(), which does real network fetches).
const cacheSnapshot = await cacheSnapshotPromise;
let loaded = 0;
for (const dir of allDirs) {
const stub = resolveStub(dir.author, dir.dirName, dir.mtime, dir.authorRoot, cacheSnapshot);
for (const file of allFiles) {
const stub = resolveStub(file.author, file.fileName, file.mtime, file.authorRoot, cacheSnapshot);
if (stub) {
loaded += 1;
partialResolved.push(stub);
}
}
await pruneCache(allDirs.map(dir => `${dir.authorRoot}${encodeURIComponent(dir.dirName)}/`));
crawlStats = { total: allDirs.length, loaded, authorsTotal: authors.length, authorsFailed };
await pruneCache(allFiles.map(file => `${file.authorRoot}${encodeURIComponent(file.fileName)}`));
crawlStats = { total: allFiles.length, loaded, authorsTotal: authors.length, authorsFailed };
const finalArticles = sortAndFilterPublished(partialResolved);
notifyProgress(true);
+1 -1
View File
@@ -494,7 +494,7 @@ export async function loadArticle(hash) {
const element = document.importNode(templates.article, true);
const field = name => element.querySelector(`[data-field="${name}"]`);
field('title').textContent = meta.title || article.slug;
// Publish date comes from the post directory's own name (already
// Publish date comes from the article file's own name (already
// resolved onto `article.date`), not the raw frontmatter.
setField(field('date'), article.date, (time, value) => { time.dateTime = String(value); time.textContent = formatDate(value); });
setField(field('description'), meta.description, (description, value) => { description.textContent = value; });